2023-11-16T16:16:26.408 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-16T16:16:26.408 DEBUG:teuthology.run:Teuthology command: teuthology --name choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi --verbose --owner scheduled_choffman@teuthology --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} --archive /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920 -- /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920/orig.config.yaml 2023-11-16T16:16:26.465 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-16T16:16:26.543 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920 branch: wip-choffman-62793 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} email: null first_in_suite: false job_id: '7460920' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-choffman-62793 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 5e4a743ad7fd1422ebd6c4dc950a21a303279207 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 5e4a743ad7fd1422ebd6c4dc950a21a303279207 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-choffman-62793 sha1: 5e4a743ad7fd1422ebd6c4dc950a21a303279207 owner: scheduled_choffman@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4433 sha1: 5e4a743ad7fd1422ebd6c4dc950a21a303279207 sleep_before_teardown: 0 suite: fs:workload suite_branch: wip-choffman-62793 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 5e4a743ad7fd1422ebd6c4dc950a21a303279207 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi044.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - fs/misc teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-16_16:09:17 tube: smithi user: choffman verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1224587 2023-11-16T16:16:26.543 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa; will attempt to use it 2023-11-16T16:16:26.544 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa/tasks 2023-11-16T16:16:26.544 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-16T16:16:26.545 INFO:teuthology.task.internal:Checking packages... 2023-11-16T16:16:26.567 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '5e4a743ad7fd1422ebd6c4dc950a21a303279207' 2023-11-16T16:16:26.568 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-16T16:16:26.568 INFO:teuthology.packaging:ref: None 2023-11-16T16:16:26.568 INFO:teuthology.packaging:tag: None 2023-11-16T16:16:26.568 INFO:teuthology.packaging:branch: wip-choffman-62793 2023-11-16T16:16:26.568 INFO:teuthology.packaging:sha1: 5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:16:26.568 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-choffman-62793 2023-11-16T16:16:26.801 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-3905.g5e4a743a 2023-11-16T16:16:26.801 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-16T16:16:26.807 INFO:teuthology.task.internal:no buildpackages task found 2023-11-16T16:16:26.807 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-16T16:16:26.815 INFO:teuthology.task.internal:Saving configuration 2023-11-16T16:16:26.835 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-16T16:16:26.840 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T16:16:26.864 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 16:10:26.568556', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T16:16:26.887 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 16:10:26.570346', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T16:16:26.912 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 16:10:26.571771', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T16:16:26.912 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-16T16:16:26.918 INFO:teuthology.task.internal:roles: ubuntu@smithi044.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-11-16T16:16:26.918 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-11-16T16:16:26.918 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-11-16T16:16:26.918 INFO:teuthology.run_tasks:Running task console_log... 2023-11-16T16:16:27.020 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd41645cb80>, signals=[15]) 2023-11-16T16:16:27.020 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-16T16:16:27.026 INFO:teuthology.task.internal:Opening connections... 2023-11-16T16:16:27.026 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi044.front.sepia.ceph.com 2023-11-16T16:16:27.028 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:16:27.096 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2023-11-16T16:16:27.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:16:27.164 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2023-11-16T16:16:27.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:16:27.231 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-16T16:16:27.237 DEBUG:teuthology.orchestra.run.smithi044:> uname -m 2023-11-16T16:16:27.260 INFO:teuthology.orchestra.run.smithi044.stdout:x86_64 2023-11-16T16:16:27.260 DEBUG:teuthology.orchestra.run.smithi044:> cat /etc/os-release 2023-11-16T16:16:27.319 INFO:teuthology.orchestra.run.smithi044.stdout:NAME="Red Hat Enterprise Linux" 2023-11-16T16:16:27.319 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION="8.6 (Ootpa)" 2023-11-16T16:16:27.319 INFO:teuthology.orchestra.run.smithi044.stdout:ID="rhel" 2023-11-16T16:16:27.319 INFO:teuthology.orchestra.run.smithi044.stdout:ID_LIKE="fedora" 2023-11-16T16:16:27.319 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION_ID="8.6" 2023-11-16T16:16:27.319 INFO:teuthology.orchestra.run.smithi044.stdout:PLATFORM_ID="platform:el8" 2023-11-16T16:16:27.320 INFO:teuthology.orchestra.run.smithi044.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-16T16:16:27.320 INFO:teuthology.orchestra.run.smithi044.stdout:ANSI_COLOR="0;31" 2023-11-16T16:16:27.320 INFO:teuthology.orchestra.run.smithi044.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-16T16:16:27.320 INFO:teuthology.orchestra.run.smithi044.stdout:HOME_URL="https://www.redhat.com/" 2023-11-16T16:16:27.320 INFO:teuthology.orchestra.run.smithi044.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-16T16:16:27.320 INFO:teuthology.orchestra.run.smithi044.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-16T16:16:27.320 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:16:27.320 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-16T16:16:27.320 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-16T16:16:27.320 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-16T16:16:27.320 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-16T16:16:27.321 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server 2023-11-16T16:16:27.340 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2023-11-16T16:16:27.360 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2023-11-16T16:16:27.360 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2023-11-16T16:16:27.420 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="Red Hat Enterprise Linux" 2023-11-16T16:16:27.420 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="8.6 (Ootpa)" 2023-11-16T16:16:27.420 INFO:teuthology.orchestra.run.smithi077.stdout:ID="rhel" 2023-11-16T16:16:27.420 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="fedora" 2023-11-16T16:16:27.420 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="8.6" 2023-11-16T16:16:27.420 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el8" 2023-11-16T16:16:27.420 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-16T16:16:27.420 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2023-11-16T16:16:27.420 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-16T16:16:27.420 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://www.redhat.com/" 2023-11-16T16:16:27.421 INFO:teuthology.orchestra.run.smithi077.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-16T16:16:27.421 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-16T16:16:27.421 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-16T16:16:27.421 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-16T16:16:27.421 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-16T16:16:27.421 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-16T16:16:27.421 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-16T16:16:27.421 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2023-11-16T16:16:27.448 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2023-11-16T16:16:27.468 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2023-11-16T16:16:27.468 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2023-11-16T16:16:27.528 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="Red Hat Enterprise Linux" 2023-11-16T16:16:27.528 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="8.6 (Ootpa)" 2023-11-16T16:16:27.528 INFO:teuthology.orchestra.run.smithi176.stdout:ID="rhel" 2023-11-16T16:16:27.529 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="fedora" 2023-11-16T16:16:27.529 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="8.6" 2023-11-16T16:16:27.529 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el8" 2023-11-16T16:16:27.529 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-16T16:16:27.529 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2023-11-16T16:16:27.529 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-16T16:16:27.529 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://www.redhat.com/" 2023-11-16T16:16:27.529 INFO:teuthology.orchestra.run.smithi176.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-16T16:16:27.530 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-16T16:16:27.530 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-11-16T16:16:27.530 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-16T16:16:27.530 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-16T16:16:27.530 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-16T16:16:27.530 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-16T16:16:27.531 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2023-11-16T16:16:27.548 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-16T16:16:27.559 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-16T16:16:27.565 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-16T16:16:27.565 DEBUG:teuthology.orchestra.run.smithi044:> test '!' -e /home/ubuntu/cephtest 2023-11-16T16:16:27.567 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2023-11-16T16:16:27.569 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2023-11-16T16:16:27.587 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-16T16:16:27.594 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-16T16:16:27.594 DEBUG:teuthology.orchestra.run.smithi044:> test -z $(ls -A /var/lib/ceph) 2023-11-16T16:16:27.627 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2023-11-16T16:16:27.629 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2023-11-16T16:16:27.656 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-16T16:16:27.735 INFO:teuthology.run_tasks:Running task kernel... 2023-11-16T16:16:27.749 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-11-16T16:16:27.749 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-11-16T16:16:27.750 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-11-16T16:16:27.750 DEBUG:teuthology.orchestra.run.smithi044:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T16:16:27.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T16:16:27.770 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2023-11-16T16:16:27.925 DEBUG:teuthology.task.kernel:sha1 for client.0 is 5b73517e0e2fa132ea79290e5fc06ea5349b2288 2023-11-16T16:16:27.926 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5b73517e0e2fa132ea79290e5fc06ea5349b2288"... 2023-11-16T16:16:27.926 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2023-11-16T16:16:27.946 INFO:teuthology.orchestra.run.smithi044.stdout:4.18.0-372.9.1.el8.x86_64 2023-11-16T16:16:27.946 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-372.9.1.el8.x86_64 vs 5b73517e0e2fa132ea79290e5fc06ea5349b2288 2023-11-16T16:16:27.946 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-11-16T16:16:27.946 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum upgrade -y linux-firmware 2023-11-16T16:16:29.151 INFO:teuthology.orchestra.run.smithi044.stdout:Updating Subscription Management repositories. 2023-11-16T16:16:29.151 INFO:teuthology.orchestra.run.smithi044.stdout:Unable to read consumer identity 2023-11-16T16:16:30.549 INFO:teuthology.orchestra.run.smithi044.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-11-16T16:16:31.333 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 21 MB/s | 16 MB 00:00 2023-11-16T16:16:35.083 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 459 kB/s | 24 kB 00:00 2023-11-16T16:16:35.882 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2023-11-16T16:16:35.883 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2023-11-16T16:16:35.883 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2023-11-16T16:16:36.024 INFO:teuthology.task.kernel:Downloading kernel 5b73517e0e2fa132ea79290e5fc06ea5349b2288 on client.0... 2023-11-16T16:16:36.024 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5b73517e0e2fa132ea79290e5fc06ea5349b2288 2023-11-16T16:16:36.167 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5b73517e0e2fa132ea79290e5fc06ea5349b2288/centos/8/flavors/default/x86_64/ 2023-11-16T16:16:36.167 DEBUG:teuthology.orchestra.run.smithi044:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_rc5_g5b73517e0e2f-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5b73517e0e2fa132ea79290e5fc06ea5349b2288/centos/8/flavors/default/x86_64/ --input-file=- 2023-11-16T16:16:36.169 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi044.front.sepia.ceph.com to complete... 2023-11-16T16:17:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16 16:17:03 URL:https://4.chacra.ceph.com/r/kernel/testing/5b73517e0e2fa132ea79290e5fc06ea5349b2288/centos/8/flavors/default/x86_64/kernel-6.6.0_rc5_g5b73517e0e2f-1.x86_64.rpm [383129572/383129572] -> "/tmp/kernel.x86_64.rpm" [1] 2023-11-16T16:17:03.853 INFO:teuthology.orchestra.run.smithi044.stderr:FINISHED --2023-11-16 16:17:03-- 2023-11-16T16:17:03.853 INFO:teuthology.orchestra.run.smithi044.stderr:Total wall clock time: 28s 2023-11-16T16:17:03.853 INFO:teuthology.orchestra.run.smithi044.stderr:Downloaded: 1 files, 365M in 27s (13.5 MB/s) 2023-11-16T16:17:03.854 INFO:teuthology.task.kernel:Installing kernel 5b73517e0e2fa132ea79290e5fc06ea5349b2288 on client.0... 2023-11-16T16:17:03.854 DEBUG:teuthology.orchestra.run.smithi044:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-11-16T16:17:05.527 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying... ######################################## 2023-11-16T16:17:05.663 INFO:teuthology.orchestra.run.smithi044.stdout:Preparing... ######################################## 2023-11-16T16:17:05.759 INFO:teuthology.orchestra.run.smithi044.stdout:Updating / installing... 2023-11-16T16:17:46.936 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-6.6.0_rc5_g5b73517e0e2f-1 ######################################## 2023-11-16T16:17:50.894 INFO:teuthology.orchestra.run.smithi044.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-11-16T16:18:16.111 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi044.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-11-16T16:18:16.112 DEBUG:teuthology.orchestra.run.smithi044:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-11-16T16:18:16.170 INFO:teuthology.orchestra.run.smithi044.stdout:/boot/System.map-6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:16.170 INFO:teuthology.orchestra.run.smithi044.stdout:/boot/config-6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:16.170 INFO:teuthology.orchestra.run.smithi044.stdout:/boot/vmlinuz-6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:16.170 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:16.170 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel 2023-11-16T16:18:16.171 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch 2023-11-16T16:18:16.171 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86 2023-11-16T16:18:16.171 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto 2023-11-16T16:18:16.171 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-11-16T16:18:16.171 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-11-16T16:18:16.172 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-11-16T16:18:16.172 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-11-16T16:18:16.172 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-11-16T16:18:16.172 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-11-16T16:18:16.172 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-11-16T16:18:16.172 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/crc32c-intel.ko 2023-11-16T16:18:16.173 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-11-16T16:18:16.173 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-11-16T16:18:16.173 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-11-16T16:18:16.173 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-11-16T16:18:16.173 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/serpent-avx2.ko 2023-11-16T16:18:16.174 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-11-16T16:18:16.174 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-11-16T16:18:16.174 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-11-16T16:18:16.174 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-11-16T16:18:16.174 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-11-16T16:18:16.174 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-11-16T16:18:16.174 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-11-16T16:18:16.175 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kernel 2023-11-16T16:18:16.175 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kernel/cpu 2023-11-16T16:18:16.175 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kernel/cpu/mce 2023-11-16T16:18:16.175 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-11-16T16:18:16.175 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kvm 2023-11-16T16:18:16.175 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kvm/kvm-amd.ko 2023-11-16T16:18:16.175 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kvm/kvm-intel.ko 2023-11-16T16:18:16.175 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kvm/kvm.ko 2023-11-16T16:18:16.176 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto 2023-11-16T16:18:16.176 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/ansi_cprng.ko 2023-11-16T16:18:16.176 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/anubis.ko 2023-11-16T16:18:16.176 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/arc4.ko 2023-11-16T16:18:16.176 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx 2023-11-16T16:18:16.176 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/async_memcpy.ko 2023-11-16T16:18:16.176 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/async_pq.ko 2023-11-16T16:18:16.177 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/async_raid6_recov.ko 2023-11-16T16:18:16.177 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/async_tx.ko 2023-11-16T16:18:16.177 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/async_xor.ko 2023-11-16T16:18:16.177 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/raid6test.ko 2023-11-16T16:18:16.177 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/authenc.ko 2023-11-16T16:18:16.177 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/authencesn.ko 2023-11-16T16:18:16.177 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/blake2b_generic.ko 2023-11-16T16:18:16.178 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/blowfish_common.ko 2023-11-16T16:18:16.178 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/blowfish_generic.ko 2023-11-16T16:18:16.178 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/camellia_generic.ko 2023-11-16T16:18:16.178 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/cast5_generic.ko 2023-11-16T16:18:16.178 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/cast6_generic.ko 2023-11-16T16:18:16.178 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/cast_common.ko 2023-11-16T16:18:16.178 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/ccm.ko 2023-11-16T16:18:16.178 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/cmac.ko 2023-11-16T16:18:16.178 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/crc32_generic.ko 2023-11-16T16:18:16.178 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/crypto_engine.ko 2023-11-16T16:18:16.178 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/crypto_user.ko 2023-11-16T16:18:16.178 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/cts.ko 2023-11-16T16:18:16.178 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/des_generic.ko 2023-11-16T16:18:16.179 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/dh_generic.ko 2023-11-16T16:18:16.179 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/echainiv.ko 2023-11-16T16:18:16.179 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/essiv.ko 2023-11-16T16:18:16.179 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/fcrypt.ko 2023-11-16T16:18:16.179 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/khazad.ko 2023-11-16T16:18:16.179 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/lz4.ko 2023-11-16T16:18:16.179 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/lz4hc.ko 2023-11-16T16:18:16.179 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/md4.ko 2023-11-16T16:18:16.179 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/michael_mic.ko 2023-11-16T16:18:16.179 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/pcbc.ko 2023-11-16T16:18:16.179 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/pcrypt.ko 2023-11-16T16:18:16.179 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/rmd160.ko 2023-11-16T16:18:16.179 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/seed.ko 2023-11-16T16:18:16.180 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/serpent_generic.ko 2023-11-16T16:18:16.180 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/tcrypt.ko 2023-11-16T16:18:16.180 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/tea.ko 2023-11-16T16:18:16.180 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/twofish_common.ko 2023-11-16T16:18:16.180 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/twofish_generic.ko 2023-11-16T16:18:16.180 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/vmac.ko 2023-11-16T16:18:16.180 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/wp512.ko 2023-11-16T16:18:16.180 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/xcbc.ko 2023-11-16T16:18:16.180 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/xor.ko 2023-11-16T16:18:16.180 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/xxhash_generic.ko 2023-11-16T16:18:16.180 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers 2023-11-16T16:18:16.180 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi 2023-11-16T16:18:16.180 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/acpi_ipmi.ko 2023-11-16T16:18:16.181 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/acpi_pad.ko 2023-11-16T16:18:16.181 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/custom_method.ko 2023-11-16T16:18:16.181 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/ec_sys.ko 2023-11-16T16:18:16.181 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/platform_profile.ko 2023-11-16T16:18:16.181 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/sbs.ko 2023-11-16T16:18:16.181 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/sbshc.ko 2023-11-16T16:18:16.181 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/video.ko 2023-11-16T16:18:16.181 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata 2023-11-16T16:18:16.181 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/acard-ahci.ko 2023-11-16T16:18:16.181 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/ahci_platform.ko 2023-11-16T16:18:16.182 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/ata_generic.ko 2023-11-16T16:18:16.182 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/libahci_platform.ko 2023-11-16T16:18:16.182 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_acpi.ko 2023-11-16T16:18:16.182 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_ali.ko 2023-11-16T16:18:16.182 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_amd.ko 2023-11-16T16:18:16.182 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_artop.ko 2023-11-16T16:18:16.182 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_atiixp.ko 2023-11-16T16:18:16.182 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_atp867x.ko 2023-11-16T16:18:16.182 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_cmd640.ko 2023-11-16T16:18:16.182 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_cmd64x.ko 2023-11-16T16:18:16.182 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_cypress.ko 2023-11-16T16:18:16.183 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_efar.ko 2023-11-16T16:18:16.183 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_hpt366.ko 2023-11-16T16:18:16.183 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_hpt37x.ko 2023-11-16T16:18:16.183 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_hpt3x2n.ko 2023-11-16T16:18:16.183 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_hpt3x3.ko 2023-11-16T16:18:16.183 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_it8213.ko 2023-11-16T16:18:16.183 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_it821x.ko 2023-11-16T16:18:16.183 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_jmicron.ko 2023-11-16T16:18:16.183 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_marvell.ko 2023-11-16T16:18:16.183 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_mpiix.ko 2023-11-16T16:18:16.183 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_netcell.ko 2023-11-16T16:18:16.183 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_ninja32.ko 2023-11-16T16:18:16.183 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_ns87410.ko 2023-11-16T16:18:16.184 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_ns87415.ko 2023-11-16T16:18:16.184 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_oldpiix.ko 2023-11-16T16:18:16.184 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_opti.ko 2023-11-16T16:18:16.184 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_optidma.ko 2023-11-16T16:18:16.184 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_pcmcia.ko 2023-11-16T16:18:16.184 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_pdc2027x.ko 2023-11-16T16:18:16.184 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-11-16T16:18:16.184 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_piccolo.ko 2023-11-16T16:18:16.184 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_rdc.ko 2023-11-16T16:18:16.184 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_sch.ko 2023-11-16T16:18:16.185 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_serverworks.ko 2023-11-16T16:18:16.185 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_sil680.ko 2023-11-16T16:18:16.185 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_sis.ko 2023-11-16T16:18:16.185 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_sl82c105.ko 2023-11-16T16:18:16.185 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_triflex.ko 2023-11-16T16:18:16.185 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_via.ko 2023-11-16T16:18:16.185 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pdc_adma.ko 2023-11-16T16:18:16.185 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_inic162x.ko 2023-11-16T16:18:16.185 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_mv.ko 2023-11-16T16:18:16.185 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_nv.ko 2023-11-16T16:18:16.185 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_promise.ko 2023-11-16T16:18:16.186 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_qstor.ko 2023-11-16T16:18:16.186 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_sil.ko 2023-11-16T16:18:16.186 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_sil24.ko 2023-11-16T16:18:16.186 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_sis.ko 2023-11-16T16:18:16.186 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_svw.ko 2023-11-16T16:18:16.186 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_sx4.ko 2023-11-16T16:18:16.186 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_uli.ko 2023-11-16T16:18:16.186 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_via.ko 2023-11-16T16:18:16.186 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_vsc.ko 2023-11-16T16:18:16.186 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/auxdisplay 2023-11-16T16:18:16.187 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/auxdisplay/ks0108.ko 2023-11-16T16:18:16.187 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/base 2023-11-16T16:18:16.187 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/base/regmap 2023-11-16T16:18:16.187 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/base/regmap/regmap-i2c.ko 2023-11-16T16:18:16.187 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/base/regmap/regmap-mmio.ko 2023-11-16T16:18:16.187 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/bcma 2023-11-16T16:18:16.187 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/bcma/bcma.ko 2023-11-16T16:18:16.187 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block 2023-11-16T16:18:16.187 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/aoe 2023-11-16T16:18:16.187 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/aoe/aoe.ko 2023-11-16T16:18:16.187 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/brd.ko 2023-11-16T16:18:16.187 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/drbd 2023-11-16T16:18:16.187 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/drbd/drbd.ko 2023-11-16T16:18:16.187 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/floppy.ko 2023-11-16T16:18:16.188 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/loop.ko 2023-11-16T16:18:16.188 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/mtip32xx 2023-11-16T16:18:16.188 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-11-16T16:18:16.188 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/nbd.ko 2023-11-16T16:18:16.188 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/null_blk 2023-11-16T16:18:16.188 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/null_blk/null_blk.ko 2023-11-16T16:18:16.188 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/pktcdvd.ko 2023-11-16T16:18:16.188 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/rbd.ko 2023-11-16T16:18:16.188 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/virtio_blk.ko 2023-11-16T16:18:16.188 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/xen-blkback 2023-11-16T16:18:16.188 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-11-16T16:18:16.188 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/xen-blkfront.ko 2023-11-16T16:18:16.188 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/zram 2023-11-16T16:18:16.189 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/zram/zram.ko 2023-11-16T16:18:16.189 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char 2023-11-16T16:18:16.189 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hangcheck-timer.ko 2023-11-16T16:18:16.189 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random 2023-11-16T16:18:16.189 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random/amd-rng.ko 2023-11-16T16:18:16.189 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random/intel-rng.ko 2023-11-16T16:18:16.189 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-11-16T16:18:16.189 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random/via-rng.ko 2023-11-16T16:18:16.189 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random/virtio-rng.ko 2023-11-16T16:18:16.189 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi 2023-11-16T16:18:16.189 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-11-16T16:18:16.189 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-11-16T16:18:16.189 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-11-16T16:18:16.189 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_si.ko 2023-11-16T16:18:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-11-16T16:18:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-11-16T16:18:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/lp.ko 2023-11-16T16:18:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/mwave 2023-11-16T16:18:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/mwave/mwave.ko 2023-11-16T16:18:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ppdev.ko 2023-11-16T16:18:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tlclk.ko 2023-11-16T16:18:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm 2023-11-16T16:18:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm.ko 2023-11-16T16:18:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm_atmel.ko 2023-11-16T16:18:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm_infineon.ko 2023-11-16T16:18:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm_nsc.ko 2023-11-16T16:18:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm_tis.ko 2023-11-16T16:18:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-11-16T16:18:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/uv_mmtimer.ko 2023-11-16T16:18:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/virtio_console.ko 2023-11-16T16:18:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq 2023-11-16T16:18:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-11-16T16:18:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-11-16T16:18:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/p4-clockmod.ko 2023-11-16T16:18:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-11-16T16:18:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/powernow-k8.ko 2023-11-16T16:18:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/speedstep-lib.ko 2023-11-16T16:18:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto 2023-11-16T16:18:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/ccp 2023-11-16T16:18:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-11-16T16:18:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/ccp/ccp.ko 2023-11-16T16:18:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel 2023-11-16T16:18:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel/qat 2023-11-16T16:18:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel/qat/qat_common 2023-11-16T16:18:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-11-16T16:18:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-11-16T16:18:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-11-16T16:18:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/padlock-aes.ko 2023-11-16T16:18:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/padlock-sha.ko 2023-11-16T16:18:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/virtio 2023-11-16T16:18:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-11-16T16:18:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dca 2023-11-16T16:18:16.193 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dca/dca.ko 2023-11-16T16:18:16.193 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/devfreq 2023-11-16T16:18:16.193 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-11-16T16:18:16.193 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dma 2023-11-16T16:18:16.193 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dma/dw 2023-11-16T16:18:16.193 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dma/dw/dw_dmac.ko 2023-11-16T16:18:16.193 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dma/ioat 2023-11-16T16:18:16.193 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dma/ioat/ioatdma.ko 2023-11-16T16:18:16.193 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac 2023-11-16T16:18:16.193 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/amd64_edac.ko 2023-11-16T16:18:16.194 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/e752x_edac.ko 2023-11-16T16:18:16.194 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/edac_mce_amd.ko 2023-11-16T16:18:16.194 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i3000_edac.ko 2023-11-16T16:18:16.194 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i3200_edac.ko 2023-11-16T16:18:16.194 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i5100_edac.ko 2023-11-16T16:18:16.194 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i5400_edac.ko 2023-11-16T16:18:16.194 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i7300_edac.ko 2023-11-16T16:18:16.195 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i7core_edac.ko 2023-11-16T16:18:16.195 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i82975x_edac.ko 2023-11-16T16:18:16.195 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/ie31200_edac.ko 2023-11-16T16:18:16.195 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/sb_edac.ko 2023-11-16T16:18:16.195 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/x38_edac.ko 2023-11-16T16:18:16.195 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/firmware 2023-11-16T16:18:16.195 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/firmware/edd.ko 2023-11-16T16:18:16.196 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/firmware/iscsi_ibft.ko 2023-11-16T16:18:16.196 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/gpio 2023-11-16T16:18:16.196 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/gpio/gpio-ich.ko 2023-11-16T16:18:16.196 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/gpio/gpio-viperboard.ko 2023-11-16T16:18:16.196 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid 2023-11-16T16:18:16.196 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-appleir.ko 2023-11-16T16:18:16.197 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-aureal.ko 2023-11-16T16:18:16.197 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-axff.ko 2023-11-16T16:18:16.197 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-dr.ko 2023-11-16T16:18:16.197 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-elecom.ko 2023-11-16T16:18:16.197 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-elo.ko 2023-11-16T16:18:16.197 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-emsff.ko 2023-11-16T16:18:16.198 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-gaff.ko 2023-11-16T16:18:16.198 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-gt683r.ko 2023-11-16T16:18:16.198 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-gyration.ko 2023-11-16T16:18:16.198 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-holtek-kbd.ko 2023-11-16T16:18:16.198 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-holtek-mouse.ko 2023-11-16T16:18:16.198 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-holtekff.ko 2023-11-16T16:18:16.198 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-hyperv.ko 2023-11-16T16:18:16.198 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-icade.ko 2023-11-16T16:18:16.198 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-keytouch.ko 2023-11-16T16:18:16.199 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-kye.ko 2023-11-16T16:18:16.199 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-lcpower.ko 2023-11-16T16:18:16.199 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-led.ko 2023-11-16T16:18:16.199 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-lenovo.ko 2023-11-16T16:18:16.199 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-logitech-dj.ko 2023-11-16T16:18:16.199 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-11-16T16:18:16.199 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-multitouch.ko 2023-11-16T16:18:16.199 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-ortek.ko 2023-11-16T16:18:16.199 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-penmount.ko 2023-11-16T16:18:16.200 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-petalynx.ko 2023-11-16T16:18:16.200 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-picolcd.ko 2023-11-16T16:18:16.200 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-pl.ko 2023-11-16T16:18:16.200 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-plantronics.ko 2023-11-16T16:18:16.200 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-primax.ko 2023-11-16T16:18:16.200 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-rmi.ko 2023-11-16T16:18:16.200 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-arvo.ko 2023-11-16T16:18:16.200 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-common.ko 2023-11-16T16:18:16.200 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-isku.ko 2023-11-16T16:18:16.201 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-kone.ko 2023-11-16T16:18:16.201 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-11-16T16:18:16.201 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-konepure.ko 2023-11-16T16:18:16.201 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-11-16T16:18:16.201 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-lua.ko 2023-11-16T16:18:16.201 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-pyra.ko 2023-11-16T16:18:16.201 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-ryos.ko 2023-11-16T16:18:16.201 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-savu.ko 2023-11-16T16:18:16.202 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat.ko 2023-11-16T16:18:16.202 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-saitek.ko 2023-11-16T16:18:16.202 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-samsung.ko 2023-11-16T16:18:16.202 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-sensor-hub.ko 2023-11-16T16:18:16.202 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-sjoy.ko 2023-11-16T16:18:16.202 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-sony.ko 2023-11-16T16:18:16.202 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-speedlink.ko 2023-11-16T16:18:16.203 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-steelseries.ko 2023-11-16T16:18:16.203 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-sunplus.ko 2023-11-16T16:18:16.203 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-thrustmaster.ko 2023-11-16T16:18:16.203 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-tivo.ko 2023-11-16T16:18:16.203 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-tmff.ko 2023-11-16T16:18:16.203 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-topseed.ko 2023-11-16T16:18:16.203 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-twinhan.ko 2023-11-16T16:18:16.203 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-uclogic.ko 2023-11-16T16:18:16.203 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-waltop.ko 2023-11-16T16:18:16.203 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-wiimote.ko 2023-11-16T16:18:16.203 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-xinmo.ko 2023-11-16T16:18:16.203 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-zpff.ko 2023-11-16T16:18:16.203 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-zydacron.ko 2023-11-16T16:18:16.203 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/uhid.ko 2023-11-16T16:18:16.204 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/wacom.ko 2023-11-16T16:18:16.204 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hv 2023-11-16T16:18:16.204 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hv/hv_balloon.ko 2023-11-16T16:18:16.204 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hv/hv_utils.ko 2023-11-16T16:18:16.204 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hv/hv_vmbus.ko 2023-11-16T16:18:16.204 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon 2023-11-16T16:18:16.204 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/abituguru.ko 2023-11-16T16:18:16.204 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/abituguru3.ko 2023-11-16T16:18:16.204 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/acpi_power_meter.ko 2023-11-16T16:18:16.205 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ad7414.ko 2023-11-16T16:18:16.205 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ad7418.ko 2023-11-16T16:18:16.205 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adc128d818.ko 2023-11-16T16:18:16.205 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adm1025.ko 2023-11-16T16:18:16.205 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adm1026.ko 2023-11-16T16:18:16.205 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adm1029.ko 2023-11-16T16:18:16.205 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adm1031.ko 2023-11-16T16:18:16.205 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adm9240.ko 2023-11-16T16:18:16.206 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ads7828.ko 2023-11-16T16:18:16.206 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7410.ko 2023-11-16T16:18:16.206 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7411.ko 2023-11-16T16:18:16.206 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7462.ko 2023-11-16T16:18:16.206 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7470.ko 2023-11-16T16:18:16.206 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7475.ko 2023-11-16T16:18:16.207 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7x10.ko 2023-11-16T16:18:16.207 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/amc6821.ko 2023-11-16T16:18:16.207 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/applesmc.ko 2023-11-16T16:18:16.207 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/asb100.ko 2023-11-16T16:18:16.207 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/asc7621.ko 2023-11-16T16:18:16.207 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/asus_atk0110.ko 2023-11-16T16:18:16.208 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/atxp1.ko 2023-11-16T16:18:16.208 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/coretemp.ko 2023-11-16T16:18:16.208 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-11-16T16:18:16.208 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/dme1737.ko 2023-11-16T16:18:16.208 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ds1621.ko 2023-11-16T16:18:16.208 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ds620.ko 2023-11-16T16:18:16.208 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/emc1403.ko 2023-11-16T16:18:16.209 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/emc6w201.ko 2023-11-16T16:18:16.209 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/f71805f.ko 2023-11-16T16:18:16.209 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/f71882fg.ko 2023-11-16T16:18:16.209 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/f75375s.ko 2023-11-16T16:18:16.209 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/fam15h_power.ko 2023-11-16T16:18:16.209 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/fschmd.ko 2023-11-16T16:18:16.209 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/g760a.ko 2023-11-16T16:18:16.210 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/g762.ko 2023-11-16T16:18:16.210 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/gl518sm.ko 2023-11-16T16:18:16.210 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/gl520sm.ko 2023-11-16T16:18:16.210 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/hwmon-vid.ko 2023-11-16T16:18:16.210 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/i5500_temp.ko 2023-11-16T16:18:16.210 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/i5k_amb.ko 2023-11-16T16:18:16.210 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ibmaem.ko 2023-11-16T16:18:16.211 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ibmpex.ko 2023-11-16T16:18:16.211 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ina209.ko 2023-11-16T16:18:16.211 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ina2xx.ko 2023-11-16T16:18:16.211 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/it87.ko 2023-11-16T16:18:16.211 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/k10temp.ko 2023-11-16T16:18:16.211 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/k8temp.ko 2023-11-16T16:18:16.211 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lineage-pem.ko 2023-11-16T16:18:16.211 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm63.ko 2023-11-16T16:18:16.211 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm73.ko 2023-11-16T16:18:16.211 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm75.ko 2023-11-16T16:18:16.211 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm77.ko 2023-11-16T16:18:16.212 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm78.ko 2023-11-16T16:18:16.212 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm80.ko 2023-11-16T16:18:16.212 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm83.ko 2023-11-16T16:18:16.212 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm85.ko 2023-11-16T16:18:16.212 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm87.ko 2023-11-16T16:18:16.212 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm90.ko 2023-11-16T16:18:16.212 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm92.ko 2023-11-16T16:18:16.212 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm93.ko 2023-11-16T16:18:16.212 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm95234.ko 2023-11-16T16:18:16.212 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm95241.ko 2023-11-16T16:18:16.212 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm95245.ko 2023-11-16T16:18:16.212 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc2945.ko 2023-11-16T16:18:16.213 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4151.ko 2023-11-16T16:18:16.213 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4215.ko 2023-11-16T16:18:16.213 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4222.ko 2023-11-16T16:18:16.213 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4245.ko 2023-11-16T16:18:16.213 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4260.ko 2023-11-16T16:18:16.213 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4261.ko 2023-11-16T16:18:16.213 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max16065.ko 2023-11-16T16:18:16.213 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max1619.ko 2023-11-16T16:18:16.213 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max1668.ko 2023-11-16T16:18:16.213 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max197.ko 2023-11-16T16:18:16.213 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max6639.ko 2023-11-16T16:18:16.213 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max6650.ko 2023-11-16T16:18:16.214 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max6697.ko 2023-11-16T16:18:16.214 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/mcp3021.ko 2023-11-16T16:18:16.214 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/nct6683.ko 2023-11-16T16:18:16.214 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/nct6775-core.ko 2023-11-16T16:18:16.214 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/nct6775.ko 2023-11-16T16:18:16.214 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/nct7802.ko 2023-11-16T16:18:16.214 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pc87360.ko 2023-11-16T16:18:16.214 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pc87427.ko 2023-11-16T16:18:16.214 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pcf8591.ko 2023-11-16T16:18:16.214 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus 2023-11-16T16:18:16.214 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-11-16T16:18:16.214 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-11-16T16:18:16.215 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-11-16T16:18:16.215 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/max16064.ko 2023-11-16T16:18:16.215 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/max34440.ko 2023-11-16T16:18:16.215 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/max8688.ko 2023-11-16T16:18:16.215 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-11-16T16:18:16.215 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-11-16T16:18:16.215 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-11-16T16:18:16.215 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-11-16T16:18:16.215 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-11-16T16:18:16.215 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-11-16T16:18:16.215 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/powr1220.ko 2023-11-16T16:18:16.215 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sch5627.ko 2023-11-16T16:18:16.215 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sch5636.ko 2023-11-16T16:18:16.215 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sch56xx-common.ko 2023-11-16T16:18:16.216 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sht15.ko 2023-11-16T16:18:16.216 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sht21.ko 2023-11-16T16:18:16.216 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/shtc1.ko 2023-11-16T16:18:16.216 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sis5595.ko 2023-11-16T16:18:16.216 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/smsc47b397.ko 2023-11-16T16:18:16.216 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/smsc47m1.ko 2023-11-16T16:18:16.216 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/smsc47m192.ko 2023-11-16T16:18:16.216 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/thmc50.ko 2023-11-16T16:18:16.216 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/tmp102.ko 2023-11-16T16:18:16.216 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/tmp103.ko 2023-11-16T16:18:16.216 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/tmp401.ko 2023-11-16T16:18:16.216 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/tmp421.ko 2023-11-16T16:18:16.216 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/via-cputemp.ko 2023-11-16T16:18:16.217 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/via686a.ko 2023-11-16T16:18:16.217 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/vt1211.ko 2023-11-16T16:18:16.217 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/vt8231.ko 2023-11-16T16:18:16.217 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83627ehf.ko 2023-11-16T16:18:16.217 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83627hf.ko 2023-11-16T16:18:16.217 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83781d.ko 2023-11-16T16:18:16.217 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83791d.ko 2023-11-16T16:18:16.217 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83792d.ko 2023-11-16T16:18:16.217 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83793.ko 2023-11-16T16:18:16.217 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83795.ko 2023-11-16T16:18:16.217 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83l785ts.ko 2023-11-16T16:18:16.217 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83l786ng.ko 2023-11-16T16:18:16.217 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c 2023-11-16T16:18:16.218 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/algos 2023-11-16T16:18:16.218 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-11-16T16:18:16.218 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-11-16T16:18:16.218 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses 2023-11-16T16:18:16.218 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-11-16T16:18:16.218 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-11-16T16:18:16.218 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-11-16T16:18:16.218 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-11-16T16:18:16.218 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-11-16T16:18:16.218 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-11-16T16:18:16.218 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-11-16T16:18:16.218 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-11-16T16:18:16.219 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-i801.ko 2023-11-16T16:18:16.219 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-isch.ko 2023-11-16T16:18:16.219 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-11-16T16:18:16.219 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-11-16T16:18:16.219 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-11-16T16:18:16.219 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-parport.ko 2023-11-16T16:18:16.219 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-11-16T16:18:16.219 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-11-16T16:18:16.219 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-11-16T16:18:16.219 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-11-16T16:18:16.219 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-11-16T16:18:16.219 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-11-16T16:18:16.219 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-via.ko 2023-11-16T16:18:16.220 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-11-16T16:18:16.220 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-11-16T16:18:16.220 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/i2c-dev.ko 2023-11-16T16:18:16.220 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/i2c-mux.ko 2023-11-16T16:18:16.220 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-11-16T16:18:16.220 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/i2c-smbus.ko 2023-11-16T16:18:16.220 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/i2c-stub.ko 2023-11-16T16:18:16.220 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband 2023-11-16T16:18:16.220 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core 2023-11-16T16:18:16.220 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/ib_cm.ko 2023-11-16T16:18:16.220 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/ib_core.ko 2023-11-16T16:18:16.220 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/ib_umad.ko 2023-11-16T16:18:16.220 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-11-16T16:18:16.221 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/iw_cm.ko 2023-11-16T16:18:16.221 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/rdma_cm.ko 2023-11-16T16:18:16.221 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-11-16T16:18:16.221 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw 2023-11-16T16:18:16.221 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/cxgb4 2023-11-16T16:18:16.221 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-11-16T16:18:16.221 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mlx4 2023-11-16T16:18:16.221 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-11-16T16:18:16.221 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mlx5 2023-11-16T16:18:16.221 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-11-16T16:18:16.221 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mthca 2023-11-16T16:18:16.221 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-11-16T16:18:16.221 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp 2023-11-16T16:18:16.221 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/ipoib 2023-11-16T16:18:16.222 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-11-16T16:18:16.222 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/iser 2023-11-16T16:18:16.222 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-11-16T16:18:16.222 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/isert 2023-11-16T16:18:16.222 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-11-16T16:18:16.222 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/srp 2023-11-16T16:18:16.222 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-11-16T16:18:16.222 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/srpt 2023-11-16T16:18:16.222 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-11-16T16:18:16.222 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input 2023-11-16T16:18:16.222 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/gameport 2023-11-16T16:18:16.222 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-11-16T16:18:16.222 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/gameport/fm801-gp.ko 2023-11-16T16:18:16.223 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/gameport/gameport.ko 2023-11-16T16:18:16.223 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/joydev.ko 2023-11-16T16:18:16.223 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/keyboard 2023-11-16T16:18:16.223 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/keyboard/gpio_keys.ko 2023-11-16T16:18:16.223 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse 2023-11-16T16:18:16.223 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/appletouch.ko 2023-11-16T16:18:16.223 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/bcm5974.ko 2023-11-16T16:18:16.223 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/cyapatp.ko 2023-11-16T16:18:16.223 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/elan_i2c.ko 2023-11-16T16:18:16.223 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/sermouse.ko 2023-11-16T16:18:16.223 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-11-16T16:18:16.223 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/synaptics_usb.ko 2023-11-16T16:18:16.223 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/vsxxxaa.ko 2023-11-16T16:18:16.223 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/rmi4 2023-11-16T16:18:16.223 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/rmi4/rmi_core.ko 2023-11-16T16:18:16.224 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/serio 2023-11-16T16:18:16.224 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/serio/altera_ps2.ko 2023-11-16T16:18:16.224 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/serio/arc_ps2.ko 2023-11-16T16:18:16.224 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-11-16T16:18:16.224 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/serio/serio_raw.ko 2023-11-16T16:18:16.224 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/sparse-keymap.ko 2023-11-16T16:18:16.224 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/iommu 2023-11-16T16:18:16.224 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/iommu/amd 2023-11-16T16:18:16.224 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/iommu/amd/iommu_v2.ko 2023-11-16T16:18:16.224 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds 2023-11-16T16:18:16.224 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/leds-blinkm.ko 2023-11-16T16:18:16.224 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/leds-lm3530.ko 2023-11-16T16:18:16.224 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/leds-lp3944.ko 2023-11-16T16:18:16.225 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/leds-ss4200.ko 2023-11-16T16:18:16.225 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger 2023-11-16T16:18:16.225 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-11-16T16:18:16.225 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-11-16T16:18:16.225 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-11-16T16:18:16.225 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-11-16T16:18:16.226 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-11-16T16:18:16.226 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-11-16T16:18:16.226 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-11-16T16:18:16.226 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-11-16T16:18:16.226 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md 2023-11-16T16:18:16.226 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/bcache 2023-11-16T16:18:16.226 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/bcache/bcache.ko 2023-11-16T16:18:16.226 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-bio-prison.ko 2023-11-16T16:18:16.227 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-cache-smq.ko 2023-11-16T16:18:16.227 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-cache.ko 2023-11-16T16:18:16.227 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-crypt.ko 2023-11-16T16:18:16.227 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-delay.ko 2023-11-16T16:18:16.227 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-flakey.ko 2023-11-16T16:18:16.227 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-log-userspace.ko 2023-11-16T16:18:16.227 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-multipath.ko 2023-11-16T16:18:16.227 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-queue-length.ko 2023-11-16T16:18:16.227 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-raid.ko 2023-11-16T16:18:16.229 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-round-robin.ko 2023-11-16T16:18:16.229 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-service-time.ko 2023-11-16T16:18:16.229 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-switch.ko 2023-11-16T16:18:16.229 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-thin-pool.ko 2023-11-16T16:18:16.230 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-verity.ko 2023-11-16T16:18:16.230 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/faulty.ko 2023-11-16T16:18:16.230 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/linear.ko 2023-11-16T16:18:16.230 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/multipath.ko 2023-11-16T16:18:16.230 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/persistent-data 2023-11-16T16:18:16.230 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-11-16T16:18:16.230 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/raid0.ko 2023-11-16T16:18:16.231 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/raid1.ko 2023-11-16T16:18:16.231 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/raid10.ko 2023-11-16T16:18:16.231 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/raid456.ko 2023-11-16T16:18:16.231 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message 2023-11-16T16:18:16.231 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion 2023-11-16T16:18:16.231 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptbase.ko 2023-11-16T16:18:16.231 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptctl.ko 2023-11-16T16:18:16.231 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptfc.ko 2023-11-16T16:18:16.231 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptlan.ko 2023-11-16T16:18:16.232 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptsas.ko 2023-11-16T16:18:16.232 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptscsih.ko 2023-11-16T16:18:16.232 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptspi.ko 2023-11-16T16:18:16.232 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd 2023-11-16T16:18:16.232 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/lpc_ich.ko 2023-11-16T16:18:16.232 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/lpc_sch.ko 2023-11-16T16:18:16.232 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/mfd-core.ko 2023-11-16T16:18:16.232 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/sm501.ko 2023-11-16T16:18:16.232 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/viperboard.ko 2023-11-16T16:18:16.233 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/vx855.ko 2023-11-16T16:18:16.233 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/wl1273-core.ko 2023-11-16T16:18:16.233 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc 2023-11-16T16:18:16.233 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/altera-stapl 2023-11-16T16:18:16.233 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-11-16T16:18:16.233 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/apds9802als.ko 2023-11-16T16:18:16.233 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/apds990x.ko 2023-11-16T16:18:16.233 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/bh1770glc.ko 2023-11-16T16:18:16.233 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/cb710 2023-11-16T16:18:16.234 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/cb710/cb710.ko 2023-11-16T16:18:16.234 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/eeprom 2023-11-16T16:18:16.234 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/eeprom/at24.ko 2023-11-16T16:18:16.234 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/eeprom/eeprom.ko 2023-11-16T16:18:16.234 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-11-16T16:18:16.234 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/eeprom/max6875.ko 2023-11-16T16:18:16.235 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/enclosure.ko 2023-11-16T16:18:16.235 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/hpilo.ko 2023-11-16T16:18:16.235 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/isl29003.ko 2023-11-16T16:18:16.235 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/isl29020.ko 2023-11-16T16:18:16.235 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/lis3lv02d 2023-11-16T16:18:16.235 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-11-16T16:18:16.235 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-11-16T16:18:16.235 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/mei 2023-11-16T16:18:16.235 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/mei/mei-me.ko 2023-11-16T16:18:16.235 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/mei/mei-txe.ko 2023-11-16T16:18:16.235 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/mei/mei.ko 2023-11-16T16:18:16.235 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-gru 2023-11-16T16:18:16.235 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-gru/gru.ko 2023-11-16T16:18:16.235 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-xp 2023-11-16T16:18:16.236 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-xp/xp.ko 2023-11-16T16:18:16.236 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-xp/xpc.ko 2023-11-16T16:18:16.236 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-11-16T16:18:16.236 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/tifm_7xx1.ko 2023-11-16T16:18:16.236 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/tifm_core.ko 2023-11-16T16:18:16.236 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/tsl2550.ko 2023-11-16T16:18:16.236 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/vmw_balloon.ko 2023-11-16T16:18:16.236 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/vmw_vmci 2023-11-16T16:18:16.236 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-11-16T16:18:16.236 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net 2023-11-16T16:18:16.236 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/bonding 2023-11-16T16:18:16.236 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/bonding/bonding.ko 2023-11-16T16:18:16.236 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa 2023-11-16T16:18:16.236 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/b53 2023-11-16T16:18:16.237 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/b53/b53_common.ko 2023-11-16T16:18:16.237 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/bcm-sf2.ko 2023-11-16T16:18:16.237 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/mv88e6060.ko 2023-11-16T16:18:16.237 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/mv88e6xxx 2023-11-16T16:18:16.237 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-11-16T16:18:16.237 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dummy.ko 2023-11-16T16:18:16.237 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/eql.ko 2023-11-16T16:18:16.237 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet 2023-11-16T16:18:16.237 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/3com 2023-11-16T16:18:16.237 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-11-16T16:18:16.237 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-11-16T16:18:16.237 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-11-16T16:18:16.237 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-11-16T16:18:16.237 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/8390 2023-11-16T16:18:16.238 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/8390/8390.ko 2023-11-16T16:18:16.238 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-11-16T16:18:16.238 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-11-16T16:18:16.238 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-11-16T16:18:16.238 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/adaptec 2023-11-16T16:18:16.238 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-11-16T16:18:16.238 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/agere 2023-11-16T16:18:16.238 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/agere/et131x.ko 2023-11-16T16:18:16.238 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/alteon 2023-11-16T16:18:16.238 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-11-16T16:18:16.238 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/altera 2023-11-16T16:18:16.238 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-11-16T16:18:16.238 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/amd 2023-11-16T16:18:16.239 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-11-16T16:18:16.239 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-11-16T16:18:16.239 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-11-16T16:18:16.239 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros 2023-11-16T16:18:16.239 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/alx 2023-11-16T16:18:16.239 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-11-16T16:18:16.239 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atl1c 2023-11-16T16:18:16.239 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-11-16T16:18:16.239 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atl1e 2023-11-16T16:18:16.239 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-11-16T16:18:16.239 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atlx 2023-11-16T16:18:16.239 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-11-16T16:18:16.239 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-11-16T16:18:16.240 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom 2023-11-16T16:18:16.240 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-11-16T16:18:16.240 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-11-16T16:18:16.240 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-11-16T16:18:16.240 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-11-16T16:18:16.240 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-11-16T16:18:16.240 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/genet 2023-11-16T16:18:16.240 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-11-16T16:18:16.240 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-11-16T16:18:16.240 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/brocade 2023-11-16T16:18:16.240 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/brocade/bna 2023-11-16T16:18:16.240 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-11-16T16:18:16.240 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio 2023-11-16T16:18:16.240 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb 2023-11-16T16:18:16.241 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-11-16T16:18:16.241 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-11-16T16:18:16.241 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-11-16T16:18:16.241 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-11-16T16:18:16.241 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-11-16T16:18:16.241 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-11-16T16:18:16.241 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-11-16T16:18:16.241 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-11-16T16:18:16.241 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-11-16T16:18:16.241 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/cisco 2023-11-16T16:18:16.241 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/cisco/enic 2023-11-16T16:18:16.241 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-11-16T16:18:16.241 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec 2023-11-16T16:18:16.241 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip 2023-11-16T16:18:16.242 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-11-16T16:18:16.242 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-11-16T16:18:16.242 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-11-16T16:18:16.242 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-11-16T16:18:16.242 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-11-16T16:18:16.242 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-11-16T16:18:16.242 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dlink 2023-11-16T16:18:16.242 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-11-16T16:18:16.242 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-11-16T16:18:16.242 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dnet.ko 2023-11-16T16:18:16.242 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/emulex 2023-11-16T16:18:16.242 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/emulex/benet 2023-11-16T16:18:16.242 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-11-16T16:18:16.243 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/ethoc.ko 2023-11-16T16:18:16.243 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/fealnx.ko 2023-11-16T16:18:16.243 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel 2023-11-16T16:18:16.243 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/e100.ko 2023-11-16T16:18:16.243 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/e1000 2023-11-16T16:18:16.243 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-11-16T16:18:16.243 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/e1000e 2023-11-16T16:18:16.243 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-11-16T16:18:16.243 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/fm10k 2023-11-16T16:18:16.243 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-11-16T16:18:16.243 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/i40e 2023-11-16T16:18:16.243 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-11-16T16:18:16.243 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/iavf 2023-11-16T16:18:16.243 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-11-16T16:18:16.243 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/igb 2023-11-16T16:18:16.244 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-11-16T16:18:16.244 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/igbvf 2023-11-16T16:18:16.244 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-11-16T16:18:16.244 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/ixgbe 2023-11-16T16:18:16.244 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-11-16T16:18:16.244 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/ixgbevf 2023-11-16T16:18:16.244 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-11-16T16:18:16.244 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/jme.ko 2023-11-16T16:18:16.244 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/marvell 2023-11-16T16:18:16.244 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-11-16T16:18:16.244 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/marvell/skge.ko 2023-11-16T16:18:16.244 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-11-16T16:18:16.244 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox 2023-11-16T16:18:16.244 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx4 2023-11-16T16:18:16.245 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-11-16T16:18:16.245 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-11-16T16:18:16.245 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx5 2023-11-16T16:18:16.245 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-11-16T16:18:16.245 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-11-16T16:18:16.245 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/micrel 2023-11-16T16:18:16.245 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-11-16T16:18:16.245 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/myricom 2023-11-16T16:18:16.245 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/myricom/myri10ge 2023-11-16T16:18:16.245 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-11-16T16:18:16.245 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/natsemi 2023-11-16T16:18:16.245 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-11-16T16:18:16.245 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-11-16T16:18:16.245 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/neterion 2023-11-16T16:18:16.246 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-11-16T16:18:16.246 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/nvidia 2023-11-16T16:18:16.246 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-11-16T16:18:16.246 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/packetengines 2023-11-16T16:18:16.246 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-11-16T16:18:16.246 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-11-16T16:18:16.246 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic 2023-11-16T16:18:16.246 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic/netxen 2023-11-16T16:18:16.247 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-11-16T16:18:16.247 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-11-16T16:18:16.247 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-11-16T16:18:16.247 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-11-16T16:18:16.247 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/rdc 2023-11-16T16:18:16.247 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-11-16T16:18:16.247 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/realtek 2023-11-16T16:18:16.247 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-11-16T16:18:16.248 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-11-16T16:18:16.248 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/realtek/atp.ko 2023-11-16T16:18:16.248 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-11-16T16:18:16.248 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/rocker 2023-11-16T16:18:16.248 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-11-16T16:18:16.248 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sfc 2023-11-16T16:18:16.249 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-11-16T16:18:16.249 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/silan 2023-11-16T16:18:16.249 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-11-16T16:18:16.249 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sis 2023-11-16T16:18:16.249 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sis/sis190.ko 2023-11-16T16:18:16.249 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sis/sis900.ko 2023-11-16T16:18:16.250 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/smsc 2023-11-16T16:18:16.250 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-11-16T16:18:16.250 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-11-16T16:18:16.250 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-11-16T16:18:16.250 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-11-16T16:18:16.250 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/stmicro 2023-11-16T16:18:16.251 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/stmicro/stmmac 2023-11-16T16:18:16.251 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-11-16T16:18:16.251 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-11-16T16:18:16.251 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sun 2023-11-16T16:18:16.251 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sun/cassini.ko 2023-11-16T16:18:16.251 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sun/niu.ko 2023-11-16T16:18:16.252 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sun/sungem.ko 2023-11-16T16:18:16.252 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-11-16T16:18:16.252 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/tehuti 2023-11-16T16:18:16.252 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-11-16T16:18:16.252 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/ti 2023-11-16T16:18:16.252 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/ti/tlan.ko 2023-11-16T16:18:16.252 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/via 2023-11-16T16:18:16.252 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-11-16T16:18:16.253 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-11-16T16:18:16.253 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/wiznet 2023-11-16T16:18:16.253 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-11-16T16:18:16.253 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-11-16T16:18:16.253 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/xircom 2023-11-16T16:18:16.254 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-11-16T16:18:16.254 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/geneve.ko 2023-11-16T16:18:16.254 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/hyperv 2023-11-16T16:18:16.254 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-11-16T16:18:16.254 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ieee802154 2023-11-16T16:18:16.254 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ieee802154/fakelb.ko 2023-11-16T16:18:16.255 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ifb.ko 2023-11-16T16:18:16.255 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ipvlan 2023-11-16T16:18:16.255 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ipvlan/ipvlan.ko 2023-11-16T16:18:16.255 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/macvlan.ko 2023-11-16T16:18:16.255 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/macvtap.ko 2023-11-16T16:18:16.255 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mdio 2023-11-16T16:18:16.255 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mdio.ko 2023-11-16T16:18:16.255 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-11-16T16:18:16.256 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-11-16T16:18:16.256 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mdio/mdio-regmap.ko 2023-11-16T16:18:16.256 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mii.ko 2023-11-16T16:18:16.256 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/net_failover.ko 2023-11-16T16:18:16.256 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/netconsole.ko 2023-11-16T16:18:16.256 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/nlmon.ko 2023-11-16T16:18:16.256 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/pcs 2023-11-16T16:18:16.256 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/pcs/pcs-lynx.ko 2023-11-16T16:18:16.256 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-11-16T16:18:16.256 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy 2023-11-16T16:18:16.256 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/amd.ko 2023-11-16T16:18:16.256 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-11-16T16:18:16.256 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-11-16T16:18:16.257 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/bcm7xxx.ko 2023-11-16T16:18:16.257 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/bcm87xx.ko 2023-11-16T16:18:16.257 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/broadcom.ko 2023-11-16T16:18:16.257 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/cicada.ko 2023-11-16T16:18:16.257 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/davicom.ko 2023-11-16T16:18:16.257 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/dp83640.ko 2023-11-16T16:18:16.257 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/et1011c.ko 2023-11-16T16:18:16.257 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/icplus.ko 2023-11-16T16:18:16.257 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/lxt.ko 2023-11-16T16:18:16.257 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/marvell.ko 2023-11-16T16:18:16.257 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/micrel.ko 2023-11-16T16:18:16.257 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/national.ko 2023-11-16T16:18:16.257 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/phylink.ko 2023-11-16T16:18:16.258 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/qsemi.ko 2023-11-16T16:18:16.258 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/realtek.ko 2023-11-16T16:18:16.258 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/smsc.ko 2023-11-16T16:18:16.258 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/ste10Xp.ko 2023-11-16T16:18:16.258 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/vitesse.ko 2023-11-16T16:18:16.258 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp 2023-11-16T16:18:16.258 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/bsd_comp.ko 2023-11-16T16:18:16.258 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/ppp_async.ko 2023-11-16T16:18:16.258 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/ppp_deflate.ko 2023-11-16T16:18:16.258 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/ppp_generic.ko 2023-11-16T16:18:16.258 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/ppp_mppe.ko 2023-11-16T16:18:16.258 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/ppp_synctty.ko 2023-11-16T16:18:16.259 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/pppoe.ko 2023-11-16T16:18:16.259 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/pppox.ko 2023-11-16T16:18:16.259 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/pptp.ko 2023-11-16T16:18:16.259 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/slip 2023-11-16T16:18:16.259 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/slip/slhc.ko 2023-11-16T16:18:16.259 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/slip/slip.ko 2023-11-16T16:18:16.259 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/sungem_phy.ko 2023-11-16T16:18:16.259 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/tap.ko 2023-11-16T16:18:16.259 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team 2023-11-16T16:18:16.259 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team.ko 2023-11-16T16:18:16.259 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team_mode_activebackup.ko 2023-11-16T16:18:16.259 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team_mode_broadcast.ko 2023-11-16T16:18:16.260 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-11-16T16:18:16.260 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team_mode_random.ko 2023-11-16T16:18:16.260 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-11-16T16:18:16.260 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/tun.ko 2023-11-16T16:18:16.260 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/veth.ko 2023-11-16T16:18:16.260 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/virtio_net.ko 2023-11-16T16:18:16.260 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/vmxnet3 2023-11-16T16:18:16.260 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-11-16T16:18:16.260 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/vxlan 2023-11-16T16:18:16.260 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/vxlan/vxlan.ko 2023-11-16T16:18:16.260 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/xen-netback 2023-11-16T16:18:16.260 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/xen-netback/xen-netback.ko 2023-11-16T16:18:16.261 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/xen-netfront.ko 2023-11-16T16:18:16.261 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ntb 2023-11-16T16:18:16.261 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ntb/ntb.ko 2023-11-16T16:18:16.261 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/nvme 2023-11-16T16:18:16.261 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/nvme/host 2023-11-16T16:18:16.261 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/nvme/host/nvme-core.ko 2023-11-16T16:18:16.261 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/nvme/host/nvme.ko 2023-11-16T16:18:16.261 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/parport 2023-11-16T16:18:16.261 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/parport/parport.ko 2023-11-16T16:18:16.261 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/parport/parport_cs.ko 2023-11-16T16:18:16.261 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/parport/parport_pc.ko 2023-11-16T16:18:16.261 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/parport/parport_serial.ko 2023-11-16T16:18:16.262 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci 2023-11-16T16:18:16.262 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci/hotplug 2023-11-16T16:18:16.262 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-11-16T16:18:16.262 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci/pcie 2023-11-16T16:18:16.262 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci/pcie/aer_inject.ko 2023-11-16T16:18:16.262 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci/xen-pcifront.ko 2023-11-16T16:18:16.262 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pcmcia 2023-11-16T16:18:16.262 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pcmcia/i82092.ko 2023-11-16T16:18:16.262 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pcmcia/pd6729.ko 2023-11-16T16:18:16.262 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pcmcia/yenta_socket.ko 2023-11-16T16:18:16.262 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pinctrl 2023-11-16T16:18:16.262 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pinctrl/intel 2023-11-16T16:18:16.263 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-11-16T16:18:16.263 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform 2023-11-16T16:18:16.263 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86 2023-11-16T16:18:16.263 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/acer-wmi.ko 2023-11-16T16:18:16.263 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/acerhdf.ko 2023-11-16T16:18:16.263 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/apple-gmux.ko 2023-11-16T16:18:16.263 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/asus-laptop.ko 2023-11-16T16:18:16.263 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-11-16T16:18:16.263 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/asus-wmi.ko 2023-11-16T16:18:16.263 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/classmate-laptop.ko 2023-11-16T16:18:16.263 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-11-16T16:18:16.263 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-11-16T16:18:16.263 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-11-16T16:18:16.264 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-11-16T16:18:16.264 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/hdaps.ko 2023-11-16T16:18:16.264 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/intel 2023-11-16T16:18:16.264 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-11-16T16:18:16.264 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/intel_ips.ko 2023-11-16T16:18:16.264 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/msi-wmi.ko 2023-11-16T16:18:16.264 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/mxm-wmi.ko 2023-11-16T16:18:16.264 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-11-16T16:18:16.264 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/samsung-laptop.ko 2023-11-16T16:18:16.264 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/samsung-q10.ko 2023-11-16T16:18:16.264 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/topstar-laptop.ko 2023-11-16T16:18:16.264 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-11-16T16:18:16.265 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/toshiba_haps.ko 2023-11-16T16:18:16.265 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/wmi-bmof.ko 2023-11-16T16:18:16.265 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/wmi.ko 2023-11-16T16:18:16.265 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/powercap 2023-11-16T16:18:16.265 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/powercap/intel_rapl_common.ko 2023-11-16T16:18:16.265 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/powercap/intel_rapl_msr.ko 2023-11-16T16:18:16.265 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps 2023-11-16T16:18:16.265 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps/clients 2023-11-16T16:18:16.265 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps/clients/pps-gpio.ko 2023-11-16T16:18:16.265 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps/clients/pps-ldisc.ko 2023-11-16T16:18:16.265 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps/clients/pps_parport.ko 2023-11-16T16:18:16.265 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps/pps_core.ko 2023-11-16T16:18:16.265 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ptp 2023-11-16T16:18:16.265 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ptp/ptp.ko 2023-11-16T16:18:16.266 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ptp/ptp_kvm.ko 2023-11-16T16:18:16.266 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc 2023-11-16T16:18:16.266 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-bq32k.ko 2023-11-16T16:18:16.266 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1286.ko 2023-11-16T16:18:16.266 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1307.ko 2023-11-16T16:18:16.266 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1374.ko 2023-11-16T16:18:16.266 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1511.ko 2023-11-16T16:18:16.266 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1553.ko 2023-11-16T16:18:16.267 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1672.ko 2023-11-16T16:18:16.267 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1742.ko 2023-11-16T16:18:16.267 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds2404.ko 2023-11-16T16:18:16.267 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds3232.ko 2023-11-16T16:18:16.267 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-em3027.ko 2023-11-16T16:18:16.267 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-fm3130.ko 2023-11-16T16:18:16.267 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-isl12022.ko 2023-11-16T16:18:16.267 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-isl1208.ko 2023-11-16T16:18:16.267 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-m41t80.ko 2023-11-16T16:18:16.268 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-m48t35.ko 2023-11-16T16:18:16.268 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-m48t59.ko 2023-11-16T16:18:16.268 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-max6900.ko 2023-11-16T16:18:16.268 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-msm6242.ko 2023-11-16T16:18:16.268 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-pcf2127.ko 2023-11-16T16:18:16.268 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-pcf85063.ko 2023-11-16T16:18:16.268 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-pcf8523.ko 2023-11-16T16:18:16.268 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-pcf8563.ko 2023-11-16T16:18:16.268 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-pcf8583.ko 2023-11-16T16:18:16.268 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-rp5c01.ko 2023-11-16T16:18:16.268 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-rs5c372.ko 2023-11-16T16:18:16.268 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-11-16T16:18:16.268 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-rx8025.ko 2023-11-16T16:18:16.269 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-rx8581.ko 2023-11-16T16:18:16.269 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-11-16T16:18:16.269 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-x1205.ko 2023-11-16T16:18:16.269 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi 2023-11-16T16:18:16.269 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/3w-9xxx.ko 2023-11-16T16:18:16.269 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/3w-sas.ko 2023-11-16T16:18:16.269 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/3w-xxxx.ko 2023-11-16T16:18:16.269 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/BusLogic.ko 2023-11-16T16:18:16.269 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/a100u2w.ko 2023-11-16T16:18:16.269 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/aacraid 2023-11-16T16:18:16.269 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/aacraid/aacraid.ko 2023-11-16T16:18:16.269 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/advansys.ko 2023-11-16T16:18:16.269 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/aic7xxx 2023-11-16T16:18:16.270 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-11-16T16:18:16.270 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-11-16T16:18:16.270 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/am53c974.ko 2023-11-16T16:18:16.270 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/arcmsr 2023-11-16T16:18:16.270 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-11-16T16:18:16.270 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/atp870u.ko 2023-11-16T16:18:16.270 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/be2iscsi 2023-11-16T16:18:16.270 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-11-16T16:18:16.270 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bfa 2023-11-16T16:18:16.270 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bfa/bfa.ko 2023-11-16T16:18:16.271 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bnx2fc 2023-11-16T16:18:16.271 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-11-16T16:18:16.271 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bnx2i 2023-11-16T16:18:16.271 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-11-16T16:18:16.271 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/ch.ko 2023-11-16T16:18:16.271 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/csiostor 2023-11-16T16:18:16.271 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/csiostor/csiostor.ko 2023-11-16T16:18:16.271 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi 2023-11-16T16:18:16.271 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi/cxgb3i 2023-11-16T16:18:16.271 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-11-16T16:18:16.271 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi/cxgb4i 2023-11-16T16:18:16.271 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-11-16T16:18:16.272 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-11-16T16:18:16.272 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/dc395x.ko 2023-11-16T16:18:16.272 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/device_handler 2023-11-16T16:18:16.272 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-11-16T16:18:16.272 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-11-16T16:18:16.272 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-11-16T16:18:16.272 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-11-16T16:18:16.272 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/dmx3191d.ko 2023-11-16T16:18:16.272 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/esas2r 2023-11-16T16:18:16.272 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/esas2r/esas2r.ko 2023-11-16T16:18:16.272 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/esp_scsi.ko 2023-11-16T16:18:16.272 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/fcoe 2023-11-16T16:18:16.272 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/fcoe/fcoe.ko 2023-11-16T16:18:16.273 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-11-16T16:18:16.273 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/fnic 2023-11-16T16:18:16.273 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/fnic/fnic.ko 2023-11-16T16:18:16.273 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/hpsa.ko 2023-11-16T16:18:16.273 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/hptiop.ko 2023-11-16T16:18:16.273 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/hv_storvsc.ko 2023-11-16T16:18:16.273 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/initio.ko 2023-11-16T16:18:16.273 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/ipr.ko 2023-11-16T16:18:16.273 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/ips.ko 2023-11-16T16:18:16.273 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/isci 2023-11-16T16:18:16.273 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/isci/isci.ko 2023-11-16T16:18:16.273 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-11-16T16:18:16.273 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/iscsi_tcp.ko 2023-11-16T16:18:16.274 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libfc 2023-11-16T16:18:16.274 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libfc/libfc.ko 2023-11-16T16:18:16.274 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libiscsi.ko 2023-11-16T16:18:16.274 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libiscsi_tcp.ko 2023-11-16T16:18:16.274 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libsas 2023-11-16T16:18:16.274 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libsas/libsas.ko 2023-11-16T16:18:16.274 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/lpfc 2023-11-16T16:18:16.274 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/lpfc/lpfc.ko 2023-11-16T16:18:16.274 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/megaraid 2023-11-16T16:18:16.274 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/megaraid.ko 2023-11-16T16:18:16.274 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-11-16T16:18:16.274 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-11-16T16:18:16.274 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-11-16T16:18:16.275 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/mpt3sas 2023-11-16T16:18:16.275 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-11-16T16:18:16.275 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/mvsas 2023-11-16T16:18:16.275 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/mvsas/mvsas.ko 2023-11-16T16:18:16.275 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/mvumi.ko 2023-11-16T16:18:16.275 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/pm8001 2023-11-16T16:18:16.275 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-11-16T16:18:16.275 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/pmcraid.ko 2023-11-16T16:18:16.275 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla1280.ko 2023-11-16T16:18:16.275 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla2xxx 2023-11-16T16:18:16.275 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-11-16T16:18:16.275 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-11-16T16:18:16.275 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla4xxx 2023-11-16T16:18:16.276 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-11-16T16:18:16.276 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/raid_class.ko 2023-11-16T16:18:16.276 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_debug.ko 2023-11-16T16:18:16.276 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_transport_fc.ko 2023-11-16T16:18:16.276 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-11-16T16:18:16.276 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_transport_sas.ko 2023-11-16T16:18:16.276 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_transport_spi.ko 2023-11-16T16:18:16.276 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_transport_srp.ko 2023-11-16T16:18:16.276 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/ses.ko 2023-11-16T16:18:16.276 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/st.ko 2023-11-16T16:18:16.276 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/stex.ko 2023-11-16T16:18:16.276 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/sym53c8xx_2 2023-11-16T16:18:16.277 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-11-16T16:18:16.277 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/virtio_scsi.ko 2023-11-16T16:18:16.277 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/vmw_pvscsi.ko 2023-11-16T16:18:16.277 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/wd719x.ko 2023-11-16T16:18:16.277 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ssb 2023-11-16T16:18:16.277 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ssb/ssb.ko 2023-11-16T16:18:16.277 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target 2023-11-16T16:18:16.277 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/iscsi 2023-11-16T16:18:16.277 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-11-16T16:18:16.277 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/loopback 2023-11-16T16:18:16.277 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/loopback/tcm_loop.ko 2023-11-16T16:18:16.277 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/target_core_file.ko 2023-11-16T16:18:16.277 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/target_core_iblock.ko 2023-11-16T16:18:16.278 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/target_core_mod.ko 2023-11-16T16:18:16.278 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/target_core_pscsi.ko 2023-11-16T16:18:16.278 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/target_core_user.ko 2023-11-16T16:18:16.278 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/tcm_fc 2023-11-16T16:18:16.278 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-11-16T16:18:16.278 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal 2023-11-16T16:18:16.278 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel 2023-11-16T16:18:16.278 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal 2023-11-16T16:18:16.278 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-11-16T16:18:16.278 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-11-16T16:18:16.278 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-11-16T16:18:16.278 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-11-16T16:18:16.278 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-11-16T16:18:16.279 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-11-16T16:18:16.279 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-11-16T16:18:16.279 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-11-16T16:18:16.279 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-11-16T16:18:16.279 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-11-16T16:18:16.279 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-11-16T16:18:16.279 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-11-16T16:18:16.279 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-11-16T16:18:16.279 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-11-16T16:18:16.279 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-11-16T16:18:16.279 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty 2023-11-16T16:18:16.279 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/ipwireless 2023-11-16T16:18:16.279 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-11-16T16:18:16.280 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/n_gsm.ko 2023-11-16T16:18:16.280 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/n_hdlc.ko 2023-11-16T16:18:16.280 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/nozomi.ko 2023-11-16T16:18:16.280 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial 2023-11-16T16:18:16.280 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial/8250 2023-11-16T16:18:16.280 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-11-16T16:18:16.280 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial/arc_uart.ko 2023-11-16T16:18:16.280 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial/jsm 2023-11-16T16:18:16.280 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial/jsm/jsm.ko 2023-11-16T16:18:16.280 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ufs 2023-11-16T16:18:16.280 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ufs/core 2023-11-16T16:18:16.280 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ufs/core/ufshcd-core.ko 2023-11-16T16:18:16.280 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ufs/host 2023-11-16T16:18:16.281 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-11-16T16:18:16.281 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio 2023-11-16T16:18:16.281 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio/uio.ko 2023-11-16T16:18:16.281 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio/uio_aec.ko 2023-11-16T16:18:16.281 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio/uio_cif.ko 2023-11-16T16:18:16.281 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio/uio_pci_generic.ko 2023-11-16T16:18:16.281 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio/uio_sercos3.ko 2023-11-16T16:18:16.281 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb 2023-11-16T16:18:16.281 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/class 2023-11-16T16:18:16.281 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/class/cdc-acm.ko 2023-11-16T16:18:16.281 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/class/cdc-wdm.ko 2023-11-16T16:18:16.281 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/class/usblp.ko 2023-11-16T16:18:16.281 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/class/usbtmc.ko 2023-11-16T16:18:16.281 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/host 2023-11-16T16:18:16.281 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/host/sl811-hcd.ko 2023-11-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/image 2023-11-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/image/mdc800.ko 2023-11-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/image/microtek.ko 2023-11-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc 2023-11-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/adutux.ko 2023-11-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/appledisplay.ko 2023-11-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/emi26.ko 2023-11-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/emi62.ko 2023-11-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/ezusb.ko 2023-11-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/idmouse.ko 2023-11-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/iowarrior.ko 2023-11-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/isight_firmware.ko 2023-11-16T16:18:16.282 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/ldusb.ko 2023-11-16T16:18:16.283 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/legousbtower.ko 2023-11-16T16:18:16.283 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/sisusbvga 2023-11-16T16:18:16.283 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-11-16T16:18:16.283 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/trancevibrator.ko 2023-11-16T16:18:16.283 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/usb3503.ko 2023-11-16T16:18:16.283 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/usblcd.ko 2023-11-16T16:18:16.283 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/usbsevseg.ko 2023-11-16T16:18:16.283 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/uss720.ko 2023-11-16T16:18:16.283 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/yurex.ko 2023-11-16T16:18:16.283 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/phy 2023-11-16T16:18:16.283 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/phy/phy-generic.ko 2023-11-16T16:18:16.283 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial 2023-11-16T16:18:16.283 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/aircable.ko 2023-11-16T16:18:16.283 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ark3116.ko 2023-11-16T16:18:16.284 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/belkin_sa.ko 2023-11-16T16:18:16.284 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ch341.ko 2023-11-16T16:18:16.284 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/cp210x.ko 2023-11-16T16:18:16.284 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/cyberjack.ko 2023-11-16T16:18:16.284 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/cypress_m8.ko 2023-11-16T16:18:16.284 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/digi_acceleport.ko 2023-11-16T16:18:16.284 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/empeg.ko 2023-11-16T16:18:16.284 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ftdi_sio.ko 2023-11-16T16:18:16.284 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/garmin_gps.ko 2023-11-16T16:18:16.284 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/io_edgeport.ko 2023-11-16T16:18:16.284 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/io_ti.ko 2023-11-16T16:18:16.284 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ipaq.ko 2023-11-16T16:18:16.284 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ipw.ko 2023-11-16T16:18:16.285 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ir-usb.ko 2023-11-16T16:18:16.285 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-11-16T16:18:16.285 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/keyspan.ko 2023-11-16T16:18:16.285 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/keyspan_pda.ko 2023-11-16T16:18:16.285 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/kl5kusb105.ko 2023-11-16T16:18:16.285 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/kobil_sct.ko 2023-11-16T16:18:16.285 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/mct_u232.ko 2023-11-16T16:18:16.285 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/mos7720.ko 2023-11-16T16:18:16.285 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/mos7840.ko 2023-11-16T16:18:16.285 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/navman.ko 2023-11-16T16:18:16.285 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/omninet.ko 2023-11-16T16:18:16.285 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/opticon.ko 2023-11-16T16:18:16.285 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/option.ko 2023-11-16T16:18:16.285 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/oti6858.ko 2023-11-16T16:18:16.286 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/pl2303.ko 2023-11-16T16:18:16.286 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/qcaux.ko 2023-11-16T16:18:16.286 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/qcserial.ko 2023-11-16T16:18:16.286 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/quatech2.ko 2023-11-16T16:18:16.286 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/safe_serial.ko 2023-11-16T16:18:16.286 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/sierra.ko 2023-11-16T16:18:16.286 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/spcp8x5.ko 2023-11-16T16:18:16.286 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ssu100.ko 2023-11-16T16:18:16.286 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/symbolserial.ko 2023-11-16T16:18:16.286 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-11-16T16:18:16.286 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-11-16T16:18:16.286 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/usb_debug.ko 2023-11-16T16:18:16.287 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/usb_wwan.ko 2023-11-16T16:18:16.287 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/visor.ko 2023-11-16T16:18:16.287 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/whiteheat.ko 2023-11-16T16:18:16.287 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/xsens_mt.ko 2023-11-16T16:18:16.287 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage 2023-11-16T16:18:16.287 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/uas.ko 2023-11-16T16:18:16.287 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-alauda.ko 2023-11-16T16:18:16.287 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-cypress.ko 2023-11-16T16:18:16.287 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-datafab.ko 2023-11-16T16:18:16.287 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-11-16T16:18:16.287 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-freecom.ko 2023-11-16T16:18:16.287 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-isd200.ko 2023-11-16T16:18:16.287 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-11-16T16:18:16.288 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-karma.ko 2023-11-16T16:18:16.288 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-onetouch.ko 2023-11-16T16:18:16.288 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-realtek.ko 2023-11-16T16:18:16.288 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-sddr09.ko 2023-11-16T16:18:16.288 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-sddr55.ko 2023-11-16T16:18:16.288 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-usbat.ko 2023-11-16T16:18:16.288 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/usb-storage.ko 2023-11-16T16:18:16.288 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/usbip 2023-11-16T16:18:16.288 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/usbip/usbip-core.ko 2023-11-16T16:18:16.288 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/usbip/usbip-host.ko 2023-11-16T16:18:16.288 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-11-16T16:18:16.288 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio 2023-11-16T16:18:16.288 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio/pci 2023-11-16T16:18:16.289 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-11-16T16:18:16.289 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio/pci/vfio-pci.ko 2023-11-16T16:18:16.289 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio/vfio.ko 2023-11-16T16:18:16.289 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-11-16T16:18:16.289 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vhost 2023-11-16T16:18:16.289 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vhost/vhost.ko 2023-11-16T16:18:16.289 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vhost/vhost_iotlb.ko 2023-11-16T16:18:16.289 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vhost/vhost_net.ko 2023-11-16T16:18:16.289 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vhost/vhost_scsi.ko 2023-11-16T16:18:16.289 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/video 2023-11-16T16:18:16.289 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/video/backlight 2023-11-16T16:18:16.289 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/video/backlight/apple_bl.ko 2023-11-16T16:18:16.290 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/video/backlight/lcd.ko 2023-11-16T16:18:16.290 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/video/backlight/platform_lcd.ko 2023-11-16T16:18:16.290 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio 2023-11-16T16:18:16.290 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio.ko 2023-11-16T16:18:16.290 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_balloon.ko 2023-11-16T16:18:16.290 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_mmio.ko 2023-11-16T16:18:16.290 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_pci.ko 2023-11-16T16:18:16.290 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-11-16T16:18:16.290 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-11-16T16:18:16.290 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_ring.ko 2023-11-16T16:18:16.290 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1 2023-11-16T16:18:16.290 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/masters 2023-11-16T16:18:16.291 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/masters/ds2482.ko 2023-11-16T16:18:16.291 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/masters/ds2490.ko 2023-11-16T16:18:16.291 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves 2023-11-16T16:18:16.291 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-11-16T16:18:16.291 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-11-16T16:18:16.291 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-11-16T16:18:16.291 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-11-16T16:18:16.291 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-11-16T16:18:16.291 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-11-16T16:18:16.291 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-11-16T16:18:16.291 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-11-16T16:18:16.291 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-11-16T16:18:16.291 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_smem.ko 2023-11-16T16:18:16.292 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_therm.ko 2023-11-16T16:18:16.292 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/wire.ko 2023-11-16T16:18:16.292 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog 2023-11-16T16:18:16.292 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/alim1535_wdt.ko 2023-11-16T16:18:16.292 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/alim7101_wdt.ko 2023-11-16T16:18:16.292 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/f71808e_wdt.ko 2023-11-16T16:18:16.292 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/hpwdt.ko 2023-11-16T16:18:16.292 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/i6300esb.ko 2023-11-16T16:18:16.292 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-11-16T16:18:16.292 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/iTCO_wdt.ko 2023-11-16T16:18:16.292 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/ib700wdt.ko 2023-11-16T16:18:16.292 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/ibmasr.ko 2023-11-16T16:18:16.292 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-11-16T16:18:16.293 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/it8712f_wdt.ko 2023-11-16T16:18:16.293 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/it87_wdt.ko 2023-11-16T16:18:16.293 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/machzwd.ko 2023-11-16T16:18:16.293 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/nv_tco.ko 2023-11-16T16:18:16.293 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/pcwd_pci.ko 2023-11-16T16:18:16.293 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/pcwd_usb.ko 2023-11-16T16:18:16.293 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-11-16T16:18:16.293 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/sch311x_wdt.ko 2023-11-16T16:18:16.293 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/softdog.ko 2023-11-16T16:18:16.293 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/sp5100_tco.ko 2023-11-16T16:18:16.293 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/via_wdt.ko 2023-11-16T16:18:16.293 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-11-16T16:18:16.293 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/w83877f_wdt.ko 2023-11-16T16:18:16.293 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/w83977f_wdt.ko 2023-11-16T16:18:16.294 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/wdt_pci.ko 2023-11-16T16:18:16.294 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/xen_wdt.ko 2023-11-16T16:18:16.294 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen 2023-11-16T16:18:16.294 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-acpi-processor.ko 2023-11-16T16:18:16.294 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-evtchn.ko 2023-11-16T16:18:16.294 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-gntalloc.ko 2023-11-16T16:18:16.294 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-gntdev.ko 2023-11-16T16:18:16.294 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-pciback 2023-11-16T16:18:16.294 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-11-16T16:18:16.294 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-privcmd.ko 2023-11-16T16:18:16.294 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xenfs 2023-11-16T16:18:16.294 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xenfs/xenfs.ko 2023-11-16T16:18:16.295 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs 2023-11-16T16:18:16.295 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/9p 2023-11-16T16:18:16.295 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/9p/9p.ko 2023-11-16T16:18:16.295 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/affs 2023-11-16T16:18:16.295 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/affs/affs.ko 2023-11-16T16:18:16.295 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/befs 2023-11-16T16:18:16.295 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/befs/befs.ko 2023-11-16T16:18:16.295 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/binfmt_misc.ko 2023-11-16T16:18:16.295 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/btrfs 2023-11-16T16:18:16.295 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/btrfs/btrfs.ko 2023-11-16T16:18:16.295 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/cachefiles 2023-11-16T16:18:16.295 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/cachefiles/cachefiles.ko 2023-11-16T16:18:16.296 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ceph 2023-11-16T16:18:16.296 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ceph/ceph.ko 2023-11-16T16:18:16.296 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/coda 2023-11-16T16:18:16.296 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/coda/coda.ko 2023-11-16T16:18:16.296 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/cramfs 2023-11-16T16:18:16.296 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/cramfs/cramfs.ko 2023-11-16T16:18:16.296 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/dlm 2023-11-16T16:18:16.296 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/dlm/dlm.ko 2023-11-16T16:18:16.296 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ecryptfs 2023-11-16T16:18:16.296 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ecryptfs/ecryptfs.ko 2023-11-16T16:18:16.296 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/f2fs 2023-11-16T16:18:16.296 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/f2fs/f2fs.ko 2023-11-16T16:18:16.296 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fat 2023-11-16T16:18:16.297 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fat/fat.ko 2023-11-16T16:18:16.297 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fat/msdos.ko 2023-11-16T16:18:16.297 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fat/vfat.ko 2023-11-16T16:18:16.297 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fscache 2023-11-16T16:18:16.297 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fscache/fscache.ko 2023-11-16T16:18:16.297 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fuse 2023-11-16T16:18:16.297 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fuse/cuse.ko 2023-11-16T16:18:16.297 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fuse/fuse.ko 2023-11-16T16:18:16.297 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/gfs2 2023-11-16T16:18:16.297 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/gfs2/gfs2.ko 2023-11-16T16:18:16.297 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/hfs 2023-11-16T16:18:16.297 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/hfs/hfs.ko 2023-11-16T16:18:16.298 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/hfsplus 2023-11-16T16:18:16.298 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/hfsplus/hfsplus.ko 2023-11-16T16:18:16.298 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/isofs 2023-11-16T16:18:16.298 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/isofs/isofs.ko 2023-11-16T16:18:16.298 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/jfs 2023-11-16T16:18:16.298 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/jfs/jfs.ko 2023-11-16T16:18:16.298 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/lockd 2023-11-16T16:18:16.298 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/lockd/lockd.ko 2023-11-16T16:18:16.298 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/minix 2023-11-16T16:18:16.298 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/minix/minix.ko 2023-11-16T16:18:16.298 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/netfs 2023-11-16T16:18:16.298 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/netfs/netfs.ko 2023-11-16T16:18:16.299 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs 2023-11-16T16:18:16.299 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/blocklayout 2023-11-16T16:18:16.299 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-11-16T16:18:16.299 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/filelayout 2023-11-16T16:18:16.299 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-11-16T16:18:16.299 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/flexfilelayout 2023-11-16T16:18:16.299 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-11-16T16:18:16.299 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/nfs.ko 2023-11-16T16:18:16.299 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/nfsv3.ko 2023-11-16T16:18:16.299 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/nfsv4.ko 2023-11-16T16:18:16.299 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs_common 2023-11-16T16:18:16.299 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs_common/grace.ko 2023-11-16T16:18:16.299 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs_common/nfs_acl.ko 2023-11-16T16:18:16.300 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfsd 2023-11-16T16:18:16.300 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfsd/nfsd.ko 2023-11-16T16:18:16.300 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nilfs2 2023-11-16T16:18:16.300 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nilfs2/nilfs2.ko 2023-11-16T16:18:16.300 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls 2023-11-16T16:18:16.300 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-celtic.ko 2023-11-16T16:18:16.300 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-centeuro.ko 2023-11-16T16:18:16.300 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-croatian.ko 2023-11-16T16:18:16.300 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-cyrillic.ko 2023-11-16T16:18:16.300 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-gaelic.ko 2023-11-16T16:18:16.300 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-greek.ko 2023-11-16T16:18:16.300 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-iceland.ko 2023-11-16T16:18:16.300 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-inuit.ko 2023-11-16T16:18:16.301 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-roman.ko 2023-11-16T16:18:16.301 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-romanian.ko 2023-11-16T16:18:16.301 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-turkish.ko 2023-11-16T16:18:16.301 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp1250.ko 2023-11-16T16:18:16.301 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp1251.ko 2023-11-16T16:18:16.301 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp1255.ko 2023-11-16T16:18:16.301 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp737.ko 2023-11-16T16:18:16.301 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp775.ko 2023-11-16T16:18:16.301 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp850.ko 2023-11-16T16:18:16.301 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp852.ko 2023-11-16T16:18:16.301 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp855.ko 2023-11-16T16:18:16.301 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp857.ko 2023-11-16T16:18:16.301 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp860.ko 2023-11-16T16:18:16.302 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp861.ko 2023-11-16T16:18:16.302 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp862.ko 2023-11-16T16:18:16.302 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp863.ko 2023-11-16T16:18:16.302 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp864.ko 2023-11-16T16:18:16.302 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp865.ko 2023-11-16T16:18:16.302 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp866.ko 2023-11-16T16:18:16.302 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp869.ko 2023-11-16T16:18:16.302 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp874.ko 2023-11-16T16:18:16.302 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp932.ko 2023-11-16T16:18:16.302 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp936.ko 2023-11-16T16:18:16.302 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp949.ko 2023-11-16T16:18:16.302 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp950.ko 2023-11-16T16:18:16.303 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_euc-jp.ko 2023-11-16T16:18:16.303 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-1.ko 2023-11-16T16:18:16.303 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-13.ko 2023-11-16T16:18:16.303 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-14.ko 2023-11-16T16:18:16.303 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-15.ko 2023-11-16T16:18:16.303 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-2.ko 2023-11-16T16:18:16.303 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-3.ko 2023-11-16T16:18:16.303 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-4.ko 2023-11-16T16:18:16.303 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-5.ko 2023-11-16T16:18:16.303 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-6.ko 2023-11-16T16:18:16.303 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-7.ko 2023-11-16T16:18:16.303 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-9.ko 2023-11-16T16:18:16.303 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_koi8-r.ko 2023-11-16T16:18:16.304 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_koi8-ru.ko 2023-11-16T16:18:16.304 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_koi8-u.ko 2023-11-16T16:18:16.304 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_ucs2_utils.ko 2023-11-16T16:18:16.304 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_utf8.ko 2023-11-16T16:18:16.304 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2 2023-11-16T16:18:16.304 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/cluster 2023-11-16T16:18:16.304 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-11-16T16:18:16.304 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/dlm 2023-11-16T16:18:16.304 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-11-16T16:18:16.304 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/dlmfs 2023-11-16T16:18:16.304 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-11-16T16:18:16.304 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/ocfs2.ko 2023-11-16T16:18:16.304 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-11-16T16:18:16.305 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-11-16T16:18:16.305 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-11-16T16:18:16.305 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/overlayfs 2023-11-16T16:18:16.305 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/overlayfs/overlay.ko 2023-11-16T16:18:16.305 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/pstore 2023-11-16T16:18:16.305 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/pstore/ramoops.ko 2023-11-16T16:18:16.305 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/reiserfs 2023-11-16T16:18:16.305 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/reiserfs/reiserfs.ko 2023-11-16T16:18:16.305 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/romfs 2023-11-16T16:18:16.305 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/romfs/romfs.ko 2023-11-16T16:18:16.305 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb 2023-11-16T16:18:16.305 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb/client 2023-11-16T16:18:16.305 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb/client/cifs.ko 2023-11-16T16:18:16.306 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb/common 2023-11-16T16:18:16.306 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb/common/cifs_arc4.ko 2023-11-16T16:18:16.306 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb/common/cifs_md4.ko 2023-11-16T16:18:16.306 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/squashfs 2023-11-16T16:18:16.306 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/squashfs/squashfs.ko 2023-11-16T16:18:16.306 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/sysv 2023-11-16T16:18:16.306 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/sysv/sysv.ko 2023-11-16T16:18:16.306 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/udf 2023-11-16T16:18:16.306 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/udf/udf.ko 2023-11-16T16:18:16.306 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ufs 2023-11-16T16:18:16.306 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ufs/ufs.ko 2023-11-16T16:18:16.306 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/xfs 2023-11-16T16:18:16.306 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/xfs/xfs.ko 2023-11-16T16:18:16.306 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel 2023-11-16T16:18:16.307 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel/rcu 2023-11-16T16:18:16.307 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel/rcu/rcutorture.ko 2023-11-16T16:18:16.307 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel/torture.ko 2023-11-16T16:18:16.307 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel/trace 2023-11-16T16:18:16.307 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-11-16T16:18:16.307 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib 2023-11-16T16:18:16.307 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/asn1_encoder.ko 2023-11-16T16:18:16.307 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/crc-itu-t.ko 2023-11-16T16:18:16.307 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/crc8.ko 2023-11-16T16:18:16.307 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/crypto 2023-11-16T16:18:16.307 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/crypto/libarc4.ko 2023-11-16T16:18:16.307 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/crypto/libdes.ko 2023-11-16T16:18:16.307 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/libcrc32c.ko 2023-11-16T16:18:16.308 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/lru_cache.ko 2023-11-16T16:18:16.308 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/lz4 2023-11-16T16:18:16.308 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/lz4/lz4_compress.ko 2023-11-16T16:18:16.308 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/lz4/lz4hc_compress.ko 2023-11-16T16:18:16.308 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/math 2023-11-16T16:18:16.308 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/math/cordic.ko 2023-11-16T16:18:16.308 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/raid6 2023-11-16T16:18:16.308 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/raid6/raid6_pq.ko 2023-11-16T16:18:16.308 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/reed_solomon 2023-11-16T16:18:16.308 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/reed_solomon/reed_solomon.ko 2023-11-16T16:18:16.308 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/ts_bm.ko 2023-11-16T16:18:16.308 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/ts_fsm.ko 2023-11-16T16:18:16.308 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/ts_kmp.ko 2023-11-16T16:18:16.309 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/zstd 2023-11-16T16:18:16.309 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/zstd/zstd_compress.ko 2023-11-16T16:18:16.309 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/mm 2023-11-16T16:18:16.309 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/mm/hwpoison-inject.ko 2023-11-16T16:18:16.309 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net 2023-11-16T16:18:16.309 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan 2023-11-16T16:18:16.309 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/6lowpan.ko 2023-11-16T16:18:16.309 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_dest.ko 2023-11-16T16:18:16.309 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_fragment.ko 2023-11-16T16:18:16.309 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_hop.ko 2023-11-16T16:18:16.309 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_ipv6.ko 2023-11-16T16:18:16.309 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_mobility.ko 2023-11-16T16:18:16.309 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_routing.ko 2023-11-16T16:18:16.310 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_udp.ko 2023-11-16T16:18:16.310 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802 2023-11-16T16:18:16.310 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802/garp.ko 2023-11-16T16:18:16.310 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802/mrp.ko 2023-11-16T16:18:16.310 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802/p8022.ko 2023-11-16T16:18:16.310 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802/psnap.ko 2023-11-16T16:18:16.310 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802/stp.ko 2023-11-16T16:18:16.310 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/8021q 2023-11-16T16:18:16.310 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/8021q/8021q.ko 2023-11-16T16:18:16.310 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/9p 2023-11-16T16:18:16.310 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/9p/9pnet.ko 2023-11-16T16:18:16.310 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/9p/9pnet_fd.ko 2023-11-16T16:18:16.310 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/9p/9pnet_rdma.ko 2023-11-16T16:18:16.311 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/9p/9pnet_virtio.ko 2023-11-16T16:18:16.311 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/batman-adv 2023-11-16T16:18:16.311 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/batman-adv/batman-adv.ko 2023-11-16T16:18:16.311 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge 2023-11-16T16:18:16.311 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/br_netfilter.ko 2023-11-16T16:18:16.311 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/bridge.ko 2023-11-16T16:18:16.311 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter 2023-11-16T16:18:16.311 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-11-16T16:18:16.311 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_among.ko 2023-11-16T16:18:16.311 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_arp.ko 2023-11-16T16:18:16.311 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-11-16T16:18:16.311 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-11-16T16:18:16.311 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_ip.ko 2023-11-16T16:18:16.312 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-11-16T16:18:16.312 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_limit.ko 2023-11-16T16:18:16.312 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_log.ko 2023-11-16T16:18:16.312 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_mark.ko 2023-11-16T16:18:16.312 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-11-16T16:18:16.312 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-11-16T16:18:16.312 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-11-16T16:18:16.312 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-11-16T16:18:16.312 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_snat.ko 2023-11-16T16:18:16.312 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_stp.ko 2023-11-16T16:18:16.312 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-11-16T16:18:16.312 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-11-16T16:18:16.312 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-11-16T16:18:16.313 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-11-16T16:18:16.313 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebtables.ko 2023-11-16T16:18:16.313 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-11-16T16:18:16.313 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-11-16T16:18:16.313 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-11-16T16:18:16.313 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ceph 2023-11-16T16:18:16.313 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ceph/libceph.ko 2023-11-16T16:18:16.313 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/core 2023-11-16T16:18:16.313 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/core/failover.ko 2023-11-16T16:18:16.313 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/core/pktgen.ko 2023-11-16T16:18:16.313 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dccp 2023-11-16T16:18:16.313 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dccp/dccp.ko 2023-11-16T16:18:16.313 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dccp/dccp_diag.ko 2023-11-16T16:18:16.314 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dccp/dccp_ipv4.ko 2023-11-16T16:18:16.314 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dccp/dccp_ipv6.ko 2023-11-16T16:18:16.314 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dns_resolver 2023-11-16T16:18:16.314 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dns_resolver/dns_resolver.ko 2023-11-16T16:18:16.314 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa 2023-11-16T16:18:16.314 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa/dsa_core.ko 2023-11-16T16:18:16.314 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa/tag_brcm.ko 2023-11-16T16:18:16.314 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa/tag_dsa.ko 2023-11-16T16:18:16.314 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa/tag_none.ko 2023-11-16T16:18:16.314 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa/tag_trailer.ko 2023-11-16T16:18:16.314 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ieee802154 2023-11-16T16:18:16.314 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ieee802154/6lowpan 2023-11-16T16:18:16.315 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-11-16T16:18:16.315 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ieee802154/ieee802154.ko 2023-11-16T16:18:16.315 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ieee802154/ieee802154_socket.ko 2023-11-16T16:18:16.315 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4 2023-11-16T16:18:16.315 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ah4.ko 2023-11-16T16:18:16.315 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/esp4.ko 2023-11-16T16:18:16.315 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/fou.ko 2023-11-16T16:18:16.315 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/gre.ko 2023-11-16T16:18:16.315 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/inet_diag.ko 2023-11-16T16:18:16.315 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ip_gre.ko 2023-11-16T16:18:16.315 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ip_tunnel.ko 2023-11-16T16:18:16.315 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ip_vti.ko 2023-11-16T16:18:16.315 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ipcomp.ko 2023-11-16T16:18:16.315 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ipip.ko 2023-11-16T16:18:16.315 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter 2023-11-16T16:18:16.316 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/arp_tables.ko 2023-11-16T16:18:16.316 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-11-16T16:18:16.316 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-11-16T16:18:16.316 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-11-16T16:18:16.316 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-11-16T16:18:16.316 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-11-16T16:18:16.316 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-11-16T16:18:16.316 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-11-16T16:18:16.316 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-11-16T16:18:16.316 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-11-16T16:18:16.316 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/iptable_security.ko 2023-11-16T16:18:16.316 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-11-16T16:18:16.316 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-11-16T16:18:16.317 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-11-16T16:18:16.317 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-11-16T16:18:16.317 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-11-16T16:18:16.317 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-11-16T16:18:16.317 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-11-16T16:18:16.317 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-11-16T16:18:16.317 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-11-16T16:18:16.317 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-11-16T16:18:16.317 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_bic.ko 2023-11-16T16:18:16.317 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_dctcp.ko 2023-11-16T16:18:16.317 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_diag.ko 2023-11-16T16:18:16.317 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_highspeed.ko 2023-11-16T16:18:16.317 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_htcp.ko 2023-11-16T16:18:16.318 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_hybla.ko 2023-11-16T16:18:16.318 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_illinois.ko 2023-11-16T16:18:16.318 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_lp.ko 2023-11-16T16:18:16.318 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_scalable.ko 2023-11-16T16:18:16.318 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_vegas.ko 2023-11-16T16:18:16.318 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_veno.ko 2023-11-16T16:18:16.318 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_westwood.ko 2023-11-16T16:18:16.318 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_yeah.ko 2023-11-16T16:18:16.318 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tunnel4.ko 2023-11-16T16:18:16.318 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/udp_diag.ko 2023-11-16T16:18:16.318 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/udp_tunnel.ko 2023-11-16T16:18:16.318 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/xfrm4_tunnel.ko 2023-11-16T16:18:16.319 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6 2023-11-16T16:18:16.319 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/ah6.ko 2023-11-16T16:18:16.319 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/esp6.ko 2023-11-16T16:18:16.319 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/fou6.ko 2023-11-16T16:18:16.319 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/ip6_tunnel.ko 2023-11-16T16:18:16.319 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-11-16T16:18:16.319 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/ip6_vti.ko 2023-11-16T16:18:16.319 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/ipcomp6.ko 2023-11-16T16:18:16.319 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter 2023-11-16T16:18:16.319 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-11-16T16:18:16.319 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-11-16T16:18:16.319 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-11-16T16:18:16.319 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-11-16T16:18:16.320 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-11-16T16:18:16.320 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-11-16T16:18:16.320 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-11-16T16:18:16.320 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-11-16T16:18:16.320 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-11-16T16:18:16.320 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-11-16T16:18:16.320 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-11-16T16:18:16.320 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-11-16T16:18:16.320 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-11-16T16:18:16.320 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-11-16T16:18:16.320 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-11-16T16:18:16.320 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-11-16T16:18:16.320 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-11-16T16:18:16.321 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-11-16T16:18:16.321 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-11-16T16:18:16.321 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-11-16T16:18:16.321 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-11-16T16:18:16.321 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-11-16T16:18:16.321 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-11-16T16:18:16.321 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-11-16T16:18:16.321 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/sit.ko 2023-11-16T16:18:16.321 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/tunnel6.ko 2023-11-16T16:18:16.321 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/xfrm6_tunnel.ko 2023-11-16T16:18:16.321 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/key 2023-11-16T16:18:16.321 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/key/af_key.ko 2023-11-16T16:18:16.321 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp 2023-11-16T16:18:16.321 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_core.ko 2023-11-16T16:18:16.322 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_debugfs.ko 2023-11-16T16:18:16.322 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_eth.ko 2023-11-16T16:18:16.322 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_ip.ko 2023-11-16T16:18:16.322 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_ip6.ko 2023-11-16T16:18:16.322 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_netlink.ko 2023-11-16T16:18:16.322 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_ppp.ko 2023-11-16T16:18:16.322 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/llc 2023-11-16T16:18:16.322 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/llc/llc.ko 2023-11-16T16:18:16.322 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/mac802154 2023-11-16T16:18:16.322 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/mac802154/mac802154.ko 2023-11-16T16:18:16.322 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/mpls 2023-11-16T16:18:16.322 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/mpls/mpls_gso.ko 2023-11-16T16:18:16.322 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter 2023-11-16T16:18:16.323 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset 2023-11-16T16:18:16.323 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set.ko 2023-11-16T16:18:16.323 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-11-16T16:18:16.323 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-11-16T16:18:16.323 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-11-16T16:18:16.323 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-11-16T16:18:16.323 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-11-16T16:18:16.323 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-11-16T16:18:16.323 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-11-16T16:18:16.323 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-11-16T16:18:16.323 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-11-16T16:18:16.323 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-11-16T16:18:16.323 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-11-16T16:18:16.324 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-11-16T16:18:16.324 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-11-16T16:18:16.324 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-11-16T16:18:16.324 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-11-16T16:18:16.324 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs 2023-11-16T16:18:16.324 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs.ko 2023-11-16T16:18:16.324 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-11-16T16:18:16.324 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-11-16T16:18:16.324 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-11-16T16:18:16.324 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-11-16T16:18:16.324 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-11-16T16:18:16.324 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-11-16T16:18:16.324 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-11-16T16:18:16.325 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-11-16T16:18:16.325 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-11-16T16:18:16.325 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-11-16T16:18:16.325 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-11-16T16:18:16.325 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-11-16T16:18:16.325 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-11-16T16:18:16.325 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conncount.ko 2023-11-16T16:18:16.325 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack.ko 2023-11-16T16:18:16.325 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-11-16T16:18:16.325 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-11-16T16:18:16.325 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-11-16T16:18:16.325 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_h323.ko 2023-11-16T16:18:16.325 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_irc.ko 2023-11-16T16:18:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-11-16T16:18:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-11-16T16:18:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-11-16T16:18:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_sane.ko 2023-11-16T16:18:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_sip.ko 2023-11-16T16:18:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-11-16T16:18:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-11-16T16:18:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_dup_netdev.ko 2023-11-16T16:18:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_flow_table.ko 2023-11-16T16:18:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_flow_table_inet.ko 2023-11-16T16:18:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_log_syslog.ko 2023-11-16T16:18:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat.ko 2023-11-16T16:18:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat_amanda.ko 2023-11-16T16:18:16.327 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat_ftp.ko 2023-11-16T16:18:16.327 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat_irc.ko 2023-11-16T16:18:16.327 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat_sip.ko 2023-11-16T16:18:16.327 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat_tftp.ko 2023-11-16T16:18:16.327 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_synproxy_core.ko 2023-11-16T16:18:16.327 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_tables.ko 2023-11-16T16:18:16.327 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nfnetlink.ko 2023-11-16T16:18:16.327 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nfnetlink_acct.ko 2023-11-16T16:18:16.327 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nfnetlink_log.ko 2023-11-16T16:18:16.327 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nfnetlink_osf.ko 2023-11-16T16:18:16.327 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nfnetlink_queue.ko 2023-11-16T16:18:16.327 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_chain_nat.ko 2023-11-16T16:18:16.327 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_compat.ko 2023-11-16T16:18:16.328 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_ct.ko 2023-11-16T16:18:16.328 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_dup_netdev.ko 2023-11-16T16:18:16.328 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_fib.ko 2023-11-16T16:18:16.328 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_fwd_netdev.ko 2023-11-16T16:18:16.328 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_hash.ko 2023-11-16T16:18:16.328 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_limit.ko 2023-11-16T16:18:16.328 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_log.ko 2023-11-16T16:18:16.328 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_masq.ko 2023-11-16T16:18:16.328 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_nat.ko 2023-11-16T16:18:16.328 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_numgen.ko 2023-11-16T16:18:16.328 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_queue.ko 2023-11-16T16:18:16.328 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_quota.ko 2023-11-16T16:18:16.328 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_redir.ko 2023-11-16T16:18:16.329 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_reject.ko 2023-11-16T16:18:16.329 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_reject_inet.ko 2023-11-16T16:18:16.329 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_socket.ko 2023-11-16T16:18:16.329 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_tunnel.ko 2023-11-16T16:18:16.329 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_AUDIT.ko 2023-11-16T16:18:16.329 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_CHECKSUM.ko 2023-11-16T16:18:16.329 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_CLASSIFY.ko 2023-11-16T16:18:16.329 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-11-16T16:18:16.329 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_CT.ko 2023-11-16T16:18:16.329 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_DSCP.ko 2023-11-16T16:18:16.329 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_HL.ko 2023-11-16T16:18:16.329 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_HMARK.ko 2023-11-16T16:18:16.329 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_IDLETIMER.ko 2023-11-16T16:18:16.330 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_LED.ko 2023-11-16T16:18:16.330 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_LOG.ko 2023-11-16T16:18:16.330 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_MASQUERADE.ko 2023-11-16T16:18:16.330 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_NETMAP.ko 2023-11-16T16:18:16.330 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_NFLOG.ko 2023-11-16T16:18:16.330 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_NFQUEUE.ko 2023-11-16T16:18:16.330 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_RATEEST.ko 2023-11-16T16:18:16.330 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_REDIRECT.ko 2023-11-16T16:18:16.330 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_SECMARK.ko 2023-11-16T16:18:16.330 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_TCPMSS.ko 2023-11-16T16:18:16.330 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-11-16T16:18:16.330 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_TEE.ko 2023-11-16T16:18:16.331 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_TPROXY.ko 2023-11-16T16:18:16.331 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_TRACE.ko 2023-11-16T16:18:16.331 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_addrtype.ko 2023-11-16T16:18:16.331 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_bpf.ko 2023-11-16T16:18:16.331 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_cgroup.ko 2023-11-16T16:18:16.331 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_cluster.ko 2023-11-16T16:18:16.331 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_comment.ko 2023-11-16T16:18:16.331 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_connbytes.ko 2023-11-16T16:18:16.331 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_connlabel.ko 2023-11-16T16:18:16.331 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_connlimit.ko 2023-11-16T16:18:16.331 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_connmark.ko 2023-11-16T16:18:16.331 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_conntrack.ko 2023-11-16T16:18:16.331 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_cpu.ko 2023-11-16T16:18:16.332 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_dccp.ko 2023-11-16T16:18:16.332 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_devgroup.ko 2023-11-16T16:18:16.332 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_dscp.ko 2023-11-16T16:18:16.332 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_ecn.ko 2023-11-16T16:18:16.332 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_esp.ko 2023-11-16T16:18:16.332 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_hashlimit.ko 2023-11-16T16:18:16.332 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_helper.ko 2023-11-16T16:18:16.332 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_hl.ko 2023-11-16T16:18:16.332 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_ipcomp.ko 2023-11-16T16:18:16.333 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_iprange.ko 2023-11-16T16:18:16.333 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_ipvs.ko 2023-11-16T16:18:16.333 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_l2tp.ko 2023-11-16T16:18:16.333 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_length.ko 2023-11-16T16:18:16.333 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_limit.ko 2023-11-16T16:18:16.333 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_mac.ko 2023-11-16T16:18:16.333 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_mark.ko 2023-11-16T16:18:16.333 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_multiport.ko 2023-11-16T16:18:16.333 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_nat.ko 2023-11-16T16:18:16.333 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_nfacct.ko 2023-11-16T16:18:16.333 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_osf.ko 2023-11-16T16:18:16.333 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_owner.ko 2023-11-16T16:18:16.333 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_physdev.ko 2023-11-16T16:18:16.334 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_pkttype.ko 2023-11-16T16:18:16.334 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_policy.ko 2023-11-16T16:18:16.334 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_quota.ko 2023-11-16T16:18:16.334 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_rateest.ko 2023-11-16T16:18:16.334 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_realm.ko 2023-11-16T16:18:16.334 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_recent.ko 2023-11-16T16:18:16.334 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_sctp.ko 2023-11-16T16:18:16.334 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_set.ko 2023-11-16T16:18:16.334 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_socket.ko 2023-11-16T16:18:16.334 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_state.ko 2023-11-16T16:18:16.334 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_statistic.ko 2023-11-16T16:18:16.334 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_string.ko 2023-11-16T16:18:16.335 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_tcpmss.ko 2023-11-16T16:18:16.335 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_time.ko 2023-11-16T16:18:16.335 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_u32.ko 2023-11-16T16:18:16.335 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netlink 2023-11-16T16:18:16.335 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netlink/netlink_diag.ko 2023-11-16T16:18:16.335 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/nsh 2023-11-16T16:18:16.335 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/nsh/nsh.ko 2023-11-16T16:18:16.335 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/openvswitch 2023-11-16T16:18:16.335 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/openvswitch/openvswitch.ko 2023-11-16T16:18:16.335 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/openvswitch/vport-geneve.ko 2023-11-16T16:18:16.335 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/openvswitch/vport-gre.ko 2023-11-16T16:18:16.335 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/openvswitch/vport-vxlan.ko 2023-11-16T16:18:16.336 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/packet 2023-11-16T16:18:16.336 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/packet/af_packet_diag.ko 2023-11-16T16:18:16.336 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/rds 2023-11-16T16:18:16.336 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/rds/rds.ko 2023-11-16T16:18:16.336 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/rds/rds_rdma.ko 2023-11-16T16:18:16.336 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/rds/rds_tcp.ko 2023-11-16T16:18:16.336 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched 2023-11-16T16:18:16.336 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_csum.ko 2023-11-16T16:18:16.336 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_gact.ko 2023-11-16T16:18:16.336 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_ipt.ko 2023-11-16T16:18:16.336 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_mirred.ko 2023-11-16T16:18:16.337 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_nat.ko 2023-11-16T16:18:16.337 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_pedit.ko 2023-11-16T16:18:16.337 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_police.ko 2023-11-16T16:18:16.337 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_simple.ko 2023-11-16T16:18:16.337 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_skbedit.ko 2023-11-16T16:18:16.337 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_vlan.ko 2023-11-16T16:18:16.337 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_basic.ko 2023-11-16T16:18:16.337 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_bpf.ko 2023-11-16T16:18:16.337 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_flow.ko 2023-11-16T16:18:16.337 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_fw.ko 2023-11-16T16:18:16.337 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_route.ko 2023-11-16T16:18:16.337 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_u32.ko 2023-11-16T16:18:16.338 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_cmp.ko 2023-11-16T16:18:16.338 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_ipset.ko 2023-11-16T16:18:16.338 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_meta.ko 2023-11-16T16:18:16.338 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_nbyte.ko 2023-11-16T16:18:16.338 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_text.ko 2023-11-16T16:18:16.338 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_u32.ko 2023-11-16T16:18:16.338 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_choke.ko 2023-11-16T16:18:16.338 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_codel.ko 2023-11-16T16:18:16.338 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_drr.ko 2023-11-16T16:18:16.338 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_fq.ko 2023-11-16T16:18:16.338 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_gred.ko 2023-11-16T16:18:16.338 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_hfsc.ko 2023-11-16T16:18:16.339 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_hhf.ko 2023-11-16T16:18:16.339 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_htb.ko 2023-11-16T16:18:16.339 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_ingress.ko 2023-11-16T16:18:16.339 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_mqprio.ko 2023-11-16T16:18:16.339 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_mqprio_lib.ko 2023-11-16T16:18:16.339 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_multiq.ko 2023-11-16T16:18:16.339 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_netem.ko 2023-11-16T16:18:16.339 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_pie.ko 2023-11-16T16:18:16.339 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_plug.ko 2023-11-16T16:18:16.339 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_prio.ko 2023-11-16T16:18:16.339 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_qfq.ko 2023-11-16T16:18:16.339 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_red.ko 2023-11-16T16:18:16.339 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_sfb.ko 2023-11-16T16:18:16.339 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_sfq.ko 2023-11-16T16:18:16.340 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_tbf.ko 2023-11-16T16:18:16.340 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_teql.ko 2023-11-16T16:18:16.340 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sctp 2023-11-16T16:18:16.340 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sctp/sctp.ko 2023-11-16T16:18:16.340 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sctp/sctp_diag.ko 2023-11-16T16:18:16.340 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc 2023-11-16T16:18:16.340 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/auth_gss 2023-11-16T16:18:16.340 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-11-16T16:18:16.340 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-11-16T16:18:16.340 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/sunrpc.ko 2023-11-16T16:18:16.340 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/xprtrdma 2023-11-16T16:18:16.340 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-11-16T16:18:16.341 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/tipc 2023-11-16T16:18:16.341 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/tipc/diag.ko 2023-11-16T16:18:16.341 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/tipc/tipc.ko 2023-11-16T16:18:16.341 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/unix 2023-11-16T16:18:16.341 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/unix/unix_diag.ko 2023-11-16T16:18:16.341 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock 2023-11-16T16:18:16.341 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-11-16T16:18:16.341 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-11-16T16:18:16.341 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock/vsock.ko 2023-11-16T16:18:16.341 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock/vsock_diag.ko 2023-11-16T16:18:16.341 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock/vsock_loopback.ko 2023-11-16T16:18:16.341 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/xfrm 2023-11-16T16:18:16.341 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/xfrm/xfrm_ipcomp.ko 2023-11-16T16:18:16.342 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security 2023-11-16T16:18:16.342 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security/keys 2023-11-16T16:18:16.342 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security/keys/encrypted-keys 2023-11-16T16:18:16.342 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-11-16T16:18:16.342 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security/keys/trusted-keys 2023-11-16T16:18:16.342 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security/keys/trusted-keys/trusted.ko 2023-11-16T16:18:16.342 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/virt 2023-11-16T16:18:16.342 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/virt/lib 2023-11-16T16:18:16.342 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/virt/lib/irqbypass.ko 2023-11-16T16:18:16.342 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.alias 2023-11-16T16:18:16.342 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.alias.bin 2023-11-16T16:18:16.342 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.builtin 2023-11-16T16:18:16.342 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.builtin.bin 2023-11-16T16:18:16.343 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.builtin.modinfo 2023-11-16T16:18:16.343 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.dep 2023-11-16T16:18:16.343 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.dep.bin 2023-11-16T16:18:16.343 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.devname 2023-11-16T16:18:16.343 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.order 2023-11-16T16:18:16.343 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.softdep 2023-11-16T16:18:16.343 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.symbols 2023-11-16T16:18:16.343 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.symbols.bin 2023-11-16T16:18:16.344 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:16.344 DEBUG:teuthology.orchestra.run.smithi044:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-11-16T16:18:16.372 INFO:teuthology.orchestra.run.smithi044.stdout:postinstall scriptlet (using /bin/sh): 2023-11-16T16:18:16.373 INFO:teuthology.orchestra.run.smithi044.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-rc5-g5b73517e0e2f -a -r /boot/System.map-6.6.0-rc5-g5b73517e0e2f ]; then 2023-11-16T16:18:16.373 INFO:teuthology.orchestra.run.smithi044.stdout:cp /boot/vmlinuz-6.6.0-rc5-g5b73517e0e2f /boot/.vmlinuz-6.6.0-rc5-g5b73517e0e2f-rpm 2023-11-16T16:18:16.373 INFO:teuthology.orchestra.run.smithi044.stdout:cp /boot/System.map-6.6.0-rc5-g5b73517e0e2f /boot/.System.map-6.6.0-rc5-g5b73517e0e2f-rpm 2023-11-16T16:18:16.373 INFO:teuthology.orchestra.run.smithi044.stdout:rm -f /boot/vmlinuz-6.6.0-rc5-g5b73517e0e2f /boot/System.map-6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:16.373 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/installkernel 6.6.0-rc5-g5b73517e0e2f /boot/.vmlinuz-6.6.0-rc5-g5b73517e0e2f-rpm /boot/.System.map-6.6.0-rc5-g5b73517e0e2f-rpm 2023-11-16T16:18:16.373 INFO:teuthology.orchestra.run.smithi044.stdout:rm -f /boot/.vmlinuz-6.6.0-rc5-g5b73517e0e2f-rpm /boot/.System.map-6.6.0-rc5-g5b73517e0e2f-rpm 2023-11-16T16:18:16.373 INFO:teuthology.orchestra.run.smithi044.stdout:fi 2023-11-16T16:18:16.373 INFO:teuthology.orchestra.run.smithi044.stdout:preuninstall scriptlet (using /bin/sh): 2023-11-16T16:18:16.373 INFO:teuthology.orchestra.run.smithi044.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-11-16T16:18:16.373 INFO:teuthology.orchestra.run.smithi044.stdout:new-kernel-pkg --remove 6.6.0-rc5-g5b73517e0e2f --rminitrd --initrdfile=/boot/initramfs-6.6.0-rc5-g5b73517e0e2f.img 2023-11-16T16:18:16.373 INFO:teuthology.orchestra.run.smithi044.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-11-16T16:18:16.373 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-install remove 6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:16.374 INFO:teuthology.orchestra.run.smithi044.stdout:fi 2023-11-16T16:18:16.374 INFO:teuthology.orchestra.run.smithi044.stdout:postuninstall scriptlet (using /bin/sh): 2023-11-16T16:18:16.374 INFO:teuthology.orchestra.run.smithi044.stdout:if [ -x /sbin/update-bootloader ]; then 2023-11-16T16:18:16.374 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/update-bootloader --remove 6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:16.374 INFO:teuthology.orchestra.run.smithi044.stdout:fi 2023-11-16T16:18:16.375 DEBUG:teuthology.orchestra.run.smithi044:> sudo rpm -qi grub2-tools 2023-11-16T16:18:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:Name : grub2-tools 2023-11-16T16:18:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:Epoch : 1 2023-11-16T16:18:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:Version : 2.02 2023-11-16T16:18:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:Release : 123.el8 2023-11-16T16:18:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:Architecture: x86_64 2023-11-16T16:18:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:Install Date: Thu 14 Jul 2022 06:22:19 PM UTC 2023-11-16T16:18:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:Group : System Environment/Base 2023-11-16T16:18:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:Size : 9265450 2023-11-16T16:18:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:License : GPLv3+ 2023-11-16T16:18:16.458 INFO:teuthology.orchestra.run.smithi044.stdout:Signature : RSA/SHA256, Thu 31 Mar 2022 07:24:35 PM UTC, Key ID 199e2f91fd431d51 2023-11-16T16:18:16.458 INFO:teuthology.orchestra.run.smithi044.stdout:Source RPM : grub2-2.02-123.el8.src.rpm 2023-11-16T16:18:16.458 INFO:teuthology.orchestra.run.smithi044.stdout:Build Date : Mon 28 Mar 2022 02:12:09 PM UTC 2023-11-16T16:18:16.458 INFO:teuthology.orchestra.run.smithi044.stdout:Build Host : x86-vm-09.build.eng.bos.redhat.com 2023-11-16T16:18:16.458 INFO:teuthology.orchestra.run.smithi044.stdout:Relocations : (not relocatable) 2023-11-16T16:18:16.458 INFO:teuthology.orchestra.run.smithi044.stdout:Packager : Red Hat, Inc. 2023-11-16T16:18:16.459 INFO:teuthology.orchestra.run.smithi044.stdout:Vendor : Red Hat, Inc. 2023-11-16T16:18:16.459 INFO:teuthology.orchestra.run.smithi044.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-16T16:18:16.459 INFO:teuthology.orchestra.run.smithi044.stdout:Summary : Support tools for GRUB. 2023-11-16T16:18:16.459 INFO:teuthology.orchestra.run.smithi044.stdout:Description : 2023-11-16T16:18:16.459 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:18:16.460 INFO:teuthology.orchestra.run.smithi044.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-16T16:18:16.460 INFO:teuthology.orchestra.run.smithi044.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-16T16:18:16.460 INFO:teuthology.orchestra.run.smithi044.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-16T16:18:16.460 INFO:teuthology.orchestra.run.smithi044.stdout:hardware devices. 2023-11-16T16:18:16.460 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:18:16.460 INFO:teuthology.orchestra.run.smithi044.stdout:This subpackage provides tools for support of all platforms. 2023-11-16T16:18:16.462 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-16T16:18:16.462 INFO:teuthology.task.kernel:Updating grub on smithi044 to boot 6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:16.462 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-16T16:18:17.695 INFO:teuthology.orchestra.run.smithi044.stderr:Generating grub configuration file ... 2023-11-16T16:18:19.557 INFO:teuthology.orchestra.run.smithi044.stderr:done 2023-11-16T16:18:19.559 DEBUG:teuthology.orchestra.run.smithi044:> mktemp 2023-11-16T16:18:19.576 INFO:teuthology.orchestra.run.smithi044.stdout:/tmp/tmp.3HiAkLft81 2023-11-16T16:18:19.576 DEBUG:teuthology.orchestra.run.smithi044:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3HiAkLft81 2023-11-16T16:18:19.648 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0666 /tmp/tmp.3HiAkLft81 2023-11-16T16:18:19.794 DEBUG:teuthology.orchestra.remote:smithi044:/tmp/tmp.3HiAkLft81 is 6KB 2023-11-16T16:18:19.843 DEBUG:teuthology.orchestra.run.smithi044:> rm -fr /tmp/tmp.3HiAkLft81 2023-11-16T16:18:19.860 DEBUG:teuthology.orchestra.run.smithi044:> sudo /bin/ls /boot/loader/entries || true 2023-11-16T16:18:19.925 INFO:teuthology.orchestra.run.smithi044.stdout:5a84457b7b52416e95d47d02e2e4a60f-0-rescue.conf 2023-11-16T16:18:19.925 INFO:teuthology.orchestra.run.smithi044.stdout:5a84457b7b52416e95d47d02e2e4a60f-4.18.0-372.9.1.el8.x86_64.conf 2023-11-16T16:18:19.925 INFO:teuthology.orchestra.run.smithi044.stdout:5a84457b7b52416e95d47d02e2e4a60f-6.6.0-rc5-g5b73517e0e2f.conf 2023-11-16T16:18:19.926 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-set-default 5a84457b7b52416e95d47d02e2e4a60f-6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:20.083 DEBUG:teuthology.orchestra.run.smithi044:> sudo shutdown -r now 2023-11-16T16:18:50.086 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-11-16T16:18:50.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2023-11-16T16:18:50.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:19:08.538 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.44 2023-11-16T16:19:17.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2023-11-16T16:19:17.541 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:20:17.542 DEBUG:teuthology.orchestra.remote:timed out 2023-11-16T16:20:29.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2023-11-16T16:20:29.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:20:29.910 DEBUG:teuthology.orchestra.run.smithi044:> true 2023-11-16T16:20:30.273 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2023-11-16T16:20:30.273 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-rc5-g5b73517e0e2f"... 2023-11-16T16:20:30.273 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2023-11-16T16:20:30.318 INFO:teuthology.orchestra.run.smithi044.stdout:6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:20:30.318 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-g5b73517e0e2f vs 6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:20:30.318 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-16T16:20:30.318 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-11-16T16:20:31.319 DEBUG:teuthology.parallel:result is None 2023-11-16T16:20:31.319 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-16T16:20:31.329 INFO:teuthology.task.internal:Creating test directory... 2023-11-16T16:20:31.330 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T16:20:31.334 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T16:20:31.337 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T16:20:31.378 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-16T16:20:31.384 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-16T16:20:31.394 INFO:teuthology.task.internal:Creating archive directory... 2023-11-16T16:20:31.394 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T16:20:31.422 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T16:20:31.425 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T16:20:31.482 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-16T16:20:31.489 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-16T16:20:31.489 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T16:20:31.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T16:20:31.511 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T16:20:31.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T16:20:31.530 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T16:20:31.534 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T16:20:31.553 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T16:20:31.621 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T16:20:31.641 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T16:20:31.651 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T16:20:31.691 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T16:20:31.696 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T16:20:31.716 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T16:20:31.718 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-16T16:20:31.725 INFO:teuthology.task.internal:Configuring sudo... 2023-11-16T16:20:31.725 DEBUG:teuthology.orchestra.run.smithi044:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T16:20:31.730 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T16:20:31.732 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T16:20:31.796 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-16T16:20:31.806 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-16T16:20:31.806 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T16:20:31.811 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T16:20:31.813 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T16:20:31.850 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T16:20:31.962 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T16:20:32.033 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T16:20:32.105 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T16:20:32.175 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:20:32.175 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T16:20:32.244 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T16:20:32.292 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T16:20:32.352 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T16:20:32.397 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T16:20:32.443 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-16T16:20:32.443 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T16:20:32.512 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T16:20:32.560 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T16:20:32.626 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T16:20:32.672 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T16:20:32.715 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-11-16T16:20:32.715 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T16:20:32.785 DEBUG:teuthology.orchestra.run.smithi044:> sudo service rsyslog restart 2023-11-16T16:20:32.789 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2023-11-16T16:20:32.792 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2023-11-16T16:20:32.857 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T16:20:32.879 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T16:20:32.906 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T16:20:33.240 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-16T16:20:33.248 INFO:teuthology.task.internal:Starting timer... 2023-11-16T16:20:33.248 INFO:teuthology.run_tasks:Running task pcp... 2023-11-16T16:20:33.258 INFO:teuthology.run_tasks:Running task selinux... 2023-11-16T16:20:33.266 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-16T16:20:33.266 DEBUG:teuthology.orchestra.run.smithi044:> sudo service auditd rotate 2023-11-16T16:20:33.387 INFO:teuthology.orchestra.run.smithi044.stdout:Rotating logs: 2023-11-16T16:20:33.392 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2023-11-16T16:20:33.447 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2023-11-16T16:20:33.449 DEBUG:teuthology.orchestra.run.smithi176:> sudo service auditd rotate 2023-11-16T16:20:33.512 INFO:teuthology.orchestra.run.smithi176.stdout:Rotating logs: 2023-11-16T16:20:33.514 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-16T16:20:33.515 DEBUG:teuthology.orchestra.run.smithi044:> /usr/sbin/getenforce 2023-11-16T16:20:33.577 INFO:teuthology.orchestra.run.smithi044.stdout:Permissive 2023-11-16T16:20:33.577 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2023-11-16T16:20:33.603 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2023-11-16T16:20:33.603 DEBUG:teuthology.orchestra.run.smithi176:> /usr/sbin/getenforce 2023-11-16T16:20:33.630 INFO:teuthology.orchestra.run.smithi176.stdout:Permissive 2023-11-16T16:20:33.630 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi044.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi077.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi176.front.sepia.ceph.com': 'permissive'} 2023-11-16T16:20:33.630 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T16:20:33.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T16:20:33.706 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T16:20:33.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T16:20:33.734 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T16:20:33.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T16:20:33.761 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-16T16:20:33.761 DEBUG:teuthology.orchestra.run.smithi044:> sudo /usr/sbin/setenforce permissive 2023-11-16T16:20:33.834 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2023-11-16T16:20:33.862 DEBUG:teuthology.orchestra.run.smithi176:> sudo /usr/sbin/setenforce permissive 2023-11-16T16:20:33.889 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-16T16:20:33.898 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-11-16T16:20:33.903 INFO:teuthology.repo_utils:Fetching main from origin 2023-11-16T16:20:34.078 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-16T16:20:34.095 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-16T16:20:34.096 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi044.front.sepia.ceph.com,smithi077.front.sepia.ceph.com,smithi176.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-16T16:34:35.976 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi044.front.sepia.ceph.com'), Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi176.front.sepia.ceph.com')] 2023-11-16T16:34:35.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2023-11-16T16:34:35.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:34:36.068 DEBUG:teuthology.orchestra.run.smithi044:> true 2023-11-16T16:34:36.177 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2023-11-16T16:34:36.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2023-11-16T16:34:36.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:34:36.244 DEBUG:teuthology.orchestra.run.smithi077:> true 2023-11-16T16:34:36.312 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2023-11-16T16:34:36.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2023-11-16T16:34:36.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:34:36.380 DEBUG:teuthology.orchestra.run.smithi176:> true 2023-11-16T16:34:36.449 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2023-11-16T16:34:36.450 INFO:teuthology.run_tasks:Running task clock... 2023-11-16T16:34:36.463 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-16T16:34:36.463 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T16:34:36.464 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T16:34:36.468 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T16:34:36.468 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T16:34:36.470 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T16:34:36.471 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T16:34:36.503 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-16T16:34:36.519 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-16T16:34:36.522 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-16T16:34:36.537 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-16T16:34:36.540 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-16T16:34:36.575 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-16T16:34:36.585 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2023-11-16T16:34:36.597 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2023-11-16T16:34:36.600 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: ntpd: command not found 2023-11-16T16:34:36.613 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-16T16:34:36.613 INFO:teuthology.orchestra.run.smithi176.stdout:506 Cannot talk to daemon 2023-11-16T16:34:36.627 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-16T16:34:36.628 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-16T16:34:36.642 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-16T16:34:36.673 INFO:teuthology.orchestra.run.smithi044.stderr:sudo: ntpd: command not found 2023-11-16T16:34:36.705 INFO:teuthology.orchestra.run.smithi044.stdout:506 Cannot talk to daemon 2023-11-16T16:34:36.710 INFO:teuthology.orchestra.run.smithi176.stderr:bash: ntpq: command not found 2023-11-16T16:34:36.713 INFO:teuthology.orchestra.run.smithi077.stderr:bash: ntpq: command not found 2023-11-16T16:34:36.714 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T16:34:36.714 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2023-11-16T16:34:36.714 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:36.715 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:36.715 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:36.715 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:36.717 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T16:34:36.717 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2023-11-16T16:34:36.717 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:36.717 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:36.717 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:36.717 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:36.740 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-16T16:34:36.776 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-16T16:34:36.871 INFO:teuthology.orchestra.run.smithi044.stderr:bash: ntpq: command not found 2023-11-16T16:34:36.878 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T16:34:36.878 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2023-11-16T16:34:36.878 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:36.879 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:36.879 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:36.879 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:36.880 INFO:teuthology.run_tasks:Running task pexec... 2023-11-16T16:34:36.889 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-16T16:34:36.889 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi044.front.sepia.ceph.com 2023-11-16T16:34:36.889 DEBUG:teuthology.orchestra.run.smithi044:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T16:34:36.890 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi077.front.sepia.ceph.com 2023-11-16T16:34:36.890 DEBUG:teuthology.orchestra.run.smithi077:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T16:34:36.890 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi176.front.sepia.ceph.com 2023-11-16T16:34:36.890 DEBUG:teuthology.orchestra.run.smithi176:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T16:34:37.260 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2023-11-16T16:34:37.263 INFO:teuthology.orchestra.run.smithi176.stdout:Updating Subscription Management repositories. 2023-11-16T16:34:37.407 INFO:teuthology.orchestra.run.smithi044.stdout:Updating Subscription Management repositories. 2023-11-16T16:34:38.879 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:05:23 ago on Thu 16 Nov 2023 04:29:15 PM UTC. 2023-11-16T16:34:38.916 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:05:20 ago on Thu 16 Nov 2023 04:29:18 PM UTC. 2023-11-16T16:34:39.172 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:10 ago on Thu 16 Nov 2023 04:32:28 PM UTC. 2023-11-16T16:34:40.604 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-16T16:34:40.605 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-16T16:34:40.605 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2023-11-16T16:34:40.605 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-16T16:34:40.605 INFO:teuthology.orchestra.run.smithi077.stdout:Resetting modules: 2023-11-16T16:34:40.605 INFO:teuthology.orchestra.run.smithi077.stdout: container-tools 2023-11-16T16:34:40.605 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-16T16:34:40.605 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-16T16:34:40.605 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-16T16:34:40.605 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-16T16:34:40.656 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-11-16T16:34:40.657 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-11-16T16:34:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2023-11-16T16:34:40.657 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-11-16T16:34:40.657 INFO:teuthology.orchestra.run.smithi176.stdout:Resetting modules: 2023-11-16T16:34:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: container-tools 2023-11-16T16:34:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-11-16T16:34:40.657 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-11-16T16:34:40.657 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-11-16T16:34:40.658 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-11-16T16:34:41.178 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2023-11-16T16:34:41.180 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2023-11-16T16:34:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: Package Architecture Version Repository Size 2023-11-16T16:34:41.180 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2023-11-16T16:34:41.180 INFO:teuthology.orchestra.run.smithi044.stdout:Resetting modules: 2023-11-16T16:34:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: container-tools 2023-11-16T16:34:41.180 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:34:41.180 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2023-11-16T16:34:41.180 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2023-11-16T16:34:41.181 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:34:41.577 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:05:26 ago on Thu 16 Nov 2023 04:29:15 PM UTC. 2023-11-16T16:34:41.686 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:05:23 ago on Thu 16 Nov 2023 04:29:18 PM UTC. 2023-11-16T16:34:42.468 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:13 ago on Thu 16 Nov 2023 04:32:28 PM UTC. 2023-11-16T16:34:48.738 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-11-16T16:34:49.159 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2023-11-16T16:34:49.202 INFO:teuthology.orchestra.run.smithi176.stdout:Updating Subscription Management repositories. 2023-11-16T16:34:49.298 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-16T16:34:49.786 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2023-11-16T16:34:49.798 INFO:teuthology.orchestra.run.smithi044.stdout:Updating Subscription Management repositories. 2023-11-16T16:34:50.783 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:05:32 ago on Thu 16 Nov 2023 04:29:18 PM UTC. 2023-11-16T16:34:51.381 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:05:35 ago on Thu 16 Nov 2023 04:29:15 PM UTC. 2023-11-16T16:34:51.468 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:22 ago on Thu 16 Nov 2023 04:32:28 PM UTC. 2023-11-16T16:34:52.502 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-11-16T16:34:52.594 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================================ 2023-11-16T16:34:52.594 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-11-16T16:34:52.594 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================================ 2023-11-16T16:34:52.594 INFO:teuthology.orchestra.run.smithi176.stdout:Installing group/module packages: 2023-11-16T16:34:52.594 INFO:teuthology.orchestra.run.smithi176.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-11-16T16:34:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-11-16T16:34:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-11-16T16:34:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-11-16T16:34:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-11-16T16:34:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-11-16T16:34:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-11-16T16:34:52.595 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-11-16T16:34:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-11-16T16:34:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-11-16T16:34:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-11-16T16:34:52.596 INFO:teuthology.orchestra.run.smithi176.stdout:Installing module profiles: 2023-11-16T16:34:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: container-tools/common 2023-11-16T16:34:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-11-16T16:34:52.596 INFO:teuthology.orchestra.run.smithi176.stdout:Enabling module streams: 2023-11-16T16:34:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: container-tools rhel8 2023-11-16T16:34:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-11-16T16:34:52.596 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-11-16T16:34:52.596 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================================ 2023-11-16T16:34:52.596 INFO:teuthology.orchestra.run.smithi176.stdout:Install 10 Packages 2023-11-16T16:34:52.597 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-11-16T16:34:52.597 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 18 M 2023-11-16T16:34:52.597 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 65 M 2023-11-16T16:34:52.597 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-11-16T16:34:52.796 INFO:teuthology.orchestra.run.smithi176.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 470 kB/s | 94 kB 00:00 2023-11-16T16:34:52.822 INFO:teuthology.orchestra.run.smithi176.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 181 kB/s | 41 kB 00:00 2023-11-16T16:34:52.847 INFO:teuthology.orchestra.run.smithi176.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 99 kB/s | 25 kB 00:00 2023-11-16T16:34:52.939 INFO:teuthology.orchestra.run.smithi176.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 347 kB/s | 49 kB 00:00 2023-11-16T16:34:53.022 INFO:teuthology.orchestra.run.smithi176.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 2.6 MB/s | 524 kB 00:00 2023-11-16T16:34:53.064 INFO:teuthology.orchestra.run.smithi176.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.2 MB/s | 149 kB 00:00 2023-11-16T16:34:53.156 INFO:teuthology.orchestra.run.smithi176.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.6 MB/s | 213 kB 00:00 2023-11-16T16:34:53.235 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-16T16:34:53.290 INFO:teuthology.orchestra.run.smithi176.stdout:(8/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 31 MB/s | 7.0 MB 00:00 2023-11-16T16:34:53.330 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================================ 2023-11-16T16:34:53.330 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-11-16T16:34:53.331 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================================ 2023-11-16T16:34:53.331 INFO:teuthology.orchestra.run.smithi077.stdout:Installing group/module packages: 2023-11-16T16:34:53.331 INFO:teuthology.orchestra.run.smithi077.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-11-16T16:34:53.331 INFO:teuthology.orchestra.run.smithi077.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-11-16T16:34:53.331 INFO:teuthology.orchestra.run.smithi077.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-11-16T16:34:53.331 INFO:teuthology.orchestra.run.smithi077.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-11-16T16:34:53.331 INFO:teuthology.orchestra.run.smithi077.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-11-16T16:34:53.331 INFO:teuthology.orchestra.run.smithi077.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-11-16T16:34:53.331 INFO:teuthology.orchestra.run.smithi077.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-11-16T16:34:53.332 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-11-16T16:34:53.332 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-11-16T16:34:53.332 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-11-16T16:34:53.332 INFO:teuthology.orchestra.run.smithi077.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-11-16T16:34:53.332 INFO:teuthology.orchestra.run.smithi077.stdout:Installing module profiles: 2023-11-16T16:34:53.332 INFO:teuthology.orchestra.run.smithi077.stdout: container-tools/common 2023-11-16T16:34:53.332 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-16T16:34:53.332 INFO:teuthology.orchestra.run.smithi077.stdout:Enabling module streams: 2023-11-16T16:34:53.333 INFO:teuthology.orchestra.run.smithi077.stdout: container-tools rhel8 2023-11-16T16:34:53.333 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-16T16:34:53.333 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-16T16:34:53.333 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================================ 2023-11-16T16:34:53.333 INFO:teuthology.orchestra.run.smithi077.stdout:Install 10 Packages 2023-11-16T16:34:53.333 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-16T16:34:53.334 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 18 M 2023-11-16T16:34:53.334 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 65 M 2023-11-16T16:34:53.334 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-16T16:34:53.382 INFO:teuthology.orchestra.run.smithi176.stdout:(9/10): buildah-1.26.2-1.module+el8.6.0+16771+2 15 MB/s | 8.0 MB 00:00 2023-11-16T16:34:53.423 INFO:teuthology.orchestra.run.smithi176.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 8.2 MB/s | 2.2 MB 00:00 2023-11-16T16:34:53.424 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-11-16T16:34:53.424 INFO:teuthology.orchestra.run.smithi176.stdout:Total 22 MB/s | 18 MB 00:00 2023-11-16T16:34:53.472 INFO:teuthology.orchestra.run.smithi077.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 675 kB/s | 94 kB 00:00 2023-11-16T16:34:53.488 INFO:teuthology.orchestra.run.smithi077.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 159 kB/s | 25 kB 00:00 2023-11-16T16:34:53.505 INFO:teuthology.orchestra.run.smithi077.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 236 kB/s | 41 kB 00:00 2023-11-16T16:34:53.506 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2023-11-16T16:34:53.613 INFO:teuthology.orchestra.run.smithi044.stdout:============================================================================================================ 2023-11-16T16:34:53.613 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2023-11-16T16:34:53.613 INFO:teuthology.orchestra.run.smithi044.stdout:============================================================================================================ 2023-11-16T16:34:53.614 INFO:teuthology.orchestra.run.smithi044.stdout:Installing group/module packages: 2023-11-16T16:34:53.614 INFO:teuthology.orchestra.run.smithi044.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-11-16T16:34:53.614 INFO:teuthology.orchestra.run.smithi044.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-11-16T16:34:53.614 INFO:teuthology.orchestra.run.smithi044.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-11-16T16:34:53.614 INFO:teuthology.orchestra.run.smithi044.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-11-16T16:34:53.614 INFO:teuthology.orchestra.run.smithi044.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-11-16T16:34:53.614 INFO:teuthology.orchestra.run.smithi044.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-11-16T16:34:53.614 INFO:teuthology.orchestra.run.smithi044.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-11-16T16:34:53.614 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2023-11-16T16:34:53.615 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-11-16T16:34:53.615 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-11-16T16:34:53.615 INFO:teuthology.orchestra.run.smithi044.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-11-16T16:34:53.615 INFO:teuthology.orchestra.run.smithi044.stdout:Installing module profiles: 2023-11-16T16:34:53.615 INFO:teuthology.orchestra.run.smithi044.stdout: container-tools/common 2023-11-16T16:34:53.615 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:34:53.615 INFO:teuthology.orchestra.run.smithi044.stdout:Enabling module streams: 2023-11-16T16:34:53.615 INFO:teuthology.orchestra.run.smithi044.stdout: container-tools rhel8 2023-11-16T16:34:53.615 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:34:53.615 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2023-11-16T16:34:53.615 INFO:teuthology.orchestra.run.smithi044.stdout:============================================================================================================ 2023-11-16T16:34:53.616 INFO:teuthology.orchestra.run.smithi044.stdout:Install 10 Packages 2023-11-16T16:34:53.616 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:34:53.616 INFO:teuthology.orchestra.run.smithi077.stdout:(4/10): cockpit-podman-49.1-1.module+el8.6.0+16 4.1 MB/s | 524 kB 00:00 2023-11-16T16:34:53.617 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 18 M 2023-11-16T16:34:53.617 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 65 M 2023-11-16T16:34:53.617 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2023-11-16T16:34:53.625 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-11-16T16:34:53.631 INFO:teuthology.orchestra.run.smithi077.stdout:(5/10): udica-0.2.6-4.module+el8.6.0+16771+28df 311 kB/s | 49 kB 00:00 2023-11-16T16:34:53.644 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-11-16T16:34:53.644 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-11-16T16:34:53.740 INFO:teuthology.orchestra.run.smithi077.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.2 MB/s | 149 kB 00:00 2023-11-16T16:34:53.757 INFO:teuthology.orchestra.run.smithi077.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.7 MB/s | 213 kB 00:00 2023-11-16T16:34:53.779 INFO:teuthology.orchestra.run.smithi044.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 578 kB/s | 94 kB 00:00 2023-11-16T16:34:53.827 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-11-16T16:34:53.830 INFO:teuthology.orchestra.run.smithi044.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 191 kB/s | 41 kB 00:00 2023-11-16T16:34:53.831 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-11-16T16:34:53.855 INFO:teuthology.orchestra.run.smithi044.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 104 kB/s | 25 kB 00:00 2023-11-16T16:34:53.866 INFO:teuthology.orchestra.run.smithi077.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 22 MB/s | 8.0 MB 00:00 2023-11-16T16:34:53.931 INFO:teuthology.orchestra.run.smithi044.stdout:(4/10): cockpit-podman-49.1-1.module+el8.6.0+16 5.1 MB/s | 524 kB 00:00 2023-11-16T16:34:53.941 INFO:teuthology.orchestra.run.smithi077.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 12 MB/s | 2.2 MB 00:00 2023-11-16T16:34:53.956 INFO:teuthology.orchestra.run.smithi044.stdout:(5/10): udica-0.2.6-4.module+el8.6.0+16771+28df 280 kB/s | 49 kB 00:00 2023-11-16T16:34:54.041 INFO:teuthology.orchestra.run.smithi077.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 23 MB/s | 7.0 MB 00:00 2023-11-16T16:34:54.041 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-16T16:34:54.041 INFO:teuthology.orchestra.run.smithi077.stdout:Total 26 MB/s | 18 MB 00:00 2023-11-16T16:34:54.056 INFO:teuthology.orchestra.run.smithi044.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.2 MB/s | 149 kB 00:00 2023-11-16T16:34:54.123 INFO:teuthology.orchestra.run.smithi044.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.2 MB/s | 213 kB 00:00 2023-11-16T16:34:54.242 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-16T16:34:54.261 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-16T16:34:54.261 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-16T16:34:54.274 INFO:teuthology.orchestra.run.smithi044.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 19 MB/s | 8.0 MB 00:00 2023-11-16T16:34:54.345 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-11-16T16:34:54.357 INFO:teuthology.orchestra.run.smithi044.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 9.4 MB/s | 2.2 MB 00:00 2023-11-16T16:34:54.443 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-16T16:34:54.447 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-16T16:34:54.491 INFO:teuthology.orchestra.run.smithi044.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 16 MB/s | 7.0 MB 00:00 2023-11-16T16:34:54.491 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2023-11-16T16:34:54.492 INFO:teuthology.orchestra.run.smithi044.stdout:Total 21 MB/s | 18 MB 00:00 2023-11-16T16:34:54.513 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-11-16T16:34:54.695 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T16:34:54.841 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2023-11-16T16:34:54.865 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2023-11-16T16:34:54.865 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2023-11-16T16:34:54.889 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-11-16T16:34:54.943 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-16T16:34:55.110 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-11-16T16:34:55.151 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-11-16T16:34:55.312 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T16:34:55.356 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2023-11-16T16:34:55.361 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2023-11-16T16:34:55.456 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-11-16T16:34:55.532 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-11-16T16:34:55.773 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-11-16T16:34:55.950 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2023-11-16T16:34:56.141 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-11-16T16:34:56.150 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-11-16T16:34:56.226 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-11-16T16:34:56.329 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T16:34:56.583 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-11-16T16:34:56.824 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-11-16T16:34:56.905 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-11-16T16:34:57.101 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-11-16T16:34:57.256 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-11-16T16:34:57.297 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-11-16T16:34:57.494 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-11-16T16:34:57.662 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T16:34:57.783 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-11-16T16:34:57.928 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T16:34:57.929 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-11-16T16:34:57.929 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T16:34:57.929 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-11-16T16:34:57.929 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-11-16T16:34:57.929 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-11-16T16:34:57.929 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-11-16T16:34:57.929 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-11-16T16:34:57.930 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-11-16T16:34:57.930 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-11-16T16:34:58.018 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-11-16T16:34:58.122 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-11-16T16:34:58.177 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-11-16T16:34:58.338 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T16:34:58.599 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T16:34:58.600 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-11-16T16:34:58.600 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T16:34:58.600 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-11-16T16:34:58.600 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-11-16T16:34:58.600 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-11-16T16:34:58.600 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-11-16T16:34:58.600 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-11-16T16:34:58.600 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-11-16T16:34:58.601 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-11-16T16:34:59.083 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-11-16T16:34:59.083 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:05:40 ago on Thu 16 Nov 2023 04:29:18 PM UTC. 2023-11-16T16:34:59.084 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-11-16T16:34:59.326 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-11-16T16:34:59.541 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-11-16T16:34:59.724 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T16:34:59.783 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-11-16T16:34:59.783 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:05:44 ago on Thu 16 Nov 2023 04:29:15 PM UTC. 2023-11-16T16:35:00.130 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T16:35:00.131 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-11-16T16:35:00.131 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T16:35:00.131 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-11-16T16:35:00.131 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-11-16T16:35:00.131 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-11-16T16:35:00.131 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-11-16T16:35:00.131 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-11-16T16:35:00.131 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-11-16T16:35:00.131 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-11-16T16:35:01.601 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-11-16T16:35:01.601 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:33 ago on Thu 16 Nov 2023 04:32:28 PM UTC. 2023-11-16T16:35:06.507 INFO:teuthology.orchestra.run.smithi077.stdout:Installed products updated. 2023-11-16T16:35:06.702 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-16T16:35:06.703 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-16T16:35:06.703 INFO:teuthology.orchestra.run.smithi077.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:06.703 INFO:teuthology.orchestra.run.smithi077.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T16:35:06.703 INFO:teuthology.orchestra.run.smithi077.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:06.703 INFO:teuthology.orchestra.run.smithi077.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-11-16T16:35:06.703 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-16T16:35:06.703 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-16T16:35:06.703 INFO:teuthology.orchestra.run.smithi077.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:06.703 INFO:teuthology.orchestra.run.smithi077.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:06.703 INFO:teuthology.orchestra.run.smithi077.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T16:35:06.703 INFO:teuthology.orchestra.run.smithi077.stdout: yajl-2.1.0-10.el8.x86_64 2023-11-16T16:35:06.703 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-16T16:35:06.703 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-16T16:35:06.852 DEBUG:teuthology.parallel:result is None 2023-11-16T16:35:08.799 INFO:teuthology.orchestra.run.smithi176.stdout:Installed products updated. 2023-11-16T16:35:09.000 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-11-16T16:35:09.000 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-11-16T16:35:09.000 INFO:teuthology.orchestra.run.smithi176.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:09.000 INFO:teuthology.orchestra.run.smithi176.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T16:35:09.000 INFO:teuthology.orchestra.run.smithi176.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:09.000 INFO:teuthology.orchestra.run.smithi176.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-11-16T16:35:09.001 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-16T16:35:09.001 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-16T16:35:09.001 INFO:teuthology.orchestra.run.smithi176.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:09.001 INFO:teuthology.orchestra.run.smithi176.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:09.001 INFO:teuthology.orchestra.run.smithi176.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T16:35:09.001 INFO:teuthology.orchestra.run.smithi176.stdout: yajl-2.1.0-10.el8.x86_64 2023-11-16T16:35:09.001 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-11-16T16:35:09.001 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-11-16T16:35:09.143 DEBUG:teuthology.parallel:result is None 2023-11-16T16:35:09.640 INFO:teuthology.orchestra.run.smithi044.stdout:Installed products updated. 2023-11-16T16:35:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:35:09.864 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2023-11-16T16:35:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T16:35:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-11-16T16:35:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-16T16:35:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-16T16:35:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T16:35:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: yajl-2.1.0-10.el8.x86_64 2023-11-16T16:35:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:35:09.865 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2023-11-16T16:35:10.134 DEBUG:teuthology.parallel:result is None 2023-11-16T16:35:10.134 INFO:teuthology.run_tasks:Running task install... 2023-11-16T16:35:10.144 DEBUG:teuthology.task.install:project ceph 2023-11-16T16:35:10.145 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207'}} 2023-11-16T16:35:10.145 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207'} 2023-11-16T16:35:10.145 INFO:teuthology.task.install:Using flavor: default 2023-11-16T16:35:10.151 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-16T16:35:10.152 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-11-16T16:35:10.152 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207', 'tag': None, 'wait_for_package': False} 2023-11-16T16:35:10.153 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:35:10.156 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207', 'tag': None, 'wait_for_package': False} 2023-11-16T16:35:10.156 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:35:10.157 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207', 'tag': None, 'wait_for_package': False} 2023-11-16T16:35:10.157 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:35:10.303 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/ 2023-11-16T16:35:10.303 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3905.g5e4a743a 2023-11-16T16:35:10.422 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/ 2023-11-16T16:35:10.422 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3905.g5e4a743a 2023-11-16T16:35:10.427 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/ 2023-11-16T16:35:10.427 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3905.g5e4a743a 2023-11-16T16:35:10.609 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-16T16:35:10.610 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-11-16T16:35:10.610 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-16T16:35:10.637 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-16T16:35:10.637 DEBUG:teuthology.orchestra.run.smithi176:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e4a743ad7fd1422ebd6c4dc950a21a303279207/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-16T16:35:10.704 DEBUG:teuthology.orchestra.run.smithi176:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-16T16:35:10.740 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-16T16:35:10.740 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-16T16:35:10.741 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-16T16:35:10.768 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-16T16:35:10.768 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e4a743ad7fd1422ebd6c4dc950a21a303279207/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-16T16:35:10.782 DEBUG:teuthology.orchestra.run.smithi176:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-16T16:35:10.825 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-16T16:35:10.826 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:35:10.826 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-16T16:35:10.835 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-16T16:35:10.848 INFO:teuthology.orchestra.run.smithi176.stdout:check_obsoletes = 1 2023-11-16T16:35:10.849 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum clean all 2023-11-16T16:35:10.898 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-16T16:35:10.898 DEBUG:teuthology.orchestra.run.smithi044:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e4a743ad7fd1422ebd6c4dc950a21a303279207/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-16T16:35:10.912 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-16T16:35:10.978 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2023-11-16T16:35:10.980 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2023-11-16T16:35:11.011 DEBUG:teuthology.orchestra.run.smithi044:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-16T16:35:11.112 DEBUG:teuthology.orchestra.run.smithi044:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-16T16:35:11.184 INFO:teuthology.orchestra.run.smithi044.stdout:check_obsoletes = 1 2023-11-16T16:35:11.188 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum clean all 2023-11-16T16:35:11.235 INFO:teuthology.orchestra.run.smithi176.stdout:Updating Subscription Management repositories. 2023-11-16T16:35:11.365 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2023-11-16T16:35:11.680 INFO:teuthology.orchestra.run.smithi044.stdout:Updating Subscription Management repositories. 2023-11-16T16:35:12.285 INFO:teuthology.orchestra.run.smithi176.stdout:44 files removed 2023-11-16T16:35:12.331 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-16T16:35:12.339 INFO:teuthology.orchestra.run.smithi077.stdout:44 files removed 2023-11-16T16:35:12.386 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-16T16:35:12.616 INFO:teuthology.orchestra.run.smithi044.stdout:44 files removed 2023-11-16T16:35:12.662 INFO:teuthology.orchestra.run.smithi176.stdout:Updating Subscription Management repositories. 2023-11-16T16:35:12.691 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-16T16:35:12.732 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2023-11-16T16:35:13.187 INFO:teuthology.orchestra.run.smithi044.stdout:Updating Subscription Management repositories. 2023-11-16T16:35:14.496 INFO:teuthology.orchestra.run.smithi176.stdout:ceph packages for x86_64 106 kB/s | 80 kB 00:00 2023-11-16T16:35:14.643 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 96 kB/s | 80 kB 00:00 2023-11-16T16:35:15.221 INFO:teuthology.orchestra.run.smithi176.stdout:ceph noarch packages 14 kB/s | 9.4 kB 00:00 2023-11-16T16:35:15.270 INFO:teuthology.orchestra.run.smithi044.stdout:ceph packages for x86_64 80 kB/s | 80 kB 00:01 2023-11-16T16:35:15.560 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 11 kB/s | 9.4 kB 00:00 2023-11-16T16:35:15.980 INFO:teuthology.orchestra.run.smithi176.stdout:ceph source packages 2.0 kB/s | 1.5 kB 00:00 2023-11-16T16:35:16.013 INFO:teuthology.orchestra.run.smithi044.stdout:ceph noarch packages 13 kB/s | 9.4 kB 00:00 2023-11-16T16:35:16.272 INFO:teuthology.orchestra.run.smithi176.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2023-11-16T16:35:16.277 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 2.1 kB/s | 1.5 kB 00:00 2023-11-16T16:35:16.420 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-11-16T16:35:16.606 INFO:teuthology.orchestra.run.smithi044.stdout:ceph source packages 2.6 kB/s | 1.5 kB 00:00 2023-11-16T16:35:16.756 INFO:teuthology.orchestra.run.smithi044.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-11-16T16:35:18.258 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-11-16T16:35:18.425 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 16 MB 00:02 2023-11-16T16:35:19.562 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 5.1 MB/s | 16 MB 00:03 2023-11-16T16:35:21.939 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 448 kB/s | 24 kB 00:00 2023-11-16T16:35:22.895 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 410 kB/s | 24 kB 00:00 2023-11-16T16:35:23.112 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 390 kB/s | 24 kB 00:00 2023-11-16T16:35:23.150 INFO:teuthology.orchestra.run.smithi176.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 42 MB/s | 47 MB 00:01 2023-11-16T16:35:24.506 INFO:teuthology.orchestra.run.smithi077.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 40 MB/s | 53 MB 00:01 2023-11-16T16:35:24.531 INFO:teuthology.orchestra.run.smithi044.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 MB/s | 47 MB 00:01 2023-11-16T16:35:31.551 INFO:teuthology.orchestra.run.smithi176.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 39 MB/s | 53 MB 00:01 2023-11-16T16:35:32.867 INFO:teuthology.orchestra.run.smithi077.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 MB/s | 47 MB 00:01 2023-11-16T16:35:34.513 INFO:teuthology.orchestra.run.smithi044.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 37 MB/s | 53 MB 00:01 2023-11-16T16:35:42.724 INFO:teuthology.orchestra.run.smithi176.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T16:35:42.724 INFO:teuthology.orchestra.run.smithi176.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T16:35:42.739 INFO:teuthology.orchestra.run.smithi176.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-16T16:35:42.740 INFO:teuthology.orchestra.run.smithi176.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-16T16:35:42.741 INFO:teuthology.orchestra.run.smithi176.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-16T16:35:42.742 INFO:teuthology.orchestra.run.smithi176.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-16T16:35:42.744 INFO:teuthology.orchestra.run.smithi176.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-16T16:35:42.747 INFO:teuthology.orchestra.run.smithi176.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-16T16:35:42.748 INFO:teuthology.orchestra.run.smithi176.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-16T16:35:42.884 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-11-16T16:35:42.896 INFO:teuthology.orchestra.run.smithi176.stdout:=========================================================================================================================== 2023-11-16T16:35:42.896 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-11-16T16:35:42.896 INFO:teuthology.orchestra.run.smithi176.stdout:=========================================================================================================================== 2023-11-16T16:35:42.896 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-11-16T16:35:42.897 INFO:teuthology.orchestra.run.smithi176.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-16T16:35:42.897 INFO:teuthology.orchestra.run.smithi176.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-16T16:35:42.897 INFO:teuthology.orchestra.run.smithi176.stdout: ceph x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.4 k 2023-11-16T16:35:42.897 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 5.3 M 2023-11-16T16:35:42.897 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 847 k 2023-11-16T16:35:42.897 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 165 k 2023-11-16T16:35:42.897 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 1.5 M 2023-11-16T16:35:42.897 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 127 k 2023-11-16T16:35:42.897 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 1.7 M 2023-11-16T16:35:42.897 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 7.4 M 2023-11-16T16:35:42.897 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 47 k 2023-11-16T16:35:42.898 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 13 M 2023-11-16T16:35:42.898 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 41 M 2023-11-16T16:35:42.898 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-volume noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 265 k 2023-11-16T16:35:42.898 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 83 k 2023-11-16T16:35:42.898 INFO:teuthology.orchestra.run.smithi176.stdout: cephfs-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 258 k 2023-11-16T16:35:42.898 INFO:teuthology.orchestra.run.smithi176.stdout: cephfs-top noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 18 k 2023-11-16T16:35:42.898 INFO:teuthology.orchestra.run.smithi176.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-16T16:35:42.898 INFO:teuthology.orchestra.run.smithi176.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-16T16:35:42.898 INFO:teuthology.orchestra.run.smithi176.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-16T16:35:42.898 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 31 k 2023-11-16T16:35:42.898 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 771 k 2023-11-16T16:35:42.899 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 139 k 2023-11-16T16:35:42.899 INFO:teuthology.orchestra.run.smithi176.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-16T16:35:42.899 INFO:teuthology.orchestra.run.smithi176.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-16T16:35:42.899 INFO:teuthology.orchestra.run.smithi176.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-16T16:35:42.899 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-16T16:35:42.899 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-16T16:35:42.899 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-16T16:35:42.899 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 197 k 2023-11-16T16:35:42.899 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 371 k 2023-11-16T16:35:42.899 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 365 k 2023-11-16T16:35:42.899 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 107 k 2023-11-16T16:35:42.900 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 94 k 2023-11-16T16:35:42.900 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.4 M 2023-11-16T16:35:42.900 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 180 k 2023-11-16T16:35:42.900 INFO:teuthology.orchestra.run.smithi176.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-16T16:35:42.900 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrading: 2023-11-16T16:35:42.900 INFO:teuthology.orchestra.run.smithi176.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-16T16:35:42.900 INFO:teuthology.orchestra.run.smithi176.stdout: librados2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.7 M 2023-11-16T16:35:42.900 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.6 M 2023-11-16T16:35:42.900 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-11-16T16:35:42.900 INFO:teuthology.orchestra.run.smithi176.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-16T16:35:42.900 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 22 M 2023-11-16T16:35:42.901 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 23 k 2023-11-16T16:35:42.901 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 2.3 M 2023-11-16T16:35:42.901 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 240 k 2023-11-16T16:35:42.901 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 4.6 M 2023-11-16T16:35:42.901 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 18 M 2023-11-16T16:35:42.901 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 15 k 2023-11-16T16:35:42.901 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 24 k 2023-11-16T16:35:42.901 INFO:teuthology.orchestra.run.smithi176.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-16T16:35:42.901 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-16T16:35:42.901 INFO:teuthology.orchestra.run.smithi176.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-16T16:35:42.901 INFO:teuthology.orchestra.run.smithi176.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-16T16:35:42.901 INFO:teuthology.orchestra.run.smithi176.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-16T16:35:42.902 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 137 k 2023-11-16T16:35:42.902 INFO:teuthology.orchestra.run.smithi176.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-16T16:35:42.902 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-16T16:35:42.902 INFO:teuthology.orchestra.run.smithi176.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-16T16:35:42.902 INFO:teuthology.orchestra.run.smithi176.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-16T16:35:42.902 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-16T16:35:42.902 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-16T16:35:42.902 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 473 k 2023-11-16T16:35:42.902 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-16T16:35:42.902 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.5 M 2023-11-16T16:35:42.903 INFO:teuthology.orchestra.run.smithi176.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-16T16:35:42.903 INFO:teuthology.orchestra.run.smithi176.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-16T16:35:42.903 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-16T16:35:42.903 INFO:teuthology.orchestra.run.smithi176.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-16T16:35:42.903 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-16T16:35:42.903 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-16T16:35:42.903 INFO:teuthology.orchestra.run.smithi176.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-16T16:35:42.903 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-16T16:35:42.903 INFO:teuthology.orchestra.run.smithi176.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-16T16:35:42.903 INFO:teuthology.orchestra.run.smithi176.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-16T16:35:42.903 INFO:teuthology.orchestra.run.smithi176.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-16T16:35:42.903 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-16T16:35:42.903 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-16T16:35:42.904 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-16T16:35:42.904 INFO:teuthology.orchestra.run.smithi176.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-16T16:35:42.904 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-16T16:35:42.904 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 45 k 2023-11-16T16:35:42.904 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 119 k 2023-11-16T16:35:42.904 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-16T16:35:42.904 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-16T16:35:42.904 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-16T16:35:42.904 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-16T16:35:42.904 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-16T16:35:42.904 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-16T16:35:42.904 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-16T16:35:42.905 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-16T16:35:42.905 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-16T16:35:42.905 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-16T16:35:42.905 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-16T16:35:42.905 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T16:35:42.905 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-16T16:35:42.905 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-16T16:35:42.905 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-16T16:35:42.905 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-16T16:35:42.905 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-16T16:35:42.905 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-16T16:35:42.905 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-16T16:35:42.906 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-16T16:35:42.906 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-16T16:35:42.906 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-16T16:35:42.906 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-16T16:35:42.906 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-16T16:35:42.906 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-16T16:35:42.906 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-16T16:35:42.906 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-16T16:35:42.906 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-16T16:35:42.906 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-16T16:35:42.906 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-16T16:35:42.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T16:35:42.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-16T16:35:42.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-16T16:35:42.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-16T16:35:42.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-16T16:35:42.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-16T16:35:42.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-16T16:35:42.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-16T16:35:42.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-16T16:35:42.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-16T16:35:42.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-16T16:35:42.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-16T16:35:42.908 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-16T16:35:42.908 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-16T16:35:42.908 INFO:teuthology.orchestra.run.smithi176.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-16T16:35:42.908 INFO:teuthology.orchestra.run.smithi176.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-16T16:35:42.908 INFO:teuthology.orchestra.run.smithi176.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-16T16:35:42.908 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-16T16:35:42.908 INFO:teuthology.orchestra.run.smithi176.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-16T16:35:42.908 INFO:teuthology.orchestra.run.smithi176.stdout:Enabling module streams: 2023-11-16T16:35:42.908 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql 10 2023-11-16T16:35:42.908 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-11-16T16:35:42.908 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-11-16T16:35:42.908 INFO:teuthology.orchestra.run.smithi176.stdout:=========================================================================================================================== 2023-11-16T16:35:42.909 INFO:teuthology.orchestra.run.smithi176.stdout:Install 128 Packages 2023-11-16T16:35:42.909 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrade 3 Packages 2023-11-16T16:35:42.909 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-11-16T16:35:42.909 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 197 M 2023-11-16T16:35:42.909 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-11-16T16:35:43.159 INFO:teuthology.orchestra.run.smithi176.stdout:(1/131): ceph-18.0.0-3905.g5e4a743a.el8.x86_64. 34 kB/s | 6.4 kB 00:00 2023-11-16T16:35:43.501 INFO:teuthology.orchestra.run.smithi176.stdout:(2/131): ceph-fuse-18.0.0-3905.g5e4a743a.el8.x8 2.4 MB/s | 847 kB 00:00 2023-11-16T16:35:43.585 INFO:teuthology.orchestra.run.smithi176.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 1.9 MB/s | 165 kB 00:00 2023-11-16T16:35:43.768 INFO:teuthology.orchestra.run.smithi176.stdout:(4/131): ceph-base-18.0.0-3905.g5e4a743a.el8.x8 6.7 MB/s | 5.3 MB 00:00 2023-11-16T16:35:43.882 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T16:35:43.882 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T16:35:43.896 INFO:teuthology.orchestra.run.smithi077.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-16T16:35:43.898 INFO:teuthology.orchestra.run.smithi077.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-16T16:35:43.899 INFO:teuthology.orchestra.run.smithi077.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-16T16:35:43.900 INFO:teuthology.orchestra.run.smithi077.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-16T16:35:43.902 INFO:teuthology.orchestra.run.smithi077.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-16T16:35:43.905 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-16T16:35:43.906 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-16T16:35:44.036 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-16T16:35:44.044 INFO:teuthology.orchestra.run.smithi176.stdout:(5/131): ceph-mgr-18.0.0-3905.g5e4a743a.el8.x86 5.6 MB/s | 1.5 MB 00:00 2023-11-16T16:35:44.048 INFO:teuthology.orchestra.run.smithi077.stdout:=========================================================================================================================== 2023-11-16T16:35:44.048 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-11-16T16:35:44.049 INFO:teuthology.orchestra.run.smithi077.stdout:=========================================================================================================================== 2023-11-16T16:35:44.049 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-16T16:35:44.049 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-16T16:35:44.049 INFO:teuthology.orchestra.run.smithi077.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-16T16:35:44.049 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.4 k 2023-11-16T16:35:44.049 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 5.3 M 2023-11-16T16:35:44.049 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 847 k 2023-11-16T16:35:44.049 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 165 k 2023-11-16T16:35:44.049 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 1.5 M 2023-11-16T16:35:44.050 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 127 k 2023-11-16T16:35:44.050 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 1.7 M 2023-11-16T16:35:44.050 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 7.4 M 2023-11-16T16:35:44.050 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 47 k 2023-11-16T16:35:44.050 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 13 M 2023-11-16T16:35:44.050 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 41 M 2023-11-16T16:35:44.050 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 265 k 2023-11-16T16:35:44.050 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 83 k 2023-11-16T16:35:44.050 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 258 k 2023-11-16T16:35:44.050 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 18 k 2023-11-16T16:35:44.050 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-16T16:35:44.051 INFO:teuthology.orchestra.run.smithi077.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-16T16:35:44.051 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-16T16:35:44.051 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 31 k 2023-11-16T16:35:44.051 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 771 k 2023-11-16T16:35:44.051 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 139 k 2023-11-16T16:35:44.051 INFO:teuthology.orchestra.run.smithi077.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-16T16:35:44.051 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-16T16:35:44.051 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-16T16:35:44.051 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-16T16:35:44.051 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-16T16:35:44.051 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-16T16:35:44.052 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 197 k 2023-11-16T16:35:44.052 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 371 k 2023-11-16T16:35:44.052 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 365 k 2023-11-16T16:35:44.052 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 107 k 2023-11-16T16:35:44.052 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 94 k 2023-11-16T16:35:44.052 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.4 M 2023-11-16T16:35:44.052 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 180 k 2023-11-16T16:35:44.052 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-16T16:35:44.052 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2023-11-16T16:35:44.052 INFO:teuthology.orchestra.run.smithi077.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-16T16:35:44.052 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.7 M 2023-11-16T16:35:44.052 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.6 M 2023-11-16T16:35:44.053 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-11-16T16:35:44.053 INFO:teuthology.orchestra.run.smithi077.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-16T16:35:44.053 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 22 M 2023-11-16T16:35:44.053 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 23 k 2023-11-16T16:35:44.053 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 2.3 M 2023-11-16T16:35:44.053 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 240 k 2023-11-16T16:35:44.053 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 4.6 M 2023-11-16T16:35:44.053 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 18 M 2023-11-16T16:35:44.053 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 15 k 2023-11-16T16:35:44.053 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 24 k 2023-11-16T16:35:44.053 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-16T16:35:44.054 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-16T16:35:44.054 INFO:teuthology.orchestra.run.smithi077.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-16T16:35:44.054 INFO:teuthology.orchestra.run.smithi077.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-16T16:35:44.054 INFO:teuthology.orchestra.run.smithi077.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-16T16:35:44.054 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 137 k 2023-11-16T16:35:44.054 INFO:teuthology.orchestra.run.smithi077.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-16T16:35:44.054 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-16T16:35:44.054 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-16T16:35:44.054 INFO:teuthology.orchestra.run.smithi077.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-16T16:35:44.054 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-16T16:35:44.054 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-16T16:35:44.055 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 473 k 2023-11-16T16:35:44.055 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-16T16:35:44.055 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.5 M 2023-11-16T16:35:44.055 INFO:teuthology.orchestra.run.smithi077.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-16T16:35:44.055 INFO:teuthology.orchestra.run.smithi077.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-16T16:35:44.055 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-16T16:35:44.055 INFO:teuthology.orchestra.run.smithi077.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-16T16:35:44.055 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-16T16:35:44.055 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-16T16:35:44.055 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-16T16:35:44.055 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-16T16:35:44.055 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-16T16:35:44.056 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-16T16:35:44.056 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-16T16:35:44.056 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-16T16:35:44.056 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-16T16:35:44.056 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-16T16:35:44.056 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-16T16:35:44.056 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-16T16:35:44.056 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 45 k 2023-11-16T16:35:44.056 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 119 k 2023-11-16T16:35:44.056 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-16T16:35:44.056 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-16T16:35:44.057 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-16T16:35:44.057 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-16T16:35:44.057 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-16T16:35:44.057 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-16T16:35:44.057 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-16T16:35:44.057 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-16T16:35:44.057 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-16T16:35:44.057 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-16T16:35:44.057 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-16T16:35:44.057 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T16:35:44.057 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-16T16:35:44.057 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-16T16:35:44.058 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-16T16:35:44.058 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-16T16:35:44.058 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-16T16:35:44.058 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-16T16:35:44.058 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-16T16:35:44.058 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-16T16:35:44.058 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-16T16:35:44.058 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-16T16:35:44.058 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-16T16:35:44.058 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-16T16:35:44.058 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-16T16:35:44.059 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-16T16:35:44.059 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-16T16:35:44.059 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-16T16:35:44.059 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-16T16:35:44.059 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-16T16:35:44.059 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T16:35:44.059 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-16T16:35:44.059 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-16T16:35:44.059 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-16T16:35:44.059 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-16T16:35:44.059 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-16T16:35:44.059 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-16T16:35:44.060 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-16T16:35:44.060 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-16T16:35:44.060 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-16T16:35:44.060 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-16T16:35:44.060 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-16T16:35:44.060 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-16T16:35:44.060 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-16T16:35:44.060 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-16T16:35:44.060 INFO:teuthology.orchestra.run.smithi077.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-16T16:35:44.060 INFO:teuthology.orchestra.run.smithi077.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-16T16:35:44.060 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-16T16:35:44.061 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-16T16:35:44.061 INFO:teuthology.orchestra.run.smithi077.stdout:Enabling module streams: 2023-11-16T16:35:44.061 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql 10 2023-11-16T16:35:44.061 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-16T16:35:44.061 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-16T16:35:44.061 INFO:teuthology.orchestra.run.smithi077.stdout:=========================================================================================================================== 2023-11-16T16:35:44.061 INFO:teuthology.orchestra.run.smithi077.stdout:Install 128 Packages 2023-11-16T16:35:44.061 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 3 Packages 2023-11-16T16:35:44.061 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-16T16:35:44.062 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 197 M 2023-11-16T16:35:44.062 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-16T16:35:44.302 INFO:teuthology.orchestra.run.smithi176.stdout:(6/131): ceph-mds-18.0.0-3905.g5e4a743a.el8.x86 3.2 MB/s | 2.3 MB 00:00 2023-11-16T16:35:44.360 INFO:teuthology.orchestra.run.smithi077.stdout:(1/131): ceph-18.0.0-3905.g5e4a743a.el8.x86_64. 24 kB/s | 6.4 kB 00:00 2023-11-16T16:35:44.619 INFO:teuthology.orchestra.run.smithi176.stdout:(7/131): ceph-mon-18.0.0-3905.g5e4a743a.el8.x86 8.0 MB/s | 4.6 MB 00:00 2023-11-16T16:35:45.812 INFO:teuthology.orchestra.run.smithi176.stdout:(8/131): ceph-common-18.0.0-3905.g5e4a743a.el8. 7.6 MB/s | 22 MB 00:02 2023-11-16T16:35:45.818 INFO:teuthology.orchestra.run.smithi077.stdout:(2/131): ceph-fuse-18.0.0-3905.g5e4a743a.el8.x8 581 kB/s | 847 kB 00:01 2023-11-16T16:35:45.887 INFO:teuthology.orchestra.run.smithi176.stdout:(9/131): ceph-selinux-18.0.0-3905.g5e4a743a.el8 325 kB/s | 24 kB 00:00 2023-11-16T16:35:46.060 INFO:teuthology.orchestra.run.smithi077.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 685 kB/s | 165 kB 00:00 2023-11-16T16:35:46.212 INFO:teuthology.orchestra.run.smithi176.stdout:(10/131): ceph-radosgw-18.0.0-3905.g5e4a743a.el 8.2 MB/s | 13 MB 00:01 2023-11-16T16:35:46.313 INFO:teuthology.orchestra.run.smithi176.stdout:(11/131): cephfs-mirror-18.0.0-3905.g5e4a743a.e 2.5 MB/s | 258 kB 00:00 2023-11-16T16:35:46.404 INFO:teuthology.orchestra.run.smithi176.stdout:(12/131): libcephfs-devel-18.0.0-3905.g5e4a743a 339 kB/s | 31 kB 00:00 2023-11-16T16:35:46.688 INFO:teuthology.orchestra.run.smithi176.stdout:(13/131): libcephfs2-18.0.0-3905.g5e4a743a.el8. 2.7 MB/s | 771 kB 00:00 2023-11-16T16:35:46.771 INFO:teuthology.orchestra.run.smithi176.stdout:(14/131): libcephsqlite-18.0.0-3905.g5e4a743a.e 1.6 MB/s | 137 kB 00:00 2023-11-16T16:35:46.847 INFO:teuthology.orchestra.run.smithi176.stdout:(15/131): librados-devel-18.0.0-3905.g5e4a743a. 1.8 MB/s | 139 kB 00:00 2023-11-16T16:35:46.988 INFO:teuthology.orchestra.run.smithi176.stdout:(16/131): libradosstriper1-18.0.0-3905.g5e4a743 3.3 MB/s | 473 kB 00:00 2023-11-16T16:35:47.400 INFO:teuthology.orchestra.run.smithi044.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T16:35:47.401 INFO:teuthology.orchestra.run.smithi044.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T16:35:47.416 INFO:teuthology.orchestra.run.smithi044.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-16T16:35:47.417 INFO:teuthology.orchestra.run.smithi044.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-16T16:35:47.419 INFO:teuthology.orchestra.run.smithi044.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-16T16:35:47.420 INFO:teuthology.orchestra.run.smithi044.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-16T16:35:47.422 INFO:teuthology.orchestra.run.smithi044.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-16T16:35:47.426 INFO:teuthology.orchestra.run.smithi044.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-16T16:35:47.427 INFO:teuthology.orchestra.run.smithi044.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-16T16:35:47.576 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2023-11-16T16:35:47.590 INFO:teuthology.orchestra.run.smithi044.stdout:=========================================================================================================================== 2023-11-16T16:35:47.590 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2023-11-16T16:35:47.590 INFO:teuthology.orchestra.run.smithi044.stdout:=========================================================================================================================== 2023-11-16T16:35:47.591 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2023-11-16T16:35:47.591 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-16T16:35:47.591 INFO:teuthology.orchestra.run.smithi044.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-16T16:35:47.591 INFO:teuthology.orchestra.run.smithi044.stdout: ceph x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.4 k 2023-11-16T16:35:47.591 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 5.3 M 2023-11-16T16:35:47.591 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 847 k 2023-11-16T16:35:47.591 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 165 k 2023-11-16T16:35:47.591 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 1.5 M 2023-11-16T16:35:47.591 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 127 k 2023-11-16T16:35:47.591 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 1.7 M 2023-11-16T16:35:47.591 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 7.4 M 2023-11-16T16:35:47.592 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-rook noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 47 k 2023-11-16T16:35:47.592 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 13 M 2023-11-16T16:35:47.592 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 41 M 2023-11-16T16:35:47.592 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-volume noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 265 k 2023-11-16T16:35:47.592 INFO:teuthology.orchestra.run.smithi044.stdout: cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 83 k 2023-11-16T16:35:47.592 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 258 k 2023-11-16T16:35:47.592 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-top noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 18 k 2023-11-16T16:35:47.592 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-16T16:35:47.592 INFO:teuthology.orchestra.run.smithi044.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-16T16:35:47.592 INFO:teuthology.orchestra.run.smithi044.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-16T16:35:47.592 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 31 k 2023-11-16T16:35:47.593 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 771 k 2023-11-16T16:35:47.593 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 139 k 2023-11-16T16:35:47.593 INFO:teuthology.orchestra.run.smithi044.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-16T16:35:47.593 INFO:teuthology.orchestra.run.smithi044.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-16T16:35:47.593 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-16T16:35:47.593 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-16T16:35:47.593 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-16T16:35:47.593 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-16T16:35:47.593 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 197 k 2023-11-16T16:35:47.593 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 371 k 2023-11-16T16:35:47.594 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 365 k 2023-11-16T16:35:47.594 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 107 k 2023-11-16T16:35:47.594 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 94 k 2023-11-16T16:35:47.594 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.4 M 2023-11-16T16:35:47.594 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 180 k 2023-11-16T16:35:47.594 INFO:teuthology.orchestra.run.smithi044.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-16T16:35:47.594 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrading: 2023-11-16T16:35:47.594 INFO:teuthology.orchestra.run.smithi044.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-16T16:35:47.594 INFO:teuthology.orchestra.run.smithi044.stdout: librados2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.7 M 2023-11-16T16:35:47.594 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.6 M 2023-11-16T16:35:47.594 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2023-11-16T16:35:47.595 INFO:teuthology.orchestra.run.smithi044.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-16T16:35:47.595 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 22 M 2023-11-16T16:35:47.595 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 23 k 2023-11-16T16:35:47.595 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 2.3 M 2023-11-16T16:35:47.595 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 240 k 2023-11-16T16:35:47.595 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 4.6 M 2023-11-16T16:35:47.595 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 18 M 2023-11-16T16:35:47.595 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 15 k 2023-11-16T16:35:47.595 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-selinux x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 24 k 2023-11-16T16:35:47.595 INFO:teuthology.orchestra.run.smithi044.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-16T16:35:47.595 INFO:teuthology.orchestra.run.smithi044.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-16T16:35:47.596 INFO:teuthology.orchestra.run.smithi044.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-16T16:35:47.596 INFO:teuthology.orchestra.run.smithi044.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-16T16:35:47.596 INFO:teuthology.orchestra.run.smithi044.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-16T16:35:47.596 INFO:teuthology.orchestra.run.smithi044.stdout: libcephsqlite x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 137 k 2023-11-16T16:35:47.596 INFO:teuthology.orchestra.run.smithi044.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-16T16:35:47.596 INFO:teuthology.orchestra.run.smithi044.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-16T16:35:47.596 INFO:teuthology.orchestra.run.smithi044.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-16T16:35:47.596 INFO:teuthology.orchestra.run.smithi044.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-16T16:35:47.596 INFO:teuthology.orchestra.run.smithi044.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-16T16:35:47.596 INFO:teuthology.orchestra.run.smithi044.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-16T16:35:47.596 INFO:teuthology.orchestra.run.smithi044.stdout: libradosstriper1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 473 k 2023-11-16T16:35:47.597 INFO:teuthology.orchestra.run.smithi044.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-16T16:35:47.597 INFO:teuthology.orchestra.run.smithi044.stdout: librgw2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.5 M 2023-11-16T16:35:47.597 INFO:teuthology.orchestra.run.smithi044.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-16T16:35:47.597 INFO:teuthology.orchestra.run.smithi044.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-16T16:35:47.597 INFO:teuthology.orchestra.run.smithi044.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-16T16:35:47.597 INFO:teuthology.orchestra.run.smithi044.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-16T16:35:47.597 INFO:teuthology.orchestra.run.smithi044.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-16T16:35:47.597 INFO:teuthology.orchestra.run.smithi044.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-16T16:35:47.597 INFO:teuthology.orchestra.run.smithi044.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-16T16:35:47.597 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-16T16:35:47.598 INFO:teuthology.orchestra.run.smithi044.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-16T16:35:47.598 INFO:teuthology.orchestra.run.smithi044.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-16T16:35:47.598 INFO:teuthology.orchestra.run.smithi044.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-16T16:35:47.598 INFO:teuthology.orchestra.run.smithi044.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-16T16:35:47.598 INFO:teuthology.orchestra.run.smithi044.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-16T16:35:47.598 INFO:teuthology.orchestra.run.smithi044.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-16T16:35:47.598 INFO:teuthology.orchestra.run.smithi044.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-16T16:35:47.598 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-16T16:35:47.598 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-argparse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 45 k 2023-11-16T16:35:47.598 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 119 k 2023-11-16T16:35:47.599 INFO:teuthology.orchestra.run.smithi044.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-16T16:35:47.599 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-16T16:35:47.599 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-16T16:35:47.599 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-16T16:35:47.599 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-16T16:35:47.599 INFO:teuthology.orchestra.run.smithi044.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-16T16:35:47.599 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-16T16:35:47.599 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-16T16:35:47.599 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-16T16:35:47.599 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-16T16:35:47.600 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-16T16:35:47.600 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T16:35:47.600 INFO:teuthology.orchestra.run.smithi044.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-16T16:35:47.600 INFO:teuthology.orchestra.run.smithi044.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-16T16:35:47.600 INFO:teuthology.orchestra.run.smithi044.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-16T16:35:47.600 INFO:teuthology.orchestra.run.smithi044.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-16T16:35:47.600 INFO:teuthology.orchestra.run.smithi044.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-16T16:35:47.600 INFO:teuthology.orchestra.run.smithi044.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-16T16:35:47.600 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-16T16:35:47.600 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-16T16:35:47.601 INFO:teuthology.orchestra.run.smithi044.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-16T16:35:47.601 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-16T16:35:47.601 INFO:teuthology.orchestra.run.smithi044.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-16T16:35:47.601 INFO:teuthology.orchestra.run.smithi044.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-16T16:35:47.601 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-16T16:35:47.601 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-16T16:35:47.601 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-16T16:35:47.601 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-16T16:35:47.601 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-16T16:35:47.601 INFO:teuthology.orchestra.run.smithi044.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-16T16:35:47.601 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T16:35:47.602 INFO:teuthology.orchestra.run.smithi044.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-16T16:35:47.602 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-16T16:35:47.602 INFO:teuthology.orchestra.run.smithi044.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-16T16:35:47.602 INFO:teuthology.orchestra.run.smithi044.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-16T16:35:47.602 INFO:teuthology.orchestra.run.smithi044.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-16T16:35:47.602 INFO:teuthology.orchestra.run.smithi044.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-16T16:35:47.602 INFO:teuthology.orchestra.run.smithi044.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-16T16:35:47.602 INFO:teuthology.orchestra.run.smithi044.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-16T16:35:47.602 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-16T16:35:47.602 INFO:teuthology.orchestra.run.smithi044.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-16T16:35:47.602 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-16T16:35:47.603 INFO:teuthology.orchestra.run.smithi044.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-16T16:35:47.603 INFO:teuthology.orchestra.run.smithi044.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-16T16:35:47.603 INFO:teuthology.orchestra.run.smithi044.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-16T16:35:47.603 INFO:teuthology.orchestra.run.smithi044.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-16T16:35:47.603 INFO:teuthology.orchestra.run.smithi044.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-16T16:35:47.603 INFO:teuthology.orchestra.run.smithi044.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-16T16:35:47.603 INFO:teuthology.orchestra.run.smithi044.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-16T16:35:47.603 INFO:teuthology.orchestra.run.smithi044.stdout:Enabling module streams: 2023-11-16T16:35:47.603 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql 10 2023-11-16T16:35:47.603 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:35:47.603 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2023-11-16T16:35:47.604 INFO:teuthology.orchestra.run.smithi044.stdout:=========================================================================================================================== 2023-11-16T16:35:47.604 INFO:teuthology.orchestra.run.smithi044.stdout:Install 128 Packages 2023-11-16T16:35:47.604 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrade 3 Packages 2023-11-16T16:35:47.604 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:35:47.604 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 197 M 2023-11-16T16:35:47.604 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2023-11-16T16:35:47.856 INFO:teuthology.orchestra.run.smithi176.stdout:(17/131): librgw2-18.0.0-3905.g5e4a743a.el8.x86 7.5 MB/s | 6.5 MB 00:00 2023-11-16T16:35:47.860 INFO:teuthology.orchestra.run.smithi077.stdout:(4/131): ceph-base-18.0.0-3905.g5e4a743a.el8.x8 1.4 MB/s | 5.3 MB 00:03 2023-11-16T16:35:47.914 INFO:teuthology.orchestra.run.smithi176.stdout:(18/131): python3-ceph-argparse-18.0.0-3905.g5e 773 kB/s | 45 kB 00:00 2023-11-16T16:35:47.981 INFO:teuthology.orchestra.run.smithi176.stdout:(19/131): python3-ceph-common-18.0.0-3905.g5e4a 1.7 MB/s | 119 kB 00:00 2023-11-16T16:35:48.048 INFO:teuthology.orchestra.run.smithi176.stdout:(20/131): python3-cephfs-18.0.0-3905.g5e4a743a. 2.9 MB/s | 197 kB 00:00 2023-11-16T16:35:48.123 INFO:teuthology.orchestra.run.smithi176.stdout:(21/131): python3-rados-18.0.0-3905.g5e4a743a.e 4.8 MB/s | 371 kB 00:00 2023-11-16T16:35:48.198 INFO:teuthology.orchestra.run.smithi176.stdout:(22/131): python3-rbd-18.0.0-3905.g5e4a743a.el8 4.8 MB/s | 365 kB 00:00 2023-11-16T16:35:48.263 INFO:teuthology.orchestra.run.smithi044.stdout:(1/131): ceph-18.0.0-3905.g5e4a743a.el8.x86_64. 21 kB/s | 6.4 kB 00:00 2023-11-16T16:35:48.310 INFO:teuthology.orchestra.run.smithi077.stdout:(5/131): ceph-mds-18.0.0-3905.g5e4a743a.el8.x86 1.0 MB/s | 2.3 MB 00:02 2023-11-16T16:35:48.366 INFO:teuthology.orchestra.run.smithi176.stdout:(23/131): ceph-osd-18.0.0-3905.g5e4a743a.el8.x8 4.4 MB/s | 18 MB 00:04 2023-11-16T16:35:48.391 INFO:teuthology.orchestra.run.smithi176.stdout:(24/131): python3-rgw-18.0.0-3905.g5e4a743a.el8 556 kB/s | 107 kB 00:00 2023-11-16T16:35:48.457 INFO:teuthology.orchestra.run.smithi176.stdout:(25/131): rbd-fuse-18.0.0-3905.g5e4a743a.el8.x8 1.0 MB/s | 94 kB 00:00 2023-11-16T16:35:48.469 INFO:teuthology.orchestra.run.smithi077.stdout:(6/131): ceph-mgr-18.0.0-3905.g5e4a743a.el8.x86 2.5 MB/s | 1.5 MB 00:00 2023-11-16T16:35:48.558 INFO:teuthology.orchestra.run.smithi176.stdout:(26/131): rbd-nbd-18.0.0-3905.g5e4a743a.el8.x86 1.8 MB/s | 180 kB 00:00 2023-11-16T16:35:48.675 INFO:teuthology.orchestra.run.smithi176.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3905.g 198 kB/s | 23 kB 00:00 2023-11-16T16:35:48.775 INFO:teuthology.orchestra.run.smithi176.stdout:(28/131): rbd-mirror-18.0.0-3905.g5e4a743a.el8. 9.0 MB/s | 3.4 MB 00:00 2023-11-16T16:35:48.817 INFO:teuthology.orchestra.run.smithi176.stdout:(29/131): ceph-mgr-cephadm-18.0.0-3905.g5e4a743 895 kB/s | 127 kB 00:00 2023-11-16T16:35:49.017 INFO:teuthology.orchestra.run.smithi176.stdout:(30/131): ceph-mgr-dashboard-18.0.0-3905.g5e4a7 7.1 MB/s | 1.7 MB 00:00 2023-11-16T16:35:49.076 INFO:teuthology.orchestra.run.smithi176.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3905.g5e 4.0 MB/s | 240 kB 00:00 2023-11-16T16:35:49.135 INFO:teuthology.orchestra.run.smithi176.stdout:(32/131): ceph-mgr-rook-18.0.0-3905.g5e4a743a.e 812 kB/s | 47 kB 00:00 2023-11-16T16:35:49.193 INFO:teuthology.orchestra.run.smithi176.stdout:(33/131): ceph-prometheus-alerts-18.0.0-3905.g5 249 kB/s | 15 kB 00:00 2023-11-16T16:35:49.197 INFO:teuthology.orchestra.run.smithi044.stdout:(2/131): ceph-fuse-18.0.0-3905.g5e4a743a.el8.x8 908 kB/s | 847 kB 00:00 2023-11-16T16:35:49.260 INFO:teuthology.orchestra.run.smithi176.stdout:(34/131): ceph-volume-18.0.0-3905.g5e4a743a.el8 3.9 MB/s | 265 kB 00:00 2023-11-16T16:35:49.319 INFO:teuthology.orchestra.run.smithi176.stdout:(35/131): cephadm-18.0.0-3905.g5e4a743a.el8.noa 1.4 MB/s | 83 kB 00:00 2023-11-16T16:35:49.364 INFO:teuthology.orchestra.run.smithi044.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 994 kB/s | 165 kB 00:00 2023-11-16T16:35:49.378 INFO:teuthology.orchestra.run.smithi176.stdout:(36/131): cephfs-top-18.0.0-3905.g5e4a743a.el8. 307 kB/s | 18 kB 00:00 2023-11-16T16:35:49.539 INFO:teuthology.orchestra.run.smithi044.stdout:(4/131): ceph-base-18.0.0-3905.g5e4a743a.el8.x8 3.4 MB/s | 5.3 MB 00:01 2023-11-16T16:35:49.748 INFO:teuthology.orchestra.run.smithi044.stdout:(5/131): ceph-mgr-18.0.0-3905.g5e4a743a.el8.x86 7.5 MB/s | 1.5 MB 00:00 2023-11-16T16:35:50.161 INFO:teuthology.orchestra.run.smithi176.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 391 kB/s | 306 kB 00:00 2023-11-16T16:35:50.319 INFO:teuthology.orchestra.run.smithi077.stdout:(7/131): ceph-mon-18.0.0-3905.g5e4a743a.el8.x86 2.3 MB/s | 4.6 MB 00:02 2023-11-16T16:35:50.328 INFO:teuthology.orchestra.run.smithi176.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 353 kB/s | 59 kB 00:00 2023-11-16T16:35:50.356 INFO:teuthology.orchestra.run.smithi044.stdout:(6/131): ceph-mon-18.0.0-3905.g5e4a743a.el8.x86 7.6 MB/s | 4.6 MB 00:00 2023-11-16T16:35:50.453 INFO:teuthology.orchestra.run.smithi176.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 601 kB/s | 75 kB 00:00 2023-11-16T16:35:50.937 INFO:teuthology.orchestra.run.smithi176.stdout:(40/131): ceph-test-18.0.0-3905.g5e4a743a.el8.x 8.0 MB/s | 41 MB 00:05 2023-11-16T16:35:50.971 INFO:teuthology.orchestra.run.smithi176.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 921 kB/s | 476 kB 00:00 2023-11-16T16:35:51.129 INFO:teuthology.orchestra.run.smithi176.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.1 MB/s | 185 kB 00:00 2023-11-16T16:35:51.306 INFO:teuthology.orchestra.run.smithi044.stdout:(7/131): ceph-mds-18.0.0-3905.g5e4a743a.el8.x86 1.2 MB/s | 2.3 MB 00:01 2023-11-16T16:35:51.396 INFO:teuthology.orchestra.run.smithi176.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 124 kB/s | 33 kB 00:00 2023-11-16T16:35:51.455 INFO:teuthology.orchestra.run.smithi176.stdout:(44/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 84 kB/s | 44 kB 00:00 2023-11-16T16:35:51.522 INFO:teuthology.orchestra.run.smithi176.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 130 kB/s | 16 kB 00:00 2023-11-16T16:35:51.630 INFO:teuthology.orchestra.run.smithi176.stdout:(46/131): ceph-mgr-diskprediction-local-18.0.0- 2.6 MB/s | 7.4 MB 00:02 2023-11-16T16:35:51.664 INFO:teuthology.orchestra.run.smithi176.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.6 MB/s | 384 kB 00:00 2023-11-16T16:35:51.698 INFO:teuthology.orchestra.run.smithi044.stdout:(8/131): ceph-osd-18.0.0-3905.g5e4a743a.el8.x86 13 MB/s | 18 MB 00:01 2023-11-16T16:35:51.749 INFO:teuthology.orchestra.run.smithi044.stdout:(9/131): ceph-selinux-18.0.0-3905.g5e4a743a.el8 491 kB/s | 24 kB 00:00 2023-11-16T16:35:51.789 INFO:teuthology.orchestra.run.smithi176.stdout:(48/131): python3-cheroot-8.5.2-1.el8.noarch.rp 516 kB/s | 173 kB 00:00 2023-11-16T16:35:51.814 INFO:teuthology.orchestra.run.smithi176.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 74 kB/s | 11 kB 00:00 2023-11-16T16:35:51.923 INFO:teuthology.orchestra.run.smithi176.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 132 kB/s | 18 kB 00:00 2023-11-16T16:35:52.057 INFO:teuthology.orchestra.run.smithi176.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 366 kB/s | 49 kB 00:00 2023-11-16T16:35:52.207 INFO:teuthology.orchestra.run.smithi176.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 392 kB/s | 59 kB 00:00 2023-11-16T16:35:52.224 INFO:teuthology.orchestra.run.smithi176.stdout:(53/131): python3-google-auth-1.1.1-10.el8.noar 137 kB/s | 81 kB 00:00 2023-11-16T16:35:52.274 INFO:teuthology.orchestra.run.smithi176.stdout:(54/131): python3-kubernetes-11.0.0-6.el8.noarc 2.5 MB/s | 1.2 MB 00:00 2023-11-16T16:35:52.350 INFO:teuthology.orchestra.run.smithi176.stdout:(55/131): python3-natsort-7.1.1-2.el8.noarch.rp 424 kB/s | 60 kB 00:00 2023-11-16T16:35:52.400 INFO:teuthology.orchestra.run.smithi176.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 124 kB/s | 16 kB 00:00 2023-11-16T16:35:52.516 INFO:teuthology.orchestra.run.smithi176.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 202 kB/s | 34 kB 00:00 2023-11-16T16:35:52.550 INFO:teuthology.orchestra.run.smithi176.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 1.3 MB/s | 196 kB 00:00 2023-11-16T16:35:52.634 INFO:teuthology.orchestra.run.smithi176.stdout:(59/131): python3-pecan-1.3.2-9.el8.noarch.rpm 691 kB/s | 283 kB 00:00 2023-11-16T16:35:52.659 INFO:teuthology.orchestra.run.smithi176.stdout:(60/131): python3-rsa-4.9-2.el8.noarch.rpm 452 kB/s | 64 kB 00:00 2023-11-16T16:35:52.676 INFO:teuthology.orchestra.run.smithi176.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 149 kB/s | 19 kB 00:00 2023-11-16T16:35:52.768 INFO:teuthology.orchestra.run.smithi176.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 179 kB/s | 24 kB 00:00 2023-11-16T16:35:52.801 INFO:teuthology.orchestra.run.smithi176.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 206 kB/s | 29 kB 00:00 2023-11-16T16:35:52.818 INFO:teuthology.orchestra.run.smithi176.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 194 kB/s | 27 kB 00:00 2023-11-16T16:35:52.960 INFO:teuthology.orchestra.run.smithi176.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 432 kB/s | 61 kB 00:00 2023-11-16T16:35:52.994 INFO:teuthology.orchestra.run.smithi176.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.3 MB/s | 251 kB 00:00 2023-11-16T16:35:53.036 INFO:teuthology.orchestra.run.smithi176.stdout:(67/131): python3-waitress-1.4.3-1.el8.noarch.r 953 kB/s | 255 kB 00:00 2023-11-16T16:35:53.086 INFO:teuthology.orchestra.run.smithi176.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 679 kB/s | 85 kB 00:00 2023-11-16T16:35:53.111 INFO:teuthology.orchestra.run.smithi176.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-11-16T16:35:53.137 INFO:teuthology.orchestra.run.smithi176.stdout:(70/131): python3-zc-lockfile-2.0-2.el8.noarch. 161 kB/s | 23 kB 00:00 2023-11-16T16:35:53.320 INFO:teuthology.orchestra.run.smithi176.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 601 kB/s | 110 kB 00:00 2023-11-16T16:35:53.471 INFO:teuthology.orchestra.run.smithi176.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 362 kB/s | 54 kB 00:00 2023-11-16T16:35:53.538 INFO:teuthology.orchestra.run.smithi176.stdout:(73/131): thrift-0.13.0-2.el8.x86_64.rpm 3.5 MB/s | 1.7 MB 00:00 2023-11-16T16:35:53.580 INFO:teuthology.orchestra.run.smithi176.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.3 MB/s | 259 kB 00:00 2023-11-16T16:35:53.830 INFO:teuthology.orchestra.run.smithi176.stdout:(75/131): python3-scipy-1.0.0-21.module+el8.5.0 19 MB/s | 14 MB 00:00 2023-11-16T16:35:53.855 INFO:teuthology.orchestra.run.smithi176.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 144 kB/s | 39 kB 00:00 2023-11-16T16:35:53.880 INFO:teuthology.orchestra.run.smithi176.stdout:(77/131): socat-1.7.4.1-1.el8.x86_64.rpm 944 kB/s | 323 kB 00:00 2023-11-16T16:35:53.989 INFO:teuthology.orchestra.run.smithi176.stdout:(78/131): bison-3.0.4-10.el8.x86_64.rpm 4.2 MB/s | 688 kB 00:00 2023-11-16T16:35:54.007 INFO:teuthology.orchestra.run.smithi044.stdout:(10/131): ceph-test-18.0.0-3905.g5e4a743a.el8.x 18 MB/s | 41 MB 00:02 2023-11-16T16:35:54.014 INFO:teuthology.orchestra.run.smithi176.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2023-11-16T16:35:54.047 INFO:teuthology.orchestra.run.smithi176.stdout:(80/131): python3-jinja2-2.10.1-3.el8.noarch.rp 2.7 MB/s | 538 kB 00:00 2023-11-16T16:35:54.057 INFO:teuthology.orchestra.run.smithi044.stdout:(11/131): cephfs-mirror-18.0.0-3905.g5e4a743a.e 5.1 MB/s | 258 kB 00:00 2023-11-16T16:35:54.108 INFO:teuthology.orchestra.run.smithi044.stdout:(12/131): libcephfs-devel-18.0.0-3905.g5e4a743a 622 kB/s | 31 kB 00:00 2023-11-16T16:35:54.123 INFO:teuthology.orchestra.run.smithi176.stdout:(81/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2023-11-16T16:35:54.156 INFO:teuthology.orchestra.run.smithi176.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 4.9 MB/s | 713 kB 00:00 2023-11-16T16:35:54.193 INFO:teuthology.orchestra.run.smithi044.stdout:(13/131): libcephfs2-18.0.0-3905.g5e4a743a.el8. 8.9 MB/s | 771 kB 00:00 2023-11-16T16:35:54.244 INFO:teuthology.orchestra.run.smithi044.stdout:(14/131): libcephsqlite-18.0.0-3905.g5e4a743a.e 2.7 MB/s | 137 kB 00:00 2023-11-16T16:35:54.248 INFO:teuthology.orchestra.run.smithi176.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 212 kB/s | 27 kB 00:00 2023-11-16T16:35:54.282 INFO:teuthology.orchestra.run.smithi176.stdout:(84/131): flex-2.6.1-9.el8.x86_64.rpm 2.5 MB/s | 320 kB 00:00 2023-11-16T16:35:54.324 INFO:teuthology.orchestra.run.smithi044.stdout:(15/131): librados-devel-18.0.0-3905.g5e4a743a. 1.7 MB/s | 139 kB 00:00 2023-11-16T16:35:54.357 INFO:teuthology.orchestra.run.smithi176.stdout:(85/131): python3-babel-2.5.1-7.el8.noarch.rpm 15 MB/s | 4.8 MB 00:00 2023-11-16T16:35:54.386 INFO:teuthology.orchestra.run.smithi044.stdout:(16/131): libradosstriper1-18.0.0-3905.g5e4a743 7.6 MB/s | 473 kB 00:00 2023-11-16T16:35:54.424 INFO:teuthology.orchestra.run.smithi176.stdout:(86/131): python3-prettytable-0.7.2-14.el8.noar 251 kB/s | 44 kB 00:00 2023-11-16T16:35:54.491 INFO:teuthology.orchestra.run.smithi176.stdout:(87/131): python3-numpy-1.14.3-10.el8.x86_64.rp 18 MB/s | 3.7 MB 00:00 2023-11-16T16:35:54.533 INFO:teuthology.orchestra.run.smithi176.stdout:(88/131): python3-werkzeug-0.12.2-4.el8.noarch. 2.5 MB/s | 457 kB 00:00 2023-11-16T16:35:54.633 INFO:teuthology.orchestra.run.smithi176.stdout:(89/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.1 MB/s | 225 kB 00:00 2023-11-16T16:35:54.658 INFO:teuthology.orchestra.run.smithi176.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2023-11-16T16:35:54.692 INFO:teuthology.orchestra.run.smithi176.stdout:(91/131): uuid-1.6.2-43.el8.x86_64.rpm 319 kB/s | 64 kB 00:00 2023-11-16T16:35:54.751 INFO:teuthology.orchestra.run.smithi176.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.1 MB/s | 103 kB 00:00 2023-11-16T16:35:54.793 INFO:teuthology.orchestra.run.smithi176.stdout:(93/131): python3-jsonpointer-1.10-11.el8.noarc 126 kB/s | 20 kB 00:00 2023-11-16T16:35:54.794 INFO:teuthology.orchestra.run.smithi077.stdout:(8/131): ceph-radosgw-18.0.0-3905.g5e4a743a.el8 2.9 MB/s | 13 MB 00:04 2023-11-16T16:35:54.811 INFO:teuthology.orchestra.run.smithi044.stdout:(17/131): librgw2-18.0.0-3905.g5e4a743a.el8.x86 15 MB/s | 6.5 MB 00:00 2023-11-16T16:35:54.851 INFO:teuthology.orchestra.run.smithi176.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 4.4 MB/s | 709 kB 00:00 2023-11-16T16:35:54.861 INFO:teuthology.orchestra.run.smithi077.stdout:(9/131): ceph-selinux-18.0.0-3905.g5e4a743a.el8 366 kB/s | 24 kB 00:00 2023-11-16T16:35:54.861 INFO:teuthology.orchestra.run.smithi044.stdout:(18/131): python3-ceph-argparse-18.0.0-3905.g5e 903 kB/s | 45 kB 00:00 2023-11-16T16:35:54.918 INFO:teuthology.orchestra.run.smithi176.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 1.5 MB/s | 198 kB 00:00 2023-11-16T16:35:54.920 INFO:teuthology.orchestra.run.smithi044.stdout:(19/131): python3-ceph-common-18.0.0-3905.g5e4a 2.0 MB/s | 119 kB 00:00 2023-11-16T16:35:54.970 INFO:teuthology.orchestra.run.smithi044.stdout:(20/131): python3-cephfs-18.0.0-3905.g5e4a743a. 3.9 MB/s | 197 kB 00:00 2023-11-16T16:35:55.026 INFO:teuthology.orchestra.run.smithi176.stdout:(96/131): openblas-threads-0.3.15-3.el8.x86_64. 18 MB/s | 4.9 MB 00:00 2023-11-16T16:35:55.029 INFO:teuthology.orchestra.run.smithi044.stdout:(21/131): python3-rados-18.0.0-3905.g5e4a743a.e 6.2 MB/s | 371 kB 00:00 2023-11-16T16:35:55.052 INFO:teuthology.orchestra.run.smithi176.stdout:(97/131): jq-1.6-3.el8.x86_64.rpm 1.5 MB/s | 202 kB 00:00 2023-11-16T16:35:55.088 INFO:teuthology.orchestra.run.smithi044.stdout:(22/131): python3-rbd-18.0.0-3905.g5e4a743a.el8 6.1 MB/s | 365 kB 00:00 2023-11-16T16:35:55.138 INFO:teuthology.orchestra.run.smithi044.stdout:(23/131): python3-rgw-18.0.0-3905.g5e4a743a.el8 2.1 MB/s | 107 kB 00:00 2023-11-16T16:35:55.161 INFO:teuthology.orchestra.run.smithi176.stdout:(98/131): openblas-0.3.15-3.el8.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2023-11-16T16:35:55.205 INFO:teuthology.orchestra.run.smithi044.stdout:(24/131): rbd-fuse-18.0.0-3905.g5e4a743a.el8.x8 1.4 MB/s | 94 kB 00:00 2023-11-16T16:35:55.236 INFO:teuthology.orchestra.run.smithi176.stdout:(99/131): librdkafka-0.11.4-3.el8.x86_64.rpm 1.7 MB/s | 354 kB 00:00 2023-11-16T16:35:55.345 INFO:teuthology.orchestra.run.smithi176.stdout:(100/131): postgresql-server-10.21-2.module+el8 17 MB/s | 5.1 MB 00:00 2023-11-16T16:35:55.378 INFO:teuthology.orchestra.run.smithi176.stdout:(101/131): postgresql-contrib-10.21-2.module+el 3.6 MB/s | 808 kB 00:00 2023-11-16T16:35:55.420 INFO:teuthology.orchestra.run.smithi176.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 8.2 MB/s | 1.5 MB 00:00 2023-11-16T16:35:55.438 INFO:teuthology.orchestra.run.smithi044.stdout:(25/131): rbd-mirror-18.0.0-3905.g5e4a743a.el8. 15 MB/s | 3.4 MB 00:00 2023-11-16T16:35:55.479 INFO:teuthology.orchestra.run.smithi176.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 2.0 MB/s | 200 kB 00:00 2023-11-16T16:35:55.489 INFO:teuthology.orchestra.run.smithi044.stdout:(26/131): rbd-nbd-18.0.0-3905.g5e4a743a.el8.x86 3.5 MB/s | 180 kB 00:00 2023-11-16T16:35:55.504 INFO:teuthology.orchestra.run.smithi176.stdout:(104/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 295 kB/s | 47 kB 00:00 2023-11-16T16:35:55.539 INFO:teuthology.orchestra.run.smithi044.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3905.g 463 kB/s | 23 kB 00:00 2023-11-16T16:35:55.554 INFO:teuthology.orchestra.run.smithi176.stdout:(105/131): python3-jwt-1.6.1-2.el8.noarch.rpm 318 kB/s | 43 kB 00:00 2023-11-16T16:35:55.620 INFO:teuthology.orchestra.run.smithi044.stdout:(28/131): ceph-mgr-cephadm-18.0.0-3905.g5e4a743 1.6 MB/s | 127 kB 00:00 2023-11-16T16:35:55.629 INFO:teuthology.orchestra.run.smithi176.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 391 kB/s | 48 kB 00:00 2023-11-16T16:35:55.655 INFO:teuthology.orchestra.run.smithi176.stdout:(107/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.2 MB/s | 220 kB 00:00 2023-11-16T16:35:55.713 INFO:teuthology.orchestra.run.smithi176.stdout:(108/131): libsepol-devel-2.9-3.el8.x86_64.rpm 547 kB/s | 87 kB 00:00 2023-11-16T16:35:55.765 INFO:teuthology.orchestra.run.smithi044.stdout:(29/131): ceph-mgr-dashboard-18.0.0-3905.g5e4a7 12 MB/s | 1.7 MB 00:00 2023-11-16T16:35:55.772 INFO:teuthology.orchestra.run.smithi176.stdout:(109/131): python3-requests-oauthlib-1.0.0-1.el 300 kB/s | 43 kB 00:00 2023-11-16T16:35:55.897 INFO:teuthology.orchestra.run.smithi176.stdout:(110/131): python3-pycparser-2.14-14.el8.noarch 450 kB/s | 109 kB 00:00 2023-11-16T16:35:55.931 INFO:teuthology.orchestra.run.smithi176.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.4 MB/s | 560 kB 00:00 2023-11-16T16:35:55.965 INFO:teuthology.orchestra.run.smithi176.stdout:(112/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 912 kB/s | 229 kB 00:00 2023-11-16T16:35:56.056 INFO:teuthology.orchestra.run.smithi176.stdout:(113/131): libaio-devel-0.3.112-1.el8.x86_64.rp 118 kB/s | 19 kB 00:00 2023-11-16T16:35:56.082 INFO:teuthology.orchestra.run.smithi176.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 1.0 MB/s | 155 kB 00:00 2023-11-16T16:35:56.149 INFO:teuthology.orchestra.run.smithi176.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 3.0 MB/s | 559 kB 00:00 2023-11-16T16:35:56.182 INFO:teuthology.orchestra.run.smithi176.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 143 kB/s | 18 kB 00:00 2023-11-16T16:35:56.241 INFO:teuthology.orchestra.run.smithi176.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.7 MB/s | 605 kB 00:00 2023-11-16T16:35:56.291 INFO:teuthology.orchestra.run.smithi176.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.6 MB/s | 238 kB 00:00 2023-11-16T16:35:56.373 INFO:teuthology.orchestra.run.smithi044.stdout:(30/131): ceph-mgr-diskprediction-local-18.0.0- 12 MB/s | 7.4 MB 00:00 2023-11-16T16:35:56.424 INFO:teuthology.orchestra.run.smithi044.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3905.g5e 4.7 MB/s | 240 kB 00:00 2023-11-16T16:35:56.499 INFO:teuthology.orchestra.run.smithi044.stdout:(32/131): ceph-mgr-rook-18.0.0-3905.g5e4a743a.e 634 kB/s | 47 kB 00:00 2023-11-16T16:35:56.549 INFO:teuthology.orchestra.run.smithi044.stdout:(33/131): ceph-prometheus-alerts-18.0.0-3905.g5 293 kB/s | 15 kB 00:00 2023-11-16T16:35:56.550 INFO:teuthology.orchestra.run.smithi176.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 234 kB/s | 72 kB 00:00 2023-11-16T16:35:56.599 INFO:teuthology.orchestra.run.smithi044.stdout:(34/131): ceph-volume-18.0.0-3905.g5e4a743a.el8 5.2 MB/s | 265 kB 00:00 2023-11-16T16:35:56.650 INFO:teuthology.orchestra.run.smithi044.stdout:(35/131): cephadm-18.0.0-3905.g5e4a743a.el8.noa 1.6 MB/s | 83 kB 00:00 2023-11-16T16:35:56.684 INFO:teuthology.orchestra.run.smithi176.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 211 kB/s | 83 kB 00:00 2023-11-16T16:35:56.700 INFO:teuthology.orchestra.run.smithi044.stdout:(36/131): cephfs-top-18.0.0-3905.g5e4a743a.el8. 360 kB/s | 18 kB 00:00 2023-11-16T16:35:56.709 INFO:teuthology.orchestra.run.smithi176.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 378 kB/s | 60 kB 00:00 2023-11-16T16:35:56.751 INFO:teuthology.orchestra.run.smithi176.stdout:(122/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 4.1 MB/s | 2.3 MB 00:00 2023-11-16T16:35:56.837 INFO:teuthology.orchestra.run.smithi176.stdout:(123/131): libuuid-devel-2.32.1-35.el8.x86_64.r 641 kB/s | 98 kB 00:00 2023-11-16T16:35:56.868 INFO:teuthology.orchestra.run.smithi176.stdout:(124/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 363 kB/s | 58 kB 00:00 2023-11-16T16:35:56.893 INFO:teuthology.orchestra.run.smithi176.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 271 kB/s | 39 kB 00:00 2023-11-16T16:35:57.002 INFO:teuthology.orchestra.run.smithi176.stdout:(126/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 3.8 MB/s | 644 kB 00:00 2023-11-16T16:35:57.027 INFO:teuthology.orchestra.run.smithi176.stdout:(127/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.1 MB/s | 171 kB 00:00 2023-11-16T16:35:57.053 INFO:teuthology.orchestra.run.smithi176.stdout:(128/131): NetworkManager-initscripts-updown-1. 866 kB/s | 138 kB 00:00 2023-11-16T16:35:57.236 INFO:teuthology.orchestra.run.smithi176.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 4.3 MB/s | 798 kB 00:00 2023-11-16T16:35:57.475 INFO:teuthology.orchestra.run.smithi044.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 395 kB/s | 306 kB 00:00 2023-11-16T16:35:57.486 INFO:teuthology.orchestra.run.smithi176.stdout:(130/131): librbd1-18.0.0-3905.g5e4a743a.el8.x8 7.8 MB/s | 3.6 MB 00:00 2023-11-16T16:35:57.592 INFO:teuthology.orchestra.run.smithi044.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 506 kB/s | 59 kB 00:00 2023-11-16T16:35:57.659 INFO:teuthology.orchestra.run.smithi077.stdout:(10/131): ceph-osd-18.0.0-3905.g5e4a743a.el8.x8 1.9 MB/s | 18 MB 00:09 2023-11-16T16:35:57.717 INFO:teuthology.orchestra.run.smithi044.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 602 kB/s | 75 kB 00:00 2023-11-16T16:35:57.761 INFO:teuthology.orchestra.run.smithi077.stdout:(11/131): cephfs-mirror-18.0.0-3905.g5e4a743a.e 2.5 MB/s | 258 kB 00:00 2023-11-16T16:35:57.820 INFO:teuthology.orchestra.run.smithi077.stdout:(12/131): libcephfs-devel-18.0.0-3905.g5e4a743a 532 kB/s | 31 kB 00:00 2023-11-16T16:35:57.918 INFO:teuthology.orchestra.run.smithi044.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.3 MB/s | 476 kB 00:00 2023-11-16T16:35:57.945 INFO:teuthology.orchestra.run.smithi176.stdout:(131/131): librados2-18.0.0-3905.g5e4a743a.el8. 3.9 MB/s | 3.7 MB 00:00 2023-11-16T16:35:57.948 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-11-16T16:35:57.948 INFO:teuthology.orchestra.run.smithi176.stdout:Total 13 MB/s | 197 MB 00:15 2023-11-16T16:35:58.045 INFO:teuthology.orchestra.run.smithi044.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 349 kB/s | 44 kB 00:00 2023-11-16T16:35:58.179 INFO:teuthology.orchestra.run.smithi077.stdout:(13/131): ceph-common-18.0.0-3905.g5e4a743a.el8 1.5 MB/s | 22 MB 00:14 2023-11-16T16:35:58.197 INFO:teuthology.orchestra.run.smithi044.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.2 MB/s | 185 kB 00:00 2023-11-16T16:35:58.212 INFO:teuthology.orchestra.run.smithi077.stdout:(14/131): libcephfs2-18.0.0-3905.g5e4a743a.el8. 1.9 MB/s | 771 kB 00:00 2023-11-16T16:35:58.279 INFO:teuthology.orchestra.run.smithi077.stdout:(15/131): librados-devel-18.0.0-3905.g5e4a743a. 2.0 MB/s | 139 kB 00:00 2023-11-16T16:35:58.296 INFO:teuthology.orchestra.run.smithi077.stdout:(16/131): libcephsqlite-18.0.0-3905.g5e4a743a.e 1.1 MB/s | 137 kB 00:00 2023-11-16T16:35:58.335 INFO:teuthology.orchestra.run.smithi044.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 240 kB/s | 33 kB 00:00 2023-11-16T16:35:58.444 INFO:teuthology.orchestra.run.smithi044.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 151 kB/s | 16 kB 00:00 2023-11-16T16:35:58.447 INFO:teuthology.orchestra.run.smithi077.stdout:(17/131): libradosstriper1-18.0.0-3905.g5e4a743 2.8 MB/s | 473 kB 00:00 2023-11-16T16:35:58.522 INFO:teuthology.orchestra.run.smithi077.stdout:(18/131): python3-ceph-argparse-18.0.0-3905.g5e 597 kB/s | 45 kB 00:00 2023-11-16T16:35:58.589 INFO:teuthology.orchestra.run.smithi077.stdout:(19/131): python3-ceph-common-18.0.0-3905.g5e4a 1.7 MB/s | 119 kB 00:00 2023-11-16T16:35:58.595 INFO:teuthology.orchestra.run.smithi044.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.1 MB/s | 173 kB 00:00 2023-11-16T16:35:58.681 INFO:teuthology.orchestra.run.smithi077.stdout:(20/131): python3-cephfs-18.0.0-3905.g5e4a743a. 2.1 MB/s | 197 kB 00:00 2023-11-16T16:35:58.728 INFO:teuthology.orchestra.run.smithi044.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.8 MB/s | 384 kB 00:00 2023-11-16T16:35:58.804 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-11-16T16:35:58.814 INFO:teuthology.orchestra.run.smithi077.stdout:(21/131): python3-rados-18.0.0-3905.g5e4a743a.e 2.7 MB/s | 371 kB 00:00 2023-11-16T16:35:58.837 INFO:teuthology.orchestra.run.smithi044.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 752 kB/s | 81 kB 00:00 2023-11-16T16:35:58.923 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-11-16T16:35:58.923 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-11-16T16:35:58.957 INFO:teuthology.orchestra.run.smithi077.stdout:(22/131): python3-rbd-18.0.0-3905.g5e4a743a.el8 2.5 MB/s | 365 kB 00:00 2023-11-16T16:35:59.023 INFO:teuthology.orchestra.run.smithi077.stdout:(23/131): python3-rgw-18.0.0-3905.g5e4a743a.el8 1.6 MB/s | 107 kB 00:00 2023-11-16T16:35:59.073 INFO:teuthology.orchestra.run.smithi044.stdout:(48/131): ceph-radosgw-18.0.0-3905.g5e4a743a.el 1.7 MB/s | 13 MB 00:07 2023-11-16T16:35:59.096 INFO:teuthology.orchestra.run.smithi044.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 43 kB/s | 11 kB 00:00 2023-11-16T16:35:59.099 INFO:teuthology.orchestra.run.smithi077.stdout:(24/131): rbd-fuse-18.0.0-3905.g5e4a743a.el8.x8 1.2 MB/s | 94 kB 00:00 2023-11-16T16:35:59.179 INFO:teuthology.orchestra.run.smithi044.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 167 kB/s | 18 kB 00:00 2023-11-16T16:35:59.288 INFO:teuthology.orchestra.run.smithi044.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 471 kB/s | 49 kB 00:00 2023-11-16T16:35:59.389 INFO:teuthology.orchestra.run.smithi044.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 588 kB/s | 59 kB 00:00 2023-11-16T16:35:59.722 INFO:teuthology.orchestra.run.smithi044.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 181 kB/s | 60 kB 00:00 2023-11-16T16:35:59.839 INFO:teuthology.orchestra.run.smithi044.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.4 MB/s | 283 kB 00:00 2023-11-16T16:35:59.948 INFO:teuthology.orchestra.run.smithi044.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 144 kB/s | 16 kB 00:00 2023-11-16T16:36:00.007 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-11-16T16:36:00.011 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-11-16T16:36:00.048 INFO:teuthology.orchestra.run.smithi044.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 337 kB/s | 34 kB 00:00 2023-11-16T16:36:00.165 INFO:teuthology.orchestra.run.smithi044.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 1.6 MB/s | 196 kB 00:00 2023-11-16T16:36:00.265 INFO:teuthology.orchestra.run.smithi044.stdout:(58/131): python3-rsa-4.9-2.el8.noarch.rpm 643 kB/s | 64 kB 00:00 2023-11-16T16:36:00.307 INFO:teuthology.orchestra.run.smithi044.stdout:(59/131): python3-kubernetes-11.0.0-6.el8.noarc 986 kB/s | 1.2 MB 00:01 2023-11-16T16:36:00.374 INFO:teuthology.orchestra.run.smithi077.stdout:(25/131): rbd-mirror-18.0.0-3905.g5e4a743a.el8. 2.7 MB/s | 3.4 MB 00:01 2023-11-16T16:36:00.382 INFO:teuthology.orchestra.run.smithi044.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 160 kB/s | 19 kB 00:00 2023-11-16T16:36:00.408 INFO:teuthology.orchestra.run.smithi044.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 239 kB/s | 24 kB 00:00 2023-11-16T16:36:00.483 INFO:teuthology.orchestra.run.smithi044.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 291 kB/s | 29 kB 00:00 2023-11-16T16:36:00.508 INFO:teuthology.orchestra.run.smithi044.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 274 kB/s | 27 kB 00:00 2023-11-16T16:36:00.575 INFO:teuthology.orchestra.run.smithi077.stdout:(26/131): rbd-nbd-18.0.0-3905.g5e4a743a.el8.x86 899 kB/s | 180 kB 00:00 2023-11-16T16:36:00.601 INFO:teuthology.orchestra.run.smithi044.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 2.1 MB/s | 255 kB 00:00 2023-11-16T16:36:00.626 INFO:teuthology.orchestra.run.smithi044.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.1 MB/s | 251 kB 00:00 2023-11-16T16:36:00.683 INFO:teuthology.orchestra.run.smithi077.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3905.g 213 kB/s | 23 kB 00:00 2023-11-16T16:36:00.710 INFO:teuthology.orchestra.run.smithi044.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 565 kB/s | 61 kB 00:00 2023-11-16T16:36:00.743 INFO:teuthology.orchestra.run.smithi044.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 729 kB/s | 85 kB 00:00 2023-11-16T16:36:00.819 INFO:teuthology.orchestra.run.smithi044.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 211 kB/s | 23 kB 00:00 2023-11-16T16:36:00.852 INFO:teuthology.orchestra.run.smithi044.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.1 MB/s | 70 kB 00:00 2023-11-16T16:36:00.978 INFO:teuthology.orchestra.run.smithi044.stdout:(70/131): thrift-0.13.0-2.el8.x86_64.rpm 7.4 MB/s | 1.7 MB 00:00 2023-11-16T16:36:01.078 INFO:teuthology.orchestra.run.smithi044.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.1 MB/s | 110 kB 00:00 2023-11-16T16:36:01.142 INFO:teuthology.orchestra.run.smithi077.stdout:(28/131): ceph-test-18.0.0-3905.g5e4a743a.el8.x 6.5 MB/s | 41 MB 00:06 2023-11-16T16:36:01.192 INFO:teuthology.orchestra.run.smithi077.stdout:(29/131): ceph-mgr-cephadm-18.0.0-3905.g5e4a743 250 kB/s | 127 kB 00:00 2023-11-16T16:36:01.267 INFO:teuthology.orchestra.run.smithi077.stdout:(30/131): librgw2-18.0.0-3905.g5e4a743a.el8.x86 2.2 MB/s | 6.5 MB 00:02 2023-11-16T16:36:01.320 INFO:teuthology.orchestra.run.smithi044.stdout:(72/131): ceph-common-18.0.0-3905.g5e4a743a.el8 1.6 MB/s | 22 MB 00:13 2023-11-16T16:36:01.345 INFO:teuthology.orchestra.run.smithi044.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 203 kB/s | 54 kB 00:00 2023-11-16T16:36:01.495 INFO:teuthology.orchestra.run.smithi044.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2023-11-16T16:36:01.554 INFO:teuthology.orchestra.run.smithi044.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.6 MB/s | 323 kB 00:00 2023-11-16T16:36:01.709 INFO:teuthology.orchestra.run.smithi077.stdout:(31/131): ceph-mgr-dashboard-18.0.0-3905.g5e4a7 3.0 MB/s | 1.7 MB 00:00 2023-11-16T16:36:01.768 INFO:teuthology.orchestra.run.smithi077.stdout:(32/131): ceph-mgr-rook-18.0.0-3905.g5e4a743a.e 812 kB/s | 47 kB 00:00 2023-11-16T16:36:01.779 INFO:teuthology.orchestra.run.smithi044.stdout:(76/131): python3-scipy-1.0.0-21.module+el8.5.0 15 MB/s | 14 MB 00:00 2023-11-16T16:36:01.793 INFO:teuthology.orchestra.run.smithi077.stdout:(33/131): ceph-mgr-modules-core-18.0.0-3905.g5e 457 kB/s | 240 kB 00:00 2023-11-16T16:36:01.804 INFO:teuthology.orchestra.run.smithi044.stdout:(77/131): python3-markupsafe-0.23-19.el8.x86_64 126 kB/s | 39 kB 00:00 2023-11-16T16:36:01.827 INFO:teuthology.orchestra.run.smithi077.stdout:(34/131): ceph-prometheus-alerts-18.0.0-3905.g5 248 kB/s | 15 kB 00:00 2023-11-16T16:36:01.838 INFO:teuthology.orchestra.run.smithi044.stdout:(78/131): bison-3.0.4-10.el8.x86_64.rpm 2.4 MB/s | 688 kB 00:00 2023-11-16T16:36:01.927 INFO:teuthology.orchestra.run.smithi077.stdout:(35/131): cephadm-18.0.0-3905.g5e4a743a.el8.noa 835 kB/s | 83 kB 00:00 2023-11-16T16:36:01.938 INFO:teuthology.orchestra.run.smithi044.stdout:(79/131): python3-jinja2-2.10.1-3.el8.noarch.rp 3.3 MB/s | 538 kB 00:00 2023-11-16T16:36:01.964 INFO:teuthology.orchestra.run.smithi044.stdout:(80/131): python3-mako-1.0.6-13.el8.noarch.rpm 987 kB/s | 157 kB 00:00 2023-11-16T16:36:01.988 INFO:teuthology.orchestra.run.smithi044.stdout:(81/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2023-11-16T16:36:02.002 INFO:teuthology.orchestra.run.smithi077.stdout:(36/131): cephfs-top-18.0.0-3905.g5e4a743a.el8. 239 kB/s | 18 kB 00:00 2023-11-16T16:36:02.072 INFO:teuthology.orchestra.run.smithi044.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 5.2 MB/s | 713 kB 00:00 2023-11-16T16:36:02.097 INFO:teuthology.orchestra.run.smithi044.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 244 kB/s | 27 kB 00:00 2023-11-16T16:36:02.169 INFO:teuthology.orchestra.run.smithi077.stdout:(37/131): ceph-volume-18.0.0-3905.g5e4a743a.el8 704 kB/s | 265 kB 00:00 2023-11-16T16:36:02.195 INFO:teuthology.orchestra.run.smithi077.stdout:(38/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2023-11-16T16:36:02.206 INFO:teuthology.orchestra.run.smithi044.stdout:(84/131): flex-2.6.1-9.el8.x86_64.rpm 2.3 MB/s | 320 kB 00:00 2023-11-16T16:36:02.232 INFO:teuthology.orchestra.run.smithi044.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 329 kB/s | 44 kB 00:00 2023-11-16T16:36:02.245 INFO:teuthology.orchestra.run.smithi077.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2023-11-16T16:36:02.295 INFO:teuthology.orchestra.run.smithi077.stdout:(40/131): liboath-2.6.2-3.el8.x86_64.rpm 469 kB/s | 59 kB 00:00 2023-11-16T16:36:02.307 INFO:teuthology.orchestra.run.smithi044.stdout:(86/131): python3-babel-2.5.1-7.el8.noarch.rpm 14 MB/s | 4.8 MB 00:00 2023-11-16T16:36:02.337 INFO:teuthology.orchestra.run.smithi077.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 5.1 MB/s | 476 kB 00:00 2023-11-16T16:36:02.363 INFO:teuthology.orchestra.run.smithi077.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 650 kB/s | 44 kB 00:00 2023-11-16T16:36:02.382 INFO:teuthology.orchestra.run.smithi044.stdout:(87/131): python3-werkzeug-0.12.2-4.el8.noarch. 3.0 MB/s | 457 kB 00:00 2023-11-16T16:36:02.388 INFO:teuthology.orchestra.run.smithi077.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 3.6 MB/s | 185 kB 00:00 2023-11-16T16:36:02.422 INFO:teuthology.orchestra.run.smithi077.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 564 kB/s | 33 kB 00:00 2023-11-16T16:36:02.447 INFO:teuthology.orchestra.run.smithi077.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 278 kB/s | 16 kB 00:00 2023-11-16T16:36:02.466 INFO:teuthology.orchestra.run.smithi044.stdout:(88/131): python3-numpy-1.14.3-10.el8.x86_64.rp 14 MB/s | 3.7 MB 00:00 2023-11-16T16:36:02.491 INFO:teuthology.orchestra.run.smithi044.stdout:(89/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.2 MB/s | 225 kB 00:00 2023-11-16T16:36:02.506 INFO:teuthology.orchestra.run.smithi077.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 6.4 MB/s | 384 kB 00:00 2023-11-16T16:36:02.516 INFO:teuthology.orchestra.run.smithi044.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 479 kB/s | 64 kB 00:00 2023-11-16T16:36:02.531 INFO:teuthology.orchestra.run.smithi077.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.5 MB/s | 173 kB 00:00 2023-11-16T16:36:02.588 INFO:teuthology.orchestra.run.smithi077.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 993 kB/s | 81 kB 00:00 2023-11-16T16:36:02.615 INFO:teuthology.orchestra.run.smithi077.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 133 kB/s | 11 kB 00:00 2023-11-16T16:36:02.617 INFO:teuthology.orchestra.run.smithi044.stdout:(91/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.0 MB/s | 103 kB 00:00 2023-11-16T16:36:02.640 INFO:teuthology.orchestra.run.smithi077.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 339 kB/s | 18 kB 00:00 2023-11-16T16:36:02.642 INFO:teuthology.orchestra.run.smithi044.stdout:(92/131): python3-jsonpointer-1.10-11.el8.noarc 133 kB/s | 20 kB 00:00 2023-11-16T16:36:02.667 INFO:teuthology.orchestra.run.smithi044.stdout:(93/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 629 kB/s | 126 kB 00:00 2023-11-16T16:36:02.691 INFO:teuthology.orchestra.run.smithi077.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 977 kB/s | 49 kB 00:00 2023-11-16T16:36:02.726 INFO:teuthology.orchestra.run.smithi044.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 6.4 MB/s | 709 kB 00:00 2023-11-16T16:36:02.741 INFO:teuthology.orchestra.run.smithi077.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2023-11-16T16:36:02.774 INFO:teuthology.orchestra.run.smithi077.stdout:(53/131): python3-kubernetes-11.0.0-6.el8.noarc 7.3 MB/s | 1.2 MB 00:00 2023-11-16T16:36:02.800 INFO:teuthology.orchestra.run.smithi077.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.0 MB/s | 60 kB 00:00 2023-11-16T16:36:02.825 INFO:teuthology.orchestra.run.smithi077.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2023-11-16T16:36:02.834 INFO:teuthology.orchestra.run.smithi044.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 1.2 MB/s | 198 kB 00:00 2023-11-16T16:36:02.850 INFO:teuthology.orchestra.run.smithi077.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2023-11-16T16:36:02.875 INFO:teuthology.orchestra.run.smithi077.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 669 kB/s | 34 kB 00:00 2023-11-16T16:36:02.901 INFO:teuthology.orchestra.run.smithi077.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 3.8 MB/s | 196 kB 00:00 2023-11-16T16:36:02.926 INFO:teuthology.orchestra.run.smithi077.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 1.2 MB/s | 64 kB 00:00 2023-11-16T16:36:02.944 INFO:teuthology.orchestra.run.smithi044.stdout:(96/131): openblas-threads-0.3.15-3.el8.x86_64. 16 MB/s | 4.9 MB 00:00 2023-11-16T16:36:02.952 INFO:teuthology.orchestra.run.smithi077.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 372 kB/s | 19 kB 00:00 2023-11-16T16:36:02.977 INFO:teuthology.orchestra.run.smithi077.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 476 kB/s | 24 kB 00:00 2023-11-16T16:36:02.994 INFO:teuthology.orchestra.run.smithi044.stdout:(97/131): jq-1.6-3.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2023-11-16T16:36:03.002 INFO:teuthology.orchestra.run.smithi077.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 582 kB/s | 29 kB 00:00 2023-11-16T16:36:03.028 INFO:teuthology.orchestra.run.smithi077.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 545 kB/s | 27 kB 00:00 2023-11-16T16:36:03.053 INFO:teuthology.orchestra.run.smithi077.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 5.0 MB/s | 255 kB 00:00 2023-11-16T16:36:03.060 INFO:teuthology.orchestra.run.smithi044.stdout:(98/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.0 MB/s | 354 kB 00:00 2023-11-16T16:36:03.078 INFO:teuthology.orchestra.run.smithi077.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 4.9 MB/s | 251 kB 00:00 2023-11-16T16:36:03.103 INFO:teuthology.orchestra.run.smithi077.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2023-11-16T16:36:03.120 INFO:teuthology.orchestra.run.smithi077.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 2.0 MB/s | 85 kB 00:00 2023-11-16T16:36:03.146 INFO:teuthology.orchestra.run.smithi077.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 548 kB/s | 23 kB 00:00 2023-11-16T16:36:03.152 INFO:teuthology.orchestra.run.smithi044.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2023-11-16T16:36:03.171 INFO:teuthology.orchestra.run.smithi077.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-11-16T16:36:03.202 INFO:teuthology.orchestra.run.smithi044.stdout:(100/131): postgresql-contrib-10.21-2.module+el 5.6 MB/s | 808 kB 00:00 2023-11-16T16:36:03.230 INFO:teuthology.orchestra.run.smithi077.stdout:(70/131): thrift-0.13.0-2.el8.x86_64.rpm 16 MB/s | 1.7 MB 00:00 2023-11-16T16:36:03.245 INFO:teuthology.orchestra.run.smithi044.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2023-11-16T16:36:03.328 INFO:teuthology.orchestra.run.smithi044.stdout:(102/131): postgresql-server-10.21-2.module+el8 15 MB/s | 5.1 MB 00:00 2023-11-16T16:36:03.354 INFO:teuthology.orchestra.run.smithi044.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.8 MB/s | 200 kB 00:00 2023-11-16T16:36:03.363 INFO:teuthology.orchestra.run.smithi077.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.5 MB/s | 200 kB 00:00 2023-11-16T16:36:03.380 INFO:teuthology.orchestra.run.smithi077.stdout:(72/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 224 kB/s | 47 kB 00:00 2023-11-16T16:36:03.412 INFO:teuthology.orchestra.run.smithi044.stdout:(104/131): postgresql-10.21-2.module+el8.6.0+15 5.8 MB/s | 1.5 MB 00:00 2023-11-16T16:36:03.465 INFO:teuthology.orchestra.run.smithi044.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.9 MB/s | 220 kB 00:00 2023-11-16T16:36:03.496 INFO:teuthology.orchestra.run.smithi044.stdout:(106/131): python3-jwt-1.6.1-2.el8.noarch.rpm 254 kB/s | 43 kB 00:00 2023-11-16T16:36:03.497 INFO:teuthology.orchestra.run.smithi077.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 319 kB/s | 43 kB 00:00 2023-11-16T16:36:03.531 INFO:teuthology.orchestra.run.smithi077.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.4 MB/s | 220 kB 00:00 2023-11-16T16:36:03.554 INFO:teuthology.orchestra.run.smithi044.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 974 kB/s | 87 kB 00:00 2023-11-16T16:36:03.580 INFO:teuthology.orchestra.run.smithi044.stdout:(108/131): keyutils-libs-devel-1.5.10-9.el8.x86 287 kB/s | 48 kB 00:00 2023-11-16T16:36:03.581 INFO:teuthology.orchestra.run.smithi077.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 575 kB/s | 48 kB 00:00 2023-11-16T16:36:03.656 INFO:teuthology.orchestra.run.smithi077.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 695 kB/s | 87 kB 00:00 2023-11-16T16:36:03.698 INFO:teuthology.orchestra.run.smithi077.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 365 kB/s | 43 kB 00:00 2023-11-16T16:36:03.774 INFO:teuthology.orchestra.run.smithi077.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 933 kB/s | 109 kB 00:00 2023-11-16T16:36:03.832 INFO:teuthology.orchestra.run.smithi077.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.7 MB/s | 229 kB 00:00 2023-11-16T16:36:03.914 INFO:teuthology.orchestra.run.smithi044.stdout:(109/131): python3-requests-oauthlib-1.0.0-1.el 102 kB/s | 43 kB 00:00 2023-11-16T16:36:03.916 INFO:teuthology.orchestra.run.smithi077.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.9 MB/s | 560 kB 00:00 2023-11-16T16:36:03.964 INFO:teuthology.orchestra.run.smithi044.stdout:(110/131): python3-pycparser-2.14-14.el8.noarch 267 kB/s | 109 kB 00:00 2023-11-16T16:36:03.990 INFO:teuthology.orchestra.run.smithi044.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 559 kB/s | 229 kB 00:00 2023-11-16T16:36:04.008 INFO:teuthology.orchestra.run.smithi077.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 107 kB/s | 19 kB 00:00 2023-11-16T16:36:04.056 INFO:teuthology.orchestra.run.smithi044.stdout:(112/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.9 MB/s | 560 kB 00:00 2023-11-16T16:36:04.067 INFO:teuthology.orchestra.run.smithi077.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 1.0 MB/s | 155 kB 00:00 2023-11-16T16:36:04.115 INFO:teuthology.orchestra.run.smithi044.stdout:(113/131): libaio-devel-0.3.112-1.el8.x86_64.rp 124 kB/s | 19 kB 00:00 2023-11-16T16:36:04.140 INFO:teuthology.orchestra.run.smithi044.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 1.0 MB/s | 155 kB 00:00 2023-11-16T16:36:04.150 INFO:teuthology.orchestra.run.smithi077.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 3.8 MB/s | 559 kB 00:00 2023-11-16T16:36:04.209 INFO:teuthology.orchestra.run.smithi077.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 126 kB/s | 18 kB 00:00 2023-11-16T16:36:04.224 INFO:teuthology.orchestra.run.smithi044.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 3.3 MB/s | 559 kB 00:00 2023-11-16T16:36:04.249 INFO:teuthology.orchestra.run.smithi044.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 134 kB/s | 18 kB 00:00 2023-11-16T16:36:04.291 INFO:teuthology.orchestra.run.smithi044.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.9 MB/s | 605 kB 00:00 2023-11-16T16:36:04.334 INFO:teuthology.orchestra.run.smithi077.stdout:(85/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.9 MB/s | 238 kB 00:00 2023-11-16T16:36:04.341 INFO:teuthology.orchestra.run.smithi044.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.0 MB/s | 238 kB 00:00 2023-11-16T16:36:04.516 INFO:teuthology.orchestra.run.smithi044.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 321 kB/s | 72 kB 00:00 2023-11-16T16:36:04.543 INFO:teuthology.orchestra.run.smithi077.stdout:(86/131): pcre2-devel-10.32-2.el8.x86_64.rpm 1.5 MB/s | 605 kB 00:00 2023-11-16T16:36:04.575 INFO:teuthology.orchestra.run.smithi044.stdout:(120/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 7.1 MB/s | 2.3 MB 00:00 2023-11-16T16:36:04.625 INFO:teuthology.orchestra.run.smithi044.stdout:(121/131): fuse-2.9.7-15.el8.x86_64.rpm 292 kB/s | 83 kB 00:00 2023-11-16T16:36:04.626 INFO:teuthology.orchestra.run.smithi077.stdout:(87/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 8.0 MB/s | 2.3 MB 00:00 2023-11-16T16:36:04.660 INFO:teuthology.orchestra.run.smithi077.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 618 kB/s | 72 kB 00:00 2023-11-16T16:36:04.777 INFO:teuthology.orchestra.run.smithi077.stdout:(89/131): elfutils-libelf-devel-0.186-1.el8.x86 512 kB/s | 60 kB 00:00 2023-11-16T16:36:04.818 INFO:teuthology.orchestra.run.smithi044.stdout:(122/131): elfutils-libelf-devel-0.186-1.el8.x8 199 kB/s | 60 kB 00:00 2023-11-16T16:36:04.868 INFO:teuthology.orchestra.run.smithi044.stdout:(123/131): libuuid-devel-2.32.1-35.el8.x86_64.r 336 kB/s | 98 kB 00:00 2023-11-16T16:36:04.869 INFO:teuthology.orchestra.run.smithi077.stdout:(90/131): fuse-2.9.7-15.el8.x86_64.rpm 341 kB/s | 83 kB 00:00 2023-11-16T16:36:04.895 INFO:teuthology.orchestra.run.smithi077.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 838 kB/s | 98 kB 00:00 2023-11-16T16:36:04.902 INFO:teuthology.orchestra.run.smithi044.stdout:(124/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 209 kB/s | 58 kB 00:00 2023-11-16T16:36:04.953 INFO:teuthology.orchestra.run.smithi077.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 690 kB/s | 58 kB 00:00 2023-11-16T16:36:04.969 INFO:teuthology.orchestra.run.smithi044.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 257 kB/s | 39 kB 00:00 2023-11-16T16:36:04.985 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-11-16T16:36:05.011 INFO:teuthology.orchestra.run.smithi044.stdout:(126/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 4.4 MB/s | 644 kB 00:00 2023-11-16T16:36:05.012 INFO:teuthology.orchestra.run.smithi077.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 330 kB/s | 39 kB 00:00 2023-11-16T16:36:05.036 INFO:teuthology.orchestra.run.smithi044.stdout:(127/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.2 MB/s | 171 kB 00:00 2023-11-16T16:36:05.054 INFO:teuthology.orchestra.run.smithi077.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 6.3 MB/s | 644 kB 00:00 2023-11-16T16:36:05.119 INFO:teuthology.orchestra.run.smithi044.stdout:(128/131): NetworkManager-initscripts-updown-1. 915 kB/s | 138 kB 00:00 2023-11-16T16:36:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-16T16:36:05.146 INFO:teuthology.orchestra.run.smithi077.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.3 MB/s | 171 kB 00:00 2023-11-16T16:36:05.188 INFO:teuthology.orchestra.run.smithi077.stdout:(96/131): NetworkManager-initscripts-updown-1.3 1.0 MB/s | 138 kB 00:00 2023-11-16T16:36:05.288 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-16T16:36:05.320 INFO:teuthology.orchestra.run.smithi044.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 3.9 MB/s | 798 kB 00:00 2023-11-16T16:36:05.478 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-11-16T16:36:05.646 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-16T16:36:05.662 INFO:teuthology.orchestra.run.smithi077.stdout:(97/131): ceph-mgr-diskprediction-local-18.0.0- 1.7 MB/s | 7.4 MB 00:04 2023-11-16T16:36:05.681 INFO:teuthology.orchestra.run.smithi077.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 224 kB/s | 110 kB 00:00 2023-11-16T16:36:05.743 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-16T16:36:05.781 INFO:teuthology.orchestra.run.smithi077.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 459 kB/s | 54 kB 00:00 2023-11-16T16:36:05.828 INFO:teuthology.orchestra.run.smithi044.stdout:(130/131): librbd1-18.0.0-3905.g5e4a743a.el8.x8 4.5 MB/s | 3.6 MB 00:00 2023-11-16T16:36:05.856 INFO:teuthology.orchestra.run.smithi077.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2023-11-16T16:36:05.954 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-16T16:36:05.998 INFO:teuthology.orchestra.run.smithi077.stdout:(101/131): python3-scipy-1.0.0-21.module+el8.5. 16 MB/s | 14 MB 00:00 2023-11-16T16:36:06.023 INFO:teuthology.orchestra.run.smithi077.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 233 kB/s | 39 kB 00:00 2023-11-16T16:36:06.049 INFO:teuthology.orchestra.run.smithi077.stdout:(103/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.2 MB/s | 323 kB 00:00 2023-11-16T16:36:06.141 INFO:teuthology.orchestra.run.smithi077.stdout:(104/131): bison-3.0.4-10.el8.x86_64.rpm 4.7 MB/s | 688 kB 00:00 2023-11-16T16:36:06.174 INFO:teuthology.orchestra.run.smithi077.stdout:(105/131): python3-jinja2-2.10.1-3.el8.noarch.r 3.5 MB/s | 538 kB 00:00 2023-11-16T16:36:06.199 INFO:teuthology.orchestra.run.smithi077.stdout:(106/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.0 MB/s | 157 kB 00:00 2023-11-16T16:36:06.250 INFO:teuthology.orchestra.run.smithi077.stdout:(107/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2023-11-16T16:36:06.333 INFO:teuthology.orchestra.run.smithi077.stdout:(108/131): automake-1.16.1-7.el8.noarch.rpm 4.4 MB/s | 713 kB 00:00 2023-11-16T16:36:06.357 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-16T16:36:06.409 INFO:teuthology.orchestra.run.smithi077.stdout:(109/131): python3-jsonpatch-1.21-2.el8.noarch. 167 kB/s | 27 kB 00:00 2023-11-16T16:36:06.492 INFO:teuthology.orchestra.run.smithi077.stdout:(110/131): flex-2.6.1-9.el8.x86_64.rpm 2.0 MB/s | 320 kB 00:00 2023-11-16T16:36:06.545 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:36:06.567 INFO:teuthology.orchestra.run.smithi077.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 13 MB/s | 4.8 MB 00:00 2023-11-16T16:36:06.592 INFO:teuthology.orchestra.run.smithi077.stdout:(112/131): python3-prettytable-0.7.2-14.el8.noa 239 kB/s | 44 kB 00:00 2023-11-16T16:36:06.676 INFO:teuthology.orchestra.run.smithi077.stdout:(113/131): python3-werkzeug-0.12.2-4.el8.noarch 4.1 MB/s | 457 kB 00:00 2023-11-16T16:36:06.701 INFO:teuthology.orchestra.run.smithi077.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 2.0 MB/s | 225 kB 00:00 2023-11-16T16:36:06.808 INFO:teuthology.orchestra.run.smithi077.stdout:(115/131): python3-numpy-1.14.3-10.el8.x86_64.r 12 MB/s | 3.7 MB 00:00 2023-11-16T16:36:06.835 INFO:teuthology.orchestra.run.smithi077.stdout:(116/131): uuid-1.6.2-43.el8.x86_64.rpm 403 kB/s | 64 kB 00:00 2023-11-16T16:36:06.869 INFO:teuthology.orchestra.run.smithi077.stdout:(117/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 754 kB/s | 126 kB 00:00 2023-11-16T16:36:06.920 INFO:teuthology.orchestra.run.smithi077.stdout:(118/131): python3-jsonpointer-1.10-11.el8.noar 180 kB/s | 20 kB 00:00 2023-11-16T16:36:06.941 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:36:07.003 INFO:teuthology.orchestra.run.smithi077.stdout:(119/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 618 kB/s | 103 kB 00:00 2023-11-16T16:36:07.095 INFO:teuthology.orchestra.run.smithi077.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 2.1 MB/s | 198 kB 00:00 2023-11-16T16:36:07.096 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:36:07.121 INFO:teuthology.orchestra.run.smithi077.stdout:(121/131): libtool-2.4.6-25.el8.x86_64.rpm 2.8 MB/s | 709 kB 00:00 2023-11-16T16:36:07.211 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:36:07.279 INFO:teuthology.orchestra.run.smithi077.stdout:(122/131): openblas-threads-0.3.15-3.el8.x86_64 14 MB/s | 4.9 MB 00:00 2023-11-16T16:36:07.304 INFO:teuthology.orchestra.run.smithi077.stdout:(123/131): jq-1.6-3.el8.x86_64.rpm 1.1 MB/s | 202 kB 00:00 2023-11-16T16:36:07.370 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:36:07.372 INFO:teuthology.orchestra.run.smithi077.stdout:(124/131): openblas-0.3.15-3.el8.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-11-16T16:36:07.422 INFO:teuthology.orchestra.run.smithi077.stdout:(125/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2023-11-16T16:36:07.466 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:36:07.564 INFO:teuthology.orchestra.run.smithi077.stdout:(126/131): postgresql-server-10.21-2.module+el8 20 MB/s | 5.1 MB 00:00 2023-11-16T16:36:07.605 INFO:teuthology.orchestra.run.smithi077.stdout:(127/131): postgresql-contrib-10.21-2.module+el 3.4 MB/s | 808 kB 00:00 2023-11-16T16:36:07.656 INFO:teuthology.orchestra.run.smithi077.stdout:(128/131): postgresql-10.21-2.module+el8.6.0+15 6.4 MB/s | 1.5 MB 00:00 2023-11-16T16:36:07.705 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 9/134 2023-11-16T16:36:07.781 INFO:teuthology.orchestra.run.smithi077.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 6.2 MB/s | 798 kB 00:00 2023-11-16T16:36:07.900 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-16T16:36:08.140 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-16T16:36:08.211 INFO:teuthology.orchestra.run.smithi044.stdout:(131/131): librados2-18.0.0-3905.g5e4a743a.el8. 1.1 MB/s | 3.7 MB 00:03 2023-11-16T16:36:08.216 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2023-11-16T16:36:08.216 INFO:teuthology.orchestra.run.smithi044.stdout:Total 9.5 MB/s | 197 MB 00:20 2023-11-16T16:36:08.540 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-16T16:36:08.749 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-16T16:36:08.773 INFO:teuthology.orchestra.run.smithi077.stdout:(130/131): librados2-18.0.0-3905.g5e4a743a.el8. 3.0 MB/s | 3.7 MB 00:01 2023-11-16T16:36:08.862 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T16:36:08.955 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T16:36:09.079 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T16:36:09.143 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T16:36:09.270 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-16T16:36:09.406 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-16T16:36:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-16T16:36:09.630 INFO:teuthology.orchestra.run.smithi077.stdout:(131/131): librbd1-18.0.0-3905.g5e4a743a.el8.x8 1.8 MB/s | 3.6 MB 00:02 2023-11-16T16:36:09.633 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-16T16:36:09.633 INFO:teuthology.orchestra.run.smithi077.stdout:Total 7.7 MB/s | 197 MB 00:25 2023-11-16T16:36:09.988 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2023-11-16T16:36:10.135 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2023-11-16T16:36:10.135 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2023-11-16T16:36:10.167 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-16T16:36:10.337 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-16T16:36:10.494 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-16T16:36:10.614 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-16T16:36:10.614 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-16T16:36:10.934 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-16T16:36:11.095 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-11-16T16:36:11.702 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-16T16:36:11.706 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-16T16:36:11.715 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2023-11-16T16:36:11.720 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2023-11-16T16:36:11.783 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-11-16T16:36:12.034 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-11-16T16:36:12.262 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-11-16T16:36:12.482 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-11-16T16:36:12.660 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-11-16T16:36:13.106 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-11-16T16:36:13.359 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-16T16:36:13.535 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 26/134 2023-11-16T16:36:13.664 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 27/134 2023-11-16T16:36:14.531 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 28/134 2023-11-16T16:36:14.858 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-11-16T16:36:15.120 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-11-16T16:36:15.358 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-11-16T16:36:15.561 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-16T16:36:16.474 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-16T16:36:16.687 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-16T16:36:16.878 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-16T16:36:17.008 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2023-11-16T16:36:17.039 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-11-16T16:36:17.142 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-11-16T16:36:17.172 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-16T16:36:17.307 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-16T16:36:17.358 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-16T16:36:17.422 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-11-16T16:36:17.568 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-11-16T16:36:17.645 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-11-16T16:36:17.764 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-16T16:36:17.810 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-11-16T16:36:17.872 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-16T16:36:17.922 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-16T16:36:17.972 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 5/134 2023-11-16T16:36:18.054 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-16T16:36:18.140 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-16T16:36:18.250 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 36/134 2023-11-16T16:36:18.349 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 5/134 2023-11-16T16:36:18.422 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 37/134 2023-11-16T16:36:18.547 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:36:18.575 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-11-16T16:36:18.638 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-16T16:36:18.673 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:36:18.762 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-11-16T16:36:18.856 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:36:18.897 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:36:18.929 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-11-16T16:36:18.995 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:36:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-11-16T16:36:19.177 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:36:19.292 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-11-16T16:36:19.324 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:36:19.339 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-11-16T16:36:19.514 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:36:19.525 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-11-16T16:36:19.616 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-16T16:36:19.656 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:36:19.825 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-16T16:36:19.839 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:36:19.900 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-11-16T16:36:19.991 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:36:20.161 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 9/134 2023-11-16T16:36:20.205 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-11-16T16:36:20.212 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-16T16:36:20.395 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-11-16T16:36:20.449 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-16T16:36:20.451 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-16T16:36:20.588 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T16:36:20.596 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-11-16T16:36:20.678 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-16T16:36:20.693 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T16:36:20.815 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-11-16T16:36:20.822 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T16:36:21.002 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T16:36:21.114 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-16T16:36:21.151 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-11-16T16:36:21.208 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-11-16T16:36:21.353 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-16T16:36:21.359 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-11-16T16:36:21.386 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-11-16T16:36:21.489 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T16:36:21.545 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-11-16T16:36:21.556 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-11-16T16:36:21.606 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T16:36:21.731 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-11-16T16:36:21.781 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T16:36:21.806 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-11-16T16:36:21.869 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T16:36:21.905 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-11-16T16:36:22.005 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-16T16:36:22.046 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-11-16T16:36:22.063 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-11-16T16:36:22.196 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-11-16T16:36:22.217 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-16T16:36:22.344 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-11-16T16:36:22.389 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-16T16:36:22.597 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-11-16T16:36:22.809 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-11-16T16:36:22.855 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-11-16T16:36:22.963 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-11-16T16:36:23.066 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-11-16T16:36:23.116 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-16T16:36:23.233 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-11-16T16:36:23.289 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-16T16:36:23.394 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-11-16T16:36:23.568 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-11-16T16:36:23.569 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-11-16T16:36:23.640 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-11-16T16:36:23.771 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-11-16T16:36:23.864 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-11-16T16:36:24.047 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-16T16:36:24.054 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-11-16T16:36:24.218 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-16T16:36:24.280 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-11-16T16:36:24.302 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-16T16:36:24.466 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-11-16T16:36:24.517 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-11-16T16:36:24.693 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-11-16T16:36:24.694 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-11-16T16:36:24.825 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-16T16:36:24.877 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-11-16T16:36:25.003 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-16T16:36:25.114 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-16T16:36:25.148 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-11-16T16:36:25.242 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-16T16:36:25.311 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 26/134 2023-11-16T16:36:25.440 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 27/134 2023-11-16T16:36:25.440 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-16T16:36:25.509 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-11-16T16:36:25.580 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-16T16:36:25.667 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 28/134 2023-11-16T16:36:25.711 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-16T16:36:25.789 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-11-16T16:36:25.888 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-11-16T16:36:25.933 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-16T16:36:26.055 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-11-16T16:36:26.111 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-16T16:36:26.241 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-11-16T16:36:26.285 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-16T16:36:26.445 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-16T16:36:26.505 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-11-16T16:36:26.573 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-16T16:36:26.763 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-16T16:36:26.795 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-16T16:36:26.995 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-16T16:36:27.088 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 26/134 2023-11-16T16:36:27.173 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-16T16:36:27.228 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 27/134 2023-11-16T16:36:27.403 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-16T16:36:27.625 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-16T16:36:27.644 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-11-16T16:36:27.782 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-16T16:36:27.999 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-16T16:36:28.334 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 28/134 2023-11-16T16:36:28.658 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-11-16T16:36:28.674 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-11-16T16:36:28.958 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-11-16T16:36:28.967 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-11-16T16:36:29.082 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-16T16:36:29.222 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-11-16T16:36:29.289 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-11-16T16:36:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-16T16:36:29.466 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-11-16T16:36:29.488 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-16T16:36:29.574 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-16T16:36:29.682 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-16T16:36:29.872 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-16T16:36:29.874 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-16T16:36:30.120 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 36/134 2023-11-16T16:36:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-16T16:36:30.277 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 37/134 2023-11-16T16:36:30.314 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-16T16:36:30.436 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-11-16T16:36:30.588 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-11-16T16:36:30.770 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-11-16T16:36:30.846 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-11-16T16:36:31.042 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-16T16:36:31.080 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-11-16T16:36:31.183 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T16:36:31.245 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T16:36:31.291 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-11-16T16:36:31.392 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 93/134 2023-11-16T16:36:31.462 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-11-16T16:36:31.525 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-11-16T16:36:31.686 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-16T16:36:31.752 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-11-16T16:36:31.917 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-11-16T16:36:32.022 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-11-16T16:36:32.078 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-11-16T16:36:32.261 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-16T16:36:32.273 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-11-16T16:36:32.448 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-11-16T16:36:32.454 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-16T16:36:32.599 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-11-16T16:36:32.697 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 36/134 2023-11-16T16:36:32.795 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-11-16T16:36:32.901 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 37/134 2023-11-16T16:36:33.010 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-11-16T16:36:33.097 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-11-16T16:36:33.275 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-11-16T16:36:33.376 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-11-16T16:36:33.500 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-11-16T16:36:33.654 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-11-16T16:36:33.722 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-11-16T16:36:33.881 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-11-16T16:36:33.963 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-11-16T16:36:34.089 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-11-16T16:36:34.206 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-11-16T16:36:34.442 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-11-16T16:36:34.541 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-11-16T16:36:34.762 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-11-16T16:36:34.863 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-11-16T16:36:34.981 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:36:35.020 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-11-16T16:36:35.055 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-11-16T16:36:35.200 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:36:35.223 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-11-16T16:36:35.314 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-11-16T16:36:35.467 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-11-16T16:36:35.541 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-11-16T16:36:35.633 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-11-16T16:36:35.785 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:36:35.824 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-11-16T16:36:35.906 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-11-16T16:36:36.027 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T16:36:36.108 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T16:36:36.123 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-11-16T16:36:36.140 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-11-16T16:36:36.258 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T16:36:36.325 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-11-16T16:36:36.367 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-11-16T16:36:36.559 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-11-16T16:36:36.591 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-11-16T16:36:36.747 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-11-16T16:36:36.804 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-11-16T16:36:36.902 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-11-16T16:36:36.969 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-16T16:36:37.197 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-16T16:36:37.255 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-11-16T16:36:37.456 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-16T16:36:37.693 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-16T16:36:37.707 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-11-16T16:36:37.891 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-16T16:36:37.998 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-11-16T16:36:38.047 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-16T16:36:38.246 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-16T16:36:38.246 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-11-16T16:36:38.405 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-11-16T16:36:38.457 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-16T16:36:38.583 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-11-16T16:36:38.673 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-16T16:36:38.812 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-11-16T16:36:38.909 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-11-16T16:36:38.923 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-16T16:36:39.059 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-16T16:36:39.149 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-11-16T16:36:39.325 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-16T16:36:39.352 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-11-16T16:36:39.557 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-16T16:36:39.587 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-16T16:36:39.688 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-16T16:36:39.750 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-16T16:36:39.888 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-11-16T16:36:40.051 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-16T16:36:40.090 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-11-16T16:36:40.256 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-16T16:36:40.268 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-16T16:36:40.452 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-16T16:36:40.624 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-16T16:36:40.720 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-16T16:36:40.925 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-16T16:36:41.129 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-16T16:36:41.318 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-16T16:36:41.469 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-16T16:36:41.690 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-16T16:36:41.829 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-16T16:36:41.898 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-16T16:36:42.097 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-16T16:36:42.148 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-16T16:36:42.291 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-16T16:36:42.410 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-16T16:36:42.428 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-16T16:36:42.675 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-16T16:36:42.754 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-16T16:36:43.028 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-16T16:36:43.082 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-16T16:36:43.211 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-16T16:36:43.265 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-16T16:36:43.476 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-16T16:36:43.680 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-16T16:36:43.881 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-16T16:36:44.048 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-16T16:36:44.122 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-16T16:36:44.252 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T16:36:44.325 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T16:36:44.511 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 93/134 2023-11-16T16:36:45.541 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-16T16:36:45.850 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-16T16:36:46.127 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-16T16:36:46.486 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-16T16:36:46.780 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-16T16:36:47.018 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-16T16:36:47.855 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-16T16:36:48.019 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T16:36:48.110 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:36:48.115 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T16:36:48.319 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 93/134 2023-11-16T16:36:48.344 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:36:48.932 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:36:49.239 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T16:36:49.320 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T16:36:49.460 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T16:36:53.823 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:36:54.061 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:36:54.789 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:36:54.931 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T16:36:54.931 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /sys 2023-11-16T16:36:54.931 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /proc 2023-11-16T16:36:54.931 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev 2023-11-16T16:36:54.931 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /run 2023-11-16T16:36:54.931 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2023-11-16T16:36:54.931 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /var/tmp 2023-11-16T16:36:54.931 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /home 2023-11-16T16:36:54.932 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /tmp 2023-11-16T16:36:54.932 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev 2023-11-16T16:36:54.932 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-11-16T16:36:55.047 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T16:36:55.221 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T16:36:55.271 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T16:36:56.057 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T16:36:56.831 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T16:36:56.982 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T16:36:57.200 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T16:36:57.354 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T16:36:57.804 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T16:36:57.979 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T16:36:57.997 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T16:36:59.739 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T16:36:59.942 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T16:37:00.068 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 102/134 2023-11-16T16:37:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 103/134 2023-11-16T16:37:00.445 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T16:37:00.656 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T16:37:00.906 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T16:37:01.866 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T16:37:02.012 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T16:37:02.101 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T16:37:02.468 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 107/134 2023-11-16T16:37:02.754 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T16:37:02.789 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T16:37:02.953 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T16:37:02.965 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T16:37:04.210 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 110/134 2023-11-16T16:37:04.442 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T16:37:04.476 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T16:37:04.636 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T16:37:04.685 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T16:37:04.861 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T16:37:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T16:37:05.345 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T16:37:08.603 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T16:37:08.603 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2023-11-16T16:37:08.603 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2023-11-16T16:37:08.603 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2023-11-16T16:37:08.603 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /run 2023-11-16T16:37:08.603 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2023-11-16T16:37:08.603 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2023-11-16T16:37:08.603 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2023-11-16T16:37:08.603 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2023-11-16T16:37:08.603 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2023-11-16T16:37:08.603 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-16T16:37:08.966 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T16:37:09.240 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T16:37:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 115/134 2023-11-16T16:37:09.668 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-11-16T16:37:09.868 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-11-16T16:37:10.524 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T16:37:10.686 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T16:37:10.722 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-11-16T16:37:10.904 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T16:37:11.041 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-11-16T16:37:11.082 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T16:37:11.238 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 119/134 2023-11-16T16:37:11.397 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-16T16:37:11.531 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T16:37:11.580 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 121/134 2023-11-16T16:37:11.666 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T16:37:11.683 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T16:37:11.719 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T16:37:13.360 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T16:37:13.571 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T16:37:13.696 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 102/134 2023-11-16T16:37:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:13.919 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 103/134 2023-11-16T16:37:14.110 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:14.124 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T16:37:14.128 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:14.269 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 124/134 2023-11-16T16:37:14.337 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T16:37:14.568 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 125/134 2023-11-16T16:37:14.647 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T16:37:14.857 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 126/134 2023-11-16T16:37:15.128 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-16T16:37:15.259 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T16:37:15.268 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T16:37:15.451 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-11-16T16:37:15.607 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T16:37:15.634 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-11-16T16:37:15.720 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-11-16T16:37:15.782 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T16:37:15.872 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T16:37:15.884 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-11-16T16:37:15.897 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-11-16T16:37:15.897 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:16.087 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:16.087 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T16:37:16.272 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 107/134 2023-11-16T16:37:16.297 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T16:37:16.483 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:16.508 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T16:37:16.542 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T16:37:16.698 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T16:37:16.710 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T16:37:17.002 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:17.003 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 1/134 2023-11-16T16:37:17.003 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2/134 2023-11-16T16:37:17.003 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 3/134 2023-11-16T16:37:17.003 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 4/134 2023-11-16T16:37:17.003 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 5/134 2023-11-16T16:37:17.003 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:37:17.003 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:37:17.003 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:37:17.004 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 9/134 2023-11-16T16:37:17.004 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 10/134 2023-11-16T16:37:17.004 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 11/134 2023-11-16T16:37:17.004 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 12/134 2023-11-16T16:37:17.004 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 13/134 2023-11-16T16:37:17.004 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 14/134 2023-11-16T16:37:17.004 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 15/134 2023-11-16T16:37:17.004 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 16/134 2023-11-16T16:37:17.005 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 17/134 2023-11-16T16:37:17.005 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 18/134 2023-11-16T16:37:17.005 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 19/134 2023-11-16T16:37:17.005 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 20/134 2023-11-16T16:37:17.005 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 21/134 2023-11-16T16:37:17.005 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 22/134 2023-11-16T16:37:17.005 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 23/134 2023-11-16T16:37:17.005 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 24/134 2023-11-16T16:37:17.006 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 25/134 2023-11-16T16:37:17.006 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 26/134 2023-11-16T16:37:17.006 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 27/134 2023-11-16T16:37:17.006 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 28/134 2023-11-16T16:37:17.006 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 29/134 2023-11-16T16:37:17.006 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 30/134 2023-11-16T16:37:17.006 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 31/134 2023-11-16T16:37:17.006 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 32/134 2023-11-16T16:37:17.007 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 33/134 2023-11-16T16:37:17.007 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 34/134 2023-11-16T16:37:17.007 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 35/134 2023-11-16T16:37:17.007 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 36/134 2023-11-16T16:37:17.007 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 37/134 2023-11-16T16:37:17.007 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 38/134 2023-11-16T16:37:17.007 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-16T16:37:17.007 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-16T16:37:17.008 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-16T16:37:17.008 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-16T16:37:17.008 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-16T16:37:17.008 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-16T16:37:17.008 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-16T16:37:17.008 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-16T16:37:17.008 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-16T16:37:17.008 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-16T16:37:17.008 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-16T16:37:17.009 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-16T16:37:17.009 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-16T16:37:17.009 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-16T16:37:17.009 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-16T16:37:17.009 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-16T16:37:17.009 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-16T16:37:17.009 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-16T16:37:17.009 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-16T16:37:17.010 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-16T16:37:17.010 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-16T16:37:17.010 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-16T16:37:17.010 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-16T16:37:17.010 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-16T16:37:17.010 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-16T16:37:17.010 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-16T16:37:17.010 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-16T16:37:17.011 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-16T16:37:17.011 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-16T16:37:17.011 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-16T16:37:17.011 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-16T16:37:17.011 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-16T16:37:17.011 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-16T16:37:17.011 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-11-16T16:37:17.011 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-11-16T16:37:17.011 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-11-16T16:37:17.012 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-11-16T16:37:17.012 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-11-16T16:37:17.012 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-11-16T16:37:17.012 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-11-16T16:37:17.012 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-11-16T16:37:17.012 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-11-16T16:37:17.012 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-11-16T16:37:17.012 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-11-16T16:37:17.012 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-11-16T16:37:17.013 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-11-16T16:37:17.013 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-11-16T16:37:17.013 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-11-16T16:37:17.013 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-11-16T16:37:17.013 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-11-16T16:37:17.013 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-11-16T16:37:17.013 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-11-16T16:37:17.013 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-11-16T16:37:17.013 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-11-16T16:37:17.014 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-11-16T16:37:17.014 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-11-16T16:37:17.014 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-11-16T16:37:17.014 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-11-16T16:37:17.014 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-11-16T16:37:17.014 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-11-16T16:37:17.014 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-11-16T16:37:17.014 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-11-16T16:37:17.015 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-11-16T16:37:17.015 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-11-16T16:37:17.015 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-11-16T16:37:17.015 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-11-16T16:37:17.015 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-11-16T16:37:17.015 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-11-16T16:37:17.015 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-11-16T16:37:17.015 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-11-16T16:37:17.015 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-11-16T16:37:17.016 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-11-16T16:37:17.016 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-11-16T16:37:17.016 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-11-16T16:37:17.016 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-11-16T16:37:17.016 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-11-16T16:37:17.016 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-11-16T16:37:17.016 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-11-16T16:37:17.016 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-11-16T16:37:17.016 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-11-16T16:37:17.017 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-11-16T16:37:17.017 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-16T16:37:17.017 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-11-16T16:37:17.017 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T16:37:17.017 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-11-16T16:37:17.017 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-11-16T16:37:17.017 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-11-16T16:37:17.017 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-11-16T16:37:17.017 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-11-16T16:37:17.018 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T16:37:17.018 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 129/134 2023-11-16T16:37:17.018 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-16T16:37:17.018 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 131/134 2023-11-16T16:37:17.018 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:17.018 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-16T16:37:17.260 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T16:37:17.260 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /sys 2023-11-16T16:37:17.260 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /proc 2023-11-16T16:37:17.260 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /dev 2023-11-16T16:37:17.260 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /run 2023-11-16T16:37:17.260 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /mnt 2023-11-16T16:37:17.261 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /var/tmp 2023-11-16T16:37:17.261 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /home 2023-11-16T16:37:17.261 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /tmp 2023-11-16T16:37:17.261 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /dev 2023-11-16T16:37:17.261 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:37:17.683 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T16:37:17.978 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 110/134 2023-11-16T16:37:18.205 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T16:37:18.239 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T16:37:18.373 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T16:37:18.421 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T16:37:18.565 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T16:37:18.909 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T16:37:19.056 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T16:37:19.490 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T16:37:19.554 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:19.554 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:48 ago on Thu 16 Nov 2023 04:35:31 PM UTC. 2023-11-16T16:37:19.669 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T16:37:19.949 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T16:37:20.165 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T16:37:20.714 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T16:37:20.924 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T16:37:20.968 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T16:37:22.961 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T16:37:23.860 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 115/134 2023-11-16T16:37:23.890 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T16:37:24.105 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T16:37:24.187 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-11-16T16:37:24.231 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 102/134 2023-11-16T16:37:24.358 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 117/134 2023-11-16T16:37:24.412 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 103/134 2023-11-16T16:37:24.606 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T16:37:24.645 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-11-16T16:37:24.805 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-11-16T16:37:24.877 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T16:37:25.002 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-11-16T16:37:25.201 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T16:37:25.268 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-11-16T16:37:25.442 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 121/134 2023-11-16T16:37:25.590 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T16:37:26.362 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T16:37:26.523 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T16:37:26.658 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T16:37:27.042 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 107/134 2023-11-16T16:37:27.269 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T16:37:27.337 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T16:37:27.556 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T16:37:27.586 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T16:37:27.778 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:27.911 INFO:teuthology.orchestra.run.smithi176.stdout:Installed products updated. 2023-11-16T16:37:28.063 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:28.081 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-11-16T16:37:28.124 INFO:teuthology.orchestra.run.smithi176.stdout:Upgraded: 2023-11-16T16:37:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-16T16:37:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.125 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-11-16T16:37:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-16T16:37:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: automake-1.16.1-7.el8.noarch 2023-11-16T16:37:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: bison-3.0.4-10.el8.x86_64 2023-11-16T16:37:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-16T16:37:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: flex-2.6.1-9.el8.x86_64 2023-11-16T16:37:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-16T16:37:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-16T16:37:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: jq-1.6-3.el8.x86_64 2023-11-16T16:37:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-16T16:37:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-16T16:37:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-16T16:37:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-16T16:37:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-16T16:37:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-16T16:37:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: libpq-13.5-1.el8.x86_64 2023-11-16T16:37:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-16T16:37:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-16T16:37:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-16T16:37:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-16T16:37:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-16T16:37:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-16T16:37:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-16T16:37:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-16T16:37:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-16T16:37:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-16T16:37:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-16T16:37:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-16T16:37:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-16T16:37:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-16T16:37:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-16T16:37:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-16T16:37:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-16T16:37:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:37:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:37:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:37:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-16T16:37:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-16T16:37:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-16T16:37:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-16T16:37:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-16T16:37:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-16T16:37:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-16T16:37:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-16T16:37:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-16T16:37:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-16T16:37:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-16T16:37:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-16T16:37:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-16T16:37:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-16T16:37:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-16T16:37:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-16T16:37:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-16T16:37:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-16T16:37:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-16T16:37:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-16T16:37:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-16T16:37:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-16T16:37:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-16T16:37:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-16T16:37:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-16T16:37:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-16T16:37:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-16T16:37:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend-2.6-1.el8.noarch 2023-11-16T16:37:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-16T16:37:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-16T16:37:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-16T16:37:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-16T16:37:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-16T16:37:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-16T16:37:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-16T16:37:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-16T16:37:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-16T16:37:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-16T16:37:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-16T16:37:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-16T16:37:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-16T16:37:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-16T16:37:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-16T16:37:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-16T16:37:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-16T16:37:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-16T16:37:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-16T16:37:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-16T16:37:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-16T16:37:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-16T16:37:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-16T16:37:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-16T16:37:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-16T16:37:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-16T16:37:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-16T16:37:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-11-16T16:37:28.135 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-11-16T16:37:28.264 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 124/134 2023-11-16T16:37:28.301 DEBUG:teuthology.parallel:result is None 2023-11-16T16:37:28.554 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 125/134 2023-11-16T16:37:28.900 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 126/134 2023-11-16T16:37:29.005 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 110/134 2023-11-16T16:37:29.225 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T16:37:29.259 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-16T16:37:29.289 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T16:37:29.431 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T16:37:29.480 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-11-16T16:37:29.508 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T16:37:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-11-16T16:37:29.689 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T16:37:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-11-16T16:37:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-11-16T16:37:29.921 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-11-16T16:37:29.930 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-11-16T16:37:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-11-16T16:37:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:30.128 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T16:37:30.272 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T16:37:30.308 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:30.308 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T16:37:30.554 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T16:37:30.746 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:31.294 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:31.294 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 1/134 2023-11-16T16:37:31.294 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2/134 2023-11-16T16:37:31.294 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 3/134 2023-11-16T16:37:31.295 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 4/134 2023-11-16T16:37:31.295 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 5/134 2023-11-16T16:37:31.295 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:37:31.295 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:37:31.295 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:37:31.295 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 9/134 2023-11-16T16:37:31.295 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 10/134 2023-11-16T16:37:31.295 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 11/134 2023-11-16T16:37:31.295 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 12/134 2023-11-16T16:37:31.296 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 13/134 2023-11-16T16:37:31.296 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 14/134 2023-11-16T16:37:31.296 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 15/134 2023-11-16T16:37:31.296 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 16/134 2023-11-16T16:37:31.296 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 17/134 2023-11-16T16:37:31.296 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 18/134 2023-11-16T16:37:31.296 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 19/134 2023-11-16T16:37:31.296 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 20/134 2023-11-16T16:37:31.296 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 21/134 2023-11-16T16:37:31.296 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 22/134 2023-11-16T16:37:31.297 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 23/134 2023-11-16T16:37:31.297 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 24/134 2023-11-16T16:37:31.297 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 25/134 2023-11-16T16:37:31.297 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 26/134 2023-11-16T16:37:31.297 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 27/134 2023-11-16T16:37:31.297 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 28/134 2023-11-16T16:37:31.297 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 29/134 2023-11-16T16:37:31.297 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 30/134 2023-11-16T16:37:31.297 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 31/134 2023-11-16T16:37:31.298 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 32/134 2023-11-16T16:37:31.298 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 33/134 2023-11-16T16:37:31.298 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 34/134 2023-11-16T16:37:31.298 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 35/134 2023-11-16T16:37:31.298 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 36/134 2023-11-16T16:37:31.298 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 37/134 2023-11-16T16:37:31.298 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 38/134 2023-11-16T16:37:31.298 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-16T16:37:31.298 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-16T16:37:31.299 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-16T16:37:31.299 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-16T16:37:31.299 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-16T16:37:31.299 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-16T16:37:31.299 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-16T16:37:31.299 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-16T16:37:31.299 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-16T16:37:31.299 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-16T16:37:31.299 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-16T16:37:31.300 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-16T16:37:31.300 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-16T16:37:31.300 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-16T16:37:31.300 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-16T16:37:31.300 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-16T16:37:31.300 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-16T16:37:31.300 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-16T16:37:31.300 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-16T16:37:31.300 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-16T16:37:31.300 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-16T16:37:31.301 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-16T16:37:31.301 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-16T16:37:31.301 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-16T16:37:31.301 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-16T16:37:31.301 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-16T16:37:31.301 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-16T16:37:31.301 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-16T16:37:31.301 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-16T16:37:31.301 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-16T16:37:31.301 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-16T16:37:31.302 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-16T16:37:31.302 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-16T16:37:31.302 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-11-16T16:37:31.302 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-11-16T16:37:31.302 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-11-16T16:37:31.302 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-11-16T16:37:31.302 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-11-16T16:37:31.302 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-11-16T16:37:31.302 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-11-16T16:37:31.303 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-11-16T16:37:31.303 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-11-16T16:37:31.303 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-11-16T16:37:31.303 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-11-16T16:37:31.303 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-11-16T16:37:31.303 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-11-16T16:37:31.303 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-11-16T16:37:31.303 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-11-16T16:37:31.303 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-11-16T16:37:31.303 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-11-16T16:37:31.303 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-11-16T16:37:31.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-11-16T16:37:31.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-11-16T16:37:31.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-11-16T16:37:31.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-11-16T16:37:31.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-11-16T16:37:31.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-11-16T16:37:31.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-11-16T16:37:31.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-11-16T16:37:31.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-11-16T16:37:31.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-11-16T16:37:31.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-11-16T16:37:31.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-11-16T16:37:31.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-11-16T16:37:31.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-11-16T16:37:31.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-11-16T16:37:31.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-11-16T16:37:31.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-11-16T16:37:31.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-11-16T16:37:31.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-11-16T16:37:31.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-11-16T16:37:31.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-11-16T16:37:31.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-11-16T16:37:31.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-11-16T16:37:31.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-11-16T16:37:31.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-11-16T16:37:31.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-11-16T16:37:31.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-11-16T16:37:31.307 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-11-16T16:37:31.307 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-11-16T16:37:31.307 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-11-16T16:37:31.307 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-11-16T16:37:31.307 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-11-16T16:37:31.307 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-11-16T16:37:31.307 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-11-16T16:37:31.307 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-11-16T16:37:31.307 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-11-16T16:37:31.307 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-11-16T16:37:31.308 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-11-16T16:37:31.308 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-11-16T16:37:31.308 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 129/134 2023-11-16T16:37:31.308 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-16T16:37:31.308 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 131/134 2023-11-16T16:37:31.308 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:31.308 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-16T16:37:33.794 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:33.794 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:01 ago on Thu 16 Nov 2023 04:35:32 PM UTC. 2023-11-16T16:37:34.797 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T16:37:35.108 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 115/134 2023-11-16T16:37:35.313 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-11-16T16:37:35.585 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-11-16T16:37:36.918 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-11-16T16:37:37.247 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-11-16T16:37:37.453 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 119/134 2023-11-16T16:37:37.627 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-16T16:37:37.826 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 121/134 2023-11-16T16:37:37.972 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T16:37:41.693 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:41.980 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:42.018 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:42.082 INFO:teuthology.orchestra.run.smithi077.stdout:Installed products updated. 2023-11-16T16:37:42.292 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-16T16:37:42.292 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2023-11-16T16:37:42.292 INFO:teuthology.orchestra.run.smithi077.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-16T16:37:42.292 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.292 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.292 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-16T16:37:42.293 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-16T16:37:42.293 INFO:teuthology.orchestra.run.smithi077.stdout: automake-1.16.1-7.el8.noarch 2023-11-16T16:37:42.293 INFO:teuthology.orchestra.run.smithi077.stdout: bison-3.0.4-10.el8.x86_64 2023-11-16T16:37:42.293 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.293 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.293 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.293 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.293 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:42.293 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.294 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.294 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.294 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:42.294 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:42.294 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:42.294 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:42.294 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:42.294 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.294 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.295 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:42.295 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.295 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.295 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.295 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:42.295 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:42.295 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.295 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:42.295 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-16T16:37:42.296 INFO:teuthology.orchestra.run.smithi077.stdout: flex-2.6.1-9.el8.x86_64 2023-11-16T16:37:42.296 INFO:teuthology.orchestra.run.smithi077.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-16T16:37:42.296 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-16T16:37:42.296 INFO:teuthology.orchestra.run.smithi077.stdout: jq-1.6-3.el8.x86_64 2023-11-16T16:37:42.296 INFO:teuthology.orchestra.run.smithi077.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-16T16:37:42.296 INFO:teuthology.orchestra.run.smithi077.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-16T16:37:42.296 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-16T16:37:42.296 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.296 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.297 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.297 INFO:teuthology.orchestra.run.smithi077.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-16T16:37:42.297 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-16T16:37:42.297 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-16T16:37:42.297 INFO:teuthology.orchestra.run.smithi077.stdout: libpq-13.5-1.el8.x86_64 2023-11-16T16:37:42.297 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-16T16:37:42.297 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-16T16:37:42.297 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.298 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.298 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-16T16:37:42.298 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.298 INFO:teuthology.orchestra.run.smithi077.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-16T16:37:42.298 INFO:teuthology.orchestra.run.smithi077.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-16T16:37:42.298 INFO:teuthology.orchestra.run.smithi077.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-16T16:37:42.298 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-16T16:37:42.298 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-16T16:37:42.298 INFO:teuthology.orchestra.run.smithi077.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-16T16:37:42.299 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-16T16:37:42.299 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-16T16:37:42.299 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-16T16:37:42.299 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-16T16:37:42.299 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-16T16:37:42.299 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-16T16:37:42.299 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-16T16:37:42.299 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-16T16:37:42.299 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:37:42.300 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:37:42.300 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:37:42.300 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-16T16:37:42.300 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-16T16:37:42.300 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-16T16:37:42.300 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-16T16:37:42.300 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-16T16:37:42.300 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.300 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.301 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.301 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-16T16:37:42.301 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-16T16:37:42.301 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-16T16:37:42.301 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-16T16:37:42.301 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-16T16:37:42.301 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-16T16:37:42.301 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-16T16:37:42.301 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-16T16:37:42.302 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-16T16:37:42.302 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-16T16:37:42.302 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-16T16:37:42.302 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-16T16:37:42.302 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-16T16:37:42.302 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-16T16:37:42.302 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-16T16:37:42.302 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-16T16:37:42.303 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-16T16:37:42.303 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-16T16:37:42.303 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-16T16:37:42.303 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-16T16:37:42.303 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-16T16:37:42.303 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-16T16:37:42.303 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-2.6-1.el8.noarch 2023-11-16T16:37:42.303 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-16T16:37:42.303 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-16T16:37:42.304 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-16T16:37:42.304 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-16T16:37:42.304 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-16T16:37:42.304 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-16T16:37:42.304 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.304 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.304 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-16T16:37:42.304 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-16T16:37:42.305 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.305 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-16T16:37:42.305 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-16T16:37:42.305 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-16T16:37:42.305 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-16T16:37:42.305 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-16T16:37:42.305 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-16T16:37:42.305 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-16T16:37:42.305 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-16T16:37:42.306 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-16T16:37:42.306 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-16T16:37:42.306 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-16T16:37:42.306 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-16T16:37:42.306 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-16T16:37:42.306 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.306 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.306 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:42.306 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-16T16:37:42.307 INFO:teuthology.orchestra.run.smithi077.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-16T16:37:42.307 INFO:teuthology.orchestra.run.smithi077.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-16T16:37:42.307 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-16T16:37:42.307 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-16T16:37:42.307 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 124/134 2023-11-16T16:37:42.308 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-16T16:37:42.308 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-16T16:37:42.308 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-16T16:37:42.490 DEBUG:teuthology.parallel:result is None 2023-11-16T16:37:42.506 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 125/134 2023-11-16T16:37:42.865 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 126/134 2023-11-16T16:37:43.189 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-16T16:37:43.370 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T16:37:43.395 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T16:37:43.640 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-11-16T16:37:43.828 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-11-16T16:37:43.945 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-11-16T16:37:44.145 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-11-16T16:37:44.176 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-11-16T16:37:44.176 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:44.411 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:44.411 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T16:37:44.661 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T16:37:44.887 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:45.178 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:45.178 INFO:teuthology.orchestra.run.smithi044.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-11-16T16:37:45.178 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:37:45.852 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 1/134 2023-11-16T16:37:45.852 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2/134 2023-11-16T16:37:45.852 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 3/134 2023-11-16T16:37:45.852 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 4/134 2023-11-16T16:37:45.853 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 5/134 2023-11-16T16:37:45.853 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:37:45.853 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:37:45.853 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:37:45.853 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 9/134 2023-11-16T16:37:45.853 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 10/134 2023-11-16T16:37:45.853 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 11/134 2023-11-16T16:37:45.853 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 12/134 2023-11-16T16:37:45.853 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 13/134 2023-11-16T16:37:45.853 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 14/134 2023-11-16T16:37:45.854 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 15/134 2023-11-16T16:37:45.854 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 16/134 2023-11-16T16:37:45.854 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 17/134 2023-11-16T16:37:45.854 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 18/134 2023-11-16T16:37:45.854 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 19/134 2023-11-16T16:37:45.854 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 20/134 2023-11-16T16:37:45.854 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 21/134 2023-11-16T16:37:45.854 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 22/134 2023-11-16T16:37:45.854 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 23/134 2023-11-16T16:37:45.854 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 24/134 2023-11-16T16:37:45.855 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 25/134 2023-11-16T16:37:45.865 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 26/134 2023-11-16T16:37:45.865 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 27/134 2023-11-16T16:37:45.865 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 28/134 2023-11-16T16:37:45.866 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 29/134 2023-11-16T16:37:45.866 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 30/134 2023-11-16T16:37:45.866 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 31/134 2023-11-16T16:37:45.866 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 32/134 2023-11-16T16:37:45.866 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 33/134 2023-11-16T16:37:45.866 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 34/134 2023-11-16T16:37:45.866 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 35/134 2023-11-16T16:37:45.866 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 36/134 2023-11-16T16:37:45.866 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 37/134 2023-11-16T16:37:45.867 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 38/134 2023-11-16T16:37:45.867 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-16T16:37:45.867 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-16T16:37:45.867 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-16T16:37:45.867 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-16T16:37:45.867 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-16T16:37:45.867 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-16T16:37:45.867 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-16T16:37:45.867 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-16T16:37:45.867 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-16T16:37:45.868 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-16T16:37:45.868 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-16T16:37:45.868 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-16T16:37:45.868 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-16T16:37:45.868 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-16T16:37:45.868 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-16T16:37:45.868 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-16T16:37:45.868 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-16T16:37:45.868 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-16T16:37:45.868 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-16T16:37:45.869 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-16T16:37:45.869 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-16T16:37:45.869 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-16T16:37:45.869 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-16T16:37:45.869 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-16T16:37:45.869 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-16T16:37:45.869 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-16T16:37:45.869 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-16T16:37:45.869 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-16T16:37:45.869 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-16T16:37:45.869 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-16T16:37:45.870 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-16T16:37:45.870 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-16T16:37:45.870 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-16T16:37:45.870 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-11-16T16:37:45.870 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-11-16T16:37:45.870 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-11-16T16:37:45.870 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-11-16T16:37:45.870 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-11-16T16:37:45.870 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-11-16T16:37:45.871 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-11-16T16:37:45.871 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-11-16T16:37:45.871 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-11-16T16:37:45.871 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-11-16T16:37:45.871 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-11-16T16:37:45.871 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-11-16T16:37:45.871 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-11-16T16:37:45.871 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-11-16T16:37:45.871 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-11-16T16:37:45.871 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-11-16T16:37:45.871 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-11-16T16:37:45.872 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-11-16T16:37:45.872 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-11-16T16:37:45.872 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-11-16T16:37:45.872 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-11-16T16:37:45.872 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-11-16T16:37:45.872 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-11-16T16:37:45.872 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-11-16T16:37:45.872 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-11-16T16:37:45.872 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-11-16T16:37:45.872 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-11-16T16:37:45.872 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-11-16T16:37:45.873 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-11-16T16:37:45.873 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-11-16T16:37:45.873 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-11-16T16:37:45.873 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-11-16T16:37:45.873 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-11-16T16:37:45.873 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-11-16T16:37:45.873 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-11-16T16:37:45.873 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-11-16T16:37:45.873 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-11-16T16:37:45.873 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-11-16T16:37:45.873 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-11-16T16:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-11-16T16:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-11-16T16:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-11-16T16:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-11-16T16:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-11-16T16:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-11-16T16:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-11-16T16:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-11-16T16:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-11-16T16:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-16T16:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-11-16T16:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T16:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-11-16T16:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-11-16T16:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-11-16T16:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-11-16T16:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-11-16T16:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T16:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 129/134 2023-11-16T16:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-16T16:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 131/134 2023-11-16T16:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:45.876 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-16T16:37:48.605 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:48.605 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:14 ago on Thu 16 Nov 2023 04:35:34 PM UTC. 2023-11-16T16:37:54.299 INFO:teuthology.orchestra.run.smithi044.stdout:Installed products updated. 2023-11-16T16:37:54.543 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:37:54.543 INFO:teuthology.orchestra.run.smithi044.stdout:Upgraded: 2023-11-16T16:37:54.543 INFO:teuthology.orchestra.run.smithi044.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-16T16:37:54.543 INFO:teuthology.orchestra.run.smithi044.stdout: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.543 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.543 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2023-11-16T16:37:54.543 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-16T16:37:54.543 INFO:teuthology.orchestra.run.smithi044.stdout: automake-1.16.1-7.el8.noarch 2023-11-16T16:37:54.543 INFO:teuthology.orchestra.run.smithi044.stdout: bison-3.0.4-10.el8.x86_64 2023-11-16T16:37:54.544 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.544 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.544 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.544 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.544 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:54.544 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.544 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.544 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.544 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:54.544 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:54.544 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:54.544 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:54.544 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:54.545 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.545 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.545 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:54.545 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.545 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.545 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.545 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:54.545 INFO:teuthology.orchestra.run.smithi044.stdout: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:54.545 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.545 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:54.545 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-16T16:37:54.545 INFO:teuthology.orchestra.run.smithi044.stdout: flex-2.6.1-9.el8.x86_64 2023-11-16T16:37:54.545 INFO:teuthology.orchestra.run.smithi044.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-16T16:37:54.546 INFO:teuthology.orchestra.run.smithi044.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-16T16:37:54.546 INFO:teuthology.orchestra.run.smithi044.stdout: jq-1.6-3.el8.x86_64 2023-11-16T16:37:54.546 INFO:teuthology.orchestra.run.smithi044.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-16T16:37:54.546 INFO:teuthology.orchestra.run.smithi044.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-16T16:37:54.546 INFO:teuthology.orchestra.run.smithi044.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-16T16:37:54.546 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.546 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.546 INFO:teuthology.orchestra.run.smithi044.stdout: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.546 INFO:teuthology.orchestra.run.smithi044.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-16T16:37:54.546 INFO:teuthology.orchestra.run.smithi044.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-16T16:37:54.546 INFO:teuthology.orchestra.run.smithi044.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-16T16:37:54.546 INFO:teuthology.orchestra.run.smithi044.stdout: libpq-13.5-1.el8.x86_64 2023-11-16T16:37:54.546 INFO:teuthology.orchestra.run.smithi044.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-16T16:37:54.547 INFO:teuthology.orchestra.run.smithi044.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-16T16:37:54.547 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.547 INFO:teuthology.orchestra.run.smithi044.stdout: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.547 INFO:teuthology.orchestra.run.smithi044.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-16T16:37:54.547 INFO:teuthology.orchestra.run.smithi044.stdout: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.547 INFO:teuthology.orchestra.run.smithi044.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-16T16:37:54.547 INFO:teuthology.orchestra.run.smithi044.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-16T16:37:54.547 INFO:teuthology.orchestra.run.smithi044.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-16T16:37:54.547 INFO:teuthology.orchestra.run.smithi044.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-16T16:37:54.547 INFO:teuthology.orchestra.run.smithi044.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-16T16:37:54.547 INFO:teuthology.orchestra.run.smithi044.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-16T16:37:54.547 INFO:teuthology.orchestra.run.smithi044.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-16T16:37:54.547 INFO:teuthology.orchestra.run.smithi044.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-16T16:37:54.547 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-16T16:37:54.548 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-16T16:37:54.548 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-16T16:37:54.548 INFO:teuthology.orchestra.run.smithi044.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-16T16:37:54.548 INFO:teuthology.orchestra.run.smithi044.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-16T16:37:54.548 INFO:teuthology.orchestra.run.smithi044.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-16T16:37:54.548 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:37:54.548 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:37:54.548 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:37:54.548 INFO:teuthology.orchestra.run.smithi044.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-16T16:37:54.548 INFO:teuthology.orchestra.run.smithi044.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-16T16:37:54.548 INFO:teuthology.orchestra.run.smithi044.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-16T16:37:54.549 INFO:teuthology.orchestra.run.smithi044.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-16T16:37:54.549 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-16T16:37:54.549 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.549 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.549 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.549 INFO:teuthology.orchestra.run.smithi044.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-16T16:37:54.549 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-16T16:37:54.549 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-16T16:37:54.549 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-16T16:37:54.549 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-16T16:37:54.549 INFO:teuthology.orchestra.run.smithi044.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-16T16:37:54.550 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-16T16:37:54.550 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-16T16:37:54.550 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-16T16:37:54.550 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-16T16:37:54.550 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-16T16:37:54.550 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-16T16:37:54.550 INFO:teuthology.orchestra.run.smithi044.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-16T16:37:54.550 INFO:teuthology.orchestra.run.smithi044.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-16T16:37:54.550 INFO:teuthology.orchestra.run.smithi044.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-16T16:37:54.550 INFO:teuthology.orchestra.run.smithi044.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-16T16:37:54.550 INFO:teuthology.orchestra.run.smithi044.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-16T16:37:54.550 INFO:teuthology.orchestra.run.smithi044.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-16T16:37:54.551 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-16T16:37:54.551 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-16T16:37:54.551 INFO:teuthology.orchestra.run.smithi044.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-16T16:37:54.551 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-16T16:37:54.551 INFO:teuthology.orchestra.run.smithi044.stdout: python3-portend-2.6-1.el8.noarch 2023-11-16T16:37:54.551 INFO:teuthology.orchestra.run.smithi044.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-16T16:37:54.551 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-16T16:37:54.551 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-16T16:37:54.551 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-16T16:37:54.551 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-16T16:37:54.551 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-16T16:37:54.551 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.552 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.552 INFO:teuthology.orchestra.run.smithi044.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-16T16:37:54.552 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-16T16:37:54.552 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.552 INFO:teuthology.orchestra.run.smithi044.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-16T16:37:54.552 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-16T16:37:54.552 INFO:teuthology.orchestra.run.smithi044.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-16T16:37:54.552 INFO:teuthology.orchestra.run.smithi044.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-16T16:37:54.552 INFO:teuthology.orchestra.run.smithi044.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-16T16:37:54.552 INFO:teuthology.orchestra.run.smithi044.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-16T16:37:54.552 INFO:teuthology.orchestra.run.smithi044.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-16T16:37:54.552 INFO:teuthology.orchestra.run.smithi044.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-16T16:37:54.552 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-16T16:37:54.553 INFO:teuthology.orchestra.run.smithi044.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-16T16:37:54.553 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-16T16:37:54.553 INFO:teuthology.orchestra.run.smithi044.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-16T16:37:54.553 INFO:teuthology.orchestra.run.smithi044.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-16T16:37:54.553 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.553 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.553 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:54.553 INFO:teuthology.orchestra.run.smithi044.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-16T16:37:54.553 INFO:teuthology.orchestra.run.smithi044.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-16T16:37:54.553 INFO:teuthology.orchestra.run.smithi044.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-16T16:37:54.553 INFO:teuthology.orchestra.run.smithi044.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-16T16:37:54.553 INFO:teuthology.orchestra.run.smithi044.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-16T16:37:54.554 INFO:teuthology.orchestra.run.smithi044.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-16T16:37:54.554 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:37:54.554 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2023-11-16T16:37:54.834 DEBUG:teuthology.parallel:result is None 2023-11-16T16:37:54.835 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:37:54.978 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-16T16:37:55.034 INFO:teuthology.orchestra.run.smithi044.stdout:18.0.0-3905.g5e4a743a.el8 2023-11-16T16:37:55.035 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3905.g5e4a743a.el8 2023-11-16T16:37:55.035 INFO:teuthology.task.install:The correct ceph version 18.0.0-3905.g5e4a743a is installed. 2023-11-16T16:37:55.035 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:37:55.170 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-16T16:37:55.199 INFO:teuthology.orchestra.run.smithi077.stdout:18.0.0-3905.g5e4a743a.el8 2023-11-16T16:37:55.199 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3905.g5e4a743a.el8 2023-11-16T16:37:55.199 INFO:teuthology.task.install:The correct ceph version 18.0.0-3905.g5e4a743a is installed. 2023-11-16T16:37:55.199 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:37:55.332 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-16T16:37:55.362 INFO:teuthology.orchestra.run.smithi176.stdout:18.0.0-3905.g5e4a743a.el8 2023-11-16T16:37:55.362 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3905.g5e4a743a.el8 2023-11-16T16:37:55.362 INFO:teuthology.task.install:The correct ceph version 18.0.0-3905.g5e4a743a is installed. 2023-11-16T16:37:55.362 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-16T16:37:55.363 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:37:55.363 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T16:37:55.453 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-16T16:37:55.453 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T16:37:55.500 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-11-16T16:37:55.500 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T16:37:55.546 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-16T16:37:55.546 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:37:55.546 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T16:37:55.616 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T16:37:55.724 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-16T16:37:55.724 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T16:37:55.756 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T16:37:55.828 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-11-16T16:37:55.829 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T16:37:55.861 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T16:37:55.930 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-16T16:37:55.930 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:37:55.930 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T16:37:56.000 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T16:37:56.109 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-16T16:37:56.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T16:37:56.141 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T16:37:56.210 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-11-16T16:37:56.210 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T16:37:56.241 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T16:37:56.310 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-16T16:37:56.310 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:37:56.310 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T16:37:56.445 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T16:37:56.555 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-16T16:37:56.555 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T16:37:56.587 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T16:37:56.657 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-11-16T16:37:56.657 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T16:37:56.688 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T16:37:56.758 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-16T16:37:56.828 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2023-11-16T16:37:56.828 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:37:56.829 INFO:tasks.cephadm:Cluster fsid is 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:37:56.829 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-16T16:37:56.829 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.44', 'mon.b': '172.21.15.77', 'mon.c': '172.21.15.176'} 2023-11-16T16:37:56.829 INFO:tasks.cephadm:First mon is mon.a on smithi044 2023-11-16T16:37:56.829 INFO:tasks.cephadm:First mgr is x 2023-11-16T16:37:56.830 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-16T16:37:56.830 DEBUG:teuthology.orchestra.run.smithi044:> sudo hostname $(hostname -s) 2023-11-16T16:37:56.903 DEBUG:teuthology.orchestra.run.smithi077:> sudo hostname $(hostname -s) 2023-11-16T16:37:56.935 DEBUG:teuthology.orchestra.run.smithi176:> sudo hostname $(hostname -s) 2023-11-16T16:37:56.966 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-11-16T16:37:56.967 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:37:57.225 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/74186/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-3905-g5e4a743a', 'node_name': '172.21.5.32+adami02', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-3905.g5e4a743a'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-11-16 00:50:05.467524', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-choffman-62793', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2023-11-16T16:37:57.361 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-choffman-62793, sha1 5e4a743ad7fd1422ebd6c4dc950a21a303279207 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:37:57.361 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/x86_64/flavors/default/cephadm 2023-11-16T16:37:57.361 DEBUG:teuthology.orchestra.run.smithi044:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-16T16:37:57.985 INFO:teuthology.orchestra.run.smithi044.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Nov 16 16:37 /home/ubuntu/cephtest/cephadm 2023-11-16T16:37:57.986 DEBUG:teuthology.orchestra.run.smithi077:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-16T16:37:58.376 INFO:teuthology.orchestra.run.smithi077.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Nov 16 16:37 /home/ubuntu/cephtest/cephadm 2023-11-16T16:37:58.377 DEBUG:teuthology.orchestra.run.smithi176:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-16T16:37:58.927 INFO:teuthology.orchestra.run.smithi176.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Nov 16 16:37 /home/ubuntu/cephtest/cephadm 2023-11-16T16:37:58.927 DEBUG:teuthology.orchestra.run.smithi044:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-16T16:37:58.978 DEBUG:teuthology.orchestra.run.smithi077:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-16T16:37:58.999 DEBUG:teuthology.orchestra.run.smithi176:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-16T16:37:59.034 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 on all hosts... 2023-11-16T16:37:59.034 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 pull 2023-11-16T16:37:59.038 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 pull 2023-11-16T16:37:59.042 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 pull 2023-11-16T16:38:00.426 INFO:teuthology.orchestra.run.smithi176.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207... 2023-11-16T16:38:01.150 INFO:teuthology.orchestra.run.smithi077.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207... 2023-11-16T16:38:03.192 INFO:teuthology.orchestra.run.smithi044.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207... 2023-11-16T16:39:28.373 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2023-11-16T16:39:28.374 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev)", 2023-11-16T16:39:28.374 INFO:teuthology.orchestra.run.smithi077.stdout: "image_id": "d5b1f852d682af5926771a6e4760786b6147f99518b76fd91b07a1b9e99bfe99", 2023-11-16T16:39:28.374 INFO:teuthology.orchestra.run.smithi077.stdout: "repo_digests": [ 2023-11-16T16:39:28.374 INFO:teuthology.orchestra.run.smithi077.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:209515a30c262c9be4300b2075ceb5fd8912e5a4eccd3c9021b5585705456b42" 2023-11-16T16:39:28.374 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2023-11-16T16:39:28.374 INFO:teuthology.orchestra.run.smithi077.stdout:} 2023-11-16T16:39:29.101 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2023-11-16T16:39:29.101 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev)", 2023-11-16T16:39:29.102 INFO:teuthology.orchestra.run.smithi176.stdout: "image_id": "d5b1f852d682af5926771a6e4760786b6147f99518b76fd91b07a1b9e99bfe99", 2023-11-16T16:39:29.102 INFO:teuthology.orchestra.run.smithi176.stdout: "repo_digests": [ 2023-11-16T16:39:29.102 INFO:teuthology.orchestra.run.smithi176.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:209515a30c262c9be4300b2075ceb5fd8912e5a4eccd3c9021b5585705456b42" 2023-11-16T16:39:29.102 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2023-11-16T16:39:29.102 INFO:teuthology.orchestra.run.smithi176.stdout:} 2023-11-16T16:39:31.560 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:39:31.560 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev)", 2023-11-16T16:39:31.560 INFO:teuthology.orchestra.run.smithi044.stdout: "image_id": "d5b1f852d682af5926771a6e4760786b6147f99518b76fd91b07a1b9e99bfe99", 2023-11-16T16:39:31.560 INFO:teuthology.orchestra.run.smithi044.stdout: "repo_digests": [ 2023-11-16T16:39:31.560 INFO:teuthology.orchestra.run.smithi044.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:209515a30c262c9be4300b2075ceb5fd8912e5a4eccd3c9021b5585705456b42" 2023-11-16T16:39:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2023-11-16T16:39:31.561 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:39:31.586 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /etc/ceph 2023-11-16T16:39:31.660 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph 2023-11-16T16:39:31.695 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph 2023-11-16T16:39:31.731 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 777 /etc/ceph 2023-11-16T16:39:31.801 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /etc/ceph 2023-11-16T16:39:31.832 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /etc/ceph 2023-11-16T16:39:31.862 INFO:tasks.cephadm:Writing seed config... 2023-11-16T16:39:31.863 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-11-16T16:39:31.864 INFO:tasks.cephadm: override: [client] debug client = 20 2023-11-16T16:39:31.864 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-16T16:39:31.864 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-11-16T16:39:31.864 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-11-16T16:39:31.864 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-11-16T16:39:31.864 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-11-16T16:39:31.864 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-11-16T16:39:31.865 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-11-16T16:39:31.865 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-11-16T16:39:31.865 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-11-16T16:39:31.865 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-11-16T16:39:31.865 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-11-16T16:39:31.865 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-11-16T16:39:31.865 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-11-16T16:39:31.865 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-11-16T16:39:31.866 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-11-16T16:39:31.866 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-11-16T16:39:31.866 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-11-16T16:39:31.866 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-11-16T16:39:31.866 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-11-16T16:39:31.866 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2023-11-16T16:39:31.866 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-16T16:39:31.867 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-16T16:39:31.867 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-16T16:39:31.867 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-16T16:39:31.867 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-16T16:39:31.867 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-11-16T16:39:31.867 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-11-16T16:39:31.867 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2023-11-16T16:39:31.867 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-11-16T16:39:31.868 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-11-16T16:39:31.868 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-11-16T16:39:31.868 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-16T16:39:31.868 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-16T16:39:31.868 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-11-16T16:39:31.868 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-11-16T16:39:31.868 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-11-16T16:39:31.868 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-11-16T16:39:31.869 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-11-16T16:39:31.869 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-11-16T16:39:31.869 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-11-16T16:39:31.869 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-11-16T16:39:31.869 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2023-11-16T16:39:31.870 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:39:31.870 DEBUG:teuthology.orchestra.run.smithi044:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-16T16:39:31.913 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-11-16T16:39:31.914 DEBUG:teuthology.orchestra.run.smithi044:mon.a> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mon.a.service 2023-11-16T16:39:31.958 DEBUG:teuthology.orchestra.run.smithi044:mgr.x> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mgr.x.service 2023-11-16T16:39:32.005 INFO:tasks.cephadm:Bootstrapping... 2023-11-16T16:39:32.005 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 -v bootstrap --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.44 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-16T16:39:32.028 INFO:journalctl@ceph.mon.a.smithi044.stdout:-- Logs begin at Thu 2023-11-16 16:19:45 UTC. -- 2023-11-16T16:39:32.075 INFO:journalctl@ceph.mgr.x.smithi044.stdout:-- Logs begin at Thu 2023-11-16 16:19:45 UTC. -- 2023-11-16T16:39:32.357 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2023-11-16T16:39:32.357 INFO:teuthology.orchestra.run.smithi044.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207', '-v', 'bootstrap', '--fsid', '7f2b12a8-849e-11ee-8dbe-212e2dc638e7', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.44', '--skip-admin-label'] 2023-11-16T16:39:32.534 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout 4.1.1 2023-11-16T16:39:32.534 INFO:teuthology.orchestra.run.smithi044.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-11-16T16:39:32.535 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying podman|docker is present... 2023-11-16T16:39:32.707 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout 4.1.1 2023-11-16T16:39:32.707 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying lvm2 is present... 2023-11-16T16:39:32.707 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying time synchronization is in place... 2023-11-16T16:39:32.722 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-11-16T16:39:32.722 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-11-16T16:39:32.735 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-11-16T16:39:32.735 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2023-11-16T16:39:32.749 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout enabled 2023-11-16T16:39:32.762 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout active 2023-11-16T16:39:32.763 INFO:teuthology.orchestra.run.smithi044.stdout:Unit chronyd.service is enabled and running 2023-11-16T16:39:32.763 INFO:teuthology.orchestra.run.smithi044.stdout:Repeating the final host check... 2023-11-16T16:39:32.936 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout 4.1.1 2023-11-16T16:39:32.936 INFO:teuthology.orchestra.run.smithi044.stdout:podman (/bin/podman) version 4.1.1 is present 2023-11-16T16:39:32.937 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl is present 2023-11-16T16:39:32.937 INFO:teuthology.orchestra.run.smithi044.stdout:lvcreate is present 2023-11-16T16:39:32.950 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-11-16T16:39:32.950 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-11-16T16:39:32.963 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-11-16T16:39:32.963 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2023-11-16T16:39:32.977 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout enabled 2023-11-16T16:39:32.990 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout active 2023-11-16T16:39:32.990 INFO:teuthology.orchestra.run.smithi044.stdout:Unit chronyd.service is enabled and running 2023-11-16T16:39:32.990 INFO:teuthology.orchestra.run.smithi044.stdout:Host looks OK 2023-11-16T16:39:32.991 INFO:teuthology.orchestra.run.smithi044.stdout:Cluster fsid: 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:39:32.991 INFO:teuthology.orchestra.run.smithi044.stdout:Acquiring lock 140638972946920 on /run/cephadm/7f2b12a8-849e-11ee-8dbe-212e2dc638e7.lock 2023-11-16T16:39:32.991 INFO:teuthology.orchestra.run.smithi044.stdout:Lock 140638972946920 acquired on /run/cephadm/7f2b12a8-849e-11ee-8dbe-212e2dc638e7.lock 2023-11-16T16:39:32.991 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying IP 172.21.15.44 port 3300 ... 2023-11-16T16:39:32.992 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying IP 172.21.15.44 port 6789 ... 2023-11-16T16:39:32.992 INFO:teuthology.orchestra.run.smithi044.stdout:Base mon IP(s) is [172.21.15.44:3300, 172.21.15.44:6789], mon addrv is [v2:172.21.15.44:3300,v1:172.21.15.44:6789] 2023-11-16T16:39:33.000 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-11-16T16:39:33.000 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.44 metric 100 2023-11-16T16:39:33.007 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-11-16T16:39:33.007 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-11-16T16:39:33.007 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1697sec hoplimit 64 pref medium 2023-11-16T16:39:33.014 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-16T16:39:33.014 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-11-16T16:39:33.014 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-11-16T16:39:33.014 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-11-16T16:39:33.014 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1572/64 scope link 2023-11-16T16:39:33.014 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-11-16T16:39:33.015 INFO:teuthology.orchestra.run.smithi044.stdout:Mon IP `172.21.15.44` is in CIDR network `172.21.0.0/20` 2023-11-16T16:39:33.015 INFO:teuthology.orchestra.run.smithi044.stdout:Mon IP `172.21.15.44` is in CIDR network `172.21.0.0/20` 2023-11-16T16:39:33.015 INFO:teuthology.orchestra.run.smithi044.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-11-16T16:39:33.016 INFO:teuthology.orchestra.run.smithi044.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-16T16:39:33.017 INFO:teuthology.orchestra.run.smithi044.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207... 2023-11-16T16:39:33.493 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout d5b1f852d682af5926771a6e4760786b6147f99518b76fd91b07a1b9e99bfe99 2023-11-16T16:39:33.493 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207... 2023-11-16T16:39:33.494 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Getting image source signatures 2023-11-16T16:39:33.494 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Copying blob sha256:3a8f6e475435f9e3b5d92795b454ceb30085f6221d261249c26042723227dea7 2023-11-16T16:39:33.494 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Copying blob sha256:837e66e3ff89804ed1bb42786f80e6c23534ff65fe48526d5007449f4e1f79b0 2023-11-16T16:39:33.494 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Copying config sha256:d5b1f852d682af5926771a6e4760786b6147f99518b76fd91b07a1b9e99bfe99 2023-11-16T16:39:33.494 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Writing manifest to image destination 2023-11-16T16:39:33.494 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Storing signatures 2023-11-16T16:39:34.471 INFO:teuthology.orchestra.run.smithi044.stdout:ceph: stdout ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev) 2023-11-16T16:39:34.471 INFO:teuthology.orchestra.run.smithi044.stdout:Ceph version: ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev) 2023-11-16T16:39:34.471 INFO:teuthology.orchestra.run.smithi044.stdout:Extracting ceph user uid/gid from container image... 2023-11-16T16:39:35.249 INFO:teuthology.orchestra.run.smithi044.stdout:stat: stdout 167 167 2023-11-16T16:39:35.249 INFO:teuthology.orchestra.run.smithi044.stdout:Creating initial keys... 2023-11-16T16:39:36.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph-authtool: stdout AQDHRVZluQ6XJRAAU9zPsAlBLTxD8P5UBOPajA== 2023-11-16T16:39:36.904 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph-authtool: stdout AQDIRVZlrCcYHxAAp/nqefYeHhrMHwGanb61sw== 2023-11-16T16:39:37.885 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph-authtool: stdout AQDJRVZl9gEuGhAAlPWOCtdg9i9inIGfve9jtQ== 2023-11-16T16:39:37.886 INFO:teuthology.orchestra.run.smithi044.stdout:Creating initial monmap... 2023-11-16T16:39:38.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-16T16:39:38.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-11-16T16:39:38.765 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:39:38.765 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-16T16:39:38.765 INFO:teuthology.orchestra.run.smithi044.stdout:monmaptool for a [v2:172.21.15.44:3300,v1:172.21.15.44:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-16T16:39:38.765 INFO:teuthology.orchestra.run.smithi044.stdout:setting min_mon_release = pacific 2023-11-16T16:39:38.765 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: set fsid to 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:39:38.765 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-16T16:39:38.765 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:39:38.765 INFO:teuthology.orchestra.run.smithi044.stdout:Creating mon... 2023-11-16T16:39:39.850 INFO:teuthology.orchestra.run.smithi044.stdout:create mon.a on 2023-11-16T16:39:40.087 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-16T16:39:40.296 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-16T16:39:40.519 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7.target. 2023-11-16T16:39:40.520 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7.target. 2023-11-16T16:39:40.850 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mon.a 2023-11-16T16:39:40.850 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mon.a.service: Unit ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mon.a.service not loaded. 2023-11-16T16:39:41.055 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7.target.wants/ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mon.a.service → /etc/systemd/system/ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@.service. 2023-11-16T16:39:41.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:41 smithi044 systemd[1]: Starting Ceph mon.a for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:39:41.810 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:41 smithi044 podman[128198]: 2023-11-16T16:39:41.810 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:41 smithi044 bash[128198]: b3794ff1bd23df8d440dd928682add1d9e841bf6dd36311acc51f37bd94e4200 2023-11-16T16:39:41.872 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-16T16:39:41.872 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2023-11-16T16:39:41.889 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-16T16:39:41.889 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2023-11-16T16:39:41.889 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2023-11-16T16:39:41.889 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to enable service . firewalld.service is not available 2023-11-16T16:39:41.889 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mon to start... 2023-11-16T16:39:41.889 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mon... 2023-11-16T16:39:42.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:41 smithi044 ceph-mon[128238]: mkfs 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:39:42.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:41 smithi044 ceph-mon[128238]: mon.a is new leader, mons a in quorum (ranks 0) 2023-11-16T16:39:42.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:41 smithi044 systemd[1]: Started Ceph mon.a for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:39:42.975 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout cluster: 2023-11-16T16:39:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout id: 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:39:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-11-16T16:39:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2023-11-16T16:39:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout services: 2023-11-16T16:39:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.717946s) 2023-11-16T16:39:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-11-16T16:39:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-11-16T16:39:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2023-11-16T16:39:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout data: 2023-11-16T16:39:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-11-16T16:39:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-11-16T16:39:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-11-16T16:39:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout pgs: 2023-11-16T16:39:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2023-11-16T16:39:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.422+0000 7fad1a42d700 1 Processor -- start 2023-11-16T16:39:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.423+0000 7fad1a42d700 1 -- start start 2023-11-16T16:39:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.423+0000 7fad1a42d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad1407bda0 0x7fad1407a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.423+0000 7fad1a42d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1407a7e0 con 0x7fad1407bda0 2023-11-16T16:39:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.424+0000 7fad13fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad1407bda0 0x7fad1407a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.424+0000 7fad13fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad1407bda0 0x7fad1407a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46232/0 (socket says 172.21.15.44:46232) 2023-11-16T16:39:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.424+0000 7fad13fff700 1 -- 172.21.15.44:0/2924477697 learned_addr learned my addr 172.21.15.44:0/2924477697 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:39:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.424+0000 7fad13fff700 1 -- 172.21.15.44:0/2924477697 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1407a920 con 0x7fad1407bda0 2023-11-16T16:39:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.425+0000 7fad13fff700 1 --2- 172.21.15.44:0/2924477697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad1407bda0 0x7fad1407a2a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fad040098a0 tx=0x7fad04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a05d0b3ff7cd990 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.425+0000 7fad12ffd700 1 -- 172.21.15.44:0/2924477697 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad04004160 con 0x7fad1407bda0 2023-11-16T16:39:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.425+0000 7fad12ffd700 1 -- 172.21.15.44:0/2924477697 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad040042c0 con 0x7fad1407bda0 2023-11-16T16:39:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.425+0000 7fad12ffd700 1 -- 172.21.15.44:0/2924477697 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad040045a0 con 0x7fad1407bda0 2023-11-16T16:39:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.425+0000 7fad1a42d700 1 -- 172.21.15.44:0/2924477697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad1407bda0 msgr2=0x7fad1407a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.425+0000 7fad1a42d700 1 --2- 172.21.15.44:0/2924477697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad1407bda0 0x7fad1407a2a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fad040098a0 tx=0x7fad04009bb0 comp rx=0 tx=0).stop 2023-11-16T16:39:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.425+0000 7fad1a42d700 1 -- 172.21.15.44:0/2924477697 shutdown_connections 2023-11-16T16:39:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.425+0000 7fad1a42d700 1 --2- 172.21.15.44:0/2924477697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad1407bda0 0x7fad1407a2a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.425+0000 7fad1a42d700 1 -- 172.21.15.44:0/2924477697 >> 172.21.15.44:0/2924477697 conn(0x7fad14100fe0 msgr2=0x7fad141033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:42.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.426+0000 7fad1a42d700 1 -- 172.21.15.44:0/2924477697 shutdown_connections 2023-11-16T16:39:42.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.426+0000 7fad1a42d700 1 -- 172.21.15.44:0/2924477697 wait complete. 2023-11-16T16:39:42.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.427+0000 7fad1a42d700 1 Processor -- start 2023-11-16T16:39:42.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.427+0000 7fad1a42d700 1 -- start start 2023-11-16T16:39:42.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.427+0000 7fad1a42d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad1407bda0 0x7fad1419b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:42.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.427+0000 7fad1a42d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1419b810 con 0x7fad1407bda0 2023-11-16T16:39:42.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.428+0000 7fad13fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad1407bda0 0x7fad1419b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:42.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.428+0000 7fad13fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad1407bda0 0x7fad1419b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46236/0 (socket says 172.21.15.44:46236) 2023-11-16T16:39:42.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.428+0000 7fad13fff700 1 -- 172.21.15.44:0/2635086545 learned_addr learned my addr 172.21.15.44:0/2635086545 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:39:42.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.428+0000 7fad13fff700 1 -- 172.21.15.44:0/2635086545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad04009550 con 0x7fad1407bda0 2023-11-16T16:39:42.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.428+0000 7fad13fff700 1 --2- 172.21.15.44:0/2635086545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad1407bda0 0x7fad1419b2d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fad04004000 tx=0x7fad04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:42.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.429+0000 7fad117fa700 1 -- 172.21.15.44:0/2635086545 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad04011630 con 0x7fad1407bda0 2023-11-16T16:39:42.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.429+0000 7fad117fa700 1 -- 172.21.15.44:0/2635086545 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad04011790 con 0x7fad1407bda0 2023-11-16T16:39:42.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.429+0000 7fad117fa700 1 -- 172.21.15.44:0/2635086545 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad04011a30 con 0x7fad1407bda0 2023-11-16T16:39:42.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.429+0000 7fad1a42d700 1 -- 172.21.15.44:0/2635086545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad1407ae20 con 0x7fad1407bda0 2023-11-16T16:39:42.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.429+0000 7fad1a42d700 1 -- 172.21.15.44:0/2635086545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad1407ac90 con 0x7fad1407bda0 2023-11-16T16:39:42.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.429+0000 7fad117fa700 1 -- 172.21.15.44:0/2635086545 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fad04022070 con 0x7fad1407bda0 2023-11-16T16:39:42.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.429+0000 7fad117fa700 1 -- 172.21.15.44:0/2635086545 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fad0401c450 con 0x7fad1407bda0 2023-11-16T16:39:42.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.430+0000 7fad1a42d700 1 -- 172.21.15.44:0/2635086545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad1407afb0 con 0x7fad1407bda0 2023-11-16T16:39:42.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.433+0000 7fad117fa700 1 -- 172.21.15.44:0/2635086545 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fad0401b450 con 0x7fad1407bda0 2023-11-16T16:39:42.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.510+0000 7fad1a42d700 1 -- 172.21.15.44:0/2635086545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fad1404e920 con 0x7fad1407bda0 2023-11-16T16:39:42.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.510+0000 7fad117fa700 1 -- 172.21.15.44:0/2635086545 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fad0401b5f0 con 0x7fad1407bda0 2023-11-16T16:39:42.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.511+0000 7fad1a42d700 1 -- 172.21.15.44:0/2635086545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad1407bda0 msgr2=0x7fad1419b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:42.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.511+0000 7fad1a42d700 1 --2- 172.21.15.44:0/2635086545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad1407bda0 0x7fad1419b2d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fad04004000 tx=0x7fad04004750 comp rx=0 tx=0).stop 2023-11-16T16:39:42.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.511+0000 7fad1a42d700 1 -- 172.21.15.44:0/2635086545 shutdown_connections 2023-11-16T16:39:42.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.511+0000 7fad1a42d700 1 --2- 172.21.15.44:0/2635086545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad1407bda0 0x7fad1419b2d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:42.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.511+0000 7fad1a42d700 1 -- 172.21.15.44:0/2635086545 >> 172.21.15.44:0/2635086545 conn(0x7fad14100fe0 msgr2=0x7fad14102880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:42.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.511+0000 7fad1a42d700 1 -- 172.21.15.44:0/2635086545 shutdown_connections 2023-11-16T16:39:42.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:42.511+0000 7fad1a42d700 1 -- 172.21.15.44:0/2635086545 wait complete. 2023-11-16T16:39:42.986 INFO:teuthology.orchestra.run.smithi044.stdout:mon is available 2023-11-16T16:39:42.986 INFO:teuthology.orchestra.run.smithi044.stdout:Assimilating anything we can from ceph.conf... 2023-11-16T16:39:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:42 smithi044 ceph-mon[128238]: mon.a is new leader, mons a in quorum (ranks 0) 2023-11-16T16:39:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:42 smithi044 ceph-mon[128238]: monmap e1: 1 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0]} removed_ranks: {} 2023-11-16T16:39:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:42 smithi044 ceph-mon[128238]: fsmap 2023-11-16T16:39:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:42 smithi044 ceph-mon[128238]: osdmap e1: 0 total, 0 up, 0 in 2023-11-16T16:39:43.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:42 smithi044 ceph-mon[128238]: mgrmap e1: no daemons active 2023-11-16T16:39:43.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:42 smithi044 ceph-mon[128238]: from='client.? 172.21.15.44:0/2635086545' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-16T16:39:44.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:43 smithi044 ceph-mon[128238]: from='client.? 172.21.15.44:0/3474470874' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-16T16:39:44.008 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:43 smithi044 ceph-mon[128238]: from='client.? 172.21.15.44:0/3474470874' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-16T16:39:44.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2023-11-16T16:39:44.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [global] 2023-11-16T16:39:44.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout fsid = 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:39:44.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.44:3300,v1:172.21.15.44:6789] 2023-11-16T16:39:44.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-11-16T16:39:44.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-11-16T16:39:44.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-11-16T16:39:44.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-11-16T16:39:44.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2023-11-16T16:39:44.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [mgr] 2023-11-16T16:39:44.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-11-16T16:39:44.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2023-11-16T16:39:44.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [osd] 2023-11-16T16:39:44.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-11-16T16:39:44.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-11-16T16:39:44.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-11-16T16:39:44.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-11-16T16:39:44.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-11-16T16:39:44.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.519+0000 7f3ea0fea700 1 Processor -- start 2023-11-16T16:39:44.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.519+0000 7f3ea0fea700 1 -- start start 2023-11-16T16:39:44.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.520+0000 7f3ea0fea700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e9c07bdc0 0x7f3e9c07a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:44.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.520+0000 7f3ea0fea700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c07a800 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.520+0000 7f3e9a59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e9c07bdc0 0x7f3e9c07a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:44.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.520+0000 7f3e9a59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e9c07bdc0 0x7f3e9c07a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46248/0 (socket says 172.21.15.44:46248) 2023-11-16T16:39:44.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.520+0000 7f3e9a59c700 1 -- 172.21.15.44:0/603300100 learned_addr learned my addr 172.21.15.44:0/603300100 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:39:44.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.521+0000 7f3e9a59c700 1 -- 172.21.15.44:0/603300100 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9c07a940 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.521+0000 7f3e9a59c700 1 --2- 172.21.15.44:0/603300100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e9c07bdc0 0x7f3e9c07a2c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c0098a0 tx=0x7f3e8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a2670867ad9559 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:44.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.521+0000 7f3e9959a700 1 -- 172.21.15.44:0/603300100 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e8c004160 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.521+0000 7f3e9959a700 1 -- 172.21.15.44:0/603300100 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8c0042c0 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.521+0000 7f3e9959a700 1 -- 172.21.15.44:0/603300100 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e8c0045a0 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.522+0000 7f3ea0fea700 1 -- 172.21.15.44:0/603300100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e9c07bdc0 msgr2=0x7f3e9c07a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:44.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.522+0000 7f3ea0fea700 1 --2- 172.21.15.44:0/603300100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e9c07bdc0 0x7f3e9c07a2c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c0098a0 tx=0x7f3e8c009bb0 comp rx=0 tx=0).stop 2023-11-16T16:39:44.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.522+0000 7f3ea0fea700 1 -- 172.21.15.44:0/603300100 shutdown_connections 2023-11-16T16:39:44.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.522+0000 7f3ea0fea700 1 --2- 172.21.15.44:0/603300100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e9c07bdc0 0x7f3e9c07a2c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:44.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.522+0000 7f3ea0fea700 1 -- 172.21.15.44:0/603300100 >> 172.21.15.44:0/603300100 conn(0x7f3e9c100fe0 msgr2=0x7f3e9c1033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:44.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.522+0000 7f3ea0fea700 1 -- 172.21.15.44:0/603300100 shutdown_connections 2023-11-16T16:39:44.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.522+0000 7f3ea0fea700 1 -- 172.21.15.44:0/603300100 wait complete. 2023-11-16T16:39:44.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.523+0000 7f3ea0fea700 1 Processor -- start 2023-11-16T16:39:44.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.523+0000 7f3ea0fea700 1 -- start start 2023-11-16T16:39:44.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.523+0000 7f3ea0fea700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e9c07bdc0 0x7f3e9c19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:44.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.523+0000 7f3ea0fea700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c19b840 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.469 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.524+0000 7f3e9a59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e9c07bdc0 0x7f3e9c19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:44.469 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.524+0000 7f3e9a59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e9c07bdc0 0x7f3e9c19b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46262/0 (socket says 172.21.15.44:46262) 2023-11-16T16:39:44.469 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.524+0000 7f3e9a59c700 1 -- 172.21.15.44:0/3474470874 learned_addr learned my addr 172.21.15.44:0/3474470874 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:39:44.469 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.524+0000 7f3e9a59c700 1 -- 172.21.15.44:0/3474470874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c009550 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.469 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.525+0000 7f3e9a59c700 1 --2- 172.21.15.44:0/3474470874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e9c07bdc0 0x7f3e9c19b300 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c004000 tx=0x7f3e8c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:44.469 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.525+0000 7f3e8b7fe700 1 -- 172.21.15.44:0/3474470874 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e8c011630 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.469 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.525+0000 7f3e8b7fe700 1 -- 172.21.15.44:0/3474470874 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8c011790 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.469 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.525+0000 7f3e8b7fe700 1 -- 172.21.15.44:0/3474470874 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e8c011a30 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.469 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.525+0000 7f3ea0fea700 1 -- 172.21.15.44:0/3474470874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9c07ae40 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.469 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.525+0000 7f3ea0fea700 1 -- 172.21.15.44:0/3474470874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9c07acb0 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.525+0000 7f3e8b7fe700 1 -- 172.21.15.44:0/3474470874 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3e8c022070 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.525+0000 7f3e8b7fe700 1 -- 172.21.15.44:0/3474470874 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3e8c01c450 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.526+0000 7f3ea0fea700 1 -- 172.21.15.44:0/3474470874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e9c07afd0 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.529+0000 7f3e8b7fe700 1 -- 172.21.15.44:0/3474470874 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f3e8c01b450 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.603+0000 7f3ea0fea700 1 -- 172.21.15.44:0/3474470874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f3e9c04e920 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.607+0000 7f3e8b7fe700 1 -- 172.21.15.44:0/3474470874 <== mon.0 v2:172.21.15.44:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e8c01b5f0 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.607+0000 7f3e8b7fe700 1 -- 172.21.15.44:0/3474470874 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f3e8c024630 con 0x7f3e9c07bdc0 2023-11-16T16:39:44.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.608+0000 7f3ea0fea700 1 -- 172.21.15.44:0/3474470874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e9c07bdc0 msgr2=0x7f3e9c19b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:44.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.608+0000 7f3ea0fea700 1 --2- 172.21.15.44:0/3474470874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e9c07bdc0 0x7f3e9c19b300 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c004000 tx=0x7f3e8c004750 comp rx=0 tx=0).stop 2023-11-16T16:39:44.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.608+0000 7f3ea0fea700 1 -- 172.21.15.44:0/3474470874 shutdown_connections 2023-11-16T16:39:44.471 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.608+0000 7f3ea0fea700 1 --2- 172.21.15.44:0/3474470874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e9c07bdc0 0x7f3e9c19b300 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:44.471 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.608+0000 7f3ea0fea700 1 -- 172.21.15.44:0/3474470874 >> 172.21.15.44:0/3474470874 conn(0x7f3e9c100fe0 msgr2=0x7f3e9c1028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:44.471 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.608+0000 7f3ea0fea700 1 -- 172.21.15.44:0/3474470874 shutdown_connections 2023-11-16T16:39:44.471 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.608+0000 7f3ea0fea700 1 -- 172.21.15.44:0/3474470874 wait complete. 2023-11-16T16:39:44.471 INFO:teuthology.orchestra.run.smithi044.stdout:Generating new minimal ceph.conf... 2023-11-16T16:39:45.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:45 smithi044 ceph-mon[128238]: from='client.? 172.21.15.44:0/1627044413' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:39:45.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.986+0000 7f1010e41700 1 Processor -- start 2023-11-16T16:39:45.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.986+0000 7f1010e41700 1 -- start start 2023-11-16T16:39:45.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.986+0000 7f1010e41700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f100c105950 0x7f100c105d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:45.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.986+0000 7f1010e41700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c1062a0 con 0x7f100c105950 2023-11-16T16:39:45.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.987+0000 7f100a59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f100c105950 0x7f100c105d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:45.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.987+0000 7f100a59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f100c105950 0x7f100c105d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51626/0 (socket says 172.21.15.44:51626) 2023-11-16T16:39:45.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.987+0000 7f100a59c700 1 -- 172.21.15.44:0/3887723990 learned_addr learned my addr 172.21.15.44:0/3887723990 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:39:45.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.987+0000 7f100a59c700 1 -- 172.21.15.44:0/3887723990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100c1063e0 con 0x7f100c105950 2023-11-16T16:39:45.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.988+0000 7f100a59c700 1 --2- 172.21.15.44:0/3887723990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f100c105950 0x7f100c105d60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc0098a0 tx=0x7f0ffc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d254c4902d1c7a6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:45.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.988+0000 7f100959a700 1 -- 172.21.15.44:0/3887723990 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ffc004160 con 0x7f100c105950 2023-11-16T16:39:45.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.988+0000 7f100959a700 1 -- 172.21.15.44:0/3887723990 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ffc0042c0 con 0x7f100c105950 2023-11-16T16:39:45.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.988+0000 7f100959a700 1 -- 172.21.15.44:0/3887723990 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ffc011a00 con 0x7f100c105950 2023-11-16T16:39:45.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.988+0000 7f1010e41700 1 -- 172.21.15.44:0/3887723990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f100c105950 msgr2=0x7f100c105d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:45.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.988+0000 7f1010e41700 1 --2- 172.21.15.44:0/3887723990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f100c105950 0x7f100c105d60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc0098a0 tx=0x7f0ffc009bb0 comp rx=0 tx=0).stop 2023-11-16T16:39:45.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.989+0000 7f1010e41700 1 -- 172.21.15.44:0/3887723990 shutdown_connections 2023-11-16T16:39:45.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.989+0000 7f1010e41700 1 --2- 172.21.15.44:0/3887723990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f100c105950 0x7f100c105d60 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:45.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.989+0000 7f1010e41700 1 -- 172.21.15.44:0/3887723990 >> 172.21.15.44:0/3887723990 conn(0x7f100c100fe0 msgr2=0x7f100c1033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:45.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.989+0000 7f1010e41700 1 -- 172.21.15.44:0/3887723990 shutdown_connections 2023-11-16T16:39:45.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.989+0000 7f1010e41700 1 -- 172.21.15.44:0/3887723990 wait complete. 2023-11-16T16:39:45.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.990+0000 7f1010e41700 1 Processor -- start 2023-11-16T16:39:45.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.990+0000 7f1010e41700 1 -- start start 2023-11-16T16:39:45.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.991+0000 7f1010e41700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f100c105950 0x7f100c199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:45.551 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.991+0000 7f1010e41700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c199dc0 con 0x7f100c105950 2023-11-16T16:39:45.551 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.991+0000 7f100a59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f100c105950 0x7f100c199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:45.551 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.991+0000 7f100a59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f100c105950 0x7f100c199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51636/0 (socket says 172.21.15.44:51636) 2023-11-16T16:39:45.551 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.991+0000 7f100a59c700 1 -- 172.21.15.44:0/1627044413 learned_addr learned my addr 172.21.15.44:0/1627044413 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:39:45.551 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.991+0000 7f100a59c700 1 -- 172.21.15.44:0/1627044413 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc009550 con 0x7f100c105950 2023-11-16T16:39:45.551 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.992+0000 7f100a59c700 1 --2- 172.21.15.44:0/1627044413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f100c105950 0x7f100c199880 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc006af0 tx=0x7f0ffc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:45.551 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.992+0000 7f0ffb7fe700 1 -- 172.21.15.44:0/1627044413 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ffc003a40 con 0x7f100c105950 2023-11-16T16:39:45.551 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.992+0000 7f0ffb7fe700 1 -- 172.21.15.44:0/1627044413 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ffc003ba0 con 0x7f100c105950 2023-11-16T16:39:45.551 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.992+0000 7f0ffb7fe700 1 -- 172.21.15.44:0/1627044413 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ffc01a6e0 con 0x7f100c105950 2023-11-16T16:39:45.552 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.992+0000 7f1010e41700 1 -- 172.21.15.44:0/1627044413 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100c1079a0 con 0x7f100c105950 2023-11-16T16:39:45.552 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.992+0000 7f1010e41700 1 -- 172.21.15.44:0/1627044413 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100c107810 con 0x7f100c105950 2023-11-16T16:39:45.552 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.992+0000 7f0ffb7fe700 1 -- 172.21.15.44:0/1627044413 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0ffc01ab40 con 0x7f100c105950 2023-11-16T16:39:45.552 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.993+0000 7f0ffb7fe700 1 -- 172.21.15.44:0/1627044413 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0ffc019700 con 0x7f100c105950 2023-11-16T16:39:45.552 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.993+0000 7f1010e41700 1 -- 172.21.15.44:0/1627044413 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f100c109e20 con 0x7f100c105950 2023-11-16T16:39:45.552 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.996+0000 7f0ffb7fe700 1 -- 172.21.15.44:0/1627044413 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f0ffc02de20 con 0x7f100c105950 2023-11-16T16:39:45.552 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.070+0000 7f1010e41700 1 -- 172.21.15.44:0/1627044413 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f100c04e8c0 con 0x7f100c105950 2023-11-16T16:39:45.552 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.070+0000 7f0ffb7fe700 1 -- 172.21.15.44:0/1627044413 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f0ffc024a60 con 0x7f100c105950 2023-11-16T16:39:45.553 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.071+0000 7f1010e41700 1 -- 172.21.15.44:0/1627044413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f100c105950 msgr2=0x7f100c199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:45.553 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.071+0000 7f1010e41700 1 --2- 172.21.15.44:0/1627044413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f100c105950 0x7f100c199880 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc006af0 tx=0x7f0ffc011ad0 comp rx=0 tx=0).stop 2023-11-16T16:39:45.553 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.071+0000 7f1010e41700 1 -- 172.21.15.44:0/1627044413 shutdown_connections 2023-11-16T16:39:45.553 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.071+0000 7f1010e41700 1 --2- 172.21.15.44:0/1627044413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f100c105950 0x7f100c199880 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:45.553 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.071+0000 7f1010e41700 1 -- 172.21.15.44:0/1627044413 >> 172.21.15.44:0/1627044413 conn(0x7f100c100fe0 msgr2=0x7f100c101c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:45.553 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.071+0000 7f1010e41700 1 -- 172.21.15.44:0/1627044413 shutdown_connections 2023-11-16T16:39:45.553 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.071+0000 7f1010e41700 1 -- 172.21.15.44:0/1627044413 wait complete. 2023-11-16T16:39:45.553 INFO:teuthology.orchestra.run.smithi044.stdout:Restarting the monitor... 2023-11-16T16:39:45.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:45 smithi044 systemd[1]: Stopping Ceph mon.a for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:39:46.329 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:46 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mon-a[128214]: 2023-11-16T16:39:46.020+0000 7fb790382700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T16:39:46.329 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:46 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mon-a[128214]: 2023-11-16T16:39:46.020+0000 7fb790382700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-11-16T16:39:46.580 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:46 smithi044 bash[128577]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mon-a 2023-11-16T16:39:47.359 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:46 smithi044 systemd[1]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mon.a.service: Succeeded. 2023-11-16T16:39:47.359 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:46 smithi044 systemd[1]: Stopped Ceph mon.a for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:39:47.359 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:46 smithi044 systemd[1]: Starting Ceph mon.a for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:39:47.595 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 podman[128710]: 2023-11-16T16:39:47.595 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 bash[128710]: 4abeb7b1fb7307b95e00f07a797c169c1f5a897f306fd33fa5be94dc289e5eb8 2023-11-16T16:39:47.595 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: set uid:gid to 167:167 (ceph:ceph) 2023-11-16T16:39:47.609 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev), process ceph-mon, pid 7 2023-11-16T16:39:47.609 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: pidfile_write: ignore empty --pid-file 2023-11-16T16:39:47.610 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: load: jerasure load: lrc 2023-11-16T16:39:47.610 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: RocksDB version: 7.9.2 2023-11-16T16:39:47.610 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Git sha 0 2023-11-16T16:39:47.610 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Compile date 2023-11-15 23:53:54 2023-11-16T16:39:47.610 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: DB SUMMARY 2023-11-16T16:39:47.610 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: DB Session ID: W8UWGCHM6L5VSG2OMT97 2023-11-16T16:39:47.610 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: CURRENT file: CURRENT 2023-11-16T16:39:47.610 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: IDENTITY file: IDENTITY 2023-11-16T16:39:47.611 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-11-16T16:39:47.611 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-11-16T16:39:47.611 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 98180 ; 2023-11-16T16:39:47.611 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.error_if_exists: 0 2023-11-16T16:39:47.611 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.create_if_missing: 0 2023-11-16T16:39:47.611 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.paranoid_checks: 1 2023-11-16T16:39:47.611 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.flush_verify_memtable_count: 1 2023-11-16T16:39:47.611 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-16T16:39:47.611 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-11-16T16:39:47.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.env: 0x565553b29720 2023-11-16T16:39:47.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.fs: PosixFileSystem 2023-11-16T16:39:47.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.info_log: 0x5655559d93c0 2023-11-16T16:39:47.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_file_opening_threads: 16 2023-11-16T16:39:47.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.statistics: (nil) 2023-11-16T16:39:47.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.use_fsync: 0 2023-11-16T16:39:47.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_log_file_size: 0 2023-11-16T16:39:47.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-16T16:39:47.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.log_file_time_to_roll: 0 2023-11-16T16:39:47.613 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.keep_log_file_num: 1000 2023-11-16T16:39:47.613 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.recycle_log_file_num: 0 2023-11-16T16:39:47.613 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.allow_fallocate: 1 2023-11-16T16:39:47.613 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.allow_mmap_reads: 0 2023-11-16T16:39:47.613 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.allow_mmap_writes: 0 2023-11-16T16:39:47.613 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.use_direct_reads: 0 2023-11-16T16:39:47.613 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-16T16:39:47.613 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.create_missing_column_families: 0 2023-11-16T16:39:47.613 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.db_log_dir: 2023-11-16T16:39:47.614 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.wal_dir: 2023-11-16T16:39:47.614 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.table_cache_numshardbits: 6 2023-11-16T16:39:47.614 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.WAL_ttl_seconds: 0 2023-11-16T16:39:47.614 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.WAL_size_limit_MB: 0 2023-11-16T16:39:47.614 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-16T16:39:47.614 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-16T16:39:47.614 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.is_fd_close_on_exec: 1 2023-11-16T16:39:47.614 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.advise_random_on_open: 1 2023-11-16T16:39:47.614 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.db_write_buffer_size: 0 2023-11-16T16:39:47.615 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.write_buffer_manager: 0x565554c4a3c0 2023-11-16T16:39:47.615 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-16T16:39:47.615 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-16T16:39:47.615 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.use_adaptive_mutex: 0 2023-11-16T16:39:47.615 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.rate_limiter: (nil) 2023-11-16T16:39:47.615 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-16T16:39:47.615 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.wal_recovery_mode: 2 2023-11-16T16:39:47.615 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.enable_thread_tracking: 0 2023-11-16T16:39:47.615 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.enable_pipelined_write: 0 2023-11-16T16:39:47.616 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.unordered_write: 0 2023-11-16T16:39:47.616 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-16T16:39:47.616 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-16T16:39:47.653 INFO:teuthology.orchestra.run.smithi044.stdout:Setting mon public_network to 172.21.0.0/20 2023-11-16T16:39:47.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-16T16:39:47.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-16T16:39:47.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.row_cache: None 2023-11-16T16:39:47.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.wal_filter: None 2023-11-16T16:39:47.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-16T16:39:47.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.allow_ingest_behind: 0 2023-11-16T16:39:47.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.two_write_queues: 0 2023-11-16T16:39:47.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.manual_wal_flush: 0 2023-11-16T16:39:47.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.wal_compression: 0 2023-11-16T16:39:47.872 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.atomic_flush: 0 2023-11-16T16:39:47.872 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-16T16:39:47.872 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.persist_stats_to_disk: 0 2023-11-16T16:39:47.872 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.write_dbid_to_manifest: 0 2023-11-16T16:39:47.872 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.log_readahead_size: 0 2023-11-16T16:39:47.872 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-16T16:39:47.872 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.best_efforts_recovery: 0 2023-11-16T16:39:47.872 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-16T16:39:47.872 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-16T16:39:47.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.allow_data_in_errors: 0 2023-11-16T16:39:47.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.db_host_id: __hostname__ 2023-11-16T16:39:47.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.enforce_single_del_contracts: true 2023-11-16T16:39:47.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_background_jobs: 2 2023-11-16T16:39:47.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_background_compactions: -1 2023-11-16T16:39:47.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_subcompactions: 1 2023-11-16T16:39:47.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-16T16:39:47.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-16T16:39:47.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.delayed_write_rate : 16777216 2023-11-16T16:39:47.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_total_wal_size: 0 2023-11-16T16:39:47.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-16T16:39:47.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.stats_dump_period_sec: 600 2023-11-16T16:39:47.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.stats_persist_period_sec: 600 2023-11-16T16:39:47.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-16T16:39:47.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_open_files: -1 2023-11-16T16:39:47.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.bytes_per_sync: 0 2023-11-16T16:39:47.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.wal_bytes_per_sync: 0 2023-11-16T16:39:47.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.strict_bytes_per_sync: 0 2023-11-16T16:39:47.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compaction_readahead_size: 0 2023-11-16T16:39:47.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_background_flushes: -1 2023-11-16T16:39:47.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Compression algorithms supported: 2023-11-16T16:39:47.875 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-16T16:39:47.875 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: kZSTD supported: 0 2023-11-16T16:39:47.875 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: kXpressCompression supported: 0 2023-11-16T16:39:47.875 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: kLZ4HCCompression supported: 1 2023-11-16T16:39:47.875 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: kZlibCompression supported: 1 2023-11-16T16:39:47.875 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: kSnappyCompression supported: 1 2023-11-16T16:39:47.875 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: kLZ4Compression supported: 1 2023-11-16T16:39:47.875 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: kBZip2Compression supported: 0 2023-11-16T16:39:47.875 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Fast CRC32 supported: Supported on x86 2023-11-16T16:39:47.875 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: DMutex implementation: pthread_mutex_t 2023-11-16T16:39:47.875 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-11-16T16:39:47.876 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-11-16T16:39:47.876 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-16T16:39:47.876 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.merge_operator: 2023-11-16T16:39:47.876 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compaction_filter: None 2023-11-16T16:39:47.876 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compaction_filter_factory: None 2023-11-16T16:39:47.876 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.sst_partitioner_factory: None 2023-11-16T16:39:47.876 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.memtable_factory: SkipListFactory 2023-11-16T16:39:47.876 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.table_factory: BlockBasedTable 2023-11-16T16:39:47.876 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5655559d94e0) 2023-11-16T16:39:47.876 INFO:journalctl@ceph.mon.a.smithi044.stdout: cache_index_and_filter_blocks: 1 2023-11-16T16:39:47.876 INFO:journalctl@ceph.mon.a.smithi044.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-16T16:39:47.877 INFO:journalctl@ceph.mon.a.smithi044.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-16T16:39:47.877 INFO:journalctl@ceph.mon.a.smithi044.stdout: pin_top_level_index_and_filter: 1 2023-11-16T16:39:47.877 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_type: 0 2023-11-16T16:39:47.877 INFO:journalctl@ceph.mon.a.smithi044.stdout: data_block_index_type: 0 2023-11-16T16:39:47.877 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_shortening: 1 2023-11-16T16:39:47.877 INFO:journalctl@ceph.mon.a.smithi044.stdout: data_block_hash_table_util_ratio: 0.750000 2023-11-16T16:39:47.877 INFO:journalctl@ceph.mon.a.smithi044.stdout: checksum: 4 2023-11-16T16:39:47.877 INFO:journalctl@ceph.mon.a.smithi044.stdout: no_block_cache: 0 2023-11-16T16:39:47.877 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache: 0x565554ceb350 2023-11-16T16:39:47.877 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_name: BinnedLRUCache 2023-11-16T16:39:47.877 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_options: 2023-11-16T16:39:47.878 INFO:journalctl@ceph.mon.a.smithi044.stdout: capacity : 536870912 2023-11-16T16:39:47.878 INFO:journalctl@ceph.mon.a.smithi044.stdout: num_shard_bits : 4 2023-11-16T16:39:47.878 INFO:journalctl@ceph.mon.a.smithi044.stdout: strict_capacity_limit : 0 2023-11-16T16:39:47.878 INFO:journalctl@ceph.mon.a.smithi044.stdout: high_pri_pool_ratio: 0.000 2023-11-16T16:39:47.878 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_compressed: (nil) 2023-11-16T16:39:47.878 INFO:journalctl@ceph.mon.a.smithi044.stdout: persistent_cache: (nil) 2023-11-16T16:39:47.878 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_size: 4096 2023-11-16T16:39:47.878 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_size_deviation: 10 2023-11-16T16:39:47.878 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_restart_interval: 16 2023-11-16T16:39:47.878 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_block_restart_interval: 1 2023-11-16T16:39:47.879 INFO:journalctl@ceph.mon.a.smithi044.stdout: metadata_block_size: 4096 2023-11-16T16:39:47.879 INFO:journalctl@ceph.mon.a.smithi044.stdout: partition_filters: 0 2023-11-16T16:39:47.879 INFO:journalctl@ceph.mon.a.smithi044.stdout: use_delta_encoding: 1 2023-11-16T16:39:47.879 INFO:journalctl@ceph.mon.a.smithi044.stdout: filter_policy: bloomfilter 2023-11-16T16:39:47.879 INFO:journalctl@ceph.mon.a.smithi044.stdout: whole_key_filtering: 1 2023-11-16T16:39:47.879 INFO:journalctl@ceph.mon.a.smithi044.stdout: verify_compression: 0 2023-11-16T16:39:47.879 INFO:journalctl@ceph.mon.a.smithi044.stdout: read_amp_bytes_per_bit: 0 2023-11-16T16:39:47.879 INFO:journalctl@ceph.mon.a.smithi044.stdout: format_version: 5 2023-11-16T16:39:47.879 INFO:journalctl@ceph.mon.a.smithi044.stdout: enable_index_compression: 1 2023-11-16T16:39:47.879 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_align: 0 2023-11-16T16:39:47.880 INFO:journalctl@ceph.mon.a.smithi044.stdout: max_auto_readahead_size: 262144 2023-11-16T16:39:47.880 INFO:journalctl@ceph.mon.a.smithi044.stdout: prepopulate_block_cache: 0 2023-11-16T16:39:47.880 INFO:journalctl@ceph.mon.a.smithi044.stdout: initial_auto_readahead_size: 8192 2023-11-16T16:39:47.880 INFO:journalctl@ceph.mon.a.smithi044.stdout: num_file_reads_for_auto_readahead: 2 2023-11-16T16:39:47.880 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.write_buffer_size: 33554432 2023-11-16T16:39:47.880 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_write_buffer_number: 2 2023-11-16T16:39:47.880 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compression: NoCompression 2023-11-16T16:39:47.880 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.bottommost_compression: Disabled 2023-11-16T16:39:47.880 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.prefix_extractor: nullptr 2023-11-16T16:39:47.880 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-16T16:39:47.880 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.num_levels: 7 2023-11-16T16:39:47.881 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-16T16:39:47.881 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-16T16:39:47.881 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-16T16:39:47.881 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-16T16:39:47.881 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-16T16:39:47.881 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-16T16:39:47.881 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-16T16:39:47.881 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-16T16:39:47.882 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-16T16:39:47.882 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-16T16:39:47.882 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-11-16T16:39:47.882 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-11-16T16:39:47.882 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compression_opts.window_bits: -14 2023-11-16T16:39:47.882 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compression_opts.level: 32767 2023-11-16T16:39:47.882 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compression_opts.strategy: 0 2023-11-16T16:39:47.882 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-16T16:39:47.882 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-16T16:39:47.882 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-11-16T16:39:47.883 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-16T16:39:47.883 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compression_opts.enabled: false 2023-11-16T16:39:47.883 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-11-16T16:39:47.883 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-16T16:39:47.883 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-16T16:39:47.883 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-16T16:39:47.883 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.target_file_size_base: 67108864 2023-11-16T16:39:47.883 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.target_file_size_multiplier: 1 2023-11-16T16:39:47.883 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-16T16:39:47.883 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-16T16:39:47.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-16T16:39:47.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-16T16:39:47.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-16T16:39:47.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-16T16:39:47.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-16T16:39:47.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-16T16:39:47.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-16T16:39:47.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-16T16:39:47.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-16T16:39:47.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-16T16:39:47.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-11-16T16:39:47.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.arena_block_size: 1048576 2023-11-16T16:39:47.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-16T16:39:47.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-16T16:39:47.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.disable_auto_compactions: 0 2023-11-16T16:39:47.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-16T16:39:47.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-16T16:39:47.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-16T16:39:47.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-16T16:39:47.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-16T16:39:47.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-16T16:39:47.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-16T16:39:47.886 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-16T16:39:47.886 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-16T16:39:47.886 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-16T16:39:47.886 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.table_properties_collectors: 2023-11-16T16:39:47.886 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.inplace_update_support: 0 2023-11-16T16:39:47.886 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.inplace_update_num_locks: 10000 2023-11-16T16:39:47.886 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-16T16:39:47.886 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-16T16:39:47.886 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.memtable_huge_page_size: 0 2023-11-16T16:39:47.887 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.bloom_locality: 0 2023-11-16T16:39:47.887 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.max_successive_merges: 0 2023-11-16T16:39:47.887 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.optimize_filters_for_hits: 0 2023-11-16T16:39:47.887 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.paranoid_file_checks: 0 2023-11-16T16:39:47.887 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.force_consistency_checks: 1 2023-11-16T16:39:47.887 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.report_bg_io_stats: 0 2023-11-16T16:39:47.887 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.ttl: 2592000 2023-11-16T16:39:47.887 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.periodic_compaction_seconds: 0 2023-11-16T16:39:47.887 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-11-16T16:39:47.887 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-11-16T16:39:47.887 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.enable_blob_files: false 2023-11-16T16:39:47.888 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.min_blob_size: 0 2023-11-16T16:39:47.888 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.blob_file_size: 268435456 2023-11-16T16:39:47.888 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.blob_compression_type: NoCompression 2023-11-16T16:39:47.888 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.enable_blob_garbage_collection: false 2023-11-16T16:39:47.888 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-16T16:39:47.888 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-11-16T16:39:47.888 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-11-16T16:39:47.888 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.blob_file_starting_level: 0 2023-11-16T16:39:47.888 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-11-16T16:39:47.888 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-11-16T16:39:47.889 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-11-16T16:39:47.889 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d89d6c1e-5184-4fb8-bdc5-fe34c1d3a42d 2023-11-16T16:39:47.889 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152787602470, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-11-16T16:39:47.889 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-11-16T16:39:47.889 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152787604520, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93460, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 312, "table_properties": {"data_size": 91470, "index_size": 272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13525, "raw_average_key_size": 49, "raw_value_size": 84103, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 271, "num_filter_entries": 271, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1700152787, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89d6c1e-5184-4fb8-bdc5-fe34c1d3a42d", "db_session_id": "W8UWGCHM6L5VSG2OMT97", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-11-16T16:39:47.889 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152787604654, "job": 1, "event": "recovery_finished"} 2023-11-16T16:39:47.889 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-11-16T16:39:47.889 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-16T16:39:47.889 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x565554d88000 2023-11-16T16:39:47.889 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: DB pointer 0x565554d74000 2023-11-16T16:39:47.890 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-11-16T16:39:47.890 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-11-16T16:39:47.890 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** DB Stats ** 2023-11-16T16:39:47.890 INFO:journalctl@ceph.mon.a.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T16:39:47.890 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-16T16:39:47.890 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T16:39:47.890 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T16:39:47.890 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-16T16:39:47.890 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T16:39:47.890 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T16:39:47.891 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2023-11-16T16:39:47.891 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** Compaction Stats [default] ** 2023-11-16T16:39:47.891 INFO:journalctl@ceph.mon.a.smithi044.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T16:39:47.891 INFO:journalctl@ceph.mon.a.smithi044.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-11-16T16:39:47.891 INFO:journalctl@ceph.mon.a.smithi044.stdout: L0 2/0 93.10 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 62.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T16:39:47.891 INFO:journalctl@ceph.mon.a.smithi044.stdout: Sum 2/0 93.10 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 62.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T16:39:47.891 INFO:journalctl@ceph.mon.a.smithi044.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 62.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T16:39:47.891 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2023-11-16T16:39:47.891 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** Compaction Stats [default] ** 2023-11-16T16:39:47.891 INFO:journalctl@ceph.mon.a.smithi044.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T16:39:47.891 INFO:journalctl@ceph.mon.a.smithi044.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T16:39:47.892 INFO:journalctl@ceph.mon.a.smithi044.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 62.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T16:39:47.892 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2023-11-16T16:39:47.892 INFO:journalctl@ceph.mon.a.smithi044.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-11-16T16:39:47.892 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2023-11-16T16:39:47.892 INFO:journalctl@ceph.mon.a.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T16:39:47.892 INFO:journalctl@ceph.mon.a.smithi044.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T16:39:47.892 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T16:39:47.892 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T16:39:47.892 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T16:39:47.892 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(Keys): cumulative 0, interval 0 2023-11-16T16:39:47.892 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative compaction: 0.00 GB write, 14.04 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T16:39:47.893 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval compaction: 0.00 GB write, 14.04 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T16:39:47.893 INFO:journalctl@ceph.mon.a.smithi044.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T16:39:47.893 INFO:journalctl@ceph.mon.a.smithi044.stdout: Block cache BinnedLRUCache@0x565554ceb350#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2023-11-16T16:39:47.893 INFO:journalctl@ceph.mon.a.smithi044.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-11-16T16:39:47.893 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2023-11-16T16:39:47.893 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** File Read Latency Histogram By Level [default] ** 2023-11-16T16:39:47.893 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: starting mon.a rank 0 at public addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] at bind addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:39:47.893 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: mon.a@-1(???) e1 preinit fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:39:47.893 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: mon.a@-1(???).mds e1 new map 2023-11-16T16:39:47.893 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: mon.a@-1(???).mds e1 print_map 2023-11-16T16:39:47.893 INFO:journalctl@ceph.mon.a.smithi044.stdout: e1 2023-11-16T16:39:47.894 INFO:journalctl@ceph.mon.a.smithi044.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T16:39:47.894 INFO:journalctl@ceph.mon.a.smithi044.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T16:39:47.894 INFO:journalctl@ceph.mon.a.smithi044.stdout: legacy client fscid: -1 2023-11-16T16:39:47.894 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2023-11-16T16:39:47.894 INFO:journalctl@ceph.mon.a.smithi044.stdout: No filesystems configured 2023-11-16T16:39:47.894 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-16T16:39:47.894 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:39:47.894 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:39:47.894 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:39:47.894 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-16T16:39:47.894 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 systemd[1]: Started Ceph mon.a for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:39:47.895 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: mon.a is new leader, mons a in quorum (ranks 0) 2023-11-16T16:39:47.895 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: monmap e1: 1 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0]} removed_ranks: {} 2023-11-16T16:39:47.895 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: fsmap 2023-11-16T16:39:47.895 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: osdmap e1: 0 total, 0 up, 0 in 2023-11-16T16:39:47.895 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:47 smithi044 ceph-mon[128750]: mgrmap e1: no daemons active 2023-11-16T16:39:48.643 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.182+0000 7fc3647d2700 1 Processor -- start 2023-11-16T16:39:48.643 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.182+0000 7fc3647d2700 1 -- start start 2023-11-16T16:39:48.643 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.183+0000 7fc3647d2700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc35c105950 0x7fc35c105d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:48.643 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.183+0000 7fc3647d2700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c1062a0 con 0x7fc35c105950 2023-11-16T16:39:48.644 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.183+0000 7fc36256e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc35c105950 0x7fc35c105d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:48.644 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.184+0000 7fc36256e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc35c105950 0x7fc35c105d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51638/0 (socket says 172.21.15.44:51638) 2023-11-16T16:39:48.644 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.184+0000 7fc36256e700 1 -- 172.21.15.44:0/3214764943 learned_addr learned my addr 172.21.15.44:0/3214764943 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:39:48.644 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.184+0000 7fc36256e700 1 -- 172.21.15.44:0/3214764943 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c1063e0 con 0x7fc35c105950 2023-11-16T16:39:48.644 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.184+0000 7fc36256e700 1 --2- 172.21.15.44:0/3214764943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc35c105950 0x7fc35c105d60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc3580098a0 tx=0x7fc358009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e08adf61f8df4d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:48.644 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.184+0000 7fc36156c700 1 -- 172.21.15.44:0/3214764943 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc358004160 con 0x7fc35c105950 2023-11-16T16:39:48.644 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.185+0000 7fc36156c700 1 -- 172.21.15.44:0/3214764943 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3580042c0 con 0x7fc35c105950 2023-11-16T16:39:48.644 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.185+0000 7fc36156c700 1 -- 172.21.15.44:0/3214764943 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc358011a00 con 0x7fc35c105950 2023-11-16T16:39:48.644 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.185+0000 7fc3647d2700 1 -- 172.21.15.44:0/3214764943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc35c105950 msgr2=0x7fc35c105d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:48.645 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.185+0000 7fc3647d2700 1 --2- 172.21.15.44:0/3214764943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc35c105950 0x7fc35c105d60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc3580098a0 tx=0x7fc358009bb0 comp rx=0 tx=0).stop 2023-11-16T16:39:48.645 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.185+0000 7fc3647d2700 1 -- 172.21.15.44:0/3214764943 shutdown_connections 2023-11-16T16:39:48.645 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.185+0000 7fc3647d2700 1 --2- 172.21.15.44:0/3214764943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc35c105950 0x7fc35c105d60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:48.645 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.185+0000 7fc3647d2700 1 -- 172.21.15.44:0/3214764943 >> 172.21.15.44:0/3214764943 conn(0x7fc35c100fe0 msgr2=0x7fc35c1033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:48.645 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.185+0000 7fc3647d2700 1 -- 172.21.15.44:0/3214764943 shutdown_connections 2023-11-16T16:39:48.645 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.185+0000 7fc3647d2700 1 -- 172.21.15.44:0/3214764943 wait complete. 2023-11-16T16:39:48.645 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.186+0000 7fc3647d2700 1 Processor -- start 2023-11-16T16:39:48.645 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.186+0000 7fc3647d2700 1 -- start start 2023-11-16T16:39:48.645 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.187+0000 7fc3647d2700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc35c105950 0x7fc35c199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:48.646 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.187+0000 7fc3647d2700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c199dc0 con 0x7fc35c105950 2023-11-16T16:39:48.646 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.187+0000 7fc36256e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc35c105950 0x7fc35c199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:48.646 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.187+0000 7fc36256e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc35c105950 0x7fc35c199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51640/0 (socket says 172.21.15.44:51640) 2023-11-16T16:39:48.646 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.187+0000 7fc36256e700 1 -- 172.21.15.44:0/3104678513 learned_addr learned my addr 172.21.15.44:0/3104678513 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:39:48.646 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.187+0000 7fc36256e700 1 -- 172.21.15.44:0/3104678513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc358009550 con 0x7fc35c105950 2023-11-16T16:39:48.646 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.188+0000 7fc36256e700 1 --2- 172.21.15.44:0/3104678513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc35c105950 0x7fc35c199880 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc358006af0 tx=0x7fc358011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:48.646 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.188+0000 7fc3537fe700 1 -- 172.21.15.44:0/3104678513 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc358003a40 con 0x7fc35c105950 2023-11-16T16:39:48.646 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.188+0000 7fc3647d2700 1 -- 172.21.15.44:0/3104678513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc35c1079a0 con 0x7fc35c105950 2023-11-16T16:39:48.647 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.188+0000 7fc3537fe700 1 -- 172.21.15.44:0/3104678513 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc358003ba0 con 0x7fc35c105950 2023-11-16T16:39:48.647 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.188+0000 7fc3537fe700 1 -- 172.21.15.44:0/3104678513 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc35801a6e0 con 0x7fc35c105950 2023-11-16T16:39:48.647 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.188+0000 7fc3647d2700 1 -- 172.21.15.44:0/3104678513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35c107810 con 0x7fc35c105950 2023-11-16T16:39:48.647 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.189+0000 7fc3537fe700 1 -- 172.21.15.44:0/3104678513 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc35801ac50 con 0x7fc35c105950 2023-11-16T16:39:48.647 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.189+0000 7fc3537fe700 1 -- 172.21.15.44:0/3104678513 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc3580193c0 con 0x7fc35c105950 2023-11-16T16:39:48.647 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.189+0000 7fc3647d2700 1 -- 172.21.15.44:0/3104678513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc35c062120 con 0x7fc35c105950 2023-11-16T16:39:48.647 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.192+0000 7fc3537fe700 1 -- 172.21.15.44:0/3104678513 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fc35802de30 con 0x7fc35c105950 2023-11-16T16:39:48.647 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.264+0000 7fc3647d2700 1 -- 172.21.15.44:0/3104678513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fc35c109df0 con 0x7fc35c105950 2023-11-16T16:39:48.648 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.267+0000 7fc3537fe700 1 -- 172.21.15.44:0/3104678513 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc358024770 con 0x7fc35c105950 2023-11-16T16:39:48.648 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.268+0000 7fc3647d2700 1 -- 172.21.15.44:0/3104678513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc35c105950 msgr2=0x7fc35c199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:48.648 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.268+0000 7fc3647d2700 1 --2- 172.21.15.44:0/3104678513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc35c105950 0x7fc35c199880 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc358006af0 tx=0x7fc358011ad0 comp rx=0 tx=0).stop 2023-11-16T16:39:48.648 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.268+0000 7fc3647d2700 1 -- 172.21.15.44:0/3104678513 shutdown_connections 2023-11-16T16:39:48.648 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.268+0000 7fc3647d2700 1 --2- 172.21.15.44:0/3104678513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc35c105950 0x7fc35c199880 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:48.648 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.268+0000 7fc3647d2700 1 -- 172.21.15.44:0/3104678513 >> 172.21.15.44:0/3104678513 conn(0x7fc35c100fe0 msgr2=0x7fc35c101c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:48.648 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.268+0000 7fc3647d2700 1 -- 172.21.15.44:0/3104678513 shutdown_connections 2023-11-16T16:39:48.648 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:48.268+0000 7fc3647d2700 1 -- 172.21.15.44:0/3104678513 wait complete. 2023-11-16T16:39:48.648 INFO:teuthology.orchestra.run.smithi044.stdout:Wrote config to /etc/ceph/ceph.conf 2023-11-16T16:39:48.649 INFO:teuthology.orchestra.run.smithi044.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-16T16:39:48.649 INFO:teuthology.orchestra.run.smithi044.stdout:Creating mgr... 2023-11-16T16:39:48.649 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying port 9283 ... 2023-11-16T16:39:48.649 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying port 8765 ... 2023-11-16T16:39:48.972 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mgr.x 2023-11-16T16:39:48.972 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mgr.x.service: Unit ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mgr.x.service not loaded. 2023-11-16T16:39:49.184 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7.target.wants/ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mgr.x.service → /etc/systemd/system/ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@.service. 2023-11-16T16:39:49.268 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:49 smithi044 systemd[1]: Starting Ceph mgr.x for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:39:49.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:49 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3104678513' entity='client.admin' 2023-11-16T16:39:49.988 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:49 smithi044 podman[128984]: 2023-11-16T16:39:49.988 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:49 smithi044 bash[128984]: f85ee589f567b5139075c1cec6dd375e1dda870db10b73a82f3d5ab5c0e6df19 2023-11-16T16:39:50.011 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-16T16:39:50.011 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2023-11-16T16:39:50.028 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-16T16:39:50.029 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2023-11-16T16:39:50.029 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2023-11-16T16:39:50.029 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to enable service . firewalld.service is not available 2023-11-16T16:39:50.047 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-16T16:39:50.047 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2023-11-16T16:39:50.064 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-16T16:39:50.064 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2023-11-16T16:39:50.064 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2023-11-16T16:39:50.064 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-11-16T16:39:50.064 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr to start... 2023-11-16T16:39:50.064 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr... 2023-11-16T16:39:50.255 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:49 smithi044 systemd[1]: Started Ceph mgr.x for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:39:50.255 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:50 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:50.145+0000 7fe943b3c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T16:39:50.256 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:50 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:50.253+0000 7fe943b3c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T16:39:50.512 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:50 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:50.373+0000 7fe943b3c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T16:39:50.785 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:50 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:50.510+0000 7fe943b3c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T16:39:50.785 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:50 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:50.745+0000 7fe943b3c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T16:39:51.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:50 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1612378762' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T16:39:51.255 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2023-11-16T16:39:51.258 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2023-11-16T16:39:51.258 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "7f2b12a8-849e-11ee-8dbe-212e2dc638e7", 2023-11-16T16:39:51.258 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2023-11-16T16:39:51.258 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-16T16:39:51.259 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-16T16:39:51.259 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-16T16:39:51.259 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:39:51.259 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-16T16:39:51.259 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-16T16:39:51.259 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2023-11-16T16:39:51.259 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:39:51.259 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-16T16:39:51.259 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "a" 2023-11-16T16:39:51.260 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:39:51.260 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2023-11-16T16:39:51.260 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-16T16:39:51.260 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:39:51.260 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-11-16T16:39:51.260 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-16T16:39:51.260 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:39:51.260 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-16T16:39:51.260 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:39:51.261 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-16T16:39:51.261 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-16T16:39:51.261 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-16T16:39:51.271 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-16T16:39:51.271 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-16T16:39:51.271 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-16T16:39:51.272 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:39:51.272 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-16T16:39:51.272 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-16T16:39:51.272 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-16T16:39:51.272 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-16T16:39:51.272 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-16T16:39:51.272 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-16T16:39:51.272 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-16T16:39:51.272 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-16T16:39:51.272 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-16T16:39:51.273 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:39:51.273 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-16T16:39:51.273 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:39:51.273 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-16T16:39:51.273 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-16T16:39:51.273 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:39:51.273 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-16T16:39:51.273 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": false, 2023-11-16T16:39:51.273 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-16T16:39:51.274 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-16T16:39:51.274 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-16T16:39:51.274 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-16T16:39:51.274 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2023-11-16T16:39:51.274 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:39:51.274 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T16:39:51.274 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:39:51.274 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-16T16:39:51.274 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:39:51.275 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2023-11-16T16:39:41.797141+0000", 2023-11-16T16:39:51.275 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T16:39:51.275 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:39:51.275 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-16T16:39:51.275 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2023-11-16T16:39:51.275 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.673+0000 7f19cc3cc700 1 Processor -- start 2023-11-16T16:39:51.275 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.673+0000 7f19cc3cc700 1 -- start start 2023-11-16T16:39:51.275 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.673+0000 7f19cc3cc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c407bdc0 0x7f19c407a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:51.275 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.673+0000 7f19cc3cc700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c407a800 con 0x7f19c407bdc0 2023-11-16T16:39:51.276 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.674+0000 7f19ca168700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c407bdc0 0x7f19c407a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:51.276 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.674+0000 7f19ca168700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c407bdc0 0x7f19c407a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51666/0 (socket says 172.21.15.44:51666) 2023-11-16T16:39:51.276 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.674+0000 7f19ca168700 1 -- 172.21.15.44:0/2198209202 learned_addr learned my addr 172.21.15.44:0/2198209202 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:39:51.276 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.674+0000 7f19ca168700 1 -- 172.21.15.44:0/2198209202 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c407a940 con 0x7f19c407bdc0 2023-11-16T16:39:51.276 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.675+0000 7f19ca168700 1 --2- 172.21.15.44:0/2198209202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c407bdc0 0x7f19c407a2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f19c00098a0 tx=0x7f19c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec14a0f4cc7c55e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:51.276 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.675+0000 7f19c9166700 1 -- 172.21.15.44:0/2198209202 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19c0004160 con 0x7f19c407bdc0 2023-11-16T16:39:51.276 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.675+0000 7f19c9166700 1 -- 172.21.15.44:0/2198209202 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19c00042c0 con 0x7f19c407bdc0 2023-11-16T16:39:51.276 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.675+0000 7f19c9166700 1 -- 172.21.15.44:0/2198209202 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19c0011a00 con 0x7f19c407bdc0 2023-11-16T16:39:51.276 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.675+0000 7f19cc3cc700 1 -- 172.21.15.44:0/2198209202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c407bdc0 msgr2=0x7f19c407a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:51.276 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.675+0000 7f19cc3cc700 1 --2- 172.21.15.44:0/2198209202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c407bdc0 0x7f19c407a2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f19c00098a0 tx=0x7f19c0009bb0 comp rx=0 tx=0).stop 2023-11-16T16:39:51.277 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.676+0000 7f19cc3cc700 1 -- 172.21.15.44:0/2198209202 shutdown_connections 2023-11-16T16:39:51.277 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.676+0000 7f19cc3cc700 1 --2- 172.21.15.44:0/2198209202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c407bdc0 0x7f19c407a2c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:51.277 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.676+0000 7f19cc3cc700 1 -- 172.21.15.44:0/2198209202 >> 172.21.15.44:0/2198209202 conn(0x7f19c4100fe0 msgr2=0x7f19c41033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:51.277 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.676+0000 7f19cc3cc700 1 -- 172.21.15.44:0/2198209202 shutdown_connections 2023-11-16T16:39:51.277 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.676+0000 7f19cc3cc700 1 -- 172.21.15.44:0/2198209202 wait complete. 2023-11-16T16:39:51.277 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.677+0000 7f19cc3cc700 1 Processor -- start 2023-11-16T16:39:51.277 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.677+0000 7f19cc3cc700 1 -- start start 2023-11-16T16:39:51.277 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.677+0000 7f19cc3cc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c407bdc0 0x7f19c419bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:51.277 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.677+0000 7f19cc3cc700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c419bff0 con 0x7f19c407bdc0 2023-11-16T16:39:51.277 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.678+0000 7f19ca168700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c407bdc0 0x7f19c419bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:51.278 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.678+0000 7f19ca168700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c407bdc0 0x7f19c419bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51682/0 (socket says 172.21.15.44:51682) 2023-11-16T16:39:51.278 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.678+0000 7f19ca168700 1 -- 172.21.15.44:0/1612378762 learned_addr learned my addr 172.21.15.44:0/1612378762 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:39:51.278 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.678+0000 7f19ca168700 1 -- 172.21.15.44:0/1612378762 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c0009550 con 0x7f19c407bdc0 2023-11-16T16:39:51.278 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.678+0000 7f19ca168700 1 --2- 172.21.15.44:0/1612378762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c407bdc0 0x7f19c419bab0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f19c0006b20 tx=0x7f19c0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:51.278 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.678+0000 7f19bb7fe700 1 -- 172.21.15.44:0/1612378762 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19c0003b70 con 0x7f19c407bdc0 2023-11-16T16:39:51.278 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.678+0000 7f19bb7fe700 1 -- 172.21.15.44:0/1612378762 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19c0003cd0 con 0x7f19c407bdc0 2023-11-16T16:39:51.278 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.678+0000 7f19cc3cc700 1 -- 172.21.15.44:0/1612378762 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c410be20 con 0x7f19c407bdc0 2023-11-16T16:39:51.278 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.679+0000 7f19bb7fe700 1 -- 172.21.15.44:0/1612378762 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19c00247c0 con 0x7f19c407bdc0 2023-11-16T16:39:51.279 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.679+0000 7f19cc3cc700 1 -- 172.21.15.44:0/1612378762 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c410bc90 con 0x7f19c407bdc0 2023-11-16T16:39:51.279 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.679+0000 7f19bb7fe700 1 -- 172.21.15.44:0/1612378762 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f19c0024c80 con 0x7f19c407bdc0 2023-11-16T16:39:51.279 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.679+0000 7f19bb7fe700 1 -- 172.21.15.44:0/1612378762 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f19c00193c0 con 0x7f19c407bdc0 2023-11-16T16:39:51.279 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.680+0000 7f19cc3cc700 1 -- 172.21.15.44:0/1612378762 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c4062120 con 0x7f19c407bdc0 2023-11-16T16:39:51.279 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.682+0000 7f19bb7fe700 1 -- 172.21.15.44:0/1612378762 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f19c001ab40 con 0x7f19c407bdc0 2023-11-16T16:39:51.279 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.759+0000 7f19cc3cc700 1 -- 172.21.15.44:0/1612378762 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f19c404e8c0 con 0x7f19c407bdc0 2023-11-16T16:39:51.279 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.760+0000 7f19bb7fe700 1 -- 172.21.15.44:0/1612378762 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f19c001ace0 con 0x7f19c407bdc0 2023-11-16T16:39:51.279 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.761+0000 7f19cc3cc700 1 -- 172.21.15.44:0/1612378762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c407bdc0 msgr2=0x7f19c419bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:51.279 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.761+0000 7f19cc3cc700 1 --2- 172.21.15.44:0/1612378762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c407bdc0 0x7f19c419bab0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f19c0006b20 tx=0x7f19c0011ad0 comp rx=0 tx=0).stop 2023-11-16T16:39:51.279 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.761+0000 7f19cc3cc700 1 -- 172.21.15.44:0/1612378762 shutdown_connections 2023-11-16T16:39:51.280 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.761+0000 7f19cc3cc700 1 --2- 172.21.15.44:0/1612378762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c407bdc0 0x7f19c419bab0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:51.280 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.761+0000 7f19cc3cc700 1 -- 172.21.15.44:0/1612378762 >> 172.21.15.44:0/1612378762 conn(0x7f19c4100fe0 msgr2=0x7f19c4101e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:51.280 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.761+0000 7f19cc3cc700 1 -- 172.21.15.44:0/1612378762 shutdown_connections 2023-11-16T16:39:51.280 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:50.761+0000 7f19cc3cc700 1 -- 172.21.15.44:0/1612378762 wait complete. 2023-11-16T16:39:51.280 INFO:teuthology.orchestra.run.smithi044.stdout:mgr not available, waiting (1/15)... 2023-11-16T16:39:51.468 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:51.171+0000 7fe943b3c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T16:39:51.468 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:51.297+0000 7fe943b3c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T16:39:51.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:51.465+0000 7fe943b3c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T16:39:51.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:51.593+0000 7fe943b3c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T16:39:52.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:52 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:52.159+0000 7fe943b3c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T16:39:53.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:52 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:52.973+0000 7fe943b3c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T16:39:53.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:53 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:53.187+0000 7fe943b3c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T16:39:54.214 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:53 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3505586118' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T16:39:54.573 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:54 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:54.212+0000 7fe943b3c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:39:54.574 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:54 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:54.339+0000 7fe943b3c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T16:39:54.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:54 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:54.571+0000 7fe943b3c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:39:55.262 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:54 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:54.962+0000 7fe943b3c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T16:39:55.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:55.259+0000 7fe943b3c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T16:39:55.619 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:55.460+0000 7fe943b3c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T16:39:56.118 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:55.817+0000 7fe943b3c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T16:39:57.118 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:56 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:56.785+0000 7fe943b3c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T16:39:57.119 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:56 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:56.965+0000 7fe943b3c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T16:39:57.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:57 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:57.222+0000 7fe943b3c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T16:39:57.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:57 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:57.376+0000 7fe943b3c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T16:39:58.118 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:57 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:57.665+0000 7fe943b3c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T16:39:59.118 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:39:58 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:39:58.777+0000 7fe943b3c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T16:40:02.970 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:58 smithi044 ceph-mon[128750]: Activating manager daemon x 2023-11-16T16:40:02.970 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:58 smithi044 ceph-mon[128750]: mgrmap e2: x(active, starting, since 0.00390935s) 2023-11-16T16:40:02.970 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:58 smithi044 ceph-mon[128750]: from='mgr.14100 172.21.15.44:0/2927725239' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T16:40:02.970 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:58 smithi044 ceph-mon[128750]: from='mgr.14100 172.21.15.44:0/2927725239' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T16:40:02.970 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:58 smithi044 ceph-mon[128750]: from='mgr.14100 172.21.15.44:0/2927725239' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T16:40:02.970 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:58 smithi044 ceph-mon[128750]: from='mgr.14100 172.21.15.44:0/2927725239' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:40:02.970 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:58 smithi044 ceph-mon[128750]: from='mgr.14100 172.21.15.44:0/2927725239' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T16:40:02.970 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:58 smithi044 ceph-mon[128750]: Manager daemon x is now available 2023-11-16T16:40:02.971 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:58 smithi044 ceph-mon[128750]: from='mgr.14100 172.21.15.44:0/2927725239' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T16:40:02.971 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:59 smithi044 ceph-mon[128750]: from='mgr.14100 172.21.15.44:0/2927725239' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T16:40:02.971 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:59 smithi044 ceph-mon[128750]: from='mgr.14100 172.21.15.44:0/2927725239' entity='mgr.x' 2023-11-16T16:40:02.971 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:59 smithi044 ceph-mon[128750]: from='mgr.14100 172.21.15.44:0/2927725239' entity='mgr.x' 2023-11-16T16:40:02.971 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:59 smithi044 ceph-mon[128750]: from='mgr.14100 172.21.15.44:0/2927725239' entity='mgr.x' 2023-11-16T16:40:02.971 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:39:59 smithi044 ceph-mon[128750]: mgrmap e3: x(active, since 1.00909s) 2023-11-16T16:40:02.980 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2023-11-16T16:40:03.001 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2023-11-16T16:40:03.001 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "7f2b12a8-849e-11ee-8dbe-212e2dc638e7", 2023-11-16T16:40:03.001 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2023-11-16T16:40:03.001 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-16T16:40:03.001 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-16T16:40:03.001 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-16T16:40:03.001 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:03.001 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-16T16:40:03.001 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-16T16:40:03.001 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2023-11-16T16:40:03.002 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:40:03.002 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-16T16:40:03.002 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "a" 2023-11-16T16:40:03.002 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:40:03.002 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2023-11-16T16:40:03.002 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-16T16:40:03.002 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:40:03.002 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-11-16T16:40:03.002 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-16T16:40:03.002 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:03.002 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-16T16:40:03.002 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:40:03.002 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-16T16:40:03.003 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-16T16:40:03.003 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-16T16:40:03.003 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-16T16:40:03.003 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-16T16:40:03.003 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-16T16:40:03.003 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:03.003 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-16T16:40:03.003 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-16T16:40:03.003 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-16T16:40:03.003 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-16T16:40:03.003 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-16T16:40:03.003 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-16T16:40:03.003 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-16T16:40:03.004 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-16T16:40:03.004 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-16T16:40:03.004 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:03.004 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-16T16:40:03.004 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:40:03.004 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-16T16:40:03.004 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-16T16:40:03.004 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:03.004 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-16T16:40:03.004 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": false, 2023-11-16T16:40:03.004 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-16T16:40:03.004 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-16T16:40:03.004 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-16T16:40:03.005 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-16T16:40:03.005 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2023-11-16T16:40:03.005 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:40:03.005 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T16:40:03.005 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:03.005 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-16T16:40:03.005 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:40:03.005 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2023-11-16T16:39:41.797141+0000", 2023-11-16T16:40:03.005 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T16:40:03.005 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:03.005 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-16T16:40:03.005 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2023-11-16T16:40:03.005 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.786+0000 7f556dfaf700 1 Processor -- start 2023-11-16T16:40:03.006 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.786+0000 7f556dfaf700 1 -- start start 2023-11-16T16:40:03.006 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.786+0000 7f556dfaf700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f556807bd80 0x7f556807a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:03.006 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.786+0000 7f556dfaf700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556807a7c0 con 0x7f556807bd80 2023-11-16T16:40:03.006 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.787+0000 7f55677fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f556807bd80 0x7f556807a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:03.006 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.787+0000 7f55677fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f556807bd80 0x7f556807a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51684/0 (socket says 172.21.15.44:51684) 2023-11-16T16:40:03.006 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.787+0000 7f55677fe700 1 -- 172.21.15.44:0/1460283669 learned_addr learned my addr 172.21.15.44:0/1460283669 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:03.006 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.787+0000 7f55677fe700 1 -- 172.21.15.44:0/1460283669 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556807a900 con 0x7f556807bd80 2023-11-16T16:40:03.006 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.788+0000 7f55677fe700 1 --2- 172.21.15.44:0/1460283669 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f556807bd80 0x7f556807a280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f55580098a0 tx=0x7f5558009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd91a74394eb73a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:03.006 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.788+0000 7f55667fc700 1 -- 172.21.15.44:0/1460283669 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5558004160 con 0x7f556807bd80 2023-11-16T16:40:03.006 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.788+0000 7f55667fc700 1 -- 172.21.15.44:0/1460283669 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55580042c0 con 0x7f556807bd80 2023-11-16T16:40:03.006 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.788+0000 7f55667fc700 1 -- 172.21.15.44:0/1460283669 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5558011a00 con 0x7f556807bd80 2023-11-16T16:40:03.006 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.789+0000 7f556dfaf700 1 -- 172.21.15.44:0/1460283669 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f556807bd80 msgr2=0x7f556807a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:03.007 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.789+0000 7f556dfaf700 1 --2- 172.21.15.44:0/1460283669 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f556807bd80 0x7f556807a280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f55580098a0 tx=0x7f5558009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:03.007 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.789+0000 7f556dfaf700 1 -- 172.21.15.44:0/1460283669 shutdown_connections 2023-11-16T16:40:03.007 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.789+0000 7f556dfaf700 1 --2- 172.21.15.44:0/1460283669 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f556807bd80 0x7f556807a280 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:03.007 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.789+0000 7f556dfaf700 1 -- 172.21.15.44:0/1460283669 >> 172.21.15.44:0/1460283669 conn(0x7f5568100fe0 msgr2=0x7f55681033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:03.007 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.789+0000 7f556dfaf700 1 -- 172.21.15.44:0/1460283669 shutdown_connections 2023-11-16T16:40:03.007 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.789+0000 7f556dfaf700 1 -- 172.21.15.44:0/1460283669 wait complete. 2023-11-16T16:40:03.007 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.790+0000 7f556dfaf700 1 Processor -- start 2023-11-16T16:40:03.007 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.791+0000 7f556dfaf700 1 -- start start 2023-11-16T16:40:03.007 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.791+0000 7f556dfaf700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f556807bd80 0x7f556819baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:03.007 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.791+0000 7f556dfaf700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556819c030 con 0x7f556807bd80 2023-11-16T16:40:03.007 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.791+0000 7f55677fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f556807bd80 0x7f556819baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:03.007 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.791+0000 7f55677fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f556807bd80 0x7f556819baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51698/0 (socket says 172.21.15.44:51698) 2023-11-16T16:40:03.007 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.792+0000 7f55677fe700 1 -- 172.21.15.44:0/3505586118 learned_addr learned my addr 172.21.15.44:0/3505586118 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:03.007 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.792+0000 7f55677fe700 1 -- 172.21.15.44:0/3505586118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5558009550 con 0x7f556807bd80 2023-11-16T16:40:03.008 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.792+0000 7f55677fe700 1 --2- 172.21.15.44:0/3505586118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f556807bd80 0x7f556819baf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5558006af0 tx=0x7f5558011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:03.008 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.792+0000 7f5564ff9700 1 -- 172.21.15.44:0/3505586118 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5558003a40 con 0x7f556807bd80 2023-11-16T16:40:03.008 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.792+0000 7f5564ff9700 1 -- 172.21.15.44:0/3505586118 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5558003ba0 con 0x7f556807bd80 2023-11-16T16:40:03.008 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.792+0000 7f556dfaf700 1 -- 172.21.15.44:0/3505586118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556810be00 con 0x7f556807bd80 2023-11-16T16:40:03.008 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.792+0000 7f5564ff9700 1 -- 172.21.15.44:0/3505586118 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f555801a700 con 0x7f556807bd80 2023-11-16T16:40:03.008 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.793+0000 7f556dfaf700 1 -- 172.21.15.44:0/3505586118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556810bc70 con 0x7f556807bd80 2023-11-16T16:40:03.008 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.793+0000 7f5564ff9700 1 -- 172.21.15.44:0/3505586118 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f555801ac80 con 0x7f556807bd80 2023-11-16T16:40:03.008 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.793+0000 7f5564ff9700 1 -- 172.21.15.44:0/3505586118 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5558019700 con 0x7f556807bd80 2023-11-16T16:40:03.008 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.794+0000 7f556dfaf700 1 -- 172.21.15.44:0/3505586118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556810bf90 con 0x7f556807bd80 2023-11-16T16:40:03.008 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.797+0000 7f5564ff9700 1 -- 172.21.15.44:0/3505586118 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f5558024ad0 con 0x7f556807bd80 2023-11-16T16:40:03.008 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.888+0000 7f556dfaf700 1 -- 172.21.15.44:0/3505586118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5568062120 con 0x7f556807bd80 2023-11-16T16:40:03.008 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.888+0000 7f5564ff9700 1 -- 172.21.15.44:0/3505586118 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f5558024c70 con 0x7f556807bd80 2023-11-16T16:40:03.008 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.889+0000 7f556dfaf700 1 -- 172.21.15.44:0/3505586118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f556807bd80 msgr2=0x7f556819baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:03.009 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.889+0000 7f556dfaf700 1 --2- 172.21.15.44:0/3505586118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f556807bd80 0x7f556819baf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5558006af0 tx=0x7f5558011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:03.009 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.889+0000 7f556dfaf700 1 -- 172.21.15.44:0/3505586118 shutdown_connections 2023-11-16T16:40:03.009 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.889+0000 7f556dfaf700 1 --2- 172.21.15.44:0/3505586118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f556807bd80 0x7f556819baf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:03.009 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.889+0000 7f556dfaf700 1 -- 172.21.15.44:0/3505586118 >> 172.21.15.44:0/3505586118 conn(0x7f5568100fe0 msgr2=0x7f5568101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:03.009 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.889+0000 7f556dfaf700 1 -- 172.21.15.44:0/3505586118 shutdown_connections 2023-11-16T16:40:03.009 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:53.890+0000 7f556dfaf700 1 -- 172.21.15.44:0/3505586118 wait complete. 2023-11-16T16:40:03.009 INFO:teuthology.orchestra.run.smithi044.stdout:mgr not available, waiting (2/15)... 2023-11-16T16:40:03.009 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:01 smithi044 ceph-mon[128750]: mgrmap e4: x(active, since 2s) 2023-11-16T16:40:04.867 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2023-11-16T16:40:04.867 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2023-11-16T16:40:04.867 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "7f2b12a8-849e-11ee-8dbe-212e2dc638e7", 2023-11-16T16:40:04.867 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2023-11-16T16:40:04.867 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-16T16:40:04.867 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-16T16:40:04.867 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-16T16:40:04.867 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:04.868 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-16T16:40:04.868 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-16T16:40:04.868 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2023-11-16T16:40:04.868 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:40:04.868 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-16T16:40:04.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:04 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3141669214' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T16:40:04.870 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "a" 2023-11-16T16:40:04.870 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:40:04.870 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2023-11-16T16:40:04.870 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-16T16:40:04.870 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:40:04.870 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-11-16T16:40:04.870 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-16T16:40:04.870 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:04.870 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-16T16:40:04.870 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:40:04.870 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-16T16:40:04.871 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-16T16:40:04.871 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-16T16:40:04.871 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-16T16:40:04.871 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-16T16:40:04.871 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-16T16:40:04.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:04.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-16T16:40:04.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-16T16:40:04.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-16T16:40:04.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-16T16:40:04.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-16T16:40:04.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-16T16:40:04.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-16T16:40:04.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-16T16:40:04.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-16T16:40:04.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:04.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-16T16:40:04.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:40:04.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-16T16:40:04.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-16T16:40:04.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:04.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-16T16:40:04.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": true, 2023-11-16T16:40:04.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-16T16:40:04.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-16T16:40:04.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-16T16:40:04.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-16T16:40:04.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2023-11-16T16:40:04.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:40:04.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T16:40:04.881 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:04.881 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-16T16:40:04.881 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:40:04.881 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2023-11-16T16:39:41.797141+0000", 2023-11-16T16:40:04.881 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T16:40:04.881 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:04.881 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-16T16:40:04.881 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2023-11-16T16:40:04.881 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.127+0000 7f6908f40700 1 Processor -- start 2023-11-16T16:40:04.881 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.127+0000 7f6908f40700 1 -- start start 2023-11-16T16:40:04.881 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.127+0000 7f6908f40700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6904105950 0x7f6904105d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:04.881 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.127+0000 7f6908f40700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69041062a0 con 0x7f6904105950 2023-11-16T16:40:04.881 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.128+0000 7f690259c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6904105950 0x7f6904105d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:04.882 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.128+0000 7f690259c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6904105950 0x7f6904105d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35848/0 (socket says 172.21.15.44:35848) 2023-11-16T16:40:04.882 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.128+0000 7f690259c700 1 -- 172.21.15.44:0/3080243195 learned_addr learned my addr 172.21.15.44:0/3080243195 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:04.882 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.128+0000 7f690259c700 1 -- 172.21.15.44:0/3080243195 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69041063e0 con 0x7f6904105950 2023-11-16T16:40:04.882 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.128+0000 7f690259c700 1 --2- 172.21.15.44:0/3080243195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6904105950 0x7f6904105d60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f68f40098a0 tx=0x7f68f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee26cd68b257a020 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:04.882 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.129+0000 7f690159a700 1 -- 172.21.15.44:0/3080243195 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68f4004160 con 0x7f6904105950 2023-11-16T16:40:04.882 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.129+0000 7f690159a700 1 -- 172.21.15.44:0/3080243195 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68f40042c0 con 0x7f6904105950 2023-11-16T16:40:04.882 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.129+0000 7f690159a700 1 -- 172.21.15.44:0/3080243195 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68f4011a00 con 0x7f6904105950 2023-11-16T16:40:04.882 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.129+0000 7f6908f40700 1 -- 172.21.15.44:0/3080243195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6904105950 msgr2=0x7f6904105d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:04.882 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.129+0000 7f6908f40700 1 --2- 172.21.15.44:0/3080243195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6904105950 0x7f6904105d60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f68f40098a0 tx=0x7f68f4009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:04.882 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.129+0000 7f6908f40700 1 -- 172.21.15.44:0/3080243195 shutdown_connections 2023-11-16T16:40:04.882 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.129+0000 7f6908f40700 1 --2- 172.21.15.44:0/3080243195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6904105950 0x7f6904105d60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:04.882 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.129+0000 7f6908f40700 1 -- 172.21.15.44:0/3080243195 >> 172.21.15.44:0/3080243195 conn(0x7f6904100fe0 msgr2=0x7f69041033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:04.883 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.129+0000 7f6908f40700 1 -- 172.21.15.44:0/3080243195 shutdown_connections 2023-11-16T16:40:04.883 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.130+0000 7f6908f40700 1 -- 172.21.15.44:0/3080243195 wait complete. 2023-11-16T16:40:04.883 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.131+0000 7f6908f40700 1 Processor -- start 2023-11-16T16:40:04.883 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.131+0000 7f6908f40700 1 -- start start 2023-11-16T16:40:04.883 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.132+0000 7f6908f40700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6904105950 0x7f690407bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:04.883 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.132+0000 7f6908f40700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690407c150 con 0x7f6904105950 2023-11-16T16:40:04.883 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.132+0000 7f690259c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6904105950 0x7f690407bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:04.883 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.132+0000 7f690259c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6904105950 0x7f690407bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35850/0 (socket says 172.21.15.44:35850) 2023-11-16T16:40:04.883 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.132+0000 7f690259c700 1 -- 172.21.15.44:0/3141669214 learned_addr learned my addr 172.21.15.44:0/3141669214 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:04.883 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.132+0000 7f690259c700 1 -- 172.21.15.44:0/3141669214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f4009550 con 0x7f6904105950 2023-11-16T16:40:04.883 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.132+0000 7f690259c700 1 --2- 172.21.15.44:0/3141669214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6904105950 0x7f690407bc10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f68f4006af0 tx=0x7f68f4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:04.883 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.133+0000 7f68f37fe700 1 -- 172.21.15.44:0/3141669214 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68f4003a40 con 0x7f6904105950 2023-11-16T16:40:04.883 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.133+0000 7f68f37fe700 1 -- 172.21.15.44:0/3141669214 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68f4003ba0 con 0x7f6904105950 2023-11-16T16:40:04.884 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.133+0000 7f68f37fe700 1 -- 172.21.15.44:0/3141669214 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68f401a6e0 con 0x7f6904105950 2023-11-16T16:40:04.884 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.133+0000 7f6908f40700 1 -- 172.21.15.44:0/3141669214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69041079a0 con 0x7f6904105950 2023-11-16T16:40:04.884 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.133+0000 7f6908f40700 1 -- 172.21.15.44:0/3141669214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6904107810 con 0x7f6904105950 2023-11-16T16:40:04.884 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.134+0000 7f6908f40700 1 -- 172.21.15.44:0/3141669214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6904062120 con 0x7f6904105950 2023-11-16T16:40:04.884 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.134+0000 7f68f37fe700 1 -- 172.21.15.44:0/3141669214 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f68f4024a60 con 0x7f6904105950 2023-11-16T16:40:04.884 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.134+0000 7f68f37fe700 1 --2- 172.21.15.44:0/3141669214 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7f68ec0387d0 0x7f68ec03ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:04.884 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.134+0000 7f68f37fe700 1 -- 172.21.15.44:0/3141669214 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f68f404c690 con 0x7f6904105950 2023-11-16T16:40:04.884 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.135+0000 7f6901d9b700 1 --2- 172.21.15.44:0/3141669214 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7f68ec0387d0 0x7f68ec03ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:04.884 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.135+0000 7f6901d9b700 1 --2- 172.21.15.44:0/3141669214 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7f68ec0387d0 0x7f68ec03ac80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f68f8006fd0 tx=0x7f68f8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:04.884 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.140+0000 7f68f37fe700 1 -- 172.21.15.44:0/3141669214 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f68f4024430 con 0x7f6904105950 2023-11-16T16:40:04.884 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.424+0000 7f6908f40700 1 -- 172.21.15.44:0/3141669214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f690404e8c0 con 0x7f6904105950 2023-11-16T16:40:04.884 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.424+0000 7f68f37fe700 1 -- 172.21.15.44:0/3141669214 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f68f4018350 con 0x7f6904105950 2023-11-16T16:40:04.885 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.427+0000 7f6908f40700 1 -- 172.21.15.44:0/3141669214 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7f68ec0387d0 msgr2=0x7f68ec03ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:04.885 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.427+0000 7f6908f40700 1 --2- 172.21.15.44:0/3141669214 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7f68ec0387d0 0x7f68ec03ac80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f68f8006fd0 tx=0x7f68f8006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:04.885 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.427+0000 7f6908f40700 1 -- 172.21.15.44:0/3141669214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6904105950 msgr2=0x7f690407bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:04.885 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.427+0000 7f6908f40700 1 --2- 172.21.15.44:0/3141669214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6904105950 0x7f690407bc10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f68f4006af0 tx=0x7f68f4011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:04.885 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.427+0000 7f6908f40700 1 -- 172.21.15.44:0/3141669214 shutdown_connections 2023-11-16T16:40:04.885 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.427+0000 7f6908f40700 1 --2- 172.21.15.44:0/3141669214 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7f68ec0387d0 0x7f68ec03ac80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:04.885 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.427+0000 7f6908f40700 1 --2- 172.21.15.44:0/3141669214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6904105950 0x7f690407bc10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:04.885 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.427+0000 7f6908f40700 1 -- 172.21.15.44:0/3141669214 >> 172.21.15.44:0/3141669214 conn(0x7f6904100fe0 msgr2=0x7f6904101c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:04.885 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.427+0000 7f6908f40700 1 -- 172.21.15.44:0/3141669214 shutdown_connections 2023-11-16T16:40:04.885 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.427+0000 7f6908f40700 1 -- 172.21.15.44:0/3141669214 wait complete. 2023-11-16T16:40:04.885 INFO:teuthology.orchestra.run.smithi044.stdout:mgr is available 2023-11-16T16:40:06.060 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2023-11-16T16:40:06.060 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [global] 2023-11-16T16:40:06.060 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout fsid = 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:40:06.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-11-16T16:40:06.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-11-16T16:40:06.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-11-16T16:40:06.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-11-16T16:40:06.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2023-11-16T16:40:06.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [mgr] 2023-11-16T16:40:06.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-11-16T16:40:06.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2023-11-16T16:40:06.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [osd] 2023-11-16T16:40:06.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-11-16T16:40:06.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-11-16T16:40:06.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-11-16T16:40:06.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-11-16T16:40:06.062 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-11-16T16:40:06.062 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.400+0000 7fcd5f925700 1 Processor -- start 2023-11-16T16:40:06.062 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.400+0000 7fcd5f925700 1 -- start start 2023-11-16T16:40:06.062 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.400+0000 7fcd5f925700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd5807bd80 0x7fcd5807a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:06.062 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.400+0000 7fcd5f925700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5807a7c0 con 0x7fcd5807bd80 2023-11-16T16:40:06.062 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.401+0000 7fcd5d6c1700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd5807bd80 0x7fcd5807a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:06.062 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.401+0000 7fcd5d6c1700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd5807bd80 0x7fcd5807a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39168/0 (socket says 172.21.15.44:39168) 2023-11-16T16:40:06.062 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.401+0000 7fcd5d6c1700 1 -- 172.21.15.44:0/332250677 learned_addr learned my addr 172.21.15.44:0/332250677 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:06.062 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.401+0000 7fcd5d6c1700 1 -- 172.21.15.44:0/332250677 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5807a900 con 0x7fcd5807bd80 2023-11-16T16:40:06.062 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.401+0000 7fcd5d6c1700 1 --2- 172.21.15.44:0/332250677 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd5807bd80 0x7fcd5807a280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcd540098a0 tx=0x7fcd54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=274c8bdc87b2eed9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:06.062 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.402+0000 7fcd4ffff700 1 -- 172.21.15.44:0/332250677 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd54004160 con 0x7fcd5807bd80 2023-11-16T16:40:06.062 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.402+0000 7fcd4ffff700 1 -- 172.21.15.44:0/332250677 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd540042c0 con 0x7fcd5807bd80 2023-11-16T16:40:06.063 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.402+0000 7fcd4ffff700 1 -- 172.21.15.44:0/332250677 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd54011a00 con 0x7fcd5807bd80 2023-11-16T16:40:06.063 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.402+0000 7fcd5f925700 1 -- 172.21.15.44:0/332250677 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd5807bd80 msgr2=0x7fcd5807a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:06.063 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.402+0000 7fcd5f925700 1 --2- 172.21.15.44:0/332250677 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd5807bd80 0x7fcd5807a280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcd540098a0 tx=0x7fcd54009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:06.063 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.402+0000 7fcd5f925700 1 -- 172.21.15.44:0/332250677 shutdown_connections 2023-11-16T16:40:06.063 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.402+0000 7fcd5f925700 1 --2- 172.21.15.44:0/332250677 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd5807bd80 0x7fcd5807a280 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:06.063 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.402+0000 7fcd5f925700 1 -- 172.21.15.44:0/332250677 >> 172.21.15.44:0/332250677 conn(0x7fcd58100fe0 msgr2=0x7fcd581033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:06.063 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.402+0000 7fcd5f925700 1 -- 172.21.15.44:0/332250677 shutdown_connections 2023-11-16T16:40:06.063 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.402+0000 7fcd5f925700 1 -- 172.21.15.44:0/332250677 wait complete. 2023-11-16T16:40:06.063 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.403+0000 7fcd5f925700 1 Processor -- start 2023-11-16T16:40:06.063 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.404+0000 7fcd5f925700 1 -- start start 2023-11-16T16:40:06.063 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.404+0000 7fcd5f925700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd5807bd80 0x7fcd5819baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:06.063 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.404+0000 7fcd5f925700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5819c030 con 0x7fcd5807bd80 2023-11-16T16:40:06.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.404+0000 7fcd5d6c1700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd5807bd80 0x7fcd5819baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:06.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.404+0000 7fcd5d6c1700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd5807bd80 0x7fcd5819baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39178/0 (socket says 172.21.15.44:39178) 2023-11-16T16:40:06.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.404+0000 7fcd5d6c1700 1 -- 172.21.15.44:0/832448138 learned_addr learned my addr 172.21.15.44:0/832448138 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:06.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.404+0000 7fcd5d6c1700 1 -- 172.21.15.44:0/832448138 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd54009550 con 0x7fcd5807bd80 2023-11-16T16:40:06.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.405+0000 7fcd5d6c1700 1 --2- 172.21.15.44:0/832448138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd5807bd80 0x7fcd5819baf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcd54006af0 tx=0x7fcd54011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:06.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.405+0000 7fcd4e7fc700 1 -- 172.21.15.44:0/832448138 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd54003a40 con 0x7fcd5807bd80 2023-11-16T16:40:06.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.405+0000 7fcd4e7fc700 1 -- 172.21.15.44:0/832448138 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd54003ba0 con 0x7fcd5807bd80 2023-11-16T16:40:06.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.405+0000 7fcd4e7fc700 1 -- 172.21.15.44:0/832448138 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd5401a700 con 0x7fcd5807bd80 2023-11-16T16:40:06.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.405+0000 7fcd5f925700 1 -- 172.21.15.44:0/832448138 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd5810be00 con 0x7fcd5807bd80 2023-11-16T16:40:06.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.405+0000 7fcd5f925700 1 -- 172.21.15.44:0/832448138 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd5810bc70 con 0x7fcd5807bd80 2023-11-16T16:40:06.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.406+0000 7fcd4e7fc700 1 -- 172.21.15.44:0/832448138 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fcd54024ad0 con 0x7fcd5807bd80 2023-11-16T16:40:06.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.406+0000 7fcd5f925700 1 -- 172.21.15.44:0/832448138 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd58062120 con 0x7fcd5807bd80 2023-11-16T16:40:06.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.407+0000 7fcd4e7fc700 1 --2- 172.21.15.44:0/832448138 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fcd440387d0 0x7fcd4403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:06.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.407+0000 7fcd4e7fc700 1 -- 172.21.15.44:0/832448138 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcd5404c630 con 0x7fcd5807bd80 2023-11-16T16:40:06.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.407+0000 7fcd5cec0700 1 --2- 172.21.15.44:0/832448138 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fcd440387d0 0x7fcd4403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:06.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.408+0000 7fcd5cec0700 1 --2- 172.21.15.44:0/832448138 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fcd440387d0 0x7fcd4403ac80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcd48006fd0 tx=0x7fcd48006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:06.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.413+0000 7fcd4e7fc700 1 -- 172.21.15.44:0/832448138 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcd54018bd0 con 0x7fcd5807bd80 2023-11-16T16:40:06.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.618+0000 7fcd5f925700 1 -- 172.21.15.44:0/832448138 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fcd5804e8c0 con 0x7fcd5807bd80 2023-11-16T16:40:06.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.619+0000 7fcd4e7fc700 1 -- 172.21.15.44:0/832448138 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fcd540187c0 con 0x7fcd5807bd80 2023-11-16T16:40:06.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.621+0000 7fcd5f925700 1 -- 172.21.15.44:0/832448138 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fcd440387d0 msgr2=0x7fcd4403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:06.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.621+0000 7fcd5f925700 1 --2- 172.21.15.44:0/832448138 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fcd440387d0 0x7fcd4403ac80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcd48006fd0 tx=0x7fcd48006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:06.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.621+0000 7fcd5f925700 1 -- 172.21.15.44:0/832448138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd5807bd80 msgr2=0x7fcd5819baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:06.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.621+0000 7fcd5f925700 1 --2- 172.21.15.44:0/832448138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd5807bd80 0x7fcd5819baf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcd54006af0 tx=0x7fcd54011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:06.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.621+0000 7fcd5f925700 1 -- 172.21.15.44:0/832448138 shutdown_connections 2023-11-16T16:40:06.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.621+0000 7fcd5f925700 1 --2- 172.21.15.44:0/832448138 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fcd440387d0 0x7fcd4403ac80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:06.066 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.621+0000 7fcd5f925700 1 --2- 172.21.15.44:0/832448138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd5807bd80 0x7fcd5819baf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:06.066 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.621+0000 7fcd5f925700 1 -- 172.21.15.44:0/832448138 >> 172.21.15.44:0/832448138 conn(0x7fcd58100fe0 msgr2=0x7fcd58101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:06.066 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.621+0000 7fcd5f925700 1 -- 172.21.15.44:0/832448138 shutdown_connections 2023-11-16T16:40:06.066 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.621+0000 7fcd5f925700 1 -- 172.21.15.44:0/832448138 wait complete. 2023-11-16T16:40:06.066 INFO:teuthology.orchestra.run.smithi044.stdout:Enabling cephadm module... 2023-11-16T16:40:06.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:05 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/832448138' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-16T16:40:07.196 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:06 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1593366694' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-16T16:40:07.197 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:07 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: ignoring --setuser ceph since I am not root 2023-11-16T16:40:07.197 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:07 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: ignoring --setgroup ceph since I am not root 2023-11-16T16:40:07.197 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:07 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:07.195+0000 7f0a97e52200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T16:40:07.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.601+0000 7fcd2c361700 1 Processor -- start 2023-11-16T16:40:07.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.602+0000 7fcd2c361700 1 -- start start 2023-11-16T16:40:07.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.602+0000 7fcd2c361700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd2407bde0 0x7fcd2407a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:07.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.602+0000 7fcd2c361700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2407a820 con 0x7fcd2407bde0 2023-11-16T16:40:07.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.602+0000 7fcd2a0fd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd2407bde0 0x7fcd2407a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:07.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.602+0000 7fcd2a0fd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd2407bde0 0x7fcd2407a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39186/0 (socket says 172.21.15.44:39186) 2023-11-16T16:40:07.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.603+0000 7fcd2a0fd700 1 -- 172.21.15.44:0/698645352 learned_addr learned my addr 172.21.15.44:0/698645352 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:07.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.603+0000 7fcd2a0fd700 1 -- 172.21.15.44:0/698645352 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2407a960 con 0x7fcd2407bde0 2023-11-16T16:40:07.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.603+0000 7fcd2a0fd700 1 --2- 172.21.15.44:0/698645352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd2407bde0 0x7fcd2407a2e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcd200098a0 tx=0x7fcd20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a22acfb4cd0d1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:07.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.603+0000 7fcd290fb700 1 -- 172.21.15.44:0/698645352 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd20004160 con 0x7fcd2407bde0 2023-11-16T16:40:07.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.603+0000 7fcd290fb700 1 -- 172.21.15.44:0/698645352 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd200042c0 con 0x7fcd2407bde0 2023-11-16T16:40:07.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.603+0000 7fcd290fb700 1 -- 172.21.15.44:0/698645352 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd20011a00 con 0x7fcd2407bde0 2023-11-16T16:40:07.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.604+0000 7fcd2c361700 1 -- 172.21.15.44:0/698645352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd2407bde0 msgr2=0x7fcd2407a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:07.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.604+0000 7fcd2c361700 1 --2- 172.21.15.44:0/698645352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd2407bde0 0x7fcd2407a2e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcd200098a0 tx=0x7fcd20009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:07.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.604+0000 7fcd2c361700 1 -- 172.21.15.44:0/698645352 shutdown_connections 2023-11-16T16:40:07.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.604+0000 7fcd2c361700 1 --2- 172.21.15.44:0/698645352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd2407bde0 0x7fcd2407a2e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:07.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.604+0000 7fcd2c361700 1 -- 172.21.15.44:0/698645352 >> 172.21.15.44:0/698645352 conn(0x7fcd24100fe0 msgr2=0x7fcd241033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:07.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.604+0000 7fcd2c361700 1 -- 172.21.15.44:0/698645352 shutdown_connections 2023-11-16T16:40:07.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.604+0000 7fcd2c361700 1 -- 172.21.15.44:0/698645352 wait complete. 2023-11-16T16:40:07.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.605+0000 7fcd2c361700 1 Processor -- start 2023-11-16T16:40:07.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.605+0000 7fcd2c361700 1 -- start start 2023-11-16T16:40:07.446 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.606+0000 7fcd2c361700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd2407bde0 0x7fcd2419bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:07.446 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.606+0000 7fcd2c361700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2419c050 con 0x7fcd2407bde0 2023-11-16T16:40:07.446 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.606+0000 7fcd2a0fd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd2407bde0 0x7fcd2419bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:07.446 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.606+0000 7fcd2a0fd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd2407bde0 0x7fcd2419bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39196/0 (socket says 172.21.15.44:39196) 2023-11-16T16:40:07.446 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.606+0000 7fcd2a0fd700 1 -- 172.21.15.44:0/1593366694 learned_addr learned my addr 172.21.15.44:0/1593366694 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:07.446 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.606+0000 7fcd2a0fd700 1 -- 172.21.15.44:0/1593366694 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd20009550 con 0x7fcd2407bde0 2023-11-16T16:40:07.446 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.606+0000 7fcd2a0fd700 1 --2- 172.21.15.44:0/1593366694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd2407bde0 0x7fcd2419bb10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcd20006af0 tx=0x7fcd20011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:07.446 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.607+0000 7fcd1b7fe700 1 -- 172.21.15.44:0/1593366694 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd20003a40 con 0x7fcd2407bde0 2023-11-16T16:40:07.446 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.607+0000 7fcd1b7fe700 1 -- 172.21.15.44:0/1593366694 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd20003ba0 con 0x7fcd2407bde0 2023-11-16T16:40:07.446 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.607+0000 7fcd1b7fe700 1 -- 172.21.15.44:0/1593366694 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd2001a700 con 0x7fcd2407bde0 2023-11-16T16:40:07.446 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.607+0000 7fcd2c361700 1 -- 172.21.15.44:0/1593366694 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2410be00 con 0x7fcd2407bde0 2023-11-16T16:40:07.446 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.607+0000 7fcd2c361700 1 -- 172.21.15.44:0/1593366694 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2410bc70 con 0x7fcd2407bde0 2023-11-16T16:40:07.446 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.608+0000 7fcd1b7fe700 1 -- 172.21.15.44:0/1593366694 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fcd20024ad0 con 0x7fcd2407bde0 2023-11-16T16:40:07.447 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.608+0000 7fcd2c361700 1 -- 172.21.15.44:0/1593366694 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd24195960 con 0x7fcd2407bde0 2023-11-16T16:40:07.447 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.608+0000 7fcd1b7fe700 1 --2- 172.21.15.44:0/1593366694 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fcd100387d0 0x7fcd1003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:07.447 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.608+0000 7fcd1b7fe700 1 -- 172.21.15.44:0/1593366694 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcd2004c630 con 0x7fcd2407bde0 2023-11-16T16:40:07.447 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.609+0000 7fcd298fc700 1 --2- 172.21.15.44:0/1593366694 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fcd100387d0 0x7fcd1003ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:07.447 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.609+0000 7fcd298fc700 1 --2- 172.21.15.44:0/1593366694 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fcd100387d0 0x7fcd1003ac80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcd14006fd0 tx=0x7fcd14006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:07.447 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.613+0000 7fcd1b7fe700 1 -- 172.21.15.44:0/1593366694 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcd20018bd0 con 0x7fcd2407bde0 2023-11-16T16:40:07.447 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.874+0000 7fcd2c361700 1 -- 172.21.15.44:0/1593366694 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fcd2402cfe0 con 0x7fcd2407bde0 2023-11-16T16:40:07.447 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.928+0000 7fcd1b7fe700 1 -- 172.21.15.44:0/1593366694 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fcd2002d770 con 0x7fcd2407bde0 2023-11-16T16:40:07.447 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.929+0000 7fcd1b7fe700 1 -- 172.21.15.44:0/1593366694 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fcd200172b0 con 0x7fcd2407bde0 2023-11-16T16:40:07.447 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.932+0000 7fcd2c361700 1 -- 172.21.15.44:0/1593366694 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fcd100387d0 msgr2=0x7fcd1003ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:07.447 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.932+0000 7fcd2c361700 1 --2- 172.21.15.44:0/1593366694 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fcd100387d0 0x7fcd1003ac80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcd14006fd0 tx=0x7fcd14006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:07.447 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.932+0000 7fcd2c361700 1 -- 172.21.15.44:0/1593366694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd2407bde0 msgr2=0x7fcd2419bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:07.447 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.932+0000 7fcd2c361700 1 --2- 172.21.15.44:0/1593366694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd2407bde0 0x7fcd2419bb10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcd20006af0 tx=0x7fcd20011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:07.447 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.932+0000 7fcd2c361700 1 -- 172.21.15.44:0/1593366694 shutdown_connections 2023-11-16T16:40:07.448 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.932+0000 7fcd2c361700 1 --2- 172.21.15.44:0/1593366694 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fcd100387d0 0x7fcd1003ac80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:07.448 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.932+0000 7fcd2c361700 1 --2- 172.21.15.44:0/1593366694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd2407bde0 0x7fcd2419bb10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:07.448 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.932+0000 7fcd2c361700 1 -- 172.21.15.44:0/1593366694 >> 172.21.15.44:0/1593366694 conn(0x7fcd24100fe0 msgr2=0x7fcd24101de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:07.448 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.932+0000 7fcd2c361700 1 -- 172.21.15.44:0/1593366694 shutdown_connections 2023-11-16T16:40:07.448 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.932+0000 7fcd2c361700 1 -- 172.21.15.44:0/1593366694 wait complete. 2023-11-16T16:40:07.471 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:07 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:07.303+0000 7f0a97e52200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T16:40:07.471 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:07 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:07.420+0000 7f0a97e52200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T16:40:07.741 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:07 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:07.557+0000 7f0a97e52200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T16:40:08.118 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:07 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:07.792+0000 7f0a97e52200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T16:40:08.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:07 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1593366694' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-16T16:40:08.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:07 smithi044 ceph-mon[128750]: mgrmap e5: x(active, since 8s) 2023-11-16T16:40:08.499 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:08 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:08.201+0000 7f0a97e52200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T16:40:08.499 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:08 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:08.324+0000 7f0a97e52200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T16:40:08.730 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2023-11-16T16:40:08.730 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-11-16T16:40:08.730 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": true, 2023-11-16T16:40:08.731 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-11-16T16:40:08.731 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-11-16T16:40:08.731 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2023-11-16T16:40:08.731 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.971+0000 7f315afed700 1 Processor -- start 2023-11-16T16:40:08.731 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.972+0000 7f315afed700 1 -- start start 2023-11-16T16:40:08.731 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.972+0000 7f315afed700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f315407bda0 0x7f315407a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:08.731 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.972+0000 7f315afed700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315407a7e0 con 0x7f315407bda0 2023-11-16T16:40:08.731 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.972+0000 7f3158d89700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f315407bda0 0x7f315407a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:08.731 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.972+0000 7f3158d89700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f315407bda0 0x7f315407a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39224/0 (socket says 172.21.15.44:39224) 2023-11-16T16:40:08.731 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.972+0000 7f3158d89700 1 -- 172.21.15.44:0/1913808983 learned_addr learned my addr 172.21.15.44:0/1913808983 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:08.731 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.973+0000 7f3158d89700 1 -- 172.21.15.44:0/1913808983 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315407a920 con 0x7f315407bda0 2023-11-16T16:40:08.732 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.973+0000 7f3158d89700 1 --2- 172.21.15.44:0/1913808983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f315407bda0 0x7f315407a2a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f31480098a0 tx=0x7f3148009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0896aa49984000 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:08.732 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.973+0000 7f31537fe700 1 -- 172.21.15.44:0/1913808983 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3148004160 con 0x7f315407bda0 2023-11-16T16:40:08.732 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.973+0000 7f31537fe700 1 -- 172.21.15.44:0/1913808983 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31480042c0 con 0x7f315407bda0 2023-11-16T16:40:08.732 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.973+0000 7f31537fe700 1 -- 172.21.15.44:0/1913808983 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3148011a00 con 0x7f315407bda0 2023-11-16T16:40:08.732 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.974+0000 7f315afed700 1 -- 172.21.15.44:0/1913808983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f315407bda0 msgr2=0x7f315407a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:08.732 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.974+0000 7f315afed700 1 --2- 172.21.15.44:0/1913808983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f315407bda0 0x7f315407a2a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f31480098a0 tx=0x7f3148009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:08.732 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.974+0000 7f315afed700 1 -- 172.21.15.44:0/1913808983 shutdown_connections 2023-11-16T16:40:08.732 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.974+0000 7f315afed700 1 --2- 172.21.15.44:0/1913808983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f315407bda0 0x7f315407a2a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:08.732 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.974+0000 7f315afed700 1 -- 172.21.15.44:0/1913808983 >> 172.21.15.44:0/1913808983 conn(0x7f3154100fe0 msgr2=0x7f31541033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:08.732 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.974+0000 7f315afed700 1 -- 172.21.15.44:0/1913808983 shutdown_connections 2023-11-16T16:40:08.732 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.974+0000 7f315afed700 1 -- 172.21.15.44:0/1913808983 wait complete. 2023-11-16T16:40:08.732 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.976+0000 7f315afed700 1 Processor -- start 2023-11-16T16:40:08.733 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.976+0000 7f315afed700 1 -- start start 2023-11-16T16:40:08.733 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.976+0000 7f315afed700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f315407bda0 0x7f315419baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:08.733 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.976+0000 7f315afed700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315419c030 con 0x7f315407bda0 2023-11-16T16:40:08.733 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.977+0000 7f3158d89700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f315407bda0 0x7f315419baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:08.733 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.977+0000 7f3158d89700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f315407bda0 0x7f315419baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39230/0 (socket says 172.21.15.44:39230) 2023-11-16T16:40:08.733 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.977+0000 7f3158d89700 1 -- 172.21.15.44:0/3047313590 learned_addr learned my addr 172.21.15.44:0/3047313590 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:08.733 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.977+0000 7f3158d89700 1 -- 172.21.15.44:0/3047313590 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3148009550 con 0x7f315407bda0 2023-11-16T16:40:08.733 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.977+0000 7f3158d89700 1 --2- 172.21.15.44:0/3047313590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f315407bda0 0x7f315419baf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3148006af0 tx=0x7f3148011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:08.733 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.977+0000 7f3151ffb700 1 -- 172.21.15.44:0/3047313590 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3148003a40 con 0x7f315407bda0 2023-11-16T16:40:08.733 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.978+0000 7f3151ffb700 1 -- 172.21.15.44:0/3047313590 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3148003ba0 con 0x7f315407bda0 2023-11-16T16:40:08.733 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.978+0000 7f315afed700 1 -- 172.21.15.44:0/3047313590 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f315410be40 con 0x7f315407bda0 2023-11-16T16:40:08.733 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.978+0000 7f3151ffb700 1 -- 172.21.15.44:0/3047313590 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f314801a700 con 0x7f315407bda0 2023-11-16T16:40:08.734 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.978+0000 7f315afed700 1 -- 172.21.15.44:0/3047313590 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f315410bcb0 con 0x7f315407bda0 2023-11-16T16:40:08.734 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.979+0000 7f3151ffb700 1 -- 172.21.15.44:0/3047313590 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f3148024430 con 0x7f315407bda0 2023-11-16T16:40:08.734 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.979+0000 7f315afed700 1 -- 172.21.15.44:0/3047313590 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3154062120 con 0x7f315407bda0 2023-11-16T16:40:08.734 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.980+0000 7f3151ffb700 1 --2- 172.21.15.44:0/3047313590 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7f3144038820 0x7f314403acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:08.734 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.980+0000 7f3151ffb700 1 -- 172.21.15.44:0/3047313590 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f314804c680 con 0x7f315407bda0 2023-11-16T16:40:08.734 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.980+0000 7f3153fff700 1 -- 172.21.15.44:0/3047313590 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7f3144038820 msgr2=0x7f314403acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1852059921 2023-11-16T16:40:08.734 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.980+0000 7f3153fff700 1 --2- 172.21.15.44:0/3047313590 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7f3144038820 0x7f314403acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:40:08.734 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.985+0000 7f3151ffb700 1 -- 172.21.15.44:0/3047313590 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f31480189a0 con 0x7f315407bda0 2023-11-16T16:40:08.734 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.181+0000 7f3153fff700 1 -- 172.21.15.44:0/3047313590 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7f3144038820 msgr2=0x7f314403acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1852059921 2023-11-16T16:40:08.734 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.181+0000 7f3153fff700 1 --2- 172.21.15.44:0/3047313590 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7f3144038820 0x7f314403acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T16:40:08.734 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.249+0000 7f315afed700 1 -- 172.21.15.44:0/3047313590 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f315404e8c0 con 0x7f315407bda0 2023-11-16T16:40:08.735 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.249+0000 7f3151ffb700 1 -- 172.21.15.44:0/3047313590 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f31480187c0 con 0x7f315407bda0 2023-11-16T16:40:08.735 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.252+0000 7f315afed700 1 -- 172.21.15.44:0/3047313590 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7f3144038820 msgr2=0x7f314403acd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:40:08.735 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.252+0000 7f315afed700 1 --2- 172.21.15.44:0/3047313590 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7f3144038820 0x7f314403acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:08.735 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.252+0000 7f315afed700 1 -- 172.21.15.44:0/3047313590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f315407bda0 msgr2=0x7f315419baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:08.735 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.252+0000 7f315afed700 1 --2- 172.21.15.44:0/3047313590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f315407bda0 0x7f315419baf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3148006af0 tx=0x7f3148011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:08.735 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.252+0000 7f315afed700 1 -- 172.21.15.44:0/3047313590 shutdown_connections 2023-11-16T16:40:08.735 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.252+0000 7f315afed700 1 --2- 172.21.15.44:0/3047313590 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7f3144038820 0x7f314403acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:08.735 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.252+0000 7f315afed700 1 --2- 172.21.15.44:0/3047313590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f315407bda0 0x7f315419baf0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:08.735 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.252+0000 7f315afed700 1 -- 172.21.15.44:0/3047313590 >> 172.21.15.44:0/3047313590 conn(0x7f3154100fe0 msgr2=0x7f3154101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:08.735 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.252+0000 7f315afed700 1 -- 172.21.15.44:0/3047313590 shutdown_connections 2023-11-16T16:40:08.735 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.252+0000 7f315afed700 1 -- 172.21.15.44:0/3047313590 wait complete. 2023-11-16T16:40:08.736 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for the mgr to restart... 2023-11-16T16:40:08.736 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr epoch 5... 2023-11-16T16:40:08.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:08 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:08.498+0000 7f0a97e52200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T16:40:08.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:08 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:08.627+0000 7f0a97e52200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T16:40:09.239 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:08 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3047313590' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-16T16:40:09.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:09 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:09.238+0000 7f0a97e52200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T16:40:10.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:10 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:10.008+0000 7f0a97e52200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T16:40:10.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:10 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:10.220+0000 7f0a97e52200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T16:40:11.551 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:11 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:11.197+0000 7f0a97e52200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:40:11.552 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:11 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:11.320+0000 7f0a97e52200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T16:40:11.552 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:11 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:11.550+0000 7f0a97e52200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:40:12.208 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:11 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:11.924+0000 7f0a97e52200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T16:40:12.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:12 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:12.207+0000 7f0a97e52200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T16:40:12.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:12 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:12.406+0000 7f0a97e52200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T16:40:13.118 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:12 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:12.760+0000 7f0a97e52200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T16:40:14.118 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:13 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:13.718+0000 7f0a97e52200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T16:40:14.118 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:13 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:13.895+0000 7f0a97e52200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T16:40:14.589 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:14 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:14.150+0000 7f0a97e52200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T16:40:14.589 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:14 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:14.303+0000 7f0a97e52200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T16:40:14.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:14 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:14.588+0000 7f0a97e52200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T16:40:16.118 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:15 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:15.682+0000 7f0a97e52200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T16:40:16.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:15 smithi044 ceph-mon[128750]: Active manager daemon x restarted 2023-11-16T16:40:16.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:15 smithi044 ceph-mon[128750]: Activating manager daemon x 2023-11-16T16:40:16.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:15 smithi044 ceph-mon[128750]: osdmap e2: 0 total, 0 up, 0 in 2023-11-16T16:40:16.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:15 smithi044 ceph-mon[128750]: mgrmap e6: x(active, starting, since 0.00422474s) 2023-11-16T16:40:16.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:15 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:40:16.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:15 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T16:40:16.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:15 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T16:40:16.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:15 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T16:40:16.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:15 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T16:40:16.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:15 smithi044 ceph-mon[128750]: Manager daemon x is now available 2023-11-16T16:40:16.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:15 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' 2023-11-16T16:40:16.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:15 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' 2023-11-16T16:40:16.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:15 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:16.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:15 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:17.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:16 smithi044 ceph-mon[128750]: Found migration_current of "None". Setting to last migration. 2023-11-16T16:40:17.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:16 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:17.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:16 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T16:40:17.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:16 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T16:40:17.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:16 smithi044 ceph-mon[128750]: mgrmap e7: x(active, since 1.00736s) 2023-11-16T16:40:17.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2023-11-16T16:40:17.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-11-16T16:40:17.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "initialized": true 2023-11-16T16:40:17.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2023-11-16T16:40:17.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.287+0000 7fb2c44cf700 1 Processor -- start 2023-11-16T16:40:17.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.288+0000 7fb2c44cf700 1 -- start start 2023-11-16T16:40:17.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.288+0000 7fb2c44cf700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2bc07bd80 0x7fb2bc07a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:17.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.288+0000 7fb2c44cf700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc07a7c0 con 0x7fb2bc07bd80 2023-11-16T16:40:17.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.288+0000 7fb2c226b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2bc07bd80 0x7fb2bc07a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:17.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.288+0000 7fb2c226b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2bc07bd80 0x7fb2bc07a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39234/0 (socket says 172.21.15.44:39234) 2023-11-16T16:40:17.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.288+0000 7fb2c226b700 1 -- 172.21.15.44:0/4149252266 learned_addr learned my addr 172.21.15.44:0/4149252266 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:17.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.289+0000 7fb2c226b700 1 -- 172.21.15.44:0/4149252266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc07a900 con 0x7fb2bc07bd80 2023-11-16T16:40:17.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.289+0000 7fb2c226b700 1 --2- 172.21.15.44:0/4149252266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2bc07bd80 0x7fb2bc07a280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80098a0 tx=0x7fb2b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdffdca0d9e0839 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:17.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.289+0000 7fb2c1269700 1 -- 172.21.15.44:0/4149252266 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2b8004160 con 0x7fb2bc07bd80 2023-11-16T16:40:17.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.289+0000 7fb2c1269700 1 -- 172.21.15.44:0/4149252266 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2b80042c0 con 0x7fb2bc07bd80 2023-11-16T16:40:17.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.289+0000 7fb2c1269700 1 -- 172.21.15.44:0/4149252266 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2b8011a00 con 0x7fb2bc07bd80 2023-11-16T16:40:17.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.290+0000 7fb2c44cf700 1 -- 172.21.15.44:0/4149252266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2bc07bd80 msgr2=0x7fb2bc07a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:17.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.290+0000 7fb2c44cf700 1 --2- 172.21.15.44:0/4149252266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2bc07bd80 0x7fb2bc07a280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80098a0 tx=0x7fb2b8009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:17.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.290+0000 7fb2c44cf700 1 -- 172.21.15.44:0/4149252266 shutdown_connections 2023-11-16T16:40:17.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.290+0000 7fb2c44cf700 1 --2- 172.21.15.44:0/4149252266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2bc07bd80 0x7fb2bc07a280 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:17.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.290+0000 7fb2c44cf700 1 -- 172.21.15.44:0/4149252266 >> 172.21.15.44:0/4149252266 conn(0x7fb2bc100fe0 msgr2=0x7fb2bc1033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:17.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.290+0000 7fb2c44cf700 1 -- 172.21.15.44:0/4149252266 shutdown_connections 2023-11-16T16:40:17.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.290+0000 7fb2c44cf700 1 -- 172.21.15.44:0/4149252266 wait complete. 2023-11-16T16:40:17.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.292+0000 7fb2c44cf700 1 Processor -- start 2023-11-16T16:40:17.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.292+0000 7fb2c44cf700 1 -- start start 2023-11-16T16:40:17.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.292+0000 7fb2c44cf700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2bc07bd80 0x7fb2bc19baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:17.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.292+0000 7fb2c44cf700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc19c030 con 0x7fb2bc07bd80 2023-11-16T16:40:17.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.293+0000 7fb2c226b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2bc07bd80 0x7fb2bc19baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:17.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.293+0000 7fb2c226b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2bc07bd80 0x7fb2bc19baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39236/0 (socket says 172.21.15.44:39236) 2023-11-16T16:40:17.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.293+0000 7fb2c226b700 1 -- 172.21.15.44:0/589611256 learned_addr learned my addr 172.21.15.44:0/589611256 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:17.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.293+0000 7fb2c226b700 1 -- 172.21.15.44:0/589611256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b8009550 con 0x7fb2bc07bd80 2023-11-16T16:40:17.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.293+0000 7fb2c226b700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2bc07bd80 0x7fb2bc19baf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8006af0 tx=0x7fb2b8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:17.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.293+0000 7fb2b37fe700 1 -- 172.21.15.44:0/589611256 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2b8003a40 con 0x7fb2bc07bd80 2023-11-16T16:40:17.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.294+0000 7fb2b37fe700 1 -- 172.21.15.44:0/589611256 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2b8003ba0 con 0x7fb2bc07bd80 2023-11-16T16:40:17.162 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.294+0000 7fb2b37fe700 1 -- 172.21.15.44:0/589611256 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2b801a700 con 0x7fb2bc07bd80 2023-11-16T16:40:17.162 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.294+0000 7fb2c44cf700 1 -- 172.21.15.44:0/589611256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2bc10be00 con 0x7fb2bc07bd80 2023-11-16T16:40:17.162 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.294+0000 7fb2c44cf700 1 -- 172.21.15.44:0/589611256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2bc10bc70 con 0x7fb2bc07bd80 2023-11-16T16:40:17.162 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.295+0000 7fb2b37fe700 1 -- 172.21.15.44:0/589611256 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fb2b8024430 con 0x7fb2bc07bd80 2023-11-16T16:40:17.162 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.295+0000 7fb2b37fe700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fb2a80387d0 0x7fb2a803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:17.162 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.295+0000 7fb2b37fe700 1 -- 172.21.15.44:0/589611256 --> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2a803b390 con 0x7fb2a80387d0 2023-11-16T16:40:17.162 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.295+0000 7fb2b37fe700 1 -- 172.21.15.44:0/589611256 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb2b804c6d0 con 0x7fb2bc07bd80 2023-11-16T16:40:17.162 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.295+0000 7fb2c1a6a700 1 -- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fb2a80387d0 msgr2=0x7fb2a803ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1852059921 2023-11-16T16:40:17.162 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.295+0000 7fb2c1a6a700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fb2a80387d0 0x7fb2a803ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:40:17.163 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.496+0000 7fb2c1a6a700 1 -- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fb2a80387d0 msgr2=0x7fb2a803ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1852059921 2023-11-16T16:40:17.163 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.496+0000 7fb2c1a6a700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fb2a80387d0 0x7fb2a803ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T16:40:17.163 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.897+0000 7fb2c1a6a700 1 -- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fb2a80387d0 msgr2=0x7fb2a803ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1852059921 2023-11-16T16:40:17.163 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.897+0000 7fb2c1a6a700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fb2a80387d0 0x7fb2a803ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-16T16:40:17.163 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.698+0000 7fb2c1a6a700 1 -- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fb2a80387d0 msgr2=0x7fb2a803ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1852059921 2023-11-16T16:40:17.163 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.698+0000 7fb2c1a6a700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fb2a80387d0 0x7fb2a803ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-16T16:40:17.163 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:12.299+0000 7fb2c1a6a700 1 -- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fb2a80387d0 msgr2=0x7fb2a803ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1852059921 2023-11-16T16:40:17.163 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:12.299+0000 7fb2c1a6a700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fb2a80387d0 0x7fb2a803ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-16T16:40:17.163 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:15.503+0000 7fb2c1a6a700 1 -- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fb2a80387d0 msgr2=0x7fb2a803ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1852059921 2023-11-16T16:40:17.164 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:15.503+0000 7fb2c1a6a700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fb2a80387d0 0x7fb2a803ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-16T16:40:17.164 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:15.687+0000 7fb2b37fe700 1 -- 172.21.15.44:0/589611256 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7fb2b80246e0 con 0x7fb2bc07bd80 2023-11-16T16:40:17.164 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:15.687+0000 7fb2b37fe700 1 -- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fb2a80387d0 msgr2=0x7fb2a803ac80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:40:17.164 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:15.687+0000 7fb2b37fe700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fb2a80387d0 0x7fb2a803ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:17.164 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.688+0000 7fb2b37fe700 1 -- 172.21.15.44:0/589611256 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fb2b804dc90 con 0x7fb2bc07bd80 2023-11-16T16:40:17.164 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.689+0000 7fb2b37fe700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fb2a803bfb0 0x7fb2a803e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:17.164 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.689+0000 7fb2b37fe700 1 -- 172.21.15.44:0/589611256 --> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2a803b390 con 0x7fb2a803bfb0 2023-11-16T16:40:17.164 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.689+0000 7fb2c1a6a700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fb2a803bfb0 0x7fb2a803e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:17.164 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.691+0000 7fb2c1a6a700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fb2a803bfb0 0x7fb2a803e390 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac003a10 tx=0x7fb2ac009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:17.164 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.692+0000 7fb2b37fe700 1 -- 172.21.15.44:0/589611256 <== mgr.14118 v2:172.21.15.44:6800/4234382393 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fb2a803b390 con 0x7fb2a803bfb0 2023-11-16T16:40:17.165 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.699+0000 7fb2c44cf700 1 -- 172.21.15.44:0/589611256 --> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb2bc19ce30 con 0x7fb2a803bfb0 2023-11-16T16:40:17.165 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.699+0000 7fb2b37fe700 1 -- 172.21.15.44:0/589611256 <== mgr.14118 v2:172.21.15.44:6800/4234382393 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb2bc19ce30 con 0x7fb2a803bfb0 2023-11-16T16:40:17.165 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.699+0000 7fb2c44cf700 1 -- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fb2a803bfb0 msgr2=0x7fb2a803e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:17.165 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.699+0000 7fb2c44cf700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fb2a803bfb0 0x7fb2a803e390 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac003a10 tx=0x7fb2ac009290 comp rx=0 tx=0).stop 2023-11-16T16:40:17.165 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.699+0000 7fb2c44cf700 1 -- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2bc07bd80 msgr2=0x7fb2bc19baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:17.165 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.699+0000 7fb2c44cf700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2bc07bd80 0x7fb2bc19baf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8006af0 tx=0x7fb2b8011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:17.165 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.700+0000 7fb2c44cf700 1 -- 172.21.15.44:0/589611256 shutdown_connections 2023-11-16T16:40:17.165 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.700+0000 7fb2c44cf700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fb2a803bfb0 0x7fb2a803e390 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:17.165 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.700+0000 7fb2c44cf700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2bc07bd80 0x7fb2bc19baf0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:17.165 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.700+0000 7fb2c44cf700 1 --2- 172.21.15.44:0/589611256 >> [v2:172.21.15.44:6800/1852059921,v1:172.21.15.44:6801/1852059921] conn(0x7fb2a80387d0 0x7fb2a803ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:17.166 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.700+0000 7fb2c44cf700 1 -- 172.21.15.44:0/589611256 >> 172.21.15.44:0/589611256 conn(0x7fb2bc100fe0 msgr2=0x7fb2bc101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:17.166 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.700+0000 7fb2c44cf700 1 -- 172.21.15.44:0/589611256 shutdown_connections 2023-11-16T16:40:17.166 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.700+0000 7fb2c44cf700 1 -- 172.21.15.44:0/589611256 wait complete. 2023-11-16T16:40:17.166 INFO:teuthology.orchestra.run.smithi044.stdout:mgr epoch 5 is available 2023-11-16T16:40:17.166 INFO:teuthology.orchestra.run.smithi044.stdout:Setting orchestrator backend to cephadm... 2023-11-16T16:40:18.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:17 smithi044 ceph-mon[128750]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-16T16:40:18.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:17 smithi044 ceph-mon[128750]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-16T16:40:18.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:17 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' 2023-11-16T16:40:18.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:17 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' 2023-11-16T16:40:18.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:17 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:18.441 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.685+0000 7f66e280e700 1 Processor -- start 2023-11-16T16:40:18.441 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.685+0000 7f66e280e700 1 -- start start 2023-11-16T16:40:18.441 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.685+0000 7f66e280e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66dc07bdc0 0x7f66dc07a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:18.441 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.685+0000 7f66e280e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc07a800 con 0x7f66dc07bdc0 2023-11-16T16:40:18.441 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.685+0000 7f66dbfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66dc07bdc0 0x7f66dc07a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:18.441 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.685+0000 7f66dbfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66dc07bdc0 0x7f66dc07a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41892/0 (socket says 172.21.15.44:41892) 2023-11-16T16:40:18.441 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.685+0000 7f66dbfff700 1 -- 172.21.15.44:0/706106266 learned_addr learned my addr 172.21.15.44:0/706106266 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:18.441 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.686+0000 7f66dbfff700 1 -- 172.21.15.44:0/706106266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc07a940 con 0x7f66dc07bdc0 2023-11-16T16:40:18.442 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.686+0000 7f66dbfff700 1 --2- 172.21.15.44:0/706106266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66dc07bdc0 0x7f66dc07a2c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f66cc0098a0 tx=0x7f66cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d434ea87eb52cd79 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:18.442 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.686+0000 7f66daffd700 1 -- 172.21.15.44:0/706106266 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66cc004160 con 0x7f66dc07bdc0 2023-11-16T16:40:18.442 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.686+0000 7f66daffd700 1 -- 172.21.15.44:0/706106266 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66cc0042c0 con 0x7f66dc07bdc0 2023-11-16T16:40:18.442 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.686+0000 7f66daffd700 1 -- 172.21.15.44:0/706106266 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66cc011a00 con 0x7f66dc07bdc0 2023-11-16T16:40:18.442 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.687+0000 7f66e280e700 1 -- 172.21.15.44:0/706106266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66dc07bdc0 msgr2=0x7f66dc07a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:18.442 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.687+0000 7f66e280e700 1 --2- 172.21.15.44:0/706106266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66dc07bdc0 0x7f66dc07a2c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f66cc0098a0 tx=0x7f66cc009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:18.442 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.687+0000 7f66e280e700 1 -- 172.21.15.44:0/706106266 shutdown_connections 2023-11-16T16:40:18.442 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.687+0000 7f66e280e700 1 --2- 172.21.15.44:0/706106266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66dc07bdc0 0x7f66dc07a2c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:18.442 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.687+0000 7f66e280e700 1 -- 172.21.15.44:0/706106266 >> 172.21.15.44:0/706106266 conn(0x7f66dc100fe0 msgr2=0x7f66dc1033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:18.442 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.687+0000 7f66e280e700 1 -- 172.21.15.44:0/706106266 shutdown_connections 2023-11-16T16:40:18.442 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.687+0000 7f66e280e700 1 -- 172.21.15.44:0/706106266 wait complete. 2023-11-16T16:40:18.443 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.688+0000 7f66e280e700 1 Processor -- start 2023-11-16T16:40:18.443 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.688+0000 7f66e280e700 1 -- start start 2023-11-16T16:40:18.443 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.689+0000 7f66e280e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66dc07bdc0 0x7f66dc19bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:18.443 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.689+0000 7f66e280e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc19c060 con 0x7f66dc07bdc0 2023-11-16T16:40:18.443 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.689+0000 7f66dbfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66dc07bdc0 0x7f66dc19bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:18.443 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.689+0000 7f66dbfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66dc07bdc0 0x7f66dc19bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41900/0 (socket says 172.21.15.44:41900) 2023-11-16T16:40:18.443 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.689+0000 7f66dbfff700 1 -- 172.21.15.44:0/2247099683 learned_addr learned my addr 172.21.15.44:0/2247099683 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:18.443 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.689+0000 7f66dbfff700 1 -- 172.21.15.44:0/2247099683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66cc009550 con 0x7f66dc07bdc0 2023-11-16T16:40:18.443 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.689+0000 7f66dbfff700 1 --2- 172.21.15.44:0/2247099683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66dc07bdc0 0x7f66dc19bb20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f66cc006b20 tx=0x7f66cc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:18.443 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.690+0000 7f66d97fa700 1 -- 172.21.15.44:0/2247099683 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66cc003b70 con 0x7f66dc07bdc0 2023-11-16T16:40:18.443 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.690+0000 7f66d97fa700 1 -- 172.21.15.44:0/2247099683 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66cc003cd0 con 0x7f66dc07bdc0 2023-11-16T16:40:18.443 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.690+0000 7f66e280e700 1 -- 172.21.15.44:0/2247099683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66dc10be20 con 0x7f66dc07bdc0 2023-11-16T16:40:18.443 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.690+0000 7f66d97fa700 1 -- 172.21.15.44:0/2247099683 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66cc0247c0 con 0x7f66dc07bdc0 2023-11-16T16:40:18.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.690+0000 7f66e280e700 1 -- 172.21.15.44:0/2247099683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66dc10bc90 con 0x7f66dc07bdc0 2023-11-16T16:40:18.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.691+0000 7f66d97fa700 1 -- 172.21.15.44:0/2247099683 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7f66cc01ab40 con 0x7f66dc07bdc0 2023-11-16T16:40:18.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.691+0000 7f66e280e700 1 -- 172.21.15.44:0/2247099683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66dc10bfb0 con 0x7f66dc07bdc0 2023-11-16T16:40:18.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.691+0000 7f66d97fa700 1 --2- 172.21.15.44:0/2247099683 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f66c40386f0 0x7f66c403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:18.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.691+0000 7f66d97fa700 1 -- 172.21.15.44:0/2247099683 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f66cc04c740 con 0x7f66dc07bdc0 2023-11-16T16:40:18.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.692+0000 7f66db7fe700 1 --2- 172.21.15.44:0/2247099683 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f66c40386f0 0x7f66c403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:18.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.692+0000 7f66db7fe700 1 --2- 172.21.15.44:0/2247099683 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f66c40386f0 0x7f66c403aba0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f66d0006fd0 tx=0x7f66d0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:18.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.697+0000 7f66d97fa700 1 -- 172.21.15.44:0/2247099683 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f66cc01a950 con 0x7f66dc07bdc0 2023-11-16T16:40:18.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.919+0000 7f66e280e700 1 -- 172.21.15.44:0/2247099683 --> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f66dc102560 con 0x7f66c40386f0 2023-11-16T16:40:18.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.926+0000 7f66d97fa700 1 -- 172.21.15.44:0/2247099683 <== mgr.14118 v2:172.21.15.44:6800/4234382393 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f66dc102560 con 0x7f66c40386f0 2023-11-16T16:40:18.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.929+0000 7f66e280e700 1 -- 172.21.15.44:0/2247099683 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f66c40386f0 msgr2=0x7f66c403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:18.444 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.929+0000 7f66e280e700 1 --2- 172.21.15.44:0/2247099683 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f66c40386f0 0x7f66c403aba0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f66d0006fd0 tx=0x7f66d0006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:18.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.929+0000 7f66e280e700 1 -- 172.21.15.44:0/2247099683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66dc07bdc0 msgr2=0x7f66dc19bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:18.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.929+0000 7f66e280e700 1 --2- 172.21.15.44:0/2247099683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66dc07bdc0 0x7f66dc19bb20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f66cc006b20 tx=0x7f66cc011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:18.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.929+0000 7f66e280e700 1 -- 172.21.15.44:0/2247099683 shutdown_connections 2023-11-16T16:40:18.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.929+0000 7f66e280e700 1 --2- 172.21.15.44:0/2247099683 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f66c40386f0 0x7f66c403aba0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:18.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.929+0000 7f66e280e700 1 --2- 172.21.15.44:0/2247099683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66dc07bdc0 0x7f66dc19bb20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:18.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.929+0000 7f66e280e700 1 -- 172.21.15.44:0/2247099683 >> 172.21.15.44:0/2247099683 conn(0x7f66dc100fe0 msgr2=0x7f66dc101e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:18.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.929+0000 7f66e280e700 1 -- 172.21.15.44:0/2247099683 shutdown_connections 2023-11-16T16:40:18.445 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.929+0000 7f66e280e700 1 -- 172.21.15.44:0/2247099683 wait complete. 2023-11-16T16:40:19.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:18 smithi044 ceph-mon[128750]: [16/Nov/2023:16:40:17] ENGINE Bus STARTING 2023-11-16T16:40:19.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:18 smithi044 ceph-mon[128750]: [16/Nov/2023:16:40:17] ENGINE Serving on https://172.21.15.44:7150 2023-11-16T16:40:19.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:18 smithi044 ceph-mon[128750]: [16/Nov/2023:16:40:17] ENGINE Serving on http://172.21.15.44:8765 2023-11-16T16:40:19.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:18 smithi044 ceph-mon[128750]: [16/Nov/2023:16:40:17] ENGINE Bus STARTED 2023-11-16T16:40:19.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:18 smithi044 ceph-mon[128750]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:19.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:18 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' 2023-11-16T16:40:19.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:18 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:19.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:18 smithi044 ceph-mon[128750]: mgrmap e8: x(active, since 2s) 2023-11-16T16:40:19.624 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout value unchanged 2023-11-16T16:40:19.624 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.976+0000 7fed96852700 1 Processor -- start 2023-11-16T16:40:19.624 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.976+0000 7fed96852700 1 -- start start 2023-11-16T16:40:19.624 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.976+0000 7fed96852700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed9007bd80 0x7fed9007a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:19.624 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.976+0000 7fed96852700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9007a7c0 con 0x7fed9007bd80 2023-11-16T16:40:19.624 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.976+0000 7fed8ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed9007bd80 0x7fed9007a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:19.625 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.977+0000 7fed8ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed9007bd80 0x7fed9007a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41906/0 (socket says 172.21.15.44:41906) 2023-11-16T16:40:19.625 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.977+0000 7fed8ffff700 1 -- 172.21.15.44:0/2116108460 learned_addr learned my addr 172.21.15.44:0/2116108460 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:19.625 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.977+0000 7fed8ffff700 1 -- 172.21.15.44:0/2116108460 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9007a900 con 0x7fed9007bd80 2023-11-16T16:40:19.625 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.977+0000 7fed8ffff700 1 --2- 172.21.15.44:0/2116108460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed9007bd80 0x7fed9007a280 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fed800098a0 tx=0x7fed80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965848b122b9fbec server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:19.625 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.977+0000 7fed8effd700 1 -- 172.21.15.44:0/2116108460 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed80004160 con 0x7fed9007bd80 2023-11-16T16:40:19.625 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.977+0000 7fed8effd700 1 -- 172.21.15.44:0/2116108460 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed800042c0 con 0x7fed9007bd80 2023-11-16T16:40:19.625 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.978+0000 7fed8effd700 1 -- 172.21.15.44:0/2116108460 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed80011a00 con 0x7fed9007bd80 2023-11-16T16:40:19.625 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.978+0000 7fed96852700 1 -- 172.21.15.44:0/2116108460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed9007bd80 msgr2=0x7fed9007a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:19.625 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.978+0000 7fed96852700 1 --2- 172.21.15.44:0/2116108460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed9007bd80 0x7fed9007a280 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fed800098a0 tx=0x7fed80009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:19.625 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.978+0000 7fed96852700 1 -- 172.21.15.44:0/2116108460 shutdown_connections 2023-11-16T16:40:19.625 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.978+0000 7fed96852700 1 --2- 172.21.15.44:0/2116108460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed9007bd80 0x7fed9007a280 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:19.625 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.978+0000 7fed96852700 1 -- 172.21.15.44:0/2116108460 >> 172.21.15.44:0/2116108460 conn(0x7fed90100fe0 msgr2=0x7fed901033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:19.626 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.978+0000 7fed96852700 1 -- 172.21.15.44:0/2116108460 shutdown_connections 2023-11-16T16:40:19.626 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.978+0000 7fed96852700 1 -- 172.21.15.44:0/2116108460 wait complete. 2023-11-16T16:40:19.626 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.979+0000 7fed96852700 1 Processor -- start 2023-11-16T16:40:19.626 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.979+0000 7fed96852700 1 -- start start 2023-11-16T16:40:19.626 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.980+0000 7fed96852700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed9007bd80 0x7fed9019baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:19.626 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.980+0000 7fed96852700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9019c030 con 0x7fed9007bd80 2023-11-16T16:40:19.626 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.980+0000 7fed8ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed9007bd80 0x7fed9019baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:19.626 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.980+0000 7fed8ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed9007bd80 0x7fed9019baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41910/0 (socket says 172.21.15.44:41910) 2023-11-16T16:40:19.626 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.980+0000 7fed8ffff700 1 -- 172.21.15.44:0/2495123378 learned_addr learned my addr 172.21.15.44:0/2495123378 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:19.626 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.980+0000 7fed8ffff700 1 -- 172.21.15.44:0/2495123378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed80009550 con 0x7fed9007bd80 2023-11-16T16:40:19.626 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.981+0000 7fed8ffff700 1 --2- 172.21.15.44:0/2495123378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed9007bd80 0x7fed9019baf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fed80009fb0 tx=0x7fed80011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:19.627 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.981+0000 7fed8d7fa700 1 -- 172.21.15.44:0/2495123378 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed80003930 con 0x7fed9007bd80 2023-11-16T16:40:19.627 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.981+0000 7fed8d7fa700 1 -- 172.21.15.44:0/2495123378 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed80003a90 con 0x7fed9007bd80 2023-11-16T16:40:19.627 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.981+0000 7fed8d7fa700 1 -- 172.21.15.44:0/2495123378 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed800195e0 con 0x7fed9007bd80 2023-11-16T16:40:19.627 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.981+0000 7fed96852700 1 -- 172.21.15.44:0/2495123378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed9010be00 con 0x7fed9007bd80 2023-11-16T16:40:19.627 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.981+0000 7fed96852700 1 -- 172.21.15.44:0/2495123378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9010bc70 con 0x7fed9007bd80 2023-11-16T16:40:19.627 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.982+0000 7fed96852700 1 -- 172.21.15.44:0/2495123378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed90062120 con 0x7fed9007bd80 2023-11-16T16:40:19.627 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.982+0000 7fed8d7fa700 1 -- 172.21.15.44:0/2495123378 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fed8001a9d0 con 0x7fed9007bd80 2023-11-16T16:40:19.627 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.983+0000 7fed8d7fa700 1 --2- 172.21.15.44:0/2495123378 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fed780387d0 0x7fed7803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:19.627 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.983+0000 7fed8d7fa700 1 -- 172.21.15.44:0/2495123378 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fed8004c890 con 0x7fed9007bd80 2023-11-16T16:40:19.627 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.983+0000 7fed8f7fe700 1 --2- 172.21.15.44:0/2495123378 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fed780387d0 0x7fed7803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:19.627 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.984+0000 7fed8f7fe700 1 --2- 172.21.15.44:0/2495123378 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fed780387d0 0x7fed7803ac80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fed84006fd0 tx=0x7fed84006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:19.627 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.988+0000 7fed8d7fa700 1 -- 172.21.15.44:0/2495123378 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fed80023b40 con 0x7fed9007bd80 2023-11-16T16:40:19.627 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:19.197+0000 7fed96852700 1 -- 172.21.15.44:0/2495123378 --> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fed901024e0 con 0x7fed780387d0 2023-11-16T16:40:19.628 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:19.198+0000 7fed8d7fa700 1 -- 172.21.15.44:0/2495123378 <== mgr.14118 v2:172.21.15.44:6800/4234382393 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fed901024e0 con 0x7fed780387d0 2023-11-16T16:40:19.628 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:19.200+0000 7fed96852700 1 -- 172.21.15.44:0/2495123378 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fed780387d0 msgr2=0x7fed7803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:19.628 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:19.200+0000 7fed96852700 1 --2- 172.21.15.44:0/2495123378 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fed780387d0 0x7fed7803ac80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fed84006fd0 tx=0x7fed84006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:19.628 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:19.200+0000 7fed96852700 1 -- 172.21.15.44:0/2495123378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed9007bd80 msgr2=0x7fed9019baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:19.628 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:19.200+0000 7fed96852700 1 --2- 172.21.15.44:0/2495123378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed9007bd80 0x7fed9019baf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fed80009fb0 tx=0x7fed80011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:19.628 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:19.200+0000 7fed96852700 1 -- 172.21.15.44:0/2495123378 shutdown_connections 2023-11-16T16:40:19.628 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:19.200+0000 7fed96852700 1 --2- 172.21.15.44:0/2495123378 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fed780387d0 0x7fed7803ac80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:19.628 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:19.200+0000 7fed96852700 1 --2- 172.21.15.44:0/2495123378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed9007bd80 0x7fed9019baf0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:19.628 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:19.200+0000 7fed96852700 1 -- 172.21.15.44:0/2495123378 >> 172.21.15.44:0/2495123378 conn(0x7fed90100fe0 msgr2=0x7fed90101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:19.628 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:19.200+0000 7fed96852700 1 -- 172.21.15.44:0/2495123378 shutdown_connections 2023-11-16T16:40:19.628 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:19.200+0000 7fed96852700 1 -- 172.21.15.44:0/2495123378 wait complete. 2023-11-16T16:40:19.628 INFO:teuthology.orchestra.run.smithi044.stdout:Generating ssh key... 2023-11-16T16:40:20.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-mon[128750]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:20.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: Generating public/private rsa key pair. 2023-11-16T16:40:20.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: Your identification has been saved in /tmp/tmph5hud_gq/key. 2023-11-16T16:40:20.869 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: Your public key has been saved in /tmp/tmph5hud_gq/key.pub. 2023-11-16T16:40:20.869 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: The key fingerprint is: 2023-11-16T16:40:20.869 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: SHA256:soKsFZpex50Pnv/1cyC+r/FQCAAVI7b0ER8JWiE4PHs ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:40:20.869 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: The key's randomart image is: 2023-11-16T16:40:20.869 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: +---[RSA 3072]----+ 2023-11-16T16:40:20.869 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: | . .*o@=.. | 2023-11-16T16:40:20.869 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: | =o B =.. | 2023-11-16T16:40:20.869 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: | +o . o | 2023-11-16T16:40:20.869 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: | . E . . | 2023-11-16T16:40:20.869 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: | . .. S . . | 2023-11-16T16:40:20.869 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: | + o. .o. ... | 2023-11-16T16:40:20.870 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: |o +..o.+ .oo . | 2023-11-16T16:40:20.870 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: |.o. ... + o+...| 2023-11-16T16:40:20.870 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: |.. o.o..o+o.o| 2023-11-16T16:40:20.870 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: +----[SHA256]-----+ 2023-11-16T16:40:21.009 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.133+0000 7fd556b85700 1 Processor -- start 2023-11-16T16:40:21.009 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.133+0000 7fd556b85700 1 -- start start 2023-11-16T16:40:21.009 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.134+0000 7fd556b85700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd550105950 0x7fd550105d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:21.010 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.134+0000 7fd556b85700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5501062a0 con 0x7fd550105950 2023-11-16T16:40:21.010 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.134+0000 7fd554921700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd550105950 0x7fd550105d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:21.010 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.134+0000 7fd554921700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd550105950 0x7fd550105d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41926/0 (socket says 172.21.15.44:41926) 2023-11-16T16:40:21.010 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.134+0000 7fd554921700 1 -- 172.21.15.44:0/1167930750 learned_addr learned my addr 172.21.15.44:0/1167930750 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:21.010 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.134+0000 7fd554921700 1 -- 172.21.15.44:0/1167930750 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5501063e0 con 0x7fd550105950 2023-11-16T16:40:21.010 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.135+0000 7fd554921700 1 --2- 172.21.15.44:0/1167930750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd550105950 0x7fd550105d60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd5440098a0 tx=0x7fd544009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=101960baa3d5b793 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:21.010 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.135+0000 7fd54f7fe700 1 -- 172.21.15.44:0/1167930750 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd544004160 con 0x7fd550105950 2023-11-16T16:40:21.010 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.135+0000 7fd54f7fe700 1 -- 172.21.15.44:0/1167930750 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5440042c0 con 0x7fd550105950 2023-11-16T16:40:21.010 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.135+0000 7fd54f7fe700 1 -- 172.21.15.44:0/1167930750 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd544011a00 con 0x7fd550105950 2023-11-16T16:40:21.010 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.135+0000 7fd556b85700 1 -- 172.21.15.44:0/1167930750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd550105950 msgr2=0x7fd550105d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:21.011 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.135+0000 7fd556b85700 1 --2- 172.21.15.44:0/1167930750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd550105950 0x7fd550105d60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd5440098a0 tx=0x7fd544009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:21.011 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.136+0000 7fd556b85700 1 -- 172.21.15.44:0/1167930750 shutdown_connections 2023-11-16T16:40:21.011 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.136+0000 7fd556b85700 1 --2- 172.21.15.44:0/1167930750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd550105950 0x7fd550105d60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:21.011 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.136+0000 7fd556b85700 1 -- 172.21.15.44:0/1167930750 >> 172.21.15.44:0/1167930750 conn(0x7fd550100fe0 msgr2=0x7fd5501033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:21.011 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.136+0000 7fd556b85700 1 -- 172.21.15.44:0/1167930750 shutdown_connections 2023-11-16T16:40:21.011 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.136+0000 7fd556b85700 1 -- 172.21.15.44:0/1167930750 wait complete. 2023-11-16T16:40:21.011 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.137+0000 7fd556b85700 1 Processor -- start 2023-11-16T16:40:21.011 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.137+0000 7fd556b85700 1 -- start start 2023-11-16T16:40:21.011 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.137+0000 7fd556b85700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd550105950 0x7fd55019baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:21.011 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.137+0000 7fd556b85700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55019bfe0 con 0x7fd550105950 2023-11-16T16:40:21.011 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.138+0000 7fd554921700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd550105950 0x7fd55019baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:21.011 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.138+0000 7fd554921700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd550105950 0x7fd55019baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41936/0 (socket says 172.21.15.44:41936) 2023-11-16T16:40:21.012 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.138+0000 7fd554921700 1 -- 172.21.15.44:0/2408628131 learned_addr learned my addr 172.21.15.44:0/2408628131 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:21.012 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.138+0000 7fd554921700 1 -- 172.21.15.44:0/2408628131 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd544009550 con 0x7fd550105950 2023-11-16T16:40:21.012 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.138+0000 7fd554921700 1 --2- 172.21.15.44:0/2408628131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd550105950 0x7fd55019baa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd544006af0 tx=0x7fd544011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:21.012 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.138+0000 7fd54dffb700 1 -- 172.21.15.44:0/2408628131 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd544003a40 con 0x7fd550105950 2023-11-16T16:40:21.012 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.138+0000 7fd54dffb700 1 -- 172.21.15.44:0/2408628131 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd544003ba0 con 0x7fd550105950 2023-11-16T16:40:21.012 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.138+0000 7fd556b85700 1 -- 172.21.15.44:0/2408628131 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5501079a0 con 0x7fd550105950 2023-11-16T16:40:21.012 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.138+0000 7fd54dffb700 1 -- 172.21.15.44:0/2408628131 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd54401a6e0 con 0x7fd550105950 2023-11-16T16:40:21.012 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.139+0000 7fd556b85700 1 -- 172.21.15.44:0/2408628131 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd550107810 con 0x7fd550105950 2023-11-16T16:40:21.012 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.140+0000 7fd54dffb700 1 -- 172.21.15.44:0/2408628131 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fd544024a60 con 0x7fd550105950 2023-11-16T16:40:21.012 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.140+0000 7fd556b85700 1 -- 172.21.15.44:0/2408628131 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd550062120 con 0x7fd550105950 2023-11-16T16:40:21.012 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.140+0000 7fd54dffb700 1 --2- 172.21.15.44:0/2408628131 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fd540038820 0x7fd54003acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:21.013 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.140+0000 7fd54dffb700 1 -- 172.21.15.44:0/2408628131 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd54404c7c0 con 0x7fd550105950 2023-11-16T16:40:21.013 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.140+0000 7fd54ffff700 1 --2- 172.21.15.44:0/2408628131 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fd540038820 0x7fd54003acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:21.013 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.141+0000 7fd54ffff700 1 --2- 172.21.15.44:0/2408628131 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fd540038820 0x7fd54003acd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd53c006fd0 tx=0x7fd53c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:21.013 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.145+0000 7fd54dffb700 1 -- 172.21.15.44:0/2408628131 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd544024430 con 0x7fd550105950 2023-11-16T16:40:21.013 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.354+0000 7fd556b85700 1 -- 172.21.15.44:0/2408628131 --> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fd550109df0 con 0x7fd540038820 2023-11-16T16:40:21.013 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.515+0000 7fd54dffb700 1 -- 172.21.15.44:0/2408628131 <== mgr.14118 v2:172.21.15.44:6800/4234382393 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd53c009e20 con 0x7fd540038820 2023-11-16T16:40:21.013 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.517+0000 7fd556b85700 1 -- 172.21.15.44:0/2408628131 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fd540038820 msgr2=0x7fd54003acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:21.013 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.517+0000 7fd556b85700 1 --2- 172.21.15.44:0/2408628131 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fd540038820 0x7fd54003acd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd53c006fd0 tx=0x7fd53c006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:21.013 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.517+0000 7fd556b85700 1 -- 172.21.15.44:0/2408628131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd550105950 msgr2=0x7fd55019baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:21.013 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.517+0000 7fd556b85700 1 --2- 172.21.15.44:0/2408628131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd550105950 0x7fd55019baa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd544006af0 tx=0x7fd544011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:21.013 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.518+0000 7fd556b85700 1 -- 172.21.15.44:0/2408628131 shutdown_connections 2023-11-16T16:40:21.013 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.518+0000 7fd556b85700 1 --2- 172.21.15.44:0/2408628131 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fd540038820 0x7fd54003acd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:21.014 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.518+0000 7fd556b85700 1 --2- 172.21.15.44:0/2408628131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd550105950 0x7fd55019baa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:21.014 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.518+0000 7fd556b85700 1 -- 172.21.15.44:0/2408628131 >> 172.21.15.44:0/2408628131 conn(0x7fd550100fe0 msgr2=0x7fd550101cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:21.014 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.518+0000 7fd556b85700 1 -- 172.21.15.44:0/2408628131 shutdown_connections 2023-11-16T16:40:21.014 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.518+0000 7fd556b85700 1 -- 172.21.15.44:0/2408628131 wait complete. 2023-11-16T16:40:21.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:21 smithi044 ceph-mon[128750]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:21.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:21 smithi044 ceph-mon[128750]: Generating ssh key... 2023-11-16T16:40:21.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:21 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' 2023-11-16T16:40:21.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:21 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' 2023-11-16T16:40:22.193 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWqZCN0b0Ts1go+60x2zF0Bszy8UH/qk+2xX1G31+FkXNGOxYkyMfJ+INT2FXjfE162RKn6x1J4nGlRR5VUaqhq47tmvZ/1rUtvsY7gYXJjdSRjRsubyEsv2cQxERbqN9GkPiO0SaHXLvlN8pttMa/44G8k6E7IapkNJrYcA+AvlsToTa8RF85rwHU0U3aJFcY5W1fuan06nE1rxVPCuk/JBOyu7NzmQ+rUGYqk6Yt2mrxXsmtqgSevYBTn/bXIpmYLls5+8DSgYceDvcUjZiZ7s2UKSYNyh8EV7PlbZpf/ofBKSmXCPlH+1BXyHGG9K5T3Ho860BjWZW9KY/HJqHn7cd50LpZNv6yBT5yMomrXvOF4qEVibEQIhgXxr3yIghGrep93FX8znyzirsSoHQGm5/KcPY7BONLltPO42gbqDWqBpyffO2XPN8WPuMkfiEwOilzu05aVFyKuO3AMnDE3qx7Gl9FFrcFUoFN6GOu9Oz64O1+3CoVdUl6ze7EFtM= ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:40:22.193 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.549+0000 7feba1c0a700 1 Processor -- start 2023-11-16T16:40:22.193 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.549+0000 7feba1c0a700 1 -- start start 2023-11-16T16:40:22.193 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.550+0000 7feba1c0a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb9c105950 0x7feb9c105d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:22.193 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.550+0000 7feba1c0a700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c1062a0 con 0x7feb9c105950 2023-11-16T16:40:22.193 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.550+0000 7feb9b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb9c105950 0x7feb9c105d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:22.193 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.550+0000 7feb9b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb9c105950 0x7feb9c105d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41948/0 (socket says 172.21.15.44:41948) 2023-11-16T16:40:22.194 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.550+0000 7feb9b7fe700 1 -- 172.21.15.44:0/2418068868 learned_addr learned my addr 172.21.15.44:0/2418068868 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:22.194 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.551+0000 7feb9b7fe700 1 -- 172.21.15.44:0/2418068868 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9c1063e0 con 0x7feb9c105950 2023-11-16T16:40:22.194 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.551+0000 7feb9b7fe700 1 --2- 172.21.15.44:0/2418068868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb9c105950 0x7feb9c105d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0098a0 tx=0x7feb8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee49232ace26f1fd server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:22.194 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.551+0000 7feb9a7fc700 1 -- 172.21.15.44:0/2418068868 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb8c004160 con 0x7feb9c105950 2023-11-16T16:40:22.194 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.551+0000 7feb9a7fc700 1 -- 172.21.15.44:0/2418068868 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb8c0042c0 con 0x7feb9c105950 2023-11-16T16:40:22.194 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.551+0000 7feb9a7fc700 1 -- 172.21.15.44:0/2418068868 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb8c011a00 con 0x7feb9c105950 2023-11-16T16:40:22.194 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.552+0000 7feba1c0a700 1 -- 172.21.15.44:0/2418068868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb9c105950 msgr2=0x7feb9c105d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:22.194 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.552+0000 7feba1c0a700 1 --2- 172.21.15.44:0/2418068868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb9c105950 0x7feb9c105d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0098a0 tx=0x7feb8c009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:22.194 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.552+0000 7feba1c0a700 1 -- 172.21.15.44:0/2418068868 shutdown_connections 2023-11-16T16:40:22.194 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.552+0000 7feba1c0a700 1 --2- 172.21.15.44:0/2418068868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb9c105950 0x7feb9c105d60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:22.194 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.552+0000 7feba1c0a700 1 -- 172.21.15.44:0/2418068868 >> 172.21.15.44:0/2418068868 conn(0x7feb9c100fe0 msgr2=0x7feb9c1033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:22.195 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.552+0000 7feba1c0a700 1 -- 172.21.15.44:0/2418068868 shutdown_connections 2023-11-16T16:40:22.195 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.552+0000 7feba1c0a700 1 -- 172.21.15.44:0/2418068868 wait complete. 2023-11-16T16:40:22.195 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.553+0000 7feba1c0a700 1 Processor -- start 2023-11-16T16:40:22.195 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.553+0000 7feba1c0a700 1 -- start start 2023-11-16T16:40:22.195 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.553+0000 7feba1c0a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb9c105950 0x7feb9c07bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:22.195 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.553+0000 7feba1c0a700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c07c150 con 0x7feb9c105950 2023-11-16T16:40:22.195 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.554+0000 7feb9b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb9c105950 0x7feb9c07bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:22.195 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.554+0000 7feb9b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb9c105950 0x7feb9c07bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41950/0 (socket says 172.21.15.44:41950) 2023-11-16T16:40:22.195 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.554+0000 7feb9b7fe700 1 -- 172.21.15.44:0/1031066716 learned_addr learned my addr 172.21.15.44:0/1031066716 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:22.195 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.554+0000 7feb9b7fe700 1 -- 172.21.15.44:0/1031066716 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c009550 con 0x7feb9c105950 2023-11-16T16:40:22.195 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.554+0000 7feb9b7fe700 1 --2- 172.21.15.44:0/1031066716 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb9c105950 0x7feb9c07bc10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feb8c006af0 tx=0x7feb8c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:22.195 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.554+0000 7feb98ff9700 1 -- 172.21.15.44:0/1031066716 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb8c003a40 con 0x7feb9c105950 2023-11-16T16:40:22.196 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.554+0000 7feb98ff9700 1 -- 172.21.15.44:0/1031066716 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb8c003ba0 con 0x7feb9c105950 2023-11-16T16:40:22.196 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.554+0000 7feb98ff9700 1 -- 172.21.15.44:0/1031066716 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb8c01a6e0 con 0x7feb9c105950 2023-11-16T16:40:22.196 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.554+0000 7feba1c0a700 1 -- 172.21.15.44:0/1031066716 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb9c1079a0 con 0x7feb9c105950 2023-11-16T16:40:22.196 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.555+0000 7feba1c0a700 1 -- 172.21.15.44:0/1031066716 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb9c107810 con 0x7feb9c105950 2023-11-16T16:40:22.196 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.556+0000 7feba1c0a700 1 -- 172.21.15.44:0/1031066716 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb9c062120 con 0x7feb9c105950 2023-11-16T16:40:22.196 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.556+0000 7feb98ff9700 1 -- 172.21.15.44:0/1031066716 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7feb8c024a60 con 0x7feb9c105950 2023-11-16T16:40:22.196 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.556+0000 7feb98ff9700 1 --2- 172.21.15.44:0/1031066716 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7feb840387d0 0x7feb8403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:22.196 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.556+0000 7feb98ff9700 1 -- 172.21.15.44:0/1031066716 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7feb8c04c830 con 0x7feb9c105950 2023-11-16T16:40:22.196 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.556+0000 7feb9affd700 1 --2- 172.21.15.44:0/1031066716 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7feb840387d0 0x7feb8403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:22.196 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.557+0000 7feb9affd700 1 --2- 172.21.15.44:0/1031066716 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7feb840387d0 0x7feb8403ac80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7feb90006fd0 tx=0x7feb90006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:22.196 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.561+0000 7feb98ff9700 1 -- 172.21.15.44:0/1031066716 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7feb8c024430 con 0x7feb9c105950 2023-11-16T16:40:22.196 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.769+0000 7feba1c0a700 1 -- 172.21.15.44:0/1031066716 --> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7feb9c07ae90 con 0x7feb840387d0 2023-11-16T16:40:22.197 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.770+0000 7feb98ff9700 1 -- 172.21.15.44:0/1031066716 <== mgr.14118 v2:172.21.15.44:6800/4234382393 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7feb9c07ae90 con 0x7feb840387d0 2023-11-16T16:40:22.197 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.772+0000 7feba1c0a700 1 -- 172.21.15.44:0/1031066716 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7feb840387d0 msgr2=0x7feb8403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:22.197 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.772+0000 7feba1c0a700 1 --2- 172.21.15.44:0/1031066716 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7feb840387d0 0x7feb8403ac80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7feb90006fd0 tx=0x7feb90006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:22.197 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.772+0000 7feba1c0a700 1 -- 172.21.15.44:0/1031066716 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb9c105950 msgr2=0x7feb9c07bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:22.197 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.772+0000 7feba1c0a700 1 --2- 172.21.15.44:0/1031066716 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb9c105950 0x7feb9c07bc10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feb8c006af0 tx=0x7feb8c011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:22.197 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.772+0000 7feba1c0a700 1 -- 172.21.15.44:0/1031066716 shutdown_connections 2023-11-16T16:40:22.197 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.772+0000 7feba1c0a700 1 --2- 172.21.15.44:0/1031066716 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7feb840387d0 0x7feb8403ac80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:22.197 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.772+0000 7feba1c0a700 1 --2- 172.21.15.44:0/1031066716 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb9c105950 0x7feb9c07bc10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:22.197 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.773+0000 7feba1c0a700 1 -- 172.21.15.44:0/1031066716 >> 172.21.15.44:0/1031066716 conn(0x7feb9c100fe0 msgr2=0x7feb9c101c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:22.197 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.773+0000 7feba1c0a700 1 -- 172.21.15.44:0/1031066716 shutdown_connections 2023-11-16T16:40:22.197 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.773+0000 7feba1c0a700 1 -- 172.21.15.44:0/1031066716 wait complete. 2023-11-16T16:40:22.197 INFO:teuthology.orchestra.run.smithi044.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-16T16:40:22.198 INFO:teuthology.orchestra.run.smithi044.stdout:Adding key to root@localhost authorized_keys... 2023-11-16T16:40:22.198 INFO:teuthology.orchestra.run.smithi044.stdout:Adding host smithi044... 2023-11-16T16:40:22.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:22 smithi044 ceph-mon[128750]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:23.654 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:23 smithi044 ceph-mon[128750]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi044", "addr": "172.21.15.44", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:25.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:24 smithi044 ceph-mon[128750]: Deploying cephadm binary to smithi044 2023-11-16T16:40:26.384 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Added host 'smithi044' with addr '172.21.15.44' 2023-11-16T16:40:26.384 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.710+0000 7f298e56e700 1 Processor -- start 2023-11-16T16:40:26.384 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.711+0000 7f298e56e700 1 -- start start 2023-11-16T16:40:26.384 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.711+0000 7f298e56e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f298807bd80 0x7f298807a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:26.384 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.711+0000 7f298e56e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298807a7c0 con 0x7f298807bd80 2023-11-16T16:40:26.384 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.711+0000 7f2987fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f298807bd80 0x7f298807a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:26.384 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.711+0000 7f2987fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f298807bd80 0x7f298807a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41952/0 (socket says 172.21.15.44:41952) 2023-11-16T16:40:26.385 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.711+0000 7f2987fff700 1 -- 172.21.15.44:0/2561600132 learned_addr learned my addr 172.21.15.44:0/2561600132 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:26.385 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.712+0000 7f2987fff700 1 -- 172.21.15.44:0/2561600132 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298807a900 con 0x7f298807bd80 2023-11-16T16:40:26.385 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.712+0000 7f2987fff700 1 --2- 172.21.15.44:0/2561600132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f298807bd80 0x7f298807a280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f29780098a0 tx=0x7f2978009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af047a5f61ecfed2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:26.385 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.712+0000 7f2986ffd700 1 -- 172.21.15.44:0/2561600132 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2978004160 con 0x7f298807bd80 2023-11-16T16:40:26.385 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.712+0000 7f2986ffd700 1 -- 172.21.15.44:0/2561600132 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29780042c0 con 0x7f298807bd80 2023-11-16T16:40:26.385 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.712+0000 7f2986ffd700 1 -- 172.21.15.44:0/2561600132 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2978011a00 con 0x7f298807bd80 2023-11-16T16:40:26.385 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.713+0000 7f298e56e700 1 -- 172.21.15.44:0/2561600132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f298807bd80 msgr2=0x7f298807a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:26.385 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.713+0000 7f298e56e700 1 --2- 172.21.15.44:0/2561600132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f298807bd80 0x7f298807a280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f29780098a0 tx=0x7f2978009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:26.385 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.713+0000 7f298e56e700 1 -- 172.21.15.44:0/2561600132 shutdown_connections 2023-11-16T16:40:26.385 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.713+0000 7f298e56e700 1 --2- 172.21.15.44:0/2561600132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f298807bd80 0x7f298807a280 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:26.385 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.713+0000 7f298e56e700 1 -- 172.21.15.44:0/2561600132 >> 172.21.15.44:0/2561600132 conn(0x7f2988100fe0 msgr2=0x7f29881033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:26.385 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.713+0000 7f298e56e700 1 -- 172.21.15.44:0/2561600132 shutdown_connections 2023-11-16T16:40:26.386 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.713+0000 7f298e56e700 1 -- 172.21.15.44:0/2561600132 wait complete. 2023-11-16T16:40:26.386 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.714+0000 7f298e56e700 1 Processor -- start 2023-11-16T16:40:26.386 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.715+0000 7f298e56e700 1 -- start start 2023-11-16T16:40:26.386 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.715+0000 7f298e56e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f298807bd80 0x7f298819baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:26.386 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.715+0000 7f298e56e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298819c030 con 0x7f298807bd80 2023-11-16T16:40:26.386 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.715+0000 7f2987fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f298807bd80 0x7f298819baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:26.386 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.715+0000 7f2987fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f298807bd80 0x7f298819baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41958/0 (socket says 172.21.15.44:41958) 2023-11-16T16:40:26.386 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.715+0000 7f2987fff700 1 -- 172.21.15.44:0/1210083423 learned_addr learned my addr 172.21.15.44:0/1210083423 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:26.386 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.716+0000 7f2987fff700 1 -- 172.21.15.44:0/1210083423 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2978009550 con 0x7f298807bd80 2023-11-16T16:40:26.386 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.716+0000 7f2987fff700 1 --2- 172.21.15.44:0/1210083423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f298807bd80 0x7f298819baf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2978006af0 tx=0x7f2978011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:26.386 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.716+0000 7f29857fa700 1 -- 172.21.15.44:0/1210083423 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2978003a40 con 0x7f298807bd80 2023-11-16T16:40:26.387 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.716+0000 7f29857fa700 1 -- 172.21.15.44:0/1210083423 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2978003ba0 con 0x7f298807bd80 2023-11-16T16:40:26.387 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.716+0000 7f298e56e700 1 -- 172.21.15.44:0/1210083423 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298810be00 con 0x7f298807bd80 2023-11-16T16:40:26.387 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.716+0000 7f29857fa700 1 -- 172.21.15.44:0/1210083423 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f297801a700 con 0x7f298807bd80 2023-11-16T16:40:26.387 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.716+0000 7f298e56e700 1 -- 172.21.15.44:0/1210083423 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f298810bc70 con 0x7f298807bd80 2023-11-16T16:40:26.387 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.718+0000 7f298e56e700 1 -- 172.21.15.44:0/1210083423 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2988062120 con 0x7f298807bd80 2023-11-16T16:40:26.387 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.718+0000 7f29857fa700 1 -- 172.21.15.44:0/1210083423 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f2978024ad0 con 0x7f298807bd80 2023-11-16T16:40:26.387 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.718+0000 7f29857fa700 1 --2- 172.21.15.44:0/1210083423 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f2970038820 0x7f297003acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:26.387 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.718+0000 7f29857fa700 1 -- 172.21.15.44:0/1210083423 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f297804caf0 con 0x7f298807bd80 2023-11-16T16:40:26.387 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.718+0000 7f29877fe700 1 --2- 172.21.15.44:0/1210083423 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f2970038820 0x7f297003acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:26.387 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.719+0000 7f29877fe700 1 --2- 172.21.15.44:0/1210083423 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f2970038820 0x7f297003acd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f297c006fd0 tx=0x7f297c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:26.387 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.724+0000 7f29857fa700 1 -- 172.21.15.44:0/1210083423 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2978024430 con 0x7f298807bd80 2023-11-16T16:40:26.387 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:22.934+0000 7f298e56e700 1 -- 172.21.15.44:0/1210083423 --> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi044", "addr": "172.21.15.44", "target": ["mon-mgr", ""]}) v1 -- 0x7f29881024e0 con 0x7f2970038820 2023-11-16T16:40:26.388 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:25.980+0000 7f29857fa700 1 -- 172.21.15.44:0/1210083423 <== mgr.14118 v2:172.21.15.44:6800/4234382393 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f29881024e0 con 0x7f2970038820 2023-11-16T16:40:26.388 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:25.983+0000 7f298e56e700 1 -- 172.21.15.44:0/1210083423 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f2970038820 msgr2=0x7f297003acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:26.388 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:25.983+0000 7f298e56e700 1 --2- 172.21.15.44:0/1210083423 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f2970038820 0x7f297003acd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f297c006fd0 tx=0x7f297c006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:26.388 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:25.983+0000 7f298e56e700 1 -- 172.21.15.44:0/1210083423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f298807bd80 msgr2=0x7f298819baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:26.388 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:25.983+0000 7f298e56e700 1 --2- 172.21.15.44:0/1210083423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f298807bd80 0x7f298819baf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2978006af0 tx=0x7f2978011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:26.388 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:25.984+0000 7f298e56e700 1 -- 172.21.15.44:0/1210083423 shutdown_connections 2023-11-16T16:40:26.388 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:25.984+0000 7f298e56e700 1 --2- 172.21.15.44:0/1210083423 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f2970038820 0x7f297003acd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:26.388 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:25.984+0000 7f298e56e700 1 --2- 172.21.15.44:0/1210083423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f298807bd80 0x7f298819baf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:26.388 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:25.984+0000 7f298e56e700 1 -- 172.21.15.44:0/1210083423 >> 172.21.15.44:0/1210083423 conn(0x7f2988100fe0 msgr2=0x7f2988101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:26.388 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:25.984+0000 7f298e56e700 1 -- 172.21.15.44:0/1210083423 shutdown_connections 2023-11-16T16:40:26.388 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:25.984+0000 7f298e56e700 1 -- 172.21.15.44:0/1210083423 wait complete. 2023-11-16T16:40:26.388 INFO:teuthology.orchestra.run.smithi044.stdout:Deploying unmanaged mon service... 2023-11-16T16:40:27.234 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:26 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' 2023-11-16T16:40:27.235 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:26 smithi044 ceph-mon[128750]: Added host smithi044 2023-11-16T16:40:27.235 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:26 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:27.772 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-11-16T16:40:27.778 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.968+0000 7fa331920700 1 Processor -- start 2023-11-16T16:40:27.779 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.968+0000 7fa331920700 1 -- start start 2023-11-16T16:40:27.779 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.968+0000 7fa331920700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa32c105950 0x7fa32c105d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:27.779 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.968+0000 7fa331920700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c1062a0 con 0x7fa32c105950 2023-11-16T16:40:27.779 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.968+0000 7fa32affd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa32c105950 0x7fa32c105d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:27.779 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.969+0000 7fa32affd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa32c105950 0x7fa32c105d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50678/0 (socket says 172.21.15.44:50678) 2023-11-16T16:40:27.779 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.969+0000 7fa32affd700 1 -- 172.21.15.44:0/272513955 learned_addr learned my addr 172.21.15.44:0/272513955 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:27.779 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.969+0000 7fa32affd700 1 -- 172.21.15.44:0/272513955 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32c1063e0 con 0x7fa32c105950 2023-11-16T16:40:27.779 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.969+0000 7fa32affd700 1 --2- 172.21.15.44:0/272513955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa32c105950 0x7fa32c105d60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa31c0098a0 tx=0x7fa31c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1db396d37c4b7d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:27.780 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.969+0000 7fa329ffb700 1 -- 172.21.15.44:0/272513955 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa31c004160 con 0x7fa32c105950 2023-11-16T16:40:27.780 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.969+0000 7fa329ffb700 1 -- 172.21.15.44:0/272513955 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa31c0042c0 con 0x7fa32c105950 2023-11-16T16:40:27.780 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.969+0000 7fa329ffb700 1 -- 172.21.15.44:0/272513955 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa31c011a00 con 0x7fa32c105950 2023-11-16T16:40:27.780 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.970+0000 7fa331920700 1 -- 172.21.15.44:0/272513955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa32c105950 msgr2=0x7fa32c105d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:27.780 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.970+0000 7fa331920700 1 --2- 172.21.15.44:0/272513955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa32c105950 0x7fa32c105d60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa31c0098a0 tx=0x7fa31c009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:27.780 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.970+0000 7fa331920700 1 -- 172.21.15.44:0/272513955 shutdown_connections 2023-11-16T16:40:27.780 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.970+0000 7fa331920700 1 --2- 172.21.15.44:0/272513955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa32c105950 0x7fa32c105d60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:27.780 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.970+0000 7fa331920700 1 -- 172.21.15.44:0/272513955 >> 172.21.15.44:0/272513955 conn(0x7fa32c100fe0 msgr2=0x7fa32c1033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:27.780 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.970+0000 7fa331920700 1 -- 172.21.15.44:0/272513955 shutdown_connections 2023-11-16T16:40:27.781 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.970+0000 7fa331920700 1 -- 172.21.15.44:0/272513955 wait complete. 2023-11-16T16:40:27.781 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.971+0000 7fa331920700 1 Processor -- start 2023-11-16T16:40:27.781 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.971+0000 7fa331920700 1 -- start start 2023-11-16T16:40:27.781 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.972+0000 7fa331920700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa32c105950 0x7fa32c19baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:27.781 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.972+0000 7fa331920700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c19bfe0 con 0x7fa32c105950 2023-11-16T16:40:27.781 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.972+0000 7fa32affd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa32c105950 0x7fa32c19baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:27.781 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.972+0000 7fa32affd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa32c105950 0x7fa32c19baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50682/0 (socket says 172.21.15.44:50682) 2023-11-16T16:40:27.781 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.972+0000 7fa32affd700 1 -- 172.21.15.44:0/2179293182 learned_addr learned my addr 172.21.15.44:0/2179293182 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:27.781 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.972+0000 7fa32affd700 1 -- 172.21.15.44:0/2179293182 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa31c009550 con 0x7fa32c105950 2023-11-16T16:40:27.781 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.973+0000 7fa32affd700 1 --2- 172.21.15.44:0/2179293182 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa32c105950 0x7fa32c19baa0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa31c009fb0 tx=0x7fa31c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:27.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.973+0000 7fa33091e700 1 -- 172.21.15.44:0/2179293182 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa31c003930 con 0x7fa32c105950 2023-11-16T16:40:27.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.973+0000 7fa33091e700 1 -- 172.21.15.44:0/2179293182 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa31c003a90 con 0x7fa32c105950 2023-11-16T16:40:27.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.973+0000 7fa331920700 1 -- 172.21.15.44:0/2179293182 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32c1079a0 con 0x7fa32c105950 2023-11-16T16:40:27.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.973+0000 7fa33091e700 1 -- 172.21.15.44:0/2179293182 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa31c019610 con 0x7fa32c105950 2023-11-16T16:40:27.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.973+0000 7fa331920700 1 -- 172.21.15.44:0/2179293182 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa32c107810 con 0x7fa32c105950 2023-11-16T16:40:27.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.974+0000 7fa331920700 1 -- 172.21.15.44:0/2179293182 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa32c062120 con 0x7fa32c105950 2023-11-16T16:40:27.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.974+0000 7fa33091e700 1 -- 172.21.15.44:0/2179293182 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fa31c01a9d0 con 0x7fa32c105950 2023-11-16T16:40:27.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.974+0000 7fa33091e700 1 --2- 172.21.15.44:0/2179293182 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fa3140387d0 0x7fa31403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:27.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.975+0000 7fa33091e700 1 -- 172.21.15.44:0/2179293182 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa31c04c860 con 0x7fa32c105950 2023-11-16T16:40:27.783 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.975+0000 7fa32a7fc700 1 --2- 172.21.15.44:0/2179293182 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fa3140387d0 0x7fa31403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:27.783 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.975+0000 7fa32a7fc700 1 --2- 172.21.15.44:0/2179293182 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fa3140387d0 0x7fa31403ac80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa320006fd0 tx=0x7fa320006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:27.783 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:26.980+0000 7fa33091e700 1 -- 172.21.15.44:0/2179293182 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa31c003c00 con 0x7fa32c105950 2023-11-16T16:40:27.783 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.201+0000 7fa331920700 1 -- 172.21.15.44:0/2179293182 --> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa32c109df0 con 0x7fa3140387d0 2023-11-16T16:40:27.783 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.208+0000 7fa33091e700 1 -- 172.21.15.44:0/2179293182 <== mgr.14118 v2:172.21.15.44:6800/4234382393 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa320009e20 con 0x7fa3140387d0 2023-11-16T16:40:27.783 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.211+0000 7fa331920700 1 -- 172.21.15.44:0/2179293182 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fa3140387d0 msgr2=0x7fa31403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:27.783 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.211+0000 7fa331920700 1 --2- 172.21.15.44:0/2179293182 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fa3140387d0 0x7fa31403ac80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa320006fd0 tx=0x7fa320006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:27.783 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.211+0000 7fa331920700 1 -- 172.21.15.44:0/2179293182 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa32c105950 msgr2=0x7fa32c19baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:27.783 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.211+0000 7fa331920700 1 --2- 172.21.15.44:0/2179293182 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa32c105950 0x7fa32c19baa0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa31c009fb0 tx=0x7fa31c011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:27.784 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.211+0000 7fa331920700 1 -- 172.21.15.44:0/2179293182 shutdown_connections 2023-11-16T16:40:27.784 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.211+0000 7fa331920700 1 --2- 172.21.15.44:0/2179293182 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fa3140387d0 0x7fa31403ac80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:27.784 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.211+0000 7fa331920700 1 --2- 172.21.15.44:0/2179293182 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa32c105950 0x7fa32c19baa0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:27.784 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.211+0000 7fa331920700 1 -- 172.21.15.44:0/2179293182 >> 172.21.15.44:0/2179293182 conn(0x7fa32c100fe0 msgr2=0x7fa32c101cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:27.784 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.211+0000 7fa331920700 1 -- 172.21.15.44:0/2179293182 shutdown_connections 2023-11-16T16:40:27.784 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.211+0000 7fa331920700 1 -- 172.21.15.44:0/2179293182 wait complete. 2023-11-16T16:40:27.784 INFO:teuthology.orchestra.run.smithi044.stdout:Deploying unmanaged mgr service... 2023-11-16T16:40:28.316 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:28 smithi044 ceph-mon[128750]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:28.316 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:28 smithi044 ceph-mon[128750]: Saving service mon spec with placement count:5 2023-11-16T16:40:28.316 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:28 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' 2023-11-16T16:40:29.372 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-11-16T16:40:29.372 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.327+0000 7f79e619e700 1 Processor -- start 2023-11-16T16:40:29.372 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.327+0000 7f79e619e700 1 -- start start 2023-11-16T16:40:29.372 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.328+0000 7f79e619e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e0105950 0x7f79e0105d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:29.372 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.328+0000 7f79e619e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e01062a0 con 0x7f79e0105950 2023-11-16T16:40:29.372 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.328+0000 7f79df7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e0105950 0x7f79e0105d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:29.373 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.328+0000 7f79df7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e0105950 0x7f79e0105d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50684/0 (socket says 172.21.15.44:50684) 2023-11-16T16:40:29.373 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.328+0000 7f79df7fe700 1 -- 172.21.15.44:0/2415079764 learned_addr learned my addr 172.21.15.44:0/2415079764 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:29.373 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.328+0000 7f79df7fe700 1 -- 172.21.15.44:0/2415079764 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e01063e0 con 0x7f79e0105950 2023-11-16T16:40:29.373 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.329+0000 7f79df7fe700 1 --2- 172.21.15.44:0/2415079764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e0105950 0x7f79e0105d60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f79d00098a0 tx=0x7f79d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f7a63655a9b77 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:29.373 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.329+0000 7f79de7fc700 1 -- 172.21.15.44:0/2415079764 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79d0004160 con 0x7f79e0105950 2023-11-16T16:40:29.373 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.329+0000 7f79de7fc700 1 -- 172.21.15.44:0/2415079764 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79d00042c0 con 0x7f79e0105950 2023-11-16T16:40:29.373 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.329+0000 7f79de7fc700 1 -- 172.21.15.44:0/2415079764 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79d0011a00 con 0x7f79e0105950 2023-11-16T16:40:29.373 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.330+0000 7f79e619e700 1 -- 172.21.15.44:0/2415079764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e0105950 msgr2=0x7f79e0105d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:29.373 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.330+0000 7f79e619e700 1 --2- 172.21.15.44:0/2415079764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e0105950 0x7f79e0105d60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f79d00098a0 tx=0x7f79d0009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:29.373 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.330+0000 7f79e619e700 1 -- 172.21.15.44:0/2415079764 shutdown_connections 2023-11-16T16:40:29.373 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.330+0000 7f79e619e700 1 --2- 172.21.15.44:0/2415079764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e0105950 0x7f79e0105d60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:29.373 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.330+0000 7f79e619e700 1 -- 172.21.15.44:0/2415079764 >> 172.21.15.44:0/2415079764 conn(0x7f79e0100fe0 msgr2=0x7f79e01033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:29.373 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.330+0000 7f79e619e700 1 -- 172.21.15.44:0/2415079764 shutdown_connections 2023-11-16T16:40:29.373 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.330+0000 7f79e619e700 1 -- 172.21.15.44:0/2415079764 wait complete. 2023-11-16T16:40:29.374 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.331+0000 7f79e619e700 1 Processor -- start 2023-11-16T16:40:29.374 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.331+0000 7f79e619e700 1 -- start start 2023-11-16T16:40:29.374 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.331+0000 7f79e619e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e0105950 0x7f79e019baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:29.374 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.331+0000 7f79e619e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e019bfe0 con 0x7f79e0105950 2023-11-16T16:40:29.374 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.332+0000 7f79df7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e0105950 0x7f79e019baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:29.374 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.332+0000 7f79df7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e0105950 0x7f79e019baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50688/0 (socket says 172.21.15.44:50688) 2023-11-16T16:40:29.374 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.332+0000 7f79df7fe700 1 -- 172.21.15.44:0/4015461513 learned_addr learned my addr 172.21.15.44:0/4015461513 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:29.374 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.332+0000 7f79df7fe700 1 -- 172.21.15.44:0/4015461513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d0009550 con 0x7f79e0105950 2023-11-16T16:40:29.374 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.332+0000 7f79df7fe700 1 --2- 172.21.15.44:0/4015461513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e0105950 0x7f79e019baa0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f79d0006af0 tx=0x7f79d0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:29.374 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.333+0000 7f79dcff9700 1 -- 172.21.15.44:0/4015461513 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79d0003a40 con 0x7f79e0105950 2023-11-16T16:40:29.374 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.333+0000 7f79dcff9700 1 -- 172.21.15.44:0/4015461513 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79d0003ba0 con 0x7f79e0105950 2023-11-16T16:40:29.374 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.333+0000 7f79e619e700 1 -- 172.21.15.44:0/4015461513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e01079a0 con 0x7f79e0105950 2023-11-16T16:40:29.374 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.333+0000 7f79dcff9700 1 -- 172.21.15.44:0/4015461513 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79d001a6e0 con 0x7f79e0105950 2023-11-16T16:40:29.375 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.333+0000 7f79e619e700 1 -- 172.21.15.44:0/4015461513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e0107810 con 0x7f79e0105950 2023-11-16T16:40:29.375 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.334+0000 7f79dcff9700 1 -- 172.21.15.44:0/4015461513 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f79d0024a60 con 0x7f79e0105950 2023-11-16T16:40:29.375 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.334+0000 7f79e619e700 1 -- 172.21.15.44:0/4015461513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e0109e20 con 0x7f79e0105950 2023-11-16T16:40:29.375 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.334+0000 7f79dcff9700 1 --2- 172.21.15.44:0/4015461513 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f79c80387d0 0x7f79c803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:29.375 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.335+0000 7f79dcff9700 1 -- 172.21.15.44:0/4015461513 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f79d004c770 con 0x7f79e0105950 2023-11-16T16:40:29.375 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.335+0000 7f79deffd700 1 --2- 172.21.15.44:0/4015461513 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f79c80387d0 0x7f79c803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:29.375 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.336+0000 7f79deffd700 1 --2- 172.21.15.44:0/4015461513 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f79c80387d0 0x7f79c803ac80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f79d4006fd0 tx=0x7f79d4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:29.375 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.341+0000 7f79dcff9700 1 -- 172.21.15.44:0/4015461513 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f79d002d910 con 0x7f79e0105950 2023-11-16T16:40:29.375 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.569+0000 7f79e619e700 1 -- 172.21.15.44:0/4015461513 --> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f79e01023d0 con 0x7f79c80387d0 2023-11-16T16:40:29.375 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.575+0000 7f79dcff9700 1 -- 172.21.15.44:0/4015461513 <== mgr.14118 v2:172.21.15.44:6800/4234382393 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f79e01023d0 con 0x7f79c80387d0 2023-11-16T16:40:29.375 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.578+0000 7f79e619e700 1 -- 172.21.15.44:0/4015461513 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f79c80387d0 msgr2=0x7f79c803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:29.375 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.578+0000 7f79e619e700 1 --2- 172.21.15.44:0/4015461513 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f79c80387d0 0x7f79c803ac80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f79d4006fd0 tx=0x7f79d4006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:29.375 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.578+0000 7f79e619e700 1 -- 172.21.15.44:0/4015461513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e0105950 msgr2=0x7f79e019baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:29.375 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.578+0000 7f79e619e700 1 --2- 172.21.15.44:0/4015461513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e0105950 0x7f79e019baa0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f79d0006af0 tx=0x7f79d0011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:29.376 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.578+0000 7f79e619e700 1 -- 172.21.15.44:0/4015461513 shutdown_connections 2023-11-16T16:40:29.376 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.578+0000 7f79e619e700 1 --2- 172.21.15.44:0/4015461513 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f79c80387d0 0x7f79c803ac80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:29.376 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.578+0000 7f79e619e700 1 --2- 172.21.15.44:0/4015461513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e0105950 0x7f79e019baa0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:29.376 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.578+0000 7f79e619e700 1 -- 172.21.15.44:0/4015461513 >> 172.21.15.44:0/4015461513 conn(0x7f79e0100fe0 msgr2=0x7f79e0101cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:29.376 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.578+0000 7f79e619e700 1 -- 172.21.15.44:0/4015461513 shutdown_connections 2023-11-16T16:40:29.376 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.578+0000 7f79e619e700 1 -- 172.21.15.44:0/4015461513 wait complete. 2023-11-16T16:40:29.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:29 smithi044 ceph-mon[128750]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:29.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:29 smithi044 ceph-mon[128750]: Saving service mgr spec with placement count:2 2023-11-16T16:40:29.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:29 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' 2023-11-16T16:40:29.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:29 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' 2023-11-16T16:40:30.555 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.891+0000 7f53392db700 1 Processor -- start 2023-11-16T16:40:30.555 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.891+0000 7f53392db700 1 -- start start 2023-11-16T16:40:30.555 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.892+0000 7f53392db700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334105950 0x7f5334105d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:30.555 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.892+0000 7f53392db700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53341062a0 con 0x7f5334105950 2023-11-16T16:40:30.555 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.892+0000 7f5332ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334105950 0x7f5334105d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:30.555 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.892+0000 7f5332ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334105950 0x7f5334105d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50690/0 (socket says 172.21.15.44:50690) 2023-11-16T16:40:30.555 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.892+0000 7f5332ffd700 1 -- 172.21.15.44:0/1033661649 learned_addr learned my addr 172.21.15.44:0/1033661649 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:30.556 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.892+0000 7f5332ffd700 1 -- 172.21.15.44:0/1033661649 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53341063e0 con 0x7f5334105950 2023-11-16T16:40:30.556 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.893+0000 7f5332ffd700 1 --2- 172.21.15.44:0/1033661649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334105950 0x7f5334105d60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f53240098a0 tx=0x7f5324009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b68a3a807fe77313 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:30.556 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.893+0000 7f5331ffb700 1 -- 172.21.15.44:0/1033661649 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5324004160 con 0x7f5334105950 2023-11-16T16:40:30.556 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.893+0000 7f5331ffb700 1 -- 172.21.15.44:0/1033661649 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53240042c0 con 0x7f5334105950 2023-11-16T16:40:30.556 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.893+0000 7f5331ffb700 1 -- 172.21.15.44:0/1033661649 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f53240119b0 con 0x7f5334105950 2023-11-16T16:40:30.556 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.894+0000 7f53392db700 1 -- 172.21.15.44:0/1033661649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334105950 msgr2=0x7f5334105d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:30.556 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.894+0000 7f53392db700 1 --2- 172.21.15.44:0/1033661649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334105950 0x7f5334105d60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f53240098a0 tx=0x7f5324009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:30.556 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.894+0000 7f53392db700 1 -- 172.21.15.44:0/1033661649 shutdown_connections 2023-11-16T16:40:30.556 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.894+0000 7f53392db700 1 --2- 172.21.15.44:0/1033661649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334105950 0x7f5334105d60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:30.556 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.894+0000 7f53392db700 1 -- 172.21.15.44:0/1033661649 >> 172.21.15.44:0/1033661649 conn(0x7f5334100fe0 msgr2=0x7f53341033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:30.556 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.894+0000 7f53392db700 1 -- 172.21.15.44:0/1033661649 shutdown_connections 2023-11-16T16:40:30.556 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.894+0000 7f53392db700 1 -- 172.21.15.44:0/1033661649 wait complete. 2023-11-16T16:40:30.556 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.895+0000 7f53392db700 1 Processor -- start 2023-11-16T16:40:30.556 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.895+0000 7f53392db700 1 -- start start 2023-11-16T16:40:30.557 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.895+0000 7f53392db700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334105950 0x7f533419bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:30.557 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.895+0000 7f53392db700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533419c020 con 0x7f5334105950 2023-11-16T16:40:30.557 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.895+0000 7f5332ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334105950 0x7f533419bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:30.557 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.895+0000 7f5332ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334105950 0x7f533419bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50704/0 (socket says 172.21.15.44:50704) 2023-11-16T16:40:30.557 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.895+0000 7f5332ffd700 1 -- 172.21.15.44:0/1717261963 learned_addr learned my addr 172.21.15.44:0/1717261963 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:30.557 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.896+0000 7f5332ffd700 1 -- 172.21.15.44:0/1717261963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5324009550 con 0x7f5334105950 2023-11-16T16:40:30.557 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.896+0000 7f5332ffd700 1 --2- 172.21.15.44:0/1717261963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334105950 0x7f533419bae0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5324011980 tx=0x7f5324011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:30.557 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.896+0000 7f531bfff700 1 -- 172.21.15.44:0/1717261963 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5324003a60 con 0x7f5334105950 2023-11-16T16:40:30.557 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.896+0000 7f531bfff700 1 -- 172.21.15.44:0/1717261963 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5324003bc0 con 0x7f5334105950 2023-11-16T16:40:30.557 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.896+0000 7f531bfff700 1 -- 172.21.15.44:0/1717261963 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f532401a670 con 0x7f5334105950 2023-11-16T16:40:30.557 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.896+0000 7f53392db700 1 -- 172.21.15.44:0/1717261963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53341079e0 con 0x7f5334105950 2023-11-16T16:40:30.557 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.897+0000 7f53392db700 1 -- 172.21.15.44:0/1717261963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5334107850 con 0x7f5334105950 2023-11-16T16:40:30.557 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.898+0000 7f53392db700 1 -- 172.21.15.44:0/1717261963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5334109e20 con 0x7f5334105950 2023-11-16T16:40:30.557 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.898+0000 7f531bfff700 1 -- 172.21.15.44:0/1717261963 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f5324024a00 con 0x7f5334105950 2023-11-16T16:40:30.558 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.898+0000 7f531bfff700 1 --2- 172.21.15.44:0/1717261963 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f531c038820 0x7f531c03acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:30.558 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.898+0000 7f531bfff700 1 -- 172.21.15.44:0/1717261963 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f532404c8a0 con 0x7f5334105950 2023-11-16T16:40:30.558 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.898+0000 7f53327fc700 1 --2- 172.21.15.44:0/1717261963 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f531c038820 0x7f531c03acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:30.558 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.899+0000 7f53327fc700 1 --2- 172.21.15.44:0/1717261963 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f531c038820 0x7f531c03acd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5328006fd0 tx=0x7f5328006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:30.558 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:29.903+0000 7f531bfff700 1 -- 172.21.15.44:0/1717261963 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5324024440 con 0x7f5334105950 2023-11-16T16:40:30.558 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.111+0000 7f53392db700 1 -- 172.21.15.44:0/1717261963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f533419c5e0 con 0x7f5334105950 2023-11-16T16:40:30.558 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.116+0000 7f531bfff700 1 -- 172.21.15.44:0/1717261963 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f5324018350 con 0x7f5334105950 2023-11-16T16:40:30.558 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.118+0000 7f53392db700 1 -- 172.21.15.44:0/1717261963 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f531c038820 msgr2=0x7f531c03acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:30.558 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.118+0000 7f53392db700 1 --2- 172.21.15.44:0/1717261963 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f531c038820 0x7f531c03acd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5328006fd0 tx=0x7f5328006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:30.558 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.118+0000 7f53392db700 1 -- 172.21.15.44:0/1717261963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334105950 msgr2=0x7f533419bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:30.558 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.118+0000 7f53392db700 1 --2- 172.21.15.44:0/1717261963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334105950 0x7f533419bae0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5324011980 tx=0x7f5324011a60 comp rx=0 tx=0).stop 2023-11-16T16:40:30.558 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.119+0000 7f53392db700 1 -- 172.21.15.44:0/1717261963 shutdown_connections 2023-11-16T16:40:30.558 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.119+0000 7f53392db700 1 --2- 172.21.15.44:0/1717261963 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f531c038820 0x7f531c03acd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:30.559 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.119+0000 7f53392db700 1 --2- 172.21.15.44:0/1717261963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334105950 0x7f533419bae0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:30.559 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.119+0000 7f53392db700 1 -- 172.21.15.44:0/1717261963 >> 172.21.15.44:0/1717261963 conn(0x7f5334100fe0 msgr2=0x7f5334101d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:30.559 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.119+0000 7f53392db700 1 -- 172.21.15.44:0/1717261963 shutdown_connections 2023-11-16T16:40:30.559 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.119+0000 7f53392db700 1 -- 172.21.15.44:0/1717261963 wait complete. 2023-11-16T16:40:31.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:31 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' 2023-11-16T16:40:31.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:31 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1717261963' entity='client.admin' 2023-11-16T16:40:31.940 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.132+0000 7ff009e93700 1 Processor -- start 2023-11-16T16:40:31.940 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.132+0000 7ff009e93700 1 -- start start 2023-11-16T16:40:31.940 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.132+0000 7ff009e93700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff00407bde0 0x7ff00407a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:31.940 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.132+0000 7ff009e93700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00407a820 con 0x7ff00407bde0 2023-11-16T16:40:31.940 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.133+0000 7ff0037fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff00407bde0 0x7ff00407a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:31.940 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.133+0000 7ff0037fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff00407bde0 0x7ff00407a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50720/0 (socket says 172.21.15.44:50720) 2023-11-16T16:40:31.940 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.133+0000 7ff0037fe700 1 -- 172.21.15.44:0/2484863236 learned_addr learned my addr 172.21.15.44:0/2484863236 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:31.940 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.133+0000 7ff0037fe700 1 -- 172.21.15.44:0/2484863236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00407a960 con 0x7ff00407bde0 2023-11-16T16:40:31.940 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.133+0000 7ff0037fe700 1 --2- 172.21.15.44:0/2484863236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff00407bde0 0x7ff00407a2e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feff40098a0 tx=0x7feff4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5031f0d7bcaf3da6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:31.941 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.134+0000 7ff0027fc700 1 -- 172.21.15.44:0/2484863236 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feff4004160 con 0x7ff00407bde0 2023-11-16T16:40:31.941 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.134+0000 7ff0027fc700 1 -- 172.21.15.44:0/2484863236 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feff40042c0 con 0x7ff00407bde0 2023-11-16T16:40:31.941 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.134+0000 7ff0027fc700 1 -- 172.21.15.44:0/2484863236 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feff40119b0 con 0x7ff00407bde0 2023-11-16T16:40:31.941 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.134+0000 7ff009e93700 1 -- 172.21.15.44:0/2484863236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff00407bde0 msgr2=0x7ff00407a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:31.941 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.134+0000 7ff009e93700 1 --2- 172.21.15.44:0/2484863236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff00407bde0 0x7ff00407a2e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feff40098a0 tx=0x7feff4009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:31.941 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.134+0000 7ff009e93700 1 -- 172.21.15.44:0/2484863236 shutdown_connections 2023-11-16T16:40:31.941 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.134+0000 7ff009e93700 1 --2- 172.21.15.44:0/2484863236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff00407bde0 0x7ff00407a2e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:31.941 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.134+0000 7ff009e93700 1 -- 172.21.15.44:0/2484863236 >> 172.21.15.44:0/2484863236 conn(0x7ff004100fe0 msgr2=0x7ff0041033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:31.941 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.135+0000 7ff009e93700 1 -- 172.21.15.44:0/2484863236 shutdown_connections 2023-11-16T16:40:31.941 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.135+0000 7ff009e93700 1 -- 172.21.15.44:0/2484863236 wait complete. 2023-11-16T16:40:31.941 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.136+0000 7ff009e93700 1 Processor -- start 2023-11-16T16:40:31.941 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.136+0000 7ff009e93700 1 -- start start 2023-11-16T16:40:31.941 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.136+0000 7ff009e93700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff00407bde0 0x7ff00419bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:31.941 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.136+0000 7ff009e93700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00419c060 con 0x7ff00407bde0 2023-11-16T16:40:31.941 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.137+0000 7ff0037fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff00407bde0 0x7ff00419bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:31.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.137+0000 7ff0037fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff00407bde0 0x7ff00419bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50736/0 (socket says 172.21.15.44:50736) 2023-11-16T16:40:31.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.137+0000 7ff0037fe700 1 -- 172.21.15.44:0/1153547147 learned_addr learned my addr 172.21.15.44:0/1153547147 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:31.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.137+0000 7ff0037fe700 1 -- 172.21.15.44:0/1153547147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff4009550 con 0x7ff00407bde0 2023-11-16T16:40:31.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.137+0000 7ff0037fe700 1 --2- 172.21.15.44:0/1153547147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff00407bde0 0x7ff00419bb20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feff4000c00 tx=0x7feff4011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:31.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.138+0000 7ff000ff9700 1 -- 172.21.15.44:0/1153547147 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feff4003ac0 con 0x7ff00407bde0 2023-11-16T16:40:31.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.138+0000 7ff000ff9700 1 -- 172.21.15.44:0/1153547147 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feff4003c20 con 0x7ff00407bde0 2023-11-16T16:40:31.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.138+0000 7ff000ff9700 1 -- 172.21.15.44:0/1153547147 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feff401a720 con 0x7ff00407bde0 2023-11-16T16:40:31.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.138+0000 7ff009e93700 1 -- 172.21.15.44:0/1153547147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00410be50 con 0x7ff00407bde0 2023-11-16T16:40:31.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.138+0000 7ff009e93700 1 -- 172.21.15.44:0/1153547147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff00410bcc0 con 0x7ff00407bde0 2023-11-16T16:40:31.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.140+0000 7ff009e93700 1 -- 172.21.15.44:0/1153547147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff0005320 con 0x7ff00407bde0 2023-11-16T16:40:31.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.140+0000 7ff000ff9700 1 -- 172.21.15.44:0/1153547147 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7feff4024a90 con 0x7ff00407bde0 2023-11-16T16:40:31.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.141+0000 7ff000ff9700 1 --2- 172.21.15.44:0/1153547147 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fefec038870 0x7fefec03ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:31.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.141+0000 7ff000ff9700 1 -- 172.21.15.44:0/1153547147 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7feff404c9e0 con 0x7ff00407bde0 2023-11-16T16:40:31.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.141+0000 7ff002ffd700 1 --2- 172.21.15.44:0/1153547147 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fefec038870 0x7fefec03ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:31.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.142+0000 7ff002ffd700 1 --2- 172.21.15.44:0/1153547147 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fefec038870 0x7fefec03ad20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feff8006fd0 tx=0x7feff8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:31.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.146+0000 7ff000ff9700 1 -- 172.21.15.44:0/1153547147 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7feff4024440 con 0x7ff00407bde0 2023-11-16T16:40:31.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.360+0000 7ff009e93700 1 -- 172.21.15.44:0/1153547147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7feff0005f70 con 0x7ff00407bde0 2023-11-16T16:40:31.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.364+0000 7ff000ff9700 1 -- 172.21.15.44:0/1153547147 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7feff402c390 con 0x7ff00407bde0 2023-11-16T16:40:31.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.367+0000 7ff009e93700 1 -- 172.21.15.44:0/1153547147 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fefec038870 msgr2=0x7fefec03ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:31.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.367+0000 7ff009e93700 1 --2- 172.21.15.44:0/1153547147 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fefec038870 0x7fefec03ad20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feff8006fd0 tx=0x7feff8006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:31.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.367+0000 7ff009e93700 1 -- 172.21.15.44:0/1153547147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff00407bde0 msgr2=0x7ff00419bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:31.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.367+0000 7ff009e93700 1 --2- 172.21.15.44:0/1153547147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff00407bde0 0x7ff00419bb20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feff4000c00 tx=0x7feff4011a80 comp rx=0 tx=0).stop 2023-11-16T16:40:31.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.367+0000 7ff009e93700 1 -- 172.21.15.44:0/1153547147 shutdown_connections 2023-11-16T16:40:31.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.367+0000 7ff009e93700 1 --2- 172.21.15.44:0/1153547147 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7fefec038870 0x7fefec03ad20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:31.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.367+0000 7ff009e93700 1 --2- 172.21.15.44:0/1153547147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff00407bde0 0x7ff00419bb20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:31.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.367+0000 7ff009e93700 1 -- 172.21.15.44:0/1153547147 >> 172.21.15.44:0/1153547147 conn(0x7ff004100fe0 msgr2=0x7ff004101de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:31.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.367+0000 7ff009e93700 1 -- 172.21.15.44:0/1153547147 shutdown_connections 2023-11-16T16:40:31.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.367+0000 7ff009e93700 1 -- 172.21.15.44:0/1153547147 wait complete. 2023-11-16T16:40:31.943 INFO:teuthology.orchestra.run.smithi044.stdout:Enabling the dashboard module... 2023-11-16T16:40:32.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:32 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1153547147' entity='client.admin' 2023-11-16T16:40:33.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:33 smithi044 ceph-mon[128750]: from='mgr.14118 172.21.15.44:0/638267248' entity='mgr.x' 2023-11-16T16:40:33.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:33 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/4197833722' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-16T16:40:34.036 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:33 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: ignoring --setuser ceph since I am not root 2023-11-16T16:40:34.036 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:33 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: ignoring --setgroup ceph since I am not root 2023-11-16T16:40:34.036 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:33 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:33.925+0000 7f765769e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T16:40:34.076 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.519+0000 7f0102bb1700 1 Processor -- start 2023-11-16T16:40:34.077 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.519+0000 7f0102bb1700 1 -- start start 2023-11-16T16:40:34.077 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.520+0000 7f0102bb1700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc07bde0 0x7f00fc07a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:34.077 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.520+0000 7f0102bb1700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc07a820 con 0x7f00fc07bde0 2023-11-16T16:40:34.077 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.520+0000 7f010094d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc07bde0 0x7f00fc07a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:34.077 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.520+0000 7f010094d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc07bde0 0x7f00fc07a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50740/0 (socket says 172.21.15.44:50740) 2023-11-16T16:40:34.077 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.520+0000 7f010094d700 1 -- 172.21.15.44:0/1955596509 learned_addr learned my addr 172.21.15.44:0/1955596509 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:34.077 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.520+0000 7f010094d700 1 -- 172.21.15.44:0/1955596509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc07a960 con 0x7f00fc07bde0 2023-11-16T16:40:34.077 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.521+0000 7f010094d700 1 --2- 172.21.15.44:0/1955596509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc07bde0 0x7f00fc07a2e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f00f00098a0 tx=0x7f00f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5486d5bcee5adcef server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:34.077 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.521+0000 7f00fb7fe700 1 -- 172.21.15.44:0/1955596509 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00f0004160 con 0x7f00fc07bde0 2023-11-16T16:40:34.077 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.521+0000 7f00fb7fe700 1 -- 172.21.15.44:0/1955596509 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00f00042c0 con 0x7f00fc07bde0 2023-11-16T16:40:34.077 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.521+0000 7f00fb7fe700 1 -- 172.21.15.44:0/1955596509 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00f00119b0 con 0x7f00fc07bde0 2023-11-16T16:40:34.078 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.522+0000 7f0102bb1700 1 -- 172.21.15.44:0/1955596509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc07bde0 msgr2=0x7f00fc07a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:34.078 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.522+0000 7f0102bb1700 1 --2- 172.21.15.44:0/1955596509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc07bde0 0x7f00fc07a2e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f00f00098a0 tx=0x7f00f0009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:34.078 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.522+0000 7f0102bb1700 1 -- 172.21.15.44:0/1955596509 shutdown_connections 2023-11-16T16:40:34.078 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.522+0000 7f0102bb1700 1 --2- 172.21.15.44:0/1955596509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc07bde0 0x7f00fc07a2e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:34.078 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.522+0000 7f0102bb1700 1 -- 172.21.15.44:0/1955596509 >> 172.21.15.44:0/1955596509 conn(0x7f00fc100fe0 msgr2=0x7f00fc1033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:34.078 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.522+0000 7f0102bb1700 1 -- 172.21.15.44:0/1955596509 shutdown_connections 2023-11-16T16:40:34.078 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.522+0000 7f0102bb1700 1 -- 172.21.15.44:0/1955596509 wait complete. 2023-11-16T16:40:34.078 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.523+0000 7f0102bb1700 1 Processor -- start 2023-11-16T16:40:34.078 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.523+0000 7f0102bb1700 1 -- start start 2023-11-16T16:40:34.078 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.523+0000 7f0102bb1700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc07bde0 0x7f00fc19bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:34.078 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.524+0000 7f0102bb1700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc19c060 con 0x7f00fc07bde0 2023-11-16T16:40:34.078 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.524+0000 7f010094d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc07bde0 0x7f00fc19bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:34.078 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.524+0000 7f010094d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc07bde0 0x7f00fc19bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50754/0 (socket says 172.21.15.44:50754) 2023-11-16T16:40:34.079 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.524+0000 7f010094d700 1 -- 172.21.15.44:0/4197833722 learned_addr learned my addr 172.21.15.44:0/4197833722 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:34.079 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.524+0000 7f010094d700 1 -- 172.21.15.44:0/4197833722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f0009550 con 0x7f00fc07bde0 2023-11-16T16:40:34.079 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.524+0000 7f010094d700 1 --2- 172.21.15.44:0/4197833722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc07bde0 0x7f00fc19bb20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f00f0000c00 tx=0x7f00f0011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:34.079 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.525+0000 7f00f9ffb700 1 -- 172.21.15.44:0/4197833722 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00f0003ac0 con 0x7f00fc07bde0 2023-11-16T16:40:34.079 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.525+0000 7f00f9ffb700 1 -- 172.21.15.44:0/4197833722 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00f0003c20 con 0x7f00fc07bde0 2023-11-16T16:40:34.079 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.525+0000 7f00f9ffb700 1 -- 172.21.15.44:0/4197833722 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00f001a720 con 0x7f00fc07bde0 2023-11-16T16:40:34.079 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.525+0000 7f0102bb1700 1 -- 172.21.15.44:0/4197833722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00fc10be50 con 0x7f00fc07bde0 2023-11-16T16:40:34.079 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.525+0000 7f0102bb1700 1 -- 172.21.15.44:0/4197833722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00fc10bcc0 con 0x7f00fc07bde0 2023-11-16T16:40:34.079 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.526+0000 7f0102bb1700 1 -- 172.21.15.44:0/4197833722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00fc10bfe0 con 0x7f00fc07bde0 2023-11-16T16:40:34.079 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.527+0000 7f00f9ffb700 1 -- 172.21.15.44:0/4197833722 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f00f0024a90 con 0x7f00fc07bde0 2023-11-16T16:40:34.079 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.527+0000 7f00f9ffb700 1 --2- 172.21.15.44:0/4197833722 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f00ec038820 0x7f00ec03acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:34.079 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.527+0000 7f00f9ffb700 1 -- 172.21.15.44:0/4197833722 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f00f004c890 con 0x7f00fc07bde0 2023-11-16T16:40:34.079 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.527+0000 7f00fbfff700 1 --2- 172.21.15.44:0/4197833722 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f00ec038820 0x7f00ec03acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:34.080 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.528+0000 7f00fbfff700 1 --2- 172.21.15.44:0/4197833722 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f00ec038820 0x7f00ec03acd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f00e8006fd0 tx=0x7f00e8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:34.080 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.532+0000 7f00f9ffb700 1 -- 172.21.15.44:0/4197833722 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f00f0024440 con 0x7f00fc07bde0 2023-11-16T16:40:34.080 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.809+0000 7f0102bb1700 1 -- 172.21.15.44:0/4197833722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f00fc02cfe0 con 0x7f00fc07bde0 2023-11-16T16:40:34.080 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:33.623+0000 7f00f9ffb700 1 -- 172.21.15.44:0/4197833722 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f00f004ecd0 con 0x7f00fc07bde0 2023-11-16T16:40:34.080 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:33.625+0000 7f00f9ffb700 1 -- 172.21.15.44:0/4197833722 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f00f004c220 con 0x7f00fc07bde0 2023-11-16T16:40:34.080 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:33.628+0000 7f0102bb1700 1 -- 172.21.15.44:0/4197833722 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f00ec038820 msgr2=0x7f00ec03acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:34.080 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:33.628+0000 7f0102bb1700 1 --2- 172.21.15.44:0/4197833722 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f00ec038820 0x7f00ec03acd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f00e8006fd0 tx=0x7f00e8006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:34.080 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:33.628+0000 7f0102bb1700 1 -- 172.21.15.44:0/4197833722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc07bde0 msgr2=0x7f00fc19bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:34.080 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:33.628+0000 7f0102bb1700 1 --2- 172.21.15.44:0/4197833722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc07bde0 0x7f00fc19bb20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f00f0000c00 tx=0x7f00f0011a80 comp rx=0 tx=0).stop 2023-11-16T16:40:34.080 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:33.628+0000 7f0102bb1700 1 -- 172.21.15.44:0/4197833722 shutdown_connections 2023-11-16T16:40:34.080 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:33.628+0000 7f0102bb1700 1 --2- 172.21.15.44:0/4197833722 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f00ec038820 0x7f00ec03acd0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:34.080 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:33.628+0000 7f0102bb1700 1 --2- 172.21.15.44:0/4197833722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc07bde0 0x7f00fc19bb20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:34.080 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:33.628+0000 7f0102bb1700 1 -- 172.21.15.44:0/4197833722 >> 172.21.15.44:0/4197833722 conn(0x7f00fc100fe0 msgr2=0x7f00fc101de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:34.081 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:33.629+0000 7f0102bb1700 1 -- 172.21.15.44:0/4197833722 shutdown_connections 2023-11-16T16:40:34.081 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:33.629+0000 7f0102bb1700 1 -- 172.21.15.44:0/4197833722 wait complete. 2023-11-16T16:40:34.287 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:34 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:34.033+0000 7f765769e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T16:40:34.287 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:34 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:34.161+0000 7f765769e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T16:40:34.287 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:34 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:34.285+0000 7f765769e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T16:40:34.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:34 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:34.526+0000 7f765769e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T16:40:34.930 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:34 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/4197833722' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-16T16:40:34.930 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:34 smithi044 ceph-mon[128750]: mgrmap e9: x(active, since 17s) 2023-11-16T16:40:35.246 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:34 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:34.936+0000 7f765769e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T16:40:35.246 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:35 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:35.066+0000 7f765769e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T16:40:35.246 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:35 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:35.244+0000 7f765769e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T16:40:35.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2023-11-16T16:40:35.363 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-11-16T16:40:35.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": true, 2023-11-16T16:40:35.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-11-16T16:40:35.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-11-16T16:40:35.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2023-11-16T16:40:35.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.629+0000 7f625b847700 1 Processor -- start 2023-11-16T16:40:35.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.629+0000 7f625b847700 1 -- start start 2023-11-16T16:40:35.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.630+0000 7f625b847700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6254104250 0x7f6254104660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:35.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.630+0000 7f625b847700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6254079d40 con 0x7f6254104250 2023-11-16T16:40:35.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.630+0000 7f62595e3700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6254104250 0x7f6254104660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:35.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.630+0000 7f62595e3700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6254104250 0x7f6254104660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50790/0 (socket says 172.21.15.44:50790) 2023-11-16T16:40:35.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.630+0000 7f62595e3700 1 -- 172.21.15.44:0/1510694621 learned_addr learned my addr 172.21.15.44:0/1510694621 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:35.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.631+0000 7f62595e3700 1 -- 172.21.15.44:0/1510694621 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6254079e80 con 0x7f6254104250 2023-11-16T16:40:35.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.631+0000 7f62595e3700 1 --2- 172.21.15.44:0/1510694621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6254104250 0x7f6254104660 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f62500098a0 tx=0x7f6250009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cbcea547a85590e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:35.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.631+0000 7f624bfff700 1 -- 172.21.15.44:0/1510694621 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6250004160 con 0x7f6254104250 2023-11-16T16:40:35.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.631+0000 7f624bfff700 1 -- 172.21.15.44:0/1510694621 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62500042c0 con 0x7f6254104250 2023-11-16T16:40:35.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.631+0000 7f624bfff700 1 -- 172.21.15.44:0/1510694621 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62500119b0 con 0x7f6254104250 2023-11-16T16:40:35.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.632+0000 7f625b847700 1 -- 172.21.15.44:0/1510694621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6254104250 msgr2=0x7f6254104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:35.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.632+0000 7f625b847700 1 --2- 172.21.15.44:0/1510694621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6254104250 0x7f6254104660 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f62500098a0 tx=0x7f6250009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:35.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.632+0000 7f625b847700 1 -- 172.21.15.44:0/1510694621 shutdown_connections 2023-11-16T16:40:35.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.632+0000 7f625b847700 1 --2- 172.21.15.44:0/1510694621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6254104250 0x7f6254104660 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:35.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.632+0000 7f625b847700 1 -- 172.21.15.44:0/1510694621 >> 172.21.15.44:0/1510694621 conn(0x7f62540ffea0 msgr2=0x7f62541022d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:35.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.632+0000 7f625b847700 1 -- 172.21.15.44:0/1510694621 shutdown_connections 2023-11-16T16:40:35.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.632+0000 7f625b847700 1 -- 172.21.15.44:0/1510694621 wait complete. 2023-11-16T16:40:35.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.633+0000 7f625b847700 1 Processor -- start 2023-11-16T16:40:35.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.633+0000 7f625b847700 1 -- start start 2023-11-16T16:40:35.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.634+0000 7f625b847700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6254104250 0x7f625419b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:35.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.634+0000 7f625b847700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625419bd80 con 0x7f6254104250 2023-11-16T16:40:35.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.634+0000 7f62595e3700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6254104250 0x7f625419b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:35.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.634+0000 7f62595e3700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6254104250 0x7f625419b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50804/0 (socket says 172.21.15.44:50804) 2023-11-16T16:40:35.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.634+0000 7f62595e3700 1 -- 172.21.15.44:0/1180624998 learned_addr learned my addr 172.21.15.44:0/1180624998 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:35.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.634+0000 7f62595e3700 1 -- 172.21.15.44:0/1180624998 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6250009550 con 0x7f6254104250 2023-11-16T16:40:35.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.635+0000 7f62595e3700 1 --2- 172.21.15.44:0/1180624998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6254104250 0x7f625419b840 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6250000c00 tx=0x7f6250011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:35.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.635+0000 7f624a7fc700 1 -- 172.21.15.44:0/1180624998 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62500039f0 con 0x7f6254104250 2023-11-16T16:40:35.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.635+0000 7f624a7fc700 1 -- 172.21.15.44:0/1180624998 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6250003b50 con 0x7f6254104250 2023-11-16T16:40:35.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.635+0000 7f624a7fc700 1 -- 172.21.15.44:0/1180624998 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f625001a780 con 0x7f6254104250 2023-11-16T16:40:35.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.635+0000 7f625b847700 1 -- 172.21.15.44:0/1180624998 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6254105a40 con 0x7f6254104250 2023-11-16T16:40:35.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.635+0000 7f625b847700 1 -- 172.21.15.44:0/1180624998 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62541058b0 con 0x7f6254104250 2023-11-16T16:40:35.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.636+0000 7f625b847700 1 -- 172.21.15.44:0/1180624998 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6254105bd0 con 0x7f6254104250 2023-11-16T16:40:35.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.637+0000 7f624a7fc700 1 -- 172.21.15.44:0/1180624998 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f6250024440 con 0x7f6254104250 2023-11-16T16:40:35.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.637+0000 7f624a7fc700 1 --2- 172.21.15.44:0/1180624998 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f6240038870 0x7f624003ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:35.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.637+0000 7f624a7fc700 1 -- 172.21.15.44:0/1180624998 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f625004c7a0 con 0x7f6254104250 2023-11-16T16:40:35.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.637+0000 7f6258de2700 1 -- 172.21.15.44:0/1180624998 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f6240038870 msgr2=0x7f624003ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4234382393 2023-11-16T16:40:35.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.637+0000 7f6258de2700 1 --2- 172.21.15.44:0/1180624998 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f6240038870 0x7f624003ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:40:35.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.642+0000 7f624a7fc700 1 -- 172.21.15.44:0/1180624998 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f62500189a0 con 0x7f6254104250 2023-11-16T16:40:35.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.838+0000 7f6258de2700 1 -- 172.21.15.44:0/1180624998 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f6240038870 msgr2=0x7f624003ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4234382393 2023-11-16T16:40:35.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.838+0000 7f6258de2700 1 --2- 172.21.15.44:0/1180624998 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f6240038870 0x7f624003ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T16:40:35.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.903+0000 7f625b847700 1 -- 172.21.15.44:0/1180624998 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f625402cfe0 con 0x7f6254104250 2023-11-16T16:40:35.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.904+0000 7f624a7fc700 1 -- 172.21.15.44:0/1180624998 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f6250018350 con 0x7f6254104250 2023-11-16T16:40:35.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.906+0000 7f625b847700 1 -- 172.21.15.44:0/1180624998 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f6240038870 msgr2=0x7f624003ad20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:40:35.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.906+0000 7f625b847700 1 --2- 172.21.15.44:0/1180624998 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f6240038870 0x7f624003ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:35.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.906+0000 7f625b847700 1 -- 172.21.15.44:0/1180624998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6254104250 msgr2=0x7f625419b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:35.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.906+0000 7f625b847700 1 --2- 172.21.15.44:0/1180624998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6254104250 0x7f625419b840 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6250000c00 tx=0x7f6250011a30 comp rx=0 tx=0).stop 2023-11-16T16:40:35.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.906+0000 7f625b847700 1 -- 172.21.15.44:0/1180624998 shutdown_connections 2023-11-16T16:40:35.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.906+0000 7f625b847700 1 --2- 172.21.15.44:0/1180624998 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f6240038870 0x7f624003ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:35.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.906+0000 7f625b847700 1 --2- 172.21.15.44:0/1180624998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6254104250 0x7f625419b840 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:35.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.906+0000 7f625b847700 1 -- 172.21.15.44:0/1180624998 >> 172.21.15.44:0/1180624998 conn(0x7f62540ffea0 msgr2=0x7f625418fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:35.368 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.906+0000 7f625b847700 1 -- 172.21.15.44:0/1180624998 shutdown_connections 2023-11-16T16:40:35.368 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.906+0000 7f625b847700 1 -- 172.21.15.44:0/1180624998 wait complete. 2023-11-16T16:40:35.368 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for the mgr to restart... 2023-11-16T16:40:35.368 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr epoch 9... 2023-11-16T16:40:35.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:35 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:35.374+0000 7f765769e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T16:40:35.958 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:35 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1180624998' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-16T16:40:36.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:35 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:35.956+0000 7f765769e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T16:40:37.118 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:36 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:36.727+0000 7f765769e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T16:40:37.118 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:36 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:36.942+0000 7f765769e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T16:40:38.286 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:37 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:37.927+0000 7f765769e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:40:38.286 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:38 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:38.051+0000 7f765769e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T16:40:38.946 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:38 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:38.284+0000 7f765769e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:40:38.946 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:38 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:38.659+0000 7f765769e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T16:40:39.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:38 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:38.944+0000 7f765769e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T16:40:39.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:39 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:39.144+0000 7f765769e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T16:40:39.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:39 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:39.501+0000 7f765769e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T16:40:40.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:40 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:40.469+0000 7f765769e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T16:40:40.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:40 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:40.649+0000 7f765769e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T16:40:41.348 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:40 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:40.905+0000 7f765769e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T16:40:41.349 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:41 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:41.060+0000 7f765769e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T16:40:41.349 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:41 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:41.346+0000 7f765769e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T16:40:42.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:40:42 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:40:42.446+0000 7f765769e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T16:40:42.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:42 smithi044 ceph-mon[128750]: Active manager daemon x restarted 2023-11-16T16:40:42.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:42 smithi044 ceph-mon[128750]: Activating manager daemon x 2023-11-16T16:40:42.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:42 smithi044 ceph-mon[128750]: osdmap e3: 0 total, 0 up, 0 in 2023-11-16T16:40:42.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:42 smithi044 ceph-mon[128750]: mgrmap e10: x(active, starting, since 0.00368391s) 2023-11-16T16:40:42.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:40:42.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T16:40:42.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T16:40:42.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T16:40:42.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T16:40:42.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:42 smithi044 ceph-mon[128750]: Manager daemon x is now available 2023-11-16T16:40:42.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:42.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:43.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T16:40:43.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T16:40:43.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:43 smithi044 ceph-mon[128750]: mgrmap e11: x(active, since 1.00772s) 2023-11-16T16:40:43.963 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2023-11-16T16:40:43.970 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-11-16T16:40:43.970 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "initialized": true 2023-11-16T16:40:43.970 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2023-11-16T16:40:43.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.907+0000 7f41b9d67700 1 Processor -- start 2023-11-16T16:40:43.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.907+0000 7f41b9d67700 1 -- start start 2023-11-16T16:40:43.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.908+0000 7f41b9d67700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41b4105950 0x7f41b4105d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:43.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.908+0000 7f41b9d67700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b41062a0 con 0x7f41b4105950 2023-11-16T16:40:43.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.908+0000 7f41b37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41b4105950 0x7f41b4105d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:43.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.908+0000 7f41b37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41b4105950 0x7f41b4105d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52406/0 (socket says 172.21.15.44:52406) 2023-11-16T16:40:43.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.908+0000 7f41b37fe700 1 -- 172.21.15.44:0/907963922 learned_addr learned my addr 172.21.15.44:0/907963922 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:43.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.908+0000 7f41b37fe700 1 -- 172.21.15.44:0/907963922 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b41063e0 con 0x7f41b4105950 2023-11-16T16:40:43.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.909+0000 7f41b37fe700 1 --2- 172.21.15.44:0/907963922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41b4105950 0x7f41b4105d60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f41a40098a0 tx=0x7f41a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a43f266bcb0de7f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:43.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.909+0000 7f41b27fc700 1 -- 172.21.15.44:0/907963922 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41a4004160 con 0x7f41b4105950 2023-11-16T16:40:43.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.909+0000 7f41b27fc700 1 -- 172.21.15.44:0/907963922 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41a40042c0 con 0x7f41b4105950 2023-11-16T16:40:43.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.909+0000 7f41b27fc700 1 -- 172.21.15.44:0/907963922 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41a40119b0 con 0x7f41b4105950 2023-11-16T16:40:43.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.910+0000 7f41b9d67700 1 -- 172.21.15.44:0/907963922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41b4105950 msgr2=0x7f41b4105d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:43.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.910+0000 7f41b9d67700 1 --2- 172.21.15.44:0/907963922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41b4105950 0x7f41b4105d60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f41a40098a0 tx=0x7f41a4009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:43.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.910+0000 7f41b9d67700 1 -- 172.21.15.44:0/907963922 shutdown_connections 2023-11-16T16:40:43.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.910+0000 7f41b9d67700 1 --2- 172.21.15.44:0/907963922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41b4105950 0x7f41b4105d60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:43.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.910+0000 7f41b9d67700 1 -- 172.21.15.44:0/907963922 >> 172.21.15.44:0/907963922 conn(0x7f41b4100fe0 msgr2=0x7f41b41033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:43.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.910+0000 7f41b9d67700 1 -- 172.21.15.44:0/907963922 shutdown_connections 2023-11-16T16:40:43.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.910+0000 7f41b9d67700 1 -- 172.21.15.44:0/907963922 wait complete. 2023-11-16T16:40:43.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.911+0000 7f41b9d67700 1 Processor -- start 2023-11-16T16:40:43.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.911+0000 7f41b9d67700 1 -- start start 2023-11-16T16:40:43.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.911+0000 7f41b9d67700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41b4105950 0x7f41b407bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:43.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.911+0000 7f41b9d67700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b407c150 con 0x7f41b4105950 2023-11-16T16:40:43.973 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.912+0000 7f41b37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41b4105950 0x7f41b407bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:43.973 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.912+0000 7f41b37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41b4105950 0x7f41b407bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52414/0 (socket says 172.21.15.44:52414) 2023-11-16T16:40:43.973 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.912+0000 7f41b37fe700 1 -- 172.21.15.44:0/2150504800 learned_addr learned my addr 172.21.15.44:0/2150504800 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:43.973 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.912+0000 7f41b37fe700 1 -- 172.21.15.44:0/2150504800 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a4009550 con 0x7f41b4105950 2023-11-16T16:40:43.973 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.912+0000 7f41b37fe700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41b4105950 0x7f41b407bc10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f41a4011980 tx=0x7f41a4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:43.973 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.912+0000 7f41b0ff9700 1 -- 172.21.15.44:0/2150504800 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41a4003a60 con 0x7f41b4105950 2023-11-16T16:40:43.973 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.913+0000 7f41b0ff9700 1 -- 172.21.15.44:0/2150504800 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41a4003bc0 con 0x7f41b4105950 2023-11-16T16:40:43.973 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.913+0000 7f41b0ff9700 1 -- 172.21.15.44:0/2150504800 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41a401a670 con 0x7f41b4105950 2023-11-16T16:40:43.973 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.913+0000 7f41b9d67700 1 -- 172.21.15.44:0/2150504800 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b41079e0 con 0x7f41b4105950 2023-11-16T16:40:43.973 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.913+0000 7f41b9d67700 1 -- 172.21.15.44:0/2150504800 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b4107850 con 0x7f41b4105950 2023-11-16T16:40:43.973 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.914+0000 7f41b0ff9700 1 -- 172.21.15.44:0/2150504800 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f41a4024a00 con 0x7f41b4105950 2023-11-16T16:40:43.973 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.914+0000 7f41b0ff9700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f419c038870 0x7f419c03ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:43.973 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.914+0000 7f41b0ff9700 1 -- 172.21.15.44:0/2150504800 --> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f419c03b430 con 0x7f419c038870 2023-11-16T16:40:43.974 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.914+0000 7f41b0ff9700 1 -- 172.21.15.44:0/2150504800 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f41a404c870 con 0x7f41b4105950 2023-11-16T16:40:43.974 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.914+0000 7f41b2ffd700 1 -- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f419c038870 msgr2=0x7f419c03ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4234382393 2023-11-16T16:40:43.974 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.915+0000 7f41b2ffd700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f419c038870 0x7f419c03ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:40:43.974 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.116+0000 7f41b2ffd700 1 -- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f419c038870 msgr2=0x7f419c03ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4234382393 2023-11-16T16:40:43.974 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.116+0000 7f41b2ffd700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f419c038870 0x7f419c03ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T16:40:43.974 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.517+0000 7f41b2ffd700 1 -- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f419c038870 msgr2=0x7f419c03ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4234382393 2023-11-16T16:40:43.974 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.517+0000 7f41b2ffd700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f419c038870 0x7f419c03ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-16T16:40:43.974 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.318+0000 7f41b2ffd700 1 -- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f419c038870 msgr2=0x7f419c03ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4234382393 2023-11-16T16:40:43.974 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.318+0000 7f41b2ffd700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f419c038870 0x7f419c03ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-16T16:40:43.974 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:38.920+0000 7f41b2ffd700 1 -- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f419c038870 msgr2=0x7f419c03ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4234382393 2023-11-16T16:40:43.974 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:38.920+0000 7f41b2ffd700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f419c038870 0x7f419c03ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-16T16:40:43.974 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:42.123+0000 7f41b2ffd700 1 -- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f419c038870 msgr2=0x7f419c03ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4234382393 2023-11-16T16:40:43.975 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:42.123+0000 7f41b2ffd700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f419c038870 0x7f419c03ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-16T16:40:43.975 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:42.451+0000 7f41b0ff9700 1 -- 172.21.15.44:0/2150504800 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7f41a4024620 con 0x7f41b4105950 2023-11-16T16:40:43.975 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:42.451+0000 7f41b0ff9700 1 -- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f419c038870 msgr2=0x7f419c03ad20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:40:43.975 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:42.451+0000 7f41b0ff9700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f419c038870 0x7f419c03ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:43.975 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.453+0000 7f41b0ff9700 1 -- 172.21.15.44:0/2150504800 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f41a404e350 con 0x7f41b4105950 2023-11-16T16:40:43.975 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.454+0000 7f41b0ff9700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f419c03c1a0 0x7f419c03e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:43.975 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.454+0000 7f41b0ff9700 1 -- 172.21.15.44:0/2150504800 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f419c03b430 con 0x7f419c03c1a0 2023-11-16T16:40:43.975 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.454+0000 7f41b2ffd700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f419c03c1a0 0x7f419c03e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:43.975 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.456+0000 7f41b2ffd700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f419c03c1a0 0x7f419c03e580 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f41a8003a10 tx=0x7f41a8009270 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:43.976 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.457+0000 7f41b0ff9700 1 -- 172.21.15.44:0/2150504800 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f419c03b430 con 0x7f419c03c1a0 2023-11-16T16:40:43.976 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.464+0000 7f41b9d67700 1 -- 172.21.15.44:0/2150504800 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f41b407b7b0 con 0x7f419c03c1a0 2023-11-16T16:40:43.976 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.464+0000 7f41b0ff9700 1 -- 172.21.15.44:0/2150504800 <== mgr.14150 v2:172.21.15.44:6800/304422349 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f41b407b7b0 con 0x7f419c03c1a0 2023-11-16T16:40:43.976 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.464+0000 7f41b9d67700 1 -- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f419c03c1a0 msgr2=0x7f419c03e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:43.976 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.464+0000 7f41b9d67700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f419c03c1a0 0x7f419c03e580 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f41a8003a10 tx=0x7f41a8009270 comp rx=0 tx=0).stop 2023-11-16T16:40:43.976 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.464+0000 7f41b9d67700 1 -- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41b4105950 msgr2=0x7f41b407bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:43.976 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.464+0000 7f41b9d67700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41b4105950 0x7f41b407bc10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f41a4011980 tx=0x7f41a4011a60 comp rx=0 tx=0).stop 2023-11-16T16:40:43.976 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.465+0000 7f41b9d67700 1 -- 172.21.15.44:0/2150504800 shutdown_connections 2023-11-16T16:40:43.976 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.465+0000 7f41b9d67700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f419c03c1a0 0x7f419c03e580 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:43.976 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.465+0000 7f41b9d67700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41b4105950 0x7f41b407bc10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:43.976 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.465+0000 7f41b9d67700 1 --2- 172.21.15.44:0/2150504800 >> [v2:172.21.15.44:6800/4234382393,v1:172.21.15.44:6801/4234382393] conn(0x7f419c038870 0x7f419c03ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:43.976 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.465+0000 7f41b9d67700 1 -- 172.21.15.44:0/2150504800 >> 172.21.15.44:0/2150504800 conn(0x7f41b4100fe0 msgr2=0x7f41b4101cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:43.977 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.465+0000 7f41b9d67700 1 -- 172.21.15.44:0/2150504800 shutdown_connections 2023-11-16T16:40:43.977 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.465+0000 7f41b9d67700 1 -- 172.21.15.44:0/2150504800 wait complete. 2023-11-16T16:40:43.977 INFO:teuthology.orchestra.run.smithi044.stdout:mgr epoch 9 is available 2023-11-16T16:40:43.977 INFO:teuthology.orchestra.run.smithi044.stdout:Generating a dashboard self-signed certificate... 2023-11-16T16:40:44.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:44 smithi044 ceph-mon[128750]: [16/Nov/2023:16:40:43] ENGINE Bus STARTING 2023-11-16T16:40:44.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:44 smithi044 ceph-mon[128750]: [16/Nov/2023:16:40:43] ENGINE Serving on https://172.21.15.44:7150 2023-11-16T16:40:44.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:44 smithi044 ceph-mon[128750]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-16T16:40:44.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:44 smithi044 ceph-mon[128750]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-16T16:40:44.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:44 smithi044 ceph-mon[128750]: [16/Nov/2023:16:40:43] ENGINE Serving on http://172.21.15.44:8765 2023-11-16T16:40:44.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:44 smithi044 ceph-mon[128750]: [16/Nov/2023:16:40:43] ENGINE Bus STARTED 2023-11-16T16:40:45.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-11-16T16:40:45.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.585+0000 7fd88de0d700 1 Processor -- start 2023-11-16T16:40:45.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.586+0000 7fd88de0d700 1 -- start start 2023-11-16T16:40:45.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.586+0000 7fd88de0d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88807bd80 0x7fd88807a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:45.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.586+0000 7fd88de0d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88807a7c0 con 0x7fd88807bd80 2023-11-16T16:40:45.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.586+0000 7fd8877fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88807bd80 0x7fd88807a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:45.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.586+0000 7fd8877fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88807bd80 0x7fd88807a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52460/0 (socket says 172.21.15.44:52460) 2023-11-16T16:40:45.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.586+0000 7fd8877fe700 1 -- 172.21.15.44:0/1734630818 learned_addr learned my addr 172.21.15.44:0/1734630818 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:45.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.587+0000 7fd8877fe700 1 -- 172.21.15.44:0/1734630818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88807a900 con 0x7fd88807bd80 2023-11-16T16:40:45.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.587+0000 7fd8877fe700 1 --2- 172.21.15.44:0/1734630818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88807bd80 0x7fd88807a280 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd8780098a0 tx=0x7fd878009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e695d76955a26b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:45.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.587+0000 7fd8867fc700 1 -- 172.21.15.44:0/1734630818 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd878004160 con 0x7fd88807bd80 2023-11-16T16:40:45.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.587+0000 7fd8867fc700 1 -- 172.21.15.44:0/1734630818 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8780042c0 con 0x7fd88807bd80 2023-11-16T16:40:45.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.587+0000 7fd8867fc700 1 -- 172.21.15.44:0/1734630818 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd8780119b0 con 0x7fd88807bd80 2023-11-16T16:40:45.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.588+0000 7fd88de0d700 1 -- 172.21.15.44:0/1734630818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88807bd80 msgr2=0x7fd88807a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:45.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.588+0000 7fd88de0d700 1 --2- 172.21.15.44:0/1734630818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88807bd80 0x7fd88807a280 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd8780098a0 tx=0x7fd878009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:45.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.588+0000 7fd88de0d700 1 -- 172.21.15.44:0/1734630818 shutdown_connections 2023-11-16T16:40:45.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.588+0000 7fd88de0d700 1 --2- 172.21.15.44:0/1734630818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88807bd80 0x7fd88807a280 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:45.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.588+0000 7fd88de0d700 1 -- 172.21.15.44:0/1734630818 >> 172.21.15.44:0/1734630818 conn(0x7fd888100fe0 msgr2=0x7fd8881033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:45.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.588+0000 7fd88de0d700 1 -- 172.21.15.44:0/1734630818 shutdown_connections 2023-11-16T16:40:45.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.588+0000 7fd88de0d700 1 -- 172.21.15.44:0/1734630818 wait complete. 2023-11-16T16:40:45.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.589+0000 7fd88de0d700 1 Processor -- start 2023-11-16T16:40:45.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.589+0000 7fd88de0d700 1 -- start start 2023-11-16T16:40:45.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.589+0000 7fd88de0d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88807bd80 0x7fd88819bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:45.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.590+0000 7fd88de0d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88819c040 con 0x7fd88807bd80 2023-11-16T16:40:45.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.590+0000 7fd8877fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88807bd80 0x7fd88819bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:45.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.590+0000 7fd8877fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88807bd80 0x7fd88819bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52472/0 (socket says 172.21.15.44:52472) 2023-11-16T16:40:45.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.590+0000 7fd8877fe700 1 -- 172.21.15.44:0/3892669522 learned_addr learned my addr 172.21.15.44:0/3892669522 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:45.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.590+0000 7fd8877fe700 1 -- 172.21.15.44:0/3892669522 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd878009550 con 0x7fd88807bd80 2023-11-16T16:40:45.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.590+0000 7fd8877fe700 1 --2- 172.21.15.44:0/3892669522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88807bd80 0x7fd88819bb00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd878000c00 tx=0x7fd878011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:45.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.591+0000 7fd884ff9700 1 -- 172.21.15.44:0/3892669522 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd8780039f0 con 0x7fd88807bd80 2023-11-16T16:40:45.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.591+0000 7fd884ff9700 1 -- 172.21.15.44:0/3892669522 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd878003b50 con 0x7fd88807bd80 2023-11-16T16:40:45.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.591+0000 7fd884ff9700 1 -- 172.21.15.44:0/3892669522 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd87801a780 con 0x7fd88807bd80 2023-11-16T16:40:45.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.591+0000 7fd88de0d700 1 -- 172.21.15.44:0/3892669522 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd88810be50 con 0x7fd88807bd80 2023-11-16T16:40:45.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.591+0000 7fd88de0d700 1 -- 172.21.15.44:0/3892669522 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd88810bcc0 con 0x7fd88807bd80 2023-11-16T16:40:45.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.592+0000 7fd88de0d700 1 -- 172.21.15.44:0/3892669522 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd88810bfe0 con 0x7fd88807bd80 2023-11-16T16:40:45.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.592+0000 7fd884ff9700 1 -- 172.21.15.44:0/3892669522 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fd878024440 con 0x7fd88807bd80 2023-11-16T16:40:45.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.592+0000 7fd884ff9700 1 --2- 172.21.15.44:0/3892669522 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd870038870 0x7fd87003ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:45.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.592+0000 7fd884ff9700 1 -- 172.21.15.44:0/3892669522 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd87804c940 con 0x7fd88807bd80 2023-11-16T16:40:45.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.593+0000 7fd886ffd700 1 --2- 172.21.15.44:0/3892669522 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd870038870 0x7fd87003ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:45.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.593+0000 7fd886ffd700 1 --2- 172.21.15.44:0/3892669522 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd870038870 0x7fd87003ad20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd87c006fd0 tx=0x7fd87c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:45.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.597+0000 7fd884ff9700 1 -- 172.21.15.44:0/3892669522 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd878018db0 con 0x7fd88807bd80 2023-11-16T16:40:45.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.833+0000 7fd88de0d700 1 -- 172.21.15.44:0/3892669522 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8881024e0 con 0x7fd870038870 2023-11-16T16:40:45.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.906+0000 7fd884ff9700 1 -- 172.21.15.44:0/3892669522 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fd8881024e0 con 0x7fd870038870 2023-11-16T16:40:45.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.908+0000 7fd88de0d700 1 -- 172.21.15.44:0/3892669522 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd870038870 msgr2=0x7fd87003ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:45.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.908+0000 7fd88de0d700 1 --2- 172.21.15.44:0/3892669522 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd870038870 0x7fd87003ad20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd87c006fd0 tx=0x7fd87c006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:45.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.908+0000 7fd88de0d700 1 -- 172.21.15.44:0/3892669522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88807bd80 msgr2=0x7fd88819bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:45.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.908+0000 7fd88de0d700 1 --2- 172.21.15.44:0/3892669522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88807bd80 0x7fd88819bb00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd878000c00 tx=0x7fd878011a30 comp rx=0 tx=0).stop 2023-11-16T16:40:45.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.909+0000 7fd88de0d700 1 -- 172.21.15.44:0/3892669522 shutdown_connections 2023-11-16T16:40:45.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.909+0000 7fd88de0d700 1 --2- 172.21.15.44:0/3892669522 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd870038870 0x7fd87003ad20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:45.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.909+0000 7fd88de0d700 1 --2- 172.21.15.44:0/3892669522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88807bd80 0x7fd88819bb00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:45.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.909+0000 7fd88de0d700 1 -- 172.21.15.44:0/3892669522 >> 172.21.15.44:0/3892669522 conn(0x7fd888100fe0 msgr2=0x7fd888101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:45.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.909+0000 7fd88de0d700 1 -- 172.21.15.44:0/3892669522 shutdown_connections 2023-11-16T16:40:45.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.909+0000 7fd88de0d700 1 -- 172.21.15.44:0/3892669522 wait complete. 2023-11-16T16:40:45.360 INFO:teuthology.orchestra.run.smithi044.stdout:Creating initial admin user... 2023-11-16T16:40:45.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:45 smithi044 ceph-mon[128750]: mgrmap e12: x(active, since 2s) 2023-11-16T16:40:45.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:40:45.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:40:45.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:40:46.841 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$P2LSfiB2D5NBQXCAc6C79.cBC2dKhNFurTg8kG3AVMdUaEVMTcg/q", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700152846, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-16T16:40:46.841 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.947+0000 7f6f7b66e700 1 Processor -- start 2023-11-16T16:40:46.841 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.948+0000 7f6f7b66e700 1 -- start start 2023-11-16T16:40:46.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.948+0000 7f6f7b66e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7407bda0 0x7f6f7407a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:46.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.948+0000 7f6f7b66e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7407a7e0 con 0x7f6f7407bda0 2023-11-16T16:40:46.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.948+0000 7f6f7940a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7407bda0 0x7f6f7407a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:46.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.948+0000 7f6f7940a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7407bda0 0x7f6f7407a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49656/0 (socket says 172.21.15.44:49656) 2023-11-16T16:40:46.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.948+0000 7f6f7940a700 1 -- 172.21.15.44:0/357798178 learned_addr learned my addr 172.21.15.44:0/357798178 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:46.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.949+0000 7f6f7940a700 1 -- 172.21.15.44:0/357798178 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7407a920 con 0x7f6f7407bda0 2023-11-16T16:40:46.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.949+0000 7f6f7940a700 1 --2- 172.21.15.44:0/357798178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7407bda0 0x7f6f7407a2a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6f700098a0 tx=0x7f6f70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8537b93406680e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:46.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.949+0000 7f6f6bfff700 1 -- 172.21.15.44:0/357798178 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f70004160 con 0x7f6f7407bda0 2023-11-16T16:40:46.843 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.949+0000 7f6f6bfff700 1 -- 172.21.15.44:0/357798178 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f700042c0 con 0x7f6f7407bda0 2023-11-16T16:40:46.843 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.949+0000 7f6f6bfff700 1 -- 172.21.15.44:0/357798178 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f700119b0 con 0x7f6f7407bda0 2023-11-16T16:40:46.843 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.950+0000 7f6f7b66e700 1 -- 172.21.15.44:0/357798178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7407bda0 msgr2=0x7f6f7407a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:46.843 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.950+0000 7f6f7b66e700 1 --2- 172.21.15.44:0/357798178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7407bda0 0x7f6f7407a2a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6f700098a0 tx=0x7f6f70009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:46.843 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.950+0000 7f6f7b66e700 1 -- 172.21.15.44:0/357798178 shutdown_connections 2023-11-16T16:40:46.843 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.950+0000 7f6f7b66e700 1 --2- 172.21.15.44:0/357798178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7407bda0 0x7f6f7407a2a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:46.843 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.950+0000 7f6f7b66e700 1 -- 172.21.15.44:0/357798178 >> 172.21.15.44:0/357798178 conn(0x7f6f74100fe0 msgr2=0x7f6f741033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:46.843 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.950+0000 7f6f7b66e700 1 -- 172.21.15.44:0/357798178 shutdown_connections 2023-11-16T16:40:46.844 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.950+0000 7f6f7b66e700 1 -- 172.21.15.44:0/357798178 wait complete. 2023-11-16T16:40:46.844 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.952+0000 7f6f7b66e700 1 Processor -- start 2023-11-16T16:40:46.844 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.952+0000 7f6f7b66e700 1 -- start start 2023-11-16T16:40:46.844 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.952+0000 7f6f7b66e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7407bda0 0x7f6f7419bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:46.844 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.952+0000 7f6f7b66e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7419c060 con 0x7f6f7407bda0 2023-11-16T16:40:46.844 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.953+0000 7f6f7940a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7407bda0 0x7f6f7419bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:46.844 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.953+0000 7f6f7940a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7407bda0 0x7f6f7419bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49660/0 (socket says 172.21.15.44:49660) 2023-11-16T16:40:46.844 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.953+0000 7f6f7940a700 1 -- 172.21.15.44:0/3748683241 learned_addr learned my addr 172.21.15.44:0/3748683241 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:46.845 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.953+0000 7f6f7940a700 1 -- 172.21.15.44:0/3748683241 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f70009550 con 0x7f6f7407bda0 2023-11-16T16:40:46.845 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.953+0000 7f6f7940a700 1 --2- 172.21.15.44:0/3748683241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7407bda0 0x7f6f7419bb20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6f70000c00 tx=0x7f6f70011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:46.845 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.953+0000 7f6f6a7fc700 1 -- 172.21.15.44:0/3748683241 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f700039f0 con 0x7f6f7407bda0 2023-11-16T16:40:46.845 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.954+0000 7f6f6a7fc700 1 -- 172.21.15.44:0/3748683241 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f70003b50 con 0x7f6f7407bda0 2023-11-16T16:40:46.845 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.954+0000 7f6f7b66e700 1 -- 172.21.15.44:0/3748683241 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7410be90 con 0x7f6f7407bda0 2023-11-16T16:40:46.845 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.954+0000 7f6f6a7fc700 1 -- 172.21.15.44:0/3748683241 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f7001a780 con 0x7f6f7407bda0 2023-11-16T16:40:46.845 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.954+0000 7f6f7b66e700 1 -- 172.21.15.44:0/3748683241 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7410bd00 con 0x7f6f7407bda0 2023-11-16T16:40:46.846 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.955+0000 7f6f6a7fc700 1 -- 172.21.15.44:0/3748683241 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f6f7001aa40 con 0x7f6f7407bda0 2023-11-16T16:40:46.846 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.955+0000 7f6f7b66e700 1 -- 172.21.15.44:0/3748683241 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f7410c020 con 0x7f6f7407bda0 2023-11-16T16:40:46.846 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.955+0000 7f6f6a7fc700 1 --2- 172.21.15.44:0/3748683241 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6f60038870 0x7f6f6003ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:46.846 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.955+0000 7f6f6a7fc700 1 -- 172.21.15.44:0/3748683241 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6f7004cb70 con 0x7f6f7407bda0 2023-11-16T16:40:46.846 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.956+0000 7f6f78c09700 1 --2- 172.21.15.44:0/3748683241 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6f60038870 0x7f6f6003ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:46.846 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.956+0000 7f6f78c09700 1 --2- 172.21.15.44:0/3748683241 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6f60038870 0x7f6f6003ad20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6f64006fd0 tx=0x7f6f64006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:46.846 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.961+0000 7f6f6a7fc700 1 -- 172.21.15.44:0/3748683241 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6f7004ee60 con 0x7f6f7407bda0 2023-11-16T16:40:46.846 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.193+0000 7f6f7b66e700 1 -- 172.21.15.44:0/3748683241 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6f74061f40 con 0x7f6f60038870 2023-11-16T16:40:46.847 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.440+0000 7f6f6a7fc700 1 -- 172.21.15.44:0/3748683241 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f6f74061f40 con 0x7f6f60038870 2023-11-16T16:40:46.847 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.443+0000 7f6f7b66e700 1 -- 172.21.15.44:0/3748683241 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6f60038870 msgr2=0x7f6f6003ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:46.847 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.443+0000 7f6f7b66e700 1 --2- 172.21.15.44:0/3748683241 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6f60038870 0x7f6f6003ad20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6f64006fd0 tx=0x7f6f64006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:46.847 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.443+0000 7f6f7b66e700 1 -- 172.21.15.44:0/3748683241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7407bda0 msgr2=0x7f6f7419bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:46.847 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.443+0000 7f6f7b66e700 1 --2- 172.21.15.44:0/3748683241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7407bda0 0x7f6f7419bb20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6f70000c00 tx=0x7f6f70011a30 comp rx=0 tx=0).stop 2023-11-16T16:40:46.847 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.443+0000 7f6f7b66e700 1 -- 172.21.15.44:0/3748683241 shutdown_connections 2023-11-16T16:40:46.847 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.443+0000 7f6f7b66e700 1 --2- 172.21.15.44:0/3748683241 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6f60038870 0x7f6f6003ad20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:46.847 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.443+0000 7f6f7b66e700 1 --2- 172.21.15.44:0/3748683241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f7407bda0 0x7f6f7419bb20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:46.848 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.443+0000 7f6f7b66e700 1 -- 172.21.15.44:0/3748683241 >> 172.21.15.44:0/3748683241 conn(0x7f6f74100fe0 msgr2=0x7f6f74101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:46.848 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.443+0000 7f6f7b66e700 1 -- 172.21.15.44:0/3748683241 shutdown_connections 2023-11-16T16:40:46.848 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.443+0000 7f6f7b66e700 1 -- 172.21.15.44:0/3748683241 wait complete. 2023-11-16T16:40:46.848 INFO:teuthology.orchestra.run.smithi044.stdout:Fetching dashboard port number... 2023-11-16T16:40:46.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:46 smithi044 ceph-mon[128750]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:46.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:46 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:40:47.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:47 smithi044 ceph-mon[128750]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:47.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:40:47.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:47 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1287659489' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-16T16:40:48.126 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 8443 2023-11-16T16:40:48.126 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.336+0000 7f501d51e700 1 Processor -- start 2023-11-16T16:40:48.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.336+0000 7f501d51e700 1 -- start start 2023-11-16T16:40:48.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.336+0000 7f501d51e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f501807bdc0 0x7f501807a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:48.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.336+0000 7f501d51e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501807a800 con 0x7f501807bdc0 2023-11-16T16:40:48.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.337+0000 7f5016ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f501807bdc0 0x7f501807a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:48.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.337+0000 7f5016ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f501807bdc0 0x7f501807a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49666/0 (socket says 172.21.15.44:49666) 2023-11-16T16:40:48.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.337+0000 7f5016ffd700 1 -- 172.21.15.44:0/34887619 learned_addr learned my addr 172.21.15.44:0/34887619 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:48.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.337+0000 7f5016ffd700 1 -- 172.21.15.44:0/34887619 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501807a940 con 0x7f501807bdc0 2023-11-16T16:40:48.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.337+0000 7f5016ffd700 1 --2- 172.21.15.44:0/34887619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f501807bdc0 0x7f501807a2c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f50080098a0 tx=0x7f5008009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbdb347ff93d7ae2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:48.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.338+0000 7f5015ffb700 1 -- 172.21.15.44:0/34887619 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5008004160 con 0x7f501807bdc0 2023-11-16T16:40:48.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.338+0000 7f5015ffb700 1 -- 172.21.15.44:0/34887619 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50080042c0 con 0x7f501807bdc0 2023-11-16T16:40:48.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.338+0000 7f5015ffb700 1 -- 172.21.15.44:0/34887619 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50080119b0 con 0x7f501807bdc0 2023-11-16T16:40:48.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.338+0000 7f501d51e700 1 -- 172.21.15.44:0/34887619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f501807bdc0 msgr2=0x7f501807a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:48.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.338+0000 7f501d51e700 1 --2- 172.21.15.44:0/34887619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f501807bdc0 0x7f501807a2c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f50080098a0 tx=0x7f5008009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:48.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.338+0000 7f501d51e700 1 -- 172.21.15.44:0/34887619 shutdown_connections 2023-11-16T16:40:48.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.338+0000 7f501d51e700 1 --2- 172.21.15.44:0/34887619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f501807bdc0 0x7f501807a2c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:48.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.338+0000 7f501d51e700 1 -- 172.21.15.44:0/34887619 >> 172.21.15.44:0/34887619 conn(0x7f5018100fe0 msgr2=0x7f50181033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:48.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.338+0000 7f501d51e700 1 -- 172.21.15.44:0/34887619 shutdown_connections 2023-11-16T16:40:48.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.339+0000 7f501d51e700 1 -- 172.21.15.44:0/34887619 wait complete. 2023-11-16T16:40:48.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.339+0000 7f501d51e700 1 Processor -- start 2023-11-16T16:40:48.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.340+0000 7f501d51e700 1 -- start start 2023-11-16T16:40:48.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.340+0000 7f501d51e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f501807bdc0 0x7f501819bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:48.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.340+0000 7f501d51e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501819c070 con 0x7f501807bdc0 2023-11-16T16:40:48.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.340+0000 7f5016ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f501807bdc0 0x7f501819bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:48.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.340+0000 7f5016ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f501807bdc0 0x7f501819bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49670/0 (socket says 172.21.15.44:49670) 2023-11-16T16:40:48.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.340+0000 7f5016ffd700 1 -- 172.21.15.44:0/1287659489 learned_addr learned my addr 172.21.15.44:0/1287659489 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:48.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.340+0000 7f5016ffd700 1 -- 172.21.15.44:0/1287659489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5008009550 con 0x7f501807bdc0 2023-11-16T16:40:48.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.341+0000 7f5016ffd700 1 --2- 172.21.15.44:0/1287659489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f501807bdc0 0x7f501819bb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5008003e60 tx=0x7f5008011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:48.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.341+0000 7f4ffffff700 1 -- 172.21.15.44:0/1287659489 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5008003af0 con 0x7f501807bdc0 2023-11-16T16:40:48.130 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.341+0000 7f4ffffff700 1 -- 172.21.15.44:0/1287659489 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5008003c50 con 0x7f501807bdc0 2023-11-16T16:40:48.130 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.341+0000 7f501d51e700 1 -- 172.21.15.44:0/1287659489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f501810be70 con 0x7f501807bdc0 2023-11-16T16:40:48.130 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.341+0000 7f4ffffff700 1 -- 172.21.15.44:0/1287659489 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5008024800 con 0x7f501807bdc0 2023-11-16T16:40:48.130 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.341+0000 7f501d51e700 1 -- 172.21.15.44:0/1287659489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f501810bce0 con 0x7f501807bdc0 2023-11-16T16:40:48.130 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.342+0000 7f4ffffff700 1 -- 172.21.15.44:0/1287659489 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f500801a430 con 0x7f501807bdc0 2023-11-16T16:40:48.130 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.342+0000 7f501d51e700 1 -- 172.21.15.44:0/1287659489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501810c000 con 0x7f501807bdc0 2023-11-16T16:40:48.130 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.343+0000 7f4ffffff700 1 --2- 172.21.15.44:0/1287659489 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5000038820 0x7f500003acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:48.130 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.343+0000 7f4ffffff700 1 -- 172.21.15.44:0/1287659489 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f500804cb50 con 0x7f501807bdc0 2023-11-16T16:40:48.130 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.343+0000 7f50167fc700 1 --2- 172.21.15.44:0/1287659489 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5000038820 0x7f500003acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:48.130 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.344+0000 7f50167fc700 1 --2- 172.21.15.44:0/1287659489 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5000038820 0x7f500003acd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f500c006fd0 tx=0x7f500c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:48.131 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.348+0000 7f4ffffff700 1 -- 172.21.15.44:0/1287659489 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5008018530 con 0x7f501807bdc0 2023-11-16T16:40:48.131 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.562+0000 7f501d51e700 1 -- 172.21.15.44:0/1287659489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f501819c5b0 con 0x7f501807bdc0 2023-11-16T16:40:48.131 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.563+0000 7f4ffffff700 1 -- 172.21.15.44:0/1287659489 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f5008018350 con 0x7f501807bdc0 2023-11-16T16:40:48.131 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.565+0000 7f501d51e700 1 -- 172.21.15.44:0/1287659489 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5000038820 msgr2=0x7f500003acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:48.131 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.565+0000 7f501d51e700 1 --2- 172.21.15.44:0/1287659489 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5000038820 0x7f500003acd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f500c006fd0 tx=0x7f500c006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:48.131 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.565+0000 7f501d51e700 1 -- 172.21.15.44:0/1287659489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f501807bdc0 msgr2=0x7f501819bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:48.131 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.565+0000 7f501d51e700 1 --2- 172.21.15.44:0/1287659489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f501807bdc0 0x7f501819bb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5008003e60 tx=0x7f5008011980 comp rx=0 tx=0).stop 2023-11-16T16:40:48.131 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.565+0000 7f501d51e700 1 -- 172.21.15.44:0/1287659489 shutdown_connections 2023-11-16T16:40:48.131 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.565+0000 7f501d51e700 1 --2- 172.21.15.44:0/1287659489 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5000038820 0x7f500003acd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:48.131 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.566+0000 7f501d51e700 1 --2- 172.21.15.44:0/1287659489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f501807bdc0 0x7f501819bb30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:48.132 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.566+0000 7f501d51e700 1 -- 172.21.15.44:0/1287659489 >> 172.21.15.44:0/1287659489 conn(0x7f5018100fe0 msgr2=0x7f5018101e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:48.132 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.566+0000 7f501d51e700 1 -- 172.21.15.44:0/1287659489 shutdown_connections 2023-11-16T16:40:48.132 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.566+0000 7f501d51e700 1 -- 172.21.15.44:0/1287659489 wait complete. 2023-11-16T16:40:48.149 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-16T16:40:48.149 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2023-11-16T16:40:48.167 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-16T16:40:48.167 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2023-11-16T16:40:48.167 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2023-11-16T16:40:48.167 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-16T16:40:48.170 INFO:teuthology.orchestra.run.smithi044.stdout:Ceph Dashboard is now available at: 2023-11-16T16:40:48.170 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:40:48.170 INFO:teuthology.orchestra.run.smithi044.stdout: URL: https://smithi044.front.sepia.ceph.com:8443/ 2023-11-16T16:40:48.170 INFO:teuthology.orchestra.run.smithi044.stdout: User: admin 2023-11-16T16:40:48.170 INFO:teuthology.orchestra.run.smithi044.stdout: Password: 9opwcamqkx 2023-11-16T16:40:48.170 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:40:48.170 INFO:teuthology.orchestra.run.smithi044.stdout:Saving cluster configuration to /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config directory 2023-11-16T16:40:48.171 INFO:teuthology.orchestra.run.smithi044.stdout:Enabling autotune for osd_memory_target 2023-11-16T16:40:49.562 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.925+0000 7fd1a3a01700 1 Processor -- start 2023-11-16T16:40:49.562 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.925+0000 7fd1a3a01700 1 -- start start 2023-11-16T16:40:49.562 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.925+0000 7fd1a3a01700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd19c07bd80 0x7fd19c07a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:49.562 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.925+0000 7fd1a3a01700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c07a7c0 con 0x7fd19c07bd80 2023-11-16T16:40:49.562 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.926+0000 7fd1a179d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd19c07bd80 0x7fd19c07a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:49.562 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.926+0000 7fd1a179d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd19c07bd80 0x7fd19c07a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49684/0 (socket says 172.21.15.44:49684) 2023-11-16T16:40:49.562 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.926+0000 7fd1a179d700 1 -- 172.21.15.44:0/3976673441 learned_addr learned my addr 172.21.15.44:0/3976673441 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:49.562 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.926+0000 7fd1a179d700 1 -- 172.21.15.44:0/3976673441 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c07a900 con 0x7fd19c07bd80 2023-11-16T16:40:49.562 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.926+0000 7fd1a179d700 1 --2- 172.21.15.44:0/3976673441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd19c07bd80 0x7fd19c07a280 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd1980098a0 tx=0x7fd198009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=142f26c37677955e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:49.562 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.926+0000 7fd193fff700 1 -- 172.21.15.44:0/3976673441 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd198004160 con 0x7fd19c07bd80 2023-11-16T16:40:49.563 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.926+0000 7fd193fff700 1 -- 172.21.15.44:0/3976673441 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1980042c0 con 0x7fd19c07bd80 2023-11-16T16:40:49.563 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.926+0000 7fd193fff700 1 -- 172.21.15.44:0/3976673441 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd1980119b0 con 0x7fd19c07bd80 2023-11-16T16:40:49.563 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.927+0000 7fd1a3a01700 1 -- 172.21.15.44:0/3976673441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd19c07bd80 msgr2=0x7fd19c07a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:49.563 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.927+0000 7fd1a3a01700 1 --2- 172.21.15.44:0/3976673441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd19c07bd80 0x7fd19c07a280 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd1980098a0 tx=0x7fd198009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:49.563 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.927+0000 7fd1a3a01700 1 -- 172.21.15.44:0/3976673441 shutdown_connections 2023-11-16T16:40:49.563 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.927+0000 7fd1a3a01700 1 --2- 172.21.15.44:0/3976673441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd19c07bd80 0x7fd19c07a280 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:49.563 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.927+0000 7fd1a3a01700 1 -- 172.21.15.44:0/3976673441 >> 172.21.15.44:0/3976673441 conn(0x7fd19c100fe0 msgr2=0x7fd19c1033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:49.563 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.927+0000 7fd1a3a01700 1 -- 172.21.15.44:0/3976673441 shutdown_connections 2023-11-16T16:40:49.563 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.927+0000 7fd1a3a01700 1 -- 172.21.15.44:0/3976673441 wait complete. 2023-11-16T16:40:49.563 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.928+0000 7fd1a3a01700 1 Processor -- start 2023-11-16T16:40:49.563 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.928+0000 7fd1a3a01700 1 -- start start 2023-11-16T16:40:49.563 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.929+0000 7fd1a3a01700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd19c07bd80 0x7fd19c19bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:49.564 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.929+0000 7fd1a3a01700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c19c040 con 0x7fd19c07bd80 2023-11-16T16:40:49.564 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.929+0000 7fd1a179d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd19c07bd80 0x7fd19c19bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:49.564 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.929+0000 7fd1a179d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd19c07bd80 0x7fd19c19bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49692/0 (socket says 172.21.15.44:49692) 2023-11-16T16:40:49.564 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.929+0000 7fd1a179d700 1 -- 172.21.15.44:0/2266264160 learned_addr learned my addr 172.21.15.44:0/2266264160 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:49.564 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.929+0000 7fd1a179d700 1 -- 172.21.15.44:0/2266264160 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd198009550 con 0x7fd19c07bd80 2023-11-16T16:40:49.564 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.930+0000 7fd1a179d700 1 --2- 172.21.15.44:0/2266264160 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd19c07bd80 0x7fd19c19bb00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd198000c00 tx=0x7fd198011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:49.564 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.930+0000 7fd1927fc700 1 -- 172.21.15.44:0/2266264160 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd1980039f0 con 0x7fd19c07bd80 2023-11-16T16:40:49.564 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.930+0000 7fd1927fc700 1 -- 172.21.15.44:0/2266264160 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd198003b50 con 0x7fd19c07bd80 2023-11-16T16:40:49.564 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.930+0000 7fd1a3a01700 1 -- 172.21.15.44:0/2266264160 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd19c10be50 con 0x7fd19c07bd80 2023-11-16T16:40:49.564 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.930+0000 7fd1927fc700 1 -- 172.21.15.44:0/2266264160 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd19801a780 con 0x7fd19c07bd80 2023-11-16T16:40:49.564 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.930+0000 7fd1a3a01700 1 -- 172.21.15.44:0/2266264160 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd19c10bcc0 con 0x7fd19c07bd80 2023-11-16T16:40:49.564 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.931+0000 7fd1927fc700 1 -- 172.21.15.44:0/2266264160 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fd198024440 con 0x7fd19c07bd80 2023-11-16T16:40:49.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.931+0000 7fd1a3a01700 1 -- 172.21.15.44:0/2266264160 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19c10bfe0 con 0x7fd19c07bd80 2023-11-16T16:40:49.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.932+0000 7fd1927fc700 1 --2- 172.21.15.44:0/2266264160 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd188038910 0x7fd18803adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:49.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.932+0000 7fd1927fc700 1 -- 172.21.15.44:0/2266264160 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd19804c9a0 con 0x7fd19c07bd80 2023-11-16T16:40:49.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.932+0000 7fd1a0f9c700 1 --2- 172.21.15.44:0/2266264160 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd188038910 0x7fd18803adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:49.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.932+0000 7fd1a0f9c700 1 --2- 172.21.15.44:0/2266264160 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd188038910 0x7fd18803adc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd18c006fd0 tx=0x7fd18c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:49.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.937+0000 7fd1927fc700 1 -- 172.21.15.44:0/2266264160 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd1980189a0 con 0x7fd19c07bd80 2023-11-16T16:40:49.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:49.146+0000 7fd1a3a01700 1 -- 172.21.15.44:0/2266264160 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fd19c19c580 con 0x7fd19c07bd80 2023-11-16T16:40:49.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:49.146+0000 7fd1927fc700 1 -- 172.21.15.44:0/2266264160 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fd198018350 con 0x7fd19c07bd80 2023-11-16T16:40:49.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:49.149+0000 7fd1a3a01700 1 -- 172.21.15.44:0/2266264160 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd188038910 msgr2=0x7fd18803adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:49.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:49.149+0000 7fd1a3a01700 1 --2- 172.21.15.44:0/2266264160 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd188038910 0x7fd18803adc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd18c006fd0 tx=0x7fd18c006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:49.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:49.149+0000 7fd1a3a01700 1 -- 172.21.15.44:0/2266264160 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd19c07bd80 msgr2=0x7fd19c19bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:49.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:49.149+0000 7fd1a3a01700 1 --2- 172.21.15.44:0/2266264160 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd19c07bd80 0x7fd19c19bb00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd198000c00 tx=0x7fd198011a30 comp rx=0 tx=0).stop 2023-11-16T16:40:49.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:49.149+0000 7fd1a3a01700 1 -- 172.21.15.44:0/2266264160 shutdown_connections 2023-11-16T16:40:49.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:49.149+0000 7fd1a3a01700 1 --2- 172.21.15.44:0/2266264160 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd188038910 0x7fd18803adc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:49.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:49.149+0000 7fd1a3a01700 1 --2- 172.21.15.44:0/2266264160 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd19c07bd80 0x7fd19c19bb00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:49.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:49.149+0000 7fd1a3a01700 1 -- 172.21.15.44:0/2266264160 >> 172.21.15.44:0/2266264160 conn(0x7fd19c100fe0 msgr2=0x7fd19c101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:49.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:49.149+0000 7fd1a3a01700 1 -- 172.21.15.44:0/2266264160 shutdown_connections 2023-11-16T16:40:49.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:49.149+0000 7fd1a3a01700 1 -- 172.21.15.44:0/2266264160 wait complete. 2023-11-16T16:40:49.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:49 smithi044 ceph-mon[128750]: mgrmap e13: x(active, since 6s) 2023-11-16T16:40:51.047 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.267+0000 7f4a1e1bb700 1 Processor -- start 2023-11-16T16:40:51.047 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.267+0000 7f4a1e1bb700 1 -- start start 2023-11-16T16:40:51.047 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.268+0000 7f4a1e1bb700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a1807bd80 0x7f4a1807a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:51.047 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.268+0000 7f4a1e1bb700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1807a7c0 con 0x7f4a1807bd80 2023-11-16T16:40:51.047 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.268+0000 7f4a177fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a1807bd80 0x7f4a1807a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:51.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.268+0000 7f4a177fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a1807bd80 0x7f4a1807a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49706/0 (socket says 172.21.15.44:49706) 2023-11-16T16:40:51.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.268+0000 7f4a177fe700 1 -- 172.21.15.44:0/1503707681 learned_addr learned my addr 172.21.15.44:0/1503707681 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:51.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.268+0000 7f4a177fe700 1 -- 172.21.15.44:0/1503707681 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1807a900 con 0x7f4a1807bd80 2023-11-16T16:40:51.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.269+0000 7f4a177fe700 1 --2- 172.21.15.44:0/1503707681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a1807bd80 0x7f4a1807a280 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4a080098a0 tx=0x7f4a08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30500912ebc80d59 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:51.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.269+0000 7f4a167fc700 1 -- 172.21.15.44:0/1503707681 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a08004160 con 0x7f4a1807bd80 2023-11-16T16:40:51.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.269+0000 7f4a167fc700 1 -- 172.21.15.44:0/1503707681 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a080042c0 con 0x7f4a1807bd80 2023-11-16T16:40:51.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.269+0000 7f4a167fc700 1 -- 172.21.15.44:0/1503707681 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a080119b0 con 0x7f4a1807bd80 2023-11-16T16:40:51.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.269+0000 7f4a1e1bb700 1 -- 172.21.15.44:0/1503707681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a1807bd80 msgr2=0x7f4a1807a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:51.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.269+0000 7f4a1e1bb700 1 --2- 172.21.15.44:0/1503707681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a1807bd80 0x7f4a1807a280 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4a080098a0 tx=0x7f4a08009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:51.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.270+0000 7f4a1e1bb700 1 -- 172.21.15.44:0/1503707681 shutdown_connections 2023-11-16T16:40:51.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.270+0000 7f4a1e1bb700 1 --2- 172.21.15.44:0/1503707681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a1807bd80 0x7f4a1807a280 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:51.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.270+0000 7f4a1e1bb700 1 -- 172.21.15.44:0/1503707681 >> 172.21.15.44:0/1503707681 conn(0x7f4a18100fe0 msgr2=0x7f4a181033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:51.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.270+0000 7f4a1e1bb700 1 -- 172.21.15.44:0/1503707681 shutdown_connections 2023-11-16T16:40:51.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.270+0000 7f4a1e1bb700 1 -- 172.21.15.44:0/1503707681 wait complete. 2023-11-16T16:40:51.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.271+0000 7f4a1e1bb700 1 Processor -- start 2023-11-16T16:40:51.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.271+0000 7f4a1e1bb700 1 -- start start 2023-11-16T16:40:51.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.271+0000 7f4a1e1bb700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a1807bd80 0x7f4a1819bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:51.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.271+0000 7f4a1e1bb700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1819c040 con 0x7f4a1807bd80 2023-11-16T16:40:51.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.271+0000 7f4a177fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a1807bd80 0x7f4a1819bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:51.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.271+0000 7f4a177fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a1807bd80 0x7f4a1819bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49712/0 (socket says 172.21.15.44:49712) 2023-11-16T16:40:51.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.271+0000 7f4a177fe700 1 -- 172.21.15.44:0/2696207203 learned_addr learned my addr 172.21.15.44:0/2696207203 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:51.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.272+0000 7f4a177fe700 1 -- 172.21.15.44:0/2696207203 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a08009550 con 0x7f4a1807bd80 2023-11-16T16:40:51.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.272+0000 7f4a177fe700 1 --2- 172.21.15.44:0/2696207203 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a1807bd80 0x7f4a1819bb00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4a08003e60 tx=0x7f4a08011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:51.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.272+0000 7f4a14ff9700 1 -- 172.21.15.44:0/2696207203 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a08003970 con 0x7f4a1807bd80 2023-11-16T16:40:51.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.272+0000 7f4a14ff9700 1 -- 172.21.15.44:0/2696207203 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a08003ad0 con 0x7f4a1807bd80 2023-11-16T16:40:51.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.272+0000 7f4a14ff9700 1 -- 172.21.15.44:0/2696207203 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a080195e0 con 0x7f4a1807bd80 2023-11-16T16:40:51.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.272+0000 7f4a1e1bb700 1 -- 172.21.15.44:0/2696207203 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a1810be50 con 0x7f4a1807bd80 2023-11-16T16:40:51.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.273+0000 7f4a1e1bb700 1 -- 172.21.15.44:0/2696207203 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1810bcc0 con 0x7f4a1807bd80 2023-11-16T16:40:51.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.273+0000 7f4a1e1bb700 1 -- 172.21.15.44:0/2696207203 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a1810bfe0 con 0x7f4a1807bd80 2023-11-16T16:40:51.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.273+0000 7f4a14ff9700 1 -- 172.21.15.44:0/2696207203 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f4a08019850 con 0x7f4a1807bd80 2023-11-16T16:40:51.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.274+0000 7f4a14ff9700 1 --2- 172.21.15.44:0/2696207203 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4a000388c0 0x7f4a0003ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:51.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.274+0000 7f4a14ff9700 1 -- 172.21.15.44:0/2696207203 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4a0804cab0 con 0x7f4a1807bd80 2023-11-16T16:40:51.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.274+0000 7f4a16ffd700 1 --2- 172.21.15.44:0/2696207203 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4a000388c0 0x7f4a0003ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:51.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.275+0000 7f4a16ffd700 1 --2- 172.21.15.44:0/2696207203 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4a000388c0 0x7f4a0003ad70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c006fd0 tx=0x7f4a0c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:51.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.279+0000 7f4a14ff9700 1 -- 172.21.15.44:0/2696207203 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4a08003c40 con 0x7f4a1807bd80 2023-11-16T16:40:51.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.598+0000 7f4a1e1bb700 1 -- 172.21.15.44:0/2696207203 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f4a1819c580 con 0x7f4a1807bd80 2023-11-16T16:40:51.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.600+0000 7f4a14ff9700 1 -- 172.21.15.44:0/2696207203 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) v1 ==== 153+0+0 (secure 0 0 0) 0x7f4a08023b40 con 0x7f4a1807bd80 2023-11-16T16:40:51.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.602+0000 7f4a1e1bb700 1 -- 172.21.15.44:0/2696207203 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4a000388c0 msgr2=0x7f4a0003ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:51.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.602+0000 7f4a1e1bb700 1 --2- 172.21.15.44:0/2696207203 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4a000388c0 0x7f4a0003ad70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c006fd0 tx=0x7f4a0c006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:51.051 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.602+0000 7f4a1e1bb700 1 -- 172.21.15.44:0/2696207203 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a1807bd80 msgr2=0x7f4a1819bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:51.051 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.602+0000 7f4a1e1bb700 1 --2- 172.21.15.44:0/2696207203 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a1807bd80 0x7f4a1819bb00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4a08003e60 tx=0x7f4a08011aa0 comp rx=0 tx=0).stop 2023-11-16T16:40:51.051 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.603+0000 7f4a1e1bb700 1 -- 172.21.15.44:0/2696207203 shutdown_connections 2023-11-16T16:40:51.051 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.603+0000 7f4a1e1bb700 1 --2- 172.21.15.44:0/2696207203 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4a000388c0 0x7f4a0003ad70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:51.051 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.603+0000 7f4a1e1bb700 1 --2- 172.21.15.44:0/2696207203 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a1807bd80 0x7f4a1819bb00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:51.051 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.603+0000 7f4a1e1bb700 1 -- 172.21.15.44:0/2696207203 >> 172.21.15.44:0/2696207203 conn(0x7f4a18100fe0 msgr2=0x7f4a18101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:51.051 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.603+0000 7f4a1e1bb700 1 -- 172.21.15.44:0/2696207203 shutdown_connections 2023-11-16T16:40:51.051 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.603+0000 7f4a1e1bb700 1 -- 172.21.15.44:0/2696207203 wait complete. 2023-11-16T16:40:51.051 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-11-16T16:40:51.051 INFO:teuthology.orchestra.run.smithi044.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-11-16T16:40:51.051 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:40:51.051 INFO:teuthology.orchestra.run.smithi044.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-16T16:40:51.052 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:40:51.052 INFO:teuthology.orchestra.run.smithi044.stdout:Or, if you are only running a single cluster on this host: 2023-11-16T16:40:51.052 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:40:51.052 INFO:teuthology.orchestra.run.smithi044.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-11-16T16:40:51.052 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:40:51.052 INFO:teuthology.orchestra.run.smithi044.stdout:Please consider enabling telemetry to help improve Ceph: 2023-11-16T16:40:51.052 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:40:51.052 INFO:teuthology.orchestra.run.smithi044.stdout: ceph telemetry on 2023-11-16T16:40:51.052 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:40:51.052 INFO:teuthology.orchestra.run.smithi044.stdout:For more information see: 2023-11-16T16:40:51.052 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:40:51.052 INFO:teuthology.orchestra.run.smithi044.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-11-16T16:40:51.052 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:40:51.053 INFO:teuthology.orchestra.run.smithi044.stdout:Bootstrap complete. 2023-11-16T16:40:51.107 INFO:tasks.cephadm:Fetching config... 2023-11-16T16:40:51.107 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:40:51.107 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-16T16:40:51.150 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-16T16:40:51.150 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:40:51.150 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-16T16:40:51.243 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-16T16:40:51.243 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:40:51.243 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/keyring of=/dev/stdout 2023-11-16T16:40:51.315 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-16T16:40:51.315 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:40:51.315 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-16T16:40:51.400 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-16T16:40:51.400 DEBUG:teuthology.orchestra.run.smithi044:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWqZCN0b0Ts1go+60x2zF0Bszy8UH/qk+2xX1G31+FkXNGOxYkyMfJ+INT2FXjfE162RKn6x1J4nGlRR5VUaqhq47tmvZ/1rUtvsY7gYXJjdSRjRsubyEsv2cQxERbqN9GkPiO0SaHXLvlN8pttMa/44G8k6E7IapkNJrYcA+AvlsToTa8RF85rwHU0U3aJFcY5W1fuan06nE1rxVPCuk/JBOyu7NzmQ+rUGYqk6Yt2mrxXsmtqgSevYBTn/bXIpmYLls5+8DSgYceDvcUjZiZ7s2UKSYNyh8EV7PlbZpf/ofBKSmXCPlH+1BXyHGG9K5T3Ho860BjWZW9KY/HJqHn7cd50LpZNv6yBT5yMomrXvOF4qEVibEQIhgXxr3yIghGrep93FX8znyzirsSoHQGm5/KcPY7BONLltPO42gbqDWqBpyffO2XPN8WPuMkfiEwOilzu05aVFyKuO3AMnDE3qx7Gl9FFrcFUoFN6GOu9Oz64O1+3CoVdUl6ze7EFtM= ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-16T16:40:51.557 INFO:teuthology.orchestra.run.smithi044.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWqZCN0b0Ts1go+60x2zF0Bszy8UH/qk+2xX1G31+FkXNGOxYkyMfJ+INT2FXjfE162RKn6x1J4nGlRR5VUaqhq47tmvZ/1rUtvsY7gYXJjdSRjRsubyEsv2cQxERbqN9GkPiO0SaHXLvlN8pttMa/44G8k6E7IapkNJrYcA+AvlsToTa8RF85rwHU0U3aJFcY5W1fuan06nE1rxVPCuk/JBOyu7NzmQ+rUGYqk6Yt2mrxXsmtqgSevYBTn/bXIpmYLls5+8DSgYceDvcUjZiZ7s2UKSYNyh8EV7PlbZpf/ofBKSmXCPlH+1BXyHGG9K5T3Ho860BjWZW9KY/HJqHn7cd50LpZNv6yBT5yMomrXvOF4qEVibEQIhgXxr3yIghGrep93FX8znyzirsSoHQGm5/KcPY7BONLltPO42gbqDWqBpyffO2XPN8WPuMkfiEwOilzu05aVFyKuO3AMnDE3qx7Gl9FFrcFUoFN6GOu9Oz64O1+3CoVdUl6ze7EFtM= ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:40:51.588 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWqZCN0b0Ts1go+60x2zF0Bszy8UH/qk+2xX1G31+FkXNGOxYkyMfJ+INT2FXjfE162RKn6x1J4nGlRR5VUaqhq47tmvZ/1rUtvsY7gYXJjdSRjRsubyEsv2cQxERbqN9GkPiO0SaHXLvlN8pttMa/44G8k6E7IapkNJrYcA+AvlsToTa8RF85rwHU0U3aJFcY5W1fuan06nE1rxVPCuk/JBOyu7NzmQ+rUGYqk6Yt2mrxXsmtqgSevYBTn/bXIpmYLls5+8DSgYceDvcUjZiZ7s2UKSYNyh8EV7PlbZpf/ofBKSmXCPlH+1BXyHGG9K5T3Ho860BjWZW9KY/HJqHn7cd50LpZNv6yBT5yMomrXvOF4qEVibEQIhgXxr3yIghGrep93FX8znyzirsSoHQGm5/KcPY7BONLltPO42gbqDWqBpyffO2XPN8WPuMkfiEwOilzu05aVFyKuO3AMnDE3qx7Gl9FFrcFUoFN6GOu9Oz64O1+3CoVdUl6ze7EFtM= ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-16T16:40:51.634 INFO:teuthology.orchestra.run.smithi077.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWqZCN0b0Ts1go+60x2zF0Bszy8UH/qk+2xX1G31+FkXNGOxYkyMfJ+INT2FXjfE162RKn6x1J4nGlRR5VUaqhq47tmvZ/1rUtvsY7gYXJjdSRjRsubyEsv2cQxERbqN9GkPiO0SaHXLvlN8pttMa/44G8k6E7IapkNJrYcA+AvlsToTa8RF85rwHU0U3aJFcY5W1fuan06nE1rxVPCuk/JBOyu7NzmQ+rUGYqk6Yt2mrxXsmtqgSevYBTn/bXIpmYLls5+8DSgYceDvcUjZiZ7s2UKSYNyh8EV7PlbZpf/ofBKSmXCPlH+1BXyHGG9K5T3Ho860BjWZW9KY/HJqHn7cd50LpZNv6yBT5yMomrXvOF4qEVibEQIhgXxr3yIghGrep93FX8znyzirsSoHQGm5/KcPY7BONLltPO42gbqDWqBpyffO2XPN8WPuMkfiEwOilzu05aVFyKuO3AMnDE3qx7Gl9FFrcFUoFN6GOu9Oz64O1+3CoVdUl6ze7EFtM= ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:40:51.647 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWqZCN0b0Ts1go+60x2zF0Bszy8UH/qk+2xX1G31+FkXNGOxYkyMfJ+INT2FXjfE162RKn6x1J4nGlRR5VUaqhq47tmvZ/1rUtvsY7gYXJjdSRjRsubyEsv2cQxERbqN9GkPiO0SaHXLvlN8pttMa/44G8k6E7IapkNJrYcA+AvlsToTa8RF85rwHU0U3aJFcY5W1fuan06nE1rxVPCuk/JBOyu7NzmQ+rUGYqk6Yt2mrxXsmtqgSevYBTn/bXIpmYLls5+8DSgYceDvcUjZiZ7s2UKSYNyh8EV7PlbZpf/ofBKSmXCPlH+1BXyHGG9K5T3Ho860BjWZW9KY/HJqHn7cd50LpZNv6yBT5yMomrXvOF4qEVibEQIhgXxr3yIghGrep93FX8znyzirsSoHQGm5/KcPY7BONLltPO42gbqDWqBpyffO2XPN8WPuMkfiEwOilzu05aVFyKuO3AMnDE3qx7Gl9FFrcFUoFN6GOu9Oz64O1+3CoVdUl6ze7EFtM= ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-16T16:40:51.693 INFO:teuthology.orchestra.run.smithi176.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWqZCN0b0Ts1go+60x2zF0Bszy8UH/qk+2xX1G31+FkXNGOxYkyMfJ+INT2FXjfE162RKn6x1J4nGlRR5VUaqhq47tmvZ/1rUtvsY7gYXJjdSRjRsubyEsv2cQxERbqN9GkPiO0SaHXLvlN8pttMa/44G8k6E7IapkNJrYcA+AvlsToTa8RF85rwHU0U3aJFcY5W1fuan06nE1rxVPCuk/JBOyu7NzmQ+rUGYqk6Yt2mrxXsmtqgSevYBTn/bXIpmYLls5+8DSgYceDvcUjZiZ7s2UKSYNyh8EV7PlbZpf/ofBKSmXCPlH+1BXyHGG9K5T3Ho860BjWZW9KY/HJqHn7cd50LpZNv6yBT5yMomrXvOF4qEVibEQIhgXxr3yIghGrep93FX8znyzirsSoHQGm5/KcPY7BONLltPO42gbqDWqBpyffO2XPN8WPuMkfiEwOilzu05aVFyKuO3AMnDE3qx7Gl9FFrcFUoFN6GOu9Oz64O1+3CoVdUl6ze7EFtM= ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:40:51.705 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-16T16:40:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:51 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2696207203' entity='client.admin' 2023-11-16T16:40:52.688 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:40:53.981 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:40:53.982 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:40:53.982 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi044", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:40:53.982 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:40:53.982 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:53.982 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:40:54.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.193+0000 7f14513e0700 1 -- 172.21.15.44:0/1417988662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f144c0ffd00 msgr2=0x7f144c100110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:54.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.193+0000 7f14513e0700 1 --2- 172.21.15.44:0/1417988662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f144c0ffd00 0x7f144c100110 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f143c009910 tx=0x7f143c009c20 comp rx=0 tx=0).stop 2023-11-16T16:40:54.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.193+0000 7f14513e0700 1 -- 172.21.15.44:0/1417988662 shutdown_connections 2023-11-16T16:40:54.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.193+0000 7f14513e0700 1 --2- 172.21.15.44:0/1417988662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f144c0ffd00 0x7f144c100110 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:54.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.193+0000 7f14513e0700 1 -- 172.21.15.44:0/1417988662 >> 172.21.15.44:0/1417988662 conn(0x7f144c0fb2b0 msgr2=0x7f144c0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:54.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.193+0000 7f14513e0700 1 -- 172.21.15.44:0/1417988662 shutdown_connections 2023-11-16T16:40:54.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.194+0000 7f14513e0700 1 -- 172.21.15.44:0/1417988662 wait complete. 2023-11-16T16:40:54.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.194+0000 7f14513e0700 1 Processor -- start 2023-11-16T16:40:54.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.195+0000 7f14513e0700 1 -- start start 2023-11-16T16:40:54.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.195+0000 7f14513e0700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f144c0ffd00 0x7f144c196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:54.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.195+0000 7f14513e0700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c197530 con 0x7f144c0ffd00 2023-11-16T16:40:54.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.195+0000 7f144affd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f144c0ffd00 0x7f144c196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:54.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.195+0000 7f144affd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f144c0ffd00 0x7f144c196ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49718/0 (socket says 172.21.15.44:49718) 2023-11-16T16:40:54.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.195+0000 7f144affd700 1 -- 172.21.15.44:0/3237880348 learned_addr learned my addr 172.21.15.44:0/3237880348 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:54.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.196+0000 7f144affd700 1 -- 172.21.15.44:0/3237880348 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143c0095f0 con 0x7f144c0ffd00 2023-11-16T16:40:54.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.196+0000 7f144affd700 1 --2- 172.21.15.44:0/3237880348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f144c0ffd00 0x7f144c196ff0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f143c006010 tx=0x7f143c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:54.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.196+0000 7f1433fff700 1 -- 172.21.15.44:0/3237880348 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f143c01d070 con 0x7f144c0ffd00 2023-11-16T16:40:54.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.196+0000 7f1433fff700 1 -- 172.21.15.44:0/3237880348 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f143c0044e0 con 0x7f144c0ffd00 2023-11-16T16:40:54.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.196+0000 7f14513e0700 1 -- 172.21.15.44:0/3237880348 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144c105bd0 con 0x7f144c0ffd00 2023-11-16T16:40:54.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.196+0000 7f1433fff700 1 -- 172.21.15.44:0/3237880348 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f143c0039b0 con 0x7f144c0ffd00 2023-11-16T16:40:54.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.196+0000 7f14513e0700 1 -- 172.21.15.44:0/3237880348 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144c105a40 con 0x7f144c0ffd00 2023-11-16T16:40:54.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.197+0000 7f1433fff700 1 -- 172.21.15.44:0/3237880348 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f143c00baf0 con 0x7f144c0ffd00 2023-11-16T16:40:54.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.197+0000 7f14513e0700 1 -- 172.21.15.44:0/3237880348 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f144c105d60 con 0x7f144c0ffd00 2023-11-16T16:40:54.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.198+0000 7f1433fff700 1 --2- 172.21.15.44:0/3237880348 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f14340388c0 0x7f143403ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:54.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.198+0000 7f1433fff700 1 -- 172.21.15.44:0/3237880348 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f143c04c790 con 0x7f144c0ffd00 2023-11-16T16:40:54.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.198+0000 7f144a7fc700 1 --2- 172.21.15.44:0/3237880348 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f14340388c0 0x7f143403ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:54.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.199+0000 7f144a7fc700 1 --2- 172.21.15.44:0/3237880348 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f14340388c0 0x7f143403ad70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1440006fd0 tx=0x7f1440006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:54.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.203+0000 7f1433fff700 1 -- 172.21.15.44:0/3237880348 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f143c02a950 con 0x7f144c0ffd00 2023-11-16T16:40:54.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.413+0000 7f14513e0700 1 -- 172.21.15.44:0/3237880348 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f144c190d00 con 0x7f144c0ffd00 2023-11-16T16:40:54.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.417+0000 7f1433fff700 1 -- 172.21.15.44:0/3237880348 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f143c02a360 con 0x7f144c0ffd00 2023-11-16T16:40:54.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.419+0000 7f14513e0700 1 -- 172.21.15.44:0/3237880348 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f14340388c0 msgr2=0x7f143403ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:54.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.419+0000 7f14513e0700 1 --2- 172.21.15.44:0/3237880348 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f14340388c0 0x7f143403ad70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1440006fd0 tx=0x7f1440006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:54.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.419+0000 7f14513e0700 1 -- 172.21.15.44:0/3237880348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f144c0ffd00 msgr2=0x7f144c196ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:54.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.419+0000 7f14513e0700 1 --2- 172.21.15.44:0/3237880348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f144c0ffd00 0x7f144c196ff0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f143c006010 tx=0x7f143c0057c0 comp rx=0 tx=0).stop 2023-11-16T16:40:54.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.419+0000 7f14513e0700 1 -- 172.21.15.44:0/3237880348 shutdown_connections 2023-11-16T16:40:54.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.419+0000 7f14513e0700 1 --2- 172.21.15.44:0/3237880348 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f14340388c0 0x7f143403ad70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:54.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.419+0000 7f14513e0700 1 --2- 172.21.15.44:0/3237880348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f144c0ffd00 0x7f144c196ff0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:54.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.419+0000 7f14513e0700 1 -- 172.21.15.44:0/3237880348 >> 172.21.15.44:0/3237880348 conn(0x7f144c0fb2b0 msgr2=0x7f144c0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:54.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.420+0000 7f14513e0700 1 -- 172.21.15.44:0/3237880348 shutdown_connections 2023-11-16T16:40:54.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:54.420+0000 7f14513e0700 1 -- 172.21.15.44:0/3237880348 wait complete. 2023-11-16T16:40:54.785 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-11-16T16:40:54.785 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-11-16T16:40:55.644 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:40:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:55 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3237880348' entity='client.admin' 2023-11-16T16:40:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:55.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:40:57.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.153+0000 7f4f265dc700 1 -- 172.21.15.44:0/712504123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f200ffd00 msgr2=0x7f4f20100110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:57.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.153+0000 7f4f265dc700 1 --2- 172.21.15.44:0/712504123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f200ffd00 0x7f4f20100110 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4f10009910 tx=0x7f4f10009c20 comp rx=0 tx=0).stop 2023-11-16T16:40:57.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.153+0000 7f4f265dc700 1 -- 172.21.15.44:0/712504123 shutdown_connections 2023-11-16T16:40:57.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.153+0000 7f4f265dc700 1 --2- 172.21.15.44:0/712504123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f200ffd00 0x7f4f20100110 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:57.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.153+0000 7f4f265dc700 1 -- 172.21.15.44:0/712504123 >> 172.21.15.44:0/712504123 conn(0x7f4f200fb2b0 msgr2=0x7f4f200fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:57.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.153+0000 7f4f265dc700 1 -- 172.21.15.44:0/712504123 shutdown_connections 2023-11-16T16:40:57.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.153+0000 7f4f265dc700 1 -- 172.21.15.44:0/712504123 wait complete. 2023-11-16T16:40:57.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.154+0000 7f4f265dc700 1 Processor -- start 2023-11-16T16:40:57.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.154+0000 7f4f265dc700 1 -- start start 2023-11-16T16:40:57.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.155+0000 7f4f265dc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f200ffd00 0x7f4f20196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:57.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.155+0000 7f4f265dc700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f20197530 con 0x7f4f200ffd00 2023-11-16T16:40:57.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.155+0000 7f4f1ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f200ffd00 0x7f4f20196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:57.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.155+0000 7f4f1ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f200ffd00 0x7f4f20196ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59140/0 (socket says 172.21.15.44:59140) 2023-11-16T16:40:57.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.155+0000 7f4f1ffff700 1 -- 172.21.15.44:0/4012898254 learned_addr learned my addr 172.21.15.44:0/4012898254 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:40:57.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.155+0000 7f4f1ffff700 1 -- 172.21.15.44:0/4012898254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f100095f0 con 0x7f4f200ffd00 2023-11-16T16:40:57.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.156+0000 7f4f1ffff700 1 --2- 172.21.15.44:0/4012898254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f200ffd00 0x7f4f20196ff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4f10004f40 tx=0x7f4f100057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:57.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.156+0000 7f4f1d7fa700 1 -- 172.21.15.44:0/4012898254 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f1001d070 con 0x7f4f200ffd00 2023-11-16T16:40:57.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.156+0000 7f4f1d7fa700 1 -- 172.21.15.44:0/4012898254 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f1000bc10 con 0x7f4f200ffd00 2023-11-16T16:40:57.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.156+0000 7f4f265dc700 1 -- 172.21.15.44:0/4012898254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f20105bd0 con 0x7f4f200ffd00 2023-11-16T16:40:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.156+0000 7f4f1d7fa700 1 -- 172.21.15.44:0/4012898254 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f10003a70 con 0x7f4f200ffd00 2023-11-16T16:40:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.156+0000 7f4f265dc700 1 -- 172.21.15.44:0/4012898254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f20105a40 con 0x7f4f200ffd00 2023-11-16T16:40:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.157+0000 7f4f1d7fa700 1 -- 172.21.15.44:0/4012898254 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f4f10003ce0 con 0x7f4f200ffd00 2023-11-16T16:40:57.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.157+0000 7f4f265dc700 1 -- 172.21.15.44:0/4012898254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f20105d60 con 0x7f4f200ffd00 2023-11-16T16:40:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.158+0000 7f4f1d7fa700 1 --2- 172.21.15.44:0/4012898254 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4f080388c0 0x7f4f0803ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.158+0000 7f4f1d7fa700 1 -- 172.21.15.44:0/4012898254 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4f1004c470 con 0x7f4f200ffd00 2023-11-16T16:40:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.158+0000 7f4f1f7fe700 1 --2- 172.21.15.44:0/4012898254 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4f080388c0 0x7f4f0803ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:57.165 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.158+0000 7f4f1f7fe700 1 --2- 172.21.15.44:0/4012898254 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4f080388c0 0x7f4f0803ad70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4f14006fd0 tx=0x7f4f14006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:57.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.162+0000 7f4f1d7fa700 1 -- 172.21.15.44:0/4012898254 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4f1002aba0 con 0x7f4f200ffd00 2023-11-16T16:40:57.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.372+0000 7f4f265dc700 1 -- 172.21.15.44:0/4012898254 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f20060f30 con 0x7f4f080388c0 2023-11-16T16:40:57.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.376+0000 7f4f1d7fa700 1 -- 172.21.15.44:0/4012898254 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4f20060f30 con 0x7f4f080388c0 2023-11-16T16:40:57.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.378+0000 7f4f265dc700 1 -- 172.21.15.44:0/4012898254 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4f080388c0 msgr2=0x7f4f0803ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:57.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.378+0000 7f4f265dc700 1 --2- 172.21.15.44:0/4012898254 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4f080388c0 0x7f4f0803ad70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4f14006fd0 tx=0x7f4f14006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:57.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.378+0000 7f4f265dc700 1 -- 172.21.15.44:0/4012898254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f200ffd00 msgr2=0x7f4f20196ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:57.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.378+0000 7f4f265dc700 1 --2- 172.21.15.44:0/4012898254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f200ffd00 0x7f4f20196ff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4f10004f40 tx=0x7f4f100057c0 comp rx=0 tx=0).stop 2023-11-16T16:40:57.382 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.378+0000 7f4f265dc700 1 -- 172.21.15.44:0/4012898254 shutdown_connections 2023-11-16T16:40:57.382 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.378+0000 7f4f265dc700 1 --2- 172.21.15.44:0/4012898254 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4f080388c0 0x7f4f0803ad70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:57.382 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.378+0000 7f4f265dc700 1 --2- 172.21.15.44:0/4012898254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f200ffd00 0x7f4f20196ff0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:57.382 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.378+0000 7f4f265dc700 1 -- 172.21.15.44:0/4012898254 >> 172.21.15.44:0/4012898254 conn(0x7f4f200fb2b0 msgr2=0x7f4f200fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:57.382 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.379+0000 7f4f265dc700 1 -- 172.21.15.44:0/4012898254 shutdown_connections 2023-11-16T16:40:57.383 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:40:57.379+0000 7f4f265dc700 1 -- 172.21.15.44:0/4012898254 wait complete. 2023-11-16T16:40:57.832 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi077 2023-11-16T16:40:57.833 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-16T16:40:57.833 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/etc/ceph/ceph.conf 2023-11-16T16:40:57.854 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-16T16:40:57.854 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:40:57.913 INFO:tasks.cephadm:Adding host smithi077 to orchestrator... 2023-11-16T16:40:57.914 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi077 2023-11-16T16:40:58.816 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:40:58.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:58 smithi044 ceph-mon[128750]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:58.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:58 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:40:58.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:58 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:58.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:58 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:40:58.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:58 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:40:58.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:58 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:40:58.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:58 smithi044 ceph-mon[128750]: Updating smithi044:/etc/ceph/ceph.conf 2023-11-16T16:40:59.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:59 smithi044 ceph-mon[128750]: Updating smithi044:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:40:59.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:40:59 smithi044 ceph-mon[128750]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:00.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.357+0000 7fc33c868700 1 -- 172.21.15.44:0/453216735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3380fff90 msgr2=0x7fc338100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:00.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.357+0000 7fc33c868700 1 --2- 172.21.15.44:0/453216735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3380fff90 0x7fc338100360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc328009910 tx=0x7fc328009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:00.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.357+0000 7fc33c868700 1 -- 172.21.15.44:0/453216735 shutdown_connections 2023-11-16T16:41:00.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.358+0000 7fc33c868700 1 --2- 172.21.15.44:0/453216735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3380fff90 0x7fc338100360 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:00.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.358+0000 7fc33c868700 1 -- 172.21.15.44:0/453216735 >> 172.21.15.44:0/453216735 conn(0x7fc3380fb850 msgr2=0x7fc3380fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:00.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.358+0000 7fc33c868700 1 -- 172.21.15.44:0/453216735 shutdown_connections 2023-11-16T16:41:00.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.358+0000 7fc33c868700 1 -- 172.21.15.44:0/453216735 wait complete. 2023-11-16T16:41:00.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.359+0000 7fc33c868700 1 Processor -- start 2023-11-16T16:41:00.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.359+0000 7fc33c868700 1 -- start start 2023-11-16T16:41:00.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.359+0000 7fc33c868700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3380fff90 0x7fc338072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:00.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.359+0000 7fc33c868700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc338074d70 con 0x7fc3380fff90 2023-11-16T16:41:00.362 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.359+0000 7fc33659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3380fff90 0x7fc338072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:00.362 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.359+0000 7fc33659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3380fff90 0x7fc338072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59144/0 (socket says 172.21.15.44:59144) 2023-11-16T16:41:00.362 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.359+0000 7fc33659c700 1 -- 172.21.15.44:0/635288765 learned_addr learned my addr 172.21.15.44:0/635288765 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:41:00.362 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.360+0000 7fc33659c700 1 -- 172.21.15.44:0/635288765 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3280095f0 con 0x7fc3380fff90 2023-11-16T16:41:00.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.360+0000 7fc33659c700 1 --2- 172.21.15.44:0/635288765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3380fff90 0x7fc338072df0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc328006010 tx=0x7fc328004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:00.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.360+0000 7fc3277fe700 1 -- 172.21.15.44:0/635288765 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc32801d070 con 0x7fc3380fff90 2023-11-16T16:41:00.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.360+0000 7fc3277fe700 1 -- 172.21.15.44:0/635288765 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc32800bc10 con 0x7fc3380fff90 2023-11-16T16:41:00.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.360+0000 7fc3277fe700 1 -- 172.21.15.44:0/635288765 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc328003910 con 0x7fc3380fff90 2023-11-16T16:41:00.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.360+0000 7fc33c868700 1 -- 172.21.15.44:0/635288765 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc33810a170 con 0x7fc3380fff90 2023-11-16T16:41:00.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.360+0000 7fc33c868700 1 -- 172.21.15.44:0/635288765 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc338109fe0 con 0x7fc3380fff90 2023-11-16T16:41:00.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.361+0000 7fc3277fe700 1 -- 172.21.15.44:0/635288765 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fc32800bd80 con 0x7fc3380fff90 2023-11-16T16:41:00.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.361+0000 7fc33c868700 1 -- 172.21.15.44:0/635288765 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33810a300 con 0x7fc3380fff90 2023-11-16T16:41:00.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.362+0000 7fc3277fe700 1 --2- 172.21.15.44:0/635288765 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc3200388c0 0x7fc32003ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:00.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.362+0000 7fc3277fe700 1 -- 172.21.15.44:0/635288765 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc32804c5a0 con 0x7fc3380fff90 2023-11-16T16:41:00.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.362+0000 7fc335d9b700 1 --2- 172.21.15.44:0/635288765 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc3200388c0 0x7fc32003ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:00.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.363+0000 7fc335d9b700 1 --2- 172.21.15.44:0/635288765 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc3200388c0 0x7fc32003ad70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc32c006fd0 tx=0x7fc32c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:00.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.367+0000 7fc3277fe700 1 -- 172.21.15.44:0/635288765 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc328029360 con 0x7fc3380fff90 2023-11-16T16:41:00.585 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:00.583+0000 7fc33c868700 1 -- 172.21.15.44:0/635288765 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi077", "target": ["mon-mgr", ""]}) v1 -- 0x7fc338074320 con 0x7fc3200388c0 2023-11-16T16:41:01.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:00 smithi044 ceph-mon[128750]: Updating smithi044:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:41:01.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:00 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:01.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:00 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:01.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:00 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:02.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:01 smithi044 ceph-mon[128750]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi077", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:02.764 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:02.762+0000 7fc3277fe700 1 -- 172.21.15.44:0/635288765 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fc338074320 con 0x7fc3200388c0 2023-11-16T16:41:02.764 INFO:teuthology.orchestra.run.smithi044.stdout:Added host 'smithi077' with addr '172.21.15.77' 2023-11-16T16:41:02.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:02.765+0000 7fc33c868700 1 -- 172.21.15.44:0/635288765 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc3200388c0 msgr2=0x7fc32003ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:02.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:02.765+0000 7fc33c868700 1 --2- 172.21.15.44:0/635288765 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc3200388c0 0x7fc32003ad70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc32c006fd0 tx=0x7fc32c006e40 comp rx=0 tx=0).stop 2023-11-16T16:41:02.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:02.765+0000 7fc33c868700 1 -- 172.21.15.44:0/635288765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3380fff90 msgr2=0x7fc338072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:02.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:02.765+0000 7fc33c868700 1 --2- 172.21.15.44:0/635288765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3380fff90 0x7fc338072df0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc328006010 tx=0x7fc328004ff0 comp rx=0 tx=0).stop 2023-11-16T16:41:02.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:02.765+0000 7fc33c868700 1 -- 172.21.15.44:0/635288765 shutdown_connections 2023-11-16T16:41:02.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:02.765+0000 7fc33c868700 1 --2- 172.21.15.44:0/635288765 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc3200388c0 0x7fc32003ad70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:02.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:02.765+0000 7fc33c868700 1 --2- 172.21.15.44:0/635288765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3380fff90 0x7fc338072df0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:02.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:02.765+0000 7fc33c868700 1 -- 172.21.15.44:0/635288765 >> 172.21.15.44:0/635288765 conn(0x7fc3380fb850 msgr2=0x7fc338104170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:02.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:02.765+0000 7fc33c868700 1 -- 172.21.15.44:0/635288765 shutdown_connections 2023-11-16T16:41:02.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:02.765+0000 7fc33c868700 1 -- 172.21.15.44:0/635288765 wait complete. 2023-11-16T16:41:02.959 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:02 smithi044 ceph-mon[128750]: Deploying cephadm binary to smithi077 2023-11-16T16:41:03.323 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-16T16:41:04.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:03 smithi044 ceph-mon[128750]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:04.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:03 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:04.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:03 smithi044 ceph-mon[128750]: Added host smithi077 2023-11-16T16:41:04.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:03 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:04.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:03 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:04.186 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:41:05.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:04 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:05.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:04 smithi044 ceph-mon[128750]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:05.683 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.680+0000 7f88c3818700 1 -- 172.21.15.44:0/625587194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc0fff90 msgr2=0x7f88bc100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:05.683 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.680+0000 7f88c3818700 1 --2- 172.21.15.44:0/625587194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc0fff90 0x7f88bc100360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f88b8009910 tx=0x7f88b8009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:05.683 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.681+0000 7f88c3818700 1 -- 172.21.15.44:0/625587194 shutdown_connections 2023-11-16T16:41:05.683 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.681+0000 7f88c3818700 1 --2- 172.21.15.44:0/625587194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc0fff90 0x7f88bc100360 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:05.683 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.681+0000 7f88c3818700 1 -- 172.21.15.44:0/625587194 >> 172.21.15.44:0/625587194 conn(0x7f88bc0fb850 msgr2=0x7f88bc0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.681+0000 7f88c3818700 1 -- 172.21.15.44:0/625587194 shutdown_connections 2023-11-16T16:41:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.681+0000 7f88c3818700 1 -- 172.21.15.44:0/625587194 wait complete. 2023-11-16T16:41:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.682+0000 7f88c3818700 1 Processor -- start 2023-11-16T16:41:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.682+0000 7f88c3818700 1 -- start start 2023-11-16T16:41:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.682+0000 7f88c3818700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc0fff90 0x7f88bc1973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.682+0000 7f88c3818700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc1978e0 con 0x7f88bc0fff90 2023-11-16T16:41:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.682+0000 7f88c15b4700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc0fff90 0x7f88bc1973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.682+0000 7f88c15b4700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc0fff90 0x7f88bc1973a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50516/0 (socket says 172.21.15.44:50516) 2023-11-16T16:41:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.682+0000 7f88c15b4700 1 -- 172.21.15.44:0/1085402980 learned_addr learned my addr 172.21.15.44:0/1085402980 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:41:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.683+0000 7f88c15b4700 1 -- 172.21.15.44:0/1085402980 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b80095f0 con 0x7f88bc0fff90 2023-11-16T16:41:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.683+0000 7f88c15b4700 1 --2- 172.21.15.44:0/1085402980 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc0fff90 0x7f88bc1973a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f88b8006010 tx=0x7f88b8004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.683+0000 7f88b27fc700 1 -- 172.21.15.44:0/1085402980 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f88b801d070 con 0x7f88bc0fff90 2023-11-16T16:41:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.683+0000 7f88b27fc700 1 -- 172.21.15.44:0/1085402980 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f88b800bc10 con 0x7f88bc0fff90 2023-11-16T16:41:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.683+0000 7f88b27fc700 1 -- 172.21.15.44:0/1085402980 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f88b8003910 con 0x7f88bc0fff90 2023-11-16T16:41:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.683+0000 7f88c3818700 1 -- 172.21.15.44:0/1085402980 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88bc10a170 con 0x7f88bc0fff90 2023-11-16T16:41:05.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.683+0000 7f88c3818700 1 -- 172.21.15.44:0/1085402980 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88bc109fe0 con 0x7f88bc0fff90 2023-11-16T16:41:05.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.684+0000 7f88c3818700 1 -- 172.21.15.44:0/1085402980 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88bc10a300 con 0x7f88bc0fff90 2023-11-16T16:41:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.684+0000 7f88b27fc700 1 -- 172.21.15.44:0/1085402980 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f88b800bd80 con 0x7f88bc0fff90 2023-11-16T16:41:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.685+0000 7f88b27fc700 1 --2- 172.21.15.44:0/1085402980 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f88a80388c0 0x7f88a803ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.685+0000 7f88b27fc700 1 -- 172.21.15.44:0/1085402980 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f88b804c5a0 con 0x7f88bc0fff90 2023-11-16T16:41:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.685+0000 7f88c0db3700 1 --2- 172.21.15.44:0/1085402980 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f88a80388c0 0x7f88a803ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.686+0000 7f88c0db3700 1 --2- 172.21.15.44:0/1085402980 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f88a80388c0 0x7f88a803ad70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f88ac006fd0 tx=0x7f88ac006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:05.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.690+0000 7f88b27fc700 1 -- 172.21.15.44:0/1085402980 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f88b8029360 con 0x7f88bc0fff90 2023-11-16T16:41:05.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.908+0000 7f88c3818700 1 -- 172.21.15.44:0/1085402980 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f88bc197ee0 con 0x7f88a80388c0 2023-11-16T16:41:05.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.909+0000 7f88b27fc700 1 -- 172.21.15.44:0/1085402980 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f88bc197ee0 con 0x7f88a80388c0 2023-11-16T16:41:05.912 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:41:05.912 INFO:teuthology.orchestra.run.smithi044.stdout:[{"addr": "172.21.15.44", "hostname": "smithi044", "labels": [], "status": ""}, {"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}] 2023-11-16T16:41:05.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.912+0000 7f88c3818700 1 -- 172.21.15.44:0/1085402980 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f88a80388c0 msgr2=0x7f88a803ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:05.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.912+0000 7f88c3818700 1 --2- 172.21.15.44:0/1085402980 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f88a80388c0 0x7f88a803ad70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f88ac006fd0 tx=0x7f88ac006e40 comp rx=0 tx=0).stop 2023-11-16T16:41:05.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.912+0000 7f88c3818700 1 -- 172.21.15.44:0/1085402980 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc0fff90 msgr2=0x7f88bc1973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:05.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.912+0000 7f88c3818700 1 --2- 172.21.15.44:0/1085402980 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc0fff90 0x7f88bc1973a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f88b8006010 tx=0x7f88b8004ff0 comp rx=0 tx=0).stop 2023-11-16T16:41:05.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.912+0000 7f88c3818700 1 -- 172.21.15.44:0/1085402980 shutdown_connections 2023-11-16T16:41:05.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.912+0000 7f88c3818700 1 --2- 172.21.15.44:0/1085402980 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f88a80388c0 0x7f88a803ad70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:05.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.912+0000 7f88c3818700 1 --2- 172.21.15.44:0/1085402980 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc0fff90 0x7f88bc1973a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:05.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.912+0000 7f88c3818700 1 -- 172.21.15.44:0/1085402980 >> 172.21.15.44:0/1085402980 conn(0x7f88bc0fb850 msgr2=0x7f88bc1038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:05.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.912+0000 7f88c3818700 1 -- 172.21.15.44:0/1085402980 shutdown_connections 2023-11-16T16:41:05.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:05.912+0000 7f88c3818700 1 -- 172.21.15.44:0/1085402980 wait complete. 2023-11-16T16:41:06.371 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi176 2023-11-16T16:41:06.371 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-11-16T16:41:06.371 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/etc/ceph/ceph.conf 2023-11-16T16:41:06.392 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-11-16T16:41:06.392 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:06.452 INFO:tasks.cephadm:Adding host smithi176 to orchestrator... 2023-11-16T16:41:06.452 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi176 2023-11-16T16:41:06.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:06 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:07.313 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:41:07.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:07 smithi044 ceph-mon[128750]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:41:07.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:07 smithi044 ceph-mon[128750]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:08.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.839+0000 7ff461c8c700 1 -- 172.21.15.44:0/4156129787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff45c072e10 msgr2=0x7ff45c0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:08.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.839+0000 7ff461c8c700 1 --2- 172.21.15.44:0/4156129787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff45c072e10 0x7ff45c0731e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff44c009910 tx=0x7ff44c009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:08.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.839+0000 7ff461c8c700 1 -- 172.21.15.44:0/4156129787 shutdown_connections 2023-11-16T16:41:08.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.839+0000 7ff461c8c700 1 --2- 172.21.15.44:0/4156129787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff45c072e10 0x7ff45c0731e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:08.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.839+0000 7ff461c8c700 1 -- 172.21.15.44:0/4156129787 >> 172.21.15.44:0/4156129787 conn(0x7ff45c0fb850 msgr2=0x7ff45c0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:08.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.839+0000 7ff461c8c700 1 -- 172.21.15.44:0/4156129787 shutdown_connections 2023-11-16T16:41:08.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.839+0000 7ff461c8c700 1 -- 172.21.15.44:0/4156129787 wait complete. 2023-11-16T16:41:08.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.840+0000 7ff461c8c700 1 Processor -- start 2023-11-16T16:41:08.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.840+0000 7ff461c8c700 1 -- start start 2023-11-16T16:41:08.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.841+0000 7ff461c8c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff45c072e10 0x7ff45c197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:08.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.841+0000 7ff461c8c700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c1978d0 con 0x7ff45c072e10 2023-11-16T16:41:08.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.841+0000 7ff45b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff45c072e10 0x7ff45c197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:08.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.841+0000 7ff45b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff45c072e10 0x7ff45c197390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50536/0 (socket says 172.21.15.44:50536) 2023-11-16T16:41:08.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.841+0000 7ff45b7fe700 1 -- 172.21.15.44:0/1124623985 learned_addr learned my addr 172.21.15.44:0/1124623985 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:41:08.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.841+0000 7ff45b7fe700 1 -- 172.21.15.44:0/1124623985 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44c0095f0 con 0x7ff45c072e10 2023-11-16T16:41:08.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.841+0000 7ff45b7fe700 1 --2- 172.21.15.44:0/1124623985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff45c072e10 0x7ff45c197390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff44c004f40 tx=0x7ff44c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:08.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.842+0000 7ff458ff9700 1 -- 172.21.15.44:0/1124623985 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff44c01d070 con 0x7ff45c072e10 2023-11-16T16:41:08.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.842+0000 7ff461c8c700 1 -- 172.21.15.44:0/1124623985 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45c10c370 con 0x7ff45c072e10 2023-11-16T16:41:08.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.842+0000 7ff458ff9700 1 -- 172.21.15.44:0/1124623985 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff44c004370 con 0x7ff45c072e10 2023-11-16T16:41:08.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.842+0000 7ff458ff9700 1 -- 172.21.15.44:0/1124623985 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff44c0039b0 con 0x7ff45c072e10 2023-11-16T16:41:08.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.842+0000 7ff461c8c700 1 -- 172.21.15.44:0/1124623985 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45c10c1e0 con 0x7ff45c072e10 2023-11-16T16:41:08.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.843+0000 7ff458ff9700 1 -- 172.21.15.44:0/1124623985 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7ff44c003c70 con 0x7ff45c072e10 2023-11-16T16:41:08.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.843+0000 7ff461c8c700 1 -- 172.21.15.44:0/1124623985 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff45c10c500 con 0x7ff45c072e10 2023-11-16T16:41:08.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.843+0000 7ff458ff9700 1 --2- 172.21.15.44:0/1124623985 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff444038870 0x7ff44403ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:08.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.843+0000 7ff458ff9700 1 -- 172.21.15.44:0/1124623985 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff44c04c5e0 con 0x7ff45c072e10 2023-11-16T16:41:08.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.844+0000 7ff45affd700 1 --2- 172.21.15.44:0/1124623985 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff444038870 0x7ff44403ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:08.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.844+0000 7ff45affd700 1 --2- 172.21.15.44:0/1124623985 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff444038870 0x7ff44403ad20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff450006fd0 tx=0x7ff450006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:08.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:08.851+0000 7ff458ff9700 1 -- 172.21.15.44:0/1124623985 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff44c02a980 con 0x7ff45c072e10 2023-11-16T16:41:08.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:41:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:08 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:41:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:08 smithi044 ceph-mon[128750]: Updating smithi077:/etc/ceph/ceph.conf 2023-11-16T16:41:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:08 smithi044 ceph-mon[128750]: Updating smithi077:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:08 smithi044 ceph-mon[128750]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:08 smithi044 ceph-mon[128750]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:08 smithi044 ceph-mon[128750]: Updating smithi077:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:41:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:08.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:09.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:09.063+0000 7ff461c8c700 1 -- 172.21.15.44:0/1124623985 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi176", "target": ["mon-mgr", ""]}) v1 -- 0x7ff45c109650 con 0x7ff444038870 2023-11-16T16:41:10.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:09 smithi044 ceph-mon[128750]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi176", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:10.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:09 smithi044 ceph-mon[128750]: Deploying cephadm binary to smithi176 2023-11-16T16:41:11.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:10 smithi044 ceph-mon[128750]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:11.202 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:11.200+0000 7ff458ff9700 1 -- 172.21.15.44:0/1124623985 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ff45c109650 con 0x7ff444038870 2023-11-16T16:41:11.202 INFO:teuthology.orchestra.run.smithi044.stdout:Added host 'smithi176' with addr '172.21.15.176' 2023-11-16T16:41:11.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:11.203+0000 7ff461c8c700 1 -- 172.21.15.44:0/1124623985 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff444038870 msgr2=0x7ff44403ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:11.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:11.203+0000 7ff461c8c700 1 --2- 172.21.15.44:0/1124623985 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff444038870 0x7ff44403ad20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff450006fd0 tx=0x7ff450006e40 comp rx=0 tx=0).stop 2023-11-16T16:41:11.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:11.203+0000 7ff461c8c700 1 -- 172.21.15.44:0/1124623985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff45c072e10 msgr2=0x7ff45c197390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:11.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:11.203+0000 7ff461c8c700 1 --2- 172.21.15.44:0/1124623985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff45c072e10 0x7ff45c197390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff44c004f40 tx=0x7ff44c005020 comp rx=0 tx=0).stop 2023-11-16T16:41:11.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:11.203+0000 7ff461c8c700 1 -- 172.21.15.44:0/1124623985 shutdown_connections 2023-11-16T16:41:11.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:11.203+0000 7ff461c8c700 1 --2- 172.21.15.44:0/1124623985 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff444038870 0x7ff44403ad20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:11.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:11.203+0000 7ff461c8c700 1 --2- 172.21.15.44:0/1124623985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff45c072e10 0x7ff45c197390 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:11.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:11.203+0000 7ff461c8c700 1 -- 172.21.15.44:0/1124623985 >> 172.21.15.44:0/1124623985 conn(0x7ff45c0fb850 msgr2=0x7ff45c1068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:11.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:11.203+0000 7ff461c8c700 1 -- 172.21.15.44:0/1124623985 shutdown_connections 2023-11-16T16:41:11.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:11.203+0000 7ff461c8c700 1 -- 172.21.15.44:0/1124623985 wait complete. 2023-11-16T16:41:11.704 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-16T16:41:12.564 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:41:12.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:12 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:12.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:12 smithi044 ceph-mon[128750]: Added host smithi176 2023-11-16T16:41:12.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:12 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:12.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:12 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:13.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:13 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:13.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:13 smithi044 ceph-mon[128750]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:14.090 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.089+0000 7f81094d7700 1 -- 172.21.15.44:0/2472183057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81040fff90 msgr2=0x7f8104100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:14.090 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.089+0000 7f81094d7700 1 --2- 172.21.15.44:0/2472183057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81040fff90 0x7f8104100360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009910 tx=0x7f80f4009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:14.090 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.089+0000 7f81094d7700 1 -- 172.21.15.44:0/2472183057 shutdown_connections 2023-11-16T16:41:14.090 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.089+0000 7f81094d7700 1 --2- 172.21.15.44:0/2472183057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81040fff90 0x7f8104100360 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:14.090 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.089+0000 7f81094d7700 1 -- 172.21.15.44:0/2472183057 >> 172.21.15.44:0/2472183057 conn(0x7f81040fb850 msgr2=0x7f81040fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:14.090 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.089+0000 7f81094d7700 1 -- 172.21.15.44:0/2472183057 shutdown_connections 2023-11-16T16:41:14.091 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.089+0000 7f81094d7700 1 -- 172.21.15.44:0/2472183057 wait complete. 2023-11-16T16:41:14.091 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.090+0000 7f81094d7700 1 Processor -- start 2023-11-16T16:41:14.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.090+0000 7f81094d7700 1 -- start start 2023-11-16T16:41:14.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.091+0000 7f81094d7700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81040fff90 0x7f81041973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:14.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.091+0000 7f81094d7700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81041978e0 con 0x7f81040fff90 2023-11-16T16:41:14.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.091+0000 7f8102ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81040fff90 0x7f81041973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:14.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.091+0000 7f8102ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81040fff90 0x7f81041973a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50554/0 (socket says 172.21.15.44:50554) 2023-11-16T16:41:14.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.091+0000 7f8102ffd700 1 -- 172.21.15.44:0/1786010007 learned_addr learned my addr 172.21.15.44:0/1786010007 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:41:14.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.091+0000 7f8102ffd700 1 -- 172.21.15.44:0/1786010007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f40095f0 con 0x7f81040fff90 2023-11-16T16:41:14.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.091+0000 7f8102ffd700 1 --2- 172.21.15.44:0/1786010007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81040fff90 0x7f81041973a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f80f4006010 tx=0x7f80f4004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:14.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.092+0000 7f80ebfff700 1 -- 172.21.15.44:0/1786010007 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80f401d070 con 0x7f81040fff90 2023-11-16T16:41:14.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.092+0000 7f80ebfff700 1 -- 172.21.15.44:0/1786010007 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80f400bc10 con 0x7f81040fff90 2023-11-16T16:41:14.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.092+0000 7f81094d7700 1 -- 172.21.15.44:0/1786010007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810410a170 con 0x7f81040fff90 2023-11-16T16:41:14.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.092+0000 7f80ebfff700 1 -- 172.21.15.44:0/1786010007 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80f4003910 con 0x7f81040fff90 2023-11-16T16:41:14.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.092+0000 7f81094d7700 1 -- 172.21.15.44:0/1786010007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8104109fe0 con 0x7f81040fff90 2023-11-16T16:41:14.094 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.093+0000 7f80ebfff700 1 -- 172.21.15.44:0/1786010007 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f80f4003b80 con 0x7f81040fff90 2023-11-16T16:41:14.094 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.093+0000 7f81094d7700 1 -- 172.21.15.44:0/1786010007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810410a300 con 0x7f81040fff90 2023-11-16T16:41:14.094 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.093+0000 7f80ebfff700 1 --2- 172.21.15.44:0/1786010007 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f80ec0388c0 0x7f80ec03ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:14.095 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.093+0000 7f80ebfff700 1 -- 172.21.15.44:0/1786010007 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f80f404c460 con 0x7f81040fff90 2023-11-16T16:41:14.095 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.094+0000 7f81027fc700 1 --2- 172.21.15.44:0/1786010007 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f80ec0388c0 0x7f80ec03ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:14.095 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.094+0000 7f81027fc700 1 --2- 172.21.15.44:0/1786010007 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f80ec0388c0 0x7f80ec03ad70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f80f8006fd0 tx=0x7f80f8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:14.100 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.099+0000 7f80ebfff700 1 -- 172.21.15.44:0/1786010007 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f80f400ba70 con 0x7f81040fff90 2023-11-16T16:41:14.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.310+0000 7f81094d7700 1 -- 172.21.15.44:0/1786010007 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8104197ee0 con 0x7f80ec0388c0 2023-11-16T16:41:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.312+0000 7f80ebfff700 1 -- 172.21.15.44:0/1786010007 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+239 (secure 0 0 0) 0x7f8104197ee0 con 0x7f80ec0388c0 2023-11-16T16:41:14.313 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:41:14.313 INFO:teuthology.orchestra.run.smithi044.stdout:[{"addr": "172.21.15.44", "hostname": "smithi044", "labels": [], "status": ""}, {"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}, {"addr": "172.21.15.176", "hostname": "smithi176", "labels": [], "status": ""}] 2023-11-16T16:41:14.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.314+0000 7f81094d7700 1 -- 172.21.15.44:0/1786010007 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f80ec0388c0 msgr2=0x7f80ec03ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:14.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.314+0000 7f81094d7700 1 --2- 172.21.15.44:0/1786010007 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f80ec0388c0 0x7f80ec03ad70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f80f8006fd0 tx=0x7f80f8006e40 comp rx=0 tx=0).stop 2023-11-16T16:41:14.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.314+0000 7f81094d7700 1 -- 172.21.15.44:0/1786010007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81040fff90 msgr2=0x7f81041973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:14.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.314+0000 7f81094d7700 1 --2- 172.21.15.44:0/1786010007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81040fff90 0x7f81041973a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f80f4006010 tx=0x7f80f4004ff0 comp rx=0 tx=0).stop 2023-11-16T16:41:14.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.314+0000 7f81094d7700 1 -- 172.21.15.44:0/1786010007 shutdown_connections 2023-11-16T16:41:14.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.314+0000 7f81094d7700 1 --2- 172.21.15.44:0/1786010007 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f80ec0388c0 0x7f80ec03ad70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:14.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.314+0000 7f81094d7700 1 --2- 172.21.15.44:0/1786010007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81040fff90 0x7f81041973a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:14.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.314+0000 7f81094d7700 1 -- 172.21.15.44:0/1786010007 >> 172.21.15.44:0/1786010007 conn(0x7f81040fb850 msgr2=0x7f81041038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:14.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.314+0000 7f81094d7700 1 -- 172.21.15.44:0/1786010007 shutdown_connections 2023-11-16T16:41:14.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:14.315+0000 7f81094d7700 1 -- 172.21.15.44:0/1786010007 wait complete. 2023-11-16T16:41:14.848 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-16T16:41:14.849 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd crush tunables default 2023-11-16T16:41:15.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:14 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:15.705 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:41:16.038 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:15 smithi044 ceph-mon[128750]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:41:16.038 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:15 smithi044 ceph-mon[128750]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:17.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.206+0000 7f9acd153700 1 -- 172.21.15.44:0/2721091679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ac80fff90 msgr2=0x7f9ac8100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:17.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.206+0000 7f9acd153700 1 --2- 172.21.15.44:0/2721091679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ac80fff90 0x7f9ac8100360 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8009910 tx=0x7f9ab8009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:17.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.207+0000 7f9acd153700 1 -- 172.21.15.44:0/2721091679 shutdown_connections 2023-11-16T16:41:17.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.207+0000 7f9acd153700 1 --2- 172.21.15.44:0/2721091679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ac80fff90 0x7f9ac8100360 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:17.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.207+0000 7f9acd153700 1 -- 172.21.15.44:0/2721091679 >> 172.21.15.44:0/2721091679 conn(0x7f9ac80fb850 msgr2=0x7f9ac80fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:17.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.207+0000 7f9acd153700 1 -- 172.21.15.44:0/2721091679 shutdown_connections 2023-11-16T16:41:17.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.207+0000 7f9acd153700 1 -- 172.21.15.44:0/2721091679 wait complete. 2023-11-16T16:41:17.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.208+0000 7f9acd153700 1 Processor -- start 2023-11-16T16:41:17.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.208+0000 7f9acd153700 1 -- start start 2023-11-16T16:41:17.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.208+0000 7f9acd153700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ac80fff90 0x7f9ac81951c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:17.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.208+0000 7f9acd153700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac8195700 con 0x7f9ac80fff90 2023-11-16T16:41:17.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.209+0000 7f9ac6d9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ac80fff90 0x7f9ac81951c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:17.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.209+0000 7f9ac6d9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ac80fff90 0x7f9ac81951c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53604/0 (socket says 172.21.15.44:53604) 2023-11-16T16:41:17.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.209+0000 7f9ac6d9d700 1 -- 172.21.15.44:0/3840079481 learned_addr learned my addr 172.21.15.44:0/3840079481 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:41:17.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.209+0000 7f9ac6d9d700 1 -- 172.21.15.44:0/3840079481 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab80095f0 con 0x7f9ac80fff90 2023-11-16T16:41:17.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.209+0000 7f9ac6d9d700 1 --2- 172.21.15.44:0/3840079481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ac80fff90 0x7f9ac81951c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8006010 tx=0x7f9ab8004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:17.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.209+0000 7f9aaffff700 1 -- 172.21.15.44:0/3840079481 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ab801d070 con 0x7f9ac80fff90 2023-11-16T16:41:17.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.209+0000 7f9aaffff700 1 -- 172.21.15.44:0/3840079481 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ab800bc10 con 0x7f9ac80fff90 2023-11-16T16:41:17.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.209+0000 7f9aaffff700 1 -- 172.21.15.44:0/3840079481 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ab8003910 con 0x7f9ac80fff90 2023-11-16T16:41:17.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.209+0000 7f9acd153700 1 -- 172.21.15.44:0/3840079481 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac810a170 con 0x7f9ac80fff90 2023-11-16T16:41:17.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.209+0000 7f9acd153700 1 -- 172.21.15.44:0/3840079481 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac8109fe0 con 0x7f9ac80fff90 2023-11-16T16:41:17.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.210+0000 7f9aaffff700 1 -- 172.21.15.44:0/3840079481 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f9ab800bd80 con 0x7f9ac80fff90 2023-11-16T16:41:17.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.210+0000 7f9acd153700 1 -- 172.21.15.44:0/3840079481 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac810a300 con 0x7f9ac80fff90 2023-11-16T16:41:17.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.211+0000 7f9aaffff700 1 --2- 172.21.15.44:0/3840079481 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9ab00388c0 0x7f9ab003ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:17.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.211+0000 7f9aaffff700 1 -- 172.21.15.44:0/3840079481 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9ab804c560 con 0x7f9ac80fff90 2023-11-16T16:41:17.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.211+0000 7f9ac659c700 1 --2- 172.21.15.44:0/3840079481 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9ab00388c0 0x7f9ab003ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:17.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.212+0000 7f9ac659c700 1 --2- 172.21.15.44:0/3840079481 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9ab00388c0 0x7f9ab003ad70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9abc006fd0 tx=0x7f9abc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:17.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.216+0000 7f9aaffff700 1 -- 172.21.15.44:0/3840079481 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9ab8029360 con 0x7f9ac80fff90 2023-11-16T16:41:17.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:17.454+0000 7f9acd153700 1 -- 172.21.15.44:0/3840079481 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f9ac8062120 con 0x7f9ac80fff90 2023-11-16T16:41:17.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:17.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:17.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:17.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:17.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:41:17.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:17.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:17.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:17 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:41:17.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:17 smithi044 ceph-mon[128750]: Updating smithi176:/etc/ceph/ceph.conf 2023-11-16T16:41:17.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:17 smithi044 ceph-mon[128750]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:17.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:17 smithi044 ceph-mon[128750]: Updating smithi176:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:17.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:17.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:17.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:18.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:18.358+0000 7f9aaffff700 1 -- 172.21.15.44:0/3840079481 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9ab8026020 con 0x7f9ac80fff90 2023-11-16T16:41:18.362 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:18.361+0000 7f9acd153700 1 -- 172.21.15.44:0/3840079481 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9ab00388c0 msgr2=0x7f9ab003ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:18.362 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:18.361+0000 7f9acd153700 1 --2- 172.21.15.44:0/3840079481 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9ab00388c0 0x7f9ab003ad70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9abc006fd0 tx=0x7f9abc006e40 comp rx=0 tx=0).stop 2023-11-16T16:41:18.362 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:18.361+0000 7f9acd153700 1 -- 172.21.15.44:0/3840079481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ac80fff90 msgr2=0x7f9ac81951c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:18.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:18.361+0000 7f9acd153700 1 --2- 172.21.15.44:0/3840079481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ac80fff90 0x7f9ac81951c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8006010 tx=0x7f9ab8004ff0 comp rx=0 tx=0).stop 2023-11-16T16:41:18.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:18.361+0000 7f9acd153700 1 -- 172.21.15.44:0/3840079481 shutdown_connections 2023-11-16T16:41:18.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:18.361+0000 7f9acd153700 1 --2- 172.21.15.44:0/3840079481 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9ab00388c0 0x7f9ab003ad70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:18.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:18.361+0000 7f9acd153700 1 --2- 172.21.15.44:0/3840079481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ac80fff90 0x7f9ac81951c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:18.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:18.362+0000 7f9acd153700 1 -- 172.21.15.44:0/3840079481 >> 172.21.15.44:0/3840079481 conn(0x7f9ac80fb850 msgr2=0x7f9ac8104170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:18.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:18.362+0000 7f9acd153700 1 -- 172.21.15.44:0/3840079481 shutdown_connections 2023-11-16T16:41:18.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:18.362+0000 7f9acd153700 1 -- 172.21.15.44:0/3840079481 wait complete. 2023-11-16T16:41:18.363 INFO:teuthology.orchestra.run.smithi044.stderr:adjusted tunables profile to default 2023-11-16T16:41:18.602 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:18 smithi044 ceph-mon[128750]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:18.602 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:18 smithi044 ceph-mon[128750]: Updating smithi176:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:41:18.602 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:18 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3840079481' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-16T16:41:18.894 INFO:tasks.cephadm:Adding mon.a on smithi044 2023-11-16T16:41:18.895 INFO:tasks.cephadm:Adding mon.b on smithi077 2023-11-16T16:41:18.895 INFO:tasks.cephadm:Adding mon.c on smithi176 2023-11-16T16:41:18.895 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch apply mon '3;smithi044:172.21.15.44=a;smithi077:172.21.15.77=b;smithi176:172.21.15.176=c' 2023-11-16T16:41:19.569 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:19.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:19 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3840079481' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-16T16:41:19.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:19 smithi044 ceph-mon[128750]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:19.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:19 smithi044 ceph-mon[128750]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:20.732 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.730+0000 7ff539664700 1 -- 172.21.15.176:0/1780092155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5340fff90 msgr2=0x7ff534100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:20.733 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.730+0000 7ff539664700 1 --2- 172.21.15.176:0/1780092155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5340fff90 0x7ff534100360 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff524009910 tx=0x7ff524009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:20.733 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.730+0000 7ff539664700 1 -- 172.21.15.176:0/1780092155 shutdown_connections 2023-11-16T16:41:20.733 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.730+0000 7ff539664700 1 --2- 172.21.15.176:0/1780092155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5340fff90 0x7ff534100360 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:20.733 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.730+0000 7ff539664700 1 -- 172.21.15.176:0/1780092155 >> 172.21.15.176:0/1780092155 conn(0x7ff5340fb870 msgr2=0x7ff5340fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:20.733 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.730+0000 7ff539664700 1 -- 172.21.15.176:0/1780092155 shutdown_connections 2023-11-16T16:41:20.733 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.730+0000 7ff539664700 1 -- 172.21.15.176:0/1780092155 wait complete. 2023-11-16T16:41:20.733 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.731+0000 7ff539664700 1 Processor -- start 2023-11-16T16:41:20.733 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.731+0000 7ff539664700 1 -- start start 2023-11-16T16:41:20.734 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.731+0000 7ff539664700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5340fff90 0x7ff534195160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:20.734 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.731+0000 7ff539664700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5341956a0 con 0x7ff5340fff90 2023-11-16T16:41:20.735 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.732+0000 7ff532ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5340fff90 0x7ff534195160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:20.735 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.732+0000 7ff532ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5340fff90 0x7ff534195160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.176:33166/0 (socket says 172.21.15.176:33166) 2023-11-16T16:41:20.735 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.732+0000 7ff532ffd700 1 -- 172.21.15.176:0/1206502509 learned_addr learned my addr 172.21.15.176:0/1206502509 (peer_addr_for_me v2:172.21.15.176:0/0) 2023-11-16T16:41:20.735 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.732+0000 7ff532ffd700 1 -- 172.21.15.176:0/1206502509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5240095f0 con 0x7ff5340fff90 2023-11-16T16:41:20.736 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.732+0000 7ff532ffd700 1 --2- 172.21.15.176:0/1206502509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5340fff90 0x7ff534195160 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff524004f40 tx=0x7ff524005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:20.736 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.733+0000 7ff51bfff700 1 -- 172.21.15.176:0/1206502509 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff52401d070 con 0x7ff5340fff90 2023-11-16T16:41:20.736 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.733+0000 7ff539664700 1 -- 172.21.15.176:0/1206502509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53410a170 con 0x7ff5340fff90 2023-11-16T16:41:20.736 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.733+0000 7ff51bfff700 1 -- 172.21.15.176:0/1206502509 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff524004370 con 0x7ff5340fff90 2023-11-16T16:41:20.736 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.733+0000 7ff539664700 1 -- 172.21.15.176:0/1206502509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff534109fe0 con 0x7ff5340fff90 2023-11-16T16:41:20.736 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.733+0000 7ff51bfff700 1 -- 172.21.15.176:0/1206502509 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5240039a0 con 0x7ff5340fff90 2023-11-16T16:41:20.736 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.734+0000 7ff51bfff700 1 -- 172.21.15.176:0/1206502509 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7ff524003b00 con 0x7ff5340fff90 2023-11-16T16:41:20.736 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.734+0000 7ff539664700 1 -- 172.21.15.176:0/1206502509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff53410a300 con 0x7ff5340fff90 2023-11-16T16:41:20.736 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.734+0000 7ff51bfff700 1 --2- 172.21.15.176:0/1206502509 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff51c038870 0x7ff51c03ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:20.736 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.734+0000 7ff51bfff700 1 -- 172.21.15.176:0/1206502509 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff52404c7a0 con 0x7ff5340fff90 2023-11-16T16:41:20.736 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.734+0000 7ff5327fc700 1 --2- 172.21.15.176:0/1206502509 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff51c038870 0x7ff51c03ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:20.737 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.735+0000 7ff5327fc700 1 --2- 172.21.15.176:0/1206502509 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff51c038870 0x7ff51c03ad20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff528006fd0 tx=0x7ff528006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:20.741 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.740+0000 7ff51bfff700 1 -- 172.21.15.176:0/1206502509 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff524005c00 con 0x7ff5340fff90 2023-11-16T16:41:20.936 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.935+0000 7ff539664700 1 -- 172.21.15.176:0/1206502509 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi044:172.21.15.44=a;smithi077:172.21.15.77=b;smithi176:172.21.15.176=c", "target": ["mon-mgr", ""]}) v1 -- 0x7ff534108190 con 0x7ff51c038870 2023-11-16T16:41:20.943 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.942+0000 7ff51bfff700 1 -- 172.21.15.176:0/1206502509 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff534108190 con 0x7ff51c038870 2023-11-16T16:41:20.944 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled mon update... 2023-11-16T16:41:20.945 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.944+0000 7ff539664700 1 -- 172.21.15.176:0/1206502509 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff51c038870 msgr2=0x7ff51c03ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:20.946 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.944+0000 7ff539664700 1 --2- 172.21.15.176:0/1206502509 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff51c038870 0x7ff51c03ad20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff528006fd0 tx=0x7ff528006e40 comp rx=0 tx=0).stop 2023-11-16T16:41:20.946 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.944+0000 7ff539664700 1 -- 172.21.15.176:0/1206502509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5340fff90 msgr2=0x7ff534195160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:20.946 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.944+0000 7ff539664700 1 --2- 172.21.15.176:0/1206502509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5340fff90 0x7ff534195160 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff524004f40 tx=0x7ff524005020 comp rx=0 tx=0).stop 2023-11-16T16:41:20.946 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.944+0000 7ff539664700 1 -- 172.21.15.176:0/1206502509 shutdown_connections 2023-11-16T16:41:20.946 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.944+0000 7ff539664700 1 --2- 172.21.15.176:0/1206502509 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff51c038870 0x7ff51c03ad20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:20.946 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.944+0000 7ff539664700 1 --2- 172.21.15.176:0/1206502509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5340fff90 0x7ff534195160 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:20.946 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.944+0000 7ff539664700 1 -- 172.21.15.176:0/1206502509 >> 172.21.15.176:0/1206502509 conn(0x7ff5340fb870 msgr2=0x7ff534104170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:20.946 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.944+0000 7ff539664700 1 -- 172.21.15.176:0/1206502509 shutdown_connections 2023-11-16T16:41:20.946 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:20.944+0000 7ff539664700 1 -- 172.21.15.176:0/1206502509 wait complete. 2023-11-16T16:41:21.321 DEBUG:teuthology.orchestra.run.smithi077:mon.b> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mon.b.service 2023-11-16T16:41:21.323 DEBUG:teuthology.orchestra.run.smithi176:mon.c> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mon.c.service 2023-11-16T16:41:21.325 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-11-16T16:41:21.326 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T16:41:21.354 INFO:journalctl@ceph.mon.c.smithi176.stdout:-- Logs begin at Thu 2023-11-16 16:14:40 UTC. -- 2023-11-16T16:41:21.358 INFO:journalctl@ceph.mon.b.smithi077.stdout:-- Logs begin at Thu 2023-11-16 16:15:33 UTC. -- 2023-11-16T16:41:21.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:21 smithi044 ceph-mon[128750]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:21.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:21 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:21.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:21 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:21.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:21 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:21.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:21 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:21.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:21 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:21.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:21 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:21.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:21 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:22.170 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:22.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:22 smithi044 ceph-mon[128750]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi044:172.21.15.44=a;smithi077:172.21.15.77=b;smithi176:172.21.15.176=c", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:22.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:22 smithi044 ceph-mon[128750]: Saving service mon spec with placement smithi044:172.21.15.44=a;smithi077:172.21.15.77=b;smithi176:172.21.15.176=c;count:3 2023-11-16T16:41:22.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:22 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:41:22.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:22 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:41:22.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:22 smithi044 ceph-mon[128750]: Deploying daemon mon.c on smithi176 2023-11-16T16:41:23.809 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.807+0000 7f3f9f7c2700 1 -- 172.21.15.176:0/2114908315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f98100df0 msgr2=0x7f3f981011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:23.809 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.807+0000 7f3f9f7c2700 1 --2- 172.21.15.176:0/2114908315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f98100df0 0x7f3f981011c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3f94009910 tx=0x7f3f94009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:23.809 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.807+0000 7f3f9f7c2700 1 -- 172.21.15.176:0/2114908315 shutdown_connections 2023-11-16T16:41:23.809 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.807+0000 7f3f9f7c2700 1 --2- 172.21.15.176:0/2114908315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f98100df0 0x7f3f981011c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:23.810 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.807+0000 7f3f9f7c2700 1 -- 172.21.15.176:0/2114908315 >> 172.21.15.176:0/2114908315 conn(0x7f3f98074930 msgr2=0x7f3f98074d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:23.810 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.807+0000 7f3f9f7c2700 1 -- 172.21.15.176:0/2114908315 shutdown_connections 2023-11-16T16:41:23.810 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.807+0000 7f3f9f7c2700 1 -- 172.21.15.176:0/2114908315 wait complete. 2023-11-16T16:41:23.810 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.808+0000 7f3f9f7c2700 1 Processor -- start 2023-11-16T16:41:23.810 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.808+0000 7f3f9f7c2700 1 -- start start 2023-11-16T16:41:23.810 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.808+0000 7f3f9f7c2700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f98100df0 0x7f3f981951c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:23.810 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.808+0000 7f3f9f7c2700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f98195700 con 0x7f3f98100df0 2023-11-16T16:41:23.811 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.809+0000 7f3f9d55e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f98100df0 0x7f3f981951c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:23.811 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.809+0000 7f3f9d55e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f98100df0 0x7f3f981951c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.176:33172/0 (socket says 172.21.15.176:33172) 2023-11-16T16:41:23.811 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.809+0000 7f3f9d55e700 1 -- 172.21.15.176:0/3176756607 learned_addr learned my addr 172.21.15.176:0/3176756607 (peer_addr_for_me v2:172.21.15.176:0/0) 2023-11-16T16:41:23.812 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.809+0000 7f3f9d55e700 1 -- 172.21.15.176:0/3176756607 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f940095f0 con 0x7f3f98100df0 2023-11-16T16:41:23.812 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.809+0000 7f3f9d55e700 1 --2- 172.21.15.176:0/3176756607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f98100df0 0x7f3f981951c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3f94005850 tx=0x7f3f94004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:23.812 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.809+0000 7f3f8e7fc700 1 -- 172.21.15.176:0/3176756607 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f9401d070 con 0x7f3f98100df0 2023-11-16T16:41:23.812 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.809+0000 7f3f9f7c2700 1 -- 172.21.15.176:0/3176756607 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f9810a190 con 0x7f3f98100df0 2023-11-16T16:41:23.812 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.809+0000 7f3f8e7fc700 1 -- 172.21.15.176:0/3176756607 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f94004370 con 0x7f3f98100df0 2023-11-16T16:41:23.812 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.809+0000 7f3f8e7fc700 1 -- 172.21.15.176:0/3176756607 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f94003ab0 con 0x7f3f98100df0 2023-11-16T16:41:23.812 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.810+0000 7f3f9f7c2700 1 -- 172.21.15.176:0/3176756607 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f9810a000 con 0x7f3f98100df0 2023-11-16T16:41:23.812 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.810+0000 7f3f9f7c2700 1 -- 172.21.15.176:0/3176756607 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f9810a320 con 0x7f3f98100df0 2023-11-16T16:41:23.812 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.810+0000 7f3f8e7fc700 1 -- 172.21.15.176:0/3176756607 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f3f9400b6f0 con 0x7f3f98100df0 2023-11-16T16:41:23.813 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.811+0000 7f3f8e7fc700 1 --2- 172.21.15.176:0/3176756607 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3f84038870 0x7f3f8403ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:23.813 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.811+0000 7f3f8e7fc700 1 -- 172.21.15.176:0/3176756607 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3f9404c550 con 0x7f3f98100df0 2023-11-16T16:41:23.813 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.811+0000 7f3f9cd5d700 1 --2- 172.21.15.176:0/3176756607 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3f84038870 0x7f3f8403ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:23.813 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.812+0000 7f3f9cd5d700 1 --2- 172.21.15.176:0/3176756607 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3f84038870 0x7f3f8403ad20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3f88006fd0 tx=0x7f3f88006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:23.817 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:23.816+0000 7f3f8e7fc700 1 -- 172.21.15.176:0/3176756607 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3f9402a590 con 0x7f3f98100df0 2023-11-16T16:41:24.101 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:24.100+0000 7f3f9f7c2700 1 -- 172.21.15.176:0/3176756607 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f98107370 con 0x7f3f98100df0 2023-11-16T16:41:24.102 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:24.101+0000 7f3f8e7fc700 1 -- 172.21.15.176:0/3176756607 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7f3f94026030 con 0x7f3f98100df0 2023-11-16T16:41:24.102 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-11-16T16:41:24.102 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T16:39:38.269339Z","created":"2023-11-16T16:39:38.269339Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T16:41:24.104 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:24.103+0000 7f3f9f7c2700 1 -- 172.21.15.176:0/3176756607 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3f84038870 msgr2=0x7f3f8403ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:24.104 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:24.103+0000 7f3f9f7c2700 1 --2- 172.21.15.176:0/3176756607 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3f84038870 0x7f3f8403ad20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3f88006fd0 tx=0x7f3f88006e40 comp rx=0 tx=0).stop 2023-11-16T16:41:24.104 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:24.103+0000 7f3f9f7c2700 1 -- 172.21.15.176:0/3176756607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f98100df0 msgr2=0x7f3f981951c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:24.105 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:24.103+0000 7f3f9f7c2700 1 --2- 172.21.15.176:0/3176756607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f98100df0 0x7f3f981951c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3f94005850 tx=0x7f3f94004ff0 comp rx=0 tx=0).stop 2023-11-16T16:41:24.105 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:24.103+0000 7f3f9f7c2700 1 -- 172.21.15.176:0/3176756607 shutdown_connections 2023-11-16T16:41:24.105 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:24.103+0000 7f3f9f7c2700 1 --2- 172.21.15.176:0/3176756607 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3f84038870 0x7f3f8403ad20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:24.105 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:24.103+0000 7f3f9f7c2700 1 --2- 172.21.15.176:0/3176756607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f98100df0 0x7f3f981951c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:24.105 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:24.103+0000 7f3f9f7c2700 1 -- 172.21.15.176:0/3176756607 >> 172.21.15.176:0/3176756607 conn(0x7f3f98074930 msgr2=0x7f3f98104610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:24.105 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:24.103+0000 7f3f9f7c2700 1 -- 172.21.15.176:0/3176756607 shutdown_connections 2023-11-16T16:41:24.105 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:24.103+0000 7f3f9f7c2700 1 -- 172.21.15.176:0/3176756607 wait complete. 2023-11-16T16:41:24.105 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2023-11-16T16:41:24.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:23 smithi044 ceph-mon[128750]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:24.131 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:23 smithi176 systemd[1]: Starting Ceph mon.c for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:41:24.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 podman[131569]: 2023-11-16T16:41:24.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: set uid:gid to 167:167 (ceph:ceph) 2023-11-16T16:41:24.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev), process ceph-mon, pid 7 2023-11-16T16:41:24.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: pidfile_write: ignore empty --pid-file 2023-11-16T16:41:24.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: load: jerasure load: lrc 2023-11-16T16:41:24.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: RocksDB version: 7.9.2 2023-11-16T16:41:24.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Git sha 0 2023-11-16T16:41:24.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Compile date 2023-11-15 23:53:54 2023-11-16T16:41:24.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: DB SUMMARY 2023-11-16T16:41:24.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: DB Session ID: 69DQIQRWL3MF4K0VSASR 2023-11-16T16:41:24.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: CURRENT file: CURRENT 2023-11-16T16:41:24.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: IDENTITY file: IDENTITY 2023-11-16T16:41:24.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-11-16T16:41:24.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-11-16T16:41:24.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-11-16T16:41:24.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.error_if_exists: 0 2023-11-16T16:41:24.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.create_if_missing: 0 2023-11-16T16:41:24.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.paranoid_checks: 1 2023-11-16T16:41:24.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.flush_verify_memtable_count: 1 2023-11-16T16:41:24.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-16T16:41:24.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-11-16T16:41:24.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.env: 0x559bc0070720 2023-11-16T16:41:24.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.fs: PosixFileSystem 2023-11-16T16:41:24.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.info_log: 0x559bc33413c0 2023-11-16T16:41:24.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_file_opening_threads: 16 2023-11-16T16:41:24.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.statistics: (nil) 2023-11-16T16:41:24.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.use_fsync: 0 2023-11-16T16:41:24.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_log_file_size: 0 2023-11-16T16:41:24.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-16T16:41:24.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.log_file_time_to_roll: 0 2023-11-16T16:41:24.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.keep_log_file_num: 1000 2023-11-16T16:41:24.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.recycle_log_file_num: 0 2023-11-16T16:41:24.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.allow_fallocate: 1 2023-11-16T16:41:24.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.allow_mmap_reads: 0 2023-11-16T16:41:24.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.allow_mmap_writes: 0 2023-11-16T16:41:24.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.use_direct_reads: 0 2023-11-16T16:41:24.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-16T16:41:24.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.create_missing_column_families: 0 2023-11-16T16:41:24.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.db_log_dir: 2023-11-16T16:41:24.899 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.wal_dir: 2023-11-16T16:41:24.899 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.table_cache_numshardbits: 6 2023-11-16T16:41:24.899 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.WAL_ttl_seconds: 0 2023-11-16T16:41:24.899 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.WAL_size_limit_MB: 0 2023-11-16T16:41:24.899 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-16T16:41:24.899 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-16T16:41:24.899 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.is_fd_close_on_exec: 1 2023-11-16T16:41:24.899 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.advise_random_on_open: 1 2023-11-16T16:41:24.899 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.db_write_buffer_size: 0 2023-11-16T16:41:24.899 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.write_buffer_manager: 0x559bc25b23c0 2023-11-16T16:41:24.899 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-16T16:41:24.900 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-16T16:41:24.900 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.use_adaptive_mutex: 0 2023-11-16T16:41:24.900 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.rate_limiter: (nil) 2023-11-16T16:41:24.900 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-16T16:41:24.900 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.wal_recovery_mode: 2 2023-11-16T16:41:24.900 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.enable_thread_tracking: 0 2023-11-16T16:41:24.900 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.enable_pipelined_write: 0 2023-11-16T16:41:24.900 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.unordered_write: 0 2023-11-16T16:41:24.900 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-16T16:41:24.900 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-16T16:41:24.900 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-16T16:41:24.900 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-16T16:41:24.900 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.row_cache: None 2023-11-16T16:41:24.901 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.wal_filter: None 2023-11-16T16:41:24.901 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-16T16:41:24.901 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.allow_ingest_behind: 0 2023-11-16T16:41:24.901 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.two_write_queues: 0 2023-11-16T16:41:24.901 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.manual_wal_flush: 0 2023-11-16T16:41:24.901 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.wal_compression: 0 2023-11-16T16:41:24.901 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.atomic_flush: 0 2023-11-16T16:41:24.901 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-16T16:41:24.901 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.persist_stats_to_disk: 0 2023-11-16T16:41:24.901 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.write_dbid_to_manifest: 0 2023-11-16T16:41:24.901 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.log_readahead_size: 0 2023-11-16T16:41:24.901 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-16T16:41:24.901 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.best_efforts_recovery: 0 2023-11-16T16:41:24.901 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-16T16:41:24.901 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-16T16:41:24.902 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.allow_data_in_errors: 0 2023-11-16T16:41:24.902 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.db_host_id: __hostname__ 2023-11-16T16:41:24.902 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.enforce_single_del_contracts: true 2023-11-16T16:41:24.902 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_background_jobs: 2 2023-11-16T16:41:24.902 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_background_compactions: -1 2023-11-16T16:41:24.902 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_subcompactions: 1 2023-11-16T16:41:24.902 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-16T16:41:24.902 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-16T16:41:24.902 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.delayed_write_rate : 16777216 2023-11-16T16:41:24.902 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_total_wal_size: 0 2023-11-16T16:41:24.902 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-16T16:41:24.902 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.stats_dump_period_sec: 600 2023-11-16T16:41:24.902 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.stats_persist_period_sec: 600 2023-11-16T16:41:24.902 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-16T16:41:24.903 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_open_files: -1 2023-11-16T16:41:24.903 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.bytes_per_sync: 0 2023-11-16T16:41:24.903 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.wal_bytes_per_sync: 0 2023-11-16T16:41:24.903 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.strict_bytes_per_sync: 0 2023-11-16T16:41:24.903 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compaction_readahead_size: 0 2023-11-16T16:41:24.903 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_background_flushes: -1 2023-11-16T16:41:24.903 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Compression algorithms supported: 2023-11-16T16:41:24.903 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-16T16:41:24.903 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: kZSTD supported: 0 2023-11-16T16:41:24.903 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: kXpressCompression supported: 0 2023-11-16T16:41:24.903 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: kLZ4HCCompression supported: 1 2023-11-16T16:41:24.903 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: kZlibCompression supported: 1 2023-11-16T16:41:24.903 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: kSnappyCompression supported: 1 2023-11-16T16:41:24.903 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: kLZ4Compression supported: 1 2023-11-16T16:41:24.904 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: kBZip2Compression supported: 0 2023-11-16T16:41:24.904 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Fast CRC32 supported: Supported on x86 2023-11-16T16:41:24.904 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: DMutex implementation: pthread_mutex_t 2023-11-16T16:41:24.904 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-11-16T16:41:24.904 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-11-16T16:41:24.904 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-16T16:41:24.904 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.merge_operator: 2023-11-16T16:41:24.904 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compaction_filter: None 2023-11-16T16:41:24.904 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compaction_filter_factory: None 2023-11-16T16:41:24.904 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.sst_partitioner_factory: None 2023-11-16T16:41:24.904 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.memtable_factory: SkipListFactory 2023-11-16T16:41:24.904 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.table_factory: BlockBasedTable 2023-11-16T16:41:24.904 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559bc33414e0) 2023-11-16T16:41:24.904 INFO:journalctl@ceph.mon.c.smithi176.stdout: cache_index_and_filter_blocks: 1 2023-11-16T16:41:24.905 INFO:journalctl@ceph.mon.c.smithi176.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-16T16:41:24.905 INFO:journalctl@ceph.mon.c.smithi176.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-16T16:41:24.905 INFO:journalctl@ceph.mon.c.smithi176.stdout: pin_top_level_index_and_filter: 1 2023-11-16T16:41:24.905 INFO:journalctl@ceph.mon.c.smithi176.stdout: index_type: 0 2023-11-16T16:41:24.905 INFO:journalctl@ceph.mon.c.smithi176.stdout: data_block_index_type: 0 2023-11-16T16:41:24.905 INFO:journalctl@ceph.mon.c.smithi176.stdout: index_shortening: 1 2023-11-16T16:41:24.905 INFO:journalctl@ceph.mon.c.smithi176.stdout: data_block_hash_table_util_ratio: 0.750000 2023-11-16T16:41:24.905 INFO:journalctl@ceph.mon.c.smithi176.stdout: checksum: 4 2023-11-16T16:41:24.905 INFO:journalctl@ceph.mon.c.smithi176.stdout: no_block_cache: 0 2023-11-16T16:41:24.905 INFO:journalctl@ceph.mon.c.smithi176.stdout: block_cache: 0x559bc26531f0 2023-11-16T16:41:24.905 INFO:journalctl@ceph.mon.c.smithi176.stdout: block_cache_name: BinnedLRUCache 2023-11-16T16:41:24.905 INFO:journalctl@ceph.mon.c.smithi176.stdout: block_cache_options: 2023-11-16T16:41:24.905 INFO:journalctl@ceph.mon.c.smithi176.stdout: capacity : 536870912 2023-11-16T16:41:24.905 INFO:journalctl@ceph.mon.c.smithi176.stdout: num_shard_bits : 4 2023-11-16T16:41:24.905 INFO:journalctl@ceph.mon.c.smithi176.stdout: strict_capacity_limit : 0 2023-11-16T16:41:24.906 INFO:journalctl@ceph.mon.c.smithi176.stdout: high_pri_pool_ratio: 0.000 2023-11-16T16:41:24.906 INFO:journalctl@ceph.mon.c.smithi176.stdout: block_cache_compressed: (nil) 2023-11-16T16:41:24.906 INFO:journalctl@ceph.mon.c.smithi176.stdout: persistent_cache: (nil) 2023-11-16T16:41:24.906 INFO:journalctl@ceph.mon.c.smithi176.stdout: block_size: 4096 2023-11-16T16:41:24.906 INFO:journalctl@ceph.mon.c.smithi176.stdout: block_size_deviation: 10 2023-11-16T16:41:24.906 INFO:journalctl@ceph.mon.c.smithi176.stdout: block_restart_interval: 16 2023-11-16T16:41:24.906 INFO:journalctl@ceph.mon.c.smithi176.stdout: index_block_restart_interval: 1 2023-11-16T16:41:24.906 INFO:journalctl@ceph.mon.c.smithi176.stdout: metadata_block_size: 4096 2023-11-16T16:41:24.906 INFO:journalctl@ceph.mon.c.smithi176.stdout: partition_filters: 0 2023-11-16T16:41:24.906 INFO:journalctl@ceph.mon.c.smithi176.stdout: use_delta_encoding: 1 2023-11-16T16:41:24.906 INFO:journalctl@ceph.mon.c.smithi176.stdout: filter_policy: bloomfilter 2023-11-16T16:41:24.906 INFO:journalctl@ceph.mon.c.smithi176.stdout: whole_key_filtering: 1 2023-11-16T16:41:24.906 INFO:journalctl@ceph.mon.c.smithi176.stdout: verify_compression: 0 2023-11-16T16:41:24.906 INFO:journalctl@ceph.mon.c.smithi176.stdout: read_amp_bytes_per_bit: 0 2023-11-16T16:41:24.906 INFO:journalctl@ceph.mon.c.smithi176.stdout: format_version: 5 2023-11-16T16:41:24.907 INFO:journalctl@ceph.mon.c.smithi176.stdout: enable_index_compression: 1 2023-11-16T16:41:24.907 INFO:journalctl@ceph.mon.c.smithi176.stdout: block_align: 0 2023-11-16T16:41:24.907 INFO:journalctl@ceph.mon.c.smithi176.stdout: max_auto_readahead_size: 262144 2023-11-16T16:41:24.907 INFO:journalctl@ceph.mon.c.smithi176.stdout: prepopulate_block_cache: 0 2023-11-16T16:41:24.907 INFO:journalctl@ceph.mon.c.smithi176.stdout: initial_auto_readahead_size: 8192 2023-11-16T16:41:24.907 INFO:journalctl@ceph.mon.c.smithi176.stdout: num_file_reads_for_auto_readahead: 2 2023-11-16T16:41:24.907 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.write_buffer_size: 33554432 2023-11-16T16:41:24.907 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_write_buffer_number: 2 2023-11-16T16:41:24.907 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compression: NoCompression 2023-11-16T16:41:24.907 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.bottommost_compression: Disabled 2023-11-16T16:41:24.907 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.prefix_extractor: nullptr 2023-11-16T16:41:24.907 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-16T16:41:24.907 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.num_levels: 7 2023-11-16T16:41:24.907 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-16T16:41:24.907 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-16T16:41:24.908 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-16T16:41:24.908 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-16T16:41:24.908 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-16T16:41:24.908 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-16T16:41:24.908 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-16T16:41:24.909 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-16T16:41:24.909 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-16T16:41:24.909 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-16T16:41:24.909 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-11-16T16:41:24.909 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-11-16T16:41:24.909 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compression_opts.window_bits: -14 2023-11-16T16:41:24.909 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compression_opts.level: 32767 2023-11-16T16:41:24.909 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compression_opts.strategy: 0 2023-11-16T16:41:24.910 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-16T16:41:24.910 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-16T16:41:24.910 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-11-16T16:41:24.910 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-16T16:41:24.910 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compression_opts.enabled: false 2023-11-16T16:41:24.910 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-11-16T16:41:24.910 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-16T16:41:24.910 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-16T16:41:24.910 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-16T16:41:24.910 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.target_file_size_base: 67108864 2023-11-16T16:41:24.910 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.target_file_size_multiplier: 1 2023-11-16T16:41:24.910 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-16T16:41:24.911 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-16T16:41:24.911 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-16T16:41:24.911 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-16T16:41:24.911 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-16T16:41:24.911 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-16T16:41:24.911 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-16T16:41:24.911 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-16T16:41:24.911 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-16T16:41:24.911 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-16T16:41:24.911 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-16T16:41:24.911 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-16T16:41:24.911 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-11-16T16:41:24.913 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.arena_block_size: 1048576 2023-11-16T16:41:24.913 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-16T16:41:24.913 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-16T16:41:24.913 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.disable_auto_compactions: 0 2023-11-16T16:41:24.913 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-16T16:41:24.913 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-16T16:41:24.913 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-16T16:41:24.913 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-16T16:41:24.913 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-16T16:41:24.913 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-16T16:41:24.913 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-16T16:41:24.913 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-16T16:41:24.913 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-16T16:41:24.913 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-16T16:41:24.914 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.table_properties_collectors: 2023-11-16T16:41:24.914 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.inplace_update_support: 0 2023-11-16T16:41:24.914 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.inplace_update_num_locks: 10000 2023-11-16T16:41:24.914 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-16T16:41:24.914 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-16T16:41:24.914 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.memtable_huge_page_size: 0 2023-11-16T16:41:24.914 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.bloom_locality: 0 2023-11-16T16:41:24.914 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.max_successive_merges: 0 2023-11-16T16:41:24.914 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.optimize_filters_for_hits: 0 2023-11-16T16:41:24.914 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.paranoid_file_checks: 0 2023-11-16T16:41:24.914 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.force_consistency_checks: 1 2023-11-16T16:41:24.914 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.report_bg_io_stats: 0 2023-11-16T16:41:24.914 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.ttl: 2592000 2023-11-16T16:41:24.914 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.periodic_compaction_seconds: 0 2023-11-16T16:41:24.915 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-11-16T16:41:24.915 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-11-16T16:41:24.915 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.enable_blob_files: false 2023-11-16T16:41:24.915 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.min_blob_size: 0 2023-11-16T16:41:24.915 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.blob_file_size: 268435456 2023-11-16T16:41:24.915 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.blob_compression_type: NoCompression 2023-11-16T16:41:24.915 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.enable_blob_garbage_collection: false 2023-11-16T16:41:24.915 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-16T16:41:24.915 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-11-16T16:41:24.915 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-11-16T16:41:24.915 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.blob_file_starting_level: 0 2023-11-16T16:41:24.915 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-11-16T16:41:24.915 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-16T16:41:24.916 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-11-16T16:41:24.916 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 80d5add6-e124-431b-9b27-4853a8229fdf 2023-11-16T16:41:24.916 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152884439193, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-11-16T16:41:24.916 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-11-16T16:41:24.916 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152884439615, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1700152884, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "80d5add6-e124-431b-9b27-4853a8229fdf", "db_session_id": "69DQIQRWL3MF4K0VSASR", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-11-16T16:41:24.916 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152884439683, "job": 1, "event": "recovery_finished"} 2023-11-16T16:41:24.916 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-11-16T16:41:24.916 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-16T16:41:24.916 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559bc26f0000 2023-11-16T16:41:24.916 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: DB pointer 0x559bc26dc000 2023-11-16T16:41:24.916 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-11-16T16:41:24.916 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-11-16T16:41:24.916 INFO:journalctl@ceph.mon.c.smithi176.stdout: ** DB Stats ** 2023-11-16T16:41:24.917 INFO:journalctl@ceph.mon.c.smithi176.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T16:41:24.917 INFO:journalctl@ceph.mon.c.smithi176.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-16T16:41:24.917 INFO:journalctl@ceph.mon.c.smithi176.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T16:41:24.917 INFO:journalctl@ceph.mon.c.smithi176.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T16:41:24.917 INFO:journalctl@ceph.mon.c.smithi176.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-16T16:41:24.917 INFO:journalctl@ceph.mon.c.smithi176.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T16:41:24.917 INFO:journalctl@ceph.mon.c.smithi176.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T16:41:24.917 INFO:journalctl@ceph.mon.c.smithi176.stdout: 2023-11-16T16:41:24.917 INFO:journalctl@ceph.mon.c.smithi176.stdout: ** Compaction Stats [default] ** 2023-11-16T16:41:24.917 INFO:journalctl@ceph.mon.c.smithi176.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T16:41:24.917 INFO:journalctl@ceph.mon.c.smithi176.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-11-16T16:41:24.917 INFO:journalctl@ceph.mon.c.smithi176.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T16:41:24.917 INFO:journalctl@ceph.mon.c.smithi176.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T16:41:24.917 INFO:journalctl@ceph.mon.c.smithi176.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T16:41:24.917 INFO:journalctl@ceph.mon.c.smithi176.stdout: 2023-11-16T16:41:24.918 INFO:journalctl@ceph.mon.c.smithi176.stdout: ** Compaction Stats [default] ** 2023-11-16T16:41:24.918 INFO:journalctl@ceph.mon.c.smithi176.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T16:41:24.918 INFO:journalctl@ceph.mon.c.smithi176.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T16:41:24.918 INFO:journalctl@ceph.mon.c.smithi176.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T16:41:24.918 INFO:journalctl@ceph.mon.c.smithi176.stdout: 2023-11-16T16:41:24.918 INFO:journalctl@ceph.mon.c.smithi176.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-11-16T16:41:24.918 INFO:journalctl@ceph.mon.c.smithi176.stdout: 2023-11-16T16:41:24.918 INFO:journalctl@ceph.mon.c.smithi176.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T16:41:24.918 INFO:journalctl@ceph.mon.c.smithi176.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T16:41:24.918 INFO:journalctl@ceph.mon.c.smithi176.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T16:41:24.918 INFO:journalctl@ceph.mon.c.smithi176.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T16:41:24.918 INFO:journalctl@ceph.mon.c.smithi176.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T16:41:24.918 INFO:journalctl@ceph.mon.c.smithi176.stdout: AddFile(Keys): cumulative 0, interval 0 2023-11-16T16:41:24.918 INFO:journalctl@ceph.mon.c.smithi176.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T16:41:24.918 INFO:journalctl@ceph.mon.c.smithi176.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T16:41:24.919 INFO:journalctl@ceph.mon.c.smithi176.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T16:41:24.919 INFO:journalctl@ceph.mon.c.smithi176.stdout: Block cache BinnedLRUCache@0x559bc26531f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-11-16T16:41:24.919 INFO:journalctl@ceph.mon.c.smithi176.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-11-16T16:41:24.919 INFO:journalctl@ceph.mon.c.smithi176.stdout: 2023-11-16T16:41:24.919 INFO:journalctl@ceph.mon.c.smithi176.stdout: ** File Read Latency Histogram By Level [default] ** 2023-11-16T16:41:24.919 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-11-16T16:41:24.919 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: using public_addr v2:172.21.15.176:0/0 -> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2023-11-16T16:41:24.919 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: starting mon.c rank -1 at public addrs [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] at bind addrs [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:41:24.919 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: mon.c@-1(???) e0 preinit fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:41:24.919 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: mon.c@-1(synchronizing).mds e1 new map 2023-11-16T16:41:24.919 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: mon.c@-1(synchronizing).mds e1 print_map 2023-11-16T16:41:24.919 INFO:journalctl@ceph.mon.c.smithi176.stdout: e1 2023-11-16T16:41:24.919 INFO:journalctl@ceph.mon.c.smithi176.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T16:41:24.920 INFO:journalctl@ceph.mon.c.smithi176.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T16:41:24.920 INFO:journalctl@ceph.mon.c.smithi176.stdout: legacy client fscid: -1 2023-11-16T16:41:24.920 INFO:journalctl@ceph.mon.c.smithi176.stdout: 2023-11-16T16:41:24.920 INFO:journalctl@ceph.mon.c.smithi176.stdout: No filesystems configured 2023-11-16T16:41:24.920 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-11-16T16:41:24.920 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-11-16T16:41:24.920 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-11-16T16:41:24.920 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-11-16T16:41:24.920 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-11-16T16:41:24.920 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-11-16T16:41:24.920 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-16T16:41:24.920 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:41:24.920 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:41:24.920 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:41:24.921 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.921 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:41:24.921 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:24.921 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.921 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.921 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.921 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.921 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:41:24.921 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:24.921 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:24.921 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:41:24.921 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: Updating smithi077:/etc/ceph/ceph.conf 2023-11-16T16:41:24.921 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: Updating smithi077:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:24.921 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:24.921 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:24.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: Updating smithi077:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:41:24.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi176", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:24.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: Deploying cephadm binary to smithi176 2023-11-16T16:41:24.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:24.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: Added host smithi176 2023-11-16T16:41:24.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:24.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:24.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.923 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:41:24.923 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:24.923 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.923 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.923 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.923 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.923 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:41:24.923 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:24.923 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:24.923 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:41:24.923 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: Updating smithi176:/etc/ceph/ceph.conf 2023-11-16T16:41:24.923 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:24.923 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: Updating smithi176:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:24.923 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.924 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.924 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.924 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:24.924 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: Updating smithi176:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:41:24.924 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3840079481' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-16T16:41:24.924 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3840079481' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-16T16:41:24.924 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:24.924 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:24.924 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:24.924 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.924 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:24.924 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:24.924 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:24.924 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:24.924 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:24.925 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:24.925 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi044:172.21.15.44=a;smithi077:172.21.15.77=b;smithi176:172.21.15.176=c", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:24.925 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: Saving service mon spec with placement smithi044:172.21.15.44=a;smithi077:172.21.15.77=b;smithi176:172.21.15.176=c;count:3 2023-11-16T16:41:24.925 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:41:24.925 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:41:24.925 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: Deploying daemon mon.c on smithi176 2023-11-16T16:41:24.925 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:24.925 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-11-16T16:41:24.925 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-11-16T16:41:24.925 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: expand_channel_meta expand map: {default=false} 2023-11-16T16:41:24.925 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: expand_channel_meta from 'false' to 'false' 2023-11-16T16:41:24.925 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: expand_channel_meta expanded map: {default=false} 2023-11-16T16:41:24.925 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: expand_channel_meta expand map: {default=info} 2023-11-16T16:41:24.925 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: expand_channel_meta from 'info' to 'info' 2023-11-16T16:41:24.925 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: expand_channel_meta expanded map: {default=info} 2023-11-16T16:41:24.926 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: expand_channel_meta expand map: {default=daemon} 2023-11-16T16:41:24.926 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: expand_channel_meta from 'daemon' to 'daemon' 2023-11-16T16:41:24.926 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: expand_channel_meta expanded map: {default=daemon} 2023-11-16T16:41:24.926 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: expand_channel_meta expand map: {default=debug} 2023-11-16T16:41:24.926 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: expand_channel_meta from 'debug' to 'debug' 2023-11-16T16:41:24.926 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 ceph-mon[131630]: expand_channel_meta expanded map: {default=debug} 2023-11-16T16:41:24.926 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 bash[131569]: bf242255d40bc9596917ad68fecf0260208af284842acadf93406ea86281b456 2023-11-16T16:41:24.926 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:24 smithi176 systemd[1]: Started Ceph mon.c for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:41:25.616 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-11-16T16:41:25.617 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T16:41:26.263 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:41:29.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:29.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:41:29.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: mon.a calling monitor election 2023-11-16T16:41:29.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:29.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:29.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:29.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:29.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: mon.c calling monitor election 2023-11-16T16:41:29.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:29.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:29.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:29.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:29.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T16:41:29.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: monmap e2: 2 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],c=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0]} removed_ranks: {} 2023-11-16T16:41:29.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: fsmap 2023-11-16T16:41:29.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:29.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: mgrmap e13: x(active, since 47s) 2023-11-16T16:41:29.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: overall HEALTH_OK 2023-11-16T16:41:29.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:29.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:29.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:29.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:29.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:29.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:29.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:41:29.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: mon.a calling monitor election 2023-11-16T16:41:29.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:29.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:29.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:29.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:29.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: mon.c calling monitor election 2023-11-16T16:41:29.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:29.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:29.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:29.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:29.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T16:41:29.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: monmap e2: 2 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],c=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0]} removed_ranks: {} 2023-11-16T16:41:29.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: fsmap 2023-11-16T16:41:29.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:29.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: mgrmap e13: x(active, since 47s) 2023-11-16T16:41:29.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: overall HEALTH_OK 2023-11-16T16:41:29.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:29.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:29.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:29.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:29.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:30.461 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.460+0000 7f4e1a711700 1 -- 172.21.15.176:0/1251343999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e000035c0 msgr2=0x7f4e00005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:30.461 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.460+0000 7f4e1a711700 1 --2- 172.21.15.176:0/1251343999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e000035c0 0x7f4e00005a40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4e140ffd80 tx=0x7f4e0400feb0 comp rx=0 tx=0).stop 2023-11-16T16:41:30.462 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.460+0000 7f4e1a711700 1 -- 172.21.15.176:0/1251343999 shutdown_connections 2023-11-16T16:41:30.462 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.460+0000 7f4e1a711700 1 --2- 172.21.15.176:0/1251343999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e000035c0 0x7f4e00005a40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:30.462 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.460+0000 7f4e1a711700 1 -- 172.21.15.176:0/1251343999 >> 172.21.15.176:0/1251343999 conn(0x7f4e140fb970 msgr2=0x7f4e140fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:30.462 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.460+0000 7f4e1a711700 1 -- 172.21.15.176:0/1251343999 shutdown_connections 2023-11-16T16:41:30.462 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.460+0000 7f4e1a711700 1 -- 172.21.15.176:0/1251343999 wait complete. 2023-11-16T16:41:30.464 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.461+0000 7f4e1a711700 1 Processor -- start 2023-11-16T16:41:30.464 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.461+0000 7f4e1a711700 1 -- start start 2023-11-16T16:41:30.464 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.461+0000 7f4e1a711700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e000035c0 0x7f4e14195300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:30.464 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.461+0000 7f4e1a711700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e14195840 0x7f4e14199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:30.464 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.461+0000 7f4e1a711700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e14195dc0 con 0x7f4e000035c0 2023-11-16T16:41:30.464 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.461+0000 7f4e1a711700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e14195f30 con 0x7f4e14195840 2023-11-16T16:41:30.464 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.462+0000 7f4e137fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e14195840 0x7f4e14199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:30.464 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.462+0000 7f4e137fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e14195840 0x7f4e14199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37722/0 (socket says 172.21.15.176:37722) 2023-11-16T16:41:30.464 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.462+0000 7f4e137fe700 1 -- 172.21.15.176:0/64436742 learned_addr learned my addr 172.21.15.176:0/64436742 (peer_addr_for_me v2:172.21.15.176:0/0) 2023-11-16T16:41:30.465 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.462+0000 7f4e13fff700 1 --2- 172.21.15.176:0/64436742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e000035c0 0x7f4e14195300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:30.465 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.462+0000 7f4e137fe700 1 -- 172.21.15.176:0/64436742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e14195840 msgr2=0x7f4e14199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-11-16T16:41:30.465 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.462+0000 7f4e137fe700 1 -- 172.21.15.176:0/64436742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e14195840 msgr2=0x7f4e14199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-16T16:41:30.465 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.462+0000 7f4e137fe700 1 --2- 172.21.15.176:0/64436742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e14195840 0x7f4e14199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-16T16:41:30.465 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.462+0000 7f4e137fe700 1 --2- 172.21.15.176:0/64436742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e14195840 0x7f4e14199cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:41:30.465 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.462+0000 7f4e13fff700 1 -- 172.21.15.176:0/64436742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e14195840 msgr2=0x7f4e14199cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:41:30.465 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.462+0000 7f4e13fff700 1 --2- 172.21.15.176:0/64436742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e14195840 0x7f4e14199cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:30.465 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.462+0000 7f4e13fff700 1 -- 172.21.15.176:0/64436742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0400f900 con 0x7f4e000035c0 2023-11-16T16:41:30.465 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.462+0000 7f4e13fff700 1 --2- 172.21.15.176:0/64436742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e000035c0 0x7f4e14195300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4e04011510 tx=0x7f4e04005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:30.465 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.463+0000 7f4e117fa700 1 -- 172.21.15.176:0/64436742 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f4e04013b00 con 0x7f4e000035c0 2023-11-16T16:41:30.466 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.463+0000 7f4e117fa700 1 -- 172.21.15.176:0/64436742 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e040044f0 con 0x7f4e000035c0 2023-11-16T16:41:30.466 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.463+0000 7f4e1a711700 1 -- 172.21.15.176:0/64436742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e14107ca0 con 0x7f4e000035c0 2023-11-16T16:41:30.466 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.463+0000 7f4e117fa700 1 -- 172.21.15.176:0/64436742 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f4e04003b50 con 0x7f4e000035c0 2023-11-16T16:41:30.466 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.463+0000 7f4e1a711700 1 -- 172.21.15.176:0/64436742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e14107b10 con 0x7f4e000035c0 2023-11-16T16:41:30.467 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.464+0000 7f4e117fa700 1 -- 172.21.15.176:0/64436742 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f4e04011590 con 0x7f4e000035c0 2023-11-16T16:41:30.467 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.464+0000 7f4e1a711700 1 -- 172.21.15.176:0/64436742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e14107e30 con 0x7f4e000035c0 2023-11-16T16:41:30.467 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.464+0000 7f4e117fa700 1 --2- 172.21.15.176:0/64436742 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4df40386b0 0x7f4df403ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:30.467 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.464+0000 7f4e117fa700 1 -- 172.21.15.176:0/64436742 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4e0404dd00 con 0x7f4e000035c0 2023-11-16T16:41:30.467 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.465+0000 7f4e137fe700 1 --2- 172.21.15.176:0/64436742 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4df40386b0 0x7f4df403ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:30.467 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.465+0000 7f4e137fe700 1 --2- 172.21.15.176:0/64436742 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4df40386b0 0x7f4df403ab60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc009180 tx=0x7f4dfc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:30.471 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.470+0000 7f4e117fa700 1 -- 172.21.15.176:0/64436742 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4e04011ea0 con 0x7f4e000035c0 2023-11-16T16:41:30.738 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.736+0000 7f4e1a711700 1 -- 172.21.15.176:0/64436742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e14196820 con 0x7f4e000035c0 2023-11-16T16:41:30.738 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.737+0000 7f4e117fa700 1 -- 172.21.15.176:0/64436742 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7f4e040103d0 con 0x7f4e000035c0 2023-11-16T16:41:30.738 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-11-16T16:41:30.739 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":2,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T16:41:24.463879Z","created":"2023-11-16T16:39:38.269339Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-11-16T16:41:30.741 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.739+0000 7f4e1a711700 1 -- 172.21.15.176:0/64436742 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4df40386b0 msgr2=0x7f4df403ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:30.741 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.739+0000 7f4e1a711700 1 --2- 172.21.15.176:0/64436742 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4df40386b0 0x7f4df403ab60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc009180 tx=0x7f4dfc008040 comp rx=0 tx=0).stop 2023-11-16T16:41:30.741 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.739+0000 7f4e1a711700 1 -- 172.21.15.176:0/64436742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e000035c0 msgr2=0x7f4e14195300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:30.741 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.739+0000 7f4e1a711700 1 --2- 172.21.15.176:0/64436742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e000035c0 0x7f4e14195300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4e04011510 tx=0x7f4e04005250 comp rx=0 tx=0).stop 2023-11-16T16:41:30.741 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.739+0000 7f4e1a711700 1 -- 172.21.15.176:0/64436742 shutdown_connections 2023-11-16T16:41:30.741 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.739+0000 7f4e1a711700 1 --2- 172.21.15.176:0/64436742 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4df40386b0 0x7f4df403ab60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:30.741 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.739+0000 7f4e1a711700 1 --2- 172.21.15.176:0/64436742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e000035c0 0x7f4e14195300 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:30.741 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.739+0000 7f4e1a711700 1 --2- 172.21.15.176:0/64436742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e14195840 0x7f4e14199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:30.741 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.739+0000 7f4e1a711700 1 -- 172.21.15.176:0/64436742 >> 172.21.15.176:0/64436742 conn(0x7f4e140fb970 msgr2=0x7f4e141019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:30.741 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.739+0000 7f4e1a711700 1 -- 172.21.15.176:0/64436742 shutdown_connections 2023-11-16T16:41:30.741 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:30.739+0000 7f4e1a711700 1 -- 172.21.15.176:0/64436742 wait complete. 2023-11-16T16:41:30.742 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 2 2023-11-16T16:41:30.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:30 smithi044 ceph-mon[128750]: Deploying daemon mon.b on smithi077 2023-11-16T16:41:30.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:30 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:30.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:30 smithi176 ceph-mon[131630]: Deploying daemon mon.b on smithi077 2023-11-16T16:41:30.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:30 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:31.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:41:31 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:41:31.463+0000 7f7611026700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-16T16:41:31.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:31 smithi044 ceph-mon[128750]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:31.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:31 smithi044 ceph-mon[128750]: from='client.? 172.21.15.176:0/64436742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T16:41:31.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:31 smithi176 ceph-mon[131630]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:31.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:31 smithi176 ceph-mon[131630]: from='client.? 172.21.15.176:0/64436742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T16:41:32.126 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-11-16T16:41:32.127 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T16:41:32.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 systemd[1]: Starting Ceph mon.b for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:41:32.776 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:41:32.822 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 podman[131164]: 2023-11-16T16:41:32.822 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: set uid:gid to 167:167 (ceph:ceph) 2023-11-16T16:41:32.822 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev), process ceph-mon, pid 7 2023-11-16T16:41:32.822 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: pidfile_write: ignore empty --pid-file 2023-11-16T16:41:32.822 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: load: jerasure load: lrc 2023-11-16T16:41:32.823 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: RocksDB version: 7.9.2 2023-11-16T16:41:32.823 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Git sha 0 2023-11-16T16:41:32.823 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Compile date 2023-11-15 23:53:54 2023-11-16T16:41:32.823 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: DB SUMMARY 2023-11-16T16:41:32.823 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: DB Session ID: BCBR8J0OTYDO2T05BWC4 2023-11-16T16:41:32.823 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: CURRENT file: CURRENT 2023-11-16T16:41:32.823 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: IDENTITY file: IDENTITY 2023-11-16T16:41:32.824 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-11-16T16:41:32.824 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-11-16T16:41:32.824 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-11-16T16:41:32.824 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.error_if_exists: 0 2023-11-16T16:41:32.824 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.create_if_missing: 0 2023-11-16T16:41:32.824 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.paranoid_checks: 1 2023-11-16T16:41:32.824 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.flush_verify_memtable_count: 1 2023-11-16T16:41:32.825 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-16T16:41:32.825 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-11-16T16:41:32.825 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.env: 0x55902d394720 2023-11-16T16:41:32.825 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.fs: PosixFileSystem 2023-11-16T16:41:32.825 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.info_log: 0x55902f6af3c0 2023-11-16T16:41:32.825 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_file_opening_threads: 16 2023-11-16T16:41:32.825 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.statistics: (nil) 2023-11-16T16:41:32.825 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.use_fsync: 0 2023-11-16T16:41:32.825 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_log_file_size: 0 2023-11-16T16:41:32.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-16T16:41:32.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.log_file_time_to_roll: 0 2023-11-16T16:41:32.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.keep_log_file_num: 1000 2023-11-16T16:41:32.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.recycle_log_file_num: 0 2023-11-16T16:41:32.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.allow_fallocate: 1 2023-11-16T16:41:32.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.allow_mmap_reads: 0 2023-11-16T16:41:32.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.allow_mmap_writes: 0 2023-11-16T16:41:32.827 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.use_direct_reads: 0 2023-11-16T16:41:32.827 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-16T16:41:32.827 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.create_missing_column_families: 0 2023-11-16T16:41:32.827 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.db_log_dir: 2023-11-16T16:41:32.827 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.wal_dir: 2023-11-16T16:41:32.827 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.table_cache_numshardbits: 6 2023-11-16T16:41:32.827 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.WAL_ttl_seconds: 0 2023-11-16T16:41:32.827 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.WAL_size_limit_MB: 0 2023-11-16T16:41:32.827 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-16T16:41:32.828 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-16T16:41:32.828 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.is_fd_close_on_exec: 1 2023-11-16T16:41:32.828 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.advise_random_on_open: 1 2023-11-16T16:41:32.828 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.db_write_buffer_size: 0 2023-11-16T16:41:32.828 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.write_buffer_manager: 0x55902e9203c0 2023-11-16T16:41:32.828 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-16T16:41:32.828 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-16T16:41:32.829 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.use_adaptive_mutex: 0 2023-11-16T16:41:32.829 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.rate_limiter: (nil) 2023-11-16T16:41:32.829 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-16T16:41:32.829 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.wal_recovery_mode: 2 2023-11-16T16:41:32.829 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.enable_thread_tracking: 0 2023-11-16T16:41:32.829 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.enable_pipelined_write: 0 2023-11-16T16:41:32.829 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.unordered_write: 0 2023-11-16T16:41:32.830 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-16T16:41:32.830 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-16T16:41:32.830 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-16T16:41:32.830 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-16T16:41:32.830 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.row_cache: None 2023-11-16T16:41:32.830 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.wal_filter: None 2023-11-16T16:41:32.830 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-16T16:41:32.831 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.allow_ingest_behind: 0 2023-11-16T16:41:32.831 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.two_write_queues: 0 2023-11-16T16:41:32.831 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.manual_wal_flush: 0 2023-11-16T16:41:32.831 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.wal_compression: 0 2023-11-16T16:41:32.831 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.atomic_flush: 0 2023-11-16T16:41:32.831 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-16T16:41:32.831 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.persist_stats_to_disk: 0 2023-11-16T16:41:32.831 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.write_dbid_to_manifest: 0 2023-11-16T16:41:32.831 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.log_readahead_size: 0 2023-11-16T16:41:32.832 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-16T16:41:32.832 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.best_efforts_recovery: 0 2023-11-16T16:41:32.832 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-16T16:41:32.832 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-16T16:41:32.832 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.allow_data_in_errors: 0 2023-11-16T16:41:32.832 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.db_host_id: __hostname__ 2023-11-16T16:41:32.832 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.enforce_single_del_contracts: true 2023-11-16T16:41:32.832 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_background_jobs: 2 2023-11-16T16:41:32.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_background_compactions: -1 2023-11-16T16:41:32.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_subcompactions: 1 2023-11-16T16:41:32.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-16T16:41:32.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-16T16:41:32.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.delayed_write_rate : 16777216 2023-11-16T16:41:32.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_total_wal_size: 0 2023-11-16T16:41:32.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-16T16:41:32.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.stats_dump_period_sec: 600 2023-11-16T16:41:32.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.stats_persist_period_sec: 600 2023-11-16T16:41:32.834 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-16T16:41:32.834 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_open_files: -1 2023-11-16T16:41:32.834 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.bytes_per_sync: 0 2023-11-16T16:41:32.834 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.wal_bytes_per_sync: 0 2023-11-16T16:41:32.834 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.strict_bytes_per_sync: 0 2023-11-16T16:41:32.834 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compaction_readahead_size: 0 2023-11-16T16:41:32.834 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_background_flushes: -1 2023-11-16T16:41:32.835 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Compression algorithms supported: 2023-11-16T16:41:32.835 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-16T16:41:32.835 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: kZSTD supported: 0 2023-11-16T16:41:32.835 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: kXpressCompression supported: 0 2023-11-16T16:41:32.835 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: kLZ4HCCompression supported: 1 2023-11-16T16:41:32.835 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: kZlibCompression supported: 1 2023-11-16T16:41:32.835 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: kSnappyCompression supported: 1 2023-11-16T16:41:32.835 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: kLZ4Compression supported: 1 2023-11-16T16:41:32.835 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: kBZip2Compression supported: 0 2023-11-16T16:41:32.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Fast CRC32 supported: Supported on x86 2023-11-16T16:41:32.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: DMutex implementation: pthread_mutex_t 2023-11-16T16:41:32.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-11-16T16:41:32.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-11-16T16:41:32.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-16T16:41:32.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.merge_operator: 2023-11-16T16:41:32.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compaction_filter: None 2023-11-16T16:41:32.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compaction_filter_factory: None 2023-11-16T16:41:32.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.sst_partitioner_factory: None 2023-11-16T16:41:32.837 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.memtable_factory: SkipListFactory 2023-11-16T16:41:32.837 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.table_factory: BlockBasedTable 2023-11-16T16:41:32.837 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55902f6af4e0) 2023-11-16T16:41:32.837 INFO:journalctl@ceph.mon.b.smithi077.stdout: cache_index_and_filter_blocks: 1 2023-11-16T16:41:32.837 INFO:journalctl@ceph.mon.b.smithi077.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-16T16:41:32.837 INFO:journalctl@ceph.mon.b.smithi077.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-16T16:41:32.837 INFO:journalctl@ceph.mon.b.smithi077.stdout: pin_top_level_index_and_filter: 1 2023-11-16T16:41:32.837 INFO:journalctl@ceph.mon.b.smithi077.stdout: index_type: 0 2023-11-16T16:41:32.837 INFO:journalctl@ceph.mon.b.smithi077.stdout: data_block_index_type: 0 2023-11-16T16:41:32.838 INFO:journalctl@ceph.mon.b.smithi077.stdout: index_shortening: 1 2023-11-16T16:41:32.838 INFO:journalctl@ceph.mon.b.smithi077.stdout: data_block_hash_table_util_ratio: 0.750000 2023-11-16T16:41:32.838 INFO:journalctl@ceph.mon.b.smithi077.stdout: checksum: 4 2023-11-16T16:41:32.838 INFO:journalctl@ceph.mon.b.smithi077.stdout: no_block_cache: 0 2023-11-16T16:41:32.838 INFO:journalctl@ceph.mon.b.smithi077.stdout: block_cache: 0x55902e9c11f0 2023-11-16T16:41:32.838 INFO:journalctl@ceph.mon.b.smithi077.stdout: block_cache_name: BinnedLRUCache 2023-11-16T16:41:32.838 INFO:journalctl@ceph.mon.b.smithi077.stdout: block_cache_options: 2023-11-16T16:41:32.838 INFO:journalctl@ceph.mon.b.smithi077.stdout: capacity : 536870912 2023-11-16T16:41:32.839 INFO:journalctl@ceph.mon.b.smithi077.stdout: num_shard_bits : 4 2023-11-16T16:41:32.839 INFO:journalctl@ceph.mon.b.smithi077.stdout: strict_capacity_limit : 0 2023-11-16T16:41:32.839 INFO:journalctl@ceph.mon.b.smithi077.stdout: high_pri_pool_ratio: 0.000 2023-11-16T16:41:32.839 INFO:journalctl@ceph.mon.b.smithi077.stdout: block_cache_compressed: (nil) 2023-11-16T16:41:32.839 INFO:journalctl@ceph.mon.b.smithi077.stdout: persistent_cache: (nil) 2023-11-16T16:41:32.839 INFO:journalctl@ceph.mon.b.smithi077.stdout: block_size: 4096 2023-11-16T16:41:32.839 INFO:journalctl@ceph.mon.b.smithi077.stdout: block_size_deviation: 10 2023-11-16T16:41:32.839 INFO:journalctl@ceph.mon.b.smithi077.stdout: block_restart_interval: 16 2023-11-16T16:41:32.839 INFO:journalctl@ceph.mon.b.smithi077.stdout: index_block_restart_interval: 1 2023-11-16T16:41:32.840 INFO:journalctl@ceph.mon.b.smithi077.stdout: metadata_block_size: 4096 2023-11-16T16:41:32.840 INFO:journalctl@ceph.mon.b.smithi077.stdout: partition_filters: 0 2023-11-16T16:41:32.840 INFO:journalctl@ceph.mon.b.smithi077.stdout: use_delta_encoding: 1 2023-11-16T16:41:32.840 INFO:journalctl@ceph.mon.b.smithi077.stdout: filter_policy: bloomfilter 2023-11-16T16:41:32.840 INFO:journalctl@ceph.mon.b.smithi077.stdout: whole_key_filtering: 1 2023-11-16T16:41:32.840 INFO:journalctl@ceph.mon.b.smithi077.stdout: verify_compression: 0 2023-11-16T16:41:32.840 INFO:journalctl@ceph.mon.b.smithi077.stdout: read_amp_bytes_per_bit: 0 2023-11-16T16:41:32.840 INFO:journalctl@ceph.mon.b.smithi077.stdout: format_version: 5 2023-11-16T16:41:32.841 INFO:journalctl@ceph.mon.b.smithi077.stdout: enable_index_compression: 1 2023-11-16T16:41:32.841 INFO:journalctl@ceph.mon.b.smithi077.stdout: block_align: 0 2023-11-16T16:41:32.841 INFO:journalctl@ceph.mon.b.smithi077.stdout: max_auto_readahead_size: 262144 2023-11-16T16:41:32.841 INFO:journalctl@ceph.mon.b.smithi077.stdout: prepopulate_block_cache: 0 2023-11-16T16:41:32.841 INFO:journalctl@ceph.mon.b.smithi077.stdout: initial_auto_readahead_size: 8192 2023-11-16T16:41:32.841 INFO:journalctl@ceph.mon.b.smithi077.stdout: num_file_reads_for_auto_readahead: 2 2023-11-16T16:41:32.841 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.write_buffer_size: 33554432 2023-11-16T16:41:32.841 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_write_buffer_number: 2 2023-11-16T16:41:32.841 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compression: NoCompression 2023-11-16T16:41:32.842 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.bottommost_compression: Disabled 2023-11-16T16:41:32.842 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.prefix_extractor: nullptr 2023-11-16T16:41:32.842 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-16T16:41:32.842 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.num_levels: 7 2023-11-16T16:41:32.842 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-16T16:41:32.842 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-16T16:41:32.842 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-16T16:41:32.842 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-16T16:41:32.843 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-16T16:41:32.843 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-16T16:41:32.843 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-16T16:41:32.843 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-16T16:41:32.843 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-16T16:41:32.843 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-16T16:41:32.843 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-11-16T16:41:32.843 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-11-16T16:41:32.843 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compression_opts.window_bits: -14 2023-11-16T16:41:32.844 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compression_opts.level: 32767 2023-11-16T16:41:32.844 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compression_opts.strategy: 0 2023-11-16T16:41:32.844 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-16T16:41:32.844 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-16T16:41:32.844 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-11-16T16:41:32.844 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-16T16:41:32.844 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compression_opts.enabled: false 2023-11-16T16:41:32.844 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-11-16T16:41:32.845 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-16T16:41:32.845 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-16T16:41:32.845 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-16T16:41:32.845 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.target_file_size_base: 67108864 2023-11-16T16:41:32.845 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.target_file_size_multiplier: 1 2023-11-16T16:41:32.845 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-16T16:41:32.845 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-16T16:41:32.845 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-16T16:41:32.845 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-16T16:41:32.846 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-16T16:41:32.846 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-16T16:41:32.846 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-16T16:41:32.846 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-16T16:41:32.846 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-16T16:41:32.846 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-16T16:41:32.846 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-16T16:41:32.846 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-16T16:41:32.846 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-11-16T16:41:32.847 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.arena_block_size: 1048576 2023-11-16T16:41:32.847 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-16T16:41:32.847 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-16T16:41:32.847 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.disable_auto_compactions: 0 2023-11-16T16:41:32.847 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-16T16:41:32.848 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-16T16:41:32.848 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-16T16:41:32.848 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-16T16:41:32.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-16T16:41:32.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-16T16:41:32.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-16T16:41:32.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-16T16:41:32.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-16T16:41:32.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-16T16:41:32.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.table_properties_collectors: 2023-11-16T16:41:32.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.inplace_update_support: 0 2023-11-16T16:41:32.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.inplace_update_num_locks: 10000 2023-11-16T16:41:32.850 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-16T16:41:32.850 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-16T16:41:32.850 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.memtable_huge_page_size: 0 2023-11-16T16:41:32.850 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.bloom_locality: 0 2023-11-16T16:41:32.850 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.max_successive_merges: 0 2023-11-16T16:41:32.850 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.optimize_filters_for_hits: 0 2023-11-16T16:41:32.850 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.paranoid_file_checks: 0 2023-11-16T16:41:32.851 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.force_consistency_checks: 1 2023-11-16T16:41:32.851 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.report_bg_io_stats: 0 2023-11-16T16:41:32.851 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.ttl: 2592000 2023-11-16T16:41:32.851 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.periodic_compaction_seconds: 0 2023-11-16T16:41:32.851 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-11-16T16:41:32.851 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-11-16T16:41:32.851 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.enable_blob_files: false 2023-11-16T16:41:32.851 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.min_blob_size: 0 2023-11-16T16:41:32.851 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.blob_file_size: 268435456 2023-11-16T16:41:32.852 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.blob_compression_type: NoCompression 2023-11-16T16:41:32.852 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.enable_blob_garbage_collection: false 2023-11-16T16:41:32.852 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-16T16:41:32.852 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-11-16T16:41:32.852 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-11-16T16:41:32.852 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.blob_file_starting_level: 0 2023-11-16T16:41:32.852 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-11-16T16:41:32.852 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-16T16:41:32.852 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-11-16T16:41:32.852 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f9bad1bf-df96-416a-8882-5a23f449ad95 2023-11-16T16:41:32.853 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152892710463, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-11-16T16:41:32.853 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-11-16T16:41:32.853 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152892711028, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1700152892, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f9bad1bf-df96-416a-8882-5a23f449ad95", "db_session_id": "BCBR8J0OTYDO2T05BWC4", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-11-16T16:41:32.853 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152892711122, "job": 1, "event": "recovery_finished"} 2023-11-16T16:41:32.853 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-11-16T16:41:32.853 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-16T16:41:32.853 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55902ea5e000 2023-11-16T16:41:32.853 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: DB pointer 0x55902ea4a000 2023-11-16T16:41:32.853 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-11-16T16:41:32.853 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-11-16T16:41:32.854 INFO:journalctl@ceph.mon.b.smithi077.stdout: ** DB Stats ** 2023-11-16T16:41:32.854 INFO:journalctl@ceph.mon.b.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T16:41:32.854 INFO:journalctl@ceph.mon.b.smithi077.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-16T16:41:32.854 INFO:journalctl@ceph.mon.b.smithi077.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T16:41:32.854 INFO:journalctl@ceph.mon.b.smithi077.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T16:41:32.854 INFO:journalctl@ceph.mon.b.smithi077.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-16T16:41:32.854 INFO:journalctl@ceph.mon.b.smithi077.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T16:41:32.854 INFO:journalctl@ceph.mon.b.smithi077.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T16:41:32.854 INFO:journalctl@ceph.mon.b.smithi077.stdout: 2023-11-16T16:41:32.855 INFO:journalctl@ceph.mon.b.smithi077.stdout: ** Compaction Stats [default] ** 2023-11-16T16:41:32.855 INFO:journalctl@ceph.mon.b.smithi077.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T16:41:32.855 INFO:journalctl@ceph.mon.b.smithi077.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-11-16T16:41:32.855 INFO:journalctl@ceph.mon.b.smithi077.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T16:41:32.855 INFO:journalctl@ceph.mon.b.smithi077.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T16:41:32.855 INFO:journalctl@ceph.mon.b.smithi077.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T16:41:32.855 INFO:journalctl@ceph.mon.b.smithi077.stdout: 2023-11-16T16:41:32.855 INFO:journalctl@ceph.mon.b.smithi077.stdout: ** Compaction Stats [default] ** 2023-11-16T16:41:32.855 INFO:journalctl@ceph.mon.b.smithi077.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T16:41:32.855 INFO:journalctl@ceph.mon.b.smithi077.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T16:41:32.856 INFO:journalctl@ceph.mon.b.smithi077.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T16:41:32.856 INFO:journalctl@ceph.mon.b.smithi077.stdout: 2023-11-16T16:41:32.856 INFO:journalctl@ceph.mon.b.smithi077.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-11-16T16:41:32.856 INFO:journalctl@ceph.mon.b.smithi077.stdout: 2023-11-16T16:41:32.856 INFO:journalctl@ceph.mon.b.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T16:41:32.856 INFO:journalctl@ceph.mon.b.smithi077.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T16:41:32.856 INFO:journalctl@ceph.mon.b.smithi077.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T16:41:32.856 INFO:journalctl@ceph.mon.b.smithi077.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T16:41:32.856 INFO:journalctl@ceph.mon.b.smithi077.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T16:41:32.856 INFO:journalctl@ceph.mon.b.smithi077.stdout: AddFile(Keys): cumulative 0, interval 0 2023-11-16T16:41:32.856 INFO:journalctl@ceph.mon.b.smithi077.stdout: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T16:41:32.857 INFO:journalctl@ceph.mon.b.smithi077.stdout: Interval compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T16:41:32.857 INFO:journalctl@ceph.mon.b.smithi077.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T16:41:32.857 INFO:journalctl@ceph.mon.b.smithi077.stdout: Block cache BinnedLRUCache@0x55902e9c11f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-11-16T16:41:32.857 INFO:journalctl@ceph.mon.b.smithi077.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-11-16T16:41:32.857 INFO:journalctl@ceph.mon.b.smithi077.stdout: 2023-11-16T16:41:32.857 INFO:journalctl@ceph.mon.b.smithi077.stdout: ** File Read Latency Histogram By Level [default] ** 2023-11-16T16:41:32.857 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-11-16T16:41:32.857 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: using public_addr v2:172.21.15.77:0/0 -> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] 2023-11-16T16:41:32.857 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: starting mon.b rank -1 at public addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] at bind addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:41:32.857 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.b@-1(???) e0 preinit fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:41:32.858 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 bash[131164]: 90599dd7dadd401b4b2ac54498e98971bf11bf7e91a750f62290c37184940aac 2023-11-16T16:41:32.858 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.b@-1(synchronizing).mds e1 new map 2023-11-16T16:41:32.858 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.b@-1(synchronizing).mds e1 print_map 2023-11-16T16:41:32.858 INFO:journalctl@ceph.mon.b.smithi077.stdout: e1 2023-11-16T16:41:32.858 INFO:journalctl@ceph.mon.b.smithi077.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T16:41:32.858 INFO:journalctl@ceph.mon.b.smithi077.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T16:41:32.858 INFO:journalctl@ceph.mon.b.smithi077.stdout: legacy client fscid: -1 2023-11-16T16:41:32.858 INFO:journalctl@ceph.mon.b.smithi077.stdout: 2023-11-16T16:41:32.858 INFO:journalctl@ceph.mon.b.smithi077.stdout: No filesystems configured 2023-11-16T16:41:32.859 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-11-16T16:41:32.859 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-11-16T16:41:32.859 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-11-16T16:41:32.859 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-11-16T16:41:32.859 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-11-16T16:41:32.859 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-11-16T16:41:32.859 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-16T16:41:32.859 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:41:32.859 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:41:32.860 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:41:32.860 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.860 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:41:32.860 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:32.860 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.860 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.860 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.860 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.860 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:41:32.860 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:32.860 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:32.861 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:41:32.861 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Updating smithi077:/etc/ceph/ceph.conf 2023-11-16T16:41:32.861 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Updating smithi077:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:32.861 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:32.861 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:32.861 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Updating smithi077:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:41:32.861 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.861 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.861 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.862 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi176", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:32.862 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Deploying cephadm binary to smithi176 2023-11-16T16:41:32.862 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:32.862 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.862 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Added host smithi176 2023-11-16T16:41:32.862 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:32.862 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.862 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.862 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:32.862 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.863 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:41:32.863 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:32.863 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.863 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.863 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.863 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.863 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:41:32.863 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:32.863 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:32.863 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:41:32.863 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Updating smithi176:/etc/ceph/ceph.conf 2023-11-16T16:41:32.864 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:32.864 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Updating smithi176:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:32.864 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.864 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.864 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.864 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:32.864 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Updating smithi176:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:41:32.864 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3840079481' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-16T16:41:32.864 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3840079481' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-16T16:41:32.864 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:32.865 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:32.865 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:32.865 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.865 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:32.865 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:32.865 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:32.865 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.865 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:32.865 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:32.865 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi044:172.21.15.44=a;smithi077:172.21.15.77=b;smithi176:172.21.15.176=c", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:32.865 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Saving service mon spec with placement smithi044:172.21.15.44=a;smithi077:172.21.15.77=b;smithi176:172.21.15.176=c;count:3 2023-11-16T16:41:32.865 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:41:32.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:41:32.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Deploying daemon mon.c on smithi176 2023-11-16T16:41:32.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:32.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:32.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:41:32.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.a calling monitor election 2023-11-16T16:41:32.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:32.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:32.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:32.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:32.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.c calling monitor election 2023-11-16T16:41:32.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:32.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:32.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:32.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:32.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T16:41:32.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: monmap e2: 2 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],c=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0]} removed_ranks: {} 2023-11-16T16:41:32.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: fsmap 2023-11-16T16:41:32.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:32.868 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mgrmap e13: x(active, since 47s) 2023-11-16T16:41:32.868 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: overall HEALTH_OK 2023-11-16T16:41:32.868 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.868 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.868 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:32.868 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:32.868 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:32.868 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: Deploying daemon mon.b on smithi077 2023-11-16T16:41:32.868 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:32.868 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:32.868 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: from='client.? 172.21.15.176:0/64436742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T16:41:32.869 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-11-16T16:41:32.869 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-11-16T16:41:32.869 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: expand_channel_meta expand map: {default=false} 2023-11-16T16:41:32.869 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: expand_channel_meta from 'false' to 'false' 2023-11-16T16:41:32.869 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: expand_channel_meta expanded map: {default=false} 2023-11-16T16:41:32.869 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: expand_channel_meta expand map: {default=info} 2023-11-16T16:41:32.869 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: expand_channel_meta from 'info' to 'info' 2023-11-16T16:41:32.869 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: expand_channel_meta expanded map: {default=info} 2023-11-16T16:41:32.869 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: expand_channel_meta expand map: {default=daemon} 2023-11-16T16:41:32.869 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: expand_channel_meta from 'daemon' to 'daemon' 2023-11-16T16:41:32.870 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: expand_channel_meta expanded map: {default=daemon} 2023-11-16T16:41:32.870 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: expand_channel_meta expand map: {default=debug} 2023-11-16T16:41:32.870 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: expand_channel_meta from 'debug' to 'debug' 2023-11-16T16:41:32.870 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 ceph-mon[131204]: expand_channel_meta expanded map: {default=debug} 2023-11-16T16:41:32.870 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:32 smithi077 systemd[1]: Started Ceph mon.b for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:41:38.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:38.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:41:38.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:38.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: mon.a calling monitor election 2023-11-16T16:41:38.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: mon.c calling monitor election 2023-11-16T16:41:38.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:38.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:38.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:38.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:38.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:38.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:38.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:38.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:38.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T16:41:38.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: monmap e3: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],c=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0]} removed_ranks: {} 2023-11-16T16:41:38.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: fsmap 2023-11-16T16:41:38.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:38.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: mgrmap e13: x(active, since 55s) 2023-11-16T16:41:38.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-11-16T16:41:38.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-11-16T16:41:38.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-11-16T16:41:38.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: mon.b (rank 2) addr [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] is down (out of quorum) 2023-11-16T16:41:38.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:38.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:38.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:38.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:41:38.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:38.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: mon.a calling monitor election 2023-11-16T16:41:38.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: mon.c calling monitor election 2023-11-16T16:41:38.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:38.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:38.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:38.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:38.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:38.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:38.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:38.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:38.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T16:41:38.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: monmap e3: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],c=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0]} removed_ranks: {} 2023-11-16T16:41:38.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: fsmap 2023-11-16T16:41:38.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:38.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: mgrmap e13: x(active, since 55s) 2023-11-16T16:41:38.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-11-16T16:41:38.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-11-16T16:41:38.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-11-16T16:41:38.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: mon.b (rank 2) addr [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] is down (out of quorum) 2023-11-16T16:41:38.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:38.148 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:38.349 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.348+0000 7f0c6bdd1700 1 -- 172.21.15.176:0/2639281111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c500035c0 msgr2=0x7f0c50005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:38.350 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.348+0000 7f0c6bdd1700 1 --2- 172.21.15.176:0/2639281111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c500035c0 0x7f0c50005a40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0c64075f20 tx=0x7f0c5400fe10 comp rx=0 tx=0).stop 2023-11-16T16:41:38.350 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.348+0000 7f0c6bdd1700 1 -- 172.21.15.176:0/2639281111 shutdown_connections 2023-11-16T16:41:38.350 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.348+0000 7f0c6bdd1700 1 --2- 172.21.15.176:0/2639281111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c500035c0 0x7f0c50005a40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:38.350 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.348+0000 7f0c6bdd1700 1 -- 172.21.15.176:0/2639281111 >> 172.21.15.176:0/2639281111 conn(0x7f0c64074c40 msgr2=0x7f0c64075040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:38.350 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.348+0000 7f0c6bdd1700 1 -- 172.21.15.176:0/2639281111 shutdown_connections 2023-11-16T16:41:38.350 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.348+0000 7f0c6bdd1700 1 -- 172.21.15.176:0/2639281111 wait complete. 2023-11-16T16:41:38.354 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.349+0000 7f0c6bdd1700 1 Processor -- start 2023-11-16T16:41:38.354 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.350+0000 7f0c6bdd1700 1 -- start start 2023-11-16T16:41:38.354 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.350+0000 7f0c6bdd1700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c500035c0 0x7f0c64196180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:38.355 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.350+0000 7f0c6bdd1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c641966c0 0x7f0c64196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:38.355 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.350+0000 7f0c6bdd1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6419f1f0 0x7f0c641a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:38.355 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.350+0000 7f0c6bdd1700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c641a1d80 con 0x7f0c500035c0 2023-11-16T16:41:38.355 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.350+0000 7f0c6bdd1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c641a1ef0 con 0x7f0c6419f1f0 2023-11-16T16:41:38.355 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.350+0000 7f0c6bdd1700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c641a20b0 con 0x7f0c641966c0 2023-11-16T16:41:38.355 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.350+0000 7f0c6936c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c641966c0 0x7f0c64196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:38.355 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.350+0000 7f0c6936c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c641966c0 0x7f0c64196b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37750/0 (socket says 172.21.15.176:37750) 2023-11-16T16:41:38.355 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.350+0000 7f0c6936c700 1 -- 172.21.15.176:0/3008949158 learned_addr learned my addr 172.21.15.176:0/3008949158 (peer_addr_for_me v2:172.21.15.176:0/0) 2023-11-16T16:41:38.355 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.350+0000 7f0c69b6d700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c500035c0 0x7f0c64196180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:38.356 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c6936c700 1 -- 172.21.15.176:0/3008949158 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c641966c0 msgr2=0x7f0c64196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-11-16T16:41:38.356 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c6936c700 1 -- 172.21.15.176:0/3008949158 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c641966c0 msgr2=0x7f0c64196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-16T16:41:38.356 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c6936c700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c641966c0 0x7f0c64196b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-16T16:41:38.356 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c6936c700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c641966c0 0x7f0c64196b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:41:38.356 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c6a36e700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6419f1f0 0x7f0c641a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:38.356 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c6a36e700 1 -- 172.21.15.176:0/3008949158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6419f1f0 msgr2=0x7f0c641a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-11-16T16:41:38.356 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c6a36e700 1 -- 172.21.15.176:0/3008949158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6419f1f0 msgr2=0x7f0c641a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-16T16:41:38.356 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c6a36e700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6419f1f0 0x7f0c641a15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-16T16:41:38.356 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c6a36e700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6419f1f0 0x7f0c641a15d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:41:38.357 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c69b6d700 1 -- 172.21.15.176:0/3008949158 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c641966c0 msgr2=0x7f0c64196b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:41:38.357 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c69b6d700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c641966c0 0x7f0c64196b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:38.357 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c69b6d700 1 -- 172.21.15.176:0/3008949158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6419f1f0 msgr2=0x7f0c641a15d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:41:38.357 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c69b6d700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6419f1f0 0x7f0c641a15d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:38.357 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c69b6d700 1 -- 172.21.15.176:0/3008949158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5400f860 con 0x7f0c500035c0 2023-11-16T16:41:38.357 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c69b6d700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c500035c0 0x7f0c64196180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0c64075f20 tx=0x7f0c54011bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:38.357 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c5affd700 1 -- 172.21.15.176:0/3008949158 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c540054d0 con 0x7f0c500035c0 2023-11-16T16:41:38.357 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c5affd700 1 -- 172.21.15.176:0/3008949158 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c54011eb0 con 0x7f0c500035c0 2023-11-16T16:41:38.357 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c6bdd1700 1 -- 172.21.15.176:0/3008949158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c64106bf0 con 0x7f0c500035c0 2023-11-16T16:41:38.358 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c5affd700 1 -- 172.21.15.176:0/3008949158 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5401d8a0 con 0x7f0c500035c0 2023-11-16T16:41:38.358 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.351+0000 7f0c6bdd1700 1 -- 172.21.15.176:0/3008949158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c64106a60 con 0x7f0c500035c0 2023-11-16T16:41:38.358 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.352+0000 7f0c5affd700 1 -- 172.21.15.176:0/3008949158 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f0c54024040 con 0x7f0c500035c0 2023-11-16T16:41:38.358 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.352+0000 7f0c6bdd1700 1 -- 172.21.15.176:0/3008949158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c64106d80 con 0x7f0c500035c0 2023-11-16T16:41:38.358 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.352+0000 7f0c5affd700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0c48038d40 0x7f0c4803b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:38.358 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.352+0000 7f0c5affd700 1 -- 172.21.15.176:0/3008949158 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0c5404e530 con 0x7f0c500035c0 2023-11-16T16:41:38.358 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.353+0000 7f0c6936c700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0c48038d40 0x7f0c4803b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:38.359 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.353+0000 7f0c6936c700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0c48038d40 0x7f0c4803b1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009180 tx=0x7f0c60008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:38.359 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.358+0000 7f0c5affd700 1 -- 172.21.15.176:0/3008949158 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0c540103d0 con 0x7f0c500035c0 2023-11-16T16:41:38.634 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.632+0000 7f0c6bdd1700 1 -- 172.21.15.176:0/3008949158 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c64062120 con 0x7f0c500035c0 2023-11-16T16:41:38.634 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.633+0000 7f0c5affd700 1 -- 172.21.15.176:0/3008949158 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1270 (secure 0 0 0) 0x7f0c5402c090 con 0x7f0c500035c0 2023-11-16T16:41:38.635 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-11-16T16:41:38.636 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":3,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T16:41:32.734980Z","created":"2023-11-16T16:39:38.269339Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-11-16T16:41:38.637 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.636+0000 7f0c6bdd1700 1 -- 172.21.15.176:0/3008949158 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0c48038d40 msgr2=0x7f0c4803b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:38.638 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.636+0000 7f0c6bdd1700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0c48038d40 0x7f0c4803b1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009180 tx=0x7f0c60008040 comp rx=0 tx=0).stop 2023-11-16T16:41:38.638 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.636+0000 7f0c6bdd1700 1 -- 172.21.15.176:0/3008949158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c500035c0 msgr2=0x7f0c64196180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:38.638 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.636+0000 7f0c6bdd1700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c500035c0 0x7f0c64196180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0c64075f20 tx=0x7f0c54011bd0 comp rx=0 tx=0).stop 2023-11-16T16:41:38.638 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.636+0000 7f0c6bdd1700 1 -- 172.21.15.176:0/3008949158 shutdown_connections 2023-11-16T16:41:38.638 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.636+0000 7f0c6bdd1700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0c48038d40 0x7f0c4803b1f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:38.638 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.636+0000 7f0c6bdd1700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6419f1f0 0x7f0c641a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:38.638 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.636+0000 7f0c6bdd1700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c500035c0 0x7f0c64196180 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:38.639 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.636+0000 7f0c6bdd1700 1 --2- 172.21.15.176:0/3008949158 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c641966c0 0x7f0c64196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:38.639 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.636+0000 7f0c6bdd1700 1 -- 172.21.15.176:0/3008949158 >> 172.21.15.176:0/3008949158 conn(0x7f0c64074c40 msgr2=0x7f0c64102950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:38.639 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.636+0000 7f0c6bdd1700 1 -- 172.21.15.176:0/3008949158 shutdown_connections 2023-11-16T16:41:38.639 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:38.636+0000 7f0c6bdd1700 1 -- 172.21.15.176:0/3008949158 wait complete. 2023-11-16T16:41:38.639 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 3 2023-11-16T16:41:38.971 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:38 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:38.971 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:38 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:38.971 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:38 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:38.971 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:38 smithi176 ceph-mon[131630]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:38.971 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:38 smithi176 ceph-mon[131630]: from='client.? 172.21.15.176:0/3008949158' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T16:41:38.972 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:38 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:39.029 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-11-16T16:41:39.030 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph config generate-minimal-conf 2023-11-16T16:41:39.101 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:38 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:39.101 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:38 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:39.101 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:38 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:39.101 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:38 smithi044 ceph-mon[128750]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:39.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:38 smithi044 ceph-mon[128750]: from='client.? 172.21.15.176:0/3008949158' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T16:41:39.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:38 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:39.909 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:41:40.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:40.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:41:40.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:40.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: mon.a calling monitor election 2023-11-16T16:41:40.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: mon.c calling monitor election 2023-11-16T16:41:40.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:40.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:40.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:40.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:40.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:40.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:40.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:40.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:40.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T16:41:40.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: monmap e3: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],c=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0]} removed_ranks: {} 2023-11-16T16:41:40.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: fsmap 2023-11-16T16:41:40.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:40.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: mgrmap e13: x(active, since 55s) 2023-11-16T16:41:40.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-11-16T16:41:40.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-11-16T16:41:40.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-11-16T16:41:40.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: mon.b (rank 2) addr [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] is down (out of quorum) 2023-11-16T16:41:40.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:40.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:40.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:40.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:40.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:40.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:40.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: from='client.? 172.21.15.176:0/3008949158' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T16:41:40.232 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:40.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: mon.b calling monitor election 2023-11-16T16:41:40.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:41:40.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: Updating smithi044:/etc/ceph/ceph.conf 2023-11-16T16:41:40.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: Updating smithi077:/etc/ceph/ceph.conf 2023-11-16T16:41:40.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: Updating smithi176:/etc/ceph/ceph.conf 2023-11-16T16:41:40.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: mon.b calling monitor election 2023-11-16T16:41:40.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: mon.a calling monitor election 2023-11-16T16:41:40.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: mon.c calling monitor election 2023-11-16T16:41:40.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-11-16T16:41:40.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: monmap e3: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],c=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0]} removed_ranks: {} 2023-11-16T16:41:40.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: fsmap 2023-11-16T16:41:40.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:40.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: mgrmap e13: x(active, since 57s) 2023-11-16T16:41:40.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-11-16T16:41:40.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: Cluster is now healthy 2023-11-16T16:41:40.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: overall HEALTH_OK 2023-11-16T16:41:40.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:40.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:40.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:40.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:40 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:40.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: mon.b calling monitor election 2023-11-16T16:41:40.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:41:40.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: Updating smithi044:/etc/ceph/ceph.conf 2023-11-16T16:41:40.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: Updating smithi077:/etc/ceph/ceph.conf 2023-11-16T16:41:40.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: Updating smithi176:/etc/ceph/ceph.conf 2023-11-16T16:41:40.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: mon.b calling monitor election 2023-11-16T16:41:40.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: mon.a calling monitor election 2023-11-16T16:41:40.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: mon.c calling monitor election 2023-11-16T16:41:40.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-11-16T16:41:40.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: monmap e3: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],c=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0]} removed_ranks: {} 2023-11-16T16:41:40.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: fsmap 2023-11-16T16:41:40.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:40.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: mgrmap e13: x(active, since 57s) 2023-11-16T16:41:40.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-11-16T16:41:40.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: Cluster is now healthy 2023-11-16T16:41:40.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: overall HEALTH_OK 2023-11-16T16:41:40.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:40.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:40.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:40.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:40 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: mon.b calling monitor election 2023-11-16T16:41:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:41:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: Updating smithi044:/etc/ceph/ceph.conf 2023-11-16T16:41:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: Updating smithi077:/etc/ceph/ceph.conf 2023-11-16T16:41:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: Updating smithi176:/etc/ceph/ceph.conf 2023-11-16T16:41:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: mon.b calling monitor election 2023-11-16T16:41:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: mon.a calling monitor election 2023-11-16T16:41:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: mon.c calling monitor election 2023-11-16T16:41:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-11-16T16:41:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: monmap e3: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],c=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0]} removed_ranks: {} 2023-11-16T16:41:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: fsmap 2023-11-16T16:41:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: mgrmap e13: x(active, since 57s) 2023-11-16T16:41:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-11-16T16:41:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: Cluster is now healthy 2023-11-16T16:41:40.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: overall HEALTH_OK 2023-11-16T16:41:40.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:40.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:40.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:40.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:40 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:41.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.426+0000 7f1b806a9700 1 -- 172.21.15.44:0/3357217618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b78100df0 msgr2=0x7f1b781011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:41.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.426+0000 7f1b806a9700 1 --2- 172.21.15.44:0/3357217618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b78100df0 0x7f1b781011c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009910 tx=0x7f1b74009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:41.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.426+0000 7f1b806a9700 1 -- 172.21.15.44:0/3357217618 shutdown_connections 2023-11-16T16:41:41.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.426+0000 7f1b806a9700 1 --2- 172.21.15.44:0/3357217618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b78100df0 0x7f1b781011c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:41.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.426+0000 7f1b806a9700 1 -- 172.21.15.44:0/3357217618 >> 172.21.15.44:0/3357217618 conn(0x7f1b78074930 msgr2=0x7f1b78074d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:41.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.427+0000 7f1b806a9700 1 -- 172.21.15.44:0/3357217618 shutdown_connections 2023-11-16T16:41:41.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.427+0000 7f1b806a9700 1 -- 172.21.15.44:0/3357217618 wait complete. 2023-11-16T16:41:41.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.428+0000 7f1b806a9700 1 Processor -- start 2023-11-16T16:41:41.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.428+0000 7f1b806a9700 1 -- start start 2023-11-16T16:41:41.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.428+0000 7f1b806a9700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b78100df0 0x7f1b78195360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:41.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.428+0000 7f1b806a9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b781958a0 0x7f1b78199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:41.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.428+0000 7f1b806a9700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b7819a250 0x7f1b7819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:41.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.428+0000 7f1b806a9700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b78195f70 con 0x7f1b78100df0 2023-11-16T16:41:41.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.428+0000 7f1b806a9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b781a1fc0 con 0x7f1b781958a0 2023-11-16T16:41:41.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.428+0000 7f1b806a9700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b781a2100 con 0x7f1b7819a250 2023-11-16T16:41:41.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.428+0000 7f1b7e445700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b78100df0 0x7f1b78195360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:41.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.429+0000 7f1b7e445700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b78100df0 0x7f1b78195360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44144/0 (socket says 172.21.15.44:44144) 2023-11-16T16:41:41.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.429+0000 7f1b7e445700 1 -- 172.21.15.44:0/1274858297 learned_addr learned my addr 172.21.15.44:0/1274858297 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:41:41.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.429+0000 7f1b7dc44700 1 --2- 172.21.15.44:0/1274858297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b781958a0 0x7f1b78199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:41.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.429+0000 7f1b7ec46700 1 --2- 172.21.15.44:0/1274858297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b7819a250 0x7f1b7819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:41.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.429+0000 7f1b7e445700 1 -- 172.21.15.44:0/1274858297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b7819a250 msgr2=0x7f1b7819a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:41.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.429+0000 7f1b7e445700 1 --2- 172.21.15.44:0/1274858297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b7819a250 0x7f1b7819a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:41.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.429+0000 7f1b7e445700 1 -- 172.21.15.44:0/1274858297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b781958a0 msgr2=0x7f1b78199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:41.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.429+0000 7f1b7e445700 1 --2- 172.21.15.44:0/1274858297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b781958a0 0x7f1b78199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:41.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.429+0000 7f1b7e445700 1 -- 172.21.15.44:0/1274858297 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b740095f0 con 0x7f1b78100df0 2023-11-16T16:41:41.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.429+0000 7f1b7e445700 1 --2- 172.21.15.44:0/1274858297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b78100df0 0x7f1b78195360 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1b74004d00 tx=0x7f1b74004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:41.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.430+0000 7f1b6f7fe700 1 -- 172.21.15.44:0/1274858297 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7401d070 con 0x7f1b78100df0 2023-11-16T16:41:41.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.430+0000 7f1b6f7fe700 1 -- 172.21.15.44:0/1274858297 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b7400be20 con 0x7f1b78100df0 2023-11-16T16:41:41.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.430+0000 7f1b806a9700 1 -- 172.21.15.44:0/1274858297 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7810a190 con 0x7f1b78100df0 2023-11-16T16:41:41.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.430+0000 7f1b6f7fe700 1 -- 172.21.15.44:0/1274858297 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b74022db0 con 0x7f1b78100df0 2023-11-16T16:41:41.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.430+0000 7f1b806a9700 1 -- 172.21.15.44:0/1274858297 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7810a000 con 0x7f1b78100df0 2023-11-16T16:41:41.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.431+0000 7f1b6f7fe700 1 -- 172.21.15.44:0/1274858297 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f1b7400b960 con 0x7f1b78100df0 2023-11-16T16:41:41.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.431+0000 7f1b806a9700 1 -- 172.21.15.44:0/1274858297 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7810a320 con 0x7f1b78100df0 2023-11-16T16:41:41.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.431+0000 7f1b6f7fe700 1 --2- 172.21.15.44:0/1274858297 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1b64038d40 0x7f1b6403b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:41.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.431+0000 7f1b6f7fe700 1 -- 172.21.15.44:0/1274858297 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1b7404c990 con 0x7f1b78100df0 2023-11-16T16:41:41.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.432+0000 7f1b7dc44700 1 --2- 172.21.15.44:0/1274858297 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1b64038d40 0x7f1b6403b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:41.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.432+0000 7f1b7dc44700 1 --2- 172.21.15.44:0/1274858297 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1b64038d40 0x7f1b6403b1f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009ad0 tx=0x7f1b68009210 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:41.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.440+0000 7f1b6f7fe700 1 -- 172.21.15.44:0/1274858297 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1b7400edb0 con 0x7f1b78100df0 2023-11-16T16:41:41.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.660+0000 7f1b806a9700 1 -- 172.21.15.44:0/1274858297 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f1b781966c0 con 0x7f1b78100df0 2023-11-16T16:41:41.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.661+0000 7f1b6f7fe700 1 -- 172.21.15.44:0/1274858297 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7f1b74027020 con 0x7f1b78100df0 2023-11-16T16:41:41.663 INFO:teuthology.orchestra.run.smithi044.stdout:# minimal ceph.conf for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:41:41.663 INFO:teuthology.orchestra.run.smithi044.stdout:[global] 2023-11-16T16:41:41.663 INFO:teuthology.orchestra.run.smithi044.stdout: fsid = 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:41:41.663 INFO:teuthology.orchestra.run.smithi044.stdout: mon_host = [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2023-11-16T16:41:41.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.663+0000 7f1b806a9700 1 -- 172.21.15.44:0/1274858297 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1b64038d40 msgr2=0x7f1b6403b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:41.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.663+0000 7f1b806a9700 1 --2- 172.21.15.44:0/1274858297 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1b64038d40 0x7f1b6403b1f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009ad0 tx=0x7f1b68009210 comp rx=0 tx=0).stop 2023-11-16T16:41:41.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.663+0000 7f1b806a9700 1 -- 172.21.15.44:0/1274858297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b78100df0 msgr2=0x7f1b78195360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:41.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.663+0000 7f1b806a9700 1 --2- 172.21.15.44:0/1274858297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b78100df0 0x7f1b78195360 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1b74004d00 tx=0x7f1b74004d30 comp rx=0 tx=0).stop 2023-11-16T16:41:41.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.663+0000 7f1b806a9700 1 -- 172.21.15.44:0/1274858297 shutdown_connections 2023-11-16T16:41:41.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.663+0000 7f1b806a9700 1 --2- 172.21.15.44:0/1274858297 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1b64038d40 0x7f1b6403b1f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:41.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.663+0000 7f1b806a9700 1 --2- 172.21.15.44:0/1274858297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b7819a250 0x7f1b7819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:41.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.663+0000 7f1b806a9700 1 --2- 172.21.15.44:0/1274858297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b78100df0 0x7f1b78195360 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:41.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.663+0000 7f1b806a9700 1 --2- 172.21.15.44:0/1274858297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b781958a0 0x7f1b78199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:41.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.663+0000 7f1b806a9700 1 -- 172.21.15.44:0/1274858297 >> 172.21.15.44:0/1274858297 conn(0x7f1b78074930 msgr2=0x7f1b78104850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:41.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.664+0000 7f1b806a9700 1 -- 172.21.15.44:0/1274858297 shutdown_connections 2023-11-16T16:41:41.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:41.664+0000 7f1b806a9700 1 -- 172.21.15.44:0/1274858297 wait complete. 2023-11-16T16:41:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:41 smithi077 ceph-mon[131204]: Updating smithi176:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:41 smithi077 ceph-mon[131204]: Updating smithi077:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:41.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:41 smithi077 ceph-mon[131204]: Updating smithi044:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:41.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:41 smithi077 ceph-mon[131204]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:41.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:41 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:41.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:41 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:41.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:41 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:41.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:41 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:41:41.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:41 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:41.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:41 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:41.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:41 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:41.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:41 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:41.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:41 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1274858297' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:42.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:41 smithi044 ceph-mon[128750]: Updating smithi176:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:42.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:41 smithi044 ceph-mon[128750]: Updating smithi077:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:42.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:41 smithi044 ceph-mon[128750]: Updating smithi044:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:42.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:41 smithi044 ceph-mon[128750]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:42.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:41 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:42.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:41 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:42.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:41 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:42.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:41 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:41:42.103 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:41 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:42.103 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:41 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:42.103 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:41 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:42.103 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:41 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:42.103 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:41 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1274858297' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:41 smithi176 ceph-mon[131630]: Updating smithi176:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:41 smithi176 ceph-mon[131630]: Updating smithi077:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:41 smithi176 ceph-mon[131630]: Updating smithi044:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:41 smithi176 ceph-mon[131630]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:41 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:41 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:41 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:41 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:41:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:41 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:41 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:41 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:42.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:41 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:42.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:41 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1274858297' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:42.200 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-11-16T16:41:42.200 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:41:42.200 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.conf 2023-11-16T16:41:42.273 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:41:42.273 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:42.391 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-16T16:41:42.391 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.conf 2023-11-16T16:41:42.426 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-16T16:41:42.426 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:42.498 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-11-16T16:41:42.498 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.conf 2023-11-16T16:41:42.535 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-11-16T16:41:42.535 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:42.606 INFO:tasks.cephadm:Adding mgr.x on smithi044 2023-11-16T16:41:42.607 INFO:tasks.cephadm:Adding mgr.y on smithi077 2023-11-16T16:41:42.607 INFO:tasks.cephadm:Adding mgr.z on smithi176 2023-11-16T16:41:42.607 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch apply mgr '3;smithi044=x;smithi077=y;smithi176=z' 2023-11-16T16:41:42.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:42 smithi176 ceph-mon[131630]: Reconfiguring mon.a (unknown last config time)... 2023-11-16T16:41:42.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:42 smithi176 ceph-mon[131630]: Reconfiguring daemon mon.a on smithi044 2023-11-16T16:41:42.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:42 smithi044 ceph-mon[128750]: Reconfiguring mon.a (unknown last config time)... 2023-11-16T16:41:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:42 smithi044 ceph-mon[128750]: Reconfiguring daemon mon.a on smithi044 2023-11-16T16:41:43.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:43.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:42 smithi077 ceph-mon[131204]: Reconfiguring mon.a (unknown last config time)... 2023-11-16T16:41:43.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:42 smithi077 ceph-mon[131204]: Reconfiguring daemon mon.a on smithi044 2023-11-16T16:41:43.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:43.286 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:41:43.973 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:43 smithi077 ceph-mon[131204]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:43.973 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:43.973 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:43.973 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:43.974 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:43.974 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:44.035 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:43 smithi176 ceph-mon[131630]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:44.035 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:44.035 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:44.035 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:44.035 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:44.035 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:44.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:43 smithi044 ceph-mon[128750]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:44.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:44.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:44.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:44.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:44.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:44.489 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.488+0000 7f955ac6d700 1 -- 172.21.15.176:0/1645285903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95540fe110 msgr2=0x7f95540fe520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:44.489 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.488+0000 7f955ac6d700 1 --2- 172.21.15.176:0/1645285903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95540fe110 0x7f95540fe520 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9548009910 tx=0x7f9548009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:44.489 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.488+0000 7f955ac6d700 1 -- 172.21.15.176:0/1645285903 shutdown_connections 2023-11-16T16:41:44.489 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.488+0000 7f955ac6d700 1 --2- 172.21.15.176:0/1645285903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95540fe110 0x7f95540fe520 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:44.489 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.488+0000 7f955ac6d700 1 -- 172.21.15.176:0/1645285903 >> 172.21.15.176:0/1645285903 conn(0x7f95540f9a60 msgr2=0x7f95540fbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:44.489 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.488+0000 7f955ac6d700 1 -- 172.21.15.176:0/1645285903 shutdown_connections 2023-11-16T16:41:44.490 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.488+0000 7f955ac6d700 1 -- 172.21.15.176:0/1645285903 wait complete. 2023-11-16T16:41:44.491 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.488+0000 7f955ac6d700 1 Processor -- start 2023-11-16T16:41:44.491 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.488+0000 7f955ac6d700 1 -- start start 2023-11-16T16:41:44.492 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f955ac6d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95540fe110 0x7f9554100af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:44.492 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f955ac6d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95540ff140 0x7f95540ff5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:44.492 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f955ac6d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95540ffaf0 0x7f955419f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:44.492 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f955ac6d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955419f630 con 0x7f95540fe110 2023-11-16T16:41:44.492 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f955ac6d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955419f770 con 0x7f95540ff140 2023-11-16T16:41:44.492 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f955ac6d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955419f8b0 con 0x7f95540ffaf0 2023-11-16T16:41:44.492 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f955920a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95540ffaf0 0x7f955419f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:44.492 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f955920a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95540ffaf0 0x7f955419f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37760/0 (socket says 172.21.15.176:37760) 2023-11-16T16:41:44.492 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f955920a700 1 -- 172.21.15.176:0/3813179385 learned_addr learned my addr 172.21.15.176:0/3813179385 (peer_addr_for_me v2:172.21.15.176:0/0) 2023-11-16T16:41:44.492 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f9558a09700 1 --2- 172.21.15.176:0/3813179385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95540fe110 0x7f9554100af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:44.492 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f9553fff700 1 --2- 172.21.15.176:0/3813179385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95540ff140 0x7f95540ff5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:44.492 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f955920a700 1 -- 172.21.15.176:0/3813179385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95540ff140 msgr2=0x7f95540ff5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:44.493 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f955920a700 1 --2- 172.21.15.176:0/3813179385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95540ff140 0x7f95540ff5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:44.493 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f955920a700 1 -- 172.21.15.176:0/3813179385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95540fe110 msgr2=0x7f9554100af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:44.493 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f955920a700 1 --2- 172.21.15.176:0/3813179385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95540fe110 0x7f9554100af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:44.493 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f955920a700 1 -- 172.21.15.176:0/3813179385 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95480095f0 con 0x7f95540ffaf0 2023-11-16T16:41:44.493 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f9553fff700 1 --2- 172.21.15.176:0/3813179385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95540ff140 0x7f95540ff5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:41:44.493 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f955920a700 1 --2- 172.21.15.176:0/3813179385 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95540ffaf0 0x7f955419f0f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f954c00ba50 tx=0x7f954c00bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:44.493 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.489+0000 7f9551ffb700 1 -- 172.21.15.176:0/3813179385 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c00c9b0 con 0x7f95540ffaf0 2023-11-16T16:41:44.493 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.490+0000 7f955ac6d700 1 -- 172.21.15.176:0/3813179385 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9554105cc0 con 0x7f95540ffaf0 2023-11-16T16:41:44.493 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.490+0000 7f9551ffb700 1 -- 172.21.15.176:0/3813179385 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f954c014440 con 0x7f95540ffaf0 2023-11-16T16:41:44.493 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.490+0000 7f9551ffb700 1 -- 172.21.15.176:0/3813179385 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c007590 con 0x7f95540ffaf0 2023-11-16T16:41:44.493 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.490+0000 7f955ac6d700 1 -- 172.21.15.176:0/3813179385 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9554105b30 con 0x7f95540ffaf0 2023-11-16T16:41:44.494 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.491+0000 7f9551ffb700 1 -- 172.21.15.176:0/3813179385 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f954c0077b0 con 0x7f95540ffaf0 2023-11-16T16:41:44.494 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.491+0000 7f955ac6d700 1 -- 172.21.15.176:0/3813179385 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f955418e9f0 con 0x7f95540ffaf0 2023-11-16T16:41:44.494 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.491+0000 7f9551ffb700 1 --2- 172.21.15.176:0/3813179385 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9544038cf0 0x7f954403b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:44.495 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.491+0000 7f9551ffb700 1 -- 172.21.15.176:0/3813179385 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f954c04aa70 con 0x7f95540ffaf0 2023-11-16T16:41:44.495 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.492+0000 7f9558a09700 1 --2- 172.21.15.176:0/3813179385 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9544038cf0 0x7f954403b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:44.495 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.492+0000 7f9558a09700 1 --2- 172.21.15.176:0/3813179385 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9544038cf0 0x7f954403b1a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9548005130 tx=0x7f9548005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:44.497 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.496+0000 7f9551ffb700 1 -- 172.21.15.176:0/3813179385 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f954c01b8b0 con 0x7f95540ffaf0 2023-11-16T16:41:44.703 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.701+0000 7f955ac6d700 1 -- 172.21.15.176:0/3813179385 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi044=x;smithi077=y;smithi176=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f9554061bf0 con 0x7f9544038cf0 2023-11-16T16:41:44.710 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.708+0000 7f9551ffb700 1 -- 172.21.15.176:0/3813179385 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9554061bf0 con 0x7f9544038cf0 2023-11-16T16:41:44.710 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled mgr update... 2023-11-16T16:41:44.712 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.711+0000 7f955ac6d700 1 -- 172.21.15.176:0/3813179385 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9544038cf0 msgr2=0x7f954403b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:44.712 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.711+0000 7f955ac6d700 1 --2- 172.21.15.176:0/3813179385 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9544038cf0 0x7f954403b1a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9548005130 tx=0x7f9548005fd0 comp rx=0 tx=0).stop 2023-11-16T16:41:44.712 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.711+0000 7f955ac6d700 1 -- 172.21.15.176:0/3813179385 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95540ffaf0 msgr2=0x7f955419f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:44.712 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.711+0000 7f955ac6d700 1 --2- 172.21.15.176:0/3813179385 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95540ffaf0 0x7f955419f0f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f954c00ba50 tx=0x7f954c00bd60 comp rx=0 tx=0).stop 2023-11-16T16:41:44.713 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.711+0000 7f955ac6d700 1 -- 172.21.15.176:0/3813179385 shutdown_connections 2023-11-16T16:41:44.713 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.711+0000 7f955ac6d700 1 --2- 172.21.15.176:0/3813179385 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9544038cf0 0x7f954403b1a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:44.713 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.711+0000 7f955ac6d700 1 --2- 172.21.15.176:0/3813179385 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95540ffaf0 0x7f955419f0f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:44.713 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.711+0000 7f955ac6d700 1 --2- 172.21.15.176:0/3813179385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95540fe110 0x7f9554100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:44.713 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.711+0000 7f955ac6d700 1 --2- 172.21.15.176:0/3813179385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95540ff140 0x7f95540ff5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:44.713 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.711+0000 7f955ac6d700 1 -- 172.21.15.176:0/3813179385 >> 172.21.15.176:0/3813179385 conn(0x7f95540f9a60 msgr2=0x7f95540fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:44.714 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.711+0000 7f955ac6d700 1 -- 172.21.15.176:0/3813179385 shutdown_connections 2023-11-16T16:41:44.714 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:41:44.711+0000 7f955ac6d700 1 -- 172.21.15.176:0/3813179385 wait complete. 2023-11-16T16:41:44.931 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:44 smithi176 ceph-mon[131630]: Reconfiguring mon.b (monmap changed)... 2023-11-16T16:41:44.931 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:44 smithi176 ceph-mon[131630]: Reconfiguring daemon mon.b on smithi077 2023-11-16T16:41:44.931 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:44 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:44.931 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:44 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:44.931 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:44 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:44.931 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:44 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:44.931 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:44 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:44.931 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:44 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:45.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:44 smithi044 ceph-mon[128750]: Reconfiguring mon.b (monmap changed)... 2023-11-16T16:41:45.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:44 smithi044 ceph-mon[128750]: Reconfiguring daemon mon.b on smithi077 2023-11-16T16:41:45.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:44 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:45.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:44 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:45.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:44 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:45.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:44 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:45.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:44 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:45.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:44 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:45.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:44 smithi077 ceph-mon[131204]: Reconfiguring mon.b (monmap changed)... 2023-11-16T16:41:45.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:44 smithi077 ceph-mon[131204]: Reconfiguring daemon mon.b on smithi077 2023-11-16T16:41:45.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:44 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:45.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:44 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:45.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:44 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:45.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:44 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:45.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:44 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:45.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:44 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:45.448 DEBUG:teuthology.orchestra.run.smithi077:mgr.y> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mgr.y.service 2023-11-16T16:41:45.451 DEBUG:teuthology.orchestra.run.smithi176:mgr.z> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mgr.z.service 2023-11-16T16:41:45.453 INFO:tasks.cephadm:Deploying OSDs... 2023-11-16T16:41:45.453 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:41:45.453 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/scratch_devs of=/dev/stdout 2023-11-16T16:41:45.482 INFO:journalctl@ceph.mgr.z.smithi176.stdout:-- Logs begin at Thu 2023-11-16 16:14:40 UTC. -- 2023-11-16T16:41:45.485 INFO:journalctl@ceph.mgr.y.smithi077.stdout:-- Logs begin at Thu 2023-11-16 16:15:33 UTC. -- 2023-11-16T16:41:45.502 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-16T16:41:45.503 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_1 2023-11-16T16:41:45.590 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-16T16:41:45.590 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:45.591 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 452 Links: 1 2023-11-16T16:41:45.591 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:45.591 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:45.591 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2023-11-16 16:40:52.095471784 +0000 2023-11-16T16:41:45.591 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2023-11-16 16:33:54.196783266 +0000 2023-11-16T16:41:45.592 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2023-11-16 16:33:54.196783266 +0000 2023-11-16T16:41:45.592 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2023-11-16 16:33:54.169783676 +0000 2023-11-16T16:41:45.592 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-16T16:41:45.663 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2023-11-16T16:41:45.663 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2023-11-16T16:41:45.663 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000291542 s, 1.8 MB/s 2023-11-16T16:41:45.666 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-16T16:41:45.713 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_2 2023-11-16T16:41:45.798 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-16T16:41:45.799 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:45.799 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 458 Links: 1 2023-11-16T16:41:45.799 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:45.799 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:45.799 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2023-11-16 16:40:52.095471784 +0000 2023-11-16T16:41:45.800 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2023-11-16 16:33:55.044770392 +0000 2023-11-16T16:41:45.800 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2023-11-16 16:33:55.044770392 +0000 2023-11-16T16:41:45.800 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2023-11-16 16:33:55.017770802 +0000 2023-11-16T16:41:45.801 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-16T16:41:45.906 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2023-11-16T16:41:45.906 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2023-11-16T16:41:45.907 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000302416 s, 1.7 MB/s 2023-11-16T16:41:45.908 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:45 smithi044 ceph-mon[128750]: Reconfiguring mon.c (monmap changed)... 2023-11-16T16:41:45.908 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:45 smithi044 ceph-mon[128750]: Reconfiguring daemon mon.c on smithi176 2023-11-16T16:41:45.908 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:45 smithi044 ceph-mon[128750]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:45.908 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:45 smithi044 ceph-mon[128750]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi044=x;smithi077=y;smithi176=z", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:45.909 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:45 smithi044 ceph-mon[128750]: Saving service mgr spec with placement smithi044=x;smithi077=y;smithi176=z;count:3 2023-11-16T16:41:45.909 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:45.909 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:45.909 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:45.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:45.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:45.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:45.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:45.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T16:41:45.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:45.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:45.913 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-16T16:41:45.959 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_3 2023-11-16T16:41:45.995 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:45 smithi176 ceph-mon[131630]: Reconfiguring mon.c (monmap changed)... 2023-11-16T16:41:45.996 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:45 smithi176 ceph-mon[131630]: Reconfiguring daemon mon.c on smithi176 2023-11-16T16:41:45.996 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:45 smithi176 ceph-mon[131630]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:45.996 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:45 smithi176 ceph-mon[131630]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi044=x;smithi077=y;smithi176=z", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:45.996 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:45 smithi176 ceph-mon[131630]: Saving service mgr spec with placement smithi044=x;smithi077=y;smithi176=z;count:3 2023-11-16T16:41:45.997 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:45.997 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:45.997 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:45.997 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:45.998 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:45.998 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:45.998 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:45.998 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T16:41:45.999 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:45.999 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:46.043 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-16T16:41:46.044 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:46.044 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 464 Links: 1 2023-11-16T16:41:46.044 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:46.044 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:46.044 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2023-11-16 16:40:52.095471784 +0000 2023-11-16T16:41:46.045 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2023-11-16 16:33:55.883757655 +0000 2023-11-16T16:41:46.045 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2023-11-16 16:33:55.883757655 +0000 2023-11-16T16:41:46.045 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2023-11-16 16:33:55.853758111 +0000 2023-11-16T16:41:46.046 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-16T16:41:46.150 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2023-11-16T16:41:46.150 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2023-11-16T16:41:46.150 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000315018 s, 1.6 MB/s 2023-11-16T16:41:46.153 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-16T16:41:46.199 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_4 2023-11-16T16:41:46.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:45 smithi077 ceph-mon[131204]: Reconfiguring mon.c (monmap changed)... 2023-11-16T16:41:46.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:45 smithi077 ceph-mon[131204]: Reconfiguring daemon mon.c on smithi176 2023-11-16T16:41:46.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:45 smithi077 ceph-mon[131204]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:46.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:45 smithi077 ceph-mon[131204]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi044=x;smithi077=y;smithi176=z", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:46.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:45 smithi077 ceph-mon[131204]: Saving service mgr spec with placement smithi044=x;smithi077=y;smithi176=z;count:3 2023-11-16T16:41:46.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:46.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:46.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:46.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:46.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:46.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:46.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:46.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T16:41:46.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:46.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:46.284 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-16T16:41:46.285 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:46.285 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 470 Links: 1 2023-11-16T16:41:46.285 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:46.285 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:46.285 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2023-11-16 16:40:52.095471784 +0000 2023-11-16T16:41:46.285 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2023-11-16 16:33:56.711745085 +0000 2023-11-16T16:41:46.286 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2023-11-16 16:33:56.711745085 +0000 2023-11-16T16:41:46.286 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2023-11-16 16:33:56.680745556 +0000 2023-11-16T16:41:46.286 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-16T16:41:46.391 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2023-11-16T16:41:46.391 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2023-11-16T16:41:46.391 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.00029843 s, 1.7 MB/s 2023-11-16T16:41:46.394 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-16T16:41:46.441 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-16T16:41:46.441 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2023-11-16T16:41:46.462 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-16T16:41:46.462 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2023-11-16T16:41:46.523 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-16T16:41:46.523 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:46.523 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 386000 Links: 1 2023-11-16T16:41:46.523 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:46.524 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:46.524 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2023-11-16 16:41:32.189479637 +0000 2023-11-16T16:41:46.524 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2023-11-16 16:30:32.413305031 +0000 2023-11-16T16:41:46.524 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2023-11-16 16:30:32.413305031 +0000 2023-11-16T16:41:46.524 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2023-11-16T16:41:46.524 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-16T16:41:46.593 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2023-11-16T16:41:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2023-11-16T16:41:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000277303 s, 1.8 MB/s 2023-11-16T16:41:46.595 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-16T16:41:46.656 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2023-11-16T16:41:46.714 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-16T16:41:46.715 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:46.715 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 392660 Links: 1 2023-11-16T16:41:46.715 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:46.715 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:46.715 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2023-11-16 16:41:32.189479637 +0000 2023-11-16T16:41:46.715 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2023-11-16 16:30:32.904293739 +0000 2023-11-16T16:41:46.716 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2023-11-16 16:30:32.904293739 +0000 2023-11-16T16:41:46.716 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2023-11-16T16:41:46.716 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-16T16:41:46.782 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2023-11-16T16:41:46.783 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2023-11-16T16:41:46.783 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000225225 s, 2.3 MB/s 2023-11-16T16:41:46.784 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-16T16:41:46.842 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2023-11-16T16:41:46.901 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-16T16:41:46.901 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:46.901 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 393340 Links: 1 2023-11-16T16:41:46.901 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:46.901 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:46.901 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2023-11-16 16:41:32.189479637 +0000 2023-11-16T16:41:46.901 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2023-11-16 16:30:33.409282125 +0000 2023-11-16T16:41:46.901 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2023-11-16 16:30:33.409282125 +0000 2023-11-16T16:41:46.901 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2023-11-16T16:41:46.902 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-16T16:41:46.965 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:46 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:41:46.965 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:46 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:41:46.965 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:46 smithi077 ceph-mon[131204]: Deploying daemon mgr.z on smithi176 2023-11-16T16:41:46.967 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2023-11-16T16:41:46.967 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2023-11-16T16:41:46.967 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000221364 s, 2.3 MB/s 2023-11-16T16:41:46.968 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-16T16:41:47.027 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2023-11-16T16:41:47.034 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:46 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:41:47.035 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:46 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:41:47.035 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:46 smithi176 ceph-mon[131630]: Deploying daemon mgr.z on smithi176 2023-11-16T16:41:47.035 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:46 smithi176 systemd[1]: Starting Ceph mgr.z for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:41:47.084 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-16T16:41:47.084 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:47.085 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 389542 Links: 1 2023-11-16T16:41:47.085 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:47.085 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:47.085 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2023-11-16 16:41:32.189479637 +0000 2023-11-16T16:41:47.085 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2023-11-16 16:30:33.914270510 +0000 2023-11-16T16:41:47.085 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2023-11-16 16:30:33.914270510 +0000 2023-11-16T16:41:47.085 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2023-11-16T16:41:47.086 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-16T16:41:47.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:46 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:41:47.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:46 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:41:47.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:46 smithi044 ceph-mon[128750]: Deploying daemon mgr.z on smithi176 2023-11-16T16:41:47.150 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2023-11-16T16:41:47.151 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2023-11-16T16:41:47.151 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000186836 s, 2.7 MB/s 2023-11-16T16:41:47.152 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-16T16:41:47.210 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-11-16T16:41:47.210 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2023-11-16T16:41:47.228 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-16T16:41:47.228 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2023-11-16T16:41:47.288 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-16T16:41:47.288 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:47.288 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 6h/6d Inode: 381806 Links: 1 2023-11-16T16:41:47.288 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:47.289 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:47.289 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2023-11-16 16:41:46.717196425 +0000 2023-11-16T16:41:47.289 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2023-11-16 16:30:32.399947537 +0000 2023-11-16T16:41:47.289 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2023-11-16 16:30:32.399947537 +0000 2023-11-16T16:41:47.289 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2023-11-16T16:41:47.289 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-16T16:41:47.353 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:47 smithi176 podman[133463]: 2023-11-16T16:41:47.353 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:47 smithi176 bash[133463]: 1fc33ae431a21ffacdeed870f00038e810f24621c90bd895d3d5e3bab95591df 2023-11-16T16:41:47.353 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:47 smithi176 ceph-mgr[133522]: -- 172.21.15.176:0/1116785821 <== mon.1 v2:172.21.15.176:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5557f7b4d520 con 0x5557f7b22c00 2023-11-16T16:41:47.356 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2023-11-16T16:41:47.357 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2023-11-16T16:41:47.357 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000244211 s, 2.1 MB/s 2023-11-16T16:41:47.358 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-16T16:41:47.417 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2023-11-16T16:41:47.474 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-16T16:41:47.474 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:47.475 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 6h/6d Inode: 392339 Links: 1 2023-11-16T16:41:47.475 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:47.475 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:47.475 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2023-11-16 16:41:46.717196425 +0000 2023-11-16T16:41:47.475 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2023-11-16 16:30:32.890943791 +0000 2023-11-16T16:41:47.475 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2023-11-16 16:30:32.890943791 +0000 2023-11-16T16:41:47.475 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2023-11-16T16:41:47.476 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-16T16:41:47.540 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2023-11-16T16:41:47.541 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2023-11-16T16:41:47.541 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000248602 s, 2.1 MB/s 2023-11-16T16:41:47.542 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-16T16:41:47.600 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2023-11-16T16:41:47.645 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:47 smithi176 systemd[1]: Started Ceph mgr.z for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:41:47.645 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:47.406+0000 7f66d8d6e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T16:41:47.645 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:47.490+0000 7f66d8d6e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T16:41:47.645 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:47.581+0000 7f66d8d6e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T16:41:47.657 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-16T16:41:47.658 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:47.658 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 6h/6d Inode: 381948 Links: 1 2023-11-16T16:41:47.658 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:47.658 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:47.658 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2023-11-16 16:41:46.717196425 +0000 2023-11-16T16:41:47.658 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2023-11-16 16:30:33.393939955 +0000 2023-11-16T16:41:47.658 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2023-11-16 16:30:33.393939955 +0000 2023-11-16T16:41:47.658 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2023-11-16T16:41:47.659 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-16T16:41:47.724 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2023-11-16T16:41:47.724 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2023-11-16T16:41:47.724 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000223892 s, 2.3 MB/s 2023-11-16T16:41:47.725 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-16T16:41:47.783 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2023-11-16T16:41:47.841 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-16T16:41:47.842 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:47.842 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 6h/6d Inode: 393709 Links: 1 2023-11-16T16:41:47.842 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:47.842 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:47.843 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2023-11-16 16:41:46.718196417 +0000 2023-11-16T16:41:47.843 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2023-11-16 16:30:33.914935981 +0000 2023-11-16T16:41:47.843 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2023-11-16 16:30:33.914935981 +0000 2023-11-16T16:41:47.844 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2023-11-16T16:41:47.844 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-16T16:41:47.908 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2023-11-16T16:41:47.908 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2023-11-16T16:41:47.908 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000201681 s, 2.5 MB/s 2023-11-16T16:41:47.910 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-16T16:41:47.968 INFO:tasks.cephadm:Deploying osd.0 on smithi044 with /dev/vg_nvme/lv_4... 2023-11-16T16:41:47.968 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-11-16T16:41:48.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:47 smithi044 ceph-mon[128750]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:48.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:48.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:48.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:48.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:48.038 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T16:41:48.038 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:48.038 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:48.136 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:47 smithi077 ceph-mon[131204]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:48.136 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:48.136 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:48.136 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:48.137 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:48.137 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T16:41:48.137 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:48.138 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:48.145 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:47.683+0000 7f66d8d6e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T16:41:48.145 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:47.865+0000 7f66d8d6e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T16:41:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:47 smithi176 ceph-mon[131630]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:48.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:48.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T16:41:48.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:48.148 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:48.490 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:48 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:48.181+0000 7f66d8d6e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T16:41:48.491 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:48 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:48.268+0000 7f66d8d6e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T16:41:48.491 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:48 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:48.396+0000 7f66d8d6e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T16:41:48.749 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:48 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:48.489+0000 7f66d8d6e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T16:41:48.832 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:41:48.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:48 smithi077 ceph-mon[131204]: Deploying daemon mgr.y on smithi077 2023-11-16T16:41:48.977 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:48 smithi077 systemd[1]: Starting Ceph mgr.y for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:41:49.091 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:48 smithi044 ceph-mon[128750]: Deploying daemon mgr.y on smithi077 2023-11-16T16:41:49.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:48 smithi176 ceph-mon[131630]: Deploying daemon mgr.y on smithi077 2023-11-16T16:41:49.145 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:48 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:48.908+0000 7f66d8d6e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T16:41:49.476 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:49 smithi077 podman[132649]: 2023-11-16T16:41:49.476 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:49 smithi077 bash[132649]: 4241444eddf8b019c281015d97bdfb39a61c2ee927da833608747172387c30b5 2023-11-16T16:41:49.477 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:49 smithi077 ceph-mgr[132689]: -- 172.21.15.77:0/4156958247 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ab63079520 con 0x55ab6304e800 2023-11-16T16:41:49.477 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:49 smithi077 systemd[1]: Started Ceph mgr.y for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:41:49.750 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:49 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:49.482+0000 7f66d8d6e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T16:41:49.750 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:49 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:49.654+0000 7f66d8d6e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T16:41:49.802 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:49 smithi077 ceph-mon[131204]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:49.802 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:49.803 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:49.803 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:49.803 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:49.803 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:49.804 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:49.535+0000 7fd3b9955200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T16:41:49.804 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:49.613+0000 7fd3b9955200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T16:41:49.804 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:49.704+0000 7fd3b9955200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T16:41:49.804 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:49.801+0000 7fd3b9955200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T16:41:50.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:49 smithi044 ceph-mon[128750]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:50.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:50.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:50.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:50.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:50.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:50.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:49 smithi176 ceph-mon[131630]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:50.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:50.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:50.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:50.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:50.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:50.226 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:49.975+0000 7fd3b9955200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T16:41:50.619 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:50.295+0000 7fd3b9955200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T16:41:50.620 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:50.384+0000 7fd3b9955200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T16:41:50.620 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:50.519+0000 7fd3b9955200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T16:41:50.740 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:50 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:50.471+0000 7f66d8d6e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:41:50.740 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:50 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:50.561+0000 7f66d8d6e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T16:41:50.886 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-11-16T16:41:50.886 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-11-16T16:41:50.886 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2023-11-16T16:41:50.886 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2023-11-16T16:41:50.886 INFO:teuthology.orchestra.run.smithi044.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0320051 s, 328 MB/s 2023-11-16T16:41:50.886 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2023-11-16T16:41:50.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:50 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:41:50.977 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:50.618+0000 7fd3b9955200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T16:41:51.037 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:50 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:41:51.038 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:50 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:50.739+0000 7f66d8d6e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:41:51.044 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:50 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:41:51.328 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi044:vg_nvme/lv_4 2023-11-16T16:41:51.395 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:51 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:51.036+0000 7f66d8d6e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T16:41:51.395 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:51 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:51.271+0000 7f66d8d6e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T16:41:51.476 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:51 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:51.060+0000 7fd3b9955200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T16:41:51.683 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:51 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:51.422+0000 7f66d8d6e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T16:41:51.977 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:51 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:51.648+0000 7fd3b9955200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T16:41:51.977 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:51 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:51.818+0000 7fd3b9955200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T16:41:51.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:51 smithi077 ceph-mon[131204]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:51.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:51 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:51.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:51 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:51.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:51 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:51.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:51 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:51.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:51 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:51.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:51 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:51.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:51 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:51.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:51 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:51.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:51 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:51.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:51 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:52.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:51 smithi044 ceph-mon[128750]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:52.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:51 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:52.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:51 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:52.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:51 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:52.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:51 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:52.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:51 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:52.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:51 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:52.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:51 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:52.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:51 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:52.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:51 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:52.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:51 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:52.145 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:51 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:51.682+0000 7f66d8d6e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T16:41:52.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:51 smithi176 ceph-mon[131630]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:52.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:51 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:52.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:51 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:52.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:51 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:52.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:51 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:52.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:51 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:52.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:51 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:52.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:51 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:52.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:51 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:52.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:51 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:52.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:51 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:52.215 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:41:52.645 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:52 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:52.379+0000 7f66d8d6e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T16:41:52.645 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:52 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:52.507+0000 7f66d8d6e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T16:41:52.829 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:52 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:52.567+0000 7fd3b9955200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:41:52.829 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:52 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:52.659+0000 7fd3b9955200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T16:41:52.829 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:52 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:41:52.829 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:52 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:41:52.829 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:52 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:41:52.830 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:52 smithi077 ceph-mon[131204]: Reconfiguring mgr.x (unknown last config time)... 2023-11-16T16:41:52.830 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:52 smithi077 ceph-mon[131204]: Reconfiguring daemon mgr.x on smithi044 2023-11-16T16:41:52.830 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:52 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:52.961 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:52 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:41:52.961 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:52 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:41:52.961 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:52 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:41:52.961 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:52 smithi044 ceph-mon[128750]: Reconfiguring mgr.x (unknown last config time)... 2023-11-16T16:41:52.961 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:52 smithi044 ceph-mon[128750]: Reconfiguring daemon mgr.x on smithi044 2023-11-16T16:41:52.961 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:52 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:53.020 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:52 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:52.695+0000 7f66d8d6e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T16:41:53.020 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:52 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:52.808+0000 7f66d8d6e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T16:41:53.020 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:52 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:41:53.020 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:52 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:41:53.020 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:52 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:41:53.020 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:52 smithi176 ceph-mon[131630]: Reconfiguring mgr.x (unknown last config time)... 2023-11-16T16:41:53.020 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:52 smithi176 ceph-mon[131630]: Reconfiguring daemon mgr.x on smithi044 2023-11-16T16:41:53.020 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:52 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:53.102 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:52 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:52.827+0000 7fd3b9955200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:41:53.470 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:53 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:53.100+0000 7fd3b9955200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T16:41:53.470 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:53 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:53.323+0000 7fd3b9955200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T16:41:53.726 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:53 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:53.468+0000 7fd3b9955200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T16:41:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:53 smithi044 ceph-mon[128750]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:54.145 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:53 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:53.018+0000 7f66d8d6e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T16:41:54.145 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:41:53 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:41:53.901+0000 7f66d8d6e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T16:41:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:53 smithi176 ceph-mon[131630]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:54.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.168+0000 7f3d26639700 1 -- 172.21.15.44:0/3989464262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d201014f0 msgr2=0x7f3d20101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:54.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.168+0000 7f3d26639700 1 --2- 172.21.15.44:0/3989464262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d201014f0 0x7f3d20101940 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3d14009990 tx=0x7f3d14009ca0 comp rx=0 tx=0).stop 2023-11-16T16:41:54.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.168+0000 7f3d26639700 1 -- 172.21.15.44:0/3989464262 shutdown_connections 2023-11-16T16:41:54.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.168+0000 7f3d26639700 1 --2- 172.21.15.44:0/3989464262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d20101e80 0x7f3d20108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:54.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.168+0000 7f3d26639700 1 --2- 172.21.15.44:0/3989464262 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d201002f0 0x7f3d20100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:54.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.168+0000 7f3d26639700 1 --2- 172.21.15.44:0/3989464262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d201014f0 0x7f3d20101940 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:54.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.168+0000 7f3d26639700 1 -- 172.21.15.44:0/3989464262 >> 172.21.15.44:0/3989464262 conn(0x7f3d200fb900 msgr2=0x7f3d200fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:54.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.168+0000 7f3d26639700 1 -- 172.21.15.44:0/3989464262 shutdown_connections 2023-11-16T16:41:54.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.168+0000 7f3d26639700 1 -- 172.21.15.44:0/3989464262 wait complete. 2023-11-16T16:41:54.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.169+0000 7f3d26639700 1 Processor -- start 2023-11-16T16:41:54.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.169+0000 7f3d26639700 1 -- start start 2023-11-16T16:41:54.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.169+0000 7f3d26639700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d201002f0 0x7f3d20196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:54.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.169+0000 7f3d26639700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d201014f0 0x7f3d201965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:54.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.169+0000 7f3d26639700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d20101e80 0x7f3d2019d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:54.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.170+0000 7f3d26639700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2019dc20 con 0x7f3d20101e80 2023-11-16T16:41:54.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.170+0000 7f3d26639700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2019dd60 con 0x7f3d201002f0 2023-11-16T16:41:54.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.170+0000 7f3d26639700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2019def0 con 0x7f3d201014f0 2023-11-16T16:41:54.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.170+0000 7f3d1ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d201002f0 0x7f3d20196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:54.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.170+0000 7f3d1ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d201002f0 0x7f3d20196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:48308/0 (socket says 172.21.15.44:48308) 2023-11-16T16:41:54.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.170+0000 7f3d1ffff700 1 -- 172.21.15.44:0/3784772234 learned_addr learned my addr 172.21.15.44:0/3784772234 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:41:54.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.170+0000 7f3d1f7fe700 1 --2- 172.21.15.44:0/3784772234 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d201014f0 0x7f3d201965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:54.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.170+0000 7f3d24bd6700 1 --2- 172.21.15.44:0/3784772234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d20101e80 0x7f3d2019d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:54.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.170+0000 7f3d1ffff700 1 -- 172.21.15.44:0/3784772234 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d201014f0 msgr2=0x7f3d201965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:54.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.170+0000 7f3d1ffff700 1 --2- 172.21.15.44:0/3784772234 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d201014f0 0x7f3d201965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:54.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.170+0000 7f3d1ffff700 1 -- 172.21.15.44:0/3784772234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d20101e80 msgr2=0x7f3d2019d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:54.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.170+0000 7f3d1ffff700 1 --2- 172.21.15.44:0/3784772234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d20101e80 0x7f3d2019d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:54.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.170+0000 7f3d1ffff700 1 -- 172.21.15.44:0/3784772234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d140095f0 con 0x7f3d201002f0 2023-11-16T16:41:54.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.170+0000 7f3d1ffff700 1 --2- 172.21.15.44:0/3784772234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d201002f0 0x7f3d20196090 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3d1000e950 tx=0x7f3d1000ec60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:54.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.171+0000 7f3d1d7fa700 1 -- 172.21.15.44:0/3784772234 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d10009770 con 0x7f3d201002f0 2023-11-16T16:41:54.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.171+0000 7f3d1d7fa700 1 -- 172.21.15.44:0/3784772234 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d1000c7e0 con 0x7f3d201002f0 2023-11-16T16:41:54.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.171+0000 7f3d26639700 1 -- 172.21.15.44:0/3784772234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d2010bcb0 con 0x7f3d201002f0 2023-11-16T16:41:54.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.171+0000 7f3d1d7fa700 1 -- 172.21.15.44:0/3784772234 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d10003c80 con 0x7f3d201002f0 2023-11-16T16:41:54.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.171+0000 7f3d26639700 1 -- 172.21.15.44:0/3784772234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2010bb20 con 0x7f3d201002f0 2023-11-16T16:41:54.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.172+0000 7f3d1d7fa700 1 -- 172.21.15.44:0/3784772234 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f3d10021440 con 0x7f3d201002f0 2023-11-16T16:41:54.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.172+0000 7f3d26639700 1 -- 172.21.15.44:0/3784772234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d20107230 con 0x7f3d201002f0 2023-11-16T16:41:54.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.172+0000 7f3d1d7fa700 1 --2- 172.21.15.44:0/3784772234 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3d080389e0 0x7f3d0803ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:54.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.172+0000 7f3d1d7fa700 1 -- 172.21.15.44:0/3784772234 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3d10014070 con 0x7f3d201002f0 2023-11-16T16:41:54.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.173+0000 7f3d1f7fe700 1 --2- 172.21.15.44:0/3784772234 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3d080389e0 0x7f3d0803ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:54.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.173+0000 7f3d1f7fe700 1 --2- 172.21.15.44:0/3784772234 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3d080389e0 0x7f3d0803ae90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3d1400b2e0 tx=0x7f3d14005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:54.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.178+0000 7f3d1d7fa700 1 -- 172.21.15.44:0/3784772234 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3d10016960 con 0x7f3d201002f0 2023-11-16T16:41:54.226 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:53 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:53.731+0000 7fd3b9955200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T16:41:54.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:53 smithi077 ceph-mon[131204]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:54.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:54.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:54.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:54.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:54.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:54.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:41:54.394 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.392+0000 7f3d26639700 1 -- 172.21.15.44:0/3784772234 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d20061ae0 con 0x7f3d080389e0 2023-11-16T16:41:54.726 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:54 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:54.422+0000 7fd3b9955200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T16:41:54.726 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:54 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:54.549+0000 7fd3b9955200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T16:41:54.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:54.776+0000 7f3d1d7fa700 1 -- 172.21.15.44:0/3784772234 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 14) v1 ==== 90599+0+0 (secure 0 0 0) 0x7f3d1000c950 con 0x7f3d201002f0 2023-11-16T16:41:55.066 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:54 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:54.737+0000 7fd3b9955200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T16:41:55.066 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:54 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:54.851+0000 7fd3b9955200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T16:41:55.066 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:54 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:41:55.067 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:54 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:41:55.067 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:54 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:41:55.067 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:54 smithi077 ceph-mon[131204]: Standby manager daemon z started 2023-11-16T16:41:55.067 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:54 smithi077 ceph-mon[131204]: from='mgr.? 172.21.15.176:0/2808004416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T16:41:55.067 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:54 smithi077 ceph-mon[131204]: from='mgr.? 172.21.15.176:0/2808004416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:41:55.067 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:54 smithi077 ceph-mon[131204]: from='mgr.? 172.21.15.176:0/2808004416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T16:41:55.067 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:54 smithi077 ceph-mon[131204]: from='mgr.? 172.21.15.176:0/2808004416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:41:55.067 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:54 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:41:55.067 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:54 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:41:55.067 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:54 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:55.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:54 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:41:55.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:54 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:41:55.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:54 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:41:55.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:54 smithi044 ceph-mon[128750]: Standby manager daemon z started 2023-11-16T16:41:55.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:54 smithi044 ceph-mon[128750]: from='mgr.? 172.21.15.176:0/2808004416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T16:41:55.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:54 smithi044 ceph-mon[128750]: from='mgr.? 172.21.15.176:0/2808004416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:41:55.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:54 smithi044 ceph-mon[128750]: from='mgr.? 172.21.15.176:0/2808004416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T16:41:55.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:54 smithi044 ceph-mon[128750]: from='mgr.? 172.21.15.176:0/2808004416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:41:55.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:54 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:41:55.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:54 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:41:55.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:54 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:55.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:54 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:41:55.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:54 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:41:55.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:54 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:41:55.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:54 smithi176 ceph-mon[131630]: Standby manager daemon z started 2023-11-16T16:41:55.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:54 smithi176 ceph-mon[131630]: from='mgr.? 172.21.15.176:0/2808004416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T16:41:55.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:54 smithi176 ceph-mon[131630]: from='mgr.? 172.21.15.176:0/2808004416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:41:55.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:54 smithi176 ceph-mon[131630]: from='mgr.? 172.21.15.176:0/2808004416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T16:41:55.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:54 smithi176 ceph-mon[131630]: from='mgr.? 172.21.15.176:0/2808004416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:41:55.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:54 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:41:55.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:54 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:41:55.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:54 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:55.476 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:55 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:55.064+0000 7fd3b9955200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T16:41:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:55 smithi044 ceph-mon[128750]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:55 smithi044 ceph-mon[128750]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:55 smithi044 ceph-mon[128750]: mgrmap e14: x(active, since 72s), standbys: z 2023-11-16T16:41:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T16:41:56.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:55 smithi176 ceph-mon[131630]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:56.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:55 smithi176 ceph-mon[131630]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:56.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:55 smithi176 ceph-mon[131630]: mgrmap e14: x(active, since 72s), standbys: z 2023-11-16T16:41:56.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:55 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T16:41:56.226 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:41:55 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:41:55.902+0000 7fd3b9955200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T16:41:56.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:55 smithi077 ceph-mon[131204]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:55 smithi077 ceph-mon[131204]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:55 smithi077 ceph-mon[131204]: mgrmap e14: x(active, since 72s), standbys: z 2023-11-16T16:41:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:55 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T16:41:56.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:41:56.789+0000 7f3d1d7fa700 1 -- 172.21.15.44:0/3784772234 <== mon.2 v2:172.21.15.77:3300/0 8 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f3d10021b10 con 0x7f3d201002f0 2023-11-16T16:41:57.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:56 smithi044 ceph-mon[128750]: Standby manager daemon y started 2023-11-16T16:41:57.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:56 smithi044 ceph-mon[128750]: from='mgr.? 172.21.15.77:0/1140083911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T16:41:57.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:56 smithi044 ceph-mon[128750]: from='mgr.? 172.21.15.77:0/1140083911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:41:57.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:56 smithi044 ceph-mon[128750]: from='mgr.? 172.21.15.77:0/1140083911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T16:41:57.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:56 smithi044 ceph-mon[128750]: from='mgr.? 172.21.15.77:0/1140083911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:41:57.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:56 smithi176 ceph-mon[131630]: Standby manager daemon y started 2023-11-16T16:41:57.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:56 smithi176 ceph-mon[131630]: from='mgr.? 172.21.15.77:0/1140083911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T16:41:57.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:56 smithi176 ceph-mon[131630]: from='mgr.? 172.21.15.77:0/1140083911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:41:57.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:56 smithi176 ceph-mon[131630]: from='mgr.? 172.21.15.77:0/1140083911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T16:41:57.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:56 smithi176 ceph-mon[131630]: from='mgr.? 172.21.15.77:0/1140083911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:41:57.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:56 smithi077 ceph-mon[131204]: Standby manager daemon y started 2023-11-16T16:41:57.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:56 smithi077 ceph-mon[131204]: from='mgr.? 172.21.15.77:0/1140083911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T16:41:57.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:56 smithi077 ceph-mon[131204]: from='mgr.? 172.21.15.77:0/1140083911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:41:57.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:56 smithi077 ceph-mon[131204]: from='mgr.? 172.21.15.77:0/1140083911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T16:41:57.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:56 smithi077 ceph-mon[131204]: from='mgr.? 172.21.15.77:0/1140083911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:41:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:57 smithi044 ceph-mon[128750]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:57 smithi044 ceph-mon[128750]: mgrmap e15: x(active, since 74s), standbys: z, y 2023-11-16T16:41:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:57 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T16:41:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:57 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3746389376' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a062dd5b-f36d-4686-86b7-b93b81bf317d"}]: dispatch 2023-11-16T16:41:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:57 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3746389376' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a062dd5b-f36d-4686-86b7-b93b81bf317d"}]': finished 2023-11-16T16:41:58.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:57 smithi044 ceph-mon[128750]: osdmap e5: 1 total, 0 up, 1 in 2023-11-16T16:41:58.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:57 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:41:58.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:57 smithi176 ceph-mon[131630]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:58.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:57 smithi176 ceph-mon[131630]: mgrmap e15: x(active, since 74s), standbys: z, y 2023-11-16T16:41:58.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:57 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T16:41:58.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:57 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3746389376' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a062dd5b-f36d-4686-86b7-b93b81bf317d"}]: dispatch 2023-11-16T16:41:58.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:57 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3746389376' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a062dd5b-f36d-4686-86b7-b93b81bf317d"}]': finished 2023-11-16T16:41:58.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:57 smithi176 ceph-mon[131630]: osdmap e5: 1 total, 0 up, 1 in 2023-11-16T16:41:58.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:57 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:41:58.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:57 smithi077 ceph-mon[131204]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:57 smithi077 ceph-mon[131204]: mgrmap e15: x(active, since 74s), standbys: z, y 2023-11-16T16:41:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:57 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T16:41:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:57 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3746389376' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a062dd5b-f36d-4686-86b7-b93b81bf317d"}]: dispatch 2023-11-16T16:41:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:57 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3746389376' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a062dd5b-f36d-4686-86b7-b93b81bf317d"}]': finished 2023-11-16T16:41:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:57 smithi077 ceph-mon[131204]: osdmap e5: 1 total, 0 up, 1 in 2023-11-16T16:41:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:57 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:41:59.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:58 smithi044 ceph-mon[128750]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:59.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:41:58 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1255103007' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:41:59.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:58 smithi176 ceph-mon[131630]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:59.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:41:58 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1255103007' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:41:59.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:58 smithi077 ceph-mon[131204]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:59.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:41:58 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1255103007' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:42:01.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:01 smithi044 ceph-mon[128750]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:01.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:01 smithi176 ceph-mon[131630]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:01.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:01 smithi077 ceph-mon[131204]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:03.743 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:03 smithi044 ceph-mon[128750]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:03.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:03 smithi176 ceph-mon[131630]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:03.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:03 smithi077 ceph-mon[131204]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:05.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:05 smithi044 ceph-mon[128750]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:05.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:05 smithi176 ceph-mon[131630]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:05.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:05 smithi077 ceph-mon[131204]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:06.906 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:06 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-16T16:42:06.907 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:06 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:06.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:06 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-16T16:42:06.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:06 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:07.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:06 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-16T16:42:07.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:06 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:07 smithi044 ceph-mon[128750]: Deploying daemon osd.0 on smithi044 2023-11-16T16:42:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:07 smithi044 ceph-mon[128750]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:07.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:07 smithi077 ceph-mon[131204]: Deploying daemon osd.0 on smithi044 2023-11-16T16:42:07.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:07 smithi077 ceph-mon[131204]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:08.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:07 smithi176 ceph-mon[131630]: Deploying daemon osd.0 on smithi044 2023-11-16T16:42:08.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:07 smithi176 ceph-mon[131630]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:09.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:09 smithi044 ceph-mon[128750]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:09.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:09 smithi077 ceph-mon[131204]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:10.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:09 smithi176 ceph-mon[131630]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:11.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:11 smithi077 ceph-mon[131204]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:12.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:11 smithi044 ceph-mon[128750]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:12.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:11 smithi176 ceph-mon[131630]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:13.675 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:13 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:13.676 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:13 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:13.676 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:13 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:13.676 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:13 smithi044 ceph-mon[128750]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:13.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:13 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:13.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:13 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:13.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:13 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:13.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:13 smithi077 ceph-mon[131204]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:13.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:13 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:13.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:13 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:13.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:13 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:13.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:13 smithi176 ceph-mon[131630]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:15.777 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:15 smithi044 ceph-mon[128750]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:15.777 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:15 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:15.778 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:15 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:15.778 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:15 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:15.778 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:15 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:15.778 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:15 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:15 smithi176 ceph-mon[131630]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:15 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:15 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:15 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:15 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:15 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:15.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:15 smithi077 ceph-mon[131204]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:15 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:15 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:15 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:15 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:15 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:16.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:42:16.151+0000 7f3d1d7fa700 1 -- 172.21.15.44:0/3784772234 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3d20061ae0 con 0x7f3d080389e0 2023-11-16T16:42:16.154 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 0 on host 'smithi044' 2023-11-16T16:42:16.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:42:16.153+0000 7f3d26639700 1 -- 172.21.15.44:0/3784772234 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3d080389e0 msgr2=0x7f3d0803ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:16.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:42:16.153+0000 7f3d26639700 1 --2- 172.21.15.44:0/3784772234 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3d080389e0 0x7f3d0803ae90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3d1400b2e0 tx=0x7f3d14005fb0 comp rx=0 tx=0).stop 2023-11-16T16:42:16.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:42:16.153+0000 7f3d26639700 1 -- 172.21.15.44:0/3784772234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d201002f0 msgr2=0x7f3d20196090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:16.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:42:16.153+0000 7f3d26639700 1 --2- 172.21.15.44:0/3784772234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d201002f0 0x7f3d20196090 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3d1000e950 tx=0x7f3d1000ec60 comp rx=0 tx=0).stop 2023-11-16T16:42:16.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:42:16.153+0000 7f3d26639700 1 -- 172.21.15.44:0/3784772234 shutdown_connections 2023-11-16T16:42:16.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:42:16.153+0000 7f3d26639700 1 --2- 172.21.15.44:0/3784772234 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3d080389e0 0x7f3d0803ae90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:16.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:42:16.153+0000 7f3d26639700 1 --2- 172.21.15.44:0/3784772234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d20101e80 0x7f3d2019d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:16.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:42:16.153+0000 7f3d26639700 1 --2- 172.21.15.44:0/3784772234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d201002f0 0x7f3d20196090 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:16.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:42:16.153+0000 7f3d26639700 1 --2- 172.21.15.44:0/3784772234 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d201014f0 0x7f3d201965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:16.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:42:16.153+0000 7f3d26639700 1 -- 172.21.15.44:0/3784772234 >> 172.21.15.44:0/3784772234 conn(0x7f3d200fb900 msgr2=0x7f3d200fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:42:16.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:42:16.154+0000 7f3d26639700 1 -- 172.21.15.44:0/3784772234 shutdown_connections 2023-11-16T16:42:16.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:42:16.154+0000 7f3d26639700 1 -- 172.21.15.44:0/3784772234 wait complete. 2023-11-16T16:42:16.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:16 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:42:16.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:16 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:42:16.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:16 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:42:16.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:16 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:16.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:16 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:16.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:16 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:16.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:16 smithi044 ceph-mon[128750]: from='osd.0 [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-16T16:42:16.669 DEBUG:teuthology.orchestra.run.smithi044:osd.0> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.0.service 2023-11-16T16:42:16.676 INFO:tasks.cephadm:Deploying osd.1 on smithi077 with /dev/vg_nvme/lv_4... 2023-11-16T16:42:16.677 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-11-16T16:42:16.751 INFO:journalctl@ceph.osd.0.smithi044.stdout:-- Logs begin at Thu 2023-11-16 16:19:45 UTC. -- 2023-11-16T16:42:16.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:16 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:42:16.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:16 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:42:16.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:16 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:42:16.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:16 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:16.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:16 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:16.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:16 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:16.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:16 smithi176 ceph-mon[131630]: from='osd.0 [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-16T16:42:16.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:16 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:42:16.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:16 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:42:16.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:16 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:42:16.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:16 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:16.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:16 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:16.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:16 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:16.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:16 smithi077 ceph-mon[131204]: from='osd.0 [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-16T16:42:17.322 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T16:42:17.659 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:17 smithi044 ceph-mon[128750]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:17.659 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:17 smithi044 ceph-mon[128750]: from='osd.0 [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-16T16:42:17.659 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:17 smithi044 ceph-mon[128750]: osdmap e6: 1 total, 0 up, 1 in 2023-11-16T16:42:17.660 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:17.660 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:17 smithi044 ceph-mon[128750]: from='osd.0 [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2023-11-16T16:42:17.850 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:17 smithi077 ceph-mon[131204]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:17.851 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:17 smithi077 ceph-mon[131204]: from='osd.0 [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-16T16:42:17.851 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:17 smithi077 ceph-mon[131204]: osdmap e6: 1 total, 0 up, 1 in 2023-11-16T16:42:17.851 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:17 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:17.851 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:17 smithi077 ceph-mon[131204]: from='osd.0 [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2023-11-16T16:42:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:17 smithi176 ceph-mon[131630]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:17 smithi176 ceph-mon[131630]: from='osd.0 [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-16T16:42:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:17 smithi176 ceph-mon[131630]: osdmap e6: 1 total, 0 up, 1 in 2023-11-16T16:42:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:17 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:17 smithi176 ceph-mon[131630]: from='osd.0 [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2023-11-16T16:42:18.619 INFO:journalctl@ceph.osd.0.smithi044.stdout:Nov 16 16:42:18 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-0[140968]: 2023-11-16T16:42:18.123+0000 7f9b59ae1700 -1 osd.0 0 waiting for initial osdmap 2023-11-16T16:42:18.619 INFO:journalctl@ceph.osd.0.smithi044.stdout:Nov 16 16:42:18 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-0[140968]: 2023-11-16T16:42:18.127+0000 7f9b538d2700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:42:18.799 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-11-16T16:42:18.799 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-11-16T16:42:18.799 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2023-11-16T16:42:18.799 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2023-11-16T16:42:18.799 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110025 s, 953 MB/s 2023-11-16T16:42:18.800 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2023-11-16T16:42:19.174 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi077:vg_nvme/lv_4 2023-11-16T16:42:19.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:19 smithi044 ceph-mon[128750]: from='osd.0 [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2023-11-16T16:42:19.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:19 smithi044 ceph-mon[128750]: osdmap e7: 1 total, 0 up, 1 in 2023-11-16T16:42:19.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:19.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:19.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:19 smithi044 ceph-mon[128750]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:19.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:19 smithi176 ceph-mon[131630]: from='osd.0 [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2023-11-16T16:42:19.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:19 smithi176 ceph-mon[131630]: osdmap e7: 1 total, 0 up, 1 in 2023-11-16T16:42:19.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:19.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:19.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:19 smithi176 ceph-mon[131630]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:19.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:19 smithi077 ceph-mon[131204]: from='osd.0 [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2023-11-16T16:42:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:19 smithi077 ceph-mon[131204]: osdmap e7: 1 total, 0 up, 1 in 2023-11-16T16:42:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:19 smithi077 ceph-mon[131204]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:19.810 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T16:42:20.284 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: purged_snaps scrub starts 2023-11-16T16:42:20.284 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: purged_snaps scrub ok 2023-11-16T16:42:20.284 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: osd.0 [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157] boot 2023-11-16T16:42:20.284 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: osdmap e8: 1 total, 1 up, 1 in 2023-11-16T16:42:20.285 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:20.285 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: Detected new or changed devices on smithi044 2023-11-16T16:42:20.285 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:20.285 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:20.285 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:42:20.285 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi044 to 17276M 2023-11-16T16:42:20.285 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:20.285 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:20.285 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:20.285 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:42:20.285 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:20.285 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:42:20.285 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:20 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:42:20.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: purged_snaps scrub starts 2023-11-16T16:42:20.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: purged_snaps scrub ok 2023-11-16T16:42:20.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: osd.0 [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157] boot 2023-11-16T16:42:20.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: osdmap e8: 1 total, 1 up, 1 in 2023-11-16T16:42:20.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:20.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: Detected new or changed devices on smithi044 2023-11-16T16:42:20.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:20.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:20.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:42:20.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi044 to 17276M 2023-11-16T16:42:20.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:20.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:20.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:20.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:42:20.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:20.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:42:20.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:20 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:42:20.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: purged_snaps scrub starts 2023-11-16T16:42:20.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: purged_snaps scrub ok 2023-11-16T16:42:20.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: osd.0 [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157] boot 2023-11-16T16:42:20.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: osdmap e8: 1 total, 1 up, 1 in 2023-11-16T16:42:20.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:20.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: Detected new or changed devices on smithi044 2023-11-16T16:42:20.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:20.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:20.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:42:20.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi044 to 17276M 2023-11-16T16:42:20.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:20.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:20.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:20.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:42:20.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:20.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:42:20.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:20 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:42:20.996 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.994+0000 7ff9e58a7700 1 -- 172.21.15.77:0/1654509415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e00ff3f0 msgr2=0x7ff9e00ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:20.996 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.994+0000 7ff9e58a7700 1 --2- 172.21.15.77:0/1654509415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e00ff3f0 0x7ff9e00ff800 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009990 tx=0x7ff9d4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:42:20.996 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.994+0000 7ff9e58a7700 1 -- 172.21.15.77:0/1654509415 shutdown_connections 2023-11-16T16:42:20.996 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.994+0000 7ff9e58a7700 1 --2- 172.21.15.77:0/1654509415 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9e0100f80 0x7ff9e0107840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:20.996 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.994+0000 7ff9e58a7700 1 --2- 172.21.15.77:0/1654509415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e00ff3f0 0x7ff9e00ff800 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:20.996 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.994+0000 7ff9e58a7700 1 --2- 172.21.15.77:0/1654509415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9e01005f0 0x7ff9e0100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:20.996 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.994+0000 7ff9e58a7700 1 -- 172.21.15.77:0/1654509415 >> 172.21.15.77:0/1654509415 conn(0x7ff9e00faa20 msgr2=0x7ff9e00fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:42:20.997 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.994+0000 7ff9e58a7700 1 -- 172.21.15.77:0/1654509415 shutdown_connections 2023-11-16T16:42:20.997 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.994+0000 7ff9e58a7700 1 -- 172.21.15.77:0/1654509415 wait complete. 2023-11-16T16:42:20.997 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.995+0000 7ff9e58a7700 1 Processor -- start 2023-11-16T16:42:20.998 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.995+0000 7ff9e58a7700 1 -- start start 2023-11-16T16:42:20.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9e58a7700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9e00ff3f0 0x7ff9e0069ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:42:20.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9e58a7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e01005f0 0x7ff9e019afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:42:20.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9e58a7700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9e0100f80 0x7ff9e019d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:42:20.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9e58a7700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e019ddd0 con 0x7ff9e0100f80 2023-11-16T16:42:20.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9e58a7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e019df40 con 0x7ff9e01005f0 2023-11-16T16:42:20.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9e58a7700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e019e100 con 0x7ff9e00ff3f0 2023-11-16T16:42:20.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9de7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e01005f0 0x7ff9e019afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:42:20.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9de7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e01005f0 0x7ff9e019afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50790/0 (socket says 172.21.15.77:50790) 2023-11-16T16:42:20.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9de7fc700 1 -- 172.21.15.77:0/364344782 learned_addr learned my addr 172.21.15.77:0/364344782 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-11-16T16:42:20.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9deffd700 1 --2- 172.21.15.77:0/364344782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9e00ff3f0 0x7ff9e0069ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:42:21.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9df7fe700 1 --2- 172.21.15.77:0/364344782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9e0100f80 0x7ff9e019d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:42:21.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9de7fc700 1 -- 172.21.15.77:0/364344782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9e00ff3f0 msgr2=0x7ff9e0069ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:21.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9de7fc700 1 --2- 172.21.15.77:0/364344782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9e00ff3f0 0x7ff9e0069ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:21.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9de7fc700 1 -- 172.21.15.77:0/364344782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9e0100f80 msgr2=0x7ff9e019d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:21.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9de7fc700 1 --2- 172.21.15.77:0/364344782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9e0100f80 0x7ff9e019d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:21.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9de7fc700 1 -- 172.21.15.77:0/364344782 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d40095f0 con 0x7ff9e01005f0 2023-11-16T16:42:21.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9de7fc700 1 --2- 172.21.15.77:0/364344782 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e01005f0 0x7ff9e019afe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000b530 tx=0x7ff9d000b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:42:21.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9e48a5700 1 -- 172.21.15.77:0/364344782 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0011830 con 0x7ff9e01005f0 2023-11-16T16:42:21.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9e58a7700 1 -- 172.21.15.77:0/364344782 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e010ae10 con 0x7ff9e01005f0 2023-11-16T16:42:21.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9e48a5700 1 -- 172.21.15.77:0/364344782 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9d0011e50 con 0x7ff9e01005f0 2023-11-16T16:42:21.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9e48a5700 1 -- 172.21.15.77:0/364344782 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d000f2e0 con 0x7ff9e01005f0 2023-11-16T16:42:21.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.996+0000 7ff9e58a7700 1 -- 172.21.15.77:0/364344782 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e010ac80 con 0x7ff9e01005f0 2023-11-16T16:42:21.001 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.997+0000 7ff9e58a7700 1 -- 172.21.15.77:0/364344782 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e01069c0 con 0x7ff9e01005f0 2023-11-16T16:42:21.001 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.998+0000 7ff9e48a5700 1 -- 172.21.15.77:0/364344782 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7ff9d0010380 con 0x7ff9e01005f0 2023-11-16T16:42:21.001 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.999+0000 7ff9e48a5700 1 --2- 172.21.15.77:0/364344782 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff9c80a1140 0x7ff9c80a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:42:21.001 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.999+0000 7ff9e48a5700 1 -- 172.21.15.77:0/364344782 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7ff9d00caa90 con 0x7ff9e01005f0 2023-11-16T16:42:21.002 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:20.999+0000 7ff9deffd700 1 --2- 172.21.15.77:0/364344782 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff9c80a1140 0x7ff9c80a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:42:21.002 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:21.000+0000 7ff9deffd700 1 --2- 172.21.15.77:0/364344782 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff9c80a1140 0x7ff9c80a35f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400b5c0 tx=0x7ff9d4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:42:21.004 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:21.003+0000 7ff9e48a5700 1 -- 172.21.15.77:0/364344782 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff9d0095bb0 con 0x7ff9e01005f0 2023-11-16T16:42:21.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:21.198+0000 7ff9e58a7700 1 -- 172.21.15.77:0/364344782 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9e0060f30 con 0x7ff9c80a1140 2023-11-16T16:42:21.394 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:21 smithi176 ceph-mon[131630]: osdmap e9: 1 total, 1 up, 1 in 2023-11-16T16:42:21.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:21 smithi176 ceph-mon[131630]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:21.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:21 smithi077 ceph-mon[131204]: osdmap e9: 1 total, 1 up, 1 in 2023-11-16T16:42:21.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:21 smithi077 ceph-mon[131204]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:21.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:21 smithi044 ceph-mon[128750]: osdmap e9: 1 total, 1 up, 1 in 2023-11-16T16:42:21.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:21 smithi044 ceph-mon[128750]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:22.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:22 smithi176 ceph-mon[131630]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:42:22.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:22 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:42:22.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:22 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:42:22.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:22 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:22.425 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:22 smithi077 ceph-mon[131204]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:42:22.425 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:22 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:42:22.425 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:22 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:42:22.425 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:22 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:22.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:22 smithi044 ceph-mon[128750]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:42:22.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:22 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:42:22.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:22 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:42:22.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:22 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:23 smithi077 ceph-mon[131204]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:23 smithi077 ceph-mon[131204]: from='client.? 172.21.15.77:0/4025435822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48ba3e8d-abd4-4253-a4aa-bc3d123a8bec"}]: dispatch 2023-11-16T16:42:23.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:23 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48ba3e8d-abd4-4253-a4aa-bc3d123a8bec"}]: dispatch 2023-11-16T16:42:23.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:23 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48ba3e8d-abd4-4253-a4aa-bc3d123a8bec"}]': finished 2023-11-16T16:42:23.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:23 smithi077 ceph-mon[131204]: osdmap e10: 2 total, 1 up, 2 in 2023-11-16T16:42:23.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:23 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:24.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:23 smithi044 ceph-mon[128750]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:24.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:23 smithi044 ceph-mon[128750]: from='client.? 172.21.15.77:0/4025435822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48ba3e8d-abd4-4253-a4aa-bc3d123a8bec"}]: dispatch 2023-11-16T16:42:24.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:23 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48ba3e8d-abd4-4253-a4aa-bc3d123a8bec"}]: dispatch 2023-11-16T16:42:24.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:23 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48ba3e8d-abd4-4253-a4aa-bc3d123a8bec"}]': finished 2023-11-16T16:42:24.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:23 smithi044 ceph-mon[128750]: osdmap e10: 2 total, 1 up, 2 in 2023-11-16T16:42:24.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:23 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:24.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:23 smithi176 ceph-mon[131630]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:24.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:23 smithi176 ceph-mon[131630]: from='client.? 172.21.15.77:0/4025435822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48ba3e8d-abd4-4253-a4aa-bc3d123a8bec"}]: dispatch 2023-11-16T16:42:24.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:23 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48ba3e8d-abd4-4253-a4aa-bc3d123a8bec"}]: dispatch 2023-11-16T16:42:24.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:23 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48ba3e8d-abd4-4253-a4aa-bc3d123a8bec"}]': finished 2023-11-16T16:42:24.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:23 smithi176 ceph-mon[131630]: osdmap e10: 2 total, 1 up, 2 in 2023-11-16T16:42:24.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:23 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:24.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:24 smithi077 ceph-mon[131204]: from='client.? 172.21.15.77:0/3284823905' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:42:25.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:24 smithi044 ceph-mon[128750]: from='client.? 172.21.15.77:0/3284823905' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:42:25.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:24 smithi176 ceph-mon[131630]: from='client.? 172.21.15.77:0/3284823905' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:42:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:25 smithi077 ceph-mon[131204]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:26.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:25 smithi044 ceph-mon[128750]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:26.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:25 smithi176 ceph-mon[131630]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:27.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:27 smithi077 ceph-mon[131204]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:28.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:27 smithi044 ceph-mon[128750]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:28.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:27 smithi176 ceph-mon[131630]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:29.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:29 smithi077 ceph-mon[131204]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:30.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:29 smithi044 ceph-mon[128750]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:30.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:29 smithi176 ceph-mon[131630]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:30.907 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:30 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-16T16:42:30.907 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:30 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:31.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:30 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-16T16:42:31.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:30 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:31.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:30 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-16T16:42:31.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:30 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:31.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:31 smithi077 ceph-mon[131204]: Deploying daemon osd.1 on smithi077 2023-11-16T16:42:31.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:31 smithi077 ceph-mon[131204]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:32.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:31 smithi044 ceph-mon[128750]: Deploying daemon osd.1 on smithi077 2023-11-16T16:42:32.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:31 smithi044 ceph-mon[128750]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:32.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:31 smithi176 ceph-mon[131630]: Deploying daemon osd.1 on smithi077 2023-11-16T16:42:32.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:31 smithi176 ceph-mon[131630]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:33.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:33 smithi077 ceph-mon[131204]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:34.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:33 smithi044 ceph-mon[128750]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:34.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:33 smithi176 ceph-mon[131630]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:35.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:35 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:35.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:35 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:35.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:35 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:35.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:35 smithi077 ceph-mon[131204]: pgmap v56: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:35.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:35 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:35.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:35 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:35.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:35 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:35.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:35 smithi044 ceph-mon[128750]: pgmap v56: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:35.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:35 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:35.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:35 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:35.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:35 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:35.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:35 smithi176 ceph-mon[131630]: pgmap v56: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:37.113 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:37.110+0000 7ff9e48a5700 1 -- 172.21.15.77:0/364344782 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff9e0060f30 con 0x7ff9c80a1140 2023-11-16T16:42:37.113 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 1 on host 'smithi077' 2023-11-16T16:42:37.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:37.113+0000 7ff9e58a7700 1 -- 172.21.15.77:0/364344782 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff9c80a1140 msgr2=0x7ff9c80a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:37.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:37.113+0000 7ff9e58a7700 1 --2- 172.21.15.77:0/364344782 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff9c80a1140 0x7ff9c80a35f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400b5c0 tx=0x7ff9d4005fd0 comp rx=0 tx=0).stop 2023-11-16T16:42:37.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:37.113+0000 7ff9e58a7700 1 -- 172.21.15.77:0/364344782 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e01005f0 msgr2=0x7ff9e019afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:37.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:37.113+0000 7ff9e58a7700 1 --2- 172.21.15.77:0/364344782 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e01005f0 0x7ff9e019afe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000b530 tx=0x7ff9d000b840 comp rx=0 tx=0).stop 2023-11-16T16:42:37.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:37.113+0000 7ff9e58a7700 1 -- 172.21.15.77:0/364344782 shutdown_connections 2023-11-16T16:42:37.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:37.113+0000 7ff9e58a7700 1 --2- 172.21.15.77:0/364344782 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff9c80a1140 0x7ff9c80a35f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:37.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:37.113+0000 7ff9e58a7700 1 --2- 172.21.15.77:0/364344782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9e0100f80 0x7ff9e019d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:37.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:37.113+0000 7ff9e58a7700 1 --2- 172.21.15.77:0/364344782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9e00ff3f0 0x7ff9e0069ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:37.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:37.113+0000 7ff9e58a7700 1 --2- 172.21.15.77:0/364344782 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e01005f0 0x7ff9e019afe0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:37.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:37.113+0000 7ff9e58a7700 1 -- 172.21.15.77:0/364344782 >> 172.21.15.77:0/364344782 conn(0x7ff9e00faa20 msgr2=0x7ff9e0103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:42:37.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:37.113+0000 7ff9e58a7700 1 -- 172.21.15.77:0/364344782 shutdown_connections 2023-11-16T16:42:37.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:42:37.113+0000 7ff9e58a7700 1 -- 172.21.15.77:0/364344782 wait complete. 2023-11-16T16:42:37.493 DEBUG:teuthology.orchestra.run.smithi077:osd.1> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.1.service 2023-11-16T16:42:37.495 INFO:tasks.cephadm:Deploying osd.2 on smithi176 with /dev/vg_nvme/lv_4... 2023-11-16T16:42:37.496 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-11-16T16:42:37.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:-- Logs begin at Thu 2023-11-16 16:15:33 UTC. -- 2023-11-16T16:42:37.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:37.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:37.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:37.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:37.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:37 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:42:37.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:37.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:37 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:42:37.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:37 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:42:37.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:37 smithi044 ceph-mon[128750]: pgmap v57: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:37.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:37.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:37.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:37.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:37.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:37.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:37.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:37.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:37 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:42:37.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:37.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:37 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:42:37.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:37 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:42:37.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:37 smithi176 ceph-mon[131630]: pgmap v57: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:37.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:37.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:37.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:37.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:37 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:37.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:37 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:37.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:37 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:37.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:37 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:37 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:42:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:37 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:37 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:42:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:37 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:42:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:37 smithi077 ceph-mon[131204]: pgmap v57: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:37 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:37 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:37 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:38.142 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:42:38.450 INFO:journalctl@ceph.osd.1.smithi077.stdout:Nov 16 16:42:38 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-1[136286]: 2023-11-16T16:42:38.179+0000 7f140b4dc640 -1 osd.1 0 log_to_monitors true 2023-11-16T16:42:38.713 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:38 smithi077 ceph-mon[131204]: from='osd.1 [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T16:42:38.713 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:38 smithi077 ceph-mon[131204]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T16:42:38.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:38 smithi176 ceph-mon[131630]: from='osd.1 [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T16:42:38.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:38 smithi176 ceph-mon[131630]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T16:42:39.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:38 smithi044 ceph-mon[128750]: from='osd.1 [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T16:42:39.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:38 smithi044 ceph-mon[128750]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T16:42:39.606 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-11-16T16:42:39.606 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-11-16T16:42:39.606 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2023-11-16T16:42:39.606 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2023-11-16T16:42:39.607 INFO:teuthology.orchestra.run.smithi176.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0108731 s, 964 MB/s 2023-11-16T16:42:39.607 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 2023-11-16T16:42:39.607 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2023-11-16T16:42:39.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:39 smithi176 ceph-mon[131630]: pgmap v58: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:39.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:39 smithi176 ceph-mon[131630]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-16T16:42:39.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:39 smithi176 ceph-mon[131630]: osdmap e11: 2 total, 1 up, 2 in 2023-11-16T16:42:39.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:39 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:39.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:39 smithi176 ceph-mon[131630]: from='osd.1 [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:42:39.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:39 smithi176 ceph-mon[131630]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:42:39.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Nov 16 16:42:39 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-1[136286]: 2023-11-16T16:42:39.667+0000 7f1400b53700 -1 osd.1 0 waiting for initial osdmap 2023-11-16T16:42:39.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Nov 16 16:42:39 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-1[136286]: 2023-11-16T16:42:39.670+0000 7f13fb145700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:42:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:39 smithi077 ceph-mon[131204]: pgmap v58: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:39 smithi077 ceph-mon[131204]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-16T16:42:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:39 smithi077 ceph-mon[131204]: osdmap e11: 2 total, 1 up, 2 in 2023-11-16T16:42:39.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:39.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:39 smithi077 ceph-mon[131204]: from='osd.1 [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:42:39.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:39 smithi077 ceph-mon[131204]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:42:39.982 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi176:vg_nvme/lv_4 2023-11-16T16:42:40.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:39 smithi044 ceph-mon[128750]: pgmap v58: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:40.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:39 smithi044 ceph-mon[128750]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-16T16:42:40.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:39 smithi044 ceph-mon[128750]: osdmap e11: 2 total, 1 up, 2 in 2023-11-16T16:42:40.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:39 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:40.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:39 smithi044 ceph-mon[128750]: from='osd.1 [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:42:40.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:39 smithi044 ceph-mon[128750]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:42:40.624 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:42:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:40 smithi077 ceph-mon[131204]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2023-11-16T16:42:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:40 smithi077 ceph-mon[131204]: osdmap e12: 2 total, 1 up, 2 in 2023-11-16T16:42:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:40 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:40 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:40 smithi077 ceph-mon[131204]: Detected new or changed devices on smithi077 2023-11-16T16:42:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:40 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:40 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:40 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:42:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:40 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi077 to 17168M 2023-11-16T16:42:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:40 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:40 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:40 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:40 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:42:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:40 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:40 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:42:40.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:40 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:42:41.000 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:40 smithi176 ceph-mon[131630]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2023-11-16T16:42:41.001 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:40 smithi176 ceph-mon[131630]: osdmap e12: 2 total, 1 up, 2 in 2023-11-16T16:42:41.001 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:40 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:41.001 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:40 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:41.001 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:40 smithi176 ceph-mon[131630]: Detected new or changed devices on smithi077 2023-11-16T16:42:41.001 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:40 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:41.001 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:40 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:41.001 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:40 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:42:41.001 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:40 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi077 to 17168M 2023-11-16T16:42:41.001 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:40 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:41.001 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:40 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:41.001 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:40 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:41.001 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:40 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:42:41.002 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:40 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:41.002 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:40 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:42:41.002 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:40 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:42:41.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:40 smithi044 ceph-mon[128750]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2023-11-16T16:42:41.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:40 smithi044 ceph-mon[128750]: osdmap e12: 2 total, 1 up, 2 in 2023-11-16T16:42:41.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:40 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:41.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:40 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:41.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:40 smithi044 ceph-mon[128750]: Detected new or changed devices on smithi077 2023-11-16T16:42:41.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:40 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:41.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:40 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:41.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:40 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:42:41.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:40 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi077 to 17168M 2023-11-16T16:42:41.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:40 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:41.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:40 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:41.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:40 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:41.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:40 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:42:41.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:40 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:41.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:40 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:42:41.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:40 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:42:41.796 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.794+0000 7f6ac23ed700 1 -- 172.21.15.176:0/3802393077 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6abc0ffe40 msgr2=0x7f6abc107710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:41.796 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.794+0000 7f6ac23ed700 1 --2- 172.21.15.176:0/3802393077 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6abc0ffe40 0x7f6abc107710 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400b0d0 tx=0x7f6ab400b3e0 comp rx=0 tx=0).stop 2023-11-16T16:42:41.796 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.794+0000 7f6ac23ed700 1 -- 172.21.15.176:0/3802393077 shutdown_connections 2023-11-16T16:42:41.796 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.794+0000 7f6ac23ed700 1 --2- 172.21.15.176:0/3802393077 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6abc0ffe40 0x7f6abc107710 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:41.796 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.794+0000 7f6ac23ed700 1 --2- 172.21.15.176:0/3802393077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6abc100770 0x7f6abc100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:41.797 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.794+0000 7f6ac23ed700 1 --2- 172.21.15.176:0/3802393077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6abc0ff4b0 0x7f6abc0ff900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:41.797 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.794+0000 7f6ac23ed700 1 -- 172.21.15.176:0/3802393077 >> 172.21.15.176:0/3802393077 conn(0x7f6abc0f9f70 msgr2=0x7f6abc0fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:42:41.797 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.794+0000 7f6ac23ed700 1 -- 172.21.15.176:0/3802393077 shutdown_connections 2023-11-16T16:42:41.797 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.794+0000 7f6ac23ed700 1 -- 172.21.15.176:0/3802393077 wait complete. 2023-11-16T16:42:41.799 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6ac23ed700 1 Processor -- start 2023-11-16T16:42:41.799 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6ac23ed700 1 -- start start 2023-11-16T16:42:41.799 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6ac23ed700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6abc0ff4b0 0x7f6abc0fef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:42:41.799 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6ac23ed700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6abc0ffe40 0x7f6abc19d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:42:41.799 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6ac23ed700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6abc100770 0x7f6abc19f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:42:41.799 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6ac23ed700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc19fe70 con 0x7f6abc100770 2023-11-16T16:42:41.799 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6ac23ed700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc19ffe0 con 0x7f6abc0ff4b0 2023-11-16T16:42:41.799 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6ac23ed700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc1a01a0 con 0x7f6abc0ffe40 2023-11-16T16:42:41.799 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6abb7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6abc0ffe40 0x7f6abc19d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:42:41.800 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6abb7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6abc0ffe40 0x7f6abc19d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37790/0 (socket says 172.21.15.176:37790) 2023-11-16T16:42:41.800 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6abb7fe700 1 -- 172.21.15.176:0/2129197668 learned_addr learned my addr 172.21.15.176:0/2129197668 (peer_addr_for_me v2:172.21.15.176:0/0) 2023-11-16T16:42:41.800 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6ac098a700 1 --2- 172.21.15.176:0/2129197668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6abc100770 0x7f6abc19f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:42:41.800 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6abbfff700 1 --2- 172.21.15.176:0/2129197668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6abc0ff4b0 0x7f6abc0fef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:42:41.800 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6abb7fe700 1 -- 172.21.15.176:0/2129197668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6abc0ff4b0 msgr2=0x7f6abc0fef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:41.800 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6abb7fe700 1 --2- 172.21.15.176:0/2129197668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6abc0ff4b0 0x7f6abc0fef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:41.800 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6abb7fe700 1 -- 172.21.15.176:0/2129197668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6abc100770 msgr2=0x7f6abc19f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:41.800 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6abb7fe700 1 --2- 172.21.15.176:0/2129197668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6abc100770 0x7f6abc19f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:41.800 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6abb7fe700 1 -- 172.21.15.176:0/2129197668 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab4009d00 con 0x7f6abc0ffe40 2023-11-16T16:42:41.800 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6ac098a700 1 --2- 172.21.15.176:0/2129197668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6abc100770 0x7f6abc19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:42:41.800 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.795+0000 7f6abb7fe700 1 --2- 172.21.15.176:0/2129197668 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6abc0ffe40 0x7f6abc19d270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400c9b0 tx=0x7f6aa400cd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:42:41.800 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.796+0000 7f6ab97fa700 1 -- 172.21.15.176:0/2129197668 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4007b00 con 0x7f6abc0ffe40 2023-11-16T16:42:41.800 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.796+0000 7f6ac23ed700 1 -- 172.21.15.176:0/2129197668 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6abc10abe0 con 0x7f6abc0ffe40 2023-11-16T16:42:41.801 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.796+0000 7f6ab97fa700 1 -- 172.21.15.176:0/2129197668 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6aa4007c60 con 0x7f6abc0ffe40 2023-11-16T16:42:41.801 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.796+0000 7f6ab97fa700 1 -- 172.21.15.176:0/2129197668 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4018660 con 0x7f6abc0ffe40 2023-11-16T16:42:41.801 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.796+0000 7f6ac23ed700 1 -- 172.21.15.176:0/2129197668 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6abc10aa50 con 0x7f6abc0ffe40 2023-11-16T16:42:41.801 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.797+0000 7f6ac23ed700 1 -- 172.21.15.176:0/2129197668 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6abc106770 con 0x7f6abc0ffe40 2023-11-16T16:42:41.801 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.798+0000 7f6ab97fa700 1 -- 172.21.15.176:0/2129197668 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f6aa4018880 con 0x7f6abc0ffe40 2023-11-16T16:42:41.802 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.798+0000 7f6ab97fa700 1 --2- 172.21.15.176:0/2129197668 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6aa80a10c0 0x7f6aa80a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:42:41.802 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.798+0000 7f6ab97fa700 1 -- 172.21.15.176:0/2129197668 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f6aa40cb3d0 con 0x7f6abc0ffe40 2023-11-16T16:42:41.802 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.799+0000 7f6abbfff700 1 --2- 172.21.15.176:0/2129197668 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6aa80a10c0 0x7f6aa80a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:42:41.802 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.799+0000 7f6abbfff700 1 --2- 172.21.15.176:0/2129197668 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6aa80a10c0 0x7f6aa80a3570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6aac006fd0 tx=0x7f6aac008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:42:41.804 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.802+0000 7f6ab97fa700 1 -- 172.21.15.176:0/2129197668 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6aa4096400 con 0x7f6abc0ffe40 2023-11-16T16:42:41.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:41 smithi176 ceph-mon[131630]: purged_snaps scrub starts 2023-11-16T16:42:41.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:41 smithi176 ceph-mon[131630]: purged_snaps scrub ok 2023-11-16T16:42:41.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:41 smithi176 ceph-mon[131630]: pgmap v61: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:41.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:41 smithi176 ceph-mon[131630]: osd.1 [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975] boot 2023-11-16T16:42:41.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:41 smithi176 ceph-mon[131630]: osdmap e13: 2 total, 2 up, 2 in 2023-11-16T16:42:41.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:41 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:41 smithi077 ceph-mon[131204]: purged_snaps scrub starts 2023-11-16T16:42:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:41 smithi077 ceph-mon[131204]: purged_snaps scrub ok 2023-11-16T16:42:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:41 smithi077 ceph-mon[131204]: pgmap v61: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:41 smithi077 ceph-mon[131204]: osd.1 [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975] boot 2023-11-16T16:42:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:41 smithi077 ceph-mon[131204]: osdmap e13: 2 total, 2 up, 2 in 2023-11-16T16:42:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:41 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:41.998 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:41.996+0000 7f6ac23ed700 1 -- 172.21.15.176:0/2129197668 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6abc061be0 con 0x7f6aa80a10c0 2023-11-16T16:42:42.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:41 smithi044 ceph-mon[128750]: purged_snaps scrub starts 2023-11-16T16:42:42.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:41 smithi044 ceph-mon[128750]: purged_snaps scrub ok 2023-11-16T16:42:42.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:41 smithi044 ceph-mon[128750]: pgmap v61: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:42.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:41 smithi044 ceph-mon[128750]: osd.1 [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975] boot 2023-11-16T16:42:42.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:41 smithi044 ceph-mon[128750]: osdmap e13: 2 total, 2 up, 2 in 2023-11-16T16:42:42.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:41 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:43.025 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:42 smithi176 ceph-mon[131630]: osdmap e14: 2 total, 2 up, 2 in 2023-11-16T16:42:43.026 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:42:43.026 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:42:43.026 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:42 smithi044 ceph-mon[128750]: osdmap e14: 2 total, 2 up, 2 in 2023-11-16T16:42:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:42:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:42:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:43.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:42 smithi077 ceph-mon[131204]: osdmap e14: 2 total, 2 up, 2 in 2023-11-16T16:42:43.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:42:43.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:42:43.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:44.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:43 smithi044 ceph-mon[128750]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:42:44.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:43 smithi044 ceph-mon[128750]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:44.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:43 smithi176 ceph-mon[131630]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:42:44.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:43 smithi176 ceph-mon[131630]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:44.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:43 smithi077 ceph-mon[131204]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:42:44.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:43 smithi077 ceph-mon[131204]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:45.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:44 smithi044 ceph-mon[128750]: from='client.? 172.21.15.176:0/1785578487' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98830193-aa89-42b5-a96b-ea72147286c1"}]: dispatch 2023-11-16T16:42:45.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:44 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98830193-aa89-42b5-a96b-ea72147286c1"}]: dispatch 2023-11-16T16:42:45.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:44 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98830193-aa89-42b5-a96b-ea72147286c1"}]': finished 2023-11-16T16:42:45.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:44 smithi044 ceph-mon[128750]: osdmap e15: 3 total, 2 up, 3 in 2023-11-16T16:42:45.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:44 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:42:45.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:44 smithi176 ceph-mon[131630]: from='client.? 172.21.15.176:0/1785578487' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98830193-aa89-42b5-a96b-ea72147286c1"}]: dispatch 2023-11-16T16:42:45.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:44 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98830193-aa89-42b5-a96b-ea72147286c1"}]: dispatch 2023-11-16T16:42:45.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:44 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98830193-aa89-42b5-a96b-ea72147286c1"}]': finished 2023-11-16T16:42:45.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:44 smithi176 ceph-mon[131630]: osdmap e15: 3 total, 2 up, 3 in 2023-11-16T16:42:45.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:44 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:42:45.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:44 smithi077 ceph-mon[131204]: from='client.? 172.21.15.176:0/1785578487' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98830193-aa89-42b5-a96b-ea72147286c1"}]: dispatch 2023-11-16T16:42:45.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:44 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98830193-aa89-42b5-a96b-ea72147286c1"}]: dispatch 2023-11-16T16:42:45.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:44 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98830193-aa89-42b5-a96b-ea72147286c1"}]': finished 2023-11-16T16:42:45.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:44 smithi077 ceph-mon[131204]: osdmap e15: 3 total, 2 up, 3 in 2023-11-16T16:42:45.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:44 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:42:46.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:45 smithi044 ceph-mon[128750]: pgmap v66: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:46.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:45 smithi044 ceph-mon[128750]: from='client.? 172.21.15.176:0/3005656047' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:42:46.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:45 smithi176 ceph-mon[131630]: pgmap v66: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:46.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:45 smithi176 ceph-mon[131630]: from='client.? 172.21.15.176:0/3005656047' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:42:46.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:45 smithi077 ceph-mon[131204]: pgmap v66: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:46.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:45 smithi077 ceph-mon[131204]: from='client.? 172.21.15.176:0/3005656047' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:42:48.118 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:47 smithi176 ceph-mon[131630]: pgmap v67: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:48.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:47 smithi044 ceph-mon[128750]: pgmap v67: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:48.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:47 smithi077 ceph-mon[131204]: pgmap v67: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:49 smithi176 ceph-mon[131630]: pgmap v68: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:50.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:49 smithi044 ceph-mon[128750]: pgmap v68: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:50.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:49 smithi077 ceph-mon[131204]: pgmap v68: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:52.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:51 smithi044 ceph-mon[128750]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:52.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:51 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-16T16:42:52.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:51 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:52.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:51 smithi044 ceph-mon[128750]: Deploying daemon osd.2 on smithi176 2023-11-16T16:42:52.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:51 smithi176 ceph-mon[131630]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:52.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:51 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-16T16:42:52.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:51 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:52.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:51 smithi176 ceph-mon[131630]: Deploying daemon osd.2 on smithi176 2023-11-16T16:42:52.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:51 smithi077 ceph-mon[131204]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:52.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:51 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-16T16:42:52.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:51 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:52.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:51 smithi077 ceph-mon[131204]: Deploying daemon osd.2 on smithi176 2023-11-16T16:42:54.005 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:53 smithi176 ceph-mon[131630]: pgmap v70: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:53 smithi044 ceph-mon[128750]: pgmap v70: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:54.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:53 smithi077 ceph-mon[131204]: pgmap v70: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:56.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:55 smithi044 ceph-mon[128750]: pgmap v71: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:56.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:56.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:56.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:56.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:55 smithi176 ceph-mon[131630]: pgmap v71: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:56.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:55 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:56.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:55 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:56.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:55 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:56.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:55 smithi077 ceph-mon[131204]: pgmap v71: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:55 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:55 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:55 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:57.618 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 2 on host 'smithi176' 2023-11-16T16:42:57.619 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:57.617+0000 7f6ab97fa700 1 -- 172.21.15.176:0/2129197668 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6abc061be0 con 0x7f6aa80a10c0 2023-11-16T16:42:57.621 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:57.618+0000 7f6ac23ed700 1 -- 172.21.15.176:0/2129197668 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6aa80a10c0 msgr2=0x7f6aa80a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:57.621 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:57.618+0000 7f6ac23ed700 1 --2- 172.21.15.176:0/2129197668 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6aa80a10c0 0x7f6aa80a3570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6aac006fd0 tx=0x7f6aac008040 comp rx=0 tx=0).stop 2023-11-16T16:42:57.621 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:57.618+0000 7f6ac23ed700 1 -- 172.21.15.176:0/2129197668 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6abc0ffe40 msgr2=0x7f6abc19d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:57.621 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:57.618+0000 7f6ac23ed700 1 --2- 172.21.15.176:0/2129197668 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6abc0ffe40 0x7f6abc19d270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400c9b0 tx=0x7f6aa400cd70 comp rx=0 tx=0).stop 2023-11-16T16:42:57.621 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:57.619+0000 7f6ac23ed700 1 -- 172.21.15.176:0/2129197668 shutdown_connections 2023-11-16T16:42:57.621 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:57.619+0000 7f6ac23ed700 1 --2- 172.21.15.176:0/2129197668 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6aa80a10c0 0x7f6aa80a3570 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:57.621 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:57.619+0000 7f6ac23ed700 1 --2- 172.21.15.176:0/2129197668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6abc100770 0x7f6abc19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:57.621 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:57.619+0000 7f6ac23ed700 1 --2- 172.21.15.176:0/2129197668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6abc0ff4b0 0x7f6abc0fef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:57.621 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:57.619+0000 7f6ac23ed700 1 --2- 172.21.15.176:0/2129197668 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6abc0ffe40 0x7f6abc19d270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:57.621 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:57.619+0000 7f6ac23ed700 1 -- 172.21.15.176:0/2129197668 >> 172.21.15.176:0/2129197668 conn(0x7f6abc0f9f70 msgr2=0x7f6abc1038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:42:57.621 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:57.619+0000 7f6ac23ed700 1 -- 172.21.15.176:0/2129197668 shutdown_connections 2023-11-16T16:42:57.622 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:42:57.619+0000 7f6ac23ed700 1 -- 172.21.15.176:0/2129197668 wait complete. 2023-11-16T16:42:58.005 DEBUG:teuthology.orchestra.run.smithi176:osd.2> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.2.service 2023-11-16T16:42:58.008 INFO:tasks.cephadm:Deploying osd.3 on smithi044 with /dev/vg_nvme/lv_3... 2023-11-16T16:42:58.008 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-11-16T16:42:58.045 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:57 smithi176 ceph-mon[131630]: pgmap v72: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:58.046 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:57 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:58.046 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:57 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:58.046 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:57 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:58.046 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:57 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:58.046 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:57 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:58.046 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:57 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:58.046 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:57 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:58.046 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:57 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:58.052 INFO:journalctl@ceph.osd.2.smithi176.stdout:-- Logs begin at Thu 2023-11-16 16:14:40 UTC. -- 2023-11-16T16:42:58.097 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:57 smithi044 ceph-mon[128750]: pgmap v72: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:58.098 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:57 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:58.098 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:57 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:58.098 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:57 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:58.098 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:57 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:58.098 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:57 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:58.098 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:57 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:58.098 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:57 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:58.098 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:57 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:58.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:57 smithi077 ceph-mon[131204]: pgmap v72: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:58.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:57 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:58.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:57 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:57 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:57 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:57 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:57 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:57 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:42:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:57 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:58.785 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:58 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:42:58.785 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:58 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:42:58.785 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:58 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:42:58.897 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:42:59.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:58 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:42:59.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:58 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:42:59.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:58 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:42:59.145 INFO:journalctl@ceph.osd.2.smithi176.stdout:Nov 16 16:42:58 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-2[137428]: 2023-11-16T16:42:58.824+0000 7f82a5870640 -1 osd.2 0 log_to_monitors true 2023-11-16T16:42:59.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:58 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:42:59.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:58 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:42:59.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:58 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:43:00.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:59 smithi044 ceph-mon[128750]: pgmap v73: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:00.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:59 smithi044 ceph-mon[128750]: from='osd.2 [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T16:43:00.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:42:59 smithi044 ceph-mon[128750]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T16:43:00.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:59 smithi176 ceph-mon[131630]: pgmap v73: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:00.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:59 smithi176 ceph-mon[131630]: from='osd.2 [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T16:43:00.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:42:59 smithi176 ceph-mon[131630]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T16:43:00.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:59 smithi077 ceph-mon[131204]: pgmap v73: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:00.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:59 smithi077 ceph-mon[131204]: from='osd.2 [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T16:43:00.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:42:59 smithi077 ceph-mon[131204]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T16:43:00.825 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-11-16T16:43:00.826 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-11-16T16:43:00.826 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2023-11-16T16:43:00.826 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2023-11-16T16:43:00.826 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0406035 s, 258 MB/s 2023-11-16T16:43:00.826 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2023-11-16T16:43:01.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:00 smithi044 ceph-mon[128750]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-16T16:43:01.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:00 smithi044 ceph-mon[128750]: osdmap e16: 3 total, 2 up, 3 in 2023-11-16T16:43:01.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:00 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:01.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:00 smithi044 ceph-mon[128750]: from='osd.2 [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:43:01.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:00 smithi044 ceph-mon[128750]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:43:01.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:00 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:01.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:00 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:01.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:00 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:01.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:00 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:01.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:00 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:01.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:00 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:01.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:00 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:01.145 INFO:journalctl@ceph.osd.2.smithi176.stdout:Nov 16 16:43:00 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-2[137428]: 2023-11-16T16:43:00.780+0000 7f829b6e8700 -1 osd.2 0 waiting for initial osdmap 2023-11-16T16:43:01.145 INFO:journalctl@ceph.osd.2.smithi176.stdout:Nov 16 16:43:00 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-2[137428]: 2023-11-16T16:43:00.785+0000 7f82954d9700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:43:01.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:00 smithi176 ceph-mon[131630]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-16T16:43:01.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:00 smithi176 ceph-mon[131630]: osdmap e16: 3 total, 2 up, 3 in 2023-11-16T16:43:01.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:00 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:01.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:00 smithi176 ceph-mon[131630]: from='osd.2 [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:43:01.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:00 smithi176 ceph-mon[131630]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:43:01.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:00 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:01.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:00 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:01.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:00 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:01.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:00 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:01.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:00 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:01.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:00 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:01.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:00 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:01.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:00 smithi077 ceph-mon[131204]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-16T16:43:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:00 smithi077 ceph-mon[131204]: osdmap e16: 3 total, 2 up, 3 in 2023-11-16T16:43:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:00 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:00 smithi077 ceph-mon[131204]: from='osd.2 [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:43:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:00 smithi077 ceph-mon[131204]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:43:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:00 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:00 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:00 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:00 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:01.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:00 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:01.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:00 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:01.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:00 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:01.300 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi044:vg_nvme/lv_3 2023-11-16T16:43:02.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:01 smithi044 ceph-mon[128750]: Detected new or changed devices on smithi176 2023-11-16T16:43:02.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:01 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi176 to 17168M 2023-11-16T16:43:02.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:01 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:43:02.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:01 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:43:02.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:01 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:43:02.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:01 smithi044 ceph-mon[128750]: pgmap v75: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:02.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:01 smithi044 ceph-mon[128750]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2023-11-16T16:43:02.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:01 smithi044 ceph-mon[128750]: osdmap e17: 3 total, 2 up, 3 in 2023-11-16T16:43:02.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:01 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:02.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:01 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:02.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:01 smithi176 ceph-mon[131630]: Detected new or changed devices on smithi176 2023-11-16T16:43:02.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:01 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi176 to 17168M 2023-11-16T16:43:02.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:01 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:43:02.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:01 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:43:02.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:01 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:43:02.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:01 smithi176 ceph-mon[131630]: pgmap v75: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:02.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:01 smithi176 ceph-mon[131630]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2023-11-16T16:43:02.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:01 smithi176 ceph-mon[131630]: osdmap e17: 3 total, 2 up, 3 in 2023-11-16T16:43:02.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:01 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:02.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:01 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:02.162 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:43:02.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:01 smithi077 ceph-mon[131204]: Detected new or changed devices on smithi176 2023-11-16T16:43:02.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:01 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi176 to 17168M 2023-11-16T16:43:02.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:01 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:43:02.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:01 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:43:02.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:01 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:43:02.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:01 smithi077 ceph-mon[131204]: pgmap v75: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:02.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:01 smithi077 ceph-mon[131204]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2023-11-16T16:43:02.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:01 smithi077 ceph-mon[131204]: osdmap e17: 3 total, 2 up, 3 in 2023-11-16T16:43:02.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:01 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:02.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:01 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:03.321 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:03 smithi044 ceph-mon[128750]: purged_snaps scrub starts 2023-11-16T16:43:03.321 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:03 smithi044 ceph-mon[128750]: purged_snaps scrub ok 2023-11-16T16:43:03.321 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:03 smithi044 ceph-mon[128750]: osd.2 [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342] boot 2023-11-16T16:43:03.321 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:03 smithi044 ceph-mon[128750]: osdmap e18: 3 total, 3 up, 3 in 2023-11-16T16:43:03.321 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:03 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:03.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:03 smithi176 ceph-mon[131630]: purged_snaps scrub starts 2023-11-16T16:43:03.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:03 smithi176 ceph-mon[131630]: purged_snaps scrub ok 2023-11-16T16:43:03.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:03 smithi176 ceph-mon[131630]: osd.2 [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342] boot 2023-11-16T16:43:03.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:03 smithi176 ceph-mon[131630]: osdmap e18: 3 total, 3 up, 3 in 2023-11-16T16:43:03.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:03 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:03.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:03 smithi077 ceph-mon[131204]: purged_snaps scrub starts 2023-11-16T16:43:03.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:03 smithi077 ceph-mon[131204]: purged_snaps scrub ok 2023-11-16T16:43:03.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:03 smithi077 ceph-mon[131204]: osd.2 [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342] boot 2023-11-16T16:43:03.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:03 smithi077 ceph-mon[131204]: osdmap e18: 3 total, 3 up, 3 in 2023-11-16T16:43:03.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:03 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:03.587 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.584+0000 7f2c29e24700 1 -- 172.21.15.44:0/2839863516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c241002f0 msgr2=0x7f2c24100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:03.587 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.584+0000 7f2c29e24700 1 --2- 172.21.15.44:0/2839863516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c241002f0 0x7f2c24100700 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2c14009990 tx=0x7f2c14009ca0 comp rx=0 tx=0).stop 2023-11-16T16:43:03.587 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.585+0000 7f2c29e24700 1 -- 172.21.15.44:0/2839863516 shutdown_connections 2023-11-16T16:43:03.587 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.585+0000 7f2c29e24700 1 --2- 172.21.15.44:0/2839863516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c24101e80 0x7f2c24108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:03.588 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.585+0000 7f2c29e24700 1 --2- 172.21.15.44:0/2839863516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c241002f0 0x7f2c24100700 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:03.588 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.585+0000 7f2c29e24700 1 --2- 172.21.15.44:0/2839863516 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c241014f0 0x7f2c24101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:03.588 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.585+0000 7f2c29e24700 1 -- 172.21.15.44:0/2839863516 >> 172.21.15.44:0/2839863516 conn(0x7f2c240fb900 msgr2=0x7f2c240fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:43:03.588 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.585+0000 7f2c29e24700 1 -- 172.21.15.44:0/2839863516 shutdown_connections 2023-11-16T16:43:03.588 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.585+0000 7f2c29e24700 1 -- 172.21.15.44:0/2839863516 wait complete. 2023-11-16T16:43:03.591 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.586+0000 7f2c29e24700 1 Processor -- start 2023-11-16T16:43:03.591 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.586+0000 7f2c29e24700 1 -- start start 2023-11-16T16:43:03.591 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.586+0000 7f2c29e24700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c241002f0 0x7f2c24074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:03.591 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.586+0000 7f2c29e24700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c241014f0 0x7f2c24072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:03.591 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.586+0000 7f2c29e24700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c24101e80 0x7f2c24073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:03.591 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.586+0000 7f2c29e24700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c24073ba0 con 0x7f2c241014f0 2023-11-16T16:43:03.591 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.586+0000 7f2c29e24700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c24073d10 con 0x7f2c24101e80 2023-11-16T16:43:03.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.586+0000 7f2c29e24700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c241a01b0 con 0x7f2c241002f0 2023-11-16T16:43:03.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.587+0000 7f2c22ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c241014f0 0x7f2c24072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:03.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.587+0000 7f2c22ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c241014f0 0x7f2c24072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44424/0 (socket says 172.21.15.44:44424) 2023-11-16T16:43:03.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.587+0000 7f2c22ffd700 1 -- 172.21.15.44:0/898991177 learned_addr learned my addr 172.21.15.44:0/898991177 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:43:03.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.587+0000 7f2c237fe700 1 --2- 172.21.15.44:0/898991177 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c241002f0 0x7f2c24074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:03.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.587+0000 7f2c23fff700 1 --2- 172.21.15.44:0/898991177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c24101e80 0x7f2c24073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:03.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.587+0000 7f2c22ffd700 1 -- 172.21.15.44:0/898991177 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c241002f0 msgr2=0x7f2c24074890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:03.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.587+0000 7f2c22ffd700 1 --2- 172.21.15.44:0/898991177 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c241002f0 0x7f2c24074890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:03.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.587+0000 7f2c22ffd700 1 -- 172.21.15.44:0/898991177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c24101e80 msgr2=0x7f2c24073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:03.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.587+0000 7f2c22ffd700 1 --2- 172.21.15.44:0/898991177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c24101e80 0x7f2c24073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:03.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.587+0000 7f2c22ffd700 1 -- 172.21.15.44:0/898991177 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c140095f0 con 0x7f2c241014f0 2023-11-16T16:43:03.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.587+0000 7f2c237fe700 1 --2- 172.21.15.44:0/898991177 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c241002f0 0x7f2c24074890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:43:03.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.587+0000 7f2c22ffd700 1 --2- 172.21.15.44:0/898991177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c241014f0 0x7f2c24072ee0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800e9a0 tx=0x7f2c1800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:43:03.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.588+0000 7f2c20ff9700 1 -- 172.21.15.44:0/898991177 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c18009790 con 0x7f2c241014f0 2023-11-16T16:43:03.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.588+0000 7f2c20ff9700 1 -- 172.21.15.44:0/898991177 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c18010430 con 0x7f2c241014f0 2023-11-16T16:43:03.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.588+0000 7f2c29e24700 1 -- 172.21.15.44:0/898991177 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2410bcb0 con 0x7f2c241014f0 2023-11-16T16:43:03.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.588+0000 7f2c20ff9700 1 -- 172.21.15.44:0/898991177 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c18018680 con 0x7f2c241014f0 2023-11-16T16:43:03.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.588+0000 7f2c29e24700 1 -- 172.21.15.44:0/898991177 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2410bb20 con 0x7f2c241014f0 2023-11-16T16:43:03.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.589+0000 7f2c29e24700 1 -- 172.21.15.44:0/898991177 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c241078d0 con 0x7f2c241014f0 2023-11-16T16:43:03.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.591+0000 7f2c20ff9700 1 -- 172.21.15.44:0/898991177 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f2c180105a0 con 0x7f2c241014f0 2023-11-16T16:43:03.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.592+0000 7f2c20ff9700 1 --2- 172.21.15.44:0/898991177 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2c0c0a1140 0x7f2c0c0a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:03.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.593+0000 7f2c20ff9700 1 -- 172.21.15.44:0/898991177 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3226+0+0 (secure 0 0 0) 0x7f2c18014070 con 0x7f2c241014f0 2023-11-16T16:43:03.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.593+0000 7f2c237fe700 1 --2- 172.21.15.44:0/898991177 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2c0c0a1140 0x7f2c0c0a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:03.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.593+0000 7f2c237fe700 1 --2- 172.21.15.44:0/898991177 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2c0c0a1140 0x7f2c0c0a35f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400b2e0 tx=0x7f2c14005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:43:03.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.595+0000 7f2c20ff9700 1 -- 172.21.15.44:0/898991177 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2c18096390 con 0x7f2c241014f0 2023-11-16T16:43:03.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:03.812+0000 7f2c29e24700 1 -- 172.21.15.44:0/898991177 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c24061c70 con 0x7f2c0c0a1140 2023-11-16T16:43:04.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:04 smithi044 ceph-mon[128750]: pgmap v78: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:04.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:04 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-11-16T16:43:04.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:04 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-11-16T16:43:04.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:04 smithi044 ceph-mon[128750]: osdmap e19: 3 total, 3 up, 3 in 2023-11-16T16:43:04.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:04 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T16:43:04.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:04 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:04.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:04 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:04.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:04 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:04.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:04 smithi176 ceph-mon[131630]: pgmap v78: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:04.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:04 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-11-16T16:43:04.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:04 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-11-16T16:43:04.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:04 smithi176 ceph-mon[131630]: osdmap e19: 3 total, 3 up, 3 in 2023-11-16T16:43:04.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:04 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T16:43:04.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:04 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:04.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:04 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:04.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:04 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:04.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:04 smithi077 ceph-mon[131204]: pgmap v78: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:04.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:04 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-11-16T16:43:04.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:04 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-11-16T16:43:04.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:04 smithi077 ceph-mon[131204]: osdmap e19: 3 total, 3 up, 3 in 2023-11-16T16:43:04.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:04 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T16:43:04.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:04 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:04.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:04 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:04.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:04 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:05.036 INFO:journalctl@ceph.osd.0.smithi044.stdout:Nov 16 16:43:04 smithi044 sudo[143997]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T16:43:05.036 INFO:journalctl@ceph.osd.0.smithi044.stdout:Nov 16 16:43:04 smithi044 sudo[143997]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:05.036 INFO:journalctl@ceph.osd.0.smithi044.stdout:Nov 16 16:43:04 smithi044 sudo[143997]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:05.036 INFO:journalctl@ceph.osd.0.smithi044.stdout:Nov 16 16:43:04 smithi044 sudo[143997]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:05.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:05.129+0000 7f2c20ff9700 1 -- 172.21.15.44:0/898991177 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2c18010880 con 0x7f2c241014f0 2023-11-16T16:43:05.343 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:05 smithi044 ceph-mon[128750]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:05.343 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:05 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-11-16T16:43:05.343 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:05 smithi044 ceph-mon[128750]: osdmap e20: 3 total, 3 up, 3 in 2023-11-16T16:43:05.343 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:05 smithi044 ceph-mon[128750]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:05.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:05 smithi176 ceph-mon[131630]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:05.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:05 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-11-16T16:43:05.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:05 smithi176 ceph-mon[131630]: osdmap e20: 3 total, 3 up, 3 in 2023-11-16T16:43:05.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:05 smithi176 ceph-mon[131630]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:05.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:05 smithi077 ceph-mon[131204]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:05.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:05 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-11-16T16:43:05.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:05 smithi077 ceph-mon[131204]: osdmap e20: 3 total, 3 up, 3 in 2023-11-16T16:43:05.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:05 smithi077 ceph-mon[131204]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:05.618 INFO:journalctl@ceph.osd.0.smithi044.stdout:Nov 16 16:43:05 smithi044 sudo[144062]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T16:43:05.618 INFO:journalctl@ceph.osd.0.smithi044.stdout:Nov 16 16:43:05 smithi044 sudo[144062]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:05.618 INFO:journalctl@ceph.osd.0.smithi044.stdout:Nov 16 16:43:05 smithi044 sudo[144062]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:05.618 INFO:journalctl@ceph.osd.0.smithi044.stdout:Nov 16 16:43:05 smithi044 sudo[144062]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:05.899 INFO:journalctl@ceph.osd.1.smithi077.stdout:Nov 16 16:43:05 smithi077 sudo[138601]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T16:43:05.899 INFO:journalctl@ceph.osd.1.smithi077.stdout:Nov 16 16:43:05 smithi077 sudo[138601]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:05.899 INFO:journalctl@ceph.osd.1.smithi077.stdout:Nov 16 16:43:05 smithi077 sudo[138601]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:05.899 INFO:journalctl@ceph.osd.1.smithi077.stdout:Nov 16 16:43:05 smithi077 sudo[138601]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:06.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:06 smithi077 ceph-mon[131204]: osdmap e21: 3 total, 3 up, 3 in 2023-11-16T16:43:06.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:06 smithi077 ceph-mon[131204]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-11-16T16:43:06.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:06 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:06.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:06 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:06.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Nov 16 16:43:05 smithi077 sudo[138604]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T16:43:06.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Nov 16 16:43:05 smithi077 sudo[138604]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:06.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Nov 16 16:43:05 smithi077 sudo[138604]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:06.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Nov 16 16:43:05 smithi077 sudo[138604]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:06.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:06 smithi044 ceph-mon[128750]: osdmap e21: 3 total, 3 up, 3 in 2023-11-16T16:43:06.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:06 smithi044 ceph-mon[128750]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-11-16T16:43:06.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:06 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:06.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:06 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:06.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:06 smithi176 ceph-mon[131630]: osdmap e21: 3 total, 3 up, 3 in 2023-11-16T16:43:06.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:06 smithi176 ceph-mon[131630]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-11-16T16:43:06.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:06 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:06.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:06 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:06.395 INFO:journalctl@ceph.osd.2.smithi176.stdout:Nov 16 16:43:06 smithi176 sudo[139650]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T16:43:06.395 INFO:journalctl@ceph.osd.2.smithi176.stdout:Nov 16 16:43:06 smithi176 sudo[139650]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:06.396 INFO:journalctl@ceph.osd.2.smithi176.stdout:Nov 16 16:43:06 smithi176 sudo[139650]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:06.396 INFO:journalctl@ceph.osd.2.smithi176.stdout:Nov 16 16:43:06 smithi176 sudo[139650]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:06.895 INFO:journalctl@ceph.osd.2.smithi176.stdout:Nov 16 16:43:06 smithi176 sudo[139653]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T16:43:06.895 INFO:journalctl@ceph.osd.2.smithi176.stdout:Nov 16 16:43:06 smithi176 sudo[139653]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:06.895 INFO:journalctl@ceph.osd.2.smithi176.stdout:Nov 16 16:43:06 smithi176 sudo[139653]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:06.895 INFO:journalctl@ceph.osd.2.smithi176.stdout:Nov 16 16:43:06 smithi176 sudo[139653]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:07.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:06 smithi044 sudo[144154]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T16:43:07.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:06 smithi044 sudo[144154]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:07.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:06 smithi044 sudo[144154]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:07.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:07 smithi044 sudo[144154]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:07.797 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:07.798 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:07 smithi176 ceph-mon[131630]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:07 smithi044 sudo[144158]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T16:43:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:07 smithi044 sudo[144158]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:07 smithi044 sudo[144158]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:07 smithi044 sudo[144158]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:07.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:07 smithi044 ceph-mon[128750]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:07.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:07.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:07 smithi077 ceph-mon[131204]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:08.049 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:07 smithi176 sudo[139656]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T16:43:08.049 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:07 smithi176 sudo[139656]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:08.049 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:07 smithi176 sudo[139656]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:08.049 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:07 smithi176 sudo[139656]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:08.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 sudo[139659]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T16:43:08.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 sudo[139659]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:08.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 sudo[139659]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:08.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 sudo[139659]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:08.544 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 sudo[138607]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T16:43:08.544 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 sudo[138607]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:08.544 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 sudo[138607]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:08.544 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 sudo[138607]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:08.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2323780668' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba9c2299-3850-4b08-b361-6ed673629019"}]: dispatch 2023-11-16T16:43:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba9c2299-3850-4b08-b361-6ed673629019"}]: dispatch 2023-11-16T16:43:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba9c2299-3850-4b08-b361-6ed673629019"}]': finished 2023-11-16T16:43:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: osdmap e22: 4 total, 3 up, 4 in 2023-11-16T16:43:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:08.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:08.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:08.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:08.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:08 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3424186714' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:08.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:08.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2323780668' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba9c2299-3850-4b08-b361-6ed673629019"}]: dispatch 2023-11-16T16:43:08.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:08.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:08.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba9c2299-3850-4b08-b361-6ed673629019"}]: dispatch 2023-11-16T16:43:08.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:08.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:08.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba9c2299-3850-4b08-b361-6ed673629019"}]': finished 2023-11-16T16:43:08.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: osdmap e22: 4 total, 3 up, 4 in 2023-11-16T16:43:08.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:08.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:08.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:08.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:08.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:08.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:08.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:08.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:08.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:08.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:08 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3424186714' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 sudo[138610]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T16:43:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 sudo[138610]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 sudo[138610]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 sudo[138610]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2323780668' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba9c2299-3850-4b08-b361-6ed673629019"}]: dispatch 2023-11-16T16:43:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:08.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba9c2299-3850-4b08-b361-6ed673629019"}]: dispatch 2023-11-16T16:43:08.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:08.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:08.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba9c2299-3850-4b08-b361-6ed673629019"}]': finished 2023-11-16T16:43:08.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: osdmap e22: 4 total, 3 up, 4 in 2023-11-16T16:43:08.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:08.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:08.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:08.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:08.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:08.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:08.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:08.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:08.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:08.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:08 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3424186714' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:09.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:09 smithi044 ceph-mon[128750]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:09.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:09 smithi044 ceph-mon[128750]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:09.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:09 smithi044 ceph-mon[128750]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:09.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:09 smithi176 ceph-mon[131630]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:09.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:09 smithi176 ceph-mon[131630]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:09.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:09 smithi176 ceph-mon[131630]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:09.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:09 smithi077 ceph-mon[131204]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:09.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:09 smithi077 ceph-mon[131204]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:09.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:09 smithi077 ceph-mon[131204]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:11.809 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:11 smithi044 ceph-mon[128750]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:11.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:11 smithi176 ceph-mon[131630]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:11.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:11 smithi077 ceph-mon[131204]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:13.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:13 smithi044 ceph-mon[128750]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 53 KiB/s wr, 5 op/s 2023-11-16T16:43:13.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:13 smithi176 ceph-mon[131630]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 53 KiB/s wr, 5 op/s 2023-11-16T16:43:13.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:13 smithi077 ceph-mon[131204]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 53 KiB/s wr, 5 op/s 2023-11-16T16:43:15.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:15 smithi044 ceph-mon[128750]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 38 KiB/s wr, 3 op/s 2023-11-16T16:43:15.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:15 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-16T16:43:15.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:15 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:15 smithi176 ceph-mon[131630]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 38 KiB/s wr, 3 op/s 2023-11-16T16:43:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:15 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-16T16:43:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:15 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:15.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:15 smithi077 ceph-mon[131204]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 38 KiB/s wr, 3 op/s 2023-11-16T16:43:15.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:15 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-16T16:43:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:15 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:16.584 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:16 smithi044 ceph-mon[128750]: Deploying daemon osd.3 on smithi044 2023-11-16T16:43:16.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:16 smithi176 ceph-mon[131630]: Deploying daemon osd.3 on smithi044 2023-11-16T16:43:16.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:16 smithi077 ceph-mon[131204]: Deploying daemon osd.3 on smithi044 2023-11-16T16:43:17.867 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:17 smithi044 ceph-mon[128750]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 29 KiB/s wr, 2 op/s 2023-11-16T16:43:17.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:17 smithi176 ceph-mon[131630]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 29 KiB/s wr, 2 op/s 2023-11-16T16:43:17.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:17 smithi077 ceph-mon[131204]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 29 KiB/s wr, 2 op/s 2023-11-16T16:43:19.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:19 smithi044 ceph-mon[128750]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 937 B/s rd, 24 KiB/s wr, 2 op/s 2023-11-16T16:43:19.394 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:19 smithi176 ceph-mon[131630]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 937 B/s rd, 24 KiB/s wr, 2 op/s 2023-11-16T16:43:19.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:19 smithi077 ceph-mon[131204]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 937 B/s rd, 24 KiB/s wr, 2 op/s 2023-11-16T16:43:21.800 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:21 smithi044 ceph-mon[128750]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-11-16T16:43:21.800 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:21 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:21.800 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:21 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:21.800 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:21 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:21 smithi176 ceph-mon[131630]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-11-16T16:43:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:21 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:21 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:21 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:21 smithi077 ceph-mon[131204]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-11-16T16:43:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:21 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:21 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:21 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:23 smithi044 ceph-mon[128750]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-11-16T16:43:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:23 smithi176 ceph-mon[131630]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-11-16T16:43:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:23 smithi077 ceph-mon[131204]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-11-16T16:43:25.085 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 3 on host 'smithi044' 2023-11-16T16:43:25.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:25.084+0000 7f2c20ff9700 1 -- 172.21.15.44:0/898991177 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2c24061c70 con 0x7f2c0c0a1140 2023-11-16T16:43:25.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:25.086+0000 7f2c29e24700 1 -- 172.21.15.44:0/898991177 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2c0c0a1140 msgr2=0x7f2c0c0a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:25.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:25.086+0000 7f2c29e24700 1 --2- 172.21.15.44:0/898991177 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2c0c0a1140 0x7f2c0c0a35f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400b2e0 tx=0x7f2c14005fb0 comp rx=0 tx=0).stop 2023-11-16T16:43:25.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:25.087+0000 7f2c29e24700 1 -- 172.21.15.44:0/898991177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c241014f0 msgr2=0x7f2c24072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:25.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:25.087+0000 7f2c29e24700 1 --2- 172.21.15.44:0/898991177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c241014f0 0x7f2c24072ee0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800e9a0 tx=0x7f2c1800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:43:25.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:25.087+0000 7f2c29e24700 1 -- 172.21.15.44:0/898991177 shutdown_connections 2023-11-16T16:43:25.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:25.087+0000 7f2c29e24700 1 --2- 172.21.15.44:0/898991177 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2c0c0a1140 0x7f2c0c0a35f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:25.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:25.087+0000 7f2c29e24700 1 --2- 172.21.15.44:0/898991177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c24101e80 0x7f2c24073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:25.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:25.087+0000 7f2c29e24700 1 --2- 172.21.15.44:0/898991177 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c241002f0 0x7f2c24074890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:25.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:25.087+0000 7f2c29e24700 1 --2- 172.21.15.44:0/898991177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c241014f0 0x7f2c24072ee0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:25.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:25.087+0000 7f2c29e24700 1 -- 172.21.15.44:0/898991177 >> 172.21.15.44:0/898991177 conn(0x7f2c240fb900 msgr2=0x7f2c24104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:43:25.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:25.087+0000 7f2c29e24700 1 -- 172.21.15.44:0/898991177 shutdown_connections 2023-11-16T16:43:25.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:43:25.087+0000 7f2c29e24700 1 -- 172.21.15.44:0/898991177 wait complete. 2023-11-16T16:43:25.499 DEBUG:teuthology.orchestra.run.smithi044:osd.3> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.3.service 2023-11-16T16:43:25.504 INFO:tasks.cephadm:Deploying osd.4 on smithi077 with /dev/vg_nvme/lv_3... 2023-11-16T16:43:25.505 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-11-16T16:43:25.592 INFO:journalctl@ceph.osd.3.smithi044.stdout:-- Logs begin at Thu 2023-11-16 16:19:45 UTC. -- 2023-11-16T16:43:25.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:25 smithi044 ceph-mon[128750]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:25.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:25 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:25.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:25 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:25.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:25 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:25.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:25 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:25.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:25 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:43:25.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:25 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:25.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:25 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:43:25.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:25 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:43:25.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:25 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:25.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:25 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:25.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:25 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:25.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:25 smithi044 ceph-mon[128750]: from='osd.3 [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-16T16:43:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:25 smithi176 ceph-mon[131630]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:25 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:25 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:25 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:25 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:25 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:43:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:25 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:25 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:43:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:25 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:43:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:25 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:25 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:25.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:25 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:25.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:25 smithi176 ceph-mon[131630]: from='osd.3 [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-16T16:43:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:25 smithi077 ceph-mon[131204]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:25 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:25 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:25 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:25 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:25 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:43:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:25 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:25 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:43:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:25 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:43:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:25 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:25 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:25 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:25 smithi077 ceph-mon[131204]: from='osd.3 [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-16T16:43:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T16:43:27.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:27 smithi044 ceph-mon[128750]: from='osd.3 [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-16T16:43:27.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:27 smithi044 ceph-mon[128750]: osdmap e23: 4 total, 3 up, 4 in 2023-11-16T16:43:27.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:27 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:27.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:27 smithi044 ceph-mon[128750]: from='osd.3 [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2023-11-16T16:43:27.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:27 smithi044 ceph-mon[128750]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:27.369 INFO:journalctl@ceph.osd.3.smithi044.stdout:Nov 16 16:43:27 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-3[146392]: 2023-11-16T16:43:27.076+0000 7fae6339b700 -1 osd.3 0 waiting for initial osdmap 2023-11-16T16:43:27.369 INFO:journalctl@ceph.osd.3.smithi044.stdout:Nov 16 16:43:27 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-3[146392]: 2023-11-16T16:43:27.083+0000 7fae5e98f700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:43:27.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:27 smithi176 ceph-mon[131630]: from='osd.3 [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-16T16:43:27.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:27 smithi176 ceph-mon[131630]: osdmap e23: 4 total, 3 up, 4 in 2023-11-16T16:43:27.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:27 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:27.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:27 smithi176 ceph-mon[131630]: from='osd.3 [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2023-11-16T16:43:27.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:27 smithi176 ceph-mon[131630]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:27.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:27 smithi077 ceph-mon[131204]: from='osd.3 [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-16T16:43:27.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:27 smithi077 ceph-mon[131204]: osdmap e23: 4 total, 3 up, 4 in 2023-11-16T16:43:27.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:27 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:27.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:27 smithi077 ceph-mon[131204]: from='osd.3 [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2023-11-16T16:43:27.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:27 smithi077 ceph-mon[131204]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:27.622 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-11-16T16:43:27.622 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-11-16T16:43:27.622 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2023-11-16T16:43:27.622 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2023-11-16T16:43:27.622 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133731 s, 784 MB/s 2023-11-16T16:43:27.623 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2023-11-16T16:43:28.000 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi077:vg_nvme/lv_3 2023-11-16T16:43:28.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:28 smithi044 ceph-mon[128750]: from='osd.3 [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2023-11-16T16:43:28.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:28 smithi044 ceph-mon[128750]: osdmap e24: 4 total, 3 up, 4 in 2023-11-16T16:43:28.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:28 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:28.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:28 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:28.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:28 smithi176 ceph-mon[131630]: from='osd.3 [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2023-11-16T16:43:28.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:28 smithi176 ceph-mon[131630]: osdmap e24: 4 total, 3 up, 4 in 2023-11-16T16:43:28.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:28 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:28.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:28 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:28.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:28 smithi077 ceph-mon[131204]: from='osd.3 [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2023-11-16T16:43:28.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:28 smithi077 ceph-mon[131204]: osdmap e24: 4 total, 3 up, 4 in 2023-11-16T16:43:28.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:28 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:28.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:28 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:28.642 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T16:43:29.275 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: purged_snaps scrub starts 2023-11-16T16:43:29.275 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: purged_snaps scrub ok 2023-11-16T16:43:29.275 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: osd.3 [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584] boot 2023-11-16T16:43:29.276 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: osdmap e25: 4 total, 4 up, 4 in 2023-11-16T16:43:29.276 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:29.276 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: Detected new or changed devices on smithi044 2023-11-16T16:43:29.276 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:29.276 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:29.276 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:29.276 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:29.276 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi044 to 8638M 2023-11-16T16:43:29.276 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:29.277 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:29.277 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:29.277 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:43:29.277 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:29.277 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:43:29.277 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:43:29.277 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:29 smithi077 ceph-mon[131204]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:29.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: purged_snaps scrub starts 2023-11-16T16:43:29.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: purged_snaps scrub ok 2023-11-16T16:43:29.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: osd.3 [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584] boot 2023-11-16T16:43:29.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: osdmap e25: 4 total, 4 up, 4 in 2023-11-16T16:43:29.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:29.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: Detected new or changed devices on smithi044 2023-11-16T16:43:29.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:29.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:29.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:29.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:29.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi044 to 8638M 2023-11-16T16:43:29.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:29.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:29.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:29.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:43:29.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:29.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:43:29.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:43:29.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:29 smithi044 ceph-mon[128750]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:29.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: purged_snaps scrub starts 2023-11-16T16:43:29.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: purged_snaps scrub ok 2023-11-16T16:43:29.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: osd.3 [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584] boot 2023-11-16T16:43:29.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: osdmap e25: 4 total, 4 up, 4 in 2023-11-16T16:43:29.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:29.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: Detected new or changed devices on smithi044 2023-11-16T16:43:29.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:29.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:29.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:29.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:29.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi044 to 8638M 2023-11-16T16:43:29.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:29.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:29.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:29.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:43:29.397 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:29.397 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:43:29.397 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:43:29.397 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:29 smithi176 ceph-mon[131630]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:29.929 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.925+0000 7ff38a9e0700 1 -- 172.21.15.77:0/3430057237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff384100d70 msgr2=0x7ff384107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:29.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.925+0000 7ff38a9e0700 1 --2- 172.21.15.77:0/3430057237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff384100d70 0x7ff384107630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00b0a0 tx=0x7ff37c00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:43:29.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.925+0000 7ff38a9e0700 1 -- 172.21.15.77:0/3430057237 shutdown_connections 2023-11-16T16:43:29.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.925+0000 7ff38a9e0700 1 --2- 172.21.15.77:0/3430057237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff384100d70 0x7ff384107630 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:29.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.925+0000 7ff38a9e0700 1 --2- 172.21.15.77:0/3430057237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3840ff1e0 0x7ff3840ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:29.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.925+0000 7ff38a9e0700 1 --2- 172.21.15.77:0/3430057237 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3841003e0 0x7ff384100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:29.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.925+0000 7ff38a9e0700 1 -- 172.21.15.77:0/3430057237 >> 172.21.15.77:0/3430057237 conn(0x7ff3840fa7f0 msgr2=0x7ff3840fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:43:29.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.925+0000 7ff38a9e0700 1 -- 172.21.15.77:0/3430057237 shutdown_connections 2023-11-16T16:43:29.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.925+0000 7ff38a9e0700 1 -- 172.21.15.77:0/3430057237 wait complete. 2023-11-16T16:43:29.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.926+0000 7ff38a9e0700 1 Processor -- start 2023-11-16T16:43:29.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.926+0000 7ff38a9e0700 1 -- start start 2023-11-16T16:43:29.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.926+0000 7ff38a9e0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3840ff1e0 0x7ff384069130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:29.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.926+0000 7ff38a9e0700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3841003e0 0x7ff38419ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:29.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.926+0000 7ff38a9e0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff384100d70 0x7ff38419d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:29.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.926+0000 7ff38a9e0700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38419dc50 con 0x7ff3841003e0 2023-11-16T16:43:29.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.926+0000 7ff38a9e0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38419ddc0 con 0x7ff384100d70 2023-11-16T16:43:29.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.926+0000 7ff38a9e0700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38419df80 con 0x7ff3840ff1e0 2023-11-16T16:43:29.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.926+0000 7ff388f7d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff384100d70 0x7ff38419d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:29.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.926+0000 7ff388f7d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff384100d70 0x7ff38419d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50872/0 (socket says 172.21.15.77:50872) 2023-11-16T16:43:29.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.927+0000 7ff388f7d700 1 -- 172.21.15.77:0/2956463544 learned_addr learned my addr 172.21.15.77:0/2956463544 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-11-16T16:43:29.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.927+0000 7ff383fff700 1 --2- 172.21.15.77:0/2956463544 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3840ff1e0 0x7ff384069130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:29.932 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.927+0000 7ff388f7d700 1 -- 172.21.15.77:0/2956463544 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3840ff1e0 msgr2=0x7ff384069130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:29.932 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.927+0000 7ff388f7d700 1 --2- 172.21.15.77:0/2956463544 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3840ff1e0 0x7ff384069130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:29.932 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.927+0000 7ff388f7d700 1 -- 172.21.15.77:0/2956463544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3841003e0 msgr2=0x7ff38419ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:29.932 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.927+0000 7ff388f7d700 1 --2- 172.21.15.77:0/2956463544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3841003e0 0x7ff38419ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:29.932 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.927+0000 7ff388f7d700 1 -- 172.21.15.77:0/2956463544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37c009d00 con 0x7ff384100d70 2023-11-16T16:43:29.932 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.927+0000 7ff388f7d700 1 --2- 172.21.15.77:0/2956463544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff384100d70 0x7ff38419d4b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00b830 tx=0x7ff37c0094e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:43:29.932 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.927+0000 7ff3817fa700 1 -- 172.21.15.77:0/2956463544 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c00e040 con 0x7ff384100d70 2023-11-16T16:43:29.932 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.927+0000 7ff3817fa700 1 -- 172.21.15.77:0/2956463544 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff37c004de0 con 0x7ff384100d70 2023-11-16T16:43:29.932 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.927+0000 7ff38a9e0700 1 -- 172.21.15.77:0/2956463544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38410ab30 con 0x7ff384100d70 2023-11-16T16:43:29.932 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.927+0000 7ff3817fa700 1 -- 172.21.15.77:0/2956463544 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c01d480 con 0x7ff384100d70 2023-11-16T16:43:29.932 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.927+0000 7ff38a9e0700 1 -- 172.21.15.77:0/2956463544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38410a9a0 con 0x7ff384100d70 2023-11-16T16:43:29.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.928+0000 7ff38a9e0700 1 -- 172.21.15.77:0/2956463544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff384106780 con 0x7ff384100d70 2023-11-16T16:43:29.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.929+0000 7ff3817fa700 1 -- 172.21.15.77:0/2956463544 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff37c003a00 con 0x7ff384100d70 2023-11-16T16:43:29.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.930+0000 7ff3817fa700 1 --2- 172.21.15.77:0/2956463544 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff36c0a1150 0x7ff36c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:29.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.930+0000 7ff3817fa700 1 -- 172.21.15.77:0/2956463544 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7ff37c0cd490 con 0x7ff384100d70 2023-11-16T16:43:29.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.931+0000 7ff383fff700 1 --2- 172.21.15.77:0/2956463544 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff36c0a1150 0x7ff36c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:29.934 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.931+0000 7ff383fff700 1 --2- 172.21.15.77:0/2956463544 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff36c0a1150 0x7ff36c0a3600 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff384100240 tx=0x7ff37400a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:43:29.936 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:29.934+0000 7ff3817fa700 1 -- 172.21.15.77:0/2956463544 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff37c097e50 con 0x7ff384100d70 2023-11-16T16:43:30.136 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:30.134+0000 7ff38a9e0700 1 -- 172.21.15.77:0/2956463544 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ff384061b20 con 0x7ff36c0a1150 2023-11-16T16:43:30.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:30 smithi044 ceph-mon[128750]: osdmap e26: 4 total, 4 up, 4 in 2023-11-16T16:43:30.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:30 smithi044 ceph-mon[128750]: osdmap e27: 4 total, 4 up, 4 in 2023-11-16T16:43:30.394 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:30 smithi176 ceph-mon[131630]: osdmap e26: 4 total, 4 up, 4 in 2023-11-16T16:43:30.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:30 smithi176 ceph-mon[131630]: osdmap e27: 4 total, 4 up, 4 in 2023-11-16T16:43:30.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:30 smithi077 ceph-mon[131204]: osdmap e26: 4 total, 4 up, 4 in 2023-11-16T16:43:30.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:30 smithi077 ceph-mon[131204]: osdmap e27: 4 total, 4 up, 4 in 2023-11-16T16:43:31.291 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:31 smithi077 ceph-mon[131204]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:31.291 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:31 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:31.292 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:31 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:31.292 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:31 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:31.292 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:31 smithi077 ceph-mon[131204]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:31.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:31 smithi044 ceph-mon[128750]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:31.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:31 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:31.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:31 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:31.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:31 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:31.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:31 smithi044 ceph-mon[128750]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:31.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:31 smithi176 ceph-mon[131630]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:31.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:31 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:31.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:31 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:31.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:31 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:31.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:31 smithi176 ceph-mon[131630]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:32.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:32 smithi044 ceph-mon[128750]: osdmap e28: 4 total, 4 up, 4 in 2023-11-16T16:43:32.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:32 smithi176 ceph-mon[131630]: osdmap e28: 4 total, 4 up, 4 in 2023-11-16T16:43:32.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:32 smithi077 ceph-mon[131204]: osdmap e28: 4 total, 4 up, 4 in 2023-11-16T16:43:33.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:33 smithi044 ceph-mon[128750]: osdmap e29: 4 total, 4 up, 4 in 2023-11-16T16:43:33.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:33 smithi044 ceph-mon[128750]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T16:43:33.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:33 smithi044 ceph-mon[128750]: from='client.? 172.21.15.77:0/1938276378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a7ee641-fb5c-4a14-83b5-07c5b5842cfc"}]: dispatch 2023-11-16T16:43:33.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:33 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a7ee641-fb5c-4a14-83b5-07c5b5842cfc"}]: dispatch 2023-11-16T16:43:33.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:33 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a7ee641-fb5c-4a14-83b5-07c5b5842cfc"}]': finished 2023-11-16T16:43:33.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:33 smithi044 ceph-mon[128750]: osdmap e30: 5 total, 4 up, 5 in 2023-11-16T16:43:33.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:33 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:33.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:33 smithi176 ceph-mon[131630]: osdmap e29: 4 total, 4 up, 4 in 2023-11-16T16:43:33.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:33 smithi176 ceph-mon[131630]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T16:43:33.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:33 smithi176 ceph-mon[131630]: from='client.? 172.21.15.77:0/1938276378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a7ee641-fb5c-4a14-83b5-07c5b5842cfc"}]: dispatch 2023-11-16T16:43:33.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:33 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a7ee641-fb5c-4a14-83b5-07c5b5842cfc"}]: dispatch 2023-11-16T16:43:33.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:33 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a7ee641-fb5c-4a14-83b5-07c5b5842cfc"}]': finished 2023-11-16T16:43:33.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:33 smithi176 ceph-mon[131630]: osdmap e30: 5 total, 4 up, 5 in 2023-11-16T16:43:33.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:33 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:33.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:33 smithi077 ceph-mon[131204]: osdmap e29: 4 total, 4 up, 4 in 2023-11-16T16:43:33.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:33 smithi077 ceph-mon[131204]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T16:43:33.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:33 smithi077 ceph-mon[131204]: from='client.? 172.21.15.77:0/1938276378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a7ee641-fb5c-4a14-83b5-07c5b5842cfc"}]: dispatch 2023-11-16T16:43:33.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:33 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a7ee641-fb5c-4a14-83b5-07c5b5842cfc"}]: dispatch 2023-11-16T16:43:33.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:33 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a7ee641-fb5c-4a14-83b5-07c5b5842cfc"}]': finished 2023-11-16T16:43:33.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:33 smithi077 ceph-mon[131204]: osdmap e30: 5 total, 4 up, 5 in 2023-11-16T16:43:33.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:33 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:34.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:34 smithi044 ceph-mon[128750]: from='client.? 172.21.15.77:0/1042163995' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:34.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:34 smithi176 ceph-mon[131630]: from='client.? 172.21.15.77:0/1042163995' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:34.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:34 smithi077 ceph-mon[131204]: from='client.? 172.21.15.77:0/1042163995' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:35.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:35 smithi044 ceph-mon[128750]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-11-16T16:43:35.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:35 smithi176 ceph-mon[131630]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-11-16T16:43:35.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:35 smithi077 ceph-mon[131204]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-11-16T16:43:37.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:37 smithi077 ceph-mon[131204]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T16:43:37.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:37 smithi044 ceph-mon[128750]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T16:43:37.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:37 smithi176 ceph-mon[131630]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T16:43:39.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:39 smithi044 ceph-mon[128750]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-11-16T16:43:39.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:39 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-16T16:43:39.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:39 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:39.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:39 smithi176 ceph-mon[131630]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-11-16T16:43:39.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:39 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-16T16:43:39.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:39 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:39.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:39 smithi077 ceph-mon[131204]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-11-16T16:43:39.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-16T16:43:39.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:40.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:40 smithi077 ceph-mon[131204]: Deploying daemon osd.4 on smithi077 2023-11-16T16:43:40.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:40 smithi044 ceph-mon[128750]: Deploying daemon osd.4 on smithi077 2023-11-16T16:43:40.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:40 smithi176 ceph-mon[131630]: Deploying daemon osd.4 on smithi077 2023-11-16T16:43:41.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:41 smithi044 ceph-mon[128750]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:41.372 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:41 smithi077 ceph-mon[131204]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:41.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:41 smithi176 ceph-mon[131630]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:43.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:43 smithi044 ceph-mon[128750]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:43.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:43.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:43.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:43 smithi176 ceph-mon[131630]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:43 smithi077 ceph-mon[131204]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:45.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:45 smithi044 ceph-mon[128750]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:45.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:45.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:45.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:45.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:45.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:45.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:45 smithi176 ceph-mon[131630]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:45.931 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:45 smithi077 ceph-mon[131204]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:45.932 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:45.932 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:45.932 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:45.932 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:45.932 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:46.251 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:46.250+0000 7ff3817fa700 1 -- 172.21.15.77:0/2956463544 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff384061b20 con 0x7ff36c0a1150 2023-11-16T16:43:46.251 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 4 on host 'smithi077' 2023-11-16T16:43:46.253 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:46.252+0000 7ff38a9e0700 1 -- 172.21.15.77:0/2956463544 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff36c0a1150 msgr2=0x7ff36c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:46.253 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:46.252+0000 7ff38a9e0700 1 --2- 172.21.15.77:0/2956463544 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff36c0a1150 0x7ff36c0a3600 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff384100240 tx=0x7ff37400a380 comp rx=0 tx=0).stop 2023-11-16T16:43:46.253 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:46.252+0000 7ff38a9e0700 1 -- 172.21.15.77:0/2956463544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff384100d70 msgr2=0x7ff38419d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:46.253 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:46.252+0000 7ff38a9e0700 1 --2- 172.21.15.77:0/2956463544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff384100d70 0x7ff38419d4b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00b830 tx=0x7ff37c0094e0 comp rx=0 tx=0).stop 2023-11-16T16:43:46.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:46.252+0000 7ff38a9e0700 1 -- 172.21.15.77:0/2956463544 shutdown_connections 2023-11-16T16:43:46.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:46.252+0000 7ff38a9e0700 1 --2- 172.21.15.77:0/2956463544 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff36c0a1150 0x7ff36c0a3600 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:46.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:46.252+0000 7ff38a9e0700 1 --2- 172.21.15.77:0/2956463544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff384100d70 0x7ff38419d4b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:46.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:46.252+0000 7ff38a9e0700 1 --2- 172.21.15.77:0/2956463544 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3840ff1e0 0x7ff384069130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:46.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:46.252+0000 7ff38a9e0700 1 --2- 172.21.15.77:0/2956463544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3841003e0 0x7ff38419ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:46.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:46.252+0000 7ff38a9e0700 1 -- 172.21.15.77:0/2956463544 >> 172.21.15.77:0/2956463544 conn(0x7ff3840fa7f0 msgr2=0x7ff3841038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:43:46.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:46.252+0000 7ff38a9e0700 1 -- 172.21.15.77:0/2956463544 shutdown_connections 2023-11-16T16:43:46.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:43:46.252+0000 7ff38a9e0700 1 -- 172.21.15.77:0/2956463544 wait complete. 2023-11-16T16:43:46.630 DEBUG:teuthology.orchestra.run.smithi077:osd.4> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.4.service 2023-11-16T16:43:46.633 INFO:tasks.cephadm:Deploying osd.5 on smithi176 with /dev/vg_nvme/lv_3... 2023-11-16T16:43:46.633 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-11-16T16:43:46.661 INFO:journalctl@ceph.osd.4.smithi077.stdout:-- Logs begin at Thu 2023-11-16 16:15:33 UTC. -- 2023-11-16T16:43:46.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:46 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:43:46.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:46 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:43:46.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:46 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:43:46.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:46 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:46.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:46 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:46.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:46 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:46.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:46 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:43:46.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:46 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:43:46.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:46 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:43:46.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:46 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:46.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:46 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:46.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:46 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:46.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:46 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:43:46.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:46 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:43:46.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:46 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:43:46.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:46 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:46.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:46 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:46.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:46 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:47.280 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:43:47.431 INFO:journalctl@ceph.osd.4.smithi077.stdout:Nov 16 16:43:47 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-4[141799]: 2023-11-16T16:43:47.191+0000 7f6cf8b65640 -1 osd.4 0 log_to_monitors true 2023-11-16T16:43:47.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:47 smithi077 ceph-mon[131204]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:47.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:47 smithi077 ceph-mon[131204]: from='osd.4 [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T16:43:47.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:47 smithi077 ceph-mon[131204]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T16:43:47.777 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:47 smithi176 ceph-mon[131630]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:47.777 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:47 smithi176 ceph-mon[131630]: from='osd.4 [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T16:43:47.777 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:47 smithi176 ceph-mon[131630]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T16:43:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:47 smithi044 ceph-mon[128750]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:47 smithi044 ceph-mon[128750]: from='osd.4 [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T16:43:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:47 smithi044 ceph-mon[128750]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T16:43:48.721 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-11-16T16:43:48.721 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-11-16T16:43:48.721 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2023-11-16T16:43:48.721 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2023-11-16T16:43:48.721 INFO:teuthology.orchestra.run.smithi176.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110595 s, 948 MB/s 2023-11-16T16:43:48.721 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 2023-11-16T16:43:48.722 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2023-11-16T16:43:48.852 INFO:journalctl@ceph.osd.4.smithi077.stdout:Nov 16 16:43:48 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-4[141799]: 2023-11-16T16:43:48.565+0000 7f6cee1dc700 -1 osd.4 0 waiting for initial osdmap 2023-11-16T16:43:48.852 INFO:journalctl@ceph.osd.4.smithi077.stdout:Nov 16 16:43:48 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-4[141799]: 2023-11-16T16:43:48.573+0000 7f6ce7fcd700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:43:48.853 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:48 smithi077 ceph-mon[131204]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-16T16:43:48.853 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:48 smithi077 ceph-mon[131204]: osdmap e31: 5 total, 4 up, 5 in 2023-11-16T16:43:48.853 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:48 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:48.853 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:48 smithi077 ceph-mon[131204]: from='osd.4 [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:43:48.853 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:48 smithi077 ceph-mon[131204]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:43:48.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:48 smithi044 ceph-mon[128750]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-16T16:43:48.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:48 smithi044 ceph-mon[128750]: osdmap e31: 5 total, 4 up, 5 in 2023-11-16T16:43:48.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:48 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:48.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:48 smithi044 ceph-mon[128750]: from='osd.4 [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:43:48.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:48 smithi044 ceph-mon[128750]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:43:48.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:48 smithi176 ceph-mon[131630]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-16T16:43:48.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:48 smithi176 ceph-mon[131630]: osdmap e31: 5 total, 4 up, 5 in 2023-11-16T16:43:48.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:48 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:48.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:48 smithi176 ceph-mon[131630]: from='osd.4 [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:43:48.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:48 smithi176 ceph-mon[131630]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:43:49.134 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi176:vg_nvme/lv_3 2023-11-16T16:43:49.773 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:43:49.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:49 smithi044 ceph-mon[128750]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:49.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:49 smithi044 ceph-mon[128750]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2023-11-16T16:43:49.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:49 smithi044 ceph-mon[128750]: osdmap e32: 5 total, 4 up, 5 in 2023-11-16T16:43:49.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:49.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:49.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:49.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:49.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:49.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:49.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:49.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:49.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:49.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:49 smithi176 ceph-mon[131630]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:49 smithi176 ceph-mon[131630]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2023-11-16T16:43:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:49 smithi176 ceph-mon[131630]: osdmap e32: 5 total, 4 up, 5 in 2023-11-16T16:43:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:49.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:49.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:49 smithi077 ceph-mon[131204]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:49.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:49 smithi077 ceph-mon[131204]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2023-11-16T16:43:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:49 smithi077 ceph-mon[131204]: osdmap e32: 5 total, 4 up, 5 in 2023-11-16T16:43:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:43:50.858 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.854+0000 7f7720aa4700 1 -- 172.21.15.176:0/258759336 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c1003e0 msgr2=0x7f771c100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:50.859 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.854+0000 7f7720aa4700 1 --2- 172.21.15.176:0/258759336 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c1003e0 0x7f771c100830 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7710009990 tx=0x7f7710009ca0 comp rx=0 tx=0).stop 2023-11-16T16:43:50.859 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.855+0000 7f7720aa4700 1 -- 172.21.15.176:0/258759336 shutdown_connections 2023-11-16T16:43:50.859 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.855+0000 7f7720aa4700 1 --2- 172.21.15.176:0/258759336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f771c100d70 0x7f771c107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:50.859 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.855+0000 7f7720aa4700 1 --2- 172.21.15.176:0/258759336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f771c0ff1e0 0x7f771c0ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:50.859 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.855+0000 7f7720aa4700 1 --2- 172.21.15.176:0/258759336 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c1003e0 0x7f771c100830 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:50.859 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.855+0000 7f7720aa4700 1 -- 172.21.15.176:0/258759336 >> 172.21.15.176:0/258759336 conn(0x7f771c0fa7f0 msgr2=0x7f771c0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:43:50.860 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.855+0000 7f7720aa4700 1 -- 172.21.15.176:0/258759336 shutdown_connections 2023-11-16T16:43:50.860 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.855+0000 7f7720aa4700 1 -- 172.21.15.176:0/258759336 wait complete. 2023-11-16T16:43:50.860 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.855+0000 7f7720aa4700 1 Processor -- start 2023-11-16T16:43:50.860 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.855+0000 7f7720aa4700 1 -- start start 2023-11-16T16:43:50.860 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.855+0000 7f7720aa4700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f771c0ff1e0 0x7f771c103170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:50.860 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f7720aa4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f771c1003e0 0x7f771c1017c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:50.860 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f7720aa4700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c100d70 0x7f771c101d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:50.860 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f7720aa4700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c102480 con 0x7f771c0ff1e0 2023-11-16T16:43:50.860 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f7720aa4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1025f0 con 0x7f771c1003e0 2023-11-16T16:43:50.861 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f7720aa4700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1a0130 con 0x7f771c100d70 2023-11-16T16:43:50.861 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f771ad9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c100d70 0x7f771c101d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:50.861 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f771ad9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c100d70 0x7f771c101d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37874/0 (socket says 172.21.15.176:37874) 2023-11-16T16:43:50.861 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f771ad9d700 1 -- 172.21.15.176:0/4161913973 learned_addr learned my addr 172.21.15.176:0/4161913973 (peer_addr_for_me v2:172.21.15.176:0/0) 2023-11-16T16:43:50.861 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f771a59c700 1 --2- 172.21.15.176:0/4161913973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f771c0ff1e0 0x7f771c103170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:50.861 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f7719d9b700 1 --2- 172.21.15.176:0/4161913973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f771c1003e0 0x7f771c1017c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:50.861 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f771ad9d700 1 -- 172.21.15.176:0/4161913973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f771c1003e0 msgr2=0x7f771c1017c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:50.861 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f771ad9d700 1 --2- 172.21.15.176:0/4161913973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f771c1003e0 0x7f771c1017c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:50.862 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f771ad9d700 1 -- 172.21.15.176:0/4161913973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f771c0ff1e0 msgr2=0x7f771c103170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:50.862 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f771ad9d700 1 --2- 172.21.15.176:0/4161913973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f771c0ff1e0 0x7f771c103170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:50.862 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f771ad9d700 1 -- 172.21.15.176:0/4161913973 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77100095f0 con 0x7f771c100d70 2023-11-16T16:43:50.862 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f7719d9b700 1 --2- 172.21.15.176:0/4161913973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f771c1003e0 0x7f771c1017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:43:50.862 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f771a59c700 1 --2- 172.21.15.176:0/4161913973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f771c0ff1e0 0x7f771c103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:43:50.862 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f771ad9d700 1 --2- 172.21.15.176:0/4161913973 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c100d70 0x7f771c101d00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f771400b970 tx=0x7f771400bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:43:50.862 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f770b7fe700 1 -- 172.21.15.176:0/4161913973 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771400c9b0 con 0x7f771c100d70 2023-11-16T16:43:50.863 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f770b7fe700 1 -- 172.21.15.176:0/4161913973 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7714014440 con 0x7f771c100d70 2023-11-16T16:43:50.863 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f770b7fe700 1 -- 172.21.15.176:0/4161913973 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77140075d0 con 0x7f771c100d70 2023-11-16T16:43:50.863 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f7720aa4700 1 -- 172.21.15.176:0/4161913973 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771c10abb0 con 0x7f771c100d70 2023-11-16T16:43:50.863 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.856+0000 7f7720aa4700 1 -- 172.21.15.176:0/4161913973 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c10aa20 con 0x7f771c100d70 2023-11-16T16:43:50.863 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.857+0000 7f7720aa4700 1 -- 172.21.15.176:0/4161913973 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771c1a0820 con 0x7f771c100d70 2023-11-16T16:43:50.864 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.858+0000 7f770b7fe700 1 -- 172.21.15.176:0/4161913973 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f77140145b0 con 0x7f771c100d70 2023-11-16T16:43:50.864 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.859+0000 7f770b7fe700 1 --2- 172.21.15.176:0/4161913973 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f77040a1150 0x7f77040a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:50.864 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.859+0000 7f770b7fe700 1 -- 172.21.15.176:0/4161913973 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f7714006e40 con 0x7f771c100d70 2023-11-16T16:43:50.865 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.859+0000 7f771a59c700 1 --2- 172.21.15.176:0/4161913973 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f77040a1150 0x7f77040a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:50.865 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.860+0000 7f771a59c700 1 --2- 172.21.15.176:0/4161913973 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f77040a1150 0x7f77040a3600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f770c005950 tx=0x7f770c0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:43:50.865 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:50.863+0000 7f770b7fe700 1 -- 172.21.15.176:0/4161913973 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f771404e730 con 0x7f771c100d70 2023-11-16T16:43:50.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:50 smithi044 ceph-mon[128750]: purged_snaps scrub starts 2023-11-16T16:43:50.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:50 smithi044 ceph-mon[128750]: purged_snaps scrub ok 2023-11-16T16:43:50.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:50 smithi044 ceph-mon[128750]: Detected new or changed devices on smithi077 2023-11-16T16:43:50.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:50 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi077 to 8584M 2023-11-16T16:43:50.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:50 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:43:50.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:50 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:43:50.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:50 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:43:50.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:50 smithi044 ceph-mon[128750]: osd.4 [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497] boot 2023-11-16T16:43:50.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:50 smithi044 ceph-mon[128750]: osdmap e33: 5 total, 5 up, 5 in 2023-11-16T16:43:50.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:50 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:50.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:50 smithi176 ceph-mon[131630]: purged_snaps scrub starts 2023-11-16T16:43:50.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:50 smithi176 ceph-mon[131630]: purged_snaps scrub ok 2023-11-16T16:43:50.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:50 smithi176 ceph-mon[131630]: Detected new or changed devices on smithi077 2023-11-16T16:43:50.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:50 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi077 to 8584M 2023-11-16T16:43:50.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:50 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:43:50.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:50 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:43:50.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:50 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:43:50.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:50 smithi176 ceph-mon[131630]: osd.4 [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497] boot 2023-11-16T16:43:50.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:50 smithi176 ceph-mon[131630]: osdmap e33: 5 total, 5 up, 5 in 2023-11-16T16:43:50.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:50 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:50.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:50 smithi077 ceph-mon[131204]: purged_snaps scrub starts 2023-11-16T16:43:50.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:50 smithi077 ceph-mon[131204]: purged_snaps scrub ok 2023-11-16T16:43:50.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:50 smithi077 ceph-mon[131204]: Detected new or changed devices on smithi077 2023-11-16T16:43:50.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:50 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi077 to 8584M 2023-11-16T16:43:50.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:50 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:43:50.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:50 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:43:50.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:50 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:43:50.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:50 smithi077 ceph-mon[131204]: osd.4 [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497] boot 2023-11-16T16:43:50.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:50 smithi077 ceph-mon[131204]: osdmap e33: 5 total, 5 up, 5 in 2023-11-16T16:43:50.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:50 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:51.059 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:43:51.058+0000 7f7720aa4700 1 -- 172.21.15.176:0/4161913973 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f771c1a0dd0 con 0x7f77040a1150 2023-11-16T16:43:51.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:51 smithi044 ceph-mon[128750]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:51.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:51 smithi044 ceph-mon[128750]: osdmap e34: 5 total, 5 up, 5 in 2023-11-16T16:43:51.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:51 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:51.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:51 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:51.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:51 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:51.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:51 smithi176 ceph-mon[131630]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:51.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:51 smithi176 ceph-mon[131630]: osdmap e34: 5 total, 5 up, 5 in 2023-11-16T16:43:51.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:51 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:51.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:51 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:51.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:51 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:51.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:51 smithi077 ceph-mon[131204]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:51.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:51 smithi077 ceph-mon[131204]: osdmap e34: 5 total, 5 up, 5 in 2023-11-16T16:43:51.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:51 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:51.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:51 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:51.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:51 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:52.777 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:52 smithi176 ceph-mon[131630]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:52.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:52 smithi044 ceph-mon[128750]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:52.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:52 smithi077 ceph-mon[131204]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:53.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:53 smithi044 ceph-mon[128750]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:43:53.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:53 smithi044 ceph-mon[128750]: from='client.? 172.21.15.176:0/947338459' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d16999a0-6bd4-4447-b88e-3271c8ab904a"}]: dispatch 2023-11-16T16:43:53.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:53 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d16999a0-6bd4-4447-b88e-3271c8ab904a"}]: dispatch 2023-11-16T16:43:53.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:53 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d16999a0-6bd4-4447-b88e-3271c8ab904a"}]': finished 2023-11-16T16:43:53.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:53 smithi044 ceph-mon[128750]: osdmap e35: 6 total, 5 up, 6 in 2023-11-16T16:43:53.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:43:53.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:53 smithi176 ceph-mon[131630]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:43:53.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:53 smithi176 ceph-mon[131630]: from='client.? 172.21.15.176:0/947338459' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d16999a0-6bd4-4447-b88e-3271c8ab904a"}]: dispatch 2023-11-16T16:43:53.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:53 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d16999a0-6bd4-4447-b88e-3271c8ab904a"}]: dispatch 2023-11-16T16:43:53.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:53 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d16999a0-6bd4-4447-b88e-3271c8ab904a"}]': finished 2023-11-16T16:43:53.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:53 smithi176 ceph-mon[131630]: osdmap e35: 6 total, 5 up, 6 in 2023-11-16T16:43:53.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:43:53.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:53 smithi077 ceph-mon[131204]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:43:53.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:53 smithi077 ceph-mon[131204]: from='client.? 172.21.15.176:0/947338459' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d16999a0-6bd4-4447-b88e-3271c8ab904a"}]: dispatch 2023-11-16T16:43:53.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:53 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d16999a0-6bd4-4447-b88e-3271c8ab904a"}]: dispatch 2023-11-16T16:43:53.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:53 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d16999a0-6bd4-4447-b88e-3271c8ab904a"}]': finished 2023-11-16T16:43:53.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:53 smithi077 ceph-mon[131204]: osdmap e35: 6 total, 5 up, 6 in 2023-11-16T16:43:53.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:43:54.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:54 smithi044 ceph-mon[128750]: from='client.? 172.21.15.176:0/2442732458' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:54.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:54 smithi176 ceph-mon[131630]: from='client.? 172.21.15.176:0/2442732458' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:54.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:54 smithi077 ceph-mon[131204]: from='client.? 172.21.15.176:0/2442732458' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:55 smithi044 ceph-mon[128750]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:43:55.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:55 smithi176 ceph-mon[131630]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:43:55.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:55 smithi077 ceph-mon[131204]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:43:57.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:57 smithi044 ceph-mon[128750]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:43:57.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:57 smithi176 ceph-mon[131630]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:43:57.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:57 smithi077 ceph-mon[131204]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:43:59.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:43:59 smithi044 ceph-mon[128750]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:43:59.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:43:59 smithi176 ceph-mon[131630]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:43:59.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:43:59 smithi077 ceph-mon[131204]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:00.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:00 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-16T16:44:00.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:00 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:00.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:00 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-16T16:44:00.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:00 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:00.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:00 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-16T16:44:00.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:00 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:01.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:01 smithi176 ceph-mon[131630]: Deploying daemon osd.5 on smithi176 2023-11-16T16:44:01.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:01 smithi176 ceph-mon[131630]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:01.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:01 smithi044 ceph-mon[128750]: Deploying daemon osd.5 on smithi176 2023-11-16T16:44:01.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:01 smithi044 ceph-mon[128750]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:01.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:01 smithi077 ceph-mon[131204]: Deploying daemon osd.5 on smithi176 2023-11-16T16:44:01.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:01 smithi077 ceph-mon[131204]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:03.858 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:03 smithi176 ceph-mon[131630]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:03 smithi044 ceph-mon[128750]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:03.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:03 smithi077 ceph-mon[131204]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:05.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:05 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:05.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:05 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:05.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:05 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:05.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:05 smithi044 ceph-mon[128750]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:05.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:05 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:05.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:05 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:05.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:05 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:05.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:05 smithi176 ceph-mon[131630]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:05.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:05 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:05.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:05 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:05.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:05 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:05.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:05 smithi077 ceph-mon[131204]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:06.973 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:44:06.972+0000 7f770b7fe700 1 -- 172.21.15.176:0/4161913973 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f771c1a0dd0 con 0x7f77040a1150 2023-11-16T16:44:06.973 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 5 on host 'smithi176' 2023-11-16T16:44:06.975 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:44:06.974+0000 7f7720aa4700 1 -- 172.21.15.176:0/4161913973 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f77040a1150 msgr2=0x7f77040a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:06.975 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:44:06.974+0000 7f7720aa4700 1 --2- 172.21.15.176:0/4161913973 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f77040a1150 0x7f77040a3600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f770c005950 tx=0x7f770c0058e0 comp rx=0 tx=0).stop 2023-11-16T16:44:06.976 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:44:06.974+0000 7f7720aa4700 1 -- 172.21.15.176:0/4161913973 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c100d70 msgr2=0x7f771c101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:06.976 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:44:06.974+0000 7f7720aa4700 1 --2- 172.21.15.176:0/4161913973 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c100d70 0x7f771c101d00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f771400b970 tx=0x7f771400bd30 comp rx=0 tx=0).stop 2023-11-16T16:44:06.976 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:44:06.974+0000 7f7720aa4700 1 -- 172.21.15.176:0/4161913973 shutdown_connections 2023-11-16T16:44:06.976 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:44:06.974+0000 7f7720aa4700 1 --2- 172.21.15.176:0/4161913973 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f77040a1150 0x7f77040a3600 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:06.976 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:44:06.974+0000 7f7720aa4700 1 --2- 172.21.15.176:0/4161913973 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f771c100d70 0x7f771c101d00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:06.976 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:44:06.974+0000 7f7720aa4700 1 --2- 172.21.15.176:0/4161913973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f771c0ff1e0 0x7f771c103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:06.976 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:44:06.974+0000 7f7720aa4700 1 --2- 172.21.15.176:0/4161913973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f771c1003e0 0x7f771c1017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:06.976 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:44:06.974+0000 7f7720aa4700 1 -- 172.21.15.176:0/4161913973 >> 172.21.15.176:0/4161913973 conn(0x7f771c0fa7f0 msgr2=0x7f771c068ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:44:06.976 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:44:06.974+0000 7f7720aa4700 1 -- 172.21.15.176:0/4161913973 shutdown_connections 2023-11-16T16:44:06.976 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:44:06.974+0000 7f7720aa4700 1 -- 172.21.15.176:0/4161913973 wait complete. 2023-11-16T16:44:07.377 DEBUG:teuthology.orchestra.run.smithi176:osd.5> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.5.service 2023-11-16T16:44:07.380 INFO:tasks.cephadm:Deploying osd.6 on smithi044 with /dev/vg_nvme/lv_2... 2023-11-16T16:44:07.380 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-11-16T16:44:07.411 INFO:journalctl@ceph.osd.5.smithi176.stdout:-- Logs begin at Thu 2023-11-16 16:14:40 UTC. -- 2023-11-16T16:44:07.451 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:07.451 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:07.451 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:07.451 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:07.451 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:07 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:44:07.451 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:07.451 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:07 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:44:07.452 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:07 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:44:07.452 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:07 smithi044 ceph-mon[128750]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:07.452 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:07.452 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:07.452 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:07.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:07.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:07 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:44:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:07 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:44:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:07 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:44:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:07 smithi077 ceph-mon[131204]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:07.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:07.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:07.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:07.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:07.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:07.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:07 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:44:07.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:07.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:07 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:44:07.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:07 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:44:07.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:07 smithi176 ceph-mon[131630]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:07.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:07.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:07.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:08.133 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 16:44:08 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T16:44:08.089+0000 7f21014c4640 -1 osd.5 0 log_to_monitors true 2023-11-16T16:44:08.251 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:44:09.089 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:09 smithi044 ceph-mon[128750]: from='osd.5 [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T16:44:09.090 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:09 smithi044 ceph-mon[128750]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T16:44:09.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:09 smithi044 ceph-mon[128750]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:09.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:09 smithi176 ceph-mon[131630]: from='osd.5 [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T16:44:09.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:09 smithi176 ceph-mon[131630]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T16:44:09.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:09 smithi176 ceph-mon[131630]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:09.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:09 smithi077 ceph-mon[131204]: from='osd.5 [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T16:44:09.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:09 smithi077 ceph-mon[131204]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T16:44:09.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:09 smithi077 ceph-mon[131204]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:10.221 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-11-16T16:44:10.222 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-11-16T16:44:10.222 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2023-11-16T16:44:10.222 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2023-11-16T16:44:10.222 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0310408 s, 338 MB/s 2023-11-16T16:44:10.222 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2023-11-16T16:44:10.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-16T16:44:10.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: osdmap e36: 6 total, 5 up, 6 in 2023-11-16T16:44:10.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:10.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: from='osd.5 [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:44:10.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:44:10.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: Detected new or changed devices on smithi176 2023-11-16T16:44:10.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:10.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:10.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:10.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:10.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi176 to 8584M 2023-11-16T16:44:10.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:10.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:10.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:10.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:44:10.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:10.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:44:10.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:10 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:44:10.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T16:44:10.101+0000 7f20f6b3b700 -1 osd.5 0 waiting for initial osdmap 2023-11-16T16:44:10.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T16:44:10.108+0000 7f20ee928700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:44:10.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-16T16:44:10.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: osdmap e36: 6 total, 5 up, 6 in 2023-11-16T16:44:10.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:10.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: from='osd.5 [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:44:10.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:44:10.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: Detected new or changed devices on smithi176 2023-11-16T16:44:10.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:10.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:10.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:10.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:10.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi176 to 8584M 2023-11-16T16:44:10.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:10.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:10.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:10.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:44:10.397 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:10.397 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:44:10.397 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:10 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:44:10.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-16T16:44:10.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: osdmap e36: 6 total, 5 up, 6 in 2023-11-16T16:44:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: from='osd.5 [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:44:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:44:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: Detected new or changed devices on smithi176 2023-11-16T16:44:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi176 to 8584M 2023-11-16T16:44:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:10.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:44:10.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:10.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:44:10.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:10 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:44:10.739 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi044:vg_nvme/lv_2 2023-11-16T16:44:11.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:11 smithi044 ceph-mon[128750]: purged_snaps scrub starts 2023-11-16T16:44:11.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:11 smithi044 ceph-mon[128750]: purged_snaps scrub ok 2023-11-16T16:44:11.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:11 smithi044 ceph-mon[128750]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2023-11-16T16:44:11.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:11 smithi044 ceph-mon[128750]: osdmap e37: 6 total, 5 up, 6 in 2023-11-16T16:44:11.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:11 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:11.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:11 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:11.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:11 smithi044 ceph-mon[128750]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:11.394 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:11 smithi176 ceph-mon[131630]: purged_snaps scrub starts 2023-11-16T16:44:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:11 smithi176 ceph-mon[131630]: purged_snaps scrub ok 2023-11-16T16:44:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:11 smithi176 ceph-mon[131630]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2023-11-16T16:44:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:11 smithi176 ceph-mon[131630]: osdmap e37: 6 total, 5 up, 6 in 2023-11-16T16:44:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:11 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:11 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:11 smithi176 ceph-mon[131630]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:11.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:11 smithi077 ceph-mon[131204]: purged_snaps scrub starts 2023-11-16T16:44:11.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:11 smithi077 ceph-mon[131204]: purged_snaps scrub ok 2023-11-16T16:44:11.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:11 smithi077 ceph-mon[131204]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2023-11-16T16:44:11.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:11 smithi077 ceph-mon[131204]: osdmap e37: 6 total, 5 up, 6 in 2023-11-16T16:44:11.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:11 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:11.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:11 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:11.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:11 smithi077 ceph-mon[131204]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:11.611 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:44:12.163 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:12 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:12.164 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:12 smithi044 ceph-mon[128750]: osd.5 [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683] boot 2023-11-16T16:44:12.164 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:12 smithi044 ceph-mon[128750]: osdmap e38: 6 total, 6 up, 6 in 2023-11-16T16:44:12.164 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:12 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:12.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:12 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:12.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:12 smithi176 ceph-mon[131630]: osd.5 [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683] boot 2023-11-16T16:44:12.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:12 smithi176 ceph-mon[131630]: osdmap e38: 6 total, 6 up, 6 in 2023-11-16T16:44:12.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:12 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:12.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:12 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:12.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:12 smithi077 ceph-mon[131204]: osd.5 [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683] boot 2023-11-16T16:44:12.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:12 smithi077 ceph-mon[131204]: osdmap e38: 6 total, 6 up, 6 in 2023-11-16T16:44:12.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:12 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:13.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.201+0000 7fef1fc8a700 1 -- 172.21.15.44:0/3516765183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef18073550 msgr2=0x7fef180739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:13.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.201+0000 7fef1fc8a700 1 --2- 172.21.15.44:0/3516765183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef18073550 0x7fef180739c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fef08009990 tx=0x7fef08009ca0 comp rx=0 tx=0).stop 2023-11-16T16:44:13.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.201+0000 7fef1fc8a700 1 -- 172.21.15.44:0/3516765183 shutdown_connections 2023-11-16T16:44:13.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.201+0000 7fef1fc8a700 1 --2- 172.21.15.44:0/3516765183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef181081c0 0x7fef1810a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:13.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.201+0000 7fef1fc8a700 1 --2- 172.21.15.44:0/3516765183 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef18074b20 0x7fef18072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:13.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.201+0000 7fef1fc8a700 1 --2- 172.21.15.44:0/3516765183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef18073550 0x7fef180739c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:13.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.201+0000 7fef1fc8a700 1 -- 172.21.15.44:0/3516765183 >> 172.21.15.44:0/3516765183 conn(0x7fef180fb900 msgr2=0x7fef180fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:44:13.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.202+0000 7fef1fc8a700 1 -- 172.21.15.44:0/3516765183 shutdown_connections 2023-11-16T16:44:13.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.202+0000 7fef1fc8a700 1 -- 172.21.15.44:0/3516765183 wait complete. 2023-11-16T16:44:13.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.203+0000 7fef1fc8a700 1 Processor -- start 2023-11-16T16:44:13.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.203+0000 7fef1fc8a700 1 -- start start 2023-11-16T16:44:13.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.203+0000 7fef1fc8a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef18073550 0x7fef18198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:44:13.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.203+0000 7fef1fc8a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef18074b20 0x7fef18198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:44:13.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.203+0000 7fef1fc8a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef181081c0 0x7fef1819f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:44:13.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.203+0000 7fef1fc8a700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1819fdc0 con 0x7fef18074b20 2023-11-16T16:44:13.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.203+0000 7fef1fc8a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1819ff00 con 0x7fef181081c0 2023-11-16T16:44:13.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.203+0000 7fef1fc8a700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef181a0090 con 0x7fef18073550 2023-11-16T16:44:13.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.204+0000 7fef1d225700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef18074b20 0x7fef18198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:44:13.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.204+0000 7fef1d225700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef18074b20 0x7fef18198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57312/0 (socket says 172.21.15.44:57312) 2023-11-16T16:44:13.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.204+0000 7fef1d225700 1 -- 172.21.15.44:0/3449965722 learned_addr learned my addr 172.21.15.44:0/3449965722 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:44:13.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.204+0000 7fef1e227700 1 --2- 172.21.15.44:0/3449965722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef181081c0 0x7fef1819f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:44:13.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.204+0000 7fef1da26700 1 --2- 172.21.15.44:0/3449965722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef18073550 0x7fef18198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:44:13.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.204+0000 7fef1d225700 1 -- 172.21.15.44:0/3449965722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef18073550 msgr2=0x7fef18198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:13.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.204+0000 7fef1d225700 1 --2- 172.21.15.44:0/3449965722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef18073550 0x7fef18198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:13.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.204+0000 7fef1d225700 1 -- 172.21.15.44:0/3449965722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef181081c0 msgr2=0x7fef1819f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:13.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.204+0000 7fef1d225700 1 --2- 172.21.15.44:0/3449965722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef181081c0 0x7fef1819f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:13.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.204+0000 7fef1d225700 1 -- 172.21.15.44:0/3449965722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef080095f0 con 0x7fef18074b20 2023-11-16T16:44:13.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.204+0000 7fef1e227700 1 --2- 172.21.15.44:0/3449965722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef181081c0 0x7fef1819f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:44:13.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.204+0000 7fef1d225700 1 --2- 172.21.15.44:0/3449965722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef18074b20 0x7fef18198770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fef08004770 tx=0x7fef08009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:44:13.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.205+0000 7fef0effd700 1 -- 172.21.15.44:0/3449965722 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0801d070 con 0x7fef18074b20 2023-11-16T16:44:13.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.205+0000 7fef0effd700 1 -- 172.21.15.44:0/3449965722 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef08005030 con 0x7fef18074b20 2023-11-16T16:44:13.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.205+0000 7fef0effd700 1 -- 172.21.15.44:0/3449965722 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef08004b50 con 0x7fef18074b20 2023-11-16T16:44:13.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.205+0000 7fef1fc8a700 1 -- 172.21.15.44:0/3449965722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef1810de90 con 0x7fef18074b20 2023-11-16T16:44:13.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.205+0000 7fef1fc8a700 1 -- 172.21.15.44:0/3449965722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1810dd00 con 0x7fef18074b20 2023-11-16T16:44:13.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.206+0000 7fef1fc8a700 1 -- 172.21.15.44:0/3449965722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef18106870 con 0x7fef18074b20 2023-11-16T16:44:13.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.208+0000 7fef0effd700 1 -- 172.21.15.44:0/3449965722 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fef08004cb0 con 0x7fef18074b20 2023-11-16T16:44:13.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.209+0000 7fef0effd700 1 --2- 172.21.15.44:0/3449965722 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fef040a11d0 0x7fef040a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:44:13.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.209+0000 7fef0effd700 1 -- 172.21.15.44:0/3449965722 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4562+0+0 (secure 0 0 0) 0x7fef080cd120 con 0x7fef18074b20 2023-11-16T16:44:13.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.210+0000 7fef1da26700 1 --2- 172.21.15.44:0/3449965722 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fef040a11d0 0x7fef040a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:44:13.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.210+0000 7fef1da26700 1 --2- 172.21.15.44:0/3449965722 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fef040a11d0 0x7fef040a3680 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fef14009c60 tx=0x7fef14009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:44:13.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.212+0000 7fef0effd700 1 -- 172.21.15.44:0/3449965722 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fef08097700 con 0x7fef18074b20 2023-11-16T16:44:13.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:13 smithi044 ceph-mon[128750]: osdmap e39: 6 total, 6 up, 6 in 2023-11-16T16:44:13.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:13 smithi044 ceph-mon[128750]: pgmap v134: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:13.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:13 smithi044 ceph-mon[128750]: osdmap e40: 6 total, 6 up, 6 in 2023-11-16T16:44:13.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:13 smithi176 ceph-mon[131630]: osdmap e39: 6 total, 6 up, 6 in 2023-11-16T16:44:13.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:13 smithi176 ceph-mon[131630]: pgmap v134: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:13.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:13 smithi176 ceph-mon[131630]: osdmap e40: 6 total, 6 up, 6 in 2023-11-16T16:44:13.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:13.429+0000 7fef1fc8a700 1 -- 172.21.15.44:0/3449965722 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fef18061c60 con 0x7fef040a11d0 2023-11-16T16:44:13.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:13 smithi077 ceph-mon[131204]: osdmap e39: 6 total, 6 up, 6 in 2023-11-16T16:44:13.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:13 smithi077 ceph-mon[131204]: pgmap v134: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:13.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:13 smithi077 ceph-mon[131204]: osdmap e40: 6 total, 6 up, 6 in 2023-11-16T16:44:14.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:14 smithi044 ceph-mon[128750]: from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:44:14.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:14 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:44:14.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:14 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:44:14.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:14 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:14.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:14 smithi176 ceph-mon[131630]: from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:44:14.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:14 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:44:14.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:14 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:44:14.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:14 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:14.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:14 smithi077 ceph-mon[131204]: from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:44:14.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:14 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:44:14.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:14 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:44:14.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:14 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:15.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:15 smithi044 ceph-mon[128750]: osdmap e41: 6 total, 6 up, 6 in 2023-11-16T16:44:15.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:15 smithi044 ceph-mon[128750]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:15.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:15 smithi176 ceph-mon[131630]: osdmap e41: 6 total, 6 up, 6 in 2023-11-16T16:44:15.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:15 smithi176 ceph-mon[131630]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:15.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:15 smithi077 ceph-mon[131204]: osdmap e41: 6 total, 6 up, 6 in 2023-11-16T16:44:15.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:15 smithi077 ceph-mon[131204]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:16.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:16 smithi176 ceph-mon[131630]: osdmap e42: 6 total, 6 up, 6 in 2023-11-16T16:44:16.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:16 smithi077 ceph-mon[131204]: osdmap e42: 6 total, 6 up, 6 in 2023-11-16T16:44:16.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:16 smithi044 ceph-mon[128750]: osdmap e42: 6 total, 6 up, 6 in 2023-11-16T16:44:17.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:17 smithi176 ceph-mon[131630]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:17.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:17 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/364362021' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a31ca92c-d241-4731-8e79-a2aa0fda2c42"}]: dispatch 2023-11-16T16:44:17.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:17 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/364362021' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a31ca92c-d241-4731-8e79-a2aa0fda2c42"}]': finished 2023-11-16T16:44:17.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:17 smithi176 ceph-mon[131630]: osdmap e43: 7 total, 6 up, 7 in 2023-11-16T16:44:17.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:17 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:17.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:17 smithi077 ceph-mon[131204]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:17.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:17 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/364362021' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a31ca92c-d241-4731-8e79-a2aa0fda2c42"}]: dispatch 2023-11-16T16:44:17.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:17 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/364362021' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a31ca92c-d241-4731-8e79-a2aa0fda2c42"}]': finished 2023-11-16T16:44:17.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:17 smithi077 ceph-mon[131204]: osdmap e43: 7 total, 6 up, 7 in 2023-11-16T16:44:17.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:17 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:17.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:17 smithi044 ceph-mon[128750]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:17.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:17 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/364362021' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a31ca92c-d241-4731-8e79-a2aa0fda2c42"}]: dispatch 2023-11-16T16:44:17.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:17 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/364362021' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a31ca92c-d241-4731-8e79-a2aa0fda2c42"}]': finished 2023-11-16T16:44:17.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:17 smithi044 ceph-mon[128750]: osdmap e43: 7 total, 6 up, 7 in 2023-11-16T16:44:17.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:18.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:18 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1048448962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:44:18.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:18 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1048448962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:44:18.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:18 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1048448962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:44:19.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:19 smithi176 ceph-mon[131630]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-11-16T16:44:19.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:19 smithi077 ceph-mon[131204]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-11-16T16:44:19.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:19 smithi044 ceph-mon[128750]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-11-16T16:44:21.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:21 smithi044 ceph-mon[128750]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-11-16T16:44:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:21 smithi176 ceph-mon[131630]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-11-16T16:44:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:21 smithi077 ceph-mon[131204]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-11-16T16:44:23.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:23 smithi044 ceph-mon[128750]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T16:44:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:23 smithi176 ceph-mon[131630]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T16:44:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:23 smithi077 ceph-mon[131204]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T16:44:24.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:24 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-16T16:44:24.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:24 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:24.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-16T16:44:24.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:24 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:24.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:24 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-16T16:44:24.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:24 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:25.732 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:25 smithi044 ceph-mon[128750]: Deploying daemon osd.6 on smithi044 2023-11-16T16:44:25.733 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:25 smithi044 ceph-mon[128750]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-11-16T16:44:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:25 smithi176 ceph-mon[131630]: Deploying daemon osd.6 on smithi044 2023-11-16T16:44:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:25 smithi176 ceph-mon[131630]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-11-16T16:44:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:25 smithi077 ceph-mon[131204]: Deploying daemon osd.6 on smithi044 2023-11-16T16:44:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:25 smithi077 ceph-mon[131204]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-11-16T16:44:27.560 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:27 smithi044 ceph-mon[128750]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-11-16T16:44:27.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:27 smithi176 ceph-mon[131630]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-11-16T16:44:27.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:27 smithi077 ceph-mon[131204]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-11-16T16:44:29.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:29 smithi044 ceph-mon[128750]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:29.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:29 smithi176 ceph-mon[131630]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:29.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:29 smithi077 ceph-mon[131204]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:31.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:31 smithi044 ceph-mon[128750]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:31.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:31 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:31.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:31 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:31.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:31 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:31.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:31 smithi077 ceph-mon[131204]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:31.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:31 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:31.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:31 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:31.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:31 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:32.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:31 smithi176 ceph-mon[131630]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:32.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:31 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:32.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:31 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:32.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:31 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:33.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:33 smithi077 ceph-mon[131204]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:34.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:33 smithi044 ceph-mon[128750]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:34.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:33 smithi176 ceph-mon[131630]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:34.648 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 6 on host 'smithi044' 2023-11-16T16:44:34.649 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:34.647+0000 7fef0effd700 1 -- 172.21.15.44:0/3449965722 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fef18061c60 con 0x7fef040a11d0 2023-11-16T16:44:34.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:34.650+0000 7fef1fc8a700 1 -- 172.21.15.44:0/3449965722 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fef040a11d0 msgr2=0x7fef040a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:34.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:34.650+0000 7fef1fc8a700 1 --2- 172.21.15.44:0/3449965722 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fef040a11d0 0x7fef040a3680 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fef14009c60 tx=0x7fef14009260 comp rx=0 tx=0).stop 2023-11-16T16:44:34.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:34.650+0000 7fef1fc8a700 1 -- 172.21.15.44:0/3449965722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef18074b20 msgr2=0x7fef18198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:34.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:34.650+0000 7fef1fc8a700 1 --2- 172.21.15.44:0/3449965722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef18074b20 0x7fef18198770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fef08004770 tx=0x7fef08009580 comp rx=0 tx=0).stop 2023-11-16T16:44:34.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:34.650+0000 7fef1fc8a700 1 -- 172.21.15.44:0/3449965722 shutdown_connections 2023-11-16T16:44:34.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:34.650+0000 7fef1fc8a700 1 --2- 172.21.15.44:0/3449965722 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fef040a11d0 0x7fef040a3680 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:34.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:34.650+0000 7fef1fc8a700 1 --2- 172.21.15.44:0/3449965722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef181081c0 0x7fef1819f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:34.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:34.650+0000 7fef1fc8a700 1 --2- 172.21.15.44:0/3449965722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef18073550 0x7fef18198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:34.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:34.650+0000 7fef1fc8a700 1 --2- 172.21.15.44:0/3449965722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef18074b20 0x7fef18198770 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fef08004770 tx=0x7fef08009580 comp rx=0 tx=0).stop 2023-11-16T16:44:34.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:34.650+0000 7fef1fc8a700 1 -- 172.21.15.44:0/3449965722 >> 172.21.15.44:0/3449965722 conn(0x7fef180fb900 msgr2=0x7fef180fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:44:34.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:34.651+0000 7fef1fc8a700 1 -- 172.21.15.44:0/3449965722 shutdown_connections 2023-11-16T16:44:34.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:44:34.651+0000 7fef1fc8a700 1 -- 172.21.15.44:0/3449965722 wait complete. 2023-11-16T16:44:35.104 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:35 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:35.104 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:35 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:35.104 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:35 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:35.104 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:35 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:35.104 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:35 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:44:35.105 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:35 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:35.105 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:35 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:44:35.105 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:35 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:44:35.105 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:35 smithi044 ceph-mon[128750]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:35.105 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:35 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:35.105 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:35 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:35.105 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:35 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:35.105 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:35 smithi044 ceph-mon[128750]: from='osd.6 [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T16:44:35.105 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:35 smithi044 ceph-mon[128750]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T16:44:35.153 DEBUG:teuthology.orchestra.run.smithi044:osd.6> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.6.service 2023-11-16T16:44:35.157 INFO:tasks.cephadm:Deploying osd.7 on smithi077 with /dev/vg_nvme/lv_2... 2023-11-16T16:44:35.157 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-11-16T16:44:35.229 INFO:journalctl@ceph.osd.6.smithi044.stdout:-- Logs begin at Thu 2023-11-16 16:19:45 UTC. -- 2023-11-16T16:44:35.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:35 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:35.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:35 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:35.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:35 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:35.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:35 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:35.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:35 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:44:35.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:35 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:35.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:35 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:44:35.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:35 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:44:35.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:35 smithi176 ceph-mon[131630]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:35.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:35 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:35.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:35 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:35.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:35 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:35.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:35 smithi176 ceph-mon[131630]: from='osd.6 [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T16:44:35.396 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:35 smithi176 ceph-mon[131630]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T16:44:35.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:35 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:35.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:35 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:35.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:35 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:35.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:35 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:35.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:35 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:44:35.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:35 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:35.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:35 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:44:35.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:35 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:44:35.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:35 smithi077 ceph-mon[131204]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:35.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:35 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:35.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:35 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:35.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:35 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:35.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:35 smithi077 ceph-mon[131204]: from='osd.6 [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T16:44:35.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:35 smithi077 ceph-mon[131204]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T16:44:35.804 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T16:44:36.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:36 smithi044 ceph-mon[128750]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-16T16:44:36.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:36 smithi044 ceph-mon[128750]: osdmap e44: 7 total, 6 up, 7 in 2023-11-16T16:44:36.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:36 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:36.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:36 smithi044 ceph-mon[128750]: from='osd.6 [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2023-11-16T16:44:36.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:36 smithi044 ceph-mon[128750]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2023-11-16T16:44:36.869 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 16:44:36 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T16:44:36.634+0000 7fbd9a081700 -1 osd.6 0 waiting for initial osdmap 2023-11-16T16:44:36.869 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 16:44:36 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T16:44:36.653+0000 7fbd95e76700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:44:36.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:36 smithi176 ceph-mon[131630]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-16T16:44:36.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:36 smithi176 ceph-mon[131630]: osdmap e44: 7 total, 6 up, 7 in 2023-11-16T16:44:36.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:36 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:36.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:36 smithi176 ceph-mon[131630]: from='osd.6 [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2023-11-16T16:44:36.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:36 smithi176 ceph-mon[131630]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2023-11-16T16:44:36.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:36 smithi077 ceph-mon[131204]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-16T16:44:36.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:36 smithi077 ceph-mon[131204]: osdmap e44: 7 total, 6 up, 7 in 2023-11-16T16:44:36.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:36 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:36.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:36 smithi077 ceph-mon[131204]: from='osd.6 [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2023-11-16T16:44:36.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:36 smithi077 ceph-mon[131204]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2023-11-16T16:44:37.259 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-11-16T16:44:37.259 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-11-16T16:44:37.259 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2023-11-16T16:44:37.259 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2023-11-16T16:44:37.260 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111059 s, 944 MB/s 2023-11-16T16:44:37.260 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2023-11-16T16:44:37.659 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi077:vg_nvme/lv_2 2023-11-16T16:44:37.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:37 smithi176 ceph-mon[131630]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:37.921 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:37 smithi176 ceph-mon[131630]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2023-11-16T16:44:37.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:37 smithi176 ceph-mon[131630]: osdmap e45: 7 total, 6 up, 7 in 2023-11-16T16:44:37.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:37.922 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:37.948 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:37 smithi044 ceph-mon[128750]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:37.948 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:37 smithi044 ceph-mon[128750]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2023-11-16T16:44:37.948 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:37 smithi044 ceph-mon[128750]: osdmap e45: 7 total, 6 up, 7 in 2023-11-16T16:44:37.948 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:37.948 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:37.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:37 smithi077 ceph-mon[131204]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:37.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:37 smithi077 ceph-mon[131204]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2023-11-16T16:44:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:37 smithi077 ceph-mon[131204]: osdmap e45: 7 total, 6 up, 7 in 2023-11-16T16:44:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:37 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:37 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:38.299 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T16:44:38.808 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:38 smithi077 ceph-mon[131204]: purged_snaps scrub starts 2023-11-16T16:44:38.808 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:38 smithi077 ceph-mon[131204]: purged_snaps scrub ok 2023-11-16T16:44:38.808 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:38 smithi077 ceph-mon[131204]: osd.6 [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849] boot 2023-11-16T16:44:38.809 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:38 smithi077 ceph-mon[131204]: osdmap e46: 7 total, 7 up, 7 in 2023-11-16T16:44:38.809 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:38 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:38.809 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:38 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:38.809 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:38 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:38.809 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:38 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:38.809 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:38 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:38.809 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:38 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:38.809 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:38 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:38.810 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:38 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:38.810 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:38 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:38.810 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:38 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:38.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:38 smithi176 ceph-mon[131630]: purged_snaps scrub starts 2023-11-16T16:44:38.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:38 smithi176 ceph-mon[131630]: purged_snaps scrub ok 2023-11-16T16:44:38.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:38 smithi176 ceph-mon[131630]: osd.6 [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849] boot 2023-11-16T16:44:38.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:38 smithi176 ceph-mon[131630]: osdmap e46: 7 total, 7 up, 7 in 2023-11-16T16:44:38.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:38 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:38.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:38 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:38.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:38 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:38.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:38 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:38.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:38 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:38.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:38 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:38.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:38 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:38.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:38 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:38.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:38 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:38.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:38 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:39.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:38 smithi044 ceph-mon[128750]: purged_snaps scrub starts 2023-11-16T16:44:39.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:38 smithi044 ceph-mon[128750]: purged_snaps scrub ok 2023-11-16T16:44:39.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:38 smithi044 ceph-mon[128750]: osd.6 [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849] boot 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:38 smithi044 ceph-mon[128750]: osdmap e46: 7 total, 7 up, 7 in 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:38 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:38 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:38 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:38 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:38 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:38 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:38 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:38 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:38 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:38 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:39.464 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.460+0000 7f7efce8f700 1 -- 172.21.15.77:0/3894464910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ef8100d70 msgr2=0x7f7ef8107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:39.464 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.460+0000 7f7efce8f700 1 --2- 172.21.15.77:0/3894464910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ef8100d70 0x7f7ef8107630 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000b0a0 tx=0x7f7ef000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:44:39.464 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.461+0000 7f7efce8f700 1 -- 172.21.15.77:0/3894464910 shutdown_connections 2023-11-16T16:44:39.464 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.461+0000 7f7efce8f700 1 --2- 172.21.15.77:0/3894464910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ef8100d70 0x7f7ef8107630 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:39.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.461+0000 7f7efce8f700 1 --2- 172.21.15.77:0/3894464910 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ef80ff1e0 0x7f7ef80ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:39.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.461+0000 7f7efce8f700 1 --2- 172.21.15.77:0/3894464910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ef81003e0 0x7f7ef8100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:39.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.461+0000 7f7efce8f700 1 -- 172.21.15.77:0/3894464910 >> 172.21.15.77:0/3894464910 conn(0x7f7ef80fa7f0 msgr2=0x7f7ef80fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:44:39.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.461+0000 7f7efce8f700 1 -- 172.21.15.77:0/3894464910 shutdown_connections 2023-11-16T16:44:39.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.461+0000 7f7efce8f700 1 -- 172.21.15.77:0/3894464910 wait complete. 2023-11-16T16:44:39.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.461+0000 7f7efce8f700 1 Processor -- start 2023-11-16T16:44:39.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.461+0000 7f7efce8f700 1 -- start start 2023-11-16T16:44:39.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.461+0000 7f7efce8f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ef80ff1e0 0x7f7ef8069110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:44:39.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7efce8f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ef81003e0 0x7f7ef819af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:44:39.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7efce8f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ef8100d70 0x7f7ef819d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:44:39.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7efce8f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef819dcf0 con 0x7f7ef8100d70 2023-11-16T16:44:39.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7efce8f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef819de60 con 0x7f7ef81003e0 2023-11-16T16:44:39.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7efce8f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef819e020 con 0x7f7ef80ff1e0 2023-11-16T16:44:39.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7ef5d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ef81003e0 0x7f7ef819af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:44:39.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7ef5d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ef81003e0 0x7f7ef819af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50984/0 (socket says 172.21.15.77:50984) 2023-11-16T16:44:39.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7ef5d9b700 1 -- 172.21.15.77:0/1505451597 learned_addr learned my addr 172.21.15.77:0/1505451597 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-11-16T16:44:39.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7ef659c700 1 --2- 172.21.15.77:0/1505451597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ef80ff1e0 0x7f7ef8069110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:44:39.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7ef6d9d700 1 --2- 172.21.15.77:0/1505451597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ef8100d70 0x7f7ef819d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:44:39.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7ef5d9b700 1 -- 172.21.15.77:0/1505451597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ef80ff1e0 msgr2=0x7f7ef8069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:39.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7ef5d9b700 1 --2- 172.21.15.77:0/1505451597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ef80ff1e0 0x7f7ef8069110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:39.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7ef5d9b700 1 -- 172.21.15.77:0/1505451597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ef8100d70 msgr2=0x7f7ef819d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:39.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7ef5d9b700 1 --2- 172.21.15.77:0/1505451597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ef8100d70 0x7f7ef819d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:39.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7ef5d9b700 1 -- 172.21.15.77:0/1505451597 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef0009d00 con 0x7f7ef81003e0 2023-11-16T16:44:39.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7ef5d9b700 1 --2- 172.21.15.77:0/1505451597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ef81003e0 0x7f7ef819af10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800b530 tx=0x7f7ee800b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:44:39.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7ee77fe700 1 -- 172.21.15.77:0/1505451597 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8011830 con 0x7f7ef81003e0 2023-11-16T16:44:39.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7efce8f700 1 -- 172.21.15.77:0/1505451597 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef810abb0 con 0x7f7ef81003e0 2023-11-16T16:44:39.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7ee77fe700 1 -- 172.21.15.77:0/1505451597 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ee8011e50 con 0x7f7ef81003e0 2023-11-16T16:44:39.468 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7ee77fe700 1 -- 172.21.15.77:0/1505451597 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee800f2e0 con 0x7f7ef81003e0 2023-11-16T16:44:39.468 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.462+0000 7f7efce8f700 1 -- 172.21.15.77:0/1505451597 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef810aa20 con 0x7f7ef81003e0 2023-11-16T16:44:39.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.463+0000 7f7efce8f700 1 -- 172.21.15.77:0/1505451597 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef819e7e0 con 0x7f7ef81003e0 2023-11-16T16:44:39.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.464+0000 7f7ee77fe700 1 -- 172.21.15.77:0/1505451597 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7ee8010380 con 0x7f7ef81003e0 2023-11-16T16:44:39.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.465+0000 7f7ee77fe700 1 --2- 172.21.15.77:0/1505451597 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7ee00a1150 0x7f7ee00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:44:39.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.465+0000 7f7ee77fe700 1 -- 172.21.15.77:0/1505451597 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f7ee80cb790 con 0x7f7ef81003e0 2023-11-16T16:44:39.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.465+0000 7f7ef659c700 1 --2- 172.21.15.77:0/1505451597 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7ee00a1150 0x7f7ee00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:44:39.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.466+0000 7f7ef659c700 1 --2- 172.21.15.77:0/1505451597 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7ee00a1150 0x7f7ee00a3600 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8100240 tx=0x7f7eec00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:44:39.470 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.469+0000 7f7ee77fe700 1 -- 172.21.15.77:0/1505451597 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7ee8095bf0 con 0x7f7ef81003e0 2023-11-16T16:44:39.665 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:39.664+0000 7f7efce8f700 1 -- 172.21.15.77:0/1505451597 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ef819ed90 con 0x7f7ee00a1150 2023-11-16T16:44:39.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:39 smithi077 ceph-mon[131204]: Detected new or changed devices on smithi044 2023-11-16T16:44:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:39 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi044 to 5758M 2023-11-16T16:44:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:39 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:44:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:39 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:44:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:39 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:44:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:39 smithi077 ceph-mon[131204]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:39 smithi077 ceph-mon[131204]: osdmap e47: 7 total, 7 up, 7 in 2023-11-16T16:44:40.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:39 smithi044 ceph-mon[128750]: Detected new or changed devices on smithi044 2023-11-16T16:44:40.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:39 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi044 to 5758M 2023-11-16T16:44:40.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:39 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:44:40.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:39 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:44:40.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:39 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:44:40.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:39 smithi044 ceph-mon[128750]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:40.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:39 smithi044 ceph-mon[128750]: osdmap e47: 7 total, 7 up, 7 in 2023-11-16T16:44:40.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:39 smithi176 ceph-mon[131630]: Detected new or changed devices on smithi044 2023-11-16T16:44:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:39 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi044 to 5758M 2023-11-16T16:44:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:39 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:44:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:39 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:44:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:39 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:44:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:39 smithi176 ceph-mon[131630]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:39 smithi176 ceph-mon[131630]: osdmap e47: 7 total, 7 up, 7 in 2023-11-16T16:44:40.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:40 smithi077 ceph-mon[131204]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:44:40.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:40 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:44:40.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:40 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:44:40.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:40 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:41.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:40 smithi044 ceph-mon[128750]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:44:41.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:40 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:44:41.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:40 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:44:41.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:40 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:41.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:40 smithi176 ceph-mon[131630]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:44:41.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:40 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:44:41.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:40 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:44:41.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:40 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:41 smithi077 ceph-mon[131204]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:42.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:41 smithi044 ceph-mon[128750]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:41 smithi176 ceph-mon[131630]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:42.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:42 smithi077 ceph-mon[131204]: from='client.? 172.21.15.77:0/1792320613' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969"}]: dispatch 2023-11-16T16:44:42.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:42 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969"}]: dispatch 2023-11-16T16:44:42.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:42 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969"}]': finished 2023-11-16T16:44:42.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:42 smithi077 ceph-mon[131204]: osdmap e48: 8 total, 7 up, 8 in 2023-11-16T16:44:42.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:42 smithi044 ceph-mon[128750]: from='client.? 172.21.15.77:0/1792320613' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969"}]: dispatch 2023-11-16T16:44:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:42 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969"}]: dispatch 2023-11-16T16:44:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:42 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969"}]': finished 2023-11-16T16:44:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:42 smithi044 ceph-mon[128750]: osdmap e48: 8 total, 7 up, 8 in 2023-11-16T16:44:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:43.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:42 smithi176 ceph-mon[131630]: from='client.? 172.21.15.77:0/1792320613' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969"}]: dispatch 2023-11-16T16:44:43.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:42 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969"}]: dispatch 2023-11-16T16:44:43.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:42 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969"}]': finished 2023-11-16T16:44:43.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:42 smithi176 ceph-mon[131630]: osdmap e48: 8 total, 7 up, 8 in 2023-11-16T16:44:43.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:43 smithi077 ceph-mon[131204]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:43 smithi077 ceph-mon[131204]: from='client.? 172.21.15.77:0/3635628440' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:44:44.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:43 smithi044 ceph-mon[128750]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:44.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:43 smithi044 ceph-mon[128750]: from='client.? 172.21.15.77:0/3635628440' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:44:44.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:43 smithi176 ceph-mon[131630]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:44.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:43 smithi176 ceph-mon[131630]: from='client.? 172.21.15.77:0/3635628440' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:44:45.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:45 smithi077 ceph-mon[131204]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:46.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:45 smithi044 ceph-mon[128750]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:46.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:45 smithi176 ceph-mon[131630]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:47.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:47 smithi077 ceph-mon[131204]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:48.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:47 smithi044 ceph-mon[128750]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:47 smithi176 ceph-mon[131630]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:49.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-16T16:44:49.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:49 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:49.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:49 smithi044 ceph-mon[128750]: Deploying daemon osd.7 on smithi077 2023-11-16T16:44:49.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:49 smithi044 ceph-mon[128750]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:49.390 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-16T16:44:49.390 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:49 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:49.390 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:49 smithi077 ceph-mon[131204]: Deploying daemon osd.7 on smithi077 2023-11-16T16:44:49.390 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:49 smithi077 ceph-mon[131204]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:49.394 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-16T16:44:49.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:49 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:49.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:49 smithi176 ceph-mon[131630]: Deploying daemon osd.7 on smithi077 2023-11-16T16:44:49.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:49 smithi176 ceph-mon[131630]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:51.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:51 smithi077 ceph-mon[131204]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:51.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:51 smithi044 ceph-mon[128750]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:51.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:51 smithi176 ceph-mon[131630]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:53.845 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:53 smithi077 ceph-mon[131204]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:53.846 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:53.846 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:53.846 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:53.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:53 smithi044 ceph-mon[128750]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:53.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:53.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:53.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:53.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:53 smithi176 ceph-mon[131630]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:53.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:53.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:53.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:55.825 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:55 smithi077 ceph-mon[131204]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:55.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:55 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:55.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:55 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:55.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:55 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:55.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:55 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:55.826 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:55 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:55 smithi044 ceph-mon[128750]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:55.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:55.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:55.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:55 smithi176 ceph-mon[131630]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:55.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:55 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:55.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:55 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:55.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:55 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:55.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:55 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:55.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:55 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:55.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:55.927+0000 7f7ee77fe700 1 -- 172.21.15.77:0/1505451597 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7ef819ed90 con 0x7f7ee00a1150 2023-11-16T16:44:55.930 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 7 on host 'smithi077' 2023-11-16T16:44:55.932 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:55.930+0000 7f7efce8f700 1 -- 172.21.15.77:0/1505451597 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7ee00a1150 msgr2=0x7f7ee00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:55.932 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:55.930+0000 7f7efce8f700 1 --2- 172.21.15.77:0/1505451597 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7ee00a1150 0x7f7ee00a3600 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8100240 tx=0x7f7eec00b410 comp rx=0 tx=0).stop 2023-11-16T16:44:55.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:55.930+0000 7f7efce8f700 1 -- 172.21.15.77:0/1505451597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ef81003e0 msgr2=0x7f7ef819af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:55.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:55.930+0000 7f7efce8f700 1 --2- 172.21.15.77:0/1505451597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ef81003e0 0x7f7ef819af10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800b530 tx=0x7f7ee800b840 comp rx=0 tx=0).stop 2023-11-16T16:44:55.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:55.930+0000 7f7efce8f700 1 -- 172.21.15.77:0/1505451597 shutdown_connections 2023-11-16T16:44:55.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:55.930+0000 7f7efce8f700 1 --2- 172.21.15.77:0/1505451597 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7ee00a1150 0x7f7ee00a3600 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:55.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:55.930+0000 7f7efce8f700 1 --2- 172.21.15.77:0/1505451597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ef8100d70 0x7f7ef819d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:55.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:55.930+0000 7f7efce8f700 1 --2- 172.21.15.77:0/1505451597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ef80ff1e0 0x7f7ef8069110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:55.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:55.930+0000 7f7efce8f700 1 --2- 172.21.15.77:0/1505451597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ef81003e0 0x7f7ef819af10 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:55.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:55.930+0000 7f7efce8f700 1 -- 172.21.15.77:0/1505451597 >> 172.21.15.77:0/1505451597 conn(0x7f7ef80fa7f0 msgr2=0x7f7ef80fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:44:55.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:55.930+0000 7f7efce8f700 1 -- 172.21.15.77:0/1505451597 shutdown_connections 2023-11-16T16:44:55.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:44:55.930+0000 7f7efce8f700 1 -- 172.21.15.77:0/1505451597 wait complete. 2023-11-16T16:44:56.380 DEBUG:teuthology.orchestra.run.smithi077:osd.7> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.7.service 2023-11-16T16:44:56.382 INFO:tasks.cephadm:Deploying osd.8 on smithi176 with /dev/vg_nvme/lv_2... 2023-11-16T16:44:56.383 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-11-16T16:44:56.413 INFO:journalctl@ceph.osd.7.smithi077.stdout:-- Logs begin at Thu 2023-11-16 16:15:33 UTC. -- 2023-11-16T16:44:56.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:56 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:44:56.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:56 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:44:56.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:56 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:44:56.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:56 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:56.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:56 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:56.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:56 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:56.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:56 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:44:56.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:56 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:44:56.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:56 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:44:56.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:56 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:56.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:56 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:56.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:56 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:56.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:56 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:44:56.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:56 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:44:56.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:56 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:44:56.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:56 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:56.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:56 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:56.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:56 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:57.034 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:44:57.051 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 16:44:56 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T16:44:56.753+0000 7f15072df640 -1 osd.7 0 log_to_monitors true 2023-11-16T16:44:57.647 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:57 smithi077 ceph-mon[131204]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:57.647 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:57 smithi077 ceph-mon[131204]: from='osd.7 [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T16:44:57.647 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:57 smithi077 ceph-mon[131204]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T16:44:57.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:57 smithi044 ceph-mon[128750]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:57.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:57 smithi044 ceph-mon[128750]: from='osd.7 [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T16:44:57.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:57 smithi044 ceph-mon[128750]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T16:44:57.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:57 smithi176 ceph-mon[131630]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:57.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:57 smithi176 ceph-mon[131630]: from='osd.7 [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T16:44:57.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:57 smithi176 ceph-mon[131630]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T16:44:58.509 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-11-16T16:44:58.509 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-11-16T16:44:58.509 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2023-11-16T16:44:58.509 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2023-11-16T16:44:58.509 INFO:teuthology.orchestra.run.smithi176.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110559 s, 948 MB/s 2023-11-16T16:44:58.509 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 2023-11-16T16:44:58.509 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2023-11-16T16:44:58.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:58 smithi044 ceph-mon[128750]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-16T16:44:58.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:58 smithi044 ceph-mon[128750]: osdmap e49: 8 total, 7 up, 8 in 2023-11-16T16:44:58.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:58 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:58.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:58 smithi044 ceph-mon[128750]: from='osd.7 [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:44:58.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:58 smithi044 ceph-mon[128750]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:44:58.887 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:58 smithi176 ceph-mon[131630]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-16T16:44:58.887 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:58 smithi176 ceph-mon[131630]: osdmap e49: 8 total, 7 up, 8 in 2023-11-16T16:44:58.887 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:58 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:58.887 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:58 smithi176 ceph-mon[131630]: from='osd.7 [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:44:58.887 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:58 smithi176 ceph-mon[131630]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:44:58.888 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi176:vg_nvme/lv_2 2023-11-16T16:44:58.976 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 16:44:58 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T16:44:58.583+0000 7f14fc956700 -1 osd.7 0 waiting for initial osdmap 2023-11-16T16:44:58.976 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 16:44:58 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T16:44:58.595+0000 7f14f4743700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:44:58.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:58 smithi077 ceph-mon[131204]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-16T16:44:58.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:58 smithi077 ceph-mon[131204]: osdmap e49: 8 total, 7 up, 8 in 2023-11-16T16:44:58.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:58 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:58.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:58 smithi077 ceph-mon[131204]: from='osd.7 [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:44:58.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:58 smithi077 ceph-mon[131204]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:44:59.530 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:44:59.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:59.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2023-11-16T16:44:59.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: osdmap e50: 8 total, 7 up, 8 in 2023-11-16T16:44:59.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:59.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:59.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: Detected new or changed devices on smithi077 2023-11-16T16:44:59.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:59.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:59.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:59.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:59.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:59.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi077 to 5722M 2023-11-16T16:44:59.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:59.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:59.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:59.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:44:59.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:59.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:44:59.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:44:59.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: osd.7 [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313] boot 2023-11-16T16:44:59.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: osdmap e51: 8 total, 8 up, 8 in 2023-11-16T16:44:59.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:44:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2023-11-16T16:44:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: osdmap e50: 8 total, 7 up, 8 in 2023-11-16T16:44:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: Detected new or changed devices on smithi077 2023-11-16T16:44:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:59.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:59.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:59.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:59.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi077 to 5722M 2023-11-16T16:44:59.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:59.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:59.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:59.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:44:59.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:59.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:44:59.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:44:59.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: osd.7 [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313] boot 2023-11-16T16:44:59.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: osdmap e51: 8 total, 8 up, 8 in 2023-11-16T16:44:59.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:44:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2023-11-16T16:44:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: osdmap e50: 8 total, 7 up, 8 in 2023-11-16T16:44:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: Detected new or changed devices on smithi077 2023-11-16T16:44:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi077 to 5722M 2023-11-16T16:44:59.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:59.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:59.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:59.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:44:59.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:44:59.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:44:59.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:44:59.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: osd.7 [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313] boot 2023-11-16T16:44:59.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: osdmap e51: 8 total, 8 up, 8 in 2023-11-16T16:44:59.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:44:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:45:00.593 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.589+0000 7f54491f9700 1 -- 172.21.15.176:0/131807517 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54440ff1e0 msgr2=0x7f54440ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:00.593 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.589+0000 7f54491f9700 1 --2- 172.21.15.176:0/131807517 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54440ff1e0 0x7f54440ff5f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5438009990 tx=0x7f5438009ca0 comp rx=0 tx=0).stop 2023-11-16T16:45:00.593 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.589+0000 7f54491f9700 1 -- 172.21.15.176:0/131807517 shutdown_connections 2023-11-16T16:45:00.593 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.589+0000 7f54491f9700 1 --2- 172.21.15.176:0/131807517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5444100d70 0x7f5444107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:00.593 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.589+0000 7f54491f9700 1 --2- 172.21.15.176:0/131807517 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54440ff1e0 0x7f54440ff5f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:00.593 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.589+0000 7f54491f9700 1 --2- 172.21.15.176:0/131807517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54441003e0 0x7f5444100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:00.593 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.589+0000 7f54491f9700 1 -- 172.21.15.176:0/131807517 >> 172.21.15.176:0/131807517 conn(0x7f54440fa7f0 msgr2=0x7f54440fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:45:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.589+0000 7f54491f9700 1 -- 172.21.15.176:0/131807517 shutdown_connections 2023-11-16T16:45:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.589+0000 7f54491f9700 1 -- 172.21.15.176:0/131807517 wait complete. 2023-11-16T16:45:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.590+0000 7f54491f9700 1 Processor -- start 2023-11-16T16:45:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.590+0000 7f54491f9700 1 -- start start 2023-11-16T16:45:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.590+0000 7f54491f9700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54440ff1e0 0x7f5444194fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.590+0000 7f54491f9700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54441003e0 0x7f5444195500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.590+0000 7f54491f9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5444100d70 0x7f544419c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.590+0000 7f54491f9700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544419cb50 con 0x7f54440ff1e0 2023-11-16T16:45:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.590+0000 7f54491f9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544419cc90 con 0x7f5444100d70 2023-11-16T16:45:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.590+0000 7f54491f9700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544419ce20 con 0x7f54441003e0 2023-11-16T16:45:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.590+0000 7f544259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54441003e0 0x7f5444195500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.590+0000 7f544259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54441003e0 0x7f5444195500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37988/0 (socket says 172.21.15.176:37988) 2023-11-16T16:45:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.590+0000 7f544259c700 1 -- 172.21.15.176:0/1277553062 learned_addr learned my addr 172.21.15.176:0/1277553062 (peer_addr_for_me v2:172.21.15.176:0/0) 2023-11-16T16:45:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.591+0000 7f5442d9d700 1 --2- 172.21.15.176:0/1277553062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54440ff1e0 0x7f5444194fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.591+0000 7f544359e700 1 --2- 172.21.15.176:0/1277553062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5444100d70 0x7f544419c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.591+0000 7f544259c700 1 -- 172.21.15.176:0/1277553062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5444100d70 msgr2=0x7f544419c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.591+0000 7f544259c700 1 --2- 172.21.15.176:0/1277553062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5444100d70 0x7f544419c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.591+0000 7f544259c700 1 -- 172.21.15.176:0/1277553062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54440ff1e0 msgr2=0x7f5444194fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.591+0000 7f544259c700 1 --2- 172.21.15.176:0/1277553062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54440ff1e0 0x7f5444194fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.591+0000 7f544259c700 1 -- 172.21.15.176:0/1277553062 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54380095f0 con 0x7f54441003e0 2023-11-16T16:45:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.591+0000 7f544259c700 1 --2- 172.21.15.176:0/1277553062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54441003e0 0x7f5444195500 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f543400da10 tx=0x7f543400dd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:45:00.596 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.592+0000 7f542bfff700 1 -- 172.21.15.176:0/1277553062 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543400eb70 con 0x7f54441003e0 2023-11-16T16:45:00.596 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.592+0000 7f542bfff700 1 -- 172.21.15.176:0/1277553062 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f543400ecd0 con 0x7f54441003e0 2023-11-16T16:45:00.596 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.592+0000 7f54491f9700 1 -- 172.21.15.176:0/1277553062 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f544410ac30 con 0x7f54441003e0 2023-11-16T16:45:00.596 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.592+0000 7f542bfff700 1 -- 172.21.15.176:0/1277553062 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54340145a0 con 0x7f54441003e0 2023-11-16T16:45:00.596 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.592+0000 7f54491f9700 1 -- 172.21.15.176:0/1277553062 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f544410aaa0 con 0x7f54441003e0 2023-11-16T16:45:00.596 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.593+0000 7f54491f9700 1 -- 172.21.15.176:0/1277553062 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5444069380 con 0x7f54441003e0 2023-11-16T16:45:00.596 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.594+0000 7f542bfff700 1 -- 172.21.15.176:0/1277553062 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5434016440 con 0x7f54441003e0 2023-11-16T16:45:00.597 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.595+0000 7f542bfff700 1 --2- 172.21.15.176:0/1277553062 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f542c0a1250 0x7f542c0a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:00.597 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.595+0000 7f542bfff700 1 -- 172.21.15.176:0/1277553062 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f54340cd5c0 con 0x7f54441003e0 2023-11-16T16:45:00.597 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.595+0000 7f5442d9d700 1 --2- 172.21.15.176:0/1277553062 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f542c0a1250 0x7f542c0a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:00.597 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.596+0000 7f5442d9d700 1 --2- 172.21.15.176:0/1277553062 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f542c0a1250 0x7f542c0a3700 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f543800b5c0 tx=0x7f5438005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:45:00.601 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.600+0000 7f542bfff700 1 -- 172.21.15.176:0/1277553062 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5434050730 con 0x7f54441003e0 2023-11-16T16:45:00.793 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:00.791+0000 7f54491f9700 1 -- 172.21.15.176:0/1277553062 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5444060f30 con 0x7f542c0a1250 2023-11-16T16:45:00.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:00 smithi044 ceph-mon[128750]: purged_snaps scrub starts 2023-11-16T16:45:00.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:00 smithi044 ceph-mon[128750]: purged_snaps scrub ok 2023-11-16T16:45:00.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:00 smithi044 ceph-mon[128750]: osdmap e52: 8 total, 8 up, 8 in 2023-11-16T16:45:00.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:00 smithi176 ceph-mon[131630]: purged_snaps scrub starts 2023-11-16T16:45:00.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:00 smithi176 ceph-mon[131630]: purged_snaps scrub ok 2023-11-16T16:45:00.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:00 smithi176 ceph-mon[131630]: osdmap e52: 8 total, 8 up, 8 in 2023-11-16T16:45:00.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:00 smithi077 ceph-mon[131204]: purged_snaps scrub starts 2023-11-16T16:45:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:00 smithi077 ceph-mon[131204]: purged_snaps scrub ok 2023-11-16T16:45:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:00 smithi077 ceph-mon[131204]: osdmap e52: 8 total, 8 up, 8 in 2023-11-16T16:45:01.853 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:01 smithi176 ceph-mon[131630]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:01.853 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:01 smithi176 ceph-mon[131630]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:01.853 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:01 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:01.853 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:01 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:01.853 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:01 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:01.853 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:01 smithi176 ceph-mon[131630]: osdmap e53: 8 total, 8 up, 8 in 2023-11-16T16:45:01.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:01 smithi044 ceph-mon[128750]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:01.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:01 smithi044 ceph-mon[128750]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:01.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:01 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:01.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:01 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:01.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:01 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:01.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:01 smithi044 ceph-mon[128750]: osdmap e53: 8 total, 8 up, 8 in 2023-11-16T16:45:01.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:01 smithi077 ceph-mon[131204]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:01.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:01 smithi077 ceph-mon[131204]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:01.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:01 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:01.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:01 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:01.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:01 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:01.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:01 smithi077 ceph-mon[131204]: osdmap e53: 8 total, 8 up, 8 in 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:03 smithi044 ceph-mon[128750]: pgmap v173: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:03 smithi044 ceph-mon[128750]: osdmap e54: 8 total, 8 up, 8 in 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:03 smithi044 ceph-mon[128750]: from='client.? 172.21.15.176:0/957323281' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba086b54-d6b6-4563-ade8-59c99f2ca473"}]: dispatch 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:03 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba086b54-d6b6-4563-ade8-59c99f2ca473"}]: dispatch 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:03 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba086b54-d6b6-4563-ade8-59c99f2ca473"}]': finished 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:03 smithi044 ceph-mon[128750]: osdmap e55: 9 total, 8 up, 9 in 2023-11-16T16:45:03.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:03 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:03 smithi176 ceph-mon[131630]: pgmap v173: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:03 smithi176 ceph-mon[131630]: osdmap e54: 8 total, 8 up, 8 in 2023-11-16T16:45:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:03 smithi176 ceph-mon[131630]: from='client.? 172.21.15.176:0/957323281' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba086b54-d6b6-4563-ade8-59c99f2ca473"}]: dispatch 2023-11-16T16:45:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:03 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba086b54-d6b6-4563-ade8-59c99f2ca473"}]: dispatch 2023-11-16T16:45:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:03 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba086b54-d6b6-4563-ade8-59c99f2ca473"}]': finished 2023-11-16T16:45:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:03 smithi176 ceph-mon[131630]: osdmap e55: 9 total, 8 up, 9 in 2023-11-16T16:45:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:03 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:03.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:03 smithi077 ceph-mon[131204]: pgmap v173: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:03.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:03 smithi077 ceph-mon[131204]: osdmap e54: 8 total, 8 up, 8 in 2023-11-16T16:45:03.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:03 smithi077 ceph-mon[131204]: from='client.? 172.21.15.176:0/957323281' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba086b54-d6b6-4563-ade8-59c99f2ca473"}]: dispatch 2023-11-16T16:45:03.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:03 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba086b54-d6b6-4563-ade8-59c99f2ca473"}]: dispatch 2023-11-16T16:45:03.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:03 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba086b54-d6b6-4563-ade8-59c99f2ca473"}]': finished 2023-11-16T16:45:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:03 smithi077 ceph-mon[131204]: osdmap e55: 9 total, 8 up, 9 in 2023-11-16T16:45:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:03 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:04.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:04 smithi044 ceph-mon[128750]: from='client.? 172.21.15.176:0/2379864948' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:04.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:04 smithi176 ceph-mon[131630]: from='client.? 172.21.15.176:0/2379864948' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:04.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:04 smithi077 ceph-mon[131204]: from='client.? 172.21.15.176:0/2379864948' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:05.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:05 smithi044 ceph-mon[128750]: pgmap v176: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:05.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:05 smithi176 ceph-mon[131630]: pgmap v176: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:05.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:05 smithi077 ceph-mon[131204]: pgmap v176: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:07 smithi044 ceph-mon[128750]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:07.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:07 smithi176 ceph-mon[131630]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:07.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:07 smithi077 ceph-mon[131204]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:09.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:09 smithi044 ceph-mon[128750]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:09.371 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:09 smithi176 ceph-mon[131630]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:09.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:09 smithi077 ceph-mon[131204]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:10.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-11-16T16:45:10.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:10.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:10 smithi044 ceph-mon[128750]: Deploying daemon osd.8 on smithi176 2023-11-16T16:45:10.394 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-11-16T16:45:10.394 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:10.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:10 smithi176 ceph-mon[131630]: Deploying daemon osd.8 on smithi176 2023-11-16T16:45:10.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-11-16T16:45:10.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:10.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:10 smithi077 ceph-mon[131204]: Deploying daemon osd.8 on smithi176 2023-11-16T16:45:11.209 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:11 smithi176 ceph-mon[131630]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:11.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:11 smithi044 ceph-mon[128750]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:11.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:11 smithi077 ceph-mon[131204]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:13.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:13 smithi044 ceph-mon[128750]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:13.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:13 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:13.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:13 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:13.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:13 smithi176 ceph-mon[131630]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:13.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:13 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:13.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:13 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:13.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:13 smithi077 ceph-mon[131204]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:13.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:13 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:13.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:13 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:14.696 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:14 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:14.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:14 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:14.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:14 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:15.822 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:15 smithi176 ceph-mon[131630]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:15.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:15 smithi044 ceph-mon[128750]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:15.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:15 smithi077 ceph-mon[131204]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:16.156 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 8 on host 'smithi176' 2023-11-16T16:45:16.156 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:16.154+0000 7f542bfff700 1 -- 172.21.15.176:0/1277553062 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5444060f30 con 0x7f542c0a1250 2023-11-16T16:45:16.158 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:16.156+0000 7f54491f9700 1 -- 172.21.15.176:0/1277553062 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f542c0a1250 msgr2=0x7f542c0a3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:16.158 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:16.156+0000 7f54491f9700 1 --2- 172.21.15.176:0/1277553062 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f542c0a1250 0x7f542c0a3700 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f543800b5c0 tx=0x7f5438005fd0 comp rx=0 tx=0).stop 2023-11-16T16:45:16.158 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:16.156+0000 7f54491f9700 1 -- 172.21.15.176:0/1277553062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54441003e0 msgr2=0x7f5444195500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:16.158 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:16.156+0000 7f54491f9700 1 --2- 172.21.15.176:0/1277553062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54441003e0 0x7f5444195500 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f543400da10 tx=0x7f543400dd20 comp rx=0 tx=0).stop 2023-11-16T16:45:16.158 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:16.156+0000 7f54491f9700 1 -- 172.21.15.176:0/1277553062 shutdown_connections 2023-11-16T16:45:16.158 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:16.156+0000 7f54491f9700 1 --2- 172.21.15.176:0/1277553062 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f542c0a1250 0x7f542c0a3700 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:16.158 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:16.156+0000 7f54491f9700 1 --2- 172.21.15.176:0/1277553062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5444100d70 0x7f544419c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:16.158 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:16.156+0000 7f54491f9700 1 --2- 172.21.15.176:0/1277553062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54440ff1e0 0x7f5444194fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:16.158 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:16.156+0000 7f54491f9700 1 --2- 172.21.15.176:0/1277553062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54441003e0 0x7f5444195500 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:16.159 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:16.156+0000 7f54491f9700 1 -- 172.21.15.176:0/1277553062 >> 172.21.15.176:0/1277553062 conn(0x7f54440fa7f0 msgr2=0x7f54440fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:45:16.159 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:16.157+0000 7f54491f9700 1 -- 172.21.15.176:0/1277553062 shutdown_connections 2023-11-16T16:45:16.159 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:45:16.157+0000 7f54491f9700 1 -- 172.21.15.176:0/1277553062 wait complete. 2023-11-16T16:45:16.517 DEBUG:teuthology.orchestra.run.smithi176:osd.8> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.8.service 2023-11-16T16:45:16.519 INFO:tasks.cephadm:Deploying osd.9 on smithi044 with /dev/vg_nvme/lv_1... 2023-11-16T16:45:16.520 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-11-16T16:45:16.549 INFO:journalctl@ceph.osd.8.smithi176.stdout:-- Logs begin at Thu 2023-11-16 16:14:40 UTC. -- 2023-11-16T16:45:17.304 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:17 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:17.304 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:17 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:17.304 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:17 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:17.304 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:17 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:17.304 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:17 smithi176 ceph-mon[131630]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:17.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:17.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:17.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:17.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:17.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:17 smithi044 ceph-mon[128750]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:17.394 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:45:17.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:17 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:17.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:17 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:17.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:17 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:17.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:17 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:17.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:17 smithi077 ceph-mon[131204]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:17.645 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 16:45:17 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T16:45:17.496+0000 7f8e18d88640 -1 osd.8 0 log_to_monitors true 2023-11-16T16:45:18.328 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:18 smithi044 ceph-mon[128750]: from='osd.8 [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T16:45:18.328 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:18 smithi044 ceph-mon[128750]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T16:45:18.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:18 smithi077 ceph-mon[131204]: from='osd.8 [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T16:45:18.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:18 smithi077 ceph-mon[131204]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T16:45:18.586 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:18 smithi176 ceph-mon[131630]: from='osd.8 [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T16:45:18.586 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:18 smithi176 ceph-mon[131630]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T16:45:19.421 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-11-16T16:45:19.421 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-11-16T16:45:19.421 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2023-11-16T16:45:19.421 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2023-11-16T16:45:19.421 INFO:teuthology.orchestra.run.smithi044.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0306584 s, 342 MB/s 2023-11-16T16:45:19.422 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 2023-11-16T16:45:19.422 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2023-11-16T16:45:19.459 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-11-16T16:45:19.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: osdmap e56: 9 total, 8 up, 9 in 2023-11-16T16:45:19.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:19.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='osd.8 [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:45:19.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:45:19.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:19.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: Detected new or changed devices on smithi176 2023-11-16T16:45:19.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:19.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:19.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:19.461 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:19.461 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:19.461 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi176 to 5722M 2023-11-16T16:45:19.461 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:19.461 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:19.461 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:19.461 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:45:19.461 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:19.461 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:45:19.461 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:45:19.461 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:19.462 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:19.462 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:19.462 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:19.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-11-16T16:45:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: osdmap e56: 9 total, 8 up, 9 in 2023-11-16T16:45:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='osd.8 [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:45:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:45:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: Detected new or changed devices on smithi176 2023-11-16T16:45:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:19.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi176 to 5722M 2023-11-16T16:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:19.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:19.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:19.645 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T16:45:19.176+0000 7f8e0e3ff700 -1 osd.8 0 waiting for initial osdmap 2023-11-16T16:45:19.645 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T16:45:19.186+0000 7f8e091f2700 -1 osd.8 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:45:19.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-11-16T16:45:19.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: osdmap e56: 9 total, 8 up, 9 in 2023-11-16T16:45:19.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:19.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='osd.8 [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:45:19.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:45:19.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:19.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: Detected new or changed devices on smithi176 2023-11-16T16:45:19.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:19.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:19.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:19.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:19.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:19.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi176 to 5722M 2023-11-16T16:45:19.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:19.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:19.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:19.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:45:19.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:19.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:45:19.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:45:19.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:19.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:19.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:19.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:19.984 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi044:vg_nvme/lv_1 2023-11-16T16:45:20.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:20 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:45:20.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:20 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:45:20.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:20 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:45:20.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:20 smithi044 ceph-mon[128750]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2023-11-16T16:45:20.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:20 smithi044 ceph-mon[128750]: osdmap e57: 9 total, 8 up, 9 in 2023-11-16T16:45:20.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:20 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:20.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:20 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:20.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:20 smithi044 ceph-mon[128750]: osd.8 [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338] boot 2023-11-16T16:45:20.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:20 smithi044 ceph-mon[128750]: osdmap e58: 9 total, 9 up, 9 in 2023-11-16T16:45:20.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:20 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:20.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:20 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:45:20.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:20 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:45:20.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:20 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:45:20.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:20 smithi077 ceph-mon[131204]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2023-11-16T16:45:20.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:20 smithi077 ceph-mon[131204]: osdmap e57: 9 total, 8 up, 9 in 2023-11-16T16:45:20.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:20 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:20.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:20 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:20.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:20 smithi077 ceph-mon[131204]: osd.8 [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338] boot 2023-11-16T16:45:20.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:20 smithi077 ceph-mon[131204]: osdmap e58: 9 total, 9 up, 9 in 2023-11-16T16:45:20.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:20 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:20.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:20 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:45:20.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:20 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:45:20.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:20 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:45:20.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:20 smithi176 ceph-mon[131630]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2023-11-16T16:45:20.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:20 smithi176 ceph-mon[131630]: osdmap e57: 9 total, 8 up, 9 in 2023-11-16T16:45:20.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:20 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:20.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:20 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:20.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:20 smithi176 ceph-mon[131630]: osd.8 [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338] boot 2023-11-16T16:45:20.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:20 smithi176 ceph-mon[131630]: osdmap e58: 9 total, 9 up, 9 in 2023-11-16T16:45:20.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:20 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:20.849 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:45:21.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:21 smithi044 ceph-mon[128750]: purged_snaps scrub starts 2023-11-16T16:45:21.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:21 smithi044 ceph-mon[128750]: purged_snaps scrub ok 2023-11-16T16:45:21.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:21 smithi044 ceph-mon[128750]: pgmap v187: 1 pgs: 1 peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:21.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:21 smithi044 ceph-mon[128750]: osdmap e59: 9 total, 9 up, 9 in 2023-11-16T16:45:21.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:21 smithi077 ceph-mon[131204]: purged_snaps scrub starts 2023-11-16T16:45:21.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:21 smithi077 ceph-mon[131204]: purged_snaps scrub ok 2023-11-16T16:45:21.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:21 smithi077 ceph-mon[131204]: pgmap v187: 1 pgs: 1 peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:21.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:21 smithi077 ceph-mon[131204]: osdmap e59: 9 total, 9 up, 9 in 2023-11-16T16:45:21.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:21 smithi176 ceph-mon[131630]: purged_snaps scrub starts 2023-11-16T16:45:21.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:21 smithi176 ceph-mon[131630]: purged_snaps scrub ok 2023-11-16T16:45:21.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:21 smithi176 ceph-mon[131630]: pgmap v187: 1 pgs: 1 peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:21.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:21 smithi176 ceph-mon[131630]: osdmap e59: 9 total, 9 up, 9 in 2023-11-16T16:45:22.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.366+0000 7fbf25041700 1 -- 172.21.15.44:0/388131759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf201014f0 msgr2=0x7fbf20101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:22.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.366+0000 7fbf25041700 1 --2- 172.21.15.44:0/388131759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf201014f0 0x7fbf20101940 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbf14009990 tx=0x7fbf14009ca0 comp rx=0 tx=0).stop 2023-11-16T16:45:22.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.367+0000 7fbf25041700 1 -- 172.21.15.44:0/388131759 shutdown_connections 2023-11-16T16:45:22.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.367+0000 7fbf25041700 1 --2- 172.21.15.44:0/388131759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf20101e80 0x7fbf20108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:22.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.367+0000 7fbf25041700 1 --2- 172.21.15.44:0/388131759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf201002f0 0x7fbf20100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:22.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.367+0000 7fbf25041700 1 --2- 172.21.15.44:0/388131759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf201014f0 0x7fbf20101940 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:22.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.367+0000 7fbf25041700 1 -- 172.21.15.44:0/388131759 >> 172.21.15.44:0/388131759 conn(0x7fbf200fb900 msgr2=0x7fbf200fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:45:22.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.367+0000 7fbf25041700 1 -- 172.21.15.44:0/388131759 shutdown_connections 2023-11-16T16:45:22.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.367+0000 7fbf25041700 1 -- 172.21.15.44:0/388131759 wait complete. 2023-11-16T16:45:22.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.368+0000 7fbf25041700 1 Processor -- start 2023-11-16T16:45:22.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.368+0000 7fbf25041700 1 -- start start 2023-11-16T16:45:22.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.369+0000 7fbf25041700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf201002f0 0x7fbf20074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:22.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.369+0000 7fbf25041700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf201014f0 0x7fbf20072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:22.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.369+0000 7fbf25041700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf20101e80 0x7fbf20073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:22.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.369+0000 7fbf25041700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf20073ba0 con 0x7fbf201002f0 2023-11-16T16:45:22.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.369+0000 7fbf25041700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf20073d10 con 0x7fbf201014f0 2023-11-16T16:45:22.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.369+0000 7fbf25041700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf201a00e0 con 0x7fbf20101e80 2023-11-16T16:45:22.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.369+0000 7fbf1ed9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf201002f0 0x7fbf20074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:22.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.369+0000 7fbf1e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf201014f0 0x7fbf20072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:22.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.369+0000 7fbf1e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf201014f0 0x7fbf20072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60242/0 (socket says 172.21.15.44:60242) 2023-11-16T16:45:22.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.369+0000 7fbf1e59c700 1 -- 172.21.15.44:0/3321092109 learned_addr learned my addr 172.21.15.44:0/3321092109 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:45:22.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.369+0000 7fbf1f59e700 1 --2- 172.21.15.44:0/3321092109 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf20101e80 0x7fbf20073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.370+0000 7fbf1e59c700 1 -- 172.21.15.44:0/3321092109 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf20101e80 msgr2=0x7fbf20073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.370+0000 7fbf1e59c700 1 --2- 172.21.15.44:0/3321092109 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf20101e80 0x7fbf20073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.370+0000 7fbf1e59c700 1 -- 172.21.15.44:0/3321092109 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf201002f0 msgr2=0x7fbf20074890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.370+0000 7fbf1e59c700 1 --2- 172.21.15.44:0/3321092109 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf201002f0 0x7fbf20074890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.370+0000 7fbf1e59c700 1 -- 172.21.15.44:0/3321092109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf140095f0 con 0x7fbf201014f0 2023-11-16T16:45:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.370+0000 7fbf1e59c700 1 --2- 172.21.15.44:0/3321092109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf201014f0 0x7fbf20072ee0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbf14004740 tx=0x7fbf14009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:45:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.370+0000 7fbf07fff700 1 -- 172.21.15.44:0/3321092109 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1401d070 con 0x7fbf201014f0 2023-11-16T16:45:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.370+0000 7fbf07fff700 1 -- 172.21.15.44:0/3321092109 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf14004830 con 0x7fbf201014f0 2023-11-16T16:45:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.370+0000 7fbf25041700 1 -- 172.21.15.44:0/3321092109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf2010bc90 con 0x7fbf201014f0 2023-11-16T16:45:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.370+0000 7fbf07fff700 1 -- 172.21.15.44:0/3321092109 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf140053c0 con 0x7fbf201014f0 2023-11-16T16:45:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.370+0000 7fbf25041700 1 -- 172.21.15.44:0/3321092109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf2010bb00 con 0x7fbf201014f0 2023-11-16T16:45:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.371+0000 7fbf25041700 1 -- 172.21.15.44:0/3321092109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf201a06d0 con 0x7fbf201014f0 2023-11-16T16:45:22.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.373+0000 7fbf07fff700 1 -- 172.21.15.44:0/3321092109 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbf14005560 con 0x7fbf201014f0 2023-11-16T16:45:22.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.375+0000 7fbf07fff700 1 --2- 172.21.15.44:0/3321092109 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbf080a11d0 0x7fbf080a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:22.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.375+0000 7fbf07fff700 1 -- 172.21.15.44:0/3321092109 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5858+0+0 (secure 0 0 0) 0x7fbf140cd5d0 con 0x7fbf201014f0 2023-11-16T16:45:22.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.375+0000 7fbf1ed9d700 1 --2- 172.21.15.44:0/3321092109 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbf080a11d0 0x7fbf080a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:22.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.376+0000 7fbf1ed9d700 1 --2- 172.21.15.44:0/3321092109 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbf080a11d0 0x7fbf080a3680 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbf20101350 tx=0x7fbf10009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:45:22.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.377+0000 7fbf07fff700 1 -- 172.21.15.44:0/3321092109 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbf140976a0 con 0x7fbf201014f0 2023-11-16T16:45:22.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:22.593+0000 7fbf25041700 1 -- 172.21.15.44:0/3321092109 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf20060f30 con 0x7fbf080a11d0 2023-11-16T16:45:23.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:23 smithi077 ceph-mon[131204]: osdmap e60: 9 total, 9 up, 9 in 2023-11-16T16:45:23.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:23 smithi077 ceph-mon[131204]: pgmap v190: 1 pgs: 1 peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:23.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:23 smithi077 ceph-mon[131204]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:23.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:23 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:23.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:23 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:23.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:23 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:23.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:23 smithi077 ceph-mon[131204]: osdmap e61: 9 total, 9 up, 9 in 2023-11-16T16:45:23.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:23 smithi044 ceph-mon[128750]: osdmap e60: 9 total, 9 up, 9 in 2023-11-16T16:45:23.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:23 smithi044 ceph-mon[128750]: pgmap v190: 1 pgs: 1 peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:23.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:23 smithi044 ceph-mon[128750]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:23.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:23 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:23.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:23 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:23.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:23 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:23.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:23 smithi044 ceph-mon[128750]: osdmap e61: 9 total, 9 up, 9 in 2023-11-16T16:45:23.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:23 smithi176 ceph-mon[131630]: osdmap e60: 9 total, 9 up, 9 in 2023-11-16T16:45:23.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:23 smithi176 ceph-mon[131630]: pgmap v190: 1 pgs: 1 peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:23.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:23 smithi176 ceph-mon[131630]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:23.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:23 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:23.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:23 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:23.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:23 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:23.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:23 smithi176 ceph-mon[131630]: osdmap e61: 9 total, 9 up, 9 in 2023-11-16T16:45:25.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:25 smithi044 ceph-mon[128750]: osdmap e62: 9 total, 9 up, 9 in 2023-11-16T16:45:25.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:25 smithi044 ceph-mon[128750]: pgmap v193: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:25.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:25 smithi176 ceph-mon[131630]: osdmap e62: 9 total, 9 up, 9 in 2023-11-16T16:45:25.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:25 smithi176 ceph-mon[131630]: pgmap v193: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:25.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:25 smithi077 ceph-mon[131204]: osdmap e62: 9 total, 9 up, 9 in 2023-11-16T16:45:25.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:25 smithi077 ceph-mon[131204]: pgmap v193: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:26.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:26 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/920219189' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88e6c45c-9bac-49a2-b71d-2a6932bae792"}]: dispatch 2023-11-16T16:45:26.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:26 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88e6c45c-9bac-49a2-b71d-2a6932bae792"}]: dispatch 2023-11-16T16:45:26.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:26 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88e6c45c-9bac-49a2-b71d-2a6932bae792"}]': finished 2023-11-16T16:45:26.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:26 smithi044 ceph-mon[128750]: osdmap e63: 10 total, 9 up, 10 in 2023-11-16T16:45:26.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:26 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:26.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:26 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/920219189' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88e6c45c-9bac-49a2-b71d-2a6932bae792"}]: dispatch 2023-11-16T16:45:26.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:26 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88e6c45c-9bac-49a2-b71d-2a6932bae792"}]: dispatch 2023-11-16T16:45:26.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:26 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88e6c45c-9bac-49a2-b71d-2a6932bae792"}]': finished 2023-11-16T16:45:26.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:26 smithi176 ceph-mon[131630]: osdmap e63: 10 total, 9 up, 10 in 2023-11-16T16:45:26.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:26 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:26.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:26 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/920219189' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88e6c45c-9bac-49a2-b71d-2a6932bae792"}]: dispatch 2023-11-16T16:45:26.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:26 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88e6c45c-9bac-49a2-b71d-2a6932bae792"}]: dispatch 2023-11-16T16:45:26.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:26 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88e6c45c-9bac-49a2-b71d-2a6932bae792"}]': finished 2023-11-16T16:45:26.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:26 smithi077 ceph-mon[131204]: osdmap e63: 10 total, 9 up, 10 in 2023-11-16T16:45:26.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:26 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:27.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:27 smithi176 ceph-mon[131630]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 133 KiB/s, 0 objects/s recovering 2023-11-16T16:45:27.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:27 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3742671944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:27.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:27 smithi077 ceph-mon[131204]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 133 KiB/s, 0 objects/s recovering 2023-11-16T16:45:27.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:27 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3742671944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:27.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:27 smithi044 ceph-mon[128750]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 133 KiB/s, 0 objects/s recovering 2023-11-16T16:45:27.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:27 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3742671944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:29.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:29 smithi044 ceph-mon[128750]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T16:45:29.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:29 smithi176 ceph-mon[131630]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T16:45:29.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:29 smithi077 ceph-mon[131204]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T16:45:31.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:31 smithi044 ceph-mon[128750]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-11-16T16:45:31.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:31 smithi176 ceph-mon[131630]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-11-16T16:45:31.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:31 smithi077 ceph-mon[131204]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-11-16T16:45:33.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:33 smithi044 ceph-mon[128750]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2023-11-16T16:45:33.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:33 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-11-16T16:45:33.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:33 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:33.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:33 smithi176 ceph-mon[131630]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2023-11-16T16:45:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:33 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-11-16T16:45:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:33 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:33.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:33 smithi077 ceph-mon[131204]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2023-11-16T16:45:33.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:33 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-11-16T16:45:33.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:33 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:34.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:34 smithi044 ceph-mon[128750]: Deploying daemon osd.9 on smithi044 2023-11-16T16:45:34.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:34 smithi176 ceph-mon[131630]: Deploying daemon osd.9 on smithi044 2023-11-16T16:45:34.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:34 smithi077 ceph-mon[131204]: Deploying daemon osd.9 on smithi044 2023-11-16T16:45:35.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:35 smithi044 ceph-mon[128750]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-11-16T16:45:35.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:35 smithi176 ceph-mon[131630]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-11-16T16:45:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:35 smithi077 ceph-mon[131204]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-11-16T16:45:37.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:37 smithi044 ceph-mon[128750]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:37.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:37 smithi176 ceph-mon[131630]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:37.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:37 smithi077 ceph-mon[131204]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:39.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:39 smithi044 ceph-mon[128750]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:39.394 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:39 smithi176 ceph-mon[131630]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:39.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:39 smithi077 ceph-mon[131204]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:41.333 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:41 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:41.333 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:41 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:41.334 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:41 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:41.334 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:41 smithi044 ceph-mon[128750]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:41.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:41 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:41.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:41 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:41.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:41 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:41.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:41 smithi176 ceph-mon[131630]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:41.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:41 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:41.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:41 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:41.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:41 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:41.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:41 smithi077 ceph-mon[131204]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:43.801 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:43 smithi044 ceph-mon[128750]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:43.801 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:43.801 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:43.802 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:43.802 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:43.802 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:43 smithi176 ceph-mon[131630]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:43.916 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 9 on host 'smithi044' 2023-11-16T16:45:43.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:43.915+0000 7fbf07fff700 1 -- 172.21.15.44:0/3321092109 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbf20060f30 con 0x7fbf080a11d0 2023-11-16T16:45:43.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:43.918+0000 7fbf25041700 1 -- 172.21.15.44:0/3321092109 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbf080a11d0 msgr2=0x7fbf080a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:43.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:43.918+0000 7fbf25041700 1 --2- 172.21.15.44:0/3321092109 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbf080a11d0 0x7fbf080a3680 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbf20101350 tx=0x7fbf10009260 comp rx=0 tx=0).stop 2023-11-16T16:45:43.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:43.918+0000 7fbf25041700 1 -- 172.21.15.44:0/3321092109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf201014f0 msgr2=0x7fbf20072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:43.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:43.918+0000 7fbf25041700 1 --2- 172.21.15.44:0/3321092109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf201014f0 0x7fbf20072ee0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbf14004740 tx=0x7fbf14009580 comp rx=0 tx=0).stop 2023-11-16T16:45:43.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:43.918+0000 7fbf25041700 1 -- 172.21.15.44:0/3321092109 shutdown_connections 2023-11-16T16:45:43.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:43.918+0000 7fbf25041700 1 --2- 172.21.15.44:0/3321092109 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbf080a11d0 0x7fbf080a3680 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:43.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:43.918+0000 7fbf25041700 1 --2- 172.21.15.44:0/3321092109 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf20101e80 0x7fbf20073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:43.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:43.918+0000 7fbf25041700 1 --2- 172.21.15.44:0/3321092109 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf201002f0 0x7fbf20074890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:43.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:43.918+0000 7fbf25041700 1 --2- 172.21.15.44:0/3321092109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf201014f0 0x7fbf20072ee0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:43.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:43.918+0000 7fbf25041700 1 -- 172.21.15.44:0/3321092109 >> 172.21.15.44:0/3321092109 conn(0x7fbf200fb900 msgr2=0x7fbf200fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:45:43.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:43.918+0000 7fbf25041700 1 -- 172.21.15.44:0/3321092109 shutdown_connections 2023-11-16T16:45:43.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:45:43.919+0000 7fbf25041700 1 -- 172.21.15.44:0/3321092109 wait complete. 2023-11-16T16:45:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:43 smithi077 ceph-mon[131204]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:44.479 DEBUG:teuthology.orchestra.run.smithi044:osd.9> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.9.service 2023-11-16T16:45:44.499 INFO:tasks.cephadm:Deploying osd.10 on smithi077 with /dev/vg_nvme/lv_1... 2023-11-16T16:45:44.500 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-11-16T16:45:44.576 INFO:journalctl@ceph.osd.9.smithi044.stdout:-- Logs begin at Thu 2023-11-16 16:19:45 UTC. -- 2023-11-16T16:45:44.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:44 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:45:44.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:44 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:45:44.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:44 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:45:44.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:44 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:44.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:44 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:44.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:44 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:44.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:44 smithi044 ceph-mon[128750]: from='osd.9 [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-11-16T16:45:44.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:44 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:45:44.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:44 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:45:44.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:44 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:45:44.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:44 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:44.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:44 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:44.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:44 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:44.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:44 smithi176 ceph-mon[131630]: from='osd.9 [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-11-16T16:45:44.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:44 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:45:44.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:44 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:45:44.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:44 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:45:44.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:44 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:44.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:44 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:44.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:44 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:44.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:44 smithi077 ceph-mon[131204]: from='osd.9 [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-11-16T16:45:45.149 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T16:45:45.658 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:45 smithi077 ceph-mon[131204]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:45.658 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:45 smithi077 ceph-mon[131204]: from='osd.9 [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-11-16T16:45:45.659 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:45 smithi077 ceph-mon[131204]: osdmap e64: 10 total, 9 up, 10 in 2023-11-16T16:45:45.659 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:45.659 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:45 smithi077 ceph-mon[131204]: from='osd.9 [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2023-11-16T16:45:45.750 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:45 smithi044 ceph-mon[128750]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:45.750 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:45 smithi044 ceph-mon[128750]: from='osd.9 [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-11-16T16:45:45.750 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:45 smithi044 ceph-mon[128750]: osdmap e64: 10 total, 9 up, 10 in 2023-11-16T16:45:45.751 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:45.751 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:45 smithi044 ceph-mon[128750]: from='osd.9 [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2023-11-16T16:45:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:45 smithi176 ceph-mon[131630]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:45 smithi176 ceph-mon[131630]: from='osd.9 [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-11-16T16:45:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:45 smithi176 ceph-mon[131630]: osdmap e64: 10 total, 9 up, 10 in 2023-11-16T16:45:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:45 smithi176 ceph-mon[131630]: from='osd.9 [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2023-11-16T16:45:46.012 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 16:45:45 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T16:45:45.889+0000 7fce866fb700 -1 osd.9 0 waiting for initial osdmap 2023-11-16T16:45:46.012 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 16:45:45 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T16:45:45.907+0000 7fce7e4e8700 -1 osd.9 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:45:46.622 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-11-16T16:45:46.623 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-11-16T16:45:46.623 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2023-11-16T16:45:46.623 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2023-11-16T16:45:46.623 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127286 s, 824 MB/s 2023-11-16T16:45:46.623 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2023-11-16T16:45:46.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:46 smithi077 ceph-mon[131204]: from='osd.9 [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2023-11-16T16:45:46.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:46 smithi077 ceph-mon[131204]: osdmap e65: 10 total, 9 up, 10 in 2023-11-16T16:45:46.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:46 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:46.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:46 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:46.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:46 smithi077 ceph-mon[131204]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:47.102 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi077:vg_nvme/lv_1 2023-11-16T16:45:47.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:46 smithi044 ceph-mon[128750]: from='osd.9 [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2023-11-16T16:45:47.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:46 smithi044 ceph-mon[128750]: osdmap e65: 10 total, 9 up, 10 in 2023-11-16T16:45:47.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:46 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:47.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:46 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:47.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:46 smithi044 ceph-mon[128750]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:47.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:46 smithi176 ceph-mon[131630]: from='osd.9 [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2023-11-16T16:45:47.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:46 smithi176 ceph-mon[131630]: osdmap e65: 10 total, 9 up, 10 in 2023-11-16T16:45:47.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:46 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:47.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:46 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:47.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:46 smithi176 ceph-mon[131630]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:47.739 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T16:45:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: purged_snaps scrub starts 2023-11-16T16:45:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: purged_snaps scrub ok 2023-11-16T16:45:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: osd.9 [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398] boot 2023-11-16T16:45:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: osdmap e66: 10 total, 10 up, 10 in 2023-11-16T16:45:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: Detected new or changed devices on smithi044 2023-11-16T16:45:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi044 to 4319M 2023-11-16T16:45:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:45:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:48.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:45:48.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:47 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:45:48.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: purged_snaps scrub starts 2023-11-16T16:45:48.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: purged_snaps scrub ok 2023-11-16T16:45:48.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: osd.9 [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398] boot 2023-11-16T16:45:48.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: osdmap e66: 10 total, 10 up, 10 in 2023-11-16T16:45:48.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:48.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: Detected new or changed devices on smithi044 2023-11-16T16:45:48.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:48.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:48.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:48.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:48.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:48.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:48.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi044 to 4319M 2023-11-16T16:45:48.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:48.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:48.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:48.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:45:48.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:48.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:45:48.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:47 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:45:48.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: purged_snaps scrub starts 2023-11-16T16:45:48.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: purged_snaps scrub ok 2023-11-16T16:45:48.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: osd.9 [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398] boot 2023-11-16T16:45:48.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: osdmap e66: 10 total, 10 up, 10 in 2023-11-16T16:45:48.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:48.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: Detected new or changed devices on smithi044 2023-11-16T16:45:48.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:48.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:48.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:48.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:48.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:48.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:48.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi044 to 4319M 2023-11-16T16:45:48.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:48.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:48.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:48.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:45:48.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:45:48.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:45:48.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:47 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:45:48.927 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.924+0000 7f8c921a9700 1 -- 172.21.15.77:0/1171102365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c100de0 msgr2=0x7f8c8c1031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:48.928 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.924+0000 7f8c921a9700 1 --2- 172.21.15.77:0/1171102365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c100de0 0x7f8c8c1031c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c009990 tx=0x7f8c7c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:45:48.928 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.924+0000 7f8c921a9700 1 -- 172.21.15.77:0/1171102365 shutdown_connections 2023-11-16T16:45:48.928 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.924+0000 7f8c921a9700 1 --2- 172.21.15.77:0/1171102365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8c106020 0x7f8c8c108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:48.928 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.924+0000 7f8c921a9700 1 --2- 172.21.15.77:0/1171102365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c100de0 0x7f8c8c1031c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:48.928 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.924+0000 7f8c921a9700 1 --2- 172.21.15.77:0/1171102365 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c8c103700 0x7f8c8c105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:48.928 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.924+0000 7f8c921a9700 1 -- 172.21.15.77:0/1171102365 >> 172.21.15.77:0/1171102365 conn(0x7f8c8c0fa820 msgr2=0x7f8c8c0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:45:48.928 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.924+0000 7f8c921a9700 1 -- 172.21.15.77:0/1171102365 shutdown_connections 2023-11-16T16:45:48.929 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.924+0000 7f8c921a9700 1 -- 172.21.15.77:0/1171102365 wait complete. 2023-11-16T16:45:48.929 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.924+0000 7f8c921a9700 1 Processor -- start 2023-11-16T16:45:48.929 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c921a9700 1 -- start start 2023-11-16T16:45:48.929 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c921a9700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8c100de0 0x7f8c8c100740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:48.929 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c921a9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c103700 0x7f8c8c0fed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:48.929 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c921a9700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c8c106020 0x7f8c8c0ff2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:48.929 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c921a9700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c0ffa50 con 0x7f8c8c100de0 2023-11-16T16:45:48.929 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c921a9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c0ffbc0 con 0x7f8c8c103700 2023-11-16T16:45:48.929 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c921a9700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c1a0070 con 0x7f8c8c106020 2023-11-16T16:45:48.929 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c8affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c103700 0x7f8c8c0fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c8affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c103700 0x7f8c8c0fed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51128/0 (socket says 172.21.15.77:51128) 2023-11-16T16:45:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c8affd700 1 -- 172.21.15.77:0/1159540569 learned_addr learned my addr 172.21.15.77:0/1159540569 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-11-16T16:45:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c8bfff700 1 --2- 172.21.15.77:0/1159540569 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c8c106020 0x7f8c8c0ff2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c8b7fe700 1 --2- 172.21.15.77:0/1159540569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8c100de0 0x7f8c8c100740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c8affd700 1 -- 172.21.15.77:0/1159540569 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c8c106020 msgr2=0x7f8c8c0ff2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c8affd700 1 --2- 172.21.15.77:0/1159540569 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c8c106020 0x7f8c8c0ff2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c8affd700 1 -- 172.21.15.77:0/1159540569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8c100de0 msgr2=0x7f8c8c100740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c8affd700 1 --2- 172.21.15.77:0/1159540569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8c100de0 0x7f8c8c100740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c8affd700 1 -- 172.21.15.77:0/1159540569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c0095f0 con 0x7f8c8c103700 2023-11-16T16:45:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c8b7fe700 1 --2- 172.21.15.77:0/1159540569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8c100de0 0x7f8c8c100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:45:48.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.925+0000 7f8c8affd700 1 --2- 172.21.15.77:0/1159540569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c103700 0x7f8c8c0fed90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000d980 tx=0x7f8c8000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:45:48.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.926+0000 7f8c88ff9700 1 -- 172.21.15.77:0/1159540569 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c80004a30 con 0x7f8c8c103700 2023-11-16T16:45:48.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.926+0000 7f8c921a9700 1 -- 172.21.15.77:0/1159540569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c8c10bcc0 con 0x7f8c8c103700 2023-11-16T16:45:48.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.926+0000 7f8c88ff9700 1 -- 172.21.15.77:0/1159540569 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c800054e0 con 0x7f8c8c103700 2023-11-16T16:45:48.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.926+0000 7f8c88ff9700 1 -- 172.21.15.77:0/1159540569 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c80009c90 con 0x7f8c8c103700 2023-11-16T16:45:48.931 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.926+0000 7f8c921a9700 1 -- 172.21.15.77:0/1159540569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8c10bb30 con 0x7f8c8c103700 2023-11-16T16:45:48.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.927+0000 7f8c921a9700 1 -- 172.21.15.77:0/1159540569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c8c0fc390 con 0x7f8c8c103700 2023-11-16T16:45:48.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.928+0000 7f8c88ff9700 1 -- 172.21.15.77:0/1159540569 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8c80004c20 con 0x7f8c8c103700 2023-11-16T16:45:48.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.928+0000 7f8c88ff9700 1 --2- 172.21.15.77:0/1159540569 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8c740a1150 0x7f8c740a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:48.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.928+0000 7f8c88ff9700 1 -- 172.21.15.77:0/1159540569 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f8c800cbff0 con 0x7f8c8c103700 2023-11-16T16:45:48.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.929+0000 7f8c8b7fe700 1 --2- 172.21.15.77:0/1159540569 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8c740a1150 0x7f8c740a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:48.933 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.929+0000 7f8c8b7fe700 1 --2- 172.21.15.77:0/1159540569 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8c740a1150 0x7f8c740a3600 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c006010 tx=0x7f8c7c00b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:45:48.934 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:48.933+0000 7f8c88ff9700 1 -- 172.21.15.77:0/1159540569 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8c80095f40 con 0x7f8c8c103700 2023-11-16T16:45:48.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:48 smithi077 ceph-mon[131204]: osdmap e67: 10 total, 10 up, 10 in 2023-11-16T16:45:48.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:48 smithi077 ceph-mon[131204]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:49.131 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:45:49.129+0000 7f8c921a9700 1 -- 172.21.15.77:0/1159540569 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c8c061a30 con 0x7f8c740a1150 2023-11-16T16:45:49.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:48 smithi044 ceph-mon[128750]: osdmap e67: 10 total, 10 up, 10 in 2023-11-16T16:45:49.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:48 smithi044 ceph-mon[128750]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:49.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:48 smithi176 ceph-mon[131630]: osdmap e67: 10 total, 10 up, 10 in 2023-11-16T16:45:49.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:48 smithi176 ceph-mon[131630]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:50.284 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:50 smithi077 ceph-mon[131204]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:50.284 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:50 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:50.284 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:50 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:50.284 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:50 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:50.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:50 smithi044 ceph-mon[128750]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:50.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:50 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:50.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:50 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:50.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:50 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:50.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:50 smithi176 ceph-mon[131630]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:50.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:50 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:50.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:50 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:50.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:50 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:51.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:51 smithi044 ceph-mon[128750]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:51.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:51 smithi176 ceph-mon[131630]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:51.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:51 smithi077 ceph-mon[131204]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:52.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:52 smithi176 ceph-mon[131630]: from='client.? 172.21.15.77:0/1657023107' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41"}]: dispatch 2023-11-16T16:45:52.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:52 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41"}]: dispatch 2023-11-16T16:45:52.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:52 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41"}]': finished 2023-11-16T16:45:52.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:52 smithi176 ceph-mon[131630]: osdmap e68: 11 total, 10 up, 11 in 2023-11-16T16:45:52.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:52 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:45:52.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:52 smithi077 ceph-mon[131204]: from='client.? 172.21.15.77:0/1657023107' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41"}]: dispatch 2023-11-16T16:45:52.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:52 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41"}]: dispatch 2023-11-16T16:45:52.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:52 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41"}]': finished 2023-11-16T16:45:52.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:52 smithi077 ceph-mon[131204]: osdmap e68: 11 total, 10 up, 11 in 2023-11-16T16:45:52.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:52 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:45:52.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:52 smithi044 ceph-mon[128750]: from='client.? 172.21.15.77:0/1657023107' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41"}]: dispatch 2023-11-16T16:45:52.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:52 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41"}]: dispatch 2023-11-16T16:45:52.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:52 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41"}]': finished 2023-11-16T16:45:52.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:52 smithi044 ceph-mon[128750]: osdmap e68: 11 total, 10 up, 11 in 2023-11-16T16:45:52.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:52 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:45:53.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:53 smithi176 ceph-mon[131630]: from='client.? 172.21.15.77:0/350613752' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:53.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:53 smithi176 ceph-mon[131630]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:53.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:53 smithi077 ceph-mon[131204]: from='client.? 172.21.15.77:0/350613752' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:53.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:53 smithi077 ceph-mon[131204]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:53.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:53 smithi044 ceph-mon[128750]: from='client.? 172.21.15.77:0/350613752' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:53.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:53 smithi044 ceph-mon[128750]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:55 smithi044 ceph-mon[128750]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:55.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:55 smithi176 ceph-mon[131630]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:55.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:55 smithi077 ceph-mon[131204]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:57.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:57 smithi044 ceph-mon[128750]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:57.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:57 smithi176 ceph-mon[131630]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:57.975 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:57 smithi077 ceph-mon[131204]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:58.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:58 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-11-16T16:45:58.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:58 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:58.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:58 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-11-16T16:45:58.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:58 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:58.956 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:58 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-11-16T16:45:58.956 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:58 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:59.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:59 smithi077 ceph-mon[131204]: Deploying daemon osd.10 on smithi077 2023-11-16T16:45:59.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:45:59 smithi077 ceph-mon[131204]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:59.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:59 smithi044 ceph-mon[128750]: Deploying daemon osd.10 on smithi077 2023-11-16T16:45:59.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:45:59 smithi044 ceph-mon[128750]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:59 smithi176 ceph-mon[131630]: Deploying daemon osd.10 on smithi077 2023-11-16T16:45:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:45:59 smithi176 ceph-mon[131630]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:01.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:01 smithi077 ceph-mon[131204]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:01.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:01 smithi044 ceph-mon[128750]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:01.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:01 smithi176 ceph-mon[131630]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:03.306 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:03 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:03.306 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:03 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:03.306 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:03 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:03.306 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:03 smithi077 ceph-mon[131204]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:03.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:03 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:03.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:03 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:03.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:03 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:03.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:03 smithi044 ceph-mon[128750]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:03.394 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:03 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:03.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:03 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:03.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:03 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:03.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:03 smithi176 ceph-mon[131630]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:04.894 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:46:04.893+0000 7f8c88ff9700 1 -- 172.21.15.77:0/1159540569 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f8c8c061a30 con 0x7f8c740a1150 2023-11-16T16:46:04.895 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 10 on host 'smithi077' 2023-11-16T16:46:04.896 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:46:04.895+0000 7f8c921a9700 1 -- 172.21.15.77:0/1159540569 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8c740a1150 msgr2=0x7f8c740a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:04.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:46:04.895+0000 7f8c921a9700 1 --2- 172.21.15.77:0/1159540569 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8c740a1150 0x7f8c740a3600 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c006010 tx=0x7f8c7c00b540 comp rx=0 tx=0).stop 2023-11-16T16:46:04.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:46:04.895+0000 7f8c921a9700 1 -- 172.21.15.77:0/1159540569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c103700 msgr2=0x7f8c8c0fed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:04.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:46:04.895+0000 7f8c921a9700 1 --2- 172.21.15.77:0/1159540569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c103700 0x7f8c8c0fed90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000d980 tx=0x7f8c8000dc90 comp rx=0 tx=0).stop 2023-11-16T16:46:04.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:46:04.895+0000 7f8c921a9700 1 -- 172.21.15.77:0/1159540569 shutdown_connections 2023-11-16T16:46:04.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:46:04.895+0000 7f8c921a9700 1 --2- 172.21.15.77:0/1159540569 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8c740a1150 0x7f8c740a3600 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:04.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:46:04.895+0000 7f8c921a9700 1 --2- 172.21.15.77:0/1159540569 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c8c106020 0x7f8c8c0ff2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:04.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:46:04.895+0000 7f8c921a9700 1 --2- 172.21.15.77:0/1159540569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8c100de0 0x7f8c8c100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:04.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:46:04.895+0000 7f8c921a9700 1 --2- 172.21.15.77:0/1159540569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c8c103700 0x7f8c8c0fed90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:04.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:46:04.895+0000 7f8c921a9700 1 -- 172.21.15.77:0/1159540569 >> 172.21.15.77:0/1159540569 conn(0x7f8c8c0fa820 msgr2=0x7f8c8c101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:04.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:46:04.896+0000 7f8c921a9700 1 -- 172.21.15.77:0/1159540569 shutdown_connections 2023-11-16T16:46:04.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-16T16:46:04.896+0000 7f8c921a9700 1 -- 172.21.15.77:0/1159540569 wait complete. 2023-11-16T16:46:05.322 DEBUG:teuthology.orchestra.run.smithi077:osd.10> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.10.service 2023-11-16T16:46:05.324 INFO:tasks.cephadm:Deploying osd.11 on smithi176 with /dev/vg_nvme/lv_1... 2023-11-16T16:46:05.324 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-11-16T16:46:05.357 INFO:journalctl@ceph.osd.10.smithi077.stdout:-- Logs begin at Thu 2023-11-16 16:15:33 UTC. -- 2023-11-16T16:46:05.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:05 smithi176 ceph-mon[131630]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:05.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:05 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:05.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:05 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:05.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:05 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:05.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:05 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:05.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:05 smithi077 ceph-mon[131204]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:05.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:05 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:05.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:05 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:05.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:05 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:05.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:05 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:05.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:05 smithi044 ceph-mon[128750]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:05.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:05 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:05.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:05 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:05.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:05 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:05.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:05 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:05.978 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:46:06.139 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 16:46:06 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T16:46:06.076+0000 7f6fbccb7640 -1 osd.10 0 log_to_monitors true 2023-11-16T16:46:06.687 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:06 smithi077 ceph-mon[131204]: from='osd.10 [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T16:46:06.687 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:06 smithi077 ceph-mon[131204]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T16:46:06.847 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:06 smithi176 ceph-mon[131630]: from='osd.10 [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T16:46:06.847 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:06 smithi176 ceph-mon[131630]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T16:46:06.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:06 smithi044 ceph-mon[128750]: from='osd.10 [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T16:46:06.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:06 smithi044 ceph-mon[128750]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T16:46:07.322 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-11-16T16:46:07.322 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-11-16T16:46:07.323 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2023-11-16T16:46:07.323 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2023-11-16T16:46:07.323 INFO:teuthology.orchestra.run.smithi176.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124583 s, 842 MB/s 2023-11-16T16:46:07.323 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2023-11-16T16:46:07.618 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:07 smithi176 ceph-mon[131630]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:07.618 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:07 smithi176 ceph-mon[131630]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-11-16T16:46:07.618 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:07 smithi176 ceph-mon[131630]: osdmap e69: 11 total, 10 up, 11 in 2023-11-16T16:46:07.618 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:07.618 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:07 smithi176 ceph-mon[131630]: from='osd.10 [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:46:07.618 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:07 smithi176 ceph-mon[131630]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:46:07.619 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi176:vg_nvme/lv_1 2023-11-16T16:46:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:07 smithi044 ceph-mon[128750]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:07 smithi044 ceph-mon[128750]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-11-16T16:46:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:07 smithi044 ceph-mon[128750]: osdmap e69: 11 total, 10 up, 11 in 2023-11-16T16:46:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:07 smithi044 ceph-mon[128750]: from='osd.10 [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:46:07.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:07 smithi044 ceph-mon[128750]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:46:07.976 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 16:46:07 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T16:46:07.584+0000 7f6fb232e700 -1 osd.10 0 waiting for initial osdmap 2023-11-16T16:46:07.976 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 16:46:07 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T16:46:07.596+0000 7f6faa11b700 -1 osd.10 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:46:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:07 smithi077 ceph-mon[131204]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:07 smithi077 ceph-mon[131204]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-11-16T16:46:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:07 smithi077 ceph-mon[131204]: osdmap e69: 11 total, 10 up, 11 in 2023-11-16T16:46:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:07 smithi077 ceph-mon[131204]: from='osd.10 [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:46:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:07 smithi077 ceph-mon[131204]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2023-11-16T16:46:08.261 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:46:08.713 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2023-11-16T16:46:08.713 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: osdmap e70: 11 total, 10 up, 11 in 2023-11-16T16:46:08.713 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:08.713 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:08.713 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: Detected new or changed devices on smithi077 2023-11-16T16:46:08.713 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:08.713 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:08.713 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:08.713 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:08.713 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:08.713 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:08.714 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi077 to 4292M 2023-11-16T16:46:08.714 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:08.714 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:08.714 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:08.714 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:46:08.714 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:08.714 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:46:08.714 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:46:08.714 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:08.714 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:08.714 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:08.714 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:08 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:08.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2023-11-16T16:46:08.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: osdmap e70: 11 total, 10 up, 11 in 2023-11-16T16:46:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: Detected new or changed devices on smithi077 2023-11-16T16:46:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi077 to 4292M 2023-11-16T16:46:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:46:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:46:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:46:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:08.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:08 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:08.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2023-11-16T16:46:08.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: osdmap e70: 11 total, 10 up, 11 in 2023-11-16T16:46:08.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:08.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: Detected new or changed devices on smithi077 2023-11-16T16:46:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi077 to 4292M 2023-11-16T16:46:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:46:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:46:08.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:46:08.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:08.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:08.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:08.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:08 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:09.344 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.343+0000 7f17fd5f3700 1 -- 172.21.15.176:0/935074862 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17f80fe5c0 msgr2=0x7f17f80fe9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:09.344 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.343+0000 7f17fd5f3700 1 --2- 172.21.15.176:0/935074862 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17f80fe5c0 0x7f17f80fe9d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f17f000b0a0 tx=0x7f17f000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:46:09.344 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.343+0000 7f17fd5f3700 1 -- 172.21.15.176:0/935074862 shutdown_connections 2023-11-16T16:46:09.344 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.343+0000 7f17fd5f3700 1 --2- 172.21.15.176:0/935074862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17f8101fd0 0x7f17f8108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:09.344 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.343+0000 7f17fd5f3700 1 --2- 172.21.15.176:0/935074862 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17f80fe5c0 0x7f17f80fe9d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:09.344 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.343+0000 7f17fd5f3700 1 --2- 172.21.15.176:0/935074862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f8101640 0x7f17f8101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:09.345 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.343+0000 7f17fd5f3700 1 -- 172.21.15.176:0/935074862 >> 172.21.15.176:0/935074862 conn(0x7f17f80f9f70 msgr2=0x7f17f80fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:09.345 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.343+0000 7f17fd5f3700 1 -- 172.21.15.176:0/935074862 shutdown_connections 2023-11-16T16:46:09.345 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.343+0000 7f17fd5f3700 1 -- 172.21.15.176:0/935074862 wait complete. 2023-11-16T16:46:09.347 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.343+0000 7f17fd5f3700 1 Processor -- start 2023-11-16T16:46:09.347 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17fd5f3700 1 -- start start 2023-11-16T16:46:09.347 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17fd5f3700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17f80fe5c0 0x7f17f8196180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:09.347 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17fd5f3700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17f8101640 0x7f17f81966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:09.347 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17fd5f3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f8101fd0 0x7f17f819d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:09.347 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17fd5f3700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f819dd10 con 0x7f17f80fe5c0 2023-11-16T16:46:09.347 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17fd5f3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f819de50 con 0x7f17f8101fd0 2023-11-16T16:46:09.347 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17fd5f3700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f819dfe0 con 0x7f17f8101640 2023-11-16T16:46:09.347 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17f67fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17f8101640 0x7f17f81966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:09.347 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17f67fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17f8101640 0x7f17f81966c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38134/0 (socket says 172.21.15.176:38134) 2023-11-16T16:46:09.347 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17f67fc700 1 -- 172.21.15.176:0/1861386414 learned_addr learned my addr 172.21.15.176:0/1861386414 (peer_addr_for_me v2:172.21.15.176:0/0) 2023-11-16T16:46:09.348 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17f77fe700 1 --2- 172.21.15.176:0/1861386414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f8101fd0 0x7f17f819d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:09.348 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17f6ffd700 1 --2- 172.21.15.176:0/1861386414 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17f80fe5c0 0x7f17f8196180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:09.348 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17f67fc700 1 -- 172.21.15.176:0/1861386414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f8101fd0 msgr2=0x7f17f819d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:09.348 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17f67fc700 1 --2- 172.21.15.176:0/1861386414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f8101fd0 0x7f17f819d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:09.348 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17f67fc700 1 -- 172.21.15.176:0/1861386414 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17f80fe5c0 msgr2=0x7f17f8196180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:09.348 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17f67fc700 1 --2- 172.21.15.176:0/1861386414 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17f80fe5c0 0x7f17f8196180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:09.348 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17f67fc700 1 -- 172.21.15.176:0/1861386414 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f0009d00 con 0x7f17f8101640 2023-11-16T16:46:09.348 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17f6ffd700 1 --2- 172.21.15.176:0/1861386414 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17f80fe5c0 0x7f17f8196180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:46:09.348 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17f67fc700 1 --2- 172.21.15.176:0/1861386414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17f8101640 0x7f17f81966c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00c9e0 tx=0x7f17ec00ccf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:09.348 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.344+0000 7f17dffff700 1 -- 172.21.15.176:0/1861386414 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ec007ac0 con 0x7f17f8101640 2023-11-16T16:46:09.348 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.345+0000 7f17dffff700 1 -- 172.21.15.176:0/1861386414 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17ec007c20 con 0x7f17f8101640 2023-11-16T16:46:09.348 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.345+0000 7f17fd5f3700 1 -- 172.21.15.176:0/1861386414 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f810bcc0 con 0x7f17f8101640 2023-11-16T16:46:09.349 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.345+0000 7f17dffff700 1 -- 172.21.15.176:0/1861386414 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ec018660 con 0x7f17f8101640 2023-11-16T16:46:09.349 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.345+0000 7f17fd5f3700 1 -- 172.21.15.176:0/1861386414 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f810bb30 con 0x7f17f8101640 2023-11-16T16:46:09.349 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.346+0000 7f17fd5f3700 1 -- 172.21.15.176:0/1861386414 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f8107940 con 0x7f17f8101640 2023-11-16T16:46:09.349 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.347+0000 7f17dffff700 1 -- 172.21.15.176:0/1861386414 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f17ec00f450 con 0x7f17f8101640 2023-11-16T16:46:09.350 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.347+0000 7f17dffff700 1 --2- 172.21.15.176:0/1861386414 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f17e00a1150 0x7f17e00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:09.350 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.347+0000 7f17dffff700 1 -- 172.21.15.176:0/1861386414 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6727+0+0 (secure 0 0 0) 0x7f17ec0cc550 con 0x7f17f8101640 2023-11-16T16:46:09.350 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.348+0000 7f17f6ffd700 1 --2- 172.21.15.176:0/1861386414 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f17e00a1150 0x7f17e00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:09.350 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.348+0000 7f17f6ffd700 1 --2- 172.21.15.176:0/1861386414 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f17e00a1150 0x7f17e00a3600 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f17f0007ae0 tx=0x7f17f0006020 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:09.351 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.351+0000 7f17dffff700 1 -- 172.21.15.176:0/1861386414 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f17ec096360 con 0x7f17f8101640 2023-11-16T16:46:09.545 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:09.544+0000 7f17fd5f3700 1 -- 172.21.15.176:0/1861386414 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f17f8061a30 con 0x7f17e00a1150 2023-11-16T16:46:09.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:09 smithi044 ceph-mon[128750]: purged_snaps scrub starts 2023-11-16T16:46:09.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:09 smithi044 ceph-mon[128750]: purged_snaps scrub ok 2023-11-16T16:46:09.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:09 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:46:09.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:09 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:46:09.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:09 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:46:09.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:09 smithi044 ceph-mon[128750]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:09.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:09 smithi044 ceph-mon[128750]: osd.10 [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093] boot 2023-11-16T16:46:09.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:09 smithi044 ceph-mon[128750]: osdmap e71: 11 total, 11 up, 11 in 2023-11-16T16:46:09.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:09 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:09.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:09 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:46:09.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:09 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:46:09.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:09 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:09.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:09 smithi176 ceph-mon[131630]: purged_snaps scrub starts 2023-11-16T16:46:09.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:09 smithi176 ceph-mon[131630]: purged_snaps scrub ok 2023-11-16T16:46:09.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:09 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:46:09.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:09 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:46:09.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:09 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:46:09.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:09 smithi176 ceph-mon[131630]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:09.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:09 smithi176 ceph-mon[131630]: osd.10 [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093] boot 2023-11-16T16:46:09.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:09 smithi176 ceph-mon[131630]: osdmap e71: 11 total, 11 up, 11 in 2023-11-16T16:46:09.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:09 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:09.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:09 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:46:09.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:09 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:46:09.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:09 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:09.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:09 smithi077 ceph-mon[131204]: purged_snaps scrub starts 2023-11-16T16:46:09.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:09 smithi077 ceph-mon[131204]: purged_snaps scrub ok 2023-11-16T16:46:09.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:09 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:46:09.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:09 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:46:09.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:09 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:46:09.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:09 smithi077 ceph-mon[131204]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:09.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:09 smithi077 ceph-mon[131204]: osd.10 [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093] boot 2023-11-16T16:46:09.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:09 smithi077 ceph-mon[131204]: osdmap e71: 11 total, 11 up, 11 in 2023-11-16T16:46:09.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:09 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:09.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:09 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:46:09.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:09 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:46:09.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:09 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:10.754 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:10 smithi176 ceph-mon[131630]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:46:10.754 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:10 smithi176 ceph-mon[131630]: osdmap e72: 11 total, 11 up, 11 in 2023-11-16T16:46:10.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:10 smithi044 ceph-mon[128750]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:46:10.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:10 smithi044 ceph-mon[128750]: osdmap e72: 11 total, 11 up, 11 in 2023-11-16T16:46:10.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:10 smithi077 ceph-mon[131204]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:46:10.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:10 smithi077 ceph-mon[131204]: osdmap e72: 11 total, 11 up, 11 in 2023-11-16T16:46:11.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:11 smithi176 ceph-mon[131630]: pgmap v226: 1 pgs: 1 remapped+peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:11.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:11 smithi176 ceph-mon[131630]: osdmap e73: 11 total, 11 up, 11 in 2023-11-16T16:46:11.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:11 smithi077 ceph-mon[131204]: pgmap v226: 1 pgs: 1 remapped+peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:11.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:11 smithi077 ceph-mon[131204]: osdmap e73: 11 total, 11 up, 11 in 2023-11-16T16:46:12.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:11 smithi044 ceph-mon[128750]: pgmap v226: 1 pgs: 1 remapped+peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:12.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:11 smithi044 ceph-mon[128750]: osdmap e73: 11 total, 11 up, 11 in 2023-11-16T16:46:12.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:12 smithi176 ceph-mon[131630]: osdmap e74: 11 total, 11 up, 11 in 2023-11-16T16:46:12.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:12 smithi176 ceph-mon[131630]: from='client.? 172.21.15.176:0/3131593038' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca5bd483-91ad-4385-922d-129f21cf955a"}]: dispatch 2023-11-16T16:46:12.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:12 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca5bd483-91ad-4385-922d-129f21cf955a"}]: dispatch 2023-11-16T16:46:12.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:12 smithi176 ceph-mon[131630]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca5bd483-91ad-4385-922d-129f21cf955a"}]': finished 2023-11-16T16:46:12.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:12 smithi176 ceph-mon[131630]: osdmap e75: 12 total, 11 up, 12 in 2023-11-16T16:46:12.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:12 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:12.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:12 smithi176 ceph-mon[131630]: from='client.? 172.21.15.176:0/197190284' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:46:12.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:12 smithi077 ceph-mon[131204]: osdmap e74: 11 total, 11 up, 11 in 2023-11-16T16:46:12.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:12 smithi077 ceph-mon[131204]: from='client.? 172.21.15.176:0/3131593038' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca5bd483-91ad-4385-922d-129f21cf955a"}]: dispatch 2023-11-16T16:46:12.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:12 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca5bd483-91ad-4385-922d-129f21cf955a"}]: dispatch 2023-11-16T16:46:12.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:12 smithi077 ceph-mon[131204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca5bd483-91ad-4385-922d-129f21cf955a"}]': finished 2023-11-16T16:46:12.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:12 smithi077 ceph-mon[131204]: osdmap e75: 12 total, 11 up, 12 in 2023-11-16T16:46:12.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:12 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:12.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:12 smithi077 ceph-mon[131204]: from='client.? 172.21.15.176:0/197190284' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:46:13.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:12 smithi044 ceph-mon[128750]: osdmap e74: 11 total, 11 up, 11 in 2023-11-16T16:46:13.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:12 smithi044 ceph-mon[128750]: from='client.? 172.21.15.176:0/3131593038' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca5bd483-91ad-4385-922d-129f21cf955a"}]: dispatch 2023-11-16T16:46:13.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:12 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca5bd483-91ad-4385-922d-129f21cf955a"}]: dispatch 2023-11-16T16:46:13.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:12 smithi044 ceph-mon[128750]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca5bd483-91ad-4385-922d-129f21cf955a"}]': finished 2023-11-16T16:46:13.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:12 smithi044 ceph-mon[128750]: osdmap e75: 12 total, 11 up, 12 in 2023-11-16T16:46:13.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:12 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:13.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:12 smithi044 ceph-mon[128750]: from='client.? 172.21.15.176:0/197190284' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:46:13.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:13 smithi176 ceph-mon[131630]: pgmap v230: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:13.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:13 smithi077 ceph-mon[131204]: pgmap v230: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:14.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:13 smithi044 ceph-mon[128750]: pgmap v230: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:15 smithi176 ceph-mon[131630]: pgmap v231: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:15.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:15 smithi077 ceph-mon[131204]: pgmap v231: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:16.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:15 smithi044 ceph-mon[128750]: pgmap v231: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:17 smithi176 ceph-mon[131630]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T16:46:17.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:17 smithi077 ceph-mon[131204]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T16:46:18.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:17 smithi044 ceph-mon[128750]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T16:46:19.311 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-11-16T16:46:19.311 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:19.311 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:19 smithi176 ceph-mon[131630]: Deploying daemon osd.11 on smithi176 2023-11-16T16:46:19.311 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:19 smithi176 ceph-mon[131630]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2023-11-16T16:46:19.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-11-16T16:46:19.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:19.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:19 smithi077 ceph-mon[131204]: Deploying daemon osd.11 on smithi176 2023-11-16T16:46:19.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:19 smithi077 ceph-mon[131204]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2023-11-16T16:46:19.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-11-16T16:46:19.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:19.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:19 smithi044 ceph-mon[128750]: Deploying daemon osd.11 on smithi176 2023-11-16T16:46:19.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:19 smithi044 ceph-mon[128750]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2023-11-16T16:46:21.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:21 smithi044 ceph-mon[128750]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-11-16T16:46:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:21 smithi176 ceph-mon[131630]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-11-16T16:46:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:21 smithi077 ceph-mon[131204]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-11-16T16:46:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:23 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.440 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:23 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.440 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:23 smithi044 ceph-mon[128750]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 54 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.440 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:23 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:30.442 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:23 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.442 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:23 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.442 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:23 smithi176 ceph-mon[131630]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 54 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.442 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:23 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:30.442 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:25.208+0000 7f17dffff700 1 -- 172.21.15.176:0/1861386414 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f17f8061a30 con 0x7f17e00a1150 2023-11-16T16:46:30.443 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 11 on host 'smithi176' 2023-11-16T16:46:30.444 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:25 smithi044 ceph-mon[128750]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.444 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:25 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.444 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:25 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.444 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:25 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:30.444 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:25 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:30.444 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:25 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.444 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:25 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.444 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:25 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.444 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:25 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:30.444 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:26 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:46:30.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:26 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:46:30.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:26 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:46:30.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:26 smithi044 ceph-mon[128750]: from='osd.11 [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T16:46:30.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:26 smithi044 ceph-mon[128750]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T16:46:30.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:27 smithi044 ceph-mon[128750]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:27 smithi044 ceph-mon[128750]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-11-16T16:46:30.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:27 smithi044 ceph-mon[128750]: osdmap e76: 12 total, 11 up, 12 in 2023-11-16T16:46:30.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:27 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:30.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:27 smithi044 ceph-mon[128750]: from='osd.11 [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:46:30.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:27 smithi044 ceph-mon[128750]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:46:30.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2023-11-16T16:46:30.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: osdmap e77: 12 total, 11 up, 12 in 2023-11-16T16:46:30.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:30.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:30.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: Detected new or changed devices on smithi176 2023-11-16T16:46:30.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:30.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:30.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:30.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:30.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi176 to 4292M 2023-11-16T16:46:30.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:30.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:30.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:46:30.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.447 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:46:30.447 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:28 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:46:30.447 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:29 smithi044 ceph-mon[128750]: purged_snaps scrub starts 2023-11-16T16:46:30.447 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:29 smithi044 ceph-mon[128750]: purged_snaps scrub ok 2023-11-16T16:46:30.447 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:29 smithi044 ceph-mon[128750]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:30.447 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:29 smithi044 ceph-mon[128750]: osd.11 [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101] boot 2023-11-16T16:46:30.447 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:29 smithi044 ceph-mon[128750]: osdmap e78: 12 total, 12 up, 12 in 2023-11-16T16:46:30.447 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:29 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:30.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:23 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:23 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:23 smithi077 ceph-mon[131204]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 54 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:23 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:30.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:25 smithi077 ceph-mon[131204]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:25 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:25 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:25 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:30.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:25 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:30.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:25 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:25 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:25 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:25 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:30.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:26 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:46:30.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:26 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:46:30.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:26 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:46:30.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:26 smithi077 ceph-mon[131204]: from='osd.11 [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T16:46:30.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:26 smithi077 ceph-mon[131204]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T16:46:30.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:27 smithi077 ceph-mon[131204]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:27 smithi077 ceph-mon[131204]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-11-16T16:46:30.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:27 smithi077 ceph-mon[131204]: osdmap e76: 12 total, 11 up, 12 in 2023-11-16T16:46:30.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:27 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:30.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:27 smithi077 ceph-mon[131204]: from='osd.11 [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:46:30.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:27 smithi077 ceph-mon[131204]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:46:30.469 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2023-11-16T16:46:30.469 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: osdmap e77: 12 total, 11 up, 12 in 2023-11-16T16:46:30.469 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:30.469 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:30.470 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: Detected new or changed devices on smithi176 2023-11-16T16:46:30.470 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.470 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.470 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:30.470 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:30.470 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:30.470 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:30.470 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi176 to 4292M 2023-11-16T16:46:30.470 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.470 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:30.470 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:30.470 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:46:30.470 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.471 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:46:30.471 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:28 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:46:30.471 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:29 smithi077 ceph-mon[131204]: purged_snaps scrub starts 2023-11-16T16:46:30.471 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:29 smithi077 ceph-mon[131204]: purged_snaps scrub ok 2023-11-16T16:46:30.471 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:29 smithi077 ceph-mon[131204]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:30.471 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:29 smithi077 ceph-mon[131204]: osd.11 [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101] boot 2023-11-16T16:46:30.471 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:29 smithi077 ceph-mon[131204]: osdmap e78: 12 total, 12 up, 12 in 2023-11-16T16:46:30.471 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:29 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:30.472 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:25.211+0000 7f17fd5f3700 1 -- 172.21.15.176:0/1861386414 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f17e00a1150 msgr2=0x7f17e00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:30.473 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:25.211+0000 7f17fd5f3700 1 --2- 172.21.15.176:0/1861386414 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f17e00a1150 0x7f17e00a3600 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f17f0007ae0 tx=0x7f17f0006020 comp rx=0 tx=0).stop 2023-11-16T16:46:30.473 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:25.211+0000 7f17fd5f3700 1 -- 172.21.15.176:0/1861386414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17f8101640 msgr2=0x7f17f81966c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:30.473 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:25.211+0000 7f17fd5f3700 1 --2- 172.21.15.176:0/1861386414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17f8101640 0x7f17f81966c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00c9e0 tx=0x7f17ec00ccf0 comp rx=0 tx=0).stop 2023-11-16T16:46:30.473 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:25.211+0000 7f17fd5f3700 1 -- 172.21.15.176:0/1861386414 shutdown_connections 2023-11-16T16:46:30.473 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:25.211+0000 7f17fd5f3700 1 --2- 172.21.15.176:0/1861386414 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f17e00a1150 0x7f17e00a3600 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:30.473 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:25.211+0000 7f17fd5f3700 1 --2- 172.21.15.176:0/1861386414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f8101fd0 0x7f17f819d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:30.473 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:25.211+0000 7f17fd5f3700 1 --2- 172.21.15.176:0/1861386414 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17f80fe5c0 0x7f17f8196180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:30.473 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:25.211+0000 7f17fd5f3700 1 --2- 172.21.15.176:0/1861386414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17f8101640 0x7f17f81966c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:30.473 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:25.211+0000 7f17fd5f3700 1 -- 172.21.15.176:0/1861386414 >> 172.21.15.176:0/1861386414 conn(0x7f17f80f9f70 msgr2=0x7f17f8104a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:30.473 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:25.211+0000 7f17fd5f3700 1 -- 172.21.15.176:0/1861386414 shutdown_connections 2023-11-16T16:46:30.473 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:25.211+0000 7f17fd5f3700 1 -- 172.21.15.176:0/1861386414 wait complete. 2023-11-16T16:46:30.474 DEBUG:teuthology.orchestra.run.smithi176:osd.11> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.11.service 2023-11-16T16:46:30.475 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:25 smithi176 ceph-mon[131630]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.475 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:25 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.475 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:25 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.475 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:25 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:30.475 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:25 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:30.475 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:25 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.475 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:25 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.475 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:25 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.475 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:25 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:30.475 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:26 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:46:30.476 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:26 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:46:30.476 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:26 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:46:30.476 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:26 smithi176 ceph-mon[131630]: from='osd.11 [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T16:46:30.476 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:26 smithi176 ceph-mon[131630]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T16:46:30.476 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:27 smithi176 ceph-mon[131630]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.476 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:27 smithi176 ceph-mon[131630]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-11-16T16:46:30.476 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:27 smithi176 ceph-mon[131630]: osdmap e76: 12 total, 11 up, 12 in 2023-11-16T16:46:30.476 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:27 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:30.476 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:27 smithi176 ceph-mon[131630]: from='osd.11 [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:46:30.476 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:27 smithi176 ceph-mon[131630]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2023-11-16T16:46:30.476 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2023-11-16T16:46:30.476 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: osdmap e77: 12 total, 11 up, 12 in 2023-11-16T16:46:30.476 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:30.476 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:30.477 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: Detected new or changed devices on smithi176 2023-11-16T16:46:30.477 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.477 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.477 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:30.477 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:30.477 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:30.477 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:30.477 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi176 to 4292M 2023-11-16T16:46:30.477 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.477 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:30.477 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:30.477 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:46:30.477 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:30.477 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:46:30.478 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:28 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:46:30.478 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:29 smithi176 ceph-mon[131630]: purged_snaps scrub starts 2023-11-16T16:46:30.478 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:29 smithi176 ceph-mon[131630]: purged_snaps scrub ok 2023-11-16T16:46:30.478 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:29 smithi176 ceph-mon[131630]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:30.478 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:29 smithi176 ceph-mon[131630]: osd.11 [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101] boot 2023-11-16T16:46:30.478 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:29 smithi176 ceph-mon[131630]: osdmap e78: 12 total, 12 up, 12 in 2023-11-16T16:46:30.478 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:29 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:30.480 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-11-16T16:46:30.480 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd stat -f json 2023-11-16T16:46:30.517 INFO:journalctl@ceph.osd.11.smithi176.stdout:-- Logs begin at Thu 2023-11-16 16:14:40 UTC. -- 2023-11-16T16:46:31.355 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:46:31.394 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:31 smithi176 ceph-mon[131630]: osdmap e79: 12 total, 12 up, 12 in 2023-11-16T16:46:31.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:31 smithi176 ceph-mon[131630]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:31.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:31 smithi077 ceph-mon[131204]: osdmap e79: 12 total, 12 up, 12 in 2023-11-16T16:46:31.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:31 smithi077 ceph-mon[131204]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:31.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:31 smithi044 ceph-mon[128750]: osdmap e79: 12 total, 12 up, 12 in 2023-11-16T16:46:31.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:31 smithi044 ceph-mon[128750]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:32.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.854+0000 7ffbc218b700 1 -- 172.21.15.44:0/1574006107 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbbc101f00 msgr2=0x7ffbbc1087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:32.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.854+0000 7ffbc218b700 1 --2- 172.21.15.44:0/1574006107 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbbc101f00 0x7ffbbc1087c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ffbb400b0a0 tx=0x7ffbb400b3b0 comp rx=0 tx=0).stop 2023-11-16T16:46:32.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.854+0000 7ffbc218b700 1 -- 172.21.15.44:0/1574006107 shutdown_connections 2023-11-16T16:46:32.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.854+0000 7ffbc218b700 1 --2- 172.21.15.44:0/1574006107 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbbc101f00 0x7ffbbc1087c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:32.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.854+0000 7ffbc218b700 1 --2- 172.21.15.44:0/1574006107 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffbbc100370 0x7ffbbc100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:32.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.854+0000 7ffbc218b700 1 --2- 172.21.15.44:0/1574006107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffbbc101570 0x7ffbbc1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:32.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.854+0000 7ffbc218b700 1 -- 172.21.15.44:0/1574006107 >> 172.21.15.44:0/1574006107 conn(0x7ffbbc0fb9a0 msgr2=0x7ffbbc0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:32.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.855+0000 7ffbc218b700 1 -- 172.21.15.44:0/1574006107 shutdown_connections 2023-11-16T16:46:32.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.855+0000 7ffbc218b700 1 -- 172.21.15.44:0/1574006107 wait complete. 2023-11-16T16:46:32.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.856+0000 7ffbc218b700 1 Processor -- start 2023-11-16T16:46:32.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.856+0000 7ffbc218b700 1 -- start start 2023-11-16T16:46:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.857+0000 7ffbc218b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffbbc100370 0x7ffbbc196130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.857+0000 7ffbc218b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbbc101570 0x7ffbbc196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.857+0000 7ffbc218b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffbbc101f00 0x7ffbbc19d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.857+0000 7ffbc218b700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc19dcc0 con 0x7ffbbc101570 2023-11-16T16:46:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.857+0000 7ffbc218b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc19de00 con 0x7ffbbc100370 2023-11-16T16:46:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.857+0000 7ffbc218b700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc19df90 con 0x7ffbbc101f00 2023-11-16T16:46:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.857+0000 7ffbbaffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbbc101570 0x7ffbbc196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.857+0000 7ffbbbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffbbc101f00 0x7ffbbc19d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.857+0000 7ffbbbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffbbc101f00 0x7ffbbc19d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:53782/0 (socket says 172.21.15.44:53782) 2023-11-16T16:46:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.857+0000 7ffbbbfff700 1 -- 172.21.15.44:0/2802522862 learned_addr learned my addr 172.21.15.44:0/2802522862 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:46:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.857+0000 7ffbbb7fe700 1 --2- 172.21.15.44:0/2802522862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffbbc100370 0x7ffbbc196130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.858+0000 7ffbbbfff700 1 -- 172.21.15.44:0/2802522862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffbbc100370 msgr2=0x7ffbbc196130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.858+0000 7ffbbbfff700 1 --2- 172.21.15.44:0/2802522862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffbbc100370 0x7ffbbc196130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.858+0000 7ffbbbfff700 1 -- 172.21.15.44:0/2802522862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbbc101570 msgr2=0x7ffbbc196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.858+0000 7ffbbbfff700 1 --2- 172.21.15.44:0/2802522862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbbc101570 0x7ffbbc196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.858+0000 7ffbbbfff700 1 -- 172.21.15.44:0/2802522862 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb4009d00 con 0x7ffbbc101f00 2023-11-16T16:46:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.858+0000 7ffbbb7fe700 1 --2- 172.21.15.44:0/2802522862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffbbc100370 0x7ffbbc196130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:46:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.858+0000 7ffbbaffd700 1 --2- 172.21.15.44:0/2802522862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbbc101570 0x7ffbbc196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:46:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.858+0000 7ffbbbfff700 1 --2- 172.21.15.44:0/2802522862 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffbbc101f00 0x7ffbbc19d6f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffbb400b070 tx=0x7ffbb400bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.858+0000 7ffbb8ff9700 1 -- 172.21.15.44:0/2802522862 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb400e040 con 0x7ffbbc101f00 2023-11-16T16:46:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.858+0000 7ffbc218b700 1 -- 172.21.15.44:0/2802522862 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbbc10bd10 con 0x7ffbbc101f00 2023-11-16T16:46:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.858+0000 7ffbb8ff9700 1 -- 172.21.15.44:0/2802522862 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffbb40094a0 con 0x7ffbbc101f00 2023-11-16T16:46:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.858+0000 7ffbb8ff9700 1 -- 172.21.15.44:0/2802522862 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb401b6c0 con 0x7ffbbc101f00 2023-11-16T16:46:32.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.858+0000 7ffbc218b700 1 -- 172.21.15.44:0/2802522862 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbbc10bb80 con 0x7ffbbc101f00 2023-11-16T16:46:32.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.859+0000 7ffbc218b700 1 -- 172.21.15.44:0/2802522862 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbbc19e580 con 0x7ffbbc101f00 2023-11-16T16:46:32.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.861+0000 7ffbb8ff9700 1 -- 172.21.15.44:0/2802522862 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ffbb401b820 con 0x7ffbbc101f00 2023-11-16T16:46:32.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.863+0000 7ffbb8ff9700 1 --2- 172.21.15.44:0/2802522862 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ffba40a11d0 0x7ffba40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.863+0000 7ffbb8ff9700 1 -- 172.21.15.44:0/2802522862 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ffbb40cd410 con 0x7ffbbc101f00 2023-11-16T16:46:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.863+0000 7ffbbb7fe700 1 --2- 172.21.15.44:0/2802522862 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ffba40a11d0 0x7ffba40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:32.865 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.863+0000 7ffbbb7fe700 1 --2- 172.21.15.44:0/2802522862 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ffba40a11d0 0x7ffba40a3680 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc1013d0 tx=0x7ffbac009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:32.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:32.865+0000 7ffbb8ff9700 1 -- 172.21.15.44:0/2802522862 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ffbb4096ff0 con 0x7ffbbc101f00 2023-11-16T16:46:33.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:33.070+0000 7ffbc218b700 1 -- 172.21.15.44:0/2802522862 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7ffbbc1072a0 con 0x7ffbbc101f00 2023-11-16T16:46:33.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:33.071+0000 7ffbb8ff9700 1 -- 172.21.15.44:0/2802522862 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) v1 ==== 74+0+133 (secure 0 0 0) 0x7ffbb409a610 con 0x7ffbbc101f00 2023-11-16T16:46:33.072 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:46:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:33.073+0000 7ffbc218b700 1 -- 172.21.15.44:0/2802522862 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ffba40a11d0 msgr2=0x7ffba40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:33.073+0000 7ffbc218b700 1 --2- 172.21.15.44:0/2802522862 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ffba40a11d0 0x7ffba40a3680 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc1013d0 tx=0x7ffbac009260 comp rx=0 tx=0).stop 2023-11-16T16:46:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:33.073+0000 7ffbc218b700 1 -- 172.21.15.44:0/2802522862 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffbbc101f00 msgr2=0x7ffbbc19d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:33.073+0000 7ffbc218b700 1 --2- 172.21.15.44:0/2802522862 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffbbc101f00 0x7ffbbc19d6f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffbb400b070 tx=0x7ffbb400bce0 comp rx=0 tx=0).stop 2023-11-16T16:46:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:33.073+0000 7ffbc218b700 1 -- 172.21.15.44:0/2802522862 shutdown_connections 2023-11-16T16:46:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:33.073+0000 7ffbc218b700 1 --2- 172.21.15.44:0/2802522862 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ffba40a11d0 0x7ffba40a3680 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:33.073+0000 7ffbc218b700 1 --2- 172.21.15.44:0/2802522862 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffbbc101f00 0x7ffbbc19d6f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:33.073+0000 7ffbc218b700 1 --2- 172.21.15.44:0/2802522862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffbbc100370 0x7ffbbc196130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:33.073+0000 7ffbc218b700 1 --2- 172.21.15.44:0/2802522862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbbc101570 0x7ffbbc196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:33.073+0000 7ffbc218b700 1 -- 172.21.15.44:0/2802522862 >> 172.21.15.44:0/2802522862 conn(0x7ffbbc0fb9a0 msgr2=0x7ffbbc0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:33.073+0000 7ffbc218b700 1 -- 172.21.15.44:0/2802522862 shutdown_connections 2023-11-16T16:46:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:33.073+0000 7ffbc218b700 1 -- 172.21.15.44:0/2802522862 wait complete. 2023-11-16T16:46:33.544 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":79,"num_osds":12,"num_up_osds":12,"osd_up_since":1700153188,"num_in_osds":12,"osd_in_since":1700153171,"num_remapped_pgs":0} 2023-11-16T16:46:33.544 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd dump --format=json 2023-11-16T16:46:33.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:33 smithi044 ceph-mon[128750]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:33.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:33 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2802522862' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T16:46:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:33 smithi176 ceph-mon[131630]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:33 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2802522862' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T16:46:33.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:33 smithi077 ceph-mon[131204]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:33.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:33 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2802522862' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T16:46:34.409 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:46:35.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:35 smithi044 ceph-mon[128750]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:35.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:35 smithi176 ceph-mon[131630]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:35.898 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.896+0000 7f1fff573700 1 -- 172.21.15.44:0/1466467958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff8100ee0 msgr2=0x7f1ff8104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:35.898 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.896+0000 7f1fff573700 1 --2- 172.21.15.44:0/1466467958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff8100ee0 0x7f1ff8104eb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80099c0 tx=0x7f1fe8009cd0 comp rx=0 tx=0).stop 2023-11-16T16:46:35.898 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.896+0000 7f1fff573700 1 -- 172.21.15.44:0/1466467958 shutdown_connections 2023-11-16T16:46:35.899 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.896+0000 7f1fff573700 1 --2- 172.21.15.44:0/1466467958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ff81055e0 0x7f1ff810d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:35.899 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.896+0000 7f1fff573700 1 --2- 172.21.15.44:0/1466467958 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ff8100540 0x7f1ff8100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:35.899 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.896+0000 7f1fff573700 1 --2- 172.21.15.44:0/1466467958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff8100ee0 0x7f1ff8104eb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:35.899 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.896+0000 7f1fff573700 1 -- 172.21.15.44:0/1466467958 >> 172.21.15.44:0/1466467958 conn(0x7f1ff80fbdc0 msgr2=0x7f1ff80fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:35.899 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.896+0000 7f1fff573700 1 -- 172.21.15.44:0/1466467958 shutdown_connections 2023-11-16T16:46:35.899 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.896+0000 7f1fff573700 1 -- 172.21.15.44:0/1466467958 wait complete. 2023-11-16T16:46:35.900 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.897+0000 7f1fff573700 1 Processor -- start 2023-11-16T16:46:35.901 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.898+0000 7f1fff573700 1 -- start start 2023-11-16T16:46:35.901 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.898+0000 7f1fff573700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ff8100540 0x7f1ff81963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:35.901 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.898+0000 7f1fff573700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff8100ee0 0x7f1ff8196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:35.901 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.898+0000 7f1fff573700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ff81055e0 0x7f1ff819ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:35.901 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.898+0000 7f1fff573700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff819b390 con 0x7f1ff8100ee0 2023-11-16T16:46:35.901 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.898+0000 7f1fff573700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff819b500 con 0x7f1ff8100540 2023-11-16T16:46:35.901 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.898+0000 7f1fff573700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff819b6c0 con 0x7f1ff81055e0 2023-11-16T16:46:35.901 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.898+0000 7f1ffcb0e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff8100ee0 0x7f1ff8196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:35.901 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.898+0000 7f1ffcb0e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff8100ee0 0x7f1ff8196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57836/0 (socket says 172.21.15.44:57836) 2023-11-16T16:46:35.901 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.898+0000 7f1ffcb0e700 1 -- 172.21.15.44:0/1297177190 learned_addr learned my addr 172.21.15.44:0/1297177190 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:46:35.901 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.898+0000 7f1ffdb10700 1 --2- 172.21.15.44:0/1297177190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ff81055e0 0x7f1ff819ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:35.901 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.898+0000 7f1ffd30f700 1 --2- 172.21.15.44:0/1297177190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ff8100540 0x7f1ff81963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:35.902 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.899+0000 7f1ffcb0e700 1 -- 172.21.15.44:0/1297177190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ff81055e0 msgr2=0x7f1ff819ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:35.902 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.899+0000 7f1ffcb0e700 1 --2- 172.21.15.44:0/1297177190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ff81055e0 0x7f1ff819ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:35.902 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.899+0000 7f1ffcb0e700 1 -- 172.21.15.44:0/1297177190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ff8100540 msgr2=0x7f1ff81963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:35.902 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.899+0000 7f1ffcb0e700 1 --2- 172.21.15.44:0/1297177190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ff8100540 0x7f1ff81963d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:35.902 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.899+0000 7f1ffcb0e700 1 -- 172.21.15.44:0/1297177190 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe80095f0 con 0x7f1ff8100ee0 2023-11-16T16:46:35.902 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.899+0000 7f1ffcb0e700 1 --2- 172.21.15.44:0/1297177190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff8100ee0 0x7f1ff8196910 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80098e0 tx=0x7f1fe800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:35.903 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.899+0000 7f1fee7fc700 1 -- 172.21.15.44:0/1297177190 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe801d070 con 0x7f1ff8100ee0 2023-11-16T16:46:35.903 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.899+0000 7f1fee7fc700 1 -- 172.21.15.44:0/1297177190 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1fe8022470 con 0x7f1ff8100ee0 2023-11-16T16:46:35.903 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.899+0000 7f1fff573700 1 -- 172.21.15.44:0/1297177190 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff8110360 con 0x7f1ff8100ee0 2023-11-16T16:46:35.903 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.899+0000 7f1fee7fc700 1 -- 172.21.15.44:0/1297177190 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8017650 con 0x7f1ff8100ee0 2023-11-16T16:46:35.903 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.899+0000 7f1fff573700 1 -- 172.21.15.44:0/1297177190 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff81101d0 con 0x7f1ff8100ee0 2023-11-16T16:46:35.903 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.900+0000 7f1fff573700 1 -- 172.21.15.44:0/1297177190 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff81104f0 con 0x7f1ff8100ee0 2023-11-16T16:46:35.905 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.904+0000 7f1fee7fc700 1 -- 172.21.15.44:0/1297177190 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1fe8022850 con 0x7f1ff8100ee0 2023-11-16T16:46:35.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.905+0000 7f1fee7fc700 1 --2- 172.21.15.44:0/1297177190 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1fe40a1150 0x7f1fe40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:35.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.905+0000 7f1fee7fc700 1 -- 172.21.15.44:0/1297177190 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1fe80cd4e0 con 0x7f1ff8100ee0 2023-11-16T16:46:35.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.905+0000 7f1ffd30f700 1 --2- 172.21.15.44:0/1297177190 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1fe40a1150 0x7f1fe40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:35.908 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.906+0000 7f1ffd30f700 1 --2- 172.21.15.44:0/1297177190 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1fe40a1150 0x7f1fe40a3600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009c60 tx=0x7f1ff4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:35.909 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:35.908+0000 7f1fee7fc700 1 -- 172.21.15.44:0/1297177190 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1fe8097150 con 0x7f1ff8100ee0 2023-11-16T16:46:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:35 smithi077 ceph-mon[131204]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:36.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:36.114+0000 7f1fff573700 1 -- 172.21.15.44:0/1297177190 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ff81a3450 con 0x7f1ff8100ee0 2023-11-16T16:46:36.117 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:36.115+0000 7f1fee7fc700 1 -- 172.21.15.44:0/1297177190 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18387 (secure 0 0 0) 0x7f1fe809a770 con 0x7f1ff8100ee0 2023-11-16T16:46:36.117 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:46:36.117 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":79,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:46:30.119708+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:46:36.119 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:36.117+0000 7f1fff573700 1 -- 172.21.15.44:0/1297177190 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1fe40a1150 msgr2=0x7f1fe40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:36.119 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:36.117+0000 7f1fff573700 1 --2- 172.21.15.44:0/1297177190 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1fe40a1150 0x7f1fe40a3600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009c60 tx=0x7f1ff4009260 comp rx=0 tx=0).stop 2023-11-16T16:46:36.119 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:36.117+0000 7f1fff573700 1 -- 172.21.15.44:0/1297177190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff8100ee0 msgr2=0x7f1ff8196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:36.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:36.117+0000 7f1fff573700 1 --2- 172.21.15.44:0/1297177190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff8100ee0 0x7f1ff8196910 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80098e0 tx=0x7f1fe800f690 comp rx=0 tx=0).stop 2023-11-16T16:46:36.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:36.117+0000 7f1fff573700 1 -- 172.21.15.44:0/1297177190 shutdown_connections 2023-11-16T16:46:36.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:36.117+0000 7f1fff573700 1 --2- 172.21.15.44:0/1297177190 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1fe40a1150 0x7f1fe40a3600 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:36.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:36.117+0000 7f1fff573700 1 --2- 172.21.15.44:0/1297177190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ff81055e0 0x7f1ff819ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:36.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:36.117+0000 7f1fff573700 1 --2- 172.21.15.44:0/1297177190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ff8100540 0x7f1ff81963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:36.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:36.117+0000 7f1fff573700 1 --2- 172.21.15.44:0/1297177190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff8100ee0 0x7f1ff8196910 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:36.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:36.118+0000 7f1fff573700 1 -- 172.21.15.44:0/1297177190 >> 172.21.15.44:0/1297177190 conn(0x7f1ff80fbdc0 msgr2=0x7f1ff80fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:36.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:36.118+0000 7f1fff573700 1 -- 172.21.15.44:0/1297177190 shutdown_connections 2023-11-16T16:46:36.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:36.118+0000 7f1fff573700 1 -- 172.21.15.44:0/1297177190 wait complete. 2023-11-16T16:46:36.595 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:03.059622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-16T16:46:36.595 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd pool get .mgr pg_num 2023-11-16T16:46:36.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:36 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1297177190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:46:36.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:36 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1297177190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:46:36.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:36 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1297177190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:46:37.458 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:46:37.856 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:37 smithi044 ceph-mon[128750]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:37.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:37 smithi176 ceph-mon[131630]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:37.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:37 smithi077 ceph-mon[131204]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:38.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.950+0000 7fba8c60e700 1 -- 172.21.15.44:0/501053595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba841055e0 msgr2=0x7fba8410d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:38.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.950+0000 7fba8c60e700 1 --2- 172.21.15.44:0/501053595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba841055e0 0x7fba8410d160 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fba7c00b0d0 tx=0x7fba7c00b3e0 comp rx=0 tx=0).stop 2023-11-16T16:46:38.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.950+0000 7fba8c60e700 1 -- 172.21.15.44:0/501053595 shutdown_connections 2023-11-16T16:46:38.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.950+0000 7fba8c60e700 1 --2- 172.21.15.44:0/501053595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba841055e0 0x7fba8410d160 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:38.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.950+0000 7fba8c60e700 1 --2- 172.21.15.44:0/501053595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba84100540 0x7fba84100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:38.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.950+0000 7fba8c60e700 1 --2- 172.21.15.44:0/501053595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba84100ee0 0x7fba84104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:38.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.950+0000 7fba8c60e700 1 -- 172.21.15.44:0/501053595 >> 172.21.15.44:0/501053595 conn(0x7fba840fbdc0 msgr2=0x7fba840fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:38.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.950+0000 7fba8c60e700 1 -- 172.21.15.44:0/501053595 shutdown_connections 2023-11-16T16:46:38.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.950+0000 7fba8c60e700 1 -- 172.21.15.44:0/501053595 wait complete. 2023-11-16T16:46:38.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.951+0000 7fba8c60e700 1 Processor -- start 2023-11-16T16:46:38.953 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.951+0000 7fba8c60e700 1 -- start start 2023-11-16T16:46:38.953 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.951+0000 7fba8c60e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba84100540 0x7fba84196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:38.953 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.951+0000 7fba8c60e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba84100ee0 0x7fba84196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:38.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.951+0000 7fba8c60e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba841055e0 0x7fba8419aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:38.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.951+0000 7fba8c60e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8419b500 con 0x7fba841055e0 2023-11-16T16:46:38.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.951+0000 7fba8c60e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8419b670 con 0x7fba84100ee0 2023-11-16T16:46:38.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.951+0000 7fba8c60e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8419b830 con 0x7fba84100540 2023-11-16T16:46:38.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.952+0000 7fba8abab700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba841055e0 0x7fba8419aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:38.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.952+0000 7fba8abab700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba841055e0 0x7fba8419aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57852/0 (socket says 172.21.15.44:57852) 2023-11-16T16:46:38.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.952+0000 7fba8abab700 1 -- 172.21.15.44:0/1091952880 learned_addr learned my addr 172.21.15.44:0/1091952880 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:46:38.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.952+0000 7fba8a3aa700 1 --2- 172.21.15.44:0/1091952880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba84100540 0x7fba84196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:38.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.952+0000 7fba89ba9700 1 --2- 172.21.15.44:0/1091952880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba84100ee0 0x7fba84196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:38.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.952+0000 7fba8abab700 1 -- 172.21.15.44:0/1091952880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba84100540 msgr2=0x7fba84196540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:38.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.952+0000 7fba8abab700 1 --2- 172.21.15.44:0/1091952880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba84100540 0x7fba84196540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:38.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.952+0000 7fba8abab700 1 -- 172.21.15.44:0/1091952880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba84100ee0 msgr2=0x7fba84196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:38.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.952+0000 7fba8abab700 1 --2- 172.21.15.44:0/1091952880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba84100ee0 0x7fba84196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:38.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.952+0000 7fba8abab700 1 -- 172.21.15.44:0/1091952880 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7c009d00 con 0x7fba841055e0 2023-11-16T16:46:38.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.952+0000 7fba8abab700 1 --2- 172.21.15.44:0/1091952880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba841055e0 0x7fba8419aef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fba7c007820 tx=0x7fba7c007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:38.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.953+0000 7fba7b7fe700 1 -- 172.21.15.44:0/1091952880 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7c00e040 con 0x7fba841055e0 2023-11-16T16:46:38.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.953+0000 7fba7b7fe700 1 -- 172.21.15.44:0/1091952880 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba7c00bcb0 con 0x7fba841055e0 2023-11-16T16:46:38.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.953+0000 7fba7b7fe700 1 -- 172.21.15.44:0/1091952880 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7c0058a0 con 0x7fba841055e0 2023-11-16T16:46:38.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.953+0000 7fba8c60e700 1 -- 172.21.15.44:0/1091952880 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba84110370 con 0x7fba841055e0 2023-11-16T16:46:38.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.953+0000 7fba8c60e700 1 -- 172.21.15.44:0/1091952880 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba841101e0 con 0x7fba841055e0 2023-11-16T16:46:38.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.954+0000 7fba8c60e700 1 -- 172.21.15.44:0/1091952880 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba84110500 con 0x7fba841055e0 2023-11-16T16:46:38.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.957+0000 7fba7b7fe700 1 -- 172.21.15.44:0/1091952880 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fba7c004a50 con 0x7fba841055e0 2023-11-16T16:46:38.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.959+0000 7fba7b7fe700 1 --2- 172.21.15.44:0/1091952880 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba700a1150 0x7fba700a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:38.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.959+0000 7fba7b7fe700 1 -- 172.21.15.44:0/1091952880 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fba7c0cd230 con 0x7fba841055e0 2023-11-16T16:46:38.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.959+0000 7fba8a3aa700 1 --2- 172.21.15.44:0/1091952880 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba700a1150 0x7fba700a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:38.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.960+0000 7fba8a3aa700 1 --2- 172.21.15.44:0/1091952880 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba700a1150 0x7fba700a3600 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fba80009c60 tx=0x7fba80009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:38.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:38.963+0000 7fba7b7fe700 1 -- 172.21.15.44:0/1091952880 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fba7c096eb0 con 0x7fba841055e0 2023-11-16T16:46:39.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:39.173+0000 7fba8c60e700 1 -- 172.21.15.44:0/1091952880 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fba841a35c0 con 0x7fba841055e0 2023-11-16T16:46:39.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:39.173+0000 7fba7b7fe700 1 -- 172.21.15.44:0/1091952880 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7fba7c09a4d0 con 0x7fba841055e0 2023-11-16T16:46:39.175 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2023-11-16T16:46:39.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:39.176+0000 7fba8c60e700 1 -- 172.21.15.44:0/1091952880 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba700a1150 msgr2=0x7fba700a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:39.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:39.176+0000 7fba8c60e700 1 --2- 172.21.15.44:0/1091952880 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba700a1150 0x7fba700a3600 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fba80009c60 tx=0x7fba80009260 comp rx=0 tx=0).stop 2023-11-16T16:46:39.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:39.176+0000 7fba8c60e700 1 -- 172.21.15.44:0/1091952880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba841055e0 msgr2=0x7fba8419aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:39.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:39.176+0000 7fba8c60e700 1 --2- 172.21.15.44:0/1091952880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba841055e0 0x7fba8419aef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fba7c007820 tx=0x7fba7c007850 comp rx=0 tx=0).stop 2023-11-16T16:46:39.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:39.176+0000 7fba8c60e700 1 -- 172.21.15.44:0/1091952880 shutdown_connections 2023-11-16T16:46:39.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:39.176+0000 7fba8c60e700 1 --2- 172.21.15.44:0/1091952880 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba700a1150 0x7fba700a3600 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:39.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:39.176+0000 7fba8c60e700 1 --2- 172.21.15.44:0/1091952880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba841055e0 0x7fba8419aef0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:39.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:39.176+0000 7fba8c60e700 1 --2- 172.21.15.44:0/1091952880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba84100540 0x7fba84196540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:39.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:39.176+0000 7fba8c60e700 1 --2- 172.21.15.44:0/1091952880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba84100ee0 0x7fba84196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:39.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:39.176+0000 7fba8c60e700 1 -- 172.21.15.44:0/1091952880 >> 172.21.15.44:0/1091952880 conn(0x7fba840fbdc0 msgr2=0x7fba840fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:39.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:39.176+0000 7fba8c60e700 1 -- 172.21.15.44:0/1091952880 shutdown_connections 2023-11-16T16:46:39.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:39.176+0000 7fba8c60e700 1 -- 172.21.15.44:0/1091952880 wait complete. 2023-11-16T16:46:39.394 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:39 smithi176 ceph-mon[131630]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:39.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:39 smithi077 ceph-mon[131204]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:39.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:39 smithi044 ceph-mon[128750]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:39.641 INFO:tasks.cephadm:Adding mds.a on smithi044 2023-11-16T16:46:39.641 INFO:tasks.cephadm:Adding mds.d on smithi044 2023-11-16T16:46:39.641 INFO:tasks.cephadm:Adding mds.g on smithi044 2023-11-16T16:46:39.641 INFO:tasks.cephadm:Adding mds.j on smithi044 2023-11-16T16:46:39.641 INFO:tasks.cephadm:Adding mds.b on smithi077 2023-11-16T16:46:39.641 INFO:tasks.cephadm:Adding mds.e on smithi077 2023-11-16T16:46:39.642 INFO:tasks.cephadm:Adding mds.h on smithi077 2023-11-16T16:46:39.642 INFO:tasks.cephadm:Adding mds.k on smithi077 2023-11-16T16:46:39.642 INFO:tasks.cephadm:Adding mds.c on smithi176 2023-11-16T16:46:39.642 INFO:tasks.cephadm:Adding mds.f on smithi176 2023-11-16T16:46:39.642 INFO:tasks.cephadm:Adding mds.i on smithi176 2023-11-16T16:46:39.642 INFO:tasks.cephadm:Adding mds.l on smithi176 2023-11-16T16:46:39.642 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph orch apply mds all '12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi176=c;smithi176=f;smithi176=i;smithi176=l' 2023-11-16T16:46:40.295 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:46:40.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:40 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1091952880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:46:40.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:40 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1091952880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:46:40.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:40 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1091952880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:46:41.394 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:41 smithi176 ceph-mon[131630]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:41.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:41 smithi077 ceph-mon[131204]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:41.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:41 smithi044 ceph-mon[128750]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:41.681 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.677+0000 7fb2686e4700 1 -- 172.21.15.176:0/4160170967 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2601003e0 msgr2=0x7fb260100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:41.682 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.677+0000 7fb2686e4700 1 --2- 172.21.15.176:0/4160170967 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2601003e0 0x7fb260100830 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb250009990 tx=0x7fb250009ca0 comp rx=0 tx=0).stop 2023-11-16T16:46:41.682 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.678+0000 7fb2686e4700 1 -- 172.21.15.176:0/4160170967 shutdown_connections 2023-11-16T16:46:41.682 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.678+0000 7fb2686e4700 1 --2- 172.21.15.176:0/4160170967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb260100d70 0x7fb260107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:41.682 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.678+0000 7fb2686e4700 1 --2- 172.21.15.176:0/4160170967 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2600ff1e0 0x7fb2600ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:41.682 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.678+0000 7fb2686e4700 1 --2- 172.21.15.176:0/4160170967 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2601003e0 0x7fb260100830 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:41.682 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.678+0000 7fb2686e4700 1 -- 172.21.15.176:0/4160170967 >> 172.21.15.176:0/4160170967 conn(0x7fb2600fa7f0 msgr2=0x7fb2600fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:41.682 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.678+0000 7fb2686e4700 1 -- 172.21.15.176:0/4160170967 shutdown_connections 2023-11-16T16:46:41.682 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.678+0000 7fb2686e4700 1 -- 172.21.15.176:0/4160170967 wait complete. 2023-11-16T16:46:41.682 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.678+0000 7fb2686e4700 1 Processor -- start 2023-11-16T16:46:41.682 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.678+0000 7fb2686e4700 1 -- start start 2023-11-16T16:46:41.682 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.678+0000 7fb2686e4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2600ff1e0 0x7fb2600699a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:41.682 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.678+0000 7fb2686e4700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2601003e0 0x7fb26019ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:41.682 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.678+0000 7fb2686e4700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb260100d70 0x7fb26019d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:41.683 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.678+0000 7fb2686e4700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26019dae0 con 0x7fb2601003e0 2023-11-16T16:46:41.683 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.678+0000 7fb2686e4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26019dc50 con 0x7fb2600ff1e0 2023-11-16T16:46:41.683 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.678+0000 7fb2686e4700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26019de10 con 0x7fb260100d70 2023-11-16T16:46:41.683 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.679+0000 7fb266c81700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb260100d70 0x7fb26019d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:41.683 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.679+0000 7fb266c81700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb260100d70 0x7fb26019d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38264/0 (socket says 172.21.15.176:38264) 2023-11-16T16:46:41.683 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.679+0000 7fb266c81700 1 -- 172.21.15.176:0/1802155568 learned_addr learned my addr 172.21.15.176:0/1802155568 (peer_addr_for_me v2:172.21.15.176:0/0) 2023-11-16T16:46:41.683 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.679+0000 7fb266480700 1 --2- 172.21.15.176:0/1802155568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2600ff1e0 0x7fb2600699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:41.683 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.679+0000 7fb265c7f700 1 --2- 172.21.15.176:0/1802155568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2601003e0 0x7fb26019ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:41.683 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.679+0000 7fb266c81700 1 -- 172.21.15.176:0/1802155568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2600ff1e0 msgr2=0x7fb2600699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:41.683 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.679+0000 7fb266c81700 1 --2- 172.21.15.176:0/1802155568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2600ff1e0 0x7fb2600699a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:41.683 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.679+0000 7fb266c81700 1 -- 172.21.15.176:0/1802155568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2601003e0 msgr2=0x7fb26019ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:41.683 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.679+0000 7fb266c81700 1 --2- 172.21.15.176:0/1802155568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2601003e0 0x7fb26019ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:41.683 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.679+0000 7fb266c81700 1 -- 172.21.15.176:0/1802155568 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2500095f0 con 0x7fb260100d70 2023-11-16T16:46:41.683 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.679+0000 7fb266c81700 1 --2- 172.21.15.176:0/1802155568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb260100d70 0x7fb26019d340 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb258007ee0 tx=0x7fb25800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:41.684 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.679+0000 7fb2577fe700 1 -- 172.21.15.176:0/1802155568 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb258017070 con 0x7fb260100d70 2023-11-16T16:46:41.684 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.679+0000 7fb2577fe700 1 -- 172.21.15.176:0/1802155568 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb25800f040 con 0x7fb260100d70 2023-11-16T16:46:41.684 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.679+0000 7fb2686e4700 1 -- 172.21.15.176:0/1802155568 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26010ab40 con 0x7fb260100d70 2023-11-16T16:46:41.684 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.679+0000 7fb2577fe700 1 -- 172.21.15.176:0/1802155568 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb258005740 con 0x7fb260100d70 2023-11-16T16:46:41.684 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.679+0000 7fb2686e4700 1 -- 172.21.15.176:0/1802155568 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb26010a9b0 con 0x7fb260100d70 2023-11-16T16:46:41.685 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.680+0000 7fb2686e4700 1 -- 172.21.15.176:0/1802155568 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb260106790 con 0x7fb260100d70 2023-11-16T16:46:41.685 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.681+0000 7fb2577fe700 1 -- 172.21.15.176:0/1802155568 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb25800dc60 con 0x7fb260100d70 2023-11-16T16:46:41.685 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.682+0000 7fb2577fe700 1 --2- 172.21.15.176:0/1802155568 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb24c0a11d0 0x7fb24c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:41.685 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.682+0000 7fb2577fe700 1 -- 172.21.15.176:0/1802155568 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb2580cc250 con 0x7fb260100d70 2023-11-16T16:46:41.685 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.682+0000 7fb266480700 1 --2- 172.21.15.176:0/1802155568 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb24c0a11d0 0x7fb24c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:41.685 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.683+0000 7fb266480700 1 --2- 172.21.15.176:0/1802155568 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb24c0a11d0 0x7fb24c0a3680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb260100240 tx=0x7fb25c00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:41.687 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.685+0000 7fb2577fe700 1 -- 172.21.15.176:0/1802155568 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb258095e40 con 0x7fb260100d70 2023-11-16T16:46:41.881 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.879+0000 7fb2686e4700 1 -- 172.21.15.176:0/1802155568 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi176=c;smithi176=f;smithi176=i;smithi176=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fb260061d70 con 0x7fb24c0a11d0 2023-11-16T16:46:41.890 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.888+0000 7fb2577fe700 1 -- 172.21.15.176:0/1802155568 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fb260061d70 con 0x7fb24c0a11d0 2023-11-16T16:46:41.890 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled mds.all update... 2023-11-16T16:46:41.892 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.890+0000 7fb2686e4700 1 -- 172.21.15.176:0/1802155568 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb24c0a11d0 msgr2=0x7fb24c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:41.892 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.890+0000 7fb2686e4700 1 --2- 172.21.15.176:0/1802155568 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb24c0a11d0 0x7fb24c0a3680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb260100240 tx=0x7fb25c00a380 comp rx=0 tx=0).stop 2023-11-16T16:46:41.892 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.890+0000 7fb2686e4700 1 -- 172.21.15.176:0/1802155568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb260100d70 msgr2=0x7fb26019d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:41.892 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.890+0000 7fb2686e4700 1 --2- 172.21.15.176:0/1802155568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb260100d70 0x7fb26019d340 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb258007ee0 tx=0x7fb25800d3b0 comp rx=0 tx=0).stop 2023-11-16T16:46:41.892 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.890+0000 7fb2686e4700 1 -- 172.21.15.176:0/1802155568 shutdown_connections 2023-11-16T16:46:41.893 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.890+0000 7fb2686e4700 1 --2- 172.21.15.176:0/1802155568 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb24c0a11d0 0x7fb24c0a3680 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:41.893 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.890+0000 7fb2686e4700 1 --2- 172.21.15.176:0/1802155568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb260100d70 0x7fb26019d340 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:41.893 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.890+0000 7fb2686e4700 1 --2- 172.21.15.176:0/1802155568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2600ff1e0 0x7fb2600699a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:41.893 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.890+0000 7fb2686e4700 1 --2- 172.21.15.176:0/1802155568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2601003e0 0x7fb26019ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:41.893 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.890+0000 7fb2686e4700 1 -- 172.21.15.176:0/1802155568 >> 172.21.15.176:0/1802155568 conn(0x7fb2600fa7f0 msgr2=0x7fb2601038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:41.893 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.890+0000 7fb2686e4700 1 -- 172.21.15.176:0/1802155568 shutdown_connections 2023-11-16T16:46:41.893 INFO:teuthology.orchestra.run.smithi176.stderr:2023-11-16T16:46:41.890+0000 7fb2686e4700 1 -- 172.21.15.176:0/1802155568 wait complete. 2023-11-16T16:46:42.341 DEBUG:teuthology.orchestra.run.smithi044:mds.a> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.a.service 2023-11-16T16:46:44.303 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:42 smithi044 ceph-mon[128750]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi176=c;smithi176=f;smithi176=i;smithi176=l", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:46:44.303 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:42 smithi044 ceph-mon[128750]: Saving service mds.all spec with placement smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi176=c;smithi176=f;smithi176=i;smithi176=l;count:12 2023-11-16T16:46:44.303 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:44.303 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:44.304 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:44.304 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:44.304 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:42 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:46:44.304 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:44.304 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:42 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:46:44.304 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:42 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:46:44.304 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:42 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T16:46:44.304 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:44.304 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:44.304 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:44.304 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:42 smithi044 ceph-mon[128750]: Deploying daemon mds.l on smithi176 2023-11-16T16:46:44.304 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:42 smithi044 ceph-mon[128750]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:44.305 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:42 smithi077 ceph-mon[131204]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi176=c;smithi176=f;smithi176=i;smithi176=l", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:46:44.305 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:42 smithi077 ceph-mon[131204]: Saving service mds.all spec with placement smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi176=c;smithi176=f;smithi176=i;smithi176=l;count:12 2023-11-16T16:46:44.305 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:44.305 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:44.305 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:44.305 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:44.305 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:42 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:46:44.305 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:44.306 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:42 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:46:44.306 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:42 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:46:44.306 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:42 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T16:46:44.306 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:44.306 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:44.306 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:44.306 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:42 smithi077 ceph-mon[131204]: Deploying daemon mds.l on smithi176 2023-11-16T16:46:44.306 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:42 smithi077 ceph-mon[131204]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:44.306 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:42 smithi176 ceph-mon[131630]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi176=c;smithi176=f;smithi176=i;smithi176=l", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:46:44.307 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:42 smithi176 ceph-mon[131630]: Saving service mds.all spec with placement smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi176=c;smithi176=f;smithi176=i;smithi176=l;count:12 2023-11-16T16:46:44.307 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:44.307 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:44.307 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:44.307 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:44.307 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:42 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:46:44.307 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:44.307 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:42 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:46:44.307 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:42 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:46:44.307 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:42 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T16:46:44.307 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:44.308 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:44.308 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:44.308 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:42 smithi176 ceph-mon[131630]: Deploying daemon mds.l on smithi176 2023-11-16T16:46:44.308 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:42 smithi176 ceph-mon[131630]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:44.309 DEBUG:teuthology.orchestra.run.smithi044:mds.d> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.d.service 2023-11-16T16:46:44.354 DEBUG:teuthology.orchestra.run.smithi044:mds.g> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.g.service 2023-11-16T16:46:44.391 INFO:journalctl@ceph.mds.a.smithi044.stdout:-- Logs begin at Thu 2023-11-16 16:19:45 UTC. -- 2023-11-16T16:46:44.395 DEBUG:teuthology.orchestra.run.smithi044:mds.j> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.j.service 2023-11-16T16:46:44.432 INFO:journalctl@ceph.mds.d.smithi044.stdout:-- Logs begin at Thu 2023-11-16 16:19:45 UTC. -- 2023-11-16T16:46:44.435 DEBUG:teuthology.orchestra.run.smithi077:mds.b> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.b.service 2023-11-16T16:46:44.438 DEBUG:teuthology.orchestra.run.smithi077:mds.e> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.e.service 2023-11-16T16:46:44.470 INFO:journalctl@ceph.mds.g.smithi044.stdout:-- Logs begin at Thu 2023-11-16 16:19:45 UTC. -- 2023-11-16T16:46:44.474 INFO:journalctl@ceph.mds.b.smithi077.stdout:-- Logs begin at Thu 2023-11-16 16:15:33 UTC. -- 2023-11-16T16:46:44.476 DEBUG:teuthology.orchestra.run.smithi077:mds.h> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.h.service 2023-11-16T16:46:44.505 INFO:journalctl@ceph.mds.j.smithi044.stdout:-- Logs begin at Thu 2023-11-16 16:19:45 UTC. -- 2023-11-16T16:46:44.505 INFO:journalctl@ceph.mds.e.smithi077.stdout:-- Logs begin at Thu 2023-11-16 16:15:33 UTC. -- 2023-11-16T16:46:44.506 DEBUG:teuthology.orchestra.run.smithi077:mds.k> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.k.service 2023-11-16T16:46:44.533 INFO:journalctl@ceph.mds.h.smithi077.stdout:-- Logs begin at Thu 2023-11-16 16:15:33 UTC. -- 2023-11-16T16:46:44.535 DEBUG:teuthology.orchestra.run.smithi176:mds.c> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.c.service 2023-11-16T16:46:44.537 DEBUG:teuthology.orchestra.run.smithi176:mds.f> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.f.service 2023-11-16T16:46:44.562 INFO:journalctl@ceph.mds.k.smithi077.stdout:-- Logs begin at Thu 2023-11-16 16:15:33 UTC. -- 2023-11-16T16:46:44.568 INFO:journalctl@ceph.mds.c.smithi176.stdout:-- Logs begin at Thu 2023-11-16 16:14:40 UTC. -- 2023-11-16T16:46:44.569 DEBUG:teuthology.orchestra.run.smithi176:mds.i> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.i.service 2023-11-16T16:46:44.595 INFO:journalctl@ceph.mds.f.smithi176.stdout:-- Logs begin at Thu 2023-11-16 16:14:40 UTC. -- 2023-11-16T16:46:44.596 DEBUG:teuthology.orchestra.run.smithi176:mds.l> sudo journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.l.service 2023-11-16T16:46:44.621 INFO:journalctl@ceph.mds.i.smithi176.stdout:-- Logs begin at Thu 2023-11-16 16:14:40 UTC. -- 2023-11-16T16:46:44.622 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-11-16T16:46:44.623 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:46:44.648 INFO:journalctl@ceph.mds.l.smithi176.stdout:-- Logs begin at Thu 2023-11-16 16:14:40 UTC. -- 2023-11-16T16:46:44.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.840+0000 7f723efff700 1 -- 172.21.15.44:0/1864108342 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f723810c860 msgr2=0x7f723810ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:44.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.840+0000 7f723efff700 1 --2- 172.21.15.44:0/1864108342 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f723810c860 0x7f723810ec40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7230009f90 tx=0x7f723000b330 comp rx=0 tx=0).stop 2023-11-16T16:46:44.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.841+0000 7f723efff700 1 -- 172.21.15.44:0/1864108342 shutdown_connections 2023-11-16T16:46:44.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.841+0000 7f723efff700 1 --2- 172.21.15.44:0/1864108342 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f723810c860 0x7f723810ec40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:44.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.841+0000 7f723efff700 1 --2- 172.21.15.44:0/1864108342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7238074b30 0x7f7238072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:44.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.841+0000 7f723efff700 1 --2- 172.21.15.44:0/1864108342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7238073550 0x7f72380739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:44.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.841+0000 7f723efff700 1 -- 172.21.15.44:0/1864108342 >> 172.21.15.44:0/1864108342 conn(0x7f72380fb870 msgr2=0x7f72380fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:44.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.841+0000 7f723efff700 1 -- 172.21.15.44:0/1864108342 shutdown_connections 2023-11-16T16:46:44.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.841+0000 7f723efff700 1 -- 172.21.15.44:0/1864108342 wait complete. 2023-11-16T16:46:44.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.842+0000 7f723efff700 1 Processor -- start 2023-11-16T16:46:44.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.842+0000 7f723efff700 1 -- start start 2023-11-16T16:46:44.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.842+0000 7f723efff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7238073550 0x7f723819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:44.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.842+0000 7f723efff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7238074b30 0x7f723819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:44.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.842+0000 7f723efff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f723810c860 0x7f72381a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:44.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.842+0000 7f723efff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72381a4210 con 0x7f7238074b30 2023-11-16T16:46:44.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.842+0000 7f723efff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72381a4350 con 0x7f7238073550 2023-11-16T16:46:44.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.842+0000 7f723efff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72381a44e0 con 0x7f723810c860 2023-11-16T16:46:44.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.842+0000 7f723cd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7238073550 0x7f723819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:44.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.842+0000 7f7237fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7238074b30 0x7f723819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:44.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.842+0000 7f723cd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7238073550 0x7f723819c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:51038/0 (socket says 172.21.15.44:51038) 2023-11-16T16:46:44.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.842+0000 7f723cd9b700 1 -- 172.21.15.44:0/159175648 learned_addr learned my addr 172.21.15.44:0/159175648 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:46:44.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.843+0000 7f723d59c700 1 --2- 172.21.15.44:0/159175648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f723810c860 0x7f72381a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:44.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.843+0000 7f723cd9b700 1 -- 172.21.15.44:0/159175648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f723810c860 msgr2=0x7f72381a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:44.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.843+0000 7f723cd9b700 1 --2- 172.21.15.44:0/159175648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f723810c860 0x7f72381a3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:44.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.843+0000 7f723cd9b700 1 -- 172.21.15.44:0/159175648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7238074b30 msgr2=0x7f723819cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:44.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.843+0000 7f723cd9b700 1 --2- 172.21.15.44:0/159175648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7238074b30 0x7f723819cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:44.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.843+0000 7f723cd9b700 1 -- 172.21.15.44:0/159175648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7230009c40 con 0x7f7238073550 2023-11-16T16:46:44.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.843+0000 7f723cd9b700 1 --2- 172.21.15.44:0/159175648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7238073550 0x7f723819c690 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f722c00e9f0 tx=0x7f722c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:44.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.843+0000 7f7235ffb700 1 -- 172.21.15.44:0/159175648 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c009690 con 0x7f7238073550 2023-11-16T16:46:44.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.843+0000 7f7235ffb700 1 -- 172.21.15.44:0/159175648 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f722c00ce80 con 0x7f7238073550 2023-11-16T16:46:44.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.843+0000 7f723efff700 1 -- 172.21.15.44:0/159175648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7238112270 con 0x7f7238073550 2023-11-16T16:46:44.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.843+0000 7f7235ffb700 1 -- 172.21.15.44:0/159175648 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c018680 con 0x7f7238073550 2023-11-16T16:46:44.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.843+0000 7f723efff700 1 -- 172.21.15.44:0/159175648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72381120e0 con 0x7f7238073550 2023-11-16T16:46:44.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.844+0000 7f723efff700 1 -- 172.21.15.44:0/159175648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72381023b0 con 0x7f7238073550 2023-11-16T16:46:44.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.846+0000 7f7235ffb700 1 -- 172.21.15.44:0/159175648 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f722c010430 con 0x7f7238073550 2023-11-16T16:46:44.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.848+0000 7f7235ffb700 1 --2- 172.21.15.44:0/159175648 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f72280a1150 0x7f72280a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:44.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.848+0000 7f7235ffb700 1 -- 172.21.15.44:0/159175648 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f722c014070 con 0x7f7238073550 2023-11-16T16:46:44.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.848+0000 7f7237fff700 1 --2- 172.21.15.44:0/159175648 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f72280a1150 0x7f72280a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:44.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.849+0000 7f7237fff700 1 --2- 172.21.15.44:0/159175648 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f72280a1150 0x7f72280a3600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7238074850 tx=0x7f7224009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:44.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:44.850+0000 7f7235ffb700 1 -- 172.21.15.44:0/159175648 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f722c096830 con 0x7f7238073550 2023-11-16T16:46:45.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.059+0000 7f723efff700 1 -- 172.21.15.44:0/159175648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f723802cfe0 con 0x7f7238073550 2023-11-16T16:46:45.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.061+0000 7f7235ffb700 1 -- 172.21.15.44:0/159175648 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18387 (secure 0 0 0) 0x7f722c099e50 con 0x7f7238073550 2023-11-16T16:46:45.062 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:46:45.063 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":79,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:46:30.119708+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:46:45.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.064+0000 7f723efff700 1 -- 172.21.15.44:0/159175648 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f72280a1150 msgr2=0x7f72280a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:45.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.064+0000 7f723efff700 1 --2- 172.21.15.44:0/159175648 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f72280a1150 0x7f72280a3600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7238074850 tx=0x7f7224009260 comp rx=0 tx=0).stop 2023-11-16T16:46:45.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.064+0000 7f723efff700 1 -- 172.21.15.44:0/159175648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7238073550 msgr2=0x7f723819c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:45.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.064+0000 7f723efff700 1 --2- 172.21.15.44:0/159175648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7238073550 0x7f723819c690 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f722c00e9f0 tx=0x7f722c00ed00 comp rx=0 tx=0).stop 2023-11-16T16:46:45.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.064+0000 7f723efff700 1 -- 172.21.15.44:0/159175648 shutdown_connections 2023-11-16T16:46:45.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.064+0000 7f723efff700 1 --2- 172.21.15.44:0/159175648 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f72280a1150 0x7f72280a3600 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.064+0000 7f723efff700 1 --2- 172.21.15.44:0/159175648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f723810c860 0x7f72381a3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.064+0000 7f723efff700 1 --2- 172.21.15.44:0/159175648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7238073550 0x7f723819c690 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.064+0000 7f723efff700 1 --2- 172.21.15.44:0/159175648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7238074b30 0x7f723819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.064+0000 7f723efff700 1 -- 172.21.15.44:0/159175648 >> 172.21.15.44:0/159175648 conn(0x7f72380fb870 msgr2=0x7f72380fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:45.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.064+0000 7f723efff700 1 -- 172.21.15.44:0/159175648 shutdown_connections 2023-11-16T16:46:45.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.064+0000 7f723efff700 1 -- 172.21.15.44:0/159175648 wait complete. 2023-11-16T16:46:45.085 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:03.059622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-16T16:46:45.085 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:46:45.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.292+0000 7fba87bff700 1 -- 172.21.15.44:0/3002567212 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba80101410 msgr2=0x7fba80101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:45.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.292+0000 7fba87bff700 1 --2- 172.21.15.44:0/3002567212 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba80101410 0x7fba80101860 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fba70009990 tx=0x7fba70009ca0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.293+0000 7fba87bff700 1 -- 172.21.15.44:0/3002567212 shutdown_connections 2023-11-16T16:46:45.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.293+0000 7fba87bff700 1 --2- 172.21.15.44:0/3002567212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba80101da0 0x7fba80108660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.293+0000 7fba87bff700 1 --2- 172.21.15.44:0/3002567212 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba80100220 0x7fba80100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.293+0000 7fba87bff700 1 --2- 172.21.15.44:0/3002567212 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba80101410 0x7fba80101860 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.293+0000 7fba87bff700 1 -- 172.21.15.44:0/3002567212 >> 172.21.15.44:0/3002567212 conn(0x7fba800fb800 msgr2=0x7fba800fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:45.295 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.293+0000 7fba87bff700 1 -- 172.21.15.44:0/3002567212 shutdown_connections 2023-11-16T16:46:45.295 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.293+0000 7fba87bff700 1 -- 172.21.15.44:0/3002567212 wait complete. 2023-11-16T16:46:45.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.294+0000 7fba87bff700 1 Processor -- start 2023-11-16T16:46:45.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.294+0000 7fba87bff700 1 -- start start 2023-11-16T16:46:45.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.294+0000 7fba87bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba80100220 0x7fba80198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:45.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.294+0000 7fba87bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba80101410 0x7fba80198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:45.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.294+0000 7fba87bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba80101da0 0x7fba8019f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:45.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.294+0000 7fba87bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8019fca0 con 0x7fba80101410 2023-11-16T16:46:45.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.294+0000 7fba87bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8019fde0 con 0x7fba80101da0 2023-11-16T16:46:45.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.294+0000 7fba87bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8019ff70 con 0x7fba80100220 2023-11-16T16:46:45.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.294+0000 7fba8519a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba80101410 0x7fba80198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:45.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.294+0000 7fba8619c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba80101da0 0x7fba8019f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:45.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.294+0000 7fba8519a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba80101410 0x7fba80198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45796/0 (socket says 172.21.15.44:45796) 2023-11-16T16:46:45.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.294+0000 7fba8619c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba80101da0 0x7fba8019f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59940/0 (socket says 172.21.15.44:59940) 2023-11-16T16:46:45.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.294+0000 7fba8519a700 1 -- 172.21.15.44:0/2842933544 learned_addr learned my addr 172.21.15.44:0/2842933544 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:46:45.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.294+0000 7fba8599b700 1 --2- 172.21.15.44:0/2842933544 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba80100220 0x7fba80198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:45.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.295+0000 7fba8619c700 1 -- 172.21.15.44:0/2842933544 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba80100220 msgr2=0x7fba80198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:45.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.295+0000 7fba8619c700 1 --2- 172.21.15.44:0/2842933544 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba80100220 0x7fba80198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.295+0000 7fba8619c700 1 -- 172.21.15.44:0/2842933544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba80101410 msgr2=0x7fba80198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:45.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.295+0000 7fba8619c700 1 --2- 172.21.15.44:0/2842933544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba80101410 0x7fba80198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.295+0000 7fba8619c700 1 -- 172.21.15.44:0/2842933544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba700095f0 con 0x7fba80101da0 2023-11-16T16:46:45.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.295+0000 7fba8519a700 1 --2- 172.21.15.44:0/2842933544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba80101410 0x7fba80198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:46:45.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.295+0000 7fba8619c700 1 --2- 172.21.15.44:0/2842933544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba80101da0 0x7fba8019f6d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fba7800efc0 tx=0x7fba7800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:45.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.295+0000 7fba76ffd700 1 -- 172.21.15.44:0/2842933544 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78018070 con 0x7fba80101da0 2023-11-16T16:46:45.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.295+0000 7fba87bff700 1 -- 172.21.15.44:0/2842933544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba8010bbe0 con 0x7fba80101da0 2023-11-16T16:46:45.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.295+0000 7fba76ffd700 1 -- 172.21.15.44:0/2842933544 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba7800f040 con 0x7fba80101da0 2023-11-16T16:46:45.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.295+0000 7fba76ffd700 1 -- 172.21.15.44:0/2842933544 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7800cd40 con 0x7fba80101da0 2023-11-16T16:46:45.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.295+0000 7fba87bff700 1 -- 172.21.15.44:0/2842933544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba8010ba50 con 0x7fba80101da0 2023-11-16T16:46:45.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.297+0000 7fba87bff700 1 -- 172.21.15.44:0/2842933544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba80192870 con 0x7fba80101da0 2023-11-16T16:46:45.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.298+0000 7fba76ffd700 1 -- 172.21.15.44:0/2842933544 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fba78003a00 con 0x7fba80101da0 2023-11-16T16:46:45.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.300+0000 7fba76ffd700 1 --2- 172.21.15.44:0/2842933544 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba6c0a1150 0x7fba6c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:45.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.300+0000 7fba76ffd700 1 -- 172.21.15.44:0/2842933544 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fba780cc0d0 con 0x7fba80101da0 2023-11-16T16:46:45.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.300+0000 7fba8599b700 1 --2- 172.21.15.44:0/2842933544 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba6c0a1150 0x7fba6c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:45.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.300+0000 7fba8599b700 1 --2- 172.21.15.44:0/2842933544 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba6c0a1150 0x7fba6c0a3600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fba80101270 tx=0x7fba7c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:45.304 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.303+0000 7fba76ffd700 1 -- 172.21.15.44:0/2842933544 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fba78095cb0 con 0x7fba80101da0 2023-11-16T16:46:45.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:45 smithi077 ceph-mon[131204]: mds.? [v2:172.21.15.176:6832/1226611,v1:172.21.15.176:6833/1226611] up:boot 2023-11-16T16:46:45.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:45 smithi077 ceph-mon[131204]: fsmap 1 up:standby 2023-11-16T16:46:45.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T16:46:45.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:45.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:45.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:45 smithi077 ceph-mon[131204]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:45.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:45.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:45.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:45.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:45 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:45.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:45 smithi077 ceph-mon[131204]: Deploying daemon mds.k on smithi077 2023-11-16T16:46:45.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:45 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/159175648' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:46:45.512 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.511+0000 7fba87bff700 1 -- 172.21.15.44:0/2842933544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fba80061cc0 con 0x7fba80101da0 2023-11-16T16:46:45.512 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.511+0000 7fba76ffd700 1 -- 172.21.15.44:0/2842933544 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7fba780992d0 con 0x7fba80101da0 2023-11-16T16:46:45.512 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2023-11-16T16:46:45.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.514+0000 7fba87bff700 1 -- 172.21.15.44:0/2842933544 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba6c0a1150 msgr2=0x7fba6c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:45.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.514+0000 7fba87bff700 1 --2- 172.21.15.44:0/2842933544 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba6c0a1150 0x7fba6c0a3600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fba80101270 tx=0x7fba7c009260 comp rx=0 tx=0).stop 2023-11-16T16:46:45.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.514+0000 7fba87bff700 1 -- 172.21.15.44:0/2842933544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba80101da0 msgr2=0x7fba8019f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:45.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.514+0000 7fba87bff700 1 --2- 172.21.15.44:0/2842933544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba80101da0 0x7fba8019f6d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fba7800efc0 tx=0x7fba7800c3a0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.514+0000 7fba87bff700 1 -- 172.21.15.44:0/2842933544 shutdown_connections 2023-11-16T16:46:45.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.514+0000 7fba87bff700 1 --2- 172.21.15.44:0/2842933544 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba6c0a1150 0x7fba6c0a3600 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.514+0000 7fba87bff700 1 --2- 172.21.15.44:0/2842933544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba80101da0 0x7fba8019f6d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.514+0000 7fba87bff700 1 --2- 172.21.15.44:0/2842933544 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba80100220 0x7fba80198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.514+0000 7fba87bff700 1 --2- 172.21.15.44:0/2842933544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba80101410 0x7fba80198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.514+0000 7fba87bff700 1 -- 172.21.15.44:0/2842933544 >> 172.21.15.44:0/2842933544 conn(0x7fba800fb800 msgr2=0x7fba80106d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:45.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.514+0000 7fba87bff700 1 -- 172.21.15.44:0/2842933544 shutdown_connections 2023-11-16T16:46:45.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.514+0000 7fba87bff700 1 -- 172.21.15.44:0/2842933544 wait complete. 2023-11-16T16:46:45.535 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:46:45.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.739+0000 7f4d525ff700 1 -- 172.21.15.44:0/704334550 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d4c1002b0 msgr2=0x7f4d4c1006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:45.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.739+0000 7f4d525ff700 1 --2- 172.21.15.44:0/704334550 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d4c1002b0 0x7f4d4c1006c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c009990 tx=0x7f4d3c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.739+0000 7f4d525ff700 1 -- 172.21.15.44:0/704334550 shutdown_connections 2023-11-16T16:46:45.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.739+0000 7f4d525ff700 1 --2- 172.21.15.44:0/704334550 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d4c101e30 0x7f4d4c1086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.739+0000 7f4d525ff700 1 --2- 172.21.15.44:0/704334550 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d4c1002b0 0x7f4d4c1006c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.739+0000 7f4d525ff700 1 --2- 172.21.15.44:0/704334550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d4c1014a0 0x7f4d4c1018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.739+0000 7f4d525ff700 1 -- 172.21.15.44:0/704334550 >> 172.21.15.44:0/704334550 conn(0x7f4d4c0fb890 msgr2=0x7f4d4c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:45.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.739+0000 7f4d525ff700 1 -- 172.21.15.44:0/704334550 shutdown_connections 2023-11-16T16:46:45.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.739+0000 7f4d525ff700 1 -- 172.21.15.44:0/704334550 wait complete. 2023-11-16T16:46:45.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.740+0000 7f4d525ff700 1 Processor -- start 2023-11-16T16:46:45.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.740+0000 7f4d525ff700 1 -- start start 2023-11-16T16:46:45.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.740+0000 7f4d525ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d4c1002b0 0x7f4d4c198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:45.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.740+0000 7f4d525ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d4c1014a0 0x7f4d4c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:45.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.740+0000 7f4d525ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d4c101e30 0x7f4d4c19f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:45.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.740+0000 7f4d525ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c19fd10 con 0x7f4d4c1014a0 2023-11-16T16:46:45.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.740+0000 7f4d525ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c19fe50 con 0x7f4d4c1002b0 2023-11-16T16:46:45.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.740+0000 7f4d525ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c19ffe0 con 0x7f4d4c101e30 2023-11-16T16:46:45.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.741+0000 7f4d4b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d4c1014a0 0x7f4d4c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:45.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.741+0000 7f4d4bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d4c1002b0 0x7f4d4c198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:45.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.741+0000 7f4d4b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d4c1014a0 0x7f4d4c1986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45810/0 (socket says 172.21.15.44:45810) 2023-11-16T16:46:45.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.741+0000 7f4d4b7fe700 1 -- 172.21.15.44:0/3930407085 learned_addr learned my addr 172.21.15.44:0/3930407085 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:46:45.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.741+0000 7f4d4bfff700 1 --2- 172.21.15.44:0/3930407085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d4c1002b0 0x7f4d4c198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59964/0 (socket says 172.21.15.44:59964) 2023-11-16T16:46:45.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.741+0000 7f4d50b9c700 1 --2- 172.21.15.44:0/3930407085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d4c101e30 0x7f4d4c19f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:45.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.741+0000 7f4d4bfff700 1 -- 172.21.15.44:0/3930407085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d4c101e30 msgr2=0x7f4d4c19f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:45.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.741+0000 7f4d4bfff700 1 --2- 172.21.15.44:0/3930407085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d4c101e30 0x7f4d4c19f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.741+0000 7f4d4bfff700 1 -- 172.21.15.44:0/3930407085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d4c1014a0 msgr2=0x7f4d4c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:45.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.741+0000 7f4d4bfff700 1 --2- 172.21.15.44:0/3930407085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d4c1014a0 0x7f4d4c1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:45.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.741+0000 7f4d4bfff700 1 -- 172.21.15.44:0/3930407085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d3c0095f0 con 0x7f4d4c1002b0 2023-11-16T16:46:45.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.741+0000 7f4d4b7fe700 1 --2- 172.21.15.44:0/3930407085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d4c1014a0 0x7f4d4c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:46:45.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.741+0000 7f4d4bfff700 1 --2- 172.21.15.44:0/3930407085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d4c1002b0 0x7f4d4c198190 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c0059d0 tx=0x7f4d3c005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:45.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.741+0000 7f4d497fa700 1 -- 172.21.15.44:0/3930407085 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c01d070 con 0x7f4d4c1002b0 2023-11-16T16:46:45.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.742+0000 7f4d497fa700 1 -- 172.21.15.44:0/3930407085 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d3c00fb90 con 0x7f4d4c1002b0 2023-11-16T16:46:45.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.742+0000 7f4d525ff700 1 -- 172.21.15.44:0/3930407085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4c10bc30 con 0x7f4d4c1002b0 2023-11-16T16:46:45.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.742+0000 7f4d497fa700 1 -- 172.21.15.44:0/3930407085 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c0173f0 con 0x7f4d4c1002b0 2023-11-16T16:46:45.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.742+0000 7f4d525ff700 1 -- 172.21.15.44:0/3930407085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4c10baa0 con 0x7f4d4c1002b0 2023-11-16T16:46:45.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.743+0000 7f4d525ff700 1 -- 172.21.15.44:0/3930407085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4c1928c0 con 0x7f4d4c1002b0 2023-11-16T16:46:45.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.744+0000 7f4d497fa700 1 -- 172.21.15.44:0/3930407085 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4d3c022470 con 0x7f4d4c1002b0 2023-11-16T16:46:45.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.746+0000 7f4d497fa700 1 --2- 172.21.15.44:0/3930407085 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4d340a1150 0x7f4d340a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:45.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.746+0000 7f4d497fa700 1 -- 172.21.15.44:0/3930407085 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4d3c0cd9d0 con 0x7f4d4c1002b0 2023-11-16T16:46:45.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.746+0000 7f4d4b7fe700 1 --2- 172.21.15.44:0/3930407085 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4d340a1150 0x7f4d340a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:45.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.747+0000 7f4d4b7fe700 1 --2- 172.21.15.44:0/3930407085 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4d340a1150 0x7f4d340a3600 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4d40011550 tx=0x7f4d40009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:45.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:45.748+0000 7f4d497fa700 1 -- 172.21.15.44:0/3930407085 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4d3c0975b0 con 0x7f4d4c1002b0 2023-11-16T16:46:45.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:45 smithi044 ceph-mon[128750]: mds.? [v2:172.21.15.176:6832/1226611,v1:172.21.15.176:6833/1226611] up:boot 2023-11-16T16:46:45.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:45 smithi044 ceph-mon[128750]: fsmap 1 up:standby 2023-11-16T16:46:45.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T16:46:45.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:45.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:45.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:45 smithi044 ceph-mon[128750]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:45.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:45.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:45.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:45.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:45 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:45.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:45 smithi044 ceph-mon[128750]: Deploying daemon mds.k on smithi077 2023-11-16T16:46:45.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:45 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/159175648' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:46:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:45 smithi176 ceph-mon[131630]: mds.? [v2:172.21.15.176:6832/1226611,v1:172.21.15.176:6833/1226611] up:boot 2023-11-16T16:46:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:45 smithi176 ceph-mon[131630]: fsmap 1 up:standby 2023-11-16T16:46:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T16:46:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:45.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:45 smithi176 ceph-mon[131630]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:45.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:45.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:45.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:45.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:45 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:45.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:45 smithi176 ceph-mon[131630]: Deploying daemon mds.k on smithi077 2023-11-16T16:46:45.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:45 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/159175648' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:46:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:46.004+0000 7f4d525ff700 1 -- 172.21.15.44:0/3930407085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d4c061700 con 0x7f4d4c1002b0 2023-11-16T16:46:46.005 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:46.004+0000 7f4d497fa700 1 -- 172.21.15.44:0/3930407085 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1251 (secure 0 0 0) 0x7f4d3c09abd0 con 0x7f4d4c1002b0 2023-11-16T16:46:46.005 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:46:46.005 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-11-16T16:46:46.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:46.007+0000 7f4d525ff700 1 -- 172.21.15.44:0/3930407085 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4d340a1150 msgr2=0x7f4d340a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:46.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:46.007+0000 7f4d525ff700 1 --2- 172.21.15.44:0/3930407085 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4d340a1150 0x7f4d340a3600 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4d40011550 tx=0x7f4d40009260 comp rx=0 tx=0).stop 2023-11-16T16:46:46.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:46.007+0000 7f4d525ff700 1 -- 172.21.15.44:0/3930407085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d4c1002b0 msgr2=0x7f4d4c198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:46.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:46.007+0000 7f4d525ff700 1 --2- 172.21.15.44:0/3930407085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d4c1002b0 0x7f4d4c198190 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c0059d0 tx=0x7f4d3c005740 comp rx=0 tx=0).stop 2023-11-16T16:46:46.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:46.007+0000 7f4d525ff700 1 -- 172.21.15.44:0/3930407085 shutdown_connections 2023-11-16T16:46:46.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:46.007+0000 7f4d525ff700 1 --2- 172.21.15.44:0/3930407085 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4d340a1150 0x7f4d340a3600 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:46.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:46.007+0000 7f4d525ff700 1 --2- 172.21.15.44:0/3930407085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d4c101e30 0x7f4d4c19f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:46.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:46.007+0000 7f4d525ff700 1 --2- 172.21.15.44:0/3930407085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d4c1002b0 0x7f4d4c198190 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:46.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:46.007+0000 7f4d525ff700 1 --2- 172.21.15.44:0/3930407085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d4c1014a0 0x7f4d4c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:46.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:46.007+0000 7f4d525ff700 1 -- 172.21.15.44:0/3930407085 >> 172.21.15.44:0/3930407085 conn(0x7f4d4c0fb890 msgr2=0x7f4d4c106dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:46.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:46.007+0000 7f4d525ff700 1 -- 172.21.15.44:0/3930407085 shutdown_connections 2023-11-16T16:46:46.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:46.007+0000 7f4d525ff700 1 -- 172.21.15.44:0/3930407085 wait complete. 2023-11-16T16:46:46.010 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 2 2023-11-16T16:46:46.331 INFO:journalctl@ceph.mds.k.smithi077.stdout:Nov 16 16:46:46 smithi077 systemd[1]: Starting Ceph mds.k for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:46:46.623 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:46 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2842933544' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:46:46.623 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:46 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3930407085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:46.624 INFO:journalctl@ceph.mds.k.smithi077.stdout:Nov 16 16:46:46 smithi077 podman[155490]: 2023-11-16T16:46:46.624 INFO:journalctl@ceph.mds.k.smithi077.stdout:Nov 16 16:46:46 smithi077 bash[155490]: ffa8502df4d4aa5ecb776fdb258ee27cae2baadfb3569fc0d06ee14c9a77e98d 2023-11-16T16:46:46.624 INFO:journalctl@ceph.mds.k.smithi077.stdout:Nov 16 16:46:46 smithi077 ceph-mds[155531]: -- 172.21.15.77:0/116617499 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556bfab31520 con 0x556bfab06800 2023-11-16T16:46:46.624 INFO:journalctl@ceph.mds.k.smithi077.stdout:Nov 16 16:46:46 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-k[155507]: starting mds.k at 2023-11-16T16:46:46.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:46 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2842933544' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:46:46.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:46 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3930407085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:46.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:46 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2842933544' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:46:46.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:46 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3930407085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:46.976 INFO:journalctl@ceph.mds.k.smithi077.stdout:Nov 16 16:46:46 smithi077 systemd[1]: Started Ceph mds.k for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:46:47.965 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:47 smithi044 ceph-mon[128750]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:47.965 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:47.965 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:47.965 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:47.966 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:47.966 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:47.966 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:47.966 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:47 smithi044 ceph-mon[128750]: Deploying daemon mds.j on smithi044 2023-11-16T16:46:47.966 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:47 smithi044 ceph-mon[128750]: mds.? [v2:172.21.15.77:6832/2897858995,v1:172.21.15.77:6833/2897858995] up:boot 2023-11-16T16:46:47.966 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:47 smithi044 ceph-mon[128750]: fsmap 2 up:standby 2023-11-16T16:46:47.966 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:47 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T16:46:47.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:47 smithi077 ceph-mon[131204]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:47.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:47.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:47.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:47.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:47.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:47.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:47.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:47 smithi077 ceph-mon[131204]: Deploying daemon mds.j on smithi044 2023-11-16T16:46:47.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:47 smithi077 ceph-mon[131204]: mds.? [v2:172.21.15.77:6832/2897858995,v1:172.21.15.77:6833/2897858995] up:boot 2023-11-16T16:46:47.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:47 smithi077 ceph-mon[131204]: fsmap 2 up:standby 2023-11-16T16:46:47.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:47 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T16:46:48.027 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:46:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:47 smithi176 ceph-mon[131630]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:47 smithi176 ceph-mon[131630]: Deploying daemon mds.j on smithi044 2023-11-16T16:46:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:47 smithi176 ceph-mon[131630]: mds.? [v2:172.21.15.77:6832/2897858995,v1:172.21.15.77:6833/2897858995] up:boot 2023-11-16T16:46:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:47 smithi176 ceph-mon[131630]: fsmap 2 up:standby 2023-11-16T16:46:48.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:47 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T16:46:48.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.239+0000 7f00b69ff700 1 -- 172.21.15.44:0/578494682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b0100290 msgr2=0x7f00b01006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:48.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.239+0000 7f00b69ff700 1 --2- 172.21.15.44:0/578494682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b0100290 0x7f00b01006a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f00a00098f0 tx=0x7f00a0009c00 comp rx=0 tx=0).stop 2023-11-16T16:46:48.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.239+0000 7f00b69ff700 1 -- 172.21.15.44:0/578494682 shutdown_connections 2023-11-16T16:46:48.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.239+0000 7f00b69ff700 1 --2- 172.21.15.44:0/578494682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00b0101e10 0x7f00b01086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:48.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.239+0000 7f00b69ff700 1 --2- 172.21.15.44:0/578494682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b0100290 0x7f00b01006a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:48.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.239+0000 7f00b69ff700 1 --2- 172.21.15.44:0/578494682 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00b0101480 0x7f00b01018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:48.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.239+0000 7f00b69ff700 1 -- 172.21.15.44:0/578494682 >> 172.21.15.44:0/578494682 conn(0x7f00b00fb870 msgr2=0x7f00b00fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:48.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.240+0000 7f00b69ff700 1 -- 172.21.15.44:0/578494682 shutdown_connections 2023-11-16T16:46:48.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.240+0000 7f00b69ff700 1 -- 172.21.15.44:0/578494682 wait complete. 2023-11-16T16:46:48.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.241+0000 7f00b69ff700 1 Processor -- start 2023-11-16T16:46:48.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.241+0000 7f00b69ff700 1 -- start start 2023-11-16T16:46:48.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.241+0000 7f00b69ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b0100290 0x7f00b0198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:48.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.241+0000 7f00b69ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00b0101480 0x7f00b0198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:48.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.241+0000 7f00b69ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00b0101e10 0x7f00b019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:48.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.241+0000 7f00b69ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b019fdd0 con 0x7f00b0100290 2023-11-16T16:46:48.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.241+0000 7f00b69ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b019ff10 con 0x7f00b0101e10 2023-11-16T16:46:48.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.241+0000 7f00b69ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b01a00a0 con 0x7f00b0101480 2023-11-16T16:46:48.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.241+0000 7f00affff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b0100290 0x7f00b0198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:48.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.241+0000 7f00affff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b0100290 0x7f00b0198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45824/0 (socket says 172.21.15.44:45824) 2023-11-16T16:46:48.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.241+0000 7f00affff700 1 -- 172.21.15.44:0/3823112292 learned_addr learned my addr 172.21.15.44:0/3823112292 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:46:48.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.241+0000 7f00af7fe700 1 --2- 172.21.15.44:0/3823112292 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00b0101480 0x7f00b0198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:48.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.241+0000 7f00b4f9c700 1 --2- 172.21.15.44:0/3823112292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00b0101e10 0x7f00b019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:48.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.242+0000 7f00affff700 1 -- 172.21.15.44:0/3823112292 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00b0101480 msgr2=0x7f00b0198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:48.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.242+0000 7f00affff700 1 --2- 172.21.15.44:0/3823112292 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00b0101480 0x7f00b0198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:48.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.242+0000 7f00affff700 1 -- 172.21.15.44:0/3823112292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00b0101e10 msgr2=0x7f00b019f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:48.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.242+0000 7f00affff700 1 --2- 172.21.15.44:0/3823112292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00b0101e10 0x7f00b019f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:48.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.242+0000 7f00affff700 1 -- 172.21.15.44:0/3823112292 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a8009d00 con 0x7f00b0100290 2023-11-16T16:46:48.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.242+0000 7f00affff700 1 --2- 172.21.15.44:0/3823112292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b0100290 0x7f00b0198250 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f00a0005e90 tx=0x7f00a000b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:48.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.242+0000 7f00ad7fa700 1 -- 172.21.15.44:0/3823112292 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a001c070 con 0x7f00b0100290 2023-11-16T16:46:48.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.242+0000 7f00b69ff700 1 -- 172.21.15.44:0/3823112292 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a0009520 con 0x7f00b0100290 2023-11-16T16:46:48.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.242+0000 7f00ad7fa700 1 -- 172.21.15.44:0/3823112292 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00a000f7f0 con 0x7f00b0100290 2023-11-16T16:46:48.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.242+0000 7f00ad7fa700 1 -- 172.21.15.44:0/3823112292 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a0003990 con 0x7f00b0100290 2023-11-16T16:46:48.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.242+0000 7f00b69ff700 1 -- 172.21.15.44:0/3823112292 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b010bc40 con 0x7f00b0100290 2023-11-16T16:46:48.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.244+0000 7f00b69ff700 1 -- 172.21.15.44:0/3823112292 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b01928d0 con 0x7f00b0100290 2023-11-16T16:46:48.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.246+0000 7f00ad7fa700 1 -- 172.21.15.44:0/3823112292 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f00a0020030 con 0x7f00b0100290 2023-11-16T16:46:48.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.247+0000 7f00ad7fa700 1 --2- 172.21.15.44:0/3823112292 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f00980a11d0 0x7f00980a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:48.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.247+0000 7f00ad7fa700 1 -- 172.21.15.44:0/3823112292 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f00a00cd320 con 0x7f00b0100290 2023-11-16T16:46:48.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.247+0000 7f00af7fe700 1 --2- 172.21.15.44:0/3823112292 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f00980a11d0 0x7f00980a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:48.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.248+0000 7f00af7fe700 1 --2- 172.21.15.44:0/3823112292 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f00980a11d0 0x7f00980a3680 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009c60 tx=0x7f00a4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:48.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.251+0000 7f00ad7fa700 1 -- 172.21.15.44:0/3823112292 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f00a0096fa0 con 0x7f00b0100290 2023-11-16T16:46:48.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.527+0000 7f00b69ff700 1 -- 172.21.15.44:0/3823112292 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00b0067a60 con 0x7f00b0100290 2023-11-16T16:46:48.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.527+0000 7f00ad7fa700 1 -- 172.21.15.44:0/3823112292 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2008 (secure 0 0 0) 0x7f00a009a5c0 con 0x7f00b0100290 2023-11-16T16:46:48.528 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:46:48.529 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-11-16T16:46:48.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.530+0000 7f00b69ff700 1 -- 172.21.15.44:0/3823112292 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f00980a11d0 msgr2=0x7f00980a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:48.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.530+0000 7f00b69ff700 1 --2- 172.21.15.44:0/3823112292 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f00980a11d0 0x7f00980a3680 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009c60 tx=0x7f00a4009260 comp rx=0 tx=0).stop 2023-11-16T16:46:48.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.530+0000 7f00b69ff700 1 -- 172.21.15.44:0/3823112292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b0100290 msgr2=0x7f00b0198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:48.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.530+0000 7f00b69ff700 1 --2- 172.21.15.44:0/3823112292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b0100290 0x7f00b0198250 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f00a0005e90 tx=0x7f00a000b730 comp rx=0 tx=0).stop 2023-11-16T16:46:48.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.530+0000 7f00b69ff700 1 -- 172.21.15.44:0/3823112292 shutdown_connections 2023-11-16T16:46:48.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.530+0000 7f00b69ff700 1 --2- 172.21.15.44:0/3823112292 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f00980a11d0 0x7f00980a3680 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:48.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.530+0000 7f00b69ff700 1 --2- 172.21.15.44:0/3823112292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00b0101e10 0x7f00b019f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:48.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.530+0000 7f00b69ff700 1 --2- 172.21.15.44:0/3823112292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b0100290 0x7f00b0198250 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:48.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.530+0000 7f00b69ff700 1 --2- 172.21.15.44:0/3823112292 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00b0101480 0x7f00b0198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:48.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.530+0000 7f00b69ff700 1 -- 172.21.15.44:0/3823112292 >> 172.21.15.44:0/3823112292 conn(0x7f00b00fb870 msgr2=0x7f00b0106df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:48.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.530+0000 7f00b69ff700 1 -- 172.21.15.44:0/3823112292 shutdown_connections 2023-11-16T16:46:48.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:48.530+0000 7f00b69ff700 1 -- 172.21.15.44:0/3823112292 wait complete. 2023-11-16T16:46:48.532 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 3 2023-11-16T16:46:49.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:49 smithi044 ceph-mon[128750]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:49.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:49 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3823112292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:49.369 INFO:journalctl@ceph.mds.j.smithi044.stdout:Nov 16 16:46:48 smithi044 systemd[1]: Starting Ceph mds.j for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:46:49.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:49 smithi176 ceph-mon[131630]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:49.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:49 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3823112292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:49.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:49 smithi077 ceph-mon[131204]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:49.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:49 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3823112292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:49.672 INFO:journalctl@ceph.mds.j.smithi044.stdout:Nov 16 16:46:49 smithi044 podman[161252]: 2023-11-16T16:46:49.672 INFO:journalctl@ceph.mds.j.smithi044.stdout:Nov 16 16:46:49 smithi044 bash[161252]: bfaef8baf56b38797ffe528d5fd40f2976e58794efdd31b04d97e29c74cfba1a 2023-11-16T16:46:49.672 INFO:journalctl@ceph.mds.j.smithi044.stdout:Nov 16 16:46:49 smithi044 ceph-mds[161292]: -- 172.21.15.44:0/2901235436 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e612c73520 con 0x55e612c4a800 2023-11-16T16:46:49.672 INFO:journalctl@ceph.mds.j.smithi044.stdout:Nov 16 16:46:49 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-j[161268]: starting mds.j at 2023-11-16T16:46:50.118 INFO:journalctl@ceph.mds.j.smithi044.stdout:Nov 16 16:46:49 smithi044 systemd[1]: Started Ceph mds.j for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:46:50.551 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:46:50.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.769+0000 7ff5ca9ff700 1 -- 172.21.15.44:0/3806929515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5c4101e10 msgr2=0x7ff5c41086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:50.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.769+0000 7ff5ca9ff700 1 --2- 172.21.15.44:0/3806929515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5c4101e10 0x7ff5c41086d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00b0a0 tx=0x7ff5bc00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.769+0000 7ff5ca9ff700 1 -- 172.21.15.44:0/3806929515 shutdown_connections 2023-11-16T16:46:50.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.769+0000 7ff5ca9ff700 1 --2- 172.21.15.44:0/3806929515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5c4101e10 0x7ff5c41086d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.769+0000 7ff5ca9ff700 1 --2- 172.21.15.44:0/3806929515 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5c4100290 0x7ff5c41006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.769+0000 7ff5ca9ff700 1 --2- 172.21.15.44:0/3806929515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5c4101480 0x7ff5c41018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.769+0000 7ff5ca9ff700 1 -- 172.21.15.44:0/3806929515 >> 172.21.15.44:0/3806929515 conn(0x7ff5c40fb870 msgr2=0x7ff5c40fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:50.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.769+0000 7ff5ca9ff700 1 -- 172.21.15.44:0/3806929515 shutdown_connections 2023-11-16T16:46:50.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.769+0000 7ff5ca9ff700 1 -- 172.21.15.44:0/3806929515 wait complete. 2023-11-16T16:46:50.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.771+0000 7ff5ca9ff700 1 Processor -- start 2023-11-16T16:46:50.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.771+0000 7ff5ca9ff700 1 -- start start 2023-11-16T16:46:50.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.771+0000 7ff5ca9ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5c4100290 0x7ff5c40748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:50.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.771+0000 7ff5ca9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5c4101480 0x7ff5c4072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:50.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.771+0000 7ff5ca9ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5c4101e10 0x7ff5c4073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:50.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.771+0000 7ff5ca9ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c4073bc0 con 0x7ff5c4100290 2023-11-16T16:46:50.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.771+0000 7ff5ca9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c4073d30 con 0x7ff5c4101480 2023-11-16T16:46:50.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.771+0000 7ff5ca9ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c41a00f0 con 0x7ff5c4101e10 2023-11-16T16:46:50.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.771+0000 7ff5c3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5c4100290 0x7ff5c40748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:50.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.771+0000 7ff5c3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5c4100290 0x7ff5c40748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45850/0 (socket says 172.21.15.44:45850) 2023-11-16T16:46:50.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.771+0000 7ff5c3fff700 1 -- 172.21.15.44:0/3052168572 learned_addr learned my addr 172.21.15.44:0/3052168572 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:46:50.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.772+0000 7ff5c37fe700 1 --2- 172.21.15.44:0/3052168572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5c4101480 0x7ff5c4072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:50.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.772+0000 7ff5c8f9c700 1 --2- 172.21.15.44:0/3052168572 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5c4101e10 0x7ff5c4073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:50.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.772+0000 7ff5c3fff700 1 -- 172.21.15.44:0/3052168572 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5c4101e10 msgr2=0x7ff5c4073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:50.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.772+0000 7ff5c3fff700 1 --2- 172.21.15.44:0/3052168572 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5c4101e10 0x7ff5c4073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.772+0000 7ff5c3fff700 1 -- 172.21.15.44:0/3052168572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5c4101480 msgr2=0x7ff5c4072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:50.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.772+0000 7ff5c3fff700 1 --2- 172.21.15.44:0/3052168572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5c4101480 0x7ff5c4072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.772+0000 7ff5c3fff700 1 -- 172.21.15.44:0/3052168572 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5bc009d00 con 0x7ff5c4100290 2023-11-16T16:46:50.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.772+0000 7ff5c3fff700 1 --2- 172.21.15.44:0/3052168572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5c4100290 0x7ff5c40748a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff5b400e970 tx=0x7ff5b400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:50.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.772+0000 7ff5c17fa700 1 -- 172.21.15.44:0/3052168572 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b4009790 con 0x7ff5c4100290 2023-11-16T16:46:50.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.772+0000 7ff5c17fa700 1 -- 172.21.15.44:0/3052168572 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff5b400c7e0 con 0x7ff5c4100290 2023-11-16T16:46:50.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.772+0000 7ff5ca9ff700 1 -- 172.21.15.44:0/3052168572 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c410bc70 con 0x7ff5c4100290 2023-11-16T16:46:50.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.773+0000 7ff5c17fa700 1 -- 172.21.15.44:0/3052168572 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b4003c70 con 0x7ff5c4100290 2023-11-16T16:46:50.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.773+0000 7ff5ca9ff700 1 -- 172.21.15.44:0/3052168572 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c410bae0 con 0x7ff5c4100290 2023-11-16T16:46:50.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.774+0000 7ff5ca9ff700 1 -- 172.21.15.44:0/3052168572 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c41a0860 con 0x7ff5c4100290 2023-11-16T16:46:50.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.778+0000 7ff5c17fa700 1 -- 172.21.15.44:0/3052168572 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff5b400cc50 con 0x7ff5c4100290 2023-11-16T16:46:50.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.779+0000 7ff5c17fa700 1 --2- 172.21.15.44:0/3052168572 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff5ac0a11d0 0x7ff5ac0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:50.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.779+0000 7ff5c17fa700 1 -- 172.21.15.44:0/3052168572 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff5b4014070 con 0x7ff5c4100290 2023-11-16T16:46:50.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.780+0000 7ff5c37fe700 1 --2- 172.21.15.44:0/3052168572 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff5ac0a11d0 0x7ff5ac0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:50.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.780+0000 7ff5c37fe700 1 --2- 172.21.15.44:0/3052168572 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff5ac0a11d0 0x7ff5ac0a3680 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8009c60 tx=0x7ff5b8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:50.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:50.785+0000 7ff5c17fa700 1 -- 172.21.15.44:0/3052168572 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff5b40961a0 con 0x7ff5c4100290 2023-11-16T16:46:50.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:50 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:50.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:50 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:50.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:50 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:50.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:50 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:50.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:50 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:50.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:50 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:50.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:50 smithi044 ceph-mon[128750]: Deploying daemon mds.d on smithi044 2023-11-16T16:46:50.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:50 smithi044 ceph-mon[128750]: mds.? [v2:172.21.15.44:6834/3970258683,v1:172.21.15.44:6835/3970258683] up:boot 2023-11-16T16:46:50.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:50 smithi044 ceph-mon[128750]: fsmap 3 up:standby 2023-11-16T16:46:50.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:50 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T16:46:51.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:51.051+0000 7ff5ca9ff700 1 -- 172.21.15.44:0/3052168572 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5c4107100 con 0x7ff5c4100290 2023-11-16T16:46:51.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:51.052+0000 7ff5c17fa700 1 -- 172.21.15.44:0/3052168572 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2765 (secure 0 0 0) 0x7ff5b4003dd0 con 0x7ff5c4100290 2023-11-16T16:46:51.053 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:46:51.053 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-11-16T16:46:51.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:51.055+0000 7ff5ca9ff700 1 -- 172.21.15.44:0/3052168572 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff5ac0a11d0 msgr2=0x7ff5ac0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:51.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:51.055+0000 7ff5ca9ff700 1 --2- 172.21.15.44:0/3052168572 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff5ac0a11d0 0x7ff5ac0a3680 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8009c60 tx=0x7ff5b8009260 comp rx=0 tx=0).stop 2023-11-16T16:46:51.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:51.055+0000 7ff5ca9ff700 1 -- 172.21.15.44:0/3052168572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5c4100290 msgr2=0x7ff5c40748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:51.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:51.055+0000 7ff5ca9ff700 1 --2- 172.21.15.44:0/3052168572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5c4100290 0x7ff5c40748a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff5b400e970 tx=0x7ff5b400ec80 comp rx=0 tx=0).stop 2023-11-16T16:46:51.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:51.055+0000 7ff5ca9ff700 1 -- 172.21.15.44:0/3052168572 shutdown_connections 2023-11-16T16:46:51.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:51.055+0000 7ff5ca9ff700 1 --2- 172.21.15.44:0/3052168572 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff5ac0a11d0 0x7ff5ac0a3680 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:51.055+0000 7ff5ca9ff700 1 --2- 172.21.15.44:0/3052168572 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5c4101e10 0x7ff5c4073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:51.055+0000 7ff5ca9ff700 1 --2- 172.21.15.44:0/3052168572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5c4100290 0x7ff5c40748a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:51.055+0000 7ff5ca9ff700 1 --2- 172.21.15.44:0/3052168572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5c4101480 0x7ff5c4072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:51.055+0000 7ff5ca9ff700 1 -- 172.21.15.44:0/3052168572 >> 172.21.15.44:0/3052168572 conn(0x7ff5c40fb870 msgr2=0x7ff5c40fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:51.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:51.055+0000 7ff5ca9ff700 1 -- 172.21.15.44:0/3052168572 shutdown_connections 2023-11-16T16:46:51.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:51.055+0000 7ff5ca9ff700 1 -- 172.21.15.44:0/3052168572 wait complete. 2023-11-16T16:46:51.057 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 4 2023-11-16T16:46:51.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:50 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:51.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:50 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:51.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:50 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:51.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:50 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:51.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:50 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:51.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:50 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:51.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:50 smithi176 ceph-mon[131630]: Deploying daemon mds.d on smithi044 2023-11-16T16:46:51.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:50 smithi176 ceph-mon[131630]: mds.? [v2:172.21.15.44:6834/3970258683,v1:172.21.15.44:6835/3970258683] up:boot 2023-11-16T16:46:51.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:50 smithi176 ceph-mon[131630]: fsmap 3 up:standby 2023-11-16T16:46:51.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:50 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T16:46:51.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:50 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:51.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:50 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:51.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:50 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:51.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:50 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:51.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:50 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:51.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:50 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:51.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:50 smithi077 ceph-mon[131204]: Deploying daemon mds.d on smithi044 2023-11-16T16:46:51.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:50 smithi077 ceph-mon[131204]: mds.? [v2:172.21.15.44:6834/3970258683,v1:172.21.15.44:6835/3970258683] up:boot 2023-11-16T16:46:51.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:50 smithi077 ceph-mon[131204]: fsmap 3 up:standby 2023-11-16T16:46:51.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:50 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T16:46:51.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:51 smithi044 ceph-mon[128750]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:51.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:51 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3052168572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:52.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:51 smithi176 ceph-mon[131630]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:52.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:51 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3052168572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:52.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:51 smithi077 ceph-mon[131204]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:52.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:51 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3052168572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:52.368 INFO:journalctl@ceph.mds.d.smithi044.stdout:Nov 16 16:46:51 smithi044 systemd[1]: Starting Ceph mds.d for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:46:52.724 INFO:journalctl@ceph.mds.d.smithi044.stdout:Nov 16 16:46:52 smithi044 podman[161646]: 2023-11-16T16:46:52.724 INFO:journalctl@ceph.mds.d.smithi044.stdout:Nov 16 16:46:52 smithi044 bash[161646]: 0f118334eb98fdf9a5b5cc05d633c773985ac6714a3abf7f83dbad7e39d21c6a 2023-11-16T16:46:52.724 INFO:journalctl@ceph.mds.d.smithi044.stdout:Nov 16 16:46:52 smithi044 ceph-mds[161686]: -- 172.21.15.44:0/2767042604 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563e81343520 con 0x563e8131ac00 2023-11-16T16:46:52.725 INFO:journalctl@ceph.mds.d.smithi044.stdout:Nov 16 16:46:52 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-d[161662]: starting mds.d at 2023-11-16T16:46:53.079 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:46:53.118 INFO:journalctl@ceph.mds.d.smithi044.stdout:Nov 16 16:46:52 smithi044 systemd[1]: Started Ceph mds.d for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:46:53.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.288+0000 7fedd63ff700 1 -- 172.21.15.44:0/3980349303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedd0105e50 msgr2=0x7fedd0108300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:53.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.288+0000 7fedd63ff700 1 --2- 172.21.15.44:0/3980349303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedd0105e50 0x7fedd0108300 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fedc800b0a0 tx=0x7fedc800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.288+0000 7fedd63ff700 1 -- 172.21.15.44:0/3980349303 shutdown_connections 2023-11-16T16:46:53.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.288+0000 7fedd63ff700 1 --2- 172.21.15.44:0/3980349303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedd0105e50 0x7fedd0108300 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.288+0000 7fedd63ff700 1 --2- 172.21.15.44:0/3980349303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedd0068ad0 0x7fedd0102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.288+0000 7fedd63ff700 1 --2- 172.21.15.44:0/3980349303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedd0103380 0x7fedd01057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.288+0000 7fedd63ff700 1 -- 172.21.15.44:0/3980349303 >> 172.21.15.44:0/3980349303 conn(0x7fedd00fa750 msgr2=0x7fedd00fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:53.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.289+0000 7fedd63ff700 1 -- 172.21.15.44:0/3980349303 shutdown_connections 2023-11-16T16:46:53.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.289+0000 7fedd63ff700 1 -- 172.21.15.44:0/3980349303 wait complete. 2023-11-16T16:46:53.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.290+0000 7fedd63ff700 1 Processor -- start 2023-11-16T16:46:53.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.290+0000 7fedd63ff700 1 -- start start 2023-11-16T16:46:53.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.290+0000 7fedd63ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedd0068ad0 0x7fedd01960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:53.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.290+0000 7fedd63ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedd0103380 0x7fedd0196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:53.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.290+0000 7fedd63ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedd0105e50 0x7fedd019d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:53.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.290+0000 7fedd63ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd019dc50 con 0x7fedd0103380 2023-11-16T16:46:53.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.290+0000 7fedd63ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd019dd90 con 0x7fedd0068ad0 2023-11-16T16:46:53.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.290+0000 7fedd63ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd019df20 con 0x7fedd0105e50 2023-11-16T16:46:53.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.290+0000 7fedcf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedd0103380 0x7fedd0196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:53.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.290+0000 7fedcffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedd0068ad0 0x7fedd01960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:53.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.290+0000 7fedcf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedd0103380 0x7fedd0196610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45896/0 (socket says 172.21.15.44:45896) 2023-11-16T16:46:53.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.290+0000 7fedcf7fe700 1 -- 172.21.15.44:0/971534018 learned_addr learned my addr 172.21.15.44:0/971534018 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:46:53.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.291+0000 7fedd499c700 1 --2- 172.21.15.44:0/971534018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedd0105e50 0x7fedd019d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:53.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.291+0000 7fedcf7fe700 1 -- 172.21.15.44:0/971534018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedd0105e50 msgr2=0x7fedd019d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:53.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.291+0000 7fedcf7fe700 1 --2- 172.21.15.44:0/971534018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedd0105e50 0x7fedd019d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.291+0000 7fedcf7fe700 1 -- 172.21.15.44:0/971534018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedd0068ad0 msgr2=0x7fedd01960d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:53.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.291+0000 7fedcf7fe700 1 --2- 172.21.15.44:0/971534018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedd0068ad0 0x7fedd01960d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.295 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.291+0000 7fedcf7fe700 1 -- 172.21.15.44:0/971534018 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc8009d00 con 0x7fedd0103380 2023-11-16T16:46:53.295 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.291+0000 7fedcf7fe700 1 --2- 172.21.15.44:0/971534018 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedd0103380 0x7fedd0196610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fedc400e9a0 tx=0x7fedc400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:53.295 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.291+0000 7fedcd7fa700 1 -- 172.21.15.44:0/971534018 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc4009790 con 0x7fedd0103380 2023-11-16T16:46:53.295 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.291+0000 7fedcd7fa700 1 -- 172.21.15.44:0/971534018 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fedc400cbd0 con 0x7fedd0103380 2023-11-16T16:46:53.295 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.291+0000 7fedd63ff700 1 -- 172.21.15.44:0/971534018 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd010bc50 con 0x7fedd0103380 2023-11-16T16:46:53.295 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.291+0000 7fedcd7fa700 1 -- 172.21.15.44:0/971534018 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc4018680 con 0x7fedd0103380 2023-11-16T16:46:53.295 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.291+0000 7fedd63ff700 1 -- 172.21.15.44:0/971534018 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd010bac0 con 0x7fedd0103380 2023-11-16T16:46:53.295 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.292+0000 7fedd63ff700 1 -- 172.21.15.44:0/971534018 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd00fc2c0 con 0x7fedd0103380 2023-11-16T16:46:53.296 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.295+0000 7fedcd7fa700 1 -- 172.21.15.44:0/971534018 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fedc400cd40 con 0x7fedd0103380 2023-11-16T16:46:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.296+0000 7fedcd7fa700 1 --2- 172.21.15.44:0/971534018 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fedb80a11d0 0x7fedb80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.296+0000 7fedcd7fa700 1 -- 172.21.15.44:0/971534018 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fedc4014070 con 0x7fedd0103380 2023-11-16T16:46:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.296+0000 7fedcffff700 1 --2- 172.21.15.44:0/971534018 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fedb80a11d0 0x7fedb80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.297+0000 7fedcffff700 1 --2- 172.21.15.44:0/971534018 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fedb80a11d0 0x7fedb80a3680 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fedd00687f0 tx=0x7fedc0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:53.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.299+0000 7fedcd7fa700 1 -- 172.21.15.44:0/971534018 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fedc40964f0 con 0x7fedd0103380 2023-11-16T16:46:53.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.554+0000 7fedd63ff700 1 -- 172.21.15.44:0/971534018 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedd002cfe0 con 0x7fedd0103380 2023-11-16T16:46:53.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.555+0000 7fedcd7fa700 1 -- 172.21.15.44:0/971534018 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3519 (secure 0 0 0) 0x7fedc4026dd0 con 0x7fedd0103380 2023-11-16T16:46:53.556 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:46:53.557 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-11-16T16:46:53.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.557+0000 7fedd63ff700 1 -- 172.21.15.44:0/971534018 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fedb80a11d0 msgr2=0x7fedb80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:53.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.557+0000 7fedd63ff700 1 --2- 172.21.15.44:0/971534018 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fedb80a11d0 0x7fedb80a3680 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fedd00687f0 tx=0x7fedc0009260 comp rx=0 tx=0).stop 2023-11-16T16:46:53.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.557+0000 7fedd63ff700 1 -- 172.21.15.44:0/971534018 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedd0103380 msgr2=0x7fedd0196610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:53.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.557+0000 7fedd63ff700 1 --2- 172.21.15.44:0/971534018 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedd0103380 0x7fedd0196610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fedc400e9a0 tx=0x7fedc400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.557+0000 7fedd63ff700 1 -- 172.21.15.44:0/971534018 shutdown_connections 2023-11-16T16:46:53.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.557+0000 7fedd63ff700 1 --2- 172.21.15.44:0/971534018 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fedb80a11d0 0x7fedb80a3680 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.557+0000 7fedd63ff700 1 --2- 172.21.15.44:0/971534018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedd0105e50 0x7fedd019d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.557+0000 7fedd63ff700 1 --2- 172.21.15.44:0/971534018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedd0068ad0 0x7fedd01960d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.557+0000 7fedd63ff700 1 --2- 172.21.15.44:0/971534018 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedd0103380 0x7fedd0196610 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.557+0000 7fedd63ff700 1 -- 172.21.15.44:0/971534018 >> 172.21.15.44:0/971534018 conn(0x7fedd00fa750 msgr2=0x7fedd01016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:53.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.557+0000 7fedd63ff700 1 -- 172.21.15.44:0/971534018 shutdown_connections 2023-11-16T16:46:53.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:53.557+0000 7fedd63ff700 1 -- 172.21.15.44:0/971534018 wait complete. 2023-11-16T16:46:53.560 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 5 2023-11-16T16:46:53.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:53 smithi044 ceph-mon[128750]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:53.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:53 smithi044 ceph-mon[128750]: mds.? [v2:172.21.15.44:6836/159805710,v1:172.21.15.44:6837/159805710] up:boot 2023-11-16T16:46:53.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:53 smithi044 ceph-mon[128750]: fsmap 4 up:standby 2023-11-16T16:46:53.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T16:46:53.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:53.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:53.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:53.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:53.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:53.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:53 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:53.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:53 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/971534018' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:53.941 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:53 smithi077 ceph-mon[131204]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:53.941 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:53 smithi077 ceph-mon[131204]: mds.? [v2:172.21.15.44:6836/159805710,v1:172.21.15.44:6837/159805710] up:boot 2023-11-16T16:46:53.941 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:53 smithi077 ceph-mon[131204]: fsmap 4 up:standby 2023-11-16T16:46:53.941 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T16:46:53.942 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:53.942 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:53.942 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:53.942 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:53.942 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:53.942 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:53 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:53.942 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:53 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/971534018' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:53 smithi176 ceph-mon[131630]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:53 smithi176 ceph-mon[131630]: mds.? [v2:172.21.15.44:6836/159805710,v1:172.21.15.44:6837/159805710] up:boot 2023-11-16T16:46:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:53 smithi176 ceph-mon[131630]: fsmap 4 up:standby 2023-11-16T16:46:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T16:46:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:53 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:53 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/971534018' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:54.477 INFO:journalctl@ceph.mds.e.smithi077.stdout:Nov 16 16:46:54 smithi077 systemd[1]: Starting Ceph mds.e for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:46:54.964 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:54 smithi077 ceph-mon[131204]: Deploying daemon mds.e on smithi077 2023-11-16T16:46:54.964 INFO:journalctl@ceph.mds.e.smithi077.stdout:Nov 16 16:46:54 smithi077 podman[155830]: 2023-11-16T16:46:55.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:54 smithi044 ceph-mon[128750]: Deploying daemon mds.e on smithi077 2023-11-16T16:46:55.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:54 smithi176 ceph-mon[131630]: Deploying daemon mds.e on smithi077 2023-11-16T16:46:55.230 INFO:journalctl@ceph.mds.e.smithi077.stdout:Nov 16 16:46:54 smithi077 ceph-mds[155870]: -- 172.21.15.77:0/3416765171 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d5875f3520 con 0x55d5875c9000 2023-11-16T16:46:55.230 INFO:journalctl@ceph.mds.e.smithi077.stdout:Nov 16 16:46:54 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-e[155846]: starting mds.e at 2023-11-16T16:46:55.230 INFO:journalctl@ceph.mds.e.smithi077.stdout:Nov 16 16:46:54 smithi077 bash[155830]: dafc8b7cc14e946e14517e8aedbef1ac75b9e36ae9a3fa65796aea184d971805 2023-11-16T16:46:55.231 INFO:journalctl@ceph.mds.e.smithi077.stdout:Nov 16 16:46:55 smithi077 systemd[1]: Started Ceph mds.e for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:46:55.580 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:46:55.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.798+0000 7f7afebff700 1 -- 172.21.15.44:0/1653358286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7af8101480 msgr2=0x7f7af81018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:55.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.798+0000 7f7afebff700 1 --2- 172.21.15.44:0/1653358286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7af8101480 0x7f7af81018d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009990 tx=0x7f7ae4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:46:55.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.798+0000 7f7afebff700 1 -- 172.21.15.44:0/1653358286 shutdown_connections 2023-11-16T16:46:55.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.798+0000 7f7afebff700 1 --2- 172.21.15.44:0/1653358286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af8101e10 0x7f7af81086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:55.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.798+0000 7f7afebff700 1 --2- 172.21.15.44:0/1653358286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7af8100290 0x7f7af81006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:55.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.798+0000 7f7afebff700 1 --2- 172.21.15.44:0/1653358286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7af8101480 0x7f7af81018d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:55.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.798+0000 7f7afebff700 1 -- 172.21.15.44:0/1653358286 >> 172.21.15.44:0/1653358286 conn(0x7f7af80fb870 msgr2=0x7f7af80fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:55.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.798+0000 7f7afebff700 1 -- 172.21.15.44:0/1653358286 shutdown_connections 2023-11-16T16:46:55.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.798+0000 7f7afebff700 1 -- 172.21.15.44:0/1653358286 wait complete. 2023-11-16T16:46:55.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.799+0000 7f7afebff700 1 Processor -- start 2023-11-16T16:46:55.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.799+0000 7f7afebff700 1 -- start start 2023-11-16T16:46:55.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7afebff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af8100290 0x7f7af81981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:55.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7afebff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7af8101480 0x7f7af8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7afebff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7af8101e10 0x7f7af819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7afebff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af819fd50 con 0x7f7af8101e10 2023-11-16T16:46:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7afebff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af819fe90 con 0x7f7af8100290 2023-11-16T16:46:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7afebff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af81a0020 con 0x7f7af8101480 2023-11-16T16:46:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7afd19c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7af8101e10 0x7f7af819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7afc99b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af8100290 0x7f7af81981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7afd19c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7af8101e10 0x7f7af819f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44702/0 (socket says 172.21.15.44:44702) 2023-11-16T16:46:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7afd19c700 1 -- 172.21.15.44:0/4049396635 learned_addr learned my addr 172.21.15.44:0/4049396635 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:46:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7af7fff700 1 --2- 172.21.15.44:0/4049396635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7af8101480 0x7f7af8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7afc99b700 1 -- 172.21.15.44:0/4049396635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7af8101480 msgr2=0x7f7af8198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7afc99b700 1 --2- 172.21.15.44:0/4049396635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7af8101480 0x7f7af8198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7afc99b700 1 -- 172.21.15.44:0/4049396635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7af8101e10 msgr2=0x7f7af819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7afc99b700 1 --2- 172.21.15.44:0/4049396635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7af8101e10 0x7f7af819f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:55.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7afc99b700 1 -- 172.21.15.44:0/4049396635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aec009520 con 0x7f7af8100290 2023-11-16T16:46:55.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.800+0000 7f7afd19c700 1 --2- 172.21.15.44:0/4049396635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7af8101e10 0x7f7af819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:46:55.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.801+0000 7f7afc99b700 1 --2- 172.21.15.44:0/4049396635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af8100290 0x7f7af81981d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00e9c0 tx=0x7f7aec00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:55.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.801+0000 7f7af5ffb700 1 -- 172.21.15.44:0/4049396635 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec00cb70 con 0x7f7af8100290 2023-11-16T16:46:55.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.801+0000 7f7af5ffb700 1 -- 172.21.15.44:0/4049396635 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7aec00ee60 con 0x7f7af8100290 2023-11-16T16:46:55.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.801+0000 7f7afebff700 1 -- 172.21.15.44:0/4049396635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae40095f0 con 0x7f7af8100290 2023-11-16T16:46:55.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.801+0000 7f7af5ffb700 1 -- 172.21.15.44:0/4049396635 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec018680 con 0x7f7af8100290 2023-11-16T16:46:55.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.801+0000 7f7afebff700 1 -- 172.21.15.44:0/4049396635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af810bc50 con 0x7f7af8100290 2023-11-16T16:46:55.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.802+0000 7f7afebff700 1 -- 172.21.15.44:0/4049396635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af81928e0 con 0x7f7af8100290 2023-11-16T16:46:55.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.804+0000 7f7af5ffb700 1 -- 172.21.15.44:0/4049396635 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7aec010430 con 0x7f7af8100290 2023-11-16T16:46:55.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.805+0000 7f7af5ffb700 1 --2- 172.21.15.44:0/4049396635 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7ae80a11d0 0x7f7ae80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:55.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.806+0000 7f7af5ffb700 1 -- 172.21.15.44:0/4049396635 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7aec014070 con 0x7f7af8100290 2023-11-16T16:46:55.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.806+0000 7f7af7fff700 1 --2- 172.21.15.44:0/4049396635 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7ae80a11d0 0x7f7ae80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:55.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.806+0000 7f7af7fff700 1 --2- 172.21.15.44:0/4049396635 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7ae80a11d0 0x7f7ae80a3680 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400b2e0 tx=0x7f7ae4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:55.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:55.808+0000 7f7af5ffb700 1 -- 172.21.15.44:0/4049396635 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7aec096360 con 0x7f7af8100290 2023-11-16T16:46:56.035 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:55 smithi176 ceph-mon[131630]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:56.035 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:55 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:56.035 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:55 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:56.035 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:55 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:56.035 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:55 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:56.035 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:55 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:56.035 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:55 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:56.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:56.065+0000 7f7afebff700 1 -- 172.21.15.44:0/4049396635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7af802cfe0 con 0x7f7af8100290 2023-11-16T16:46:56.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:56.066+0000 7f7af5ffb700 1 -- 172.21.15.44:0/4049396635 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4276 (secure 0 0 0) 0x7f7aec00ce60 con 0x7f7af8100290 2023-11-16T16:46:56.068 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:46:56.068 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-11-16T16:46:56.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:56.068+0000 7f7afebff700 1 -- 172.21.15.44:0/4049396635 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7ae80a11d0 msgr2=0x7f7ae80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:56.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:56.068+0000 7f7afebff700 1 --2- 172.21.15.44:0/4049396635 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7ae80a11d0 0x7f7ae80a3680 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400b2e0 tx=0x7f7ae4005fb0 comp rx=0 tx=0).stop 2023-11-16T16:46:56.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:56.068+0000 7f7afebff700 1 -- 172.21.15.44:0/4049396635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af8100290 msgr2=0x7f7af81981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:56.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:56.068+0000 7f7afebff700 1 --2- 172.21.15.44:0/4049396635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af8100290 0x7f7af81981d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00e9c0 tx=0x7f7aec00ecd0 comp rx=0 tx=0).stop 2023-11-16T16:46:56.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:56.068+0000 7f7afebff700 1 -- 172.21.15.44:0/4049396635 shutdown_connections 2023-11-16T16:46:56.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:56.068+0000 7f7afebff700 1 --2- 172.21.15.44:0/4049396635 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7ae80a11d0 0x7f7ae80a3680 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:56.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:56.068+0000 7f7afebff700 1 --2- 172.21.15.44:0/4049396635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7af8101e10 0x7f7af819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:56.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:56.068+0000 7f7afebff700 1 --2- 172.21.15.44:0/4049396635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af8100290 0x7f7af81981d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:56.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:56.068+0000 7f7afebff700 1 --2- 172.21.15.44:0/4049396635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7af8101480 0x7f7af8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:56.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:56.068+0000 7f7afebff700 1 -- 172.21.15.44:0/4049396635 >> 172.21.15.44:0/4049396635 conn(0x7f7af80fb870 msgr2=0x7f7af8106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:56.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:56.068+0000 7f7afebff700 1 -- 172.21.15.44:0/4049396635 shutdown_connections 2023-11-16T16:46:56.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:56.068+0000 7f7afebff700 1 -- 172.21.15.44:0/4049396635 wait complete. 2023-11-16T16:46:56.072 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 6 2023-11-16T16:46:56.087 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:55 smithi044 ceph-mon[128750]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:56.087 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:56.087 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:56.087 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:56.087 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:56.087 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:56.087 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:55 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:56.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:55 smithi077 ceph-mon[131204]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:56.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:55 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:56.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:55 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:56.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:55 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:56.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:55 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:55 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:55 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:56.645 INFO:journalctl@ceph.mds.f.smithi176.stdout:Nov 16 16:46:56 smithi176 systemd[1]: Starting Ceph mds.f for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:46:57.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:56 smithi044 ceph-mon[128750]: Deploying daemon mds.f on smithi176 2023-11-16T16:46:57.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:56 smithi044 ceph-mon[128750]: mds.? [v2:172.21.15.77:6834/1355935860,v1:172.21.15.77:6835/1355935860] up:boot 2023-11-16T16:46:57.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:56 smithi044 ceph-mon[128750]: fsmap 5 up:standby 2023-11-16T16:46:57.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:56 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T16:46:57.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:56 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/4049396635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:57.127 INFO:journalctl@ceph.mds.f.smithi176.stdout:Nov 16 16:46:56 smithi176 podman[157297]: 2023-11-16T16:46:57.127 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:56 smithi176 ceph-mon[131630]: Deploying daemon mds.f on smithi176 2023-11-16T16:46:57.127 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:56 smithi176 ceph-mon[131630]: mds.? [v2:172.21.15.77:6834/1355935860,v1:172.21.15.77:6835/1355935860] up:boot 2023-11-16T16:46:57.128 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:56 smithi176 ceph-mon[131630]: fsmap 5 up:standby 2023-11-16T16:46:57.128 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:56 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T16:46:57.128 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:56 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/4049396635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:57.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:56 smithi077 ceph-mon[131204]: Deploying daemon mds.f on smithi176 2023-11-16T16:46:57.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:56 smithi077 ceph-mon[131204]: mds.? [v2:172.21.15.77:6834/1355935860,v1:172.21.15.77:6835/1355935860] up:boot 2023-11-16T16:46:57.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:56 smithi077 ceph-mon[131204]: fsmap 5 up:standby 2023-11-16T16:46:57.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:56 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T16:46:57.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:56 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/4049396635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:57.395 INFO:journalctl@ceph.mds.f.smithi176.stdout:Nov 16 16:46:57 smithi176 ceph-mds[157337]: -- 172.21.15.176:0/1352236471 <== mon.1 v2:172.21.15.176:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556163139520 con 0x55616310ec00 2023-11-16T16:46:57.395 INFO:journalctl@ceph.mds.f.smithi176.stdout:Nov 16 16:46:57 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-f[157313]: starting mds.f at 2023-11-16T16:46:57.395 INFO:journalctl@ceph.mds.f.smithi176.stdout:Nov 16 16:46:57 smithi176 bash[157297]: 402c4a9c283bb3888dc70918caafeb4392946bb7a77b9a56cf70b9bbbc17882d 2023-11-16T16:46:57.395 INFO:journalctl@ceph.mds.f.smithi176.stdout:Nov 16 16:46:57 smithi176 systemd[1]: Started Ceph mds.f for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:46:58.033 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:57 smithi176 ceph-mon[131630]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:58.033 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:57 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:58.033 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:57 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:58.033 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:57 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:58.033 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:57 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:58.033 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:57 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:58.033 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:57 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:58.090 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:46:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:57 smithi044 ceph-mon[128750]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:57 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:57 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:57 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:58.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:57 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:58.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:57 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:58.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:57 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:58.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:57 smithi077 ceph-mon[131204]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:58.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:57 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:57 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:57 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:46:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:57 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:57 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:58.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:57 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:58.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.303+0000 7f77f33ff700 1 -- 172.21.15.44:0/3637976882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec101480 msgr2=0x7f77ec1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:58.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.303+0000 7f77f33ff700 1 --2- 172.21.15.44:0/3637976882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec101480 0x7f77ec1018d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009990 tx=0x7f77dc009ca0 comp rx=0 tx=0).stop 2023-11-16T16:46:58.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.304+0000 7f77f33ff700 1 -- 172.21.15.44:0/3637976882 shutdown_connections 2023-11-16T16:46:58.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.304+0000 7f77f33ff700 1 --2- 172.21.15.44:0/3637976882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77ec101e10 0x7f77ec1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:58.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.304+0000 7f77f33ff700 1 --2- 172.21.15.44:0/3637976882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77ec100290 0x7f77ec1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:58.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.304+0000 7f77f33ff700 1 --2- 172.21.15.44:0/3637976882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec101480 0x7f77ec1018d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:58.307 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.304+0000 7f77f33ff700 1 -- 172.21.15.44:0/3637976882 >> 172.21.15.44:0/3637976882 conn(0x7f77ec0fb870 msgr2=0x7f77ec0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:58.307 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.304+0000 7f77f33ff700 1 -- 172.21.15.44:0/3637976882 shutdown_connections 2023-11-16T16:46:58.307 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.304+0000 7f77f33ff700 1 -- 172.21.15.44:0/3637976882 wait complete. 2023-11-16T16:46:58.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.305+0000 7f77f33ff700 1 Processor -- start 2023-11-16T16:46:58.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.305+0000 7f77f33ff700 1 -- start start 2023-11-16T16:46:58.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.305+0000 7f77f33ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77ec100290 0x7f77ec0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:58.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.305+0000 7f77f33ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec101480 0x7f77ec072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:58.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.305+0000 7f77f33ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77ec101e10 0x7f77ec073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:58.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.305+0000 7f77f33ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec073bc0 con 0x7f77ec101480 2023-11-16T16:46:58.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.305+0000 7f77f33ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec073d30 con 0x7f77ec101e10 2023-11-16T16:46:58.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.305+0000 7f77f33ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec19fff0 con 0x7f77ec100290 2023-11-16T16:46:58.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.305+0000 7f77f099a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec101480 0x7f77ec072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:58.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.305+0000 7f77f099a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec101480 0x7f77ec072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44732/0 (socket says 172.21.15.44:44732) 2023-11-16T16:46:58.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.305+0000 7f77f099a700 1 -- 172.21.15.44:0/1658834741 learned_addr learned my addr 172.21.15.44:0/1658834741 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:46:58.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.306+0000 7f77f119b700 1 --2- 172.21.15.44:0/1658834741 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77ec100290 0x7f77ec0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:58.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.306+0000 7f77f199c700 1 --2- 172.21.15.44:0/1658834741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77ec101e10 0x7f77ec073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:58.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.306+0000 7f77f099a700 1 -- 172.21.15.44:0/1658834741 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77ec100290 msgr2=0x7f77ec0748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:58.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.306+0000 7f77f099a700 1 --2- 172.21.15.44:0/1658834741 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77ec100290 0x7f77ec0748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:58.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.306+0000 7f77f099a700 1 -- 172.21.15.44:0/1658834741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77ec101e10 msgr2=0x7f77ec073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:58.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.306+0000 7f77f099a700 1 --2- 172.21.15.44:0/1658834741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77ec101e10 0x7f77ec073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:58.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.306+0000 7f77f099a700 1 -- 172.21.15.44:0/1658834741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77dc0095f0 con 0x7f77ec101480 2023-11-16T16:46:58.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.306+0000 7f77f099a700 1 --2- 172.21.15.44:0/1658834741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec101480 0x7f77ec072ee0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00b620 tx=0x7f77dc005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:58.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.306+0000 7f77e27fc700 1 -- 172.21.15.44:0/1658834741 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc01d070 con 0x7f77ec101480 2023-11-16T16:46:58.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.306+0000 7f77f33ff700 1 -- 172.21.15.44:0/1658834741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ec10bc50 con 0x7f77ec101480 2023-11-16T16:46:58.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.306+0000 7f77e27fc700 1 -- 172.21.15.44:0/1658834741 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77dc022470 con 0x7f77ec101480 2023-11-16T16:46:58.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.306+0000 7f77e27fc700 1 -- 172.21.15.44:0/1658834741 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc017650 con 0x7f77ec101480 2023-11-16T16:46:58.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.307+0000 7f77f33ff700 1 -- 172.21.15.44:0/1658834741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ec10bac0 con 0x7f77ec101480 2023-11-16T16:46:58.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.307+0000 7f77f33ff700 1 -- 172.21.15.44:0/1658834741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ec1a05a0 con 0x7f77ec101480 2023-11-16T16:46:58.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.310+0000 7f77e27fc700 1 -- 172.21.15.44:0/1658834741 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f77dc0225e0 con 0x7f77ec101480 2023-11-16T16:46:58.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.311+0000 7f77e27fc700 1 --2- 172.21.15.44:0/1658834741 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f77d80a1150 0x7f77d80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:58.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.311+0000 7f77e27fc700 1 -- 172.21.15.44:0/1658834741 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f77dc0ce180 con 0x7f77ec101480 2023-11-16T16:46:58.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.311+0000 7f77f119b700 1 --2- 172.21.15.44:0/1658834741 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f77d80a1150 0x7f77d80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:58.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.312+0000 7f77f119b700 1 --2- 172.21.15.44:0/1658834741 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f77d80a1150 0x7f77d80a3600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f77ec1012e0 tx=0x7f77e8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:58.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.313+0000 7f77e27fc700 1 -- 172.21.15.44:0/1658834741 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f77dc097d60 con 0x7f77ec101480 2023-11-16T16:46:58.574 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.573+0000 7f77f33ff700 1 -- 172.21.15.44:0/1658834741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77ec1071d0 con 0x7f77ec101480 2023-11-16T16:46:58.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.573+0000 7f77e27fc700 1 -- 172.21.15.44:0/1658834741 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5036 (secure 0 0 0) 0x7f77dc09b380 con 0x7f77ec101480 2023-11-16T16:46:58.575 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:46:58.576 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2023-11-16T16:46:58.578 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.576+0000 7f77f33ff700 1 -- 172.21.15.44:0/1658834741 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f77d80a1150 msgr2=0x7f77d80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:58.578 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.576+0000 7f77f33ff700 1 --2- 172.21.15.44:0/1658834741 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f77d80a1150 0x7f77d80a3600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f77ec1012e0 tx=0x7f77e8009260 comp rx=0 tx=0).stop 2023-11-16T16:46:58.578 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.576+0000 7f77f33ff700 1 -- 172.21.15.44:0/1658834741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec101480 msgr2=0x7f77ec072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:58.578 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.576+0000 7f77f33ff700 1 --2- 172.21.15.44:0/1658834741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec101480 0x7f77ec072ee0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00b620 tx=0x7f77dc005740 comp rx=0 tx=0).stop 2023-11-16T16:46:58.579 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.576+0000 7f77f33ff700 1 -- 172.21.15.44:0/1658834741 shutdown_connections 2023-11-16T16:46:58.579 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.576+0000 7f77f33ff700 1 --2- 172.21.15.44:0/1658834741 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f77d80a1150 0x7f77d80a3600 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:58.579 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.576+0000 7f77f33ff700 1 --2- 172.21.15.44:0/1658834741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77ec101e10 0x7f77ec073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:58.579 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.576+0000 7f77f33ff700 1 --2- 172.21.15.44:0/1658834741 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77ec100290 0x7f77ec0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:58.579 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.576+0000 7f77f33ff700 1 --2- 172.21.15.44:0/1658834741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec101480 0x7f77ec072ee0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:58.579 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.576+0000 7f77f33ff700 1 -- 172.21.15.44:0/1658834741 >> 172.21.15.44:0/1658834741 conn(0x7f77ec0fb870 msgr2=0x7f77ec0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:58.579 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.576+0000 7f77f33ff700 1 -- 172.21.15.44:0/1658834741 shutdown_connections 2023-11-16T16:46:58.579 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:46:58.576+0000 7f77f33ff700 1 -- 172.21.15.44:0/1658834741 wait complete. 2023-11-16T16:46:58.580 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 7 2023-11-16T16:46:58.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:58 smithi044 ceph-mon[128750]: Deploying daemon mds.c on smithi176 2023-11-16T16:46:58.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:58 smithi044 ceph-mon[128750]: mds.? [v2:172.21.15.176:6834/1665062606,v1:172.21.15.176:6835/1665062606] up:boot 2023-11-16T16:46:58.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:58 smithi044 ceph-mon[128750]: fsmap 6 up:standby 2023-11-16T16:46:58.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:58 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T16:46:58.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:58 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1658834741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:59.102 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:58 smithi176 ceph-mon[131630]: Deploying daemon mds.c on smithi176 2023-11-16T16:46:59.102 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:58 smithi176 ceph-mon[131630]: mds.? [v2:172.21.15.176:6834/1665062606,v1:172.21.15.176:6835/1665062606] up:boot 2023-11-16T16:46:59.103 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:58 smithi176 ceph-mon[131630]: fsmap 6 up:standby 2023-11-16T16:46:59.103 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:58 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T16:46:59.103 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:58 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1658834741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:59.103 INFO:journalctl@ceph.mds.c.smithi176.stdout:Nov 16 16:46:58 smithi176 systemd[1]: Starting Ceph mds.c for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:46:59.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:58 smithi077 ceph-mon[131204]: Deploying daemon mds.c on smithi176 2023-11-16T16:46:59.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:58 smithi077 ceph-mon[131204]: mds.? [v2:172.21.15.176:6834/1665062606,v1:172.21.15.176:6835/1665062606] up:boot 2023-11-16T16:46:59.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:58 smithi077 ceph-mon[131204]: fsmap 6 up:standby 2023-11-16T16:46:59.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:58 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T16:46:59.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:58 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1658834741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:59.388 INFO:journalctl@ceph.mds.c.smithi176.stdout:Nov 16 16:46:59 smithi176 podman[157636]: 2023-11-16T16:46:59.389 INFO:journalctl@ceph.mds.c.smithi176.stdout:Nov 16 16:46:59 smithi176 ceph-mds[157676]: -- 172.21.15.176:0/1535127737 <== mon.1 v2:172.21.15.176:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557c637df520 con 0x557c637b4800 2023-11-16T16:46:59.389 INFO:journalctl@ceph.mds.c.smithi176.stdout:Nov 16 16:46:59 smithi176 bash[157636]: d553725e39cfecd8d38bf47f1efb53df49970bf201449fda5bdfbc062e487a2e 2023-11-16T16:46:59.389 INFO:journalctl@ceph.mds.c.smithi176.stdout:Nov 16 16:46:59 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-c[157652]: starting mds.c at 2023-11-16T16:46:59.645 INFO:journalctl@ceph.mds.c.smithi176.stdout:Nov 16 16:46:59 smithi176 systemd[1]: Started Ceph mds.c for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:47:00.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:59 smithi044 ceph-mon[128750]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:00.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:00.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:00.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:00.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:00.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:00.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:46:59 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:00.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:59 smithi176 ceph-mon[131630]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:00.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:00.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:00.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:00.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:00.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:00.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:46:59 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:00.196 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:59 smithi077 ceph-mon[131204]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:00.196 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:00.196 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:00.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:00.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:00.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:00.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:46:59 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:00.598 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:00.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.812+0000 7ff037dff700 1 -- 172.21.15.44:0/228357696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff030073550 msgr2=0x7ff0300739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:00.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.812+0000 7ff037dff700 1 --2- 172.21.15.44:0/228357696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff030073550 0x7ff0300739c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff020009990 tx=0x7ff020009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:00.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.812+0000 7ff037dff700 1 -- 172.21.15.44:0/228357696 shutdown_connections 2023-11-16T16:47:00.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.812+0000 7ff037dff700 1 --2- 172.21.15.44:0/228357696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff03010c880 0x7ff03010ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:00.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.812+0000 7ff037dff700 1 --2- 172.21.15.44:0/228357696 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff030074b30 0x7ff030072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:00.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.812+0000 7ff037dff700 1 --2- 172.21.15.44:0/228357696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff030073550 0x7ff0300739c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:00.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.812+0000 7ff037dff700 1 -- 172.21.15.44:0/228357696 >> 172.21.15.44:0/228357696 conn(0x7ff0300fb890 msgr2=0x7ff0300fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:00.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.813+0000 7ff037dff700 1 -- 172.21.15.44:0/228357696 shutdown_connections 2023-11-16T16:47:00.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.813+0000 7ff037dff700 1 -- 172.21.15.44:0/228357696 wait complete. 2023-11-16T16:47:00.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.814+0000 7ff037dff700 1 Processor -- start 2023-11-16T16:47:00.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.814+0000 7ff037dff700 1 -- start start 2023-11-16T16:47:00.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.814+0000 7ff037dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff030073550 0x7ff03019c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:00.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.814+0000 7ff037dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff030074b30 0x7ff03019cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:00.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.814+0000 7ff037dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03010c880 0x7ff0301a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:00.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.814+0000 7ff037dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0301a4170 con 0x7ff030074b30 2023-11-16T16:47:00.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.814+0000 7ff037dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0301a42b0 con 0x7ff030073550 2023-11-16T16:47:00.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.814+0000 7ff037dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0301a4440 con 0x7ff03010c880 2023-11-16T16:47:00.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.814+0000 7ff035b9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff030073550 0x7ff03019c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:00.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.814+0000 7ff035b9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff030073550 0x7ff03019c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:34318/0 (socket says 172.21.15.44:34318) 2023-11-16T16:47:00.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.814+0000 7ff035b9b700 1 -- 172.21.15.44:0/3567716709 learned_addr learned my addr 172.21.15.44:0/3567716709 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:00.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.814+0000 7ff03539a700 1 --2- 172.21.15.44:0/3567716709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff030074b30 0x7ff03019cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:00.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.815+0000 7ff03639c700 1 --2- 172.21.15.44:0/3567716709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03010c880 0x7ff0301a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:00.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.815+0000 7ff035b9b700 1 -- 172.21.15.44:0/3567716709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03010c880 msgr2=0x7ff0301a3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:00.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.815+0000 7ff035b9b700 1 --2- 172.21.15.44:0/3567716709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03010c880 0x7ff0301a3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:00.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.815+0000 7ff035b9b700 1 -- 172.21.15.44:0/3567716709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff030074b30 msgr2=0x7ff03019cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:00.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.815+0000 7ff035b9b700 1 --2- 172.21.15.44:0/3567716709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff030074b30 0x7ff03019cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:00.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.815+0000 7ff035b9b700 1 -- 172.21.15.44:0/3567716709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0200095f0 con 0x7ff030073550 2023-11-16T16:47:00.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.815+0000 7ff035b9b700 1 --2- 172.21.15.44:0/3567716709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff030073550 0x7ff03019c5f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff02c00e970 tx=0x7ff02c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:00.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.815+0000 7ff026ffd700 1 -- 172.21.15.44:0/3567716709 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c009790 con 0x7ff030073550 2023-11-16T16:47:00.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.815+0000 7ff037dff700 1 -- 172.21.15.44:0/3567716709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff030112290 con 0x7ff030073550 2023-11-16T16:47:00.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.815+0000 7ff026ffd700 1 -- 172.21.15.44:0/3567716709 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff02c00eec0 con 0x7ff030073550 2023-11-16T16:47:00.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.815+0000 7ff026ffd700 1 -- 172.21.15.44:0/3567716709 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c005590 con 0x7ff030073550 2023-11-16T16:47:00.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.815+0000 7ff037dff700 1 -- 172.21.15.44:0/3567716709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff030112100 con 0x7ff030073550 2023-11-16T16:47:00.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.816+0000 7ff037dff700 1 -- 172.21.15.44:0/3567716709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0301023d0 con 0x7ff030073550 2023-11-16T16:47:00.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.818+0000 7ff026ffd700 1 -- 172.21.15.44:0/3567716709 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff02c00cbd0 con 0x7ff030073550 2023-11-16T16:47:00.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.820+0000 7ff026ffd700 1 --2- 172.21.15.44:0/3567716709 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff01c0a11d0 0x7ff01c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:00.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.820+0000 7ff026ffd700 1 -- 172.21.15.44:0/3567716709 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff02c014070 con 0x7ff030073550 2023-11-16T16:47:00.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.820+0000 7ff03539a700 1 --2- 172.21.15.44:0/3567716709 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff01c0a11d0 0x7ff01c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:00.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.820+0000 7ff03539a700 1 --2- 172.21.15.44:0/3567716709 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff01c0a11d0 0x7ff01c0a3680 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff02000b2e0 tx=0x7ff020005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:00.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:00.824+0000 7ff026ffd700 1 -- 172.21.15.44:0/3567716709 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff02c096250 con 0x7ff030073550 2023-11-16T16:47:01.082 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:01.080+0000 7ff037dff700 1 -- 172.21.15.44:0/3567716709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff03002cfe0 con 0x7ff030073550 2023-11-16T16:47:01.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:01.081+0000 7ff026ffd700 1 -- 172.21.15.44:0/3567716709 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5796 (secure 0 0 0) 0x7ff02c004ef0 con 0x7ff030073550 2023-11-16T16:47:01.083 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:01.083 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-11-16T16:47:01.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:01.083+0000 7ff037dff700 1 -- 172.21.15.44:0/3567716709 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff01c0a11d0 msgr2=0x7ff01c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:01.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:01.083+0000 7ff037dff700 1 --2- 172.21.15.44:0/3567716709 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff01c0a11d0 0x7ff01c0a3680 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff02000b2e0 tx=0x7ff020005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:01.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:01.083+0000 7ff037dff700 1 -- 172.21.15.44:0/3567716709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff030073550 msgr2=0x7ff03019c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:01.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:01.083+0000 7ff037dff700 1 --2- 172.21.15.44:0/3567716709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff030073550 0x7ff03019c5f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff02c00e970 tx=0x7ff02c00ec80 comp rx=0 tx=0).stop 2023-11-16T16:47:01.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:01.083+0000 7ff037dff700 1 -- 172.21.15.44:0/3567716709 shutdown_connections 2023-11-16T16:47:01.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:01.083+0000 7ff037dff700 1 --2- 172.21.15.44:0/3567716709 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff01c0a11d0 0x7ff01c0a3680 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:01.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:01.083+0000 7ff037dff700 1 --2- 172.21.15.44:0/3567716709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03010c880 0x7ff0301a3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:01.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:01.083+0000 7ff037dff700 1 --2- 172.21.15.44:0/3567716709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff030073550 0x7ff03019c5f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:01.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:01.083+0000 7ff037dff700 1 --2- 172.21.15.44:0/3567716709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff030074b30 0x7ff03019cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:01.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:01.083+0000 7ff037dff700 1 -- 172.21.15.44:0/3567716709 >> 172.21.15.44:0/3567716709 conn(0x7ff0300fb890 msgr2=0x7ff0300fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:01.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:01.084+0000 7ff037dff700 1 -- 172.21.15.44:0/3567716709 shutdown_connections 2023-11-16T16:47:01.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:01.084+0000 7ff037dff700 1 -- 172.21.15.44:0/3567716709 wait complete. 2023-11-16T16:47:01.087 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 8 2023-11-16T16:47:01.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:00 smithi044 ceph-mon[128750]: Deploying daemon mds.h on smithi077 2023-11-16T16:47:01.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:00 smithi044 ceph-mon[128750]: mds.? [v2:172.21.15.176:6836/3753678422,v1:172.21.15.176:6837/3753678422] up:boot 2023-11-16T16:47:01.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:00 smithi044 ceph-mon[128750]: fsmap 7 up:standby 2023-11-16T16:47:01.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:00 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T16:47:01.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:00 smithi176 ceph-mon[131630]: Deploying daemon mds.h on smithi077 2023-11-16T16:47:01.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:00 smithi176 ceph-mon[131630]: mds.? [v2:172.21.15.176:6836/3753678422,v1:172.21.15.176:6837/3753678422] up:boot 2023-11-16T16:47:01.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:00 smithi176 ceph-mon[131630]: fsmap 7 up:standby 2023-11-16T16:47:01.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:00 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T16:47:01.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:00 smithi077 ceph-mon[131204]: Deploying daemon mds.h on smithi077 2023-11-16T16:47:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:00 smithi077 ceph-mon[131204]: mds.? [v2:172.21.15.176:6836/3753678422,v1:172.21.15.176:6837/3753678422] up:boot 2023-11-16T16:47:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:00 smithi077 ceph-mon[131204]: fsmap 7 up:standby 2023-11-16T16:47:01.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:00 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T16:47:01.227 INFO:journalctl@ceph.mds.h.smithi077.stdout:Nov 16 16:47:00 smithi077 systemd[1]: Starting Ceph mds.h for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:47:01.564 INFO:journalctl@ceph.mds.h.smithi077.stdout:Nov 16 16:47:01 smithi077 podman[156168]: 2023-11-16T16:47:01.564 INFO:journalctl@ceph.mds.h.smithi077.stdout:Nov 16 16:47:01 smithi077 ceph-mds[156208]: -- 172.21.15.77:0/207073263 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f6d78e5520 con 0x55f6d78bb000 2023-11-16T16:47:01.565 INFO:journalctl@ceph.mds.h.smithi077.stdout:Nov 16 16:47:01 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-h[156184]: starting mds.h at 2023-11-16T16:47:01.976 INFO:journalctl@ceph.mds.h.smithi077.stdout:Nov 16 16:47:01 smithi077 bash[156168]: 5b9e8bd097f33c88a441816d7fd334d8c6e7995d0cd7d825c3dfd45ab08edb61 2023-11-16T16:47:01.976 INFO:journalctl@ceph.mds.h.smithi077.stdout:Nov 16 16:47:01 smithi077 systemd[1]: Started Ceph mds.h for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:47:01.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:01 smithi077 ceph-mon[131204]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:01.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:01 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3567716709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:01.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:01 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:01.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:01 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:01.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:01 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:01.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:01 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:01.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:01 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:01.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:01 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:02.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:01 smithi044 ceph-mon[128750]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:02.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:01 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3567716709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:02.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:01 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:02.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:01 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:02.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:01 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:02.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:01 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:02.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:01 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:02.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:01 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:02.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:01 smithi176 ceph-mon[131630]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:02.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:01 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3567716709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:02.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:01 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:02.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:01 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:02.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:01 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:02.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:01 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:02.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:01 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:02.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:01 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:02 smithi176 ceph-mon[131630]: Deploying daemon mds.i on smithi176 2023-11-16T16:47:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:02 smithi176 ceph-mon[131630]: mds.? [v2:172.21.15.77:6836/440017436,v1:172.21.15.77:6837/440017436] up:boot 2023-11-16T16:47:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:02 smithi176 ceph-mon[131630]: fsmap 8 up:standby 2023-11-16T16:47:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:02 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T16:47:03.105 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:03.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:02 smithi044 ceph-mon[128750]: Deploying daemon mds.i on smithi176 2023-11-16T16:47:03.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:02 smithi044 ceph-mon[128750]: mds.? [v2:172.21.15.77:6836/440017436,v1:172.21.15.77:6837/440017436] up:boot 2023-11-16T16:47:03.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:02 smithi044 ceph-mon[128750]: fsmap 8 up:standby 2023-11-16T16:47:03.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:02 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T16:47:03.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:02 smithi077 ceph-mon[131204]: Deploying daemon mds.i on smithi176 2023-11-16T16:47:03.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:02 smithi077 ceph-mon[131204]: mds.? [v2:172.21.15.77:6836/440017436,v1:172.21.15.77:6837/440017436] up:boot 2023-11-16T16:47:03.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:02 smithi077 ceph-mon[131204]: fsmap 8 up:standby 2023-11-16T16:47:03.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:02 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T16:47:03.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.321+0000 7fb376fff700 1 -- 172.21.15.44:0/1381070803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb370101e10 msgr2=0x7fb3701086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:03.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.321+0000 7fb376fff700 1 --2- 172.21.15.44:0/1381070803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb370101e10 0x7fb3701086d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb36800b0a0 tx=0x7fb36800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.321+0000 7fb376fff700 1 -- 172.21.15.44:0/1381070803 shutdown_connections 2023-11-16T16:47:03.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.321+0000 7fb376fff700 1 --2- 172.21.15.44:0/1381070803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb370101e10 0x7fb3701086d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.321+0000 7fb376fff700 1 --2- 172.21.15.44:0/1381070803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb370100290 0x7fb3701006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.321+0000 7fb376fff700 1 --2- 172.21.15.44:0/1381070803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb370101480 0x7fb3701018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.321+0000 7fb376fff700 1 -- 172.21.15.44:0/1381070803 >> 172.21.15.44:0/1381070803 conn(0x7fb3700fb870 msgr2=0x7fb3700fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:03.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.321+0000 7fb376fff700 1 -- 172.21.15.44:0/1381070803 shutdown_connections 2023-11-16T16:47:03.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.321+0000 7fb376fff700 1 -- 172.21.15.44:0/1381070803 wait complete. 2023-11-16T16:47:03.325 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.322+0000 7fb376fff700 1 Processor -- start 2023-11-16T16:47:03.325 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.322+0000 7fb376fff700 1 -- start start 2023-11-16T16:47:03.325 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.322+0000 7fb376fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb370100290 0x7fb370196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:03.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.322+0000 7fb376fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb370101480 0x7fb3701965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:03.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.322+0000 7fb376fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb370101e10 0x7fb37019d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:03.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.322+0000 7fb376fff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37019dc10 con 0x7fb370101480 2023-11-16T16:47:03.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.322+0000 7fb376fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37019dd50 con 0x7fb370101e10 2023-11-16T16:47:03.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.322+0000 7fb376fff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37019dee0 con 0x7fb370100290 2023-11-16T16:47:03.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.323+0000 7fb36ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb370101480 0x7fb3701965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:03.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.323+0000 7fb374d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb370100290 0x7fb370196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:03.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.323+0000 7fb36ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb370101480 0x7fb3701965d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44784/0 (socket says 172.21.15.44:44784) 2023-11-16T16:47:03.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.323+0000 7fb36ffff700 1 -- 172.21.15.44:0/1988945384 learned_addr learned my addr 172.21.15.44:0/1988945384 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:03.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.323+0000 7fb37559c700 1 --2- 172.21.15.44:0/1988945384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb370101e10 0x7fb37019d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:03.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.323+0000 7fb36ffff700 1 -- 172.21.15.44:0/1988945384 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb370100290 msgr2=0x7fb370196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:03.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.323+0000 7fb36ffff700 1 --2- 172.21.15.44:0/1988945384 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb370100290 0x7fb370196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.323+0000 7fb36ffff700 1 -- 172.21.15.44:0/1988945384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb370101e10 msgr2=0x7fb37019d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:03.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.323+0000 7fb36ffff700 1 --2- 172.21.15.44:0/1988945384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb370101e10 0x7fb37019d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.323+0000 7fb36ffff700 1 -- 172.21.15.44:0/1988945384 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb368009d00 con 0x7fb370101480 2023-11-16T16:47:03.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.323+0000 7fb374d9b700 1 --2- 172.21.15.44:0/1988945384 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb370100290 0x7fb370196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:03.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.323+0000 7fb37559c700 1 --2- 172.21.15.44:0/1988945384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb370101e10 0x7fb37019d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:03.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.323+0000 7fb36ffff700 1 --2- 172.21.15.44:0/1988945384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb370101480 0x7fb3701965d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00e9a0 tx=0x7fb35c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:03.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.323+0000 7fb36dffb700 1 -- 172.21.15.44:0/1988945384 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c009790 con 0x7fb370101480 2023-11-16T16:47:03.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.324+0000 7fb36dffb700 1 -- 172.21.15.44:0/1988945384 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb35c00cbd0 con 0x7fb370101480 2023-11-16T16:47:03.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.324+0000 7fb376fff700 1 -- 172.21.15.44:0/1988945384 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37010bc70 con 0x7fb370101480 2023-11-16T16:47:03.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.324+0000 7fb36dffb700 1 -- 172.21.15.44:0/1988945384 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c018680 con 0x7fb370101480 2023-11-16T16:47:03.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.324+0000 7fb376fff700 1 -- 172.21.15.44:0/1988945384 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37010bae0 con 0x7fb370101480 2023-11-16T16:47:03.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.325+0000 7fb376fff700 1 -- 172.21.15.44:0/1988945384 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb37019e620 con 0x7fb370101480 2023-11-16T16:47:03.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.327+0000 7fb36dffb700 1 -- 172.21.15.44:0/1988945384 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb35c00cd40 con 0x7fb370101480 2023-11-16T16:47:03.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.328+0000 7fb36dffb700 1 --2- 172.21.15.44:0/1988945384 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb3600a11d0 0x7fb3600a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:03.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.328+0000 7fb36dffb700 1 -- 172.21.15.44:0/1988945384 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb35c014070 con 0x7fb370101480 2023-11-16T16:47:03.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.328+0000 7fb374d9b700 1 --2- 172.21.15.44:0/1988945384 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb3600a11d0 0x7fb3600a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:03.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.329+0000 7fb374d9b700 1 --2- 172.21.15.44:0/1988945384 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb3600a11d0 0x7fb3600a3680 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb3701012e0 tx=0x7fb364009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:03.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.331+0000 7fb36dffb700 1 -- 172.21.15.44:0/1988945384 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb35c0964f0 con 0x7fb370101480 2023-11-16T16:47:03.350 INFO:journalctl@ceph.mds.i.smithi176.stdout:Nov 16 16:47:03 smithi176 systemd[1]: Starting Ceph mds.i for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:47:03.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.592+0000 7fb376fff700 1 -- 172.21.15.44:0/1988945384 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb370107100 con 0x7fb370101480 2023-11-16T16:47:03.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.593+0000 7fb36dffb700 1 -- 172.21.15.44:0/1988945384 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6550 (secure 0 0 0) 0x7fb35c099b10 con 0x7fb370101480 2023-11-16T16:47:03.594 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:03.594 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-11-16T16:47:03.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.595+0000 7fb376fff700 1 -- 172.21.15.44:0/1988945384 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb3600a11d0 msgr2=0x7fb3600a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:03.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.595+0000 7fb376fff700 1 --2- 172.21.15.44:0/1988945384 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb3600a11d0 0x7fb3600a3680 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb3701012e0 tx=0x7fb364009260 comp rx=0 tx=0).stop 2023-11-16T16:47:03.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.595+0000 7fb376fff700 1 -- 172.21.15.44:0/1988945384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb370101480 msgr2=0x7fb3701965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:03.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.595+0000 7fb376fff700 1 --2- 172.21.15.44:0/1988945384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb370101480 0x7fb3701965d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00e9a0 tx=0x7fb35c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.595+0000 7fb376fff700 1 -- 172.21.15.44:0/1988945384 shutdown_connections 2023-11-16T16:47:03.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.595+0000 7fb376fff700 1 --2- 172.21.15.44:0/1988945384 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb3600a11d0 0x7fb3600a3680 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.595+0000 7fb376fff700 1 --2- 172.21.15.44:0/1988945384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb370101e10 0x7fb37019d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.595+0000 7fb376fff700 1 --2- 172.21.15.44:0/1988945384 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb370100290 0x7fb370196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.595+0000 7fb376fff700 1 --2- 172.21.15.44:0/1988945384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb370101480 0x7fb3701965d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.595+0000 7fb376fff700 1 -- 172.21.15.44:0/1988945384 >> 172.21.15.44:0/1988945384 conn(0x7fb3700fb870 msgr2=0x7fb3700fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:03.598 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.595+0000 7fb376fff700 1 -- 172.21.15.44:0/1988945384 shutdown_connections 2023-11-16T16:47:03.598 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:03.595+0000 7fb376fff700 1 -- 172.21.15.44:0/1988945384 wait complete. 2023-11-16T16:47:03.598 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 9 2023-11-16T16:47:03.600 INFO:journalctl@ceph.mds.i.smithi176.stdout:Nov 16 16:47:03 smithi176 podman[157975]: 2023-11-16T16:47:03.600 INFO:journalctl@ceph.mds.i.smithi176.stdout:Nov 16 16:47:03 smithi176 ceph-mds[158015]: -- 172.21.15.176:0/2572153287 <== mon.1 v2:172.21.15.176:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a7d3487520 con 0x55a7d345c800 2023-11-16T16:47:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:03 smithi044 ceph-mon[128750]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:03 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1988945384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:03 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:03 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:03 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:03 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:03 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:03 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:03.894 INFO:journalctl@ceph.mds.i.smithi176.stdout:Nov 16 16:47:03 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-i[157991]: starting mds.i at 2023-11-16T16:47:03.895 INFO:journalctl@ceph.mds.i.smithi176.stdout:Nov 16 16:47:03 smithi176 bash[157975]: 1c4a3f323f7b8188017ee5315f0b631ffeef72a350c191f375e82f19dc4c1c4b 2023-11-16T16:47:03.895 INFO:journalctl@ceph.mds.i.smithi176.stdout:Nov 16 16:47:03 smithi176 systemd[1]: Started Ceph mds.i for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:47:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:03 smithi176 ceph-mon[131630]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:03 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1988945384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:03 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:03 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:03 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:03 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:03 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:03 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:04.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:03 smithi077 ceph-mon[131204]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:04.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:03 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1988945384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:04.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:03 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:04.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:03 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:04.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:03 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:04.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:03 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:04.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:03 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:04.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:03 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:05.095 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:04 smithi044 ceph-mon[128750]: Deploying daemon mds.g on smithi044 2023-11-16T16:47:05.095 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:04 smithi044 ceph-mon[128750]: mds.? [v2:172.21.15.176:6838/2828920421,v1:172.21.15.176:6839/2828920421] up:boot 2023-11-16T16:47:05.096 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:04 smithi044 ceph-mon[128750]: fsmap 9 up:standby 2023-11-16T16:47:05.096 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:04 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T16:47:05.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:04 smithi176 ceph-mon[131630]: Deploying daemon mds.g on smithi044 2023-11-16T16:47:05.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:04 smithi176 ceph-mon[131630]: mds.? [v2:172.21.15.176:6838/2828920421,v1:172.21.15.176:6839/2828920421] up:boot 2023-11-16T16:47:05.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:04 smithi176 ceph-mon[131630]: fsmap 9 up:standby 2023-11-16T16:47:05.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:04 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T16:47:05.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:04 smithi077 ceph-mon[131204]: Deploying daemon mds.g on smithi044 2023-11-16T16:47:05.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:04 smithi077 ceph-mon[131204]: mds.? [v2:172.21.15.176:6838/2828920421,v1:172.21.15.176:6839/2828920421] up:boot 2023-11-16T16:47:05.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:04 smithi077 ceph-mon[131204]: fsmap 9 up:standby 2023-11-16T16:47:05.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:04 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T16:47:05.617 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:05.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.856+0000 7f49b23ff700 1 -- 172.21.15.44:0/1705918851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ac073550 msgr2=0x7f49ac0739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:05.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.856+0000 7f49b23ff700 1 --2- 172.21.15.44:0/1705918851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ac073550 0x7f49ac0739c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009990 tx=0x7f49a0009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:05.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.856+0000 7f49b23ff700 1 -- 172.21.15.44:0/1705918851 shutdown_connections 2023-11-16T16:47:05.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.856+0000 7f49b23ff700 1 --2- 172.21.15.44:0/1705918851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49ac10c860 0x7f49ac10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:05.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.856+0000 7f49b23ff700 1 --2- 172.21.15.44:0/1705918851 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49ac074b30 0x7f49ac072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:05.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.856+0000 7f49b23ff700 1 --2- 172.21.15.44:0/1705918851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ac073550 0x7f49ac0739c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:05.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.856+0000 7f49b23ff700 1 -- 172.21.15.44:0/1705918851 >> 172.21.15.44:0/1705918851 conn(0x7f49ac0fb870 msgr2=0x7f49ac0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:05.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.856+0000 7f49b23ff700 1 -- 172.21.15.44:0/1705918851 shutdown_connections 2023-11-16T16:47:05.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.856+0000 7f49b23ff700 1 -- 172.21.15.44:0/1705918851 wait complete. 2023-11-16T16:47:05.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.857+0000 7f49b23ff700 1 Processor -- start 2023-11-16T16:47:05.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.857+0000 7f49b23ff700 1 -- start start 2023-11-16T16:47:05.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.858+0000 7f49b23ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49ac073550 0x7f49ac19c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:05.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.858+0000 7f49b23ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ac074b30 0x7f49ac19cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:05.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.858+0000 7f49b23ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49ac10c860 0x7f49ac1a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:05.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.858+0000 7f49b23ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac1a4170 con 0x7f49ac074b30 2023-11-16T16:47:05.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.858+0000 7f49b23ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac1a42b0 con 0x7f49ac073550 2023-11-16T16:47:05.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.858+0000 7f49b23ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac1a4440 con 0x7f49ac10c860 2023-11-16T16:47:05.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.858+0000 7f49ab7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ac074b30 0x7f49ac19cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:05.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.858+0000 7f49ab7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ac074b30 0x7f49ac19cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52270/0 (socket says 172.21.15.44:52270) 2023-11-16T16:47:05.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.858+0000 7f49ab7fe700 1 -- 172.21.15.44:0/1009234367 learned_addr learned my addr 172.21.15.44:0/1009234367 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:05.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.858+0000 7f49abfff700 1 --2- 172.21.15.44:0/1009234367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49ac073550 0x7f49ac19c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:05.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.858+0000 7f49b099c700 1 --2- 172.21.15.44:0/1009234367 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49ac10c860 0x7f49ac1a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:05.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.859+0000 7f49ab7fe700 1 -- 172.21.15.44:0/1009234367 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49ac10c860 msgr2=0x7f49ac1a3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:05.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.859+0000 7f49ab7fe700 1 --2- 172.21.15.44:0/1009234367 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49ac10c860 0x7f49ac1a3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:05.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.859+0000 7f49ab7fe700 1 -- 172.21.15.44:0/1009234367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49ac073550 msgr2=0x7f49ac19c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:05.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.859+0000 7f49ab7fe700 1 --2- 172.21.15.44:0/1009234367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49ac073550 0x7f49ac19c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:05.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.859+0000 7f49ab7fe700 1 -- 172.21.15.44:0/1009234367 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a00095f0 con 0x7f49ac074b30 2023-11-16T16:47:05.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.859+0000 7f49ab7fe700 1 --2- 172.21.15.44:0/1009234367 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ac074b30 0x7f49ac19cb30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f49a000b680 tx=0x7f49a0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:05.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.859+0000 7f49a97fa700 1 -- 172.21.15.44:0/1009234367 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a001d070 con 0x7f49ac074b30 2023-11-16T16:47:05.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.859+0000 7f49a97fa700 1 -- 172.21.15.44:0/1009234367 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49a0021410 con 0x7f49ac074b30 2023-11-16T16:47:05.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.859+0000 7f49a97fa700 1 -- 172.21.15.44:0/1009234367 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a0017650 con 0x7f49ac074b30 2023-11-16T16:47:05.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.859+0000 7f49b23ff700 1 -- 172.21.15.44:0/1009234367 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ac112270 con 0x7f49ac074b30 2023-11-16T16:47:05.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.859+0000 7f49b23ff700 1 -- 172.21.15.44:0/1009234367 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ac1120e0 con 0x7f49ac074b30 2023-11-16T16:47:05.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.860+0000 7f49b23ff700 1 -- 172.21.15.44:0/1009234367 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ac1023b0 con 0x7f49ac074b30 2023-11-16T16:47:05.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.862+0000 7f49a97fa700 1 -- 172.21.15.44:0/1009234367 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f49a0021580 con 0x7f49ac074b30 2023-11-16T16:47:05.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.864+0000 7f49a97fa700 1 --2- 172.21.15.44:0/1009234367 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49940a1150 0x7f49940a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:05.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.864+0000 7f49a97fa700 1 -- 172.21.15.44:0/1009234367 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f49a00cd960 con 0x7f49ac074b30 2023-11-16T16:47:05.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.864+0000 7f49abfff700 1 --2- 172.21.15.44:0/1009234367 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49940a1150 0x7f49940a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:05.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.865+0000 7f49abfff700 1 --2- 172.21.15.44:0/1009234367 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49940a1150 0x7f49940a3600 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f49ac074f00 tx=0x7f499c0073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:05.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:05.867+0000 7f49a97fa700 1 -- 172.21.15.44:0/1009234367 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f49a0097540 con 0x7f49ac074b30 2023-11-16T16:47:06.028 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:05 smithi044 ceph-mon[128750]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:06.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:06.134+0000 7f49b23ff700 1 -- 172.21.15.44:0/1009234367 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49ac02cfe0 con 0x7f49ac074b30 2023-11-16T16:47:06.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:06.135+0000 7f49a97fa700 1 -- 172.21.15.44:0/1009234367 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7312 (secure 0 0 0) 0x7f49a009ab60 con 0x7f49ac074b30 2023-11-16T16:47:06.137 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:06.137 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-11-16T16:47:06.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:06.137+0000 7f49b23ff700 1 -- 172.21.15.44:0/1009234367 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49940a1150 msgr2=0x7f49940a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:06.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:06.137+0000 7f49b23ff700 1 --2- 172.21.15.44:0/1009234367 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49940a1150 0x7f49940a3600 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f49ac074f00 tx=0x7f499c0073d0 comp rx=0 tx=0).stop 2023-11-16T16:47:06.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:06.137+0000 7f49b23ff700 1 -- 172.21.15.44:0/1009234367 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ac074b30 msgr2=0x7f49ac19cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:06.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:06.137+0000 7f49b23ff700 1 --2- 172.21.15.44:0/1009234367 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ac074b30 0x7f49ac19cb30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f49a000b680 tx=0x7f49a0005740 comp rx=0 tx=0).stop 2023-11-16T16:47:06.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:06.137+0000 7f49b23ff700 1 -- 172.21.15.44:0/1009234367 shutdown_connections 2023-11-16T16:47:06.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:06.137+0000 7f49b23ff700 1 --2- 172.21.15.44:0/1009234367 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49940a1150 0x7f49940a3600 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:06.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:06.137+0000 7f49b23ff700 1 --2- 172.21.15.44:0/1009234367 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49ac10c860 0x7f49ac1a3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:06.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:06.137+0000 7f49b23ff700 1 --2- 172.21.15.44:0/1009234367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49ac073550 0x7f49ac19c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:06.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:06.137+0000 7f49b23ff700 1 --2- 172.21.15.44:0/1009234367 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ac074b30 0x7f49ac19cb30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:06.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:06.137+0000 7f49b23ff700 1 -- 172.21.15.44:0/1009234367 >> 172.21.15.44:0/1009234367 conn(0x7f49ac0fb870 msgr2=0x7f49ac0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:06.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:06.137+0000 7f49b23ff700 1 -- 172.21.15.44:0/1009234367 shutdown_connections 2023-11-16T16:47:06.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:06.137+0000 7f49b23ff700 1 -- 172.21.15.44:0/1009234367 wait complete. 2023-11-16T16:47:06.141 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 10 2023-11-16T16:47:06.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:05 smithi176 ceph-mon[131630]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:06.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:05 smithi077 ceph-mon[131204]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:06.368 INFO:journalctl@ceph.mds.g.smithi044.stdout:Nov 16 16:47:06 smithi044 systemd[1]: Starting Ceph mds.g for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:47:06.787 INFO:journalctl@ceph.mds.g.smithi044.stdout:Nov 16 16:47:06 smithi044 podman[162269]: 2023-11-16T16:47:06.788 INFO:journalctl@ceph.mds.g.smithi044.stdout:Nov 16 16:47:06 smithi044 bash[162269]: ecc4738c1543b0a7ce2c65480c56592bb4bdf6f5197e94c47ec71c8f3fc63eba 2023-11-16T16:47:07.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:06 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1009234367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:07.119 INFO:journalctl@ceph.mds.g.smithi044.stdout:Nov 16 16:47:06 smithi044 ceph-mds[162309]: -- 172.21.15.44:0/1942769785 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f1f8307520 con 0x55f1f82dec00 2023-11-16T16:47:07.119 INFO:journalctl@ceph.mds.g.smithi044.stdout:Nov 16 16:47:06 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-g[162285]: starting mds.g at 2023-11-16T16:47:07.119 INFO:journalctl@ceph.mds.g.smithi044.stdout:Nov 16 16:47:06 smithi044 systemd[1]: Started Ceph mds.g for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:47:07.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:06 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1009234367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:07.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:06 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1009234367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:08.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:07 smithi044 ceph-mon[128750]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:08.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:07 smithi044 ceph-mon[128750]: mds.? [v2:172.21.15.44:6838/1792053180,v1:172.21.15.44:6839/1792053180] up:boot 2023-11-16T16:47:08.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:07 smithi044 ceph-mon[128750]: fsmap 10 up:standby 2023-11-16T16:47:08.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T16:47:08.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:08.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:08.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:08.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:08.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:08.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:07 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:08.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:07 smithi176 ceph-mon[131630]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:08.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:07 smithi176 ceph-mon[131630]: mds.? [v2:172.21.15.44:6838/1792053180,v1:172.21.15.44:6839/1792053180] up:boot 2023-11-16T16:47:08.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:07 smithi176 ceph-mon[131630]: fsmap 10 up:standby 2023-11-16T16:47:08.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T16:47:08.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:08.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:08.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:08.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:08.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:08.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:07 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:08.162 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:08.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:07 smithi077 ceph-mon[131204]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:08.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:07 smithi077 ceph-mon[131204]: mds.? [v2:172.21.15.44:6838/1792053180,v1:172.21.15.44:6839/1792053180] up:boot 2023-11-16T16:47:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:07 smithi077 ceph-mon[131204]: fsmap 10 up:standby 2023-11-16T16:47:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T16:47:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:07 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:08.382 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.380+0000 7fcb645ff700 1 -- 172.21.15.44:0/4105718393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb5c101e10 msgr2=0x7fcb5c1086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:08.383 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.380+0000 7fcb645ff700 1 --2- 172.21.15.44:0/4105718393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb5c101e10 0x7fcb5c1086d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400b0a0 tx=0x7fcb5400b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.383 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.380+0000 7fcb645ff700 1 -- 172.21.15.44:0/4105718393 shutdown_connections 2023-11-16T16:47:08.383 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.380+0000 7fcb645ff700 1 --2- 172.21.15.44:0/4105718393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb5c101e10 0x7fcb5c1086d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.383 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.380+0000 7fcb645ff700 1 --2- 172.21.15.44:0/4105718393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb5c100290 0x7fcb5c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.383 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.380+0000 7fcb645ff700 1 --2- 172.21.15.44:0/4105718393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb5c101480 0x7fcb5c1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.383 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.380+0000 7fcb645ff700 1 -- 172.21.15.44:0/4105718393 >> 172.21.15.44:0/4105718393 conn(0x7fcb5c0fb870 msgr2=0x7fcb5c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:08.383 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.381+0000 7fcb645ff700 1 -- 172.21.15.44:0/4105718393 shutdown_connections 2023-11-16T16:47:08.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.381+0000 7fcb645ff700 1 -- 172.21.15.44:0/4105718393 wait complete. 2023-11-16T16:47:08.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.382+0000 7fcb645ff700 1 Processor -- start 2023-11-16T16:47:08.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.382+0000 7fcb645ff700 1 -- start start 2023-11-16T16:47:08.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.382+0000 7fcb645ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb5c100290 0x7fcb5c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:08.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.382+0000 7fcb645ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb5c101480 0x7fcb5c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:08.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.382+0000 7fcb645ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb5c101e10 0x7fcb5c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:08.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.382+0000 7fcb645ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c073bc0 con 0x7fcb5c101480 2023-11-16T16:47:08.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.382+0000 7fcb645ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c073d30 con 0x7fcb5c101e10 2023-11-16T16:47:08.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.382+0000 7fcb645ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c1a00f0 con 0x7fcb5c100290 2023-11-16T16:47:08.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.382+0000 7fcb61b9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb5c101480 0x7fcb5c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:08.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.382+0000 7fcb6239b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb5c100290 0x7fcb5c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:08.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.382+0000 7fcb6239b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb5c100290 0x7fcb5c0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:41742/0 (socket says 172.21.15.44:41742) 2023-11-16T16:47:08.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.382+0000 7fcb61b9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb5c101480 0x7fcb5c072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52298/0 (socket says 172.21.15.44:52298) 2023-11-16T16:47:08.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.382+0000 7fcb6239b700 1 -- 172.21.15.44:0/2351825103 learned_addr learned my addr 172.21.15.44:0/2351825103 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:08.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.383+0000 7fcb62b9c700 1 --2- 172.21.15.44:0/2351825103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb5c101e10 0x7fcb5c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:08.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.383+0000 7fcb61b9a700 1 -- 172.21.15.44:0/2351825103 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb5c100290 msgr2=0x7fcb5c0748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:08.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.383+0000 7fcb61b9a700 1 --2- 172.21.15.44:0/2351825103 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb5c100290 0x7fcb5c0748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.383+0000 7fcb61b9a700 1 -- 172.21.15.44:0/2351825103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb5c101e10 msgr2=0x7fcb5c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:08.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.383+0000 7fcb61b9a700 1 --2- 172.21.15.44:0/2351825103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb5c101e10 0x7fcb5c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.383+0000 7fcb61b9a700 1 -- 172.21.15.44:0/2351825103 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb54009d00 con 0x7fcb5c101480 2023-11-16T16:47:08.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.383+0000 7fcb62b9c700 1 --2- 172.21.15.44:0/2351825103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb5c101e10 0x7fcb5c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:08.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.383+0000 7fcb61b9a700 1 --2- 172.21.15.44:0/2351825103 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb5c101480 0x7fcb5c072ee0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c00e9f0 tx=0x7fcb4c00edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:08.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.384+0000 7fcb537fe700 1 -- 172.21.15.44:0/2351825103 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c00cbd0 con 0x7fcb5c101480 2023-11-16T16:47:08.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.384+0000 7fcb645ff700 1 -- 172.21.15.44:0/2351825103 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5c10bc70 con 0x7fcb5c101480 2023-11-16T16:47:08.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.384+0000 7fcb537fe700 1 -- 172.21.15.44:0/2351825103 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb4c00cd30 con 0x7fcb5c101480 2023-11-16T16:47:08.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.384+0000 7fcb537fe700 1 -- 172.21.15.44:0/2351825103 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c018680 con 0x7fcb5c101480 2023-11-16T16:47:08.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.384+0000 7fcb645ff700 1 -- 172.21.15.44:0/2351825103 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb5c10bae0 con 0x7fcb5c101480 2023-11-16T16:47:08.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.385+0000 7fcb645ff700 1 -- 172.21.15.44:0/2351825103 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb5c107770 con 0x7fcb5c101480 2023-11-16T16:47:08.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.387+0000 7fcb537fe700 1 -- 172.21.15.44:0/2351825103 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcb4c010430 con 0x7fcb5c101480 2023-11-16T16:47:08.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.388+0000 7fcb537fe700 1 --2- 172.21.15.44:0/2351825103 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fcb480a1150 0x7fcb480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:08.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.388+0000 7fcb537fe700 1 -- 172.21.15.44:0/2351825103 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcb4c014070 con 0x7fcb5c101480 2023-11-16T16:47:08.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.388+0000 7fcb6239b700 1 --2- 172.21.15.44:0/2351825103 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fcb480a1150 0x7fcb480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:08.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.389+0000 7fcb6239b700 1 --2- 172.21.15.44:0/2351825103 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fcb480a1150 0x7fcb480a3600 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcb58005950 tx=0x7fcb58009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:08.394 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.392+0000 7fcb537fe700 1 -- 172.21.15.44:0/2351825103 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcb4c051730 con 0x7fcb5c101480 2023-11-16T16:47:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.705+0000 7fcb645ff700 1 -- 172.21.15.44:0/2351825103 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb5c061930 con 0x7fcb5c101480 2023-11-16T16:47:08.707 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.705+0000 7fcb537fe700 1 -- 172.21.15.44:0/2351825103 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8070 (secure 0 0 0) 0x7fcb4c096340 con 0x7fcb5c101480 2023-11-16T16:47:08.707 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:08.709 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-11-16T16:47:08.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.708+0000 7fcb645ff700 1 -- 172.21.15.44:0/2351825103 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fcb480a1150 msgr2=0x7fcb480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:08.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.708+0000 7fcb645ff700 1 --2- 172.21.15.44:0/2351825103 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fcb480a1150 0x7fcb480a3600 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcb58005950 tx=0x7fcb58009260 comp rx=0 tx=0).stop 2023-11-16T16:47:08.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.708+0000 7fcb645ff700 1 -- 172.21.15.44:0/2351825103 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb5c101480 msgr2=0x7fcb5c072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:08.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.708+0000 7fcb645ff700 1 --2- 172.21.15.44:0/2351825103 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb5c101480 0x7fcb5c072ee0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c00e9f0 tx=0x7fcb4c00edb0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.708+0000 7fcb645ff700 1 -- 172.21.15.44:0/2351825103 shutdown_connections 2023-11-16T16:47:08.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.708+0000 7fcb645ff700 1 --2- 172.21.15.44:0/2351825103 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fcb480a1150 0x7fcb480a3600 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.708+0000 7fcb645ff700 1 --2- 172.21.15.44:0/2351825103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb5c101e10 0x7fcb5c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.708+0000 7fcb645ff700 1 --2- 172.21.15.44:0/2351825103 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb5c100290 0x7fcb5c0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.708+0000 7fcb645ff700 1 --2- 172.21.15.44:0/2351825103 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb5c101480 0x7fcb5c072ee0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.708+0000 7fcb645ff700 1 -- 172.21.15.44:0/2351825103 >> 172.21.15.44:0/2351825103 conn(0x7fcb5c0fb870 msgr2=0x7fcb5c1048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:08.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.708+0000 7fcb645ff700 1 -- 172.21.15.44:0/2351825103 shutdown_connections 2023-11-16T16:47:08.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:08.708+0000 7fcb645ff700 1 -- 172.21.15.44:0/2351825103 wait complete. 2023-11-16T16:47:08.712 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 11 2023-11-16T16:47:09.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:08 smithi044 ceph-mon[128750]: Deploying daemon mds.a on smithi044 2023-11-16T16:47:09.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:08 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2351825103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:09.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:08 smithi176 ceph-mon[131630]: Deploying daemon mds.a on smithi044 2023-11-16T16:47:09.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:08 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2351825103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:09.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:08 smithi077 ceph-mon[131204]: Deploying daemon mds.a on smithi044 2023-11-16T16:47:09.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:08 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2351825103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:09.619 INFO:journalctl@ceph.mds.a.smithi044.stdout:Nov 16 16:47:09 smithi044 systemd[1]: Starting Ceph mds.a for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:47:09.903 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:09 smithi044 ceph-mon[128750]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:09.903 INFO:journalctl@ceph.mds.a.smithi044.stdout:Nov 16 16:47:09 smithi044 podman[162657]: 2023-11-16T16:47:09.903 INFO:journalctl@ceph.mds.a.smithi044.stdout:Nov 16 16:47:09 smithi044 bash[162657]: 33941a07c921443af0915f74e6e110a4bf7782580a94ffdb3c749b60286d8c88 2023-11-16T16:47:10.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:09 smithi176 ceph-mon[131630]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:10.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:09 smithi077 ceph-mon[131204]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:10.368 INFO:journalctl@ceph.mds.a.smithi044.stdout:Nov 16 16:47:09 smithi044 ceph-mds[162697]: -- 172.21.15.44:0/2682101758 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560007485520 con 0x56000745d000 2023-11-16T16:47:10.368 INFO:journalctl@ceph.mds.a.smithi044.stdout:Nov 16 16:47:09 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-a[162673]: starting mds.a at 2023-11-16T16:47:10.368 INFO:journalctl@ceph.mds.a.smithi044.stdout:Nov 16 16:47:09 smithi044 systemd[1]: Started Ceph mds.a for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:47:10.731 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:10.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.949+0000 7f2b4a99d700 1 -- 172.21.15.44:0/1828739043 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b44068b10 msgr2=0x7f2b44102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:10.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.949+0000 7f2b4a99d700 1 --2- 172.21.15.44:0/1828739043 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b44068b10 0x7f2b44102c30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2b40009990 tx=0x7f2b40009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:10.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.950+0000 7f2b4a99d700 1 -- 172.21.15.44:0/1828739043 shutdown_connections 2023-11-16T16:47:10.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.950+0000 7f2b4a99d700 1 --2- 172.21.15.44:0/1828739043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b44105c80 0x7f2b44108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:10.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.950+0000 7f2b4a99d700 1 --2- 172.21.15.44:0/1828739043 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b44068b10 0x7f2b44102c30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:10.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.950+0000 7f2b4a99d700 1 --2- 172.21.15.44:0/1828739043 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b44103170 0x7f2b441055f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:10.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.950+0000 7f2b4a99d700 1 -- 172.21.15.44:0/1828739043 >> 172.21.15.44:0/1828739043 conn(0x7f2b440fa620 msgr2=0x7f2b440fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:10.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.950+0000 7f2b4a99d700 1 -- 172.21.15.44:0/1828739043 shutdown_connections 2023-11-16T16:47:10.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.950+0000 7f2b4a99d700 1 -- 172.21.15.44:0/1828739043 wait complete. 2023-11-16T16:47:10.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.951+0000 7f2b4a99d700 1 Processor -- start 2023-11-16T16:47:10.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.951+0000 7f2b4a99d700 1 -- start start 2023-11-16T16:47:10.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.951+0000 7f2b4a99d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b44068b10 0x7f2b44100590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:10.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.951+0000 7f2b4a99d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b44103170 0x7f2b440febd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:10.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.951+0000 7f2b4a99d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b44105c80 0x7f2b440ff110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:10.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.951+0000 7f2b4a99d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b440ff8b0 con 0x7f2b44068b10 2023-11-16T16:47:10.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.951+0000 7f2b4a99d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b440ffa20 con 0x7f2b44103170 2023-11-16T16:47:10.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.951+0000 7f2b4a99d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4419fe30 con 0x7f2b44105c80 2023-11-16T16:47:10.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.951+0000 7f2b4999b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b44068b10 0x7f2b44100590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:10.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.951+0000 7f2b4999b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b44068b10 0x7f2b44100590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52332/0 (socket says 172.21.15.44:52332) 2023-11-16T16:47:10.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.951+0000 7f2b4999b700 1 -- 172.21.15.44:0/2091182076 learned_addr learned my addr 172.21.15.44:0/2091182076 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:10.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.952+0000 7f2b4a19c700 1 --2- 172.21.15.44:0/2091182076 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b44105c80 0x7f2b440ff110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:10.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.952+0000 7f2b4919a700 1 --2- 172.21.15.44:0/2091182076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b44103170 0x7f2b440febd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:10.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.952+0000 7f2b4999b700 1 -- 172.21.15.44:0/2091182076 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b44105c80 msgr2=0x7f2b440ff110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:10.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.952+0000 7f2b4999b700 1 --2- 172.21.15.44:0/2091182076 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b44105c80 0x7f2b440ff110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:10.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.952+0000 7f2b4999b700 1 -- 172.21.15.44:0/2091182076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b44103170 msgr2=0x7f2b440febd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:10.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.952+0000 7f2b4999b700 1 --2- 172.21.15.44:0/2091182076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b44103170 0x7f2b440febd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:10.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.952+0000 7f2b4999b700 1 -- 172.21.15.44:0/2091182076 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b400095f0 con 0x7f2b44068b10 2023-11-16T16:47:10.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.952+0000 7f2b4999b700 1 --2- 172.21.15.44:0/2091182076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b44068b10 0x7f2b44100590 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2b40005720 tx=0x7f2b40009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:10.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.952+0000 7f2b3affd700 1 -- 172.21.15.44:0/2091182076 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4001d070 con 0x7f2b44068b10 2023-11-16T16:47:10.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.952+0000 7f2b3affd700 1 -- 172.21.15.44:0/2091182076 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b40021410 con 0x7f2b44068b10 2023-11-16T16:47:10.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.952+0000 7f2b4a99d700 1 -- 172.21.15.44:0/2091182076 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b4410ba50 con 0x7f2b44068b10 2023-11-16T16:47:10.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.952+0000 7f2b3affd700 1 -- 172.21.15.44:0/2091182076 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b40017650 con 0x7f2b44068b10 2023-11-16T16:47:10.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.953+0000 7f2b4a99d700 1 -- 172.21.15.44:0/2091182076 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b4410b8c0 con 0x7f2b44068b10 2023-11-16T16:47:10.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.954+0000 7f2b4a99d700 1 -- 172.21.15.44:0/2091182076 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b440fc190 con 0x7f2b44068b10 2023-11-16T16:47:10.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.956+0000 7f2b3affd700 1 -- 172.21.15.44:0/2091182076 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2b40021580 con 0x7f2b44068b10 2023-11-16T16:47:10.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.957+0000 7f2b3affd700 1 --2- 172.21.15.44:0/2091182076 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2b300a1150 0x7f2b300a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:10.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.957+0000 7f2b3affd700 1 -- 172.21.15.44:0/2091182076 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2b400cdb50 con 0x7f2b44068b10 2023-11-16T16:47:10.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.958+0000 7f2b4919a700 1 --2- 172.21.15.44:0/2091182076 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2b300a1150 0x7f2b300a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:10.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.958+0000 7f2b4919a700 1 --2- 172.21.15.44:0/2091182076 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2b300a1150 0x7f2b300a3600 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009c60 tx=0x7f2b34009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:10.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:10.959+0000 7f2b3affd700 1 -- 172.21.15.44:0/2091182076 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2b40097730 con 0x7f2b44068b10 2023-11-16T16:47:11.097 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:11.097 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:11.097 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:11.098 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:11.098 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:11.098 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:11.098 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:10 smithi077 ceph-mon[131204]: Deploying daemon mds.b on smithi077 2023-11-16T16:47:11.098 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:10 smithi077 ceph-mon[131204]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:11.098 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:10 smithi077 ceph-mon[131204]: mds.? [v2:172.21.15.44:6840/2080079420,v1:172.21.15.44:6841/2080079420] up:boot 2023-11-16T16:47:11.098 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:10 smithi077 ceph-mon[131204]: fsmap 11 up:standby 2023-11-16T16:47:11.098 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:10 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T16:47:11.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:11.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:11.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:11.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:11.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:11.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:11.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:10 smithi044 ceph-mon[128750]: Deploying daemon mds.b on smithi077 2023-11-16T16:47:11.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:10 smithi044 ceph-mon[128750]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:11.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:10 smithi044 ceph-mon[128750]: mds.? [v2:172.21.15.44:6840/2080079420,v1:172.21.15.44:6841/2080079420] up:boot 2023-11-16T16:47:11.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:10 smithi044 ceph-mon[128750]: fsmap 11 up:standby 2023-11-16T16:47:11.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:10 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T16:47:11.221 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:11.220+0000 7f2b4a99d700 1 -- 172.21.15.44:0/2091182076 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b4402cfe0 con 0x7f2b44068b10 2023-11-16T16:47:11.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:11.221+0000 7f2b3affd700 1 -- 172.21.15.44:0/2091182076 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8828 (secure 0 0 0) 0x7f2b4009ad50 con 0x7f2b44068b10 2023-11-16T16:47:11.222 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:11.223 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-11-16T16:47:11.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:11.223+0000 7f2b4a99d700 1 -- 172.21.15.44:0/2091182076 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2b300a1150 msgr2=0x7f2b300a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:11.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:11.223+0000 7f2b4a99d700 1 --2- 172.21.15.44:0/2091182076 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2b300a1150 0x7f2b300a3600 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009c60 tx=0x7f2b34009260 comp rx=0 tx=0).stop 2023-11-16T16:47:11.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:11.223+0000 7f2b4a99d700 1 -- 172.21.15.44:0/2091182076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b44068b10 msgr2=0x7f2b44100590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:11.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:11.223+0000 7f2b4a99d700 1 --2- 172.21.15.44:0/2091182076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b44068b10 0x7f2b44100590 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2b40005720 tx=0x7f2b40009580 comp rx=0 tx=0).stop 2023-11-16T16:47:11.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:11.223+0000 7f2b4a99d700 1 -- 172.21.15.44:0/2091182076 shutdown_connections 2023-11-16T16:47:11.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:11.223+0000 7f2b4a99d700 1 --2- 172.21.15.44:0/2091182076 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2b300a1150 0x7f2b300a3600 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:11.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:11.223+0000 7f2b4a99d700 1 --2- 172.21.15.44:0/2091182076 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b44105c80 0x7f2b440ff110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:11.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:11.223+0000 7f2b4a99d700 1 --2- 172.21.15.44:0/2091182076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b44068b10 0x7f2b44100590 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:11.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:11.223+0000 7f2b4a99d700 1 --2- 172.21.15.44:0/2091182076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b44103170 0x7f2b440febd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:11.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:11.223+0000 7f2b4a99d700 1 -- 172.21.15.44:0/2091182076 >> 172.21.15.44:0/2091182076 conn(0x7f2b440fa620 msgr2=0x7f2b441014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:11.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:11.223+0000 7f2b4a99d700 1 -- 172.21.15.44:0/2091182076 shutdown_connections 2023-11-16T16:47:11.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:11.223+0000 7f2b4a99d700 1 -- 172.21.15.44:0/2091182076 wait complete. 2023-11-16T16:47:11.226 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 12 2023-11-16T16:47:11.394 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:10 smithi176 ceph-mon[131630]: Deploying daemon mds.b on smithi077 2023-11-16T16:47:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:10 smithi176 ceph-mon[131630]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:10 smithi176 ceph-mon[131630]: mds.? [v2:172.21.15.44:6840/2080079420,v1:172.21.15.44:6841/2080079420] up:boot 2023-11-16T16:47:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:10 smithi176 ceph-mon[131630]: fsmap 11 up:standby 2023-11-16T16:47:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:10 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T16:47:11.726 INFO:journalctl@ceph.mds.b.smithi077.stdout:Nov 16 16:47:11 smithi077 systemd[1]: Starting Ceph mds.b for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:47:12.121 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:12 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2091182076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:12.121 INFO:journalctl@ceph.mds.b.smithi077.stdout:Nov 16 16:47:11 smithi077 podman[156507]: 2023-11-16T16:47:12.121 INFO:journalctl@ceph.mds.b.smithi077.stdout:Nov 16 16:47:12 smithi077 ceph-mds[156547]: -- 172.21.15.77:0/2334412013 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555d2c319520 con 0x555d2c2eec00 2023-11-16T16:47:12.121 INFO:journalctl@ceph.mds.b.smithi077.stdout:Nov 16 16:47:12 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-b[156523]: starting mds.b at 2023-11-16T16:47:12.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:12 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2091182076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:12.394 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:12 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2091182076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:12.476 INFO:journalctl@ceph.mds.b.smithi077.stdout:Nov 16 16:47:12 smithi077 bash[156507]: a51f508774480c7479e21b5fe3daaee1703ff6ffd56c51dfcc3ca910fe4465c6 2023-11-16T16:47:12.476 INFO:journalctl@ceph.mds.b.smithi077.stdout:Nov 16 16:47:12 smithi077 systemd[1]: Started Ceph mds.b for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:47:13.244 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:13.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.465+0000 7f9f959ff700 1 -- 172.21.15.44:0/2289432161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f901036b0 msgr2=0x7f9f90105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:13.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.465+0000 7f9f959ff700 1 --2- 172.21.15.44:0/2289432161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f901036b0 0x7f9f90105a90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9f78009990 tx=0x7f9f78009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.466+0000 7f9f959ff700 1 -- 172.21.15.44:0/2289432161 shutdown_connections 2023-11-16T16:47:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.466+0000 7f9f959ff700 1 --2- 172.21.15.44:0/2289432161 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f90105fd0 0x7f9f901083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.466+0000 7f9f959ff700 1 --2- 172.21.15.44:0/2289432161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f90100d90 0x7f9f90103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.466+0000 7f9f959ff700 1 --2- 172.21.15.44:0/2289432161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f901036b0 0x7f9f90105a90 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.466+0000 7f9f959ff700 1 -- 172.21.15.44:0/2289432161 >> 172.21.15.44:0/2289432161 conn(0x7f9f900fa750 msgr2=0x7f9f900fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.466+0000 7f9f959ff700 1 -- 172.21.15.44:0/2289432161 shutdown_connections 2023-11-16T16:47:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.466+0000 7f9f959ff700 1 -- 172.21.15.44:0/2289432161 wait complete. 2023-11-16T16:47:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.467+0000 7f9f959ff700 1 Processor -- start 2023-11-16T16:47:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.467+0000 7f9f959ff700 1 -- start start 2023-11-16T16:47:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.467+0000 7f9f959ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f90100d90 0x7f9f901981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.467+0000 7f9f959ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f901036b0 0x7f9f90198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.467+0000 7f9f959ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f90105fd0 0x7f9f9019f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.467+0000 7f9f959ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9019fd70 con 0x7f9f90105fd0 2023-11-16T16:47:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.467+0000 7f9f959ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9019feb0 con 0x7f9f90100d90 2023-11-16T16:47:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.467+0000 7f9f959ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f901a0040 con 0x7f9f901036b0 2023-11-16T16:47:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.468+0000 7f9f87fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f901036b0 0x7f9f90198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.468+0000 7f9f8f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f90105fd0 0x7f9f9019f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:13.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.468+0000 7f9f8effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f90100d90 0x7f9f901981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:13.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.468+0000 7f9f87fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f901036b0 0x7f9f90198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:41802/0 (socket says 172.21.15.44:41802) 2023-11-16T16:47:13.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.468+0000 7f9f87fff700 1 -- 172.21.15.44:0/2838965558 learned_addr learned my addr 172.21.15.44:0/2838965558 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:13.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.468+0000 7f9f8effd700 1 -- 172.21.15.44:0/2838965558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f901036b0 msgr2=0x7f9f90198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:13.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.468+0000 7f9f8effd700 1 --2- 172.21.15.44:0/2838965558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f901036b0 0x7f9f90198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:13.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.468+0000 7f9f8effd700 1 -- 172.21.15.44:0/2838965558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f90105fd0 msgr2=0x7f9f9019f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:13.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.468+0000 7f9f8effd700 1 --2- 172.21.15.44:0/2838965558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f90105fd0 0x7f9f9019f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:13.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.468+0000 7f9f8effd700 1 -- 172.21.15.44:0/2838965558 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f80009520 con 0x7f9f90100d90 2023-11-16T16:47:13.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.468+0000 7f9f87fff700 1 --2- 172.21.15.44:0/2838965558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f901036b0 0x7f9f90198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:13.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.468+0000 7f9f8effd700 1 --2- 172.21.15.44:0/2838965558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f90100d90 0x7f9f901981f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f8000e9c0 tx=0x7f9f8000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:13.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.468+0000 7f9f949fd700 1 -- 172.21.15.44:0/2838965558 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8000cb70 con 0x7f9f90100d90 2023-11-16T16:47:13.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.469+0000 7f9f949fd700 1 -- 172.21.15.44:0/2838965558 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f8000ee60 con 0x7f9f90100d90 2023-11-16T16:47:13.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.469+0000 7f9f949fd700 1 -- 172.21.15.44:0/2838965558 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f80003c60 con 0x7f9f90100d90 2023-11-16T16:47:13.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.469+0000 7f9f959ff700 1 -- 172.21.15.44:0/2838965558 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f88009c40 con 0x7f9f90100d90 2023-11-16T16:47:13.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.469+0000 7f9f959ff700 1 -- 172.21.15.44:0/2838965558 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f780095f0 con 0x7f9f90100d90 2023-11-16T16:47:13.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.470+0000 7f9f959ff700 1 -- 172.21.15.44:0/2838965558 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f90107b40 con 0x7f9f90100d90 2023-11-16T16:47:13.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.472+0000 7f9f949fd700 1 -- 172.21.15.44:0/2838965558 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9f80003e00 con 0x7f9f90100d90 2023-11-16T16:47:13.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.474+0000 7f9f949fd700 1 --2- 172.21.15.44:0/2838965558 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9f7c0a1150 0x7f9f7c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:13.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.474+0000 7f9f949fd700 1 -- 172.21.15.44:0/2838965558 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9f80014070 con 0x7f9f90100d90 2023-11-16T16:47:13.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.474+0000 7f9f87fff700 1 --2- 172.21.15.44:0/2838965558 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9f7c0a1150 0x7f9f7c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:13.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.475+0000 7f9f87fff700 1 --2- 172.21.15.44:0/2838965558 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9f7c0a1150 0x7f9f7c0a3600 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9f78006010 tx=0x7f9f7800b260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:13.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:13 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:13 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:13 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:13 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:13 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:13 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:47:13.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:13 smithi077 ceph-mon[131204]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:13.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:13 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:13 smithi077 ceph-mon[131204]: mds.? [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] up:boot 2023-11-16T16:47:13.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:13 smithi077 ceph-mon[131204]: fsmap 12 up:standby 2023-11-16T16:47:13.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:13 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T16:47:13.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.477+0000 7f9f949fd700 1 -- 172.21.15.44:0/2838965558 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9f80096330 con 0x7f9f90100d90 2023-11-16T16:47:13.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:13 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:13 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:13 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:13 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:13 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:13 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:47:13.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:13 smithi044 ceph-mon[128750]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:13.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:13 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:13 smithi044 ceph-mon[128750]: mds.? [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] up:boot 2023-11-16T16:47:13.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:13 smithi044 ceph-mon[128750]: fsmap 12 up:standby 2023-11-16T16:47:13.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:13 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T16:47:13.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:13 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:13 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:13 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:13 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:13 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:13 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:47:13.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:13 smithi176 ceph-mon[131630]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:13.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:13 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:13.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:13 smithi176 ceph-mon[131630]: mds.? [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] up:boot 2023-11-16T16:47:13.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:13 smithi176 ceph-mon[131630]: fsmap 12 up:standby 2023-11-16T16:47:13.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:13 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T16:47:13.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.780+0000 7f9f959ff700 1 -- 172.21.15.44:0/2838965558 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f9002cbc0 con 0x7f9f90100d90 2023-11-16T16:47:13.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.781+0000 7f9f949fd700 1 -- 172.21.15.44:0/2838965558 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9586 (secure 0 0 0) 0x7f9f80099950 con 0x7f9f90100d90 2023-11-16T16:47:13.783 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:13.783 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24551,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-11-16T16:47:13.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.784+0000 7f9f959ff700 1 -- 172.21.15.44:0/2838965558 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9f7c0a1150 msgr2=0x7f9f7c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:13.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.784+0000 7f9f959ff700 1 --2- 172.21.15.44:0/2838965558 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9f7c0a1150 0x7f9f7c0a3600 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9f78006010 tx=0x7f9f7800b260 comp rx=0 tx=0).stop 2023-11-16T16:47:13.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.784+0000 7f9f959ff700 1 -- 172.21.15.44:0/2838965558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f90100d90 msgr2=0x7f9f901981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:13.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.784+0000 7f9f959ff700 1 --2- 172.21.15.44:0/2838965558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f90100d90 0x7f9f901981f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f8000e9c0 tx=0x7f9f8000ecd0 comp rx=0 tx=0).stop 2023-11-16T16:47:13.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.784+0000 7f9f959ff700 1 -- 172.21.15.44:0/2838965558 shutdown_connections 2023-11-16T16:47:13.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.784+0000 7f9f959ff700 1 --2- 172.21.15.44:0/2838965558 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9f7c0a1150 0x7f9f7c0a3600 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:13.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.784+0000 7f9f959ff700 1 --2- 172.21.15.44:0/2838965558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f90105fd0 0x7f9f9019f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:13.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.784+0000 7f9f959ff700 1 --2- 172.21.15.44:0/2838965558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f90100d90 0x7f9f901981f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:13.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.784+0000 7f9f959ff700 1 --2- 172.21.15.44:0/2838965558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f901036b0 0x7f9f90198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:13.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.784+0000 7f9f959ff700 1 -- 172.21.15.44:0/2838965558 >> 172.21.15.44:0/2838965558 conn(0x7f9f900fa750 msgr2=0x7f9f901019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:13.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.784+0000 7f9f959ff700 1 -- 172.21.15.44:0/2838965558 shutdown_connections 2023-11-16T16:47:13.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:13.784+0000 7f9f959ff700 1 -- 172.21.15.44:0/2838965558 wait complete. 2023-11-16T16:47:13.787 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2023-11-16T16:47:13.805 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:14.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.021+0000 7f43ddbff700 1 -- 172.21.15.44:0/3506579171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43d81036b0 msgr2=0x7f43d8105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.021+0000 7f43ddbff700 1 --2- 172.21.15.44:0/3506579171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43d81036b0 0x7f43d8105a90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f43cc0098f0 tx=0x7f43cc009c00 comp rx=0 tx=0).stop 2023-11-16T16:47:14.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.021+0000 7f43ddbff700 1 -- 172.21.15.44:0/3506579171 shutdown_connections 2023-11-16T16:47:14.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.021+0000 7f43ddbff700 1 --2- 172.21.15.44:0/3506579171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43d8105fd0 0x7f43d81083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.021+0000 7f43ddbff700 1 --2- 172.21.15.44:0/3506579171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43d8100d90 0x7f43d8103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.021+0000 7f43ddbff700 1 --2- 172.21.15.44:0/3506579171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43d81036b0 0x7f43d8105a90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.021+0000 7f43ddbff700 1 -- 172.21.15.44:0/3506579171 >> 172.21.15.44:0/3506579171 conn(0x7f43d80fa750 msgr2=0x7f43d80fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:14.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.021+0000 7f43ddbff700 1 -- 172.21.15.44:0/3506579171 shutdown_connections 2023-11-16T16:47:14.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.021+0000 7f43ddbff700 1 -- 172.21.15.44:0/3506579171 wait complete. 2023-11-16T16:47:14.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.023+0000 7f43ddbff700 1 Processor -- start 2023-11-16T16:47:14.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.023+0000 7f43ddbff700 1 -- start start 2023-11-16T16:47:14.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.023+0000 7f43ddbff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43d8100d90 0x7f43d8196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:14.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.023+0000 7f43ddbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43d81036b0 0x7f43d8196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:14.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.024+0000 7f43ddbff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43d8105fd0 0x7f43d819d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:14.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.024+0000 7f43ddbff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d819dbd0 con 0x7f43d8105fd0 2023-11-16T16:47:14.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.024+0000 7f43ddbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d819dd10 con 0x7f43d81036b0 2023-11-16T16:47:14.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.024+0000 7f43ddbff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d819dea0 con 0x7f43d8100d90 2023-11-16T16:47:14.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.024+0000 7f43d6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43d81036b0 0x7f43d8196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:14.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.024+0000 7f43d77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43d8100d90 0x7f43d8196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:14.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.024+0000 7f43d7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43d8105fd0 0x7f43d819d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:14.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.024+0000 7f43d6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43d81036b0 0x7f43d8196590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:34706/0 (socket says 172.21.15.44:34706) 2023-11-16T16:47:14.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.024+0000 7f43d6ffd700 1 -- 172.21.15.44:0/1562021637 learned_addr learned my addr 172.21.15.44:0/1562021637 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:14.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.024+0000 7f43d77fe700 1 -- 172.21.15.44:0/1562021637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43d81036b0 msgr2=0x7f43d8196590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.024+0000 7f43d77fe700 1 --2- 172.21.15.44:0/1562021637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43d81036b0 0x7f43d8196590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.024+0000 7f43d77fe700 1 -- 172.21.15.44:0/1562021637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43d8105fd0 msgr2=0x7f43d819d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.024+0000 7f43d77fe700 1 --2- 172.21.15.44:0/1562021637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43d8105fd0 0x7f43d819d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.024+0000 7f43d77fe700 1 -- 172.21.15.44:0/1562021637 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43cc009520 con 0x7f43d8100d90 2023-11-16T16:47:14.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.024+0000 7f43d6ffd700 1 --2- 172.21.15.44:0/1562021637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43d81036b0 0x7f43d8196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:14.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.025+0000 7f43d77fe700 1 --2- 172.21.15.44:0/1562021637 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43d8100d90 0x7f43d8196050 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f43c8005fd0 tx=0x7f43c800eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:14.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.025+0000 7f43d4ff9700 1 -- 172.21.15.44:0/1562021637 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c8010900 con 0x7f43d8100d90 2023-11-16T16:47:14.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.025+0000 7f43d4ff9700 1 -- 172.21.15.44:0/1562021637 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43c8018420 con 0x7f43d8100d90 2023-11-16T16:47:14.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.025+0000 7f43ddbff700 1 -- 172.21.15.44:0/1562021637 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d810bc60 con 0x7f43d8100d90 2023-11-16T16:47:14.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.025+0000 7f43d4ff9700 1 -- 172.21.15.44:0/1562021637 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c800f680 con 0x7f43d8100d90 2023-11-16T16:47:14.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.025+0000 7f43ddbff700 1 -- 172.21.15.44:0/1562021637 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d810bad0 con 0x7f43d8100d90 2023-11-16T16:47:14.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.026+0000 7f43ddbff700 1 -- 172.21.15.44:0/1562021637 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d80fc2c0 con 0x7f43d8100d90 2023-11-16T16:47:14.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.028+0000 7f43d4ff9700 1 -- 172.21.15.44:0/1562021637 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f43c8018590 con 0x7f43d8100d90 2023-11-16T16:47:14.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.030+0000 7f43d4ff9700 1 --2- 172.21.15.44:0/1562021637 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f43c00a11d0 0x7f43c00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:14.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.030+0000 7f43d4ff9700 1 -- 172.21.15.44:0/1562021637 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f43c8014070 con 0x7f43d8100d90 2023-11-16T16:47:14.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.030+0000 7f43d6ffd700 1 --2- 172.21.15.44:0/1562021637 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f43c00a11d0 0x7f43c00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:14.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.030+0000 7f43d6ffd700 1 --2- 172.21.15.44:0/1562021637 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f43c00a11d0 0x7f43c00a3680 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00c010 tx=0x7f43cc019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:14.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.034+0000 7f43d4ff9700 1 -- 172.21.15.44:0/1562021637 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f43c8051730 con 0x7f43d8100d90 2023-11-16T16:47:14.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.253+0000 7f43ddbff700 1 -- 172.21.15.44:0/1562021637 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43d8067a60 con 0x7f43d8100d90 2023-11-16T16:47:14.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.254+0000 7f43d4ff9700 1 -- 172.21.15.44:0/1562021637 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18387 (secure 0 0 0) 0x7f43c8096020 con 0x7f43d8100d90 2023-11-16T16:47:14.256 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:14.256 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":79,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:46:30.119708+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:14.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.256+0000 7f43ddbff700 1 -- 172.21.15.44:0/1562021637 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f43c00a11d0 msgr2=0x7f43c00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.256+0000 7f43ddbff700 1 --2- 172.21.15.44:0/1562021637 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f43c00a11d0 0x7f43c00a3680 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00c010 tx=0x7f43cc019040 comp rx=0 tx=0).stop 2023-11-16T16:47:14.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.256+0000 7f43ddbff700 1 -- 172.21.15.44:0/1562021637 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43d8100d90 msgr2=0x7f43d8196050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.256+0000 7f43ddbff700 1 --2- 172.21.15.44:0/1562021637 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43d8100d90 0x7f43d8196050 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f43c8005fd0 tx=0x7f43c800eca0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.256+0000 7f43ddbff700 1 -- 172.21.15.44:0/1562021637 shutdown_connections 2023-11-16T16:47:14.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.256+0000 7f43ddbff700 1 --2- 172.21.15.44:0/1562021637 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f43c00a11d0 0x7f43c00a3680 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.256+0000 7f43ddbff700 1 --2- 172.21.15.44:0/1562021637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43d8105fd0 0x7f43d819d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.256+0000 7f43ddbff700 1 --2- 172.21.15.44:0/1562021637 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43d8100d90 0x7f43d8196050 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.256+0000 7f43ddbff700 1 --2- 172.21.15.44:0/1562021637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43d81036b0 0x7f43d8196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.256+0000 7f43ddbff700 1 -- 172.21.15.44:0/1562021637 >> 172.21.15.44:0/1562021637 conn(0x7f43d80fa750 msgr2=0x7f43d81019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:14.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.257+0000 7f43ddbff700 1 -- 172.21.15.44:0/1562021637 shutdown_connections 2023-11-16T16:47:14.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.257+0000 7f43ddbff700 1 -- 172.21.15.44:0/1562021637 wait complete. 2023-11-16T16:47:14.278 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:03.059622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-16T16:47:14.278 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:47:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.487+0000 7f3c835ff700 1 -- 172.21.15.44:0/2263900974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c7c1002e0 msgr2=0x7f3c7c1006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.487+0000 7f3c835ff700 1 --2- 172.21.15.44:0/2263900974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c7c1002e0 0x7f3c7c1006f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009990 tx=0x7f3c78009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.488+0000 7f3c835ff700 1 -- 172.21.15.44:0/2263900974 shutdown_connections 2023-11-16T16:47:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.488+0000 7f3c835ff700 1 --2- 172.21.15.44:0/2263900974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c7c101e60 0x7f3c7c108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.488+0000 7f3c835ff700 1 --2- 172.21.15.44:0/2263900974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c7c1002e0 0x7f3c7c1006f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.488+0000 7f3c835ff700 1 --2- 172.21.15.44:0/2263900974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c7c1014d0 0x7f3c7c101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.488+0000 7f3c835ff700 1 -- 172.21.15.44:0/2263900974 >> 172.21.15.44:0/2263900974 conn(0x7f3c7c0fb8e0 msgr2=0x7f3c7c0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.488+0000 7f3c835ff700 1 -- 172.21.15.44:0/2263900974 shutdown_connections 2023-11-16T16:47:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.488+0000 7f3c835ff700 1 -- 172.21.15.44:0/2263900974 wait complete. 2023-11-16T16:47:14.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.489+0000 7f3c835ff700 1 Processor -- start 2023-11-16T16:47:14.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.489+0000 7f3c835ff700 1 -- start start 2023-11-16T16:47:14.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.489+0000 7f3c835ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c7c1002e0 0x7f3c7c1960e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:14.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.489+0000 7f3c835ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c7c1014d0 0x7f3c7c196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:14.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.489+0000 7f3c835ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c7c101e60 0x7f3c7c19d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:14.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.489+0000 7f3c835ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c19dc60 con 0x7f3c7c101e60 2023-11-16T16:47:14.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.489+0000 7f3c835ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c19dda0 con 0x7f3c7c1014d0 2023-11-16T16:47:14.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.489+0000 7f3c835ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c19df30 con 0x7f3c7c1002e0 2023-11-16T16:47:14.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.490+0000 7f3c80b9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c7c1014d0 0x7f3c7c196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:14.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.490+0000 7f3c8139b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c7c1002e0 0x7f3c7c1960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:14.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.490+0000 7f3c80b9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c7c1014d0 0x7f3c7c196620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:34728/0 (socket says 172.21.15.44:34728) 2023-11-16T16:47:14.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.490+0000 7f3c8139b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c7c1002e0 0x7f3c7c1960e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:41826/0 (socket says 172.21.15.44:41826) 2023-11-16T16:47:14.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.490+0000 7f3c80b9a700 1 -- 172.21.15.44:0/1310187463 learned_addr learned my addr 172.21.15.44:0/1310187463 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:14.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.490+0000 7f3c81b9c700 1 --2- 172.21.15.44:0/1310187463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c7c101e60 0x7f3c7c19d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:14.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.490+0000 7f3c80b9a700 1 -- 172.21.15.44:0/1310187463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c7c1002e0 msgr2=0x7f3c7c1960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.490+0000 7f3c80b9a700 1 --2- 172.21.15.44:0/1310187463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c7c1002e0 0x7f3c7c1960e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.490+0000 7f3c80b9a700 1 -- 172.21.15.44:0/1310187463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c7c101e60 msgr2=0x7f3c7c19d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.490+0000 7f3c80b9a700 1 --2- 172.21.15.44:0/1310187463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c7c101e60 0x7f3c7c19d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.490+0000 7f3c80b9a700 1 -- 172.21.15.44:0/1310187463 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c780095f0 con 0x7f3c7c1014d0 2023-11-16T16:47:14.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.490+0000 7f3c80b9a700 1 --2- 172.21.15.44:0/1310187463 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c7c1014d0 0x7f3c7c196620 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c00e9a0 tx=0x7f3c6c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:14.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.491+0000 7f3c727fc700 1 -- 172.21.15.44:0/1310187463 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c00ca10 con 0x7f3c7c1014d0 2023-11-16T16:47:14.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.491+0000 7f3c727fc700 1 -- 172.21.15.44:0/1310187463 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c6c00cb70 con 0x7f3c7c1014d0 2023-11-16T16:47:14.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.491+0000 7f3c835ff700 1 -- 172.21.15.44:0/1310187463 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7c10bcb0 con 0x7f3c7c1014d0 2023-11-16T16:47:14.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.491+0000 7f3c727fc700 1 -- 172.21.15.44:0/1310187463 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c018680 con 0x7f3c7c1014d0 2023-11-16T16:47:14.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.491+0000 7f3c835ff700 1 -- 172.21.15.44:0/1310187463 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7c10bb20 con 0x7f3c7c1014d0 2023-11-16T16:47:14.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.492+0000 7f3c835ff700 1 -- 172.21.15.44:0/1310187463 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7c1078a0 con 0x7f3c7c1014d0 2023-11-16T16:47:14.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.494+0000 7f3c727fc700 1 -- 172.21.15.44:0/1310187463 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3c6c00cce0 con 0x7f3c7c1014d0 2023-11-16T16:47:14.497 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.495+0000 7f3c727fc700 1 --2- 172.21.15.44:0/1310187463 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3c680a11d0 0x7f3c680a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:14.497 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.496+0000 7f3c727fc700 1 -- 172.21.15.44:0/1310187463 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3c6c014070 con 0x7f3c7c1014d0 2023-11-16T16:47:14.497 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.496+0000 7f3c8139b700 1 --2- 172.21.15.44:0/1310187463 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3c680a11d0 0x7f3c680a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:14.498 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.496+0000 7f3c8139b700 1 --2- 172.21.15.44:0/1310187463 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3c680a11d0 0x7f3c680a3680 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3c7800b2e0 tx=0x7f3c78005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:14.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.497+0000 7f3c727fc700 1 -- 172.21.15.44:0/1310187463 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3c6c0961a0 con 0x7f3c7c1014d0 2023-11-16T16:47:14.737 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.735+0000 7f3c835ff700 1 -- 172.21.15.44:0/1310187463 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3c7c061700 con 0x7f3c7c1014d0 2023-11-16T16:47:14.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.736+0000 7f3c727fc700 1 -- 172.21.15.44:0/1310187463 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3c6c0997c0 con 0x7f3c7c1014d0 2023-11-16T16:47:14.738 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2023-11-16T16:47:14.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.739+0000 7f3c835ff700 1 -- 172.21.15.44:0/1310187463 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3c680a11d0 msgr2=0x7f3c680a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.739+0000 7f3c835ff700 1 --2- 172.21.15.44:0/1310187463 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3c680a11d0 0x7f3c680a3680 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3c7800b2e0 tx=0x7f3c78005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.739+0000 7f3c835ff700 1 -- 172.21.15.44:0/1310187463 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c7c1014d0 msgr2=0x7f3c7c196620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.739+0000 7f3c835ff700 1 --2- 172.21.15.44:0/1310187463 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c7c1014d0 0x7f3c7c196620 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c00e9a0 tx=0x7f3c6c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.739+0000 7f3c835ff700 1 -- 172.21.15.44:0/1310187463 shutdown_connections 2023-11-16T16:47:14.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.739+0000 7f3c835ff700 1 --2- 172.21.15.44:0/1310187463 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3c680a11d0 0x7f3c680a3680 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.739+0000 7f3c835ff700 1 --2- 172.21.15.44:0/1310187463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c7c101e60 0x7f3c7c19d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.739+0000 7f3c835ff700 1 --2- 172.21.15.44:0/1310187463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c7c1002e0 0x7f3c7c1960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.739+0000 7f3c835ff700 1 --2- 172.21.15.44:0/1310187463 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c7c1014d0 0x7f3c7c196620 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.739+0000 7f3c835ff700 1 -- 172.21.15.44:0/1310187463 >> 172.21.15.44:0/1310187463 conn(0x7f3c7c0fb8e0 msgr2=0x7f3c7c1049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:14.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.739+0000 7f3c835ff700 1 -- 172.21.15.44:0/1310187463 shutdown_connections 2023-11-16T16:47:14.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.739+0000 7f3c835ff700 1 -- 172.21.15.44:0/1310187463 wait complete. 2023-11-16T16:47:14.760 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-16T16:47:14.822 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:14 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2838965558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:14.822 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:14 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1562021637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:14.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:14 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2838965558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:14.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:14 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1562021637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:14.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.970+0000 7f05711ff700 1 -- 172.21.15.44:0/2159914230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f056c073550 msgr2=0x7f056c0739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.970+0000 7f05711ff700 1 --2- 172.21.15.44:0/2159914230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f056c073550 0x7f056c0739c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0554009990 tx=0x7f0554009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.970+0000 7f05711ff700 1 -- 172.21.15.44:0/2159914230 shutdown_connections 2023-11-16T16:47:14.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.970+0000 7f05711ff700 1 --2- 172.21.15.44:0/2159914230 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f056c108150 0x7f056c10a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.970+0000 7f05711ff700 1 --2- 172.21.15.44:0/2159914230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f056c074b30 0x7f056c072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.970+0000 7f05711ff700 1 --2- 172.21.15.44:0/2159914230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f056c073550 0x7f056c0739c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.970+0000 7f05711ff700 1 -- 172.21.15.44:0/2159914230 >> 172.21.15.44:0/2159914230 conn(0x7f056c0fb870 msgr2=0x7f056c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:14.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.970+0000 7f05711ff700 1 -- 172.21.15.44:0/2159914230 shutdown_connections 2023-11-16T16:47:14.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.970+0000 7f05711ff700 1 -- 172.21.15.44:0/2159914230 wait complete. 2023-11-16T16:47:14.974 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.971+0000 7f05711ff700 1 Processor -- start 2023-11-16T16:47:14.974 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.971+0000 7f05711ff700 1 -- start start 2023-11-16T16:47:14.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.971+0000 7f05711ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f056c073550 0x7f056c1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:14.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.971+0000 7f05711ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f056c074b30 0x7f056c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:14.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.971+0000 7f05711ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f056c108150 0x7f056c19f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:14.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.971+0000 7f05711ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c19fd60 con 0x7f056c073550 2023-11-16T16:47:14.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.971+0000 7f05711ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c19fea0 con 0x7f056c108150 2023-11-16T16:47:14.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.971+0000 7f05711ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c1a0030 con 0x7f056c074b30 2023-11-16T16:47:14.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.971+0000 7f056ad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f056c073550 0x7f056c1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:14.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.972+0000 7f0563fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f056c074b30 0x7f056c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:14.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.972+0000 7f056ad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f056c073550 0x7f056c1981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43752/0 (socket says 172.21.15.44:43752) 2023-11-16T16:47:14.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.972+0000 7f056ad9d700 1 -- 172.21.15.44:0/871816390 learned_addr learned my addr 172.21.15.44:0/871816390 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:14.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.972+0000 7f056b59e700 1 --2- 172.21.15.44:0/871816390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f056c108150 0x7f056c19f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:14.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.972+0000 7f056ad9d700 1 -- 172.21.15.44:0/871816390 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f056c074b30 msgr2=0x7f056c198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.972+0000 7f056ad9d700 1 --2- 172.21.15.44:0/871816390 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f056c074b30 0x7f056c198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.972+0000 7f056ad9d700 1 -- 172.21.15.44:0/871816390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f056c108150 msgr2=0x7f056c19f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.972+0000 7f056ad9d700 1 --2- 172.21.15.44:0/871816390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f056c108150 0x7f056c19f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.972+0000 7f056ad9d700 1 -- 172.21.15.44:0/871816390 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05540095f0 con 0x7f056c073550 2023-11-16T16:47:14.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.972+0000 7f056ad9d700 1 --2- 172.21.15.44:0/871816390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f056c073550 0x7f056c1981e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f055c00e970 tx=0x7f055c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:14.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.972+0000 7f0568d99700 1 -- 172.21.15.44:0/871816390 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c009790 con 0x7f056c073550 2023-11-16T16:47:14.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.972+0000 7f0568d99700 1 -- 172.21.15.44:0/871816390 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f055c00c7e0 con 0x7f056c073550 2023-11-16T16:47:14.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.972+0000 7f05711ff700 1 -- 172.21.15.44:0/871816390 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056c10de70 con 0x7f056c073550 2023-11-16T16:47:14.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.972+0000 7f0568d99700 1 -- 172.21.15.44:0/871816390 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c0054e0 con 0x7f056c073550 2023-11-16T16:47:14.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.973+0000 7f05711ff700 1 -- 172.21.15.44:0/871816390 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056c10dce0 con 0x7f056c073550 2023-11-16T16:47:14.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:14 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2838965558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:14.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:14 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1562021637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:14.977 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.973+0000 7f05711ff700 1 -- 172.21.15.44:0/871816390 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056c106800 con 0x7f056c073550 2023-11-16T16:47:14.977 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.976+0000 7f0568d99700 1 -- 172.21.15.44:0/871816390 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f055c00c950 con 0x7f056c073550 2023-11-16T16:47:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.977+0000 7f0568d99700 1 --2- 172.21.15.44:0/871816390 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f05580a11d0 0x7f05580a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.977+0000 7f0568d99700 1 -- 172.21.15.44:0/871816390 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f055c014070 con 0x7f056c073550 2023-11-16T16:47:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.977+0000 7f0563fff700 1 --2- 172.21.15.44:0/871816390 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f05580a11d0 0x7f05580a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.978+0000 7f0563fff700 1 --2- 172.21.15.44:0/871816390 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f05580a11d0 0x7f05580a3680 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f055400b2e0 tx=0x7f0554005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:14.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:14.979+0000 7f0568d99700 1 -- 172.21.15.44:0/871816390 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f055c096460 con 0x7f056c073550 2023-11-16T16:47:15.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.206+0000 7f05711ff700 1 -- 172.21.15.44:0/871816390 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f056c02cfe0 con 0x7f056c073550 2023-11-16T16:47:15.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.207+0000 7f0568d99700 1 -- 172.21.15.44:0/871816390 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) v1 ==== 84+0+65 (secure 0 0 0) 0x7f055c099a80 con 0x7f056c073550 2023-11-16T16:47:15.209 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:15.209 INFO:teuthology.orchestra.run.smithi044.stdout:[ 2023-11-16T16:47:15.209 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T16:47:15.209 INFO:teuthology.orchestra.run.smithi044.stdout: "poolnum": 1, 2023-11-16T16:47:15.209 INFO:teuthology.orchestra.run.smithi044.stdout: "poolname": ".mgr" 2023-11-16T16:47:15.209 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:47:15.209 INFO:teuthology.orchestra.run.smithi044.stdout:] 2023-11-16T16:47:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.209+0000 7f05711ff700 1 -- 172.21.15.44:0/871816390 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f05580a11d0 msgr2=0x7f05580a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.209+0000 7f05711ff700 1 --2- 172.21.15.44:0/871816390 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f05580a11d0 0x7f05580a3680 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f055400b2e0 tx=0x7f0554005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.209+0000 7f05711ff700 1 -- 172.21.15.44:0/871816390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f056c073550 msgr2=0x7f056c1981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.209+0000 7f05711ff700 1 --2- 172.21.15.44:0/871816390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f056c073550 0x7f056c1981e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f055c00e970 tx=0x7f055c00ec80 comp rx=0 tx=0).stop 2023-11-16T16:47:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.210+0000 7f05711ff700 1 -- 172.21.15.44:0/871816390 shutdown_connections 2023-11-16T16:47:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.210+0000 7f05711ff700 1 --2- 172.21.15.44:0/871816390 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f05580a11d0 0x7f05580a3680 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.210+0000 7f05711ff700 1 --2- 172.21.15.44:0/871816390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f056c108150 0x7f056c19f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.210+0000 7f05711ff700 1 --2- 172.21.15.44:0/871816390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f056c073550 0x7f056c1981e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.210+0000 7f05711ff700 1 --2- 172.21.15.44:0/871816390 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f056c074b30 0x7f056c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.210+0000 7f05711ff700 1 -- 172.21.15.44:0/871816390 >> 172.21.15.44:0/871816390 conn(0x7f056c0fb870 msgr2=0x7f056c0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.210+0000 7f05711ff700 1 -- 172.21.15.44:0/871816390 shutdown_connections 2023-11-16T16:47:15.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.210+0000 7f05711ff700 1 -- 172.21.15.44:0/871816390 wait complete. 2023-11-16T16:47:15.230 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-16T16:47:15.230 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-16T16:47:15.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.447+0000 7ff5f2fff700 1 -- 172.21.15.44:0/1321967652 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5ec1002c0 msgr2=0x7ff5ec1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:15.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.447+0000 7ff5f2fff700 1 --2- 172.21.15.44:0/1321967652 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5ec1002c0 0x7ff5ec1006d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff5e00098f0 tx=0x7ff5e0009c00 comp rx=0 tx=0).stop 2023-11-16T16:47:15.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.447+0000 7ff5f2fff700 1 -- 172.21.15.44:0/1321967652 shutdown_connections 2023-11-16T16:47:15.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.447+0000 7ff5f2fff700 1 --2- 172.21.15.44:0/1321967652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5ec101e40 0x7ff5ec108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:15.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.447+0000 7ff5f2fff700 1 --2- 172.21.15.44:0/1321967652 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5ec1002c0 0x7ff5ec1006d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:15.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.447+0000 7ff5f2fff700 1 --2- 172.21.15.44:0/1321967652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5ec1014b0 0x7ff5ec101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:15.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.447+0000 7ff5f2fff700 1 -- 172.21.15.44:0/1321967652 >> 172.21.15.44:0/1321967652 conn(0x7ff5ec0fb8a0 msgr2=0x7ff5ec0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:15.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.448+0000 7ff5f2fff700 1 -- 172.21.15.44:0/1321967652 shutdown_connections 2023-11-16T16:47:15.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.448+0000 7ff5f2fff700 1 -- 172.21.15.44:0/1321967652 wait complete. 2023-11-16T16:47:15.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.448+0000 7ff5f2fff700 1 Processor -- start 2023-11-16T16:47:15.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.449+0000 7ff5f2fff700 1 -- start start 2023-11-16T16:47:15.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.449+0000 7ff5f2fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5ec1002c0 0x7ff5ec198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:15.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.449+0000 7ff5f2fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5ec1014b0 0x7ff5ec1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:15.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.449+0000 7ff5f2fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5ec101e40 0x7ff5ec19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:15.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.449+0000 7ff5f2fff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ec19fdf0 con 0x7ff5ec1014b0 2023-11-16T16:47:15.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.449+0000 7ff5f2fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ec19ff30 con 0x7ff5ec1002c0 2023-11-16T16:47:15.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.449+0000 7ff5f2fff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ec1a00c0 con 0x7ff5ec101e40 2023-11-16T16:47:15.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.449+0000 7ff5f0d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5ec1002c0 0x7ff5ec198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:15.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.449+0000 7ff5f0d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5ec1002c0 0x7ff5ec198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:46340/0 (socket says 172.21.15.44:46340) 2023-11-16T16:47:15.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.449+0000 7ff5f0d9b700 1 -- 172.21.15.44:0/615880553 learned_addr learned my addr 172.21.15.44:0/615880553 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:15.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.449+0000 7ff5f159c700 1 --2- 172.21.15.44:0/615880553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5ec101e40 0x7ff5ec19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:15.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.449+0000 7ff5ebfff700 1 --2- 172.21.15.44:0/615880553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5ec1014b0 0x7ff5ec1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:15.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.450+0000 7ff5f0d9b700 1 -- 172.21.15.44:0/615880553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5ec101e40 msgr2=0x7ff5ec19f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:15.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.450+0000 7ff5f0d9b700 1 --2- 172.21.15.44:0/615880553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5ec101e40 0x7ff5ec19f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:15.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.450+0000 7ff5f0d9b700 1 -- 172.21.15.44:0/615880553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5ec1014b0 msgr2=0x7ff5ec1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:15.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.450+0000 7ff5f0d9b700 1 --2- 172.21.15.44:0/615880553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5ec1014b0 0x7ff5ec1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:15.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.450+0000 7ff5f0d9b700 1 -- 172.21.15.44:0/615880553 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d8009520 con 0x7ff5ec1002c0 2023-11-16T16:47:15.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.450+0000 7ff5f159c700 1 --2- 172.21.15.44:0/615880553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5ec101e40 0x7ff5ec19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:15.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.450+0000 7ff5f0d9b700 1 --2- 172.21.15.44:0/615880553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5ec1002c0 0x7ff5ec198270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0009fd0 tx=0x7ff5e000b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:15.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.450+0000 7ff5e9ffb700 1 -- 172.21.15.44:0/615880553 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e001c070 con 0x7ff5ec1002c0 2023-11-16T16:47:15.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.450+0000 7ff5e9ffb700 1 -- 172.21.15.44:0/615880553 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff5e000f7d0 con 0x7ff5ec1002c0 2023-11-16T16:47:15.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.450+0000 7ff5f2fff700 1 -- 172.21.15.44:0/615880553 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e0009520 con 0x7ff5ec1002c0 2023-11-16T16:47:15.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.450+0000 7ff5e9ffb700 1 -- 172.21.15.44:0/615880553 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0003a40 con 0x7ff5ec1002c0 2023-11-16T16:47:15.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.450+0000 7ff5f2fff700 1 -- 172.21.15.44:0/615880553 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5ec10bc10 con 0x7ff5ec1002c0 2023-11-16T16:47:15.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.451+0000 7ff5f2fff700 1 -- 172.21.15.44:0/615880553 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5ec067b00 con 0x7ff5ec1002c0 2023-11-16T16:47:15.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.453+0000 7ff5e9ffb700 1 -- 172.21.15.44:0/615880553 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff5e0020030 con 0x7ff5ec1002c0 2023-11-16T16:47:15.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.454+0000 7ff5e9ffb700 1 --2- 172.21.15.44:0/615880553 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff5dc0a11d0 0x7ff5dc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:15.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.454+0000 7ff5e9ffb700 1 -- 172.21.15.44:0/615880553 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff5e00cd3d0 con 0x7ff5ec1002c0 2023-11-16T16:47:15.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.454+0000 7ff5ebfff700 1 --2- 172.21.15.44:0/615880553 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff5dc0a11d0 0x7ff5dc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:15.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.455+0000 7ff5ebfff700 1 --2- 172.21.15.44:0/615880553 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff5dc0a11d0 0x7ff5dc0a3680 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8009df0 tx=0x7ff5d8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:15.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.458+0000 7ff5e9ffb700 1 -- 172.21.15.44:0/615880553 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff5e0096fb0 con 0x7ff5ec1002c0 2023-11-16T16:47:15.676 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:15.674+0000 7ff5f2fff700 1 -- 172.21.15.44:0/615880553 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff5ec1a0b30 con 0x7ff5ec1002c0 2023-11-16T16:47:15.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:15 smithi044 ceph-mon[128750]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:15.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:15 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1310187463' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:15.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:15 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:15.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:15 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:15.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:15 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/871816390' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-11-16T16:47:15.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:15 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:15.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:15 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:15 smithi176 ceph-mon[131630]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:15 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1310187463' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:15 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:15 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:15 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/871816390' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-11-16T16:47:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:15 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:15 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:15.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:15 smithi077 ceph-mon[131204]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:15.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:15 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1310187463' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:15.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:15 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:15 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:15 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/871816390' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-11-16T16:47:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:15 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:15 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:16.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.591+0000 7ff5e9ffb700 1 -- 172.21.15.44:0/615880553 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v80) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff5e009a5d0 con 0x7ff5ec1002c0 2023-11-16T16:47:16.598 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.596+0000 7ff5f2fff700 1 -- 172.21.15.44:0/615880553 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff5dc0a11d0 msgr2=0x7ff5dc0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:16.598 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.596+0000 7ff5f2fff700 1 --2- 172.21.15.44:0/615880553 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff5dc0a11d0 0x7ff5dc0a3680 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8009df0 tx=0x7ff5d8009260 comp rx=0 tx=0).stop 2023-11-16T16:47:16.598 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.596+0000 7ff5f2fff700 1 -- 172.21.15.44:0/615880553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5ec1002c0 msgr2=0x7ff5ec198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:16.598 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.596+0000 7ff5f2fff700 1 --2- 172.21.15.44:0/615880553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5ec1002c0 0x7ff5ec198270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0009fd0 tx=0x7ff5e000b790 comp rx=0 tx=0).stop 2023-11-16T16:47:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.596+0000 7ff5f2fff700 1 -- 172.21.15.44:0/615880553 shutdown_connections 2023-11-16T16:47:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.596+0000 7ff5f2fff700 1 --2- 172.21.15.44:0/615880553 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff5dc0a11d0 0x7ff5dc0a3680 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.596+0000 7ff5f2fff700 1 --2- 172.21.15.44:0/615880553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5ec101e40 0x7ff5ec19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.596+0000 7ff5f2fff700 1 --2- 172.21.15.44:0/615880553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5ec1002c0 0x7ff5ec198270 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.596+0000 7ff5f2fff700 1 --2- 172.21.15.44:0/615880553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5ec1014b0 0x7ff5ec1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.596+0000 7ff5f2fff700 1 -- 172.21.15.44:0/615880553 >> 172.21.15.44:0/615880553 conn(0x7ff5ec0fb8a0 msgr2=0x7ff5ec0fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.597+0000 7ff5f2fff700 1 -- 172.21.15.44:0/615880553 shutdown_connections 2023-11-16T16:47:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.597+0000 7ff5f2fff700 1 -- 172.21.15.44:0/615880553 wait complete. 2023-11-16T16:47:16.600 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' created 2023-11-16T16:47:16.620 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-11-16T16:47:16.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.841+0000 7f6539dff700 1 -- 172.21.15.44:0/287051038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6534105fc0 msgr2=0x7f6534108420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:16.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.841+0000 7f6539dff700 1 --2- 172.21.15.44:0/287051038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6534105fc0 0x7f6534108420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f652c00b0a0 tx=0x7f652c00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.841+0000 7f6539dff700 1 -- 172.21.15.44:0/287051038 shutdown_connections 2023-11-16T16:47:16.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.841+0000 7f6539dff700 1 --2- 172.21.15.44:0/287051038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6534105fc0 0x7f6534108420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.841+0000 7f6539dff700 1 --2- 172.21.15.44:0/287051038 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6534100d80 0x7f6534103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.841+0000 7f6539dff700 1 --2- 172.21.15.44:0/287051038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65341036a0 0x7f6534105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.841+0000 7f6539dff700 1 -- 172.21.15.44:0/287051038 >> 172.21.15.44:0/287051038 conn(0x7f65340fa820 msgr2=0x7f65340fcc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:16.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.841+0000 7f6539dff700 1 -- 172.21.15.44:0/287051038 shutdown_connections 2023-11-16T16:47:16.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.841+0000 7f6539dff700 1 -- 172.21.15.44:0/287051038 wait complete. 2023-11-16T16:47:16.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.842+0000 7f6539dff700 1 Processor -- start 2023-11-16T16:47:16.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.842+0000 7f6539dff700 1 -- start start 2023-11-16T16:47:16.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f6539dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6534100d80 0x7f6534198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:16.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f6539dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65341036a0 0x7f6534198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:16.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f6539dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6534105fc0 0x7f653419f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:16.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f6539dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653419fed0 con 0x7f65341036a0 2023-11-16T16:47:16.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f6539dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65341a0010 con 0x7f6534100d80 2023-11-16T16:47:16.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f6539dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65341a01a0 con 0x7f6534105fc0 2023-11-16T16:47:16.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f65337fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6534100d80 0x7f6534198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:16.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f65337fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6534100d80 0x7f6534198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:46356/0 (socket says 172.21.15.44:46356) 2023-11-16T16:47:16.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f65337fe700 1 -- 172.21.15.44:0/545376724 learned_addr learned my addr 172.21.15.44:0/545376724 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:16.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f652bfff700 1 --2- 172.21.15.44:0/545376724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65341036a0 0x7f6534198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:16.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f6533fff700 1 --2- 172.21.15.44:0/545376724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6534105fc0 0x7f653419f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:16.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f65337fe700 1 -- 172.21.15.44:0/545376724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6534105fc0 msgr2=0x7f653419f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:16.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f65337fe700 1 --2- 172.21.15.44:0/545376724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6534105fc0 0x7f653419f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f65337fe700 1 -- 172.21.15.44:0/545376724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65341036a0 msgr2=0x7f6534198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:16.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f65337fe700 1 --2- 172.21.15.44:0/545376724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65341036a0 0x7f6534198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f65337fe700 1 -- 172.21.15.44:0/545376724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f652c009d00 con 0x7f6534100d80 2023-11-16T16:47:16.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f6533fff700 1 --2- 172.21.15.44:0/545376724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6534105fc0 0x7f653419f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:16.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f652bfff700 1 --2- 172.21.15.44:0/545376724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65341036a0 0x7f6534198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:16.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.843+0000 7f65337fe700 1 --2- 172.21.15.44:0/545376724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6534100d80 0x7f6534198350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f652400e9a0 tx=0x7f652400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:16.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.844+0000 7f65317fa700 1 -- 172.21.15.44:0/545376724 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6524009790 con 0x7f6534100d80 2023-11-16T16:47:16.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.844+0000 7f65317fa700 1 -- 172.21.15.44:0/545376724 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f652400cbd0 con 0x7f6534100d80 2023-11-16T16:47:16.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.844+0000 7f65317fa700 1 -- 172.21.15.44:0/545376724 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6524018680 con 0x7f6534100d80 2023-11-16T16:47:16.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.844+0000 7f6539dff700 1 -- 172.21.15.44:0/545376724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f653410bcb0 con 0x7f6534100d80 2023-11-16T16:47:16.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.844+0000 7f6539dff700 1 -- 172.21.15.44:0/545376724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f653410bb20 con 0x7f6534100d80 2023-11-16T16:47:16.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.845+0000 7f6539dff700 1 -- 172.21.15.44:0/545376724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6534107b30 con 0x7f6534100d80 2023-11-16T16:47:16.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.847+0000 7f65317fa700 1 -- 172.21.15.44:0/545376724 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f652400cd40 con 0x7f6534100d80 2023-11-16T16:47:16.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.848+0000 7f65317fa700 1 --2- 172.21.15.44:0/545376724 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f65200a11d0 0x7f65200a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:16.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.848+0000 7f65317fa700 1 -- 172.21.15.44:0/545376724 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f6524014070 con 0x7f6534100d80 2023-11-16T16:47:16.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.848+0000 7f652bfff700 1 --2- 172.21.15.44:0/545376724 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f65200a11d0 0x7f65200a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:16.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.849+0000 7f652bfff700 1 --2- 172.21.15.44:0/545376724 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f65200a11d0 0x7f65200a3680 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f651c009cb0 tx=0x7f651c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:16.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:16.852+0000 7f65317fa700 1 -- 172.21.15.44:0/545376724 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6524051730 con 0x7f6534100d80 2023-11-16T16:47:16.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:16 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/615880553' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T16:47:16.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:16 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T16:47:16.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:16 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:16.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:16 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:16.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:16 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:16.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:16 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:47:16.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:16 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:16.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:16 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/615880553' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T16:47:16.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:16 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T16:47:16.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:16 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:16.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:16 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:16.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:16 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:16.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:16 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:47:16.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:16 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:16.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:16 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/615880553' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T16:47:16.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:16 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T16:47:16.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:16 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:16.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:16 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:16.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:16 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:16.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:16 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:47:16.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:16 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:17.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.064+0000 7f6539dff700 1 -- 172.21.15.44:0/545376724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f653402cfe0 con 0x7f6534100d80 2023-11-16T16:47:17.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.617+0000 7f65317fa700 1 -- 172.21.15.44:0/545376724 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v81) v1 ==== 171+0+0 (secure 0 0 0) 0x7f6524099df0 con 0x7f6534100d80 2023-11-16T16:47:17.630 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.628+0000 7f6539dff700 1 -- 172.21.15.44:0/545376724 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f65200a11d0 msgr2=0x7f65200a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:17.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.628+0000 7f6539dff700 1 --2- 172.21.15.44:0/545376724 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f65200a11d0 0x7f65200a3680 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f651c009cb0 tx=0x7f651c009260 comp rx=0 tx=0).stop 2023-11-16T16:47:17.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.628+0000 7f6539dff700 1 -- 172.21.15.44:0/545376724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6534100d80 msgr2=0x7f6534198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:17.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.628+0000 7f6539dff700 1 --2- 172.21.15.44:0/545376724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6534100d80 0x7f6534198350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f652400e9a0 tx=0x7f652400ed60 comp rx=0 tx=0).stop 2023-11-16T16:47:17.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.629+0000 7f6539dff700 1 -- 172.21.15.44:0/545376724 shutdown_connections 2023-11-16T16:47:17.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.629+0000 7f6539dff700 1 --2- 172.21.15.44:0/545376724 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f65200a11d0 0x7f65200a3680 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:17.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.629+0000 7f6539dff700 1 --2- 172.21.15.44:0/545376724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6534105fc0 0x7f653419f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:17.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.629+0000 7f6539dff700 1 --2- 172.21.15.44:0/545376724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6534100d80 0x7f6534198350 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:17.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.629+0000 7f6539dff700 1 --2- 172.21.15.44:0/545376724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65341036a0 0x7f6534198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:17.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.629+0000 7f6539dff700 1 -- 172.21.15.44:0/545376724 >> 172.21.15.44:0/545376724 conn(0x7f65340fa820 msgr2=0x7f65340ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:17.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.630+0000 7f6539dff700 1 -- 172.21.15.44:0/545376724 shutdown_connections 2023-11-16T16:47:17.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.630+0000 7f6539dff700 1 -- 172.21.15.44:0/545376724 wait complete. 2023-11-16T16:47:17.638 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' created 2023-11-16T16:47:17.661 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-16T16:47:17.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:17 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:47:17.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:17 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:47:17.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:17 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:47:17.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:17 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T16:47:17.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:17 smithi044 ceph-mon[128750]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:17.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:17 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-11-16T16:47:17.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:17 smithi044 ceph-mon[128750]: osdmap e80: 12 total, 12 up, 12 in 2023-11-16T16:47:17.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:17 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:47:17.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.876+0000 7fb4981ff700 1 -- 172.21.15.44:0/1158372854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb490101e40 msgr2=0x7fb490108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:17.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.876+0000 7fb4981ff700 1 --2- 172.21.15.44:0/1158372854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb490101e40 0x7fb490108700 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb48800b0a0 tx=0x7fb48800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:17.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.877+0000 7fb4981ff700 1 -- 172.21.15.44:0/1158372854 shutdown_connections 2023-11-16T16:47:17.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.877+0000 7fb4981ff700 1 --2- 172.21.15.44:0/1158372854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb490101e40 0x7fb490108700 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:17.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.877+0000 7fb4981ff700 1 --2- 172.21.15.44:0/1158372854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4901002c0 0x7fb4901006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:17.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.877+0000 7fb4981ff700 1 --2- 172.21.15.44:0/1158372854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4901014b0 0x7fb490101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:17.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.877+0000 7fb4981ff700 1 -- 172.21.15.44:0/1158372854 >> 172.21.15.44:0/1158372854 conn(0x7fb4900fb8a0 msgr2=0x7fb4900fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:17.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.877+0000 7fb4981ff700 1 -- 172.21.15.44:0/1158372854 shutdown_connections 2023-11-16T16:47:17.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.877+0000 7fb4981ff700 1 -- 172.21.15.44:0/1158372854 wait complete. 2023-11-16T16:47:17.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.878+0000 7fb4981ff700 1 Processor -- start 2023-11-16T16:47:17.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.878+0000 7fb4981ff700 1 -- start start 2023-11-16T16:47:17.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.878+0000 7fb4981ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4901002c0 0x7fb490196170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:17.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.878+0000 7fb4981ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4901014b0 0x7fb4901966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:17.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.878+0000 7fb4981ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb490101e40 0x7fb49019d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:17.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.878+0000 7fb4981ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49019dcf0 con 0x7fb490101e40 2023-11-16T16:47:17.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.878+0000 7fb4981ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49019de30 con 0x7fb4901014b0 2023-11-16T16:47:17.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.878+0000 7fb4981ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49019dfc0 con 0x7fb4901002c0 2023-11-16T16:47:17.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.878+0000 7fb49579a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4901014b0 0x7fb4901966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:17.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.878+0000 7fb49679c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb490101e40 0x7fb49019d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:17.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.878+0000 7fb49579a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4901014b0 0x7fb4901966b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:46386/0 (socket says 172.21.15.44:46386) 2023-11-16T16:47:17.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.878+0000 7fb49579a700 1 -- 172.21.15.44:0/4273869729 learned_addr learned my addr 172.21.15.44:0/4273869729 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:17.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.878+0000 7fb495f9b700 1 --2- 172.21.15.44:0/4273869729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4901002c0 0x7fb490196170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:17.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.879+0000 7fb49579a700 1 -- 172.21.15.44:0/4273869729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4901002c0 msgr2=0x7fb490196170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:17.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.879+0000 7fb49579a700 1 --2- 172.21.15.44:0/4273869729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4901002c0 0x7fb490196170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:17.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.879+0000 7fb49579a700 1 -- 172.21.15.44:0/4273869729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb490101e40 msgr2=0x7fb49019d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:17.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.879+0000 7fb49579a700 1 --2- 172.21.15.44:0/4273869729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb490101e40 0x7fb49019d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:17.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.879+0000 7fb49579a700 1 -- 172.21.15.44:0/4273869729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb488009d00 con 0x7fb4901014b0 2023-11-16T16:47:17.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.879+0000 7fb495f9b700 1 --2- 172.21.15.44:0/4273869729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4901002c0 0x7fb490196170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:17.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.879+0000 7fb49579a700 1 --2- 172.21.15.44:0/4273869729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4901014b0 0x7fb4901966b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb48000e9a0 tx=0x7fb48000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:17.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.879+0000 7fb486ffd700 1 -- 172.21.15.44:0/4273869729 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480009790 con 0x7fb4901014b0 2023-11-16T16:47:17.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.879+0000 7fb486ffd700 1 -- 172.21.15.44:0/4273869729 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb48000cbd0 con 0x7fb4901014b0 2023-11-16T16:47:17.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.879+0000 7fb4981ff700 1 -- 172.21.15.44:0/4273869729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49010bc90 con 0x7fb4901014b0 2023-11-16T16:47:17.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.879+0000 7fb486ffd700 1 -- 172.21.15.44:0/4273869729 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480018680 con 0x7fb4901014b0 2023-11-16T16:47:17.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.879+0000 7fb4981ff700 1 -- 172.21.15.44:0/4273869729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49010bb00 con 0x7fb4901014b0 2023-11-16T16:47:17.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.880+0000 7fb4981ff700 1 -- 172.21.15.44:0/4273869729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4901077e0 con 0x7fb4901014b0 2023-11-16T16:47:17.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.882+0000 7fb486ffd700 1 -- 172.21.15.44:0/4273869729 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb48000cd40 con 0x7fb4901014b0 2023-11-16T16:47:17.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.883+0000 7fb486ffd700 1 --2- 172.21.15.44:0/4273869729 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb47c0a1150 0x7fb47c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:17.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.883+0000 7fb486ffd700 1 -- 172.21.15.44:0/4273869729 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fb480014070 con 0x7fb4901014b0 2023-11-16T16:47:17.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.883+0000 7fb495f9b700 1 --2- 172.21.15.44:0/4273869729 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb47c0a1150 0x7fb47c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:17.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.884+0000 7fb495f9b700 1 --2- 172.21.15.44:0/4273869729 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb47c0a1150 0x7fb47c0a3600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb490101310 tx=0x7fb48c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:17.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:17.887+0000 7fb486ffd700 1 -- 172.21.15.44:0/4273869729 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb480051730 con 0x7fb4901014b0 2023-11-16T16:47:17.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:17 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:47:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:17 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:47:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:17 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:47:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:17 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T16:47:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:17 smithi176 ceph-mon[131630]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:17 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-11-16T16:47:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:17 smithi176 ceph-mon[131630]: osdmap e80: 12 total, 12 up, 12 in 2023-11-16T16:47:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:17 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:47:17.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:17 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:47:17.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:17 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:47:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:17 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:47:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:17 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T16:47:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:17 smithi077 ceph-mon[131204]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:17 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-11-16T16:47:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:17 smithi077 ceph-mon[131204]: osdmap e80: 12 total, 12 up, 12 in 2023-11-16T16:47:17.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:17 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:47:18.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.148+0000 7fb4981ff700 1 -- 172.21.15.44:0/4273869729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb49002cfe0 con 0x7fb4901014b0 2023-11-16T16:47:18.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.161+0000 7fb486ffd700 1 -- 172.21.15.44:0/4273869729 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fb4800962b0 con 0x7fb4901014b0 2023-11-16T16:47:18.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.170+0000 7fb4981ff700 1 -- 172.21.15.44:0/4273869729 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb47c0a1150 msgr2=0x7fb47c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:18.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.170+0000 7fb4981ff700 1 --2- 172.21.15.44:0/4273869729 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb47c0a1150 0x7fb47c0a3600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb490101310 tx=0x7fb48c009260 comp rx=0 tx=0).stop 2023-11-16T16:47:18.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.170+0000 7fb4981ff700 1 -- 172.21.15.44:0/4273869729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4901014b0 msgr2=0x7fb4901966b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:18.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.170+0000 7fb4981ff700 1 --2- 172.21.15.44:0/4273869729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4901014b0 0x7fb4901966b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb48000e9a0 tx=0x7fb48000ecb0 comp rx=0 tx=0).stop 2023-11-16T16:47:18.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.170+0000 7fb4981ff700 1 -- 172.21.15.44:0/4273869729 shutdown_connections 2023-11-16T16:47:18.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.170+0000 7fb4981ff700 1 --2- 172.21.15.44:0/4273869729 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb47c0a1150 0x7fb47c0a3600 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:18.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.170+0000 7fb4981ff700 1 --2- 172.21.15.44:0/4273869729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb490101e40 0x7fb49019d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:18.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.170+0000 7fb4981ff700 1 --2- 172.21.15.44:0/4273869729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4901002c0 0x7fb490196170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:18.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.170+0000 7fb4981ff700 1 --2- 172.21.15.44:0/4273869729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4901014b0 0x7fb4901966b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:18.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.170+0000 7fb4981ff700 1 -- 172.21.15.44:0/4273869729 >> 172.21.15.44:0/4273869729 conn(0x7fb4900fb8a0 msgr2=0x7fb490104920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:18.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.172+0000 7fb4981ff700 1 -- 172.21.15.44:0/4273869729 shutdown_connections 2023-11-16T16:47:18.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.172+0000 7fb4981ff700 1 -- 172.21.15.44:0/4273869729 wait complete. 2023-11-16T16:47:18.182 INFO:teuthology.orchestra.run.smithi044.stderr:new fs with metadata pool 2 and data pool 3 2023-11-16T16:47:18.212 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-11-16T16:47:18.212 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-11-16T16:47:18.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.427+0000 7f2d7a5ff700 1 -- 172.21.15.44:0/1712394984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d74101e50 msgr2=0x7f2d74108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:18.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.427+0000 7f2d7a5ff700 1 --2- 172.21.15.44:0/1712394984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d74101e50 0x7f2d74108710 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00b0a0 tx=0x7f2d6c00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:18.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.427+0000 7f2d7a5ff700 1 -- 172.21.15.44:0/1712394984 shutdown_connections 2023-11-16T16:47:18.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.427+0000 7f2d7a5ff700 1 --2- 172.21.15.44:0/1712394984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d74101e50 0x7f2d74108710 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:18.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.427+0000 7f2d7a5ff700 1 --2- 172.21.15.44:0/1712394984 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d741002d0 0x7f2d741006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:18.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.427+0000 7f2d7a5ff700 1 --2- 172.21.15.44:0/1712394984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d741014c0 0x7f2d74101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:18.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.427+0000 7f2d7a5ff700 1 -- 172.21.15.44:0/1712394984 >> 172.21.15.44:0/1712394984 conn(0x7f2d740fb8b0 msgr2=0x7f2d740fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:18.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.427+0000 7f2d7a5ff700 1 -- 172.21.15.44:0/1712394984 shutdown_connections 2023-11-16T16:47:18.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.427+0000 7f2d7a5ff700 1 -- 172.21.15.44:0/1712394984 wait complete. 2023-11-16T16:47:18.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.428+0000 7f2d7a5ff700 1 Processor -- start 2023-11-16T16:47:18.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.429+0000 7f2d7a5ff700 1 -- start start 2023-11-16T16:47:18.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.429+0000 7f2d7a5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d741002d0 0x7f2d740748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:18.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.429+0000 7f2d7a5ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d741014c0 0x7f2d74072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:18.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.429+0000 7f2d7a5ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d74101e50 0x7f2d74073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:18.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.429+0000 7f2d7a5ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d74073bc0 con 0x7f2d74101e50 2023-11-16T16:47:18.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.429+0000 7f2d7a5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d74073d30 con 0x7f2d741002d0 2023-11-16T16:47:18.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.429+0000 7f2d7a5ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d741a0220 con 0x7f2d741014c0 2023-11-16T16:47:18.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.429+0000 7f2d73fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d741002d0 0x7f2d740748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:18.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.429+0000 7f2d73fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d741002d0 0x7f2d740748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:46394/0 (socket says 172.21.15.44:46394) 2023-11-16T16:47:18.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.429+0000 7f2d73fff700 1 -- 172.21.15.44:0/2312510971 learned_addr learned my addr 172.21.15.44:0/2312510971 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:18.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.429+0000 7f2d737fe700 1 --2- 172.21.15.44:0/2312510971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d741014c0 0x7f2d74072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:18.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.429+0000 7f2d78b9c700 1 --2- 172.21.15.44:0/2312510971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d74101e50 0x7f2d74073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:18.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.429+0000 7f2d73fff700 1 -- 172.21.15.44:0/2312510971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d741014c0 msgr2=0x7f2d74072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:18.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.430+0000 7f2d73fff700 1 --2- 172.21.15.44:0/2312510971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d741014c0 0x7f2d74072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:18.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.430+0000 7f2d73fff700 1 -- 172.21.15.44:0/2312510971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d74101e50 msgr2=0x7f2d74073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:18.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.430+0000 7f2d73fff700 1 --2- 172.21.15.44:0/2312510971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d74101e50 0x7f2d74073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:18.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.430+0000 7f2d73fff700 1 -- 172.21.15.44:0/2312510971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6c009d00 con 0x7f2d741002d0 2023-11-16T16:47:18.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.430+0000 7f2d73fff700 1 --2- 172.21.15.44:0/2312510971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d741002d0 0x7f2d740748a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400e970 tx=0x7f2d6400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:18.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.430+0000 7f2d717fa700 1 -- 172.21.15.44:0/2312510971 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64009790 con 0x7f2d741002d0 2023-11-16T16:47:18.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.430+0000 7f2d717fa700 1 -- 172.21.15.44:0/2312510971 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d6400c7e0 con 0x7f2d741002d0 2023-11-16T16:47:18.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.430+0000 7f2d7a5ff700 1 -- 172.21.15.44:0/2312510971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7410bca0 con 0x7f2d741002d0 2023-11-16T16:47:18.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.430+0000 7f2d717fa700 1 -- 172.21.15.44:0/2312510971 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64003c70 con 0x7f2d741002d0 2023-11-16T16:47:18.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.430+0000 7f2d7a5ff700 1 -- 172.21.15.44:0/2312510971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7410bb10 con 0x7f2d741002d0 2023-11-16T16:47:18.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.431+0000 7f2d7a5ff700 1 -- 172.21.15.44:0/2312510971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d741a0940 con 0x7f2d741002d0 2023-11-16T16:47:18.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.433+0000 7f2d717fa700 1 -- 172.21.15.44:0/2312510971 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2d6400cc50 con 0x7f2d741002d0 2023-11-16T16:47:18.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.434+0000 7f2d717fa700 1 --2- 172.21.15.44:0/2312510971 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2d5c0a1150 0x7f2d5c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:18.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.435+0000 7f2d717fa700 1 -- 172.21.15.44:0/2312510971 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f2d64014070 con 0x7f2d741002d0 2023-11-16T16:47:18.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.435+0000 7f2d737fe700 1 --2- 172.21.15.44:0/2312510971 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2d5c0a1150 0x7f2d5c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:18.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.435+0000 7f2d737fe700 1 --2- 172.21.15.44:0/2312510971 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2d5c0a1150 0x7f2d5c0a3600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2d68009c60 tx=0x7f2d68009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:18.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.436+0000 7f2d717fa700 1 -- 172.21.15.44:0/2312510971 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2d640967f0 con 0x7f2d741002d0 2023-11-16T16:47:18.668 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:18.666+0000 7f2d7a5ff700 1 -- 172.21.15.44:0/2312510971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f2d741a1070 con 0x7f2d741002d0 2023-11-16T16:47:18.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:18 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/545376724' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T16:47:18.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:18 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T16:47:18.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:18 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-11-16T16:47:18.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:18 smithi044 ceph-mon[128750]: osdmap e81: 12 total, 12 up, 12 in 2023-11-16T16:47:18.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:18 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/4273869729' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T16:47:18.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:18 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T16:47:18.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:18 smithi044 ceph-mon[128750]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-11-16T16:47:18.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:18 smithi044 ceph-mon[128750]: osdmap e82: 12 total, 12 up, 12 in 2023-11-16T16:47:18.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:18 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-11-16T16:47:18.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:18 smithi044 ceph-mon[128750]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-11-16T16:47:18.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:18 smithi044 ceph-mon[128750]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-11-16T16:47:18.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:18 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/545376724' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T16:47:18.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:18 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T16:47:18.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:18 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-11-16T16:47:18.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:18 smithi176 ceph-mon[131630]: osdmap e81: 12 total, 12 up, 12 in 2023-11-16T16:47:18.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:18 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/4273869729' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T16:47:18.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:18 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T16:47:18.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:18 smithi176 ceph-mon[131630]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-11-16T16:47:18.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:18 smithi176 ceph-mon[131630]: osdmap e82: 12 total, 12 up, 12 in 2023-11-16T16:47:18.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:18 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-11-16T16:47:18.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:18 smithi176 ceph-mon[131630]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-11-16T16:47:18.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:18 smithi176 ceph-mon[131630]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-11-16T16:47:18.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:18 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/545376724' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T16:47:18.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:18 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T16:47:18.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:18 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-11-16T16:47:18.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:18 smithi077 ceph-mon[131204]: osdmap e81: 12 total, 12 up, 12 in 2023-11-16T16:47:18.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:18 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/4273869729' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T16:47:18.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:18 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T16:47:18.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:18 smithi077 ceph-mon[131204]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-11-16T16:47:18.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:18 smithi077 ceph-mon[131204]: osdmap e82: 12 total, 12 up, 12 in 2023-11-16T16:47:18.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:18 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-11-16T16:47:18.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:18 smithi077 ceph-mon[131204]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-11-16T16:47:18.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:18 smithi077 ceph-mon[131204]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-11-16T16:47:19.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.161+0000 7f2d717fa700 1 -- 172.21.15.44:0/2312510971 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v83) v1 ==== 157+0+0 (secure 0 0 0) 0x7f2d64003dd0 con 0x7f2d741002d0 2023-11-16T16:47:19.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.164+0000 7f2d7a5ff700 1 -- 172.21.15.44:0/2312510971 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2d5c0a1150 msgr2=0x7f2d5c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:19.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.164+0000 7f2d7a5ff700 1 --2- 172.21.15.44:0/2312510971 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2d5c0a1150 0x7f2d5c0a3600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2d68009c60 tx=0x7f2d68009260 comp rx=0 tx=0).stop 2023-11-16T16:47:19.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.164+0000 7f2d7a5ff700 1 -- 172.21.15.44:0/2312510971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d741002d0 msgr2=0x7f2d740748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:19.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.164+0000 7f2d7a5ff700 1 --2- 172.21.15.44:0/2312510971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d741002d0 0x7f2d740748a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400e970 tx=0x7f2d6400ec80 comp rx=0 tx=0).stop 2023-11-16T16:47:19.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.164+0000 7f2d7a5ff700 1 -- 172.21.15.44:0/2312510971 shutdown_connections 2023-11-16T16:47:19.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.164+0000 7f2d7a5ff700 1 --2- 172.21.15.44:0/2312510971 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2d5c0a1150 0x7f2d5c0a3600 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.164+0000 7f2d7a5ff700 1 --2- 172.21.15.44:0/2312510971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d74101e50 0x7f2d74073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.164+0000 7f2d7a5ff700 1 --2- 172.21.15.44:0/2312510971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d741002d0 0x7f2d740748a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.164+0000 7f2d7a5ff700 1 --2- 172.21.15.44:0/2312510971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d741014c0 0x7f2d74072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.164+0000 7f2d7a5ff700 1 -- 172.21.15.44:0/2312510971 >> 172.21.15.44:0/2312510971 conn(0x7f2d740fb8b0 msgr2=0x7f2d740fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:19.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.164+0000 7f2d7a5ff700 1 -- 172.21.15.44:0/2312510971 shutdown_connections 2023-11-16T16:47:19.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.164+0000 7f2d7a5ff700 1 -- 172.21.15.44:0/2312510971 wait complete. 2023-11-16T16:47:19.187 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-11-16T16:47:19.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.419+0000 7efec1dff700 1 -- 172.21.15.44:0/4169666732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efebc074b30 msgr2=0x7efebc072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:19.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.419+0000 7efec1dff700 1 --2- 172.21.15.44:0/4169666732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efebc074b30 0x7efebc072f80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efeac009990 tx=0x7efeac009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.419+0000 7efec1dff700 1 -- 172.21.15.44:0/4169666732 shutdown_connections 2023-11-16T16:47:19.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.419+0000 7efec1dff700 1 --2- 172.21.15.44:0/4169666732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efebc108190 0x7efebc10a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.419+0000 7efec1dff700 1 --2- 172.21.15.44:0/4169666732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efebc074b30 0x7efebc072f80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.419+0000 7efec1dff700 1 --2- 172.21.15.44:0/4169666732 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efebc073550 0x7efebc0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.419+0000 7efec1dff700 1 -- 172.21.15.44:0/4169666732 >> 172.21.15.44:0/4169666732 conn(0x7efebc0fb8b0 msgr2=0x7efebc0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:19.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.419+0000 7efec1dff700 1 -- 172.21.15.44:0/4169666732 shutdown_connections 2023-11-16T16:47:19.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.419+0000 7efec1dff700 1 -- 172.21.15.44:0/4169666732 wait complete. 2023-11-16T16:47:19.423 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.420+0000 7efec1dff700 1 Processor -- start 2023-11-16T16:47:19.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.421+0000 7efec1dff700 1 -- start start 2023-11-16T16:47:19.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.421+0000 7efec1dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efebc073550 0x7efebc1982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:19.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.421+0000 7efec1dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efebc074b30 0x7efebc1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:19.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.421+0000 7efec1dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efebc108190 0x7efebc19f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:19.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.421+0000 7efec1dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc19fe20 con 0x7efebc074b30 2023-11-16T16:47:19.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.421+0000 7efec1dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc19ff60 con 0x7efebc073550 2023-11-16T16:47:19.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.421+0000 7efec1dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc1a00f0 con 0x7efebc108190 2023-11-16T16:47:19.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.421+0000 7efebb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efebc073550 0x7efebc1982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:19.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.421+0000 7efebb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efebc073550 0x7efebc1982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:46408/0 (socket says 172.21.15.44:46408) 2023-11-16T16:47:19.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.421+0000 7efebaffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efebc074b30 0x7efebc1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:19.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.421+0000 7efebb7fe700 1 -- 172.21.15.44:0/310722486 learned_addr learned my addr 172.21.15.44:0/310722486 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:19.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.421+0000 7efebbfff700 1 --2- 172.21.15.44:0/310722486 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efebc108190 0x7efebc19f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:19.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.422+0000 7efebb7fe700 1 -- 172.21.15.44:0/310722486 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efebc108190 msgr2=0x7efebc19f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:19.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.422+0000 7efebb7fe700 1 --2- 172.21.15.44:0/310722486 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efebc108190 0x7efebc19f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.422+0000 7efebb7fe700 1 -- 172.21.15.44:0/310722486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efebc074b30 msgr2=0x7efebc1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:19.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.422+0000 7efebb7fe700 1 --2- 172.21.15.44:0/310722486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efebc074b30 0x7efebc1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.422+0000 7efebb7fe700 1 -- 172.21.15.44:0/310722486 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb0009520 con 0x7efebc073550 2023-11-16T16:47:19.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.422+0000 7efebbfff700 1 --2- 172.21.15.44:0/310722486 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efebc108190 0x7efebc19f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:19.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.422+0000 7efebb7fe700 1 --2- 172.21.15.44:0/310722486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efebc073550 0x7efebc1982a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efeac005720 tx=0x7efeac009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:19.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.422+0000 7efeb8ff9700 1 -- 172.21.15.44:0/310722486 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac01d070 con 0x7efebc073550 2023-11-16T16:47:19.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.422+0000 7efeb8ff9700 1 -- 172.21.15.44:0/310722486 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efeac021410 con 0x7efebc073550 2023-11-16T16:47:19.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.422+0000 7efec1dff700 1 -- 172.21.15.44:0/310722486 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb4009c40 con 0x7efebc073550 2023-11-16T16:47:19.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.422+0000 7efeb8ff9700 1 -- 172.21.15.44:0/310722486 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac017650 con 0x7efebc073550 2023-11-16T16:47:19.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.422+0000 7efec1dff700 1 -- 172.21.15.44:0/310722486 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeac0095f0 con 0x7efebc073550 2023-11-16T16:47:19.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.424+0000 7efec1dff700 1 -- 172.21.15.44:0/310722486 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efebc106840 con 0x7efebc073550 2023-11-16T16:47:19.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.426+0000 7efeb8ff9700 1 -- 172.21.15.44:0/310722486 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7efeac021580 con 0x7efebc073550 2023-11-16T16:47:19.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.427+0000 7efeb8ff9700 1 --2- 172.21.15.44:0/310722486 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efea40a1150 0x7efea40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:19.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.427+0000 7efeb8ff9700 1 -- 172.21.15.44:0/310722486 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8205+0+0 (secure 0 0 0) 0x7efeac0cdbb0 con 0x7efebc073550 2023-11-16T16:47:19.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.427+0000 7efebaffd700 1 --2- 172.21.15.44:0/310722486 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efea40a1150 0x7efea40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:19.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.428+0000 7efebaffd700 1 --2- 172.21.15.44:0/310722486 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efea40a1150 0x7efea40a3600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efebc074850 tx=0x7efeb0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:19.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.432+0000 7efeb8ff9700 1 -- 172.21.15.44:0/310722486 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efeac097470 con 0x7efebc073550 2023-11-16T16:47:19.656 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:19.655+0000 7efec1dff700 1 -- 172.21.15.44:0/310722486 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7efebc02cbc0 con 0x7efebc073550 2023-11-16T16:47:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:19 smithi044 ceph-mon[128750]: pgmap v270: 97 pgs: 21 creating+peering, 6 active+clean, 70 unknown; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:19 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2312510971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-11-16T16:47:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:19 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-11-16T16:47:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:19 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-11-16T16:47:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:19 smithi044 ceph-mon[128750]: osdmap e83: 12 total, 12 up, 12 in 2023-11-16T16:47:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:19 smithi044 ceph-mon[128750]: mds.? [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] up:active 2023-11-16T16:47:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:19 smithi044 ceph-mon[128750]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:19.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:19 smithi044 ceph-mon[128750]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:19.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:19.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:19.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:19.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:19 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:19.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:19 smithi176 ceph-mon[131630]: pgmap v270: 97 pgs: 21 creating+peering, 6 active+clean, 70 unknown; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:19.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:19 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2312510971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-11-16T16:47:19.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:19 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-11-16T16:47:19.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:19 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-11-16T16:47:19.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:19 smithi176 ceph-mon[131630]: osdmap e83: 12 total, 12 up, 12 in 2023-11-16T16:47:19.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:19 smithi176 ceph-mon[131630]: mds.? [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] up:active 2023-11-16T16:47:19.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:19 smithi176 ceph-mon[131630]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:19.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:19 smithi176 ceph-mon[131630]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:19.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:19.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:19.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:19.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:19 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:19.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:19 smithi077 ceph-mon[131204]: pgmap v270: 97 pgs: 21 creating+peering, 6 active+clean, 70 unknown; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:19.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:19 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2312510971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-11-16T16:47:19.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:19 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-11-16T16:47:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:19 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-11-16T16:47:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:19 smithi077 ceph-mon[131204]: osdmap e83: 12 total, 12 up, 12 in 2023-11-16T16:47:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:19 smithi077 ceph-mon[131204]: mds.? [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] up:active 2023-11-16T16:47:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:19 smithi077 ceph-mon[131204]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:19 smithi077 ceph-mon[131204]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:19 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:20.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:20 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/310722486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-11-16T16:47:20.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:20 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-11-16T16:47:20.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:20 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:20.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:20 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:20.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:20 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:20.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:20 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:47:20.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:20 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:20.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:20 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/310722486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-11-16T16:47:20.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:20 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-11-16T16:47:20.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:20 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:20.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:20 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:20.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:20 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:20.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:20 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:47:20.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:20 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:20.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:20 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/310722486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-11-16T16:47:20.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:20 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-11-16T16:47:20.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:20 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:20.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:20 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:20.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:20 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:20.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:20 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:47:20.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:20 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:47:21.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.629+0000 7efeb8ff9700 1 -- 172.21.15.44:0/310722486 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v85) v1 ==== 228+0+0 (secure 0 0 0) 0x7efeac026100 con 0x7efebc073550 2023-11-16T16:47:21.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.637+0000 7efec1dff700 1 -- 172.21.15.44:0/310722486 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efea40a1150 msgr2=0x7efea40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:21.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.637+0000 7efec1dff700 1 --2- 172.21.15.44:0/310722486 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efea40a1150 0x7efea40a3600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efebc074850 tx=0x7efeb0009260 comp rx=0 tx=0).stop 2023-11-16T16:47:21.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.637+0000 7efec1dff700 1 -- 172.21.15.44:0/310722486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efebc073550 msgr2=0x7efebc1982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:21.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.637+0000 7efec1dff700 1 --2- 172.21.15.44:0/310722486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efebc073550 0x7efebc1982a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efeac005720 tx=0x7efeac009580 comp rx=0 tx=0).stop 2023-11-16T16:47:21.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.637+0000 7efec1dff700 1 -- 172.21.15.44:0/310722486 shutdown_connections 2023-11-16T16:47:21.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.637+0000 7efec1dff700 1 --2- 172.21.15.44:0/310722486 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efea40a1150 0x7efea40a3600 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:21.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.637+0000 7efec1dff700 1 --2- 172.21.15.44:0/310722486 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efebc108190 0x7efebc19f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:21.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.637+0000 7efec1dff700 1 --2- 172.21.15.44:0/310722486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efebc073550 0x7efebc1982a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:21.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.637+0000 7efec1dff700 1 --2- 172.21.15.44:0/310722486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efebc074b30 0x7efebc1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:21.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.637+0000 7efec1dff700 1 -- 172.21.15.44:0/310722486 >> 172.21.15.44:0/310722486 conn(0x7efebc0fb8b0 msgr2=0x7efebc0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:21.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.637+0000 7efec1dff700 1 -- 172.21.15.44:0/310722486 shutdown_connections 2023-11-16T16:47:21.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.638+0000 7efec1dff700 1 -- 172.21.15.44:0/310722486 wait complete. 2023-11-16T16:47:21.641 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data_ec' created 2023-11-16T16:47:21.661 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-11-16T16:47:21.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.870+0000 7f7a72dff700 1 -- 172.21.15.44:0/1968706719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a6c100de0 msgr2=0x7f7a6c1031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:21.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.870+0000 7f7a72dff700 1 --2- 172.21.15.44:0/1968706719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a6c100de0 0x7f7a6c1031c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7a60009990 tx=0x7f7a60009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:21.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.870+0000 7f7a72dff700 1 -- 172.21.15.44:0/1968706719 shutdown_connections 2023-11-16T16:47:21.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.870+0000 7f7a72dff700 1 --2- 172.21.15.44:0/1968706719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c106020 0x7f7a6c108400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:21.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.870+0000 7f7a72dff700 1 --2- 172.21.15.44:0/1968706719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a6c100de0 0x7f7a6c1031c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:21.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.870+0000 7f7a72dff700 1 --2- 172.21.15.44:0/1968706719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a6c103700 0x7f7a6c105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:21.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.870+0000 7f7a72dff700 1 -- 172.21.15.44:0/1968706719 >> 172.21.15.44:0/1968706719 conn(0x7f7a6c0fa7c0 msgr2=0x7f7a6c0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:21.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.870+0000 7f7a72dff700 1 -- 172.21.15.44:0/1968706719 shutdown_connections 2023-11-16T16:47:21.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.870+0000 7f7a72dff700 1 -- 172.21.15.44:0/1968706719 wait complete. 2023-11-16T16:47:21.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.871+0000 7f7a72dff700 1 Processor -- start 2023-11-16T16:47:21.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.871+0000 7f7a72dff700 1 -- start start 2023-11-16T16:47:21.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.871+0000 7f7a72dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a6c100de0 0x7f7a6c100750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:21.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.872+0000 7f7a72dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c103700 0x7f7a6c0fed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:21.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.872+0000 7f7a72dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a6c106020 0x7f7a6c0ff2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:21.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.872+0000 7f7a72dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c0ffa70 con 0x7f7a6c103700 2023-11-16T16:47:21.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.872+0000 7f7a72dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c0ffbe0 con 0x7f7a6c106020 2023-11-16T16:47:21.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.872+0000 7f7a72dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c1a0110 con 0x7f7a6c100de0 2023-11-16T16:47:21.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.872+0000 7f7a6bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c103700 0x7f7a6c0fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:21.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.872+0000 7f7a6bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c103700 0x7f7a6c0fed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43872/0 (socket says 172.21.15.44:43872) 2023-11-16T16:47:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.872+0000 7f7a6bfff700 1 -- 172.21.15.44:0/2590533369 learned_addr learned my addr 172.21.15.44:0/2590533369 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.872+0000 7f7a70b9b700 1 --2- 172.21.15.44:0/2590533369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a6c100de0 0x7f7a6c100750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.872+0000 7f7a7139c700 1 --2- 172.21.15.44:0/2590533369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a6c106020 0x7f7a6c0ff2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.872+0000 7f7a6bfff700 1 -- 172.21.15.44:0/2590533369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a6c100de0 msgr2=0x7f7a6c100750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.872+0000 7f7a6bfff700 1 --2- 172.21.15.44:0/2590533369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a6c100de0 0x7f7a6c100750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.872+0000 7f7a6bfff700 1 -- 172.21.15.44:0/2590533369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a6c106020 msgr2=0x7f7a6c0ff2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.872+0000 7f7a6bfff700 1 --2- 172.21.15.44:0/2590533369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a6c106020 0x7f7a6c0ff2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.872+0000 7f7a6bfff700 1 -- 172.21.15.44:0/2590533369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a600095f0 con 0x7f7a6c103700 2023-11-16T16:47:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.872+0000 7f7a7139c700 1 --2- 172.21.15.44:0/2590533369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a6c106020 0x7f7a6c0ff2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.873+0000 7f7a6bfff700 1 --2- 172.21.15.44:0/2590533369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c103700 0x7f7a6c0fed90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7a5800d980 tx=0x7f7a5800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.873+0000 7f7a69ffb700 1 -- 172.21.15.44:0/2590533369 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a58004220 con 0x7f7a6c103700 2023-11-16T16:47:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.873+0000 7f7a69ffb700 1 -- 172.21.15.44:0/2590533369 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a58007640 con 0x7f7a6c103700 2023-11-16T16:47:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.873+0000 7f7a72dff700 1 -- 172.21.15.44:0/2590533369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6c10bc70 con 0x7f7a6c103700 2023-11-16T16:47:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.873+0000 7f7a69ffb700 1 -- 172.21.15.44:0/2590533369 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a58005690 con 0x7f7a6c103700 2023-11-16T16:47:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.873+0000 7f7a72dff700 1 -- 172.21.15.44:0/2590533369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6c10bae0 con 0x7f7a6c103700 2023-11-16T16:47:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.874+0000 7f7a72dff700 1 -- 172.21.15.44:0/2590533369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6c0fc330 con 0x7f7a6c103700 2023-11-16T16:47:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.876+0000 7f7a69ffb700 1 -- 172.21.15.44:0/2590533369 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7a58004750 con 0x7f7a6c103700 2023-11-16T16:47:21.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.877+0000 7f7a69ffb700 1 --2- 172.21.15.44:0/2590533369 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7a5c0a11d0 0x7f7a5c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:21.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.878+0000 7f7a69ffb700 1 -- 172.21.15.44:0/2590533369 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f7a580cc8e0 con 0x7f7a6c103700 2023-11-16T16:47:21.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.878+0000 7f7a70b9b700 1 --2- 172.21.15.44:0/2590533369 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7a5c0a11d0 0x7f7a5c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:21.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.878+0000 7f7a70b9b700 1 --2- 172.21.15.44:0/2590533369 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7a5c0a11d0 0x7f7a5c0a3680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000b2e0 tx=0x7f7a60005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:21.882+0000 7f7a69ffb700 1 -- 172.21.15.44:0/2590533369 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7a58095e00 con 0x7f7a6c103700 2023-11-16T16:47:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:21 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:47:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:21 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:47:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:21 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:47:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:21 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T16:47:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:21 smithi176 ceph-mon[131630]: pgmap v272: 97 pgs: 21 creating+peering, 58 active+clean, 18 unknown; 578 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2023-11-16T16:47:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:21 smithi176 ceph-mon[131630]: osdmap e84: 12 total, 12 up, 12 in 2023-11-16T16:47:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:21 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:47:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:21 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:47:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:21 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:47:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:21 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T16:47:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:21 smithi077 ceph-mon[131204]: pgmap v272: 97 pgs: 21 creating+peering, 58 active+clean, 18 unknown; 578 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2023-11-16T16:47:21.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:21 smithi077 ceph-mon[131204]: osdmap e84: 12 total, 12 up, 12 in 2023-11-16T16:47:22.095 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.094+0000 7f7a72dff700 1 -- 172.21.15.44:0/2590533369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f7a6c02cfe0 con 0x7f7a6c103700 2023-11-16T16:47:22.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:21 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:47:22.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:21 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:47:22.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:21 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:47:22.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:21 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T16:47:22.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:21 smithi044 ceph-mon[128750]: pgmap v272: 97 pgs: 21 creating+peering, 58 active+clean, 18 unknown; 578 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2023-11-16T16:47:22.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:21 smithi044 ceph-mon[128750]: osdmap e84: 12 total, 12 up, 12 in 2023-11-16T16:47:22.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.637+0000 7f7a69ffb700 1 -- 172.21.15.44:0/2590533369 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v86) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7a58016020 con 0x7f7a6c103700 2023-11-16T16:47:22.644 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.643+0000 7f7a72dff700 1 -- 172.21.15.44:0/2590533369 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7a5c0a11d0 msgr2=0x7f7a5c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.643+0000 7f7a72dff700 1 --2- 172.21.15.44:0/2590533369 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7a5c0a11d0 0x7f7a5c0a3680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000b2e0 tx=0x7f7a60005fd0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.643+0000 7f7a72dff700 1 -- 172.21.15.44:0/2590533369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c103700 msgr2=0x7f7a6c0fed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.643+0000 7f7a72dff700 1 --2- 172.21.15.44:0/2590533369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c103700 0x7f7a6c0fed90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7a5800d980 tx=0x7f7a5800dc90 comp rx=0 tx=0).stop 2023-11-16T16:47:22.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.643+0000 7f7a72dff700 1 -- 172.21.15.44:0/2590533369 shutdown_connections 2023-11-16T16:47:22.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.643+0000 7f7a72dff700 1 --2- 172.21.15.44:0/2590533369 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7a5c0a11d0 0x7f7a5c0a3680 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.643+0000 7f7a72dff700 1 --2- 172.21.15.44:0/2590533369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a6c106020 0x7f7a6c0ff2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.643+0000 7f7a72dff700 1 --2- 172.21.15.44:0/2590533369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a6c100de0 0x7f7a6c100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.643+0000 7f7a72dff700 1 --2- 172.21.15.44:0/2590533369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c103700 0x7f7a6c0fed90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.643+0000 7f7a72dff700 1 -- 172.21.15.44:0/2590533369 >> 172.21.15.44:0/2590533369 conn(0x7f7a6c0fa7c0 msgr2=0x7f7a6c101aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:22.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.644+0000 7f7a72dff700 1 -- 172.21.15.44:0/2590533369 shutdown_connections 2023-11-16T16:47:22.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.644+0000 7f7a72dff700 1 -- 172.21.15.44:0/2590533369 wait complete. 2023-11-16T16:47:22.649 INFO:teuthology.orchestra.run.smithi044.stderr:set pool 4 allow_ec_overwrites to true 2023-11-16T16:47:22.677 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-11-16T16:47:22.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.886+0000 7f4f45dff700 1 -- 172.21.15.44:0/3269703002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f401002c0 msgr2=0x7f4f401006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.886+0000 7f4f45dff700 1 --2- 172.21.15.44:0/3269703002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f401002c0 0x7f4f401006d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4f30009990 tx=0x7f4f30009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.886+0000 7f4f45dff700 1 -- 172.21.15.44:0/3269703002 shutdown_connections 2023-11-16T16:47:22.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.886+0000 7f4f45dff700 1 --2- 172.21.15.44:0/3269703002 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f40101e40 0x7f4f40108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.886+0000 7f4f45dff700 1 --2- 172.21.15.44:0/3269703002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f401002c0 0x7f4f401006d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.886+0000 7f4f45dff700 1 --2- 172.21.15.44:0/3269703002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f401014b0 0x7f4f40101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.886+0000 7f4f45dff700 1 -- 172.21.15.44:0/3269703002 >> 172.21.15.44:0/3269703002 conn(0x7f4f400fb8a0 msgr2=0x7f4f400fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:22.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.886+0000 7f4f45dff700 1 -- 172.21.15.44:0/3269703002 shutdown_connections 2023-11-16T16:47:22.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.886+0000 7f4f45dff700 1 -- 172.21.15.44:0/3269703002 wait complete. 2023-11-16T16:47:22.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.887+0000 7f4f45dff700 1 Processor -- start 2023-11-16T16:47:22.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.887+0000 7f4f45dff700 1 -- start start 2023-11-16T16:47:22.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.887+0000 7f4f45dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f401002c0 0x7f4f400748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:22.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.887+0000 7f4f45dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f401014b0 0x7f4f40072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:22.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.887+0000 7f4f45dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f40101e40 0x7f4f40073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:22.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.887+0000 7f4f45dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f40073bc0 con 0x7f4f40101e40 2023-11-16T16:47:22.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.887+0000 7f4f45dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f40073d30 con 0x7f4f401014b0 2023-11-16T16:47:22.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.887+0000 7f4f45dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f401a00f0 con 0x7f4f401002c0 2023-11-16T16:47:22.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.888+0000 7f4f3ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f40101e40 0x7f4f40073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:22.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.888+0000 7f4f3f7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f401002c0 0x7f4f400748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:22.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.888+0000 7f4f3ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f40101e40 0x7f4f40073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43884/0 (socket says 172.21.15.44:43884) 2023-11-16T16:47:22.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.888+0000 7f4f3ffff700 1 -- 172.21.15.44:0/4156486178 learned_addr learned my addr 172.21.15.44:0/4156486178 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:22.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.888+0000 7f4f3effd700 1 --2- 172.21.15.44:0/4156486178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f401014b0 0x7f4f40072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:22.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.888+0000 7f4f3f7fe700 1 -- 172.21.15.44:0/4156486178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f401014b0 msgr2=0x7f4f40072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.888+0000 7f4f3f7fe700 1 --2- 172.21.15.44:0/4156486178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f401014b0 0x7f4f40072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.888+0000 7f4f3f7fe700 1 -- 172.21.15.44:0/4156486178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f40101e40 msgr2=0x7f4f40073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.888+0000 7f4f3f7fe700 1 --2- 172.21.15.44:0/4156486178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f40101e40 0x7f4f40073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.888+0000 7f4f3f7fe700 1 -- 172.21.15.44:0/4156486178 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f300095f0 con 0x7f4f401002c0 2023-11-16T16:47:22.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.888+0000 7f4f3ffff700 1 --2- 172.21.15.44:0/4156486178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f40101e40 0x7f4f40073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:22.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.888+0000 7f4f3f7fe700 1 --2- 172.21.15.44:0/4156486178 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f401002c0 0x7f4f400748a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000b620 tx=0x7f4f30004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:22.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.889+0000 7f4f3cff9700 1 -- 172.21.15.44:0/4156486178 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3001d070 con 0x7f4f401002c0 2023-11-16T16:47:22.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.889+0000 7f4f45dff700 1 -- 172.21.15.44:0/4156486178 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4010bc90 con 0x7f4f401002c0 2023-11-16T16:47:22.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.889+0000 7f4f3cff9700 1 -- 172.21.15.44:0/4156486178 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f30021410 con 0x7f4f401002c0 2023-11-16T16:47:22.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.889+0000 7f4f3cff9700 1 -- 172.21.15.44:0/4156486178 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f30017650 con 0x7f4f401002c0 2023-11-16T16:47:22.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.889+0000 7f4f45dff700 1 -- 172.21.15.44:0/4156486178 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4010bb00 con 0x7f4f401002c0 2023-11-16T16:47:22.894 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.890+0000 7f4f45dff700 1 -- 172.21.15.44:0/4156486178 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f4010be20 con 0x7f4f401002c0 2023-11-16T16:47:22.894 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.892+0000 7f4f3cff9700 1 -- 172.21.15.44:0/4156486178 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4f300217f0 con 0x7f4f401002c0 2023-11-16T16:47:22.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:22 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-11-16T16:47:22.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:22 smithi176 ceph-mon[131630]: osdmap e85: 12 total, 12 up, 12 in 2023-11-16T16:47:22.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:22 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2590533369' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-11-16T16:47:22.895 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.893+0000 7f4f3cff9700 1 --2- 172.21.15.44:0/4156486178 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4f280a1150 0x7f4f280a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:22.895 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.894+0000 7f4f3cff9700 1 -- 172.21.15.44:0/4156486178 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f4f300ce400 con 0x7f4f401002c0 2023-11-16T16:47:22.895 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.894+0000 7f4f3effd700 1 --2- 172.21.15.44:0/4156486178 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4f280a1150 0x7f4f280a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:22.896 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.894+0000 7f4f3effd700 1 --2- 172.21.15.44:0/4156486178 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4f280a1150 0x7f4f280a3600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009c60 tx=0x7f4f34009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:22.897 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:22.896+0000 7f4f3cff9700 1 -- 172.21.15.44:0/4156486178 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4f300979b0 con 0x7f4f401002c0 2023-11-16T16:47:22.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:22 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-11-16T16:47:22.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:22 smithi077 ceph-mon[131204]: osdmap e85: 12 total, 12 up, 12 in 2023-11-16T16:47:22.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:22 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2590533369' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-11-16T16:47:23.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:22 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-11-16T16:47:23.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:22 smithi044 ceph-mon[128750]: osdmap e85: 12 total, 12 up, 12 in 2023-11-16T16:47:23.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:22 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2590533369' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-11-16T16:47:23.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.156+0000 7f4f45dff700 1 -- 172.21.15.44:0/4156486178 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f4f40107850 con 0x7f4f401002c0 2023-11-16T16:47:23.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.165+0000 7f4f3cff9700 1 -- 172.21.15.44:0/4156486178 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7f4f30021e60 con 0x7f4f401002c0 2023-11-16T16:47:23.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.167+0000 7f4f45dff700 1 -- 172.21.15.44:0/4156486178 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4f280a1150 msgr2=0x7f4f280a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.167+0000 7f4f45dff700 1 --2- 172.21.15.44:0/4156486178 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4f280a1150 0x7f4f280a3600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009c60 tx=0x7f4f34009260 comp rx=0 tx=0).stop 2023-11-16T16:47:23.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.168+0000 7f4f45dff700 1 -- 172.21.15.44:0/4156486178 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f401002c0 msgr2=0x7f4f400748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.168+0000 7f4f45dff700 1 --2- 172.21.15.44:0/4156486178 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f401002c0 0x7f4f400748a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000b620 tx=0x7f4f30004730 comp rx=0 tx=0).stop 2023-11-16T16:47:23.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.168+0000 7f4f45dff700 1 -- 172.21.15.44:0/4156486178 shutdown_connections 2023-11-16T16:47:23.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.168+0000 7f4f45dff700 1 --2- 172.21.15.44:0/4156486178 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4f280a1150 0x7f4f280a3600 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.168+0000 7f4f45dff700 1 --2- 172.21.15.44:0/4156486178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4f40101e40 0x7f4f40073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.168+0000 7f4f45dff700 1 --2- 172.21.15.44:0/4156486178 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f401002c0 0x7f4f400748a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.168+0000 7f4f45dff700 1 --2- 172.21.15.44:0/4156486178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f401014b0 0x7f4f40072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.168+0000 7f4f45dff700 1 -- 172.21.15.44:0/4156486178 >> 172.21.15.44:0/4156486178 conn(0x7f4f400fb8a0 msgr2=0x7f4f40104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:23.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.168+0000 7f4f45dff700 1 -- 172.21.15.44:0/4156486178 shutdown_connections 2023-11-16T16:47:23.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.168+0000 7f4f45dff700 1 -- 172.21.15.44:0/4156486178 wait complete. 2023-11-16T16:47:23.171 INFO:teuthology.orchestra.run.smithi044.stderr:added data pool 4 to fsmap 2023-11-16T16:47:23.188 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:23.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.390+0000 7feed05ff700 1 -- 172.21.15.44:0/91343756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec8100290 msgr2=0x7feec81006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.390+0000 7feed05ff700 1 --2- 172.21.15.44:0/91343756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec8100290 0x7feec81006a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7feec4009990 tx=0x7feec4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.390+0000 7feed05ff700 1 -- 172.21.15.44:0/91343756 shutdown_connections 2023-11-16T16:47:23.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.390+0000 7feed05ff700 1 --2- 172.21.15.44:0/91343756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec8101e10 0x7feec81086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.390+0000 7feed05ff700 1 --2- 172.21.15.44:0/91343756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec8100290 0x7feec81006a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.390+0000 7feed05ff700 1 --2- 172.21.15.44:0/91343756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feec8101480 0x7feec81018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.390+0000 7feed05ff700 1 -- 172.21.15.44:0/91343756 >> 172.21.15.44:0/91343756 conn(0x7feec80fb870 msgr2=0x7feec80fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:23.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.390+0000 7feed05ff700 1 -- 172.21.15.44:0/91343756 shutdown_connections 2023-11-16T16:47:23.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.390+0000 7feed05ff700 1 -- 172.21.15.44:0/91343756 wait complete. 2023-11-16T16:47:23.394 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.391+0000 7feed05ff700 1 Processor -- start 2023-11-16T16:47:23.394 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.391+0000 7feed05ff700 1 -- start start 2023-11-16T16:47:23.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.391+0000 7feed05ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feec8100290 0x7feec81981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:23.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.391+0000 7feed05ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec8101480 0x7feec8198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:23.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.391+0000 7feed05ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec8101e10 0x7feec819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:23.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.391+0000 7feed05ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec819fd40 con 0x7feec8101480 2023-11-16T16:47:23.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.391+0000 7feed05ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec819fe80 con 0x7feec8101e10 2023-11-16T16:47:23.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.391+0000 7feed05ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec81a0010 con 0x7feec8100290 2023-11-16T16:47:23.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.392+0000 7feecdb9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec8101480 0x7feec8198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:23.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.392+0000 7feecdb9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec8101480 0x7feec8198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43898/0 (socket says 172.21.15.44:43898) 2023-11-16T16:47:23.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.392+0000 7feecdb9a700 1 -- 172.21.15.44:0/1297960570 learned_addr learned my addr 172.21.15.44:0/1297960570 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:23.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.392+0000 7feeceb9c700 1 --2- 172.21.15.44:0/1297960570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec8101e10 0x7feec819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:23.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.392+0000 7feece39b700 1 --2- 172.21.15.44:0/1297960570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feec8100290 0x7feec81981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:23.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.392+0000 7feecdb9a700 1 -- 172.21.15.44:0/1297960570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feec8100290 msgr2=0x7feec81981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.392+0000 7feecdb9a700 1 --2- 172.21.15.44:0/1297960570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feec8100290 0x7feec81981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.392+0000 7feecdb9a700 1 -- 172.21.15.44:0/1297960570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec8101e10 msgr2=0x7feec819f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.392+0000 7feecdb9a700 1 --2- 172.21.15.44:0/1297960570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec8101e10 0x7feec819f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.392+0000 7feecdb9a700 1 -- 172.21.15.44:0/1297960570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec40095f0 con 0x7feec8101480 2023-11-16T16:47:23.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.392+0000 7feecdb9a700 1 --2- 172.21.15.44:0/1297960570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec8101480 0x7feec8198700 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7feeb800e9a0 tx=0x7feeb800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:23.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.393+0000 7feebf7fe700 1 -- 172.21.15.44:0/1297960570 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb8009790 con 0x7feec8101480 2023-11-16T16:47:23.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.393+0000 7feebf7fe700 1 -- 172.21.15.44:0/1297960570 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feeb8010430 con 0x7feec8101480 2023-11-16T16:47:23.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.393+0000 7feed05ff700 1 -- 172.21.15.44:0/1297960570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec810bc40 con 0x7feec8101480 2023-11-16T16:47:23.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.393+0000 7feebf7fe700 1 -- 172.21.15.44:0/1297960570 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb8018680 con 0x7feec8101480 2023-11-16T16:47:23.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.393+0000 7feed05ff700 1 -- 172.21.15.44:0/1297960570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec810bab0 con 0x7feec8101480 2023-11-16T16:47:23.397 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.394+0000 7feed05ff700 1 -- 172.21.15.44:0/1297960570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec81928d0 con 0x7feec8101480 2023-11-16T16:47:23.397 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.396+0000 7feebf7fe700 1 -- 172.21.15.44:0/1297960570 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7feeb80105a0 con 0x7feec8101480 2023-11-16T16:47:23.399 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.397+0000 7feebf7fe700 1 --2- 172.21.15.44:0/1297960570 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7feeb40a11d0 0x7feeb40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:23.399 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.397+0000 7feebf7fe700 1 -- 172.21.15.44:0/1297960570 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7feeb8014070 con 0x7feec8101480 2023-11-16T16:47:23.399 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.398+0000 7feece39b700 1 --2- 172.21.15.44:0/1297960570 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7feeb40a11d0 0x7feeb40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:23.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.398+0000 7feece39b700 1 --2- 172.21.15.44:0/1297960570 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7feeb40a11d0 0x7feeb40a3680 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feec400b2e0 tx=0x7feec4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:23.403 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.402+0000 7feebf7fe700 1 -- 172.21.15.44:0/1297960570 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7feeb8051730 con 0x7feec8101480 2023-11-16T16:47:23.476 INFO:journalctl@ceph.mds.b.smithi077.stdout:Nov 16 16:47:23 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-b[156523]: 2023-11-16T16:47:23.066+0000 7ff7f12ca700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-16T16:47:23.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.658+0000 7feed05ff700 1 -- 172.21.15.44:0/1297960570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feec802cfe0 con 0x7feec8101480 2023-11-16T16:47:23.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.659+0000 7feebf7fe700 1 -- 172.21.15.44:0/1297960570 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10732 (secure 0 0 0) 0x7feeb80961d0 con 0x7feec8101480 2023-11-16T16:47:23.661 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:23.661 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:23.159198+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-11-16T16:47:23.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.661+0000 7feed05ff700 1 -- 172.21.15.44:0/1297960570 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7feeb40a11d0 msgr2=0x7feeb40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.661+0000 7feed05ff700 1 --2- 172.21.15.44:0/1297960570 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7feeb40a11d0 0x7feeb40a3680 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feec400b2e0 tx=0x7feec4005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.661+0000 7feed05ff700 1 -- 172.21.15.44:0/1297960570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec8101480 msgr2=0x7feec8198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.661+0000 7feed05ff700 1 --2- 172.21.15.44:0/1297960570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec8101480 0x7feec8198700 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7feeb800e9a0 tx=0x7feeb800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.661+0000 7feed05ff700 1 -- 172.21.15.44:0/1297960570 shutdown_connections 2023-11-16T16:47:23.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.661+0000 7feed05ff700 1 --2- 172.21.15.44:0/1297960570 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7feeb40a11d0 0x7feeb40a3680 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.664 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.661+0000 7feed05ff700 1 --2- 172.21.15.44:0/1297960570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec8101e10 0x7feec819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.664 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.661+0000 7feed05ff700 1 --2- 172.21.15.44:0/1297960570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feec8100290 0x7feec81981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.664 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.661+0000 7feed05ff700 1 --2- 172.21.15.44:0/1297960570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec8101480 0x7feec8198700 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.664 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.661+0000 7feed05ff700 1 -- 172.21.15.44:0/1297960570 >> 172.21.15.44:0/1297960570 conn(0x7feec80fb870 msgr2=0x7feec8106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:23.664 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.661+0000 7feed05ff700 1 -- 172.21.15.44:0/1297960570 shutdown_connections 2023-11-16T16:47:23.664 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.662+0000 7feed05ff700 1 -- 172.21.15.44:0/1297960570 wait complete. 2023-11-16T16:47:23.664 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 17 2023-11-16T16:47:23.682 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:23.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.884+0000 7f11447ff700 1 -- 172.21.15.44:0/2568279634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f113c1036b0 msgr2=0x7f113c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.884+0000 7f11447ff700 1 --2- 172.21.15.44:0/2568279634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f113c1036b0 0x7f113c105a90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f112c009990 tx=0x7f112c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.884+0000 7f11447ff700 1 -- 172.21.15.44:0/2568279634 shutdown_connections 2023-11-16T16:47:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.884+0000 7f11447ff700 1 --2- 172.21.15.44:0/2568279634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f113c105fd0 0x7f113c1083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.884+0000 7f11447ff700 1 --2- 172.21.15.44:0/2568279634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c100d90 0x7f113c103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.884+0000 7f11447ff700 1 --2- 172.21.15.44:0/2568279634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f113c1036b0 0x7f113c105a90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.884+0000 7f11447ff700 1 -- 172.21.15.44:0/2568279634 >> 172.21.15.44:0/2568279634 conn(0x7f113c0fa750 msgr2=0x7f113c0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.885+0000 7f11447ff700 1 -- 172.21.15.44:0/2568279634 shutdown_connections 2023-11-16T16:47:23.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.885+0000 7f11447ff700 1 -- 172.21.15.44:0/2568279634 wait complete. 2023-11-16T16:47:23.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.886+0000 7f11447ff700 1 Processor -- start 2023-11-16T16:47:23.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.886+0000 7f11447ff700 1 -- start start 2023-11-16T16:47:23.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.886+0000 7f11447ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c100d90 0x7f113c100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:23.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.886+0000 7f11447ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f113c1036b0 0x7f113c0fed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:23.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.886+0000 7f11447ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f113c105fd0 0x7f113c0ff280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:23.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.886+0000 7f11447ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c0ffa20 con 0x7f113c1036b0 2023-11-16T16:47:23.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.886+0000 7f11447ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c0ffb90 con 0x7f113c100d90 2023-11-16T16:47:23.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.886+0000 7f11447ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c1a0020 con 0x7f113c105fd0 2023-11-16T16:47:23.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.886+0000 7f1141d9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f113c1036b0 0x7f113c0fed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:23.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.886+0000 7f114259b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c100d90 0x7f113c100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:23.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.886+0000 7f114259b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c100d90 0x7f113c100700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:46482/0 (socket says 172.21.15.44:46482) 2023-11-16T16:47:23.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.886+0000 7f1141d9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f113c1036b0 0x7f113c0fed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43916/0 (socket says 172.21.15.44:43916) 2023-11-16T16:47:23.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.886+0000 7f114259b700 1 -- 172.21.15.44:0/1722024369 learned_addr learned my addr 172.21.15.44:0/1722024369 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:23.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.886+0000 7f1142d9c700 1 --2- 172.21.15.44:0/1722024369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f113c105fd0 0x7f113c0ff280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:23.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.887+0000 7f1141d9a700 1 -- 172.21.15.44:0/1722024369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f113c105fd0 msgr2=0x7f113c0ff280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.887+0000 7f1141d9a700 1 --2- 172.21.15.44:0/1722024369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f113c105fd0 0x7f113c0ff280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.887+0000 7f1141d9a700 1 -- 172.21.15.44:0/1722024369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c100d90 msgr2=0x7f113c100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.887+0000 7f1141d9a700 1 --2- 172.21.15.44:0/1722024369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c100d90 0x7f113c100700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.887+0000 7f1141d9a700 1 -- 172.21.15.44:0/1722024369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c0095f0 con 0x7f113c1036b0 2023-11-16T16:47:23.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.887+0000 7f1141d9a700 1 --2- 172.21.15.44:0/1722024369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f113c1036b0 0x7f113c0fed40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f112c00b680 tx=0x7f112c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:23.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.887+0000 7f11337fe700 1 -- 172.21.15.44:0/1722024369 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c01d070 con 0x7f113c1036b0 2023-11-16T16:47:23.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.887+0000 7f11337fe700 1 -- 172.21.15.44:0/1722024369 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f112c021410 con 0x7f113c1036b0 2023-11-16T16:47:23.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.887+0000 7f11447ff700 1 -- 172.21.15.44:0/1722024369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113c10bc60 con 0x7f113c1036b0 2023-11-16T16:47:23.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.887+0000 7f11337fe700 1 -- 172.21.15.44:0/1722024369 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c017650 con 0x7f113c1036b0 2023-11-16T16:47:23.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.887+0000 7f11447ff700 1 -- 172.21.15.44:0/1722024369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113c10bad0 con 0x7f113c1036b0 2023-11-16T16:47:23.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.888+0000 7f11447ff700 1 -- 172.21.15.44:0/1722024369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113c0fc2c0 con 0x7f113c1036b0 2023-11-16T16:47:23.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.890+0000 7f11337fe700 1 -- 172.21.15.44:0/1722024369 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f112c021810 con 0x7f113c1036b0 2023-11-16T16:47:23.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.892+0000 7f11337fe700 1 --2- 172.21.15.44:0/1722024369 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f11280a11d0 0x7f11280a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:23.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.892+0000 7f11337fe700 1 -- 172.21.15.44:0/1722024369 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f112c0ce4e0 con 0x7f113c1036b0 2023-11-16T16:47:23.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.892+0000 7f114259b700 1 --2- 172.21.15.44:0/1722024369 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f11280a11d0 0x7f11280a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:23.894 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.892+0000 7f114259b700 1 --2- 172.21.15.44:0/1722024369 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f11280a11d0 0x7f11280a3680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1138005950 tx=0x7f1138009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:23 smithi176 ceph-mon[131630]: pgmap v275: 129 pgs: 21 creating+peering, 85 active+clean, 23 unknown; 579 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 9 op/s 2023-11-16T16:47:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:23 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2590533369' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-11-16T16:47:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:23 smithi176 ceph-mon[131630]: osdmap e86: 12 total, 12 up, 12 in 2023-11-16T16:47:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:23 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/4156486178' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-11-16T16:47:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:23 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-11-16T16:47:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:23 smithi176 ceph-mon[131630]: osdmap e87: 12 total, 12 up, 12 in 2023-11-16T16:47:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:23 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-11-16T16:47:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:23 smithi176 ceph-mon[131630]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:23.896 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:23.894+0000 7f11337fe700 1 -- 172.21.15.44:0/1722024369 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f112c097a70 con 0x7f113c1036b0 2023-11-16T16:47:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:23 smithi077 ceph-mon[131204]: pgmap v275: 129 pgs: 21 creating+peering, 85 active+clean, 23 unknown; 579 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 9 op/s 2023-11-16T16:47:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:23 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2590533369' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-11-16T16:47:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:23 smithi077 ceph-mon[131204]: osdmap e86: 12 total, 12 up, 12 in 2023-11-16T16:47:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:23 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/4156486178' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-11-16T16:47:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:23 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-11-16T16:47:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:23 smithi077 ceph-mon[131204]: osdmap e87: 12 total, 12 up, 12 in 2023-11-16T16:47:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:23 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-11-16T16:47:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:23 smithi077 ceph-mon[131204]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:24.106 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.104+0000 7f11447ff700 1 -- 172.21.15.44:0/1722024369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f113c02cfe0 con 0x7f113c1036b0 2023-11-16T16:47:24.107 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.106+0000 7f11337fe700 1 -- 172.21.15.44:0/1722024369 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23925 (secure 0 0 0) 0x7f112c09b090 con 0x7f113c1036b0 2023-11-16T16:47:24.108 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:24.108 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":87,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:47:23.158961+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:24.110 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.108+0000 7f11447ff700 1 -- 172.21.15.44:0/1722024369 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f11280a11d0 msgr2=0x7f11280a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:24.110 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.108+0000 7f11447ff700 1 --2- 172.21.15.44:0/1722024369 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f11280a11d0 0x7f11280a3680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1138005950 tx=0x7f1138009260 comp rx=0 tx=0).stop 2023-11-16T16:47:24.110 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.108+0000 7f11447ff700 1 -- 172.21.15.44:0/1722024369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f113c1036b0 msgr2=0x7f113c0fed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:24.110 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.108+0000 7f11447ff700 1 --2- 172.21.15.44:0/1722024369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f113c1036b0 0x7f113c0fed40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f112c00b680 tx=0x7f112c009580 comp rx=0 tx=0).stop 2023-11-16T16:47:24.110 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.108+0000 7f11447ff700 1 -- 172.21.15.44:0/1722024369 shutdown_connections 2023-11-16T16:47:24.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.108+0000 7f11447ff700 1 --2- 172.21.15.44:0/1722024369 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f11280a11d0 0x7f11280a3680 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.108+0000 7f11447ff700 1 --2- 172.21.15.44:0/1722024369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f113c105fd0 0x7f113c0ff280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.108+0000 7f11447ff700 1 --2- 172.21.15.44:0/1722024369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c100d90 0x7f113c100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.108+0000 7f11447ff700 1 --2- 172.21.15.44:0/1722024369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f113c1036b0 0x7f113c0fed40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.108+0000 7f11447ff700 1 -- 172.21.15.44:0/1722024369 >> 172.21.15.44:0/1722024369 conn(0x7f113c0fa750 msgr2=0x7f113c1019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:24.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.108+0000 7f11447ff700 1 -- 172.21.15.44:0/1722024369 shutdown_connections 2023-11-16T16:47:24.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.108+0000 7f11447ff700 1 -- 172.21.15.44:0/1722024369 wait complete. 2023-11-16T16:47:24.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:23 smithi044 ceph-mon[128750]: pgmap v275: 129 pgs: 21 creating+peering, 85 active+clean, 23 unknown; 579 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 9 op/s 2023-11-16T16:47:24.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:23 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2590533369' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-11-16T16:47:24.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:23 smithi044 ceph-mon[128750]: osdmap e86: 12 total, 12 up, 12 in 2023-11-16T16:47:24.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:23 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/4156486178' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-11-16T16:47:24.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:23 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-11-16T16:47:24.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:23 smithi044 ceph-mon[128750]: osdmap e87: 12 total, 12 up, 12 in 2023-11-16T16:47:24.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:23 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-11-16T16:47:24.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:23 smithi044 ceph-mon[128750]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:24.130 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:24.334 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.333+0000 7f04097ff700 1 -- 172.21.15.44:0/4090356969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0404100290 msgr2=0x7f04041006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:24.335 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.333+0000 7f04097ff700 1 --2- 172.21.15.44:0/4090356969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0404100290 0x7f04041006a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f03f4009990 tx=0x7f03f4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.335 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.333+0000 7f04097ff700 1 -- 172.21.15.44:0/4090356969 shutdown_connections 2023-11-16T16:47:24.335 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.333+0000 7f04097ff700 1 --2- 172.21.15.44:0/4090356969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0404101e10 0x7f04041086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.335 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.333+0000 7f04097ff700 1 --2- 172.21.15.44:0/4090356969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0404100290 0x7f04041006a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.335 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.333+0000 7f04097ff700 1 --2- 172.21.15.44:0/4090356969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0404101480 0x7f04041018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.335 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.333+0000 7f04097ff700 1 -- 172.21.15.44:0/4090356969 >> 172.21.15.44:0/4090356969 conn(0x7f04040fb870 msgr2=0x7f04040fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:24.335 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.333+0000 7f04097ff700 1 -- 172.21.15.44:0/4090356969 shutdown_connections 2023-11-16T16:47:24.335 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.333+0000 7f04097ff700 1 -- 172.21.15.44:0/4090356969 wait complete. 2023-11-16T16:47:24.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.334+0000 7f04097ff700 1 Processor -- start 2023-11-16T16:47:24.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.334+0000 7f04097ff700 1 -- start start 2023-11-16T16:47:24.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.334+0000 7f04097ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0404100290 0x7f04041981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:24.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.334+0000 7f04097ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0404101480 0x7f0404198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:24.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.334+0000 7f04097ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0404101e10 0x7f040419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:24.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.334+0000 7f04097ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040419fd50 con 0x7f0404101e10 2023-11-16T16:47:24.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.334+0000 7f04097ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040419fe90 con 0x7f0404100290 2023-11-16T16:47:24.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.334+0000 7f04097ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04041a0020 con 0x7f0404101480 2023-11-16T16:47:24.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.335+0000 7f04037fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0404101e10 0x7f040419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:24.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.335+0000 7f03fbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0404101480 0x7f0404198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:24.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.335+0000 7f03fbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0404101480 0x7f0404198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:41404/0 (socket says 172.21.15.44:41404) 2023-11-16T16:47:24.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.335+0000 7f03fbfff700 1 -- 172.21.15.44:0/4027541230 learned_addr learned my addr 172.21.15.44:0/4027541230 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:24.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.335+0000 7f0402ffd700 1 --2- 172.21.15.44:0/4027541230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0404100290 0x7f04041981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:24.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.335+0000 7f03fbfff700 1 -- 172.21.15.44:0/4027541230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0404100290 msgr2=0x7f04041981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:24.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.335+0000 7f03fbfff700 1 --2- 172.21.15.44:0/4027541230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0404100290 0x7f04041981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.335+0000 7f03fbfff700 1 -- 172.21.15.44:0/4027541230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0404101e10 msgr2=0x7f040419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:24.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.335+0000 7f03fbfff700 1 --2- 172.21.15.44:0/4027541230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0404101e10 0x7f040419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.335+0000 7f03fbfff700 1 -- 172.21.15.44:0/4027541230 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f40095f0 con 0x7f0404101480 2023-11-16T16:47:24.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.335+0000 7f04037fe700 1 --2- 172.21.15.44:0/4027541230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0404101e10 0x7f040419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:24.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.335+0000 7f03fbfff700 1 --2- 172.21.15.44:0/4027541230 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0404101480 0x7f0404198710 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00e9a0 tx=0x7f03ec00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:24.339 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.336+0000 7f0400ff9700 1 -- 172.21.15.44:0/4027541230 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec009790 con 0x7f0404101480 2023-11-16T16:47:24.339 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.336+0000 7f0400ff9700 1 -- 172.21.15.44:0/4027541230 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03ec010430 con 0x7f0404101480 2023-11-16T16:47:24.339 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.336+0000 7f04097ff700 1 -- 172.21.15.44:0/4027541230 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040410bc40 con 0x7f0404101480 2023-11-16T16:47:24.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.336+0000 7f0400ff9700 1 -- 172.21.15.44:0/4027541230 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec018680 con 0x7f0404101480 2023-11-16T16:47:24.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.336+0000 7f04097ff700 1 -- 172.21.15.44:0/4027541230 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f040410bab0 con 0x7f0404101480 2023-11-16T16:47:24.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.337+0000 7f04097ff700 1 -- 172.21.15.44:0/4027541230 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04041928d0 con 0x7f0404101480 2023-11-16T16:47:24.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.339+0000 7f0400ff9700 1 -- 172.21.15.44:0/4027541230 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f03ec0105a0 con 0x7f0404101480 2023-11-16T16:47:24.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.340+0000 7f0400ff9700 1 --2- 172.21.15.44:0/4027541230 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f03f00a11d0 0x7f03f00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:24.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.340+0000 7f0400ff9700 1 -- 172.21.15.44:0/4027541230 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f03ec014070 con 0x7f0404101480 2023-11-16T16:47:24.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.341+0000 7f0402ffd700 1 --2- 172.21.15.44:0/4027541230 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f03f00a11d0 0x7f03f00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:24.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.341+0000 7f0402ffd700 1 --2- 172.21.15.44:0/4027541230 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f03f00a11d0 0x7f03f00a3680 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f03f400b2e0 tx=0x7f03f4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:24.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.342+0000 7f0400ff9700 1 -- 172.21.15.44:0/4027541230 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f03ec0963d0 con 0x7f0404101480 2023-11-16T16:47:24.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.555+0000 7f04097ff700 1 -- 172.21.15.44:0/4027541230 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f040402cfe0 con 0x7f0404101480 2023-11-16T16:47:24.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.557+0000 7f0400ff9700 1 -- 172.21.15.44:0/4027541230 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23925 (secure 0 0 0) 0x7f03ec026020 con 0x7f0404101480 2023-11-16T16:47:24.559 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:24.559 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":87,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:47:23.158961+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:24.561 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.559+0000 7f04097ff700 1 -- 172.21.15.44:0/4027541230 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f03f00a11d0 msgr2=0x7f03f00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:24.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.559+0000 7f04097ff700 1 --2- 172.21.15.44:0/4027541230 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f03f00a11d0 0x7f03f00a3680 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f03f400b2e0 tx=0x7f03f4005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.560+0000 7f04097ff700 1 -- 172.21.15.44:0/4027541230 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0404101480 msgr2=0x7f0404198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:24.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.560+0000 7f04097ff700 1 --2- 172.21.15.44:0/4027541230 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0404101480 0x7f0404198710 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00e9a0 tx=0x7f03ec00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.560+0000 7f04097ff700 1 -- 172.21.15.44:0/4027541230 shutdown_connections 2023-11-16T16:47:24.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.560+0000 7f04097ff700 1 --2- 172.21.15.44:0/4027541230 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f03f00a11d0 0x7f03f00a3680 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.560+0000 7f04097ff700 1 --2- 172.21.15.44:0/4027541230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0404101e10 0x7f040419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.560+0000 7f04097ff700 1 --2- 172.21.15.44:0/4027541230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0404100290 0x7f04041981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.560+0000 7f04097ff700 1 --2- 172.21.15.44:0/4027541230 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0404101480 0x7f0404198710 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.560+0000 7f04097ff700 1 -- 172.21.15.44:0/4027541230 >> 172.21.15.44:0/4027541230 conn(0x7f04040fb870 msgr2=0x7f0404106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:24.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.560+0000 7f04097ff700 1 -- 172.21.15.44:0/4027541230 shutdown_connections 2023-11-16T16:47:24.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.560+0000 7f04097ff700 1 -- 172.21.15.44:0/4027541230 wait complete. 2023-11-16T16:47:24.593 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:24.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.796+0000 7f0b34bff700 1 -- 172.21.15.44:0/936540999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b30100290 msgr2=0x7f0b301006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:24.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.796+0000 7f0b34bff700 1 --2- 172.21.15.44:0/936540999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b30100290 0x7f0b301006a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0b20009990 tx=0x7f0b20009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.796+0000 7f0b34bff700 1 -- 172.21.15.44:0/936540999 shutdown_connections 2023-11-16T16:47:24.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.796+0000 7f0b34bff700 1 --2- 172.21.15.44:0/936540999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b30101e10 0x7f0b301086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.796+0000 7f0b34bff700 1 --2- 172.21.15.44:0/936540999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b30100290 0x7f0b301006a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.796+0000 7f0b34bff700 1 --2- 172.21.15.44:0/936540999 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b30101480 0x7f0b301018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.796+0000 7f0b34bff700 1 -- 172.21.15.44:0/936540999 >> 172.21.15.44:0/936540999 conn(0x7f0b300fb870 msgr2=0x7f0b300fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:24.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.797+0000 7f0b34bff700 1 -- 172.21.15.44:0/936540999 shutdown_connections 2023-11-16T16:47:24.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.797+0000 7f0b34bff700 1 -- 172.21.15.44:0/936540999 wait complete. 2023-11-16T16:47:24.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.797+0000 7f0b34bff700 1 Processor -- start 2023-11-16T16:47:24.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.798+0000 7f0b34bff700 1 -- start start 2023-11-16T16:47:24.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.798+0000 7f0b34bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b30100290 0x7f0b300748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:24.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.798+0000 7f0b34bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b30101480 0x7f0b30072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:24.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.798+0000 7f0b34bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b30101e10 0x7f0b30073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:24.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.798+0000 7f0b34bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b30073bc0 con 0x7f0b30101480 2023-11-16T16:47:24.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.798+0000 7f0b34bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b30073d30 con 0x7f0b30100290 2023-11-16T16:47:24.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.798+0000 7f0b34bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b301a0060 con 0x7f0b30101e10 2023-11-16T16:47:24.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.798+0000 7f0b2e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b30100290 0x7f0b300748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:24.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.798+0000 7f0b2dd9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b30101480 0x7f0b30072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:24.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.798+0000 7f0b2e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b30100290 0x7f0b300748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:46528/0 (socket says 172.21.15.44:46528) 2023-11-16T16:47:24.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.798+0000 7f0b2e59c700 1 -- 172.21.15.44:0/3837644872 learned_addr learned my addr 172.21.15.44:0/3837644872 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:24.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.798+0000 7f0b2ed9d700 1 --2- 172.21.15.44:0/3837644872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b30101e10 0x7f0b30073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:24.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.799+0000 7f0b2e59c700 1 -- 172.21.15.44:0/3837644872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b30101e10 msgr2=0x7f0b30073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:24.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.799+0000 7f0b2e59c700 1 --2- 172.21.15.44:0/3837644872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b30101e10 0x7f0b30073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.799+0000 7f0b2e59c700 1 -- 172.21.15.44:0/3837644872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b30101480 msgr2=0x7f0b30072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:24.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.799+0000 7f0b2e59c700 1 --2- 172.21.15.44:0/3837644872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b30101480 0x7f0b30072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.799+0000 7f0b2e59c700 1 -- 172.21.15.44:0/3837644872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b200095f0 con 0x7f0b30100290 2023-11-16T16:47:24.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.799+0000 7f0b2dd9b700 1 --2- 172.21.15.44:0/3837644872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b30101480 0x7f0b30072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:24.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.799+0000 7f0b2e59c700 1 --2- 172.21.15.44:0/3837644872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b30100290 0x7f0b300748a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0b20004730 tx=0x7f0b20004760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:24.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.799+0000 7f0b1f7fe700 1 -- 172.21.15.44:0/3837644872 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2001d070 con 0x7f0b30100290 2023-11-16T16:47:24.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.799+0000 7f0b34bff700 1 -- 172.21.15.44:0/3837644872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3010bc40 con 0x7f0b30100290 2023-11-16T16:47:24.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.799+0000 7f0b34bff700 1 -- 172.21.15.44:0/3837644872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3010bab0 con 0x7f0b30100290 2023-11-16T16:47:24.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.799+0000 7f0b1f7fe700 1 -- 172.21.15.44:0/3837644872 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b20021410 con 0x7f0b30100290 2023-11-16T16:47:24.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.799+0000 7f0b1f7fe700 1 -- 172.21.15.44:0/3837644872 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20017650 con 0x7f0b30100290 2023-11-16T16:47:24.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.800+0000 7f0b34bff700 1 -- 172.21.15.44:0/3837644872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b10005310 con 0x7f0b30100290 2023-11-16T16:47:24.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.802+0000 7f0b1f7fe700 1 -- 172.21.15.44:0/3837644872 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0b2000fb00 con 0x7f0b30100290 2023-11-16T16:47:24.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.803+0000 7f0b1f7fe700 1 --2- 172.21.15.44:0/3837644872 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0b180a1150 0x7f0b180a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:24.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.804+0000 7f0b1f7fe700 1 -- 172.21.15.44:0/3837644872 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0b200cdda0 con 0x7f0b30100290 2023-11-16T16:47:24.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.804+0000 7f0b2dd9b700 1 --2- 172.21.15.44:0/3837644872 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0b180a1150 0x7f0b180a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:24.806 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.804+0000 7f0b2dd9b700 1 --2- 172.21.15.44:0/3837644872 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0b180a1150 0x7f0b180a3600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0b24011490 tx=0x7f0b24009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:24.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:24.806+0000 7f0b1f7fe700 1 -- 172.21.15.44:0/3837644872 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0b20097330 con 0x7f0b30100290 2023-11-16T16:47:24.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:24 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1297960570' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:24.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:24 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1722024369' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:24.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:24 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/4027541230' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:24.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:24 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1297960570' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:24.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:24 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1722024369' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:24.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:24 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/4027541230' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:24.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:24 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1297960570' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:24.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:24 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1722024369' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:24.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:24 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/4027541230' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:25.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.061+0000 7f0b34bff700 1 -- 172.21.15.44:0/3837644872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b10005180 con 0x7f0b30100290 2023-11-16T16:47:25.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.062+0000 7f0b1f7fe700 1 -- 172.21.15.44:0/3837644872 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10732 (secure 0 0 0) 0x7f0b2009a950 con 0x7f0b30100290 2023-11-16T16:47:25.064 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:25.064 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:23.159198+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-11-16T16:47:25.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.064+0000 7f0b34bff700 1 -- 172.21.15.44:0/3837644872 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0b180a1150 msgr2=0x7f0b180a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.064+0000 7f0b34bff700 1 --2- 172.21.15.44:0/3837644872 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0b180a1150 0x7f0b180a3600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0b24011490 tx=0x7f0b24009260 comp rx=0 tx=0).stop 2023-11-16T16:47:25.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.064+0000 7f0b34bff700 1 -- 172.21.15.44:0/3837644872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b30100290 msgr2=0x7f0b300748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.064+0000 7f0b34bff700 1 --2- 172.21.15.44:0/3837644872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b30100290 0x7f0b300748a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0b20004730 tx=0x7f0b20004760 comp rx=0 tx=0).stop 2023-11-16T16:47:25.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.064+0000 7f0b34bff700 1 -- 172.21.15.44:0/3837644872 shutdown_connections 2023-11-16T16:47:25.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.064+0000 7f0b34bff700 1 --2- 172.21.15.44:0/3837644872 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0b180a1150 0x7f0b180a3600 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.064+0000 7f0b34bff700 1 --2- 172.21.15.44:0/3837644872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b30101e10 0x7f0b30073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.064+0000 7f0b34bff700 1 --2- 172.21.15.44:0/3837644872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b30100290 0x7f0b300748a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.064+0000 7f0b34bff700 1 --2- 172.21.15.44:0/3837644872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b30101480 0x7f0b30072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.064+0000 7f0b34bff700 1 -- 172.21.15.44:0/3837644872 >> 172.21.15.44:0/3837644872 conn(0x7f0b300fb870 msgr2=0x7f0b300fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:25.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.064+0000 7f0b34bff700 1 -- 172.21.15.44:0/3837644872 shutdown_connections 2023-11-16T16:47:25.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.064+0000 7f0b34bff700 1 -- 172.21.15.44:0/3837644872 wait complete. 2023-11-16T16:47:25.067 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 17 2023-11-16T16:47:25.085 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:25.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.287+0000 7fd8947ff700 1 -- 172.21.15.44:0/2204749585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd88c101480 msgr2=0x7fd88c1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.287+0000 7fd8947ff700 1 --2- 172.21.15.44:0/2204749585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd88c101480 0x7fd88c1018d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009990 tx=0x7fd87c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.287+0000 7fd8947ff700 1 -- 172.21.15.44:0/2204749585 shutdown_connections 2023-11-16T16:47:25.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.287+0000 7fd8947ff700 1 --2- 172.21.15.44:0/2204749585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88c101e10 0x7fd88c1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.287+0000 7fd8947ff700 1 --2- 172.21.15.44:0/2204749585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd88c100290 0x7fd88c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.287+0000 7fd8947ff700 1 --2- 172.21.15.44:0/2204749585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd88c101480 0x7fd88c1018d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.287+0000 7fd8947ff700 1 -- 172.21.15.44:0/2204749585 >> 172.21.15.44:0/2204749585 conn(0x7fd88c0fb870 msgr2=0x7fd88c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:25.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.287+0000 7fd8947ff700 1 -- 172.21.15.44:0/2204749585 shutdown_connections 2023-11-16T16:47:25.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.287+0000 7fd8947ff700 1 -- 172.21.15.44:0/2204749585 wait complete. 2023-11-16T16:47:25.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.288+0000 7fd8947ff700 1 Processor -- start 2023-11-16T16:47:25.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.288+0000 7fd8947ff700 1 -- start start 2023-11-16T16:47:25.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.288+0000 7fd8947ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd88c100290 0x7fd88c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.288+0000 7fd8947ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd88c101480 0x7fd88c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.288+0000 7fd8947ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88c101e10 0x7fd88c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.288+0000 7fd8947ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c073bc0 con 0x7fd88c101e10 2023-11-16T16:47:25.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.288+0000 7fd8947ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c073d30 con 0x7fd88c101480 2023-11-16T16:47:25.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.288+0000 7fd8947ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c1a0080 con 0x7fd88c100290 2023-11-16T16:47:25.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.288+0000 7fd892d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88c101e10 0x7fd88c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.288+0000 7fd892d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88c101e10 0x7fd88c073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39856/0 (socket says 172.21.15.44:39856) 2023-11-16T16:47:25.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.288+0000 7fd891d9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd88c101480 0x7fd88c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.288+0000 7fd892d9c700 1 -- 172.21.15.44:0/4268713993 learned_addr learned my addr 172.21.15.44:0/4268713993 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:25.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.289+0000 7fd89259b700 1 --2- 172.21.15.44:0/4268713993 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd88c100290 0x7fd88c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.289+0000 7fd892d9c700 1 -- 172.21.15.44:0/4268713993 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd88c100290 msgr2=0x7fd88c0748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.289+0000 7fd892d9c700 1 --2- 172.21.15.44:0/4268713993 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd88c100290 0x7fd88c0748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.289+0000 7fd892d9c700 1 -- 172.21.15.44:0/4268713993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd88c101480 msgr2=0x7fd88c072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.289+0000 7fd892d9c700 1 --2- 172.21.15.44:0/4268713993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd88c101480 0x7fd88c072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.289+0000 7fd892d9c700 1 -- 172.21.15.44:0/4268713993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87c0095f0 con 0x7fd88c101e10 2023-11-16T16:47:25.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.289+0000 7fd891d9a700 1 --2- 172.21.15.44:0/4268713993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd88c101480 0x7fd88c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:25.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.289+0000 7fd892d9c700 1 --2- 172.21.15.44:0/4268713993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88c101e10 0x7fd88c073420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd88400ef30 tx=0x7fd88400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:25.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.289+0000 7fd8837fe700 1 -- 172.21.15.44:0/4268713993 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884018070 con 0x7fd88c101e10 2023-11-16T16:47:25.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.289+0000 7fd8837fe700 1 -- 172.21.15.44:0/4268713993 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd88400f040 con 0x7fd88c101e10 2023-11-16T16:47:25.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.289+0000 7fd8947ff700 1 -- 172.21.15.44:0/4268713993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd88c10bc50 con 0x7fd88c101e10 2023-11-16T16:47:25.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.289+0000 7fd8837fe700 1 -- 172.21.15.44:0/4268713993 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88400cd00 con 0x7fd88c101e10 2023-11-16T16:47:25.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.290+0000 7fd8947ff700 1 -- 172.21.15.44:0/4268713993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd88c10bac0 con 0x7fd88c101e10 2023-11-16T16:47:25.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.291+0000 7fd8947ff700 1 -- 172.21.15.44:0/4268713993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd88c107870 con 0x7fd88c101e10 2023-11-16T16:47:25.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.293+0000 7fd8837fe700 1 -- 172.21.15.44:0/4268713993 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd884003a00 con 0x7fd88c101e10 2023-11-16T16:47:25.296 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.294+0000 7fd8837fe700 1 --2- 172.21.15.44:0/4268713993 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd8780a1150 0x7fd8780a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.296 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.294+0000 7fd8837fe700 1 -- 172.21.15.44:0/4268713993 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd8840cc570 con 0x7fd88c101e10 2023-11-16T16:47:25.296 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.294+0000 7fd89259b700 1 --2- 172.21.15.44:0/4268713993 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd8780a1150 0x7fd8780a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.296 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.295+0000 7fd89259b700 1 --2- 172.21.15.44:0/4268713993 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd8780a1150 0x7fd8780a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd88c1012e0 tx=0x7fd888009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:25.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.298+0000 7fd8837fe700 1 -- 172.21.15.44:0/4268713993 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd884095b60 con 0x7fd88c101e10 2023-11-16T16:47:25.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.509+0000 7fd8947ff700 1 -- 172.21.15.44:0/4268713993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd88c02cfe0 con 0x7fd88c101e10 2023-11-16T16:47:25.512 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.511+0000 7fd8837fe700 1 -- 172.21.15.44:0/4268713993 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23925 (secure 0 0 0) 0x7fd884099180 con 0x7fd88c101e10 2023-11-16T16:47:25.513 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:25.513 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":87,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:47:23.158961+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:25.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.513+0000 7fd8947ff700 1 -- 172.21.15.44:0/4268713993 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd8780a1150 msgr2=0x7fd8780a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.513+0000 7fd8947ff700 1 --2- 172.21.15.44:0/4268713993 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd8780a1150 0x7fd8780a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd88c1012e0 tx=0x7fd888009260 comp rx=0 tx=0).stop 2023-11-16T16:47:25.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.513+0000 7fd8947ff700 1 -- 172.21.15.44:0/4268713993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88c101e10 msgr2=0x7fd88c073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.513+0000 7fd8947ff700 1 --2- 172.21.15.44:0/4268713993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88c101e10 0x7fd88c073420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd88400ef30 tx=0x7fd88400c3a0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.513+0000 7fd8947ff700 1 -- 172.21.15.44:0/4268713993 shutdown_connections 2023-11-16T16:47:25.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.513+0000 7fd8947ff700 1 --2- 172.21.15.44:0/4268713993 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd8780a1150 0x7fd8780a3600 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.513+0000 7fd8947ff700 1 --2- 172.21.15.44:0/4268713993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd88c101e10 0x7fd88c073420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.513+0000 7fd8947ff700 1 --2- 172.21.15.44:0/4268713993 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd88c100290 0x7fd88c0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.513+0000 7fd8947ff700 1 --2- 172.21.15.44:0/4268713993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd88c101480 0x7fd88c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.513+0000 7fd8947ff700 1 -- 172.21.15.44:0/4268713993 >> 172.21.15.44:0/4268713993 conn(0x7fd88c0fb870 msgr2=0x7fd88c1049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:25.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.513+0000 7fd8947ff700 1 -- 172.21.15.44:0/4268713993 shutdown_connections 2023-11-16T16:47:25.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.513+0000 7fd8947ff700 1 -- 172.21.15.44:0/4268713993 wait complete. 2023-11-16T16:47:25.535 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-16T16:47:18.150861+0000', 'modified': '2023-11-16T16:47:23.159198+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24551}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24551': {'gid': 24551, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6839/1948116745', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6838', 'nonce': 1948116745}, {'type': 'v1', 'addr': '172.21.15.77:6839', 'nonce': 1948116745}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-11-16T16:47:25.535 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-16T16:47:25.535 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-16T16:47:25.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.742+0000 7f2ebf1ff700 1 -- 172.21.15.44:0/907154736 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eb8073550 msgr2=0x7f2eb80739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.742+0000 7f2ebf1ff700 1 --2- 172.21.15.44:0/907154736 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eb8073550 0x7f2eb80739c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4009990 tx=0x7f2ea4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.742+0000 7f2ebf1ff700 1 -- 172.21.15.44:0/907154736 shutdown_connections 2023-11-16T16:47:25.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.742+0000 7f2ebf1ff700 1 --2- 172.21.15.44:0/907154736 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2eb8108100 0x7f2eb810a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.742+0000 7f2ebf1ff700 1 --2- 172.21.15.44:0/907154736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2eb8074b30 0x7f2eb8072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.742+0000 7f2ebf1ff700 1 --2- 172.21.15.44:0/907154736 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eb8073550 0x7f2eb80739c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.742+0000 7f2ebf1ff700 1 -- 172.21.15.44:0/907154736 >> 172.21.15.44:0/907154736 conn(0x7f2eb80fb820 msgr2=0x7f2eb80fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:25.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.742+0000 7f2ebf1ff700 1 -- 172.21.15.44:0/907154736 shutdown_connections 2023-11-16T16:47:25.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.742+0000 7f2ebf1ff700 1 -- 172.21.15.44:0/907154736 wait complete. 2023-11-16T16:47:25.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.744+0000 7f2ebf1ff700 1 Processor -- start 2023-11-16T16:47:25.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.744+0000 7f2ebf1ff700 1 -- start start 2023-11-16T16:47:25.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.744+0000 7f2ebf1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2eb8073550 0x7f2eb81981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.744+0000 7f2ebf1ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2eb8074b30 0x7f2eb8198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.744+0000 7f2ebf1ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eb8108100 0x7f2eb819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.744+0000 7f2ebf1ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb819fd60 con 0x7f2eb8108100 2023-11-16T16:47:25.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.744+0000 7f2ebf1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb819fea0 con 0x7f2eb8073550 2023-11-16T16:47:25.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.744+0000 7f2ebf1ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb81a0030 con 0x7f2eb8074b30 2023-11-16T16:47:25.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.745+0000 7f2ebd79c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eb8108100 0x7f2eb819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.745+0000 7f2ebd79c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eb8108100 0x7f2eb819f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39880/0 (socket says 172.21.15.44:39880) 2023-11-16T16:47:25.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.745+0000 7f2ebd79c700 1 -- 172.21.15.44:0/2190384114 learned_addr learned my addr 172.21.15.44:0/2190384114 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:25.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.745+0000 7f2ebcf9b700 1 --2- 172.21.15.44:0/2190384114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2eb8073550 0x7f2eb81981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.745+0000 7f2eb7fff700 1 --2- 172.21.15.44:0/2190384114 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2eb8074b30 0x7f2eb8198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.745+0000 7f2ebd79c700 1 -- 172.21.15.44:0/2190384114 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2eb8074b30 msgr2=0x7f2eb8198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.745+0000 7f2ebd79c700 1 --2- 172.21.15.44:0/2190384114 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2eb8074b30 0x7f2eb8198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.745+0000 7f2ebd79c700 1 -- 172.21.15.44:0/2190384114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2eb8073550 msgr2=0x7f2eb81981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.745+0000 7f2ebd79c700 1 --2- 172.21.15.44:0/2190384114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2eb8073550 0x7f2eb81981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.745+0000 7f2ebd79c700 1 -- 172.21.15.44:0/2190384114 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea40095f0 con 0x7f2eb8108100 2023-11-16T16:47:25.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.745+0000 7f2ebcf9b700 1 --2- 172.21.15.44:0/2190384114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2eb8073550 0x7f2eb81981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:25.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.745+0000 7f2ebd79c700 1 --2- 172.21.15.44:0/2190384114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eb8108100 0x7f2eb819f790 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000ef30 tx=0x7f2eb000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:25.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.746+0000 7f2eb5ffb700 1 -- 172.21.15.44:0/2190384114 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0018070 con 0x7f2eb8108100 2023-11-16T16:47:25.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.746+0000 7f2eb5ffb700 1 -- 172.21.15.44:0/2190384114 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2eb000f040 con 0x7f2eb8108100 2023-11-16T16:47:25.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.746+0000 7f2eb5ffb700 1 -- 172.21.15.44:0/2190384114 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb000cd00 con 0x7f2eb8108100 2023-11-16T16:47:25.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.746+0000 7f2ebf1ff700 1 -- 172.21.15.44:0/2190384114 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb810de20 con 0x7f2eb8108100 2023-11-16T16:47:25.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.746+0000 7f2ebf1ff700 1 -- 172.21.15.44:0/2190384114 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb810dc90 con 0x7f2eb8108100 2023-11-16T16:47:25.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.749+0000 7f2eb5ffb700 1 -- 172.21.15.44:0/2190384114 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2eb0003a00 con 0x7f2eb8108100 2023-11-16T16:47:25.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.750+0000 7f2ed2b6fb80 1 Processor -- start 2023-11-16T16:47:25.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.750+0000 7f2ed2b6fb80 1 -- start start 2023-11-16T16:47:25.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.750+0000 7f2eb5ffb700 1 --2- 172.21.15.44:0/2190384114 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ea80a11d0 0x7f2ea80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.750+0000 7f2ed2b6fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 0x562508f3dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.750+0000 7f2eb5ffb700 1 -- 172.21.15.44:0/2190384114 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2eb00cc620 con 0x7f2eb8108100 2023-11-16T16:47:25.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.750+0000 7f2ed2b6fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f3eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.750+0000 7f2ed2b6fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3f0c0 0x562508f3f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.750+0000 7f2ed2b6fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562508e43010 con 0x562508f3e500 2023-11-16T16:47:25.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.750+0000 7f2ed2b6fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562508e95850 con 0x562508f3f0c0 2023-11-16T16:47:25.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.750+0000 7f2ed2b6fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x562508e8ece0 con 0x562508f3dbb0 2023-11-16T16:47:25.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.751+0000 7f2eb7fff700 1 --2- 172.21.15.44:0/2190384114 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ea80a11d0 0x7f2ea80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.751+0000 7f2eb7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f3eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.751+0000 7f2ebcf9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 0x562508f3dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.751+0000 7f2ebcf9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 0x562508f3dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:59464/0 (socket says 172.21.15.44:59464) 2023-11-16T16:47:25.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.751+0000 7f2ebcf9b700 1 -- 172.21.15.44:0/850129636 learned_addr learned my addr 172.21.15.44:0/850129636 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:25.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.751+0000 7f2ebd79c700 1 --2- 172.21.15.44:0/850129636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3f0c0 0x562508f3f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.751+0000 7f2ebcf9b700 1 -- 172.21.15.44:0/850129636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3f0c0 msgr2=0x562508f3f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:47:25.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.751+0000 7f2ebcf9b700 1 --2- 172.21.15.44:0/850129636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3f0c0 0x562508f3f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.751+0000 7f2ebcf9b700 1 -- 172.21.15.44:0/850129636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 msgr2=0x562508f3eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:47:25.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.751+0000 7f2ebcf9b700 1 --2- 172.21.15.44:0/850129636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f3eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.751+0000 7f2ebcf9b700 1 -- 172.21.15.44:0/850129636 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562508d105e0 con 0x562508f3dbb0 2023-11-16T16:47:25.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.751+0000 7f2eb7fff700 1 --2- 172.21.15.44:0/2190384114 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ea80a11d0 0x7f2ea80a3680 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400b2e0 tx=0x7f2ea4005510 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:25.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.752+0000 7f2ebcf9b700 1 --2- 172.21.15.44:0/850129636 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 0x562508f3dfc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2eac0115f0 tx=0x7f2eac011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78fe26ab8742b46d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:25.759 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.752+0000 7f2ea2ffd700 1 -- 172.21.15.44:0/850129636 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac004230 con 0x562508f3dbb0 2023-11-16T16:47:25.759 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.752+0000 7f2ea2ffd700 1 -- 172.21.15.44:0/850129636 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2eac004390 con 0x562508f3dbb0 2023-11-16T16:47:25.759 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.752+0000 7f2ea2ffd700 1 -- 172.21.15.44:0/850129636 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac019820 con 0x562508f3dbb0 2023-11-16T16:47:25.759 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.753+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/850129636 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 msgr2=0x562508f3dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.759 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.753+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/850129636 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 0x562508f3dfc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2eac0115f0 tx=0x7f2eac011900 comp rx=0 tx=0).stop 2023-11-16T16:47:25.759 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.753+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/850129636 shutdown_connections 2023-11-16T16:47:25.759 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.753+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/850129636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3f0c0 0x562508f3f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.759 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.753+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/850129636 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 0x562508f3dfc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.759 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.753+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/850129636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f3eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.753+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/850129636 >> 172.21.15.44:0/850129636 conn(0x562508f33310 msgr2=0x562508f38620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:25.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.753+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/850129636 shutdown_connections 2023-11-16T16:47:25.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.753+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/850129636 wait complete. 2023-11-16T16:47:25.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.753+0000 7f2ed2b6fb80 1 -- start start 2023-11-16T16:47:25.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.754+0000 7f2ed2b6fb80 1 Processor -- start 2023-11-16T16:47:25.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.754+0000 7f2ed2b6fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3dbb0 0x562508f47590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.754+0000 7f2ed2b6fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f47ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.754+0000 7f2ed2b6fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3f0c0 0x562508f4eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.754+0000 7f2ed2b6fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562508e30890 con 0x562508f3e500 2023-11-16T16:47:25.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.754+0000 7f2ed2b6fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562508e30530 con 0x562508f3dbb0 2023-11-16T16:47:25.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.754+0000 7f2ed2b6fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x562508d128d0 con 0x562508f3f0c0 2023-11-16T16:47:25.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.754+0000 7f2ed2b6fb80 10 client.0 _ll_register_callbacks cb 0x562508e4e730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:47:25.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.754+0000 7f2ebcf9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3dbb0 0x562508f47590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.754+0000 7f2ebcf9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3dbb0 0x562508f47590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:43896/0 (socket says 172.21.15.44:43896) 2023-11-16T16:47:25.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.754+0000 7f2ebcf9b700 1 -- 172.21.15.44:0/2112467613 learned_addr learned my addr 172.21.15.44:0/2112467613 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:25.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.754+0000 7f2eb7fff700 1 --2- 172.21.15.44:0/2112467613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f47ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.754+0000 7f2ebd79c700 1 --2- 172.21.15.44:0/2112467613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3f0c0 0x562508f4eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.755+0000 7f2eb7fff700 1 -- 172.21.15.44:0/2112467613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3f0c0 msgr2=0x562508f4eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:47:25.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.755+0000 7f2eb7fff700 1 --2- 172.21.15.44:0/2112467613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3f0c0 0x562508f4eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.755+0000 7f2eb7fff700 1 -- 172.21.15.44:0/2112467613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3dbb0 msgr2=0x562508f47590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:47:25.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.755+0000 7f2eb7fff700 1 --2- 172.21.15.44:0/2112467613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3dbb0 0x562508f47590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.755+0000 7f2eb7fff700 1 -- 172.21.15.44:0/2112467613 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eac009d50 con 0x562508f3e500 2023-11-16T16:47:25.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.755+0000 7f2ebcf9b700 1 --2- 172.21.15.44:0/2112467613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3dbb0 0x562508f47590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:25.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.755+0000 7f2eb7fff700 1 --2- 172.21.15.44:0/2112467613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f47ad0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400b680 tx=0x7f2ea4005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12991da1ba6f9dc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:25.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.755+0000 7f2e97fff700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:47:25.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.755+0000 7f2e97fff700 1 -- 172.21.15.44:0/2112467613 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea4029470 con 0x562508f3e500 2023-11-16T16:47:25.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.755+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/2112467613 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562508f30ad0 con 0x562508f3e500 2023-11-16T16:47:25.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.755+0000 7f2e97fff700 1 -- 172.21.15.44:0/2112467613 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2ea40295d0 con 0x562508f3e500 2023-11-16T16:47:25.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.756+0000 7f2e97fff700 1 -- 172.21.15.44:0/2112467613 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ea4029790 con 0x562508f3e500 2023-11-16T16:47:25.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.756+0000 7f2ed2b6fb80 10 client.14649 fetch_fsmap learned FSMap version 17 2023-11-16T16:47:25.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.756+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/2112467613 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x562508e55140 con 0x562508f3e500 2023-11-16T16:47:25.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.756+0000 7f2e97fff700 1 -- 172.21.15.44:0/2112467613 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562508f30ad0 con 0x562508f3e500 2023-11-16T16:47:25.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.756+0000 7f2e97fff700 1 -- 172.21.15.44:0/2112467613 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f2ea40177c0 con 0x562508f3e500 2023-11-16T16:47:25.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.756+0000 7f2ed2b6fb80 10 client.14649 fetch_fsmap finished waiting for FSMap version 17 2023-11-16T16:47:25.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.756+0000 7f2ed2b6fb80 10 client.14649 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T16:47:25.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.756+0000 7f2ed2b6fb80 20 client.14649 populate_metadata read hostname 'smithi044' 2023-11-16T16:47:25.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.756+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/2112467613 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562508f52870 0x562508f51c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.756+0000 7f2ed2b6fb80 4 client.14649 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:47:25.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.756+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/2112467613 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562508e51050 con 0x562508f52870 2023-11-16T16:47:25.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.757+0000 7f2ebcf9b700 1 --2- 172.21.15.44:0/2112467613 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562508f52870 0x562508f51c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.757+0000 7f2ebcf9b700 1 --2- 172.21.15.44:0/2112467613 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562508f52870 0x562508f51c60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db4fc0bdb4b2625f server_cookie=6cc2683e6b8bd55f in_seq=0 out_seq=0 2023-11-16T16:47:25.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.757+0000 7f2e97fff700 10 client.14649 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:47:25.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.758+0000 7f2e97fff700 1 -- 172.21.15.44:0/2112467613 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562508e51050 con 0x562508f52870 2023-11-16T16:47:25.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.758+0000 7f2e97fff700 10 client.14649 handle_command_reply: tid=0 2023-11-16T16:47:25.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.758+0000 7f2ed2b6fb80 1 client.14649 shutdown 2023-11-16T16:47:25.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.758+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/2112467613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 msgr2=0x562508f47ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.758+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/2112467613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f47ad0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400b680 tx=0x7f2ea4005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.758+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/2112467613 shutdown_connections 2023-11-16T16:47:25.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.758+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/2112467613 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562508f52870 0x562508f51c60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.758+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/2112467613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3f0c0 0x562508f4eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.758+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/2112467613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3dbb0 0x562508f47590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.758+0000 7f2e97fff700 0 client.14649 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:47:25.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.758+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/2112467613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f47ad0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.758+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/2112467613 >> 172.21.15.44:0/2112467613 conn(0x562508f33310 msgr2=0x562508f37d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:25.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.758+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/2112467613 shutdown_connections 2023-11-16T16:47:25.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.758+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/2112467613 wait complete. 2023-11-16T16:47:25.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.759+0000 7f2ed2b6fb80 20 client.14649 trim_cache size 0 max 16384 2023-11-16T16:47:25.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.777+0000 7f2ed2b6fb80 1 Processor -- start 2023-11-16T16:47:25.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.777+0000 7f2ed2b6fb80 1 -- start start 2023-11-16T16:47:25.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.777+0000 7f2ed2b6fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 0x562508f51c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.777+0000 7f2ed2b6fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.777+0000 7f2ed2b6fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3f0c0 0x562508f4eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.777+0000 7f2ed2b6fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562508f47160 con 0x562508f3e500 2023-11-16T16:47:25.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.777+0000 7f2ed2b6fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562508e38730 con 0x562508f3f0c0 2023-11-16T16:47:25.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.777+0000 7f2ed2b6fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x562508d9aed0 con 0x562508f3dbb0 2023-11-16T16:47:25.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.777+0000 7f2eb7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.777+0000 7f2eb7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39898/0 (socket says 172.21.15.44:39898) 2023-11-16T16:47:25.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.777+0000 7f2eb7fff700 1 -- 172.21.15.44:0/124984641 learned_addr learned my addr 172.21.15.44:0/124984641 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:25.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.778+0000 7f2ebd79c700 1 --2- 172.21.15.44:0/124984641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3f0c0 0x562508f4eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.778+0000 7f2ebcf9b700 1 --2- 172.21.15.44:0/124984641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 0x562508f51c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.778+0000 7f2eb7fff700 1 -- 172.21.15.44:0/124984641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 msgr2=0x562508f51c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:47:25.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.778+0000 7f2eb7fff700 1 --2- 172.21.15.44:0/124984641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 0x562508f51c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.778+0000 7f2eb7fff700 1 -- 172.21.15.44:0/124984641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3f0c0 msgr2=0x562508f4eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:47:25.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.778+0000 7f2eb7fff700 1 --2- 172.21.15.44:0/124984641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3f0c0 0x562508f4eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.778+0000 7f2eb7fff700 1 -- 172.21.15.44:0/124984641 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562508e54e90 con 0x562508f3e500 2023-11-16T16:47:25.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.778+0000 7f2eb7fff700 1 --2- 172.21.15.44:0/124984641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f521a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400bb10 tx=0x7f2ea40098c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=200e2b8a0460d812 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:25.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.778+0000 7f2ea37fe700 1 -- 172.21.15.44:0/124984641 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea400b390 con 0x562508f3e500 2023-11-16T16:47:25.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.778+0000 7f2ea37fe700 1 -- 172.21.15.44:0/124984641 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ea4005720 con 0x562508f3e500 2023-11-16T16:47:25.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.778+0000 7f2ea37fe700 1 -- 172.21.15.44:0/124984641 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea403ad40 con 0x562508f3e500 2023-11-16T16:47:25.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.779+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/124984641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 msgr2=0x562508f521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.779+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/124984641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f521a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400bb10 tx=0x7f2ea40098c0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.779+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/124984641 shutdown_connections 2023-11-16T16:47:25.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.779+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/124984641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3f0c0 0x562508f4eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.779+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/124984641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 0x562508f51c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.779+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/124984641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f521a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.779+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/124984641 >> 172.21.15.44:0/124984641 conn(0x562508f33310 msgr2=0x562508f45440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:25.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.779+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/124984641 shutdown_connections 2023-11-16T16:47:25.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.779+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/124984641 wait complete. 2023-11-16T16:47:25.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.779+0000 7f2ed2b6fb80 1 -- start start 2023-11-16T16:47:25.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.780+0000 7f2ed2b6fb80 1 Processor -- start 2023-11-16T16:47:25.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.780+0000 7f2ed2b6fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 0x562508f48df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.780+0000 7f2ed2b6fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f49330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.780+0000 7f2ed2b6fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3f0c0 0x562508f4e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.780+0000 7f2ed2b6fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562508f4e8a0 con 0x562508f3e500 2023-11-16T16:47:25.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.780+0000 7f2ed2b6fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562508f4ea10 con 0x562508f3f0c0 2023-11-16T16:47:25.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.780+0000 7f2ed2b6fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x562508f4ebd0 con 0x562508f3dbb0 2023-11-16T16:47:25.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.780+0000 7f2ed2b6fb80 10 client.0 _ll_register_callbacks cb 0x7f2ea400b850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:47:25.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.780+0000 7f2ebcf9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 0x562508f48df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.780+0000 7f2eb7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f49330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.780+0000 7f2ebcf9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 0x562508f48df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:59494/0 (socket says 172.21.15.44:59494) 2023-11-16T16:47:25.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.780+0000 7f2ebcf9b700 1 -- 172.21.15.44:0/331161198 learned_addr learned my addr 172.21.15.44:0/331161198 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:25.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.780+0000 7f2ebd79c700 1 --2- 172.21.15.44:0/331161198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3f0c0 0x562508f4e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.781+0000 7f2eb7fff700 1 -- 172.21.15.44:0/331161198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 msgr2=0x562508f48df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:47:25.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.781+0000 7f2eb7fff700 1 --2- 172.21.15.44:0/331161198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 0x562508f48df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.781+0000 7f2eb7fff700 1 -- 172.21.15.44:0/331161198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3f0c0 msgr2=0x562508f4e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:47:25.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.781+0000 7f2eb7fff700 1 --2- 172.21.15.44:0/331161198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3f0c0 0x562508f4e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.781+0000 7f2eb7fff700 1 -- 172.21.15.44:0/331161198 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562508e55140 con 0x562508f3e500 2023-11-16T16:47:25.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.781+0000 7f2ebcf9b700 1 --2- 172.21.15.44:0/331161198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 0x562508f48df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:25.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.781+0000 7f2ebd79c700 1 --2- 172.21.15.44:0/331161198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3f0c0 0x562508f4e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:25.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.781+0000 7f2eb7fff700 1 --2- 172.21.15.44:0/331161198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f49330 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40058b0 tx=0x7f2ea403bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0f8059212cca8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:25.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.781+0000 7f2e97fff700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:47:25.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.781+0000 7f2e97fff700 1 -- 172.21.15.44:0/331161198 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea403a6e0 con 0x562508f3e500 2023-11-16T16:47:25.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.781+0000 7f2e97fff700 1 -- 172.21.15.44:0/331161198 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2ea4017600 con 0x562508f3e500 2023-11-16T16:47:25.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.781+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/331161198 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2eac004390 con 0x562508f3e500 2023-11-16T16:47:25.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.782+0000 7f2e97fff700 1 -- 172.21.15.44:0/331161198 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ea40177c0 con 0x562508f3e500 2023-11-16T16:47:25.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.782+0000 7f2ed2b6fb80 10 client.14655 fetch_fsmap learned FSMap version 17 2023-11-16T16:47:25.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.782+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/331161198 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x562508ca5c10 con 0x562508f3e500 2023-11-16T16:47:25.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.782+0000 7f2e97fff700 1 -- 172.21.15.44:0/331161198 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2eac004390 con 0x562508f3e500 2023-11-16T16:47:25.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.782+0000 7f2e97fff700 1 -- 172.21.15.44:0/331161198 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f2ea4033440 con 0x562508f3e500 2023-11-16T16:47:25.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.782+0000 7f2ed2b6fb80 10 client.14655 fetch_fsmap finished waiting for FSMap version 17 2023-11-16T16:47:25.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.782+0000 7f2ed2b6fb80 10 client.14655 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T16:47:25.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.782+0000 7f2ed2b6fb80 20 client.14655 populate_metadata read hostname 'smithi044' 2023-11-16T16:47:25.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.782+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/331161198 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562508faaa40 0x562508fa7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.782+0000 7f2ed2b6fb80 4 client.14655 mds_command: new command op to 24551 tid=0[{"prefix": "status"}] 2023-11-16T16:47:25.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.782+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/331161198 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562508e5d610 con 0x562508faaa40 2023-11-16T16:47:25.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.783+0000 7f2ebcf9b700 1 --2- 172.21.15.44:0/331161198 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562508faaa40 0x562508fa7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.783+0000 7f2ebcf9b700 1 --2- 172.21.15.44:0/331161198 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562508faaa40 0x562508fa7480 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89ab7ab70b5f76a8 server_cookie=65a1edba3d769642 in_seq=0 out_seq=0 2023-11-16T16:47:25.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.783+0000 7f2e97fff700 10 client.14655 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:47:25.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.783+0000 7f2e97fff700 1 -- 172.21.15.44:0/331161198 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x562508e5d610 con 0x562508faaa40 2023-11-16T16:47:25.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.783+0000 7f2e97fff700 10 client.14655 handle_command_reply: tid=0 2023-11-16T16:47:25.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.783+0000 7f2ed2b6fb80 1 client.14655 shutdown 2023-11-16T16:47:25.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.784+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/331161198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 msgr2=0x562508f49330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.784+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/331161198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f49330 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40058b0 tx=0x7f2ea403bf10 comp rx=0 tx=0).stop 2023-11-16T16:47:25.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.784+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/331161198 shutdown_connections 2023-11-16T16:47:25.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.784+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/331161198 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562508faaa40 0x562508fa7480 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.784+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/331161198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562508f3f0c0 0x562508f4e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.784+0000 7f2e97fff700 0 client.14655 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:47:25.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.784+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/331161198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562508f3dbb0 0x562508f48df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.784+0000 7f2ed2b6fb80 1 --2- 172.21.15.44:0/331161198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562508f3e500 0x562508f49330 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.784+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/331161198 >> 172.21.15.44:0/331161198 conn(0x562508f33310 msgr2=0x562508f51c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:25.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.784+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/331161198 shutdown_connections 2023-11-16T16:47:25.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.784+0000 7f2ed2b6fb80 1 -- 172.21.15.44:0/331161198 wait complete. 2023-11-16T16:47:25.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.784+0000 7f2ed2b6fb80 20 client.14655 trim_cache size 0 max 16384 2023-11-16T16:47:25.792 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:47:25.792 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "7f2b12a8-849e-11ee-8dbe-212e2dc638e7", 2023-11-16T16:47:25.792 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2023-11-16T16:47:25.792 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24551, 2023-11-16T16:47:25.792 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2023-11-16T16:47:25.792 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2023-11-16T16:47:25.792 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2023-11-16T16:47:25.793 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 7.629050264, 2023-11-16T16:47:25.793 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 17, 2023-11-16T16:47:25.793 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 87, 2023-11-16T16:47:25.793 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 86, 2023-11-16T16:47:25.793 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 13.699338743 2023-11-16T16:47:25.793 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:47:25.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.784+0000 7f2ebf1ff700 1 -- 172.21.15.44:0/2190384114 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ea80a11d0 msgr2=0x7f2ea80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.784+0000 7f2ebf1ff700 1 --2- 172.21.15.44:0/2190384114 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ea80a11d0 0x7f2ea80a3680 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400b2e0 tx=0x7f2ea4005510 comp rx=0 tx=0).stop 2023-11-16T16:47:25.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.784+0000 7f2ebf1ff700 1 -- 172.21.15.44:0/2190384114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eb8108100 msgr2=0x7f2eb819f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.784+0000 7f2ebf1ff700 1 --2- 172.21.15.44:0/2190384114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eb8108100 0x7f2eb819f790 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000ef30 tx=0x7f2eb000c3a0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.785+0000 7f2ebf1ff700 1 -- 172.21.15.44:0/2190384114 shutdown_connections 2023-11-16T16:47:25.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.785+0000 7f2ebf1ff700 1 --2- 172.21.15.44:0/2190384114 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ea80a11d0 0x7f2ea80a3680 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.785+0000 7f2ebf1ff700 1 --2- 172.21.15.44:0/2190384114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eb8108100 0x7f2eb819f790 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.785+0000 7f2ebf1ff700 1 --2- 172.21.15.44:0/2190384114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2eb8073550 0x7f2eb81981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.785+0000 7f2ebf1ff700 1 --2- 172.21.15.44:0/2190384114 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2eb8074b30 0x7f2eb8198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.785+0000 7f2ebf1ff700 1 -- 172.21.15.44:0/2190384114 >> 172.21.15.44:0/2190384114 conn(0x7f2eb80fb820 msgr2=0x7f2eb80fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:25.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.785+0000 7f2ebf1ff700 1 -- 172.21.15.44:0/2190384114 shutdown_connections 2023-11-16T16:47:25.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:25.785+0000 7f2ebf1ff700 1 -- 172.21.15.44:0/2190384114 wait complete. 2023-11-16T16:47:25.802 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:25 smithi044 ceph-mon[128750]: pgmap v278: 129 pgs: 9 creating+peering, 108 active+clean, 12 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 3 op/s 2023-11-16T16:47:25.802 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:25 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3837644872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:25.802 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:25 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/4268713993' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:25.805 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-16T16:47:25.805 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-11-16T16:47:25.805 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph fs ls 2023-11-16T16:47:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:25 smithi176 ceph-mon[131630]: pgmap v278: 129 pgs: 9 creating+peering, 108 active+clean, 12 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 3 op/s 2023-11-16T16:47:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:25 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3837644872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:25 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/4268713993' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:25 smithi077 ceph-mon[131204]: pgmap v278: 129 pgs: 9 creating+peering, 108 active+clean, 12 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 3 op/s 2023-11-16T16:47:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:25 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3837644872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:25 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/4268713993' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:26.662 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:47:27.734 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:27 smithi044 ceph-mon[128750]: pgmap v279: 129 pgs: 9 creating+peering, 120 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2023-11-16T16:47:27.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:27 smithi077 ceph-mon[131204]: pgmap v279: 129 pgs: 9 creating+peering, 120 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2023-11-16T16:47:28.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:27 smithi176 ceph-mon[131630]: pgmap v279: 129 pgs: 9 creating+peering, 120 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2023-11-16T16:47:28.399 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.397+0000 7fe2d9c41700 1 -- 172.21.15.44:0/696901121 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2d4100ee0 msgr2=0x7fe2d4104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:28.399 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.397+0000 7fe2d9c41700 1 --2- 172.21.15.44:0/696901121 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2d4100ee0 0x7fe2d4104eb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8009920 tx=0x7fe2c8009c30 comp rx=0 tx=0).stop 2023-11-16T16:47:28.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.397+0000 7fe2d9c41700 1 -- 172.21.15.44:0/696901121 shutdown_connections 2023-11-16T16:47:28.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.397+0000 7fe2d9c41700 1 --2- 172.21.15.44:0/696901121 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2d41055e0 0x7fe2d410d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:28.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.397+0000 7fe2d9c41700 1 --2- 172.21.15.44:0/696901121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2d4100540 0x7fe2d4100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:28.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.397+0000 7fe2d9c41700 1 --2- 172.21.15.44:0/696901121 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2d4100ee0 0x7fe2d4104eb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:28.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.397+0000 7fe2d9c41700 1 -- 172.21.15.44:0/696901121 >> 172.21.15.44:0/696901121 conn(0x7fe2d40fbdc0 msgr2=0x7fe2d40fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:28.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.397+0000 7fe2d9c41700 1 -- 172.21.15.44:0/696901121 shutdown_connections 2023-11-16T16:47:28.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.397+0000 7fe2d9c41700 1 -- 172.21.15.44:0/696901121 wait complete. 2023-11-16T16:47:28.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.398+0000 7fe2d9c41700 1 Processor -- start 2023-11-16T16:47:28.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.399+0000 7fe2d9c41700 1 -- start start 2023-11-16T16:47:28.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.399+0000 7fe2d9c41700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2d4100540 0x7fe2d4196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:28.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.399+0000 7fe2d9c41700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2d4100ee0 0x7fe2d4196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:28.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.399+0000 7fe2d9c41700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2d41055e0 0x7fe2d419add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:28.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.399+0000 7fe2d9c41700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d419b3e0 con 0x7fe2d4100540 2023-11-16T16:47:28.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.399+0000 7fe2d9c41700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d419b550 con 0x7fe2d41055e0 2023-11-16T16:47:28.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.399+0000 7fe2d9c41700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d419b710 con 0x7fe2d4100ee0 2023-11-16T16:47:28.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.399+0000 7fe2d37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2d4100540 0x7fe2d4196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:28.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.399+0000 7fe2d37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2d4100540 0x7fe2d4196420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39924/0 (socket says 172.21.15.44:39924) 2023-11-16T16:47:28.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.399+0000 7fe2d37fe700 1 -- 172.21.15.44:0/3875067402 learned_addr learned my addr 172.21.15.44:0/3875067402 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:28.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.399+0000 7fe2d2ffd700 1 --2- 172.21.15.44:0/3875067402 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2d4100ee0 0x7fe2d4196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:28.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.399+0000 7fe2d3fff700 1 --2- 172.21.15.44:0/3875067402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2d41055e0 0x7fe2d419add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:28.403 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.400+0000 7fe2d2ffd700 1 -- 172.21.15.44:0/3875067402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2d41055e0 msgr2=0x7fe2d419add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:28.403 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.400+0000 7fe2d2ffd700 1 --2- 172.21.15.44:0/3875067402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2d41055e0 0x7fe2d419add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:28.403 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.400+0000 7fe2d2ffd700 1 -- 172.21.15.44:0/3875067402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2d4100540 msgr2=0x7fe2d4196420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:28.403 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.400+0000 7fe2d2ffd700 1 --2- 172.21.15.44:0/3875067402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2d4100540 0x7fe2d4196420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:28.403 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.400+0000 7fe2d2ffd700 1 -- 172.21.15.44:0/3875067402 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c40095f0 con 0x7fe2d4100ee0 2023-11-16T16:47:28.403 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.400+0000 7fe2d37fe700 1 --2- 172.21.15.44:0/3875067402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2d4100540 0x7fe2d4196420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:28.403 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.400+0000 7fe2d2ffd700 1 --2- 172.21.15.44:0/3875067402 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2d4100ee0 0x7fe2d4196960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8009230 tx=0x7fe2c800fa20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:28.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.400+0000 7fe2d0ff9700 1 -- 172.21.15.44:0/3875067402 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c801c070 con 0x7fe2d4100ee0 2023-11-16T16:47:28.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.400+0000 7fe2d9c41700 1 -- 172.21.15.44:0/3875067402 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c8009520 con 0x7fe2d4100ee0 2023-11-16T16:47:28.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.400+0000 7fe2d0ff9700 1 -- 172.21.15.44:0/3875067402 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2c800baa0 con 0x7fe2d4100ee0 2023-11-16T16:47:28.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.400+0000 7fe2d0ff9700 1 -- 172.21.15.44:0/3875067402 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c80038e0 con 0x7fe2d4100ee0 2023-11-16T16:47:28.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.400+0000 7fe2d9c41700 1 -- 172.21.15.44:0/3875067402 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d4110360 con 0x7fe2d4100ee0 2023-11-16T16:47:28.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.401+0000 7fe2d9c41700 1 -- 172.21.15.44:0/3875067402 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d419bc50 con 0x7fe2d4100ee0 2023-11-16T16:47:28.405 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.403+0000 7fe2d0ff9700 1 -- 172.21.15.44:0/3875067402 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe2c8020030 con 0x7fe2d4100ee0 2023-11-16T16:47:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.405+0000 7fe2d0ff9700 1 --2- 172.21.15.44:0/3875067402 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe2bc0a1150 0x7fe2bc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.405+0000 7fe2d0ff9700 1 -- 172.21.15.44:0/3875067402 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe2c80cd9d0 con 0x7fe2d4100ee0 2023-11-16T16:47:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.405+0000 7fe2d37fe700 1 --2- 172.21.15.44:0/3875067402 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe2bc0a1150 0x7fe2bc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.405+0000 7fe2d37fe700 1 --2- 172.21.15.44:0/3875067402 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe2bc0a1150 0x7fe2bc0a3600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4005d10 tx=0x7fe2c4009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:28.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.409+0000 7fe2d0ff9700 1 -- 172.21.15.44:0/3875067402 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe2c8096f60 con 0x7fe2d4100ee0 2023-11-16T16:47:28.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.660+0000 7fe2d9c41700 1 -- 172.21.15.44:0/3875067402 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe2d41104f0 con 0x7fe2d4100ee0 2023-11-16T16:47:28.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.660+0000 7fe2d0ff9700 1 -- 172.21.15.44:0/3875067402 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7fe2c809a580 con 0x7fe2d4100ee0 2023-11-16T16:47:28.662 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-16T16:47:28.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.663+0000 7fe2d9c41700 1 -- 172.21.15.44:0/3875067402 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe2bc0a1150 msgr2=0x7fe2bc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:28.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.663+0000 7fe2d9c41700 1 --2- 172.21.15.44:0/3875067402 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe2bc0a1150 0x7fe2bc0a3600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4005d10 tx=0x7fe2c4009310 comp rx=0 tx=0).stop 2023-11-16T16:47:28.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.663+0000 7fe2d9c41700 1 -- 172.21.15.44:0/3875067402 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2d4100ee0 msgr2=0x7fe2d4196960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:28.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.663+0000 7fe2d9c41700 1 --2- 172.21.15.44:0/3875067402 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2d4100ee0 0x7fe2d4196960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8009230 tx=0x7fe2c800fa20 comp rx=0 tx=0).stop 2023-11-16T16:47:28.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.663+0000 7fe2d9c41700 1 -- 172.21.15.44:0/3875067402 shutdown_connections 2023-11-16T16:47:28.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.663+0000 7fe2d9c41700 1 --2- 172.21.15.44:0/3875067402 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe2bc0a1150 0x7fe2bc0a3600 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:28.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.663+0000 7fe2d9c41700 1 --2- 172.21.15.44:0/3875067402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2d41055e0 0x7fe2d419add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:28.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.663+0000 7fe2d9c41700 1 --2- 172.21.15.44:0/3875067402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2d4100540 0x7fe2d4196420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:28.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.663+0000 7fe2d9c41700 1 --2- 172.21.15.44:0/3875067402 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2d4100ee0 0x7fe2d4196960 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:28.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.663+0000 7fe2d9c41700 1 -- 172.21.15.44:0/3875067402 >> 172.21.15.44:0/3875067402 conn(0x7fe2d40fbdc0 msgr2=0x7fe2d40fc880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:28.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.663+0000 7fe2d9c41700 1 -- 172.21.15.44:0/3875067402 shutdown_connections 2023-11-16T16:47:28.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:28.663+0000 7fe2d9c41700 1 -- 172.21.15.44:0/3875067402 wait complete. 2023-11-16T16:47:29.151 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-16T16:47:29.151 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-16T16:47:29.151 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2023-11-16T16:47:29.151 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-11-16T16:47:29.151 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-11-16T16:47:29.151 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-16T16:47:29.151 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-16T16:47:29.151 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-16T16:47:29.151 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-11-16T16:47:29.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:29.152 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2023-11-16T16:47:29.196 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-16T16:47:29.197 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-16T16:47:29.197 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2023-11-16T16:47:29.197 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2023-11-16T16:47:29.197 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2023-11-16T16:47:29.197 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2023-11-16T16:47:29.197 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T16:47:29.197 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2023-11-16T16:47:29.197 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2023-11-16T16:47:29.197 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T16:47:29.197 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2023-11-16T16:47:29.197 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2023-11-16T16:47:29.197 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-16T16:47:29.198 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2023-11-16T16:47:29.198 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2023-11-16T16:47:29.198 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-16T16:47:29.198 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 41553sec preferred_lft 41553sec 2023-11-16T16:47:29.198 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link 2023-11-16T16:47:29.198 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2023-11-16T16:47:29.198 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T16:47:29.198 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2023-11-16T16:47:29.198 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2023-11-16T16:47:29.199 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-16T16:47:29.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:29.200 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec bash -c ' 2023-11-16T16:47:29.200 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2023-11-16T16:47:29.200 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add name ceph-brx type bridge 2023-11-16T16:47:29.200 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr flush dev ceph-brx 2023-11-16T16:47:29.200 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set ceph-brx up 2023-11-16T16:47:29.200 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-16T16:47:29.200 DEBUG:teuthology.orchestra.run.smithi044:> ') 2023-11-16T16:47:29.343 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:29 smithi044 ceph-mon[128750]: pgmap v280: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 1 op/s 2023-11-16T16:47:29.344 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:29 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3875067402' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T16:47:29.394 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:29 smithi176 ceph-mon[131630]: pgmap v280: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 1 op/s 2023-11-16T16:47:29.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:29 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3875067402' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T16:47:29.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:29 smithi077 ceph-mon[131204]: pgmap v280: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 1 op/s 2023-11-16T16:47:29.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:29 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3875067402' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T16:47:29.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:29.706 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-16T16:47:29.782 INFO:teuthology.orchestra.run.smithi044.stdout:1 2023-11-16T16:47:29.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:29.785 DEBUG:teuthology.orchestra.run.smithi044:> route 2023-11-16T16:47:29.849 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2023-11-16T16:47:29.850 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-16T16:47:29.850 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-16T16:47:29.850 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-16T16:47:29.850 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-16T16:47:29.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:29.851 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec bash -c ' 2023-11-16T16:47:29.851 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2023-11-16T16:47:29.851 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-16T16:47:29.851 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-16T16:47:29.851 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-16T16:47:29.851 DEBUG:teuthology.orchestra.run.smithi044:> ') 2023-11-16T16:47:30.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:30.555 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2023-11-16T16:47:30.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:30.600 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list-id 2023-11-16T16:47:30.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:30.687 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec bash -c ' 2023-11-16T16:47:30.687 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2023-11-16T16:47:30.687 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-16T16:47:30.687 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-11-16T16:47:30.687 DEBUG:teuthology.orchestra.run.smithi044:> ') 2023-11-16T16:47:30.834 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-11-16T16:47:30.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:30.835 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec bash -c ' 2023-11-16T16:47:30.835 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2023-11-16T16:47:30.835 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-11-16T16:47:30.835 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-16T16:47:30.835 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-11-16T16:47:30.835 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-11-16T16:47:30.835 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-11-16T16:47:30.835 DEBUG:teuthology.orchestra.run.smithi044:> ') 2023-11-16T16:47:31.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:31.130 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec bash -c ' 2023-11-16T16:47:31.131 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2023-11-16T16:47:31.131 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2023-11-16T16:47:31.131 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set dev brx.0 master ceph-brx 2023-11-16T16:47:31.131 DEBUG:teuthology.orchestra.run.smithi044:> ') 2023-11-16T16:47:31.308 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-11-16T16:47:31.308 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-16T16:47:31.308 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-11-16T16:47:31.352 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-11-16T16:47:31.353 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-16T16:47:31.353 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-11-16T16:47:31.435 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2023-11-16T16:47:31.546 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-16T16:47:31.546 DEBUG:teuthology.orchestra.run.smithi044:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-16T16:47:31.616 INFO:teuthology.orchestra.run.smithi044.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-16T16:47:31.619 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-16T16:47:31.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:31.620 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2023-11-16T16:47:31.664 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-16T16:47:31.664 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-11-16T16:47:31.709 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2023-11-16T16:47:31.781 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-16T16:47:31.781 DEBUG:teuthology.orchestra.run.smithi044:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-16T16:47:31.842 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:31 smithi044 ceph-mon[128750]: pgmap v281: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s wr, 0 op/s 2023-11-16T16:47:31.852 INFO:teuthology.orchestra.run.smithi044.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-16T16:47:31.854 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-16T16:47:31.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:31.855 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2023-11-16T16:47:31.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:31 smithi176 ceph-mon[131630]: pgmap v281: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s wr, 0 op/s 2023-11-16T16:47:31.959 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi044.stderr:2023-11-16T16:47:31.957+0000 7f86f1b395c0 -1 init, newargv = 0x563eca596390 newargc=15 2023-11-16T16:47:31.959 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi044.stderr:ceph-fuse[165166]: starting ceph client 2023-11-16T16:47:31.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi044.stderr:ceph-fuse[165166]: starting fuse 2023-11-16T16:47:31.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:31 smithi077 ceph-mon[131204]: pgmap v281: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s wr, 0 op/s 2023-11-16T16:47:33.001 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2023-11-16T16:47:33.080 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-16T16:47:33.081 DEBUG:teuthology.orchestra.run.smithi044:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-16T16:47:33.152 INFO:teuthology.orchestra.run.smithi044.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-16T16:47:33.155 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-16T16:47:33.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:33.155 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2023-11-16T16:47:33.200 INFO:teuthology.orchestra.run.smithi044.stdout:121 2023-11-16T16:47:33.200 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [121] 2023-11-16T16:47:33.200 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-11-16T16:47:33.200 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2023-11-16T16:47:33.200 DEBUG:teuthology.orchestra.run.smithi044:> import re 2023-11-16T16:47:33.200 DEBUG:teuthology.orchestra.run.smithi044:> import os 2023-11-16T16:47:33.201 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2023-11-16T16:47:33.201 DEBUG:teuthology.orchestra.run.smithi044:> 2023-11-16T16:47:33.201 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2023-11-16T16:47:33.201 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-11-16T16:47:33.201 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2023-11-16T16:47:33.201 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-11-16T16:47:33.201 DEBUG:teuthology.orchestra.run.smithi044:> 2023-11-16T16:47:33.201 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2023-11-16T16:47:33.201 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2023-11-16T16:47:33.201 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2023-11-16T16:47:33.201 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2023-11-16T16:47:33.201 DEBUG:teuthology.orchestra.run.smithi044:> 2023-11-16T16:47:33.202 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2023-11-16T16:47:33.202 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-16T16:47:33.202 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-16T16:47:33.202 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-16T16:47:33.202 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2023-11-16T16:47:33.202 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2023-11-16T16:47:33.202 DEBUG:teuthology.orchestra.run.smithi044:> return f 2023-11-16T16:47:33.202 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-16T16:47:33.202 DEBUG:teuthology.orchestra.run.smithi044:> 2023-11-16T16:47:33.202 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.admin")) 2023-11-16T16:47:33.202 DEBUG:teuthology.orchestra.run.smithi044:> ' 2023-11-16T16:47:33.387 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.admin.165166.asok 2023-11-16T16:47:33.565 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.165166.asok 2023-11-16T16:47:33.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:33.565 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.165166.asok status 2023-11-16T16:47:33.799 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:47:33.800 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2023-11-16T16:47:33.800 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "5e4a743ad7fd1422ebd6c4dc950a21a303279207", 2023-11-16T16:47:33.800 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev)", 2023-11-16T16:47:33.800 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "admin", 2023-11-16T16:47:33.800 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2023-11-16T16:47:33.800 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-11-16T16:47:33.800 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "165166", 2023-11-16T16:47:33.800 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2023-11-16T16:47:33.801 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T16:47:33.801 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2023-11-16T16:47:33.801 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2023-11-16T16:47:33.801 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24694, 2023-11-16T16:47:33.801 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2023-11-16T16:47:33.801 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2023-11-16T16:47:33.801 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2023-11-16T16:47:33.801 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 24694 2023-11-16T16:47:33.801 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T16:47:33.802 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2023-11-16T16:47:33.802 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2023-11-16T16:47:33.802 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2023-11-16T16:47:33.802 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 61699200 2023-11-16T16:47:33.802 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:47:33.802 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T16:47:33.802 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2023-11-16T16:47:33.802 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2023-11-16T16:47:33.802 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2023-11-16T16:47:33.803 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 61699200 2023-11-16T16:47:33.803 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T16:47:33.803 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.24694 172.21.15.44:0/61699200", 2023-11-16T16:47:33.803 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "172.21.15.44:0/61699200", 2023-11-16T16:47:33.803 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2023-11-16T16:47:33.803 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 17, 2023-11-16T16:47:33.803 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 87, 2023-11-16T16:47:33.803 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2023-11-16T16:47:33.803 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2023-11-16T16:47:33.803 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2023-11-16T16:47:33.804 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:47:33.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:33.815 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2023-11-16T16:47:33.815 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-11-16T16:47:33.860 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2023-11-16T16:47:33.860 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-11-16T16:47:33.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:33.861 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-11-16T16:47:33.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:33 smithi044 ceph-mon[128750]: pgmap v282: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 724 B/s wr, 0 op/s 2023-11-16T16:47:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:33 smithi176 ceph-mon[131630]: pgmap v282: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 724 B/s wr, 0 op/s 2023-11-16T16:47:33.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:33.939 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-11-16T16:47:33.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:33 smithi077 ceph-mon[131204]: pgmap v282: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 724 B/s wr, 0 op/s 2023-11-16T16:47:34.084 INFO:teuthology.orchestra.run.smithi044.stdout:# file: . 2023-11-16T16:47:34.084 INFO:teuthology.orchestra.run.smithi044.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-11-16T16:47:34.084 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:34.085 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:47:34.085 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T16:47:34.131 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:34.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.378+0000 7fbafdfff700 1 -- 172.21.15.44:0/1877467028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf81001b0 msgr2=0x7fbaf81005c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:34.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.378+0000 7fbafdfff700 1 --2- 172.21.15.44:0/1877467028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf81001b0 0x7fbaf81005c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbae8009990 tx=0x7fbae8009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:34.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.379+0000 7fbafdfff700 1 -- 172.21.15.44:0/1877467028 shutdown_connections 2023-11-16T16:47:34.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.379+0000 7fbafdfff700 1 --2- 172.21.15.44:0/1877467028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaf8101d30 0x7fbaf81085f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:34.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.379+0000 7fbafdfff700 1 --2- 172.21.15.44:0/1877467028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf81001b0 0x7fbaf81005c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:34.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.379+0000 7fbafdfff700 1 --2- 172.21.15.44:0/1877467028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf81013a0 0x7fbaf81017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:34.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.379+0000 7fbafdfff700 1 -- 172.21.15.44:0/1877467028 >> 172.21.15.44:0/1877467028 conn(0x7fbaf80fb790 msgr2=0x7fbaf80fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:34.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.379+0000 7fbafdfff700 1 -- 172.21.15.44:0/1877467028 shutdown_connections 2023-11-16T16:47:34.382 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.379+0000 7fbafdfff700 1 -- 172.21.15.44:0/1877467028 wait complete. 2023-11-16T16:47:34.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.380+0000 7fbafdfff700 1 Processor -- start 2023-11-16T16:47:34.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.380+0000 7fbafdfff700 1 -- start start 2023-11-16T16:47:34.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.380+0000 7fbafdfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf81001b0 0x7fbaf80748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:34.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.380+0000 7fbafdfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf81013a0 0x7fbaf8072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:34.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.380+0000 7fbafdfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaf8101d30 0x7fbaf8073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:34.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.380+0000 7fbafdfff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf8073bc0 con 0x7fbaf81013a0 2023-11-16T16:47:34.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.380+0000 7fbafdfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf8073d30 con 0x7fbaf8101d30 2023-11-16T16:47:34.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.380+0000 7fbafdfff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf819ffd0 con 0x7fbaf81001b0 2023-11-16T16:47:34.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.380+0000 7fbaeffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf81013a0 0x7fbaf8072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:34.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.380+0000 7fbaf77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf81001b0 0x7fbaf80748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:34.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.381+0000 7fbaf77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf81001b0 0x7fbaf80748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:59528/0 (socket says 172.21.15.44:59528) 2023-11-16T16:47:34.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.381+0000 7fbaf77fe700 1 -- 172.21.15.44:0/147874742 learned_addr learned my addr 172.21.15.44:0/147874742 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:34.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.381+0000 7fbaf7fff700 1 --2- 172.21.15.44:0/147874742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaf8101d30 0x7fbaf8073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:34.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.381+0000 7fbaeffff700 1 -- 172.21.15.44:0/147874742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf81001b0 msgr2=0x7fbaf80748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:34.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.381+0000 7fbaeffff700 1 --2- 172.21.15.44:0/147874742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf81001b0 0x7fbaf80748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:34.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.381+0000 7fbaeffff700 1 -- 172.21.15.44:0/147874742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaf8101d30 msgr2=0x7fbaf8073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:34.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.381+0000 7fbaeffff700 1 --2- 172.21.15.44:0/147874742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaf8101d30 0x7fbaf8073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:34.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.381+0000 7fbaeffff700 1 -- 172.21.15.44:0/147874742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae80095f0 con 0x7fbaf81013a0 2023-11-16T16:47:34.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.381+0000 7fbaeffff700 1 --2- 172.21.15.44:0/147874742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf81013a0 0x7fbaf8072ee0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbae000d950 tx=0x7fbae000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:34.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.381+0000 7fbaf57fa700 1 -- 172.21.15.44:0/147874742 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae000f890 con 0x7fbaf81013a0 2023-11-16T16:47:34.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.381+0000 7fbafdfff700 1 -- 172.21.15.44:0/147874742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf810bb70 con 0x7fbaf81013a0 2023-11-16T16:47:34.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.381+0000 7fbaf57fa700 1 -- 172.21.15.44:0/147874742 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbae0010460 con 0x7fbaf81013a0 2023-11-16T16:47:34.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.381+0000 7fbaf57fa700 1 -- 172.21.15.44:0/147874742 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae000e610 con 0x7fbaf81013a0 2023-11-16T16:47:34.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.382+0000 7fbafdfff700 1 -- 172.21.15.44:0/147874742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf810b9e0 con 0x7fbaf81013a0 2023-11-16T16:47:34.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.383+0000 7fbafdfff700 1 -- 172.21.15.44:0/147874742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf8107740 con 0x7fbaf81013a0 2023-11-16T16:47:34.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.385+0000 7fbaf57fa700 1 -- 172.21.15.44:0/147874742 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbae00108d0 con 0x7fbaf81013a0 2023-11-16T16:47:34.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.386+0000 7fbaf57fa700 1 --2- 172.21.15.44:0/147874742 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbae40a11d0 0x7fbae40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:34.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.386+0000 7fbaf57fa700 1 -- 172.21.15.44:0/147874742 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbae00cc770 con 0x7fbaf81013a0 2023-11-16T16:47:34.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.386+0000 7fbaf77fe700 1 --2- 172.21.15.44:0/147874742 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbae40a11d0 0x7fbae40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:34.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.387+0000 7fbaf77fe700 1 --2- 172.21.15.44:0/147874742 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbae40a11d0 0x7fbae40a3680 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbae8006010 tx=0x7fbae800b260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:34.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.390+0000 7fbaf57fa700 1 -- 172.21.15.44:0/147874742 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbae0095c70 con 0x7fbaf81013a0 2023-11-16T16:47:34.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.603+0000 7fbafdfff700 1 -- 172.21.15.44:0/147874742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbaf802cfe0 con 0x7fbaf81013a0 2023-11-16T16:47:34.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.605+0000 7fbaf57fa700 1 -- 172.21.15.44:0/147874742 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23925 (secure 0 0 0) 0x7fbae0099290 con 0x7fbaf81013a0 2023-11-16T16:47:34.607 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:34.607 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":87,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:47:23.158961+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:34.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.607+0000 7fbafdfff700 1 -- 172.21.15.44:0/147874742 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbae40a11d0 msgr2=0x7fbae40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:34.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.607+0000 7fbafdfff700 1 --2- 172.21.15.44:0/147874742 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbae40a11d0 0x7fbae40a3680 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbae8006010 tx=0x7fbae800b260 comp rx=0 tx=0).stop 2023-11-16T16:47:34.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.607+0000 7fbafdfff700 1 -- 172.21.15.44:0/147874742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf81013a0 msgr2=0x7fbaf8072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:34.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.607+0000 7fbafdfff700 1 --2- 172.21.15.44:0/147874742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf81013a0 0x7fbaf8072ee0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbae000d950 tx=0x7fbae000dc60 comp rx=0 tx=0).stop 2023-11-16T16:47:34.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.607+0000 7fbafdfff700 1 -- 172.21.15.44:0/147874742 shutdown_connections 2023-11-16T16:47:34.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.607+0000 7fbafdfff700 1 --2- 172.21.15.44:0/147874742 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbae40a11d0 0x7fbae40a3680 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:34.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.607+0000 7fbafdfff700 1 --2- 172.21.15.44:0/147874742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaf8101d30 0x7fbaf8073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:34.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.607+0000 7fbafdfff700 1 --2- 172.21.15.44:0/147874742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf81001b0 0x7fbaf80748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:34.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.607+0000 7fbafdfff700 1 --2- 172.21.15.44:0/147874742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf81013a0 0x7fbaf8072ee0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:34.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.607+0000 7fbafdfff700 1 -- 172.21.15.44:0/147874742 >> 172.21.15.44:0/147874742 conn(0x7fbaf80fb790 msgr2=0x7fbaf8104880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:34.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.608+0000 7fbafdfff700 1 -- 172.21.15.44:0/147874742 shutdown_connections 2023-11-16T16:47:34.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.608+0000 7fbafdfff700 1 -- 172.21.15.44:0/147874742 wait complete. 2023-11-16T16:47:34.628 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:03.059622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:16.211826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:17.601591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:21.136084+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T16:47:34.629 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:47:34.837 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.835+0000 7f2b153ff700 1 -- 172.21.15.44:0/3695558485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b101002c0 msgr2=0x7f2b101006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:34.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.835+0000 7f2b153ff700 1 --2- 172.21.15.44:0/3695558485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b101002c0 0x7f2b101006d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2b000098f0 tx=0x7f2b00009c00 comp rx=0 tx=0).stop 2023-11-16T16:47:34.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.835+0000 7f2b153ff700 1 -- 172.21.15.44:0/3695558485 shutdown_connections 2023-11-16T16:47:34.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.835+0000 7f2b153ff700 1 --2- 172.21.15.44:0/3695558485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b10101e40 0x7f2b10108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:34.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.835+0000 7f2b153ff700 1 --2- 172.21.15.44:0/3695558485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b101002c0 0x7f2b101006d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:34.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.835+0000 7f2b153ff700 1 --2- 172.21.15.44:0/3695558485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b101014b0 0x7f2b10101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:34.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.835+0000 7f2b153ff700 1 -- 172.21.15.44:0/3695558485 >> 172.21.15.44:0/3695558485 conn(0x7f2b100fb8a0 msgr2=0x7f2b100fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:34.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.835+0000 7f2b153ff700 1 -- 172.21.15.44:0/3695558485 shutdown_connections 2023-11-16T16:47:34.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.836+0000 7f2b153ff700 1 -- 172.21.15.44:0/3695558485 wait complete. 2023-11-16T16:47:34.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.836+0000 7f2b153ff700 1 Processor -- start 2023-11-16T16:47:34.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.837+0000 7f2b153ff700 1 -- start start 2023-11-16T16:47:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.837+0000 7f2b153ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b101002c0 0x7f2b101982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.837+0000 7f2b153ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b101014b0 0x7f2b10198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.837+0000 7f2b153ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b10101e40 0x7f2b1019f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.837+0000 7f2b153ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1019fe60 con 0x7f2b10101e40 2023-11-16T16:47:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.837+0000 7f2b153ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1019ffa0 con 0x7f2b101014b0 2023-11-16T16:47:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.837+0000 7f2b153ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b101a0130 con 0x7f2b101002c0 2023-11-16T16:47:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.837+0000 7f2b0f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b10101e40 0x7f2b1019f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.837+0000 7f2b0f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b10101e40 0x7f2b1019f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39950/0 (socket says 172.21.15.44:39950) 2023-11-16T16:47:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.837+0000 7f2b0effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b101002c0 0x7f2b101982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.837+0000 7f2b0f7fe700 1 -- 172.21.15.44:0/1625737134 learned_addr learned my addr 172.21.15.44:0/1625737134 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.837+0000 7f2b0e7fc700 1 --2- 172.21.15.44:0/1625737134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b101014b0 0x7f2b10198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.838+0000 7f2b0f7fe700 1 -- 172.21.15.44:0/1625737134 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b101002c0 msgr2=0x7f2b101982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.838+0000 7f2b0f7fe700 1 --2- 172.21.15.44:0/1625737134 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b101002c0 0x7f2b101982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.838+0000 7f2b0f7fe700 1 -- 172.21.15.44:0/1625737134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b101014b0 msgr2=0x7f2b10198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.838+0000 7f2b0f7fe700 1 --2- 172.21.15.44:0/1625737134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b101014b0 0x7f2b10198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.838+0000 7f2b0f7fe700 1 -- 172.21.15.44:0/1625737134 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b040095f0 con 0x7f2b10101e40 2023-11-16T16:47:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.838+0000 7f2b0f7fe700 1 --2- 172.21.15.44:0/1625737134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b10101e40 0x7f2b1019f890 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2b0800ef30 tx=0x7f2b0800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.838+0000 7f2af7fff700 1 -- 172.21.15.44:0/1625737134 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b08018070 con 0x7f2b10101e40 2023-11-16T16:47:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.838+0000 7f2af7fff700 1 -- 172.21.15.44:0/1625737134 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b0800f040 con 0x7f2b10101e40 2023-11-16T16:47:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.838+0000 7f2b153ff700 1 -- 172.21.15.44:0/1625737134 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b00009520 con 0x7f2b10101e40 2023-11-16T16:47:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.838+0000 7f2af7fff700 1 -- 172.21.15.44:0/1625737134 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0800cd00 con 0x7f2b10101e40 2023-11-16T16:47:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.838+0000 7f2b153ff700 1 -- 172.21.15.44:0/1625737134 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1010bc90 con 0x7f2b10101e40 2023-11-16T16:47:34.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.839+0000 7f2b153ff700 1 -- 172.21.15.44:0/1625737134 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b10067b00 con 0x7f2b10101e40 2023-11-16T16:47:34.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.841+0000 7f2af7fff700 1 -- 172.21.15.44:0/1625737134 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2b08003a00 con 0x7f2b10101e40 2023-11-16T16:47:34.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.843+0000 7f2af7fff700 1 --2- 172.21.15.44:0/1625737134 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2af80a11d0 0x7f2af80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:34.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.843+0000 7f2af7fff700 1 -- 172.21.15.44:0/1625737134 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2b080cc600 con 0x7f2b10101e40 2023-11-16T16:47:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.843+0000 7f2b0effd700 1 --2- 172.21.15.44:0/1625737134 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2af80a11d0 0x7f2af80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.843+0000 7f2b0effd700 1 --2- 172.21.15.44:0/1625737134 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2af80a11d0 0x7f2af80a3680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009fd0 tx=0x7f2b00019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:34.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:34.846+0000 7f2af7fff700 1 -- 172.21.15.44:0/1625737134 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2b08095b90 con 0x7f2b10101e40 2023-11-16T16:47:35.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.055+0000 7f2b153ff700 1 -- 172.21.15.44:0/1625737134 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2b101a0c30 con 0x7f2b10101e40 2023-11-16T16:47:35.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.055+0000 7f2af7fff700 1 -- 172.21.15.44:0/1625737134 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2b080991b0 con 0x7f2b10101e40 2023-11-16T16:47:35.057 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2023-11-16T16:47:35.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.057+0000 7f2b153ff700 1 -- 172.21.15.44:0/1625737134 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2af80a11d0 msgr2=0x7f2af80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:35.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.057+0000 7f2b153ff700 1 --2- 172.21.15.44:0/1625737134 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2af80a11d0 0x7f2af80a3680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009fd0 tx=0x7f2b00019040 comp rx=0 tx=0).stop 2023-11-16T16:47:35.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.057+0000 7f2b153ff700 1 -- 172.21.15.44:0/1625737134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b10101e40 msgr2=0x7f2b1019f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:35.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.057+0000 7f2b153ff700 1 --2- 172.21.15.44:0/1625737134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b10101e40 0x7f2b1019f890 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2b0800ef30 tx=0x7f2b0800c3a0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.058+0000 7f2b153ff700 1 -- 172.21.15.44:0/1625737134 shutdown_connections 2023-11-16T16:47:35.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.058+0000 7f2b153ff700 1 --2- 172.21.15.44:0/1625737134 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2af80a11d0 0x7f2af80a3680 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.058+0000 7f2b153ff700 1 --2- 172.21.15.44:0/1625737134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b10101e40 0x7f2b1019f890 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.058+0000 7f2b153ff700 1 --2- 172.21.15.44:0/1625737134 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b101002c0 0x7f2b101982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.058+0000 7f2b153ff700 1 --2- 172.21.15.44:0/1625737134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b101014b0 0x7f2b10198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.058+0000 7f2b153ff700 1 -- 172.21.15.44:0/1625737134 >> 172.21.15.44:0/1625737134 conn(0x7f2b100fb8a0 msgr2=0x7f2b100fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:35.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.058+0000 7f2b153ff700 1 -- 172.21.15.44:0/1625737134 shutdown_connections 2023-11-16T16:47:35.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.058+0000 7f2b153ff700 1 -- 172.21.15.44:0/1625737134 wait complete. 2023-11-16T16:47:35.078 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T16:47:35.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.283+0000 7f850adff700 1 -- 172.21.15.44:0/2648308674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85041002c0 msgr2=0x7f85041006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:35.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.283+0000 7f850adff700 1 --2- 172.21.15.44:0/2648308674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85041002c0 0x7f85041006d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009990 tx=0x7f84f8009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.283+0000 7f850adff700 1 -- 172.21.15.44:0/2648308674 shutdown_connections 2023-11-16T16:47:35.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.283+0000 7f850adff700 1 --2- 172.21.15.44:0/2648308674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8504101e40 0x7f8504108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.283+0000 7f850adff700 1 --2- 172.21.15.44:0/2648308674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85041002c0 0x7f85041006d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.283+0000 7f850adff700 1 --2- 172.21.15.44:0/2648308674 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85041014b0 0x7f8504101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.283+0000 7f850adff700 1 -- 172.21.15.44:0/2648308674 >> 172.21.15.44:0/2648308674 conn(0x7f85040fb8a0 msgr2=0x7f85040fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:35.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.283+0000 7f850adff700 1 -- 172.21.15.44:0/2648308674 shutdown_connections 2023-11-16T16:47:35.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.283+0000 7f850adff700 1 -- 172.21.15.44:0/2648308674 wait complete. 2023-11-16T16:47:35.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.285+0000 7f850adff700 1 Processor -- start 2023-11-16T16:47:35.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.285+0000 7f850adff700 1 -- start start 2023-11-16T16:47:35.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.286+0000 7f850adff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85041002c0 0x7f85040748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:35.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.286+0000 7f850adff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85041014b0 0x7f8504072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:35.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.286+0000 7f850adff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8504101e40 0x7f8504073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:35.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.286+0000 7f850adff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8504073bc0 con 0x7f85041014b0 2023-11-16T16:47:35.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.286+0000 7f850adff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8504073d30 con 0x7f8504101e40 2023-11-16T16:47:35.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.286+0000 7f850adff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85041a0170 con 0x7f85041002c0 2023-11-16T16:47:35.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.286+0000 7f8503fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85041014b0 0x7f8504072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:35.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.286+0000 7f8503fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85041014b0 0x7f8504072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46272/0 (socket says 172.21.15.44:46272) 2023-11-16T16:47:35.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.286+0000 7f8503fff700 1 -- 172.21.15.44:0/432128370 learned_addr learned my addr 172.21.15.44:0/432128370 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:35.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.286+0000 7f8508b9b700 1 --2- 172.21.15.44:0/432128370 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85041002c0 0x7f85040748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:35.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.286+0000 7f850939c700 1 --2- 172.21.15.44:0/432128370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8504101e40 0x7f8504073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:35.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.286+0000 7f8503fff700 1 -- 172.21.15.44:0/432128370 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85041002c0 msgr2=0x7f85040748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:35.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.286+0000 7f8503fff700 1 --2- 172.21.15.44:0/432128370 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85041002c0 0x7f85040748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.286+0000 7f8503fff700 1 -- 172.21.15.44:0/432128370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8504101e40 msgr2=0x7f8504073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:35.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.286+0000 7f8503fff700 1 --2- 172.21.15.44:0/432128370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8504101e40 0x7f8504073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.286+0000 7f8503fff700 1 -- 172.21.15.44:0/432128370 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f80095f0 con 0x7f85041014b0 2023-11-16T16:47:35.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.287+0000 7f8503fff700 1 --2- 172.21.15.44:0/432128370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85041014b0 0x7f8504072ee0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f84f000d950 tx=0x7f84f000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:35.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.287+0000 7f8501ffb700 1 -- 172.21.15.44:0/432128370 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f000f890 con 0x7f85041014b0 2023-11-16T16:47:35.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.287+0000 7f8501ffb700 1 -- 172.21.15.44:0/432128370 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84f0010460 con 0x7f85041014b0 2023-11-16T16:47:35.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.287+0000 7f850adff700 1 -- 172.21.15.44:0/432128370 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850410bc90 con 0x7f85041014b0 2023-11-16T16:47:35.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.287+0000 7f8501ffb700 1 -- 172.21.15.44:0/432128370 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f000e610 con 0x7f85041014b0 2023-11-16T16:47:35.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.287+0000 7f850adff700 1 -- 172.21.15.44:0/432128370 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850410bb00 con 0x7f85041014b0 2023-11-16T16:47:35.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.288+0000 7f850adff700 1 -- 172.21.15.44:0/432128370 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85041a08b0 con 0x7f85041014b0 2023-11-16T16:47:35.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.290+0000 7f8501ffb700 1 -- 172.21.15.44:0/432128370 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f84f00108d0 con 0x7f85041014b0 2023-11-16T16:47:35.295 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.293+0000 7f8501ffb700 1 --2- 172.21.15.44:0/432128370 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f84f40a11d0 0x7f84f40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:35.295 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.293+0000 7f8501ffb700 1 -- 172.21.15.44:0/432128370 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f84f00cc870 con 0x7f85041014b0 2023-11-16T16:47:35.295 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.293+0000 7f8508b9b700 1 --2- 172.21.15.44:0/432128370 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f84f40a11d0 0x7f84f40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:35.295 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.294+0000 7f8508b9b700 1 --2- 172.21.15.44:0/432128370 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f84f40a11d0 0x7f84f40a3680 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f84f800b5c0 tx=0x7f84f8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:35.297 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.295+0000 7f8501ffb700 1 -- 172.21.15.44:0/432128370 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f84f0095dd0 con 0x7f85041014b0 2023-11-16T16:47:35.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.504+0000 7f850adff700 1 -- 172.21.15.44:0/432128370 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8504107220 con 0x7f85041014b0 2023-11-16T16:47:35.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.505+0000 7f8501ffb700 1 -- 172.21.15.44:0/432128370 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f84f0020020 con 0x7f85041014b0 2023-11-16T16:47:35.507 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:47:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.507+0000 7f850adff700 1 -- 172.21.15.44:0/432128370 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f84f40a11d0 msgr2=0x7f84f40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.507+0000 7f850adff700 1 --2- 172.21.15.44:0/432128370 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f84f40a11d0 0x7f84f40a3680 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f84f800b5c0 tx=0x7f84f8005fd0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.507+0000 7f850adff700 1 -- 172.21.15.44:0/432128370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85041014b0 msgr2=0x7f8504072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.507+0000 7f850adff700 1 --2- 172.21.15.44:0/432128370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85041014b0 0x7f8504072ee0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f84f000d950 tx=0x7f84f000dc60 comp rx=0 tx=0).stop 2023-11-16T16:47:35.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.507+0000 7f850adff700 1 -- 172.21.15.44:0/432128370 shutdown_connections 2023-11-16T16:47:35.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.507+0000 7f850adff700 1 --2- 172.21.15.44:0/432128370 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f84f40a11d0 0x7f84f40a3680 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.507+0000 7f850adff700 1 --2- 172.21.15.44:0/432128370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8504101e40 0x7f8504073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.507+0000 7f850adff700 1 --2- 172.21.15.44:0/432128370 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85041002c0 0x7f85040748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.507+0000 7f850adff700 1 --2- 172.21.15.44:0/432128370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85041014b0 0x7f8504072ee0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.507+0000 7f850adff700 1 -- 172.21.15.44:0/432128370 >> 172.21.15.44:0/432128370 conn(0x7f85040fb8a0 msgr2=0x7f85040fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:35.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.507+0000 7f850adff700 1 -- 172.21.15.44:0/432128370 shutdown_connections 2023-11-16T16:47:35.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.507+0000 7f850adff700 1 -- 172.21.15.44:0/432128370 wait complete. 2023-11-16T16:47:35.528 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T16:47:35.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.734+0000 7f57641ff700 1 -- 172.21.15.44:0/1416187308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f575c1002d0 msgr2=0x7f575c1006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:35.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.734+0000 7f57641ff700 1 --2- 172.21.15.44:0/1416187308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f575c1002d0 0x7f575c1006e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5758009990 tx=0x7f5758009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.734+0000 7f57641ff700 1 -- 172.21.15.44:0/1416187308 shutdown_connections 2023-11-16T16:47:35.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.734+0000 7f57641ff700 1 --2- 172.21.15.44:0/1416187308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c101e50 0x7f575c108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.734+0000 7f57641ff700 1 --2- 172.21.15.44:0/1416187308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f575c1002d0 0x7f575c1006e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.734+0000 7f57641ff700 1 --2- 172.21.15.44:0/1416187308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f575c1014c0 0x7f575c101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.734+0000 7f57641ff700 1 -- 172.21.15.44:0/1416187308 >> 172.21.15.44:0/1416187308 conn(0x7f575c0fb8b0 msgr2=0x7f575c0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:35.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.734+0000 7f57641ff700 1 -- 172.21.15.44:0/1416187308 shutdown_connections 2023-11-16T16:47:35.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.734+0000 7f57641ff700 1 -- 172.21.15.44:0/1416187308 wait complete. 2023-11-16T16:47:35.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.735+0000 7f57641ff700 1 Processor -- start 2023-11-16T16:47:35.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.735+0000 7f57641ff700 1 -- start start 2023-11-16T16:47:35.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.735+0000 7f57641ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f575c1002d0 0x7f575c198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:35.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.735+0000 7f57641ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c1014c0 0x7f575c198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:35.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.735+0000 7f57641ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f575c101e50 0x7f575c19f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:35.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.736+0000 7f57641ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c19fe80 con 0x7f575c1002d0 2023-11-16T16:47:35.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.736+0000 7f57641ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c19ffc0 con 0x7f575c1014c0 2023-11-16T16:47:35.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.736+0000 7f57641ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c1a0150 con 0x7f575c101e50 2023-11-16T16:47:35.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.736+0000 7f5761f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f575c1002d0 0x7f575c198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:35.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.736+0000 7f5761f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f575c1002d0 0x7f575c198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46280/0 (socket says 172.21.15.44:46280) 2023-11-16T16:47:35.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.736+0000 7f5761f9b700 1 -- 172.21.15.44:0/1451274488 learned_addr learned my addr 172.21.15.44:0/1451274488 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:35.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.736+0000 7f576179a700 1 --2- 172.21.15.44:0/1451274488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c1014c0 0x7f575c198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:35.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.736+0000 7f576279c700 1 --2- 172.21.15.44:0/1451274488 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f575c101e50 0x7f575c19f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:35.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.736+0000 7f5761f9b700 1 -- 172.21.15.44:0/1451274488 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f575c101e50 msgr2=0x7f575c19f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:35.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.736+0000 7f5761f9b700 1 --2- 172.21.15.44:0/1451274488 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f575c101e50 0x7f575c19f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.736+0000 7f5761f9b700 1 -- 172.21.15.44:0/1451274488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c1014c0 msgr2=0x7f575c198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:35.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.736+0000 7f5761f9b700 1 --2- 172.21.15.44:0/1451274488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c1014c0 0x7f575c198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.736+0000 7f5761f9b700 1 -- 172.21.15.44:0/1451274488 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5754009c40 con 0x7f575c1002d0 2023-11-16T16:47:35.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.736+0000 7f5761f9b700 1 --2- 172.21.15.44:0/1451274488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f575c1002d0 0x7f575c198300 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5758004730 tx=0x7f5758004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:35.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.737+0000 7f5752ffd700 1 -- 172.21.15.44:0/1451274488 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575801d070 con 0x7f575c1002d0 2023-11-16T16:47:35.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.737+0000 7f5752ffd700 1 -- 172.21.15.44:0/1451274488 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5758021410 con 0x7f575c1002d0 2023-11-16T16:47:35.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.737+0000 7f57641ff700 1 -- 172.21.15.44:0/1451274488 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57580095f0 con 0x7f575c1002d0 2023-11-16T16:47:35.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.737+0000 7f5752ffd700 1 -- 172.21.15.44:0/1451274488 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758017650 con 0x7f575c1002d0 2023-11-16T16:47:35.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.737+0000 7f57641ff700 1 -- 172.21.15.44:0/1451274488 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c10bca0 con 0x7f575c1002d0 2023-11-16T16:47:35.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.738+0000 7f57641ff700 1 -- 172.21.15.44:0/1451274488 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c10be30 con 0x7f575c1002d0 2023-11-16T16:47:35.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.740+0000 7f5752ffd700 1 -- 172.21.15.44:0/1451274488 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5758021810 con 0x7f575c1002d0 2023-11-16T16:47:35.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.741+0000 7f5752ffd700 1 --2- 172.21.15.44:0/1451274488 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f57480a11d0 0x7f57480a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:35.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.741+0000 7f5752ffd700 1 -- 172.21.15.44:0/1451274488 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f57580ce590 con 0x7f575c1002d0 2023-11-16T16:47:35.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.742+0000 7f576179a700 1 --2- 172.21.15.44:0/1451274488 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f57480a11d0 0x7f57480a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:35.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.742+0000 7f576179a700 1 --2- 172.21.15.44:0/1451274488 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f57480a11d0 0x7f57480a3680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f574c009c60 tx=0x7f574c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:35.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.743+0000 7f5752ffd700 1 -- 172.21.15.44:0/1451274488 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5758097b20 con 0x7f575c1002d0 2023-11-16T16:47:35.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:35 smithi044 ceph-mon[128750]: pgmap v283: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T16:47:35.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:35 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/147874742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:35.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:35 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1625737134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:35.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:35 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/432128370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:47:35.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:35 smithi176 ceph-mon[131630]: pgmap v283: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T16:47:35.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:35 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/147874742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:35.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:35 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1625737134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:35.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:35 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/432128370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:47:35.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.954+0000 7f57641ff700 1 -- 172.21.15.44:0/1451274488 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f575c192a00 con 0x7f575c1002d0 2023-11-16T16:47:35.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.954+0000 7f5752ffd700 1 -- 172.21.15.44:0/1451274488 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f575809b140 con 0x7f575c1002d0 2023-11-16T16:47:35.956 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2023-11-16T16:47:35.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.957+0000 7f57641ff700 1 -- 172.21.15.44:0/1451274488 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f57480a11d0 msgr2=0x7f57480a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:35.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.957+0000 7f57641ff700 1 --2- 172.21.15.44:0/1451274488 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f57480a11d0 0x7f57480a3680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f574c009c60 tx=0x7f574c009260 comp rx=0 tx=0).stop 2023-11-16T16:47:35.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.957+0000 7f57641ff700 1 -- 172.21.15.44:0/1451274488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f575c1002d0 msgr2=0x7f575c198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:35.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.957+0000 7f57641ff700 1 --2- 172.21.15.44:0/1451274488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f575c1002d0 0x7f575c198300 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5758004730 tx=0x7f5758004760 comp rx=0 tx=0).stop 2023-11-16T16:47:35.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.957+0000 7f57641ff700 1 -- 172.21.15.44:0/1451274488 shutdown_connections 2023-11-16T16:47:35.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.957+0000 7f57641ff700 1 --2- 172.21.15.44:0/1451274488 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f57480a11d0 0x7f57480a3680 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.957+0000 7f57641ff700 1 --2- 172.21.15.44:0/1451274488 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f575c101e50 0x7f575c19f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.957+0000 7f57641ff700 1 --2- 172.21.15.44:0/1451274488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f575c1002d0 0x7f575c198300 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.957+0000 7f57641ff700 1 --2- 172.21.15.44:0/1451274488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c1014c0 0x7f575c198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:35.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.957+0000 7f57641ff700 1 -- 172.21.15.44:0/1451274488 >> 172.21.15.44:0/1451274488 conn(0x7f575c0fb8b0 msgr2=0x7f575c106e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:35.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.957+0000 7f57641ff700 1 -- 172.21.15.44:0/1451274488 shutdown_connections 2023-11-16T16:47:35.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:35.957+0000 7f57641ff700 1 -- 172.21.15.44:0/1451274488 wait complete. 2023-11-16T16:47:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:35 smithi077 ceph-mon[131204]: pgmap v283: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T16:47:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:35 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/147874742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:35 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1625737134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:35 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/432128370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:47:35.977 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T16:47:36.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.181+0000 7fe3d71ff700 1 -- 172.21.15.44:0/467060379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3d0102510 msgr2=0x7fe3d0102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:36.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.181+0000 7fe3d71ff700 1 --2- 172.21.15.44:0/467060379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3d0102510 0x7fe3d0102920 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4009990 tx=0x7fe3c4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.181+0000 7fe3d71ff700 1 -- 172.21.15.44:0/467060379 shutdown_connections 2023-11-16T16:47:36.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.181+0000 7fe3d71ff700 1 --2- 172.21.15.44:0/467060379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3d0104090 0x7fe3d010a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.181+0000 7fe3d71ff700 1 --2- 172.21.15.44:0/467060379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3d0102510 0x7fe3d0102920 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.181+0000 7fe3d71ff700 1 --2- 172.21.15.44:0/467060379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3d0103700 0x7fe3d0103b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.181+0000 7fe3d71ff700 1 -- 172.21.15.44:0/467060379 >> 172.21.15.44:0/467060379 conn(0x7fe3d00fdb30 msgr2=0x7fe3d00fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:36.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.181+0000 7fe3d71ff700 1 -- 172.21.15.44:0/467060379 shutdown_connections 2023-11-16T16:47:36.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.182+0000 7fe3d71ff700 1 -- 172.21.15.44:0/467060379 wait complete. 2023-11-16T16:47:36.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.183+0000 7fe3d71ff700 1 Processor -- start 2023-11-16T16:47:36.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.183+0000 7fe3d71ff700 1 -- start start 2023-11-16T16:47:36.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3d71ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3d0102510 0x7fe3d0198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:36.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3d71ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3d0103700 0x7fe3d01987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:36.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3d71ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3d0104090 0x7fe3d019f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:36.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3d71ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d019fdf0 con 0x7fe3d0102510 2023-11-16T16:47:36.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3d71ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d019ff30 con 0x7fe3d0104090 2023-11-16T16:47:36.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3d71ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d01a00c0 con 0x7fe3d0103700 2023-11-16T16:47:36.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3d4f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3d0102510 0x7fe3d0198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:36.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3d4f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3d0102510 0x7fe3d0198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46298/0 (socket says 172.21.15.44:46298) 2023-11-16T16:47:36.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3d4f9b700 1 -- 172.21.15.44:0/1629889111 learned_addr learned my addr 172.21.15.44:0/1629889111 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:36.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3cffff700 1 --2- 172.21.15.44:0/1629889111 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3d0103700 0x7fe3d01987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:36.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3d579c700 1 --2- 172.21.15.44:0/1629889111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3d0104090 0x7fe3d019f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:36.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3cffff700 1 -- 172.21.15.44:0/1629889111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3d0104090 msgr2=0x7fe3d019f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:36.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3cffff700 1 --2- 172.21.15.44:0/1629889111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3d0104090 0x7fe3d019f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3cffff700 1 -- 172.21.15.44:0/1629889111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3d0102510 msgr2=0x7fe3d0198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:36.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3cffff700 1 --2- 172.21.15.44:0/1629889111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3d0102510 0x7fe3d0198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3cffff700 1 -- 172.21.15.44:0/1629889111 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c40095f0 con 0x7fe3d0103700 2023-11-16T16:47:36.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.184+0000 7fe3d4f9b700 1 --2- 172.21.15.44:0/1629889111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3d0102510 0x7fe3d0198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:36.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.185+0000 7fe3cffff700 1 --2- 172.21.15.44:0/1629889111 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3d0103700 0x7fe3d01987b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00e9f0 tx=0x7fe3bc00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:36.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.185+0000 7fe3cdffb700 1 -- 172.21.15.44:0/1629889111 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc00c9c0 con 0x7fe3d0103700 2023-11-16T16:47:36.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.185+0000 7fe3d71ff700 1 -- 172.21.15.44:0/1629889111 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d010ded0 con 0x7fe3d0103700 2023-11-16T16:47:36.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.185+0000 7fe3cdffb700 1 -- 172.21.15.44:0/1629889111 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3bc010430 con 0x7fe3d0103700 2023-11-16T16:47:36.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.185+0000 7fe3cdffb700 1 -- 172.21.15.44:0/1629889111 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc018680 con 0x7fe3d0103700 2023-11-16T16:47:36.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.185+0000 7fe3d71ff700 1 -- 172.21.15.44:0/1629889111 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d010dd40 con 0x7fe3d0103700 2023-11-16T16:47:36.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.186+0000 7fe3d71ff700 1 -- 172.21.15.44:0/1629889111 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d0109a00 con 0x7fe3d0103700 2023-11-16T16:47:36.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.188+0000 7fe3cdffb700 1 -- 172.21.15.44:0/1629889111 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe3bc010860 con 0x7fe3d0103700 2023-11-16T16:47:36.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.190+0000 7fe3cdffb700 1 --2- 172.21.15.44:0/1629889111 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe3c00a11d0 0x7fe3c00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:36.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.190+0000 7fe3cdffb700 1 -- 172.21.15.44:0/1629889111 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe3bc014070 con 0x7fe3d0103700 2023-11-16T16:47:36.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.190+0000 7fe3d4f9b700 1 --2- 172.21.15.44:0/1629889111 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe3c00a11d0 0x7fe3c00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:36.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.190+0000 7fe3d4f9b700 1 --2- 172.21.15.44:0/1629889111 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe3c00a11d0 0x7fe3c00a3680 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400b2e0 tx=0x7fe3c4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:36.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.192+0000 7fe3cdffb700 1 -- 172.21.15.44:0/1629889111 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe3bc0960b0 con 0x7fe3d0103700 2023-11-16T16:47:36.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.405+0000 7fe3d71ff700 1 -- 172.21.15.44:0/1629889111 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe3d0067b00 con 0x7fe3d0103700 2023-11-16T16:47:36.407 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.405+0000 7fe3cdffb700 1 -- 172.21.15.44:0/1629889111 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe3bc0996d0 con 0x7fe3d0103700 2023-11-16T16:47:36.407 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.408+0000 7fe3d71ff700 1 -- 172.21.15.44:0/1629889111 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe3c00a11d0 msgr2=0x7fe3c00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.408+0000 7fe3d71ff700 1 --2- 172.21.15.44:0/1629889111 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe3c00a11d0 0x7fe3c00a3680 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400b2e0 tx=0x7fe3c4005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.408+0000 7fe3d71ff700 1 -- 172.21.15.44:0/1629889111 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3d0103700 msgr2=0x7fe3d01987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.408+0000 7fe3d71ff700 1 --2- 172.21.15.44:0/1629889111 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3d0103700 0x7fe3d01987b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00e9f0 tx=0x7fe3bc00ed00 comp rx=0 tx=0).stop 2023-11-16T16:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.408+0000 7fe3d71ff700 1 -- 172.21.15.44:0/1629889111 shutdown_connections 2023-11-16T16:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.408+0000 7fe3d71ff700 1 --2- 172.21.15.44:0/1629889111 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe3c00a11d0 0x7fe3c00a3680 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.408+0000 7fe3d71ff700 1 --2- 172.21.15.44:0/1629889111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3d0104090 0x7fe3d019f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.408+0000 7fe3d71ff700 1 --2- 172.21.15.44:0/1629889111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3d0102510 0x7fe3d0198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.408+0000 7fe3d71ff700 1 --2- 172.21.15.44:0/1629889111 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3d0103700 0x7fe3d01987b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.408+0000 7fe3d71ff700 1 -- 172.21.15.44:0/1629889111 >> 172.21.15.44:0/1629889111 conn(0x7fe3d00fdb30 msgr2=0x7fe3d0106b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:36.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.408+0000 7fe3d71ff700 1 -- 172.21.15.44:0/1629889111 shutdown_connections 2023-11-16T16:47:36.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.408+0000 7fe3d71ff700 1 -- 172.21.15.44:0/1629889111 wait complete. 2023-11-16T16:47:36.428 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.632+0000 7f7a737ff700 1 -- 172.21.15.44:0/940406039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c100230 msgr2=0x7f7a6c100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.632+0000 7f7a737ff700 1 --2- 172.21.15.44:0/940406039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c100230 0x7f7a6c100640 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009990 tx=0x7f7a68009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.633+0000 7f7a737ff700 1 -- 172.21.15.44:0/940406039 shutdown_connections 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.633+0000 7f7a737ff700 1 --2- 172.21.15.44:0/940406039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a6c101db0 0x7f7a6c108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.633+0000 7f7a737ff700 1 --2- 172.21.15.44:0/940406039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c100230 0x7f7a6c100640 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.633+0000 7f7a737ff700 1 --2- 172.21.15.44:0/940406039 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a6c101420 0x7f7a6c101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.633+0000 7f7a737ff700 1 -- 172.21.15.44:0/940406039 >> 172.21.15.44:0/940406039 conn(0x7f7a6c0fb810 msgr2=0x7f7a6c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.633+0000 7f7a737ff700 1 -- 172.21.15.44:0/940406039 shutdown_connections 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.633+0000 7f7a737ff700 1 -- 172.21.15.44:0/940406039 wait complete. 2023-11-16T16:47:36.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.634+0000 7f7a737ff700 1 Processor -- start 2023-11-16T16:47:36.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.634+0000 7f7a737ff700 1 -- start start 2023-11-16T16:47:36.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.634+0000 7f7a737ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c100230 0x7f7a6c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:36.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.634+0000 7f7a737ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a6c101420 0x7f7a6c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:36.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.634+0000 7f7a737ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a6c101db0 0x7f7a6c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:36.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.634+0000 7f7a737ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c073bc0 con 0x7f7a6c100230 2023-11-16T16:47:36.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.634+0000 7f7a737ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c073d30 con 0x7f7a6c101db0 2023-11-16T16:47:36.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.634+0000 7f7a737ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c1a0040 con 0x7f7a6c101420 2023-11-16T16:47:36.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.634+0000 7f7a7159b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c100230 0x7f7a6c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:36.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.634+0000 7f7a7159b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c100230 0x7f7a6c0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46322/0 (socket says 172.21.15.44:46322) 2023-11-16T16:47:36.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.634+0000 7f7a7159b700 1 -- 172.21.15.44:0/530744673 learned_addr learned my addr 172.21.15.44:0/530744673 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:36.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.634+0000 7f7a70d9a700 1 --2- 172.21.15.44:0/530744673 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a6c101420 0x7f7a6c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:36.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.635+0000 7f7a71d9c700 1 --2- 172.21.15.44:0/530744673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a6c101db0 0x7f7a6c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:36.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.635+0000 7f7a7159b700 1 -- 172.21.15.44:0/530744673 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a6c101420 msgr2=0x7f7a6c072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:36.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.635+0000 7f7a7159b700 1 --2- 172.21.15.44:0/530744673 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a6c101420 0x7f7a6c072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.635+0000 7f7a7159b700 1 -- 172.21.15.44:0/530744673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a6c101db0 msgr2=0x7f7a6c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:36.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.635+0000 7f7a7159b700 1 --2- 172.21.15.44:0/530744673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a6c101db0 0x7f7a6c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.635+0000 7f7a7159b700 1 -- 172.21.15.44:0/530744673 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a680095f0 con 0x7f7a6c100230 2023-11-16T16:47:36.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.635+0000 7f7a7159b700 1 --2- 172.21.15.44:0/530744673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c100230 0x7f7a6c0748a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7a68005740 tx=0x7f7a68005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:36.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.635+0000 7f7a627fc700 1 -- 172.21.15.44:0/530744673 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6801d070 con 0x7f7a6c100230 2023-11-16T16:47:36.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.635+0000 7f7a627fc700 1 -- 172.21.15.44:0/530744673 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a68021410 con 0x7f7a6c100230 2023-11-16T16:47:36.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.635+0000 7f7a737ff700 1 -- 172.21.15.44:0/530744673 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6c10bbe0 con 0x7f7a6c100230 2023-11-16T16:47:36.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.635+0000 7f7a627fc700 1 -- 172.21.15.44:0/530744673 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a68017650 con 0x7f7a6c100230 2023-11-16T16:47:36.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.636+0000 7f7a737ff700 1 -- 172.21.15.44:0/530744673 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6c10ba50 con 0x7f7a6c100230 2023-11-16T16:47:36.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.637+0000 7f7a737ff700 1 -- 172.21.15.44:0/530744673 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6c107770 con 0x7f7a6c100230 2023-11-16T16:47:36.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.638+0000 7f7a627fc700 1 -- 172.21.15.44:0/530744673 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7a68021580 con 0x7f7a6c100230 2023-11-16T16:47:36.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.640+0000 7f7a627fc700 1 --2- 172.21.15.44:0/530744673 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7a580a11d0 0x7f7a580a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:36.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.640+0000 7f7a627fc700 1 -- 172.21.15.44:0/530744673 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7a680cdc00 con 0x7f7a6c100230 2023-11-16T16:47:36.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.640+0000 7f7a70d9a700 1 --2- 172.21.15.44:0/530744673 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7a580a11d0 0x7f7a580a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:36.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.641+0000 7f7a70d9a700 1 --2- 172.21.15.44:0/530744673 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7a580a11d0 0x7f7a580a3680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009c60 tx=0x7f7a5c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:36.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.644+0000 7f7a627fc700 1 -- 172.21.15.44:0/530744673 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7a68097190 con 0x7f7a6c100230 2023-11-16T16:47:36.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:36 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1451274488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:47:36.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:36 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1629889111' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:47:36.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.867+0000 7f7a737ff700 1 -- 172.21.15.44:0/530744673 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7a6c02cfe0 con 0x7f7a6c100230 2023-11-16T16:47:36.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.868+0000 7f7a627fc700 1 -- 172.21.15.44:0/530744673 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) v1 ==== 81+0+591 (secure 0 0 0) 0x7f7a68026260 con 0x7f7a6c100230 2023-11-16T16:47:36.870 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1018560948 2023-11-17T16:40:42.449496+0000 2023-11-16T16:47:36.870 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1599940692 2023-11-17T16:40:42.449496+0000 2023-11-16T16:47:36.870 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3409868354 2023-11-17T16:40:42.449496+0000 2023-11-16T16:47:36.870 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/4234382393 2023-11-17T16:40:42.449496+0000 2023-11-16T16:47:36.870 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/1852059921 2023-11-17T16:40:15.683996+0000 2023-11-16T16:47:36.870 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3135397911 2023-11-17T16:40:15.683996+0000 2023-11-16T16:47:36.870 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/1852059921 2023-11-17T16:40:15.683996+0000 2023-11-16T16:47:36.871 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3599504154 2023-11-17T16:40:15.683996+0000 2023-11-16T16:47:36.871 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/264068429 2023-11-17T16:40:15.683996+0000 2023-11-16T16:47:36.871 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/4234382393 2023-11-17T16:40:42.449496+0000 2023-11-16T16:47:36.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.870+0000 7f7a737ff700 1 -- 172.21.15.44:0/530744673 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7a580a11d0 msgr2=0x7f7a580a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:36.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.870+0000 7f7a737ff700 1 --2- 172.21.15.44:0/530744673 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7a580a11d0 0x7f7a580a3680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009c60 tx=0x7f7a5c009260 comp rx=0 tx=0).stop 2023-11-16T16:47:36.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.870+0000 7f7a737ff700 1 -- 172.21.15.44:0/530744673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c100230 msgr2=0x7f7a6c0748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:36.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.870+0000 7f7a737ff700 1 --2- 172.21.15.44:0/530744673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c100230 0x7f7a6c0748a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7a68005740 tx=0x7f7a68005770 comp rx=0 tx=0).stop 2023-11-16T16:47:36.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.870+0000 7f7a737ff700 1 -- 172.21.15.44:0/530744673 shutdown_connections 2023-11-16T16:47:36.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.870+0000 7f7a737ff700 1 --2- 172.21.15.44:0/530744673 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7a580a11d0 0x7f7a580a3680 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.870+0000 7f7a737ff700 1 --2- 172.21.15.44:0/530744673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a6c101db0 0x7f7a6c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.870+0000 7f7a737ff700 1 --2- 172.21.15.44:0/530744673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a6c100230 0x7f7a6c0748a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.870+0000 7f7a737ff700 1 --2- 172.21.15.44:0/530744673 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a6c101420 0x7f7a6c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.870+0000 7f7a737ff700 1 -- 172.21.15.44:0/530744673 >> 172.21.15.44:0/530744673 conn(0x7f7a6c0fb810 msgr2=0x7f7a6c1048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:36.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.870+0000 7f7a737ff700 1 -- 172.21.15.44:0/530744673 shutdown_connections 2023-11-16T16:47:36.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:36.870+0000 7f7a737ff700 1 -- 172.21.15.44:0/530744673 wait complete. 2023-11-16T16:47:36.873 INFO:teuthology.orchestra.run.smithi044.stderr:listed 10 entries 2023-11-16T16:47:36.890 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:47:36.891 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T16:47:36.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:36 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1451274488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:47:36.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:36 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1629889111' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:47:36.936 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:36.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:36 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1451274488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:47:36.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:36 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1629889111' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:47:37.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.177+0000 7f81e9fff700 1 -- 172.21.15.44:0/820999609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e4073550 msgr2=0x7f81e40739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:37.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.177+0000 7f81e9fff700 1 --2- 172.21.15.44:0/820999609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e4073550 0x7f81e40739c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f81d8009990 tx=0x7f81d8009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.177+0000 7f81e9fff700 1 -- 172.21.15.44:0/820999609 shutdown_connections 2023-11-16T16:47:37.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.177+0000 7f81e9fff700 1 --2- 172.21.15.44:0/820999609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81e4108150 0x7f81e410a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.177+0000 7f81e9fff700 1 --2- 172.21.15.44:0/820999609 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81e4074b30 0x7f81e4072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.177+0000 7f81e9fff700 1 --2- 172.21.15.44:0/820999609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e4073550 0x7f81e40739c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.177+0000 7f81e9fff700 1 -- 172.21.15.44:0/820999609 >> 172.21.15.44:0/820999609 conn(0x7f81e40fb870 msgr2=0x7f81e40fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:37.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.177+0000 7f81e9fff700 1 -- 172.21.15.44:0/820999609 shutdown_connections 2023-11-16T16:47:37.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.177+0000 7f81e9fff700 1 -- 172.21.15.44:0/820999609 wait complete. 2023-11-16T16:47:37.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.178+0000 7f81e9fff700 1 Processor -- start 2023-11-16T16:47:37.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.178+0000 7f81e9fff700 1 -- start start 2023-11-16T16:47:37.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.178+0000 7f81e9fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81e4073550 0x7f81e41981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:37.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.178+0000 7f81e9fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e4074b30 0x7f81e4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:37.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.178+0000 7f81e9fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81e4108150 0x7f81e419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:37.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.179+0000 7f81e9fff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e419fd40 con 0x7f81e4074b30 2023-11-16T16:47:37.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.179+0000 7f81e9fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e419fe80 con 0x7f81e4073550 2023-11-16T16:47:37.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.179+0000 7f81e9fff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e41a0010 con 0x7f81e4108150 2023-11-16T16:47:37.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.179+0000 7f81e2ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e4074b30 0x7f81e4198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:37.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.179+0000 7f81e2ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e4074b30 0x7f81e4198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46340/0 (socket says 172.21.15.44:46340) 2023-11-16T16:47:37.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.179+0000 7f81e2ffd700 1 -- 172.21.15.44:0/697579968 learned_addr learned my addr 172.21.15.44:0/697579968 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:37.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.179+0000 7f81e37fe700 1 --2- 172.21.15.44:0/697579968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81e4073550 0x7f81e41981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:37.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.179+0000 7f81e3fff700 1 --2- 172.21.15.44:0/697579968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81e4108150 0x7f81e419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:37.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.179+0000 7f81e2ffd700 1 -- 172.21.15.44:0/697579968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81e4108150 msgr2=0x7f81e419f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:37.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.179+0000 7f81e2ffd700 1 --2- 172.21.15.44:0/697579968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81e4108150 0x7f81e419f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.179+0000 7f81e2ffd700 1 -- 172.21.15.44:0/697579968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81e4073550 msgr2=0x7f81e41981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:37.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.179+0000 7f81e2ffd700 1 --2- 172.21.15.44:0/697579968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81e4073550 0x7f81e41981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.179+0000 7f81e2ffd700 1 -- 172.21.15.44:0/697579968 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc009c40 con 0x7f81e4074b30 2023-11-16T16:47:37.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.179+0000 7f81e37fe700 1 --2- 172.21.15.44:0/697579968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81e4073550 0x7f81e41981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:37.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.179+0000 7f81e2ffd700 1 --2- 172.21.15.44:0/697579968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e4074b30 0x7f81e4198700 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f81d8004710 tx=0x7f81d8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:37.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.180+0000 7f81e0ff9700 1 -- 172.21.15.44:0/697579968 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d801d070 con 0x7f81e4074b30 2023-11-16T16:47:37.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.180+0000 7f81e0ff9700 1 -- 172.21.15.44:0/697579968 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f81d8021410 con 0x7f81e4074b30 2023-11-16T16:47:37.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.180+0000 7f81e9fff700 1 -- 172.21.15.44:0/697579968 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d80095f0 con 0x7f81e4074b30 2023-11-16T16:47:37.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.180+0000 7f81e0ff9700 1 -- 172.21.15.44:0/697579968 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d8017650 con 0x7f81e4074b30 2023-11-16T16:47:37.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.180+0000 7f81e9fff700 1 -- 172.21.15.44:0/697579968 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e410de20 con 0x7f81e4074b30 2023-11-16T16:47:37.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.181+0000 7f81e9fff700 1 -- 172.21.15.44:0/697579968 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e4106800 con 0x7f81e4074b30 2023-11-16T16:47:37.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.183+0000 7f81e0ff9700 1 -- 172.21.15.44:0/697579968 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f81d8021580 con 0x7f81e4074b30 2023-11-16T16:47:37.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.184+0000 7f81e0ff9700 1 --2- 172.21.15.44:0/697579968 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f81cc0a1150 0x7f81cc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:37.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.184+0000 7f81e0ff9700 1 -- 172.21.15.44:0/697579968 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f81d80ce230 con 0x7f81e4074b30 2023-11-16T16:47:37.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.185+0000 7f81e37fe700 1 --2- 172.21.15.44:0/697579968 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f81cc0a1150 0x7f81cc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:37.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.185+0000 7f81e37fe700 1 --2- 172.21.15.44:0/697579968 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f81cc0a1150 0x7f81cc0a3600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f81e4074f00 tx=0x7f81d4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:37.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.186+0000 7f81e0ff9700 1 -- 172.21.15.44:0/697579968 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f81d80978e0 con 0x7f81e4074b30 2023-11-16T16:47:37.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.395+0000 7f81e9fff700 1 -- 172.21.15.44:0/697579968 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81e402cfe0 con 0x7f81e4074b30 2023-11-16T16:47:37.399 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.396+0000 7f81e0ff9700 1 -- 172.21.15.44:0/697579968 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23925 (secure 0 0 0) 0x7f81d8026090 con 0x7f81e4074b30 2023-11-16T16:47:37.399 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:37.399 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":87,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:47:23.158961+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:37.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.399+0000 7f81e9fff700 1 -- 172.21.15.44:0/697579968 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f81cc0a1150 msgr2=0x7f81cc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:37.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.399+0000 7f81e9fff700 1 --2- 172.21.15.44:0/697579968 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f81cc0a1150 0x7f81cc0a3600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f81e4074f00 tx=0x7f81d4009260 comp rx=0 tx=0).stop 2023-11-16T16:47:37.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.399+0000 7f81e9fff700 1 -- 172.21.15.44:0/697579968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e4074b30 msgr2=0x7f81e4198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:37.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.399+0000 7f81e9fff700 1 --2- 172.21.15.44:0/697579968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e4074b30 0x7f81e4198700 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f81d8004710 tx=0x7f81d8009580 comp rx=0 tx=0).stop 2023-11-16T16:47:37.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.399+0000 7f81e9fff700 1 -- 172.21.15.44:0/697579968 shutdown_connections 2023-11-16T16:47:37.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.399+0000 7f81e9fff700 1 --2- 172.21.15.44:0/697579968 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f81cc0a1150 0x7f81cc0a3600 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.399+0000 7f81e9fff700 1 --2- 172.21.15.44:0/697579968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81e4108150 0x7f81e419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.399+0000 7f81e9fff700 1 --2- 172.21.15.44:0/697579968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81e4073550 0x7f81e41981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.399+0000 7f81e9fff700 1 --2- 172.21.15.44:0/697579968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81e4074b30 0x7f81e4198700 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.399+0000 7f81e9fff700 1 -- 172.21.15.44:0/697579968 >> 172.21.15.44:0/697579968 conn(0x7f81e40fb870 msgr2=0x7f81e40fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:37.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.399+0000 7f81e9fff700 1 -- 172.21.15.44:0/697579968 shutdown_connections 2023-11-16T16:47:37.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.399+0000 7f81e9fff700 1 -- 172.21.15.44:0/697579968 wait complete. 2023-11-16T16:47:37.420 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:03.059622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:16.211826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:17.601591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:21.136084+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T16:47:37.420 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:47:37.630 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.628+0000 7f243a3ff700 1 -- 172.21.15.44:0/4050927365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24341002c0 msgr2=0x7f24341006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:37.630 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.628+0000 7f243a3ff700 1 --2- 172.21.15.44:0/4050927365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24341002c0 0x7f24341006d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2424009990 tx=0x7f2424009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.628+0000 7f243a3ff700 1 -- 172.21.15.44:0/4050927365 shutdown_connections 2023-11-16T16:47:37.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.628+0000 7f243a3ff700 1 --2- 172.21.15.44:0/4050927365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2434101e40 0x7f2434108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.628+0000 7f243a3ff700 1 --2- 172.21.15.44:0/4050927365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24341002c0 0x7f24341006d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.628+0000 7f243a3ff700 1 --2- 172.21.15.44:0/4050927365 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24341014b0 0x7f2434101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.628+0000 7f243a3ff700 1 -- 172.21.15.44:0/4050927365 >> 172.21.15.44:0/4050927365 conn(0x7f24340fb8a0 msgr2=0x7f24340fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:37.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.628+0000 7f243a3ff700 1 -- 172.21.15.44:0/4050927365 shutdown_connections 2023-11-16T16:47:37.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.628+0000 7f243a3ff700 1 -- 172.21.15.44:0/4050927365 wait complete. 2023-11-16T16:47:37.633 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.629+0000 7f243a3ff700 1 Processor -- start 2023-11-16T16:47:37.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.629+0000 7f243a3ff700 1 -- start start 2023-11-16T16:47:37.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.630+0000 7f243a3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24341002c0 0x7f2434198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:37.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.630+0000 7f243a3ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24341014b0 0x7f24341987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:37.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.630+0000 7f243a3ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2434101e40 0x7f243419f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:37.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.630+0000 7f243a3ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243419fdf0 con 0x7f24341014b0 2023-11-16T16:47:37.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.630+0000 7f243a3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243419ff30 con 0x7f24341002c0 2023-11-16T16:47:37.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.630+0000 7f243a3ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24341a00c0 con 0x7f2434101e40 2023-11-16T16:47:37.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.630+0000 7f24337fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24341014b0 0x7f24341987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:37.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.630+0000 7f2433fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24341002c0 0x7f2434198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:37.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.630+0000 7f2433fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24341002c0 0x7f2434198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:43270/0 (socket says 172.21.15.44:43270) 2023-11-16T16:47:37.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.630+0000 7f24337fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24341014b0 0x7f24341987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46354/0 (socket says 172.21.15.44:46354) 2023-11-16T16:47:37.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.630+0000 7f2433fff700 1 -- 172.21.15.44:0/1042436105 learned_addr learned my addr 172.21.15.44:0/1042436105 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:37.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.630+0000 7f243899c700 1 --2- 172.21.15.44:0/1042436105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2434101e40 0x7f243419f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:37.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.631+0000 7f2433fff700 1 -- 172.21.15.44:0/1042436105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2434101e40 msgr2=0x7f243419f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:37.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.631+0000 7f2433fff700 1 --2- 172.21.15.44:0/1042436105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2434101e40 0x7f243419f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.631+0000 7f2433fff700 1 -- 172.21.15.44:0/1042436105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24341014b0 msgr2=0x7f24341987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:37.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.631+0000 7f2433fff700 1 --2- 172.21.15.44:0/1042436105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24341014b0 0x7f24341987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.631+0000 7f2433fff700 1 -- 172.21.15.44:0/1042436105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24240095f0 con 0x7f24341002c0 2023-11-16T16:47:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.631+0000 7f24337fe700 1 --2- 172.21.15.44:0/1042436105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24341014b0 0x7f24341987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.631+0000 7f2433fff700 1 --2- 172.21.15.44:0/1042436105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24341002c0 0x7f2434198270 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2424004730 tx=0x7f2424004760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.631+0000 7f24317fa700 1 -- 172.21.15.44:0/1042436105 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242401d070 con 0x7f24341002c0 2023-11-16T16:47:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.631+0000 7f243a3ff700 1 -- 172.21.15.44:0/1042436105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f243410bc90 con 0x7f24341002c0 2023-11-16T16:47:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.631+0000 7f24317fa700 1 -- 172.21.15.44:0/1042436105 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2424021410 con 0x7f24341002c0 2023-11-16T16:47:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.631+0000 7f24317fa700 1 -- 172.21.15.44:0/1042436105 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424017650 con 0x7f24341002c0 2023-11-16T16:47:37.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.631+0000 7f243a3ff700 1 -- 172.21.15.44:0/1042436105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f243410bb00 con 0x7f24341002c0 2023-11-16T16:47:37.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.632+0000 7f243a3ff700 1 -- 172.21.15.44:0/1042436105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24341929f0 con 0x7f24341002c0 2023-11-16T16:47:37.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.634+0000 7f24317fa700 1 -- 172.21.15.44:0/1042436105 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2424021580 con 0x7f24341002c0 2023-11-16T16:47:37.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.635+0000 7f24317fa700 1 --2- 172.21.15.44:0/1042436105 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f241c0a1150 0x7f241c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:37.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.636+0000 7f24317fa700 1 -- 172.21.15.44:0/1042436105 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f24240ce0d0 con 0x7f24341002c0 2023-11-16T16:47:37.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.636+0000 7f24337fe700 1 --2- 172.21.15.44:0/1042436105 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f241c0a1150 0x7f241c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:37.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.636+0000 7f24337fe700 1 --2- 172.21.15.44:0/1042436105 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f241c0a1150 0x7f241c0a3600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2428011490 tx=0x7f2428009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:37.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.638+0000 7f24317fa700 1 -- 172.21.15.44:0/1042436105 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2424097660 con 0x7f24341002c0 2023-11-16T16:47:37.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.847+0000 7f243a3ff700 1 -- 172.21.15.44:0/1042436105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2434061700 con 0x7f24341002c0 2023-11-16T16:47:37.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.848+0000 7f24317fa700 1 -- 172.21.15.44:0/1042436105 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f242409ac80 con 0x7f24341002c0 2023-11-16T16:47:37.850 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2023-11-16T16:47:37.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.850+0000 7f243a3ff700 1 -- 172.21.15.44:0/1042436105 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f241c0a1150 msgr2=0x7f241c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:37.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.850+0000 7f243a3ff700 1 --2- 172.21.15.44:0/1042436105 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f241c0a1150 0x7f241c0a3600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2428011490 tx=0x7f2428009260 comp rx=0 tx=0).stop 2023-11-16T16:47:37.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.850+0000 7f243a3ff700 1 -- 172.21.15.44:0/1042436105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24341002c0 msgr2=0x7f2434198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:37.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.850+0000 7f243a3ff700 1 --2- 172.21.15.44:0/1042436105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24341002c0 0x7f2434198270 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2424004730 tx=0x7f2424004760 comp rx=0 tx=0).stop 2023-11-16T16:47:37.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.851+0000 7f243a3ff700 1 -- 172.21.15.44:0/1042436105 shutdown_connections 2023-11-16T16:47:37.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.851+0000 7f243a3ff700 1 --2- 172.21.15.44:0/1042436105 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f241c0a1150 0x7f241c0a3600 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.851+0000 7f243a3ff700 1 --2- 172.21.15.44:0/1042436105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2434101e40 0x7f243419f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.854 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.851+0000 7f243a3ff700 1 --2- 172.21.15.44:0/1042436105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24341002c0 0x7f2434198270 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.854 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.851+0000 7f243a3ff700 1 --2- 172.21.15.44:0/1042436105 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24341014b0 0x7f24341987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:37.854 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.851+0000 7f243a3ff700 1 -- 172.21.15.44:0/1042436105 >> 172.21.15.44:0/1042436105 conn(0x7f24340fb8a0 msgr2=0x7f2434106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:37.854 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.851+0000 7f243a3ff700 1 -- 172.21.15.44:0/1042436105 shutdown_connections 2023-11-16T16:47:37.854 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:37.851+0000 7f243a3ff700 1 -- 172.21.15.44:0/1042436105 wait complete. 2023-11-16T16:47:37.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:37 smithi044 ceph-mon[128750]: pgmap v284: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T16:47:37.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:37 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/530744673' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T16:47:37.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:37 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/697579968' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:37.871 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T16:47:37.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:37 smithi176 ceph-mon[131630]: pgmap v284: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T16:47:37.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:37 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/530744673' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T16:47:37.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:37 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/697579968' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:37 smithi077 ceph-mon[131204]: pgmap v284: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T16:47:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:37 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/530744673' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T16:47:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:37 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/697579968' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.076+0000 7efeb71ff700 1 -- 172.21.15.44:0/811265801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb01002c0 msgr2=0x7efeb01006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.076+0000 7efeb71ff700 1 --2- 172.21.15.44:0/811265801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb01002c0 0x7efeb01006d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efea4009990 tx=0x7efea4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.076+0000 7efeb71ff700 1 -- 172.21.15.44:0/811265801 shutdown_connections 2023-11-16T16:47:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.076+0000 7efeb71ff700 1 --2- 172.21.15.44:0/811265801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeb0101e40 0x7efeb0108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.076+0000 7efeb71ff700 1 --2- 172.21.15.44:0/811265801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb01002c0 0x7efeb01006d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.076+0000 7efeb71ff700 1 --2- 172.21.15.44:0/811265801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efeb01014b0 0x7efeb0101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.076+0000 7efeb71ff700 1 -- 172.21.15.44:0/811265801 >> 172.21.15.44:0/811265801 conn(0x7efeb00fb8a0 msgr2=0x7efeb00fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.076+0000 7efeb71ff700 1 -- 172.21.15.44:0/811265801 shutdown_connections 2023-11-16T16:47:38.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.076+0000 7efeb71ff700 1 -- 172.21.15.44:0/811265801 wait complete. 2023-11-16T16:47:38.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.077+0000 7efeb71ff700 1 Processor -- start 2023-11-16T16:47:38.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.077+0000 7efeb71ff700 1 -- start start 2023-11-16T16:47:38.082 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.077+0000 7efeb71ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb01002c0 0x7efeb0198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:38.082 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.077+0000 7efeb71ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeb01014b0 0x7efeb0198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:38.082 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.077+0000 7efeb71ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efeb0101e40 0x7efeb019f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:38.082 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.077+0000 7efeb71ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb019fd80 con 0x7efeb0101e40 2023-11-16T16:47:38.082 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.078+0000 7efeb71ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb019fec0 con 0x7efeb01014b0 2023-11-16T16:47:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.078+0000 7efeb71ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb01a0050 con 0x7efeb01002c0 2023-11-16T16:47:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.078+0000 7efeb4f9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb01002c0 0x7efeb0198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.078+0000 7efeb4f9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb01002c0 0x7efeb0198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:46834/0 (socket says 172.21.15.44:46834) 2023-11-16T16:47:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.078+0000 7efeb579c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efeb0101e40 0x7efeb019f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.078+0000 7efeb4f9b700 1 -- 172.21.15.44:0/370204919 learned_addr learned my addr 172.21.15.44:0/370204919 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.078+0000 7efeaffff700 1 --2- 172.21.15.44:0/370204919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeb01014b0 0x7efeb0198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.078+0000 7efeb4f9b700 1 -- 172.21.15.44:0/370204919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeb01014b0 msgr2=0x7efeb0198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.078+0000 7efeb4f9b700 1 --2- 172.21.15.44:0/370204919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeb01014b0 0x7efeb0198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.078+0000 7efeb4f9b700 1 -- 172.21.15.44:0/370204919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efeb0101e40 msgr2=0x7efeb019f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.078+0000 7efeb4f9b700 1 --2- 172.21.15.44:0/370204919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efeb0101e40 0x7efeb019f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.078+0000 7efeb4f9b700 1 -- 172.21.15.44:0/370204919 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c009520 con 0x7efeb01002c0 2023-11-16T16:47:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.078+0000 7efeb579c700 1 --2- 172.21.15.44:0/370204919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efeb0101e40 0x7efeb019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.078+0000 7efeb4f9b700 1 --2- 172.21.15.44:0/370204919 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb01002c0 0x7efeb0198200 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efea4004710 tx=0x7efea4009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.079+0000 7efeadffb700 1 -- 172.21.15.44:0/370204919 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea401d070 con 0x7efeb01002c0 2023-11-16T16:47:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.079+0000 7efeb71ff700 1 -- 172.21.15.44:0/370204919 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea40095f0 con 0x7efeb01002c0 2023-11-16T16:47:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.079+0000 7efeadffb700 1 -- 172.21.15.44:0/370204919 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efea4021410 con 0x7efeb01002c0 2023-11-16T16:47:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.079+0000 7efeadffb700 1 -- 172.21.15.44:0/370204919 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea4017650 con 0x7efeb01002c0 2023-11-16T16:47:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.079+0000 7efeb71ff700 1 -- 172.21.15.44:0/370204919 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb010bc10 con 0x7efeb01002c0 2023-11-16T16:47:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.080+0000 7efeb71ff700 1 -- 172.21.15.44:0/370204919 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb010ba80 con 0x7efeb01002c0 2023-11-16T16:47:38.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.082+0000 7efeadffb700 1 -- 172.21.15.44:0/370204919 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7efea4021580 con 0x7efeb01002c0 2023-11-16T16:47:38.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.083+0000 7efeadffb700 1 --2- 172.21.15.44:0/370204919 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efea00a1150 0x7efea00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:38.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.083+0000 7efeadffb700 1 -- 172.21.15.44:0/370204919 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7efea40ce270 con 0x7efeb01002c0 2023-11-16T16:47:38.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.083+0000 7efeaffff700 1 --2- 172.21.15.44:0/370204919 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efea00a1150 0x7efea00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:38.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.084+0000 7efeaffff700 1 --2- 172.21.15.44:0/370204919 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efea00a1150 0x7efea00a3600 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009df0 tx=0x7efe9c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:38.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.085+0000 7efeadffb700 1 -- 172.21.15.44:0/370204919 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efea4097860 con 0x7efeb01002c0 2023-11-16T16:47:38.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.296+0000 7efeb71ff700 1 -- 172.21.15.44:0/370204919 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efeb010bda0 con 0x7efeb01002c0 2023-11-16T16:47:38.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.297+0000 7efeadffb700 1 -- 172.21.15.44:0/370204919 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7efea409ae80 con 0x7efeb01002c0 2023-11-16T16:47:38.299 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:47:38.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.299+0000 7efeb71ff700 1 -- 172.21.15.44:0/370204919 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efea00a1150 msgr2=0x7efea00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:38.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.299+0000 7efeb71ff700 1 --2- 172.21.15.44:0/370204919 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efea00a1150 0x7efea00a3600 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009df0 tx=0x7efe9c009260 comp rx=0 tx=0).stop 2023-11-16T16:47:38.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.299+0000 7efeb71ff700 1 -- 172.21.15.44:0/370204919 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb01002c0 msgr2=0x7efeb0198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:38.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.299+0000 7efeb71ff700 1 --2- 172.21.15.44:0/370204919 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb01002c0 0x7efeb0198200 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efea4004710 tx=0x7efea4009580 comp rx=0 tx=0).stop 2023-11-16T16:47:38.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.299+0000 7efeb71ff700 1 -- 172.21.15.44:0/370204919 shutdown_connections 2023-11-16T16:47:38.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.299+0000 7efeb71ff700 1 --2- 172.21.15.44:0/370204919 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efea00a1150 0x7efea00a3600 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.299+0000 7efeb71ff700 1 --2- 172.21.15.44:0/370204919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efeb0101e40 0x7efeb019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.299+0000 7efeb71ff700 1 --2- 172.21.15.44:0/370204919 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb01002c0 0x7efeb0198200 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.299+0000 7efeb71ff700 1 --2- 172.21.15.44:0/370204919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeb01014b0 0x7efeb0198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.299+0000 7efeb71ff700 1 -- 172.21.15.44:0/370204919 >> 172.21.15.44:0/370204919 conn(0x7efeb00fb8a0 msgr2=0x7efeb00fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:38.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.299+0000 7efeb71ff700 1 -- 172.21.15.44:0/370204919 shutdown_connections 2023-11-16T16:47:38.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.299+0000 7efeb71ff700 1 -- 172.21.15.44:0/370204919 wait complete. 2023-11-16T16:47:38.320 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T16:47:38.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.526+0000 7f3345bff700 1 -- 172.21.15.44:0/1872985551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3340101e40 msgr2=0x7f3340108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:38.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.526+0000 7f3345bff700 1 --2- 172.21.15.44:0/1872985551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3340101e40 0x7f3340108700 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3338009f90 tx=0x7f333800b330 comp rx=0 tx=0).stop 2023-11-16T16:47:38.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.526+0000 7f3345bff700 1 -- 172.21.15.44:0/1872985551 shutdown_connections 2023-11-16T16:47:38.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.526+0000 7f3345bff700 1 --2- 172.21.15.44:0/1872985551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3340101e40 0x7f3340108700 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.526+0000 7f3345bff700 1 --2- 172.21.15.44:0/1872985551 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33401002c0 0x7f33401006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.526+0000 7f3345bff700 1 --2- 172.21.15.44:0/1872985551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33401014b0 0x7f3340101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.526+0000 7f3345bff700 1 -- 172.21.15.44:0/1872985551 >> 172.21.15.44:0/1872985551 conn(0x7f33400fb8a0 msgr2=0x7f33400fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:38.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.526+0000 7f3345bff700 1 -- 172.21.15.44:0/1872985551 shutdown_connections 2023-11-16T16:47:38.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.526+0000 7f3345bff700 1 -- 172.21.15.44:0/1872985551 wait complete. 2023-11-16T16:47:38.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.527+0000 7f3345bff700 1 Processor -- start 2023-11-16T16:47:38.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.527+0000 7f3345bff700 1 -- start start 2023-11-16T16:47:38.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.527+0000 7f3345bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33401002c0 0x7f3340196170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:38.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.527+0000 7f3345bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33401014b0 0x7f33401966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:38.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.527+0000 7f3345bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3340101e40 0x7f334019d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:38.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.527+0000 7f3345bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334019dcf0 con 0x7f3340101e40 2023-11-16T16:47:38.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.527+0000 7f3345bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334019de30 con 0x7f33401002c0 2023-11-16T16:47:38.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.527+0000 7f3345bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334019dfc0 con 0x7f33401014b0 2023-11-16T16:47:38.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.528+0000 7f333ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3340101e40 0x7f334019d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:38.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.528+0000 7f333ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3340101e40 0x7f334019d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46394/0 (socket says 172.21.15.44:46394) 2023-11-16T16:47:38.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.528+0000 7f333ffff700 1 -- 172.21.15.44:0/2997708949 learned_addr learned my addr 172.21.15.44:0/2997708949 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:38.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.528+0000 7f333f7fe700 1 --2- 172.21.15.44:0/2997708949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33401002c0 0x7f3340196170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:38.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.528+0000 7f3337fff700 1 --2- 172.21.15.44:0/2997708949 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33401014b0 0x7f33401966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:38.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.528+0000 7f333ffff700 1 -- 172.21.15.44:0/2997708949 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33401014b0 msgr2=0x7f33401966b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:38.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.528+0000 7f333ffff700 1 --2- 172.21.15.44:0/2997708949 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33401014b0 0x7f33401966b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.528+0000 7f333ffff700 1 -- 172.21.15.44:0/2997708949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33401002c0 msgr2=0x7f3340196170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:38.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.528+0000 7f333ffff700 1 --2- 172.21.15.44:0/2997708949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33401002c0 0x7f3340196170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.528+0000 7f333ffff700 1 -- 172.21.15.44:0/2997708949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3328009520 con 0x7f3340101e40 2023-11-16T16:47:38.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.528+0000 7f333ffff700 1 --2- 172.21.15.44:0/2997708949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3340101e40 0x7f334019d720 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3338007930 tx=0x7f3338004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:38.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.529+0000 7f333d7fa700 1 -- 172.21.15.44:0/2997708949 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333800e040 con 0x7f3340101e40 2023-11-16T16:47:38.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.529+0000 7f333d7fa700 1 -- 172.21.15.44:0/2997708949 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f333800bda0 con 0x7f3340101e40 2023-11-16T16:47:38.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.529+0000 7f3345bff700 1 -- 172.21.15.44:0/2997708949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3338009c40 con 0x7f3340101e40 2023-11-16T16:47:38.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.529+0000 7f333d7fa700 1 -- 172.21.15.44:0/2997708949 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33380052a0 con 0x7f3340101e40 2023-11-16T16:47:38.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.529+0000 7f3345bff700 1 -- 172.21.15.44:0/2997708949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334010bc90 con 0x7f3340101e40 2023-11-16T16:47:38.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.530+0000 7f3345bff700 1 -- 172.21.15.44:0/2997708949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f334010be20 con 0x7f3340101e40 2023-11-16T16:47:38.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.532+0000 7f333d7fa700 1 -- 172.21.15.44:0/2997708949 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f333801d470 con 0x7f3340101e40 2023-11-16T16:47:38.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.533+0000 7f333d7fa700 1 --2- 172.21.15.44:0/2997708949 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f332c0a11d0 0x7f332c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:38.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.533+0000 7f333d7fa700 1 -- 172.21.15.44:0/2997708949 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f33380ce110 con 0x7f3340101e40 2023-11-16T16:47:38.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.533+0000 7f333f7fe700 1 --2- 172.21.15.44:0/2997708949 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f332c0a11d0 0x7f332c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:38.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.534+0000 7f333f7fe700 1 --2- 172.21.15.44:0/2997708949 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f332c0a11d0 0x7f332c0a3680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3340101310 tx=0x7f3330009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:38.539 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.537+0000 7f333d7fa700 1 -- 172.21.15.44:0/2997708949 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3338097700 con 0x7f3340101e40 2023-11-16T16:47:38.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.752+0000 7f3345bff700 1 -- 172.21.15.44:0/2997708949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3340061860 con 0x7f3340101e40 2023-11-16T16:47:38.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.752+0000 7f333d7fa700 1 -- 172.21.15.44:0/2997708949 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3338008f30 con 0x7f3340101e40 2023-11-16T16:47:38.755 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2023-11-16T16:47:38.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.754+0000 7f3345bff700 1 -- 172.21.15.44:0/2997708949 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f332c0a11d0 msgr2=0x7f332c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:38.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.754+0000 7f3345bff700 1 --2- 172.21.15.44:0/2997708949 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f332c0a11d0 0x7f332c0a3680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3340101310 tx=0x7f3330009310 comp rx=0 tx=0).stop 2023-11-16T16:47:38.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.754+0000 7f3345bff700 1 -- 172.21.15.44:0/2997708949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3340101e40 msgr2=0x7f334019d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:38.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.754+0000 7f3345bff700 1 --2- 172.21.15.44:0/2997708949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3340101e40 0x7f334019d720 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3338007930 tx=0x7f3338004d10 comp rx=0 tx=0).stop 2023-11-16T16:47:38.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.754+0000 7f3345bff700 1 -- 172.21.15.44:0/2997708949 shutdown_connections 2023-11-16T16:47:38.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.754+0000 7f3345bff700 1 --2- 172.21.15.44:0/2997708949 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f332c0a11d0 0x7f332c0a3680 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.754+0000 7f3345bff700 1 --2- 172.21.15.44:0/2997708949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3340101e40 0x7f334019d720 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.754+0000 7f3345bff700 1 --2- 172.21.15.44:0/2997708949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33401002c0 0x7f3340196170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.754+0000 7f3345bff700 1 --2- 172.21.15.44:0/2997708949 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33401014b0 0x7f33401966b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.754+0000 7f3345bff700 1 -- 172.21.15.44:0/2997708949 >> 172.21.15.44:0/2997708949 conn(0x7f33400fb8a0 msgr2=0x7f3340104920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:38.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.754+0000 7f3345bff700 1 -- 172.21.15.44:0/2997708949 shutdown_connections 2023-11-16T16:47:38.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.755+0000 7f3345bff700 1 -- 172.21.15.44:0/2997708949 wait complete. 2023-11-16T16:47:38.775 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T16:47:38.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:38 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1042436105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:38.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:38 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/370204919' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:47:38.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:38 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1042436105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:38.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:38 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/370204919' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:47:38.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:38 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1042436105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:38.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:38 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/370204919' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:47:38.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.981+0000 7f2457bff700 1 -- 172.21.15.44:0/1672010663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2450101e40 msgr2=0x7f2450108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:38.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.981+0000 7f2457bff700 1 --2- 172.21.15.44:0/1672010663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2450101e40 0x7f2450108700 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f244800b0a0 tx=0x7f244800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.981+0000 7f2457bff700 1 -- 172.21.15.44:0/1672010663 shutdown_connections 2023-11-16T16:47:38.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.981+0000 7f2457bff700 1 --2- 172.21.15.44:0/1672010663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2450101e40 0x7f2450108700 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.981+0000 7f2457bff700 1 --2- 172.21.15.44:0/1672010663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24501002c0 0x7f24501006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.981+0000 7f2457bff700 1 --2- 172.21.15.44:0/1672010663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24501014b0 0x7f2450101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.981+0000 7f2457bff700 1 -- 172.21.15.44:0/1672010663 >> 172.21.15.44:0/1672010663 conn(0x7f24500fb8a0 msgr2=0x7f24500fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:38.984 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.981+0000 7f2457bff700 1 -- 172.21.15.44:0/1672010663 shutdown_connections 2023-11-16T16:47:38.984 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.981+0000 7f2457bff700 1 -- 172.21.15.44:0/1672010663 wait complete. 2023-11-16T16:47:38.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.982+0000 7f2457bff700 1 Processor -- start 2023-11-16T16:47:38.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.982+0000 7f2457bff700 1 -- start start 2023-11-16T16:47:38.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.982+0000 7f2457bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24501002c0 0x7f24500748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:38.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.982+0000 7f2457bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24501014b0 0x7f2450072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:38.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.982+0000 7f2457bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2450101e40 0x7f2450073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:38.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.982+0000 7f2457bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2450073bc0 con 0x7f24501002c0 2023-11-16T16:47:38.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.982+0000 7f2457bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2450073d30 con 0x7f24501014b0 2023-11-16T16:47:38.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.982+0000 7f2457bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24501a01a0 con 0x7f2450101e40 2023-11-16T16:47:38.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.983+0000 7f245599b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24501002c0 0x7f24500748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:38.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.983+0000 7f245599b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24501002c0 0x7f24500748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46418/0 (socket says 172.21.15.44:46418) 2023-11-16T16:47:38.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.983+0000 7f245599b700 1 -- 172.21.15.44:0/420937266 learned_addr learned my addr 172.21.15.44:0/420937266 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:38.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.983+0000 7f245519a700 1 --2- 172.21.15.44:0/420937266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24501014b0 0x7f2450072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:38.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.983+0000 7f245619c700 1 --2- 172.21.15.44:0/420937266 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2450101e40 0x7f2450073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:38.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.983+0000 7f245599b700 1 -- 172.21.15.44:0/420937266 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2450101e40 msgr2=0x7f2450073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:38.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.983+0000 7f245599b700 1 --2- 172.21.15.44:0/420937266 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2450101e40 0x7f2450073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.983+0000 7f245599b700 1 -- 172.21.15.44:0/420937266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24501014b0 msgr2=0x7f2450072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:38.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.983+0000 7f245599b700 1 --2- 172.21.15.44:0/420937266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24501014b0 0x7f2450072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:38.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.983+0000 7f245599b700 1 -- 172.21.15.44:0/420937266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2448009d00 con 0x7f24501002c0 2023-11-16T16:47:38.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.983+0000 7f245619c700 1 --2- 172.21.15.44:0/420937266 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2450101e40 0x7f2450073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:38.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.983+0000 7f245599b700 1 --2- 172.21.15.44:0/420937266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24501002c0 0x7f24500748a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f244c00e970 tx=0x7f244c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:38.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.983+0000 7f2446ffd700 1 -- 172.21.15.44:0/420937266 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c009790 con 0x7f24501002c0 2023-11-16T16:47:38.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.984+0000 7f2446ffd700 1 -- 172.21.15.44:0/420937266 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f244c00eec0 con 0x7f24501002c0 2023-11-16T16:47:38.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.984+0000 7f2457bff700 1 -- 172.21.15.44:0/420937266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f245010bc90 con 0x7f24501002c0 2023-11-16T16:47:38.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.984+0000 7f2446ffd700 1 -- 172.21.15.44:0/420937266 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c003cf0 con 0x7f24501002c0 2023-11-16T16:47:38.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.984+0000 7f2457bff700 1 -- 172.21.15.44:0/420937266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f245010bb00 con 0x7f24501002c0 2023-11-16T16:47:38.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.985+0000 7f2457bff700 1 -- 172.21.15.44:0/420937266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24501077c0 con 0x7f24501002c0 2023-11-16T16:47:38.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.987+0000 7f2446ffd700 1 -- 172.21.15.44:0/420937266 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f244c00ccd0 con 0x7f24501002c0 2023-11-16T16:47:38.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.988+0000 7f2446ffd700 1 --2- 172.21.15.44:0/420937266 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f243c0a1150 0x7f243c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:38.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.988+0000 7f2446ffd700 1 -- 172.21.15.44:0/420937266 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f244c014070 con 0x7f24501002c0 2023-11-16T16:47:38.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.988+0000 7f245519a700 1 --2- 172.21.15.44:0/420937266 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f243c0a1150 0x7f243c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:38.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.989+0000 7f245519a700 1 --2- 172.21.15.44:0/420937266 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f243c0a1150 0x7f243c0a3600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2440009c60 tx=0x7f2440009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:38.994 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:38.992+0000 7f2446ffd700 1 -- 172.21.15.44:0/420937266 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f244c0962b0 con 0x7f24501002c0 2023-11-16T16:47:39.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.203+0000 7f2457bff700 1 -- 172.21.15.44:0/420937266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2450061930 con 0x7f24501002c0 2023-11-16T16:47:39.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.204+0000 7f2446ffd700 1 -- 172.21.15.44:0/420937266 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7f244c0998d0 con 0x7f24501002c0 2023-11-16T16:47:39.206 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:47:39.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.206+0000 7f2457bff700 1 -- 172.21.15.44:0/420937266 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f243c0a1150 msgr2=0x7f243c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:39.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.206+0000 7f2457bff700 1 --2- 172.21.15.44:0/420937266 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f243c0a1150 0x7f243c0a3600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2440009c60 tx=0x7f2440009260 comp rx=0 tx=0).stop 2023-11-16T16:47:39.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.206+0000 7f2457bff700 1 -- 172.21.15.44:0/420937266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24501002c0 msgr2=0x7f24500748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:39.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.206+0000 7f2457bff700 1 --2- 172.21.15.44:0/420937266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24501002c0 0x7f24500748a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f244c00e970 tx=0x7f244c00ec80 comp rx=0 tx=0).stop 2023-11-16T16:47:39.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.206+0000 7f2457bff700 1 -- 172.21.15.44:0/420937266 shutdown_connections 2023-11-16T16:47:39.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.206+0000 7f2457bff700 1 --2- 172.21.15.44:0/420937266 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f243c0a1150 0x7f243c0a3600 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:39.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.206+0000 7f2457bff700 1 --2- 172.21.15.44:0/420937266 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2450101e40 0x7f2450073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:39.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.206+0000 7f2457bff700 1 --2- 172.21.15.44:0/420937266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24501002c0 0x7f24500748a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:39.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.206+0000 7f2457bff700 1 --2- 172.21.15.44:0/420937266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24501014b0 0x7f2450072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:39.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.206+0000 7f2457bff700 1 -- 172.21.15.44:0/420937266 >> 172.21.15.44:0/420937266 conn(0x7f24500fb8a0 msgr2=0x7f2450104900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:39.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.207+0000 7f2457bff700 1 -- 172.21.15.44:0/420937266 shutdown_connections 2023-11-16T16:47:39.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.207+0000 7f2457bff700 1 -- 172.21.15.44:0/420937266 wait complete. 2023-11-16T16:47:39.227 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-16T16:47:39.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.431+0000 7fe1ad5ff700 1 -- 172.21.15.44:0/3618878244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1a8101420 msgr2=0x7fe1a8101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:39.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.431+0000 7fe1ad5ff700 1 --2- 172.21.15.44:0/3618878244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1a8101420 0x7fe1a8101870 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe19c009990 tx=0x7fe19c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:39.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.431+0000 7fe1ad5ff700 1 -- 172.21.15.44:0/3618878244 shutdown_connections 2023-11-16T16:47:39.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.431+0000 7fe1ad5ff700 1 --2- 172.21.15.44:0/3618878244 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a8101db0 0x7fe1a8108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:39.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.431+0000 7fe1ad5ff700 1 --2- 172.21.15.44:0/3618878244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1a8100230 0x7fe1a8100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:39.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.431+0000 7fe1ad5ff700 1 --2- 172.21.15.44:0/3618878244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1a8101420 0x7fe1a8101870 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:39.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.431+0000 7fe1ad5ff700 1 -- 172.21.15.44:0/3618878244 >> 172.21.15.44:0/3618878244 conn(0x7fe1a80fb810 msgr2=0x7fe1a80fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:39.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.432+0000 7fe1ad5ff700 1 -- 172.21.15.44:0/3618878244 shutdown_connections 2023-11-16T16:47:39.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.432+0000 7fe1ad5ff700 1 -- 172.21.15.44:0/3618878244 wait complete. 2023-11-16T16:47:39.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.433+0000 7fe1ad5ff700 1 Processor -- start 2023-11-16T16:47:39.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.433+0000 7fe1ad5ff700 1 -- start start 2023-11-16T16:47:39.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.434+0000 7fe1ad5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1a8100230 0x7fe1a8195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:39.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.434+0000 7fe1ad5ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1a8101420 0x7fe1a81964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:39.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.434+0000 7fe1ad5ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a8101db0 0x7fe1a819d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:39.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.434+0000 7fe1ad5ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a819db30 con 0x7fe1a8101420 2023-11-16T16:47:39.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.434+0000 7fe1ad5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a819dc70 con 0x7fe1a8100230 2023-11-16T16:47:39.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.434+0000 7fe1ad5ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a819de00 con 0x7fe1a8101db0 2023-11-16T16:47:39.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.434+0000 7fe1a67fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1a8101420 0x7fe1a81964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:39.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.434+0000 7fe1a6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1a8100230 0x7fe1a8195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:39.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.434+0000 7fe1a77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a8101db0 0x7fe1a819d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:39.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.434+0000 7fe1a6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1a8100230 0x7fe1a8195fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:43338/0 (socket says 172.21.15.44:43338) 2023-11-16T16:47:39.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.434+0000 7fe1a67fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1a8101420 0x7fe1a81964f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46428/0 (socket says 172.21.15.44:46428) 2023-11-16T16:47:39.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.434+0000 7fe1a77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a8101db0 0x7fe1a819d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:46890/0 (socket says 172.21.15.44:46890) 2023-11-16T16:47:39.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.434+0000 7fe1a6ffd700 1 -- 172.21.15.44:0/3126334848 learned_addr learned my addr 172.21.15.44:0/3126334848 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:39.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.435+0000 7fe1a77fe700 1 -- 172.21.15.44:0/3126334848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1a8100230 msgr2=0x7fe1a8195fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:39.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.435+0000 7fe1a77fe700 1 --2- 172.21.15.44:0/3126334848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1a8100230 0x7fe1a8195fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:39.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.435+0000 7fe1a77fe700 1 -- 172.21.15.44:0/3126334848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1a8101420 msgr2=0x7fe1a81964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:39.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.435+0000 7fe1a77fe700 1 --2- 172.21.15.44:0/3126334848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1a8101420 0x7fe1a81964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:39.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.435+0000 7fe1a77fe700 1 -- 172.21.15.44:0/3126334848 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c0095f0 con 0x7fe1a8101db0 2023-11-16T16:47:39.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.435+0000 7fe1a67fc700 1 --2- 172.21.15.44:0/3126334848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1a8101420 0x7fe1a81964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:39.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.435+0000 7fe1a6ffd700 1 --2- 172.21.15.44:0/3126334848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1a8100230 0x7fe1a8195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:39.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.435+0000 7fe1a77fe700 1 --2- 172.21.15.44:0/3126334848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a8101db0 0x7fe1a819d560 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0007f10 tx=0x7fe1a000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:39.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.435+0000 7fe18ffff700 1 -- 172.21.15.44:0/3126334848 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0004db0 con 0x7fe1a8101db0 2023-11-16T16:47:39.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.435+0000 7fe18ffff700 1 -- 172.21.15.44:0/3126334848 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe1a000f040 con 0x7fe1a8101db0 2023-11-16T16:47:39.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.435+0000 7fe1ad5ff700 1 -- 172.21.15.44:0/3126334848 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a810bbf0 con 0x7fe1a8101db0 2023-11-16T16:47:39.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.435+0000 7fe18ffff700 1 -- 172.21.15.44:0/3126334848 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0005990 con 0x7fe1a8101db0 2023-11-16T16:47:39.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.435+0000 7fe1ad5ff700 1 -- 172.21.15.44:0/3126334848 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a810ba60 con 0x7fe1a8101db0 2023-11-16T16:47:39.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.436+0000 7fe1ad5ff700 1 -- 172.21.15.44:0/3126334848 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a81077a0 con 0x7fe1a8101db0 2023-11-16T16:47:39.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.438+0000 7fe18ffff700 1 -- 172.21.15.44:0/3126334848 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe1a000db20 con 0x7fe1a8101db0 2023-11-16T16:47:39.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.440+0000 7fe18ffff700 1 --2- 172.21.15.44:0/3126334848 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe1900a1150 0x7fe1900a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:39.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.440+0000 7fe18ffff700 1 -- 172.21.15.44:0/3126334848 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe1a00cc520 con 0x7fe1a8101db0 2023-11-16T16:47:39.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.440+0000 7fe1a6ffd700 1 --2- 172.21.15.44:0/3126334848 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe1900a1150 0x7fe1900a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:39.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.440+0000 7fe1a6ffd700 1 --2- 172.21.15.44:0/3126334848 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe1900a1150 0x7fe1900a3600 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe198005950 tx=0x7fe198009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:39.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.442+0000 7fe18ffff700 1 -- 172.21.15.44:0/3126334848 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe1a0095a20 con 0x7fe1a8101db0 2023-11-16T16:47:39.669 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.668+0000 7fe1ad5ff700 1 -- 172.21.15.44:0/3126334848 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe1a802cfe0 con 0x7fe1a8101db0 2023-11-16T16:47:39.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.668+0000 7fe18ffff700 1 -- 172.21.15.44:0/3126334848 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) v1 ==== 81+0+591 (secure 0 0 0) 0x7fe1a0099040 con 0x7fe1a8101db0 2023-11-16T16:47:39.670 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1018560948 2023-11-17T16:40:42.449496+0000 2023-11-16T16:47:39.671 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1599940692 2023-11-17T16:40:42.449496+0000 2023-11-16T16:47:39.671 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3409868354 2023-11-17T16:40:42.449496+0000 2023-11-16T16:47:39.671 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/4234382393 2023-11-17T16:40:42.449496+0000 2023-11-16T16:47:39.671 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/1852059921 2023-11-17T16:40:15.683996+0000 2023-11-16T16:47:39.671 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3135397911 2023-11-17T16:40:15.683996+0000 2023-11-16T16:47:39.671 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/1852059921 2023-11-17T16:40:15.683996+0000 2023-11-16T16:47:39.671 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3599504154 2023-11-17T16:40:15.683996+0000 2023-11-16T16:47:39.671 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/264068429 2023-11-17T16:40:15.683996+0000 2023-11-16T16:47:39.671 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/4234382393 2023-11-17T16:40:42.449496+0000 2023-11-16T16:47:39.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.671+0000 7fe1ad5ff700 1 -- 172.21.15.44:0/3126334848 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe1900a1150 msgr2=0x7fe1900a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:39.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.671+0000 7fe1ad5ff700 1 --2- 172.21.15.44:0/3126334848 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe1900a1150 0x7fe1900a3600 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe198005950 tx=0x7fe198009260 comp rx=0 tx=0).stop 2023-11-16T16:47:39.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.671+0000 7fe1ad5ff700 1 -- 172.21.15.44:0/3126334848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a8101db0 msgr2=0x7fe1a819d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:39.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.671+0000 7fe1ad5ff700 1 --2- 172.21.15.44:0/3126334848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a8101db0 0x7fe1a819d560 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0007f10 tx=0x7fe1a000d3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:39.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.671+0000 7fe1ad5ff700 1 -- 172.21.15.44:0/3126334848 shutdown_connections 2023-11-16T16:47:39.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.671+0000 7fe1ad5ff700 1 --2- 172.21.15.44:0/3126334848 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe1900a1150 0x7fe1900a3600 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:39.674 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.671+0000 7fe1ad5ff700 1 --2- 172.21.15.44:0/3126334848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a8101db0 0x7fe1a819d560 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:39.674 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.671+0000 7fe1ad5ff700 1 --2- 172.21.15.44:0/3126334848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1a8100230 0x7fe1a8195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:39.674 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.671+0000 7fe1ad5ff700 1 --2- 172.21.15.44:0/3126334848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1a8101420 0x7fe1a81964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:39.674 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.671+0000 7fe1ad5ff700 1 -- 172.21.15.44:0/3126334848 >> 172.21.15.44:0/3126334848 conn(0x7fe1a80fb810 msgr2=0x7fe1a81048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:39.674 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.671+0000 7fe1ad5ff700 1 -- 172.21.15.44:0/3126334848 shutdown_connections 2023-11-16T16:47:39.674 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:39.671+0000 7fe1ad5ff700 1 -- 172.21.15.44:0/3126334848 wait complete. 2023-11-16T16:47:39.674 INFO:teuthology.orchestra.run.smithi044.stderr:listed 10 entries 2023-11-16T16:47:39.691 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2023-11-16T16:47:39.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:39.692 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-11-16T16:47:39.758 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi044.stderr:ceph-fuse[165166]: fuse finished with error 0 and tester_r 0 2023-11-16T16:47:39.763 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T16:47:39.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:39 smithi044 ceph-mon[128750]: pgmap v285: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T16:47:39.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:39 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2997708949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:47:39.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:39 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/420937266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:47:39.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:39 smithi176 ceph-mon[131630]: pgmap v285: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T16:47:39.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:39 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2997708949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:47:39.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:39 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/420937266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:47:39.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:39 smithi077 ceph-mon[131204]: pgmap v285: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T16:47:39.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:39 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2997708949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:47:39.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:39 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/420937266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:47:40.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:40 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3126334848' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T16:47:40.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:40 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3126334848' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T16:47:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:40 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3126334848' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T16:47:41.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:41 smithi044 ceph-mon[128750]: pgmap v286: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T16:47:41.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:41 smithi176 ceph-mon[131630]: pgmap v286: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T16:47:41.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:41 smithi077 ceph-mon[131204]: pgmap v286: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T16:47:42.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2023-11-16T16:47:42.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]: dispatch 2023-11-16T16:47:42.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2023-11-16T16:47:42.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2023-11-16T16:47:42.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2023-11-16T16:47:42.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2023-11-16T16:47:42.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]: dispatch 2023-11-16T16:47:42.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2023-11-16T16:47:42.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2023-11-16T16:47:42.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:42 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2023-11-16T16:47:42.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2023-11-16T16:47:42.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]: dispatch 2023-11-16T16:47:42.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2023-11-16T16:47:42.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2023-11-16T16:47:42.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2023-11-16T16:47:42.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2023-11-16T16:47:42.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]: dispatch 2023-11-16T16:47:42.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2023-11-16T16:47:42.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2023-11-16T16:47:42.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:42 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2023-11-16T16:47:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2023-11-16T16:47:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]: dispatch 2023-11-16T16:47:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2023-11-16T16:47:43.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2023-11-16T16:47:43.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2023-11-16T16:47:43.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2023-11-16T16:47:43.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]: dispatch 2023-11-16T16:47:43.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2023-11-16T16:47:43.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2023-11-16T16:47:43.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:42 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2023-11-16T16:47:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:43 smithi176 ceph-mon[131630]: pgmap v287: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T16:47:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2023-11-16T16:47:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]': finished 2023-11-16T16:47:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2023-11-16T16:47:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2023-11-16T16:47:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2023-11-16T16:47:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2023-11-16T16:47:43.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]': finished 2023-11-16T16:47:43.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2023-11-16T16:47:43.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2023-11-16T16:47:43.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2023-11-16T16:47:43.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:43 smithi176 ceph-mon[131630]: osdmap e88: 12 total, 12 up, 12 in 2023-11-16T16:47:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:43 smithi077 ceph-mon[131204]: pgmap v287: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T16:47:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2023-11-16T16:47:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]': finished 2023-11-16T16:47:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2023-11-16T16:47:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2023-11-16T16:47:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2023-11-16T16:47:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2023-11-16T16:47:43.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]': finished 2023-11-16T16:47:43.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2023-11-16T16:47:43.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2023-11-16T16:47:43.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2023-11-16T16:47:43.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:43 smithi077 ceph-mon[131204]: osdmap e88: 12 total, 12 up, 12 in 2023-11-16T16:47:44.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:43 smithi044 ceph-mon[128750]: pgmap v287: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T16:47:44.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2023-11-16T16:47:44.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]': finished 2023-11-16T16:47:44.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2023-11-16T16:47:44.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2023-11-16T16:47:44.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2023-11-16T16:47:44.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2023-11-16T16:47:44.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]': finished 2023-11-16T16:47:44.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2023-11-16T16:47:44.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2023-11-16T16:47:44.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2023-11-16T16:47:44.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:43 smithi044 ceph-mon[128750]: osdmap e88: 12 total, 12 up, 12 in 2023-11-16T16:47:44.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:44 smithi176 ceph-mon[131630]: osdmap e89: 12 total, 12 up, 12 in 2023-11-16T16:47:44.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:44 smithi077 ceph-mon[131204]: osdmap e89: 12 total, 12 up, 12 in 2023-11-16T16:47:45.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:44 smithi044 ceph-mon[128750]: osdmap e89: 12 total, 12 up, 12 in 2023-11-16T16:47:45.864 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2023-11-16T16:47:45.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:45.865 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-11-16T16:47:45.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:45 smithi176 ceph-mon[131630]: pgmap v290: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:45.931 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:45.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:45 smithi077 ceph-mon[131204]: pgmap v290: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:45.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:45 smithi044 ceph-mon[128750]: pgmap v290: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:46.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.141+0000 7f04ec7ff700 1 -- 172.21.15.44:0/1821406529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04e4101e10 msgr2=0x7f04e41086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.141+0000 7f04ec7ff700 1 --2- 172.21.15.44:0/1821406529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04e4101e10 0x7f04e41086d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00b0a0 tx=0x7f04dc00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.141+0000 7f04ec7ff700 1 -- 172.21.15.44:0/1821406529 shutdown_connections 2023-11-16T16:47:46.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.141+0000 7f04ec7ff700 1 --2- 172.21.15.44:0/1821406529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04e4101e10 0x7f04e41086d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.141+0000 7f04ec7ff700 1 --2- 172.21.15.44:0/1821406529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04e4100290 0x7f04e41006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.141+0000 7f04ec7ff700 1 --2- 172.21.15.44:0/1821406529 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04e4101480 0x7f04e41018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.141+0000 7f04ec7ff700 1 -- 172.21.15.44:0/1821406529 >> 172.21.15.44:0/1821406529 conn(0x7f04e40fb870 msgr2=0x7f04e40fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:46.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.141+0000 7f04ec7ff700 1 -- 172.21.15.44:0/1821406529 shutdown_connections 2023-11-16T16:47:46.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.141+0000 7f04ec7ff700 1 -- 172.21.15.44:0/1821406529 wait complete. 2023-11-16T16:47:46.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.142+0000 7f04ec7ff700 1 Processor -- start 2023-11-16T16:47:46.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.142+0000 7f04ec7ff700 1 -- start start 2023-11-16T16:47:46.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04ec7ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04e4100290 0x7f04e4198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:46.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04ec7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04e4101480 0x7f04e41987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:46.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04ec7ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04e4101e10 0x7f04e419f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:46.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04ec7ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e419fe10 con 0x7f04e4101e10 2023-11-16T16:47:46.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04ec7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e419ff50 con 0x7f04e4101480 2023-11-16T16:47:46.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04ec7ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e41a00e0 con 0x7f04e4100290 2023-11-16T16:47:46.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04ead9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04e4101e10 0x7f04e419f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:46.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04ea59b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04e4100290 0x7f04e4198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:46.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04ead9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04e4101e10 0x7f04e419f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52688/0 (socket says 172.21.15.44:52688) 2023-11-16T16:47:46.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04ead9c700 1 -- 172.21.15.44:0/3864135856 learned_addr learned my addr 172.21.15.44:0/3864135856 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:46.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04e9d9a700 1 --2- 172.21.15.44:0/3864135856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04e4101480 0x7f04e41987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:46.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04ea59b700 1 -- 172.21.15.44:0/3864135856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04e4101480 msgr2=0x7f04e41987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04ea59b700 1 --2- 172.21.15.44:0/3864135856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04e4101480 0x7f04e41987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04ea59b700 1 -- 172.21.15.44:0/3864135856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04e4101e10 msgr2=0x7f04e419f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04ea59b700 1 --2- 172.21.15.44:0/3864135856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04e4101e10 0x7f04e419f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04ea59b700 1 -- 172.21.15.44:0/3864135856 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc009d00 con 0x7f04e4100290 2023-11-16T16:47:46.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.143+0000 7f04ead9c700 1 --2- 172.21.15.44:0/3864135856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04e4101e10 0x7f04e419f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:46.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.144+0000 7f04ea59b700 1 --2- 172.21.15.44:0/3864135856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04e4100290 0x7f04e4198290 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f04e000e9c0 tx=0x7f04e000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:46.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.144+0000 7f04db7fe700 1 -- 172.21.15.44:0/3864135856 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e0004220 con 0x7f04e4100290 2023-11-16T16:47:46.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.144+0000 7f04ec7ff700 1 -- 172.21.15.44:0/3864135856 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e410bc70 con 0x7f04e4100290 2023-11-16T16:47:46.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.144+0000 7f04db7fe700 1 -- 172.21.15.44:0/3864135856 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f04e0004380 con 0x7f04e4100290 2023-11-16T16:47:46.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.144+0000 7f04db7fe700 1 -- 172.21.15.44:0/3864135856 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e0003be0 con 0x7f04e4100290 2023-11-16T16:47:46.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.144+0000 7f04ec7ff700 1 -- 172.21.15.44:0/3864135856 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e410bae0 con 0x7f04e4100290 2023-11-16T16:47:46.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.145+0000 7f04ec7ff700 1 -- 172.21.15.44:0/3864135856 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e41a0880 con 0x7f04e4100290 2023-11-16T16:47:46.148 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.147+0000 7f04db7fe700 1 -- 172.21.15.44:0/3864135856 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f04e0003d40 con 0x7f04e4100290 2023-11-16T16:47:46.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.148+0000 7f04db7fe700 1 --2- 172.21.15.44:0/3864135856 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f04d00a1150 0x7f04d00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:46.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.149+0000 7f04db7fe700 1 -- 172.21.15.44:0/3864135856 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f04e0014070 con 0x7f04e4100290 2023-11-16T16:47:46.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.149+0000 7f04e9d9a700 1 --2- 172.21.15.44:0/3864135856 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f04d00a1150 0x7f04d00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:46.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.149+0000 7f04e9d9a700 1 --2- 172.21.15.44:0/3864135856 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f04d00a1150 0x7f04d00a3600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f04d4009c60 tx=0x7f04d4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:46.151 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.151+0000 7f04db7fe700 1 -- 172.21.15.44:0/3864135856 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f04e00968d0 con 0x7f04e4100290 2023-11-16T16:47:46.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.357+0000 7f04ec7ff700 1 -- 172.21.15.44:0/3864135856 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04e41a0e30 con 0x7f04e4100290 2023-11-16T16:47:46.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.359+0000 7f04db7fe700 1 -- 172.21.15.44:0/3864135856 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24451 (secure 0 0 0) 0x7f04e001f340 con 0x7f04e4100290 2023-11-16T16:47:46.361 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:46.361 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":89,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:47:43.617828+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.362+0000 7f04ec7ff700 1 -- 172.21.15.44:0/3864135856 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f04d00a1150 msgr2=0x7f04d00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.362+0000 7f04ec7ff700 1 --2- 172.21.15.44:0/3864135856 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f04d00a1150 0x7f04d00a3600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f04d4009c60 tx=0x7f04d4009260 comp rx=0 tx=0).stop 2023-11-16T16:47:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.362+0000 7f04ec7ff700 1 -- 172.21.15.44:0/3864135856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04e4100290 msgr2=0x7f04e4198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.362+0000 7f04ec7ff700 1 --2- 172.21.15.44:0/3864135856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04e4100290 0x7f04e4198290 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f04e000e9c0 tx=0x7f04e000ecd0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.362+0000 7f04ec7ff700 1 -- 172.21.15.44:0/3864135856 shutdown_connections 2023-11-16T16:47:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.362+0000 7f04ec7ff700 1 --2- 172.21.15.44:0/3864135856 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f04d00a1150 0x7f04d00a3600 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.362+0000 7f04ec7ff700 1 --2- 172.21.15.44:0/3864135856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04e4101e10 0x7f04e419f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.362+0000 7f04ec7ff700 1 --2- 172.21.15.44:0/3864135856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04e4100290 0x7f04e4198290 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.362+0000 7f04ec7ff700 1 --2- 172.21.15.44:0/3864135856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04e4101480 0x7f04e41987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.362+0000 7f04ec7ff700 1 -- 172.21.15.44:0/3864135856 >> 172.21.15.44:0/3864135856 conn(0x7f04e40fb870 msgr2=0x7f04e40fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:46.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.362+0000 7f04ec7ff700 1 -- 172.21.15.44:0/3864135856 shutdown_connections 2023-11-16T16:47:46.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.362+0000 7f04ec7ff700 1 -- 172.21.15.44:0/3864135856 wait complete. 2023-11-16T16:47:46.381 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:46.586 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.585+0000 7fbae91ff700 1 -- 172.21.15.44:0/2583948769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbae41014a0 msgr2=0x7fbae41018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.586 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.585+0000 7fbae91ff700 1 --2- 172.21.15.44:0/2583948769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbae41014a0 0x7fbae41018f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009990 tx=0x7fbad8009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.586 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.585+0000 7fbae91ff700 1 -- 172.21.15.44:0/2583948769 shutdown_connections 2023-11-16T16:47:46.586 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.585+0000 7fbae91ff700 1 --2- 172.21.15.44:0/2583948769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbae4101e30 0x7fbae41086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.586 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.585+0000 7fbae91ff700 1 --2- 172.21.15.44:0/2583948769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbae41002b0 0x7fbae41006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.586 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.585+0000 7fbae91ff700 1 --2- 172.21.15.44:0/2583948769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbae41014a0 0x7fbae41018f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.586 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.585+0000 7fbae91ff700 1 -- 172.21.15.44:0/2583948769 >> 172.21.15.44:0/2583948769 conn(0x7fbae40fb890 msgr2=0x7fbae40fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:46.586 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.585+0000 7fbae91ff700 1 -- 172.21.15.44:0/2583948769 shutdown_connections 2023-11-16T16:47:46.587 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.585+0000 7fbae91ff700 1 -- 172.21.15.44:0/2583948769 wait complete. 2023-11-16T16:47:46.588 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.586+0000 7fbae91ff700 1 Processor -- start 2023-11-16T16:47:46.589 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.586+0000 7fbae91ff700 1 -- start start 2023-11-16T16:47:46.589 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.587+0000 7fbae91ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbae41002b0 0x7fbae4196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:46.589 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.587+0000 7fbae91ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbae41014a0 0x7fbae4196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:46.589 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.587+0000 7fbae91ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbae4101e30 0x7fbae419d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:46.589 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.587+0000 7fbae91ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae419dba0 con 0x7fbae41014a0 2023-11-16T16:47:46.589 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.587+0000 7fbae91ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae419dce0 con 0x7fbae4101e30 2023-11-16T16:47:46.589 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.587+0000 7fbae91ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae419de70 con 0x7fbae41002b0 2023-11-16T16:47:46.589 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.587+0000 7fbae259c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbae41014a0 0x7fbae4196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:46.589 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.587+0000 7fbae259c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbae41014a0 0x7fbae4196560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52710/0 (socket says 172.21.15.44:52710) 2023-11-16T16:47:46.589 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.587+0000 7fbae2d9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbae41002b0 0x7fbae4196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:46.589 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.587+0000 7fbae259c700 1 -- 172.21.15.44:0/1721475838 learned_addr learned my addr 172.21.15.44:0/1721475838 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:46.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.587+0000 7fbae359e700 1 --2- 172.21.15.44:0/1721475838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbae4101e30 0x7fbae419d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:46.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.587+0000 7fbae259c700 1 -- 172.21.15.44:0/1721475838 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbae41002b0 msgr2=0x7fbae4196020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.587+0000 7fbae259c700 1 --2- 172.21.15.44:0/1721475838 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbae41002b0 0x7fbae4196020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.587+0000 7fbae259c700 1 -- 172.21.15.44:0/1721475838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbae4101e30 msgr2=0x7fbae419d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.587+0000 7fbae259c700 1 --2- 172.21.15.44:0/1721475838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbae4101e30 0x7fbae419d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.587+0000 7fbae259c700 1 -- 172.21.15.44:0/1721475838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbadc009c40 con 0x7fbae41014a0 2023-11-16T16:47:46.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.588+0000 7fbae259c700 1 --2- 172.21.15.44:0/1721475838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbae41014a0 0x7fbae4196560 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbad800b620 tx=0x7fbad8005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:46.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.588+0000 7fbacbfff700 1 -- 172.21.15.44:0/1721475838 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad801d070 con 0x7fbae41014a0 2023-11-16T16:47:46.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.588+0000 7fbacbfff700 1 -- 172.21.15.44:0/1721475838 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbad8022470 con 0x7fbae41014a0 2023-11-16T16:47:46.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.588+0000 7fbae91ff700 1 -- 172.21.15.44:0/1721475838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad80095f0 con 0x7fbae41014a0 2023-11-16T16:47:46.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.588+0000 7fbacbfff700 1 -- 172.21.15.44:0/1721475838 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad8017650 con 0x7fbae41014a0 2023-11-16T16:47:46.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.588+0000 7fbae91ff700 1 -- 172.21.15.44:0/1721475838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae410bc60 con 0x7fbae41014a0 2023-11-16T16:47:46.591 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.589+0000 7fbae91ff700 1 -- 172.21.15.44:0/1721475838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae41078c0 con 0x7fbae41014a0 2023-11-16T16:47:46.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.591+0000 7fbacbfff700 1 -- 172.21.15.44:0/1721475838 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbad80225e0 con 0x7fbae41014a0 2023-11-16T16:47:46.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.592+0000 7fbacbfff700 1 --2- 172.21.15.44:0/1721475838 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbacc0a1150 0x7fbacc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:46.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.593+0000 7fbacbfff700 1 -- 172.21.15.44:0/1721475838 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbad80ce0e0 con 0x7fbae41014a0 2023-11-16T16:47:46.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.593+0000 7fbae2d9d700 1 --2- 172.21.15.44:0/1721475838 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbacc0a1150 0x7fbacc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:46.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.593+0000 7fbae2d9d700 1 --2- 172.21.15.44:0/1721475838 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbacc0a1150 0x7fbacc0a3600 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbae4101300 tx=0x7fbad4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:46.598 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.597+0000 7fbacbfff700 1 -- 172.21.15.44:0/1721475838 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbad8097550 con 0x7fbae41014a0 2023-11-16T16:47:46.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.815+0000 7fbae91ff700 1 -- 172.21.15.44:0/1721475838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbae4067a60 con 0x7fbae41014a0 2023-11-16T16:47:46.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.816+0000 7fbacbfff700 1 -- 172.21.15.44:0/1721475838 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24451 (secure 0 0 0) 0x7fbad809ab70 con 0x7fbae41014a0 2023-11-16T16:47:46.817 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:46.818 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":89,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:47:43.617828+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.819+0000 7fbae91ff700 1 -- 172.21.15.44:0/1721475838 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbacc0a1150 msgr2=0x7fbacc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.819+0000 7fbae91ff700 1 --2- 172.21.15.44:0/1721475838 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbacc0a1150 0x7fbacc0a3600 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbae4101300 tx=0x7fbad4009260 comp rx=0 tx=0).stop 2023-11-16T16:47:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.819+0000 7fbae91ff700 1 -- 172.21.15.44:0/1721475838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbae41014a0 msgr2=0x7fbae4196560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.819+0000 7fbae91ff700 1 --2- 172.21.15.44:0/1721475838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbae41014a0 0x7fbae4196560 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbad800b620 tx=0x7fbad8005740 comp rx=0 tx=0).stop 2023-11-16T16:47:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.819+0000 7fbae91ff700 1 -- 172.21.15.44:0/1721475838 shutdown_connections 2023-11-16T16:47:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.819+0000 7fbae91ff700 1 --2- 172.21.15.44:0/1721475838 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbacc0a1150 0x7fbacc0a3600 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.819+0000 7fbae91ff700 1 --2- 172.21.15.44:0/1721475838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbae4101e30 0x7fbae419d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.819+0000 7fbae91ff700 1 --2- 172.21.15.44:0/1721475838 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbae41002b0 0x7fbae4196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.819+0000 7fbae91ff700 1 --2- 172.21.15.44:0/1721475838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbae41014a0 0x7fbae4196560 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.819+0000 7fbae91ff700 1 -- 172.21.15.44:0/1721475838 >> 172.21.15.44:0/1721475838 conn(0x7fbae40fb890 msgr2=0x7fbae4104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.819+0000 7fbae91ff700 1 -- 172.21.15.44:0/1721475838 shutdown_connections 2023-11-16T16:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:46.819+0000 7fbae91ff700 1 -- 172.21.15.44:0/1721475838 wait complete. 2023-11-16T16:47:46.839 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-16T16:47:46.900 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:46 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3864135856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:46.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:46 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3864135856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:47.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.045+0000 7f654ebff700 1 -- 172.21.15.44:0/544876645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65481002c0 msgr2=0x7f65481006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.045+0000 7f654ebff700 1 --2- 172.21.15.44:0/544876645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65481002c0 0x7f65481006d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f653c009990 tx=0x7f653c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.045+0000 7f654ebff700 1 -- 172.21.15.44:0/544876645 shutdown_connections 2023-11-16T16:47:47.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.045+0000 7f654ebff700 1 --2- 172.21.15.44:0/544876645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6548101e40 0x7f6548108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.045+0000 7f654ebff700 1 --2- 172.21.15.44:0/544876645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65481002c0 0x7f65481006d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.045+0000 7f654ebff700 1 --2- 172.21.15.44:0/544876645 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65481014b0 0x7f6548101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.045+0000 7f654ebff700 1 -- 172.21.15.44:0/544876645 >> 172.21.15.44:0/544876645 conn(0x7f65480fb8a0 msgr2=0x7f65480fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:47.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.045+0000 7f654ebff700 1 -- 172.21.15.44:0/544876645 shutdown_connections 2023-11-16T16:47:47.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.045+0000 7f654ebff700 1 -- 172.21.15.44:0/544876645 wait complete. 2023-11-16T16:47:47.048 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.046+0000 7f654ebff700 1 Processor -- start 2023-11-16T16:47:47.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.046+0000 7f654ebff700 1 -- start start 2023-11-16T16:47:47.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f654ebff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65481002c0 0x7f65480748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:47.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f654ebff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65481014b0 0x7f6548072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:47.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f654ebff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6548101e40 0x7f6548073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:47.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f654ebff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6548073bc0 con 0x7f6548101e40 2023-11-16T16:47:47.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f654ebff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6548073d30 con 0x7f65481014b0 2023-11-16T16:47:47.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f654ebff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65481a00c0 con 0x7f65481002c0 2023-11-16T16:47:47.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f654d19c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6548101e40 0x7f6548073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:47.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f654d19c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6548101e40 0x7f6548073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52734/0 (socket says 172.21.15.44:52734) 2023-11-16T16:47:47.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f654d19c700 1 -- 172.21.15.44:0/1363561451 learned_addr learned my addr 172.21.15.44:0/1363561451 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:47.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f6547fff700 1 --2- 172.21.15.44:0/1363561451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65481014b0 0x7f6548072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:47.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f654c99b700 1 --2- 172.21.15.44:0/1363561451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65481002c0 0x7f65480748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:47.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f654d19c700 1 -- 172.21.15.44:0/1363561451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65481002c0 msgr2=0x7f65480748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f654d19c700 1 --2- 172.21.15.44:0/1363561451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65481002c0 0x7f65480748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f654d19c700 1 -- 172.21.15.44:0/1363561451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65481014b0 msgr2=0x7f6548072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f654d19c700 1 --2- 172.21.15.44:0/1363561451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65481014b0 0x7f6548072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f654d19c700 1 -- 172.21.15.44:0/1363561451 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c0095f0 con 0x7f6548101e40 2023-11-16T16:47:47.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.047+0000 7f6547fff700 1 --2- 172.21.15.44:0/1363561451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65481014b0 0x7f6548072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:47.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.048+0000 7f654d19c700 1 --2- 172.21.15.44:0/1363561451 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6548101e40 0x7f6548073420 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f654000ef30 tx=0x7f654000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:47.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.048+0000 7f6545ffb700 1 -- 172.21.15.44:0/1363561451 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540018070 con 0x7f6548101e40 2023-11-16T16:47:47.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.048+0000 7f6545ffb700 1 -- 172.21.15.44:0/1363561451 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f654000f040 con 0x7f6548101e40 2023-11-16T16:47:47.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.048+0000 7f6545ffb700 1 -- 172.21.15.44:0/1363561451 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654000cd00 con 0x7f6548101e40 2023-11-16T16:47:47.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.048+0000 7f654ebff700 1 -- 172.21.15.44:0/1363561451 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f654810bc90 con 0x7f6548101e40 2023-11-16T16:47:47.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.048+0000 7f654ebff700 1 -- 172.21.15.44:0/1363561451 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f654810bb00 con 0x7f6548101e40 2023-11-16T16:47:47.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.049+0000 7f654ebff700 1 -- 172.21.15.44:0/1363561451 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65481a0700 con 0x7f6548101e40 2023-11-16T16:47:47.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.051+0000 7f6545ffb700 1 -- 172.21.15.44:0/1363561451 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6540003a00 con 0x7f6548101e40 2023-11-16T16:47:47.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.052+0000 7f6545ffb700 1 --2- 172.21.15.44:0/1363561451 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f65380a1150 0x7f65380a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:47.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.052+0000 7f6545ffb700 1 -- 172.21.15.44:0/1363561451 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f65400cc6f0 con 0x7f6548101e40 2023-11-16T16:47:47.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.053+0000 7f654c99b700 1 --2- 172.21.15.44:0/1363561451 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f65380a1150 0x7f65380a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:47.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.053+0000 7f654c99b700 1 --2- 172.21.15.44:0/1363561451 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f65380a1150 0x7f65380a3600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f653c00b2e0 tx=0x7f653c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:47.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.056+0000 7f6545ffb700 1 -- 172.21.15.44:0/1363561451 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6540095b60 con 0x7f6548101e40 2023-11-16T16:47:47.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:46 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3864135856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:47.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.311+0000 7f654ebff700 1 -- 172.21.15.44:0/1363561451 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6548107170 con 0x7f6548101e40 2023-11-16T16:47:47.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.649+0000 7f6545ffb700 1 -- 172.21.15.44:0/1363561451 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6540099180 con 0x7f6548101e40 2023-11-16T16:47:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.652+0000 7f654ebff700 1 -- 172.21.15.44:0/1363561451 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f65380a1150 msgr2=0x7f65380a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.652+0000 7f654ebff700 1 --2- 172.21.15.44:0/1363561451 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f65380a1150 0x7f65380a3600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f653c00b2e0 tx=0x7f653c005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.652+0000 7f654ebff700 1 -- 172.21.15.44:0/1363561451 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6548101e40 msgr2=0x7f6548073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.652+0000 7f654ebff700 1 --2- 172.21.15.44:0/1363561451 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6548101e40 0x7f6548073420 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f654000ef30 tx=0x7f654000c3a0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.652+0000 7f654ebff700 1 -- 172.21.15.44:0/1363561451 shutdown_connections 2023-11-16T16:47:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.652+0000 7f654ebff700 1 --2- 172.21.15.44:0/1363561451 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f65380a1150 0x7f65380a3600 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.652+0000 7f654ebff700 1 --2- 172.21.15.44:0/1363561451 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6548101e40 0x7f6548073420 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.652+0000 7f654ebff700 1 --2- 172.21.15.44:0/1363561451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65481002c0 0x7f65480748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.652+0000 7f654ebff700 1 --2- 172.21.15.44:0/1363561451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65481014b0 0x7f6548072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.652+0000 7f654ebff700 1 -- 172.21.15.44:0/1363561451 >> 172.21.15.44:0/1363561451 conn(0x7f65480fb8a0 msgr2=0x7f65480fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:47.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.652+0000 7f654ebff700 1 -- 172.21.15.44:0/1363561451 shutdown_connections 2023-11-16T16:47:47.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.652+0000 7f654ebff700 1 -- 172.21.15.44:0/1363561451 wait complete. 2023-11-16T16:47:47.673 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2023-11-16T16:47:47.674 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-11-16T16:47:47.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.882+0000 7fea615ff700 1 -- 172.21.15.44:0/2918228124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea5c108180 msgr2=0x7fea5c10a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.882+0000 7fea615ff700 1 --2- 172.21.15.44:0/2918228124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea5c108180 0x7fea5c10a600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fea5400b0a0 tx=0x7fea5400b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.882+0000 7fea615ff700 1 -- 172.21.15.44:0/2918228124 shutdown_connections 2023-11-16T16:47:47.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.882+0000 7fea615ff700 1 --2- 172.21.15.44:0/2918228124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea5c108180 0x7fea5c10a600 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.882+0000 7fea615ff700 1 --2- 172.21.15.44:0/2918228124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5c074b30 0x7fea5c072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.882+0000 7fea615ff700 1 --2- 172.21.15.44:0/2918228124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fea5c073550 0x7fea5c0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.882+0000 7fea615ff700 1 -- 172.21.15.44:0/2918228124 >> 172.21.15.44:0/2918228124 conn(0x7fea5c0fb8a0 msgr2=0x7fea5c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:47.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.882+0000 7fea615ff700 1 -- 172.21.15.44:0/2918228124 shutdown_connections 2023-11-16T16:47:47.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.882+0000 7fea615ff700 1 -- 172.21.15.44:0/2918228124 wait complete. 2023-11-16T16:47:47.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.883+0000 7fea615ff700 1 Processor -- start 2023-11-16T16:47:47.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.883+0000 7fea615ff700 1 -- start start 2023-11-16T16:47:47.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.883+0000 7fea615ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea5c073550 0x7fea5c1982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:47.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.884+0000 7fea615ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fea5c074b30 0x7fea5c198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:47.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.884+0000 7fea615ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5c108180 0x7fea5c19f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:47.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.884+0000 7fea615ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c19fe40 con 0x7fea5c108180 2023-11-16T16:47:47.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.884+0000 7fea615ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c19ff80 con 0x7fea5c073550 2023-11-16T16:47:47.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.884+0000 7fea615ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c1a0110 con 0x7fea5c074b30 2023-11-16T16:47:47.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.884+0000 7fea5affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea5c073550 0x7fea5c1982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:47.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.884+0000 7fea5affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea5c073550 0x7fea5c1982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59794/0 (socket says 172.21.15.44:59794) 2023-11-16T16:47:47.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.884+0000 7fea5affd700 1 -- 172.21.15.44:0/1158669540 learned_addr learned my addr 172.21.15.44:0/1158669540 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:47.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.884+0000 7fea5b7fe700 1 --2- 172.21.15.44:0/1158669540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5c108180 0x7fea5c19f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:47.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.884+0000 7fea53fff700 1 --2- 172.21.15.44:0/1158669540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fea5c074b30 0x7fea5c198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:47.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.884+0000 7fea5affd700 1 -- 172.21.15.44:0/1158669540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fea5c074b30 msgr2=0x7fea5c198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.884+0000 7fea5affd700 1 --2- 172.21.15.44:0/1158669540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fea5c074b30 0x7fea5c198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.884+0000 7fea5affd700 1 -- 172.21.15.44:0/1158669540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5c108180 msgr2=0x7fea5c19f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.884+0000 7fea5affd700 1 --2- 172.21.15.44:0/1158669540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5c108180 0x7fea5c19f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.884+0000 7fea5affd700 1 -- 172.21.15.44:0/1158669540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4c009520 con 0x7fea5c073550 2023-11-16T16:47:47.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.884+0000 7fea5b7fe700 1 --2- 172.21.15.44:0/1158669540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5c108180 0x7fea5c19f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:47.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.885+0000 7fea5affd700 1 --2- 172.21.15.44:0/1158669540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea5c073550 0x7fea5c1982c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fea4c00ea30 tx=0x7fea4c00ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:47.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.885+0000 7fea58ff9700 1 -- 172.21.15.44:0/1158669540 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c00cbd0 con 0x7fea5c073550 2023-11-16T16:47:47.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.885+0000 7fea58ff9700 1 -- 172.21.15.44:0/1158669540 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea4c00eed0 con 0x7fea5c073550 2023-11-16T16:47:47.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.885+0000 7fea615ff700 1 -- 172.21.15.44:0/1158669540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea54009d00 con 0x7fea5c073550 2023-11-16T16:47:47.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.885+0000 7fea58ff9700 1 -- 172.21.15.44:0/1158669540 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c003d20 con 0x7fea5c073550 2023-11-16T16:47:47.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.885+0000 7fea615ff700 1 -- 172.21.15.44:0/1158669540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea5c10de50 con 0x7fea5c073550 2023-11-16T16:47:47.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.886+0000 7fea615ff700 1 -- 172.21.15.44:0/1158669540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea5c106830 con 0x7fea5c073550 2023-11-16T16:47:47.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.888+0000 7fea58ff9700 1 -- 172.21.15.44:0/1158669540 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fea4c003ee0 con 0x7fea5c073550 2023-11-16T16:47:47.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.890+0000 7fea58ff9700 1 --2- 172.21.15.44:0/1158669540 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fea480a1150 0x7fea480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:47.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.890+0000 7fea58ff9700 1 -- 172.21.15.44:0/1158669540 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fea4c014070 con 0x7fea5c073550 2023-11-16T16:47:47.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.890+0000 7fea53fff700 1 --2- 172.21.15.44:0/1158669540 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fea480a1150 0x7fea480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:47.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.890+0000 7fea53fff700 1 --2- 172.21.15.44:0/1158669540 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fea480a1150 0x7fea480a3600 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fea5c074850 tx=0x7fea44009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:47.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:47.892+0000 7fea58ff9700 1 -- 172.21.15.44:0/1158669540 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fea4c096920 con 0x7fea5c073550 2023-11-16T16:47:47.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:47 smithi077 ceph-mon[131204]: pgmap v291: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2023-11-16T16:47:47.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:47 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1721475838' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:47.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:47 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1363561451' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-11-16T16:47:48.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:47 smithi044 ceph-mon[128750]: pgmap v291: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2023-11-16T16:47:48.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:47 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1721475838' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:48.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:47 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1363561451' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-11-16T16:47:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:47 smithi176 ceph-mon[131630]: pgmap v291: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2023-11-16T16:47:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:47 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1721475838' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:48.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:47 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1363561451' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-11-16T16:47:48.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:48.154+0000 7fea615ff700 1 -- 172.21.15.44:0/1158669540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fea5c067b00 con 0x7fea5c073550 2023-11-16T16:47:48.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:48 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1363561451' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-11-16T16:47:48.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:48 smithi077 ceph-mon[131204]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:48.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:48 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1158669540' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T16:47:48.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:48 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T16:47:49.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:48 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1363561451' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-11-16T16:47:49.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:48 smithi044 ceph-mon[128750]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:49.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:48 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1158669540' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T16:47:49.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:48 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T16:47:49.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:48 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1363561451' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-11-16T16:47:49.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:48 smithi176 ceph-mon[131630]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:49.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:48 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1158669540' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T16:47:49.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:48 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T16:47:49.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.154+0000 7fea58ff9700 1 -- 172.21.15.44:0/1158669540 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) v1 ==== 121+0+0 (secure 0 0 0) 0x7fea4c099f40 con 0x7fea5c073550 2023-11-16T16:47:49.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.157+0000 7fea615ff700 1 -- 172.21.15.44:0/1158669540 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fea480a1150 msgr2=0x7fea480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:49.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.157+0000 7fea615ff700 1 --2- 172.21.15.44:0/1158669540 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fea480a1150 0x7fea480a3600 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fea5c074850 tx=0x7fea44009260 comp rx=0 tx=0).stop 2023-11-16T16:47:49.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.157+0000 7fea615ff700 1 -- 172.21.15.44:0/1158669540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea5c073550 msgr2=0x7fea5c1982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:49.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.157+0000 7fea615ff700 1 --2- 172.21.15.44:0/1158669540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea5c073550 0x7fea5c1982c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fea4c00ea30 tx=0x7fea4c00ed40 comp rx=0 tx=0).stop 2023-11-16T16:47:49.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.157+0000 7fea615ff700 1 -- 172.21.15.44:0/1158669540 shutdown_connections 2023-11-16T16:47:49.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.157+0000 7fea615ff700 1 --2- 172.21.15.44:0/1158669540 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fea480a1150 0x7fea480a3600 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.157+0000 7fea615ff700 1 --2- 172.21.15.44:0/1158669540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea5c108180 0x7fea5c19f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.157+0000 7fea615ff700 1 --2- 172.21.15.44:0/1158669540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea5c073550 0x7fea5c1982c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.157+0000 7fea615ff700 1 --2- 172.21.15.44:0/1158669540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fea5c074b30 0x7fea5c198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.157+0000 7fea615ff700 1 -- 172.21.15.44:0/1158669540 >> 172.21.15.44:0/1158669540 conn(0x7fea5c0fb8a0 msgr2=0x7fea5c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:49.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.158+0000 7fea615ff700 1 -- 172.21.15.44:0/1158669540 shutdown_connections 2023-11-16T16:47:49.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.158+0000 7fea615ff700 1 -- 172.21.15.44:0/1158669540 wait complete. 2023-11-16T16:47:49.181 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-11-16T16:47:49.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.391+0000 7f467a3ff700 1 -- 172.21.15.44:0/781707099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741014c0 msgr2=0x7f4674101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:49.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.391+0000 7f467a3ff700 1 --2- 172.21.15.44:0/781707099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741014c0 0x7f4674101910 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4668009990 tx=0x7f4668009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.391+0000 7f467a3ff700 1 -- 172.21.15.44:0/781707099 shutdown_connections 2023-11-16T16:47:49.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.391+0000 7f467a3ff700 1 --2- 172.21.15.44:0/781707099 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4674101e50 0x7f4674108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.391+0000 7f467a3ff700 1 --2- 172.21.15.44:0/781707099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46741002d0 0x7f46741006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.391+0000 7f467a3ff700 1 --2- 172.21.15.44:0/781707099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741014c0 0x7f4674101910 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.391+0000 7f467a3ff700 1 -- 172.21.15.44:0/781707099 >> 172.21.15.44:0/781707099 conn(0x7f46740fb8b0 msgr2=0x7f46740fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:49.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.392+0000 7f467a3ff700 1 -- 172.21.15.44:0/781707099 shutdown_connections 2023-11-16T16:47:49.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.392+0000 7f467a3ff700 1 -- 172.21.15.44:0/781707099 wait complete. 2023-11-16T16:47:49.394 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.393+0000 7f467a3ff700 1 Processor -- start 2023-11-16T16:47:49.394 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.393+0000 7f467a3ff700 1 -- start start 2023-11-16T16:47:49.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.394+0000 7f467a3ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741002d0 0x7f46740748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:49.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.394+0000 7f467a3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46741014c0 0x7f4674072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:49.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.394+0000 7f467a3ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4674101e50 0x7f4674073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:49.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.394+0000 7f467a3ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4674073bc0 con 0x7f46741002d0 2023-11-16T16:47:49.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.394+0000 7f467a3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4674073d30 con 0x7f46741014c0 2023-11-16T16:47:49.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.394+0000 7f467a3ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46741a00e0 con 0x7f4674101e50 2023-11-16T16:47:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.394+0000 7f4673fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741002d0 0x7f46740748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.394+0000 7f4673fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741002d0 0x7f46740748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52760/0 (socket says 172.21.15.44:52760) 2023-11-16T16:47:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.394+0000 7f4673fff700 1 -- 172.21.15.44:0/3013150724 learned_addr learned my addr 172.21.15.44:0/3013150724 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.394+0000 7f46737fe700 1 --2- 172.21.15.44:0/3013150724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46741014c0 0x7f4674072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.394+0000 7f467899c700 1 --2- 172.21.15.44:0/3013150724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4674101e50 0x7f4674073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.394+0000 7f4673fff700 1 -- 172.21.15.44:0/3013150724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4674101e50 msgr2=0x7f4674073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.394+0000 7f4673fff700 1 --2- 172.21.15.44:0/3013150724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4674101e50 0x7f4674073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.394+0000 7f4673fff700 1 -- 172.21.15.44:0/3013150724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46741014c0 msgr2=0x7f4674072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.394+0000 7f4673fff700 1 --2- 172.21.15.44:0/3013150724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46741014c0 0x7f4674072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.394+0000 7f4673fff700 1 -- 172.21.15.44:0/3013150724 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46680095f0 con 0x7f46741002d0 2023-11-16T16:47:49.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.395+0000 7f4673fff700 1 --2- 172.21.15.44:0/3013150724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741002d0 0x7f46740748a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f466400e970 tx=0x7f466400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.395+0000 7f46717fa700 1 -- 172.21.15.44:0/3013150724 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4664009790 con 0x7f46741002d0 2023-11-16T16:47:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.395+0000 7f46717fa700 1 -- 172.21.15.44:0/3013150724 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f466400eec0 con 0x7f46741002d0 2023-11-16T16:47:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.395+0000 7f467a3ff700 1 -- 172.21.15.44:0/3013150724 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467410bc80 con 0x7f46741002d0 2023-11-16T16:47:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.395+0000 7f46717fa700 1 -- 172.21.15.44:0/3013150724 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4664003cf0 con 0x7f46741002d0 2023-11-16T16:47:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.395+0000 7f467a3ff700 1 -- 172.21.15.44:0/3013150724 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f467410baf0 con 0x7f46741002d0 2023-11-16T16:47:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.396+0000 7f467a3ff700 1 -- 172.21.15.44:0/3013150724 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46741a0830 con 0x7f46741002d0 2023-11-16T16:47:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.398+0000 7f46717fa700 1 -- 172.21.15.44:0/3013150724 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f466400ccd0 con 0x7f46741002d0 2023-11-16T16:47:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.399+0000 7f46717fa700 1 --2- 172.21.15.44:0/3013150724 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f465c0a1150 0x7f465c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.399+0000 7f46717fa700 1 -- 172.21.15.44:0/3013150724 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4664014070 con 0x7f46741002d0 2023-11-16T16:47:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.400+0000 7f46737fe700 1 --2- 172.21.15.44:0/3013150724 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f465c0a1150 0x7f465c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.400+0000 7f46737fe700 1 --2- 172.21.15.44:0/3013150724 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f465c0a1150 0x7f465c0a3600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f466800b2e0 tx=0x7f4668005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:49.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.404+0000 7f46717fa700 1 -- 172.21.15.44:0/3013150724 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f46640d2020 con 0x7f46741002d0 2023-11-16T16:47:49.668 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:49.668+0000 7f467a3ff700 1 -- 172.21.15.44:0/3013150724 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f4674107200 con 0x7f46741002d0 2023-11-16T16:47:49.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:49 smithi077 ceph-mon[131204]: pgmap v292: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2023-11-16T16:47:49.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:49 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-11-16T16:47:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:49 smithi077 ceph-mon[131204]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:49.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:49 smithi077 ceph-mon[131204]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2023-11-16T16:47:50.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:49 smithi044 ceph-mon[128750]: pgmap v292: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2023-11-16T16:47:50.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:49 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-11-16T16:47:50.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:49 smithi044 ceph-mon[128750]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:50.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:49 smithi044 ceph-mon[128750]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2023-11-16T16:47:50.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:49 smithi176 ceph-mon[131630]: pgmap v292: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2023-11-16T16:47:50.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:49 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-11-16T16:47:50.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:49 smithi176 ceph-mon[131630]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:50.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:49 smithi176 ceph-mon[131630]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2023-11-16T16:47:50.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.665+0000 7f46717fa700 1 -- 172.21.15.44:0/3013150724 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) v1 ==== 115+0+0 (secure 0 0 0) 0x7f46640962b0 con 0x7f46741002d0 2023-11-16T16:47:50.669 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.668+0000 7f467a3ff700 1 -- 172.21.15.44:0/3013150724 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f465c0a1150 msgr2=0x7f465c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:50.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.668+0000 7f467a3ff700 1 --2- 172.21.15.44:0/3013150724 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f465c0a1150 0x7f465c0a3600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f466800b2e0 tx=0x7f4668005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.668+0000 7f467a3ff700 1 -- 172.21.15.44:0/3013150724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741002d0 msgr2=0x7f46740748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:50.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.668+0000 7f467a3ff700 1 --2- 172.21.15.44:0/3013150724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741002d0 0x7f46740748a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f466400e970 tx=0x7f466400ec80 comp rx=0 tx=0).stop 2023-11-16T16:47:50.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.668+0000 7f467a3ff700 1 -- 172.21.15.44:0/3013150724 shutdown_connections 2023-11-16T16:47:50.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.668+0000 7f467a3ff700 1 --2- 172.21.15.44:0/3013150724 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f465c0a1150 0x7f465c0a3600 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.668+0000 7f467a3ff700 1 --2- 172.21.15.44:0/3013150724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4674101e50 0x7f4674073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.668+0000 7f467a3ff700 1 --2- 172.21.15.44:0/3013150724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741002d0 0x7f46740748a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.671 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.668+0000 7f467a3ff700 1 --2- 172.21.15.44:0/3013150724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46741014c0 0x7f4674072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.671 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.668+0000 7f467a3ff700 1 -- 172.21.15.44:0/3013150724 >> 172.21.15.44:0/3013150724 conn(0x7f46740fb8b0 msgr2=0x7f46740fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:50.671 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.668+0000 7f467a3ff700 1 -- 172.21.15.44:0/3013150724 shutdown_connections 2023-11-16T16:47:50.671 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.668+0000 7f467a3ff700 1 -- 172.21.15.44:0/3013150724 wait complete. 2023-11-16T16:47:50.691 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2023-11-16T16:47:50.691 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2023-11-16T16:47:50.906 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.904+0000 7facdb3ff700 1 -- 172.21.15.44:0/4241008390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7facd4074b30 msgr2=0x7facd4072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:50.906 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.904+0000 7facdb3ff700 1 --2- 172.21.15.44:0/4241008390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7facd4074b30 0x7facd4072f80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7facd0009990 tx=0x7facd0009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.906 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.904+0000 7facdb3ff700 1 -- 172.21.15.44:0/4241008390 shutdown_connections 2023-11-16T16:47:50.906 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.904+0000 7facdb3ff700 1 --2- 172.21.15.44:0/4241008390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facd410c890 0x7facd410ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.906 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.904+0000 7facdb3ff700 1 --2- 172.21.15.44:0/4241008390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7facd4074b30 0x7facd4072f80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.906 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.904+0000 7facdb3ff700 1 --2- 172.21.15.44:0/4241008390 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7facd4073550 0x7facd40739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.906 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.904+0000 7facdb3ff700 1 -- 172.21.15.44:0/4241008390 >> 172.21.15.44:0/4241008390 conn(0x7facd40fb8a0 msgr2=0x7facd40fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:50.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.904+0000 7facdb3ff700 1 -- 172.21.15.44:0/4241008390 shutdown_connections 2023-11-16T16:47:50.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.904+0000 7facdb3ff700 1 -- 172.21.15.44:0/4241008390 wait complete. 2023-11-16T16:47:50.909 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.905+0000 7facdb3ff700 1 Processor -- start 2023-11-16T16:47:50.909 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.906+0000 7facdb3ff700 1 -- start start 2023-11-16T16:47:50.909 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.906+0000 7facdb3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facd4073550 0x7facd419c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:50.909 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.906+0000 7facdb3ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7facd4074b30 0x7facd419cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:50.909 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.906+0000 7facdb3ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7facd410c890 0x7facd41a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:50.909 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.906+0000 7facdb3ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd41a4220 con 0x7facd410c890 2023-11-16T16:47:50.909 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.906+0000 7facdb3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd41a4360 con 0x7facd4073550 2023-11-16T16:47:50.909 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.906+0000 7facdb3ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd41a44f0 con 0x7facd4074b30 2023-11-16T16:47:50.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.906+0000 7facd919b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facd4073550 0x7facd419c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:50.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.906+0000 7facd919b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facd4073550 0x7facd419c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59816/0 (socket says 172.21.15.44:59816) 2023-11-16T16:47:50.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.906+0000 7facd919b700 1 -- 172.21.15.44:0/1912388457 learned_addr learned my addr 172.21.15.44:0/1912388457 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:50.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.906+0000 7facd999c700 1 --2- 172.21.15.44:0/1912388457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7facd410c890 0x7facd41a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:50.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.906+0000 7facd899a700 1 --2- 172.21.15.44:0/1912388457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7facd4074b30 0x7facd419cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:50.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.907+0000 7facd919b700 1 -- 172.21.15.44:0/1912388457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7facd4074b30 msgr2=0x7facd419cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:50.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.907+0000 7facd919b700 1 --2- 172.21.15.44:0/1912388457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7facd4074b30 0x7facd419cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.907+0000 7facd919b700 1 -- 172.21.15.44:0/1912388457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7facd410c890 msgr2=0x7facd41a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:50.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.907+0000 7facd919b700 1 --2- 172.21.15.44:0/1912388457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7facd410c890 0x7facd41a3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.907+0000 7facd919b700 1 -- 172.21.15.44:0/1912388457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd00095f0 con 0x7facd4073550 2023-11-16T16:47:50.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.907+0000 7facd899a700 1 --2- 172.21.15.44:0/1912388457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7facd4074b30 0x7facd419cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:50.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.907+0000 7facd919b700 1 --2- 172.21.15.44:0/1912388457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facd4073550 0x7facd419c6a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7facd000b680 tx=0x7facd0005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:50.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.907+0000 7facca7fc700 1 -- 172.21.15.44:0/1912388457 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd001d070 con 0x7facd4073550 2023-11-16T16:47:50.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.907+0000 7facca7fc700 1 -- 172.21.15.44:0/1912388457 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7facd0021410 con 0x7facd4073550 2023-11-16T16:47:50.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.907+0000 7facdb3ff700 1 -- 172.21.15.44:0/1912388457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd4112280 con 0x7facd4073550 2023-11-16T16:47:50.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.907+0000 7facca7fc700 1 -- 172.21.15.44:0/1912388457 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd0017650 con 0x7facd4073550 2023-11-16T16:47:50.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.907+0000 7facdb3ff700 1 -- 172.21.15.44:0/1912388457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd41120f0 con 0x7facd4073550 2023-11-16T16:47:50.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.908+0000 7facdb3ff700 1 -- 172.21.15.44:0/1912388457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd41023e0 con 0x7facd4073550 2023-11-16T16:47:50.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.910+0000 7facca7fc700 1 -- 172.21.15.44:0/1912388457 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7facd0021580 con 0x7facd4073550 2023-11-16T16:47:50.913 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.912+0000 7facca7fc700 1 --2- 172.21.15.44:0/1912388457 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7facc00a1150 0x7facc00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:50.913 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.912+0000 7facca7fc700 1 -- 172.21.15.44:0/1912388457 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7facd00ce2c0 con 0x7facd4073550 2023-11-16T16:47:50.913 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.912+0000 7facd899a700 1 --2- 172.21.15.44:0/1912388457 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7facc00a1150 0x7facc00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:50.913 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.913+0000 7facd899a700 1 --2- 172.21.15.44:0/1912388457 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7facc00a1150 0x7facc00a3600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7facd4074850 tx=0x7facc400b670 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:50.914 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:50.914+0000 7facca7fc700 1 -- 172.21.15.44:0/1912388457 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7facd0097730 con 0x7facd4073550 2023-11-16T16:47:50.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:50 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3013150724' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-11-16T16:47:51.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:50 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3013150724' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-11-16T16:47:51.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:50 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3013150724' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-11-16T16:47:51.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.154+0000 7facdb3ff700 1 -- 172.21.15.44:0/1912388457 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7facd4061bc0 con 0x7facc00a1150 2023-11-16T16:47:51.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.194+0000 7facca7fc700 1 -- 172.21.15.44:0/1912388457 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7facd4061bc0 con 0x7facc00a1150 2023-11-16T16:47:51.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.196+0000 7facdb3ff700 1 -- 172.21.15.44:0/1912388457 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7facc00a1150 msgr2=0x7facc00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:51.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.196+0000 7facdb3ff700 1 --2- 172.21.15.44:0/1912388457 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7facc00a1150 0x7facc00a3600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7facd4074850 tx=0x7facc400b670 comp rx=0 tx=0).stop 2023-11-16T16:47:51.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.197+0000 7facdb3ff700 1 -- 172.21.15.44:0/1912388457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facd4073550 msgr2=0x7facd419c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:51.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.197+0000 7facdb3ff700 1 --2- 172.21.15.44:0/1912388457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facd4073550 0x7facd419c6a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7facd000b680 tx=0x7facd0005740 comp rx=0 tx=0).stop 2023-11-16T16:47:51.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.197+0000 7facdb3ff700 1 -- 172.21.15.44:0/1912388457 shutdown_connections 2023-11-16T16:47:51.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.197+0000 7facdb3ff700 1 --2- 172.21.15.44:0/1912388457 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7facc00a1150 0x7facc00a3600 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.197+0000 7facdb3ff700 1 --2- 172.21.15.44:0/1912388457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7facd410c890 0x7facd41a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.197+0000 7facdb3ff700 1 --2- 172.21.15.44:0/1912388457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facd4073550 0x7facd419c6a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.197+0000 7facdb3ff700 1 --2- 172.21.15.44:0/1912388457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7facd4074b30 0x7facd419cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.197+0000 7facdb3ff700 1 -- 172.21.15.44:0/1912388457 >> 172.21.15.44:0/1912388457 conn(0x7facd40fb8a0 msgr2=0x7facd40fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:51.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.197+0000 7facdb3ff700 1 -- 172.21.15.44:0/1912388457 shutdown_connections 2023-11-16T16:47:51.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.197+0000 7facdb3ff700 1 -- 172.21.15.44:0/1912388457 wait complete. 2023-11-16T16:47:51.216 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2023-11-16T16:47:51.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.424+0000 7fbc611ff700 1 -- 172.21.15.44:0/866705930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc5c1014c0 msgr2=0x7fbc5c101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:51.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.424+0000 7fbc611ff700 1 --2- 172.21.15.44:0/866705930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc5c1014c0 0x7fbc5c101910 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbc50009990 tx=0x7fbc50009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.424+0000 7fbc611ff700 1 -- 172.21.15.44:0/866705930 shutdown_connections 2023-11-16T16:47:51.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.424+0000 7fbc611ff700 1 --2- 172.21.15.44:0/866705930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc5c101e50 0x7fbc5c108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.424+0000 7fbc611ff700 1 --2- 172.21.15.44:0/866705930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc5c1002d0 0x7fbc5c1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.424+0000 7fbc611ff700 1 --2- 172.21.15.44:0/866705930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc5c1014c0 0x7fbc5c101910 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.424+0000 7fbc611ff700 1 -- 172.21.15.44:0/866705930 >> 172.21.15.44:0/866705930 conn(0x7fbc5c0fb8b0 msgr2=0x7fbc5c0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:51.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.424+0000 7fbc611ff700 1 -- 172.21.15.44:0/866705930 shutdown_connections 2023-11-16T16:47:51.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.424+0000 7fbc611ff700 1 -- 172.21.15.44:0/866705930 wait complete. 2023-11-16T16:47:51.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.425+0000 7fbc611ff700 1 Processor -- start 2023-11-16T16:47:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.425+0000 7fbc611ff700 1 -- start start 2023-11-16T16:47:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc611ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc5c1002d0 0x7fbc5c198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc611ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc5c1014c0 0x7fbc5c1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc611ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc5c101e50 0x7fbc5c19f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc611ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c19fde0 con 0x7fbc5c1002d0 2023-11-16T16:47:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc611ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c19ff20 con 0x7fbc5c1014c0 2023-11-16T16:47:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc611ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c1a00b0 con 0x7fbc5c101e50 2023-11-16T16:47:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc5ad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc5c1002d0 0x7fbc5c198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc5ad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc5c1002d0 0x7fbc5c198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52806/0 (socket says 172.21.15.44:52806) 2023-11-16T16:47:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc5ad9d700 1 -- 172.21.15.44:0/2473720960 learned_addr learned my addr 172.21.15.44:0/2473720960 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc5a59c700 1 --2- 172.21.15.44:0/2473720960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc5c1014c0 0x7fbc5c1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc5b59e700 1 --2- 172.21.15.44:0/2473720960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc5c101e50 0x7fbc5c19f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc5ad9d700 1 -- 172.21.15.44:0/2473720960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc5c101e50 msgr2=0x7fbc5c19f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc5ad9d700 1 --2- 172.21.15.44:0/2473720960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc5c101e50 0x7fbc5c19f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc5ad9d700 1 -- 172.21.15.44:0/2473720960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc5c1014c0 msgr2=0x7fbc5c1987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc5ad9d700 1 --2- 172.21.15.44:0/2473720960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc5c1014c0 0x7fbc5c1987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc5ad9d700 1 -- 172.21.15.44:0/2473720960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc500095f0 con 0x7fbc5c1002d0 2023-11-16T16:47:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.426+0000 7fbc5b59e700 1 --2- 172.21.15.44:0/2473720960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc5c101e50 0x7fbc5c19f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.427+0000 7fbc5ad9d700 1 --2- 172.21.15.44:0/2473720960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc5c1002d0 0x7fbc5c198260 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00e970 tx=0x7fbc4c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.427+0000 7fbc43fff700 1 -- 172.21.15.44:0/2473720960 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c009790 con 0x7fbc5c1002d0 2023-11-16T16:47:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.427+0000 7fbc611ff700 1 -- 172.21.15.44:0/2473720960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc5c10bc40 con 0x7fbc5c1002d0 2023-11-16T16:47:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.427+0000 7fbc43fff700 1 -- 172.21.15.44:0/2473720960 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbc4c00eec0 con 0x7fbc5c1002d0 2023-11-16T16:47:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.427+0000 7fbc43fff700 1 -- 172.21.15.44:0/2473720960 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c003cf0 con 0x7fbc5c1002d0 2023-11-16T16:47:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.427+0000 7fbc611ff700 1 -- 172.21.15.44:0/2473720960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc5c10bab0 con 0x7fbc5c1002d0 2023-11-16T16:47:51.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.428+0000 7fbc611ff700 1 -- 172.21.15.44:0/2473720960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc5c1a0850 con 0x7fbc5c1002d0 2023-11-16T16:47:51.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.430+0000 7fbc43fff700 1 -- 172.21.15.44:0/2473720960 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbc4c00ccd0 con 0x7fbc5c1002d0 2023-11-16T16:47:51.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.431+0000 7fbc43fff700 1 --2- 172.21.15.44:0/2473720960 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbc440a11d0 0x7fbc440a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:51.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.432+0000 7fbc43fff700 1 -- 172.21.15.44:0/2473720960 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbc4c014070 con 0x7fbc5c1002d0 2023-11-16T16:47:51.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.432+0000 7fbc5a59c700 1 --2- 172.21.15.44:0/2473720960 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbc440a11d0 0x7fbc440a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:51.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.432+0000 7fbc5a59c700 1 --2- 172.21.15.44:0/2473720960 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbc440a11d0 0x7fbc440a3680 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000b620 tx=0x7fbc50005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:51.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.433+0000 7fbc43fff700 1 -- 172.21.15.44:0/2473720960 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbc4c0963d0 con 0x7fbc5c1002d0 2023-11-16T16:47:51.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:47:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:47:51.161+0000 7f75ba603700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:47:51.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:47:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:47:51.161+0000 7f75ba603700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:47:51.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:47:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:47:51.161+0000 7f75ba603700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:47:51.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:47:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:47:51.161+0000 7f75ba603700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:47:51.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:47:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:47:51.161+0000 7f75ba603700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:47:51.681 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.681+0000 7fbc611ff700 1 -- 172.21.15.44:0/2473720960 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc5c1a0e00 con 0x7fbc440a11d0 2023-11-16T16:47:51.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.686+0000 7fbc43fff700 1 -- 172.21.15.44:0/2473720960 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7fbc5c1a0e00 con 0x7fbc440a11d0 2023-11-16T16:47:51.686 INFO:teuthology.orchestra.run.smithi044.stdout:/volumes/_nogroup/sv_0/60cbb175-e2d1-4fdb-87af-472261279681 2023-11-16T16:47:51.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.688+0000 7fbc611ff700 1 -- 172.21.15.44:0/2473720960 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbc440a11d0 msgr2=0x7fbc440a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:51.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.688+0000 7fbc611ff700 1 --2- 172.21.15.44:0/2473720960 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbc440a11d0 0x7fbc440a3680 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000b620 tx=0x7fbc50005960 comp rx=0 tx=0).stop 2023-11-16T16:47:51.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.688+0000 7fbc611ff700 1 -- 172.21.15.44:0/2473720960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc5c1002d0 msgr2=0x7fbc5c198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:51.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.688+0000 7fbc611ff700 1 --2- 172.21.15.44:0/2473720960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc5c1002d0 0x7fbc5c198260 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00e970 tx=0x7fbc4c00ec80 comp rx=0 tx=0).stop 2023-11-16T16:47:51.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.688+0000 7fbc611ff700 1 -- 172.21.15.44:0/2473720960 shutdown_connections 2023-11-16T16:47:51.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.688+0000 7fbc611ff700 1 --2- 172.21.15.44:0/2473720960 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbc440a11d0 0x7fbc440a3680 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.688+0000 7fbc611ff700 1 --2- 172.21.15.44:0/2473720960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc5c101e50 0x7fbc5c19f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.688+0000 7fbc611ff700 1 --2- 172.21.15.44:0/2473720960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc5c1002d0 0x7fbc5c198260 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.688+0000 7fbc611ff700 1 --2- 172.21.15.44:0/2473720960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc5c1014c0 0x7fbc5c1987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.688+0000 7fbc611ff700 1 -- 172.21.15.44:0/2473720960 >> 172.21.15.44:0/2473720960 conn(0x7fbc5c0fb8b0 msgr2=0x7fbc5c0fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:51.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.688+0000 7fbc611ff700 1 -- 172.21.15.44:0/2473720960 shutdown_connections 2023-11-16T16:47:51.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.689+0000 7fbc611ff700 1 -- 172.21.15.44:0/2473720960 wait complete. 2023-11-16T16:47:51.708 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2023-11-16T16:47:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.916+0000 7fa0043ff700 1 -- 172.21.15.44:0/732154756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ffc101e40 msgr2=0x7f9ffc108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.916+0000 7fa0043ff700 1 --2- 172.21.15.44:0/732154756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ffc101e40 0x7f9ffc108700 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400b0a0 tx=0x7f9ff400b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.916+0000 7fa0043ff700 1 -- 172.21.15.44:0/732154756 shutdown_connections 2023-11-16T16:47:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.916+0000 7fa0043ff700 1 --2- 172.21.15.44:0/732154756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ffc101e40 0x7f9ffc108700 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.916+0000 7fa0043ff700 1 --2- 172.21.15.44:0/732154756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ffc1002c0 0x7f9ffc1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.916+0000 7fa0043ff700 1 --2- 172.21.15.44:0/732154756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ffc1014b0 0x7f9ffc101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.916+0000 7fa0043ff700 1 -- 172.21.15.44:0/732154756 >> 172.21.15.44:0/732154756 conn(0x7f9ffc0fb8a0 msgr2=0x7f9ffc0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:51.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.916+0000 7fa0043ff700 1 -- 172.21.15.44:0/732154756 shutdown_connections 2023-11-16T16:47:51.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.916+0000 7fa0043ff700 1 -- 172.21.15.44:0/732154756 wait complete. 2023-11-16T16:47:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.917+0000 7fa0043ff700 1 Processor -- start 2023-11-16T16:47:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.917+0000 7fa0043ff700 1 -- start start 2023-11-16T16:47:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa0043ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ffc1002c0 0x7f9ffc0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa0043ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ffc1014b0 0x7f9ffc072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa0043ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ffc101e40 0x7f9ffc073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa0043ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc073bc0 con 0x7f9ffc1002c0 2023-11-16T16:47:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa0043ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc073d30 con 0x7f9ffc1014b0 2023-11-16T16:47:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa0043ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc1a0230 con 0x7f9ffc101e40 2023-11-16T16:47:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa00219b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ffc1002c0 0x7f9ffc0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa00219b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ffc1002c0 0x7f9ffc0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52822/0 (socket says 172.21.15.44:52822) 2023-11-16T16:47:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa00299c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ffc101e40 0x7f9ffc073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa00219b700 1 -- 172.21.15.44:0/1177681358 learned_addr learned my addr 172.21.15.44:0/1177681358 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:51.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa00199a700 1 --2- 172.21.15.44:0/1177681358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ffc1014b0 0x7f9ffc072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:51.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa00299c700 1 -- 172.21.15.44:0/1177681358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ffc1014b0 msgr2=0x7f9ffc072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:51.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa00299c700 1 --2- 172.21.15.44:0/1177681358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ffc1014b0 0x7f9ffc072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa00299c700 1 -- 172.21.15.44:0/1177681358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ffc1002c0 msgr2=0x7f9ffc0748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:51.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa00299c700 1 --2- 172.21.15.44:0/1177681358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ffc1002c0 0x7f9ffc0748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa00299c700 1 -- 172.21.15.44:0/1177681358 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff4009d00 con 0x7f9ffc101e40 2023-11-16T16:47:51.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.918+0000 7fa00219b700 1 --2- 172.21.15.44:0/1177681358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ffc1002c0 0x7f9ffc0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:51.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.919+0000 7fa00299c700 1 --2- 172.21.15.44:0/1177681358 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ffc101e40 0x7f9ffc073420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400b070 tx=0x7f9ff400bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:51.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.919+0000 7f9ff37fe700 1 -- 172.21.15.44:0/1177681358 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff400e040 con 0x7f9ffc101e40 2023-11-16T16:47:51.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.919+0000 7f9ff37fe700 1 -- 172.21.15.44:0/1177681358 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ff4009480 con 0x7f9ffc101e40 2023-11-16T16:47:51.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.919+0000 7fa0043ff700 1 -- 172.21.15.44:0/1177681358 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ffc10bc90 con 0x7f9ffc101e40 2023-11-16T16:47:51.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.919+0000 7f9ff37fe700 1 -- 172.21.15.44:0/1177681358 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff40052a0 con 0x7f9ffc101e40 2023-11-16T16:47:51.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.919+0000 7fa0043ff700 1 -- 172.21.15.44:0/1177681358 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ffc10bb00 con 0x7f9ffc101e40 2023-11-16T16:47:51.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.920+0000 7fa0043ff700 1 -- 172.21.15.44:0/1177681358 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ffc1077b0 con 0x7f9ffc101e40 2023-11-16T16:47:51.923 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.922+0000 7f9ff37fe700 1 -- 172.21.15.44:0/1177681358 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9ff401d480 con 0x7f9ffc101e40 2023-11-16T16:47:51.924 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.923+0000 7f9ff37fe700 1 --2- 172.21.15.44:0/1177681358 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9fe80a1150 0x7f9fe80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:51.924 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.924+0000 7f9ff37fe700 1 -- 172.21.15.44:0/1177681358 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9ff40ce100 con 0x7f9ffc101e40 2023-11-16T16:47:51.924 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.924+0000 7fa00219b700 1 --2- 172.21.15.44:0/1177681358 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9fe80a1150 0x7f9fe80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:51.925 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.924+0000 7fa00219b700 1 --2- 172.21.15.44:0/1177681358 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9fe80a1150 0x7f9fe80a3600 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8005950 tx=0x7f9ff8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:51.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:51.926+0000 7f9ff37fe700 1 -- 172.21.15.44:0/1177681358 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9ff4097570 con 0x7f9ffc101e40 2023-11-16T16:47:51.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:51 smithi077 ceph-mon[131204]: pgmap v293: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 127 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2023-11-16T16:47:51.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:51 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3013150724' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-11-16T16:47:51.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:51 smithi077 ceph-mon[131204]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2023-11-16T16:47:52.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:51 smithi044 ceph-mon[128750]: pgmap v293: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 127 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2023-11-16T16:47:52.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:51 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3013150724' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-11-16T16:47:52.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:51 smithi044 ceph-mon[128750]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2023-11-16T16:47:52.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:51 smithi176 ceph-mon[131630]: pgmap v293: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 127 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2023-11-16T16:47:52.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:51 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3013150724' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-11-16T16:47:52.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:51 smithi176 ceph-mon[131630]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2023-11-16T16:47:52.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.170+0000 7fa0043ff700 1 -- 172.21.15.44:0/1177681358 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ffc061c10 con 0x7f9fe80a1150 2023-11-16T16:47:52.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.192+0000 7f9ff37fe700 1 -- 172.21.15.44:0/1177681358 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9ffc061c10 con 0x7f9fe80a1150 2023-11-16T16:47:52.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.194+0000 7fa0043ff700 1 -- 172.21.15.44:0/1177681358 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9fe80a1150 msgr2=0x7f9fe80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:52.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.194+0000 7fa0043ff700 1 --2- 172.21.15.44:0/1177681358 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9fe80a1150 0x7f9fe80a3600 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8005950 tx=0x7f9ff8009260 comp rx=0 tx=0).stop 2023-11-16T16:47:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.194+0000 7fa0043ff700 1 -- 172.21.15.44:0/1177681358 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ffc101e40 msgr2=0x7f9ffc073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.194+0000 7fa0043ff700 1 --2- 172.21.15.44:0/1177681358 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ffc101e40 0x7f9ffc073420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400b070 tx=0x7f9ff400bc80 comp rx=0 tx=0).stop 2023-11-16T16:47:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.195+0000 7fa0043ff700 1 -- 172.21.15.44:0/1177681358 shutdown_connections 2023-11-16T16:47:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.195+0000 7fa0043ff700 1 --2- 172.21.15.44:0/1177681358 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9fe80a1150 0x7f9fe80a3600 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.195+0000 7fa0043ff700 1 --2- 172.21.15.44:0/1177681358 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ffc101e40 0x7f9ffc073420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.195+0000 7fa0043ff700 1 --2- 172.21.15.44:0/1177681358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ffc1002c0 0x7f9ffc0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.195+0000 7fa0043ff700 1 --2- 172.21.15.44:0/1177681358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ffc1014b0 0x7f9ffc072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.195+0000 7fa0043ff700 1 -- 172.21.15.44:0/1177681358 >> 172.21.15.44:0/1177681358 conn(0x7f9ffc0fb8a0 msgr2=0x7f9ffc1048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.195+0000 7fa0043ff700 1 -- 172.21.15.44:0/1177681358 shutdown_connections 2023-11-16T16:47:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.195+0000 7fa0043ff700 1 -- 172.21.15.44:0/1177681358 wait complete. 2023-11-16T16:47:52.215 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2023-11-16T16:47:52.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.421+0000 7f3de33ff700 1 -- 172.21.15.44:0/1080251306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ddc1002d0 msgr2=0x7f3ddc1006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:52.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.421+0000 7f3de33ff700 1 --2- 172.21.15.44:0/1080251306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ddc1002d0 0x7f3ddc1006e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8009990 tx=0x7f3dd8009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.421+0000 7f3de33ff700 1 -- 172.21.15.44:0/1080251306 shutdown_connections 2023-11-16T16:47:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.421+0000 7f3de33ff700 1 --2- 172.21.15.44:0/1080251306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ddc101e50 0x7f3ddc108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.421+0000 7f3de33ff700 1 --2- 172.21.15.44:0/1080251306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ddc1002d0 0x7f3ddc1006e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.421+0000 7f3de33ff700 1 --2- 172.21.15.44:0/1080251306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ddc1014c0 0x7f3ddc101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.421+0000 7f3de33ff700 1 -- 172.21.15.44:0/1080251306 >> 172.21.15.44:0/1080251306 conn(0x7f3ddc0fb8b0 msgr2=0x7f3ddc0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.421+0000 7f3de33ff700 1 -- 172.21.15.44:0/1080251306 shutdown_connections 2023-11-16T16:47:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.421+0000 7f3de33ff700 1 -- 172.21.15.44:0/1080251306 wait complete. 2023-11-16T16:47:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.423+0000 7f3de33ff700 1 Processor -- start 2023-11-16T16:47:52.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.423+0000 7f3de33ff700 1 -- start start 2023-11-16T16:47:52.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.423+0000 7f3de33ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ddc1002d0 0x7f3ddc1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:52.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.423+0000 7f3de33ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ddc1014c0 0x7f3ddc1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:52.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.423+0000 7f3de33ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ddc101e50 0x7f3ddc19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:52.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.423+0000 7f3de33ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc19fe30 con 0x7f3ddc101e50 2023-11-16T16:47:52.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.423+0000 7f3de33ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc19ff70 con 0x7f3ddc1014c0 2023-11-16T16:47:52.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.424+0000 7f3de33ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc1a0100 con 0x7f3ddc1002d0 2023-11-16T16:47:52.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.424+0000 7f3de199c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ddc101e50 0x7f3ddc19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:52.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.424+0000 7f3de119b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ddc1002d0 0x7f3ddc1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:52.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.424+0000 7f3de199c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ddc101e50 0x7f3ddc19f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52838/0 (socket says 172.21.15.44:52838) 2023-11-16T16:47:52.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.424+0000 7f3de199c700 1 -- 172.21.15.44:0/3838961166 learned_addr learned my addr 172.21.15.44:0/3838961166 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:52.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.424+0000 7f3de099a700 1 --2- 172.21.15.44:0/3838961166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ddc1014c0 0x7f3ddc1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:52.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.424+0000 7f3de199c700 1 -- 172.21.15.44:0/3838961166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ddc1002d0 msgr2=0x7f3ddc1982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:52.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.424+0000 7f3de199c700 1 --2- 172.21.15.44:0/3838961166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ddc1002d0 0x7f3ddc1982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.424+0000 7f3de199c700 1 -- 172.21.15.44:0/3838961166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ddc1014c0 msgr2=0x7f3ddc1987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:52.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.424+0000 7f3de199c700 1 --2- 172.21.15.44:0/3838961166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ddc1014c0 0x7f3ddc1987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.424+0000 7f3de199c700 1 -- 172.21.15.44:0/3838961166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd80095f0 con 0x7f3ddc101e50 2023-11-16T16:47:52.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.424+0000 7f3de199c700 1 --2- 172.21.15.44:0/3838961166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ddc101e50 0x7f3ddc19f860 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3dd400ef30 tx=0x7f3dd400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:52.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.425+0000 7f3dd27fc700 1 -- 172.21.15.44:0/3838961166 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd4018070 con 0x7f3ddc101e50 2023-11-16T16:47:52.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.425+0000 7f3dd27fc700 1 -- 172.21.15.44:0/3838961166 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3dd400f040 con 0x7f3ddc101e50 2023-11-16T16:47:52.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.425+0000 7f3de33ff700 1 -- 172.21.15.44:0/3838961166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ddc10bca0 con 0x7f3ddc101e50 2023-11-16T16:47:52.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.425+0000 7f3dd27fc700 1 -- 172.21.15.44:0/3838961166 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd400cd00 con 0x7f3ddc101e50 2023-11-16T16:47:52.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.425+0000 7f3de33ff700 1 -- 172.21.15.44:0/3838961166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ddc10bb10 con 0x7f3ddc101e50 2023-11-16T16:47:52.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.426+0000 7f3de33ff700 1 -- 172.21.15.44:0/3838961166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ddc192a00 con 0x7f3ddc101e50 2023-11-16T16:47:52.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.428+0000 7f3dd27fc700 1 -- 172.21.15.44:0/3838961166 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3dd4003a00 con 0x7f3ddc101e50 2023-11-16T16:47:52.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.429+0000 7f3dd27fc700 1 --2- 172.21.15.44:0/3838961166 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3dc80a1150 0x7f3dc80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:52.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.429+0000 7f3dd27fc700 1 -- 172.21.15.44:0/3838961166 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3dd40cc7a0 con 0x7f3ddc101e50 2023-11-16T16:47:52.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.430+0000 7f3de119b700 1 --2- 172.21.15.44:0/3838961166 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3dc80a1150 0x7f3dc80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:52.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.430+0000 7f3de119b700 1 --2- 172.21.15.44:0/3838961166 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3dc80a1150 0x7f3dc80a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800b2e0 tx=0x7f3dd8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:52.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.433+0000 7f3dd27fc700 1 -- 172.21.15.44:0/3838961166 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3dd4095c10 con 0x7f3ddc101e50 2023-11-16T16:47:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.681+0000 7f3dd27fc700 1 -- 172.21.15.44:0/3838961166 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3dd4003cb0 con 0x7f3ddc101e50 2023-11-16T16:47:52.685 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.684+0000 7f3de33ff700 1 -- 172.21.15.44:0/3838961166 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ddc061cc0 con 0x7f3dc80a1150 2023-11-16T16:47:52.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.688+0000 7f3dd27fc700 1 -- 172.21.15.44:0/3838961166 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f3ddc061cc0 con 0x7f3dc80a1150 2023-11-16T16:47:52.689 INFO:teuthology.orchestra.run.smithi044.stdout:/volumes/_nogroup/sv_1/fc1dc7c3-3f9b-4705-9c25-f5705d7e927f 2023-11-16T16:47:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.691+0000 7f3de33ff700 1 -- 172.21.15.44:0/3838961166 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3dc80a1150 msgr2=0x7f3dc80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.691+0000 7f3de33ff700 1 --2- 172.21.15.44:0/3838961166 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3dc80a1150 0x7f3dc80a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800b2e0 tx=0x7f3dd8005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.691+0000 7f3de33ff700 1 -- 172.21.15.44:0/3838961166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ddc101e50 msgr2=0x7f3ddc19f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.691+0000 7f3de33ff700 1 --2- 172.21.15.44:0/3838961166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ddc101e50 0x7f3ddc19f860 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3dd400ef30 tx=0x7f3dd400c3a0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.691+0000 7f3de33ff700 1 -- 172.21.15.44:0/3838961166 shutdown_connections 2023-11-16T16:47:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.691+0000 7f3de33ff700 1 --2- 172.21.15.44:0/3838961166 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3dc80a1150 0x7f3dc80a3600 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.691+0000 7f3de33ff700 1 --2- 172.21.15.44:0/3838961166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ddc101e50 0x7f3ddc19f860 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.691+0000 7f3de33ff700 1 --2- 172.21.15.44:0/3838961166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ddc1002d0 0x7f3ddc1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.691+0000 7f3de33ff700 1 --2- 172.21.15.44:0/3838961166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ddc1014c0 0x7f3ddc1987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.691+0000 7f3de33ff700 1 -- 172.21.15.44:0/3838961166 >> 172.21.15.44:0/3838961166 conn(0x7f3ddc0fb8b0 msgr2=0x7f3ddc106d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.691+0000 7f3de33ff700 1 -- 172.21.15.44:0/3838961166 shutdown_connections 2023-11-16T16:47:52.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.691+0000 7f3de33ff700 1 -- 172.21.15.44:0/3838961166 wait complete. 2023-11-16T16:47:52.711 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.914+0000 7f52a67ff700 1 -- 172.21.15.44:0/2102185995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a0074b30 msgr2=0x7f52a0072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.914+0000 7f52a67ff700 1 --2- 172.21.15.44:0/2102185995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a0074b30 0x7f52a0072f80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f52900098f0 tx=0x7f5290009c00 comp rx=0 tx=0).stop 2023-11-16T16:47:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.914+0000 7f52a67ff700 1 -- 172.21.15.44:0/2102185995 shutdown_connections 2023-11-16T16:47:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.914+0000 7f52a67ff700 1 --2- 172.21.15.44:0/2102185995 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52a010c860 0x7f52a010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.914+0000 7f52a67ff700 1 --2- 172.21.15.44:0/2102185995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a0074b30 0x7f52a0072f80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.914+0000 7f52a67ff700 1 --2- 172.21.15.44:0/2102185995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52a0073550 0x7f52a00739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.914+0000 7f52a67ff700 1 -- 172.21.15.44:0/2102185995 >> 172.21.15.44:0/2102185995 conn(0x7f52a00fb870 msgr2=0x7f52a00fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.914+0000 7f52a67ff700 1 -- 172.21.15.44:0/2102185995 shutdown_connections 2023-11-16T16:47:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.914+0000 7f52a67ff700 1 -- 172.21.15.44:0/2102185995 wait complete. 2023-11-16T16:47:52.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.915+0000 7f52a67ff700 1 Processor -- start 2023-11-16T16:47:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.915+0000 7f52a67ff700 1 -- start start 2023-11-16T16:47:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.915+0000 7f52a67ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a0073550 0x7f52a019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.915+0000 7f52a67ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52a0074b30 0x7f52a019cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.915+0000 7f52a67ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52a010c860 0x7f52a01a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.915+0000 7f52a67ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a01a41d0 con 0x7f52a010c860 2023-11-16T16:47:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.915+0000 7f52a67ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a01a4310 con 0x7f52a0073550 2023-11-16T16:47:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.915+0000 7f52a67ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a01a44a0 con 0x7f52a0074b30 2023-11-16T16:47:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.916+0000 7f529ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a0073550 0x7f52a019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.916+0000 7f529f7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52a0074b30 0x7f52a019cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.916+0000 7f52a4d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52a010c860 0x7f52a01a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.916+0000 7f529f7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52a0074b30 0x7f52a019cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:44738/0 (socket says 172.21.15.44:44738) 2023-11-16T16:47:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.916+0000 7f529ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a0073550 0x7f52a019c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59906/0 (socket says 172.21.15.44:59906) 2023-11-16T16:47:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.916+0000 7f529f7fe700 1 -- 172.21.15.44:0/4028242290 learned_addr learned my addr 172.21.15.44:0/4028242290 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.916+0000 7f529f7fe700 1 -- 172.21.15.44:0/4028242290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a0073550 msgr2=0x7f52a019c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.916+0000 7f529f7fe700 1 --2- 172.21.15.44:0/4028242290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a0073550 0x7f52a019c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.916+0000 7f529f7fe700 1 -- 172.21.15.44:0/4028242290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52a010c860 msgr2=0x7f52a01a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.916+0000 7f529f7fe700 1 --2- 172.21.15.44:0/4028242290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52a010c860 0x7f52a01a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.916+0000 7f529f7fe700 1 -- 172.21.15.44:0/4028242290 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52940095f0 con 0x7f52a0074b30 2023-11-16T16:47:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.916+0000 7f529ffff700 1 --2- 172.21.15.44:0/4028242290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a0073550 0x7f52a019c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.916+0000 7f529f7fe700 1 --2- 172.21.15.44:0/4028242290 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52a0074b30 0x7f52a019cb90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f529400ebf0 tx=0x7f529400ef00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.917+0000 7f529d7fa700 1 -- 172.21.15.44:0/4028242290 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5294009a10 con 0x7f52a0074b30 2023-11-16T16:47:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.917+0000 7f529d7fa700 1 -- 172.21.15.44:0/4028242290 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f529400cc90 con 0x7f52a0074b30 2023-11-16T16:47:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.917+0000 7f52a67ff700 1 -- 172.21.15.44:0/4028242290 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5290009520 con 0x7f52a0074b30 2023-11-16T16:47:52.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.917+0000 7f529d7fa700 1 -- 172.21.15.44:0/4028242290 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5294018750 con 0x7f52a0074b30 2023-11-16T16:47:52.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.917+0000 7f52a67ff700 1 -- 172.21.15.44:0/4028242290 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a0112250 con 0x7f52a0074b30 2023-11-16T16:47:52.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.918+0000 7f52a67ff700 1 -- 172.21.15.44:0/4028242290 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a01023b0 con 0x7f52a0074b30 2023-11-16T16:47:52.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.920+0000 7f529d7fa700 1 -- 172.21.15.44:0/4028242290 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f52940188b0 con 0x7f52a0074b30 2023-11-16T16:47:52.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.921+0000 7f529d7fa700 1 --2- 172.21.15.44:0/4028242290 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f52880a11e0 0x7f52880a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:52.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.921+0000 7f529d7fa700 1 -- 172.21.15.44:0/4028242290 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5294014070 con 0x7f52a0074b30 2023-11-16T16:47:52.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.921+0000 7f529ffff700 1 --2- 172.21.15.44:0/4028242290 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f52880a11e0 0x7f52880a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:52.923 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.922+0000 7f529ffff700 1 --2- 172.21.15.44:0/4028242290 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f52880a11e0 0x7f52880a3690 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5290009fd0 tx=0x7f5290019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:52.924 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:52.923+0000 7f529d7fa700 1 -- 172.21.15.44:0/4028242290 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5294096800 con 0x7f52a0074b30 2023-11-16T16:47:52.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:52 smithi077 ceph-mon[131204]: from='client.24785 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:52.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:52 smithi077 ceph-mon[131204]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:53.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:52 smithi044 ceph-mon[128750]: from='client.24785 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:53.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:52 smithi044 ceph-mon[128750]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:53.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:52 smithi176 ceph-mon[131630]: from='client.24785 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:53.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:52 smithi176 ceph-mon[131630]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:53.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.186+0000 7f52a67ff700 1 -- 172.21.15.44:0/4028242290 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52a002cfe0 con 0x7f52a0074b30 2023-11-16T16:47:53.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.187+0000 7f529d7fa700 1 -- 172.21.15.44:0/4028242290 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f5294099e20 con 0x7f52a0074b30 2023-11-16T16:47:53.188 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:53.188 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:47:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.190+0000 7f52a67ff700 1 -- 172.21.15.44:0/4028242290 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f52880a11e0 msgr2=0x7f52880a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.190+0000 7f52a67ff700 1 --2- 172.21.15.44:0/4028242290 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f52880a11e0 0x7f52880a3690 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5290009fd0 tx=0x7f5290019040 comp rx=0 tx=0).stop 2023-11-16T16:47:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.190+0000 7f52a67ff700 1 -- 172.21.15.44:0/4028242290 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52a0074b30 msgr2=0x7f52a019cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.190+0000 7f52a67ff700 1 --2- 172.21.15.44:0/4028242290 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52a0074b30 0x7f52a019cb90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f529400ebf0 tx=0x7f529400ef00 comp rx=0 tx=0).stop 2023-11-16T16:47:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.190+0000 7f52a67ff700 1 -- 172.21.15.44:0/4028242290 shutdown_connections 2023-11-16T16:47:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.190+0000 7f52a67ff700 1 --2- 172.21.15.44:0/4028242290 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f52880a11e0 0x7f52880a3690 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.190+0000 7f52a67ff700 1 --2- 172.21.15.44:0/4028242290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52a010c860 0x7f52a01a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.190+0000 7f52a67ff700 1 --2- 172.21.15.44:0/4028242290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a0073550 0x7f52a019c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.190+0000 7f52a67ff700 1 --2- 172.21.15.44:0/4028242290 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52a0074b30 0x7f52a019cb90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.190+0000 7f52a67ff700 1 -- 172.21.15.44:0/4028242290 >> 172.21.15.44:0/4028242290 conn(0x7f52a00fb870 msgr2=0x7f52a00fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.190+0000 7f52a67ff700 1 -- 172.21.15.44:0/4028242290 shutdown_connections 2023-11-16T16:47:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.190+0000 7f52a67ff700 1 -- 172.21.15.44:0/4028242290 wait complete. 2023-11-16T16:47:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:47:53.210 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:53.413 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.412+0000 7feee17ff700 1 -- 172.21.15.44:0/3730923784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feedc103760 msgr2=0x7feedc103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.413 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.412+0000 7feee17ff700 1 --2- 172.21.15.44:0/3730923784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feedc103760 0x7feedc103bd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7feed0009990 tx=0x7feed0009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.413 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.412+0000 7feee17ff700 1 -- 172.21.15.44:0/3730923784 shutdown_connections 2023-11-16T16:47:53.413 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.412+0000 7feee17ff700 1 --2- 172.21.15.44:0/3730923784 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feedc104110 0x7feedc10a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.414 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.412+0000 7feee17ff700 1 --2- 172.21.15.44:0/3730923784 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feedc1024d0 0x7feedc1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.414 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.412+0000 7feee17ff700 1 --2- 172.21.15.44:0/3730923784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feedc103760 0x7feedc103bd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.414 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.412+0000 7feee17ff700 1 -- 172.21.15.44:0/3730923784 >> 172.21.15.44:0/3730923784 conn(0x7feedc0fdb30 msgr2=0x7feedc0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:53.414 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.412+0000 7feee17ff700 1 -- 172.21.15.44:0/3730923784 shutdown_connections 2023-11-16T16:47:53.414 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.412+0000 7feee17ff700 1 -- 172.21.15.44:0/3730923784 wait complete. 2023-11-16T16:47:53.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.413+0000 7feee17ff700 1 Processor -- start 2023-11-16T16:47:53.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.414+0000 7feee17ff700 1 -- start start 2023-11-16T16:47:53.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.414+0000 7feee17ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feedc1024d0 0x7feedc198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:53.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.414+0000 7feee17ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feedc103760 0x7feedc198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:53.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.414+0000 7feee17ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feedc104110 0x7feedc19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:53.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.414+0000 7feee17ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc19fcb0 con 0x7feedc103760 2023-11-16T16:47:53.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.414+0000 7feee17ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc19fdf0 con 0x7feedc1024d0 2023-11-16T16:47:53.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.414+0000 7feee17ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc19ff80 con 0x7feedc104110 2023-11-16T16:47:53.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.414+0000 7feedaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feedc1024d0 0x7feedc198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:53.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.414+0000 7feeda7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feedc103760 0x7feedc198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:53.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.414+0000 7feedaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feedc1024d0 0x7feedc198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59924/0 (socket says 172.21.15.44:59924) 2023-11-16T16:47:53.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.414+0000 7feedaffd700 1 -- 172.21.15.44:0/4020113591 learned_addr learned my addr 172.21.15.44:0/4020113591 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:53.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.414+0000 7feedb7fe700 1 --2- 172.21.15.44:0/4020113591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feedc104110 0x7feedc19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:53.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.415+0000 7feeda7fc700 1 -- 172.21.15.44:0/4020113591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feedc104110 msgr2=0x7feedc19f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.415+0000 7feeda7fc700 1 --2- 172.21.15.44:0/4020113591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feedc104110 0x7feedc19f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.415+0000 7feeda7fc700 1 -- 172.21.15.44:0/4020113591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feedc1024d0 msgr2=0x7feedc198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.415+0000 7feeda7fc700 1 --2- 172.21.15.44:0/4020113591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feedc1024d0 0x7feedc198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.415+0000 7feeda7fc700 1 -- 172.21.15.44:0/4020113591 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feecc009520 con 0x7feedc103760 2023-11-16T16:47:53.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.415+0000 7feedaffd700 1 --2- 172.21.15.44:0/4020113591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feedc1024d0 0x7feedc198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:53.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.415+0000 7feeda7fc700 1 --2- 172.21.15.44:0/4020113591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feedc103760 0x7feedc198670 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7feed0000c00 tx=0x7feed0005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:53.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.415+0000 7feec3fff700 1 -- 172.21.15.44:0/4020113591 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed001d070 con 0x7feedc103760 2023-11-16T16:47:53.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.415+0000 7feec3fff700 1 -- 172.21.15.44:0/4020113591 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feed0022470 con 0x7feedc103760 2023-11-16T16:47:53.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.415+0000 7feee17ff700 1 -- 172.21.15.44:0/4020113591 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed00095f0 con 0x7feedc103760 2023-11-16T16:47:53.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.415+0000 7feec3fff700 1 -- 172.21.15.44:0/4020113591 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0017650 con 0x7feedc103760 2023-11-16T16:47:53.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.415+0000 7feee17ff700 1 -- 172.21.15.44:0/4020113591 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feedc10ded0 con 0x7feedc103760 2023-11-16T16:47:53.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.416+0000 7feee17ff700 1 -- 172.21.15.44:0/4020113591 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feedc109450 con 0x7feedc103760 2023-11-16T16:47:53.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.418+0000 7feec3fff700 1 -- 172.21.15.44:0/4020113591 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7feed00225e0 con 0x7feedc103760 2023-11-16T16:47:53.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.420+0000 7feec3fff700 1 --2- 172.21.15.44:0/4020113591 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7feec40a11e0 0x7feec40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:53.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.420+0000 7feec3fff700 1 -- 172.21.15.44:0/4020113591 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7feed00ce150 con 0x7feedc103760 2023-11-16T16:47:53.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.420+0000 7feedaffd700 1 --2- 172.21.15.44:0/4020113591 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7feec40a11e0 0x7feec40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:53.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.420+0000 7feedaffd700 1 --2- 172.21.15.44:0/4020113591 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7feec40a11e0 0x7feec40a3690 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7feecc009fd0 tx=0x7feecc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:53.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.424+0000 7feec3fff700 1 -- 172.21.15.44:0/4020113591 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7feed0097460 con 0x7feedc103760 2023-11-16T16:47:53.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.637+0000 7feee17ff700 1 -- 172.21.15.44:0/4020113591 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feedc02cfe0 con 0x7feedc103760 2023-11-16T16:47:53.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.638+0000 7feec3fff700 1 -- 172.21.15.44:0/4020113591 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24451 (secure 0 0 0) 0x7feed009aa80 con 0x7feedc103760 2023-11-16T16:47:53.640 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:53.640 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":89,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:47:43.617828+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:53.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.641+0000 7feee17ff700 1 -- 172.21.15.44:0/4020113591 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7feec40a11e0 msgr2=0x7feec40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.641+0000 7feee17ff700 1 --2- 172.21.15.44:0/4020113591 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7feec40a11e0 0x7feec40a3690 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7feecc009fd0 tx=0x7feecc009260 comp rx=0 tx=0).stop 2023-11-16T16:47:53.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.641+0000 7feee17ff700 1 -- 172.21.15.44:0/4020113591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feedc103760 msgr2=0x7feedc198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.641+0000 7feee17ff700 1 --2- 172.21.15.44:0/4020113591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feedc103760 0x7feedc198670 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7feed0000c00 tx=0x7feed0005720 comp rx=0 tx=0).stop 2023-11-16T16:47:53.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.641+0000 7feee17ff700 1 -- 172.21.15.44:0/4020113591 shutdown_connections 2023-11-16T16:47:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.641+0000 7feee17ff700 1 --2- 172.21.15.44:0/4020113591 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7feec40a11e0 0x7feec40a3690 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.641+0000 7feee17ff700 1 --2- 172.21.15.44:0/4020113591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feedc104110 0x7feedc19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.641+0000 7feee17ff700 1 --2- 172.21.15.44:0/4020113591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feedc1024d0 0x7feedc198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.641+0000 7feee17ff700 1 --2- 172.21.15.44:0/4020113591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feedc103760 0x7feedc198670 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.641+0000 7feee17ff700 1 -- 172.21.15.44:0/4020113591 >> 172.21.15.44:0/4020113591 conn(0x7feedc0fdb30 msgr2=0x7feedc0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.641+0000 7feee17ff700 1 -- 172.21.15.44:0/4020113591 shutdown_connections 2023-11-16T16:47:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.641+0000 7feee17ff700 1 -- 172.21.15.44:0/4020113591 wait complete. 2023-11-16T16:47:53.662 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-16T16:47:53.663 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T16:47:53.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.867+0000 7fee259ff700 1 -- 172.21.15.44:0/3305969409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee20100290 msgr2=0x7fee201006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.867+0000 7fee259ff700 1 --2- 172.21.15.44:0/3305969409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee20100290 0x7fee201006a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fee10009990 tx=0x7fee10009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.867+0000 7fee259ff700 1 -- 172.21.15.44:0/3305969409 shutdown_connections 2023-11-16T16:47:53.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.867+0000 7fee259ff700 1 --2- 172.21.15.44:0/3305969409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee20101e10 0x7fee201086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.867+0000 7fee259ff700 1 --2- 172.21.15.44:0/3305969409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee20100290 0x7fee201006a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.867+0000 7fee259ff700 1 --2- 172.21.15.44:0/3305969409 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee20101480 0x7fee201018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.867+0000 7fee259ff700 1 -- 172.21.15.44:0/3305969409 >> 172.21.15.44:0/3305969409 conn(0x7fee200fb870 msgr2=0x7fee200fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:53.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.867+0000 7fee259ff700 1 -- 172.21.15.44:0/3305969409 shutdown_connections 2023-11-16T16:47:53.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.867+0000 7fee259ff700 1 -- 172.21.15.44:0/3305969409 wait complete. 2023-11-16T16:47:53.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.868+0000 7fee259ff700 1 Processor -- start 2023-11-16T16:47:53.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.869+0000 7fee259ff700 1 -- start start 2023-11-16T16:47:53.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.869+0000 7fee259ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee20100290 0x7fee201981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:53.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.869+0000 7fee259ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee20101480 0x7fee20198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:53.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.869+0000 7fee259ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee20101e10 0x7fee2019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:53.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.869+0000 7fee259ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2019fd50 con 0x7fee20101e10 2023-11-16T16:47:53.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.869+0000 7fee259ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2019fe90 con 0x7fee20101480 2023-11-16T16:47:53.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.869+0000 7fee259ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee201a0020 con 0x7fee20100290 2023-11-16T16:47:53.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.869+0000 7fee1f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee20101e10 0x7fee2019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:53.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.869+0000 7fee1effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee20100290 0x7fee201981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:53.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.869+0000 7fee1effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee20100290 0x7fee201981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:44782/0 (socket says 172.21.15.44:44782) 2023-11-16T16:47:53.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.869+0000 7fee1effd700 1 -- 172.21.15.44:0/831301051 learned_addr learned my addr 172.21.15.44:0/831301051 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:53.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.869+0000 7fee1e7fc700 1 --2- 172.21.15.44:0/831301051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee20101480 0x7fee20198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:53.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.870+0000 7fee1effd700 1 -- 172.21.15.44:0/831301051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee20101480 msgr2=0x7fee20198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.870+0000 7fee1effd700 1 --2- 172.21.15.44:0/831301051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee20101480 0x7fee20198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.870+0000 7fee1effd700 1 -- 172.21.15.44:0/831301051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee20101e10 msgr2=0x7fee2019f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.870+0000 7fee1effd700 1 --2- 172.21.15.44:0/831301051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee20101e10 0x7fee2019f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.870+0000 7fee1effd700 1 -- 172.21.15.44:0/831301051 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee100095f0 con 0x7fee20100290 2023-11-16T16:47:53.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.870+0000 7fee1e7fc700 1 --2- 172.21.15.44:0/831301051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee20101480 0x7fee20198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:53.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.870+0000 7fee1f7fe700 1 --2- 172.21.15.44:0/831301051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee20101e10 0x7fee2019f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:53.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.870+0000 7fee1effd700 1 --2- 172.21.15.44:0/831301051 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee20100290 0x7fee201981d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fee10005740 tx=0x7fee10005770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:53.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.870+0000 7fee249fd700 1 -- 172.21.15.44:0/831301051 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1001d070 con 0x7fee20100290 2023-11-16T16:47:53.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.870+0000 7fee259ff700 1 -- 172.21.15.44:0/831301051 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2010bc40 con 0x7fee20100290 2023-11-16T16:47:53.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.871+0000 7fee259ff700 1 -- 172.21.15.44:0/831301051 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2010bab0 con 0x7fee20100290 2023-11-16T16:47:53.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.871+0000 7fee249fd700 1 -- 172.21.15.44:0/831301051 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee10021410 con 0x7fee20100290 2023-11-16T16:47:53.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.871+0000 7fee249fd700 1 -- 172.21.15.44:0/831301051 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee10017650 con 0x7fee20100290 2023-11-16T16:47:53.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.872+0000 7fee259ff700 1 -- 172.21.15.44:0/831301051 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee201928d0 con 0x7fee20100290 2023-11-16T16:47:53.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.874+0000 7fee249fd700 1 -- 172.21.15.44:0/831301051 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fee1000fb40 con 0x7fee20100290 2023-11-16T16:47:53.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.875+0000 7fee249fd700 1 --2- 172.21.15.44:0/831301051 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fee080a11e0 0x7fee080a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:53.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.875+0000 7fee249fd700 1 -- 172.21.15.44:0/831301051 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fee100cdf30 con 0x7fee20100290 2023-11-16T16:47:53.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.875+0000 7fee1e7fc700 1 --2- 172.21.15.44:0/831301051 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fee080a11e0 0x7fee080a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:53.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.876+0000 7fee1e7fc700 1 --2- 172.21.15.44:0/831301051 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fee080a11e0 0x7fee080a3690 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fee14011490 tx=0x7fee14009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:53.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:53.877+0000 7fee249fd700 1 -- 172.21.15.44:0/831301051 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fee10097240 con 0x7fee20100290 2023-11-16T16:47:53.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:53 smithi077 ceph-mon[131204]: from='client.24820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:53.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:53 smithi077 ceph-mon[131204]: pgmap v294: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 103 B/s wr, 1 op/s; 2 B/s, 0 objects/s recovering 2023-11-16T16:47:53.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:53 smithi077 ceph-mon[131204]: mgrmap e17: x(active, since 7m), standbys: z, y 2023-11-16T16:47:53.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:53 smithi077 ceph-mon[131204]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:53.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:53 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/4028242290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:53.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:53 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/4020113591' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:54.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.083+0000 7fee259ff700 1 -- 172.21.15.44:0/831301051 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fee20061c30 con 0x7fee080a11e0 2023-11-16T16:47:54.100 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.099+0000 7fee249fd700 1 -- 172.21.15.44:0/831301051 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422568 (secure 0 0 0) 0x7fee20061c30 con 0x7fee080a11e0 2023-11-16T16:47:54.106 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:54.110 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.105+0000 7fee259ff700 1 -- 172.21.15.44:0/831301051 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fee080a11e0 msgr2=0x7fee080a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.105+0000 7fee259ff700 1 --2- 172.21.15.44:0/831301051 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fee080a11e0 0x7fee080a3690 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fee14011490 tx=0x7fee14009260 comp rx=0 tx=0).stop 2023-11-16T16:47:54.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.105+0000 7fee259ff700 1 -- 172.21.15.44:0/831301051 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee20100290 msgr2=0x7fee201981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.105+0000 7fee259ff700 1 --2- 172.21.15.44:0/831301051 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee20100290 0x7fee201981d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fee10005740 tx=0x7fee10005770 comp rx=0 tx=0).stop 2023-11-16T16:47:54.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.105+0000 7fee259ff700 1 -- 172.21.15.44:0/831301051 shutdown_connections 2023-11-16T16:47:54.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.105+0000 7fee259ff700 1 --2- 172.21.15.44:0/831301051 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fee080a11e0 0x7fee080a3690 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.105+0000 7fee259ff700 1 --2- 172.21.15.44:0/831301051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee20101e10 0x7fee2019f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.105+0000 7fee259ff700 1 --2- 172.21.15.44:0/831301051 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee20100290 0x7fee201981d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.105+0000 7fee259ff700 1 --2- 172.21.15.44:0/831301051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee20101480 0x7fee20198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.105+0000 7fee259ff700 1 -- 172.21.15.44:0/831301051 >> 172.21.15.44:0/831301051 conn(0x7fee200fb870 msgr2=0x7fee20106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:54.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.105+0000 7fee259ff700 1 -- 172.21.15.44:0/831301051 shutdown_connections 2023-11-16T16:47:54.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.105+0000 7fee259ff700 1 -- 172.21.15.44:0/831301051 wait complete. 2023-11-16T16:47:54.112 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2023-11-16T16:47:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:53 smithi044 ceph-mon[128750]: from='client.24820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:53 smithi044 ceph-mon[128750]: pgmap v294: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 103 B/s wr, 1 op/s; 2 B/s, 0 objects/s recovering 2023-11-16T16:47:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:53 smithi044 ceph-mon[128750]: mgrmap e17: x(active, since 7m), standbys: z, y 2023-11-16T16:47:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:53 smithi044 ceph-mon[128750]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:53 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/4028242290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:54.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:53 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/4020113591' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:54.125 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2023-11-16T16:47:52.539427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594960,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":145,"num_read_kb":130,"num_write":230,"num_write_kb":2957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180,"ondisk_log_size":180,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":79908,"kb_used_data":11524,"kb_used_omap":0,"kb_used_meta":68288,"kb_avail":1124861916,"statfs":{"total":1151940427776,"available":1151858601984,"internally_reserved":0,"allocated":11800576,"data_stored":7590479,"data_compressed":318969,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":0,"internal_metadata":69926912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.920735"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625866+0000","last_change":"2023-11-16T16:47:18.183024+0000","last_active":"2023-11-16T16:47:43.625866+0000","last_peered":"2023-11-16T16:47:43.625866+0000","last_clean":"2023-11-16T16:47:43.625866+0000","last_became_active":"2023-11-16T16:47:18.182387+0000","last_became_peered":"2023-11-16T16:47:18.182387+0000","last_unstale":"2023-11-16T16:47:43.625866+0000","last_undegraded":"2023-11-16T16:47:43.625866+0000","last_fullsized":"2023-11-16T16:47:43.625866+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:33:54.818555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626848+0000","last_change":"2023-11-16T16:47:18.200135+0000","last_active":"2023-11-16T16:47:43.626848+0000","last_peered":"2023-11-16T16:47:43.626848+0000","last_clean":"2023-11-16T16:47:43.626848+0000","last_became_active":"2023-11-16T16:47:18.194732+0000","last_became_peered":"2023-11-16T16:47:18.194732+0000","last_unstale":"2023-11-16T16:47:43.626848+0000","last_undegraded":"2023-11-16T16:47:43.626848+0000","last_fullsized":"2023-11-16T16:47:43.626848+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:35:44.790080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626424+0000","last_change":"2023-11-16T16:47:18.184127+0000","last_active":"2023-11-16T16:47:43.626424+0000","last_peered":"2023-11-16T16:47:43.626424+0000","last_clean":"2023-11-16T16:47:43.626424+0000","last_became_active":"2023-11-16T16:47:18.183163+0000","last_became_peered":"2023-11-16T16:47:18.183163+0000","last_unstale":"2023-11-16T16:47:43.626424+0000","last_undegraded":"2023-11-16T16:47:43.626424+0000","last_fullsized":"2023-11-16T16:47:43.626424+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:22:40.700926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629755+0000","last_change":"2023-11-16T16:47:18.177476+0000","last_active":"2023-11-16T16:47:43.629755+0000","last_peered":"2023-11-16T16:47:43.629755+0000","last_clean":"2023-11-16T16:47:43.629755+0000","last_became_active":"2023-11-16T16:47:18.177178+0000","last_became_peered":"2023-11-16T16:47:18.177178+0000","last_unstale":"2023-11-16T16:47:43.629755+0000","last_undegraded":"2023-11-16T16:47:43.629755+0000","last_fullsized":"2023-11-16T16:47:43.629755+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:40:05.452664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626685+0000","last_change":"2023-11-16T16:47:18.178511+0000","last_active":"2023-11-16T16:47:43.626685+0000","last_peered":"2023-11-16T16:47:43.626685+0000","last_clean":"2023-11-16T16:47:43.626685+0000","last_became_active":"2023-11-16T16:47:18.178198+0000","last_became_peered":"2023-11-16T16:47:18.178198+0000","last_unstale":"2023-11-16T16:47:43.626685+0000","last_undegraded":"2023-11-16T16:47:43.626685+0000","last_fullsized":"2023-11-16T16:47:43.626685+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:53:35.196107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626326+0000","last_change":"2023-11-16T16:47:18.186552+0000","last_active":"2023-11-16T16:47:43.626326+0000","last_peered":"2023-11-16T16:47:43.626326+0000","last_clean":"2023-11-16T16:47:43.626326+0000","last_became_active":"2023-11-16T16:47:18.186238+0000","last_became_peered":"2023-11-16T16:47:18.186238+0000","last_unstale":"2023-11-16T16:47:43.626326+0000","last_undegraded":"2023-11-16T16:47:43.626326+0000","last_fullsized":"2023-11-16T16:47:43.626326+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:36:52.844919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622387+0000","last_change":"2023-11-16T16:47:18.179839+0000","last_active":"2023-11-16T16:47:43.622387+0000","last_peered":"2023-11-16T16:47:43.622387+0000","last_clean":"2023-11-16T16:47:43.622387+0000","last_became_active":"2023-11-16T16:47:18.179060+0000","last_became_peered":"2023-11-16T16:47:18.179060+0000","last_unstale":"2023-11-16T16:47:43.622387+0000","last_undegraded":"2023-11-16T16:47:43.622387+0000","last_fullsized":"2023-11-16T16:47:43.622387+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:55:30.972787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.633015+0000","last_change":"2023-11-16T16:47:43.631738+0000","last_active":"2023-11-16T16:47:43.633015+0000","last_peered":"2023-11-16T16:47:43.633015+0000","last_clean":"2023-11-16T16:47:43.633015+0000","last_became_active":"2023-11-16T16:47:43.631146+0000","last_became_peered":"2023-11-16T16:47:43.631146+0000","last_unstale":"2023-11-16T16:47:43.633015+0000","last_undegraded":"2023-11-16T16:47:43.633015+0000","last_fullsized":"2023-11-16T16:47:43.633015+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:32:26.552018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947385+0000","last_change":"2023-11-16T16:47:18.177834+0000","last_active":"2023-11-16T16:47:43.947385+0000","last_peered":"2023-11-16T16:47:43.947385+0000","last_clean":"2023-11-16T16:47:43.947385+0000","last_became_active":"2023-11-16T16:47:18.176299+0000","last_became_peered":"2023-11-16T16:47:18.176299+0000","last_unstale":"2023-11-16T16:47:43.947385+0000","last_undegraded":"2023-11-16T16:47:43.947385+0000","last_fullsized":"2023-11-16T16:47:43.947385+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:18:34.674950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626777+0000","last_change":"2023-11-16T16:47:18.179394+0000","last_active":"2023-11-16T16:47:43.626777+0000","last_peered":"2023-11-16T16:47:43.626777+0000","last_clean":"2023-11-16T16:47:43.626777+0000","last_became_active":"2023-11-16T16:47:18.179119+0000","last_became_peered":"2023-11-16T16:47:18.179119+0000","last_unstale":"2023-11-16T16:47:43.626777+0000","last_undegraded":"2023-11-16T16:47:43.626777+0000","last_fullsized":"2023-11-16T16:47:43.626777+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:32:22.770618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623315+0000","last_change":"2023-11-16T16:47:18.183585+0000","last_active":"2023-11-16T16:47:43.623315+0000","last_peered":"2023-11-16T16:47:43.623315+0000","last_clean":"2023-11-16T16:47:43.623315+0000","last_became_active":"2023-11-16T16:47:18.183209+0000","last_became_peered":"2023-11-16T16:47:18.183209+0000","last_unstale":"2023-11-16T16:47:43.623315+0000","last_undegraded":"2023-11-16T16:47:43.623315+0000","last_fullsized":"2023-11-16T16:47:43.623315+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:22:47.894810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629596+0000","last_change":"2023-11-16T16:47:18.186503+0000","last_active":"2023-11-16T16:47:43.629596+0000","last_peered":"2023-11-16T16:47:43.629596+0000","last_clean":"2023-11-16T16:47:43.629596+0000","last_became_active":"2023-11-16T16:47:18.186193+0000","last_became_peered":"2023-11-16T16:47:18.186193+0000","last_unstale":"2023-11-16T16:47:43.629596+0000","last_undegraded":"2023-11-16T16:47:43.629596+0000","last_fullsized":"2023-11-16T16:47:43.629596+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:59:03.486858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629884+0000","last_change":"2023-11-16T16:47:18.193842+0000","last_active":"2023-11-16T16:47:43.629884+0000","last_peered":"2023-11-16T16:47:43.629884+0000","last_clean":"2023-11-16T16:47:43.629884+0000","last_became_active":"2023-11-16T16:47:18.192309+0000","last_became_peered":"2023-11-16T16:47:18.192309+0000","last_unstale":"2023-11-16T16:47:43.629884+0000","last_undegraded":"2023-11-16T16:47:43.629884+0000","last_fullsized":"2023-11-16T16:47:43.629884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:53:33.884565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625812+0000","last_change":"2023-11-16T16:47:18.192359+0000","last_active":"2023-11-16T16:47:43.625812+0000","last_peered":"2023-11-16T16:47:43.625812+0000","last_clean":"2023-11-16T16:47:43.625812+0000","last_became_active":"2023-11-16T16:47:18.189765+0000","last_became_peered":"2023-11-16T16:47:18.189765+0000","last_unstale":"2023-11-16T16:47:43.625812+0000","last_undegraded":"2023-11-16T16:47:43.625812+0000","last_fullsized":"2023-11-16T16:47:43.625812+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:17:15.440725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625875+0000","last_change":"2023-11-16T16:47:18.178386+0000","last_active":"2023-11-16T16:47:43.625875+0000","last_peered":"2023-11-16T16:47:43.625875+0000","last_clean":"2023-11-16T16:47:43.625875+0000","last_became_active":"2023-11-16T16:47:18.178041+0000","last_became_peered":"2023-11-16T16:47:18.178041+0000","last_unstale":"2023-11-16T16:47:43.625875+0000","last_undegraded":"2023-11-16T16:47:43.625875+0000","last_fullsized":"2023-11-16T16:47:43.625875+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:27:47.278353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.946667+0000","last_change":"2023-11-16T16:47:18.174424+0000","last_active":"2023-11-16T16:47:43.946667+0000","last_peered":"2023-11-16T16:47:43.946667+0000","last_clean":"2023-11-16T16:47:43.946667+0000","last_became_active":"2023-11-16T16:47:18.174078+0000","last_became_peered":"2023-11-16T16:47:18.174078+0000","last_unstale":"2023-11-16T16:47:43.946667+0000","last_undegraded":"2023-11-16T16:47:43.946667+0000","last_fullsized":"2023-11-16T16:47:43.946667+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:14:13.491783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625739+0000","last_change":"2023-11-16T16:47:18.183161+0000","last_active":"2023-11-16T16:47:43.625739+0000","last_peered":"2023-11-16T16:47:43.625739+0000","last_clean":"2023-11-16T16:47:43.625739+0000","last_became_active":"2023-11-16T16:47:18.182806+0000","last_became_peered":"2023-11-16T16:47:18.182806+0000","last_unstale":"2023-11-16T16:47:43.625739+0000","last_undegraded":"2023-11-16T16:47:43.625739+0000","last_fullsized":"2023-11-16T16:47:43.625739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:50:04.072542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622713+0000","last_change":"2023-11-16T16:47:18.187225+0000","last_active":"2023-11-16T16:47:43.622713+0000","last_peered":"2023-11-16T16:47:43.622713+0000","last_clean":"2023-11-16T16:47:43.622713+0000","last_became_active":"2023-11-16T16:47:18.186858+0000","last_became_peered":"2023-11-16T16:47:18.186858+0000","last_unstale":"2023-11-16T16:47:43.622713+0000","last_undegraded":"2023-11-16T16:47:43.622713+0000","last_fullsized":"2023-11-16T16:47:43.622713+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:42:28.736883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623323+0000","last_change":"2023-11-16T16:47:18.190947+0000","last_active":"2023-11-16T16:47:43.623323+0000","last_peered":"2023-11-16T16:47:43.623323+0000","last_clean":"2023-11-16T16:47:43.623323+0000","last_became_active":"2023-11-16T16:47:18.190737+0000","last_became_peered":"2023-11-16T16:47:18.190737+0000","last_unstale":"2023-11-16T16:47:43.623323+0000","last_undegraded":"2023-11-16T16:47:43.623323+0000","last_fullsized":"2023-11-16T16:47:43.623323+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:34:59.150388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622328+0000","last_change":"2023-11-16T16:47:22.659730+0000","last_active":"2023-11-16T16:47:43.622328+0000","last_peered":"2023-11-16T16:47:43.622328+0000","last_clean":"2023-11-16T16:47:43.622328+0000","last_became_active":"2023-11-16T16:47:22.659313+0000","last_became_peered":"2023-11-16T16:47:22.659313+0000","last_unstale":"2023-11-16T16:47:43.622328+0000","last_undegraded":"2023-11-16T16:47:43.622328+0000","last_fullsized":"2023-11-16T16:47:43.622328+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:05:05.336507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'4","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622965+0000","last_change":"2023-11-16T16:47:17.629748+0000","last_active":"2023-11-16T16:47:43.622965+0000","last_peered":"2023-11-16T16:47:43.622965+0000","last_clean":"2023-11-16T16:47:43.622965+0000","last_became_active":"2023-11-16T16:47:17.629284+0000","last_became_peered":"2023-11-16T16:47:17.629284+0000","last_unstale":"2023-11-16T16:47:43.622965+0000","last_undegraded":"2023-11-16T16:47:43.622965+0000","last_fullsized":"2023-11-16T16:47:43.622965+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:41:29.574724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970501+0000","last_change":"2023-11-16T16:47:18.176239+0000","last_active":"2023-11-16T16:47:43.970501+0000","last_peered":"2023-11-16T16:47:43.970501+0000","last_clean":"2023-11-16T16:47:43.970501+0000","last_became_active":"2023-11-16T16:47:18.175879+0000","last_became_peered":"2023-11-16T16:47:18.175879+0000","last_unstale":"2023-11-16T16:47:43.970501+0000","last_undegraded":"2023-11-16T16:47:43.970501+0000","last_fullsized":"2023-11-16T16:47:43.970501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:32:09.544453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.628988+0000","last_change":"2023-11-16T16:47:22.659679+0000","last_active":"2023-11-16T16:47:43.628988+0000","last_peered":"2023-11-16T16:47:43.628988+0000","last_clean":"2023-11-16T16:47:43.628988+0000","last_became_active":"2023-11-16T16:47:22.658946+0000","last_became_peered":"2023-11-16T16:47:22.658946+0000","last_unstale":"2023-11-16T16:47:43.628988+0000","last_undegraded":"2023-11-16T16:47:43.628988+0000","last_fullsized":"2023-11-16T16:47:43.628988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:33:44.392111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"82'1","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.971259+0000","last_change":"2023-11-16T16:47:17.643944+0000","last_active":"2023-11-16T16:47:43.971259+0000","last_peered":"2023-11-16T16:47:43.971259+0000","last_clean":"2023-11-16T16:47:43.971259+0000","last_became_active":"2023-11-16T16:47:17.643470+0000","last_became_peered":"2023-11-16T16:47:17.643470+0000","last_unstale":"2023-11-16T16:47:43.971259+0000","last_undegraded":"2023-11-16T16:47:43.971259+0000","last_fullsized":"2023-11-16T16:47:43.971259+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:28:22.870692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.627110+0000","last_change":"2023-11-16T16:47:18.198139+0000","last_active":"2023-11-16T16:47:43.627110+0000","last_peered":"2023-11-16T16:47:43.627110+0000","last_clean":"2023-11-16T16:47:43.627110+0000","last_became_active":"2023-11-16T16:47:18.197762+0000","last_became_peered":"2023-11-16T16:47:18.197762+0000","last_unstale":"2023-11-16T16:47:43.627110+0000","last_undegraded":"2023-11-16T16:47:43.627110+0000","last_fullsized":"2023-11-16T16:47:43.627110+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:27:32.243043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629208+0000","last_change":"2023-11-16T16:47:22.655810+0000","last_active":"2023-11-16T16:47:43.629208+0000","last_peered":"2023-11-16T16:47:43.629208+0000","last_clean":"2023-11-16T16:47:43.629208+0000","last_became_active":"2023-11-16T16:47:22.655388+0000","last_became_peered":"2023-11-16T16:47:22.655388+0000","last_unstale":"2023-11-16T16:47:43.629208+0000","last_undegraded":"2023-11-16T16:47:43.629208+0000","last_fullsized":"2023-11-16T16:47:43.629208+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:05:46.905125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'2","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.635266+0000","last_change":"2023-11-16T16:47:43.635266+0000","last_active":"2023-11-16T16:47:43.635266+0000","last_peered":"2023-11-16T16:47:43.635266+0000","last_clean":"2023-11-16T16:47:43.635266+0000","last_became_active":"2023-11-16T16:47:43.630583+0000","last_became_peered":"2023-11-16T16:47:43.630583+0000","last_unstale":"2023-11-16T16:47:43.635266+0000","last_undegraded":"2023-11-16T16:47:43.635266+0000","last_fullsized":"2023-11-16T16:47:43.635266+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:51:43.792892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.946938+0000","last_change":"2023-11-16T16:47:18.186341+0000","last_active":"2023-11-16T16:47:43.946938+0000","last_peered":"2023-11-16T16:47:43.946938+0000","last_clean":"2023-11-16T16:47:43.946938+0000","last_became_active":"2023-11-16T16:47:18.186066+0000","last_became_peered":"2023-11-16T16:47:18.186066+0000","last_unstale":"2023-11-16T16:47:43.946938+0000","last_undegraded":"2023-11-16T16:47:43.946938+0000","last_fullsized":"2023-11-16T16:47:43.946938+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:22:46.055671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629005+0000","last_change":"2023-11-16T16:47:22.657227+0000","last_active":"2023-11-16T16:47:43.629005+0000","last_peered":"2023-11-16T16:47:43.629005+0000","last_clean":"2023-11-16T16:47:43.629005+0000","last_became_active":"2023-11-16T16:47:22.656835+0000","last_became_peered":"2023-11-16T16:47:22.656835+0000","last_unstale":"2023-11-16T16:47:43.629005+0000","last_undegraded":"2023-11-16T16:47:43.629005+0000","last_fullsized":"2023-11-16T16:47:43.629005+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:07.552894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625950+0000","last_change":"2023-11-16T16:47:17.630194+0000","last_active":"2023-11-16T16:47:43.625950+0000","last_peered":"2023-11-16T16:47:43.625950+0000","last_clean":"2023-11-16T16:47:43.625950+0000","last_became_active":"2023-11-16T16:47:17.629886+0000","last_became_peered":"2023-11-16T16:47:17.629886+0000","last_unstale":"2023-11-16T16:47:43.625950+0000","last_undegraded":"2023-11-16T16:47:43.625950+0000","last_fullsized":"2023-11-16T16:47:43.625950+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:57:09.970289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630557+0000","last_change":"2023-11-16T16:47:18.181596+0000","last_active":"2023-11-16T16:47:43.630557+0000","last_peered":"2023-11-16T16:47:43.630557+0000","last_clean":"2023-11-16T16:47:43.630557+0000","last_became_active":"2023-11-16T16:47:18.181287+0000","last_became_peered":"2023-11-16T16:47:18.181287+0000","last_unstale":"2023-11-16T16:47:43.630557+0000","last_undegraded":"2023-11-16T16:47:43.630557+0000","last_fullsized":"2023-11-16T16:47:43.630557+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:31:55.798497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622763+0000","last_change":"2023-11-16T16:47:22.649197+0000","last_active":"2023-11-16T16:47:43.622763+0000","last_peered":"2023-11-16T16:47:43.622763+0000","last_clean":"2023-11-16T16:47:43.622763+0000","last_became_active":"2023-11-16T16:47:22.648664+0000","last_became_peered":"2023-11-16T16:47:22.648664+0000","last_unstale":"2023-11-16T16:47:43.622763+0000","last_undegraded":"2023-11-16T16:47:43.622763+0000","last_fullsized":"2023-11-16T16:47:43.622763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:56:11.119940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"82'1","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630141+0000","last_change":"2023-11-16T16:47:17.627672+0000","last_active":"2023-11-16T16:47:43.630141+0000","last_peered":"2023-11-16T16:47:43.630141+0000","last_clean":"2023-11-16T16:47:43.630141+0000","last_became_active":"2023-11-16T16:47:17.627383+0000","last_became_peered":"2023-11-16T16:47:17.627383+0000","last_unstale":"2023-11-16T16:47:43.630141+0000","last_undegraded":"2023-11-16T16:47:43.630141+0000","last_fullsized":"2023-11-16T16:47:43.630141+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:35:57.379840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625043+0000","last_change":"2023-11-16T16:47:18.180983+0000","last_active":"2023-11-16T16:47:43.625043+0000","last_peered":"2023-11-16T16:47:43.625043+0000","last_clean":"2023-11-16T16:47:43.625043+0000","last_became_active":"2023-11-16T16:47:18.180702+0000","last_became_peered":"2023-11-16T16:47:18.180702+0000","last_unstale":"2023-11-16T16:47:43.625043+0000","last_undegraded":"2023-11-16T16:47:43.625043+0000","last_fullsized":"2023-11-16T16:47:43.625043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:20:27.895648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622657+0000","last_change":"2023-11-16T16:47:22.659898+0000","last_active":"2023-11-16T16:47:43.622657+0000","last_peered":"2023-11-16T16:47:43.622657+0000","last_clean":"2023-11-16T16:47:43.622657+0000","last_became_active":"2023-11-16T16:47:22.659326+0000","last_became_peered":"2023-11-16T16:47:22.659326+0000","last_unstale":"2023-11-16T16:47:43.622657+0000","last_undegraded":"2023-11-16T16:47:43.622657+0000","last_fullsized":"2023-11-16T16:47:43.622657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:57:36.289645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"82'1","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.949231+0000","last_change":"2023-11-16T16:47:43.949231+0000","last_active":"2023-11-16T16:47:43.949231+0000","last_peered":"2023-11-16T16:47:43.949231+0000","last_clean":"2023-11-16T16:47:43.949231+0000","last_became_active":"2023-11-16T16:47:43.948766+0000","last_became_peered":"2023-11-16T16:47:43.948766+0000","last_unstale":"2023-11-16T16:47:43.949231+0000","last_undegraded":"2023-11-16T16:47:43.949231+0000","last_fullsized":"2023-11-16T16:47:43.949231+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:46:57.086811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630999+0000","last_change":"2023-11-16T16:47:18.185263+0000","last_active":"2023-11-16T16:47:43.630999+0000","last_peered":"2023-11-16T16:47:43.630999+0000","last_clean":"2023-11-16T16:47:43.630999+0000","last_became_active":"2023-11-16T16:47:18.184910+0000","last_became_peered":"2023-11-16T16:47:18.184910+0000","last_unstale":"2023-11-16T16:47:43.630999+0000","last_undegraded":"2023-11-16T16:47:43.630999+0000","last_fullsized":"2023-11-16T16:47:43.630999+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:25:21.463648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625581+0000","last_change":"2023-11-16T16:47:22.659711+0000","last_active":"2023-11-16T16:47:43.625581+0000","last_peered":"2023-11-16T16:47:43.625581+0000","last_clean":"2023-11-16T16:47:43.625581+0000","last_became_active":"2023-11-16T16:47:22.659087+0000","last_became_peered":"2023-11-16T16:47:22.659087+0000","last_unstale":"2023-11-16T16:47:43.625581+0000","last_undegraded":"2023-11-16T16:47:43.625581+0000","last_fullsized":"2023-11-16T16:47:43.625581+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:13:26.339037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629045+0000","last_change":"2023-11-16T16:47:17.629703+0000","last_active":"2023-11-16T16:47:43.629045+0000","last_peered":"2023-11-16T16:47:43.629045+0000","last_clean":"2023-11-16T16:47:43.629045+0000","last_became_active":"2023-11-16T16:47:17.629245+0000","last_became_peered":"2023-11-16T16:47:17.629245+0000","last_unstale":"2023-11-16T16:47:43.629045+0000","last_undegraded":"2023-11-16T16:47:43.629045+0000","last_fullsized":"2023-11-16T16:47:43.629045+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:59:34.790629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970314+0000","last_change":"2023-11-16T16:47:18.187149+0000","last_active":"2023-11-16T16:47:43.970314+0000","last_peered":"2023-11-16T16:47:43.970314+0000","last_clean":"2023-11-16T16:47:43.970314+0000","last_became_active":"2023-11-16T16:47:18.185717+0000","last_became_peered":"2023-11-16T16:47:18.185717+0000","last_unstale":"2023-11-16T16:47:43.970314+0000","last_undegraded":"2023-11-16T16:47:43.970314+0000","last_fullsized":"2023-11-16T16:47:43.970314+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:23:04.064643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626668+0000","last_change":"2023-11-16T16:47:22.663025+0000","last_active":"2023-11-16T16:47:43.626668+0000","last_peered":"2023-11-16T16:47:43.626668+0000","last_clean":"2023-11-16T16:47:43.626668+0000","last_became_active":"2023-11-16T16:47:22.662596+0000","last_became_peered":"2023-11-16T16:47:22.662596+0000","last_unstale":"2023-11-16T16:47:43.626668+0000","last_undegraded":"2023-11-16T16:47:43.626668+0000","last_fullsized":"2023-11-16T16:47:43.626668+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:47:26.815530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629668+0000","last_change":"2023-11-16T16:47:17.626357+0000","last_active":"2023-11-16T16:47:43.629668+0000","last_peered":"2023-11-16T16:47:43.629668+0000","last_clean":"2023-11-16T16:47:43.629668+0000","last_became_active":"2023-11-16T16:47:17.626029+0000","last_became_peered":"2023-11-16T16:47:17.626029+0000","last_unstale":"2023-11-16T16:47:43.629668+0000","last_undegraded":"2023-11-16T16:47:43.629668+0000","last_fullsized":"2023-11-16T16:47:43.629668+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:55:28.659340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.621841+0000","last_change":"2023-11-16T16:47:18.177153+0000","last_active":"2023-11-16T16:47:43.621841+0000","last_peered":"2023-11-16T16:47:43.621841+0000","last_clean":"2023-11-16T16:47:43.621841+0000","last_became_active":"2023-11-16T16:47:18.176751+0000","last_became_peered":"2023-11-16T16:47:18.176751+0000","last_unstale":"2023-11-16T16:47:43.621841+0000","last_undegraded":"2023-11-16T16:47:43.621841+0000","last_fullsized":"2023-11-16T16:47:43.621841+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:17:58.058194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970584+0000","last_change":"2023-11-16T16:47:22.652718+0000","last_active":"2023-11-16T16:47:43.970584+0000","last_peered":"2023-11-16T16:47:43.970584+0000","last_clean":"2023-11-16T16:47:43.970584+0000","last_became_active":"2023-11-16T16:47:22.652278+0000","last_became_peered":"2023-11-16T16:47:22.652278+0000","last_unstale":"2023-11-16T16:47:43.970584+0000","last_undegraded":"2023-11-16T16:47:43.970584+0000","last_fullsized":"2023-11-16T16:47:43.970584+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:43:41.546583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626473+0000","last_change":"2023-11-16T16:47:17.627308+0000","last_active":"2023-11-16T16:47:43.626473+0000","last_peered":"2023-11-16T16:47:43.626473+0000","last_clean":"2023-11-16T16:47:43.626473+0000","last_became_active":"2023-11-16T16:47:17.626992+0000","last_became_peered":"2023-11-16T16:47:17.626992+0000","last_unstale":"2023-11-16T16:47:43.626473+0000","last_undegraded":"2023-11-16T16:47:43.626473+0000","last_fullsized":"2023-11-16T16:47:43.626473+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:41:51.914757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947323+0000","last_change":"2023-11-16T16:47:18.188015+0000","last_active":"2023-11-16T16:47:43.947323+0000","last_peered":"2023-11-16T16:47:43.947323+0000","last_clean":"2023-11-16T16:47:43.947323+0000","last_became_active":"2023-11-16T16:47:18.187730+0000","last_became_peered":"2023-11-16T16:47:18.187730+0000","last_unstale":"2023-11-16T16:47:43.947323+0000","last_undegraded":"2023-11-16T16:47:43.947323+0000","last_fullsized":"2023-11-16T16:47:43.947323+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:39:02.651682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.621932+0000","last_change":"2023-11-16T16:47:22.655656+0000","last_active":"2023-11-16T16:47:43.621932+0000","last_peered":"2023-11-16T16:47:43.621932+0000","last_clean":"2023-11-16T16:47:43.621932+0000","last_became_active":"2023-11-16T16:47:22.655133+0000","last_became_peered":"2023-11-16T16:47:22.655133+0000","last_unstale":"2023-11-16T16:47:43.621932+0000","last_undegraded":"2023-11-16T16:47:43.621932+0000","last_fullsized":"2023-11-16T16:47:43.621932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:19:21.932819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947947+0000","last_change":"2023-11-16T16:47:43.947947+0000","last_active":"2023-11-16T16:47:43.947947+0000","last_peered":"2023-11-16T16:47:43.947947+0000","last_clean":"2023-11-16T16:47:43.947947+0000","last_became_active":"2023-11-16T16:47:43.947478+0000","last_became_peered":"2023-11-16T16:47:43.947478+0000","last_unstale":"2023-11-16T16:47:43.947947+0000","last_undegraded":"2023-11-16T16:47:43.947947+0000","last_fullsized":"2023-11-16T16:47:43.947947+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:40:14.254873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.632657+0000","last_change":"2023-11-16T16:47:43.632657+0000","last_active":"2023-11-16T16:47:43.632657+0000","last_peered":"2023-11-16T16:47:43.632657+0000","last_clean":"2023-11-16T16:47:43.632657+0000","last_became_active":"2023-11-16T16:47:43.631573+0000","last_became_peered":"2023-11-16T16:47:43.631573+0000","last_unstale":"2023-11-16T16:47:43.632657+0000","last_undegraded":"2023-11-16T16:47:43.632657+0000","last_fullsized":"2023-11-16T16:47:43.632657+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:45:08.317787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629418+0000","last_change":"2023-11-16T16:47:22.661639+0000","last_active":"2023-11-16T16:47:43.629418+0000","last_peered":"2023-11-16T16:47:43.629418+0000","last_clean":"2023-11-16T16:47:43.629418+0000","last_became_active":"2023-11-16T16:47:22.661230+0000","last_became_peered":"2023-11-16T16:47:22.661230+0000","last_unstale":"2023-11-16T16:47:43.629418+0000","last_undegraded":"2023-11-16T16:47:43.629418+0000","last_fullsized":"2023-11-16T16:47:43.629418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:04:16.909613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"82'3","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630143+0000","last_change":"2023-11-16T16:47:17.638209+0000","last_active":"2023-11-16T16:47:43.630143+0000","last_peered":"2023-11-16T16:47:43.630143+0000","last_clean":"2023-11-16T16:47:43.630143+0000","last_became_active":"2023-11-16T16:47:17.637957+0000","last_became_peered":"2023-11-16T16:47:17.637957+0000","last_unstale":"2023-11-16T16:47:43.630143+0000","last_undegraded":"2023-11-16T16:47:43.630143+0000","last_fullsized":"2023-11-16T16:47:43.630143+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:08:43.599916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947142+0000","last_change":"2023-11-16T16:47:18.182693+0000","last_active":"2023-11-16T16:47:43.947142+0000","last_peered":"2023-11-16T16:47:43.947142+0000","last_clean":"2023-11-16T16:47:43.947142+0000","last_became_active":"2023-11-16T16:47:18.182370+0000","last_became_peered":"2023-11-16T16:47:18.182370+0000","last_unstale":"2023-11-16T16:47:43.947142+0000","last_undegraded":"2023-11-16T16:47:43.947142+0000","last_fullsized":"2023-11-16T16:47:43.947142+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:38:49.650460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.621838+0000","last_change":"2023-11-16T16:47:22.649348+0000","last_active":"2023-11-16T16:47:43.621838+0000","last_peered":"2023-11-16T16:47:43.621838+0000","last_clean":"2023-11-16T16:47:43.621838+0000","last_became_active":"2023-11-16T16:47:22.648860+0000","last_became_peered":"2023-11-16T16:47:22.648860+0000","last_unstale":"2023-11-16T16:47:43.621838+0000","last_undegraded":"2023-11-16T16:47:43.621838+0000","last_fullsized":"2023-11-16T16:47:43.621838+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:45:18.960009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'10","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:49.162885+0000","last_change":"2023-11-16T16:47:17.619417+0000","last_active":"2023-11-16T16:47:49.162885+0000","last_peered":"2023-11-16T16:47:49.162885+0000","last_clean":"2023-11-16T16:47:49.162885+0000","last_became_active":"2023-11-16T16:47:17.619086+0000","last_became_peered":"2023-11-16T16:47:17.619086+0000","last_unstale":"2023-11-16T16:47:49.162885+0000","last_undegraded":"2023-11-16T16:47:49.162885+0000","last_fullsized":"2023-11-16T16:47:49.162885+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:27:30.044686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625947+0000","last_change":"2023-11-16T16:47:18.177537+0000","last_active":"2023-11-16T16:47:43.625947+0000","last_peered":"2023-11-16T16:47:43.625947+0000","last_clean":"2023-11-16T16:47:43.625947+0000","last_became_active":"2023-11-16T16:47:18.177112+0000","last_became_peered":"2023-11-16T16:47:18.177112+0000","last_unstale":"2023-11-16T16:47:43.625947+0000","last_undegraded":"2023-11-16T16:47:43.625947+0000","last_fullsized":"2023-11-16T16:47:43.625947+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:17:18.541523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630846+0000","last_change":"2023-11-16T16:47:22.658086+0000","last_active":"2023-11-16T16:47:43.630846+0000","last_peered":"2023-11-16T16:47:43.630846+0000","last_clean":"2023-11-16T16:47:43.630846+0000","last_became_active":"2023-11-16T16:47:22.657725+0000","last_became_peered":"2023-11-16T16:47:22.657725+0000","last_unstale":"2023-11-16T16:47:43.630846+0000","last_undegraded":"2023-11-16T16:47:43.630846+0000","last_fullsized":"2023-11-16T16:47:43.630846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:53:51.729635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"82'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622784+0000","last_change":"2023-11-16T16:47:17.625405+0000","last_active":"2023-11-16T16:47:43.622784+0000","last_peered":"2023-11-16T16:47:43.622784+0000","last_clean":"2023-11-16T16:47:43.622784+0000","last_became_active":"2023-11-16T16:47:17.625026+0000","last_became_peered":"2023-11-16T16:47:17.625026+0000","last_unstale":"2023-11-16T16:47:43.622784+0000","last_undegraded":"2023-11-16T16:47:43.622784+0000","last_fullsized":"2023-11-16T16:47:43.622784+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:45:14.505117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629372+0000","last_change":"2023-11-16T16:47:18.196345+0000","last_active":"2023-11-16T16:47:43.629372+0000","last_peered":"2023-11-16T16:47:43.629372+0000","last_clean":"2023-11-16T16:47:43.629372+0000","last_became_active":"2023-11-16T16:47:18.196040+0000","last_became_peered":"2023-11-16T16:47:18.196040+0000","last_unstale":"2023-11-16T16:47:43.629372+0000","last_undegraded":"2023-11-16T16:47:43.629372+0000","last_fullsized":"2023-11-16T16:47:43.629372+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:07:42.452837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.624832+0000","last_change":"2023-11-16T16:47:22.660955+0000","last_active":"2023-11-16T16:47:43.624832+0000","last_peered":"2023-11-16T16:47:43.624832+0000","last_clean":"2023-11-16T16:47:43.624832+0000","last_became_active":"2023-11-16T16:47:22.660574+0000","last_became_peered":"2023-11-16T16:47:22.660574+0000","last_unstale":"2023-11-16T16:47:43.624832+0000","last_undegraded":"2023-11-16T16:47:43.624832+0000","last_fullsized":"2023-11-16T16:47:43.624832+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:41:43.745744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630317+0000","last_change":"2023-11-16T16:47:17.630481+0000","last_active":"2023-11-16T16:47:43.630317+0000","last_peered":"2023-11-16T16:47:43.630317+0000","last_clean":"2023-11-16T16:47:43.630317+0000","last_became_active":"2023-11-16T16:47:17.630190+0000","last_became_peered":"2023-11-16T16:47:17.630190+0000","last_unstale":"2023-11-16T16:47:43.630317+0000","last_undegraded":"2023-11-16T16:47:43.630317+0000","last_fullsized":"2023-11-16T16:47:43.630317+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:03:46.177268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622893+0000","last_change":"2023-11-16T16:47:18.179833+0000","last_active":"2023-11-16T16:47:43.622893+0000","last_peered":"2023-11-16T16:47:43.622893+0000","last_clean":"2023-11-16T16:47:43.622893+0000","last_became_active":"2023-11-16T16:47:18.179498+0000","last_became_peered":"2023-11-16T16:47:18.179498+0000","last_unstale":"2023-11-16T16:47:43.622893+0000","last_undegraded":"2023-11-16T16:47:43.622893+0000","last_fullsized":"2023-11-16T16:47:43.622893+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:29:41.782619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622702+0000","last_change":"2023-11-16T16:47:22.656312+0000","last_active":"2023-11-16T16:47:43.622702+0000","last_peered":"2023-11-16T16:47:43.622702+0000","last_clean":"2023-11-16T16:47:43.622702+0000","last_became_active":"2023-11-16T16:47:22.655870+0000","last_became_peered":"2023-11-16T16:47:22.655870+0000","last_unstale":"2023-11-16T16:47:43.622702+0000","last_undegraded":"2023-11-16T16:47:43.622702+0000","last_fullsized":"2023-11-16T16:47:43.622702+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:42:39.568364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629185+0000","last_change":"2023-11-16T16:47:17.622648+0000","last_active":"2023-11-16T16:47:43.629185+0000","last_peered":"2023-11-16T16:47:43.629185+0000","last_clean":"2023-11-16T16:47:43.629185+0000","last_became_active":"2023-11-16T16:47:17.622329+0000","last_became_peered":"2023-11-16T16:47:17.622329+0000","last_unstale":"2023-11-16T16:47:43.629185+0000","last_undegraded":"2023-11-16T16:47:43.629185+0000","last_fullsized":"2023-11-16T16:47:43.629185+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:01:26.211530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622424+0000","last_change":"2023-11-16T16:47:18.179712+0000","last_active":"2023-11-16T16:47:43.622424+0000","last_peered":"2023-11-16T16:47:43.622424+0000","last_clean":"2023-11-16T16:47:43.622424+0000","last_became_active":"2023-11-16T16:47:18.179392+0000","last_became_peered":"2023-11-16T16:47:18.179392+0000","last_unstale":"2023-11-16T16:47:43.622424+0000","last_undegraded":"2023-11-16T16:47:43.622424+0000","last_fullsized":"2023-11-16T16:47:43.622424+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:56:44.447970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.631211+0000","last_change":"2023-11-16T16:47:22.665088+0000","last_active":"2023-11-16T16:47:43.631211+0000","last_peered":"2023-11-16T16:47:43.631211+0000","last_clean":"2023-11-16T16:47:43.631211+0000","last_became_active":"2023-11-16T16:47:22.662802+0000","last_became_peered":"2023-11-16T16:47:22.662802+0000","last_unstale":"2023-11-16T16:47:43.631211+0000","last_undegraded":"2023-11-16T16:47:43.631211+0000","last_fullsized":"2023-11-16T16:47:43.631211+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:51:55.859193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"82'8","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.624624+0000","last_change":"2023-11-16T16:47:17.639246+0000","last_active":"2023-11-16T16:47:43.624624+0000","last_peered":"2023-11-16T16:47:43.624624+0000","last_clean":"2023-11-16T16:47:43.624624+0000","last_became_active":"2023-11-16T16:47:17.638928+0000","last_became_peered":"2023-11-16T16:47:17.638928+0000","last_unstale":"2023-11-16T16:47:43.624624+0000","last_undegraded":"2023-11-16T16:47:43.624624+0000","last_fullsized":"2023-11-16T16:47:43.624624+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:18:41.175512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629991+0000","last_change":"2023-11-16T16:47:18.182609+0000","last_active":"2023-11-16T16:47:43.629991+0000","last_peered":"2023-11-16T16:47:43.629991+0000","last_clean":"2023-11-16T16:47:43.629991+0000","last_became_active":"2023-11-16T16:47:18.182260+0000","last_became_peered":"2023-11-16T16:47:18.182260+0000","last_unstale":"2023-11-16T16:47:43.629991+0000","last_undegraded":"2023-11-16T16:47:43.629991+0000","last_fullsized":"2023-11-16T16:47:43.629991+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:14:14.972367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629794+0000","last_change":"2023-11-16T16:47:22.647586+0000","last_active":"2023-11-16T16:47:43.629794+0000","last_peered":"2023-11-16T16:47:43.629794+0000","last_clean":"2023-11-16T16:47:43.629794+0000","last_became_active":"2023-11-16T16:47:22.647187+0000","last_became_peered":"2023-11-16T16:47:22.647187+0000","last_unstale":"2023-11-16T16:47:43.629794+0000","last_undegraded":"2023-11-16T16:47:43.629794+0000","last_fullsized":"2023-11-16T16:47:43.629794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:01:47.702590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.595597+0000","last_change":"2023-11-16T16:47:43.933501+0000","last_active":"2023-11-16T16:47:44.595597+0000","last_peered":"2023-11-16T16:47:44.595597+0000","last_clean":"2023-11-16T16:47:44.595597+0000","last_became_active":"2023-11-16T16:47:43.631746+0000","last_became_peered":"2023-11-16T16:47:43.631746+0000","last_unstale":"2023-11-16T16:47:44.595597+0000","last_undegraded":"2023-11-16T16:47:44.595597+0000","last_fullsized":"2023-11-16T16:47:44.595597+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:11:10.737388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625043+0000","last_change":"2023-11-16T16:47:18.175210+0000","last_active":"2023-11-16T16:47:43.625043+0000","last_peered":"2023-11-16T16:47:43.625043+0000","last_clean":"2023-11-16T16:47:43.625043+0000","last_became_active":"2023-11-16T16:47:18.174926+0000","last_became_peered":"2023-11-16T16:47:18.174926+0000","last_unstale":"2023-11-16T16:47:43.625043+0000","last_undegraded":"2023-11-16T16:47:43.625043+0000","last_fullsized":"2023-11-16T16:47:43.625043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:31:59.526898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.624874+0000","last_change":"2023-11-16T16:47:22.719361+0000","last_active":"2023-11-16T16:47:43.624874+0000","last_peered":"2023-11-16T16:47:43.624874+0000","last_clean":"2023-11-16T16:47:43.624874+0000","last_became_active":"2023-11-16T16:47:22.718936+0000","last_became_peered":"2023-11-16T16:47:22.718936+0000","last_unstale":"2023-11-16T16:47:43.624874+0000","last_undegraded":"2023-11-16T16:47:43.624874+0000","last_fullsized":"2023-11-16T16:47:43.624874+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:18:52.706386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622234+0000","last_change":"2023-11-16T16:47:17.632623+0000","last_active":"2023-11-16T16:47:43.622234+0000","last_peered":"2023-11-16T16:47:43.622234+0000","last_clean":"2023-11-16T16:47:43.622234+0000","last_became_active":"2023-11-16T16:47:17.631323+0000","last_became_peered":"2023-11-16T16:47:17.631323+0000","last_unstale":"2023-11-16T16:47:43.622234+0000","last_undegraded":"2023-11-16T16:47:43.622234+0000","last_fullsized":"2023-11-16T16:47:43.622234+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:39:00.464312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626526+0000","last_change":"2023-11-16T16:47:18.190779+0000","last_active":"2023-11-16T16:47:43.626526+0000","last_peered":"2023-11-16T16:47:43.626526+0000","last_clean":"2023-11-16T16:47:43.626526+0000","last_became_active":"2023-11-16T16:47:18.190440+0000","last_became_peered":"2023-11-16T16:47:18.190440+0000","last_unstale":"2023-11-16T16:47:43.626526+0000","last_undegraded":"2023-11-16T16:47:43.626526+0000","last_fullsized":"2023-11-16T16:47:43.626526+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:41:07.686317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630403+0000","last_change":"2023-11-16T16:47:22.664847+0000","last_active":"2023-11-16T16:47:43.630403+0000","last_peered":"2023-11-16T16:47:43.630403+0000","last_clean":"2023-11-16T16:47:43.630403+0000","last_became_active":"2023-11-16T16:47:22.659382+0000","last_became_peered":"2023-11-16T16:47:22.659382+0000","last_unstale":"2023-11-16T16:47:43.630403+0000","last_undegraded":"2023-11-16T16:47:43.630403+0000","last_fullsized":"2023-11-16T16:47:43.630403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:11:46.749829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622597+0000","last_change":"2023-11-16T16:47:17.630956+0000","last_active":"2023-11-16T16:47:43.622597+0000","last_peered":"2023-11-16T16:47:43.622597+0000","last_clean":"2023-11-16T16:47:43.622597+0000","last_became_active":"2023-11-16T16:47:17.629864+0000","last_became_peered":"2023-11-16T16:47:17.629864+0000","last_unstale":"2023-11-16T16:47:43.622597+0000","last_undegraded":"2023-11-16T16:47:43.622597+0000","last_fullsized":"2023-11-16T16:47:43.622597+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:01:12.432152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626055+0000","last_change":"2023-11-16T16:47:18.183375+0000","last_active":"2023-11-16T16:47:43.626055+0000","last_peered":"2023-11-16T16:47:43.626055+0000","last_clean":"2023-11-16T16:47:43.626055+0000","last_became_active":"2023-11-16T16:47:18.183050+0000","last_became_peered":"2023-11-16T16:47:18.183050+0000","last_unstale":"2023-11-16T16:47:43.626055+0000","last_undegraded":"2023-11-16T16:47:43.626055+0000","last_fullsized":"2023-11-16T16:47:43.626055+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:27:13.855960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630861+0000","last_change":"2023-11-16T16:47:22.719109+0000","last_active":"2023-11-16T16:47:43.630861+0000","last_peered":"2023-11-16T16:47:43.630861+0000","last_clean":"2023-11-16T16:47:43.630861+0000","last_became_active":"2023-11-16T16:47:22.718680+0000","last_became_peered":"2023-11-16T16:47:22.718680+0000","last_unstale":"2023-11-16T16:47:43.630861+0000","last_undegraded":"2023-11-16T16:47:43.630861+0000","last_fullsized":"2023-11-16T16:47:43.630861+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:36:16.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629990+0000","last_change":"2023-11-16T16:47:18.181632+0000","last_active":"2023-11-16T16:47:43.629990+0000","last_peered":"2023-11-16T16:47:43.629990+0000","last_clean":"2023-11-16T16:47:43.629990+0000","last_became_active":"2023-11-16T16:47:18.181323+0000","last_became_peered":"2023-11-16T16:47:18.181323+0000","last_unstale":"2023-11-16T16:47:43.629990+0000","last_undegraded":"2023-11-16T16:47:43.629990+0000","last_fullsized":"2023-11-16T16:47:43.629990+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:29:17.626827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629025+0000","last_change":"2023-11-16T16:47:17.637269+0000","last_active":"2023-11-16T16:47:43.629025+0000","last_peered":"2023-11-16T16:47:43.629025+0000","last_clean":"2023-11-16T16:47:43.629025+0000","last_became_active":"2023-11-16T16:47:17.636881+0000","last_became_peered":"2023-11-16T16:47:17.636881+0000","last_unstale":"2023-11-16T16:47:43.629025+0000","last_undegraded":"2023-11-16T16:47:43.629025+0000","last_fullsized":"2023-11-16T16:47:43.629025+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:43.798391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623135+0000","last_change":"2023-11-16T16:47:22.659528+0000","last_active":"2023-11-16T16:47:43.623135+0000","last_peered":"2023-11-16T16:47:43.623135+0000","last_clean":"2023-11-16T16:47:43.623135+0000","last_became_active":"2023-11-16T16:47:22.659076+0000","last_became_peered":"2023-11-16T16:47:22.659076+0000","last_unstale":"2023-11-16T16:47:43.623135+0000","last_undegraded":"2023-11-16T16:47:43.623135+0000","last_fullsized":"2023-11-16T16:47:43.623135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:06:05.562183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970448+0000","last_change":"2023-11-16T16:47:18.182102+0000","last_active":"2023-11-16T16:47:43.970448+0000","last_peered":"2023-11-16T16:47:43.970448+0000","last_clean":"2023-11-16T16:47:43.970448+0000","last_became_active":"2023-11-16T16:47:18.181528+0000","last_became_peered":"2023-11-16T16:47:18.181528+0000","last_unstale":"2023-11-16T16:47:43.970448+0000","last_undegraded":"2023-11-16T16:47:43.970448+0000","last_fullsized":"2023-11-16T16:47:43.970448+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:28:03.200017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629674+0000","last_change":"2023-11-16T16:47:17.626655+0000","last_active":"2023-11-16T16:47:43.629674+0000","last_peered":"2023-11-16T16:47:43.629674+0000","last_clean":"2023-11-16T16:47:43.629674+0000","last_became_active":"2023-11-16T16:47:17.626329+0000","last_became_peered":"2023-11-16T16:47:17.626329+0000","last_unstale":"2023-11-16T16:47:43.629674+0000","last_undegraded":"2023-11-16T16:47:43.629674+0000","last_fullsized":"2023-11-16T16:47:43.629674+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:30:26.199727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622227+0000","last_change":"2023-11-16T16:46:11.943398+0000","last_active":"2023-11-16T16:47:43.622227+0000","last_peered":"2023-11-16T16:47:43.622227+0000","last_clean":"2023-11-16T16:47:43.622227+0000","last_became_active":"2023-11-16T16:46:11.942862+0000","last_became_peered":"2023-11-16T16:46:11.942862+0000","last_unstale":"2023-11-16T16:47:43.622227+0000","last_undegraded":"2023-11-16T16:47:43.622227+0000","last_fullsized":"2023-11-16T16:47:43.622227+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:43:03.109797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:43:03.109797+0000","last_clean_scrub_stamp":"2023-11-16T16:43:03.109797+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:14:00.412865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630080+0000","last_change":"2023-11-16T16:47:22.719963+0000","last_active":"2023-11-16T16:47:43.630080+0000","last_peered":"2023-11-16T16:47:43.630080+0000","last_clean":"2023-11-16T16:47:43.630080+0000","last_became_active":"2023-11-16T16:47:22.719506+0000","last_became_peered":"2023-11-16T16:47:22.719506+0000","last_unstale":"2023-11-16T16:47:43.630080+0000","last_undegraded":"2023-11-16T16:47:43.630080+0000","last_fullsized":"2023-11-16T16:47:43.630080+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:27:40.361521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.628810+0000","last_change":"2023-11-16T16:47:18.184630+0000","last_active":"2023-11-16T16:47:43.628810+0000","last_peered":"2023-11-16T16:47:43.628810+0000","last_clean":"2023-11-16T16:47:43.628810+0000","last_became_active":"2023-11-16T16:47:18.181978+0000","last_became_peered":"2023-11-16T16:47:18.181978+0000","last_unstale":"2023-11-16T16:47:43.628810+0000","last_undegraded":"2023-11-16T16:47:43.628810+0000","last_fullsized":"2023-11-16T16:47:43.628810+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:22:39.294919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.635963+0000","last_change":"2023-11-16T16:47:43.635365+0000","last_active":"2023-11-16T16:47:43.635963+0000","last_peered":"2023-11-16T16:47:43.635963+0000","last_clean":"2023-11-16T16:47:43.635963+0000","last_became_active":"2023-11-16T16:47:43.634974+0000","last_became_peered":"2023-11-16T16:47:43.634974+0000","last_unstale":"2023-11-16T16:47:43.635963+0000","last_undegraded":"2023-11-16T16:47:43.635963+0000","last_fullsized":"2023-11-16T16:47:43.635963+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:40:21.789398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625072+0000","last_change":"2023-11-16T16:47:22.657133+0000","last_active":"2023-11-16T16:47:43.625072+0000","last_peered":"2023-11-16T16:47:43.625072+0000","last_clean":"2023-11-16T16:47:43.625072+0000","last_became_active":"2023-11-16T16:47:22.656557+0000","last_became_peered":"2023-11-16T16:47:22.656557+0000","last_unstale":"2023-11-16T16:47:43.625072+0000","last_undegraded":"2023-11-16T16:47:43.625072+0000","last_fullsized":"2023-11-16T16:47:43.625072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:06:56.883403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626111+0000","last_change":"2023-11-16T16:47:17.639107+0000","last_active":"2023-11-16T16:47:43.626111+0000","last_peered":"2023-11-16T16:47:43.626111+0000","last_clean":"2023-11-16T16:47:43.626111+0000","last_became_active":"2023-11-16T16:47:17.638796+0000","last_became_peered":"2023-11-16T16:47:17.638796+0000","last_unstale":"2023-11-16T16:47:43.626111+0000","last_undegraded":"2023-11-16T16:47:43.626111+0000","last_fullsized":"2023-11-16T16:47:43.626111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:15:27.137682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629965+0000","last_change":"2023-11-16T16:47:18.186761+0000","last_active":"2023-11-16T16:47:43.629965+0000","last_peered":"2023-11-16T16:47:43.629965+0000","last_clean":"2023-11-16T16:47:43.629965+0000","last_became_active":"2023-11-16T16:47:18.185594+0000","last_became_peered":"2023-11-16T16:47:18.185594+0000","last_unstale":"2023-11-16T16:47:43.629965+0000","last_undegraded":"2023-11-16T16:47:43.629965+0000","last_fullsized":"2023-11-16T16:47:43.629965+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:30:05.305376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970230+0000","last_change":"2023-11-16T16:47:22.719534+0000","last_active":"2023-11-16T16:47:43.970230+0000","last_peered":"2023-11-16T16:47:43.970230+0000","last_clean":"2023-11-16T16:47:43.970230+0000","last_became_active":"2023-11-16T16:47:22.718738+0000","last_became_peered":"2023-11-16T16:47:22.718738+0000","last_unstale":"2023-11-16T16:47:43.970230+0000","last_undegraded":"2023-11-16T16:47:43.970230+0000","last_fullsized":"2023-11-16T16:47:43.970230+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:14:30.912366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622745+0000","last_change":"2023-11-16T16:47:17.630318+0000","last_active":"2023-11-16T16:47:43.622745+0000","last_peered":"2023-11-16T16:47:43.622745+0000","last_clean":"2023-11-16T16:47:43.622745+0000","last_became_active":"2023-11-16T16:47:17.629035+0000","last_became_peered":"2023-11-16T16:47:17.629035+0000","last_unstale":"2023-11-16T16:47:43.622745+0000","last_undegraded":"2023-11-16T16:47:43.622745+0000","last_fullsized":"2023-11-16T16:47:43.622745+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:44:38.576547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970772+0000","last_change":"2023-11-16T16:47:18.185034+0000","last_active":"2023-11-16T16:47:43.970772+0000","last_peered":"2023-11-16T16:47:43.970772+0000","last_clean":"2023-11-16T16:47:43.970772+0000","last_became_active":"2023-11-16T16:47:18.181961+0000","last_became_peered":"2023-11-16T16:47:18.181961+0000","last_unstale":"2023-11-16T16:47:43.970772+0000","last_undegraded":"2023-11-16T16:47:43.970772+0000","last_fullsized":"2023-11-16T16:47:43.970772+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:38:55.393578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622320+0000","last_change":"2023-11-16T16:47:22.649952+0000","last_active":"2023-11-16T16:47:43.622320+0000","last_peered":"2023-11-16T16:47:43.622320+0000","last_clean":"2023-11-16T16:47:43.622320+0000","last_became_active":"2023-11-16T16:47:22.649481+0000","last_became_peered":"2023-11-16T16:47:22.649481+0000","last_unstale":"2023-11-16T16:47:43.622320+0000","last_undegraded":"2023-11-16T16:47:43.622320+0000","last_fullsized":"2023-11-16T16:47:43.622320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:36:07.109965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"82'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629922+0000","last_change":"2023-11-16T16:47:17.631982+0000","last_active":"2023-11-16T16:47:43.629922+0000","last_peered":"2023-11-16T16:47:43.629922+0000","last_clean":"2023-11-16T16:47:43.629922+0000","last_became_active":"2023-11-16T16:47:17.631552+0000","last_became_peered":"2023-11-16T16:47:17.631552+0000","last_unstale":"2023-11-16T16:47:43.629922+0000","last_undegraded":"2023-11-16T16:47:43.629922+0000","last_fullsized":"2023-11-16T16:47:43.629922+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:35:56.360671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626666+0000","last_change":"2023-11-16T16:47:18.197325+0000","last_active":"2023-11-16T16:47:43.626666+0000","last_peered":"2023-11-16T16:47:43.626666+0000","last_clean":"2023-11-16T16:47:43.626666+0000","last_became_active":"2023-11-16T16:47:18.190602+0000","last_became_peered":"2023-11-16T16:47:18.190602+0000","last_unstale":"2023-11-16T16:47:43.626666+0000","last_undegraded":"2023-11-16T16:47:43.626666+0000","last_fullsized":"2023-11-16T16:47:43.626666+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:28:40.186656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629554+0000","last_change":"2023-11-16T16:47:22.720023+0000","last_active":"2023-11-16T16:47:43.629554+0000","last_peered":"2023-11-16T16:47:43.629554+0000","last_clean":"2023-11-16T16:47:43.629554+0000","last_became_active":"2023-11-16T16:47:22.719561+0000","last_became_peered":"2023-11-16T16:47:22.719561+0000","last_unstale":"2023-11-16T16:47:43.629554+0000","last_undegraded":"2023-11-16T16:47:43.629554+0000","last_fullsized":"2023-11-16T16:47:43.629554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:40:41.808974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.946763+0000","last_change":"2023-11-16T16:47:17.629379+0000","last_active":"2023-11-16T16:47:43.946763+0000","last_peered":"2023-11-16T16:47:43.946763+0000","last_clean":"2023-11-16T16:47:43.946763+0000","last_became_active":"2023-11-16T16:47:17.629025+0000","last_became_peered":"2023-11-16T16:47:17.629025+0000","last_unstale":"2023-11-16T16:47:43.946763+0000","last_undegraded":"2023-11-16T16:47:43.946763+0000","last_fullsized":"2023-11-16T16:47:43.946763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:23:29.948397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626265+0000","last_change":"2023-11-16T16:47:18.185418+0000","last_active":"2023-11-16T16:47:43.626265+0000","last_peered":"2023-11-16T16:47:43.626265+0000","last_clean":"2023-11-16T16:47:43.626265+0000","last_became_active":"2023-11-16T16:47:18.185092+0000","last_became_peered":"2023-11-16T16:47:18.185092+0000","last_unstale":"2023-11-16T16:47:43.626265+0000","last_undegraded":"2023-11-16T16:47:43.626265+0000","last_fullsized":"2023-11-16T16:47:43.626265+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:02:15.891150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622811+0000","last_change":"2023-11-16T16:47:22.650701+0000","last_active":"2023-11-16T16:47:43.622811+0000","last_peered":"2023-11-16T16:47:43.622811+0000","last_clean":"2023-11-16T16:47:43.622811+0000","last_became_active":"2023-11-16T16:47:22.648710+0000","last_became_peered":"2023-11-16T16:47:22.648710+0000","last_unstale":"2023-11-16T16:47:43.622811+0000","last_undegraded":"2023-11-16T16:47:43.622811+0000","last_fullsized":"2023-11-16T16:47:43.622811+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:50:37.989145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970985+0000","last_change":"2023-11-16T16:47:17.638622+0000","last_active":"2023-11-16T16:47:43.970985+0000","last_peered":"2023-11-16T16:47:43.970985+0000","last_clean":"2023-11-16T16:47:43.970985+0000","last_became_active":"2023-11-16T16:47:17.638303+0000","last_became_peered":"2023-11-16T16:47:17.638303+0000","last_unstale":"2023-11-16T16:47:43.970985+0000","last_undegraded":"2023-11-16T16:47:43.970985+0000","last_fullsized":"2023-11-16T16:47:43.970985+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:03:52.117172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626598+0000","last_change":"2023-11-16T16:47:18.173474+0000","last_active":"2023-11-16T16:47:43.626598+0000","last_peered":"2023-11-16T16:47:43.626598+0000","last_clean":"2023-11-16T16:47:43.626598+0000","last_became_active":"2023-11-16T16:47:18.173140+0000","last_became_peered":"2023-11-16T16:47:18.173140+0000","last_unstale":"2023-11-16T16:47:43.626598+0000","last_undegraded":"2023-11-16T16:47:43.626598+0000","last_fullsized":"2023-11-16T16:47:43.626598+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:13:23.724052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629348+0000","last_change":"2023-11-16T16:47:22.655806+0000","last_active":"2023-11-16T16:47:43.629348+0000","last_peered":"2023-11-16T16:47:43.629348+0000","last_clean":"2023-11-16T16:47:43.629348+0000","last_became_active":"2023-11-16T16:47:22.655385+0000","last_became_peered":"2023-11-16T16:47:22.655385+0000","last_unstale":"2023-11-16T16:47:43.629348+0000","last_undegraded":"2023-11-16T16:47:43.629348+0000","last_fullsized":"2023-11-16T16:47:43.629348+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:31:41.160819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622449+0000","last_change":"2023-11-16T16:47:17.631673+0000","last_active":"2023-11-16T16:47:43.622449+0000","last_peered":"2023-11-16T16:47:43.622449+0000","last_clean":"2023-11-16T16:47:43.622449+0000","last_became_active":"2023-11-16T16:47:17.631356+0000","last_became_peered":"2023-11-16T16:47:17.631356+0000","last_unstale":"2023-11-16T16:47:43.622449+0000","last_undegraded":"2023-11-16T16:47:43.622449+0000","last_fullsized":"2023-11-16T16:47:43.622449+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:29:12.540977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970538+0000","last_change":"2023-11-16T16:47:18.186706+0000","last_active":"2023-11-16T16:47:43.970538+0000","last_peered":"2023-11-16T16:47:43.970538+0000","last_clean":"2023-11-16T16:47:43.970538+0000","last_became_active":"2023-11-16T16:47:18.186360+0000","last_became_peered":"2023-11-16T16:47:18.186360+0000","last_unstale":"2023-11-16T16:47:43.970538+0000","last_undegraded":"2023-11-16T16:47:43.970538+0000","last_fullsized":"2023-11-16T16:47:43.970538+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:32:31.165136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629693+0000","last_change":"2023-11-16T16:47:22.719065+0000","last_active":"2023-11-16T16:47:43.629693+0000","last_peered":"2023-11-16T16:47:43.629693+0000","last_clean":"2023-11-16T16:47:43.629693+0000","last_became_active":"2023-11-16T16:47:22.718662+0000","last_became_peered":"2023-11-16T16:47:22.718662+0000","last_unstale":"2023-11-16T16:47:43.629693+0000","last_undegraded":"2023-11-16T16:47:43.629693+0000","last_fullsized":"2023-11-16T16:47:43.629693+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:56:10.235168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"82'2","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.638200+0000","last_change":"2023-11-16T16:47:43.637790+0000","last_active":"2023-11-16T16:47:43.638200+0000","last_peered":"2023-11-16T16:47:43.638200+0000","last_clean":"2023-11-16T16:47:43.638200+0000","last_became_active":"2023-11-16T16:47:43.632225+0000","last_became_peered":"2023-11-16T16:47:43.632225+0000","last_unstale":"2023-11-16T16:47:43.638200+0000","last_undegraded":"2023-11-16T16:47:43.638200+0000","last_fullsized":"2023-11-16T16:47:43.638200+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:31:01.698264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622703+0000","last_change":"2023-11-16T16:47:18.180999+0000","last_active":"2023-11-16T16:47:43.622703+0000","last_peered":"2023-11-16T16:47:43.622703+0000","last_clean":"2023-11-16T16:47:43.622703+0000","last_became_active":"2023-11-16T16:47:18.180724+0000","last_became_peered":"2023-11-16T16:47:18.180724+0000","last_unstale":"2023-11-16T16:47:43.622703+0000","last_undegraded":"2023-11-16T16:47:43.622703+0000","last_fullsized":"2023-11-16T16:47:43.622703+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:38:44.147293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947186+0000","last_change":"2023-11-16T16:47:22.665607+0000","last_active":"2023-11-16T16:47:43.947186+0000","last_peered":"2023-11-16T16:47:43.947186+0000","last_clean":"2023-11-16T16:47:43.947186+0000","last_became_active":"2023-11-16T16:47:22.665273+0000","last_became_peered":"2023-11-16T16:47:22.665273+0000","last_unstale":"2023-11-16T16:47:43.947186+0000","last_undegraded":"2023-11-16T16:47:43.947186+0000","last_fullsized":"2023-11-16T16:47:43.947186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:08:37.804057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622937+0000","last_change":"2023-11-16T16:47:17.632614+0000","last_active":"2023-11-16T16:47:43.622937+0000","last_peered":"2023-11-16T16:47:43.622937+0000","last_clean":"2023-11-16T16:47:43.622937+0000","last_became_active":"2023-11-16T16:47:17.632227+0000","last_became_peered":"2023-11-16T16:47:17.632227+0000","last_unstale":"2023-11-16T16:47:43.622937+0000","last_undegraded":"2023-11-16T16:47:43.622937+0000","last_fullsized":"2023-11-16T16:47:43.622937+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:58:13.465619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623388+0000","last_change":"2023-11-16T16:47:18.177607+0000","last_active":"2023-11-16T16:47:43.623388+0000","last_peered":"2023-11-16T16:47:43.623388+0000","last_clean":"2023-11-16T16:47:43.623388+0000","last_became_active":"2023-11-16T16:47:18.177266+0000","last_became_peered":"2023-11-16T16:47:18.177266+0000","last_unstale":"2023-11-16T16:47:43.623388+0000","last_undegraded":"2023-11-16T16:47:43.623388+0000","last_fullsized":"2023-11-16T16:47:43.623388+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:03:08.676930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625888+0000","last_change":"2023-11-16T16:47:22.659987+0000","last_active":"2023-11-16T16:47:43.625888+0000","last_peered":"2023-11-16T16:47:43.625888+0000","last_clean":"2023-11-16T16:47:43.625888+0000","last_became_active":"2023-11-16T16:47:22.658725+0000","last_became_peered":"2023-11-16T16:47:22.658725+0000","last_unstale":"2023-11-16T16:47:43.625888+0000","last_undegraded":"2023-11-16T16:47:43.625888+0000","last_fullsized":"2023-11-16T16:47:43.625888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:51:27.380649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630637+0000","last_change":"2023-11-16T16:47:17.640696+0000","last_active":"2023-11-16T16:47:43.630637+0000","last_peered":"2023-11-16T16:47:43.630637+0000","last_clean":"2023-11-16T16:47:43.630637+0000","last_became_active":"2023-11-16T16:47:17.640428+0000","last_became_peered":"2023-11-16T16:47:17.640428+0000","last_unstale":"2023-11-16T16:47:43.630637+0000","last_undegraded":"2023-11-16T16:47:43.630637+0000","last_fullsized":"2023-11-16T16:47:43.630637+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:47:12.331771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.972273+0000","last_change":"2023-11-16T16:47:43.972273+0000","last_active":"2023-11-16T16:47:43.972273+0000","last_peered":"2023-11-16T16:47:43.972273+0000","last_clean":"2023-11-16T16:47:43.972273+0000","last_became_active":"2023-11-16T16:47:43.971981+0000","last_became_peered":"2023-11-16T16:47:43.971981+0000","last_unstale":"2023-11-16T16:47:43.972273+0000","last_undegraded":"2023-11-16T16:47:43.972273+0000","last_fullsized":"2023-11-16T16:47:43.972273+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:38:01.080979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629465+0000","last_change":"2023-11-16T16:47:22.649436+0000","last_active":"2023-11-16T16:47:43.629465+0000","last_peered":"2023-11-16T16:47:43.629465+0000","last_clean":"2023-11-16T16:47:43.629465+0000","last_became_active":"2023-11-16T16:47:22.649034+0000","last_became_peered":"2023-11-16T16:47:22.649034+0000","last_unstale":"2023-11-16T16:47:43.629465+0000","last_undegraded":"2023-11-16T16:47:43.629465+0000","last_fullsized":"2023-11-16T16:47:43.629465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:50:06.438075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.631007+0000","last_change":"2023-11-16T16:47:17.634582+0000","last_active":"2023-11-16T16:47:43.631007+0000","last_peered":"2023-11-16T16:47:43.631007+0000","last_clean":"2023-11-16T16:47:43.631007+0000","last_became_active":"2023-11-16T16:47:17.634172+0000","last_became_peered":"2023-11-16T16:47:17.634172+0000","last_unstale":"2023-11-16T16:47:43.631007+0000","last_undegraded":"2023-11-16T16:47:43.631007+0000","last_fullsized":"2023-11-16T16:47:43.631007+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:50:55.582584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623419+0000","last_change":"2023-11-16T16:47:18.180466+0000","last_active":"2023-11-16T16:47:43.623419+0000","last_peered":"2023-11-16T16:47:43.623419+0000","last_clean":"2023-11-16T16:47:43.623419+0000","last_became_active":"2023-11-16T16:47:18.180166+0000","last_became_peered":"2023-11-16T16:47:18.180166+0000","last_unstale":"2023-11-16T16:47:43.623419+0000","last_undegraded":"2023-11-16T16:47:43.623419+0000","last_fullsized":"2023-11-16T16:47:43.623419+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:02:56.576134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626739+0000","last_change":"2023-11-16T16:47:18.179196+0000","last_active":"2023-11-16T16:47:43.626739+0000","last_peered":"2023-11-16T16:47:43.626739+0000","last_clean":"2023-11-16T16:47:43.626739+0000","last_became_active":"2023-11-16T16:47:18.178827+0000","last_became_peered":"2023-11-16T16:47:18.178827+0000","last_unstale":"2023-11-16T16:47:43.626739+0000","last_undegraded":"2023-11-16T16:47:43.626739+0000","last_fullsized":"2023-11-16T16:47:43.626739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:07:11.423464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622851+0000","last_change":"2023-11-16T16:47:18.184630+0000","last_active":"2023-11-16T16:47:43.622851+0000","last_peered":"2023-11-16T16:47:43.622851+0000","last_clean":"2023-11-16T16:47:43.622851+0000","last_became_active":"2023-11-16T16:47:18.184314+0000","last_became_peered":"2023-11-16T16:47:18.184314+0000","last_unstale":"2023-11-16T16:47:43.622851+0000","last_undegraded":"2023-11-16T16:47:43.622851+0000","last_fullsized":"2023-11-16T16:47:43.622851+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:56:13.651803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622558+0000","last_change":"2023-11-16T16:47:18.184255+0000","last_active":"2023-11-16T16:47:43.622558+0000","last_peered":"2023-11-16T16:47:43.622558+0000","last_clean":"2023-11-16T16:47:43.622558+0000","last_became_active":"2023-11-16T16:47:18.183936+0000","last_became_peered":"2023-11-16T16:47:18.183936+0000","last_unstale":"2023-11-16T16:47:43.622558+0000","last_undegraded":"2023-11-16T16:47:43.622558+0000","last_fullsized":"2023-11-16T16:47:43.622558+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:59:41.136565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626136+0000","last_change":"2023-11-16T16:47:18.196331+0000","last_active":"2023-11-16T16:47:43.626136+0000","last_peered":"2023-11-16T16:47:43.626136+0000","last_clean":"2023-11-16T16:47:43.626136+0000","last_became_active":"2023-11-16T16:47:18.195954+0000","last_became_peered":"2023-11-16T16:47:18.195954+0000","last_unstale":"2023-11-16T16:47:43.626136+0000","last_undegraded":"2023-11-16T16:47:43.626136+0000","last_fullsized":"2023-11-16T16:47:43.626136+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:31:23.208122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629719+0000","last_change":"2023-11-16T16:47:18.190537+0000","last_active":"2023-11-16T16:47:43.629719+0000","last_peered":"2023-11-16T16:47:43.629719+0000","last_clean":"2023-11-16T16:47:43.629719+0000","last_became_active":"2023-11-16T16:47:18.190118+0000","last_became_peered":"2023-11-16T16:47:18.190118+0000","last_unstale":"2023-11-16T16:47:43.629719+0000","last_undegraded":"2023-11-16T16:47:43.629719+0000","last_fullsized":"2023-11-16T16:47:43.629719+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:39:38.125056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625721+0000","last_change":"2023-11-16T16:47:18.179215+0000","last_active":"2023-11-16T16:47:43.625721+0000","last_peered":"2023-11-16T16:47:43.625721+0000","last_clean":"2023-11-16T16:47:43.625721+0000","last_became_active":"2023-11-16T16:47:18.178759+0000","last_became_peered":"2023-11-16T16:47:18.178759+0000","last_unstale":"2023-11-16T16:47:43.625721+0000","last_undegraded":"2023-11-16T16:47:43.625721+0000","last_fullsized":"2023-11-16T16:47:43.625721+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:06:54.886510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623599+0000","last_change":"2023-11-16T16:47:18.179588+0000","last_active":"2023-11-16T16:47:43.623599+0000","last_peered":"2023-11-16T16:47:43.623599+0000","last_clean":"2023-11-16T16:47:43.623599+0000","last_became_active":"2023-11-16T16:47:18.179294+0000","last_became_peered":"2023-11-16T16:47:18.179294+0000","last_unstale":"2023-11-16T16:47:43.623599+0000","last_undegraded":"2023-11-16T16:47:43.623599+0000","last_fullsized":"2023-11-16T16:47:43.623599+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:02:55.079560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.971128+0000","last_change":"2023-11-16T16:47:18.181587+0000","last_active":"2023-11-16T16:47:43.971128+0000","last_peered":"2023-11-16T16:47:43.971128+0000","last_clean":"2023-11-16T16:47:43.971128+0000","last_became_active":"2023-11-16T16:47:18.181203+0000","last_became_peered":"2023-11-16T16:47:18.181203+0000","last_unstale":"2023-11-16T16:47:43.971128+0000","last_undegraded":"2023-11-16T16:47:43.971128+0000","last_fullsized":"2023-11-16T16:47:43.971128+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:13:49.378115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629816+0000","last_change":"2023-11-16T16:47:18.182690+0000","last_active":"2023-11-16T16:47:43.629816+0000","last_peered":"2023-11-16T16:47:43.629816+0000","last_clean":"2023-11-16T16:47:43.629816+0000","last_became_active":"2023-11-16T16:47:18.182361+0000","last_became_peered":"2023-11-16T16:47:18.182361+0000","last_unstale":"2023-11-16T16:47:43.629816+0000","last_undegraded":"2023-11-16T16:47:43.629816+0000","last_fullsized":"2023-11-16T16:47:43.629816+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:55:41.987600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970993+0000","last_change":"2023-11-16T16:47:18.184599+0000","last_active":"2023-11-16T16:47:43.970993+0000","last_peered":"2023-11-16T16:47:43.970993+0000","last_clean":"2023-11-16T16:47:43.970993+0000","last_became_active":"2023-11-16T16:47:18.184288+0000","last_became_peered":"2023-11-16T16:47:18.184288+0000","last_unstale":"2023-11-16T16:47:43.970993+0000","last_undegraded":"2023-11-16T16:47:43.970993+0000","last_fullsized":"2023-11-16T16:47:43.970993+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:43:20.190571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947382+0000","last_change":"2023-11-16T16:47:18.187267+0000","last_active":"2023-11-16T16:47:43.947382+0000","last_peered":"2023-11-16T16:47:43.947382+0000","last_clean":"2023-11-16T16:47:43.947382+0000","last_became_active":"2023-11-16T16:47:18.186984+0000","last_became_peered":"2023-11-16T16:47:18.186984+0000","last_unstale":"2023-11-16T16:47:43.947382+0000","last_undegraded":"2023-11-16T16:47:43.947382+0000","last_fullsized":"2023-11-16T16:47:43.947382+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:30:20.891991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630154+0000","last_change":"2023-11-16T16:47:18.191566+0000","last_active":"2023-11-16T16:47:43.630154+0000","last_peered":"2023-11-16T16:47:43.630154+0000","last_clean":"2023-11-16T16:47:43.630154+0000","last_became_active":"2023-11-16T16:47:18.185342+0000","last_became_peered":"2023-11-16T16:47:18.185342+0000","last_unstale":"2023-11-16T16:47:43.630154+0000","last_undegraded":"2023-11-16T16:47:43.630154+0000","last_fullsized":"2023-11-16T16:47:43.630154+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:55:29.393440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629636+0000","last_change":"2023-11-16T16:47:43.629636+0000","last_active":"2023-11-16T16:47:43.629636+0000","last_peered":"2023-11-16T16:47:43.629636+0000","last_clean":"2023-11-16T16:47:43.629636+0000","last_became_active":"2023-11-16T16:47:43.629179+0000","last_became_peered":"2023-11-16T16:47:43.629179+0000","last_unstale":"2023-11-16T16:47:43.629636+0000","last_undegraded":"2023-11-16T16:47:43.629636+0000","last_fullsized":"2023-11-16T16:47:43.629636+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:21:10.581239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281399,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4592,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":9,"num_read_kb":11,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":31157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449107,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6136,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739016,"statfs":{"total":95995035648,"available":95988752384,"internally_reserved":0,"allocated":966656,"data_stored":483162,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51000000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.56100000000000005}]},{"osd":2,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.34899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.59399999999999997}]},{"osd":5,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.47799999999999998}]},{"osd":6,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.626}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":9,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.64400000000000002}]},{"osd":10,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.80900000000000005}]}]},{"osd":10,"up_from":71,"seq":304942678039,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6332,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738820,"statfs":{"total":95995035648,"available":95988551680,"internally_reserved":0,"allocated":1036288,"data_stored":1077496,"data_compressed":53424,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":1,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.47999999999999998}]},{"osd":2,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.65300000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63}]},{"osd":5,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.78300000000000003}]},{"osd":6,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.76800000000000002}]},{"osd":7,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.69999999999999996}]},{"osd":8,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.75600000000000001}]},{"osd":9,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.80700000000000005}]},{"osd":11,"last update":"Thu Nov 16 16:47:30 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.85299999999999998}]}]},{"osd":9,"up_from":66,"seq":283467841563,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6268,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738884,"statfs":{"total":95995035648,"available":95988617216,"internally_reserved":0,"allocated":970752,"data_stored":487818,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.629}]},{"osd":1,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.68799999999999994}]},{"osd":2,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.71199999999999997}]},{"osd":3,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.94399999999999995}]},{"osd":4,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.753},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.78800000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.032}]},{"osd":6,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":1.054},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.95599999999999996}]},{"osd":7,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.878},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":1.0740000000000001}]},{"osd":8,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.92200000000000004}]},{"osd":10,"last update":"Thu Nov 16 16:47:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.016},{"interface":"front","average":{"1min":1.05,"5min":1.05,"15min":1.05},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":1.0960000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]}]},{"osd":8,"up_from":58,"seq":249108103200,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6392,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738760,"statfs":{"total":95995035648,"available":95988490240,"internally_reserved":0,"allocated":966656,"data_stored":483722,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:23 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.78600000000000003}]},{"osd":1,"last update":"Thu Nov 16 16:47:23 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.60899999999999999,"15min":0.59599999999999997},"min":{"1min":0.435,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.61199999999999999,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Nov 16 16:47:23 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.46400000000000002,"15min":0.45400000000000001},"min":{"1min":0.29599999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.46500000000000002,"15min":0.46300000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.441}]},{"osd":3,"last update":"Thu Nov 16 16:47:23 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.44800000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.92200000000000004,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.75900000000000001}]},{"osd":4,"last update":"Thu Nov 16 16:47:23 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.628,"15min":0.625},"min":{"1min":0.442,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.95599999999999996,"5min":0.999,"15min":0.999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69199999999999995,"15min":0.68600000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.80100000000000005}]},{"osd":5,"last update":"Thu Nov 16 16:47:23 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53700000000000003,"15min":0.52900000000000003},"min":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56399999999999995,"15min":0.55900000000000005},"min":{"1min":0.35599999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.47199999999999998}]},{"osd":6,"last update":"Thu Nov 16 16:47:23 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.72899999999999998,"15min":0.72199999999999998},"min":{"1min":0.52800000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.68100000000000005,"15min":0.67600000000000005},"min":{"1min":0.47099999999999997,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.71399999999999997}]},{"osd":7,"last update":"Thu Nov 16 16:47:23 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.69299999999999995,"15min":0.68700000000000006},"min":{"1min":0.46800000000000003,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.751,"5min":0.72599999999999998,"15min":0.72199999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.86299999999999999}]},{"osd":9,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77200000000000002,"15min":0.77100000000000002},"min":{"1min":0.52200000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92000000000000004,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.81200000000000006,"15min":0.81399999999999995},"min":{"1min":0.55900000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0069999999999999,"5min":1.298,"15min":1.298},"last":0.93700000000000006}]},{"osd":10,"last update":"Thu Nov 16 16:47:08 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.89200000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]},{"osd":7,"up_from":51,"seq":219043332132,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6504,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738648,"statfs":{"total":95995035648,"available":95988375552,"internally_reserved":0,"allocated":1015808,"data_stored":1074090,"data_compressed":53424,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.377,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81799999999999995,"5min":0.875,"15min":0.875},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.41999999999999998,"15min":0.40799999999999997},"min":{"1min":0.32900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.44900000000000001,"15min":0.44},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73599999999999999,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.45800000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.78400000000000003}]},{"osd":3,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.56599999999999995,"15min":0.55200000000000005},"min":{"1min":0.437,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.629,"15min":0.621},"min":{"1min":0.51600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.86699999999999999,"5min":0.873,"15min":0.873},"last":0.73299999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53200000000000003,"15min":0.52900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.82499999999999996,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.56100000000000005}]},{"osd":5,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.86799999999999999,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.64100000000000001,"15min":0.63100000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.89400000000000002}]},{"osd":6,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.69099999999999995,"15min":0.67800000000000005},"min":{"1min":0.63400000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68000000000000005,"15min":0.67200000000000004},"min":{"1min":0.48699999999999999,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.74299999999999999}]},{"osd":8,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.68000000000000005,"15min":0.67200000000000004},"min":{"1min":0.53900000000000003,"5min":0.436,"15min":0.436},"max":{"1min":0.95599999999999996,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.47399999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.94099999999999995,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.878}]},{"osd":9,"last update":"Thu Nov 16 16:46:47 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.93100000000000005}]},{"osd":10,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.85199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]}]},{"osd":6,"up_from":46,"seq":197568495657,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6388,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738764,"statfs":{"total":95995035648,"available":95988494336,"internally_reserved":0,"allocated":962560,"data_stored":483140,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.38800000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.43099999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84599999999999997,"5min":1.345,"15min":1.345},"last":1.0680000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64900000000000002,"15min":0.64800000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.98199999999999998,"5min":1.111,"15min":1.111},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69699999999999995,"15min":0.70399999999999996},"min":{"1min":0.505,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0229999999999999,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.72199999999999998}]},{"osd":2,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.71799999999999997,"15min":0.70799999999999996},"min":{"1min":0.54900000000000004,"5min":0.436,"15min":0.436},"max":{"1min":0.998,"5min":1.407,"15min":1.407},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.71899999999999997,"15min":0.72499999999999998},"min":{"1min":0.48799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.98699999999999999,"5min":1.171,"15min":1.171},"last":0.59999999999999998}]},{"osd":3,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.72799999999999998,"15min":0.73999999999999999},"min":{"1min":0.49099999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0189999999999999,"5min":1.022,"15min":1.022},"last":0.77700000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.746,"15min":0.73199999999999998},"min":{"1min":0.56899999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.755,"15min":0.76000000000000001},"min":{"1min":0.58499999999999996,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.048,"5min":1.083,"15min":1.083},"last":0.83599999999999997}]},{"osd":5,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79000000000000004,"15min":0.80000000000000004},"min":{"1min":0.61099999999999999,"5min":0.503,"15min":0.503},"max":{"1min":1.0940000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":1.034},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.88300000000000001,"15min":0.89700000000000002},"min":{"1min":0.747,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.2749999999999999,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.93999999999999995}]},{"osd":7,"last update":"Thu Nov 16 16:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.80500000000000005,"15min":0.79300000000000004},"min":{"1min":0.54800000000000004,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.88},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.754,"15min":0.74399999999999999},"min":{"1min":0.55400000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.90500000000000003}]},{"osd":8,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.83899999999999997,"15min":0.84499999999999997},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.88800000000000001,"15min":0.88700000000000001},"min":{"1min":0.66200000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.3360000000000001,"5min":1.823,"15min":1.823},"last":1.284}]},{"osd":9,"last update":"Thu Nov 16 16:46:48 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":1.238},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":1.0069999999999999}]},{"osd":10,"last update":"Thu Nov 16 16:47:11 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.097},{"interface":"front","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":1.1859999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7296,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737856,"statfs":{"total":95995035648,"available":95987564544,"internally_reserved":0,"allocated":974848,"data_stored":486934,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55100000000000005,"15min":0.52500000000000002},"min":{"1min":0.39600000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.77500000000000002,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.54000000000000004,"15min":0.51400000000000001},"min":{"1min":0.40500000000000003,"5min":0.315,"15min":0.315},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.56699999999999995}]},{"osd":2,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.30099999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72599999999999998,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.51200000000000001,"15min":0.499},"min":{"1min":0.40100000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.60999999999999999}]},{"osd":3,"last update":"Thu Nov 16 16:47:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55000000000000004,"15min":0.53400000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.58899999999999997,"15min":0.57299999999999995},"min":{"1min":0.41199999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Nov 16 16:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.41899999999999998,"15min":0.41499999999999998},"min":{"1min":0.26500000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.435,"15min":0.43099999999999999},"min":{"1min":0.28499999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.78900000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:47:15 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.47099999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76600000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.72199999999999998}]},{"osd":6,"last update":"Thu Nov 16 16:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.42599999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82999999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.65800000000000003,"15min":0.64200000000000002},"min":{"1min":0.48899999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.80700000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:47:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.52700000000000002,"15min":0.51300000000000001},"min":{"1min":0.39100000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.56200000000000006,"15min":0.55200000000000005},"min":{"1min":0.36799999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.84299999999999997}]},{"osd":8,"last update":"Thu Nov 16 16:47:24 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.64400000000000002,"15min":0.63200000000000001},"min":{"1min":0.42599999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.871},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.65400000000000003,"15min":0.64400000000000002},"min":{"1min":0.46200000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.85899999999999999}]},{"osd":9,"last update":"Thu Nov 16 16:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.89400000000000002}]},{"osd":10,"last update":"Thu Nov 16 16:47:08 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.90700000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]}]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7288,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737864,"statfs":{"total":95995035648,"available":95987572736,"internally_reserved":0,"allocated":966656,"data_stored":483230,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.55800000000000005,"15min":0.53800000000000003},"min":{"1min":0.42099999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.58499999999999996,"15min":0.55500000000000005},"min":{"1min":0.50900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.73999999999999999}]},{"osd":2,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.66800000000000004,"15min":0.58999999999999997},"min":{"1min":0.50900000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.97499999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.64500000000000002,"15min":0.60999999999999999},"min":{"1min":0.47999999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.71099999999999997}]},{"osd":3,"last update":"Thu Nov 16 16:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.53800000000000003,"15min":0.52400000000000002},"min":{"1min":0.438,"5min":0.31,"15min":0.31},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.53000000000000003,"15min":0.51400000000000001},"min":{"1min":0.33400000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.46500000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.63,"15min":0.58299999999999996},"min":{"1min":0.44,"5min":0.434,"15min":0.434},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.75800000000000001,"15min":0.79100000000000004},"min":{"1min":0.55600000000000005,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0069999999999999,"5min":5.4409999999999998,"15min":5.4409999999999998},"last":0.63900000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:47:17 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72899999999999998,"15min":0.69899999999999995},"min":{"1min":0.57199999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0089999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.75},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72199999999999998,"15min":0.71599999999999997},"min":{"1min":0.44400000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.97299999999999998,"5min":1.153,"15min":1.153},"last":0.81699999999999995}]},{"osd":6,"last update":"Thu Nov 16 16:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.70299999999999996,"15min":0.68700000000000006},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.98599999999999999,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.69499999999999995,"15min":0.66600000000000004},"min":{"1min":0.42499999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.036,"5min":1.093,"15min":1.093},"last":0.68400000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.79100000000000004,"15min":0.78700000000000003},"min":{"1min":0.54400000000000004,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.83499999999999996,"15min":0.82999999999999996},"min":{"1min":0.58699999999999997,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.77000000000000002}]},{"osd":8,"last update":"Thu Nov 16 16:47:22 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.86799999999999999,"15min":0.88},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.111,"5min":1.298,"15min":1.298},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.90600000000000003,"15min":0.91800000000000004},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.1319999999999999,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.99199999999999999}]},{"osd":9,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.78700000000000003}]},{"osd":10,"last update":"Thu Nov 16 16:47:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":3.694,"5min":3.694,"15min":3.694},"last":1.016}]},{"osd":11,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.96499999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7360,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737792,"statfs":{"total":95995035648,"available":95987499008,"internally_reserved":0,"allocated":974848,"data_stored":486695,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.503,"15min":0.45900000000000002},"min":{"1min":0.34599999999999997,"5min":0.312,"15min":0.312},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51000000000000001,"15min":0.49099999999999999},"min":{"1min":0.312,"5min":0.309,"15min":0.309},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.42099999999999999}]},{"osd":1,"last update":"Thu Nov 16 16:47:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.51300000000000001,"15min":0.48499999999999999},"min":{"1min":0.35199999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.52000000000000002,"15min":0.48799999999999999},"min":{"1min":0.36699999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77000000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.77600000000000002}]},{"osd":3,"last update":"Thu Nov 16 16:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68100000000000005,"15min":0.72099999999999997},"min":{"1min":0.49299999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.89300000000000002,"5min":3.98,"15min":3.98},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.60399999999999998,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.52500000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:46:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.72799999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.64100000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:47:19 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46200000000000002,"15min":0.45600000000000002},"min":{"1min":0.29899999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.48699999999999999,"15min":0.47499999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Nov 16 16:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.59199999999999997,"15min":0.58199999999999996},"min":{"1min":0.44400000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.59299999999999997}]},{"osd":7,"last update":"Thu Nov 16 16:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.749},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65500000000000003,"15min":0.65600000000000003},"min":{"1min":0.46899999999999997,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.80200000000000005}]},{"osd":8,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59499999999999997,"15min":0.59399999999999997},"min":{"1min":0.37,"5min":0.309,"15min":0.309},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61399999999999999}]},{"osd":9,"last update":"Thu Nov 16 16:46:47 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.68300000000000005}]},{"osd":10,"last update":"Thu Nov 16 16:47:11 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.76100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6716,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738436,"statfs":{"total":95995035648,"available":95988158464,"internally_reserved":0,"allocated":970752,"data_stored":483244,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:35 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.45500000000000002,"15min":0.42599999999999999},"min":{"1min":0.40500000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57099999999999995,"5min":1.085,"15min":1.085},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54500000000000004,"15min":0.49299999999999999},"min":{"1min":0.36799999999999999,"5min":0.313,"15min":0.313},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Nov 16 16:47:35 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.60499999999999998,"15min":0.57399999999999995},"min":{"1min":0.41899999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.96999999999999997,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.63200000000000001,"15min":0.59499999999999997},"min":{"1min":0.45400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.006,"5min":1.286,"15min":1.286},"last":0.66000000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:47:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.63300000000000001,"15min":0.58699999999999997},"min":{"1min":0.53200000000000003,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.92500000000000004,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":1.02},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.64500000000000002,"15min":0.60099999999999998},"min":{"1min":0.48499999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.94199999999999995}]},{"osd":4,"last update":"Thu Nov 16 16:46:56 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.63800000000000001,"15min":0.60399999999999998},"min":{"1min":0.41499999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.93200000000000005,"5min":1.25,"15min":1.25},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.68300000000000005,"15min":0.66200000000000003},"min":{"1min":0.49199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.115,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.751}]},{"osd":5,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.71199999999999997,"15min":0.68999999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.73299999999999998,"15min":0.67700000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":5.1459999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":0.97599999999999998}]},{"osd":6,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.69399999999999995,"15min":0.65900000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0669999999999999,"5min":1.323,"15min":1.323},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.70599999999999996,"15min":0.68200000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.02,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.63700000000000001}]},{"osd":7,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.79700000000000004,"15min":0.78800000000000003},"min":{"1min":0.55200000000000005,"5min":0.504,"15min":0.504},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.85999999999999999,"15min":0.84999999999999998},"min":{"1min":0.627,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.359,"5min":1.369,"15min":1.369},"last":0.79600000000000004}]},{"osd":8,"last update":"Thu Nov 16 16:46:23 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.94199999999999995}]},{"osd":9,"last update":"Thu Nov 16 16:46:48 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.83199999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.304,"5min":1.304,"15min":1.304},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":1.0069999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92800000000000005}]}]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6520,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738632,"statfs":{"total":95995035648,"available":95988359168,"internally_reserved":0,"allocated":966656,"data_stored":483162,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.48899999999999999,"15min":0.46600000000000003},"min":{"1min":0.439,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.51100000000000001,"15min":0.48599999999999999},"min":{"1min":0.47399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.63500000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.34200000000000003,"15min":0.318},"min":{"1min":0.33800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.35899999999999999,"15min":0.34100000000000003},"min":{"1min":0.34999999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.34100000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.52800000000000002,"15min":0.504},"min":{"1min":0.41099999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.53000000000000003,"15min":0.5},"min":{"1min":0.48899999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.48399999999999999}]},{"osd":3,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.54100000000000004,"15min":0.51900000000000002},"min":{"1min":0.52000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.629},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.54400000000000004,"15min":0.52200000000000002},"min":{"1min":0.39300000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:47:16 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.56399999999999995,"15min":0.53600000000000003},"min":{"1min":0.42399999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78700000000000003,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.55800000000000005,"15min":0.52400000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.61299999999999999,"15min":0.58199999999999996},"min":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.65700000000000003,"15min":0.61899999999999999},"min":{"1min":0.45100000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.52900000000000003,"15min":0.51700000000000002},"min":{"1min":0.42199999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.58199999999999996,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.43099999999999999}]},{"osd":8,"last update":"Thu Nov 16 16:47:24 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.71599999999999997},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.7650000000000001,"5min":2.7650000000000001,"15min":2.7650000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.751,"15min":0.75},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.0219999999999998,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.57399999999999995}]},{"osd":9,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.79600000000000004,"15min":0.80500000000000005},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.77900000000000003,"15min":0.78100000000000003},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.72399999999999998}]},{"osd":10,"last update":"Thu Nov 16 16:47:11 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.71199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6708,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738444,"statfs":{"total":95995035648,"available":95988166656,"internally_reserved":0,"allocated":1028096,"data_stored":1077786,"data_compressed":53424,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52100000000000002,"15min":0.51500000000000001},"min":{"1min":0.34000000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56200000000000006,"15min":0.56599999999999995},"min":{"1min":0.37,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.67100000000000004}]},{"osd":1,"last update":"Thu Nov 16 16:47:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.55300000000000005,"15min":0.54100000000000004},"min":{"1min":0.34399999999999997,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.034,"5min":1.167,"15min":1.167},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.52700000000000002,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Nov 16 16:47:18 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.376,"15min":0.36799999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.495},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.40100000000000002,"15min":0.38600000000000001},"min":{"1min":0.27500000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.46300000000000002}]},{"osd":3,"last update":"Thu Nov 16 16:47:18 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.54200000000000004,"15min":0.53100000000000003},"min":{"1min":0.34499999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58399999999999996,"15min":0.56699999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.77700000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:47:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.40000000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.53600000000000003,"15min":0.52000000000000002},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.55900000000000005}]},{"osd":6,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.63200000000000001,"15min":0.63},"min":{"1min":0.436,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91200000000000003,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Nov 16 16:47:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.60999999999999999,"15min":0.60399999999999998},"min":{"1min":0.44700000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.63300000000000001}]},{"osd":8,"last update":"Thu Nov 16 16:47:25 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.68500000000000005}]},{"osd":9,"last update":"Thu Nov 16 16:47:50 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.748,"15min":0.76100000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.86799999999999999,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.749,"15min":0.75900000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.871,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.79600000000000004}]},{"osd":10,"last update":"Thu Nov 16 16:47:08 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.751}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":21175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T16:47:54.135 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T16:47:54.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:53 smithi176 ceph-mon[131630]: from='client.24820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:53 smithi176 ceph-mon[131630]: pgmap v294: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 103 B/s wr, 1 op/s; 2 B/s, 0 objects/s recovering 2023-11-16T16:47:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:53 smithi176 ceph-mon[131630]: mgrmap e17: x(active, since 7m), standbys: z, y 2023-11-16T16:47:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:53 smithi176 ceph-mon[131630]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:53 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/4028242290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:53 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/4020113591' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:54.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.368+0000 7fa31d3ff700 1 -- 172.21.15.44:0/2055543294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa318101e10 msgr2=0x7fa3181086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.368+0000 7fa31d3ff700 1 --2- 172.21.15.44:0/2055543294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa318101e10 0x7fa3181086d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa31000b0a0 tx=0x7fa31000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.368+0000 7fa31d3ff700 1 -- 172.21.15.44:0/2055543294 shutdown_connections 2023-11-16T16:47:54.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.368+0000 7fa31d3ff700 1 --2- 172.21.15.44:0/2055543294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa318101e10 0x7fa3181086d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.368+0000 7fa31d3ff700 1 --2- 172.21.15.44:0/2055543294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa318100290 0x7fa3181006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.368+0000 7fa31d3ff700 1 --2- 172.21.15.44:0/2055543294 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa318101480 0x7fa3181018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.368+0000 7fa31d3ff700 1 -- 172.21.15.44:0/2055543294 >> 172.21.15.44:0/2055543294 conn(0x7fa3180fb870 msgr2=0x7fa3180fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:54.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.368+0000 7fa31d3ff700 1 -- 172.21.15.44:0/2055543294 shutdown_connections 2023-11-16T16:47:54.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.368+0000 7fa31d3ff700 1 -- 172.21.15.44:0/2055543294 wait complete. 2023-11-16T16:47:54.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.369+0000 7fa31d3ff700 1 Processor -- start 2023-11-16T16:47:54.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.369+0000 7fa31d3ff700 1 -- start start 2023-11-16T16:47:54.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.370+0000 7fa31d3ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa318100290 0x7fa3181982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:54.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.370+0000 7fa31d3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa318101480 0x7fa3181987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:54.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.370+0000 7fa31d3ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa318101e10 0x7fa31819f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:54.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.370+0000 7fa31d3ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31819fe20 con 0x7fa318101e10 2023-11-16T16:47:54.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.370+0000 7fa31d3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31819ff60 con 0x7fa318101480 2023-11-16T16:47:54.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.370+0000 7fa31d3ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3181a00f0 con 0x7fa318100290 2023-11-16T16:47:54.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.370+0000 7fa3177fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa318101e10 0x7fa31819f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:54.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.370+0000 7fa3177fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa318101e10 0x7fa31819f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52902/0 (socket says 172.21.15.44:52902) 2023-11-16T16:47:54.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.370+0000 7fa3177fe700 1 -- 172.21.15.44:0/2990707478 learned_addr learned my addr 172.21.15.44:0/2990707478 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:54.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.370+0000 7fa316ffd700 1 --2- 172.21.15.44:0/2990707478 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa318100290 0x7fa3181982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:54.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.370+0000 7fa30ffff700 1 --2- 172.21.15.44:0/2990707478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa318101480 0x7fa3181987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:54.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.370+0000 7fa3177fe700 1 -- 172.21.15.44:0/2990707478 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa318100290 msgr2=0x7fa3181982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.370+0000 7fa3177fe700 1 --2- 172.21.15.44:0/2990707478 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa318100290 0x7fa3181982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.370+0000 7fa3177fe700 1 -- 172.21.15.44:0/2990707478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa318101480 msgr2=0x7fa3181987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.370+0000 7fa3177fe700 1 --2- 172.21.15.44:0/2990707478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa318101480 0x7fa3181987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.370+0000 7fa3177fe700 1 -- 172.21.15.44:0/2990707478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa310009d00 con 0x7fa318101e10 2023-11-16T16:47:54.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.371+0000 7fa3177fe700 1 --2- 172.21.15.44:0/2990707478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa318101e10 0x7fa31819f850 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa31000d010 tx=0x7fa31000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:54.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.371+0000 7fa314ff9700 1 -- 172.21.15.44:0/2990707478 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31000e040 con 0x7fa318101e10 2023-11-16T16:47:54.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.371+0000 7fa314ff9700 1 -- 172.21.15.44:0/2990707478 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa310003e10 con 0x7fa318101e10 2023-11-16T16:47:54.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.371+0000 7fa31d3ff700 1 -- 172.21.15.44:0/2990707478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31810bc70 con 0x7fa318101e10 2023-11-16T16:47:54.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.371+0000 7fa314ff9700 1 -- 172.21.15.44:0/2990707478 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3100052a0 con 0x7fa318101e10 2023-11-16T16:47:54.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.371+0000 7fa31d3ff700 1 -- 172.21.15.44:0/2990707478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31810bae0 con 0x7fa318101e10 2023-11-16T16:47:54.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.372+0000 7fa31d3ff700 1 -- 172.21.15.44:0/2990707478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa318067a60 con 0x7fa318101e10 2023-11-16T16:47:54.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.374+0000 7fa314ff9700 1 -- 172.21.15.44:0/2990707478 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa31001d480 con 0x7fa318101e10 2023-11-16T16:47:54.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.375+0000 7fa314ff9700 1 --2- 172.21.15.44:0/2990707478 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa3040a1260 0x7fa3040a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:54.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.376+0000 7fa314ff9700 1 -- 172.21.15.44:0/2990707478 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa3100cdec0 con 0x7fa318101e10 2023-11-16T16:47:54.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.376+0000 7fa316ffd700 1 --2- 172.21.15.44:0/2990707478 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa3040a1260 0x7fa3040a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:54.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.376+0000 7fa316ffd700 1 --2- 172.21.15.44:0/2990707478 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa3040a1260 0x7fa3040a3710 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa3181012e0 tx=0x7fa308009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:54.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.380+0000 7fa314ff9700 1 -- 172.21.15.44:0/2990707478 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa3100971d0 con 0x7fa318101e10 2023-11-16T16:47:54.585 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.584+0000 7fa31d3ff700 1 -- 172.21.15.44:0/2990707478 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa318108430 con 0x7fa3040a1260 2023-11-16T16:47:54.598 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.598+0000 7fa314ff9700 1 -- 172.21.15.44:0/2990707478 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422548 (secure 0 0 0) 0x7fa318108430 con 0x7fa3040a1260 2023-11-16T16:47:54.604 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:47:54.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.603+0000 7fa31d3ff700 1 -- 172.21.15.44:0/2990707478 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa3040a1260 msgr2=0x7fa3040a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.603+0000 7fa31d3ff700 1 --2- 172.21.15.44:0/2990707478 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa3040a1260 0x7fa3040a3710 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa3181012e0 tx=0x7fa308009260 comp rx=0 tx=0).stop 2023-11-16T16:47:54.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.603+0000 7fa31d3ff700 1 -- 172.21.15.44:0/2990707478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa318101e10 msgr2=0x7fa31819f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.603+0000 7fa31d3ff700 1 --2- 172.21.15.44:0/2990707478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa318101e10 0x7fa31819f850 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa31000d010 tx=0x7fa31000bc80 comp rx=0 tx=0).stop 2023-11-16T16:47:54.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.603+0000 7fa31d3ff700 1 -- 172.21.15.44:0/2990707478 shutdown_connections 2023-11-16T16:47:54.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.603+0000 7fa31d3ff700 1 --2- 172.21.15.44:0/2990707478 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa3040a1260 0x7fa3040a3710 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.603+0000 7fa31d3ff700 1 --2- 172.21.15.44:0/2990707478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa318101e10 0x7fa31819f850 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.603+0000 7fa31d3ff700 1 --2- 172.21.15.44:0/2990707478 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa318100290 0x7fa3181982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.603+0000 7fa31d3ff700 1 --2- 172.21.15.44:0/2990707478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa318101480 0x7fa3181987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.603+0000 7fa31d3ff700 1 -- 172.21.15.44:0/2990707478 >> 172.21.15.44:0/2990707478 conn(0x7fa3180fb870 msgr2=0x7fa318106d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:54.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.603+0000 7fa31d3ff700 1 -- 172.21.15.44:0/2990707478 shutdown_connections 2023-11-16T16:47:54.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:54.603+0000 7fa31d3ff700 1 -- 172.21.15.44:0/2990707478 wait complete. 2023-11-16T16:47:54.609 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2023-11-16T16:47:54.623 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2023-11-16T16:47:54.540046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594960,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":146,"num_read_kb":131,"num_write":230,"num_write_kb":2957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180,"ondisk_log_size":180,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80008,"kb_used_data":11560,"kb_used_omap":0,"kb_used_meta":68352,"kb_avail":1124861816,"statfs":{"total":1151940427776,"available":1151858499584,"internally_reserved":0,"allocated":11837440,"data_stored":7625200,"data_compressed":318969,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":0,"internal_metadata":69992448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":12,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.919777"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625866+0000","last_change":"2023-11-16T16:47:18.183024+0000","last_active":"2023-11-16T16:47:43.625866+0000","last_peered":"2023-11-16T16:47:43.625866+0000","last_clean":"2023-11-16T16:47:43.625866+0000","last_became_active":"2023-11-16T16:47:18.182387+0000","last_became_peered":"2023-11-16T16:47:18.182387+0000","last_unstale":"2023-11-16T16:47:43.625866+0000","last_undegraded":"2023-11-16T16:47:43.625866+0000","last_fullsized":"2023-11-16T16:47:43.625866+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:33:54.818555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626848+0000","last_change":"2023-11-16T16:47:18.200135+0000","last_active":"2023-11-16T16:47:43.626848+0000","last_peered":"2023-11-16T16:47:43.626848+0000","last_clean":"2023-11-16T16:47:43.626848+0000","last_became_active":"2023-11-16T16:47:18.194732+0000","last_became_peered":"2023-11-16T16:47:18.194732+0000","last_unstale":"2023-11-16T16:47:43.626848+0000","last_undegraded":"2023-11-16T16:47:43.626848+0000","last_fullsized":"2023-11-16T16:47:43.626848+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:35:44.790080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626424+0000","last_change":"2023-11-16T16:47:18.184127+0000","last_active":"2023-11-16T16:47:43.626424+0000","last_peered":"2023-11-16T16:47:43.626424+0000","last_clean":"2023-11-16T16:47:43.626424+0000","last_became_active":"2023-11-16T16:47:18.183163+0000","last_became_peered":"2023-11-16T16:47:18.183163+0000","last_unstale":"2023-11-16T16:47:43.626424+0000","last_undegraded":"2023-11-16T16:47:43.626424+0000","last_fullsized":"2023-11-16T16:47:43.626424+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:22:40.700926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629755+0000","last_change":"2023-11-16T16:47:18.177476+0000","last_active":"2023-11-16T16:47:43.629755+0000","last_peered":"2023-11-16T16:47:43.629755+0000","last_clean":"2023-11-16T16:47:43.629755+0000","last_became_active":"2023-11-16T16:47:18.177178+0000","last_became_peered":"2023-11-16T16:47:18.177178+0000","last_unstale":"2023-11-16T16:47:43.629755+0000","last_undegraded":"2023-11-16T16:47:43.629755+0000","last_fullsized":"2023-11-16T16:47:43.629755+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:40:05.452664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626685+0000","last_change":"2023-11-16T16:47:18.178511+0000","last_active":"2023-11-16T16:47:43.626685+0000","last_peered":"2023-11-16T16:47:43.626685+0000","last_clean":"2023-11-16T16:47:43.626685+0000","last_became_active":"2023-11-16T16:47:18.178198+0000","last_became_peered":"2023-11-16T16:47:18.178198+0000","last_unstale":"2023-11-16T16:47:43.626685+0000","last_undegraded":"2023-11-16T16:47:43.626685+0000","last_fullsized":"2023-11-16T16:47:43.626685+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:53:35.196107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626326+0000","last_change":"2023-11-16T16:47:18.186552+0000","last_active":"2023-11-16T16:47:43.626326+0000","last_peered":"2023-11-16T16:47:43.626326+0000","last_clean":"2023-11-16T16:47:43.626326+0000","last_became_active":"2023-11-16T16:47:18.186238+0000","last_became_peered":"2023-11-16T16:47:18.186238+0000","last_unstale":"2023-11-16T16:47:43.626326+0000","last_undegraded":"2023-11-16T16:47:43.626326+0000","last_fullsized":"2023-11-16T16:47:43.626326+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:36:52.844919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622387+0000","last_change":"2023-11-16T16:47:18.179839+0000","last_active":"2023-11-16T16:47:43.622387+0000","last_peered":"2023-11-16T16:47:43.622387+0000","last_clean":"2023-11-16T16:47:43.622387+0000","last_became_active":"2023-11-16T16:47:18.179060+0000","last_became_peered":"2023-11-16T16:47:18.179060+0000","last_unstale":"2023-11-16T16:47:43.622387+0000","last_undegraded":"2023-11-16T16:47:43.622387+0000","last_fullsized":"2023-11-16T16:47:43.622387+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:55:30.972787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.633015+0000","last_change":"2023-11-16T16:47:43.631738+0000","last_active":"2023-11-16T16:47:43.633015+0000","last_peered":"2023-11-16T16:47:43.633015+0000","last_clean":"2023-11-16T16:47:43.633015+0000","last_became_active":"2023-11-16T16:47:43.631146+0000","last_became_peered":"2023-11-16T16:47:43.631146+0000","last_unstale":"2023-11-16T16:47:43.633015+0000","last_undegraded":"2023-11-16T16:47:43.633015+0000","last_fullsized":"2023-11-16T16:47:43.633015+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:32:26.552018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947385+0000","last_change":"2023-11-16T16:47:18.177834+0000","last_active":"2023-11-16T16:47:43.947385+0000","last_peered":"2023-11-16T16:47:43.947385+0000","last_clean":"2023-11-16T16:47:43.947385+0000","last_became_active":"2023-11-16T16:47:18.176299+0000","last_became_peered":"2023-11-16T16:47:18.176299+0000","last_unstale":"2023-11-16T16:47:43.947385+0000","last_undegraded":"2023-11-16T16:47:43.947385+0000","last_fullsized":"2023-11-16T16:47:43.947385+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:18:34.674950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626777+0000","last_change":"2023-11-16T16:47:18.179394+0000","last_active":"2023-11-16T16:47:43.626777+0000","last_peered":"2023-11-16T16:47:43.626777+0000","last_clean":"2023-11-16T16:47:43.626777+0000","last_became_active":"2023-11-16T16:47:18.179119+0000","last_became_peered":"2023-11-16T16:47:18.179119+0000","last_unstale":"2023-11-16T16:47:43.626777+0000","last_undegraded":"2023-11-16T16:47:43.626777+0000","last_fullsized":"2023-11-16T16:47:43.626777+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:32:22.770618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623315+0000","last_change":"2023-11-16T16:47:18.183585+0000","last_active":"2023-11-16T16:47:43.623315+0000","last_peered":"2023-11-16T16:47:43.623315+0000","last_clean":"2023-11-16T16:47:43.623315+0000","last_became_active":"2023-11-16T16:47:18.183209+0000","last_became_peered":"2023-11-16T16:47:18.183209+0000","last_unstale":"2023-11-16T16:47:43.623315+0000","last_undegraded":"2023-11-16T16:47:43.623315+0000","last_fullsized":"2023-11-16T16:47:43.623315+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:22:47.894810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629596+0000","last_change":"2023-11-16T16:47:18.186503+0000","last_active":"2023-11-16T16:47:43.629596+0000","last_peered":"2023-11-16T16:47:43.629596+0000","last_clean":"2023-11-16T16:47:43.629596+0000","last_became_active":"2023-11-16T16:47:18.186193+0000","last_became_peered":"2023-11-16T16:47:18.186193+0000","last_unstale":"2023-11-16T16:47:43.629596+0000","last_undegraded":"2023-11-16T16:47:43.629596+0000","last_fullsized":"2023-11-16T16:47:43.629596+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:59:03.486858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629884+0000","last_change":"2023-11-16T16:47:18.193842+0000","last_active":"2023-11-16T16:47:43.629884+0000","last_peered":"2023-11-16T16:47:43.629884+0000","last_clean":"2023-11-16T16:47:43.629884+0000","last_became_active":"2023-11-16T16:47:18.192309+0000","last_became_peered":"2023-11-16T16:47:18.192309+0000","last_unstale":"2023-11-16T16:47:43.629884+0000","last_undegraded":"2023-11-16T16:47:43.629884+0000","last_fullsized":"2023-11-16T16:47:43.629884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:53:33.884565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625812+0000","last_change":"2023-11-16T16:47:18.192359+0000","last_active":"2023-11-16T16:47:43.625812+0000","last_peered":"2023-11-16T16:47:43.625812+0000","last_clean":"2023-11-16T16:47:43.625812+0000","last_became_active":"2023-11-16T16:47:18.189765+0000","last_became_peered":"2023-11-16T16:47:18.189765+0000","last_unstale":"2023-11-16T16:47:43.625812+0000","last_undegraded":"2023-11-16T16:47:43.625812+0000","last_fullsized":"2023-11-16T16:47:43.625812+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:17:15.440725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625875+0000","last_change":"2023-11-16T16:47:18.178386+0000","last_active":"2023-11-16T16:47:43.625875+0000","last_peered":"2023-11-16T16:47:43.625875+0000","last_clean":"2023-11-16T16:47:43.625875+0000","last_became_active":"2023-11-16T16:47:18.178041+0000","last_became_peered":"2023-11-16T16:47:18.178041+0000","last_unstale":"2023-11-16T16:47:43.625875+0000","last_undegraded":"2023-11-16T16:47:43.625875+0000","last_fullsized":"2023-11-16T16:47:43.625875+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:27:47.278353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.946667+0000","last_change":"2023-11-16T16:47:18.174424+0000","last_active":"2023-11-16T16:47:43.946667+0000","last_peered":"2023-11-16T16:47:43.946667+0000","last_clean":"2023-11-16T16:47:43.946667+0000","last_became_active":"2023-11-16T16:47:18.174078+0000","last_became_peered":"2023-11-16T16:47:18.174078+0000","last_unstale":"2023-11-16T16:47:43.946667+0000","last_undegraded":"2023-11-16T16:47:43.946667+0000","last_fullsized":"2023-11-16T16:47:43.946667+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:14:13.491783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625739+0000","last_change":"2023-11-16T16:47:18.183161+0000","last_active":"2023-11-16T16:47:43.625739+0000","last_peered":"2023-11-16T16:47:43.625739+0000","last_clean":"2023-11-16T16:47:43.625739+0000","last_became_active":"2023-11-16T16:47:18.182806+0000","last_became_peered":"2023-11-16T16:47:18.182806+0000","last_unstale":"2023-11-16T16:47:43.625739+0000","last_undegraded":"2023-11-16T16:47:43.625739+0000","last_fullsized":"2023-11-16T16:47:43.625739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:50:04.072542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622713+0000","last_change":"2023-11-16T16:47:18.187225+0000","last_active":"2023-11-16T16:47:43.622713+0000","last_peered":"2023-11-16T16:47:43.622713+0000","last_clean":"2023-11-16T16:47:43.622713+0000","last_became_active":"2023-11-16T16:47:18.186858+0000","last_became_peered":"2023-11-16T16:47:18.186858+0000","last_unstale":"2023-11-16T16:47:43.622713+0000","last_undegraded":"2023-11-16T16:47:43.622713+0000","last_fullsized":"2023-11-16T16:47:43.622713+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:42:28.736883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623323+0000","last_change":"2023-11-16T16:47:18.190947+0000","last_active":"2023-11-16T16:47:43.623323+0000","last_peered":"2023-11-16T16:47:43.623323+0000","last_clean":"2023-11-16T16:47:43.623323+0000","last_became_active":"2023-11-16T16:47:18.190737+0000","last_became_peered":"2023-11-16T16:47:18.190737+0000","last_unstale":"2023-11-16T16:47:43.623323+0000","last_undegraded":"2023-11-16T16:47:43.623323+0000","last_fullsized":"2023-11-16T16:47:43.623323+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:34:59.150388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622328+0000","last_change":"2023-11-16T16:47:22.659730+0000","last_active":"2023-11-16T16:47:43.622328+0000","last_peered":"2023-11-16T16:47:43.622328+0000","last_clean":"2023-11-16T16:47:43.622328+0000","last_became_active":"2023-11-16T16:47:22.659313+0000","last_became_peered":"2023-11-16T16:47:22.659313+0000","last_unstale":"2023-11-16T16:47:43.622328+0000","last_undegraded":"2023-11-16T16:47:43.622328+0000","last_fullsized":"2023-11-16T16:47:43.622328+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:05:05.336507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'4","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622965+0000","last_change":"2023-11-16T16:47:17.629748+0000","last_active":"2023-11-16T16:47:43.622965+0000","last_peered":"2023-11-16T16:47:43.622965+0000","last_clean":"2023-11-16T16:47:43.622965+0000","last_became_active":"2023-11-16T16:47:17.629284+0000","last_became_peered":"2023-11-16T16:47:17.629284+0000","last_unstale":"2023-11-16T16:47:43.622965+0000","last_undegraded":"2023-11-16T16:47:43.622965+0000","last_fullsized":"2023-11-16T16:47:43.622965+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:41:29.574724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970501+0000","last_change":"2023-11-16T16:47:18.176239+0000","last_active":"2023-11-16T16:47:43.970501+0000","last_peered":"2023-11-16T16:47:43.970501+0000","last_clean":"2023-11-16T16:47:43.970501+0000","last_became_active":"2023-11-16T16:47:18.175879+0000","last_became_peered":"2023-11-16T16:47:18.175879+0000","last_unstale":"2023-11-16T16:47:43.970501+0000","last_undegraded":"2023-11-16T16:47:43.970501+0000","last_fullsized":"2023-11-16T16:47:43.970501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:32:09.544453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.628988+0000","last_change":"2023-11-16T16:47:22.659679+0000","last_active":"2023-11-16T16:47:43.628988+0000","last_peered":"2023-11-16T16:47:43.628988+0000","last_clean":"2023-11-16T16:47:43.628988+0000","last_became_active":"2023-11-16T16:47:22.658946+0000","last_became_peered":"2023-11-16T16:47:22.658946+0000","last_unstale":"2023-11-16T16:47:43.628988+0000","last_undegraded":"2023-11-16T16:47:43.628988+0000","last_fullsized":"2023-11-16T16:47:43.628988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:33:44.392111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"82'1","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.971259+0000","last_change":"2023-11-16T16:47:17.643944+0000","last_active":"2023-11-16T16:47:43.971259+0000","last_peered":"2023-11-16T16:47:43.971259+0000","last_clean":"2023-11-16T16:47:43.971259+0000","last_became_active":"2023-11-16T16:47:17.643470+0000","last_became_peered":"2023-11-16T16:47:17.643470+0000","last_unstale":"2023-11-16T16:47:43.971259+0000","last_undegraded":"2023-11-16T16:47:43.971259+0000","last_fullsized":"2023-11-16T16:47:43.971259+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:28:22.870692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.627110+0000","last_change":"2023-11-16T16:47:18.198139+0000","last_active":"2023-11-16T16:47:43.627110+0000","last_peered":"2023-11-16T16:47:43.627110+0000","last_clean":"2023-11-16T16:47:43.627110+0000","last_became_active":"2023-11-16T16:47:18.197762+0000","last_became_peered":"2023-11-16T16:47:18.197762+0000","last_unstale":"2023-11-16T16:47:43.627110+0000","last_undegraded":"2023-11-16T16:47:43.627110+0000","last_fullsized":"2023-11-16T16:47:43.627110+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:27:32.243043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629208+0000","last_change":"2023-11-16T16:47:22.655810+0000","last_active":"2023-11-16T16:47:43.629208+0000","last_peered":"2023-11-16T16:47:43.629208+0000","last_clean":"2023-11-16T16:47:43.629208+0000","last_became_active":"2023-11-16T16:47:22.655388+0000","last_became_peered":"2023-11-16T16:47:22.655388+0000","last_unstale":"2023-11-16T16:47:43.629208+0000","last_undegraded":"2023-11-16T16:47:43.629208+0000","last_fullsized":"2023-11-16T16:47:43.629208+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:05:46.905125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'2","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.635266+0000","last_change":"2023-11-16T16:47:43.635266+0000","last_active":"2023-11-16T16:47:43.635266+0000","last_peered":"2023-11-16T16:47:43.635266+0000","last_clean":"2023-11-16T16:47:43.635266+0000","last_became_active":"2023-11-16T16:47:43.630583+0000","last_became_peered":"2023-11-16T16:47:43.630583+0000","last_unstale":"2023-11-16T16:47:43.635266+0000","last_undegraded":"2023-11-16T16:47:43.635266+0000","last_fullsized":"2023-11-16T16:47:43.635266+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:51:43.792892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.946938+0000","last_change":"2023-11-16T16:47:18.186341+0000","last_active":"2023-11-16T16:47:43.946938+0000","last_peered":"2023-11-16T16:47:43.946938+0000","last_clean":"2023-11-16T16:47:43.946938+0000","last_became_active":"2023-11-16T16:47:18.186066+0000","last_became_peered":"2023-11-16T16:47:18.186066+0000","last_unstale":"2023-11-16T16:47:43.946938+0000","last_undegraded":"2023-11-16T16:47:43.946938+0000","last_fullsized":"2023-11-16T16:47:43.946938+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:22:46.055671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629005+0000","last_change":"2023-11-16T16:47:22.657227+0000","last_active":"2023-11-16T16:47:43.629005+0000","last_peered":"2023-11-16T16:47:43.629005+0000","last_clean":"2023-11-16T16:47:43.629005+0000","last_became_active":"2023-11-16T16:47:22.656835+0000","last_became_peered":"2023-11-16T16:47:22.656835+0000","last_unstale":"2023-11-16T16:47:43.629005+0000","last_undegraded":"2023-11-16T16:47:43.629005+0000","last_fullsized":"2023-11-16T16:47:43.629005+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:07.552894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625950+0000","last_change":"2023-11-16T16:47:17.630194+0000","last_active":"2023-11-16T16:47:43.625950+0000","last_peered":"2023-11-16T16:47:43.625950+0000","last_clean":"2023-11-16T16:47:43.625950+0000","last_became_active":"2023-11-16T16:47:17.629886+0000","last_became_peered":"2023-11-16T16:47:17.629886+0000","last_unstale":"2023-11-16T16:47:43.625950+0000","last_undegraded":"2023-11-16T16:47:43.625950+0000","last_fullsized":"2023-11-16T16:47:43.625950+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:57:09.970289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630557+0000","last_change":"2023-11-16T16:47:18.181596+0000","last_active":"2023-11-16T16:47:43.630557+0000","last_peered":"2023-11-16T16:47:43.630557+0000","last_clean":"2023-11-16T16:47:43.630557+0000","last_became_active":"2023-11-16T16:47:18.181287+0000","last_became_peered":"2023-11-16T16:47:18.181287+0000","last_unstale":"2023-11-16T16:47:43.630557+0000","last_undegraded":"2023-11-16T16:47:43.630557+0000","last_fullsized":"2023-11-16T16:47:43.630557+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:31:55.798497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622763+0000","last_change":"2023-11-16T16:47:22.649197+0000","last_active":"2023-11-16T16:47:43.622763+0000","last_peered":"2023-11-16T16:47:43.622763+0000","last_clean":"2023-11-16T16:47:43.622763+0000","last_became_active":"2023-11-16T16:47:22.648664+0000","last_became_peered":"2023-11-16T16:47:22.648664+0000","last_unstale":"2023-11-16T16:47:43.622763+0000","last_undegraded":"2023-11-16T16:47:43.622763+0000","last_fullsized":"2023-11-16T16:47:43.622763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:56:11.119940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"82'1","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630141+0000","last_change":"2023-11-16T16:47:17.627672+0000","last_active":"2023-11-16T16:47:43.630141+0000","last_peered":"2023-11-16T16:47:43.630141+0000","last_clean":"2023-11-16T16:47:43.630141+0000","last_became_active":"2023-11-16T16:47:17.627383+0000","last_became_peered":"2023-11-16T16:47:17.627383+0000","last_unstale":"2023-11-16T16:47:43.630141+0000","last_undegraded":"2023-11-16T16:47:43.630141+0000","last_fullsized":"2023-11-16T16:47:43.630141+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:35:57.379840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625043+0000","last_change":"2023-11-16T16:47:18.180983+0000","last_active":"2023-11-16T16:47:43.625043+0000","last_peered":"2023-11-16T16:47:43.625043+0000","last_clean":"2023-11-16T16:47:43.625043+0000","last_became_active":"2023-11-16T16:47:18.180702+0000","last_became_peered":"2023-11-16T16:47:18.180702+0000","last_unstale":"2023-11-16T16:47:43.625043+0000","last_undegraded":"2023-11-16T16:47:43.625043+0000","last_fullsized":"2023-11-16T16:47:43.625043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:20:27.895648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622657+0000","last_change":"2023-11-16T16:47:22.659898+0000","last_active":"2023-11-16T16:47:43.622657+0000","last_peered":"2023-11-16T16:47:43.622657+0000","last_clean":"2023-11-16T16:47:43.622657+0000","last_became_active":"2023-11-16T16:47:22.659326+0000","last_became_peered":"2023-11-16T16:47:22.659326+0000","last_unstale":"2023-11-16T16:47:43.622657+0000","last_undegraded":"2023-11-16T16:47:43.622657+0000","last_fullsized":"2023-11-16T16:47:43.622657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:57:36.289645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"82'1","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.949231+0000","last_change":"2023-11-16T16:47:43.949231+0000","last_active":"2023-11-16T16:47:43.949231+0000","last_peered":"2023-11-16T16:47:43.949231+0000","last_clean":"2023-11-16T16:47:43.949231+0000","last_became_active":"2023-11-16T16:47:43.948766+0000","last_became_peered":"2023-11-16T16:47:43.948766+0000","last_unstale":"2023-11-16T16:47:43.949231+0000","last_undegraded":"2023-11-16T16:47:43.949231+0000","last_fullsized":"2023-11-16T16:47:43.949231+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:46:57.086811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630999+0000","last_change":"2023-11-16T16:47:18.185263+0000","last_active":"2023-11-16T16:47:43.630999+0000","last_peered":"2023-11-16T16:47:43.630999+0000","last_clean":"2023-11-16T16:47:43.630999+0000","last_became_active":"2023-11-16T16:47:18.184910+0000","last_became_peered":"2023-11-16T16:47:18.184910+0000","last_unstale":"2023-11-16T16:47:43.630999+0000","last_undegraded":"2023-11-16T16:47:43.630999+0000","last_fullsized":"2023-11-16T16:47:43.630999+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:25:21.463648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625581+0000","last_change":"2023-11-16T16:47:22.659711+0000","last_active":"2023-11-16T16:47:43.625581+0000","last_peered":"2023-11-16T16:47:43.625581+0000","last_clean":"2023-11-16T16:47:43.625581+0000","last_became_active":"2023-11-16T16:47:22.659087+0000","last_became_peered":"2023-11-16T16:47:22.659087+0000","last_unstale":"2023-11-16T16:47:43.625581+0000","last_undegraded":"2023-11-16T16:47:43.625581+0000","last_fullsized":"2023-11-16T16:47:43.625581+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:13:26.339037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629045+0000","last_change":"2023-11-16T16:47:17.629703+0000","last_active":"2023-11-16T16:47:43.629045+0000","last_peered":"2023-11-16T16:47:43.629045+0000","last_clean":"2023-11-16T16:47:43.629045+0000","last_became_active":"2023-11-16T16:47:17.629245+0000","last_became_peered":"2023-11-16T16:47:17.629245+0000","last_unstale":"2023-11-16T16:47:43.629045+0000","last_undegraded":"2023-11-16T16:47:43.629045+0000","last_fullsized":"2023-11-16T16:47:43.629045+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:59:34.790629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970314+0000","last_change":"2023-11-16T16:47:18.187149+0000","last_active":"2023-11-16T16:47:43.970314+0000","last_peered":"2023-11-16T16:47:43.970314+0000","last_clean":"2023-11-16T16:47:43.970314+0000","last_became_active":"2023-11-16T16:47:18.185717+0000","last_became_peered":"2023-11-16T16:47:18.185717+0000","last_unstale":"2023-11-16T16:47:43.970314+0000","last_undegraded":"2023-11-16T16:47:43.970314+0000","last_fullsized":"2023-11-16T16:47:43.970314+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:23:04.064643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626668+0000","last_change":"2023-11-16T16:47:22.663025+0000","last_active":"2023-11-16T16:47:43.626668+0000","last_peered":"2023-11-16T16:47:43.626668+0000","last_clean":"2023-11-16T16:47:43.626668+0000","last_became_active":"2023-11-16T16:47:22.662596+0000","last_became_peered":"2023-11-16T16:47:22.662596+0000","last_unstale":"2023-11-16T16:47:43.626668+0000","last_undegraded":"2023-11-16T16:47:43.626668+0000","last_fullsized":"2023-11-16T16:47:43.626668+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:47:26.815530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629668+0000","last_change":"2023-11-16T16:47:17.626357+0000","last_active":"2023-11-16T16:47:43.629668+0000","last_peered":"2023-11-16T16:47:43.629668+0000","last_clean":"2023-11-16T16:47:43.629668+0000","last_became_active":"2023-11-16T16:47:17.626029+0000","last_became_peered":"2023-11-16T16:47:17.626029+0000","last_unstale":"2023-11-16T16:47:43.629668+0000","last_undegraded":"2023-11-16T16:47:43.629668+0000","last_fullsized":"2023-11-16T16:47:43.629668+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:55:28.659340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.621841+0000","last_change":"2023-11-16T16:47:18.177153+0000","last_active":"2023-11-16T16:47:43.621841+0000","last_peered":"2023-11-16T16:47:43.621841+0000","last_clean":"2023-11-16T16:47:43.621841+0000","last_became_active":"2023-11-16T16:47:18.176751+0000","last_became_peered":"2023-11-16T16:47:18.176751+0000","last_unstale":"2023-11-16T16:47:43.621841+0000","last_undegraded":"2023-11-16T16:47:43.621841+0000","last_fullsized":"2023-11-16T16:47:43.621841+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:17:58.058194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970584+0000","last_change":"2023-11-16T16:47:22.652718+0000","last_active":"2023-11-16T16:47:43.970584+0000","last_peered":"2023-11-16T16:47:43.970584+0000","last_clean":"2023-11-16T16:47:43.970584+0000","last_became_active":"2023-11-16T16:47:22.652278+0000","last_became_peered":"2023-11-16T16:47:22.652278+0000","last_unstale":"2023-11-16T16:47:43.970584+0000","last_undegraded":"2023-11-16T16:47:43.970584+0000","last_fullsized":"2023-11-16T16:47:43.970584+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:43:41.546583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626473+0000","last_change":"2023-11-16T16:47:17.627308+0000","last_active":"2023-11-16T16:47:43.626473+0000","last_peered":"2023-11-16T16:47:43.626473+0000","last_clean":"2023-11-16T16:47:43.626473+0000","last_became_active":"2023-11-16T16:47:17.626992+0000","last_became_peered":"2023-11-16T16:47:17.626992+0000","last_unstale":"2023-11-16T16:47:43.626473+0000","last_undegraded":"2023-11-16T16:47:43.626473+0000","last_fullsized":"2023-11-16T16:47:43.626473+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:41:51.914757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947323+0000","last_change":"2023-11-16T16:47:18.188015+0000","last_active":"2023-11-16T16:47:43.947323+0000","last_peered":"2023-11-16T16:47:43.947323+0000","last_clean":"2023-11-16T16:47:43.947323+0000","last_became_active":"2023-11-16T16:47:18.187730+0000","last_became_peered":"2023-11-16T16:47:18.187730+0000","last_unstale":"2023-11-16T16:47:43.947323+0000","last_undegraded":"2023-11-16T16:47:43.947323+0000","last_fullsized":"2023-11-16T16:47:43.947323+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:39:02.651682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.621932+0000","last_change":"2023-11-16T16:47:22.655656+0000","last_active":"2023-11-16T16:47:43.621932+0000","last_peered":"2023-11-16T16:47:43.621932+0000","last_clean":"2023-11-16T16:47:43.621932+0000","last_became_active":"2023-11-16T16:47:22.655133+0000","last_became_peered":"2023-11-16T16:47:22.655133+0000","last_unstale":"2023-11-16T16:47:43.621932+0000","last_undegraded":"2023-11-16T16:47:43.621932+0000","last_fullsized":"2023-11-16T16:47:43.621932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:19:21.932819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947947+0000","last_change":"2023-11-16T16:47:43.947947+0000","last_active":"2023-11-16T16:47:43.947947+0000","last_peered":"2023-11-16T16:47:43.947947+0000","last_clean":"2023-11-16T16:47:43.947947+0000","last_became_active":"2023-11-16T16:47:43.947478+0000","last_became_peered":"2023-11-16T16:47:43.947478+0000","last_unstale":"2023-11-16T16:47:43.947947+0000","last_undegraded":"2023-11-16T16:47:43.947947+0000","last_fullsized":"2023-11-16T16:47:43.947947+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:40:14.254873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.632657+0000","last_change":"2023-11-16T16:47:43.632657+0000","last_active":"2023-11-16T16:47:43.632657+0000","last_peered":"2023-11-16T16:47:43.632657+0000","last_clean":"2023-11-16T16:47:43.632657+0000","last_became_active":"2023-11-16T16:47:43.631573+0000","last_became_peered":"2023-11-16T16:47:43.631573+0000","last_unstale":"2023-11-16T16:47:43.632657+0000","last_undegraded":"2023-11-16T16:47:43.632657+0000","last_fullsized":"2023-11-16T16:47:43.632657+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:45:08.317787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629418+0000","last_change":"2023-11-16T16:47:22.661639+0000","last_active":"2023-11-16T16:47:43.629418+0000","last_peered":"2023-11-16T16:47:43.629418+0000","last_clean":"2023-11-16T16:47:43.629418+0000","last_became_active":"2023-11-16T16:47:22.661230+0000","last_became_peered":"2023-11-16T16:47:22.661230+0000","last_unstale":"2023-11-16T16:47:43.629418+0000","last_undegraded":"2023-11-16T16:47:43.629418+0000","last_fullsized":"2023-11-16T16:47:43.629418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:04:16.909613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"82'3","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630143+0000","last_change":"2023-11-16T16:47:17.638209+0000","last_active":"2023-11-16T16:47:43.630143+0000","last_peered":"2023-11-16T16:47:43.630143+0000","last_clean":"2023-11-16T16:47:43.630143+0000","last_became_active":"2023-11-16T16:47:17.637957+0000","last_became_peered":"2023-11-16T16:47:17.637957+0000","last_unstale":"2023-11-16T16:47:43.630143+0000","last_undegraded":"2023-11-16T16:47:43.630143+0000","last_fullsized":"2023-11-16T16:47:43.630143+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:08:43.599916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947142+0000","last_change":"2023-11-16T16:47:18.182693+0000","last_active":"2023-11-16T16:47:43.947142+0000","last_peered":"2023-11-16T16:47:43.947142+0000","last_clean":"2023-11-16T16:47:43.947142+0000","last_became_active":"2023-11-16T16:47:18.182370+0000","last_became_peered":"2023-11-16T16:47:18.182370+0000","last_unstale":"2023-11-16T16:47:43.947142+0000","last_undegraded":"2023-11-16T16:47:43.947142+0000","last_fullsized":"2023-11-16T16:47:43.947142+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:38:49.650460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.621838+0000","last_change":"2023-11-16T16:47:22.649348+0000","last_active":"2023-11-16T16:47:43.621838+0000","last_peered":"2023-11-16T16:47:43.621838+0000","last_clean":"2023-11-16T16:47:43.621838+0000","last_became_active":"2023-11-16T16:47:22.648860+0000","last_became_peered":"2023-11-16T16:47:22.648860+0000","last_unstale":"2023-11-16T16:47:43.621838+0000","last_undegraded":"2023-11-16T16:47:43.621838+0000","last_fullsized":"2023-11-16T16:47:43.621838+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:45:18.960009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'10","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:49.162885+0000","last_change":"2023-11-16T16:47:17.619417+0000","last_active":"2023-11-16T16:47:49.162885+0000","last_peered":"2023-11-16T16:47:49.162885+0000","last_clean":"2023-11-16T16:47:49.162885+0000","last_became_active":"2023-11-16T16:47:17.619086+0000","last_became_peered":"2023-11-16T16:47:17.619086+0000","last_unstale":"2023-11-16T16:47:49.162885+0000","last_undegraded":"2023-11-16T16:47:49.162885+0000","last_fullsized":"2023-11-16T16:47:49.162885+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:27:30.044686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625947+0000","last_change":"2023-11-16T16:47:18.177537+0000","last_active":"2023-11-16T16:47:43.625947+0000","last_peered":"2023-11-16T16:47:43.625947+0000","last_clean":"2023-11-16T16:47:43.625947+0000","last_became_active":"2023-11-16T16:47:18.177112+0000","last_became_peered":"2023-11-16T16:47:18.177112+0000","last_unstale":"2023-11-16T16:47:43.625947+0000","last_undegraded":"2023-11-16T16:47:43.625947+0000","last_fullsized":"2023-11-16T16:47:43.625947+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:17:18.541523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630846+0000","last_change":"2023-11-16T16:47:22.658086+0000","last_active":"2023-11-16T16:47:43.630846+0000","last_peered":"2023-11-16T16:47:43.630846+0000","last_clean":"2023-11-16T16:47:43.630846+0000","last_became_active":"2023-11-16T16:47:22.657725+0000","last_became_peered":"2023-11-16T16:47:22.657725+0000","last_unstale":"2023-11-16T16:47:43.630846+0000","last_undegraded":"2023-11-16T16:47:43.630846+0000","last_fullsized":"2023-11-16T16:47:43.630846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:53:51.729635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"82'2","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622837+0000","last_change":"2023-11-16T16:47:17.625405+0000","last_active":"2023-11-16T16:47:43.622837+0000","last_peered":"2023-11-16T16:47:43.622837+0000","last_clean":"2023-11-16T16:47:43.622837+0000","last_became_active":"2023-11-16T16:47:17.625026+0000","last_became_peered":"2023-11-16T16:47:17.625026+0000","last_unstale":"2023-11-16T16:47:43.622837+0000","last_undegraded":"2023-11-16T16:47:43.622837+0000","last_fullsized":"2023-11-16T16:47:43.622837+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:45:14.505117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629372+0000","last_change":"2023-11-16T16:47:18.196345+0000","last_active":"2023-11-16T16:47:43.629372+0000","last_peered":"2023-11-16T16:47:43.629372+0000","last_clean":"2023-11-16T16:47:43.629372+0000","last_became_active":"2023-11-16T16:47:18.196040+0000","last_became_peered":"2023-11-16T16:47:18.196040+0000","last_unstale":"2023-11-16T16:47:43.629372+0000","last_undegraded":"2023-11-16T16:47:43.629372+0000","last_fullsized":"2023-11-16T16:47:43.629372+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:07:42.452837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.624832+0000","last_change":"2023-11-16T16:47:22.660955+0000","last_active":"2023-11-16T16:47:43.624832+0000","last_peered":"2023-11-16T16:47:43.624832+0000","last_clean":"2023-11-16T16:47:43.624832+0000","last_became_active":"2023-11-16T16:47:22.660574+0000","last_became_peered":"2023-11-16T16:47:22.660574+0000","last_unstale":"2023-11-16T16:47:43.624832+0000","last_undegraded":"2023-11-16T16:47:43.624832+0000","last_fullsized":"2023-11-16T16:47:43.624832+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:41:43.745744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630317+0000","last_change":"2023-11-16T16:47:17.630481+0000","last_active":"2023-11-16T16:47:43.630317+0000","last_peered":"2023-11-16T16:47:43.630317+0000","last_clean":"2023-11-16T16:47:43.630317+0000","last_became_active":"2023-11-16T16:47:17.630190+0000","last_became_peered":"2023-11-16T16:47:17.630190+0000","last_unstale":"2023-11-16T16:47:43.630317+0000","last_undegraded":"2023-11-16T16:47:43.630317+0000","last_fullsized":"2023-11-16T16:47:43.630317+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:03:46.177268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622893+0000","last_change":"2023-11-16T16:47:18.179833+0000","last_active":"2023-11-16T16:47:43.622893+0000","last_peered":"2023-11-16T16:47:43.622893+0000","last_clean":"2023-11-16T16:47:43.622893+0000","last_became_active":"2023-11-16T16:47:18.179498+0000","last_became_peered":"2023-11-16T16:47:18.179498+0000","last_unstale":"2023-11-16T16:47:43.622893+0000","last_undegraded":"2023-11-16T16:47:43.622893+0000","last_fullsized":"2023-11-16T16:47:43.622893+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:29:41.782619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622702+0000","last_change":"2023-11-16T16:47:22.656312+0000","last_active":"2023-11-16T16:47:43.622702+0000","last_peered":"2023-11-16T16:47:43.622702+0000","last_clean":"2023-11-16T16:47:43.622702+0000","last_became_active":"2023-11-16T16:47:22.655870+0000","last_became_peered":"2023-11-16T16:47:22.655870+0000","last_unstale":"2023-11-16T16:47:43.622702+0000","last_undegraded":"2023-11-16T16:47:43.622702+0000","last_fullsized":"2023-11-16T16:47:43.622702+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:42:39.568364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629185+0000","last_change":"2023-11-16T16:47:17.622648+0000","last_active":"2023-11-16T16:47:43.629185+0000","last_peered":"2023-11-16T16:47:43.629185+0000","last_clean":"2023-11-16T16:47:43.629185+0000","last_became_active":"2023-11-16T16:47:17.622329+0000","last_became_peered":"2023-11-16T16:47:17.622329+0000","last_unstale":"2023-11-16T16:47:43.629185+0000","last_undegraded":"2023-11-16T16:47:43.629185+0000","last_fullsized":"2023-11-16T16:47:43.629185+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:01:26.211530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622424+0000","last_change":"2023-11-16T16:47:18.179712+0000","last_active":"2023-11-16T16:47:43.622424+0000","last_peered":"2023-11-16T16:47:43.622424+0000","last_clean":"2023-11-16T16:47:43.622424+0000","last_became_active":"2023-11-16T16:47:18.179392+0000","last_became_peered":"2023-11-16T16:47:18.179392+0000","last_unstale":"2023-11-16T16:47:43.622424+0000","last_undegraded":"2023-11-16T16:47:43.622424+0000","last_fullsized":"2023-11-16T16:47:43.622424+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:56:44.447970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.631211+0000","last_change":"2023-11-16T16:47:22.665088+0000","last_active":"2023-11-16T16:47:43.631211+0000","last_peered":"2023-11-16T16:47:43.631211+0000","last_clean":"2023-11-16T16:47:43.631211+0000","last_became_active":"2023-11-16T16:47:22.662802+0000","last_became_peered":"2023-11-16T16:47:22.662802+0000","last_unstale":"2023-11-16T16:47:43.631211+0000","last_undegraded":"2023-11-16T16:47:43.631211+0000","last_fullsized":"2023-11-16T16:47:43.631211+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:51:55.859193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"82'8","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.624624+0000","last_change":"2023-11-16T16:47:17.639246+0000","last_active":"2023-11-16T16:47:43.624624+0000","last_peered":"2023-11-16T16:47:43.624624+0000","last_clean":"2023-11-16T16:47:43.624624+0000","last_became_active":"2023-11-16T16:47:17.638928+0000","last_became_peered":"2023-11-16T16:47:17.638928+0000","last_unstale":"2023-11-16T16:47:43.624624+0000","last_undegraded":"2023-11-16T16:47:43.624624+0000","last_fullsized":"2023-11-16T16:47:43.624624+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:18:41.175512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629991+0000","last_change":"2023-11-16T16:47:18.182609+0000","last_active":"2023-11-16T16:47:43.629991+0000","last_peered":"2023-11-16T16:47:43.629991+0000","last_clean":"2023-11-16T16:47:43.629991+0000","last_became_active":"2023-11-16T16:47:18.182260+0000","last_became_peered":"2023-11-16T16:47:18.182260+0000","last_unstale":"2023-11-16T16:47:43.629991+0000","last_undegraded":"2023-11-16T16:47:43.629991+0000","last_fullsized":"2023-11-16T16:47:43.629991+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:14:14.972367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629794+0000","last_change":"2023-11-16T16:47:22.647586+0000","last_active":"2023-11-16T16:47:43.629794+0000","last_peered":"2023-11-16T16:47:43.629794+0000","last_clean":"2023-11-16T16:47:43.629794+0000","last_became_active":"2023-11-16T16:47:22.647187+0000","last_became_peered":"2023-11-16T16:47:22.647187+0000","last_unstale":"2023-11-16T16:47:43.629794+0000","last_undegraded":"2023-11-16T16:47:43.629794+0000","last_fullsized":"2023-11-16T16:47:43.629794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:01:47.702590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.595597+0000","last_change":"2023-11-16T16:47:43.933501+0000","last_active":"2023-11-16T16:47:44.595597+0000","last_peered":"2023-11-16T16:47:44.595597+0000","last_clean":"2023-11-16T16:47:44.595597+0000","last_became_active":"2023-11-16T16:47:43.631746+0000","last_became_peered":"2023-11-16T16:47:43.631746+0000","last_unstale":"2023-11-16T16:47:44.595597+0000","last_undegraded":"2023-11-16T16:47:44.595597+0000","last_fullsized":"2023-11-16T16:47:44.595597+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:11:10.737388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625043+0000","last_change":"2023-11-16T16:47:18.175210+0000","last_active":"2023-11-16T16:47:43.625043+0000","last_peered":"2023-11-16T16:47:43.625043+0000","last_clean":"2023-11-16T16:47:43.625043+0000","last_became_active":"2023-11-16T16:47:18.174926+0000","last_became_peered":"2023-11-16T16:47:18.174926+0000","last_unstale":"2023-11-16T16:47:43.625043+0000","last_undegraded":"2023-11-16T16:47:43.625043+0000","last_fullsized":"2023-11-16T16:47:43.625043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:31:59.526898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.624874+0000","last_change":"2023-11-16T16:47:22.719361+0000","last_active":"2023-11-16T16:47:43.624874+0000","last_peered":"2023-11-16T16:47:43.624874+0000","last_clean":"2023-11-16T16:47:43.624874+0000","last_became_active":"2023-11-16T16:47:22.718936+0000","last_became_peered":"2023-11-16T16:47:22.718936+0000","last_unstale":"2023-11-16T16:47:43.624874+0000","last_undegraded":"2023-11-16T16:47:43.624874+0000","last_fullsized":"2023-11-16T16:47:43.624874+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:18:52.706386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622234+0000","last_change":"2023-11-16T16:47:17.632623+0000","last_active":"2023-11-16T16:47:43.622234+0000","last_peered":"2023-11-16T16:47:43.622234+0000","last_clean":"2023-11-16T16:47:43.622234+0000","last_became_active":"2023-11-16T16:47:17.631323+0000","last_became_peered":"2023-11-16T16:47:17.631323+0000","last_unstale":"2023-11-16T16:47:43.622234+0000","last_undegraded":"2023-11-16T16:47:43.622234+0000","last_fullsized":"2023-11-16T16:47:43.622234+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:39:00.464312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626526+0000","last_change":"2023-11-16T16:47:18.190779+0000","last_active":"2023-11-16T16:47:43.626526+0000","last_peered":"2023-11-16T16:47:43.626526+0000","last_clean":"2023-11-16T16:47:43.626526+0000","last_became_active":"2023-11-16T16:47:18.190440+0000","last_became_peered":"2023-11-16T16:47:18.190440+0000","last_unstale":"2023-11-16T16:47:43.626526+0000","last_undegraded":"2023-11-16T16:47:43.626526+0000","last_fullsized":"2023-11-16T16:47:43.626526+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:41:07.686317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630403+0000","last_change":"2023-11-16T16:47:22.664847+0000","last_active":"2023-11-16T16:47:43.630403+0000","last_peered":"2023-11-16T16:47:43.630403+0000","last_clean":"2023-11-16T16:47:43.630403+0000","last_became_active":"2023-11-16T16:47:22.659382+0000","last_became_peered":"2023-11-16T16:47:22.659382+0000","last_unstale":"2023-11-16T16:47:43.630403+0000","last_undegraded":"2023-11-16T16:47:43.630403+0000","last_fullsized":"2023-11-16T16:47:43.630403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:11:46.749829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622597+0000","last_change":"2023-11-16T16:47:17.630956+0000","last_active":"2023-11-16T16:47:43.622597+0000","last_peered":"2023-11-16T16:47:43.622597+0000","last_clean":"2023-11-16T16:47:43.622597+0000","last_became_active":"2023-11-16T16:47:17.629864+0000","last_became_peered":"2023-11-16T16:47:17.629864+0000","last_unstale":"2023-11-16T16:47:43.622597+0000","last_undegraded":"2023-11-16T16:47:43.622597+0000","last_fullsized":"2023-11-16T16:47:43.622597+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:01:12.432152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626055+0000","last_change":"2023-11-16T16:47:18.183375+0000","last_active":"2023-11-16T16:47:43.626055+0000","last_peered":"2023-11-16T16:47:43.626055+0000","last_clean":"2023-11-16T16:47:43.626055+0000","last_became_active":"2023-11-16T16:47:18.183050+0000","last_became_peered":"2023-11-16T16:47:18.183050+0000","last_unstale":"2023-11-16T16:47:43.626055+0000","last_undegraded":"2023-11-16T16:47:43.626055+0000","last_fullsized":"2023-11-16T16:47:43.626055+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:27:13.855960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630861+0000","last_change":"2023-11-16T16:47:22.719109+0000","last_active":"2023-11-16T16:47:43.630861+0000","last_peered":"2023-11-16T16:47:43.630861+0000","last_clean":"2023-11-16T16:47:43.630861+0000","last_became_active":"2023-11-16T16:47:22.718680+0000","last_became_peered":"2023-11-16T16:47:22.718680+0000","last_unstale":"2023-11-16T16:47:43.630861+0000","last_undegraded":"2023-11-16T16:47:43.630861+0000","last_fullsized":"2023-11-16T16:47:43.630861+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:36:16.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629990+0000","last_change":"2023-11-16T16:47:18.181632+0000","last_active":"2023-11-16T16:47:43.629990+0000","last_peered":"2023-11-16T16:47:43.629990+0000","last_clean":"2023-11-16T16:47:43.629990+0000","last_became_active":"2023-11-16T16:47:18.181323+0000","last_became_peered":"2023-11-16T16:47:18.181323+0000","last_unstale":"2023-11-16T16:47:43.629990+0000","last_undegraded":"2023-11-16T16:47:43.629990+0000","last_fullsized":"2023-11-16T16:47:43.629990+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:29:17.626827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629025+0000","last_change":"2023-11-16T16:47:17.637269+0000","last_active":"2023-11-16T16:47:43.629025+0000","last_peered":"2023-11-16T16:47:43.629025+0000","last_clean":"2023-11-16T16:47:43.629025+0000","last_became_active":"2023-11-16T16:47:17.636881+0000","last_became_peered":"2023-11-16T16:47:17.636881+0000","last_unstale":"2023-11-16T16:47:43.629025+0000","last_undegraded":"2023-11-16T16:47:43.629025+0000","last_fullsized":"2023-11-16T16:47:43.629025+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:43.798391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623135+0000","last_change":"2023-11-16T16:47:22.659528+0000","last_active":"2023-11-16T16:47:43.623135+0000","last_peered":"2023-11-16T16:47:43.623135+0000","last_clean":"2023-11-16T16:47:43.623135+0000","last_became_active":"2023-11-16T16:47:22.659076+0000","last_became_peered":"2023-11-16T16:47:22.659076+0000","last_unstale":"2023-11-16T16:47:43.623135+0000","last_undegraded":"2023-11-16T16:47:43.623135+0000","last_fullsized":"2023-11-16T16:47:43.623135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:06:05.562183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970448+0000","last_change":"2023-11-16T16:47:18.182102+0000","last_active":"2023-11-16T16:47:43.970448+0000","last_peered":"2023-11-16T16:47:43.970448+0000","last_clean":"2023-11-16T16:47:43.970448+0000","last_became_active":"2023-11-16T16:47:18.181528+0000","last_became_peered":"2023-11-16T16:47:18.181528+0000","last_unstale":"2023-11-16T16:47:43.970448+0000","last_undegraded":"2023-11-16T16:47:43.970448+0000","last_fullsized":"2023-11-16T16:47:43.970448+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:28:03.200017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629674+0000","last_change":"2023-11-16T16:47:17.626655+0000","last_active":"2023-11-16T16:47:43.629674+0000","last_peered":"2023-11-16T16:47:43.629674+0000","last_clean":"2023-11-16T16:47:43.629674+0000","last_became_active":"2023-11-16T16:47:17.626329+0000","last_became_peered":"2023-11-16T16:47:17.626329+0000","last_unstale":"2023-11-16T16:47:43.629674+0000","last_undegraded":"2023-11-16T16:47:43.629674+0000","last_fullsized":"2023-11-16T16:47:43.629674+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:30:26.199727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622227+0000","last_change":"2023-11-16T16:46:11.943398+0000","last_active":"2023-11-16T16:47:43.622227+0000","last_peered":"2023-11-16T16:47:43.622227+0000","last_clean":"2023-11-16T16:47:43.622227+0000","last_became_active":"2023-11-16T16:46:11.942862+0000","last_became_peered":"2023-11-16T16:46:11.942862+0000","last_unstale":"2023-11-16T16:47:43.622227+0000","last_undegraded":"2023-11-16T16:47:43.622227+0000","last_fullsized":"2023-11-16T16:47:43.622227+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:43:03.109797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:43:03.109797+0000","last_clean_scrub_stamp":"2023-11-16T16:43:03.109797+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:14:00.412865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630080+0000","last_change":"2023-11-16T16:47:22.719963+0000","last_active":"2023-11-16T16:47:43.630080+0000","last_peered":"2023-11-16T16:47:43.630080+0000","last_clean":"2023-11-16T16:47:43.630080+0000","last_became_active":"2023-11-16T16:47:22.719506+0000","last_became_peered":"2023-11-16T16:47:22.719506+0000","last_unstale":"2023-11-16T16:47:43.630080+0000","last_undegraded":"2023-11-16T16:47:43.630080+0000","last_fullsized":"2023-11-16T16:47:43.630080+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:27:40.361521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.628810+0000","last_change":"2023-11-16T16:47:18.184630+0000","last_active":"2023-11-16T16:47:43.628810+0000","last_peered":"2023-11-16T16:47:43.628810+0000","last_clean":"2023-11-16T16:47:43.628810+0000","last_became_active":"2023-11-16T16:47:18.181978+0000","last_became_peered":"2023-11-16T16:47:18.181978+0000","last_unstale":"2023-11-16T16:47:43.628810+0000","last_undegraded":"2023-11-16T16:47:43.628810+0000","last_fullsized":"2023-11-16T16:47:43.628810+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:22:39.294919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.635963+0000","last_change":"2023-11-16T16:47:43.635365+0000","last_active":"2023-11-16T16:47:43.635963+0000","last_peered":"2023-11-16T16:47:43.635963+0000","last_clean":"2023-11-16T16:47:43.635963+0000","last_became_active":"2023-11-16T16:47:43.634974+0000","last_became_peered":"2023-11-16T16:47:43.634974+0000","last_unstale":"2023-11-16T16:47:43.635963+0000","last_undegraded":"2023-11-16T16:47:43.635963+0000","last_fullsized":"2023-11-16T16:47:43.635963+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:40:21.789398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625072+0000","last_change":"2023-11-16T16:47:22.657133+0000","last_active":"2023-11-16T16:47:43.625072+0000","last_peered":"2023-11-16T16:47:43.625072+0000","last_clean":"2023-11-16T16:47:43.625072+0000","last_became_active":"2023-11-16T16:47:22.656557+0000","last_became_peered":"2023-11-16T16:47:22.656557+0000","last_unstale":"2023-11-16T16:47:43.625072+0000","last_undegraded":"2023-11-16T16:47:43.625072+0000","last_fullsized":"2023-11-16T16:47:43.625072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:06:56.883403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626111+0000","last_change":"2023-11-16T16:47:17.639107+0000","last_active":"2023-11-16T16:47:43.626111+0000","last_peered":"2023-11-16T16:47:43.626111+0000","last_clean":"2023-11-16T16:47:43.626111+0000","last_became_active":"2023-11-16T16:47:17.638796+0000","last_became_peered":"2023-11-16T16:47:17.638796+0000","last_unstale":"2023-11-16T16:47:43.626111+0000","last_undegraded":"2023-11-16T16:47:43.626111+0000","last_fullsized":"2023-11-16T16:47:43.626111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:15:27.137682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629965+0000","last_change":"2023-11-16T16:47:18.186761+0000","last_active":"2023-11-16T16:47:43.629965+0000","last_peered":"2023-11-16T16:47:43.629965+0000","last_clean":"2023-11-16T16:47:43.629965+0000","last_became_active":"2023-11-16T16:47:18.185594+0000","last_became_peered":"2023-11-16T16:47:18.185594+0000","last_unstale":"2023-11-16T16:47:43.629965+0000","last_undegraded":"2023-11-16T16:47:43.629965+0000","last_fullsized":"2023-11-16T16:47:43.629965+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:30:05.305376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970230+0000","last_change":"2023-11-16T16:47:22.719534+0000","last_active":"2023-11-16T16:47:43.970230+0000","last_peered":"2023-11-16T16:47:43.970230+0000","last_clean":"2023-11-16T16:47:43.970230+0000","last_became_active":"2023-11-16T16:47:22.718738+0000","last_became_peered":"2023-11-16T16:47:22.718738+0000","last_unstale":"2023-11-16T16:47:43.970230+0000","last_undegraded":"2023-11-16T16:47:43.970230+0000","last_fullsized":"2023-11-16T16:47:43.970230+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:14:30.912366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622745+0000","last_change":"2023-11-16T16:47:17.630318+0000","last_active":"2023-11-16T16:47:43.622745+0000","last_peered":"2023-11-16T16:47:43.622745+0000","last_clean":"2023-11-16T16:47:43.622745+0000","last_became_active":"2023-11-16T16:47:17.629035+0000","last_became_peered":"2023-11-16T16:47:17.629035+0000","last_unstale":"2023-11-16T16:47:43.622745+0000","last_undegraded":"2023-11-16T16:47:43.622745+0000","last_fullsized":"2023-11-16T16:47:43.622745+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:44:38.576547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970772+0000","last_change":"2023-11-16T16:47:18.185034+0000","last_active":"2023-11-16T16:47:43.970772+0000","last_peered":"2023-11-16T16:47:43.970772+0000","last_clean":"2023-11-16T16:47:43.970772+0000","last_became_active":"2023-11-16T16:47:18.181961+0000","last_became_peered":"2023-11-16T16:47:18.181961+0000","last_unstale":"2023-11-16T16:47:43.970772+0000","last_undegraded":"2023-11-16T16:47:43.970772+0000","last_fullsized":"2023-11-16T16:47:43.970772+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:38:55.393578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622320+0000","last_change":"2023-11-16T16:47:22.649952+0000","last_active":"2023-11-16T16:47:43.622320+0000","last_peered":"2023-11-16T16:47:43.622320+0000","last_clean":"2023-11-16T16:47:43.622320+0000","last_became_active":"2023-11-16T16:47:22.649481+0000","last_became_peered":"2023-11-16T16:47:22.649481+0000","last_unstale":"2023-11-16T16:47:43.622320+0000","last_undegraded":"2023-11-16T16:47:43.622320+0000","last_fullsized":"2023-11-16T16:47:43.622320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:36:07.109965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"82'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629922+0000","last_change":"2023-11-16T16:47:17.631982+0000","last_active":"2023-11-16T16:47:43.629922+0000","last_peered":"2023-11-16T16:47:43.629922+0000","last_clean":"2023-11-16T16:47:43.629922+0000","last_became_active":"2023-11-16T16:47:17.631552+0000","last_became_peered":"2023-11-16T16:47:17.631552+0000","last_unstale":"2023-11-16T16:47:43.629922+0000","last_undegraded":"2023-11-16T16:47:43.629922+0000","last_fullsized":"2023-11-16T16:47:43.629922+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:35:56.360671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626666+0000","last_change":"2023-11-16T16:47:18.197325+0000","last_active":"2023-11-16T16:47:43.626666+0000","last_peered":"2023-11-16T16:47:43.626666+0000","last_clean":"2023-11-16T16:47:43.626666+0000","last_became_active":"2023-11-16T16:47:18.190602+0000","last_became_peered":"2023-11-16T16:47:18.190602+0000","last_unstale":"2023-11-16T16:47:43.626666+0000","last_undegraded":"2023-11-16T16:47:43.626666+0000","last_fullsized":"2023-11-16T16:47:43.626666+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:28:40.186656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629554+0000","last_change":"2023-11-16T16:47:22.720023+0000","last_active":"2023-11-16T16:47:43.629554+0000","last_peered":"2023-11-16T16:47:43.629554+0000","last_clean":"2023-11-16T16:47:43.629554+0000","last_became_active":"2023-11-16T16:47:22.719561+0000","last_became_peered":"2023-11-16T16:47:22.719561+0000","last_unstale":"2023-11-16T16:47:43.629554+0000","last_undegraded":"2023-11-16T16:47:43.629554+0000","last_fullsized":"2023-11-16T16:47:43.629554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:40:41.808974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.946763+0000","last_change":"2023-11-16T16:47:17.629379+0000","last_active":"2023-11-16T16:47:43.946763+0000","last_peered":"2023-11-16T16:47:43.946763+0000","last_clean":"2023-11-16T16:47:43.946763+0000","last_became_active":"2023-11-16T16:47:17.629025+0000","last_became_peered":"2023-11-16T16:47:17.629025+0000","last_unstale":"2023-11-16T16:47:43.946763+0000","last_undegraded":"2023-11-16T16:47:43.946763+0000","last_fullsized":"2023-11-16T16:47:43.946763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:23:29.948397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626265+0000","last_change":"2023-11-16T16:47:18.185418+0000","last_active":"2023-11-16T16:47:43.626265+0000","last_peered":"2023-11-16T16:47:43.626265+0000","last_clean":"2023-11-16T16:47:43.626265+0000","last_became_active":"2023-11-16T16:47:18.185092+0000","last_became_peered":"2023-11-16T16:47:18.185092+0000","last_unstale":"2023-11-16T16:47:43.626265+0000","last_undegraded":"2023-11-16T16:47:43.626265+0000","last_fullsized":"2023-11-16T16:47:43.626265+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:02:15.891150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622811+0000","last_change":"2023-11-16T16:47:22.650701+0000","last_active":"2023-11-16T16:47:43.622811+0000","last_peered":"2023-11-16T16:47:43.622811+0000","last_clean":"2023-11-16T16:47:43.622811+0000","last_became_active":"2023-11-16T16:47:22.648710+0000","last_became_peered":"2023-11-16T16:47:22.648710+0000","last_unstale":"2023-11-16T16:47:43.622811+0000","last_undegraded":"2023-11-16T16:47:43.622811+0000","last_fullsized":"2023-11-16T16:47:43.622811+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:50:37.989145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970985+0000","last_change":"2023-11-16T16:47:17.638622+0000","last_active":"2023-11-16T16:47:43.970985+0000","last_peered":"2023-11-16T16:47:43.970985+0000","last_clean":"2023-11-16T16:47:43.970985+0000","last_became_active":"2023-11-16T16:47:17.638303+0000","last_became_peered":"2023-11-16T16:47:17.638303+0000","last_unstale":"2023-11-16T16:47:43.970985+0000","last_undegraded":"2023-11-16T16:47:43.970985+0000","last_fullsized":"2023-11-16T16:47:43.970985+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:03:52.117172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626598+0000","last_change":"2023-11-16T16:47:18.173474+0000","last_active":"2023-11-16T16:47:43.626598+0000","last_peered":"2023-11-16T16:47:43.626598+0000","last_clean":"2023-11-16T16:47:43.626598+0000","last_became_active":"2023-11-16T16:47:18.173140+0000","last_became_peered":"2023-11-16T16:47:18.173140+0000","last_unstale":"2023-11-16T16:47:43.626598+0000","last_undegraded":"2023-11-16T16:47:43.626598+0000","last_fullsized":"2023-11-16T16:47:43.626598+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:13:23.724052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629348+0000","last_change":"2023-11-16T16:47:22.655806+0000","last_active":"2023-11-16T16:47:43.629348+0000","last_peered":"2023-11-16T16:47:43.629348+0000","last_clean":"2023-11-16T16:47:43.629348+0000","last_became_active":"2023-11-16T16:47:22.655385+0000","last_became_peered":"2023-11-16T16:47:22.655385+0000","last_unstale":"2023-11-16T16:47:43.629348+0000","last_undegraded":"2023-11-16T16:47:43.629348+0000","last_fullsized":"2023-11-16T16:47:43.629348+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:31:41.160819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622449+0000","last_change":"2023-11-16T16:47:17.631673+0000","last_active":"2023-11-16T16:47:43.622449+0000","last_peered":"2023-11-16T16:47:43.622449+0000","last_clean":"2023-11-16T16:47:43.622449+0000","last_became_active":"2023-11-16T16:47:17.631356+0000","last_became_peered":"2023-11-16T16:47:17.631356+0000","last_unstale":"2023-11-16T16:47:43.622449+0000","last_undegraded":"2023-11-16T16:47:43.622449+0000","last_fullsized":"2023-11-16T16:47:43.622449+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:29:12.540977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970538+0000","last_change":"2023-11-16T16:47:18.186706+0000","last_active":"2023-11-16T16:47:43.970538+0000","last_peered":"2023-11-16T16:47:43.970538+0000","last_clean":"2023-11-16T16:47:43.970538+0000","last_became_active":"2023-11-16T16:47:18.186360+0000","last_became_peered":"2023-11-16T16:47:18.186360+0000","last_unstale":"2023-11-16T16:47:43.970538+0000","last_undegraded":"2023-11-16T16:47:43.970538+0000","last_fullsized":"2023-11-16T16:47:43.970538+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:32:31.165136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629693+0000","last_change":"2023-11-16T16:47:22.719065+0000","last_active":"2023-11-16T16:47:43.629693+0000","last_peered":"2023-11-16T16:47:43.629693+0000","last_clean":"2023-11-16T16:47:43.629693+0000","last_became_active":"2023-11-16T16:47:22.718662+0000","last_became_peered":"2023-11-16T16:47:22.718662+0000","last_unstale":"2023-11-16T16:47:43.629693+0000","last_undegraded":"2023-11-16T16:47:43.629693+0000","last_fullsized":"2023-11-16T16:47:43.629693+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:56:10.235168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"82'2","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.638200+0000","last_change":"2023-11-16T16:47:43.637790+0000","last_active":"2023-11-16T16:47:43.638200+0000","last_peered":"2023-11-16T16:47:43.638200+0000","last_clean":"2023-11-16T16:47:43.638200+0000","last_became_active":"2023-11-16T16:47:43.632225+0000","last_became_peered":"2023-11-16T16:47:43.632225+0000","last_unstale":"2023-11-16T16:47:43.638200+0000","last_undegraded":"2023-11-16T16:47:43.638200+0000","last_fullsized":"2023-11-16T16:47:43.638200+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:31:01.698264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622703+0000","last_change":"2023-11-16T16:47:18.180999+0000","last_active":"2023-11-16T16:47:43.622703+0000","last_peered":"2023-11-16T16:47:43.622703+0000","last_clean":"2023-11-16T16:47:43.622703+0000","last_became_active":"2023-11-16T16:47:18.180724+0000","last_became_peered":"2023-11-16T16:47:18.180724+0000","last_unstale":"2023-11-16T16:47:43.622703+0000","last_undegraded":"2023-11-16T16:47:43.622703+0000","last_fullsized":"2023-11-16T16:47:43.622703+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:38:44.147293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947186+0000","last_change":"2023-11-16T16:47:22.665607+0000","last_active":"2023-11-16T16:47:43.947186+0000","last_peered":"2023-11-16T16:47:43.947186+0000","last_clean":"2023-11-16T16:47:43.947186+0000","last_became_active":"2023-11-16T16:47:22.665273+0000","last_became_peered":"2023-11-16T16:47:22.665273+0000","last_unstale":"2023-11-16T16:47:43.947186+0000","last_undegraded":"2023-11-16T16:47:43.947186+0000","last_fullsized":"2023-11-16T16:47:43.947186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:08:37.804057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622937+0000","last_change":"2023-11-16T16:47:17.632614+0000","last_active":"2023-11-16T16:47:43.622937+0000","last_peered":"2023-11-16T16:47:43.622937+0000","last_clean":"2023-11-16T16:47:43.622937+0000","last_became_active":"2023-11-16T16:47:17.632227+0000","last_became_peered":"2023-11-16T16:47:17.632227+0000","last_unstale":"2023-11-16T16:47:43.622937+0000","last_undegraded":"2023-11-16T16:47:43.622937+0000","last_fullsized":"2023-11-16T16:47:43.622937+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:58:13.465619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623388+0000","last_change":"2023-11-16T16:47:18.177607+0000","last_active":"2023-11-16T16:47:43.623388+0000","last_peered":"2023-11-16T16:47:43.623388+0000","last_clean":"2023-11-16T16:47:43.623388+0000","last_became_active":"2023-11-16T16:47:18.177266+0000","last_became_peered":"2023-11-16T16:47:18.177266+0000","last_unstale":"2023-11-16T16:47:43.623388+0000","last_undegraded":"2023-11-16T16:47:43.623388+0000","last_fullsized":"2023-11-16T16:47:43.623388+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:03:08.676930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625888+0000","last_change":"2023-11-16T16:47:22.659987+0000","last_active":"2023-11-16T16:47:43.625888+0000","last_peered":"2023-11-16T16:47:43.625888+0000","last_clean":"2023-11-16T16:47:43.625888+0000","last_became_active":"2023-11-16T16:47:22.658725+0000","last_became_peered":"2023-11-16T16:47:22.658725+0000","last_unstale":"2023-11-16T16:47:43.625888+0000","last_undegraded":"2023-11-16T16:47:43.625888+0000","last_fullsized":"2023-11-16T16:47:43.625888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:51:27.380649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630637+0000","last_change":"2023-11-16T16:47:17.640696+0000","last_active":"2023-11-16T16:47:43.630637+0000","last_peered":"2023-11-16T16:47:43.630637+0000","last_clean":"2023-11-16T16:47:43.630637+0000","last_became_active":"2023-11-16T16:47:17.640428+0000","last_became_peered":"2023-11-16T16:47:17.640428+0000","last_unstale":"2023-11-16T16:47:43.630637+0000","last_undegraded":"2023-11-16T16:47:43.630637+0000","last_fullsized":"2023-11-16T16:47:43.630637+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:47:12.331771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.972273+0000","last_change":"2023-11-16T16:47:43.972273+0000","last_active":"2023-11-16T16:47:43.972273+0000","last_peered":"2023-11-16T16:47:43.972273+0000","last_clean":"2023-11-16T16:47:43.972273+0000","last_became_active":"2023-11-16T16:47:43.971981+0000","last_became_peered":"2023-11-16T16:47:43.971981+0000","last_unstale":"2023-11-16T16:47:43.972273+0000","last_undegraded":"2023-11-16T16:47:43.972273+0000","last_fullsized":"2023-11-16T16:47:43.972273+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:38:01.080979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629465+0000","last_change":"2023-11-16T16:47:22.649436+0000","last_active":"2023-11-16T16:47:43.629465+0000","last_peered":"2023-11-16T16:47:43.629465+0000","last_clean":"2023-11-16T16:47:43.629465+0000","last_became_active":"2023-11-16T16:47:22.649034+0000","last_became_peered":"2023-11-16T16:47:22.649034+0000","last_unstale":"2023-11-16T16:47:43.629465+0000","last_undegraded":"2023-11-16T16:47:43.629465+0000","last_fullsized":"2023-11-16T16:47:43.629465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:50:06.438075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.631007+0000","last_change":"2023-11-16T16:47:17.634582+0000","last_active":"2023-11-16T16:47:43.631007+0000","last_peered":"2023-11-16T16:47:43.631007+0000","last_clean":"2023-11-16T16:47:43.631007+0000","last_became_active":"2023-11-16T16:47:17.634172+0000","last_became_peered":"2023-11-16T16:47:17.634172+0000","last_unstale":"2023-11-16T16:47:43.631007+0000","last_undegraded":"2023-11-16T16:47:43.631007+0000","last_fullsized":"2023-11-16T16:47:43.631007+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:50:55.582584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623419+0000","last_change":"2023-11-16T16:47:18.180466+0000","last_active":"2023-11-16T16:47:43.623419+0000","last_peered":"2023-11-16T16:47:43.623419+0000","last_clean":"2023-11-16T16:47:43.623419+0000","last_became_active":"2023-11-16T16:47:18.180166+0000","last_became_peered":"2023-11-16T16:47:18.180166+0000","last_unstale":"2023-11-16T16:47:43.623419+0000","last_undegraded":"2023-11-16T16:47:43.623419+0000","last_fullsized":"2023-11-16T16:47:43.623419+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:02:56.576134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626739+0000","last_change":"2023-11-16T16:47:18.179196+0000","last_active":"2023-11-16T16:47:43.626739+0000","last_peered":"2023-11-16T16:47:43.626739+0000","last_clean":"2023-11-16T16:47:43.626739+0000","last_became_active":"2023-11-16T16:47:18.178827+0000","last_became_peered":"2023-11-16T16:47:18.178827+0000","last_unstale":"2023-11-16T16:47:43.626739+0000","last_undegraded":"2023-11-16T16:47:43.626739+0000","last_fullsized":"2023-11-16T16:47:43.626739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:07:11.423464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622851+0000","last_change":"2023-11-16T16:47:18.184630+0000","last_active":"2023-11-16T16:47:43.622851+0000","last_peered":"2023-11-16T16:47:43.622851+0000","last_clean":"2023-11-16T16:47:43.622851+0000","last_became_active":"2023-11-16T16:47:18.184314+0000","last_became_peered":"2023-11-16T16:47:18.184314+0000","last_unstale":"2023-11-16T16:47:43.622851+0000","last_undegraded":"2023-11-16T16:47:43.622851+0000","last_fullsized":"2023-11-16T16:47:43.622851+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:56:13.651803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622558+0000","last_change":"2023-11-16T16:47:18.184255+0000","last_active":"2023-11-16T16:47:43.622558+0000","last_peered":"2023-11-16T16:47:43.622558+0000","last_clean":"2023-11-16T16:47:43.622558+0000","last_became_active":"2023-11-16T16:47:18.183936+0000","last_became_peered":"2023-11-16T16:47:18.183936+0000","last_unstale":"2023-11-16T16:47:43.622558+0000","last_undegraded":"2023-11-16T16:47:43.622558+0000","last_fullsized":"2023-11-16T16:47:43.622558+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:59:41.136565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626136+0000","last_change":"2023-11-16T16:47:18.196331+0000","last_active":"2023-11-16T16:47:43.626136+0000","last_peered":"2023-11-16T16:47:43.626136+0000","last_clean":"2023-11-16T16:47:43.626136+0000","last_became_active":"2023-11-16T16:47:18.195954+0000","last_became_peered":"2023-11-16T16:47:18.195954+0000","last_unstale":"2023-11-16T16:47:43.626136+0000","last_undegraded":"2023-11-16T16:47:43.626136+0000","last_fullsized":"2023-11-16T16:47:43.626136+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:31:23.208122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629719+0000","last_change":"2023-11-16T16:47:18.190537+0000","last_active":"2023-11-16T16:47:43.629719+0000","last_peered":"2023-11-16T16:47:43.629719+0000","last_clean":"2023-11-16T16:47:43.629719+0000","last_became_active":"2023-11-16T16:47:18.190118+0000","last_became_peered":"2023-11-16T16:47:18.190118+0000","last_unstale":"2023-11-16T16:47:43.629719+0000","last_undegraded":"2023-11-16T16:47:43.629719+0000","last_fullsized":"2023-11-16T16:47:43.629719+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:39:38.125056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625721+0000","last_change":"2023-11-16T16:47:18.179215+0000","last_active":"2023-11-16T16:47:43.625721+0000","last_peered":"2023-11-16T16:47:43.625721+0000","last_clean":"2023-11-16T16:47:43.625721+0000","last_became_active":"2023-11-16T16:47:18.178759+0000","last_became_peered":"2023-11-16T16:47:18.178759+0000","last_unstale":"2023-11-16T16:47:43.625721+0000","last_undegraded":"2023-11-16T16:47:43.625721+0000","last_fullsized":"2023-11-16T16:47:43.625721+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:06:54.886510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623599+0000","last_change":"2023-11-16T16:47:18.179588+0000","last_active":"2023-11-16T16:47:43.623599+0000","last_peered":"2023-11-16T16:47:43.623599+0000","last_clean":"2023-11-16T16:47:43.623599+0000","last_became_active":"2023-11-16T16:47:18.179294+0000","last_became_peered":"2023-11-16T16:47:18.179294+0000","last_unstale":"2023-11-16T16:47:43.623599+0000","last_undegraded":"2023-11-16T16:47:43.623599+0000","last_fullsized":"2023-11-16T16:47:43.623599+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:02:55.079560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.971128+0000","last_change":"2023-11-16T16:47:18.181587+0000","last_active":"2023-11-16T16:47:43.971128+0000","last_peered":"2023-11-16T16:47:43.971128+0000","last_clean":"2023-11-16T16:47:43.971128+0000","last_became_active":"2023-11-16T16:47:18.181203+0000","last_became_peered":"2023-11-16T16:47:18.181203+0000","last_unstale":"2023-11-16T16:47:43.971128+0000","last_undegraded":"2023-11-16T16:47:43.971128+0000","last_fullsized":"2023-11-16T16:47:43.971128+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:13:49.378115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629816+0000","last_change":"2023-11-16T16:47:18.182690+0000","last_active":"2023-11-16T16:47:43.629816+0000","last_peered":"2023-11-16T16:47:43.629816+0000","last_clean":"2023-11-16T16:47:43.629816+0000","last_became_active":"2023-11-16T16:47:18.182361+0000","last_became_peered":"2023-11-16T16:47:18.182361+0000","last_unstale":"2023-11-16T16:47:43.629816+0000","last_undegraded":"2023-11-16T16:47:43.629816+0000","last_fullsized":"2023-11-16T16:47:43.629816+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:55:41.987600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970993+0000","last_change":"2023-11-16T16:47:18.184599+0000","last_active":"2023-11-16T16:47:43.970993+0000","last_peered":"2023-11-16T16:47:43.970993+0000","last_clean":"2023-11-16T16:47:43.970993+0000","last_became_active":"2023-11-16T16:47:18.184288+0000","last_became_peered":"2023-11-16T16:47:18.184288+0000","last_unstale":"2023-11-16T16:47:43.970993+0000","last_undegraded":"2023-11-16T16:47:43.970993+0000","last_fullsized":"2023-11-16T16:47:43.970993+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:43:20.190571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947382+0000","last_change":"2023-11-16T16:47:18.187267+0000","last_active":"2023-11-16T16:47:43.947382+0000","last_peered":"2023-11-16T16:47:43.947382+0000","last_clean":"2023-11-16T16:47:43.947382+0000","last_became_active":"2023-11-16T16:47:18.186984+0000","last_became_peered":"2023-11-16T16:47:18.186984+0000","last_unstale":"2023-11-16T16:47:43.947382+0000","last_undegraded":"2023-11-16T16:47:43.947382+0000","last_fullsized":"2023-11-16T16:47:43.947382+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:30:20.891991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630154+0000","last_change":"2023-11-16T16:47:18.191566+0000","last_active":"2023-11-16T16:47:43.630154+0000","last_peered":"2023-11-16T16:47:43.630154+0000","last_clean":"2023-11-16T16:47:43.630154+0000","last_became_active":"2023-11-16T16:47:18.185342+0000","last_became_peered":"2023-11-16T16:47:18.185342+0000","last_unstale":"2023-11-16T16:47:43.630154+0000","last_undegraded":"2023-11-16T16:47:43.630154+0000","last_fullsized":"2023-11-16T16:47:43.630154+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:55:29.393440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629636+0000","last_change":"2023-11-16T16:47:43.629636+0000","last_active":"2023-11-16T16:47:43.629636+0000","last_peered":"2023-11-16T16:47:43.629636+0000","last_clean":"2023-11-16T16:47:43.629636+0000","last_became_active":"2023-11-16T16:47:43.629179+0000","last_became_peered":"2023-11-16T16:47:43.629179+0000","last_unstale":"2023-11-16T16:47:43.629636+0000","last_undegraded":"2023-11-16T16:47:43.629636+0000","last_fullsized":"2023-11-16T16:47:43.629636+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:21:10.581239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281399,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4592,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":12,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":65878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449107,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6136,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739016,"statfs":{"total":95995035648,"available":95988752384,"internally_reserved":0,"allocated":966656,"data_stored":483162,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51000000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.56100000000000005}]},{"osd":2,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.34899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.59399999999999997}]},{"osd":5,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.47799999999999998}]},{"osd":6,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.626}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":9,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.64400000000000002}]},{"osd":10,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.80900000000000005}]}]},{"osd":10,"up_from":71,"seq":304942678039,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6332,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738820,"statfs":{"total":95995035648,"available":95988551680,"internally_reserved":0,"allocated":1036288,"data_stored":1077496,"data_compressed":53424,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":1,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.47999999999999998}]},{"osd":2,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.65300000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63}]},{"osd":5,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.78300000000000003}]},{"osd":6,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.76800000000000002}]},{"osd":7,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.69999999999999996}]},{"osd":8,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.75600000000000001}]},{"osd":9,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.80700000000000005}]},{"osd":11,"last update":"Thu Nov 16 16:47:30 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.85299999999999998}]}]},{"osd":9,"up_from":66,"seq":283467841563,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6268,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738884,"statfs":{"total":95995035648,"available":95988617216,"internally_reserved":0,"allocated":970752,"data_stored":487818,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.629}]},{"osd":1,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.68799999999999994}]},{"osd":2,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.71199999999999997}]},{"osd":3,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.94399999999999995}]},{"osd":4,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.753},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.78800000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.032}]},{"osd":6,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":1.054},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.95599999999999996}]},{"osd":7,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.878},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":1.0740000000000001}]},{"osd":8,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.92200000000000004}]},{"osd":10,"last update":"Thu Nov 16 16:47:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.016},{"interface":"front","average":{"1min":1.05,"5min":1.05,"15min":1.05},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":1.0960000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]}]},{"osd":8,"up_from":58,"seq":249108103200,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6392,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738760,"statfs":{"total":95995035648,"available":95988490240,"internally_reserved":0,"allocated":966656,"data_stored":483722,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:23 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.78600000000000003}]},{"osd":1,"last update":"Thu Nov 16 16:47:23 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.60899999999999999,"15min":0.59599999999999997},"min":{"1min":0.435,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.61199999999999999,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Nov 16 16:47:23 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.46400000000000002,"15min":0.45400000000000001},"min":{"1min":0.29599999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.46500000000000002,"15min":0.46300000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.441}]},{"osd":3,"last update":"Thu Nov 16 16:47:23 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.44800000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.92200000000000004,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.75900000000000001}]},{"osd":4,"last update":"Thu Nov 16 16:47:23 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.628,"15min":0.625},"min":{"1min":0.442,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.95599999999999996,"5min":0.999,"15min":0.999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69199999999999995,"15min":0.68600000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.80100000000000005}]},{"osd":5,"last update":"Thu Nov 16 16:47:23 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53700000000000003,"15min":0.52900000000000003},"min":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56399999999999995,"15min":0.55900000000000005},"min":{"1min":0.35599999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.47199999999999998}]},{"osd":6,"last update":"Thu Nov 16 16:47:23 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.72899999999999998,"15min":0.72199999999999998},"min":{"1min":0.52800000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.68100000000000005,"15min":0.67600000000000005},"min":{"1min":0.47099999999999997,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.71399999999999997}]},{"osd":7,"last update":"Thu Nov 16 16:47:23 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.69299999999999995,"15min":0.68700000000000006},"min":{"1min":0.46800000000000003,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.751,"5min":0.72599999999999998,"15min":0.72199999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.86299999999999999}]},{"osd":9,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77200000000000002,"15min":0.77100000000000002},"min":{"1min":0.52200000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92000000000000004,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.81200000000000006,"15min":0.81399999999999995},"min":{"1min":0.55900000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0069999999999999,"5min":1.298,"15min":1.298},"last":0.93700000000000006}]},{"osd":10,"last update":"Thu Nov 16 16:47:08 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.89200000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]},{"osd":7,"up_from":51,"seq":219043332132,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6504,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738648,"statfs":{"total":95995035648,"available":95988375552,"internally_reserved":0,"allocated":1015808,"data_stored":1074090,"data_compressed":53424,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.377,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81799999999999995,"5min":0.875,"15min":0.875},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.41999999999999998,"15min":0.40799999999999997},"min":{"1min":0.32900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.44900000000000001,"15min":0.44},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73599999999999999,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.45800000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.78400000000000003}]},{"osd":3,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.56599999999999995,"15min":0.55200000000000005},"min":{"1min":0.437,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.629,"15min":0.621},"min":{"1min":0.51600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.86699999999999999,"5min":0.873,"15min":0.873},"last":0.73299999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53200000000000003,"15min":0.52900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.82499999999999996,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.56100000000000005}]},{"osd":5,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.86799999999999999,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.64100000000000001,"15min":0.63100000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.89400000000000002}]},{"osd":6,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.69099999999999995,"15min":0.67800000000000005},"min":{"1min":0.63400000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68000000000000005,"15min":0.67200000000000004},"min":{"1min":0.48699999999999999,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.74299999999999999}]},{"osd":8,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.68000000000000005,"15min":0.67200000000000004},"min":{"1min":0.53900000000000003,"5min":0.436,"15min":0.436},"max":{"1min":0.95599999999999996,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.47399999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.94099999999999995,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.878}]},{"osd":9,"last update":"Thu Nov 16 16:46:47 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.93100000000000005}]},{"osd":10,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.85199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]}]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6388,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738764,"statfs":{"total":95995035648,"available":95988494336,"internally_reserved":0,"allocated":962560,"data_stored":483140,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.38800000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.43099999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84599999999999997,"5min":1.345,"15min":1.345},"last":0.44700000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64900000000000002,"15min":0.64800000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.98199999999999998,"5min":1.111,"15min":1.111},"last":1.018},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69699999999999995,"15min":0.70399999999999996},"min":{"1min":0.505,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0229999999999999,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.66900000000000004}]},{"osd":2,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.71799999999999997,"15min":0.70799999999999996},"min":{"1min":0.54900000000000004,"5min":0.436,"15min":0.436},"max":{"1min":0.998,"5min":1.407,"15min":1.407},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.71899999999999997,"15min":0.72499999999999998},"min":{"1min":0.48799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.98699999999999999,"5min":1.171,"15min":1.171},"last":0.88700000000000001}]},{"osd":3,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.72799999999999998,"15min":0.73999999999999999},"min":{"1min":0.49099999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0189999999999999,"5min":1.022,"15min":1.022},"last":0.85199999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.746,"15min":0.73199999999999998},"min":{"1min":0.56899999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.755,"15min":0.76000000000000001},"min":{"1min":0.58499999999999996,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.048,"5min":1.083,"15min":1.083},"last":0.93500000000000005}]},{"osd":5,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79000000000000004,"15min":0.80000000000000004},"min":{"1min":0.61099999999999999,"5min":0.503,"15min":0.503},"max":{"1min":1.0940000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.88300000000000001,"15min":0.89700000000000002},"min":{"1min":0.747,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.2749999999999999,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.63100000000000001}]},{"osd":7,"last update":"Thu Nov 16 16:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.80500000000000005,"15min":0.79300000000000004},"min":{"1min":0.54800000000000004,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.754,"15min":0.74399999999999999},"min":{"1min":0.55400000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":1.0820000000000001}]},{"osd":8,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.83899999999999997,"15min":0.84499999999999997},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.88800000000000001,"15min":0.88700000000000001},"min":{"1min":0.66200000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.3360000000000001,"5min":1.823,"15min":1.823},"last":0.91300000000000003}]},{"osd":9,"last update":"Thu Nov 16 16:47:50 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.96499999999999997,"15min":0.96699999999999997},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.238,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.875},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.84699999999999998,"15min":0.83899999999999997},"min":{"1min":0.61899999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.069,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.97899999999999998}]},{"osd":10,"last update":"Thu Nov 16 16:47:11 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.95999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7296,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737856,"statfs":{"total":95995035648,"available":95987564544,"internally_reserved":0,"allocated":974848,"data_stored":486934,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55100000000000005,"15min":0.52500000000000002},"min":{"1min":0.39600000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.77500000000000002,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.54000000000000004,"15min":0.51400000000000001},"min":{"1min":0.40500000000000003,"5min":0.315,"15min":0.315},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.56699999999999995}]},{"osd":2,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.30099999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72599999999999998,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.51200000000000001,"15min":0.499},"min":{"1min":0.40100000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.60999999999999999}]},{"osd":3,"last update":"Thu Nov 16 16:47:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55000000000000004,"15min":0.53400000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.58899999999999997,"15min":0.57299999999999995},"min":{"1min":0.41199999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Nov 16 16:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.41899999999999998,"15min":0.41499999999999998},"min":{"1min":0.26500000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.435,"15min":0.43099999999999999},"min":{"1min":0.28499999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.78900000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:47:15 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.47099999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76600000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.72199999999999998}]},{"osd":6,"last update":"Thu Nov 16 16:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.42599999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82999999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.65800000000000003,"15min":0.64200000000000002},"min":{"1min":0.48899999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.80700000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:47:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.52700000000000002,"15min":0.51300000000000001},"min":{"1min":0.39100000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.56200000000000006,"15min":0.55200000000000005},"min":{"1min":0.36799999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.84299999999999997}]},{"osd":8,"last update":"Thu Nov 16 16:47:24 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.64400000000000002,"15min":0.63200000000000001},"min":{"1min":0.42599999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.871},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.65400000000000003,"15min":0.64400000000000002},"min":{"1min":0.46200000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.85899999999999999}]},{"osd":9,"last update":"Thu Nov 16 16:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.89400000000000002}]},{"osd":10,"last update":"Thu Nov 16 16:47:08 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.90700000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]}]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7288,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737864,"statfs":{"total":95995035648,"available":95987572736,"internally_reserved":0,"allocated":966656,"data_stored":483230,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 16:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57099999999999995,"15min":0.54300000000000004},"min":{"1min":0.375,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.88400000000000001,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.752,"5min":0.627,"15min":0.56899999999999995},"min":{"1min":0.38100000000000001,"5min":0.316,"15min":0.316},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.71599999999999997}]},{"osd":2,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.66800000000000004,"15min":0.58999999999999997},"min":{"1min":0.50900000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.97499999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.64500000000000002,"15min":0.60999999999999999},"min":{"1min":0.47999999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.86099999999999999}]},{"osd":3,"last update":"Thu Nov 16 16:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.53800000000000003,"15min":0.52400000000000002},"min":{"1min":0.438,"5min":0.31,"15min":0.31},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.53000000000000003,"15min":0.51400000000000001},"min":{"1min":0.33400000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.63,"15min":0.58299999999999996},"min":{"1min":0.44,"5min":0.434,"15min":0.434},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.75800000000000001,"15min":0.79100000000000004},"min":{"1min":0.55600000000000005,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0069999999999999,"5min":5.4409999999999998,"15min":5.4409999999999998},"last":0.88100000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:47:17 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72899999999999998,"15min":0.69899999999999995},"min":{"1min":0.57199999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0089999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72199999999999998,"15min":0.71599999999999997},"min":{"1min":0.44400000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.97299999999999998,"5min":1.153,"15min":1.153},"last":0.754}]},{"osd":6,"last update":"Thu Nov 16 16:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.70299999999999996,"15min":0.68700000000000006},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.98599999999999999,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.69499999999999995,"15min":0.66600000000000004},"min":{"1min":0.42499999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.036,"5min":1.093,"15min":1.093},"last":0.81699999999999995}]},{"osd":7,"last update":"Thu Nov 16 16:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.79100000000000004,"15min":0.78700000000000003},"min":{"1min":0.54400000000000004,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.83499999999999996,"15min":0.82999999999999996},"min":{"1min":0.58699999999999997,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.63400000000000001}]},{"osd":8,"last update":"Thu Nov 16 16:47:22 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.86799999999999999,"15min":0.88},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.111,"5min":1.298,"15min":1.298},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.90600000000000003,"15min":0.91800000000000004},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.1319999999999999,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.83999999999999997}]},{"osd":9,"last update":"Thu Nov 16 16:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.98399999999999999},"min":{"1min":0.64100000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1930000000000001,"5min":1.278,"15min":1.278},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.91000000000000003,"15min":0.92100000000000004},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.117,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.73299999999999998}]},{"osd":10,"last update":"Thu Nov 16 16:47:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":3.694,"5min":3.694,"15min":3.694},"last":0.89700000000000002}]},{"osd":11,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.79100000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7460,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737692,"statfs":{"total":95995035648,"available":95987396608,"internally_reserved":0,"allocated":1011712,"data_stored":521416,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.503,"15min":0.45900000000000002},"min":{"1min":0.34599999999999997,"5min":0.312,"15min":0.312},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51000000000000001,"15min":0.49099999999999999},"min":{"1min":0.312,"5min":0.309,"15min":0.309},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.621}]},{"osd":1,"last update":"Thu Nov 16 16:47:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.51300000000000001,"15min":0.48499999999999999},"min":{"1min":0.35199999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.52000000000000002,"15min":0.48799999999999999},"min":{"1min":0.36699999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77000000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.88800000000000001}]},{"osd":3,"last update":"Thu Nov 16 16:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68100000000000005,"15min":0.72099999999999997},"min":{"1min":0.49299999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.89300000000000002,"5min":3.98,"15min":3.98},"last":0.753},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.60399999999999998,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.57299999999999995}]},{"osd":4,"last update":"Thu Nov 16 16:46:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.72799999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.72799999999999998}]},{"osd":5,"last update":"Thu Nov 16 16:47:19 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46200000000000002,"15min":0.45600000000000002},"min":{"1min":0.29899999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.48699999999999999,"15min":0.47499999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.497}]},{"osd":6,"last update":"Thu Nov 16 16:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.59199999999999997,"15min":0.58199999999999996},"min":{"1min":0.44400000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.66800000000000004}]},{"osd":7,"last update":"Thu Nov 16 16:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65500000000000003,"15min":0.65600000000000003},"min":{"1min":0.46899999999999997,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.85599999999999998}]},{"osd":8,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59499999999999997,"15min":0.59399999999999997},"min":{"1min":0.37,"5min":0.309,"15min":0.309},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.69399999999999995}]},{"osd":9,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.71899999999999997,"15min":0.71699999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.91300000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.73099999999999998,"15min":0.72899999999999998},"min":{"1min":0.51200000000000001,"5min":0.503,"15min":0.503},"max":{"1min":0.91500000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.78800000000000003}]},{"osd":10,"last update":"Thu Nov 16 16:47:11 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.84399999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6716,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738436,"statfs":{"total":95995035648,"available":95988158464,"internally_reserved":0,"allocated":970752,"data_stored":483244,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:35 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.45500000000000002,"15min":0.42599999999999999},"min":{"1min":0.40500000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57099999999999995,"5min":1.085,"15min":1.085},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54500000000000004,"15min":0.49299999999999999},"min":{"1min":0.36799999999999999,"5min":0.313,"15min":0.313},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.5}]},{"osd":1,"last update":"Thu Nov 16 16:47:35 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.60499999999999998,"15min":0.57399999999999995},"min":{"1min":0.41899999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.96999999999999997,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.63200000000000001,"15min":0.59499999999999997},"min":{"1min":0.45400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.006,"5min":1.286,"15min":1.286},"last":0.70599999999999996}]},{"osd":2,"last update":"Thu Nov 16 16:47:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.63300000000000001,"15min":0.58699999999999997},"min":{"1min":0.53200000000000003,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.92500000000000004,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.64500000000000002,"15min":0.60099999999999998},"min":{"1min":0.48499999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.67200000000000004}]},{"osd":4,"last update":"Thu Nov 16 16:46:56 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.63800000000000001,"15min":0.60399999999999998},"min":{"1min":0.41499999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.93200000000000005,"5min":1.25,"15min":1.25},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.68300000000000005,"15min":0.66200000000000003},"min":{"1min":0.49199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.115,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.71199999999999997,"15min":0.68999999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.73299999999999998,"15min":0.67700000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":5.1459999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":0.80000000000000004}]},{"osd":6,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.69399999999999995,"15min":0.65900000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0669999999999999,"5min":1.323,"15min":1.323},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.70599999999999996,"15min":0.68200000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.02,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.79700000000000004,"15min":0.78800000000000003},"min":{"1min":0.55200000000000005,"5min":0.504,"15min":0.504},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.755},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.85999999999999999,"15min":0.84999999999999998},"min":{"1min":0.627,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.359,"5min":1.369,"15min":1.369},"last":0.87}]},{"osd":8,"last update":"Thu Nov 16 16:46:23 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.94199999999999995}]},{"osd":9,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.90400000000000003,"15min":0.91700000000000004},"min":{"1min":0.66700000000000004,"5min":0.501,"15min":0.501},"max":{"1min":1.0900000000000001,"5min":1.395,"15min":1.395},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.97199999999999998,"15min":0.97699999999999998},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.81899999999999995}]},{"osd":10,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.304,"5min":1.304,"15min":1.304},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":1.002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]}]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6520,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738632,"statfs":{"total":95995035648,"available":95988359168,"internally_reserved":0,"allocated":966656,"data_stored":483162,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.48899999999999999,"15min":0.46600000000000003},"min":{"1min":0.439,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.51100000000000001,"15min":0.48599999999999999},"min":{"1min":0.47399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.63500000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.34200000000000003,"15min":0.318},"min":{"1min":0.33800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.35899999999999999,"15min":0.34100000000000003},"min":{"1min":0.34999999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.34100000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.52800000000000002,"15min":0.504},"min":{"1min":0.41099999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.53000000000000003,"15min":0.5},"min":{"1min":0.48899999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.48399999999999999}]},{"osd":3,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.54100000000000004,"15min":0.51900000000000002},"min":{"1min":0.52000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.629},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.54400000000000004,"15min":0.52200000000000002},"min":{"1min":0.39300000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:47:16 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.56399999999999995,"15min":0.53600000000000003},"min":{"1min":0.42399999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78700000000000003,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.55800000000000005,"15min":0.52400000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.61299999999999999,"15min":0.58199999999999996},"min":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.65700000000000003,"15min":0.61899999999999999},"min":{"1min":0.45100000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.52900000000000003,"15min":0.51700000000000002},"min":{"1min":0.42199999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.58199999999999996,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.43099999999999999}]},{"osd":8,"last update":"Thu Nov 16 16:47:24 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.71599999999999997},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.7650000000000001,"5min":2.7650000000000001,"15min":2.7650000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.751,"15min":0.75},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.0219999999999998,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.57399999999999995}]},{"osd":9,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.79600000000000004,"15min":0.80500000000000005},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.77900000000000003,"15min":0.78100000000000003},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.72399999999999998}]},{"osd":10,"last update":"Thu Nov 16 16:47:11 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.71199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6708,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738444,"statfs":{"total":95995035648,"available":95988166656,"internally_reserved":0,"allocated":1028096,"data_stored":1077786,"data_compressed":53424,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52100000000000002,"15min":0.51500000000000001},"min":{"1min":0.34000000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56200000000000006,"15min":0.56599999999999995},"min":{"1min":0.37,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.67100000000000004}]},{"osd":1,"last update":"Thu Nov 16 16:47:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.55300000000000005,"15min":0.54100000000000004},"min":{"1min":0.34399999999999997,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.034,"5min":1.167,"15min":1.167},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.52700000000000002,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Nov 16 16:47:18 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.376,"15min":0.36799999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.495},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.40100000000000002,"15min":0.38600000000000001},"min":{"1min":0.27500000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.46300000000000002}]},{"osd":3,"last update":"Thu Nov 16 16:47:18 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.54200000000000004,"15min":0.53100000000000003},"min":{"1min":0.34499999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58399999999999996,"15min":0.56699999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.77700000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:47:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.40000000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.53600000000000003,"15min":0.52000000000000002},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.55900000000000005}]},{"osd":6,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.63200000000000001,"15min":0.63},"min":{"1min":0.436,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91200000000000003,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Nov 16 16:47:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.60999999999999999,"15min":0.60399999999999998},"min":{"1min":0.44700000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.63300000000000001}]},{"osd":8,"last update":"Thu Nov 16 16:47:25 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.68500000000000005}]},{"osd":9,"last update":"Thu Nov 16 16:47:50 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.748,"15min":0.76100000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.86799999999999999,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.749,"15min":0.75900000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.871,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.79600000000000004}]},{"osd":10,"last update":"Thu Nov 16 16:47:08 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.751}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":21175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":38549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T16:47:54.632 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-16T16:47:54.632 INFO:tasks.cephadm:Setting up client nodes... 2023-11-16T16:47:54.632 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-11-16T16:47:55.531 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:47:55.957 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:55 smithi044 ceph-mon[128750]: from='client.24844 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:47:55.957 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:55 smithi044 ceph-mon[128750]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 93 B/s wr, 1 op/s; 2 B/s, 0 objects/s recovering 2023-11-16T16:47:55.957 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:55 smithi044 ceph-mon[128750]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:47:55.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:55 smithi077 ceph-mon[131204]: from='client.24844 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:47:55.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:55 smithi077 ceph-mon[131204]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 93 B/s wr, 1 op/s; 2 B/s, 0 objects/s recovering 2023-11-16T16:47:55.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:55 smithi077 ceph-mon[131204]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:47:56.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:55 smithi176 ceph-mon[131630]: from='client.24844 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:47:56.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:55 smithi176 ceph-mon[131630]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 93 B/s wr, 1 op/s; 2 B/s, 0 objects/s recovering 2023-11-16T16:47:56.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:55 smithi176 ceph-mon[131630]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:47:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.052+0000 7fc0bc314700 1 -- 172.21.15.44:0/1613302152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0b4101e80 msgr2=0x7fc0b4108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.052+0000 7fc0bc314700 1 --2- 172.21.15.44:0/1613302152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0b4101e80 0x7fc0b4108740 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac00b0a0 tx=0x7fc0ac00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.052+0000 7fc0bc314700 1 -- 172.21.15.44:0/1613302152 shutdown_connections 2023-11-16T16:47:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.052+0000 7fc0bc314700 1 --2- 172.21.15.44:0/1613302152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0b4101e80 0x7fc0b4108740 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.052+0000 7fc0bc314700 1 --2- 172.21.15.44:0/1613302152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0b41002f0 0x7fc0b4100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.052+0000 7fc0bc314700 1 --2- 172.21.15.44:0/1613302152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b41014f0 0x7fc0b4101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.052+0000 7fc0bc314700 1 -- 172.21.15.44:0/1613302152 >> 172.21.15.44:0/1613302152 conn(0x7fc0b40fb900 msgr2=0x7fc0b40fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:57.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.053+0000 7fc0bc314700 1 -- 172.21.15.44:0/1613302152 shutdown_connections 2023-11-16T16:47:57.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.053+0000 7fc0bc314700 1 -- 172.21.15.44:0/1613302152 wait complete. 2023-11-16T16:47:57.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.054+0000 7fc0bc314700 1 Processor -- start 2023-11-16T16:47:57.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.054+0000 7fc0bc314700 1 -- start start 2023-11-16T16:47:57.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.054+0000 7fc0bc314700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0b41002f0 0x7fc0b4198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:57.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.054+0000 7fc0bc314700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b41014f0 0x7fc0b4198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:57.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.054+0000 7fc0bc314700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0b4101e80 0x7fc0b419f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:57.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.054+0000 7fc0bc314700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b419fed0 con 0x7fc0b41002f0 2023-11-16T16:47:57.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.054+0000 7fc0bc314700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b41a0010 con 0x7fc0b41014f0 2023-11-16T16:47:57.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.054+0000 7fc0bc314700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b41a01a0 con 0x7fc0b4101e80 2023-11-16T16:47:57.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.054+0000 7fc0ba0b0700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0b41002f0 0x7fc0b4198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:57.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.055+0000 7fc0ba0b0700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0b41002f0 0x7fc0b4198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50162/0 (socket says 172.21.15.44:50162) 2023-11-16T16:47:57.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.055+0000 7fc0ba0b0700 1 -- 172.21.15.44:0/3825583121 learned_addr learned my addr 172.21.15.44:0/3825583121 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:47:57.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.055+0000 7fc0b98af700 1 --2- 172.21.15.44:0/3825583121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b41014f0 0x7fc0b4198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:57.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.055+0000 7fc0ba8b1700 1 --2- 172.21.15.44:0/3825583121 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0b4101e80 0x7fc0b419f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.055+0000 7fc0ba0b0700 1 -- 172.21.15.44:0/3825583121 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0b4101e80 msgr2=0x7fc0b419f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.055+0000 7fc0ba0b0700 1 --2- 172.21.15.44:0/3825583121 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0b4101e80 0x7fc0b419f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.055+0000 7fc0ba0b0700 1 -- 172.21.15.44:0/3825583121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b41014f0 msgr2=0x7fc0b4198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.055+0000 7fc0ba0b0700 1 --2- 172.21.15.44:0/3825583121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b41014f0 0x7fc0b4198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.055+0000 7fc0ba0b0700 1 -- 172.21.15.44:0/3825583121 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ac009d00 con 0x7fc0b41002f0 2023-11-16T16:47:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.055+0000 7fc0ba0b0700 1 --2- 172.21.15.44:0/3825583121 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0b41002f0 0x7fc0b4198340 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000d9b0 tx=0x7fc0b000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.056+0000 7fc0ab7fe700 1 -- 172.21.15.44:0/3825583121 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b000b950 con 0x7fc0b41002f0 2023-11-16T16:47:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.056+0000 7fc0ab7fe700 1 -- 172.21.15.44:0/3825583121 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0b0007620 con 0x7fc0b41002f0 2023-11-16T16:47:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.056+0000 7fc0ab7fe700 1 -- 172.21.15.44:0/3825583121 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b000f3c0 con 0x7fc0b41002f0 2023-11-16T16:47:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.056+0000 7fc0bc314700 1 -- 172.21.15.44:0/3825583121 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b410bcb0 con 0x7fc0b41002f0 2023-11-16T16:47:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.056+0000 7fc0bc314700 1 -- 172.21.15.44:0/3825583121 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b410bb20 con 0x7fc0b41002f0 2023-11-16T16:47:57.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.057+0000 7fc0bc314700 1 -- 172.21.15.44:0/3825583121 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b41a0830 con 0x7fc0b41002f0 2023-11-16T16:47:57.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.059+0000 7fc0ab7fe700 1 -- 172.21.15.44:0/3825583121 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc0b0010460 con 0x7fc0b41002f0 2023-11-16T16:47:57.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.060+0000 7fc0ab7fe700 1 --2- 172.21.15.44:0/3825583121 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc0a00a11e0 0x7fc0a00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:57.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.060+0000 7fc0ab7fe700 1 -- 172.21.15.44:0/3825583121 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc0b00ccc10 con 0x7fc0b41002f0 2023-11-16T16:47:57.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.061+0000 7fc0b98af700 1 --2- 172.21.15.44:0/3825583121 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc0a00a11e0 0x7fc0a00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:57.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.061+0000 7fc0b98af700 1 --2- 172.21.15.44:0/3825583121 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc0a00a11e0 0x7fc0a00a3690 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4009be0 tx=0x7fc0a4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:57.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.064+0000 7fc0ab7fe700 1 -- 172.21.15.44:0/3825583121 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc0b0095e90 con 0x7fc0b41002f0 2023-11-16T16:47:57.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.354+0000 7fc0bc314700 1 -- 172.21.15.44:0/3825583121 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fc0b41a0e80 con 0x7fc0b41002f0 2023-11-16T16:47:57.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.359+0000 7fc0ab7fe700 1 -- 172.21.15.44:0/3825583121 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7fc0b0016020 con 0x7fc0b41002f0 2023-11-16T16:47:57.361 INFO:teuthology.orchestra.run.smithi044.stdout:[client.0] 2023-11-16T16:47:57.361 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQC9R1ZlrM08FRAABE4Xy68TXR+jCvcuTpiZIA== 2023-11-16T16:47:57.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.361+0000 7fc0bc314700 1 -- 172.21.15.44:0/3825583121 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc0a00a11e0 msgr2=0x7fc0a00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:57.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.361+0000 7fc0bc314700 1 --2- 172.21.15.44:0/3825583121 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc0a00a11e0 0x7fc0a00a3690 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4009be0 tx=0x7fc0a4009260 comp rx=0 tx=0).stop 2023-11-16T16:47:57.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.361+0000 7fc0bc314700 1 -- 172.21.15.44:0/3825583121 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0b41002f0 msgr2=0x7fc0b4198340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:57.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.361+0000 7fc0bc314700 1 --2- 172.21.15.44:0/3825583121 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0b41002f0 0x7fc0b4198340 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000d9b0 tx=0x7fc0b000dcc0 comp rx=0 tx=0).stop 2023-11-16T16:47:57.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.362+0000 7fc0bc314700 1 -- 172.21.15.44:0/3825583121 shutdown_connections 2023-11-16T16:47:57.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.362+0000 7fc0bc314700 1 --2- 172.21.15.44:0/3825583121 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc0a00a11e0 0x7fc0a00a3690 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:57.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.362+0000 7fc0bc314700 1 --2- 172.21.15.44:0/3825583121 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0b4101e80 0x7fc0b419f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:57.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.362+0000 7fc0bc314700 1 --2- 172.21.15.44:0/3825583121 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0b41002f0 0x7fc0b4198340 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:57.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.362+0000 7fc0bc314700 1 --2- 172.21.15.44:0/3825583121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b41014f0 0x7fc0b4198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:57.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.362+0000 7fc0bc314700 1 -- 172.21.15.44:0/3825583121 >> 172.21.15.44:0/3825583121 conn(0x7fc0b40fb900 msgr2=0x7fc0b4106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:57.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.362+0000 7fc0bc314700 1 -- 172.21.15.44:0/3825583121 shutdown_connections 2023-11-16T16:47:57.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:47:57.362+0000 7fc0bc314700 1 -- 172.21.15.44:0/3825583121 wait complete. 2023-11-16T16:47:57.906 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:57 smithi044 ceph-mon[128750]: pgmap v296: 129 pgs: 129 active+clean; 615 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.5 KiB/s wr, 3 op/s; 1 B/s, 0 objects/s recovering 2023-11-16T16:47:57.906 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:57 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3825583121' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-16T16:47:57.906 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:57 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3825583121' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-16T16:47:57.932 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:47:57.932 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-11-16T16:47:57.932 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-16T16:47:57.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:57 smithi077 ceph-mon[131204]: pgmap v296: 129 pgs: 129 active+clean; 615 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.5 KiB/s wr, 3 op/s; 1 B/s, 0 objects/s recovering 2023-11-16T16:47:57.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:57 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3825583121' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-16T16:47:57.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:57 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3825583121' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-16T16:47:58.036 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-16T16:47:58.036 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-11-16T16:47:58.036 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph mgr dump --format=json 2023-11-16T16:47:58.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:57 smithi176 ceph-mon[131630]: pgmap v296: 129 pgs: 129 active+clean; 615 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.5 KiB/s wr, 3 op/s; 1 B/s, 0 objects/s recovering 2023-11-16T16:47:58.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:57 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3825583121' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-16T16:47:58.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:57 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3825583121' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-16T16:47:58.938 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:47:59.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:47:59 smithi077 ceph-mon[131204]: pgmap v297: 129 pgs: 129 active+clean; 615 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.5 KiB/s wr, 3 op/s 2023-11-16T16:47:59.538 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:47:59 smithi044 ceph-mon[128750]: pgmap v297: 129 pgs: 129 active+clean; 615 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.5 KiB/s wr, 3 op/s 2023-11-16T16:47:59.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:47:59 smithi176 ceph-mon[131630]: pgmap v297: 129 pgs: 129 active+clean; 615 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.5 KiB/s wr, 3 op/s 2023-11-16T16:48:00.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.451+0000 7fe87cdb2700 1 -- 172.21.15.44:0/3306469110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8781055e0 msgr2=0x7fe87810d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:00.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.451+0000 7fe87cdb2700 1 --2- 172.21.15.44:0/3306469110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8781055e0 0x7fe87810d160 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe87000b0d0 tx=0x7fe87000b3e0 comp rx=0 tx=0).stop 2023-11-16T16:48:00.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.451+0000 7fe87cdb2700 1 -- 172.21.15.44:0/3306469110 shutdown_connections 2023-11-16T16:48:00.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.451+0000 7fe87cdb2700 1 --2- 172.21.15.44:0/3306469110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8781055e0 0x7fe87810d160 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:00.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.451+0000 7fe87cdb2700 1 --2- 172.21.15.44:0/3306469110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe878100540 0x7fe878100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:00.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.451+0000 7fe87cdb2700 1 --2- 172.21.15.44:0/3306469110 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe878100ee0 0x7fe878104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:00.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.451+0000 7fe87cdb2700 1 -- 172.21.15.44:0/3306469110 >> 172.21.15.44:0/3306469110 conn(0x7fe8780fbdc0 msgr2=0x7fe8780fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:00.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.451+0000 7fe87cdb2700 1 -- 172.21.15.44:0/3306469110 shutdown_connections 2023-11-16T16:48:00.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.451+0000 7fe87cdb2700 1 -- 172.21.15.44:0/3306469110 wait complete. 2023-11-16T16:48:00.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.452+0000 7fe87cdb2700 1 Processor -- start 2023-11-16T16:48:00.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.453+0000 7fe87cdb2700 1 -- start start 2023-11-16T16:48:00.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.453+0000 7fe87cdb2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe878100540 0x7fe878196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:00.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.453+0000 7fe87cdb2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe878100ee0 0x7fe878196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:00.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.453+0000 7fe87cdb2700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8781055e0 0x7fe87819aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:00.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.453+0000 7fe87cdb2700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87819b500 con 0x7fe8781055e0 2023-11-16T16:48:00.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.453+0000 7fe87cdb2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87819b670 con 0x7fe878100540 2023-11-16T16:48:00.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.453+0000 7fe87cdb2700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87819b830 con 0x7fe878100ee0 2023-11-16T16:48:00.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.454+0000 7fe87659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe878100540 0x7fe878196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:00.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.454+0000 7fe875d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe878100ee0 0x7fe878196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:00.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.454+0000 7fe876d9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8781055e0 0x7fe87819aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:00.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.454+0000 7fe87659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe878100540 0x7fe878196540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59790/0 (socket says 172.21.15.44:59790) 2023-11-16T16:48:00.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.454+0000 7fe87659c700 1 -- 172.21.15.44:0/3984430282 learned_addr learned my addr 172.21.15.44:0/3984430282 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:00.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.454+0000 7fe87659c700 1 -- 172.21.15.44:0/3984430282 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe878100ee0 msgr2=0x7fe878196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:00.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.454+0000 7fe87659c700 1 --2- 172.21.15.44:0/3984430282 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe878100ee0 0x7fe878196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:00.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.454+0000 7fe87659c700 1 -- 172.21.15.44:0/3984430282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8781055e0 msgr2=0x7fe87819aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:00.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.454+0000 7fe87659c700 1 --2- 172.21.15.44:0/3984430282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8781055e0 0x7fe87819aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:00.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.454+0000 7fe87659c700 1 -- 172.21.15.44:0/3984430282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe870009d00 con 0x7fe878100540 2023-11-16T16:48:00.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.454+0000 7fe87659c700 1 --2- 172.21.15.44:0/3984430282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe878100540 0x7fe878196540 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe86800e9a0 tx=0x7fe86800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:00.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.455+0000 7fe8677fe700 1 -- 172.21.15.44:0/3984430282 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe868009790 con 0x7fe878100540 2023-11-16T16:48:00.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.455+0000 7fe8677fe700 1 -- 172.21.15.44:0/3984430282 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe868010430 con 0x7fe878100540 2023-11-16T16:48:00.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.455+0000 7fe87cdb2700 1 -- 172.21.15.44:0/3984430282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe878110370 con 0x7fe878100540 2023-11-16T16:48:00.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.455+0000 7fe8677fe700 1 -- 172.21.15.44:0/3984430282 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe868018680 con 0x7fe878100540 2023-11-16T16:48:00.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.455+0000 7fe87cdb2700 1 -- 172.21.15.44:0/3984430282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8781101e0 con 0x7fe878100540 2023-11-16T16:48:00.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.456+0000 7fe87cdb2700 1 -- 172.21.15.44:0/3984430282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe878110500 con 0x7fe878100540 2023-11-16T16:48:00.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.458+0000 7fe8677fe700 1 -- 172.21.15.44:0/3984430282 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe8680105a0 con 0x7fe878100540 2023-11-16T16:48:00.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.459+0000 7fe8677fe700 1 --2- 172.21.15.44:0/3984430282 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe8600a11e0 0x7fe8600a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:00.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.460+0000 7fe8677fe700 1 -- 172.21.15.44:0/3984430282 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe868014070 con 0x7fe878100540 2023-11-16T16:48:00.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.460+0000 7fe875d9b700 1 --2- 172.21.15.44:0/3984430282 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe8600a11e0 0x7fe8600a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:00.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.460+0000 7fe875d9b700 1 --2- 172.21.15.44:0/3984430282 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe8600a11e0 0x7fe8600a3690 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe878197b60 tx=0x7fe86c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:00.465 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.463+0000 7fe8677fe700 1 -- 172.21.15.44:0/3984430282 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe868051730 con 0x7fe878100540 2023-11-16T16:48:00.732 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.731+0000 7fe87cdb2700 1 -- 172.21.15.44:0/3984430282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe8781a3620 con 0x7fe878100540 2023-11-16T16:48:00.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.736+0000 7fe8677fe700 1 -- 172.21.15.44:0/3984430282 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260123 (secure 0 0 0) 0x7fe8680d0030 con 0x7fe878100540 2023-11-16T16:48:00.741 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:48:00.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.740+0000 7fe87cdb2700 1 -- 172.21.15.44:0/3984430282 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe8600a11e0 msgr2=0x7fe8600a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:00.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.740+0000 7fe87cdb2700 1 --2- 172.21.15.44:0/3984430282 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe8600a11e0 0x7fe8600a3690 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe878197b60 tx=0x7fe86c009260 comp rx=0 tx=0).stop 2023-11-16T16:48:00.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.740+0000 7fe87cdb2700 1 -- 172.21.15.44:0/3984430282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe878100540 msgr2=0x7fe878196540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:00.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.740+0000 7fe87cdb2700 1 --2- 172.21.15.44:0/3984430282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe878100540 0x7fe878196540 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe86800e9a0 tx=0x7fe86800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:48:00.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.740+0000 7fe87cdb2700 1 -- 172.21.15.44:0/3984430282 shutdown_connections 2023-11-16T16:48:00.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.740+0000 7fe87cdb2700 1 --2- 172.21.15.44:0/3984430282 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe8600a11e0 0x7fe8600a3690 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:00.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.740+0000 7fe87cdb2700 1 --2- 172.21.15.44:0/3984430282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8781055e0 0x7fe87819aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:00.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.740+0000 7fe87cdb2700 1 --2- 172.21.15.44:0/3984430282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe878100540 0x7fe878196540 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:00.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.740+0000 7fe87cdb2700 1 --2- 172.21.15.44:0/3984430282 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe878100ee0 0x7fe878196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:00.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.740+0000 7fe87cdb2700 1 -- 172.21.15.44:0/3984430282 >> 172.21.15.44:0/3984430282 conn(0x7fe8780fbdc0 msgr2=0x7fe8780fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:00.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.740+0000 7fe87cdb2700 1 -- 172.21.15.44:0/3984430282 shutdown_connections 2023-11-16T16:48:00.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:00.740+0000 7fe87cdb2700 1 -- 172.21.15.44:0/3984430282 wait complete. 2023-11-16T16:48:01.239 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":17,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":304422349},{"type":"v1","addr":"172.21.15.44:6801","nonce":304422349}]},"active_addr":"172.21.15.44:6801/304422349","active_change":"2023-11-16T16:40:42.449626+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.44:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":4218362219}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":2937650254}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":1983838729}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":1915598320}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":3442862721}]}]} 2023-11-16T16:48:01.244 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-11-16T16:48:01.244 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-11-16T16:48:01.244 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd dump --format=json 2023-11-16T16:48:01.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:01 smithi044 ceph-mon[128750]: pgmap v298: 129 pgs: 129 active+clean; 628 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.8 KiB/s wr, 4 op/s 2023-11-16T16:48:01.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:01 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3984430282' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-11-16T16:48:01.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:01 smithi176 ceph-mon[131630]: pgmap v298: 129 pgs: 129 active+clean; 628 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.8 KiB/s wr, 4 op/s 2023-11-16T16:48:01.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:01 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3984430282' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-11-16T16:48:01.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:01 smithi077 ceph-mon[131204]: pgmap v298: 129 pgs: 129 active+clean; 628 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.8 KiB/s wr, 4 op/s 2023-11-16T16:48:01.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:01 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3984430282' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-11-16T16:48:02.126 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:03.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.743+0000 7f5d4606f700 1 -- 172.21.15.44:0/1488977908 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d40100ee0 msgr2=0x7f5d40104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:03.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.743+0000 7f5d4606f700 1 --2- 172.21.15.44:0/1488977908 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d40100ee0 0x7f5d40104eb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009920 tx=0x7f5d34009c30 comp rx=0 tx=0).stop 2023-11-16T16:48:03.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.744+0000 7f5d4606f700 1 -- 172.21.15.44:0/1488977908 shutdown_connections 2023-11-16T16:48:03.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.744+0000 7f5d4606f700 1 --2- 172.21.15.44:0/1488977908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d401055e0 0x7f5d4010d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:03.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.744+0000 7f5d4606f700 1 --2- 172.21.15.44:0/1488977908 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d40100540 0x7f5d40100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:03.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.744+0000 7f5d4606f700 1 --2- 172.21.15.44:0/1488977908 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d40100ee0 0x7f5d40104eb0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:03.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.744+0000 7f5d4606f700 1 -- 172.21.15.44:0/1488977908 >> 172.21.15.44:0/1488977908 conn(0x7f5d400fbdc0 msgr2=0x7f5d400fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:03.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.744+0000 7f5d4606f700 1 -- 172.21.15.44:0/1488977908 shutdown_connections 2023-11-16T16:48:03.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.744+0000 7f5d4606f700 1 -- 172.21.15.44:0/1488977908 wait complete. 2023-11-16T16:48:03.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.745+0000 7f5d4606f700 1 Processor -- start 2023-11-16T16:48:03.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.745+0000 7f5d4606f700 1 -- start start 2023-11-16T16:48:03.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.745+0000 7f5d4606f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d40100540 0x7f5d40198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:03.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.745+0000 7f5d4606f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d40100ee0 0x7f5d40198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:03.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.745+0000 7f5d4606f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d401055e0 0x7f5d4019bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:03.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.745+0000 7f5d4606f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4019c580 con 0x7f5d40100ee0 2023-11-16T16:48:03.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.745+0000 7f5d4606f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4019c6f0 con 0x7f5d401055e0 2023-11-16T16:48:03.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.745+0000 7f5d4606f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4019c8b0 con 0x7f5d40100540 2023-11-16T16:48:03.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.746+0000 7f5d3effd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d40100ee0 0x7f5d40198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:03.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.746+0000 7f5d3effd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d40100ee0 0x7f5d40198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50206/0 (socket says 172.21.15.44:50206) 2023-11-16T16:48:03.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.746+0000 7f5d3effd700 1 -- 172.21.15.44:0/1393310112 learned_addr learned my addr 172.21.15.44:0/1393310112 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:03.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.746+0000 7f5d3f7fe700 1 --2- 172.21.15.44:0/1393310112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d40100540 0x7f5d40198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:03.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.746+0000 7f5d3ffff700 1 --2- 172.21.15.44:0/1393310112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d401055e0 0x7f5d4019bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:03.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.746+0000 7f5d3effd700 1 -- 172.21.15.44:0/1393310112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d40100540 msgr2=0x7f5d40198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:03.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.746+0000 7f5d3effd700 1 --2- 172.21.15.44:0/1393310112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d40100540 0x7f5d40198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:03.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.746+0000 7f5d3effd700 1 -- 172.21.15.44:0/1393310112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d401055e0 msgr2=0x7f5d4019bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:03.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.746+0000 7f5d3effd700 1 --2- 172.21.15.44:0/1393310112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d401055e0 0x7f5d4019bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:03.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.746+0000 7f5d3effd700 1 -- 172.21.15.44:0/1393310112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d38009d00 con 0x7f5d40100ee0 2023-11-16T16:48:03.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.746+0000 7f5d3f7fe700 1 --2- 172.21.15.44:0/1393310112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d40100540 0x7f5d40198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:03.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.747+0000 7f5d3effd700 1 --2- 172.21.15.44:0/1393310112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d40100ee0 0x7f5d40198bc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009230 tx=0x7f5d3400f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:03.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.747+0000 7f5d3cff9700 1 -- 172.21.15.44:0/1393310112 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3401c070 con 0x7f5d40100ee0 2023-11-16T16:48:03.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.747+0000 7f5d3cff9700 1 -- 172.21.15.44:0/1393310112 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d3400baa0 con 0x7f5d40100ee0 2023-11-16T16:48:03.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.747+0000 7f5d4606f700 1 -- 172.21.15.44:0/1393310112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d34009520 con 0x7f5d40100ee0 2023-11-16T16:48:03.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.747+0000 7f5d3cff9700 1 -- 172.21.15.44:0/1393310112 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d340038e0 con 0x7f5d40100ee0 2023-11-16T16:48:03.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.747+0000 7f5d4606f700 1 -- 172.21.15.44:0/1393310112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d40110360 con 0x7f5d40100ee0 2023-11-16T16:48:03.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.748+0000 7f5d4606f700 1 -- 172.21.15.44:0/1393310112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d401101d0 con 0x7f5d40100ee0 2023-11-16T16:48:03.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.750+0000 7f5d3cff9700 1 -- 172.21.15.44:0/1393310112 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5d34020030 con 0x7f5d40100ee0 2023-11-16T16:48:03.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.751+0000 7f5d3cff9700 1 --2- 172.21.15.44:0/1393310112 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5d280a11e0 0x7f5d280a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:03.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.751+0000 7f5d3cff9700 1 -- 172.21.15.44:0/1393310112 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5d340cdc30 con 0x7f5d40100ee0 2023-11-16T16:48:03.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.752+0000 7f5d3f7fe700 1 --2- 172.21.15.44:0/1393310112 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5d280a11e0 0x7f5d280a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:03.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.752+0000 7f5d3f7fe700 1 --2- 172.21.15.44:0/1393310112 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5d280a11e0 0x7f5d280a3690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5d30011490 tx=0x7f5d30009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:03.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.756+0000 7f5d3cff9700 1 -- 172.21.15.44:0/1393310112 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5d34096f40 con 0x7f5d40100ee0 2023-11-16T16:48:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:03 smithi044 ceph-mon[128750]: pgmap v299: 129 pgs: 129 active+clean; 628 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.9 KiB/s wr, 4 op/s 2023-11-16T16:48:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:03 smithi176 ceph-mon[131630]: pgmap v299: 129 pgs: 129 active+clean; 628 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.9 KiB/s wr, 4 op/s 2023-11-16T16:48:03.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.965+0000 7f5d4606f700 1 -- 172.21.15.44:0/1393310112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d401104f0 con 0x7f5d40100ee0 2023-11-16T16:48:03.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.967+0000 7f5d3cff9700 1 -- 172.21.15.44:0/1393310112 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24451 (secure 0 0 0) 0x7f5d3409a560 con 0x7f5d40100ee0 2023-11-16T16:48:03.970 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:48:03.970 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":89,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:47:43.617828+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:48:03.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.969+0000 7f5d4606f700 1 -- 172.21.15.44:0/1393310112 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5d280a11e0 msgr2=0x7f5d280a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:03.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.969+0000 7f5d4606f700 1 --2- 172.21.15.44:0/1393310112 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5d280a11e0 0x7f5d280a3690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5d30011490 tx=0x7f5d30009260 comp rx=0 tx=0).stop 2023-11-16T16:48:03.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.969+0000 7f5d4606f700 1 -- 172.21.15.44:0/1393310112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d40100ee0 msgr2=0x7f5d40198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:03.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.969+0000 7f5d4606f700 1 --2- 172.21.15.44:0/1393310112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d40100ee0 0x7f5d40198bc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009230 tx=0x7f5d3400f9e0 comp rx=0 tx=0).stop 2023-11-16T16:48:03.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.969+0000 7f5d4606f700 1 -- 172.21.15.44:0/1393310112 shutdown_connections 2023-11-16T16:48:03.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.969+0000 7f5d4606f700 1 --2- 172.21.15.44:0/1393310112 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5d280a11e0 0x7f5d280a3690 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:03.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.969+0000 7f5d4606f700 1 --2- 172.21.15.44:0/1393310112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d401055e0 0x7f5d4019bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:03.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.969+0000 7f5d4606f700 1 --2- 172.21.15.44:0/1393310112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d40100540 0x7f5d40198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:03.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.969+0000 7f5d4606f700 1 --2- 172.21.15.44:0/1393310112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d40100ee0 0x7f5d40198bc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:03.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.969+0000 7f5d4606f700 1 -- 172.21.15.44:0/1393310112 >> 172.21.15.44:0/1393310112 conn(0x7f5d400fbdc0 msgr2=0x7f5d400fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:03.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.969+0000 7f5d4606f700 1 -- 172.21.15.44:0/1393310112 shutdown_connections 2023-11-16T16:48:03.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:03.970+0000 7f5d4606f700 1 -- 172.21.15.44:0/1393310112 wait complete. 2023-11-16T16:48:03.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:03 smithi077 ceph-mon[131204]: pgmap v299: 129 pgs: 129 active+clean; 628 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.9 KiB/s wr, 4 op/s 2023-11-16T16:48:04.430 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-11-16T16:48:04.431 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd dump --format=json 2023-11-16T16:48:04.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:04 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1393310112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:04.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:04 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1393310112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:04.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:04 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1393310112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:05.292 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:05.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:05 smithi044 ceph-mon[128750]: pgmap v300: 129 pgs: 129 active+clean; 628 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 4.9 KiB/s wr, 4 op/s 2023-11-16T16:48:05.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:05 smithi176 ceph-mon[131630]: pgmap v300: 129 pgs: 129 active+clean; 628 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 4.9 KiB/s wr, 4 op/s 2023-11-16T16:48:05.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:05 smithi077 ceph-mon[131204]: pgmap v300: 129 pgs: 129 active+clean; 628 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 4.9 KiB/s wr, 4 op/s 2023-11-16T16:48:06.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.809+0000 7f494b061700 1 -- 172.21.15.44:0/3606200076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4944100ee0 msgr2=0x7f4944104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:06.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.809+0000 7f494b061700 1 --2- 172.21.15.44:0/3606200076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4944100ee0 0x7f4944104eb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f49300099c0 tx=0x7f4930009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:06.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.809+0000 7f494b061700 1 -- 172.21.15.44:0/3606200076 shutdown_connections 2023-11-16T16:48:06.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.809+0000 7f494b061700 1 --2- 172.21.15.44:0/3606200076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49441055e0 0x7f494410d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:06.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.809+0000 7f494b061700 1 --2- 172.21.15.44:0/3606200076 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4944100540 0x7f4944100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:06.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.809+0000 7f494b061700 1 --2- 172.21.15.44:0/3606200076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4944100ee0 0x7f4944104eb0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:06.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.809+0000 7f494b061700 1 -- 172.21.15.44:0/3606200076 >> 172.21.15.44:0/3606200076 conn(0x7f49440fbdc0 msgr2=0x7f49440fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:06.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.809+0000 7f494b061700 1 -- 172.21.15.44:0/3606200076 shutdown_connections 2023-11-16T16:48:06.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.809+0000 7f494b061700 1 -- 172.21.15.44:0/3606200076 wait complete. 2023-11-16T16:48:06.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.811+0000 7f494b061700 1 Processor -- start 2023-11-16T16:48:06.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.811+0000 7f494b061700 1 -- start start 2023-11-16T16:48:06.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.811+0000 7f494b061700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4944100540 0x7f49441963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:06.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.811+0000 7f494b061700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4944100ee0 0x7f4944196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:06.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.811+0000 7f494b061700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49441055e0 0x7f494419ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:06.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.811+0000 7f494b061700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494419b390 con 0x7f4944100ee0 2023-11-16T16:48:06.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.811+0000 7f494b061700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494419b500 con 0x7f4944100540 2023-11-16T16:48:06.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.811+0000 7f494b061700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494419b6c0 con 0x7f49441055e0 2023-11-16T16:48:06.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.812+0000 7f4943fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4944100ee0 0x7f4944196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:06.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.812+0000 7f4943fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4944100ee0 0x7f4944196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33218/0 (socket says 172.21.15.44:33218) 2023-11-16T16:48:06.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.812+0000 7f4943fff700 1 -- 172.21.15.44:0/78374695 learned_addr learned my addr 172.21.15.44:0/78374695 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:06.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.812+0000 7f4948dfd700 1 --2- 172.21.15.44:0/78374695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4944100540 0x7f49441963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:06.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.812+0000 7f49495fe700 1 --2- 172.21.15.44:0/78374695 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49441055e0 0x7f494419ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:06.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.812+0000 7f4943fff700 1 -- 172.21.15.44:0/78374695 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49441055e0 msgr2=0x7f494419ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:06.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.812+0000 7f4943fff700 1 --2- 172.21.15.44:0/78374695 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49441055e0 0x7f494419ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:06.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.812+0000 7f4943fff700 1 -- 172.21.15.44:0/78374695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4944100540 msgr2=0x7f49441963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:06.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.812+0000 7f4943fff700 1 --2- 172.21.15.44:0/78374695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4944100540 0x7f49441963d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:06.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.812+0000 7f4943fff700 1 -- 172.21.15.44:0/78374695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49300095f0 con 0x7f4944100ee0 2023-11-16T16:48:06.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.812+0000 7f4948dfd700 1 --2- 172.21.15.44:0/78374695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4944100540 0x7f49441963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:06.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.813+0000 7f4943fff700 1 --2- 172.21.15.44:0/78374695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4944100ee0 0x7f4944196910 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f49300098e0 tx=0x7f493000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:06.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.813+0000 7f4941ffb700 1 -- 172.21.15.44:0/78374695 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493001d070 con 0x7f4944100ee0 2023-11-16T16:48:06.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.813+0000 7f4941ffb700 1 -- 172.21.15.44:0/78374695 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4930022470 con 0x7f4944100ee0 2023-11-16T16:48:06.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.813+0000 7f4941ffb700 1 -- 172.21.15.44:0/78374695 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4930017650 con 0x7f4944100ee0 2023-11-16T16:48:06.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.813+0000 7f494b061700 1 -- 172.21.15.44:0/78374695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4944110360 con 0x7f4944100ee0 2023-11-16T16:48:06.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.813+0000 7f494b061700 1 -- 172.21.15.44:0/78374695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49441101d0 con 0x7f4944100ee0 2023-11-16T16:48:06.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.814+0000 7f494b061700 1 -- 172.21.15.44:0/78374695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49441104f0 con 0x7f4944100ee0 2023-11-16T16:48:06.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.818+0000 7f4941ffb700 1 -- 172.21.15.44:0/78374695 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4930022850 con 0x7f4944100ee0 2023-11-16T16:48:06.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.819+0000 7f4941ffb700 1 --2- 172.21.15.44:0/78374695 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49340a1260 0x7f49340a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:06.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.819+0000 7f4941ffb700 1 -- 172.21.15.44:0/78374695 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f49300cdd10 con 0x7f4944100ee0 2023-11-16T16:48:06.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.819+0000 7f4948dfd700 1 --2- 172.21.15.44:0/78374695 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49340a1260 0x7f49340a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:06.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.820+0000 7f4948dfd700 1 --2- 172.21.15.44:0/78374695 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49340a1260 0x7f49340a3710 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4938011490 tx=0x7f4938009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:06.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:06.821+0000 7f4941ffb700 1 -- 172.21.15.44:0/78374695 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4930097130 con 0x7f4944100ee0 2023-11-16T16:48:07.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:07.029+0000 7f494b061700 1 -- 172.21.15.44:0/78374695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4944066ea0 con 0x7f4944100ee0 2023-11-16T16:48:07.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:07.030+0000 7f4941ffb700 1 -- 172.21.15.44:0/78374695 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24451 (secure 0 0 0) 0x7f493009a750 con 0x7f4944100ee0 2023-11-16T16:48:07.033 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:48:07.033 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":89,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:47:43.617828+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:48:07.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:07.033+0000 7f494b061700 1 -- 172.21.15.44:0/78374695 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49340a1260 msgr2=0x7f49340a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:07.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:07.033+0000 7f494b061700 1 --2- 172.21.15.44:0/78374695 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49340a1260 0x7f49340a3710 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4938011490 tx=0x7f4938009260 comp rx=0 tx=0).stop 2023-11-16T16:48:07.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:07.033+0000 7f494b061700 1 -- 172.21.15.44:0/78374695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4944100ee0 msgr2=0x7f4944196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:07.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:07.033+0000 7f494b061700 1 --2- 172.21.15.44:0/78374695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4944100ee0 0x7f4944196910 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f49300098e0 tx=0x7f493000f690 comp rx=0 tx=0).stop 2023-11-16T16:48:07.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:07.033+0000 7f494b061700 1 -- 172.21.15.44:0/78374695 shutdown_connections 2023-11-16T16:48:07.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:07.033+0000 7f494b061700 1 --2- 172.21.15.44:0/78374695 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49340a1260 0x7f49340a3710 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:07.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:07.033+0000 7f494b061700 1 --2- 172.21.15.44:0/78374695 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49441055e0 0x7f494419ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:07.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:07.033+0000 7f494b061700 1 --2- 172.21.15.44:0/78374695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4944100540 0x7f49441963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:07.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:07.033+0000 7f494b061700 1 --2- 172.21.15.44:0/78374695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4944100ee0 0x7f4944196910 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:07.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:07.033+0000 7f494b061700 1 -- 172.21.15.44:0/78374695 >> 172.21.15.44:0/78374695 conn(0x7f49440fbdc0 msgr2=0x7f49440fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:07.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:07.033+0000 7f494b061700 1 -- 172.21.15.44:0/78374695 shutdown_connections 2023-11-16T16:48:07.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:07.033+0000 7f494b061700 1 -- 172.21.15.44:0/78374695 wait complete. 2023-11-16T16:48:07.498 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.0 flush_pg_stats 2023-11-16T16:48:07.498 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.1 flush_pg_stats 2023-11-16T16:48:07.499 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.2 flush_pg_stats 2023-11-16T16:48:07.499 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.3 flush_pg_stats 2023-11-16T16:48:07.500 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.4 flush_pg_stats 2023-11-16T16:48:07.500 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.5 flush_pg_stats 2023-11-16T16:48:07.501 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.6 flush_pg_stats 2023-11-16T16:48:07.502 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.7 flush_pg_stats 2023-11-16T16:48:07.502 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.8 flush_pg_stats 2023-11-16T16:48:07.503 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.9 flush_pg_stats 2023-11-16T16:48:07.503 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.10 flush_pg_stats 2023-11-16T16:48:07.504 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.11 flush_pg_stats 2023-11-16T16:48:07.745 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:07 smithi044 ceph-mon[128750]: pgmap v301: 129 pgs: 129 active+clean; 636 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.7 KiB/s wr, 4 op/s 2023-11-16T16:48:07.745 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:07 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/78374695' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:07.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:07 smithi176 ceph-mon[131630]: pgmap v301: 129 pgs: 129 active+clean; 636 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.7 KiB/s wr, 4 op/s 2023-11-16T16:48:07.908 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:07 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/78374695' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:07.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:07 smithi077 ceph-mon[131204]: pgmap v301: 129 pgs: 129 active+clean; 636 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.7 KiB/s wr, 4 op/s 2023-11-16T16:48:07.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:07 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/78374695' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:09.365 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:09.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:09 smithi077 ceph-mon[131204]: pgmap v302: 129 pgs: 129 active+clean; 636 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2023-11-16T16:48:09.581 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:09.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:09 smithi044 ceph-mon[128750]: pgmap v302: 129 pgs: 129 active+clean; 636 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2023-11-16T16:48:09.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:09 smithi176 ceph-mon[131630]: pgmap v302: 129 pgs: 129 active+clean; 636 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2023-11-16T16:48:09.667 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:09.887 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:10.091 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:10.533 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:10.536 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:10.762 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:10.809 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:10.810 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:11.043 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:11.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:11 smithi044 ceph-mon[128750]: pgmap v303: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.4 KiB/s wr, 3 op/s 2023-11-16T16:48:11.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:11 smithi176 ceph-mon[131630]: pgmap v303: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.4 KiB/s wr, 3 op/s 2023-11-16T16:48:11.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:11 smithi077 ceph-mon[131204]: pgmap v303: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.4 KiB/s wr, 3 op/s 2023-11-16T16:48:12.133 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.384+0000 7f4ce777b700 1 -- 172.21.15.44:0/1672824291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ce0100540 msgr2=0x7f4ce0100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.384+0000 7f4ce777b700 1 --2- 172.21.15.44:0/1672824291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ce0100540 0x7f4ce0100910 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc0099c0 tx=0x7f4cdc009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.384+0000 7f4ce777b700 1 -- 172.21.15.44:0/1672824291 shutdown_connections 2023-11-16T16:48:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.384+0000 7f4ce777b700 1 --2- 172.21.15.44:0/1672824291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ce01055e0 0x7f4ce010d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.384+0000 7f4ce777b700 1 --2- 172.21.15.44:0/1672824291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ce0100540 0x7f4ce0100910 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.384+0000 7f4ce777b700 1 --2- 172.21.15.44:0/1672824291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ce0100ee0 0x7f4ce0104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.384+0000 7f4ce777b700 1 -- 172.21.15.44:0/1672824291 >> 172.21.15.44:0/1672824291 conn(0x7f4ce00fbdc0 msgr2=0x7f4ce00fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.385+0000 7f4ce777b700 1 -- 172.21.15.44:0/1672824291 shutdown_connections 2023-11-16T16:48:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.385+0000 7f4ce777b700 1 -- 172.21.15.44:0/1672824291 wait complete. 2023-11-16T16:48:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.386+0000 7f4ce777b700 1 Processor -- start 2023-11-16T16:48:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.386+0000 7f4ce777b700 1 -- start start 2023-11-16T16:48:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.386+0000 7f4ce777b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ce0100540 0x7f4ce0072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.386+0000 7f4ce777b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ce0100ee0 0x7f4ce0073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.386+0000 7f4ce777b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ce01055e0 0x7f4ce00738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.386+0000 7f4ce777b700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce01a4890 con 0x7f4ce01055e0 2023-11-16T16:48:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.386+0000 7f4ce777b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce01a4a00 con 0x7f4ce0100540 2023-11-16T16:48:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.386+0000 7f4ce777b700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce01a4bc0 con 0x7f4ce0100ee0 2023-11-16T16:48:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.386+0000 7f4ce5d18700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ce01055e0 0x7f4ce00738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.387+0000 7f4ce5d18700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ce01055e0 0x7f4ce00738b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33240/0 (socket says 172.21.15.44:33240) 2023-11-16T16:48:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.387+0000 7f4ce5d18700 1 -- 172.21.15.44:0/1810545811 learned_addr learned my addr 172.21.15.44:0/1810545811 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.387+0000 7f4ce5517700 1 --2- 172.21.15.44:0/1810545811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ce0100540 0x7f4ce0072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.387+0000 7f4ce4d16700 1 --2- 172.21.15.44:0/1810545811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ce0100ee0 0x7f4ce0073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.387+0000 7f4ce5d18700 1 -- 172.21.15.44:0/1810545811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ce0100ee0 msgr2=0x7f4ce0073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.387+0000 7f4ce5d18700 1 --2- 172.21.15.44:0/1810545811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ce0100ee0 0x7f4ce0073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.387+0000 7f4ce5d18700 1 -- 172.21.15.44:0/1810545811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ce0100540 msgr2=0x7f4ce0072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.387+0000 7f4ce5d18700 1 --2- 172.21.15.44:0/1810545811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ce0100540 0x7f4ce0072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.387+0000 7f4ce5d18700 1 -- 172.21.15.44:0/1810545811 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cdc0095f0 con 0x7f4ce01055e0 2023-11-16T16:48:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.387+0000 7f4ce5d18700 1 --2- 172.21.15.44:0/1810545811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ce01055e0 0x7f4ce00738b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8007ee0 tx=0x7f4cd800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.387+0000 7f4cd67fc700 1 -- 172.21.15.44:0/1810545811 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8017070 con 0x7f4ce01055e0 2023-11-16T16:48:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.387+0000 7f4cd67fc700 1 -- 172.21.15.44:0/1810545811 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4cd800f040 con 0x7f4ce01055e0 2023-11-16T16:48:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.387+0000 7f4cd67fc700 1 -- 172.21.15.44:0/1810545811 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8005580 con 0x7f4ce01055e0 2023-11-16T16:48:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.387+0000 7f4ce777b700 1 -- 172.21.15.44:0/1810545811 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce0110320 con 0x7f4ce01055e0 2023-11-16T16:48:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.388+0000 7f4ce777b700 1 -- 172.21.15.44:0/1810545811 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce0110190 con 0x7f4ce01055e0 2023-11-16T16:48:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.388+0000 7f4ce777b700 1 -- 172.21.15.44:0/1810545811 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4ce01047e0 con 0x7f4ce01055e0 2023-11-16T16:48:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.390+0000 7f4cd67fc700 1 -- 172.21.15.44:0/1810545811 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4cd800db20 con 0x7f4ce01055e0 2023-11-16T16:48:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.392+0000 7f4cd67fc700 1 --2- 172.21.15.44:0/1810545811 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4ccc0a1260 0x7f4ccc0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.392+0000 7f4cd67fc700 1 -- 172.21.15.44:0/1810545811 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4cd80cc450 con 0x7f4ce01055e0 2023-11-16T16:48:13.542 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.392+0000 7f4ce5517700 1 --2- 172.21.15.44:0/1810545811 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4ccc0a1260 0x7f4ccc0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:13.542 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.392+0000 7f4cd67fc700 1 --2- 172.21.15.44:0/1810545811 >> [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342] conn(0x7f4ccc0ac270 0x7f4ccc0ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:13.542 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.392+0000 7f4cd67fc700 1 -- 172.21.15.44:0/1810545811 --> [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ccc0aedd0 con 0x7f4ccc0ac270 2023-11-16T16:48:13.542 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.392+0000 7f4cd67fc700 1 -- 172.21.15.44:0/1810545811 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4cd80cea30 con 0x7f4ce01055e0 2023-11-16T16:48:13.542 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.393+0000 7f4ce5517700 1 --2- 172.21.15.44:0/1810545811 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4ccc0a1260 0x7f4ccc0a3710 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009990 tx=0x7f4cdc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:13.542 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.393+0000 7f4ce4d16700 1 --2- 172.21.15.44:0/1810545811 >> [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342] conn(0x7f4ccc0ac270 0x7f4ccc0ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:13.542 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.393+0000 7f4ce4d16700 1 --2- 172.21.15.44:0/1810545811 >> [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342] conn(0x7f4ccc0ac270 0x7f4ccc0ae700 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:13.542 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.394+0000 7f4cd67fc700 1 -- 172.21.15.44:0/1810545811 <== osd.2 v2:172.21.15.176:6800/1877349342 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f4ccc0aedd0 con 0x7f4ccc0ac270 2023-11-16T16:48:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.411+0000 7f4ce777b700 1 -- 172.21.15.44:0/1810545811 --> [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4ce01104b0 con 0x7f4ccc0ac270 2023-11-16T16:48:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.411+0000 7f4cd67fc700 1 -- 172.21.15.44:0/1810545811 <== osd.2 v2:172.21.15.176:6800/1877349342 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4ce01104b0 con 0x7f4ccc0ac270 2023-11-16T16:48:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.412+0000 7f4ce777b700 1 -- 172.21.15.44:0/1810545811 >> [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342] conn(0x7f4ccc0ac270 msgr2=0x7f4ccc0ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.412+0000 7f4ce777b700 1 --2- 172.21.15.44:0/1810545811 >> [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342] conn(0x7f4ccc0ac270 0x7f4ccc0ae700 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.412+0000 7f4ce777b700 1 -- 172.21.15.44:0/1810545811 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4ccc0a1260 msgr2=0x7f4ccc0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.412+0000 7f4ce777b700 1 --2- 172.21.15.44:0/1810545811 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4ccc0a1260 0x7f4ccc0a3710 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009990 tx=0x7f4cdc005fb0 comp rx=0 tx=0).stop 2023-11-16T16:48:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.412+0000 7f4ce777b700 1 -- 172.21.15.44:0/1810545811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ce01055e0 msgr2=0x7f4ce00738b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.412+0000 7f4ce777b700 1 --2- 172.21.15.44:0/1810545811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ce01055e0 0x7f4ce00738b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8007ee0 tx=0x7f4cd800d3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.412+0000 7f4ce5d18700 1 -- 172.21.15.44:0/1810545811 reap_dead start 2023-11-16T16:48:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.412+0000 7f4ce777b700 1 -- 172.21.15.44:0/1810545811 shutdown_connections 2023-11-16T16:48:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.412+0000 7f4ce777b700 1 -- 172.21.15.44:0/1810545811 >> 172.21.15.44:0/1810545811 conn(0x7f4ce00fbdc0 msgr2=0x7f4ce0103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.412+0000 7f4ce777b700 1 -- 172.21.15.44:0/1810545811 shutdown_connections 2023-11-16T16:48:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:13.412+0000 7f4ce777b700 1 -- 172.21.15.44:0/1810545811 wait complete. 2023-11-16T16:48:13.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:13 smithi044 ceph-mon[128750]: pgmap v304: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2023-11-16T16:48:13.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:13 smithi176 ceph-mon[131630]: pgmap v304: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2023-11-16T16:48:13.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:13 smithi077 ceph-mon[131204]: pgmap v304: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2023-11-16T16:48:15.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:15 smithi044 ceph-mon[128750]: pgmap v305: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2023-11-16T16:48:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:15 smithi176 ceph-mon[131630]: pgmap v305: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2023-11-16T16:48:15.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:15 smithi077 ceph-mon[131204]: pgmap v305: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2023-11-16T16:48:16.153 INFO:teuthology.orchestra.run.smithi044.stdout:77309411394 2023-11-16T16:48:16.154 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.2 2023-11-16T16:48:16.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.431+0000 7fc87b724700 1 -- 172.21.15.44:0/3636184353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc874101540 msgr2=0x7fc874101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.431+0000 7fc87b724700 1 --2- 172.21.15.44:0/3636184353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc874101540 0x7fc874101910 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc870009990 tx=0x7fc870009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.431+0000 7fc87b724700 1 -- 172.21.15.44:0/3636184353 shutdown_connections 2023-11-16T16:48:16.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.431+0000 7fc87b724700 1 --2- 172.21.15.44:0/3636184353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc87410a890 0x7fc87410ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.431+0000 7fc87b724700 1 --2- 172.21.15.44:0/3636184353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc874101540 0x7fc874101910 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.431+0000 7fc87b724700 1 --2- 172.21.15.44:0/3636184353 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc874101e50 0x7fc87410a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.431+0000 7fc87b724700 1 -- 172.21.15.44:0/3636184353 >> 172.21.15.44:0/3636184353 conn(0x7fc8740770a0 msgr2=0x7fc8740774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:16.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.431+0000 7fc87b724700 1 -- 172.21.15.44:0/3636184353 shutdown_connections 2023-11-16T16:48:16.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.431+0000 7fc87b724700 1 -- 172.21.15.44:0/3636184353 wait complete. 2023-11-16T16:48:16.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.432+0000 7fc87b724700 1 Processor -- start 2023-11-16T16:48:16.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.432+0000 7fc87b724700 1 -- start start 2023-11-16T16:48:16.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.433+0000 7fc87b724700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc874101540 0x7fc87419e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:16.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.433+0000 7fc87b724700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc874101e50 0x7fc87419ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:16.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.433+0000 7fc87b724700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc87410a890 0x7fc874198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:16.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.433+0000 7fc87b724700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc874198be0 con 0x7fc87410a890 2023-11-16T16:48:16.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.433+0000 7fc87b724700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc874198d50 con 0x7fc874101e50 2023-11-16T16:48:16.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.433+0000 7fc87b724700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc874198f10 con 0x7fc874101540 2023-11-16T16:48:16.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.433+0000 7fc8794c0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc874101540 0x7fc87419e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:16.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.433+0000 7fc8794c0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc874101540 0x7fc87419e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:59184/0 (socket says 172.21.15.44:59184) 2023-11-16T16:48:16.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.433+0000 7fc8794c0700 1 -- 172.21.15.44:0/2542247159 learned_addr learned my addr 172.21.15.44:0/2542247159 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:16.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.433+0000 7fc879cc1700 1 --2- 172.21.15.44:0/2542247159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc87410a890 0x7fc874198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:16.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.433+0000 7fc878cbf700 1 --2- 172.21.15.44:0/2542247159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc874101e50 0x7fc87419ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:16.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.434+0000 7fc8794c0700 1 -- 172.21.15.44:0/2542247159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc874101e50 msgr2=0x7fc87419ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.434+0000 7fc8794c0700 1 --2- 172.21.15.44:0/2542247159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc874101e50 0x7fc87419ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.434+0000 7fc8794c0700 1 -- 172.21.15.44:0/2542247159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc87410a890 msgr2=0x7fc874198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.434+0000 7fc8794c0700 1 --2- 172.21.15.44:0/2542247159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc87410a890 0x7fc874198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.434+0000 7fc8794c0700 1 -- 172.21.15.44:0/2542247159 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8700095f0 con 0x7fc874101540 2023-11-16T16:48:16.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.434+0000 7fc8794c0700 1 --2- 172.21.15.44:0/2542247159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc874101540 0x7fc87419e4f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc87000b680 tx=0x7fc870009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:16.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.434+0000 7fc86a7fc700 1 -- 172.21.15.44:0/2542247159 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87001d070 con 0x7fc874101540 2023-11-16T16:48:16.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.434+0000 7fc86a7fc700 1 -- 172.21.15.44:0/2542247159 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc870021410 con 0x7fc874101540 2023-11-16T16:48:16.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.434+0000 7fc87b724700 1 -- 172.21.15.44:0/2542247159 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc874110360 con 0x7fc874101540 2023-11-16T16:48:16.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.434+0000 7fc86a7fc700 1 -- 172.21.15.44:0/2542247159 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc870017650 con 0x7fc874101540 2023-11-16T16:48:16.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.434+0000 7fc87b724700 1 -- 172.21.15.44:0/2542247159 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8741101d0 con 0x7fc874101540 2023-11-16T16:48:16.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.435+0000 7fc87b724700 1 -- 172.21.15.44:0/2542247159 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc8740ffaa0 con 0x7fc874101540 2023-11-16T16:48:16.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.437+0000 7fc86a7fc700 1 -- 172.21.15.44:0/2542247159 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc870021580 con 0x7fc874101540 2023-11-16T16:48:16.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.439+0000 7fc86a7fc700 1 --2- 172.21.15.44:0/2542247159 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc8600a1260 0x7fc8600a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:16.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.439+0000 7fc86a7fc700 1 -- 172.21.15.44:0/2542247159 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc8700ce450 con 0x7fc874101540 2023-11-16T16:48:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.439+0000 7fc878cbf700 1 --2- 172.21.15.44:0/2542247159 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc8600a1260 0x7fc8600a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.439+0000 7fc86a7fc700 1 --2- 172.21.15.44:0/2542247159 >> [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584] conn(0x7fc8600ac140 0x7fc8600ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.439+0000 7fc86a7fc700 1 -- 172.21.15.44:0/2542247159 --> [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8600aeca0 con 0x7fc8600ac140 2023-11-16T16:48:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.439+0000 7fc86a7fc700 1 -- 172.21.15.44:0/2542247159 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8700ce7c0 con 0x7fc874101540 2023-11-16T16:48:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.440+0000 7fc878cbf700 1 --2- 172.21.15.44:0/2542247159 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc8600a1260 0x7fc8600a3710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc87410c4e0 tx=0x7fc864009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.440+0000 7fc879cc1700 1 --2- 172.21.15.44:0/2542247159 >> [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584] conn(0x7fc8600ac140 0x7fc8600ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.440+0000 7fc879cc1700 1 --2- 172.21.15.44:0/2542247159 >> [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584] conn(0x7fc8600ac140 0x7fc8600ae5d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.441+0000 7fc86a7fc700 1 -- 172.21.15.44:0/2542247159 <== osd.3 v2:172.21.15.44:6810/1041267584 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc8600aeca0 con 0x7fc8600ac140 2023-11-16T16:48:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.457+0000 7fc87b724700 1 -- 172.21.15.44:0/2542247159 --> [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc8741104f0 con 0x7fc8600ac140 2023-11-16T16:48:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.458+0000 7fc86a7fc700 1 -- 172.21.15.44:0/2542247159 <== osd.3 v2:172.21.15.44:6810/1041267584 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc8741104f0 con 0x7fc8600ac140 2023-11-16T16:48:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.458+0000 7fc87b724700 1 -- 172.21.15.44:0/2542247159 >> [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584] conn(0x7fc8600ac140 msgr2=0x7fc8600ae5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.458+0000 7fc87b724700 1 --2- 172.21.15.44:0/2542247159 >> [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584] conn(0x7fc8600ac140 0x7fc8600ae5d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.458+0000 7fc87b724700 1 -- 172.21.15.44:0/2542247159 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc8600a1260 msgr2=0x7fc8600a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.458+0000 7fc87b724700 1 --2- 172.21.15.44:0/2542247159 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc8600a1260 0x7fc8600a3710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc87410c4e0 tx=0x7fc864009260 comp rx=0 tx=0).stop 2023-11-16T16:48:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.458+0000 7fc87b724700 1 -- 172.21.15.44:0/2542247159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc874101540 msgr2=0x7fc87419e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.458+0000 7fc87b724700 1 --2- 172.21.15.44:0/2542247159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc874101540 0x7fc87419e4f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc87000b680 tx=0x7fc870009580 comp rx=0 tx=0).stop 2023-11-16T16:48:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.458+0000 7fc879cc1700 1 -- 172.21.15.44:0/2542247159 reap_dead start 2023-11-16T16:48:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.458+0000 7fc87b724700 1 -- 172.21.15.44:0/2542247159 shutdown_connections 2023-11-16T16:48:16.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.458+0000 7fc87b724700 1 -- 172.21.15.44:0/2542247159 >> 172.21.15.44:0/2542247159 conn(0x7fc8740770a0 msgr2=0x7fc8740ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:16.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.459+0000 7fc87b724700 1 -- 172.21.15.44:0/2542247159 shutdown_connections 2023-11-16T16:48:16.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:15.459+0000 7fc87b724700 1 -- 172.21.15.44:0/2542247159 wait complete. 2023-11-16T16:48:16.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.250+0000 7f29caf29700 1 -- 172.21.15.44:0/487606768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29c41112f0 msgr2=0x7f29c41136d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.250+0000 7f29caf29700 1 --2- 172.21.15.44:0/487606768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29c41112f0 0x7f29c41136d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00b0a0 tx=0x7f29bc00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.250+0000 7f29caf29700 1 -- 172.21.15.44:0/487606768 shutdown_connections 2023-11-16T16:48:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.250+0000 7f29caf29700 1 --2- 172.21.15.44:0/487606768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29c41112f0 0x7f29c41136d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.250+0000 7f29caf29700 1 --2- 172.21.15.44:0/487606768 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f29c4072e90 0x7f29c4073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.250+0000 7f29caf29700 1 --2- 172.21.15.44:0/487606768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29c40737a0 0x7f29c4110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.250+0000 7f29caf29700 1 -- 172.21.15.44:0/487606768 >> 172.21.15.44:0/487606768 conn(0x7f29c40fbdc0 msgr2=0x7f29c40fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.250+0000 7f29caf29700 1 -- 172.21.15.44:0/487606768 shutdown_connections 2023-11-16T16:48:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.250+0000 7f29caf29700 1 -- 172.21.15.44:0/487606768 wait complete. 2023-11-16T16:48:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.251+0000 7f29caf29700 1 Processor -- start 2023-11-16T16:48:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.252+0000 7f29caf29700 1 -- start start 2023-11-16T16:48:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.252+0000 7f29caf29700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29c4072e90 0x7f29c41a28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.252+0000 7f29caf29700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f29c40737a0 0x7f29c41a2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.252+0000 7f29caf29700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29c41112f0 0x7f29c419c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.252+0000 7f29caf29700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c419cfa0 con 0x7f29c41112f0 2023-11-16T16:48:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.252+0000 7f29caf29700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c419d110 con 0x7f29c4072e90 2023-11-16T16:48:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.252+0000 7f29caf29700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c419d2d0 con 0x7f29c40737a0 2023-11-16T16:48:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.253+0000 7f29c94c6700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29c41112f0 0x7f29c419c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.253+0000 7f29c8cc5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29c4072e90 0x7f29c41a28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.253+0000 7f29c8cc5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29c4072e90 0x7f29c41a28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:33150/0 (socket says 172.21.15.44:33150) 2023-11-16T16:48:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.253+0000 7f29c94c6700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29c41112f0 0x7f29c419c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46904/0 (socket says 172.21.15.44:46904) 2023-11-16T16:48:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.253+0000 7f29c8cc5700 1 -- 172.21.15.44:0/464392543 learned_addr learned my addr 172.21.15.44:0/464392543 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.253+0000 7f29c3fff700 1 --2- 172.21.15.44:0/464392543 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f29c40737a0 0x7f29c41a2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.253+0000 7f29c8cc5700 1 -- 172.21.15.44:0/464392543 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f29c40737a0 msgr2=0x7f29c41a2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.253+0000 7f29c8cc5700 1 --2- 172.21.15.44:0/464392543 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f29c40737a0 0x7f29c41a2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.253+0000 7f29c8cc5700 1 -- 172.21.15.44:0/464392543 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29c41112f0 msgr2=0x7f29c419c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.253+0000 7f29c8cc5700 1 --2- 172.21.15.44:0/464392543 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29c41112f0 0x7f29c419c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.253+0000 7f29c8cc5700 1 -- 172.21.15.44:0/464392543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b0009520 con 0x7f29c4072e90 2023-11-16T16:48:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.253+0000 7f29c94c6700 1 --2- 172.21.15.44:0/464392543 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29c41112f0 0x7f29c419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.254+0000 7f29c8cc5700 1 --2- 172.21.15.44:0/464392543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29c4072e90 0x7f29c41a28b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f29b800e9a0 tx=0x7f29b800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.254+0000 7f29c1ffb700 1 -- 172.21.15.44:0/464392543 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8009790 con 0x7f29c4072e90 2023-11-16T16:48:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.254+0000 7f29caf29700 1 -- 172.21.15.44:0/464392543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29bc009d00 con 0x7f29c4072e90 2023-11-16T16:48:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.254+0000 7f29c1ffb700 1 -- 172.21.15.44:0/464392543 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f29b8010430 con 0x7f29c4072e90 2023-11-16T16:48:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.254+0000 7f29c1ffb700 1 -- 172.21.15.44:0/464392543 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8018680 con 0x7f29c4072e90 2023-11-16T16:48:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.254+0000 7f29caf29700 1 -- 172.21.15.44:0/464392543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c4116960 con 0x7f29c4072e90 2023-11-16T16:48:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.255+0000 7f29caf29700 1 -- 172.21.15.44:0/464392543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f29c40fdd30 con 0x7f29c4072e90 2023-11-16T16:48:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.257+0000 7f29c1ffb700 1 -- 172.21.15.44:0/464392543 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f29b80105a0 con 0x7f29c4072e90 2023-11-16T16:48:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.259+0000 7f29c1ffb700 1 --2- 172.21.15.44:0/464392543 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f29b40a1260 0x7f29b40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.259+0000 7f29c1ffb700 1 -- 172.21.15.44:0/464392543 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f29b8014070 con 0x7f29c4072e90 2023-11-16T16:48:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.259+0000 7f29c3fff700 1 --2- 172.21.15.44:0/464392543 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f29b40a1260 0x7f29b40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.259+0000 7f29c1ffb700 1 --2- 172.21.15.44:0/464392543 >> [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398] conn(0x7f29b40ac270 0x7f29b40ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.259+0000 7f29c1ffb700 1 -- 172.21.15.44:0/464392543 --> [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29b40aedd0 con 0x7f29b40ac270 2023-11-16T16:48:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.259+0000 7f29c1ffb700 1 -- 172.21.15.44:0/464392543 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29b80cd5a0 con 0x7f29c4072e90 2023-11-16T16:48:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.259+0000 7f29c94c6700 1 --2- 172.21.15.44:0/464392543 >> [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398] conn(0x7f29b40ac270 0x7f29b40ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.260+0000 7f29c3fff700 1 --2- 172.21.15.44:0/464392543 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f29b40a1260 0x7f29b40a3710 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f29c41129d0 tx=0x7f29b0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.260+0000 7f29c94c6700 1 --2- 172.21.15.44:0/464392543 >> [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398] conn(0x7f29b40ac270 0x7f29b40ae700 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.261+0000 7f29c1ffb700 1 -- 172.21.15.44:0/464392543 <== osd.9 v2:172.21.15.44:6826/1897410398 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f29b40aedd0 con 0x7f29b40ac270 2023-11-16T16:48:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.278+0000 7f29caf29700 1 -- 172.21.15.44:0/464392543 --> [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f29c41167d0 con 0x7f29b40ac270 2023-11-16T16:48:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.279+0000 7f29c1ffb700 1 -- 172.21.15.44:0/464392543 <== osd.9 v2:172.21.15.44:6826/1897410398 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f29c41167d0 con 0x7f29b40ac270 2023-11-16T16:48:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.279+0000 7f29caf29700 1 -- 172.21.15.44:0/464392543 >> [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398] conn(0x7f29b40ac270 msgr2=0x7f29b40ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.279+0000 7f29caf29700 1 --2- 172.21.15.44:0/464392543 >> [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398] conn(0x7f29b40ac270 0x7f29b40ae700 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.279+0000 7f29caf29700 1 -- 172.21.15.44:0/464392543 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f29b40a1260 msgr2=0x7f29b40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.279+0000 7f29caf29700 1 --2- 172.21.15.44:0/464392543 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f29b40a1260 0x7f29b40a3710 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f29c41129d0 tx=0x7f29b0009260 comp rx=0 tx=0).stop 2023-11-16T16:48:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.279+0000 7f29caf29700 1 -- 172.21.15.44:0/464392543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29c4072e90 msgr2=0x7f29c41a28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.279+0000 7f29caf29700 1 --2- 172.21.15.44:0/464392543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29c4072e90 0x7f29c41a28b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f29b800e9a0 tx=0x7f29b800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.279+0000 7f29c94c6700 1 -- 172.21.15.44:0/464392543 reap_dead start 2023-11-16T16:48:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.279+0000 7f29caf29700 1 -- 172.21.15.44:0/464392543 shutdown_connections 2023-11-16T16:48:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.279+0000 7f29caf29700 1 -- 172.21.15.44:0/464392543 >> 172.21.15.44:0/464392543 conn(0x7f29c40fbdc0 msgr2=0x7f29c41134e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.280+0000 7f29caf29700 1 -- 172.21.15.44:0/464392543 shutdown_connections 2023-11-16T16:48:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:16.280+0000 7f29caf29700 1 -- 172.21.15.44:0/464392543 wait complete. 2023-11-16T16:48:17.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.136+0000 7f4c50312700 1 -- 172.21.15.44:0/862173108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c4810a890 msgr2=0x7f4c4810ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:17.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.136+0000 7f4c50312700 1 --2- 172.21.15.44:0/862173108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c4810a890 0x7f4c4810ccd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000b0a0 tx=0x7f4c4000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:17.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.136+0000 7f4c50312700 1 -- 172.21.15.44:0/862173108 shutdown_connections 2023-11-16T16:48:17.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.136+0000 7f4c50312700 1 --2- 172.21.15.44:0/862173108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c4810a890 0x7f4c4810ccd0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:17.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.136+0000 7f4c50312700 1 --2- 172.21.15.44:0/862173108 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c48101540 0x7f4c48101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:17.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.136+0000 7f4c50312700 1 --2- 172.21.15.44:0/862173108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c48101e50 0x7f4c4810a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:17.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.136+0000 7f4c50312700 1 -- 172.21.15.44:0/862173108 >> 172.21.15.44:0/862173108 conn(0x7f4c480770a0 msgr2=0x7f4c480774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:17.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.136+0000 7f4c50312700 1 -- 172.21.15.44:0/862173108 shutdown_connections 2023-11-16T16:48:17.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.136+0000 7f4c50312700 1 -- 172.21.15.44:0/862173108 wait complete. 2023-11-16T16:48:17.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.137+0000 7f4c50312700 1 Processor -- start 2023-11-16T16:48:17.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.137+0000 7f4c50312700 1 -- start start 2023-11-16T16:48:17.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.137+0000 7f4c50312700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c48101540 0x7f4c48198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:17.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.137+0000 7f4c50312700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c48101e50 0x7f4c48198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:17.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.137+0000 7f4c50312700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c4810a890 0x7f4c4819c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:17.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.137+0000 7f4c50312700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4819c600 con 0x7f4c4810a890 2023-11-16T16:48:17.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.137+0000 7f4c50312700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4819c770 con 0x7f4c48101540 2023-11-16T16:48:17.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.137+0000 7f4c50312700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4819c930 con 0x7f4c48101e50 2023-11-16T16:48:17.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.138+0000 7f4c4e8af700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c4810a890 0x7f4c4819c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:17.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.138+0000 7f4c4e8af700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c4810a890 0x7f4c4819c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46912/0 (socket says 172.21.15.44:46912) 2023-11-16T16:48:17.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.138+0000 7f4c4e8af700 1 -- 172.21.15.44:0/1078423252 learned_addr learned my addr 172.21.15.44:0/1078423252 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:17.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.138+0000 7f4c4e0ae700 1 --2- 172.21.15.44:0/1078423252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c48101540 0x7f4c48198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:17.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.138+0000 7f4c4d8ad700 1 --2- 172.21.15.44:0/1078423252 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c48101e50 0x7f4c48198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.138+0000 7f4c4e8af700 1 -- 172.21.15.44:0/1078423252 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c48101e50 msgr2=0x7f4c48198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.138+0000 7f4c4e8af700 1 --2- 172.21.15.44:0/1078423252 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c48101e50 0x7f4c48198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.138+0000 7f4c4e8af700 1 -- 172.21.15.44:0/1078423252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c48101540 msgr2=0x7f4c48198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.138+0000 7f4c4e8af700 1 --2- 172.21.15.44:0/1078423252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c48101540 0x7f4c48198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.138+0000 7f4c4e8af700 1 -- 172.21.15.44:0/1078423252 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c40009d00 con 0x7f4c4810a890 2023-11-16T16:48:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.138+0000 7f4c4e8af700 1 --2- 172.21.15.44:0/1078423252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c4810a890 0x7f4c4819c000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009fd0 tx=0x7f4c4000bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.139+0000 7f4c3f7fe700 1 -- 172.21.15.44:0/1078423252 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4000e040 con 0x7f4c4810a890 2023-11-16T16:48:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.139+0000 7f4c3f7fe700 1 -- 172.21.15.44:0/1078423252 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c400094c0 con 0x7f4c4810a890 2023-11-16T16:48:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.139+0000 7f4c3f7fe700 1 -- 172.21.15.44:0/1078423252 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c400052a0 con 0x7f4c4810a890 2023-11-16T16:48:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.139+0000 7f4c50312700 1 -- 172.21.15.44:0/1078423252 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c48110350 con 0x7f4c4810a890 2023-11-16T16:48:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.139+0000 7f4c50312700 1 -- 172.21.15.44:0/1078423252 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c481101c0 con 0x7f4c4810a890 2023-11-16T16:48:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.140+0000 7f4c50312700 1 -- 172.21.15.44:0/1078423252 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4c480fc760 con 0x7f4c4810a890 2023-11-16T16:48:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.142+0000 7f4c3f7fe700 1 -- 172.21.15.44:0/1078423252 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4c4001d470 con 0x7f4c4810a890 2023-11-16T16:48:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.143+0000 7f4c3f7fe700 1 --2- 172.21.15.44:0/1078423252 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4c340a1260 0x7f4c340a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.143+0000 7f4c3f7fe700 1 -- 172.21.15.44:0/1078423252 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4c400ce670 con 0x7f4c4810a890 2023-11-16T16:48:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.143+0000 7f4c4e0ae700 1 --2- 172.21.15.44:0/1078423252 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4c340a1260 0x7f4c340a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.143+0000 7f4c3f7fe700 1 --2- 172.21.15.44:0/1078423252 >> [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497] conn(0x7f4c340ac270 0x7f4c340ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.143+0000 7f4c3f7fe700 1 -- 172.21.15.44:0/1078423252 --> [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c340aedd0 con 0x7f4c340ac270 2023-11-16T16:48:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.144+0000 7f4c3f7fe700 1 -- 172.21.15.44:0/1078423252 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c400ce9e0 con 0x7f4c4810a890 2023-11-16T16:48:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.144+0000 7f4c4d8ad700 1 --2- 172.21.15.44:0/1078423252 >> [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497] conn(0x7f4c340ac270 0x7f4c340ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.144+0000 7f4c4e0ae700 1 --2- 172.21.15.44:0/1078423252 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4c340a1260 0x7f4c340a3710 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009c60 tx=0x7f4c44009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.144+0000 7f4c4d8ad700 1 --2- 172.21.15.44:0/1078423252 >> [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497] conn(0x7f4c340ac270 0x7f4c340ae700 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.145+0000 7f4c3f7fe700 1 -- 172.21.15.44:0/1078423252 <== osd.4 v2:172.21.15.77:6808/2688407497 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f4c340aedd0 con 0x7f4c340ac270 2023-11-16T16:48:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.162+0000 7f4c50312700 1 -- 172.21.15.44:0/1078423252 --> [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4c481104e0 con 0x7f4c340ac270 2023-11-16T16:48:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.163+0000 7f4c3f7fe700 1 -- 172.21.15.44:0/1078423252 <== osd.4 v2:172.21.15.77:6808/2688407497 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4c481104e0 con 0x7f4c340ac270 2023-11-16T16:48:17.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.163+0000 7f4c50312700 1 -- 172.21.15.44:0/1078423252 >> [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497] conn(0x7f4c340ac270 msgr2=0x7f4c340ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:17.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.163+0000 7f4c50312700 1 --2- 172.21.15.44:0/1078423252 >> [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497] conn(0x7f4c340ac270 0x7f4c340ae700 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:17.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.164+0000 7f4c50312700 1 -- 172.21.15.44:0/1078423252 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4c340a1260 msgr2=0x7f4c340a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:17.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.164+0000 7f4c50312700 1 --2- 172.21.15.44:0/1078423252 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4c340a1260 0x7f4c340a3710 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009c60 tx=0x7f4c44009260 comp rx=0 tx=0).stop 2023-11-16T16:48:17.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.164+0000 7f4c50312700 1 -- 172.21.15.44:0/1078423252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c4810a890 msgr2=0x7f4c4819c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:17.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.164+0000 7f4c50312700 1 --2- 172.21.15.44:0/1078423252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c4810a890 0x7f4c4819c000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009fd0 tx=0x7f4c4000bce0 comp rx=0 tx=0).stop 2023-11-16T16:48:17.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.164+0000 7f4c4e8af700 1 -- 172.21.15.44:0/1078423252 reap_dead start 2023-11-16T16:48:17.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.164+0000 7f4c50312700 1 -- 172.21.15.44:0/1078423252 shutdown_connections 2023-11-16T16:48:17.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.164+0000 7f4c50312700 1 -- 172.21.15.44:0/1078423252 >> 172.21.15.44:0/1078423252 conn(0x7f4c480770a0 msgr2=0x7f4c480fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:17.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.164+0000 7f4c50312700 1 -- 172.21.15.44:0/1078423252 shutdown_connections 2023-11-16T16:48:17.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.164+0000 7f4c50312700 1 -- 172.21.15.44:0/1078423252 wait complete. 2023-11-16T16:48:17.707 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:17 smithi044 ceph-mon[128750]: pgmap v306: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1023 B/s wr, 3 op/s 2023-11-16T16:48:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:17 smithi176 ceph-mon[131630]: pgmap v306: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1023 B/s wr, 3 op/s 2023-11-16T16:48:17.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:17 smithi077 ceph-mon[131204]: pgmap v306: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1023 B/s wr, 3 op/s 2023-11-16T16:48:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.768+0000 7f789efb0700 1 -- 172.21.15.44:0/1459111871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78981055e0 msgr2=0x7f789810d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.768+0000 7f789efb0700 1 --2- 172.21.15.44:0/1459111871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78981055e0 0x7f789810d160 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f789000b0d0 tx=0x7f789000b3e0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.768+0000 7f789efb0700 1 -- 172.21.15.44:0/1459111871 shutdown_connections 2023-11-16T16:48:18.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.769+0000 7f789efb0700 1 --2- 172.21.15.44:0/1459111871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78981055e0 0x7f789810d160 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.769+0000 7f789efb0700 1 --2- 172.21.15.44:0/1459111871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7898100540 0x7f7898100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.769+0000 7f789efb0700 1 --2- 172.21.15.44:0/1459111871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7898100ee0 0x7f7898104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.769+0000 7f789efb0700 1 -- 172.21.15.44:0/1459111871 >> 172.21.15.44:0/1459111871 conn(0x7f78980fbdc0 msgr2=0x7f78980fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.769+0000 7f789efb0700 1 -- 172.21.15.44:0/1459111871 shutdown_connections 2023-11-16T16:48:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.769+0000 7f789efb0700 1 -- 172.21.15.44:0/1459111871 wait complete. 2023-11-16T16:48:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.770+0000 7f789efb0700 1 Processor -- start 2023-11-16T16:48:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.770+0000 7f789efb0700 1 -- start start 2023-11-16T16:48:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.770+0000 7f789efb0700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7898100540 0x7f78981986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.770+0000 7f789efb0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7898100ee0 0x7f7898198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.770+0000 7f789efb0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78981055e0 0x7f789819bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:18.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.770+0000 7f789efb0700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789819c5a0 con 0x7f7898100540 2023-11-16T16:48:18.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.770+0000 7f789efb0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789819c710 con 0x7f7898100ee0 2023-11-16T16:48:18.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.770+0000 7f789efb0700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789819c8d0 con 0x7f78981055e0 2023-11-16T16:48:18.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.770+0000 7f789cd4c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7898100540 0x7f78981986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:18.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.771+0000 7f789cd4c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7898100540 0x7f78981986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46940/0 (socket says 172.21.15.44:46940) 2023-11-16T16:48:18.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.771+0000 7f789cd4c700 1 -- 172.21.15.44:0/249785656 learned_addr learned my addr 172.21.15.44:0/249785656 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:18.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.771+0000 7f7897fff700 1 --2- 172.21.15.44:0/249785656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7898100ee0 0x7f7898198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:18.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.771+0000 7f789d54d700 1 --2- 172.21.15.44:0/249785656 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78981055e0 0x7f789819bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:18.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.771+0000 7f789cd4c700 1 -- 172.21.15.44:0/249785656 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78981055e0 msgr2=0x7f789819bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.771+0000 7f789cd4c700 1 --2- 172.21.15.44:0/249785656 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78981055e0 0x7f789819bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.771+0000 7f789cd4c700 1 -- 172.21.15.44:0/249785656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7898100ee0 msgr2=0x7f7898198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.771+0000 7f789cd4c700 1 --2- 172.21.15.44:0/249785656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7898100ee0 0x7f7898198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.771+0000 7f789cd4c700 1 -- 172.21.15.44:0/249785656 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c009520 con 0x7f7898100540 2023-11-16T16:48:18.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.771+0000 7f789cd4c700 1 --2- 172.21.15.44:0/249785656 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7898100540 0x7f78981986a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f788c00eb30 tx=0x7f788c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:18.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.771+0000 7f7895ffb700 1 -- 172.21.15.44:0/249785656 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c009880 con 0x7f7898100540 2023-11-16T16:48:18.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.771+0000 7f789efb0700 1 -- 172.21.15.44:0/249785656 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7890009d00 con 0x7f7898100540 2023-11-16T16:48:18.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.771+0000 7f7895ffb700 1 -- 172.21.15.44:0/249785656 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f788c0044e0 con 0x7f7898100540 2023-11-16T16:48:18.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.772+0000 7f7895ffb700 1 -- 172.21.15.44:0/249785656 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c003d60 con 0x7f7898100540 2023-11-16T16:48:18.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.772+0000 7f789efb0700 1 -- 172.21.15.44:0/249785656 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7898110370 con 0x7f7898100540 2023-11-16T16:48:18.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.772+0000 7f789efb0700 1 -- 172.21.15.44:0/249785656 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f78980fd190 con 0x7f7898100540 2023-11-16T16:48:18.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.775+0000 7f7895ffb700 1 -- 172.21.15.44:0/249785656 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f788c003f00 con 0x7f7898100540 2023-11-16T16:48:18.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.776+0000 7f7895ffb700 1 --2- 172.21.15.44:0/249785656 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f78880a11e0 0x7f78880a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:18.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.776+0000 7f7895ffb700 1 -- 172.21.15.44:0/249785656 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f788c014070 con 0x7f7898100540 2023-11-16T16:48:18.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.776+0000 7f7897fff700 1 --2- 172.21.15.44:0/249785656 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f78880a11e0 0x7f78880a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:18.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.776+0000 7f7895ffb700 1 --2- 172.21.15.44:0/249785656 >> [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313] conn(0x7f78880ac1f0 0x7f78880ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:18.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.776+0000 7f7895ffb700 1 -- 172.21.15.44:0/249785656 --> [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78880aed50 con 0x7f78880ac1f0 2023-11-16T16:48:18.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.776+0000 7f7895ffb700 1 -- 172.21.15.44:0/249785656 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f788c0cda50 con 0x7f7898100540 2023-11-16T16:48:18.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.777+0000 7f7897fff700 1 --2- 172.21.15.44:0/249785656 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f78880a11e0 0x7f78880a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7898199cc0 tx=0x7f7884009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:18.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.777+0000 7f789d54d700 1 --2- 172.21.15.44:0/249785656 >> [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313] conn(0x7f78880ac1f0 0x7f78880ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:18.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.778+0000 7f789d54d700 1 --2- 172.21.15.44:0/249785656 >> [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313] conn(0x7f78880ac1f0 0x7f78880ae680 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:18.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.779+0000 7f7895ffb700 1 -- 172.21.15.44:0/249785656 <== osd.7 v2:172.21.15.77:6816/1651452313 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f78880aed50 con 0x7f78880ac1f0 2023-11-16T16:48:18.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.795+0000 7f789efb0700 1 -- 172.21.15.44:0/249785656 --> [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f78981101e0 con 0x7f78880ac1f0 2023-11-16T16:48:18.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.796+0000 7f7895ffb700 1 -- 172.21.15.44:0/249785656 <== osd.7 v2:172.21.15.77:6816/1651452313 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f78981101e0 con 0x7f78880ac1f0 2023-11-16T16:48:18.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.796+0000 7f789efb0700 1 -- 172.21.15.44:0/249785656 >> [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313] conn(0x7f78880ac1f0 msgr2=0x7f78880ae680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.796+0000 7f789efb0700 1 --2- 172.21.15.44:0/249785656 >> [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313] conn(0x7f78880ac1f0 0x7f78880ae680 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.796+0000 7f789efb0700 1 -- 172.21.15.44:0/249785656 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f78880a11e0 msgr2=0x7f78880a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.796+0000 7f789efb0700 1 --2- 172.21.15.44:0/249785656 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f78880a11e0 0x7f78880a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7898199cc0 tx=0x7f7884009260 comp rx=0 tx=0).stop 2023-11-16T16:48:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.796+0000 7f789efb0700 1 -- 172.21.15.44:0/249785656 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7898100540 msgr2=0x7f78981986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.796+0000 7f789efb0700 1 --2- 172.21.15.44:0/249785656 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7898100540 0x7f78981986a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f788c00eb30 tx=0x7f788c00ee40 comp rx=0 tx=0).stop 2023-11-16T16:48:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.797+0000 7f789d54d700 1 -- 172.21.15.44:0/249785656 reap_dead start 2023-11-16T16:48:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.797+0000 7f789efb0700 1 -- 172.21.15.44:0/249785656 shutdown_connections 2023-11-16T16:48:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.797+0000 7f789efb0700 1 -- 172.21.15.44:0/249785656 >> 172.21.15.44:0/249785656 conn(0x7f78980fbdc0 msgr2=0x7f78980fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.797+0000 7f789efb0700 1 -- 172.21.15.44:0/249785656 shutdown_connections 2023-11-16T16:48:18.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.797+0000 7f789efb0700 1 -- 172.21.15.44:0/249785656 wait complete. 2023-11-16T16:48:18.350 INFO:teuthology.orchestra.run.smithi044.stdout:107374182459 2023-11-16T16:48:18.351 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.3 2023-11-16T16:48:18.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.581+0000 7f533ad82700 1 -- 172.21.15.44:0/1732539568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334100540 msgr2=0x7f5334100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.581+0000 7f533ad82700 1 --2- 172.21.15.44:0/1732539568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334100540 0x7f5334100910 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f53280099c0 tx=0x7f5328009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.581+0000 7f533ad82700 1 -- 172.21.15.44:0/1732539568 shutdown_connections 2023-11-16T16:48:18.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.581+0000 7f533ad82700 1 --2- 172.21.15.44:0/1732539568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53341055e0 0x7f533410d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.581+0000 7f533ad82700 1 --2- 172.21.15.44:0/1732539568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334100540 0x7f5334100910 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.581+0000 7f533ad82700 1 --2- 172.21.15.44:0/1732539568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5334100ee0 0x7f5334104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.581+0000 7f533ad82700 1 -- 172.21.15.44:0/1732539568 >> 172.21.15.44:0/1732539568 conn(0x7f53340fbdc0 msgr2=0x7f53340fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:18.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.581+0000 7f533ad82700 1 -- 172.21.15.44:0/1732539568 shutdown_connections 2023-11-16T16:48:18.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.582+0000 7f533ad82700 1 -- 172.21.15.44:0/1732539568 wait complete. 2023-11-16T16:48:18.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.583+0000 7f533ad82700 1 Processor -- start 2023-11-16T16:48:18.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.583+0000 7f533ad82700 1 -- start start 2023-11-16T16:48:18.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.584+0000 7f533ad82700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334100540 0x7f5334198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:18.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.584+0000 7f533ad82700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5334100ee0 0x7f5334198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:18.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.584+0000 7f533ad82700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53341055e0 0x7f533419cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:18.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.584+0000 7f533ad82700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533419d520 con 0x7f5334100540 2023-11-16T16:48:18.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.584+0000 7f533ad82700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533419d690 con 0x7f53341055e0 2023-11-16T16:48:18.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.584+0000 7f533ad82700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533419d850 con 0x7f5334100ee0 2023-11-16T16:48:18.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.584+0000 7f5333fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5334100ee0 0x7f5334198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:18.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.584+0000 7f5338b1e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334100540 0x7f5334198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:18.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.584+0000 7f5333fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5334100ee0 0x7f5334198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:59268/0 (socket says 172.21.15.44:59268) 2023-11-16T16:48:18.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.584+0000 7f5333fff700 1 -- 172.21.15.44:0/2491518790 learned_addr learned my addr 172.21.15.44:0/2491518790 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:18.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.584+0000 7f533931f700 1 --2- 172.21.15.44:0/2491518790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53341055e0 0x7f533419cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:18.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.585+0000 7f5333fff700 1 -- 172.21.15.44:0/2491518790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53341055e0 msgr2=0x7f533419cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.585+0000 7f5333fff700 1 --2- 172.21.15.44:0/2491518790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53341055e0 0x7f533419cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.585+0000 7f5333fff700 1 -- 172.21.15.44:0/2491518790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334100540 msgr2=0x7f5334198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.585+0000 7f5333fff700 1 --2- 172.21.15.44:0/2491518790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5334100540 0x7f5334198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.585+0000 7f5333fff700 1 -- 172.21.15.44:0/2491518790 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53280095f0 con 0x7f5334100ee0 2023-11-16T16:48:18.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.585+0000 7f5333fff700 1 --2- 172.21.15.44:0/2491518790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5334100ee0 0x7f5334198aa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f532000e970 tx=0x7f532000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:18.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.585+0000 7f5331ffb700 1 -- 172.21.15.44:0/2491518790 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5320009790 con 0x7f5334100ee0 2023-11-16T16:48:18.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.585+0000 7f5331ffb700 1 -- 172.21.15.44:0/2491518790 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f532000c7e0 con 0x7f5334100ee0 2023-11-16T16:48:18.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.585+0000 7f533ad82700 1 -- 172.21.15.44:0/2491518790 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5334110320 con 0x7f5334100ee0 2023-11-16T16:48:18.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.585+0000 7f5331ffb700 1 -- 172.21.15.44:0/2491518790 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53200054e0 con 0x7f5334100ee0 2023-11-16T16:48:18.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.585+0000 7f533ad82700 1 -- 172.21.15.44:0/2491518790 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5334110190 con 0x7f5334100ee0 2023-11-16T16:48:18.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.586+0000 7f533ad82700 1 -- 172.21.15.44:0/2491518790 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f53340fd380 con 0x7f5334100ee0 2023-11-16T16:48:18.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.588+0000 7f5331ffb700 1 -- 172.21.15.44:0/2491518790 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f532000c950 con 0x7f5334100ee0 2023-11-16T16:48:18.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.590+0000 7f5331ffb700 1 --2- 172.21.15.44:0/2491518790 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f53240a11e0 0x7f53240a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:18.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.590+0000 7f5331ffb700 1 -- 172.21.15.44:0/2491518790 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5320014070 con 0x7f5334100ee0 2023-11-16T16:48:18.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.590+0000 7f5338b1e700 1 --2- 172.21.15.44:0/2491518790 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f53240a11e0 0x7f53240a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:18.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.590+0000 7f5331ffb700 1 --2- 172.21.15.44:0/2491518790 >> [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157] conn(0x7f53240ac1f0 0x7f53240ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:18.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.590+0000 7f5331ffb700 1 -- 172.21.15.44:0/2491518790 --> [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53240aed50 con 0x7f53240ac1f0 2023-11-16T16:48:18.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.590+0000 7f5331ffb700 1 -- 172.21.15.44:0/2491518790 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53200cd5f0 con 0x7f5334100ee0 2023-11-16T16:48:18.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.590+0000 7f533931f700 1 --2- 172.21.15.44:0/2491518790 >> [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157] conn(0x7f53240ac1f0 0x7f53240ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:18.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.591+0000 7f5338b1e700 1 --2- 172.21.15.44:0/2491518790 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f53240a11e0 0x7f53240a3690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5328009fd0 tx=0x7f5328005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:18.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.591+0000 7f533931f700 1 --2- 172.21.15.44:0/2491518790 >> [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157] conn(0x7f53240ac1f0 0x7f53240ae680 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:18.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.592+0000 7f5331ffb700 1 -- 172.21.15.44:0/2491518790 <== osd.0 v2:172.21.15.44:6802/2561264157 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f53240aed50 con 0x7f53240ac1f0 2023-11-16T16:48:18.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.609+0000 7f533ad82700 1 -- 172.21.15.44:0/2491518790 --> [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f53341104b0 con 0x7f53240ac1f0 2023-11-16T16:48:18.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.610+0000 7f5331ffb700 1 -- 172.21.15.44:0/2491518790 <== osd.0 v2:172.21.15.44:6802/2561264157 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f53341104b0 con 0x7f53240ac1f0 2023-11-16T16:48:18.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.611+0000 7f533ad82700 1 -- 172.21.15.44:0/2491518790 >> [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157] conn(0x7f53240ac1f0 msgr2=0x7f53240ae680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.611+0000 7f533ad82700 1 --2- 172.21.15.44:0/2491518790 >> [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157] conn(0x7f53240ac1f0 0x7f53240ae680 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.611+0000 7f533ad82700 1 -- 172.21.15.44:0/2491518790 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f53240a11e0 msgr2=0x7f53240a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.611+0000 7f533ad82700 1 --2- 172.21.15.44:0/2491518790 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f53240a11e0 0x7f53240a3690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5328009fd0 tx=0x7f5328005fb0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.611+0000 7f533ad82700 1 -- 172.21.15.44:0/2491518790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5334100ee0 msgr2=0x7f5334198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.611+0000 7f533ad82700 1 --2- 172.21.15.44:0/2491518790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5334100ee0 0x7f5334198aa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f532000e970 tx=0x7f532000ec80 comp rx=0 tx=0).stop 2023-11-16T16:48:18.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.611+0000 7f533931f700 1 -- 172.21.15.44:0/2491518790 reap_dead start 2023-11-16T16:48:18.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.611+0000 7f533ad82700 1 -- 172.21.15.44:0/2491518790 shutdown_connections 2023-11-16T16:48:18.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.611+0000 7f533ad82700 1 -- 172.21.15.44:0/2491518790 >> 172.21.15.44:0/2491518790 conn(0x7f53340fbdc0 msgr2=0x7f53340fc930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:18.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.613+0000 7f533ad82700 1 -- 172.21.15.44:0/2491518790 shutdown_connections 2023-11-16T16:48:18.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:18.613+0000 7f533ad82700 1 -- 172.21.15.44:0/2491518790 wait complete. 2023-11-16T16:48:18.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.318+0000 7f80ccfda700 1 -- 172.21.15.44:0/2509324831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80c81055e0 msgr2=0x7f80c810d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.318+0000 7f80ccfda700 1 --2- 172.21.15.44:0/2509324831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80c81055e0 0x7f80c810d160 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f80c000b0d0 tx=0x7f80c000b3e0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.318+0000 7f80ccfda700 1 -- 172.21.15.44:0/2509324831 shutdown_connections 2023-11-16T16:48:18.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.318+0000 7f80ccfda700 1 --2- 172.21.15.44:0/2509324831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80c81055e0 0x7f80c810d160 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.318+0000 7f80ccfda700 1 --2- 172.21.15.44:0/2509324831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c8100540 0x7f80c8100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.318+0000 7f80ccfda700 1 --2- 172.21.15.44:0/2509324831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80c8100ee0 0x7f80c8104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.318+0000 7f80ccfda700 1 -- 172.21.15.44:0/2509324831 >> 172.21.15.44:0/2509324831 conn(0x7f80c80fbdc0 msgr2=0x7f80c80fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:18.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.319+0000 7f80ccfda700 1 -- 172.21.15.44:0/2509324831 shutdown_connections 2023-11-16T16:48:18.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.319+0000 7f80ccfda700 1 -- 172.21.15.44:0/2509324831 wait complete. 2023-11-16T16:48:18.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.320+0000 7f80ccfda700 1 Processor -- start 2023-11-16T16:48:18.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.320+0000 7f80ccfda700 1 -- start start 2023-11-16T16:48:18.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.320+0000 7f80ccfda700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80c8100540 0x7f80c819e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:18.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.320+0000 7f80ccfda700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c8100ee0 0x7f80c819eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:18.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.320+0000 7f80ccfda700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80c81055e0 0x7f80c8198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:18.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.320+0000 7f80ccfda700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c8198d10 con 0x7f80c81055e0 2023-11-16T16:48:18.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.320+0000 7f80ccfda700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c8198e80 con 0x7f80c8100ee0 2023-11-16T16:48:18.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.320+0000 7f80ccfda700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c8199040 con 0x7f80c8100540 2023-11-16T16:48:18.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.321+0000 7f80c6d9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80c81055e0 0x7f80c8198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:18.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.321+0000 7f80c5d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c8100ee0 0x7f80c819eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:18.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.321+0000 7f80c5d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c8100ee0 0x7f80c819eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:33190/0 (socket says 172.21.15.44:33190) 2023-11-16T16:48:18.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.321+0000 7f80c5d9b700 1 -- 172.21.15.44:0/328428289 learned_addr learned my addr 172.21.15.44:0/328428289 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:18.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.321+0000 7f80c659c700 1 --2- 172.21.15.44:0/328428289 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80c8100540 0x7f80c819e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:18.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.321+0000 7f80c5d9b700 1 -- 172.21.15.44:0/328428289 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80c8100540 msgr2=0x7f80c819e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.321+0000 7f80c5d9b700 1 --2- 172.21.15.44:0/328428289 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80c8100540 0x7f80c819e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.321+0000 7f80c5d9b700 1 -- 172.21.15.44:0/328428289 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80c81055e0 msgr2=0x7f80c8198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.321+0000 7f80c5d9b700 1 --2- 172.21.15.44:0/328428289 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80c81055e0 0x7f80c8198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.321+0000 7f80c5d9b700 1 -- 172.21.15.44:0/328428289 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c0009d00 con 0x7f80c8100ee0 2023-11-16T16:48:18.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.321+0000 7f80c5d9b700 1 --2- 172.21.15.44:0/328428289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c8100ee0 0x7f80c819eab0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f80bc00e970 tx=0x7f80bc00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:18.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.322+0000 7f80b77fe700 1 -- 172.21.15.44:0/328428289 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc009790 con 0x7f80c8100ee0 2023-11-16T16:48:18.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.322+0000 7f80ccfda700 1 -- 172.21.15.44:0/328428289 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c8110370 con 0x7f80c8100ee0 2023-11-16T16:48:18.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.322+0000 7f80b77fe700 1 -- 172.21.15.44:0/328428289 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80bc00c7e0 con 0x7f80c8100ee0 2023-11-16T16:48:18.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.322+0000 7f80b77fe700 1 -- 172.21.15.44:0/328428289 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc0054e0 con 0x7f80c8100ee0 2023-11-16T16:48:18.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.322+0000 7f80ccfda700 1 -- 172.21.15.44:0/328428289 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c81101e0 con 0x7f80c8100ee0 2023-11-16T16:48:18.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.323+0000 7f80ccfda700 1 -- 172.21.15.44:0/328428289 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f80c80fd0b0 con 0x7f80c8100ee0 2023-11-16T16:48:18.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.325+0000 7f80b77fe700 1 -- 172.21.15.44:0/328428289 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f80bc00c950 con 0x7f80c8100ee0 2023-11-16T16:48:18.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.326+0000 7f80b77fe700 1 --2- 172.21.15.44:0/328428289 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f80b00a11e0 0x7f80b00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:18.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.326+0000 7f80b77fe700 1 -- 172.21.15.44:0/328428289 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f80bc014070 con 0x7f80c8100ee0 2023-11-16T16:48:18.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.326+0000 7f80c659c700 1 --2- 172.21.15.44:0/328428289 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f80b00a11e0 0x7f80b00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:18.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.327+0000 7f80b77fe700 1 --2- 172.21.15.44:0/328428289 >> [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849] conn(0x7f80b00ac1f0 0x7f80b00ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:18.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.327+0000 7f80b77fe700 1 -- 172.21.15.44:0/328428289 --> [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f80b00aed50 con 0x7f80b00ac1f0 2023-11-16T16:48:18.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.327+0000 7f80b77fe700 1 -- 172.21.15.44:0/328428289 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80bc0cd5b0 con 0x7f80c8100ee0 2023-11-16T16:48:18.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.327+0000 7f80c6d9d700 1 --2- 172.21.15.44:0/328428289 >> [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849] conn(0x7f80b00ac1f0 0x7f80b00ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:18.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.327+0000 7f80c659c700 1 --2- 172.21.15.44:0/328428289 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f80b00a11e0 0x7f80b00a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f80b8009c60 tx=0x7f80b8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:18.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.327+0000 7f80c6d9d700 1 --2- 172.21.15.44:0/328428289 >> [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849] conn(0x7f80b00ac1f0 0x7f80b00ae680 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:18.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.328+0000 7f80b77fe700 1 -- 172.21.15.44:0/328428289 <== osd.6 v2:172.21.15.44:6818/2634951849 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f80b00aed50 con 0x7f80b00ac1f0 2023-11-16T16:48:18.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.345+0000 7f80ccfda700 1 -- 172.21.15.44:0/328428289 --> [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f80c8110500 con 0x7f80b00ac1f0 2023-11-16T16:48:18.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.345+0000 7f80b77fe700 1 -- 172.21.15.44:0/328428289 <== osd.6 v2:172.21.15.44:6818/2634951849 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f80c8110500 con 0x7f80b00ac1f0 2023-11-16T16:48:18.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.345+0000 7f80ccfda700 1 -- 172.21.15.44:0/328428289 >> [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849] conn(0x7f80b00ac1f0 msgr2=0x7f80b00ae680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.345+0000 7f80ccfda700 1 --2- 172.21.15.44:0/328428289 >> [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849] conn(0x7f80b00ac1f0 0x7f80b00ae680 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:18.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.345+0000 7f80ccfda700 1 -- 172.21.15.44:0/328428289 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f80b00a11e0 msgr2=0x7f80b00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.345+0000 7f80ccfda700 1 --2- 172.21.15.44:0/328428289 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f80b00a11e0 0x7f80b00a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f80b8009c60 tx=0x7f80b8009260 comp rx=0 tx=0).stop 2023-11-16T16:48:18.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.345+0000 7f80ccfda700 1 -- 172.21.15.44:0/328428289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c8100ee0 msgr2=0x7f80c819eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:18.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.345+0000 7f80ccfda700 1 --2- 172.21.15.44:0/328428289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c8100ee0 0x7f80c819eab0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f80bc00e970 tx=0x7f80bc00ec80 comp rx=0 tx=0).stop 2023-11-16T16:48:18.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.346+0000 7f80c6d9d700 1 -- 172.21.15.44:0/328428289 reap_dead start 2023-11-16T16:48:18.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.346+0000 7f80ccfda700 1 -- 172.21.15.44:0/328428289 shutdown_connections 2023-11-16T16:48:18.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.346+0000 7f80ccfda700 1 -- 172.21.15.44:0/328428289 >> 172.21.15.44:0/328428289 conn(0x7f80c80fbdc0 msgr2=0x7f80c80fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:18.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.346+0000 7f80ccfda700 1 -- 172.21.15.44:0/328428289 shutdown_connections 2023-11-16T16:48:18.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:17.346+0000 7f80ccfda700 1 -- 172.21.15.44:0/328428289 wait complete. 2023-11-16T16:48:19.255 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:19 smithi044 ceph-mon[128750]: pgmap v307: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:48:19.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:19 smithi077 ceph-mon[131204]: pgmap v307: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:48:19.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.302+0000 7f9ad4900700 1 -- 172.21.15.44:0/3601855962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ad00ff460 msgr2=0x7f9ad00ff830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.302+0000 7f9ad4900700 1 --2- 172.21.15.44:0/3601855962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ad00ff460 0x7f9ad00ff830 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0009990 tx=0x7f9ac0009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.303+0000 7f9ad4900700 1 -- 172.21.15.44:0/3601855962 shutdown_connections 2023-11-16T16:48:19.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.303+0000 7f9ad4900700 1 --2- 172.21.15.44:0/3601855962 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0104500 0x7f9ad010c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.303+0000 7f9ad4900700 1 --2- 172.21.15.44:0/3601855962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ad00ff460 0x7f9ad00ff830 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.303+0000 7f9ad4900700 1 --2- 172.21.15.44:0/3601855962 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ad00ffe00 0x7f9ad0103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.303+0000 7f9ad4900700 1 -- 172.21.15.44:0/3601855962 >> 172.21.15.44:0/3601855962 conn(0x7f9ad00770a0 msgr2=0x7f9ad00774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:19.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.303+0000 7f9ad4900700 1 -- 172.21.15.44:0/3601855962 shutdown_connections 2023-11-16T16:48:19.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.303+0000 7f9ad4900700 1 -- 172.21.15.44:0/3601855962 wait complete. 2023-11-16T16:48:19.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.304+0000 7f9ad4900700 1 Processor -- start 2023-11-16T16:48:19.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.304+0000 7f9ad4900700 1 -- start start 2023-11-16T16:48:19.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.305+0000 7f9ad4900700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ad00ff460 0x7f9ad01060a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.306+0000 7f9ace59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ad00ff460 0x7f9ad01060a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.306+0000 7f9ace59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ad00ff460 0x7f9ad01060a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:59296/0 (socket says 172.21.15.44:59296) 2023-11-16T16:48:19.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.306+0000 7f9ad4900700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ad00ffe00 0x7f9ad01065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.306+0000 7f9ad4900700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0104500 0x7f9ad0106b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.306+0000 7f9ad4900700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad01a8c60 con 0x7f9ad0104500 2023-11-16T16:48:19.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.306+0000 7f9ad4900700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad01a8dd0 con 0x7f9ad00ffe00 2023-11-16T16:48:19.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.306+0000 7f9ad4900700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad01a8f90 con 0x7f9ad00ff460 2023-11-16T16:48:19.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.306+0000 7f9ace59c700 1 -- 172.21.15.44:0/2958962110 learned_addr learned my addr 172.21.15.44:0/2958962110 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:19.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.306+0000 7f9aced9d700 1 --2- 172.21.15.44:0/2958962110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0104500 0x7f9ad0106b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.307+0000 7f9acdd9b700 1 --2- 172.21.15.44:0/2958962110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ad00ffe00 0x7f9ad01065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.307+0000 7f9aced9d700 1 -- 172.21.15.44:0/2958962110 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ad00ff460 msgr2=0x7f9ad01060a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.307+0000 7f9aced9d700 1 --2- 172.21.15.44:0/2958962110 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ad00ff460 0x7f9ad01060a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.307+0000 7f9aced9d700 1 -- 172.21.15.44:0/2958962110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ad00ffe00 msgr2=0x7f9ad01065e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.307+0000 7f9aced9d700 1 --2- 172.21.15.44:0/2958962110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ad00ffe00 0x7f9ad01065e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.307+0000 7f9aced9d700 1 -- 172.21.15.44:0/2958962110 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac00095f0 con 0x7f9ad0104500 2023-11-16T16:48:19.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.307+0000 7f9aced9d700 1 --2- 172.21.15.44:0/2958962110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0104500 0x7f9ad0106b20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9ac800cc50 tx=0x7f9ac800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:19.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.308+0000 7f9abf7fe700 1 -- 172.21.15.44:0/2958962110 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac800aa90 con 0x7f9ad0104500 2023-11-16T16:48:19.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.308+0000 7f9ad4900700 1 -- 172.21.15.44:0/2958962110 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad010f290 con 0x7f9ad0104500 2023-11-16T16:48:19.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.308+0000 7f9abf7fe700 1 -- 172.21.15.44:0/2958962110 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ac800abf0 con 0x7f9ad0104500 2023-11-16T16:48:19.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.309+0000 7f9abf7fe700 1 -- 172.21.15.44:0/2958962110 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac80115e0 con 0x7f9ad0104500 2023-11-16T16:48:19.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.309+0000 7f9ad4900700 1 -- 172.21.15.44:0/2958962110 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad010f100 con 0x7f9ad0104500 2023-11-16T16:48:19.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.311+0000 7f9ad4900700 1 -- 172.21.15.44:0/2958962110 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9ad00fe4e0 con 0x7f9ad0104500 2023-11-16T16:48:19.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.314+0000 7f9abf7fe700 1 -- 172.21.15.44:0/2958962110 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9ac8012420 con 0x7f9ad0104500 2023-11-16T16:48:19.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.317+0000 7f9abf7fe700 1 --2- 172.21.15.44:0/2958962110 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9ab80a1260 0x7f9ab80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.317+0000 7f9abf7fe700 1 -- 172.21.15.44:0/2958962110 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9ac80cc9a0 con 0x7f9ad0104500 2023-11-16T16:48:19.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.318+0000 7f9abf7fe700 1 --2- 172.21.15.44:0/2958962110 >> [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093] conn(0x7f9ab80ac270 0x7f9ab80ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.318+0000 7f9abf7fe700 1 -- 172.21.15.44:0/2958962110 --> [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ab80aedd0 con 0x7f9ab80ac270 2023-11-16T16:48:19.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.318+0000 7f9abf7fe700 1 -- 172.21.15.44:0/2958962110 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ac80d12a0 con 0x7f9ad0104500 2023-11-16T16:48:19.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.318+0000 7f9ace59c700 1 --2- 172.21.15.44:0/2958962110 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9ab80a1260 0x7f9ab80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.319+0000 7f9acdd9b700 1 --2- 172.21.15.44:0/2958962110 >> [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093] conn(0x7f9ab80ac270 0x7f9ab80ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.319+0000 7f9ace59c700 1 --2- 172.21.15.44:0/2958962110 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9ab80a1260 0x7f9ab80a3710 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0006010 tx=0x7f9ac000b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:19.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.319+0000 7f9acdd9b700 1 --2- 172.21.15.44:0/2958962110 >> [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093] conn(0x7f9ab80ac270 0x7f9ab80ae700 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:19.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.320+0000 7f9abf7fe700 1 -- 172.21.15.44:0/2958962110 <== osd.10 v2:172.21.15.77:6824/3918521093 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f9ab80aedd0 con 0x7f9ab80ac270 2023-11-16T16:48:19.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.357+0000 7f9ad4900700 1 -- 172.21.15.44:0/2958962110 --> [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9ad010f420 con 0x7f9ab80ac270 2023-11-16T16:48:19.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.358+0000 7f9abf7fe700 1 -- 172.21.15.44:0/2958962110 <== osd.10 v2:172.21.15.77:6824/3918521093 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9ad010f420 con 0x7f9ab80ac270 2023-11-16T16:48:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.359+0000 7f9ad4900700 1 -- 172.21.15.44:0/2958962110 >> [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093] conn(0x7f9ab80ac270 msgr2=0x7f9ab80ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.359+0000 7f9ad4900700 1 --2- 172.21.15.44:0/2958962110 >> [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093] conn(0x7f9ab80ac270 0x7f9ab80ae700 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.359+0000 7f9ad4900700 1 -- 172.21.15.44:0/2958962110 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9ab80a1260 msgr2=0x7f9ab80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.359+0000 7f9ad4900700 1 --2- 172.21.15.44:0/2958962110 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9ab80a1260 0x7f9ab80a3710 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0006010 tx=0x7f9ac000b540 comp rx=0 tx=0).stop 2023-11-16T16:48:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.359+0000 7f9ad4900700 1 -- 172.21.15.44:0/2958962110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0104500 msgr2=0x7f9ad0106b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.359+0000 7f9ad4900700 1 --2- 172.21.15.44:0/2958962110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ad0104500 0x7f9ad0106b20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9ac800cc50 tx=0x7f9ac800cf60 comp rx=0 tx=0).stop 2023-11-16T16:48:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.359+0000 7f9aced9d700 1 -- 172.21.15.44:0/2958962110 reap_dead start 2023-11-16T16:48:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.359+0000 7f9ad4900700 1 -- 172.21.15.44:0/2958962110 shutdown_connections 2023-11-16T16:48:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.359+0000 7f9ad4900700 1 -- 172.21.15.44:0/2958962110 >> 172.21.15.44:0/2958962110 conn(0x7f9ad00770a0 msgr2=0x7f9ad0068ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.359+0000 7f9ad4900700 1 -- 172.21.15.44:0/2958962110 shutdown_connections 2023-11-16T16:48:19.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.360+0000 7f9ad4900700 1 -- 172.21.15.44:0/2958962110 wait complete. 2023-11-16T16:48:19.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:19 smithi176 ceph-mon[131630]: pgmap v307: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:48:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.241+0000 7f61e4ef3700 1 -- 172.21.15.44:0/3799864258 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61e0100540 msgr2=0x7f61e0100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.241+0000 7f61e4ef3700 1 --2- 172.21.15.44:0/3799864258 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61e0100540 0x7f61e0100910 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f61d00099c0 tx=0x7f61d0009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.241+0000 7f61e4ef3700 1 -- 172.21.15.44:0/3799864258 shutdown_connections 2023-11-16T16:48:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.241+0000 7f61e4ef3700 1 --2- 172.21.15.44:0/3799864258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61e01055e0 0x7f61e010d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.241+0000 7f61e4ef3700 1 --2- 172.21.15.44:0/3799864258 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61e0100540 0x7f61e0100910 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.241+0000 7f61e4ef3700 1 --2- 172.21.15.44:0/3799864258 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61e0100ee0 0x7f61e0104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.241+0000 7f61e4ef3700 1 -- 172.21.15.44:0/3799864258 >> 172.21.15.44:0/3799864258 conn(0x7f61e00fbdc0 msgr2=0x7f61e00fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.242+0000 7f61e4ef3700 1 -- 172.21.15.44:0/3799864258 shutdown_connections 2023-11-16T16:48:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.242+0000 7f61e4ef3700 1 -- 172.21.15.44:0/3799864258 wait complete. 2023-11-16T16:48:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.243+0000 7f61e4ef3700 1 Processor -- start 2023-11-16T16:48:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.243+0000 7f61e4ef3700 1 -- start start 2023-11-16T16:48:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.244+0000 7f61e4ef3700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61e0100540 0x7f61e010fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.244+0000 7f61e4ef3700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61e0100ee0 0x7f61e0110410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.244+0000 7f61e4ef3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61e01055e0 0x7f61e01a2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.244+0000 7f61e4ef3700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e01a2c10 con 0x7f61e0100ee0 2023-11-16T16:48:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.244+0000 7f61e4ef3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e01a2d80 con 0x7f61e01055e0 2023-11-16T16:48:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.244+0000 7f61e4ef3700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e01a2f40 con 0x7f61e0100540 2023-11-16T16:48:19.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.244+0000 7f61ddd9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61e0100ee0 0x7f61e0110410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.244+0000 7f61ddd9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61e0100ee0 0x7f61e0110410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46982/0 (socket says 172.21.15.44:46982) 2023-11-16T16:48:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.244+0000 7f61ddd9b700 1 -- 172.21.15.44:0/4168393701 learned_addr learned my addr 172.21.15.44:0/4168393701 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.245+0000 7f61de59c700 1 --2- 172.21.15.44:0/4168393701 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61e0100540 0x7f61e010fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.245+0000 7f61ded9d700 1 --2- 172.21.15.44:0/4168393701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61e01055e0 0x7f61e01a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.245+0000 7f61ddd9b700 1 -- 172.21.15.44:0/4168393701 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61e0100540 msgr2=0x7f61e010fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.245+0000 7f61ddd9b700 1 --2- 172.21.15.44:0/4168393701 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61e0100540 0x7f61e010fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.245+0000 7f61ddd9b700 1 -- 172.21.15.44:0/4168393701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61e01055e0 msgr2=0x7f61e01a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.245+0000 7f61ddd9b700 1 --2- 172.21.15.44:0/4168393701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61e01055e0 0x7f61e01a2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.245+0000 7f61ddd9b700 1 -- 172.21.15.44:0/4168393701 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d00095f0 con 0x7f61e0100ee0 2023-11-16T16:48:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.246+0000 7f61ddd9b700 1 --2- 172.21.15.44:0/4168393701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61e0100ee0 0x7f61e0110410 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f61d400c8f0 tx=0x7f61d400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.246+0000 7f61cf7fe700 1 -- 172.21.15.44:0/4168393701 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d40078d0 con 0x7f61e0100ee0 2023-11-16T16:48:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.246+0000 7f61cf7fe700 1 -- 172.21.15.44:0/4168393701 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61d400f450 con 0x7f61e0100ee0 2023-11-16T16:48:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.246+0000 7f61cf7fe700 1 -- 172.21.15.44:0/4168393701 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d4018660 con 0x7f61e0100ee0 2023-11-16T16:48:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.246+0000 7f61e4ef3700 1 -- 172.21.15.44:0/4168393701 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e0073340 con 0x7f61e0100ee0 2023-11-16T16:48:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.246+0000 7f61e4ef3700 1 -- 172.21.15.44:0/4168393701 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e00731b0 con 0x7f61e0100ee0 2023-11-16T16:48:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.247+0000 7f61e4ef3700 1 -- 172.21.15.44:0/4168393701 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f61e0104860 con 0x7f61e0100ee0 2023-11-16T16:48:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.253+0000 7f61cf7fe700 1 -- 172.21.15.44:0/4168393701 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f61d400f5c0 con 0x7f61e0100ee0 2023-11-16T16:48:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.255+0000 7f61cf7fe700 1 --2- 172.21.15.44:0/4168393701 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f61c80a1260 0x7f61c80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.255+0000 7f61cf7fe700 1 -- 172.21.15.44:0/4168393701 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f61d40ccc60 con 0x7f61e0100ee0 2023-11-16T16:48:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.255+0000 7f61de59c700 1 --2- 172.21.15.44:0/4168393701 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f61c80a1260 0x7f61c80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.255+0000 7f61cf7fe700 1 --2- 172.21.15.44:0/4168393701 >> [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683] conn(0x7f61c80ac270 0x7f61c80ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.255+0000 7f61cf7fe700 1 -- 172.21.15.44:0/4168393701 --> [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61c80aedd0 con 0x7f61c80ac270 2023-11-16T16:48:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.255+0000 7f61cf7fe700 1 -- 172.21.15.44:0/4168393701 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61d40d12a0 con 0x7f61e0100ee0 2023-11-16T16:48:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.256+0000 7f61ded9d700 1 --2- 172.21.15.44:0/4168393701 >> [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683] conn(0x7f61c80ac270 0x7f61c80ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.257+0000 7f61de59c700 1 --2- 172.21.15.44:0/4168393701 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f61c80a1260 0x7f61c80a3710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f61e0113f00 tx=0x7f61d0003680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.257+0000 7f61ded9d700 1 --2- 172.21.15.44:0/4168393701 >> [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683] conn(0x7f61c80ac270 0x7f61c80ae700 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:19.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.258+0000 7f61cf7fe700 1 -- 172.21.15.44:0/4168393701 <== osd.5 v2:172.21.15.176:6808/609622683 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f61c80aedd0 con 0x7f61c80ac270 2023-11-16T16:48:19.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.293+0000 7f61e4ef3700 1 -- 172.21.15.44:0/4168393701 --> [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f61e00734d0 con 0x7f61c80ac270 2023-11-16T16:48:19.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.294+0000 7f61cf7fe700 1 -- 172.21.15.44:0/4168393701 <== osd.5 v2:172.21.15.176:6808/609622683 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f61e00734d0 con 0x7f61c80ac270 2023-11-16T16:48:19.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.294+0000 7f61e4ef3700 1 -- 172.21.15.44:0/4168393701 >> [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683] conn(0x7f61c80ac270 msgr2=0x7f61c80ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.294+0000 7f61e4ef3700 1 --2- 172.21.15.44:0/4168393701 >> [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683] conn(0x7f61c80ac270 0x7f61c80ae700 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.295+0000 7f61e4ef3700 1 -- 172.21.15.44:0/4168393701 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f61c80a1260 msgr2=0x7f61c80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.295+0000 7f61e4ef3700 1 --2- 172.21.15.44:0/4168393701 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f61c80a1260 0x7f61c80a3710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f61e0113f00 tx=0x7f61d0003680 comp rx=0 tx=0).stop 2023-11-16T16:48:19.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.295+0000 7f61e4ef3700 1 -- 172.21.15.44:0/4168393701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61e0100ee0 msgr2=0x7f61e0110410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.295+0000 7f61e4ef3700 1 --2- 172.21.15.44:0/4168393701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61e0100ee0 0x7f61e0110410 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f61d400c8f0 tx=0x7f61d400cc00 comp rx=0 tx=0).stop 2023-11-16T16:48:19.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.295+0000 7f61ded9d700 1 -- 172.21.15.44:0/4168393701 reap_dead start 2023-11-16T16:48:19.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.295+0000 7f61e4ef3700 1 -- 172.21.15.44:0/4168393701 shutdown_connections 2023-11-16T16:48:19.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.295+0000 7f61e4ef3700 1 -- 172.21.15.44:0/4168393701 >> 172.21.15.44:0/4168393701 conn(0x7f61e00fbdc0 msgr2=0x7f61e0103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:19.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.296+0000 7f61e4ef3700 1 -- 172.21.15.44:0/4168393701 shutdown_connections 2023-11-16T16:48:19.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.296+0000 7f61e4ef3700 1 -- 172.21.15.44:0/4168393701 wait complete. 2023-11-16T16:48:19.754 INFO:teuthology.orchestra.run.smithi044.stdout:219043332138 2023-11-16T16:48:19.754 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.7 2023-11-16T16:48:19.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.436+0000 7fd13be1d700 1 -- 172.21.15.44:0/1684914184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134100540 msgr2=0x7fd134100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.436+0000 7fd13be1d700 1 --2- 172.21.15.44:0/1684914184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134100540 0x7fd134100910 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd130009920 tx=0x7fd130009c30 comp rx=0 tx=0).stop 2023-11-16T16:48:19.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.436+0000 7fd13be1d700 1 -- 172.21.15.44:0/1684914184 shutdown_connections 2023-11-16T16:48:19.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.436+0000 7fd13be1d700 1 --2- 172.21.15.44:0/1684914184 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd1341055e0 0x7fd13410d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.436+0000 7fd13be1d700 1 --2- 172.21.15.44:0/1684914184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134100540 0x7fd134100910 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.436+0000 7fd13be1d700 1 --2- 172.21.15.44:0/1684914184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd134100ee0 0x7fd134104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.436+0000 7fd13be1d700 1 -- 172.21.15.44:0/1684914184 >> 172.21.15.44:0/1684914184 conn(0x7fd1340fbdc0 msgr2=0x7fd1340fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:19.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.436+0000 7fd13be1d700 1 -- 172.21.15.44:0/1684914184 shutdown_connections 2023-11-16T16:48:19.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.436+0000 7fd13be1d700 1 -- 172.21.15.44:0/1684914184 wait complete. 2023-11-16T16:48:19.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.437+0000 7fd13be1d700 1 Processor -- start 2023-11-16T16:48:19.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.437+0000 7fd13be1d700 1 -- start start 2023-11-16T16:48:19.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.438+0000 7fd13be1d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134100540 0x7fd1341986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.438+0000 7fd13be1d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd134100ee0 0x7fd134198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.438+0000 7fd13be1d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd1341055e0 0x7fd13419cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.438+0000 7fd13be1d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13419d5f0 con 0x7fd134100540 2023-11-16T16:48:19.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.438+0000 7fd13be1d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13419d760 con 0x7fd134100ee0 2023-11-16T16:48:19.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.438+0000 7fd13be1d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13419d920 con 0x7fd1341055e0 2023-11-16T16:48:19.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.438+0000 7fd139bb9700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134100540 0x7fd1341986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.438+0000 7fd139bb9700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134100540 0x7fd1341986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47004/0 (socket says 172.21.15.44:47004) 2023-11-16T16:48:19.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.438+0000 7fd139bb9700 1 -- 172.21.15.44:0/2350959470 learned_addr learned my addr 172.21.15.44:0/2350959470 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:19.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.438+0000 7fd1393b8700 1 --2- 172.21.15.44:0/2350959470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd134100ee0 0x7fd134198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.438+0000 7fd13a3ba700 1 --2- 172.21.15.44:0/2350959470 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd1341055e0 0x7fd13419cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.439+0000 7fd139bb9700 1 -- 172.21.15.44:0/2350959470 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd1341055e0 msgr2=0x7fd13419cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.439+0000 7fd139bb9700 1 --2- 172.21.15.44:0/2350959470 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd1341055e0 0x7fd13419cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.439+0000 7fd139bb9700 1 -- 172.21.15.44:0/2350959470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd134100ee0 msgr2=0x7fd134198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.439+0000 7fd139bb9700 1 --2- 172.21.15.44:0/2350959470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd134100ee0 0x7fd134198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.439+0000 7fd139bb9700 1 -- 172.21.15.44:0/2350959470 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd130009520 con 0x7fd134100540 2023-11-16T16:48:19.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.439+0000 7fd139bb9700 1 --2- 172.21.15.44:0/2350959470 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134100540 0x7fd1341986d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd130009230 tx=0x7fd13000bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:19.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.439+0000 7fd12affd700 1 -- 172.21.15.44:0/2350959470 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13001c070 con 0x7fd134100540 2023-11-16T16:48:19.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.439+0000 7fd13be1d700 1 -- 172.21.15.44:0/2350959470 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd134110370 con 0x7fd134100540 2023-11-16T16:48:19.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.439+0000 7fd12affd700 1 -- 172.21.15.44:0/2350959470 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd130005040 con 0x7fd134100540 2023-11-16T16:48:19.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.439+0000 7fd12affd700 1 -- 172.21.15.44:0/2350959470 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130004af0 con 0x7fd134100540 2023-11-16T16:48:19.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.440+0000 7fd13be1d700 1 -- 172.21.15.44:0/2350959470 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1341101e0 con 0x7fd134100540 2023-11-16T16:48:19.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.441+0000 7fd13be1d700 1 -- 172.21.15.44:0/2350959470 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd134073ff0 con 0x7fd134100540 2023-11-16T16:48:19.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.444+0000 7fd12affd700 1 -- 172.21.15.44:0/2350959470 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd130020070 con 0x7fd134100540 2023-11-16T16:48:19.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.446+0000 7fd12affd700 1 --2- 172.21.15.44:0/2350959470 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd1200a1260 0x7fd1200a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.446+0000 7fd12affd700 1 -- 172.21.15.44:0/2350959470 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd1300cde90 con 0x7fd134100540 2023-11-16T16:48:19.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.446+0000 7fd12affd700 1 --2- 172.21.15.44:0/2350959470 >> [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975] conn(0x7fd1200ac270 0x7fd1200ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.446+0000 7fd12affd700 1 -- 172.21.15.44:0/2350959470 --> [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1200aedd0 con 0x7fd1200ac270 2023-11-16T16:48:19.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.446+0000 7fd12affd700 1 -- 172.21.15.44:0/2350959470 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1300ce2a0 con 0x7fd134100540 2023-11-16T16:48:19.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.447+0000 7fd1393b8700 1 --2- 172.21.15.44:0/2350959470 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd1200a1260 0x7fd1200a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.447+0000 7fd13a3ba700 1 --2- 172.21.15.44:0/2350959470 >> [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975] conn(0x7fd1200ac270 0x7fd1200ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.447+0000 7fd1393b8700 1 --2- 172.21.15.44:0/2350959470 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd1200a1260 0x7fd1200a3710 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd134199cf0 tx=0x7fd124009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.447+0000 7fd13a3ba700 1 --2- 172.21.15.44:0/2350959470 >> [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975] conn(0x7fd1200ac270 0x7fd1200ae700 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.448+0000 7fd12affd700 1 -- 172.21.15.44:0/2350959470 <== osd.1 v2:172.21.15.77:6800/1505198975 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd1200aedd0 con 0x7fd1200ac270 2023-11-16T16:48:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.466+0000 7fd13be1d700 1 -- 172.21.15.44:0/2350959470 --> [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd134110500 con 0x7fd1200ac270 2023-11-16T16:48:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.467+0000 7fd12affd700 1 -- 172.21.15.44:0/2350959470 <== osd.1 v2:172.21.15.77:6800/1505198975 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd134110500 con 0x7fd1200ac270 2023-11-16T16:48:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.467+0000 7fd13be1d700 1 -- 172.21.15.44:0/2350959470 >> [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975] conn(0x7fd1200ac270 msgr2=0x7fd1200ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.467+0000 7fd13be1d700 1 --2- 172.21.15.44:0/2350959470 >> [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975] conn(0x7fd1200ac270 0x7fd1200ae700 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.467+0000 7fd13be1d700 1 -- 172.21.15.44:0/2350959470 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd1200a1260 msgr2=0x7fd1200a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.467+0000 7fd13be1d700 1 --2- 172.21.15.44:0/2350959470 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd1200a1260 0x7fd1200a3710 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd134199cf0 tx=0x7fd124009310 comp rx=0 tx=0).stop 2023-11-16T16:48:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.467+0000 7fd13be1d700 1 -- 172.21.15.44:0/2350959470 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134100540 msgr2=0x7fd1341986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.467+0000 7fd13be1d700 1 --2- 172.21.15.44:0/2350959470 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134100540 0x7fd1341986d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd130009230 tx=0x7fd13000bf60 comp rx=0 tx=0).stop 2023-11-16T16:48:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.468+0000 7fd13a3ba700 1 -- 172.21.15.44:0/2350959470 reap_dead start 2023-11-16T16:48:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.468+0000 7fd13be1d700 1 -- 172.21.15.44:0/2350959470 shutdown_connections 2023-11-16T16:48:19.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.468+0000 7fd13be1d700 1 -- 172.21.15.44:0/2350959470 >> 172.21.15.44:0/2350959470 conn(0x7fd1340fbdc0 msgr2=0x7fd1340fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:19.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.468+0000 7fd13be1d700 1 -- 172.21.15.44:0/2350959470 shutdown_connections 2023-11-16T16:48:19.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:19.468+0000 7fd13be1d700 1 -- 172.21.15.44:0/2350959470 wait complete. 2023-11-16T16:48:20.131 INFO:teuthology.orchestra.run.smithi044.stdout:283467841568 2023-11-16T16:48:20.131 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.9 2023-11-16T16:48:20.363 INFO:teuthology.orchestra.run.smithi044.stdout:141733920823 2023-11-16T16:48:20.363 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.4 2023-11-16T16:48:20.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.644+0000 7fca7e214700 1 -- 172.21.15.44:0/3315338134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca780ffe00 msgr2=0x7fca78103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:20.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.644+0000 7fca7e214700 1 --2- 172.21.15.44:0/3315338134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca780ffe00 0x7fca78103dd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0099c0 tx=0x7fca6c009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:20.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.644+0000 7fca7e214700 1 -- 172.21.15.44:0/3315338134 shutdown_connections 2023-11-16T16:48:20.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.644+0000 7fca7e214700 1 --2- 172.21.15.44:0/3315338134 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca78104500 0x7fca7810c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:20.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.644+0000 7fca7e214700 1 --2- 172.21.15.44:0/3315338134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca780ff460 0x7fca780ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:20.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.644+0000 7fca7e214700 1 --2- 172.21.15.44:0/3315338134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca780ffe00 0x7fca78103dd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:20.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.644+0000 7fca7e214700 1 -- 172.21.15.44:0/3315338134 >> 172.21.15.44:0/3315338134 conn(0x7fca780770a0 msgr2=0x7fca780774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:20.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.644+0000 7fca7e214700 1 -- 172.21.15.44:0/3315338134 shutdown_connections 2023-11-16T16:48:20.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.644+0000 7fca7e214700 1 -- 172.21.15.44:0/3315338134 wait complete. 2023-11-16T16:48:20.649 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.645+0000 7fca7e214700 1 Processor -- start 2023-11-16T16:48:20.649 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.646+0000 7fca7e214700 1 -- start start 2023-11-16T16:48:20.649 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.646+0000 7fca7e214700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca780ff460 0x7fca78198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:20.649 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.646+0000 7fca7e214700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca780ffe00 0x7fca78198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:20.649 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.646+0000 7fca7e214700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca78104500 0x7fca7819cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:20.649 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.646+0000 7fca7e214700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7819d5e0 con 0x7fca780ffe00 2023-11-16T16:48:20.649 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.646+0000 7fca7e214700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7819d750 con 0x7fca780ff460 2023-11-16T16:48:20.649 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.646+0000 7fca7e214700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7819d910 con 0x7fca78104500 2023-11-16T16:48:20.649 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.646+0000 7fca76ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca780ffe00 0x7fca78198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:20.649 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.646+0000 7fca76ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca780ffe00 0x7fca78198b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47032/0 (socket says 172.21.15.44:47032) 2023-11-16T16:48:20.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.646+0000 7fca76ffd700 1 -- 172.21.15.44:0/3564252234 learned_addr learned my addr 172.21.15.44:0/3564252234 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:20.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.646+0000 7fca777fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca780ff460 0x7fca78198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:20.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.647+0000 7fca77fff700 1 --2- 172.21.15.44:0/3564252234 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca78104500 0x7fca7819cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:20.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.647+0000 7fca76ffd700 1 -- 172.21.15.44:0/3564252234 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca78104500 msgr2=0x7fca7819cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:20.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.647+0000 7fca76ffd700 1 --2- 172.21.15.44:0/3564252234 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca78104500 0x7fca7819cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:20.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.647+0000 7fca76ffd700 1 -- 172.21.15.44:0/3564252234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca780ff460 msgr2=0x7fca78198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:20.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.647+0000 7fca76ffd700 1 --2- 172.21.15.44:0/3564252234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca780ff460 0x7fca78198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:20.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.647+0000 7fca76ffd700 1 -- 172.21.15.44:0/3564252234 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6c0095f0 con 0x7fca780ffe00 2023-11-16T16:48:20.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.647+0000 7fca777fe700 1 --2- 172.21.15.44:0/3564252234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca780ff460 0x7fca78198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:20.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.647+0000 7fca76ffd700 1 --2- 172.21.15.44:0/3564252234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca780ffe00 0x7fca78198b60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009fd0 tx=0x7fca6c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:20.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.647+0000 7fca74ff9700 1 -- 172.21.15.44:0/3564252234 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c01d070 con 0x7fca780ffe00 2023-11-16T16:48:20.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.647+0000 7fca74ff9700 1 -- 172.21.15.44:0/3564252234 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca6c005050 con 0x7fca780ffe00 2023-11-16T16:48:20.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.647+0000 7fca7e214700 1 -- 172.21.15.44:0/3564252234 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7810f280 con 0x7fca780ffe00 2023-11-16T16:48:20.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.647+0000 7fca74ff9700 1 -- 172.21.15.44:0/3564252234 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c004a90 con 0x7fca780ffe00 2023-11-16T16:48:20.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.648+0000 7fca7e214700 1 -- 172.21.15.44:0/3564252234 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7810f0f0 con 0x7fca780ffe00 2023-11-16T16:48:20.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.648+0000 7fca7e214700 1 -- 172.21.15.44:0/3564252234 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fca780fe4b0 con 0x7fca780ffe00 2023-11-16T16:48:20.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.652+0000 7fca74ff9700 1 -- 172.21.15.44:0/3564252234 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fca6c005470 con 0x7fca780ffe00 2023-11-16T16:48:20.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.654+0000 7fca74ff9700 1 --2- 172.21.15.44:0/3564252234 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fca600a11e0 0x7fca600a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:20.656 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.654+0000 7fca74ff9700 1 -- 172.21.15.44:0/3564252234 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fca6c0ce2a0 con 0x7fca780ffe00 2023-11-16T16:48:20.656 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.654+0000 7fca777fe700 1 --2- 172.21.15.44:0/3564252234 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fca600a11e0 0x7fca600a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:20.656 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.654+0000 7fca74ff9700 1 --2- 172.21.15.44:0/3564252234 >> [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101] conn(0x7fca600ac1f0 0x7fca600ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:20.656 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.654+0000 7fca74ff9700 1 -- 172.21.15.44:0/3564252234 --> [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fca600aed50 con 0x7fca600ac1f0 2023-11-16T16:48:20.656 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.654+0000 7fca74ff9700 1 -- 172.21.15.44:0/3564252234 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca6c0ce610 con 0x7fca780ffe00 2023-11-16T16:48:20.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.655+0000 7fca77fff700 1 --2- 172.21.15.44:0/3564252234 >> [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101] conn(0x7fca600ac1f0 0x7fca600ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:20.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.655+0000 7fca777fe700 1 --2- 172.21.15.44:0/3564252234 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fca600a11e0 0x7fca600a3690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fca68007fd0 tx=0x7fca680073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:20.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.655+0000 7fca77fff700 1 --2- 172.21.15.44:0/3564252234 >> [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101] conn(0x7fca600ac1f0 0x7fca600ae680 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:20.658 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.656+0000 7fca74ff9700 1 -- 172.21.15.44:0/3564252234 <== osd.11 v2:172.21.15.176:6824/1098488101 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fca600aed50 con 0x7fca600ac1f0 2023-11-16T16:48:20.675 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.673+0000 7fca7e214700 1 -- 172.21.15.44:0/3564252234 --> [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fca7810f410 con 0x7fca600ac1f0 2023-11-16T16:48:20.676 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.674+0000 7fca74ff9700 1 -- 172.21.15.44:0/3564252234 <== osd.11 v2:172.21.15.176:6824/1098488101 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fca7810f410 con 0x7fca600ac1f0 2023-11-16T16:48:20.677 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.675+0000 7fca7e214700 1 -- 172.21.15.44:0/3564252234 >> [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101] conn(0x7fca600ac1f0 msgr2=0x7fca600ae680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:20.677 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.675+0000 7fca7e214700 1 --2- 172.21.15.44:0/3564252234 >> [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101] conn(0x7fca600ac1f0 0x7fca600ae680 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:20.677 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.675+0000 7fca7e214700 1 -- 172.21.15.44:0/3564252234 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fca600a11e0 msgr2=0x7fca600a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:20.677 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.675+0000 7fca7e214700 1 --2- 172.21.15.44:0/3564252234 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fca600a11e0 0x7fca600a3690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fca68007fd0 tx=0x7fca680073d0 comp rx=0 tx=0).stop 2023-11-16T16:48:20.677 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.675+0000 7fca7e214700 1 -- 172.21.15.44:0/3564252234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca780ffe00 msgr2=0x7fca78198b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:20.677 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.675+0000 7fca7e214700 1 --2- 172.21.15.44:0/3564252234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca780ffe00 0x7fca78198b60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009fd0 tx=0x7fca6c00f690 comp rx=0 tx=0).stop 2023-11-16T16:48:20.677 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.675+0000 7fca77fff700 1 -- 172.21.15.44:0/3564252234 reap_dead start 2023-11-16T16:48:20.677 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.675+0000 7fca7e214700 1 -- 172.21.15.44:0/3564252234 shutdown_connections 2023-11-16T16:48:20.677 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.675+0000 7fca7e214700 1 -- 172.21.15.44:0/3564252234 >> 172.21.15.44:0/3564252234 conn(0x7fca780770a0 msgr2=0x7fca78068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:20.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.675+0000 7fca7e214700 1 -- 172.21.15.44:0/3564252234 shutdown_connections 2023-11-16T16:48:20.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:20.675+0000 7fca7e214700 1 -- 172.21.15.44:0/3564252234 wait complete. 2023-11-16T16:48:20.896 INFO:teuthology.orchestra.run.smithi044.stdout:197568495663 2023-11-16T16:48:20.897 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.6 2023-11-16T16:48:20.967 INFO:teuthology.orchestra.run.smithi044.stdout:55834574917 2023-11-16T16:48:20.967 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.1 2023-11-16T16:48:21.383 INFO:teuthology.orchestra.run.smithi044.stdout:34359738441 2023-11-16T16:48:21.384 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.0 2023-11-16T16:48:21.610 INFO:teuthology.orchestra.run.smithi044.stdout:335007449114 2023-11-16T16:48:21.610 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.11 2023-11-16T16:48:21.678 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:21 smithi044 ceph-mon[128750]: pgmap v308: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:48:21.678 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:21 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:48:21.679 INFO:teuthology.orchestra.run.smithi044.stdout:304942678045 2023-11-16T16:48:21.680 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.10 2023-11-16T16:48:21.805 INFO:teuthology.orchestra.run.smithi044.stdout:163208757301 2023-11-16T16:48:21.805 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.5 2023-11-16T16:48:21.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:21 smithi176 ceph-mon[131630]: pgmap v308: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:48:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:21 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:48:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:21 smithi077 ceph-mon[131204]: pgmap v308: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:48:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:21 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:48:22.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.307+0000 7f8c67fd2700 1 -- 172.21.15.44:0/2770882446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c60119a40 msgr2=0x7f8c6011be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.307+0000 7f8c67fd2700 1 --2- 172.21.15.44:0/2770882446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c60119a40 0x7f8c6011be20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8c58009f90 tx=0x7f8c5800b330 comp rx=0 tx=0).stop 2023-11-16T16:48:22.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.308+0000 7f8c67fd2700 1 -- 172.21.15.44:0/2770882446 shutdown_connections 2023-11-16T16:48:22.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.308+0000 7f8c67fd2700 1 --2- 172.21.15.44:0/2770882446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c60119a40 0x7f8c6011be20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.308+0000 7f8c67fd2700 1 --2- 172.21.15.44:0/2770882446 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c60107af0 0x7f8c6010eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.308+0000 7f8c67fd2700 1 --2- 172.21.15.44:0/2770882446 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c60107ec0 0x7f8c60108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.308+0000 7f8c67fd2700 1 -- 172.21.15.44:0/2770882446 >> 172.21.15.44:0/2770882446 conn(0x7f8c6006c600 msgr2=0x7f8c600717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:22.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.308+0000 7f8c67fd2700 1 -- 172.21.15.44:0/2770882446 shutdown_connections 2023-11-16T16:48:22.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.308+0000 7f8c67fd2700 1 -- 172.21.15.44:0/2770882446 wait complete. 2023-11-16T16:48:22.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.310+0000 7f8c67fd2700 1 Processor -- start 2023-11-16T16:48:22.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.310+0000 7f8c67fd2700 1 -- start start 2023-11-16T16:48:22.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.311+0000 7f8c67fd2700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c60107af0 0x7f8c601a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:22.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.311+0000 7f8c67fd2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c60107ec0 0x7f8c601a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:22.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.311+0000 7f8c67fd2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c60119a40 0x7f8c601a9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:22.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.311+0000 7f8c67fd2700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c601aa0c0 con 0x7f8c60107af0 2023-11-16T16:48:22.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.311+0000 7f8c67fd2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c601aa230 con 0x7f8c60107ec0 2023-11-16T16:48:22.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.311+0000 7f8c67fd2700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c601aa3f0 con 0x7f8c60119a40 2023-11-16T16:48:22.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.311+0000 7f8c6556d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c60107ec0 0x7f8c601a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:22.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.311+0000 7f8c6556d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c60107ec0 0x7f8c601a5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:33322/0 (socket says 172.21.15.44:33322) 2023-11-16T16:48:22.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.311+0000 7f8c6556d700 1 -- 172.21.15.44:0/4001150987 learned_addr learned my addr 172.21.15.44:0/4001150987 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:22.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.312+0000 7f8c65d6e700 1 --2- 172.21.15.44:0/4001150987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c60107af0 0x7f8c601a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:22.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.312+0000 7f8c6656f700 1 --2- 172.21.15.44:0/4001150987 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c60119a40 0x7f8c601a9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:22.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.312+0000 7f8c6556d700 1 -- 172.21.15.44:0/4001150987 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c60119a40 msgr2=0x7f8c601a9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.312+0000 7f8c6556d700 1 --2- 172.21.15.44:0/4001150987 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c60119a40 0x7f8c601a9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.312+0000 7f8c6556d700 1 -- 172.21.15.44:0/4001150987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c60107af0 msgr2=0x7f8c601a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.312+0000 7f8c6556d700 1 --2- 172.21.15.44:0/4001150987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c60107af0 0x7f8c601a5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.312+0000 7f8c6556d700 1 -- 172.21.15.44:0/4001150987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c500095f0 con 0x7f8c60107ec0 2023-11-16T16:48:22.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.312+0000 7f8c6656f700 1 --2- 172.21.15.44:0/4001150987 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c60119a40 0x7f8c601a9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:48:22.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.312+0000 7f8c65d6e700 1 --2- 172.21.15.44:0/4001150987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c60107af0 0x7f8c601a5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:22.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.313+0000 7f8c6556d700 1 --2- 172.21.15.44:0/4001150987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c60107ec0 0x7f8c601a5640 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8c5000ec10 tx=0x7f8c5000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.313+0000 7f8c56ffd700 1 -- 172.21.15.44:0/4001150987 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5000cd40 con 0x7f8c60107ec0 2023-11-16T16:48:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.313+0000 7f8c56ffd700 1 -- 172.21.15.44:0/4001150987 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c500044e0 con 0x7f8c60107ec0 2023-11-16T16:48:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.313+0000 7f8c56ffd700 1 -- 172.21.15.44:0/4001150987 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c50010690 con 0x7f8c60107ec0 2023-11-16T16:48:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.313+0000 7f8c67fd2700 1 -- 172.21.15.44:0/4001150987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c58009c40 con 0x7f8c60107ec0 2023-11-16T16:48:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.313+0000 7f8c67fd2700 1 -- 172.21.15.44:0/4001150987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6011f160 con 0x7f8c60107ec0 2023-11-16T16:48:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.315+0000 7f8c67fd2700 1 -- 172.21.15.44:0/4001150987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8c60061230 con 0x7f8c60107ec0 2023-11-16T16:48:22.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.319+0000 7f8c56ffd700 1 -- 172.21.15.44:0/4001150987 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8c50003780 con 0x7f8c60107ec0 2023-11-16T16:48:22.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.321+0000 7f8c56ffd700 1 --2- 172.21.15.44:0/4001150987 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8c4c0a1260 0x7f8c4c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:22.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.321+0000 7f8c56ffd700 1 -- 172.21.15.44:0/4001150987 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8c50014070 con 0x7f8c60107ec0 2023-11-16T16:48:22.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.321+0000 7f8c56ffd700 1 --2- 172.21.15.44:0/4001150987 >> [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338] conn(0x7f8c4c0ac270 0x7f8c4c0ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:22.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.321+0000 7f8c56ffd700 1 -- 172.21.15.44:0/4001150987 --> [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c4c0aedd0 con 0x7f8c4c0ac270 2023-11-16T16:48:22.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.321+0000 7f8c56ffd700 1 -- 172.21.15.44:0/4001150987 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c500cd8d0 con 0x7f8c60107ec0 2023-11-16T16:48:22.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.321+0000 7f8c65d6e700 1 --2- 172.21.15.44:0/4001150987 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8c4c0a1260 0x7f8c4c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:22.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.321+0000 7f8c6656f700 1 --2- 172.21.15.44:0/4001150987 >> [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338] conn(0x7f8c4c0ac270 0x7f8c4c0ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:22.324 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.322+0000 7f8c6656f700 1 --2- 172.21.15.44:0/4001150987 >> [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338] conn(0x7f8c4c0ac270 0x7f8c4c0ae700 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:22.324 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.322+0000 7f8c65d6e700 1 --2- 172.21.15.44:0/4001150987 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8c4c0a1260 0x7f8c4c0a3710 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00afd0 tx=0x7f8c5c00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:22.325 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.323+0000 7f8c56ffd700 1 -- 172.21.15.44:0/4001150987 <== osd.8 v2:172.21.15.176:6816/832350338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8c4c0aedd0 con 0x7f8c4c0ac270 2023-11-16T16:48:22.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.341+0000 7f8c67fd2700 1 -- 172.21.15.44:0/4001150987 --> [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8c6011efd0 con 0x7f8c4c0ac270 2023-11-16T16:48:22.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.342+0000 7f8c56ffd700 1 -- 172.21.15.44:0/4001150987 <== osd.8 v2:172.21.15.176:6816/832350338 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8c6011efd0 con 0x7f8c4c0ac270 2023-11-16T16:48:22.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.343+0000 7f8c67fd2700 1 -- 172.21.15.44:0/4001150987 >> [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338] conn(0x7f8c4c0ac270 msgr2=0x7f8c4c0ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.343+0000 7f8c67fd2700 1 --2- 172.21.15.44:0/4001150987 >> [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338] conn(0x7f8c4c0ac270 0x7f8c4c0ae700 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.343+0000 7f8c67fd2700 1 -- 172.21.15.44:0/4001150987 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8c4c0a1260 msgr2=0x7f8c4c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.343+0000 7f8c67fd2700 1 --2- 172.21.15.44:0/4001150987 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8c4c0a1260 0x7f8c4c0a3710 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00afd0 tx=0x7f8c5c00a380 comp rx=0 tx=0).stop 2023-11-16T16:48:22.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.344+0000 7f8c67fd2700 1 -- 172.21.15.44:0/4001150987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c60107ec0 msgr2=0x7f8c601a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.344+0000 7f8c67fd2700 1 --2- 172.21.15.44:0/4001150987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c60107ec0 0x7f8c601a5640 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8c5000ec10 tx=0x7f8c5000ef20 comp rx=0 tx=0).stop 2023-11-16T16:48:22.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.344+0000 7f8c6656f700 1 -- 172.21.15.44:0/4001150987 reap_dead start 2023-11-16T16:48:22.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.344+0000 7f8c67fd2700 1 -- 172.21.15.44:0/4001150987 shutdown_connections 2023-11-16T16:48:22.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.344+0000 7f8c67fd2700 1 -- 172.21.15.44:0/4001150987 >> 172.21.15.44:0/4001150987 conn(0x7f8c6006c600 msgr2=0x7f8c6011a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:22.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.344+0000 7f8c67fd2700 1 -- 172.21.15.44:0/4001150987 shutdown_connections 2023-11-16T16:48:22.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:22.344+0000 7f8c67fd2700 1 -- 172.21.15.44:0/4001150987 wait complete. 2023-11-16T16:48:22.706 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:22.724 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:22.856 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:23.112 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:23.159 INFO:teuthology.orchestra.run.smithi044.stdout:249108103207 2023-11-16T16:48:23.160 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.8 2023-11-16T16:48:23.732 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:23 smithi044 ceph-mon[128750]: pgmap v309: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:23.733 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:23 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:23.733 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:23 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:23.733 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:23 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:23.733 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:23 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:23.764 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:23.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:23 smithi176 ceph-mon[131630]: pgmap v309: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:23.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:23 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:23 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:23 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:23 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:23.896 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:23 smithi077 ceph-mon[131204]: pgmap v309: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:23 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:23 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:23 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:23 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:25.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:25 smithi044 ceph-mon[128750]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:25 smithi176 ceph-mon[131630]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:25 smithi077 ceph-mon[131204]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:26.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.630+0000 7fa31e786700 1 -- 172.21.15.44:0/680203622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa31810cc80 msgr2=0x7fa31810f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:26.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.630+0000 7fa31e786700 1 --2- 172.21.15.44:0/680203622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa31810cc80 0x7fa31810f060 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa30800b0a0 tx=0x7fa30800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:26.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.631+0000 7fa31e786700 1 -- 172.21.15.44:0/680203622 shutdown_connections 2023-11-16T16:48:26.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.631+0000 7fa31e786700 1 --2- 172.21.15.44:0/680203622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa31810cc80 0x7fa31810f060 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:26.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.631+0000 7fa31e786700 1 --2- 172.21.15.44:0/680203622 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3180ff0d0 0x7fa3180ff4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:26.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.631+0000 7fa31e786700 1 --2- 172.21.15.44:0/680203622 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3180ff9e0 0x7fa31810c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:26.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.631+0000 7fa31e786700 1 -- 172.21.15.44:0/680203622 >> 172.21.15.44:0/680203622 conn(0x7fa3180770a0 msgr2=0x7fa3180774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:26.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.632+0000 7fa31e786700 1 -- 172.21.15.44:0/680203622 shutdown_connections 2023-11-16T16:48:26.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.632+0000 7fa31e786700 1 -- 172.21.15.44:0/680203622 wait complete. 2023-11-16T16:48:26.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.633+0000 7fa31e786700 1 Processor -- start 2023-11-16T16:48:26.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.633+0000 7fa31e786700 1 -- start start 2023-11-16T16:48:26.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.633+0000 7fa31e786700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3180ff0d0 0x7fa31806bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:26.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.633+0000 7fa31e786700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3180ff9e0 0x7fa31806c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:26.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.633+0000 7fa31e786700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa31810cc80 0x7fa318070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:26.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.634+0000 7fa31e786700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa318070db0 con 0x7fa31810cc80 2023-11-16T16:48:26.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.634+0000 7fa31e786700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa318070f20 con 0x7fa3180ff0d0 2023-11-16T16:48:26.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.634+0000 7fa31e786700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3180710e0 con 0x7fa3180ff9e0 2023-11-16T16:48:26.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.634+0000 7fa317fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3180ff0d0 0x7fa31806bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:26.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.634+0000 7fa31cd23700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa31810cc80 0x7fa318070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:26.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.634+0000 7fa317fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3180ff0d0 0x7fa31806bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:50050/0 (socket says 172.21.15.44:50050) 2023-11-16T16:48:26.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.634+0000 7fa31cd23700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa31810cc80 0x7fa318070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35724/0 (socket says 172.21.15.44:35724) 2023-11-16T16:48:26.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.634+0000 7fa317fff700 1 -- 172.21.15.44:0/1084816265 learned_addr learned my addr 172.21.15.44:0/1084816265 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:26.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.634+0000 7fa31cd23700 1 -- 172.21.15.44:0/1084816265 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3180ff9e0 msgr2=0x7fa31806c230 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-16T16:48:26.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.634+0000 7fa31cd23700 1 --2- 172.21.15.44:0/1084816265 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3180ff9e0 0x7fa31806c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:26.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.634+0000 7fa31cd23700 1 -- 172.21.15.44:0/1084816265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3180ff0d0 msgr2=0x7fa31806bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:26.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.634+0000 7fa31cd23700 1 --2- 172.21.15.44:0/1084816265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3180ff0d0 0x7fa31806bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:26.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.634+0000 7fa31cd23700 1 -- 172.21.15.44:0/1084816265 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa308009d00 con 0x7fa31810cc80 2023-11-16T16:48:26.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.635+0000 7fa31cd23700 1 --2- 172.21.15.44:0/1084816265 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa31810cc80 0x7fa318070870 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa30800b070 tx=0x7fa308004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:26.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.635+0000 7fa3157fa700 1 -- 172.21.15.44:0/1084816265 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30800e040 con 0x7fa31810cc80 2023-11-16T16:48:26.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.635+0000 7fa3157fa700 1 -- 172.21.15.44:0/1084816265 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa308003a00 con 0x7fa31810cc80 2023-11-16T16:48:26.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.635+0000 7fa3157fa700 1 -- 172.21.15.44:0/1084816265 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa308004590 con 0x7fa31810cc80 2023-11-16T16:48:26.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.636+0000 7fa31e786700 1 -- 172.21.15.44:0/1084816265 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31806ab30 con 0x7fa31810cc80 2023-11-16T16:48:26.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.637+0000 7fa31e786700 1 -- 172.21.15.44:0/1084816265 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31806a9a0 con 0x7fa31810cc80 2023-11-16T16:48:26.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.650+0000 7fa31e786700 1 -- 172.21.15.44:0/1084816265 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31806acc0 con 0x7fa31810cc80 2023-11-16T16:48:26.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.655+0000 7fa3157fa700 1 -- 172.21.15.44:0/1084816265 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa308003b70 con 0x7fa31810cc80 2023-11-16T16:48:26.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.658+0000 7fa3157fa700 1 --2- 172.21.15.44:0/1084816265 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa3040a1330 0x7fa3040a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:26.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.659+0000 7fa3157fa700 1 -- 172.21.15.44:0/1084816265 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa3080cdb90 con 0x7fa31810cc80 2023-11-16T16:48:26.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.660+0000 7fa317fff700 1 --2- 172.21.15.44:0/1084816265 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa3040a1330 0x7fa3040a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:26.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.661+0000 7fa317fff700 1 --2- 172.21.15.44:0/1084816265 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa3040a1330 0x7fa3040a37e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa3100099d0 tx=0x7fa310008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:26.674 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.672+0000 7fa3157fa700 1 -- 172.21.15.44:0/1084816265 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa308096ea0 con 0x7fa31810cc80 2023-11-16T16:48:26.848 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:26.927 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.925+0000 7fa31e786700 1 -- 172.21.15.44:0/1084816265 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa318071680 con 0x7fa31810cc80 2023-11-16T16:48:26.927 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.926+0000 7fa3157fa700 1 -- 172.21.15.44:0/1084816265 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa30809a4c0 con 0x7fa31810cc80 2023-11-16T16:48:26.927 INFO:teuthology.orchestra.run.smithi044.stdout:197568495665 2023-11-16T16:48:26.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.928+0000 7fa31e786700 1 -- 172.21.15.44:0/1084816265 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa3040a1330 msgr2=0x7fa3040a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:26.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.928+0000 7fa31e786700 1 --2- 172.21.15.44:0/1084816265 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa3040a1330 0x7fa3040a37e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa3100099d0 tx=0x7fa310008040 comp rx=0 tx=0).stop 2023-11-16T16:48:26.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.928+0000 7fa31e786700 1 -- 172.21.15.44:0/1084816265 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa31810cc80 msgr2=0x7fa318070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:26.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.928+0000 7fa31e786700 1 --2- 172.21.15.44:0/1084816265 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa31810cc80 0x7fa318070870 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa30800b070 tx=0x7fa308004b00 comp rx=0 tx=0).stop 2023-11-16T16:48:26.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.928+0000 7fa31e786700 1 -- 172.21.15.44:0/1084816265 shutdown_connections 2023-11-16T16:48:26.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.928+0000 7fa31e786700 1 --2- 172.21.15.44:0/1084816265 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa3040a1330 0x7fa3040a37e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:26.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.928+0000 7fa31e786700 1 --2- 172.21.15.44:0/1084816265 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa31810cc80 0x7fa318070870 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:26.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.928+0000 7fa31e786700 1 --2- 172.21.15.44:0/1084816265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3180ff0d0 0x7fa31806bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:26.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.928+0000 7fa31e786700 1 --2- 172.21.15.44:0/1084816265 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3180ff9e0 0x7fa31806c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:26.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.928+0000 7fa31e786700 1 -- 172.21.15.44:0/1084816265 >> 172.21.15.44:0/1084816265 conn(0x7fa3180770a0 msgr2=0x7fa31810e090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:26.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.928+0000 7fa31e786700 1 -- 172.21.15.44:0/1084816265 shutdown_connections 2023-11-16T16:48:26.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:26.928+0000 7fa31e786700 1 -- 172.21.15.44:0/1084816265 wait complete. 2023-11-16T16:48:27.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.139+0000 7f5f289f3700 1 -- 172.21.15.44:0/545992910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f24100ee0 msgr2=0x7f5f24104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:27.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.139+0000 7f5f289f3700 1 --2- 172.21.15.44:0/545992910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f24100ee0 0x7f5f24104eb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5f180099c0 tx=0x7f5f18009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.139+0000 7f5f289f3700 1 -- 172.21.15.44:0/545992910 shutdown_connections 2023-11-16T16:48:27.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.139+0000 7f5f289f3700 1 --2- 172.21.15.44:0/545992910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f241055e0 0x7f5f2410d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.139+0000 7f5f289f3700 1 --2- 172.21.15.44:0/545992910 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f24100540 0x7f5f24100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.139+0000 7f5f289f3700 1 --2- 172.21.15.44:0/545992910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f24100ee0 0x7f5f24104eb0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.139+0000 7f5f289f3700 1 -- 172.21.15.44:0/545992910 >> 172.21.15.44:0/545992910 conn(0x7f5f240fbdc0 msgr2=0x7f5f240fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:27.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.139+0000 7f5f289f3700 1 -- 172.21.15.44:0/545992910 shutdown_connections 2023-11-16T16:48:27.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.139+0000 7f5f289f3700 1 -- 172.21.15.44:0/545992910 wait complete. 2023-11-16T16:48:27.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.140+0000 7f5f289f3700 1 Processor -- start 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.140+0000 7f5f289f3700 1 -- start start 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.140+0000 7f5f289f3700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f24100540 0x7f5f24072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.140+0000 7f5f289f3700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f24100ee0 0x7f5f24073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.140+0000 7f5f289f3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f241055e0 0x7f5f24073870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.140+0000 7f5f289f3700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f241a8d80 con 0x7f5f24100ee0 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.141+0000 7f5f289f3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f241a8ec0 con 0x7f5f241055e0 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.141+0000 7f5f289f3700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f241a9000 con 0x7f5f24100540 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.141+0000 7f5f22d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f241055e0 0x7f5f24073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.141+0000 7f5f2259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f24100540 0x7f5f24072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.141+0000 7f5f21d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f24100ee0 0x7f5f24073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.141+0000 7f5f2259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f24100540 0x7f5f24072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:40148/0 (socket says 172.21.15.44:40148) 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.141+0000 7f5f2259c700 1 -- 172.21.15.44:0/2454835485 learned_addr learned my addr 172.21.15.44:0/2454835485 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:27.144 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.141+0000 7f5f21d9b700 1 -- 172.21.15.44:0/2454835485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f24100540 msgr2=0x7f5f24072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:27.144 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.141+0000 7f5f21d9b700 1 --2- 172.21.15.44:0/2454835485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f24100540 0x7f5f24072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.144 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.141+0000 7f5f21d9b700 1 -- 172.21.15.44:0/2454835485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f241055e0 msgr2=0x7f5f24073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:27.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.141+0000 7f5f21d9b700 1 --2- 172.21.15.44:0/2454835485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f241055e0 0x7f5f24073870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.141+0000 7f5f21d9b700 1 -- 172.21.15.44:0/2454835485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f180095f0 con 0x7f5f24100ee0 2023-11-16T16:48:27.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.141+0000 7f5f22d9d700 1 --2- 172.21.15.44:0/2454835485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f241055e0 0x7f5f24073870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:27.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.142+0000 7f5f21d9b700 1 --2- 172.21.15.44:0/2454835485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f24100ee0 0x7f5f24073330 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5f2406a3c0 tx=0x7f5f1800bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:27.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.142+0000 7f5f137fe700 1 -- 172.21.15.44:0/2454835485 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1801c070 con 0x7f5f24100ee0 2023-11-16T16:48:27.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.142+0000 7f5f137fe700 1 -- 172.21.15.44:0/2454835485 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f1800fc50 con 0x7f5f24100ee0 2023-11-16T16:48:27.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.142+0000 7f5f137fe700 1 -- 172.21.15.44:0/2454835485 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f18022bc0 con 0x7f5f24100ee0 2023-11-16T16:48:27.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.142+0000 7f5f289f3700 1 -- 172.21.15.44:0/2454835485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f24110360 con 0x7f5f24100ee0 2023-11-16T16:48:27.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.142+0000 7f5f289f3700 1 -- 172.21.15.44:0/2454835485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f241101d0 con 0x7f5f24100ee0 2023-11-16T16:48:27.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.143+0000 7f5f289f3700 1 -- 172.21.15.44:0/2454835485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f241104f0 con 0x7f5f24100ee0 2023-11-16T16:48:27.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.145+0000 7f5f137fe700 1 -- 172.21.15.44:0/2454835485 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5f18020070 con 0x7f5f24100ee0 2023-11-16T16:48:27.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.147+0000 7f5f137fe700 1 --2- 172.21.15.44:0/2454835485 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5f0c0a1260 0x7f5f0c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:27.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.147+0000 7f5f137fe700 1 -- 172.21.15.44:0/2454835485 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5f180cdbb0 con 0x7f5f24100ee0 2023-11-16T16:48:27.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.147+0000 7f5f2259c700 1 --2- 172.21.15.44:0/2454835485 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5f0c0a1260 0x7f5f0c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:27.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.148+0000 7f5f2259c700 1 --2- 172.21.15.44:0/2454835485 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5f0c0a1260 0x7f5f0c0a3710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5f14009d20 tx=0x7f5f14009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:27.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.151+0000 7f5f137fe700 1 -- 172.21.15.44:0/2454835485 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5f18096ec0 con 0x7f5f24100ee0 2023-11-16T16:48:27.342 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:27.402 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:27.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.438+0000 7f5f289f3700 1 -- 172.21.15.44:0/2454835485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f5f241a99b0 con 0x7f5f24100ee0 2023-11-16T16:48:27.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.440+0000 7f5f137fe700 1 -- 172.21.15.44:0/2454835485 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f5f1809a4e0 con 0x7f5f24100ee0 2023-11-16T16:48:27.442 INFO:teuthology.orchestra.run.smithi044.stdout:304942678047 2023-11-16T16:48:27.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.444+0000 7f5f289f3700 1 -- 172.21.15.44:0/2454835485 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5f0c0a1260 msgr2=0x7f5f0c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:27.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.444+0000 7f5f289f3700 1 --2- 172.21.15.44:0/2454835485 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5f0c0a1260 0x7f5f0c0a3710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5f14009d20 tx=0x7f5f14009260 comp rx=0 tx=0).stop 2023-11-16T16:48:27.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.444+0000 7f5f289f3700 1 -- 172.21.15.44:0/2454835485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f24100ee0 msgr2=0x7f5f24073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:27.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.444+0000 7f5f289f3700 1 --2- 172.21.15.44:0/2454835485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f24100ee0 0x7f5f24073330 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5f2406a3c0 tx=0x7f5f1800bfd0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.444+0000 7f5f289f3700 1 -- 172.21.15.44:0/2454835485 shutdown_connections 2023-11-16T16:48:27.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.444+0000 7f5f289f3700 1 --2- 172.21.15.44:0/2454835485 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5f0c0a1260 0x7f5f0c0a3710 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.444+0000 7f5f289f3700 1 --2- 172.21.15.44:0/2454835485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f241055e0 0x7f5f24073870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.444+0000 7f5f289f3700 1 --2- 172.21.15.44:0/2454835485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f24100540 0x7f5f24072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.444+0000 7f5f289f3700 1 --2- 172.21.15.44:0/2454835485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f24100ee0 0x7f5f24073330 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.444+0000 7f5f289f3700 1 -- 172.21.15.44:0/2454835485 >> 172.21.15.44:0/2454835485 conn(0x7f5f240fbdc0 msgr2=0x7f5f240fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:27.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.445+0000 7f5f289f3700 1 -- 172.21.15.44:0/2454835485 shutdown_connections 2023-11-16T16:48:27.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.445+0000 7f5f289f3700 1 -- 172.21.15.44:0/2454835485 wait complete. 2023-11-16T16:48:27.527 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:27.668 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.666+0000 7f62e5508700 1 -- 172.21.15.44:0/2456506286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62e0100ee0 msgr2=0x7f62e0104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:27.669 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.666+0000 7f62e5508700 1 --2- 172.21.15.44:0/2456506286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62e0100ee0 0x7f62e0104eb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009920 tx=0x7f62d4009c30 comp rx=0 tx=0).stop 2023-11-16T16:48:27.669 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.666+0000 7f62e5508700 1 -- 172.21.15.44:0/2456506286 shutdown_connections 2023-11-16T16:48:27.669 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.666+0000 7f62e5508700 1 --2- 172.21.15.44:0/2456506286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62e01055e0 0x7f62e010d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.669 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.666+0000 7f62e5508700 1 --2- 172.21.15.44:0/2456506286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62e0100540 0x7f62e0100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.669 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.666+0000 7f62e5508700 1 --2- 172.21.15.44:0/2456506286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62e0100ee0 0x7f62e0104eb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.669 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.666+0000 7f62e5508700 1 -- 172.21.15.44:0/2456506286 >> 172.21.15.44:0/2456506286 conn(0x7f62e00fbdc0 msgr2=0x7f62e00fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:27.669 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.667+0000 7f62e5508700 1 -- 172.21.15.44:0/2456506286 shutdown_connections 2023-11-16T16:48:27.669 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.667+0000 7f62e5508700 1 -- 172.21.15.44:0/2456506286 wait complete. 2023-11-16T16:48:27.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.668+0000 7f62e5508700 1 Processor -- start 2023-11-16T16:48:27.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.668+0000 7f62e5508700 1 -- start start 2023-11-16T16:48:27.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.668+0000 7f62e5508700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62e0100540 0x7f62e0198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:27.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.668+0000 7f62e5508700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62e0100ee0 0x7f62e0198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:27.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.668+0000 7f62e5508700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62e01055e0 0x7f62e019cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:27.671 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.668+0000 7f62e5508700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e019d600 con 0x7f62e01055e0 2023-11-16T16:48:27.671 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.668+0000 7f62e5508700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e019d770 con 0x7f62e0100ee0 2023-11-16T16:48:27.671 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.668+0000 7f62e5508700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e019d930 con 0x7f62e0100540 2023-11-16T16:48:27.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.669+0000 7f62df7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62e01055e0 0x7f62e019cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:27.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.669+0000 7f62df7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62e01055e0 0x7f62e019cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35752/0 (socket says 172.21.15.44:35752) 2023-11-16T16:48:27.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.669+0000 7f62df7fe700 1 -- 172.21.15.44:0/945619678 learned_addr learned my addr 172.21.15.44:0/945619678 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:27.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.669+0000 7f62deffd700 1 --2- 172.21.15.44:0/945619678 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62e0100540 0x7f62e0198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:27.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.669+0000 7f62de7fc700 1 --2- 172.21.15.44:0/945619678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62e0100ee0 0x7f62e0198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:27.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.669+0000 7f62df7fe700 1 -- 172.21.15.44:0/945619678 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62e0100540 msgr2=0x7f62e0198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:27.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.669+0000 7f62df7fe700 1 --2- 172.21.15.44:0/945619678 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62e0100540 0x7f62e0198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.669+0000 7f62df7fe700 1 -- 172.21.15.44:0/945619678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62e0100ee0 msgr2=0x7f62e0198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:27.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.669+0000 7f62df7fe700 1 --2- 172.21.15.44:0/945619678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62e0100ee0 0x7f62e0198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.669+0000 7f62df7fe700 1 -- 172.21.15.44:0/945619678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d4009520 con 0x7f62e01055e0 2023-11-16T16:48:27.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.670+0000 7f62df7fe700 1 --2- 172.21.15.44:0/945619678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62e01055e0 0x7f62e019cff0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f62d800ef30 tx=0x7f62d800c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:27.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.670+0000 7f62c7fff700 1 -- 172.21.15.44:0/945619678 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d8018070 con 0x7f62e01055e0 2023-11-16T16:48:27.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.670+0000 7f62c7fff700 1 -- 172.21.15.44:0/945619678 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62d800f040 con 0x7f62e01055e0 2023-11-16T16:48:27.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.670+0000 7f62c7fff700 1 -- 172.21.15.44:0/945619678 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d800cda0 con 0x7f62e01055e0 2023-11-16T16:48:27.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.670+0000 7f62e5508700 1 -- 172.21.15.44:0/945619678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e0110360 con 0x7f62e01055e0 2023-11-16T16:48:27.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.670+0000 7f62e5508700 1 -- 172.21.15.44:0/945619678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e01101d0 con 0x7f62e01055e0 2023-11-16T16:48:27.674 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.672+0000 7f62e5508700 1 -- 172.21.15.44:0/945619678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e01104f0 con 0x7f62e01055e0 2023-11-16T16:48:27.677 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.675+0000 7f62c7fff700 1 -- 172.21.15.44:0/945619678 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f62d8003a00 con 0x7f62e01055e0 2023-11-16T16:48:27.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.678+0000 7f62c7fff700 1 --2- 172.21.15.44:0/945619678 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f62c80a1260 0x7f62c80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:27.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.678+0000 7f62c7fff700 1 -- 172.21.15.44:0/945619678 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f62d80cc8c0 con 0x7f62e01055e0 2023-11-16T16:48:27.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.678+0000 7f62deffd700 1 --2- 172.21.15.44:0/945619678 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f62c80a1260 0x7f62c80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:27.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.679+0000 7f62deffd700 1 --2- 172.21.15.44:0/945619678 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f62c80a1260 0x7f62c80a3710 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f62d0005fd0 tx=0x7f62d0009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:27.682 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.681+0000 7f62c7fff700 1 -- 172.21.15.44:0/945619678 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f62d8095c30 con 0x7f62e01055e0 2023-11-16T16:48:27.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:27 smithi044 ceph-mon[128750]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:27.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:27 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1084816265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-11-16T16:48:27.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:27 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2454835485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-11-16T16:48:27.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:27 smithi176 ceph-mon[131630]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:27.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:27 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1084816265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-11-16T16:48:27.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:27 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2454835485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-11-16T16:48:27.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.913+0000 7f62e5508700 1 -- 172.21.15.44:0/945619678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f62e01a56c0 con 0x7f62e01055e0 2023-11-16T16:48:27.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.914+0000 7f62c7fff700 1 -- 172.21.15.44:0/945619678 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f62d8099250 con 0x7f62e01055e0 2023-11-16T16:48:27.915 INFO:teuthology.orchestra.run.smithi044.stdout:219043332140 2023-11-16T16:48:27.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.916+0000 7f62e5508700 1 -- 172.21.15.44:0/945619678 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f62c80a1260 msgr2=0x7f62c80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:27.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.916+0000 7f62e5508700 1 --2- 172.21.15.44:0/945619678 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f62c80a1260 0x7f62c80a3710 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f62d0005fd0 tx=0x7f62d0009310 comp rx=0 tx=0).stop 2023-11-16T16:48:27.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.916+0000 7f62e5508700 1 -- 172.21.15.44:0/945619678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62e01055e0 msgr2=0x7f62e019cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:27.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.916+0000 7f62e5508700 1 --2- 172.21.15.44:0/945619678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62e01055e0 0x7f62e019cff0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f62d800ef30 tx=0x7f62d800c450 comp rx=0 tx=0).stop 2023-11-16T16:48:27.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.916+0000 7f62e5508700 1 -- 172.21.15.44:0/945619678 shutdown_connections 2023-11-16T16:48:27.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.916+0000 7f62e5508700 1 --2- 172.21.15.44:0/945619678 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f62c80a1260 0x7f62c80a3710 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.916+0000 7f62e5508700 1 --2- 172.21.15.44:0/945619678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62e01055e0 0x7f62e019cff0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.916+0000 7f62e5508700 1 --2- 172.21.15.44:0/945619678 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62e0100540 0x7f62e0198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.916+0000 7f62e5508700 1 --2- 172.21.15.44:0/945619678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62e0100ee0 0x7f62e0198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.916+0000 7f62e5508700 1 -- 172.21.15.44:0/945619678 >> 172.21.15.44:0/945619678 conn(0x7f62e00fbdc0 msgr2=0x7f62e00fc880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:27.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.916+0000 7f62e5508700 1 -- 172.21.15.44:0/945619678 shutdown_connections 2023-11-16T16:48:27.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.916+0000 7f62e5508700 1 -- 172.21.15.44:0/945619678 wait complete. 2023-11-16T16:48:27.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:27 smithi077 ceph-mon[131204]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:27.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:27 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1084816265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-11-16T16:48:27.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:27 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2454835485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-11-16T16:48:28.269 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.791+0000 7fdc2f008700 1 -- 172.21.15.44:0/4186751378 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc28101e50 msgr2=0x7fdc2810a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:28.269 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.791+0000 7fdc2f008700 1 --2- 172.21.15.44:0/4186751378 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc28101e50 0x7fdc2810a350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdc14009990 tx=0x7fdc14009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:28.269 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.792+0000 7fdc2f008700 1 -- 172.21.15.44:0/4186751378 shutdown_connections 2023-11-16T16:48:28.269 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.792+0000 7fdc2f008700 1 --2- 172.21.15.44:0/4186751378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc2810a890 0x7fdc2810ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:28.269 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.792+0000 7fdc2f008700 1 --2- 172.21.15.44:0/4186751378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc28101540 0x7fdc28101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:28.269 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.792+0000 7fdc2f008700 1 --2- 172.21.15.44:0/4186751378 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc28101e50 0x7fdc2810a350 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.792+0000 7fdc2f008700 1 -- 172.21.15.44:0/4186751378 >> 172.21.15.44:0/4186751378 conn(0x7fdc280770a0 msgr2=0x7fdc280774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.792+0000 7fdc2f008700 1 -- 172.21.15.44:0/4186751378 shutdown_connections 2023-11-16T16:48:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.792+0000 7fdc2f008700 1 -- 172.21.15.44:0/4186751378 wait complete. 2023-11-16T16:48:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.793+0000 7fdc2f008700 1 Processor -- start 2023-11-16T16:48:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.793+0000 7fdc2f008700 1 -- start start 2023-11-16T16:48:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.794+0000 7fdc2f008700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc28101540 0x7fdc281963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.794+0000 7fdc2f008700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc28101e50 0x7fdc28196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.794+0000 7fdc2f008700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc2810a890 0x7fdc2819ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.794+0000 7fdc2f008700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2819b3b0 con 0x7fdc28101e50 2023-11-16T16:48:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.794+0000 7fdc2f008700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2819b520 con 0x7fdc28101540 2023-11-16T16:48:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.794+0000 7fdc2f008700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2819b6e0 con 0x7fdc2810a890 2023-11-16T16:48:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.794+0000 7fdc2d5a5700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc2810a890 0x7fdc2819ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.794+0000 7fdc27fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc28101e50 0x7fdc28196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.794+0000 7fdc2d5a5700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc2810a890 0x7fdc2819ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:40178/0 (socket says 172.21.15.44:40178) 2023-11-16T16:48:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.794+0000 7fdc2d5a5700 1 -- 172.21.15.44:0/4035866462 learned_addr learned my addr 172.21.15.44:0/4035866462 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.795+0000 7fdc2cda4700 1 --2- 172.21.15.44:0/4035866462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc28101540 0x7fdc281963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.795+0000 7fdc2d5a5700 1 -- 172.21.15.44:0/4035866462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc28101540 msgr2=0x7fdc281963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.795+0000 7fdc2d5a5700 1 --2- 172.21.15.44:0/4035866462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc28101540 0x7fdc281963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.795+0000 7fdc2d5a5700 1 -- 172.21.15.44:0/4035866462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc28101e50 msgr2=0x7fdc28196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.795+0000 7fdc2d5a5700 1 --2- 172.21.15.44:0/4035866462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc28101e50 0x7fdc28196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.795+0000 7fdc2d5a5700 1 -- 172.21.15.44:0/4035866462 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc20009c40 con 0x7fdc2810a890 2023-11-16T16:48:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.795+0000 7fdc2cda4700 1 --2- 172.21.15.44:0/4035866462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc28101540 0x7fdc281963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:48:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.795+0000 7fdc27fff700 1 --2- 172.21.15.44:0/4035866462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc28101e50 0x7fdc28196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.795+0000 7fdc2d5a5700 1 --2- 172.21.15.44:0/4035866462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc2810a890 0x7fdc2819ada0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdc2000c370 tx=0x7fdc2000c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.795+0000 7fdc25ffb700 1 -- 172.21.15.44:0/4035866462 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc20017070 con 0x7fdc2810a890 2023-11-16T16:48:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.796+0000 7fdc25ffb700 1 -- 172.21.15.44:0/4035866462 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc2000f040 con 0x7fdc2810a890 2023-11-16T16:48:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.796+0000 7fdc2f008700 1 -- 172.21.15.44:0/4035866462 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc140095f0 con 0x7fdc2810a890 2023-11-16T16:48:28.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.796+0000 7fdc25ffb700 1 -- 172.21.15.44:0/4035866462 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2000e050 con 0x7fdc2810a890 2023-11-16T16:48:28.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.796+0000 7fdc2f008700 1 -- 172.21.15.44:0/4035866462 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc28110310 con 0x7fdc2810a890 2023-11-16T16:48:28.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.797+0000 7fdc2f008700 1 -- 172.21.15.44:0/4035866462 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc28110180 con 0x7fdc2810a890 2023-11-16T16:48:28.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.799+0000 7fdc25ffb700 1 -- 172.21.15.44:0/4035866462 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fdc20003a00 con 0x7fdc2810a890 2023-11-16T16:48:28.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.800+0000 7fdc25ffb700 1 --2- 172.21.15.44:0/4035866462 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fdc180a11e0 0x7fdc180a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:28.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.800+0000 7fdc25ffb700 1 -- 172.21.15.44:0/4035866462 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdc200cc710 con 0x7fdc2810a890 2023-11-16T16:48:28.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.800+0000 7fdc2cda4700 1 --2- 172.21.15.44:0/4035866462 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fdc180a11e0 0x7fdc180a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:28.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.801+0000 7fdc2cda4700 1 --2- 172.21.15.44:0/4035866462 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fdc180a11e0 0x7fdc180a3690 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009cb0 tx=0x7fdc1c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:28.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:27.804+0000 7fdc25ffb700 1 -- 172.21.15.44:0/4035866462 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdc20095990 con 0x7fdc2810a890 2023-11-16T16:48:28.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.033+0000 7fdc2f008700 1 -- 172.21.15.44:0/4035866462 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fdc281104a0 con 0x7fdc2810a890 2023-11-16T16:48:28.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.034+0000 7fdc25ffb700 1 -- 172.21.15.44:0/4035866462 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fdc20098fb0 con 0x7fdc2810a890 2023-11-16T16:48:28.273 INFO:teuthology.orchestra.run.smithi044.stdout:107374182461 2023-11-16T16:48:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.036+0000 7fdc2f008700 1 -- 172.21.15.44:0/4035866462 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fdc180a11e0 msgr2=0x7fdc180a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.036+0000 7fdc2f008700 1 --2- 172.21.15.44:0/4035866462 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fdc180a11e0 0x7fdc180a3690 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009cb0 tx=0x7fdc1c009260 comp rx=0 tx=0).stop 2023-11-16T16:48:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.036+0000 7fdc2f008700 1 -- 172.21.15.44:0/4035866462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc2810a890 msgr2=0x7fdc2819ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.036+0000 7fdc2f008700 1 --2- 172.21.15.44:0/4035866462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc2810a890 0x7fdc2819ada0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdc2000c370 tx=0x7fdc2000c680 comp rx=0 tx=0).stop 2023-11-16T16:48:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.036+0000 7fdc2f008700 1 -- 172.21.15.44:0/4035866462 shutdown_connections 2023-11-16T16:48:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.036+0000 7fdc2f008700 1 --2- 172.21.15.44:0/4035866462 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fdc180a11e0 0x7fdc180a3690 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.036+0000 7fdc2f008700 1 --2- 172.21.15.44:0/4035866462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc2810a890 0x7fdc2819ada0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.036+0000 7fdc2f008700 1 --2- 172.21.15.44:0/4035866462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc28101540 0x7fdc281963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.036+0000 7fdc2f008700 1 --2- 172.21.15.44:0/4035866462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc28101e50 0x7fdc28196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:28.274 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.036+0000 7fdc2f008700 1 -- 172.21.15.44:0/4035866462 >> 172.21.15.44:0/4035866462 conn(0x7fdc280770a0 msgr2=0x7fdc280fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:28.274 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.036+0000 7fdc2f008700 1 -- 172.21.15.44:0/4035866462 shutdown_connections 2023-11-16T16:48:28.274 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.037+0000 7fdc2f008700 1 -- 172.21.15.44:0/4035866462 wait complete. 2023-11-16T16:48:28.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:28 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/945619678' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-11-16T16:48:28.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:28 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/4035866462' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-11-16T16:48:28.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:28 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/945619678' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-11-16T16:48:28.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:28 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/4035866462' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-11-16T16:48:28.943 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332138 got 219043332140 for osd.7 2023-11-16T16:48:28.943 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:28.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.944+0000 7fd277199700 1 -- 172.21.15.44:0/2736121358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2701055f0 msgr2=0x7fd27010d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:28.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.944+0000 7fd277199700 1 --2- 172.21.15.44:0/2736121358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2701055f0 0x7fd27010d170 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd26800b0d0 tx=0x7fd26800b3e0 comp rx=0 tx=0).stop 2023-11-16T16:48:28.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.944+0000 7fd277199700 1 -- 172.21.15.44:0/2736121358 shutdown_connections 2023-11-16T16:48:28.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.944+0000 7fd277199700 1 --2- 172.21.15.44:0/2736121358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2701055f0 0x7fd27010d170 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:28.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.944+0000 7fd277199700 1 --2- 172.21.15.44:0/2736121358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd270100550 0x7fd270100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:28.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.944+0000 7fd277199700 1 --2- 172.21.15.44:0/2736121358 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd270100ef0 0x7fd270104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:28.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.944+0000 7fd277199700 1 -- 172.21.15.44:0/2736121358 >> 172.21.15.44:0/2736121358 conn(0x7fd2700fbdf0 msgr2=0x7fd2700fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:28.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.944+0000 7fd277199700 1 -- 172.21.15.44:0/2736121358 shutdown_connections 2023-11-16T16:48:28.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.944+0000 7fd277199700 1 -- 172.21.15.44:0/2736121358 wait complete. 2023-11-16T16:48:28.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.945+0000 7fd277199700 1 Processor -- start 2023-11-16T16:48:28.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.945+0000 7fd277199700 1 -- start start 2023-11-16T16:48:28.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.945+0000 7fd277199700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd270100550 0x7fd2701071f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:28.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.946+0000 7fd277199700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd270100ef0 0x7fd270107730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:28.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.946+0000 7fd277199700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2701055f0 0x7fd270107c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:28.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.946+0000 7fd277199700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2701a8da0 con 0x7fd270100550 2023-11-16T16:48:28.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.946+0000 7fd277199700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2701a8f10 con 0x7fd270100ef0 2023-11-16T16:48:28.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.946+0000 7fd277199700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2701a90d0 con 0x7fd2701055f0 2023-11-16T16:48:28.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.946+0000 7fd274f35700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd270100550 0x7fd2701071f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:28.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.946+0000 7fd274f35700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd270100550 0x7fd2701071f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35780/0 (socket says 172.21.15.44:35780) 2023-11-16T16:48:28.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.946+0000 7fd274f35700 1 -- 172.21.15.44:0/1433126507 learned_addr learned my addr 172.21.15.44:0/1433126507 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:28.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.946+0000 7fd26ffff700 1 --2- 172.21.15.44:0/1433126507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd270100ef0 0x7fd270107730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:28.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.946+0000 7fd275736700 1 --2- 172.21.15.44:0/1433126507 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2701055f0 0x7fd270107c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:28.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.946+0000 7fd274f35700 1 -- 172.21.15.44:0/1433126507 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2701055f0 msgr2=0x7fd270107c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:28.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.946+0000 7fd274f35700 1 --2- 172.21.15.44:0/1433126507 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2701055f0 0x7fd270107c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:28.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.946+0000 7fd274f35700 1 -- 172.21.15.44:0/1433126507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd270100ef0 msgr2=0x7fd270107730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:28.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.946+0000 7fd274f35700 1 --2- 172.21.15.44:0/1433126507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd270100ef0 0x7fd270107730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:28.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.946+0000 7fd274f35700 1 -- 172.21.15.44:0/1433126507 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd268009d00 con 0x7fd270100550 2023-11-16T16:48:28.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.947+0000 7fd274f35700 1 --2- 172.21.15.44:0/1433126507 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd270100550 0x7fd2701071f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd26400e9a0 tx=0x7fd26400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:28.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.947+0000 7fd26dffb700 1 -- 172.21.15.44:0/1433126507 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264009790 con 0x7fd270100550 2023-11-16T16:48:28.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.947+0000 7fd26dffb700 1 -- 172.21.15.44:0/1433126507 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd264010430 con 0x7fd270100550 2023-11-16T16:48:28.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.947+0000 7fd26dffb700 1 -- 172.21.15.44:0/1433126507 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264018680 con 0x7fd270100550 2023-11-16T16:48:28.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.947+0000 7fd277199700 1 -- 172.21.15.44:0/1433126507 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2701103c0 con 0x7fd270100550 2023-11-16T16:48:28.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.947+0000 7fd277199700 1 -- 172.21.15.44:0/1433126507 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd270110230 con 0x7fd270100550 2023-11-16T16:48:28.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.948+0000 7fd277199700 1 -- 172.21.15.44:0/1433126507 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd270110550 con 0x7fd270100550 2023-11-16T16:48:28.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.950+0000 7fd26dffb700 1 -- 172.21.15.44:0/1433126507 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd2640105a0 con 0x7fd270100550 2023-11-16T16:48:28.953 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.952+0000 7fd26dffb700 1 --2- 172.21.15.44:0/1433126507 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd2600a1260 0x7fd2600a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:28.953 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.952+0000 7fd26dffb700 1 -- 172.21.15.44:0/1433126507 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd264014070 con 0x7fd270100550 2023-11-16T16:48:28.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.952+0000 7fd26ffff700 1 --2- 172.21.15.44:0/1433126507 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd2600a1260 0x7fd2600a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:28.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.952+0000 7fd26ffff700 1 --2- 172.21.15.44:0/1433126507 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd2600a1260 0x7fd2600a3710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd270108a90 tx=0x7fd25c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:28.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:28.956+0000 7fd26dffb700 1 -- 172.21.15.44:0/1433126507 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd264051730 con 0x7fd270100550 2023-11-16T16:48:28.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:28 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/945619678' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-11-16T16:48:28.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:28 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/4035866462' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-11-16T16:48:29.139 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678045 got 304942678047 for osd.10 2023-11-16T16:48:29.139 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:29.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:29.172+0000 7fd277199700 1 -- 172.21.15.44:0/1433126507 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd270066ea0 con 0x7fd270100550 2023-11-16T16:48:29.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:29.173+0000 7fd26dffb700 1 -- 172.21.15.44:0/1433126507 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd264096200 con 0x7fd270100550 2023-11-16T16:48:29.175 INFO:teuthology.orchestra.run.smithi044.stdout:77309411397 2023-11-16T16:48:29.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:29.175+0000 7fd277199700 1 -- 172.21.15.44:0/1433126507 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd2600a1260 msgr2=0x7fd2600a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:29.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:29.175+0000 7fd277199700 1 --2- 172.21.15.44:0/1433126507 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd2600a1260 0x7fd2600a3710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd270108a90 tx=0x7fd25c009260 comp rx=0 tx=0).stop 2023-11-16T16:48:29.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:29.175+0000 7fd277199700 1 -- 172.21.15.44:0/1433126507 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd270100550 msgr2=0x7fd2701071f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:29.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:29.175+0000 7fd277199700 1 --2- 172.21.15.44:0/1433126507 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd270100550 0x7fd2701071f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd26400e9a0 tx=0x7fd26400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:48:29.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:29.176+0000 7fd277199700 1 -- 172.21.15.44:0/1433126507 shutdown_connections 2023-11-16T16:48:29.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:29.176+0000 7fd277199700 1 --2- 172.21.15.44:0/1433126507 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd2600a1260 0x7fd2600a3710 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:29.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:29.176+0000 7fd277199700 1 --2- 172.21.15.44:0/1433126507 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2701055f0 0x7fd270107c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:29.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:29.176+0000 7fd277199700 1 --2- 172.21.15.44:0/1433126507 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd270100550 0x7fd2701071f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:29.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:29.176+0000 7fd277199700 1 --2- 172.21.15.44:0/1433126507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd270100ef0 0x7fd270107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:29.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:29.176+0000 7fd277199700 1 -- 172.21.15.44:0/1433126507 >> 172.21.15.44:0/1433126507 conn(0x7fd2700fbdf0 msgr2=0x7fd27010b630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:29.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:29.176+0000 7fd277199700 1 -- 172.21.15.44:0/1433126507 shutdown_connections 2023-11-16T16:48:29.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:29.176+0000 7fd277199700 1 -- 172.21.15.44:0/1433126507 wait complete. 2023-11-16T16:48:29.669 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182459 got 107374182461 for osd.3 2023-11-16T16:48:29.669 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:29.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:29 smithi044 ceph-mon[128750]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:29.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:29 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1433126507' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-11-16T16:48:29.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:29 smithi176 ceph-mon[131630]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:29.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:29 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1433126507' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-11-16T16:48:29.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:29 smithi077 ceph-mon[131204]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:29.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:29 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1433126507' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-11-16T16:48:30.007 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:30.289 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495663 got 197568495665 for osd.6 2023-11-16T16:48:30.289 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:30.460 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.571+0000 7f8195f67700 1 -- 172.21.15.44:0/2532821401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8190100ee0 msgr2=0x7f8190104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.571+0000 7f8195f67700 1 --2- 172.21.15.44:0/2532821401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8190100ee0 0x7f8190104eb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f81840099c0 tx=0x7f8184009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.571+0000 7f8195f67700 1 -- 172.21.15.44:0/2532821401 shutdown_connections 2023-11-16T16:48:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.571+0000 7f8195f67700 1 --2- 172.21.15.44:0/2532821401 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81901055e0 0x7f819010d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.571+0000 7f8195f67700 1 --2- 172.21.15.44:0/2532821401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8190100540 0x7f8190100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.571+0000 7f8195f67700 1 --2- 172.21.15.44:0/2532821401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8190100ee0 0x7f8190104eb0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.571+0000 7f8195f67700 1 -- 172.21.15.44:0/2532821401 >> 172.21.15.44:0/2532821401 conn(0x7f81900fbdc0 msgr2=0x7f81900fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.571+0000 7f8195f67700 1 -- 172.21.15.44:0/2532821401 shutdown_connections 2023-11-16T16:48:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.571+0000 7f8195f67700 1 -- 172.21.15.44:0/2532821401 wait complete. 2023-11-16T16:48:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.573+0000 7f8195f67700 1 Processor -- start 2023-11-16T16:48:31.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.573+0000 7f8195f67700 1 -- start start 2023-11-16T16:48:31.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.573+0000 7f8195f67700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8190100540 0x7f8190072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:31.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.573+0000 7f8195f67700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8190100ee0 0x7f8190073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:31.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.574+0000 7f8195f67700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81901055e0 0x7f81900738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:31.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.574+0000 7f8195f67700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81901a4840 con 0x7f8190100540 2023-11-16T16:48:31.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.574+0000 7f8195f67700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81901a49b0 con 0x7f81901055e0 2023-11-16T16:48:31.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.574+0000 7f8195f67700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81901a4b70 con 0x7f8190100ee0 2023-11-16T16:48:31.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.574+0000 7f818f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8190100540 0x7f8190072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:31.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.574+0000 7f818f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8190100540 0x7f8190072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35794/0 (socket says 172.21.15.44:35794) 2023-11-16T16:48:31.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.574+0000 7f818f7fe700 1 -- 172.21.15.44:0/545181267 learned_addr learned my addr 172.21.15.44:0/545181267 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.574+0000 7f818effd700 1 --2- 172.21.15.44:0/545181267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8190100ee0 0x7f8190073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.574+0000 7f818ffff700 1 --2- 172.21.15.44:0/545181267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81901055e0 0x7f81900738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.574+0000 7f818f7fe700 1 -- 172.21.15.44:0/545181267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8190100ee0 msgr2=0x7f8190073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.574+0000 7f818f7fe700 1 --2- 172.21.15.44:0/545181267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8190100ee0 0x7f8190073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.574+0000 7f818f7fe700 1 -- 172.21.15.44:0/545181267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81901055e0 msgr2=0x7f81900738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.574+0000 7f818f7fe700 1 --2- 172.21.15.44:0/545181267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81901055e0 0x7f81900738b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.574+0000 7f818f7fe700 1 -- 172.21.15.44:0/545181267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81840095f0 con 0x7f8190100540 2023-11-16T16:48:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.574+0000 7f818f7fe700 1 --2- 172.21.15.44:0/545181267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8190100540 0x7f8190072df0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f818000e9a0 tx=0x7f818000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.575+0000 7f818cff9700 1 -- 172.21.15.44:0/545181267 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180009790 con 0x7f8190100540 2023-11-16T16:48:31.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.575+0000 7f818cff9700 1 -- 172.21.15.44:0/545181267 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f818000cbd0 con 0x7f8190100540 2023-11-16T16:48:31.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.575+0000 7f818cff9700 1 -- 172.21.15.44:0/545181267 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180018680 con 0x7f8190100540 2023-11-16T16:48:31.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.575+0000 7f8195f67700 1 -- 172.21.15.44:0/545181267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8190110360 con 0x7f8190100540 2023-11-16T16:48:31.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.575+0000 7f8195f67700 1 -- 172.21.15.44:0/545181267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81901101d0 con 0x7f8190100540 2023-11-16T16:48:31.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.576+0000 7f8195f67700 1 -- 172.21.15.44:0/545181267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81901104f0 con 0x7f8190100540 2023-11-16T16:48:31.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.578+0000 7f818cff9700 1 -- 172.21.15.44:0/545181267 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f818000cd40 con 0x7f8190100540 2023-11-16T16:48:31.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.579+0000 7f818cff9700 1 --2- 172.21.15.44:0/545181267 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f81780a1260 0x7f81780a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:31.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.580+0000 7f818cff9700 1 -- 172.21.15.44:0/545181267 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8180014070 con 0x7f8190100540 2023-11-16T16:48:31.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.580+0000 7f818effd700 1 --2- 172.21.15.44:0/545181267 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f81780a1260 0x7f81780a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:31.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.580+0000 7f818effd700 1 --2- 172.21.15.44:0/545181267 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f81780a1260 0x7f81780a3710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8184009fd0 tx=0x7f8184005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:31.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.583+0000 7f818cff9700 1 -- 172.21.15.44:0/545181267 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8180051730 con 0x7f8190100540 2023-11-16T16:48:31.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.816+0000 7f8195f67700 1 -- 172.21.15.44:0/545181267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f8190066ea0 con 0x7f8190100540 2023-11-16T16:48:31.388 INFO:teuthology.orchestra.run.smithi044.stdout:283467841571 2023-11-16T16:48:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.816+0000 7f818cff9700 1 -- 172.21.15.44:0/545181267 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f81800962b0 con 0x7f8190100540 2023-11-16T16:48:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.818+0000 7f8195f67700 1 -- 172.21.15.44:0/545181267 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f81780a1260 msgr2=0x7f81780a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.818+0000 7f8195f67700 1 --2- 172.21.15.44:0/545181267 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f81780a1260 0x7f81780a3710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8184009fd0 tx=0x7f8184005fb0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.818+0000 7f8195f67700 1 -- 172.21.15.44:0/545181267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8190100540 msgr2=0x7f8190072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.818+0000 7f8195f67700 1 --2- 172.21.15.44:0/545181267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8190100540 0x7f8190072df0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f818000e9a0 tx=0x7f818000ecb0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.818+0000 7f8195f67700 1 -- 172.21.15.44:0/545181267 shutdown_connections 2023-11-16T16:48:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.818+0000 7f8195f67700 1 --2- 172.21.15.44:0/545181267 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f81780a1260 0x7f81780a3710 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.818+0000 7f8195f67700 1 --2- 172.21.15.44:0/545181267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81901055e0 0x7f81900738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.818+0000 7f8195f67700 1 --2- 172.21.15.44:0/545181267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8190100540 0x7f8190072df0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.818+0000 7f8195f67700 1 --2- 172.21.15.44:0/545181267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8190100ee0 0x7f8190073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.818+0000 7f8195f67700 1 -- 172.21.15.44:0/545181267 >> 172.21.15.44:0/545181267 conn(0x7f81900fbdc0 msgr2=0x7f81900fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.818+0000 7f8195f67700 1 -- 172.21.15.44:0/545181267 shutdown_connections 2023-11-16T16:48:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:30.819+0000 7f8195f67700 1 -- 172.21.15.44:0/545181267 wait complete. 2023-11-16T16:48:31.451 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411394 got 77309411397 for osd.2 2023-11-16T16:48:31.451 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:31.773 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:31 smithi044 ceph-mon[128750]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:31.773 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:31 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/545181267' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-11-16T16:48:31.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:31 smithi176 ceph-mon[131630]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:31.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:31 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/545181267' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-11-16T16:48:31.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:31 smithi077 ceph-mon[131204]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:31.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:31 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/545181267' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-11-16T16:48:32.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.790+0000 7ff4cf030700 1 -- 172.21.15.44:0/3885688632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4c81055e0 msgr2=0x7ff4c810d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:32.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.790+0000 7ff4cf030700 1 --2- 172.21.15.44:0/3885688632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4c81055e0 0x7ff4c810d160 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000b0d0 tx=0x7ff4c000b3e0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.790+0000 7ff4cf030700 1 -- 172.21.15.44:0/3885688632 shutdown_connections 2023-11-16T16:48:32.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.790+0000 7ff4cf030700 1 --2- 172.21.15.44:0/3885688632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4c81055e0 0x7ff4c810d160 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.790+0000 7ff4cf030700 1 --2- 172.21.15.44:0/3885688632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4c8100540 0x7ff4c8100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.790+0000 7ff4cf030700 1 --2- 172.21.15.44:0/3885688632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c8100ee0 0x7ff4c8104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.790+0000 7ff4cf030700 1 -- 172.21.15.44:0/3885688632 >> 172.21.15.44:0/3885688632 conn(0x7ff4c80fbdc0 msgr2=0x7ff4c80fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:32.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.790+0000 7ff4cf030700 1 -- 172.21.15.44:0/3885688632 shutdown_connections 2023-11-16T16:48:32.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.790+0000 7ff4cf030700 1 -- 172.21.15.44:0/3885688632 wait complete. 2023-11-16T16:48:32.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.791+0000 7ff4cf030700 1 Processor -- start 2023-11-16T16:48:32.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.791+0000 7ff4cf030700 1 -- start start 2023-11-16T16:48:32.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.792+0000 7ff4cf030700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4c8100540 0x7ff4c8198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:32.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.792+0000 7ff4cf030700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c8100ee0 0x7ff4c8198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:32.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.792+0000 7ff4cf030700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4c81055e0 0x7ff4c819bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:32.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.792+0000 7ff4cf030700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c819c580 con 0x7ff4c81055e0 2023-11-16T16:48:32.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.792+0000 7ff4cf030700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c819c6f0 con 0x7ff4c8100ee0 2023-11-16T16:48:32.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.792+0000 7ff4cf030700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c819c8b0 con 0x7ff4c8100540 2023-11-16T16:48:32.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.792+0000 7ff4cd5cd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4c81055e0 0x7ff4c819bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:32.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.792+0000 7ff4ccdcc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4c8100540 0x7ff4c8198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:32.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.792+0000 7ff4cd5cd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4c81055e0 0x7ff4c819bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35818/0 (socket says 172.21.15.44:35818) 2023-11-16T16:48:32.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.792+0000 7ff4cd5cd700 1 -- 172.21.15.44:0/950692396 learned_addr learned my addr 172.21.15.44:0/950692396 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:32.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.792+0000 7ff4c7fff700 1 --2- 172.21.15.44:0/950692396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c8100ee0 0x7ff4c8198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:32.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.792+0000 7ff4cd5cd700 1 -- 172.21.15.44:0/950692396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4c8100540 msgr2=0x7ff4c8198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:32.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.792+0000 7ff4cd5cd700 1 --2- 172.21.15.44:0/950692396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4c8100540 0x7ff4c8198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.792+0000 7ff4cd5cd700 1 -- 172.21.15.44:0/950692396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c8100ee0 msgr2=0x7ff4c8198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:32.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.792+0000 7ff4cd5cd700 1 --2- 172.21.15.44:0/950692396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c8100ee0 0x7ff4c8198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.792+0000 7ff4cd5cd700 1 -- 172.21.15.44:0/950692396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c0009d00 con 0x7ff4c81055e0 2023-11-16T16:48:32.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.793+0000 7ff4cd5cd700 1 --2- 172.21.15.44:0/950692396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4c81055e0 0x7ff4c819bf80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0007870 tx=0x7ff4c00078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:32.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.793+0000 7ff4c5ffb700 1 -- 172.21.15.44:0/950692396 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c000e040 con 0x7ff4c81055e0 2023-11-16T16:48:32.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.793+0000 7ff4c5ffb700 1 -- 172.21.15.44:0/950692396 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4c0005070 con 0x7ff4c81055e0 2023-11-16T16:48:32.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.793+0000 7ff4c5ffb700 1 -- 172.21.15.44:0/950692396 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c001b6c0 con 0x7ff4c81055e0 2023-11-16T16:48:32.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.793+0000 7ff4cf030700 1 -- 172.21.15.44:0/950692396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c8110320 con 0x7ff4c81055e0 2023-11-16T16:48:32.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.793+0000 7ff4cf030700 1 -- 172.21.15.44:0/950692396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c8110190 con 0x7ff4c81055e0 2023-11-16T16:48:32.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.795+0000 7ff4b37fe700 1 -- 172.21.15.44:0/950692396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c81104b0 con 0x7ff4c81055e0 2023-11-16T16:48:32.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.796+0000 7ff4c5ffb700 1 -- 172.21.15.44:0/950692396 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff4c00051e0 con 0x7ff4c81055e0 2023-11-16T16:48:32.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.798+0000 7ff4c5ffb700 1 --2- 172.21.15.44:0/950692396 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff4b80a1260 0x7ff4b80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:32.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.798+0000 7ff4c5ffb700 1 -- 172.21.15.44:0/950692396 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff4c00cde80 con 0x7ff4c81055e0 2023-11-16T16:48:32.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.798+0000 7ff4ccdcc700 1 --2- 172.21.15.44:0/950692396 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff4b80a1260 0x7ff4b80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:32.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.799+0000 7ff4ccdcc700 1 --2- 172.21.15.44:0/950692396 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff4b80a1260 0x7ff4b80a3710 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc005950 tx=0x7ff4bc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:32.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:32.802+0000 7ff4c5ffb700 1 -- 172.21.15.44:0/950692396 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff4c0097190 con 0x7ff4c81055e0 2023-11-16T16:48:33.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.059+0000 7ff4b37fe700 1 -- 172.21.15.44:0/950692396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff4c8066ea0 con 0x7ff4c81055e0 2023-11-16T16:48:33.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.059+0000 7ff4c5ffb700 1 -- 172.21.15.44:0/950692396 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff4c009a7b0 con 0x7ff4c81055e0 2023-11-16T16:48:33.061 INFO:teuthology.orchestra.run.smithi044.stdout:141733920825 2023-11-16T16:48:33.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.059+0000 7ff4b37fe700 1 -- 172.21.15.44:0/950692396 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff4b80a1260 msgr2=0x7ff4b80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:33.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.059+0000 7ff4b37fe700 1 --2- 172.21.15.44:0/950692396 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff4b80a1260 0x7ff4b80a3710 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc005950 tx=0x7ff4bc00b410 comp rx=0 tx=0).stop 2023-11-16T16:48:33.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.059+0000 7ff4b37fe700 1 -- 172.21.15.44:0/950692396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4c81055e0 msgr2=0x7ff4c819bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:33.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.059+0000 7ff4b37fe700 1 --2- 172.21.15.44:0/950692396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4c81055e0 0x7ff4c819bf80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0007870 tx=0x7ff4c00078a0 comp rx=0 tx=0).stop 2023-11-16T16:48:33.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.059+0000 7ff4b37fe700 1 -- 172.21.15.44:0/950692396 shutdown_connections 2023-11-16T16:48:33.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.059+0000 7ff4b37fe700 1 --2- 172.21.15.44:0/950692396 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff4b80a1260 0x7ff4b80a3710 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:33.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.059+0000 7ff4b37fe700 1 --2- 172.21.15.44:0/950692396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff4c81055e0 0x7ff4c819bf80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:33.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.059+0000 7ff4b37fe700 1 --2- 172.21.15.44:0/950692396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4c8100540 0x7ff4c8198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:33.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.059+0000 7ff4b37fe700 1 --2- 172.21.15.44:0/950692396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c8100ee0 0x7ff4c8198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:33.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.059+0000 7ff4b37fe700 1 -- 172.21.15.44:0/950692396 >> 172.21.15.44:0/950692396 conn(0x7ff4c80fbdc0 msgr2=0x7ff4c810b620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:33.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.064+0000 7ff4b37fe700 1 -- 172.21.15.44:0/950692396 shutdown_connections 2023-11-16T16:48:33.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.064+0000 7ff4b37fe700 1 -- 172.21.15.44:0/950692396 wait complete. 2023-11-16T16:48:33.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.115+0000 7f48a8c24700 1 -- 172.21.15.44:0/4142656773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48a40ffc40 msgr2=0x7f48a4104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:33.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.115+0000 7f48a8c24700 1 --2- 172.21.15.44:0/4142656773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48a40ffc40 0x7f48a4104120 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4898009990 tx=0x7f4898009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:33.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.115+0000 7f48a8c24700 1 -- 172.21.15.44:0/4142656773 shutdown_connections 2023-11-16T16:48:33.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.115+0000 7f48a8c24700 1 --2- 172.21.15.44:0/4142656773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48a4104660 0x7f48a4104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:33.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.115+0000 7f48a8c24700 1 --2- 172.21.15.44:0/4142656773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48a40ff330 0x7f48a40ff700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:33.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.115+0000 7f48a8c24700 1 --2- 172.21.15.44:0/4142656773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48a40ffc40 0x7f48a4104120 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:33.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.115+0000 7f48a8c24700 1 -- 172.21.15.44:0/4142656773 >> 172.21.15.44:0/4142656773 conn(0x7f48a40770a0 msgr2=0x7f48a40774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:33.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.115+0000 7f48a8c24700 1 -- 172.21.15.44:0/4142656773 shutdown_connections 2023-11-16T16:48:33.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.115+0000 7f48a8c24700 1 -- 172.21.15.44:0/4142656773 wait complete. 2023-11-16T16:48:33.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.116+0000 7f48a8c24700 1 Processor -- start 2023-11-16T16:48:33.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.116+0000 7f48a8c24700 1 -- start start 2023-11-16T16:48:33.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.117+0000 7f48a8c24700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48a40ff330 0x7f48a41a2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:33.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.117+0000 7f48a8c24700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48a40ffc40 0x7f48a41a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:33.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.117+0000 7f48a8c24700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48a4104660 0x7f48a419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:33.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.117+0000 7f48a8c24700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a419cf40 con 0x7f48a40ffc40 2023-11-16T16:48:33.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.117+0000 7f48a8c24700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a419d0b0 con 0x7f48a4104660 2023-11-16T16:48:33.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.117+0000 7f48a8c24700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a419d270 con 0x7f48a40ff330 2023-11-16T16:48:33.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.117+0000 7f48a259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48a40ff330 0x7f48a41a2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:33.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.117+0000 7f48a259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48a40ff330 0x7f48a41a2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:40236/0 (socket says 172.21.15.44:40236) 2023-11-16T16:48:33.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.117+0000 7f48a259c700 1 -- 172.21.15.44:0/3042693077 learned_addr learned my addr 172.21.15.44:0/3042693077 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:33.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.117+0000 7f48a1d9b700 1 --2- 172.21.15.44:0/3042693077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48a40ffc40 0x7f48a41a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:33.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.117+0000 7f48a2d9d700 1 --2- 172.21.15.44:0/3042693077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48a4104660 0x7f48a419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:33.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.118+0000 7f48a2d9d700 1 -- 172.21.15.44:0/3042693077 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48a40ff330 msgr2=0x7f48a41a2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:33.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.118+0000 7f48a2d9d700 1 --2- 172.21.15.44:0/3042693077 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48a40ff330 0x7f48a41a2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:33.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.118+0000 7f48a2d9d700 1 -- 172.21.15.44:0/3042693077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48a40ffc40 msgr2=0x7f48a41a2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:33.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.118+0000 7f48a2d9d700 1 --2- 172.21.15.44:0/3042693077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48a40ffc40 0x7f48a41a2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:33.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.118+0000 7f48a2d9d700 1 -- 172.21.15.44:0/3042693077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48980095f0 con 0x7f48a4104660 2023-11-16T16:48:33.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.118+0000 7f48a259c700 1 --2- 172.21.15.44:0/3042693077 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48a40ff330 0x7f48a41a2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:33.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.118+0000 7f48a2d9d700 1 --2- 172.21.15.44:0/3042693077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48a4104660 0x7f48a419c8d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f489c00efc0 tx=0x7f489c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:33.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.118+0000 7f48937fe700 1 -- 172.21.15.44:0/3042693077 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c018070 con 0x7f48a4104660 2023-11-16T16:48:33.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.119+0000 7f48937fe700 1 -- 172.21.15.44:0/3042693077 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f489c00f040 con 0x7f48a4104660 2023-11-16T16:48:33.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.119+0000 7f48a8c24700 1 -- 172.21.15.44:0/3042693077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a4116970 con 0x7f48a4104660 2023-11-16T16:48:33.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.119+0000 7f48a8c24700 1 -- 172.21.15.44:0/3042693077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a41167e0 con 0x7f48a4104660 2023-11-16T16:48:33.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.119+0000 7f48937fe700 1 -- 172.21.15.44:0/3042693077 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c00cd40 con 0x7f48a4104660 2023-11-16T16:48:33.568 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.120+0000 7f48a8c24700 1 -- 172.21.15.44:0/3042693077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4884005320 con 0x7f48a4104660 2023-11-16T16:48:33.568 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.123+0000 7f48937fe700 1 -- 172.21.15.44:0/3042693077 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f489c004ad0 con 0x7f48a4104660 2023-11-16T16:48:33.568 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.124+0000 7f48937fe700 1 --2- 172.21.15.44:0/3042693077 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f488c0a1260 0x7f488c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:33.568 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.125+0000 7f48937fe700 1 -- 172.21.15.44:0/3042693077 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f489c0cc800 con 0x7f48a4104660 2023-11-16T16:48:33.568 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.125+0000 7f48a259c700 1 --2- 172.21.15.44:0/3042693077 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f488c0a1260 0x7f488c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:33.568 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.126+0000 7f48a259c700 1 --2- 172.21.15.44:0/3042693077 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f488c0a1260 0x7f488c0a3710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4894011550 tx=0x7f4894009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:33.568 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.128+0000 7f48937fe700 1 -- 172.21.15.44:0/3042693077 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f489c095b10 con 0x7f48a4104660 2023-11-16T16:48:33.568 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.377+0000 7f48a8c24700 1 -- 172.21.15.44:0/3042693077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f4884005190 con 0x7f48a4104660 2023-11-16T16:48:33.568 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.378+0000 7f48937fe700 1 -- 172.21.15.44:0/3042693077 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f489c099130 con 0x7f48a4104660 2023-11-16T16:48:33.568 INFO:teuthology.orchestra.run.smithi044.stdout:335007449116 2023-11-16T16:48:33.569 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.381+0000 7f48a8c24700 1 -- 172.21.15.44:0/3042693077 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f488c0a1260 msgr2=0x7f488c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:33.569 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.381+0000 7f48a8c24700 1 --2- 172.21.15.44:0/3042693077 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f488c0a1260 0x7f488c0a3710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4894011550 tx=0x7f4894009260 comp rx=0 tx=0).stop 2023-11-16T16:48:33.569 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.381+0000 7f48a8c24700 1 -- 172.21.15.44:0/3042693077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48a4104660 msgr2=0x7f48a419c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:33.569 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.381+0000 7f48a8c24700 1 --2- 172.21.15.44:0/3042693077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48a4104660 0x7f48a419c8d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f489c00efc0 tx=0x7f489c00c3a0 comp rx=0 tx=0).stop 2023-11-16T16:48:33.569 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.382+0000 7f48a8c24700 1 -- 172.21.15.44:0/3042693077 shutdown_connections 2023-11-16T16:48:33.569 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.382+0000 7f48a8c24700 1 --2- 172.21.15.44:0/3042693077 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f488c0a1260 0x7f488c0a3710 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:33.569 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.382+0000 7f48a8c24700 1 --2- 172.21.15.44:0/3042693077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48a4104660 0x7f48a419c8d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:33.569 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.382+0000 7f48a8c24700 1 --2- 172.21.15.44:0/3042693077 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48a40ff330 0x7f48a41a2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:33.569 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.382+0000 7f48a8c24700 1 --2- 172.21.15.44:0/3042693077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48a40ffc40 0x7f48a41a2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:33.569 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.382+0000 7f48a8c24700 1 -- 172.21.15.44:0/3042693077 >> 172.21.15.44:0/3042693077 conn(0x7f48a40770a0 msgr2=0x7f48a41021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:33.570 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.382+0000 7f48a8c24700 1 -- 172.21.15.44:0/3042693077 shutdown_connections 2023-11-16T16:48:33.570 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:33.382+0000 7f48a8c24700 1 -- 172.21.15.44:0/3042693077 wait complete. 2023-11-16T16:48:33.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:33 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:33.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:33 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:33.712 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:33 smithi044 ceph-mon[128750]: pgmap v314: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:33.712 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:33 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/950692396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-11-16T16:48:33.712 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:33 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3042693077' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-11-16T16:48:33.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:33 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:33.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:33 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:33.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:33 smithi077 ceph-mon[131204]: pgmap v314: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:33.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:33 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/950692396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-11-16T16:48:33.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:33 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3042693077' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-11-16T16:48:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:33 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:33 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:33 smithi176 ceph-mon[131630]: pgmap v314: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:33 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/950692396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-11-16T16:48:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:33 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3042693077' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-11-16T16:48:34.152 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841568 got 283467841571 for osd.9 2023-11-16T16:48:34.153 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:34.177 INFO:tasks.cephadm.ceph_manager.ceph:need seq 335007449114 got 335007449116 for osd.11 2023-11-16T16:48:34.177 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:34.726 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:34 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:48:34.726 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:34 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:48:34.726 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:34 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:48:34.726 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:34 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:34.727 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:34 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:48:34.727 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:34 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:48:34.727 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:34 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T16:48:34.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:34 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:48:34.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:34 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:48:34.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:34 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:48:34.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:34 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:34.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:34 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:48:34.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:34 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:48:34.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:34 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T16:48:34.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.803+0000 7fb25dd2f700 1 -- 172.21.15.44:0/2696823641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb258101540 msgr2=0x7fb258101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:34.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.803+0000 7fb25dd2f700 1 --2- 172.21.15.44:0/2696823641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb258101540 0x7fb258101910 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb248009990 tx=0x7fb248009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:34.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.803+0000 7fb25dd2f700 1 -- 172.21.15.44:0/2696823641 shutdown_connections 2023-11-16T16:48:34.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.803+0000 7fb25dd2f700 1 --2- 172.21.15.44:0/2696823641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb25810a890 0x7fb25810ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:34.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.803+0000 7fb25dd2f700 1 --2- 172.21.15.44:0/2696823641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb258101540 0x7fb258101910 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:34.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.803+0000 7fb25dd2f700 1 --2- 172.21.15.44:0/2696823641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb258101e50 0x7fb25810a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:34.806 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.803+0000 7fb25dd2f700 1 -- 172.21.15.44:0/2696823641 >> 172.21.15.44:0/2696823641 conn(0x7fb2580770d0 msgr2=0x7fb2580774d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:34.806 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.803+0000 7fb25dd2f700 1 -- 172.21.15.44:0/2696823641 shutdown_connections 2023-11-16T16:48:34.806 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.803+0000 7fb25dd2f700 1 -- 172.21.15.44:0/2696823641 wait complete. 2023-11-16T16:48:34.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.804+0000 7fb25dd2f700 1 Processor -- start 2023-11-16T16:48:34.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.804+0000 7fb25dd2f700 1 -- start start 2023-11-16T16:48:34.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.804+0000 7fb25dd2f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb258101540 0x7fb258198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:34.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.804+0000 7fb25dd2f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb258101e50 0x7fb258198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:34.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.805+0000 7fb25dd2f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb25810a890 0x7fb25819cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:34.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.805+0000 7fb25dd2f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25819d550 con 0x7fb258101540 2023-11-16T16:48:34.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.805+0000 7fb25dd2f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25819d6c0 con 0x7fb258101e50 2023-11-16T16:48:34.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.805+0000 7fb25dd2f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25819d880 con 0x7fb25810a890 2023-11-16T16:48:34.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.805+0000 7fb2577fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb258101540 0x7fb258198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:34.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.805+0000 7fb2577fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb258101540 0x7fb258198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35848/0 (socket says 172.21.15.44:35848) 2023-11-16T16:48:34.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.805+0000 7fb2577fe700 1 -- 172.21.15.44:0/4019553301 learned_addr learned my addr 172.21.15.44:0/4019553301 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:34.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.805+0000 7fb256ffd700 1 --2- 172.21.15.44:0/4019553301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb258101e50 0x7fb258198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:34.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.805+0000 7fb257fff700 1 --2- 172.21.15.44:0/4019553301 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb25810a890 0x7fb25819cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:34.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.805+0000 7fb2577fe700 1 -- 172.21.15.44:0/4019553301 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb25810a890 msgr2=0x7fb25819cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:34.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.805+0000 7fb2577fe700 1 --2- 172.21.15.44:0/4019553301 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb25810a890 0x7fb25819cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:34.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.805+0000 7fb2577fe700 1 -- 172.21.15.44:0/4019553301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb258101e50 msgr2=0x7fb258198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:34.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.805+0000 7fb2577fe700 1 --2- 172.21.15.44:0/4019553301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb258101e50 0x7fb258198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:34.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.805+0000 7fb2577fe700 1 -- 172.21.15.44:0/4019553301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2480095f0 con 0x7fb258101540 2023-11-16T16:48:34.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.806+0000 7fb2577fe700 1 --2- 172.21.15.44:0/4019553301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb258101540 0x7fb258198590 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb248000c00 tx=0x7fb248005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:34.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.806+0000 7fb254ff9700 1 -- 172.21.15.44:0/4019553301 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24801d070 con 0x7fb258101540 2023-11-16T16:48:34.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.806+0000 7fb254ff9700 1 -- 172.21.15.44:0/4019553301 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb248022470 con 0x7fb258101540 2023-11-16T16:48:34.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.806+0000 7fb25dd2f700 1 -- 172.21.15.44:0/4019553301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb258110320 con 0x7fb258101540 2023-11-16T16:48:34.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.806+0000 7fb254ff9700 1 -- 172.21.15.44:0/4019553301 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb248017650 con 0x7fb258101540 2023-11-16T16:48:34.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.806+0000 7fb25dd2f700 1 -- 172.21.15.44:0/4019553301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb258110190 con 0x7fb258101540 2023-11-16T16:48:34.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.807+0000 7fb25dd2f700 1 -- 172.21.15.44:0/4019553301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2581104b0 con 0x7fb258101540 2023-11-16T16:48:34.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.809+0000 7fb254ff9700 1 -- 172.21.15.44:0/4019553301 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb2480225e0 con 0x7fb258101540 2023-11-16T16:48:34.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.810+0000 7fb254ff9700 1 --2- 172.21.15.44:0/4019553301 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb2400a11e0 0x7fb2400a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.811+0000 7fb254ff9700 1 -- 172.21.15.44:0/4019553301 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb2480ce1f0 con 0x7fb258101540 2023-11-16T16:48:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.811+0000 7fb256ffd700 1 --2- 172.21.15.44:0/4019553301 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb2400a11e0 0x7fb2400a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.811+0000 7fb256ffd700 1 --2- 172.21.15.44:0/4019553301 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb2400a11e0 0x7fb2400a3690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb258199bb0 tx=0x7fb24c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:34.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:34.815+0000 7fb254ff9700 1 -- 172.21.15.44:0/4019553301 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb248097500 con 0x7fb258101540 2023-11-16T16:48:34.862 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920823 got 141733920825 for osd.4 2023-11-16T16:48:34.862 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:34.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:34 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:48:34.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:34 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:48:34.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:34 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:48:34.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:34 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:48:34.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:34 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:48:34.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:34 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:48:34.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:34 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T16:48:35.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.101+0000 7fb25dd2f700 1 -- 172.21.15.44:0/4019553301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb2581a5670 con 0x7fb258101540 2023-11-16T16:48:35.104 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.102+0000 7fb254ff9700 1 -- 172.21.15.44:0/4019553301 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb24809ab20 con 0x7fb258101540 2023-11-16T16:48:35.104 INFO:teuthology.orchestra.run.smithi044.stdout:55834574920 2023-11-16T16:48:35.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.106+0000 7fb25dd2f700 1 -- 172.21.15.44:0/4019553301 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb2400a11e0 msgr2=0x7fb2400a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.106+0000 7fb25dd2f700 1 --2- 172.21.15.44:0/4019553301 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb2400a11e0 0x7fb2400a3690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb258199bb0 tx=0x7fb24c009260 comp rx=0 tx=0).stop 2023-11-16T16:48:35.109 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.106+0000 7fb25dd2f700 1 -- 172.21.15.44:0/4019553301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb258101540 msgr2=0x7fb258198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.109 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.106+0000 7fb25dd2f700 1 --2- 172.21.15.44:0/4019553301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb258101540 0x7fb258198590 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb248000c00 tx=0x7fb248005740 comp rx=0 tx=0).stop 2023-11-16T16:48:35.109 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.106+0000 7fb25dd2f700 1 -- 172.21.15.44:0/4019553301 shutdown_connections 2023-11-16T16:48:35.109 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.106+0000 7fb25dd2f700 1 --2- 172.21.15.44:0/4019553301 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb2400a11e0 0x7fb2400a3690 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.109 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.106+0000 7fb25dd2f700 1 --2- 172.21.15.44:0/4019553301 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb25810a890 0x7fb25819cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.109 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.106+0000 7fb25dd2f700 1 --2- 172.21.15.44:0/4019553301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb258101540 0x7fb258198590 secure :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb248000c00 tx=0x7fb248005740 comp rx=0 tx=0).stop 2023-11-16T16:48:35.109 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.106+0000 7fb25dd2f700 1 --2- 172.21.15.44:0/4019553301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb258101e50 0x7fb258198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.109 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.106+0000 7fb25dd2f700 1 -- 172.21.15.44:0/4019553301 >> 172.21.15.44:0/4019553301 conn(0x7fb2580770d0 msgr2=0x7fb2580fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:35.110 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.108+0000 7fb25dd2f700 1 -- 172.21.15.44:0/4019553301 shutdown_connections 2023-11-16T16:48:35.110 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.108+0000 7fb25dd2f700 1 -- 172.21.15.44:0/4019553301 wait complete. 2023-11-16T16:48:35.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.203+0000 7f5e743c4700 1 -- 172.21.15.44:0/3758139777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e6c0ff0d0 msgr2=0x7f5e6c0ff4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.203+0000 7f5e743c4700 1 --2- 172.21.15.44:0/3758139777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e6c0ff0d0 0x7f5e6c0ff4a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009990 tx=0x7f5e68009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.203+0000 7f5e743c4700 1 -- 172.21.15.44:0/3758139777 shutdown_connections 2023-11-16T16:48:35.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.203+0000 7f5e743c4700 1 --2- 172.21.15.44:0/3758139777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e6c10cc80 0x7f5e6c10f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.203+0000 7f5e743c4700 1 --2- 172.21.15.44:0/3758139777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e6c0ff0d0 0x7f5e6c0ff4a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.203+0000 7f5e743c4700 1 --2- 172.21.15.44:0/3758139777 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5e6c0ff9e0 0x7f5e6c10c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.203+0000 7f5e743c4700 1 -- 172.21.15.44:0/3758139777 >> 172.21.15.44:0/3758139777 conn(0x7f5e6c0770a0 msgr2=0x7f5e6c0774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:35.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.203+0000 7f5e743c4700 1 -- 172.21.15.44:0/3758139777 shutdown_connections 2023-11-16T16:48:35.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.203+0000 7f5e743c4700 1 -- 172.21.15.44:0/3758139777 wait complete. 2023-11-16T16:48:35.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.205+0000 7f5e743c4700 1 Processor -- start 2023-11-16T16:48:35.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.206+0000 7f5e743c4700 1 -- start start 2023-11-16T16:48:35.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.206+0000 7f5e743c4700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e6c0ff0d0 0x7f5e6c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.206+0000 7f5e743c4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e6c0ff9e0 0x7f5e6c070210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.206+0000 7f5e743c4700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5e6c10cc80 0x7f5e6c06aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.206+0000 7f5e743c4700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c06b4e0 con 0x7f5e6c0ff0d0 2023-11-16T16:48:35.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.206+0000 7f5e743c4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c06b620 con 0x7f5e6c0ff9e0 2023-11-16T16:48:35.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.206+0000 7f5e743c4700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c06b7e0 con 0x7f5e6c10cc80 2023-11-16T16:48:35.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.207+0000 7f5e7195f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e6c0ff9e0 0x7f5e6c070210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.207+0000 7f5e7195f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e6c0ff9e0 0x7f5e6c070210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:35496/0 (socket says 172.21.15.44:35496) 2023-11-16T16:48:35.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.207+0000 7f5e7195f700 1 -- 172.21.15.44:0/1117600721 learned_addr learned my addr 172.21.15.44:0/1117600721 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:35.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.207+0000 7f5e72160700 1 --2- 172.21.15.44:0/1117600721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e6c0ff0d0 0x7f5e6c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.207+0000 7f5e72961700 1 --2- 172.21.15.44:0/1117600721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5e6c10cc80 0x7f5e6c06aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.207+0000 7f5e72160700 1 -- 172.21.15.44:0/1117600721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5e6c10cc80 msgr2=0x7f5e6c06aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.207+0000 7f5e72160700 1 --2- 172.21.15.44:0/1117600721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5e6c10cc80 0x7f5e6c06aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.207+0000 7f5e72160700 1 -- 172.21.15.44:0/1117600721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e6c0ff9e0 msgr2=0x7f5e6c070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.207+0000 7f5e72160700 1 --2- 172.21.15.44:0/1117600721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e6c0ff9e0 0x7f5e6c070210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.207+0000 7f5e72160700 1 -- 172.21.15.44:0/1117600721 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e680095f0 con 0x7f5e6c0ff0d0 2023-11-16T16:48:35.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.208+0000 7f5e72160700 1 --2- 172.21.15.44:0/1117600721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e6c0ff0d0 0x7f5e6c06fcd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5e68000c00 tx=0x7f5e68005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:35.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.208+0000 7f5e637fe700 1 -- 172.21.15.44:0/1117600721 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6801d070 con 0x7f5e6c0ff0d0 2023-11-16T16:48:35.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.208+0000 7f5e637fe700 1 -- 172.21.15.44:0/1117600721 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e68021410 con 0x7f5e6c0ff0d0 2023-11-16T16:48:35.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.208+0000 7f5e743c4700 1 -- 172.21.15.44:0/1117600721 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6c1125b0 con 0x7f5e6c0ff0d0 2023-11-16T16:48:35.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.208+0000 7f5e637fe700 1 -- 172.21.15.44:0/1117600721 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68017650 con 0x7f5e6c0ff0d0 2023-11-16T16:48:35.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.208+0000 7f5e743c4700 1 -- 172.21.15.44:0/1117600721 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6c112420 con 0x7f5e6c0ff0d0 2023-11-16T16:48:35.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.211+0000 7f5e743c4700 1 -- 172.21.15.44:0/1117600721 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e6c112740 con 0x7f5e6c0ff0d0 2023-11-16T16:48:35.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.213+0000 7f5e637fe700 1 -- 172.21.15.44:0/1117600721 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5e680217f0 con 0x7f5e6c0ff0d0 2023-11-16T16:48:35.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.216+0000 7f5e637fe700 1 --2- 172.21.15.44:0/1117600721 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5e580a1260 0x7f5e580a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.216+0000 7f5e637fe700 1 -- 172.21.15.44:0/1117600721 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5e680cdfb0 con 0x7f5e6c0ff0d0 2023-11-16T16:48:35.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.217+0000 7f5e7195f700 1 --2- 172.21.15.44:0/1117600721 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5e580a1260 0x7f5e580a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.219 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.217+0000 7f5e7195f700 1 --2- 172.21.15.44:0/1117600721 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5e580a1260 0x7f5e580a3710 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c06c6c0 tx=0x7f5e5c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:35.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.223+0000 7f5e637fe700 1 -- 172.21.15.44:0/1117600721 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5e68097320 con 0x7f5e6c0ff0d0 2023-11-16T16:48:35.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.280+0000 7f6adbb99700 1 -- 172.21.15.44:0/1037615563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad4104500 msgr2=0x7f6ad410c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.280+0000 7f6adbb99700 1 --2- 172.21.15.44:0/1037615563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad4104500 0x7f6ad410c080 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6acc00b0a0 tx=0x7f6acc00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.280+0000 7f6adbb99700 1 -- 172.21.15.44:0/1037615563 shutdown_connections 2023-11-16T16:48:35.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.280+0000 7f6adbb99700 1 --2- 172.21.15.44:0/1037615563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad4104500 0x7f6ad410c080 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.280+0000 7f6adbb99700 1 --2- 172.21.15.44:0/1037615563 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ad40ff460 0x7f6ad40ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.280+0000 7f6adbb99700 1 --2- 172.21.15.44:0/1037615563 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ad40ffe00 0x7f6ad4103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.280+0000 7f6adbb99700 1 -- 172.21.15.44:0/1037615563 >> 172.21.15.44:0/1037615563 conn(0x7f6ad40770a0 msgr2=0x7f6ad40774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:35.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.280+0000 7f6adbb99700 1 -- 172.21.15.44:0/1037615563 shutdown_connections 2023-11-16T16:48:35.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.280+0000 7f6adbb99700 1 -- 172.21.15.44:0/1037615563 wait complete. 2023-11-16T16:48:35.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.281+0000 7f6adbb99700 1 Processor -- start 2023-11-16T16:48:35.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.282+0000 7f6adbb99700 1 -- start start 2023-11-16T16:48:35.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.282+0000 7f6adbb99700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ad40ff460 0x7f6ad410c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.282+0000 7f6adbb99700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ad40ffe00 0x7f6ad410ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.282+0000 7f6adbb99700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad4104500 0x7f6ad410d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.282+0000 7f6adbb99700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad41a4950 con 0x7f6ad40ffe00 2023-11-16T16:48:35.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.282+0000 7f6adbb99700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad41a4ac0 con 0x7f6ad4104500 2023-11-16T16:48:35.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.282+0000 7f6adbb99700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad41a4c80 con 0x7f6ad40ff460 2023-11-16T16:48:35.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.282+0000 7f6ad9134700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ad40ffe00 0x7f6ad410ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.283+0000 7f6ad9935700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ad40ff460 0x7f6ad410c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.283+0000 7f6ad9134700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ad40ffe00 0x7f6ad410ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44464/0 (socket says 172.21.15.44:44464) 2023-11-16T16:48:35.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.283+0000 7f6ad9134700 1 -- 172.21.15.44:0/3282170709 learned_addr learned my addr 172.21.15.44:0/3282170709 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:35.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.283+0000 7f6ada136700 1 --2- 172.21.15.44:0/3282170709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad4104500 0x7f6ad410d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.283+0000 7f6ad9134700 1 -- 172.21.15.44:0/3282170709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ad40ff460 msgr2=0x7f6ad410c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.283+0000 7f6ad9134700 1 --2- 172.21.15.44:0/3282170709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ad40ff460 0x7f6ad410c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.283+0000 7f6ad9134700 1 -- 172.21.15.44:0/3282170709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad4104500 msgr2=0x7f6ad410d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.283+0000 7f6ad9134700 1 --2- 172.21.15.44:0/3282170709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad4104500 0x7f6ad410d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.283+0000 7f6ad9134700 1 -- 172.21.15.44:0/3282170709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac4009520 con 0x7f6ad40ffe00 2023-11-16T16:48:35.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.283+0000 7f6ad9134700 1 --2- 172.21.15.44:0/3282170709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ad40ffe00 0x7f6ad410ced0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6ac400ea30 tx=0x7f6ac400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:35.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.284+0000 7f6acaffd700 1 -- 172.21.15.44:0/3282170709 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac400cbd0 con 0x7f6ad40ffe00 2023-11-16T16:48:35.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.284+0000 7f6adbb99700 1 -- 172.21.15.44:0/3282170709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6acc009d00 con 0x7f6ad40ffe00 2023-11-16T16:48:35.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.284+0000 7f6acaffd700 1 -- 172.21.15.44:0/3282170709 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ac400eed0 con 0x7f6ad40ffe00 2023-11-16T16:48:35.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.284+0000 7f6acaffd700 1 -- 172.21.15.44:0/3282170709 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4003d20 con 0x7f6ad40ffe00 2023-11-16T16:48:35.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.284+0000 7f6adbb99700 1 -- 172.21.15.44:0/3282170709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad41103a0 con 0x7f6ad40ffe00 2023-11-16T16:48:35.288 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.286+0000 7f6adbb99700 1 -- 172.21.15.44:0/3282170709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad4110210 con 0x7f6ad40ffe00 2023-11-16T16:48:35.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.289+0000 7f6acaffd700 1 -- 172.21.15.44:0/3282170709 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6ac4004020 con 0x7f6ad40ffe00 2023-11-16T16:48:35.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.290+0000 7f6acaffd700 1 --2- 172.21.15.44:0/3282170709 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6ac00a11e0 0x7f6ac00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.290+0000 7f6acaffd700 1 -- 172.21.15.44:0/3282170709 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6ac4014070 con 0x7f6ad40ffe00 2023-11-16T16:48:35.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.291+0000 7f6ad9935700 1 --2- 172.21.15.44:0/3282170709 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6ac00a11e0 0x7f6ac00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.292+0000 7f6ad9935700 1 --2- 172.21.15.44:0/3282170709 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6ac00a11e0 0x7f6ac00a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0007be0 tx=0x7f6ad00073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:35.296 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.294+0000 7f6acaffd700 1 -- 172.21.15.44:0/3282170709 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6ac4096310 con 0x7f6ad40ffe00 2023-11-16T16:48:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.339+0000 7fa5ce9bd700 1 -- 172.21.15.44:0/713764076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c8100370 msgr2=0x7fa5c8100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.339+0000 7fa5ce9bd700 1 --2- 172.21.15.44:0/713764076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c8100370 0x7fa5c8100740 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009990 tx=0x7fa5c4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.339+0000 7fa5ce9bd700 1 -- 172.21.15.44:0/713764076 shutdown_connections 2023-11-16T16:48:35.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.339+0000 7fa5ce9bd700 1 --2- 172.21.15.44:0/713764076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5c8105410 0x7fa5c810cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.339+0000 7fa5ce9bd700 1 --2- 172.21.15.44:0/713764076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c8100370 0x7fa5c8100740 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.339+0000 7fa5ce9bd700 1 --2- 172.21.15.44:0/713764076 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5c8100d10 0x7fa5c8104ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.339+0000 7fa5ce9bd700 1 -- 172.21.15.44:0/713764076 >> 172.21.15.44:0/713764076 conn(0x7fa5c80fbbd0 msgr2=0x7fa5c80fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:35.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.339+0000 7fa5ce9bd700 1 -- 172.21.15.44:0/713764076 shutdown_connections 2023-11-16T16:48:35.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.339+0000 7fa5ce9bd700 1 -- 172.21.15.44:0/713764076 wait complete. 2023-11-16T16:48:35.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.340+0000 7fa5ce9bd700 1 Processor -- start 2023-11-16T16:48:35.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.340+0000 7fa5ce9bd700 1 -- start start 2023-11-16T16:48:35.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.341+0000 7fa5ce9bd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c8100370 0x7fa5c8072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.341+0000 7fa5ce9bd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5c8100d10 0x7fa5c8073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.341+0000 7fa5ce9bd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5c8105410 0x7fa5c80738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.341+0000 7fa5ce9bd700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c81a4690 con 0x7fa5c8100d10 2023-11-16T16:48:35.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.341+0000 7fa5ce9bd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c81a4800 con 0x7fa5c8100370 2023-11-16T16:48:35.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.341+0000 7fa5ce9bd700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c81a49c0 con 0x7fa5c8105410 2023-11-16T16:48:35.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.341+0000 7fa5cd1ba700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5c8100d10 0x7fa5c8073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.341+0000 7fa5cd9bb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c8100370 0x7fa5c8072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.341+0000 7fa5cd1ba700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5c8100d10 0x7fa5c8073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44486/0 (socket says 172.21.15.44:44486) 2023-11-16T16:48:35.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.341+0000 7fa5cd1ba700 1 -- 172.21.15.44:0/2755522926 learned_addr learned my addr 172.21.15.44:0/2755522926 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:35.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.341+0000 7fa5ce1bc700 1 --2- 172.21.15.44:0/2755522926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5c8105410 0x7fa5c80738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.342+0000 7fa5cd9bb700 1 -- 172.21.15.44:0/2755522926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5c8105410 msgr2=0x7fa5c80738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.342+0000 7fa5cd9bb700 1 --2- 172.21.15.44:0/2755522926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5c8105410 0x7fa5c80738b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.342+0000 7fa5cd9bb700 1 -- 172.21.15.44:0/2755522926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5c8100d10 msgr2=0x7fa5c8073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.342+0000 7fa5cd9bb700 1 --2- 172.21.15.44:0/2755522926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5c8100d10 0x7fa5c8073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.342+0000 7fa5cd9bb700 1 -- 172.21.15.44:0/2755522926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c40095f0 con 0x7fa5c8100370 2023-11-16T16:48:35.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.342+0000 7fa5cd9bb700 1 --2- 172.21.15.44:0/2755522926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c8100370 0x7fa5c8072df0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009990 tx=0x7fa5c4009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:35.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.342+0000 7fa5beffd700 1 -- 172.21.15.44:0/2755522926 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c401d070 con 0x7fa5c8100370 2023-11-16T16:48:35.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.342+0000 7fa5beffd700 1 -- 172.21.15.44:0/2755522926 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5c4021410 con 0x7fa5c8100370 2023-11-16T16:48:35.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.342+0000 7fa5beffd700 1 -- 172.21.15.44:0/2755522926 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4017650 con 0x7fa5c8100370 2023-11-16T16:48:35.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.342+0000 7fa5ce9bd700 1 -- 172.21.15.44:0/2755522926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c81101a0 con 0x7fa5c8100370 2023-11-16T16:48:35.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.342+0000 7fa5ce9bd700 1 -- 172.21.15.44:0/2755522926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c8110010 con 0x7fa5c8100370 2023-11-16T16:48:35.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.343+0000 7fa5ce9bd700 1 -- 172.21.15.44:0/2755522926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c8110330 con 0x7fa5c8100370 2023-11-16T16:48:35.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.346+0000 7fa5beffd700 1 -- 172.21.15.44:0/2755522926 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa5c4021810 con 0x7fa5c8100370 2023-11-16T16:48:35.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.347+0000 7fa5beffd700 1 --2- 172.21.15.44:0/2755522926 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa5b40a1260 0x7fa5b40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.347+0000 7fa5beffd700 1 -- 172.21.15.44:0/2755522926 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa5c40ce750 con 0x7fa5c8100370 2023-11-16T16:48:35.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.347+0000 7fa5cd1ba700 1 --2- 172.21.15.44:0/2755522926 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa5b40a1260 0x7fa5b40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.348+0000 7fa5cd1ba700 1 --2- 172.21.15.44:0/2755522926 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa5b40a1260 0x7fa5b40a3710 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80745e0 tx=0x7fa5b8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:35.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.349+0000 7fa5beffd700 1 -- 172.21.15.44:0/2755522926 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa5c4097a60 con 0x7fa5c8100370 2023-11-16T16:48:35.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.498+0000 7f5e743c4700 1 -- 172.21.15.44:0/1117600721 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f5e6c06c4a0 con 0x7f5e6c0ff0d0 2023-11-16T16:48:35.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.498+0000 7f5e637fe700 1 -- 172.21.15.44:0/1117600721 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5e68026020 con 0x7f5e6c0ff0d0 2023-11-16T16:48:35.501 INFO:teuthology.orchestra.run.smithi044.stdout:34359738445 2023-11-16T16:48:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.501+0000 7f5e743c4700 1 -- 172.21.15.44:0/1117600721 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5e580a1260 msgr2=0x7f5e580a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.501+0000 7f5e743c4700 1 --2- 172.21.15.44:0/1117600721 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5e580a1260 0x7f5e580a3710 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c06c6c0 tx=0x7f5e5c009260 comp rx=0 tx=0).stop 2023-11-16T16:48:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.501+0000 7f5e743c4700 1 -- 172.21.15.44:0/1117600721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e6c0ff0d0 msgr2=0x7f5e6c06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.501+0000 7f5e743c4700 1 --2- 172.21.15.44:0/1117600721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e6c0ff0d0 0x7f5e6c06fcd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5e68000c00 tx=0x7f5e68005740 comp rx=0 tx=0).stop 2023-11-16T16:48:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.501+0000 7f5e743c4700 1 -- 172.21.15.44:0/1117600721 shutdown_connections 2023-11-16T16:48:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.501+0000 7f5e743c4700 1 --2- 172.21.15.44:0/1117600721 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f5e580a1260 0x7f5e580a3710 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.501+0000 7f5e743c4700 1 --2- 172.21.15.44:0/1117600721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5e6c10cc80 0x7f5e6c06aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.501+0000 7f5e743c4700 1 --2- 172.21.15.44:0/1117600721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e6c0ff0d0 0x7f5e6c06fcd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.501+0000 7f5e743c4700 1 --2- 172.21.15.44:0/1117600721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e6c0ff9e0 0x7f5e6c070210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.501+0000 7f5e743c4700 1 -- 172.21.15.44:0/1117600721 >> 172.21.15.44:0/1117600721 conn(0x7f5e6c0770a0 msgr2=0x7f5e6c10e120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.501+0000 7f5e743c4700 1 -- 172.21.15.44:0/1117600721 shutdown_connections 2023-11-16T16:48:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.501+0000 7f5e743c4700 1 -- 172.21.15.44:0/1117600721 wait complete. 2023-11-16T16:48:35.554 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.552+0000 7f6adbb99700 1 -- 172.21.15.44:0/3282170709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f6ad4110530 con 0x7f6ad40ffe00 2023-11-16T16:48:35.555 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.553+0000 7f6acaffd700 1 -- 172.21.15.44:0/3282170709 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6ac4099930 con 0x7f6ad40ffe00 2023-11-16T16:48:35.555 INFO:teuthology.orchestra.run.smithi044.stdout:249108103209 2023-11-16T16:48:35.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.555+0000 7f6adbb99700 1 -- 172.21.15.44:0/3282170709 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6ac00a11e0 msgr2=0x7f6ac00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.555+0000 7f6adbb99700 1 --2- 172.21.15.44:0/3282170709 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6ac00a11e0 0x7f6ac00a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0007be0 tx=0x7f6ad00073d0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.555+0000 7f6adbb99700 1 -- 172.21.15.44:0/3282170709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ad40ffe00 msgr2=0x7f6ad410ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.555+0000 7f6adbb99700 1 --2- 172.21.15.44:0/3282170709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ad40ffe00 0x7f6ad410ced0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6ac400ea30 tx=0x7f6ac400ed40 comp rx=0 tx=0).stop 2023-11-16T16:48:35.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.555+0000 7f6adbb99700 1 -- 172.21.15.44:0/3282170709 shutdown_connections 2023-11-16T16:48:35.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.555+0000 7f6adbb99700 1 --2- 172.21.15.44:0/3282170709 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6ac00a11e0 0x7f6ac00a3690 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.555+0000 7f6adbb99700 1 --2- 172.21.15.44:0/3282170709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad4104500 0x7f6ad410d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.555+0000 7f6adbb99700 1 --2- 172.21.15.44:0/3282170709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ad40ff460 0x7f6ad410c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.555+0000 7f6adbb99700 1 --2- 172.21.15.44:0/3282170709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ad40ffe00 0x7f6ad410ced0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.555+0000 7f6adbb99700 1 -- 172.21.15.44:0/3282170709 >> 172.21.15.44:0/3282170709 conn(0x7f6ad40770a0 msgr2=0x7f6ad40fcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:35.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.555+0000 7f6adbb99700 1 -- 172.21.15.44:0/3282170709 shutdown_connections 2023-11-16T16:48:35.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.555+0000 7f6adbb99700 1 -- 172.21.15.44:0/3282170709 wait complete. 2023-11-16T16:48:35.572 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.570+0000 7fa5ce9bd700 1 -- 172.21.15.44:0/2755522926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fa5c804e8c0 con 0x7fa5c8100370 2023-11-16T16:48:35.573 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.571+0000 7fa5beffd700 1 -- 172.21.15.44:0/2755522926 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa5c409b080 con 0x7fa5c8100370 2023-11-16T16:48:35.573 INFO:teuthology.orchestra.run.smithi044.stdout:163208757304 2023-11-16T16:48:35.577 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.574+0000 7fa5ce9bd700 1 -- 172.21.15.44:0/2755522926 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa5b40a1260 msgr2=0x7fa5b40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.577 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.574+0000 7fa5ce9bd700 1 --2- 172.21.15.44:0/2755522926 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa5b40a1260 0x7fa5b40a3710 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80745e0 tx=0x7fa5b8009260 comp rx=0 tx=0).stop 2023-11-16T16:48:35.577 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.574+0000 7fa5ce9bd700 1 -- 172.21.15.44:0/2755522926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c8100370 msgr2=0x7fa5c8072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.577 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.574+0000 7fa5ce9bd700 1 --2- 172.21.15.44:0/2755522926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c8100370 0x7fa5c8072df0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009990 tx=0x7fa5c4009580 comp rx=0 tx=0).stop 2023-11-16T16:48:35.577 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.574+0000 7fa5ce9bd700 1 -- 172.21.15.44:0/2755522926 shutdown_connections 2023-11-16T16:48:35.577 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.574+0000 7fa5ce9bd700 1 --2- 172.21.15.44:0/2755522926 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa5b40a1260 0x7fa5b40a3710 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.577 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.574+0000 7fa5ce9bd700 1 --2- 172.21.15.44:0/2755522926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5c8105410 0x7fa5c80738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.577 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.574+0000 7fa5ce9bd700 1 --2- 172.21.15.44:0/2755522926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c8100370 0x7fa5c8072df0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.577 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.574+0000 7fa5ce9bd700 1 --2- 172.21.15.44:0/2755522926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5c8100d10 0x7fa5c8073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.578 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.574+0000 7fa5ce9bd700 1 -- 172.21.15.44:0/2755522926 >> 172.21.15.44:0/2755522926 conn(0x7fa5c80fbbd0 msgr2=0x7fa5c80fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:35.578 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.575+0000 7fa5ce9bd700 1 -- 172.21.15.44:0/2755522926 shutdown_connections 2023-11-16T16:48:35.578 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:35.575+0000 7fa5ce9bd700 1 -- 172.21.15.44:0/2755522926 wait complete. 2023-11-16T16:48:35.579 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:35 smithi044 ceph-mon[128750]: pgmap v315: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:35.579 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:35 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/4019553301' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-11-16T16:48:35.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:35 smithi077 ceph-mon[131204]: pgmap v315: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:35.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:35 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/4019553301' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-11-16T16:48:35.741 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574917 got 55834574920 for osd.1 2023-11-16T16:48:35.742 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:35.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:35 smithi176 ceph-mon[131630]: pgmap v315: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:35.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:35 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/4019553301' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-11-16T16:48:36.257 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738441 got 34359738445 for osd.0 2023-11-16T16:48:36.258 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:36.357 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757301 got 163208757304 for osd.5 2023-11-16T16:48:36.358 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:36.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:36 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1117600721' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-11-16T16:48:36.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:36 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3282170709' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-11-16T16:48:36.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:36 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2755522926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-11-16T16:48:36.723 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103207 got 249108103209 for osd.8 2023-11-16T16:48:36.723 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:36.723 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-11-16T16:48:36.723 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph pg dump --format=json 2023-11-16T16:48:36.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:36 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1117600721' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-11-16T16:48:36.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:36 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3282170709' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-11-16T16:48:36.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:36 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2755522926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-11-16T16:48:36.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:36 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1117600721' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-11-16T16:48:36.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:36 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3282170709' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-11-16T16:48:36.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:36 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2755522926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-11-16T16:48:37.583 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:37.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:37 smithi077 ceph-mon[131204]: pgmap v316: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:37.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:37 smithi044 ceph-mon[128750]: pgmap v316: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:37.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:37 smithi176 ceph-mon[131630]: pgmap v316: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:39.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.113+0000 7fbd6600a700 1 -- 172.21.15.44:0/3282158377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd6010a890 msgr2=0x7fbd6010ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:39.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.113+0000 7fbd6600a700 1 --2- 172.21.15.44:0/3282158377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd6010a890 0x7fbd6010ccd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800b0a0 tx=0x7fbd5800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:39.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.113+0000 7fbd6600a700 1 -- 172.21.15.44:0/3282158377 shutdown_connections 2023-11-16T16:48:39.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.113+0000 7fbd6600a700 1 --2- 172.21.15.44:0/3282158377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd6010a890 0x7fbd6010ccd0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:39.117 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.113+0000 7fbd6600a700 1 --2- 172.21.15.44:0/3282158377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd60101540 0x7fbd60101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:39.117 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.113+0000 7fbd6600a700 1 --2- 172.21.15.44:0/3282158377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd60101e50 0x7fbd6010a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:39.117 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.113+0000 7fbd6600a700 1 -- 172.21.15.44:0/3282158377 >> 172.21.15.44:0/3282158377 conn(0x7fbd600770a0 msgr2=0x7fbd600774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:39.117 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.113+0000 7fbd6600a700 1 -- 172.21.15.44:0/3282158377 shutdown_connections 2023-11-16T16:48:39.117 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.113+0000 7fbd6600a700 1 -- 172.21.15.44:0/3282158377 wait complete. 2023-11-16T16:48:39.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.115+0000 7fbd6600a700 1 Processor -- start 2023-11-16T16:48:39.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.115+0000 7fbd6600a700 1 -- start start 2023-11-16T16:48:39.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.115+0000 7fbd6600a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd60101540 0x7fbd60198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:39.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.115+0000 7fbd6600a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd60101e50 0x7fbd60198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:39.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.115+0000 7fbd6600a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd6010a890 0x7fbd6019c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:39.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.115+0000 7fbd6600a700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6019c610 con 0x7fbd6010a890 2023-11-16T16:48:39.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.115+0000 7fbd6600a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6019c780 con 0x7fbd60101e50 2023-11-16T16:48:39.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.115+0000 7fbd6600a700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6019c940 con 0x7fbd60101540 2023-11-16T16:48:39.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.115+0000 7fbd5ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd6010a890 0x7fbd6019c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:39.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.115+0000 7fbd5ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd6010a890 0x7fbd6019c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44498/0 (socket says 172.21.15.44:44498) 2023-11-16T16:48:39.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.115+0000 7fbd5ffff700 1 -- 172.21.15.44:0/1318839511 learned_addr learned my addr 172.21.15.44:0/1318839511 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:39.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.115+0000 7fbd5effd700 1 --2- 172.21.15.44:0/1318839511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd60101e50 0x7fbd60198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:39.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.116+0000 7fbd5f7fe700 1 --2- 172.21.15.44:0/1318839511 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd60101540 0x7fbd60198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:39.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.116+0000 7fbd5ffff700 1 -- 172.21.15.44:0/1318839511 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd60101540 msgr2=0x7fbd60198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:39.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.116+0000 7fbd5ffff700 1 --2- 172.21.15.44:0/1318839511 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd60101540 0x7fbd60198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:39.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.116+0000 7fbd5ffff700 1 -- 172.21.15.44:0/1318839511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd60101e50 msgr2=0x7fbd60198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:39.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.116+0000 7fbd5ffff700 1 --2- 172.21.15.44:0/1318839511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd60101e50 0x7fbd60198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:39.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.116+0000 7fbd5ffff700 1 -- 172.21.15.44:0/1318839511 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd58009d00 con 0x7fbd6010a890 2023-11-16T16:48:39.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.116+0000 7fbd5ffff700 1 --2- 172.21.15.44:0/1318839511 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd6010a890 0x7fbd6019c010 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbd58009fd0 tx=0x7fbd5800bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:39.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.116+0000 7fbd5cff9700 1 -- 172.21.15.44:0/1318839511 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5800e040 con 0x7fbd6010a890 2023-11-16T16:48:39.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.117+0000 7fbd5cff9700 1 -- 172.21.15.44:0/1318839511 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd580094c0 con 0x7fbd6010a890 2023-11-16T16:48:39.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.117+0000 7fbd6600a700 1 -- 172.21.15.44:0/1318839511 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd60110350 con 0x7fbd6010a890 2023-11-16T16:48:39.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.117+0000 7fbd5cff9700 1 -- 172.21.15.44:0/1318839511 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd580052a0 con 0x7fbd6010a890 2023-11-16T16:48:39.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.117+0000 7fbd6600a700 1 -- 172.21.15.44:0/1318839511 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd601101c0 con 0x7fbd6010a890 2023-11-16T16:48:39.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.118+0000 7fbd6600a700 1 -- 172.21.15.44:0/1318839511 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd601104e0 con 0x7fbd6010a890 2023-11-16T16:48:39.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.120+0000 7fbd5cff9700 1 -- 172.21.15.44:0/1318839511 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fbd5801d470 con 0x7fbd6010a890 2023-11-16T16:48:39.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.121+0000 7fbd5cff9700 1 --2- 172.21.15.44:0/1318839511 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbd480a1260 0x7fbd480a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:39.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.121+0000 7fbd5cff9700 1 -- 172.21.15.44:0/1318839511 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbd580ce1e0 con 0x7fbd6010a890 2023-11-16T16:48:39.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.121+0000 7fbd5f7fe700 1 --2- 172.21.15.44:0/1318839511 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbd480a1260 0x7fbd480a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:39.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.122+0000 7fbd5f7fe700 1 --2- 172.21.15.44:0/1318839511 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbd480a1260 0x7fbd480a3710 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009c60 tx=0x7fbd50009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:39.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.125+0000 7fbd5cff9700 1 -- 172.21.15.44:0/1318839511 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbd580974f0 con 0x7fbd6010a890 2023-11-16T16:48:39.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.328+0000 7fbd6600a700 1 -- 172.21.15.44:0/1318839511 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbd600ff7e0 con 0x7fbd480a1260 2023-11-16T16:48:39.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.342+0000 7fbd5cff9700 1 -- 172.21.15.44:0/1318839511 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427309 (secure 0 0 0) 0x7fbd600ff7e0 con 0x7fbd480a1260 2023-11-16T16:48:39.351 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:48:39.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.348+0000 7fbd6600a700 1 -- 172.21.15.44:0/1318839511 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbd480a1260 msgr2=0x7fbd480a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:39.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.348+0000 7fbd6600a700 1 --2- 172.21.15.44:0/1318839511 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbd480a1260 0x7fbd480a3710 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009c60 tx=0x7fbd50009260 comp rx=0 tx=0).stop 2023-11-16T16:48:39.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.348+0000 7fbd6600a700 1 -- 172.21.15.44:0/1318839511 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd6010a890 msgr2=0x7fbd6019c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:39.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.348+0000 7fbd6600a700 1 --2- 172.21.15.44:0/1318839511 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd6010a890 0x7fbd6019c010 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbd58009fd0 tx=0x7fbd5800bce0 comp rx=0 tx=0).stop 2023-11-16T16:48:39.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.348+0000 7fbd6600a700 1 -- 172.21.15.44:0/1318839511 shutdown_connections 2023-11-16T16:48:39.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.348+0000 7fbd6600a700 1 --2- 172.21.15.44:0/1318839511 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fbd480a1260 0x7fbd480a3710 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:39.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.348+0000 7fbd6600a700 1 --2- 172.21.15.44:0/1318839511 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd6010a890 0x7fbd6019c010 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:39.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.348+0000 7fbd6600a700 1 --2- 172.21.15.44:0/1318839511 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd60101540 0x7fbd60198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:39.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.348+0000 7fbd6600a700 1 --2- 172.21.15.44:0/1318839511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd60101e50 0x7fbd60198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:39.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.348+0000 7fbd6600a700 1 -- 172.21.15.44:0/1318839511 >> 172.21.15.44:0/1318839511 conn(0x7fbd600770a0 msgr2=0x7fbd600ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:39.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.348+0000 7fbd6600a700 1 -- 172.21.15.44:0/1318839511 shutdown_connections 2023-11-16T16:48:39.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:39.348+0000 7fbd6600a700 1 -- 172.21.15.44:0/1318839511 wait complete. 2023-11-16T16:48:39.359 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2023-11-16T16:48:39.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:39 smithi077 ceph-mon[131204]: pgmap v317: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:39.533 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:39 smithi044 ceph-mon[128750]: pgmap v317: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:39.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:39 smithi176 ceph-mon[131630]: pgmap v317: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:39.885 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2023-11-16T16:48:38.553349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652320,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":242,"num_read_kb":236,"num_write":259,"num_write_kb":3028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80612,"kb_used_data":11716,"kb_used_omap":0,"kb_used_meta":68800,"kb_avail":1124861212,"statfs":{"total":1151940427776,"available":1151857881088,"internally_reserved":0,"allocated":11997184,"data_stored":7786397,"data_compressed":318969,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":0,"internal_metadata":70451200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003853"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625866+0000","last_change":"2023-11-16T16:47:18.183024+0000","last_active":"2023-11-16T16:47:43.625866+0000","last_peered":"2023-11-16T16:47:43.625866+0000","last_clean":"2023-11-16T16:47:43.625866+0000","last_became_active":"2023-11-16T16:47:18.182387+0000","last_became_peered":"2023-11-16T16:47:18.182387+0000","last_unstale":"2023-11-16T16:47:43.625866+0000","last_undegraded":"2023-11-16T16:47:43.625866+0000","last_fullsized":"2023-11-16T16:47:43.625866+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:33:54.818555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626848+0000","last_change":"2023-11-16T16:47:18.200135+0000","last_active":"2023-11-16T16:47:43.626848+0000","last_peered":"2023-11-16T16:47:43.626848+0000","last_clean":"2023-11-16T16:47:43.626848+0000","last_became_active":"2023-11-16T16:47:18.194732+0000","last_became_peered":"2023-11-16T16:47:18.194732+0000","last_unstale":"2023-11-16T16:47:43.626848+0000","last_undegraded":"2023-11-16T16:47:43.626848+0000","last_fullsized":"2023-11-16T16:47:43.626848+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:35:44.790080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626424+0000","last_change":"2023-11-16T16:47:18.184127+0000","last_active":"2023-11-16T16:47:43.626424+0000","last_peered":"2023-11-16T16:47:43.626424+0000","last_clean":"2023-11-16T16:47:43.626424+0000","last_became_active":"2023-11-16T16:47:18.183163+0000","last_became_peered":"2023-11-16T16:47:18.183163+0000","last_unstale":"2023-11-16T16:47:43.626424+0000","last_undegraded":"2023-11-16T16:47:43.626424+0000","last_fullsized":"2023-11-16T16:47:43.626424+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:22:40.700926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629755+0000","last_change":"2023-11-16T16:47:18.177476+0000","last_active":"2023-11-16T16:47:43.629755+0000","last_peered":"2023-11-16T16:47:43.629755+0000","last_clean":"2023-11-16T16:47:43.629755+0000","last_became_active":"2023-11-16T16:47:18.177178+0000","last_became_peered":"2023-11-16T16:47:18.177178+0000","last_unstale":"2023-11-16T16:47:43.629755+0000","last_undegraded":"2023-11-16T16:47:43.629755+0000","last_fullsized":"2023-11-16T16:47:43.629755+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:40:05.452664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626685+0000","last_change":"2023-11-16T16:47:18.178511+0000","last_active":"2023-11-16T16:47:43.626685+0000","last_peered":"2023-11-16T16:47:43.626685+0000","last_clean":"2023-11-16T16:47:43.626685+0000","last_became_active":"2023-11-16T16:47:18.178198+0000","last_became_peered":"2023-11-16T16:47:18.178198+0000","last_unstale":"2023-11-16T16:47:43.626685+0000","last_undegraded":"2023-11-16T16:47:43.626685+0000","last_fullsized":"2023-11-16T16:47:43.626685+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:53:35.196107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626326+0000","last_change":"2023-11-16T16:47:18.186552+0000","last_active":"2023-11-16T16:47:43.626326+0000","last_peered":"2023-11-16T16:47:43.626326+0000","last_clean":"2023-11-16T16:47:43.626326+0000","last_became_active":"2023-11-16T16:47:18.186238+0000","last_became_peered":"2023-11-16T16:47:18.186238+0000","last_unstale":"2023-11-16T16:47:43.626326+0000","last_undegraded":"2023-11-16T16:47:43.626326+0000","last_fullsized":"2023-11-16T16:47:43.626326+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:36:52.844919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622387+0000","last_change":"2023-11-16T16:47:18.179839+0000","last_active":"2023-11-16T16:47:43.622387+0000","last_peered":"2023-11-16T16:47:43.622387+0000","last_clean":"2023-11-16T16:47:43.622387+0000","last_became_active":"2023-11-16T16:47:18.179060+0000","last_became_peered":"2023-11-16T16:47:18.179060+0000","last_unstale":"2023-11-16T16:47:43.622387+0000","last_undegraded":"2023-11-16T16:47:43.622387+0000","last_fullsized":"2023-11-16T16:47:43.622387+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:55:30.972787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.633015+0000","last_change":"2023-11-16T16:47:43.631738+0000","last_active":"2023-11-16T16:47:43.633015+0000","last_peered":"2023-11-16T16:47:43.633015+0000","last_clean":"2023-11-16T16:47:43.633015+0000","last_became_active":"2023-11-16T16:47:43.631146+0000","last_became_peered":"2023-11-16T16:47:43.631146+0000","last_unstale":"2023-11-16T16:47:43.633015+0000","last_undegraded":"2023-11-16T16:47:43.633015+0000","last_fullsized":"2023-11-16T16:47:43.633015+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:32:26.552018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947385+0000","last_change":"2023-11-16T16:47:18.177834+0000","last_active":"2023-11-16T16:47:43.947385+0000","last_peered":"2023-11-16T16:47:43.947385+0000","last_clean":"2023-11-16T16:47:43.947385+0000","last_became_active":"2023-11-16T16:47:18.176299+0000","last_became_peered":"2023-11-16T16:47:18.176299+0000","last_unstale":"2023-11-16T16:47:43.947385+0000","last_undegraded":"2023-11-16T16:47:43.947385+0000","last_fullsized":"2023-11-16T16:47:43.947385+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:18:34.674950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626777+0000","last_change":"2023-11-16T16:47:18.179394+0000","last_active":"2023-11-16T16:47:43.626777+0000","last_peered":"2023-11-16T16:47:43.626777+0000","last_clean":"2023-11-16T16:47:43.626777+0000","last_became_active":"2023-11-16T16:47:18.179119+0000","last_became_peered":"2023-11-16T16:47:18.179119+0000","last_unstale":"2023-11-16T16:47:43.626777+0000","last_undegraded":"2023-11-16T16:47:43.626777+0000","last_fullsized":"2023-11-16T16:47:43.626777+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:32:22.770618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623315+0000","last_change":"2023-11-16T16:47:18.183585+0000","last_active":"2023-11-16T16:47:43.623315+0000","last_peered":"2023-11-16T16:47:43.623315+0000","last_clean":"2023-11-16T16:47:43.623315+0000","last_became_active":"2023-11-16T16:47:18.183209+0000","last_became_peered":"2023-11-16T16:47:18.183209+0000","last_unstale":"2023-11-16T16:47:43.623315+0000","last_undegraded":"2023-11-16T16:47:43.623315+0000","last_fullsized":"2023-11-16T16:47:43.623315+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:22:47.894810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629596+0000","last_change":"2023-11-16T16:47:18.186503+0000","last_active":"2023-11-16T16:47:43.629596+0000","last_peered":"2023-11-16T16:47:43.629596+0000","last_clean":"2023-11-16T16:47:43.629596+0000","last_became_active":"2023-11-16T16:47:18.186193+0000","last_became_peered":"2023-11-16T16:47:18.186193+0000","last_unstale":"2023-11-16T16:47:43.629596+0000","last_undegraded":"2023-11-16T16:47:43.629596+0000","last_fullsized":"2023-11-16T16:47:43.629596+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:59:03.486858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629884+0000","last_change":"2023-11-16T16:47:18.193842+0000","last_active":"2023-11-16T16:47:43.629884+0000","last_peered":"2023-11-16T16:47:43.629884+0000","last_clean":"2023-11-16T16:47:43.629884+0000","last_became_active":"2023-11-16T16:47:18.192309+0000","last_became_peered":"2023-11-16T16:47:18.192309+0000","last_unstale":"2023-11-16T16:47:43.629884+0000","last_undegraded":"2023-11-16T16:47:43.629884+0000","last_fullsized":"2023-11-16T16:47:43.629884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:53:33.884565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625812+0000","last_change":"2023-11-16T16:47:18.192359+0000","last_active":"2023-11-16T16:47:43.625812+0000","last_peered":"2023-11-16T16:47:43.625812+0000","last_clean":"2023-11-16T16:47:43.625812+0000","last_became_active":"2023-11-16T16:47:18.189765+0000","last_became_peered":"2023-11-16T16:47:18.189765+0000","last_unstale":"2023-11-16T16:47:43.625812+0000","last_undegraded":"2023-11-16T16:47:43.625812+0000","last_fullsized":"2023-11-16T16:47:43.625812+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:17:15.440725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625875+0000","last_change":"2023-11-16T16:47:18.178386+0000","last_active":"2023-11-16T16:47:43.625875+0000","last_peered":"2023-11-16T16:47:43.625875+0000","last_clean":"2023-11-16T16:47:43.625875+0000","last_became_active":"2023-11-16T16:47:18.178041+0000","last_became_peered":"2023-11-16T16:47:18.178041+0000","last_unstale":"2023-11-16T16:47:43.625875+0000","last_undegraded":"2023-11-16T16:47:43.625875+0000","last_fullsized":"2023-11-16T16:47:43.625875+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:27:47.278353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.946667+0000","last_change":"2023-11-16T16:47:18.174424+0000","last_active":"2023-11-16T16:47:43.946667+0000","last_peered":"2023-11-16T16:47:43.946667+0000","last_clean":"2023-11-16T16:47:43.946667+0000","last_became_active":"2023-11-16T16:47:18.174078+0000","last_became_peered":"2023-11-16T16:47:18.174078+0000","last_unstale":"2023-11-16T16:47:43.946667+0000","last_undegraded":"2023-11-16T16:47:43.946667+0000","last_fullsized":"2023-11-16T16:47:43.946667+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:14:13.491783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625739+0000","last_change":"2023-11-16T16:47:18.183161+0000","last_active":"2023-11-16T16:47:43.625739+0000","last_peered":"2023-11-16T16:47:43.625739+0000","last_clean":"2023-11-16T16:47:43.625739+0000","last_became_active":"2023-11-16T16:47:18.182806+0000","last_became_peered":"2023-11-16T16:47:18.182806+0000","last_unstale":"2023-11-16T16:47:43.625739+0000","last_undegraded":"2023-11-16T16:47:43.625739+0000","last_fullsized":"2023-11-16T16:47:43.625739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:50:04.072542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622713+0000","last_change":"2023-11-16T16:47:18.187225+0000","last_active":"2023-11-16T16:47:43.622713+0000","last_peered":"2023-11-16T16:47:43.622713+0000","last_clean":"2023-11-16T16:47:43.622713+0000","last_became_active":"2023-11-16T16:47:18.186858+0000","last_became_peered":"2023-11-16T16:47:18.186858+0000","last_unstale":"2023-11-16T16:47:43.622713+0000","last_undegraded":"2023-11-16T16:47:43.622713+0000","last_fullsized":"2023-11-16T16:47:43.622713+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:42:28.736883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623323+0000","last_change":"2023-11-16T16:47:18.190947+0000","last_active":"2023-11-16T16:47:43.623323+0000","last_peered":"2023-11-16T16:47:43.623323+0000","last_clean":"2023-11-16T16:47:43.623323+0000","last_became_active":"2023-11-16T16:47:18.190737+0000","last_became_peered":"2023-11-16T16:47:18.190737+0000","last_unstale":"2023-11-16T16:47:43.623323+0000","last_undegraded":"2023-11-16T16:47:43.623323+0000","last_fullsized":"2023-11-16T16:47:43.623323+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:34:59.150388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622328+0000","last_change":"2023-11-16T16:47:22.659730+0000","last_active":"2023-11-16T16:47:43.622328+0000","last_peered":"2023-11-16T16:47:43.622328+0000","last_clean":"2023-11-16T16:47:43.622328+0000","last_became_active":"2023-11-16T16:47:22.659313+0000","last_became_peered":"2023-11-16T16:47:22.659313+0000","last_unstale":"2023-11-16T16:47:43.622328+0000","last_undegraded":"2023-11-16T16:47:43.622328+0000","last_fullsized":"2023-11-16T16:47:43.622328+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:05:05.336507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'4","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622965+0000","last_change":"2023-11-16T16:47:17.629748+0000","last_active":"2023-11-16T16:47:43.622965+0000","last_peered":"2023-11-16T16:47:43.622965+0000","last_clean":"2023-11-16T16:47:43.622965+0000","last_became_active":"2023-11-16T16:47:17.629284+0000","last_became_peered":"2023-11-16T16:47:17.629284+0000","last_unstale":"2023-11-16T16:47:43.622965+0000","last_undegraded":"2023-11-16T16:47:43.622965+0000","last_fullsized":"2023-11-16T16:47:43.622965+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:41:29.574724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970501+0000","last_change":"2023-11-16T16:47:18.176239+0000","last_active":"2023-11-16T16:47:43.970501+0000","last_peered":"2023-11-16T16:47:43.970501+0000","last_clean":"2023-11-16T16:47:43.970501+0000","last_became_active":"2023-11-16T16:47:18.175879+0000","last_became_peered":"2023-11-16T16:47:18.175879+0000","last_unstale":"2023-11-16T16:47:43.970501+0000","last_undegraded":"2023-11-16T16:47:43.970501+0000","last_fullsized":"2023-11-16T16:47:43.970501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:32:09.544453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.628988+0000","last_change":"2023-11-16T16:47:22.659679+0000","last_active":"2023-11-16T16:47:43.628988+0000","last_peered":"2023-11-16T16:47:43.628988+0000","last_clean":"2023-11-16T16:47:43.628988+0000","last_became_active":"2023-11-16T16:47:22.658946+0000","last_became_peered":"2023-11-16T16:47:22.658946+0000","last_unstale":"2023-11-16T16:47:43.628988+0000","last_undegraded":"2023-11-16T16:47:43.628988+0000","last_fullsized":"2023-11-16T16:47:43.628988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:33:44.392111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"82'1","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.971259+0000","last_change":"2023-11-16T16:47:17.643944+0000","last_active":"2023-11-16T16:47:43.971259+0000","last_peered":"2023-11-16T16:47:43.971259+0000","last_clean":"2023-11-16T16:47:43.971259+0000","last_became_active":"2023-11-16T16:47:17.643470+0000","last_became_peered":"2023-11-16T16:47:17.643470+0000","last_unstale":"2023-11-16T16:47:43.971259+0000","last_undegraded":"2023-11-16T16:47:43.971259+0000","last_fullsized":"2023-11-16T16:47:43.971259+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:28:22.870692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.627110+0000","last_change":"2023-11-16T16:47:18.198139+0000","last_active":"2023-11-16T16:47:43.627110+0000","last_peered":"2023-11-16T16:47:43.627110+0000","last_clean":"2023-11-16T16:47:43.627110+0000","last_became_active":"2023-11-16T16:47:18.197762+0000","last_became_peered":"2023-11-16T16:47:18.197762+0000","last_unstale":"2023-11-16T16:47:43.627110+0000","last_undegraded":"2023-11-16T16:47:43.627110+0000","last_fullsized":"2023-11-16T16:47:43.627110+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:27:32.243043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629208+0000","last_change":"2023-11-16T16:47:22.655810+0000","last_active":"2023-11-16T16:47:43.629208+0000","last_peered":"2023-11-16T16:47:43.629208+0000","last_clean":"2023-11-16T16:47:43.629208+0000","last_became_active":"2023-11-16T16:47:22.655388+0000","last_became_peered":"2023-11-16T16:47:22.655388+0000","last_unstale":"2023-11-16T16:47:43.629208+0000","last_undegraded":"2023-11-16T16:47:43.629208+0000","last_fullsized":"2023-11-16T16:47:43.629208+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:05:46.905125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'2","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.635266+0000","last_change":"2023-11-16T16:47:43.635266+0000","last_active":"2023-11-16T16:47:43.635266+0000","last_peered":"2023-11-16T16:47:43.635266+0000","last_clean":"2023-11-16T16:47:43.635266+0000","last_became_active":"2023-11-16T16:47:43.630583+0000","last_became_peered":"2023-11-16T16:47:43.630583+0000","last_unstale":"2023-11-16T16:47:43.635266+0000","last_undegraded":"2023-11-16T16:47:43.635266+0000","last_fullsized":"2023-11-16T16:47:43.635266+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:51:43.792892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.946938+0000","last_change":"2023-11-16T16:47:18.186341+0000","last_active":"2023-11-16T16:47:43.946938+0000","last_peered":"2023-11-16T16:47:43.946938+0000","last_clean":"2023-11-16T16:47:43.946938+0000","last_became_active":"2023-11-16T16:47:18.186066+0000","last_became_peered":"2023-11-16T16:47:18.186066+0000","last_unstale":"2023-11-16T16:47:43.946938+0000","last_undegraded":"2023-11-16T16:47:43.946938+0000","last_fullsized":"2023-11-16T16:47:43.946938+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:22:46.055671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629005+0000","last_change":"2023-11-16T16:47:22.657227+0000","last_active":"2023-11-16T16:47:43.629005+0000","last_peered":"2023-11-16T16:47:43.629005+0000","last_clean":"2023-11-16T16:47:43.629005+0000","last_became_active":"2023-11-16T16:47:22.656835+0000","last_became_peered":"2023-11-16T16:47:22.656835+0000","last_unstale":"2023-11-16T16:47:43.629005+0000","last_undegraded":"2023-11-16T16:47:43.629005+0000","last_fullsized":"2023-11-16T16:47:43.629005+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:07.552894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625950+0000","last_change":"2023-11-16T16:47:17.630194+0000","last_active":"2023-11-16T16:47:43.625950+0000","last_peered":"2023-11-16T16:47:43.625950+0000","last_clean":"2023-11-16T16:47:43.625950+0000","last_became_active":"2023-11-16T16:47:17.629886+0000","last_became_peered":"2023-11-16T16:47:17.629886+0000","last_unstale":"2023-11-16T16:47:43.625950+0000","last_undegraded":"2023-11-16T16:47:43.625950+0000","last_fullsized":"2023-11-16T16:47:43.625950+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:57:09.970289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630557+0000","last_change":"2023-11-16T16:47:18.181596+0000","last_active":"2023-11-16T16:47:43.630557+0000","last_peered":"2023-11-16T16:47:43.630557+0000","last_clean":"2023-11-16T16:47:43.630557+0000","last_became_active":"2023-11-16T16:47:18.181287+0000","last_became_peered":"2023-11-16T16:47:18.181287+0000","last_unstale":"2023-11-16T16:47:43.630557+0000","last_undegraded":"2023-11-16T16:47:43.630557+0000","last_fullsized":"2023-11-16T16:47:43.630557+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:31:55.798497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622763+0000","last_change":"2023-11-16T16:47:22.649197+0000","last_active":"2023-11-16T16:47:43.622763+0000","last_peered":"2023-11-16T16:47:43.622763+0000","last_clean":"2023-11-16T16:47:43.622763+0000","last_became_active":"2023-11-16T16:47:22.648664+0000","last_became_peered":"2023-11-16T16:47:22.648664+0000","last_unstale":"2023-11-16T16:47:43.622763+0000","last_undegraded":"2023-11-16T16:47:43.622763+0000","last_fullsized":"2023-11-16T16:47:43.622763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:56:11.119940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"82'1","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630141+0000","last_change":"2023-11-16T16:47:17.627672+0000","last_active":"2023-11-16T16:47:43.630141+0000","last_peered":"2023-11-16T16:47:43.630141+0000","last_clean":"2023-11-16T16:47:43.630141+0000","last_became_active":"2023-11-16T16:47:17.627383+0000","last_became_peered":"2023-11-16T16:47:17.627383+0000","last_unstale":"2023-11-16T16:47:43.630141+0000","last_undegraded":"2023-11-16T16:47:43.630141+0000","last_fullsized":"2023-11-16T16:47:43.630141+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:35:57.379840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625043+0000","last_change":"2023-11-16T16:47:18.180983+0000","last_active":"2023-11-16T16:47:43.625043+0000","last_peered":"2023-11-16T16:47:43.625043+0000","last_clean":"2023-11-16T16:47:43.625043+0000","last_became_active":"2023-11-16T16:47:18.180702+0000","last_became_peered":"2023-11-16T16:47:18.180702+0000","last_unstale":"2023-11-16T16:47:43.625043+0000","last_undegraded":"2023-11-16T16:47:43.625043+0000","last_fullsized":"2023-11-16T16:47:43.625043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:20:27.895648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622657+0000","last_change":"2023-11-16T16:47:22.659898+0000","last_active":"2023-11-16T16:47:43.622657+0000","last_peered":"2023-11-16T16:47:43.622657+0000","last_clean":"2023-11-16T16:47:43.622657+0000","last_became_active":"2023-11-16T16:47:22.659326+0000","last_became_peered":"2023-11-16T16:47:22.659326+0000","last_unstale":"2023-11-16T16:47:43.622657+0000","last_undegraded":"2023-11-16T16:47:43.622657+0000","last_fullsized":"2023-11-16T16:47:43.622657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:57:36.289645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"82'1","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.949231+0000","last_change":"2023-11-16T16:47:43.949231+0000","last_active":"2023-11-16T16:47:43.949231+0000","last_peered":"2023-11-16T16:47:43.949231+0000","last_clean":"2023-11-16T16:47:43.949231+0000","last_became_active":"2023-11-16T16:47:43.948766+0000","last_became_peered":"2023-11-16T16:47:43.948766+0000","last_unstale":"2023-11-16T16:47:43.949231+0000","last_undegraded":"2023-11-16T16:47:43.949231+0000","last_fullsized":"2023-11-16T16:47:43.949231+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:46:57.086811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630999+0000","last_change":"2023-11-16T16:47:18.185263+0000","last_active":"2023-11-16T16:47:43.630999+0000","last_peered":"2023-11-16T16:47:43.630999+0000","last_clean":"2023-11-16T16:47:43.630999+0000","last_became_active":"2023-11-16T16:47:18.184910+0000","last_became_peered":"2023-11-16T16:47:18.184910+0000","last_unstale":"2023-11-16T16:47:43.630999+0000","last_undegraded":"2023-11-16T16:47:43.630999+0000","last_fullsized":"2023-11-16T16:47:43.630999+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:25:21.463648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625581+0000","last_change":"2023-11-16T16:47:22.659711+0000","last_active":"2023-11-16T16:47:43.625581+0000","last_peered":"2023-11-16T16:47:43.625581+0000","last_clean":"2023-11-16T16:47:43.625581+0000","last_became_active":"2023-11-16T16:47:22.659087+0000","last_became_peered":"2023-11-16T16:47:22.659087+0000","last_unstale":"2023-11-16T16:47:43.625581+0000","last_undegraded":"2023-11-16T16:47:43.625581+0000","last_fullsized":"2023-11-16T16:47:43.625581+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:13:26.339037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629045+0000","last_change":"2023-11-16T16:47:17.629703+0000","last_active":"2023-11-16T16:47:43.629045+0000","last_peered":"2023-11-16T16:47:43.629045+0000","last_clean":"2023-11-16T16:47:43.629045+0000","last_became_active":"2023-11-16T16:47:17.629245+0000","last_became_peered":"2023-11-16T16:47:17.629245+0000","last_unstale":"2023-11-16T16:47:43.629045+0000","last_undegraded":"2023-11-16T16:47:43.629045+0000","last_fullsized":"2023-11-16T16:47:43.629045+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:59:34.790629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970314+0000","last_change":"2023-11-16T16:47:18.187149+0000","last_active":"2023-11-16T16:47:43.970314+0000","last_peered":"2023-11-16T16:47:43.970314+0000","last_clean":"2023-11-16T16:47:43.970314+0000","last_became_active":"2023-11-16T16:47:18.185717+0000","last_became_peered":"2023-11-16T16:47:18.185717+0000","last_unstale":"2023-11-16T16:47:43.970314+0000","last_undegraded":"2023-11-16T16:47:43.970314+0000","last_fullsized":"2023-11-16T16:47:43.970314+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:23:04.064643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626668+0000","last_change":"2023-11-16T16:47:22.663025+0000","last_active":"2023-11-16T16:47:43.626668+0000","last_peered":"2023-11-16T16:47:43.626668+0000","last_clean":"2023-11-16T16:47:43.626668+0000","last_became_active":"2023-11-16T16:47:22.662596+0000","last_became_peered":"2023-11-16T16:47:22.662596+0000","last_unstale":"2023-11-16T16:47:43.626668+0000","last_undegraded":"2023-11-16T16:47:43.626668+0000","last_fullsized":"2023-11-16T16:47:43.626668+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:47:26.815530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629668+0000","last_change":"2023-11-16T16:47:17.626357+0000","last_active":"2023-11-16T16:47:43.629668+0000","last_peered":"2023-11-16T16:47:43.629668+0000","last_clean":"2023-11-16T16:47:43.629668+0000","last_became_active":"2023-11-16T16:47:17.626029+0000","last_became_peered":"2023-11-16T16:47:17.626029+0000","last_unstale":"2023-11-16T16:47:43.629668+0000","last_undegraded":"2023-11-16T16:47:43.629668+0000","last_fullsized":"2023-11-16T16:47:43.629668+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:55:28.659340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.621841+0000","last_change":"2023-11-16T16:47:18.177153+0000","last_active":"2023-11-16T16:47:43.621841+0000","last_peered":"2023-11-16T16:47:43.621841+0000","last_clean":"2023-11-16T16:47:43.621841+0000","last_became_active":"2023-11-16T16:47:18.176751+0000","last_became_peered":"2023-11-16T16:47:18.176751+0000","last_unstale":"2023-11-16T16:47:43.621841+0000","last_undegraded":"2023-11-16T16:47:43.621841+0000","last_fullsized":"2023-11-16T16:47:43.621841+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:17:58.058194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970584+0000","last_change":"2023-11-16T16:47:22.652718+0000","last_active":"2023-11-16T16:47:43.970584+0000","last_peered":"2023-11-16T16:47:43.970584+0000","last_clean":"2023-11-16T16:47:43.970584+0000","last_became_active":"2023-11-16T16:47:22.652278+0000","last_became_peered":"2023-11-16T16:47:22.652278+0000","last_unstale":"2023-11-16T16:47:43.970584+0000","last_undegraded":"2023-11-16T16:47:43.970584+0000","last_fullsized":"2023-11-16T16:47:43.970584+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:43:41.546583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626473+0000","last_change":"2023-11-16T16:47:17.627308+0000","last_active":"2023-11-16T16:47:43.626473+0000","last_peered":"2023-11-16T16:47:43.626473+0000","last_clean":"2023-11-16T16:47:43.626473+0000","last_became_active":"2023-11-16T16:47:17.626992+0000","last_became_peered":"2023-11-16T16:47:17.626992+0000","last_unstale":"2023-11-16T16:47:43.626473+0000","last_undegraded":"2023-11-16T16:47:43.626473+0000","last_fullsized":"2023-11-16T16:47:43.626473+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:41:51.914757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947323+0000","last_change":"2023-11-16T16:47:18.188015+0000","last_active":"2023-11-16T16:47:43.947323+0000","last_peered":"2023-11-16T16:47:43.947323+0000","last_clean":"2023-11-16T16:47:43.947323+0000","last_became_active":"2023-11-16T16:47:18.187730+0000","last_became_peered":"2023-11-16T16:47:18.187730+0000","last_unstale":"2023-11-16T16:47:43.947323+0000","last_undegraded":"2023-11-16T16:47:43.947323+0000","last_fullsized":"2023-11-16T16:47:43.947323+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:39:02.651682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.621932+0000","last_change":"2023-11-16T16:47:22.655656+0000","last_active":"2023-11-16T16:47:43.621932+0000","last_peered":"2023-11-16T16:47:43.621932+0000","last_clean":"2023-11-16T16:47:43.621932+0000","last_became_active":"2023-11-16T16:47:22.655133+0000","last_became_peered":"2023-11-16T16:47:22.655133+0000","last_unstale":"2023-11-16T16:47:43.621932+0000","last_undegraded":"2023-11-16T16:47:43.621932+0000","last_fullsized":"2023-11-16T16:47:43.621932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:19:21.932819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947947+0000","last_change":"2023-11-16T16:47:43.947947+0000","last_active":"2023-11-16T16:47:43.947947+0000","last_peered":"2023-11-16T16:47:43.947947+0000","last_clean":"2023-11-16T16:47:43.947947+0000","last_became_active":"2023-11-16T16:47:43.947478+0000","last_became_peered":"2023-11-16T16:47:43.947478+0000","last_unstale":"2023-11-16T16:47:43.947947+0000","last_undegraded":"2023-11-16T16:47:43.947947+0000","last_fullsized":"2023-11-16T16:47:43.947947+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:40:14.254873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.632657+0000","last_change":"2023-11-16T16:47:43.632657+0000","last_active":"2023-11-16T16:47:43.632657+0000","last_peered":"2023-11-16T16:47:43.632657+0000","last_clean":"2023-11-16T16:47:43.632657+0000","last_became_active":"2023-11-16T16:47:43.631573+0000","last_became_peered":"2023-11-16T16:47:43.631573+0000","last_unstale":"2023-11-16T16:47:43.632657+0000","last_undegraded":"2023-11-16T16:47:43.632657+0000","last_fullsized":"2023-11-16T16:47:43.632657+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:45:08.317787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629418+0000","last_change":"2023-11-16T16:47:22.661639+0000","last_active":"2023-11-16T16:47:43.629418+0000","last_peered":"2023-11-16T16:47:43.629418+0000","last_clean":"2023-11-16T16:47:43.629418+0000","last_became_active":"2023-11-16T16:47:22.661230+0000","last_became_peered":"2023-11-16T16:47:22.661230+0000","last_unstale":"2023-11-16T16:47:43.629418+0000","last_undegraded":"2023-11-16T16:47:43.629418+0000","last_fullsized":"2023-11-16T16:47:43.629418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:04:16.909613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"82'3","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630143+0000","last_change":"2023-11-16T16:47:17.638209+0000","last_active":"2023-11-16T16:47:43.630143+0000","last_peered":"2023-11-16T16:47:43.630143+0000","last_clean":"2023-11-16T16:47:43.630143+0000","last_became_active":"2023-11-16T16:47:17.637957+0000","last_became_peered":"2023-11-16T16:47:17.637957+0000","last_unstale":"2023-11-16T16:47:43.630143+0000","last_undegraded":"2023-11-16T16:47:43.630143+0000","last_fullsized":"2023-11-16T16:47:43.630143+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:08:43.599916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947142+0000","last_change":"2023-11-16T16:47:18.182693+0000","last_active":"2023-11-16T16:47:43.947142+0000","last_peered":"2023-11-16T16:47:43.947142+0000","last_clean":"2023-11-16T16:47:43.947142+0000","last_became_active":"2023-11-16T16:47:18.182370+0000","last_became_peered":"2023-11-16T16:47:18.182370+0000","last_unstale":"2023-11-16T16:47:43.947142+0000","last_undegraded":"2023-11-16T16:47:43.947142+0000","last_fullsized":"2023-11-16T16:47:43.947142+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:38:49.650460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.621838+0000","last_change":"2023-11-16T16:47:22.649348+0000","last_active":"2023-11-16T16:47:43.621838+0000","last_peered":"2023-11-16T16:47:43.621838+0000","last_clean":"2023-11-16T16:47:43.621838+0000","last_became_active":"2023-11-16T16:47:22.648860+0000","last_became_peered":"2023-11-16T16:47:22.648860+0000","last_unstale":"2023-11-16T16:47:43.621838+0000","last_undegraded":"2023-11-16T16:47:43.621838+0000","last_fullsized":"2023-11-16T16:47:43.621838+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:45:18.960009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'31","reported_seq":157,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:48:34.244281+0000","last_change":"2023-11-16T16:47:17.619417+0000","last_active":"2023-11-16T16:48:34.244281+0000","last_peered":"2023-11-16T16:48:34.244281+0000","last_clean":"2023-11-16T16:48:34.244281+0000","last_became_active":"2023-11-16T16:47:17.619086+0000","last_became_peered":"2023-11-16T16:47:17.619086+0000","last_unstale":"2023-11-16T16:48:34.244281+0000","last_undegraded":"2023-11-16T16:48:34.244281+0000","last_fullsized":"2023-11-16T16:48:34.244281+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:27:30.044686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60908,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":100,"num_read_kb":111,"num_write":31,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625947+0000","last_change":"2023-11-16T16:47:18.177537+0000","last_active":"2023-11-16T16:47:43.625947+0000","last_peered":"2023-11-16T16:47:43.625947+0000","last_clean":"2023-11-16T16:47:43.625947+0000","last_became_active":"2023-11-16T16:47:18.177112+0000","last_became_peered":"2023-11-16T16:47:18.177112+0000","last_unstale":"2023-11-16T16:47:43.625947+0000","last_undegraded":"2023-11-16T16:47:43.625947+0000","last_fullsized":"2023-11-16T16:47:43.625947+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:17:18.541523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630846+0000","last_change":"2023-11-16T16:47:22.658086+0000","last_active":"2023-11-16T16:47:43.630846+0000","last_peered":"2023-11-16T16:47:43.630846+0000","last_clean":"2023-11-16T16:47:43.630846+0000","last_became_active":"2023-11-16T16:47:22.657725+0000","last_became_peered":"2023-11-16T16:47:22.657725+0000","last_unstale":"2023-11-16T16:47:43.630846+0000","last_undegraded":"2023-11-16T16:47:43.630846+0000","last_fullsized":"2023-11-16T16:47:43.630846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:53:51.729635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"82'2","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622837+0000","last_change":"2023-11-16T16:47:17.625405+0000","last_active":"2023-11-16T16:47:43.622837+0000","last_peered":"2023-11-16T16:47:43.622837+0000","last_clean":"2023-11-16T16:47:43.622837+0000","last_became_active":"2023-11-16T16:47:17.625026+0000","last_became_peered":"2023-11-16T16:47:17.625026+0000","last_unstale":"2023-11-16T16:47:43.622837+0000","last_undegraded":"2023-11-16T16:47:43.622837+0000","last_fullsized":"2023-11-16T16:47:43.622837+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:45:14.505117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629372+0000","last_change":"2023-11-16T16:47:18.196345+0000","last_active":"2023-11-16T16:47:43.629372+0000","last_peered":"2023-11-16T16:47:43.629372+0000","last_clean":"2023-11-16T16:47:43.629372+0000","last_became_active":"2023-11-16T16:47:18.196040+0000","last_became_peered":"2023-11-16T16:47:18.196040+0000","last_unstale":"2023-11-16T16:47:43.629372+0000","last_undegraded":"2023-11-16T16:47:43.629372+0000","last_fullsized":"2023-11-16T16:47:43.629372+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:07:42.452837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.624832+0000","last_change":"2023-11-16T16:47:22.660955+0000","last_active":"2023-11-16T16:47:43.624832+0000","last_peered":"2023-11-16T16:47:43.624832+0000","last_clean":"2023-11-16T16:47:43.624832+0000","last_became_active":"2023-11-16T16:47:22.660574+0000","last_became_peered":"2023-11-16T16:47:22.660574+0000","last_unstale":"2023-11-16T16:47:43.624832+0000","last_undegraded":"2023-11-16T16:47:43.624832+0000","last_fullsized":"2023-11-16T16:47:43.624832+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:41:43.745744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630317+0000","last_change":"2023-11-16T16:47:17.630481+0000","last_active":"2023-11-16T16:47:43.630317+0000","last_peered":"2023-11-16T16:47:43.630317+0000","last_clean":"2023-11-16T16:47:43.630317+0000","last_became_active":"2023-11-16T16:47:17.630190+0000","last_became_peered":"2023-11-16T16:47:17.630190+0000","last_unstale":"2023-11-16T16:47:43.630317+0000","last_undegraded":"2023-11-16T16:47:43.630317+0000","last_fullsized":"2023-11-16T16:47:43.630317+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:03:46.177268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622893+0000","last_change":"2023-11-16T16:47:18.179833+0000","last_active":"2023-11-16T16:47:43.622893+0000","last_peered":"2023-11-16T16:47:43.622893+0000","last_clean":"2023-11-16T16:47:43.622893+0000","last_became_active":"2023-11-16T16:47:18.179498+0000","last_became_peered":"2023-11-16T16:47:18.179498+0000","last_unstale":"2023-11-16T16:47:43.622893+0000","last_undegraded":"2023-11-16T16:47:43.622893+0000","last_fullsized":"2023-11-16T16:47:43.622893+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:29:41.782619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622702+0000","last_change":"2023-11-16T16:47:22.656312+0000","last_active":"2023-11-16T16:47:43.622702+0000","last_peered":"2023-11-16T16:47:43.622702+0000","last_clean":"2023-11-16T16:47:43.622702+0000","last_became_active":"2023-11-16T16:47:22.655870+0000","last_became_peered":"2023-11-16T16:47:22.655870+0000","last_unstale":"2023-11-16T16:47:43.622702+0000","last_undegraded":"2023-11-16T16:47:43.622702+0000","last_fullsized":"2023-11-16T16:47:43.622702+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:42:39.568364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629185+0000","last_change":"2023-11-16T16:47:17.622648+0000","last_active":"2023-11-16T16:47:43.629185+0000","last_peered":"2023-11-16T16:47:43.629185+0000","last_clean":"2023-11-16T16:47:43.629185+0000","last_became_active":"2023-11-16T16:47:17.622329+0000","last_became_peered":"2023-11-16T16:47:17.622329+0000","last_unstale":"2023-11-16T16:47:43.629185+0000","last_undegraded":"2023-11-16T16:47:43.629185+0000","last_fullsized":"2023-11-16T16:47:43.629185+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:01:26.211530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622424+0000","last_change":"2023-11-16T16:47:18.179712+0000","last_active":"2023-11-16T16:47:43.622424+0000","last_peered":"2023-11-16T16:47:43.622424+0000","last_clean":"2023-11-16T16:47:43.622424+0000","last_became_active":"2023-11-16T16:47:18.179392+0000","last_became_peered":"2023-11-16T16:47:18.179392+0000","last_unstale":"2023-11-16T16:47:43.622424+0000","last_undegraded":"2023-11-16T16:47:43.622424+0000","last_fullsized":"2023-11-16T16:47:43.622424+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:56:44.447970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.631211+0000","last_change":"2023-11-16T16:47:22.665088+0000","last_active":"2023-11-16T16:47:43.631211+0000","last_peered":"2023-11-16T16:47:43.631211+0000","last_clean":"2023-11-16T16:47:43.631211+0000","last_became_active":"2023-11-16T16:47:22.662802+0000","last_became_peered":"2023-11-16T16:47:22.662802+0000","last_unstale":"2023-11-16T16:47:43.631211+0000","last_undegraded":"2023-11-16T16:47:43.631211+0000","last_fullsized":"2023-11-16T16:47:43.631211+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:51:55.859193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"82'8","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.624624+0000","last_change":"2023-11-16T16:47:17.639246+0000","last_active":"2023-11-16T16:47:43.624624+0000","last_peered":"2023-11-16T16:47:43.624624+0000","last_clean":"2023-11-16T16:47:43.624624+0000","last_became_active":"2023-11-16T16:47:17.638928+0000","last_became_peered":"2023-11-16T16:47:17.638928+0000","last_unstale":"2023-11-16T16:47:43.624624+0000","last_undegraded":"2023-11-16T16:47:43.624624+0000","last_fullsized":"2023-11-16T16:47:43.624624+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:18:41.175512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629991+0000","last_change":"2023-11-16T16:47:18.182609+0000","last_active":"2023-11-16T16:47:43.629991+0000","last_peered":"2023-11-16T16:47:43.629991+0000","last_clean":"2023-11-16T16:47:43.629991+0000","last_became_active":"2023-11-16T16:47:18.182260+0000","last_became_peered":"2023-11-16T16:47:18.182260+0000","last_unstale":"2023-11-16T16:47:43.629991+0000","last_undegraded":"2023-11-16T16:47:43.629991+0000","last_fullsized":"2023-11-16T16:47:43.629991+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:14:14.972367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629794+0000","last_change":"2023-11-16T16:47:22.647586+0000","last_active":"2023-11-16T16:47:43.629794+0000","last_peered":"2023-11-16T16:47:43.629794+0000","last_clean":"2023-11-16T16:47:43.629794+0000","last_became_active":"2023-11-16T16:47:22.647187+0000","last_became_peered":"2023-11-16T16:47:22.647187+0000","last_unstale":"2023-11-16T16:47:43.629794+0000","last_undegraded":"2023-11-16T16:47:43.629794+0000","last_fullsized":"2023-11-16T16:47:43.629794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:01:47.702590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.595597+0000","last_change":"2023-11-16T16:47:43.933501+0000","last_active":"2023-11-16T16:47:44.595597+0000","last_peered":"2023-11-16T16:47:44.595597+0000","last_clean":"2023-11-16T16:47:44.595597+0000","last_became_active":"2023-11-16T16:47:43.631746+0000","last_became_peered":"2023-11-16T16:47:43.631746+0000","last_unstale":"2023-11-16T16:47:44.595597+0000","last_undegraded":"2023-11-16T16:47:44.595597+0000","last_fullsized":"2023-11-16T16:47:44.595597+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:11:10.737388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625043+0000","last_change":"2023-11-16T16:47:18.175210+0000","last_active":"2023-11-16T16:47:43.625043+0000","last_peered":"2023-11-16T16:47:43.625043+0000","last_clean":"2023-11-16T16:47:43.625043+0000","last_became_active":"2023-11-16T16:47:18.174926+0000","last_became_peered":"2023-11-16T16:47:18.174926+0000","last_unstale":"2023-11-16T16:47:43.625043+0000","last_undegraded":"2023-11-16T16:47:43.625043+0000","last_fullsized":"2023-11-16T16:47:43.625043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:31:59.526898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.624874+0000","last_change":"2023-11-16T16:47:22.719361+0000","last_active":"2023-11-16T16:47:43.624874+0000","last_peered":"2023-11-16T16:47:43.624874+0000","last_clean":"2023-11-16T16:47:43.624874+0000","last_became_active":"2023-11-16T16:47:22.718936+0000","last_became_peered":"2023-11-16T16:47:22.718936+0000","last_unstale":"2023-11-16T16:47:43.624874+0000","last_undegraded":"2023-11-16T16:47:43.624874+0000","last_fullsized":"2023-11-16T16:47:43.624874+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:18:52.706386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622234+0000","last_change":"2023-11-16T16:47:17.632623+0000","last_active":"2023-11-16T16:47:43.622234+0000","last_peered":"2023-11-16T16:47:43.622234+0000","last_clean":"2023-11-16T16:47:43.622234+0000","last_became_active":"2023-11-16T16:47:17.631323+0000","last_became_peered":"2023-11-16T16:47:17.631323+0000","last_unstale":"2023-11-16T16:47:43.622234+0000","last_undegraded":"2023-11-16T16:47:43.622234+0000","last_fullsized":"2023-11-16T16:47:43.622234+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:39:00.464312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626526+0000","last_change":"2023-11-16T16:47:18.190779+0000","last_active":"2023-11-16T16:47:43.626526+0000","last_peered":"2023-11-16T16:47:43.626526+0000","last_clean":"2023-11-16T16:47:43.626526+0000","last_became_active":"2023-11-16T16:47:18.190440+0000","last_became_peered":"2023-11-16T16:47:18.190440+0000","last_unstale":"2023-11-16T16:47:43.626526+0000","last_undegraded":"2023-11-16T16:47:43.626526+0000","last_fullsized":"2023-11-16T16:47:43.626526+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:41:07.686317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630403+0000","last_change":"2023-11-16T16:47:22.664847+0000","last_active":"2023-11-16T16:47:43.630403+0000","last_peered":"2023-11-16T16:47:43.630403+0000","last_clean":"2023-11-16T16:47:43.630403+0000","last_became_active":"2023-11-16T16:47:22.659382+0000","last_became_peered":"2023-11-16T16:47:22.659382+0000","last_unstale":"2023-11-16T16:47:43.630403+0000","last_undegraded":"2023-11-16T16:47:43.630403+0000","last_fullsized":"2023-11-16T16:47:43.630403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:11:46.749829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622597+0000","last_change":"2023-11-16T16:47:17.630956+0000","last_active":"2023-11-16T16:47:43.622597+0000","last_peered":"2023-11-16T16:47:43.622597+0000","last_clean":"2023-11-16T16:47:43.622597+0000","last_became_active":"2023-11-16T16:47:17.629864+0000","last_became_peered":"2023-11-16T16:47:17.629864+0000","last_unstale":"2023-11-16T16:47:43.622597+0000","last_undegraded":"2023-11-16T16:47:43.622597+0000","last_fullsized":"2023-11-16T16:47:43.622597+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:01:12.432152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626055+0000","last_change":"2023-11-16T16:47:18.183375+0000","last_active":"2023-11-16T16:47:43.626055+0000","last_peered":"2023-11-16T16:47:43.626055+0000","last_clean":"2023-11-16T16:47:43.626055+0000","last_became_active":"2023-11-16T16:47:18.183050+0000","last_became_peered":"2023-11-16T16:47:18.183050+0000","last_unstale":"2023-11-16T16:47:43.626055+0000","last_undegraded":"2023-11-16T16:47:43.626055+0000","last_fullsized":"2023-11-16T16:47:43.626055+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:27:13.855960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630861+0000","last_change":"2023-11-16T16:47:22.719109+0000","last_active":"2023-11-16T16:47:43.630861+0000","last_peered":"2023-11-16T16:47:43.630861+0000","last_clean":"2023-11-16T16:47:43.630861+0000","last_became_active":"2023-11-16T16:47:22.718680+0000","last_became_peered":"2023-11-16T16:47:22.718680+0000","last_unstale":"2023-11-16T16:47:43.630861+0000","last_undegraded":"2023-11-16T16:47:43.630861+0000","last_fullsized":"2023-11-16T16:47:43.630861+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:36:16.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629990+0000","last_change":"2023-11-16T16:47:18.181632+0000","last_active":"2023-11-16T16:47:43.629990+0000","last_peered":"2023-11-16T16:47:43.629990+0000","last_clean":"2023-11-16T16:47:43.629990+0000","last_became_active":"2023-11-16T16:47:18.181323+0000","last_became_peered":"2023-11-16T16:47:18.181323+0000","last_unstale":"2023-11-16T16:47:43.629990+0000","last_undegraded":"2023-11-16T16:47:43.629990+0000","last_fullsized":"2023-11-16T16:47:43.629990+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:29:17.626827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629025+0000","last_change":"2023-11-16T16:47:17.637269+0000","last_active":"2023-11-16T16:47:43.629025+0000","last_peered":"2023-11-16T16:47:43.629025+0000","last_clean":"2023-11-16T16:47:43.629025+0000","last_became_active":"2023-11-16T16:47:17.636881+0000","last_became_peered":"2023-11-16T16:47:17.636881+0000","last_unstale":"2023-11-16T16:47:43.629025+0000","last_undegraded":"2023-11-16T16:47:43.629025+0000","last_fullsized":"2023-11-16T16:47:43.629025+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:43.798391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623215+0000","last_change":"2023-11-16T16:47:22.659528+0000","last_active":"2023-11-16T16:47:43.623215+0000","last_peered":"2023-11-16T16:47:43.623215+0000","last_clean":"2023-11-16T16:47:43.623215+0000","last_became_active":"2023-11-16T16:47:22.659076+0000","last_became_peered":"2023-11-16T16:47:22.659076+0000","last_unstale":"2023-11-16T16:47:43.623215+0000","last_undegraded":"2023-11-16T16:47:43.623215+0000","last_fullsized":"2023-11-16T16:47:43.623215+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:06:05.562183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970448+0000","last_change":"2023-11-16T16:47:18.182102+0000","last_active":"2023-11-16T16:47:43.970448+0000","last_peered":"2023-11-16T16:47:43.970448+0000","last_clean":"2023-11-16T16:47:43.970448+0000","last_became_active":"2023-11-16T16:47:18.181528+0000","last_became_peered":"2023-11-16T16:47:18.181528+0000","last_unstale":"2023-11-16T16:47:43.970448+0000","last_undegraded":"2023-11-16T16:47:43.970448+0000","last_fullsized":"2023-11-16T16:47:43.970448+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:28:03.200017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629674+0000","last_change":"2023-11-16T16:47:17.626655+0000","last_active":"2023-11-16T16:47:43.629674+0000","last_peered":"2023-11-16T16:47:43.629674+0000","last_clean":"2023-11-16T16:47:43.629674+0000","last_became_active":"2023-11-16T16:47:17.626329+0000","last_became_peered":"2023-11-16T16:47:17.626329+0000","last_unstale":"2023-11-16T16:47:43.629674+0000","last_undegraded":"2023-11-16T16:47:43.629674+0000","last_fullsized":"2023-11-16T16:47:43.629674+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:30:26.199727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622227+0000","last_change":"2023-11-16T16:46:11.943398+0000","last_active":"2023-11-16T16:47:43.622227+0000","last_peered":"2023-11-16T16:47:43.622227+0000","last_clean":"2023-11-16T16:47:43.622227+0000","last_became_active":"2023-11-16T16:46:11.942862+0000","last_became_peered":"2023-11-16T16:46:11.942862+0000","last_unstale":"2023-11-16T16:47:43.622227+0000","last_undegraded":"2023-11-16T16:47:43.622227+0000","last_fullsized":"2023-11-16T16:47:43.622227+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:43:03.109797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:43:03.109797+0000","last_clean_scrub_stamp":"2023-11-16T16:43:03.109797+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:14:00.412865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630080+0000","last_change":"2023-11-16T16:47:22.719963+0000","last_active":"2023-11-16T16:47:43.630080+0000","last_peered":"2023-11-16T16:47:43.630080+0000","last_clean":"2023-11-16T16:47:43.630080+0000","last_became_active":"2023-11-16T16:47:22.719506+0000","last_became_peered":"2023-11-16T16:47:22.719506+0000","last_unstale":"2023-11-16T16:47:43.630080+0000","last_undegraded":"2023-11-16T16:47:43.630080+0000","last_fullsized":"2023-11-16T16:47:43.630080+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:27:40.361521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.628810+0000","last_change":"2023-11-16T16:47:18.184630+0000","last_active":"2023-11-16T16:47:43.628810+0000","last_peered":"2023-11-16T16:47:43.628810+0000","last_clean":"2023-11-16T16:47:43.628810+0000","last_became_active":"2023-11-16T16:47:18.181978+0000","last_became_peered":"2023-11-16T16:47:18.181978+0000","last_unstale":"2023-11-16T16:47:43.628810+0000","last_undegraded":"2023-11-16T16:47:43.628810+0000","last_fullsized":"2023-11-16T16:47:43.628810+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:22:39.294919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.635963+0000","last_change":"2023-11-16T16:47:43.635365+0000","last_active":"2023-11-16T16:47:43.635963+0000","last_peered":"2023-11-16T16:47:43.635963+0000","last_clean":"2023-11-16T16:47:43.635963+0000","last_became_active":"2023-11-16T16:47:43.634974+0000","last_became_peered":"2023-11-16T16:47:43.634974+0000","last_unstale":"2023-11-16T16:47:43.635963+0000","last_undegraded":"2023-11-16T16:47:43.635963+0000","last_fullsized":"2023-11-16T16:47:43.635963+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:40:21.789398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625072+0000","last_change":"2023-11-16T16:47:22.657133+0000","last_active":"2023-11-16T16:47:43.625072+0000","last_peered":"2023-11-16T16:47:43.625072+0000","last_clean":"2023-11-16T16:47:43.625072+0000","last_became_active":"2023-11-16T16:47:22.656557+0000","last_became_peered":"2023-11-16T16:47:22.656557+0000","last_unstale":"2023-11-16T16:47:43.625072+0000","last_undegraded":"2023-11-16T16:47:43.625072+0000","last_fullsized":"2023-11-16T16:47:43.625072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:06:56.883403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626111+0000","last_change":"2023-11-16T16:47:17.639107+0000","last_active":"2023-11-16T16:47:43.626111+0000","last_peered":"2023-11-16T16:47:43.626111+0000","last_clean":"2023-11-16T16:47:43.626111+0000","last_became_active":"2023-11-16T16:47:17.638796+0000","last_became_peered":"2023-11-16T16:47:17.638796+0000","last_unstale":"2023-11-16T16:47:43.626111+0000","last_undegraded":"2023-11-16T16:47:43.626111+0000","last_fullsized":"2023-11-16T16:47:43.626111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:15:27.137682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629965+0000","last_change":"2023-11-16T16:47:18.186761+0000","last_active":"2023-11-16T16:47:43.629965+0000","last_peered":"2023-11-16T16:47:43.629965+0000","last_clean":"2023-11-16T16:47:43.629965+0000","last_became_active":"2023-11-16T16:47:18.185594+0000","last_became_peered":"2023-11-16T16:47:18.185594+0000","last_unstale":"2023-11-16T16:47:43.629965+0000","last_undegraded":"2023-11-16T16:47:43.629965+0000","last_fullsized":"2023-11-16T16:47:43.629965+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:30:05.305376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970230+0000","last_change":"2023-11-16T16:47:22.719534+0000","last_active":"2023-11-16T16:47:43.970230+0000","last_peered":"2023-11-16T16:47:43.970230+0000","last_clean":"2023-11-16T16:47:43.970230+0000","last_became_active":"2023-11-16T16:47:22.718738+0000","last_became_peered":"2023-11-16T16:47:22.718738+0000","last_unstale":"2023-11-16T16:47:43.970230+0000","last_undegraded":"2023-11-16T16:47:43.970230+0000","last_fullsized":"2023-11-16T16:47:43.970230+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:14:30.912366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'4","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:57.089897+0000","last_change":"2023-11-16T16:47:17.630318+0000","last_active":"2023-11-16T16:47:57.089897+0000","last_peered":"2023-11-16T16:47:57.089897+0000","last_clean":"2023-11-16T16:47:57.089897+0000","last_became_active":"2023-11-16T16:47:17.629035+0000","last_became_peered":"2023-11-16T16:47:17.629035+0000","last_unstale":"2023-11-16T16:47:57.089897+0000","last_undegraded":"2023-11-16T16:47:57.089897+0000","last_fullsized":"2023-11-16T16:47:57.089897+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:44:38.576547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970772+0000","last_change":"2023-11-16T16:47:18.185034+0000","last_active":"2023-11-16T16:47:43.970772+0000","last_peered":"2023-11-16T16:47:43.970772+0000","last_clean":"2023-11-16T16:47:43.970772+0000","last_became_active":"2023-11-16T16:47:18.181961+0000","last_became_peered":"2023-11-16T16:47:18.181961+0000","last_unstale":"2023-11-16T16:47:43.970772+0000","last_undegraded":"2023-11-16T16:47:43.970772+0000","last_fullsized":"2023-11-16T16:47:43.970772+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:38:55.393578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622320+0000","last_change":"2023-11-16T16:47:22.649952+0000","last_active":"2023-11-16T16:47:43.622320+0000","last_peered":"2023-11-16T16:47:43.622320+0000","last_clean":"2023-11-16T16:47:43.622320+0000","last_became_active":"2023-11-16T16:47:22.649481+0000","last_became_peered":"2023-11-16T16:47:22.649481+0000","last_unstale":"2023-11-16T16:47:43.622320+0000","last_undegraded":"2023-11-16T16:47:43.622320+0000","last_fullsized":"2023-11-16T16:47:43.622320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:36:07.109965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"82'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629922+0000","last_change":"2023-11-16T16:47:17.631982+0000","last_active":"2023-11-16T16:47:43.629922+0000","last_peered":"2023-11-16T16:47:43.629922+0000","last_clean":"2023-11-16T16:47:43.629922+0000","last_became_active":"2023-11-16T16:47:17.631552+0000","last_became_peered":"2023-11-16T16:47:17.631552+0000","last_unstale":"2023-11-16T16:47:43.629922+0000","last_undegraded":"2023-11-16T16:47:43.629922+0000","last_fullsized":"2023-11-16T16:47:43.629922+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:35:56.360671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626666+0000","last_change":"2023-11-16T16:47:18.197325+0000","last_active":"2023-11-16T16:47:43.626666+0000","last_peered":"2023-11-16T16:47:43.626666+0000","last_clean":"2023-11-16T16:47:43.626666+0000","last_became_active":"2023-11-16T16:47:18.190602+0000","last_became_peered":"2023-11-16T16:47:18.190602+0000","last_unstale":"2023-11-16T16:47:43.626666+0000","last_undegraded":"2023-11-16T16:47:43.626666+0000","last_fullsized":"2023-11-16T16:47:43.626666+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:28:40.186656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629554+0000","last_change":"2023-11-16T16:47:22.720023+0000","last_active":"2023-11-16T16:47:43.629554+0000","last_peered":"2023-11-16T16:47:43.629554+0000","last_clean":"2023-11-16T16:47:43.629554+0000","last_became_active":"2023-11-16T16:47:22.719561+0000","last_became_peered":"2023-11-16T16:47:22.719561+0000","last_unstale":"2023-11-16T16:47:43.629554+0000","last_undegraded":"2023-11-16T16:47:43.629554+0000","last_fullsized":"2023-11-16T16:47:43.629554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:40:41.808974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.946763+0000","last_change":"2023-11-16T16:47:17.629379+0000","last_active":"2023-11-16T16:47:43.946763+0000","last_peered":"2023-11-16T16:47:43.946763+0000","last_clean":"2023-11-16T16:47:43.946763+0000","last_became_active":"2023-11-16T16:47:17.629025+0000","last_became_peered":"2023-11-16T16:47:17.629025+0000","last_unstale":"2023-11-16T16:47:43.946763+0000","last_undegraded":"2023-11-16T16:47:43.946763+0000","last_fullsized":"2023-11-16T16:47:43.946763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:23:29.948397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626265+0000","last_change":"2023-11-16T16:47:18.185418+0000","last_active":"2023-11-16T16:47:43.626265+0000","last_peered":"2023-11-16T16:47:43.626265+0000","last_clean":"2023-11-16T16:47:43.626265+0000","last_became_active":"2023-11-16T16:47:18.185092+0000","last_became_peered":"2023-11-16T16:47:18.185092+0000","last_unstale":"2023-11-16T16:47:43.626265+0000","last_undegraded":"2023-11-16T16:47:43.626265+0000","last_fullsized":"2023-11-16T16:47:43.626265+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:02:15.891150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622811+0000","last_change":"2023-11-16T16:47:22.650701+0000","last_active":"2023-11-16T16:47:43.622811+0000","last_peered":"2023-11-16T16:47:43.622811+0000","last_clean":"2023-11-16T16:47:43.622811+0000","last_became_active":"2023-11-16T16:47:22.648710+0000","last_became_peered":"2023-11-16T16:47:22.648710+0000","last_unstale":"2023-11-16T16:47:43.622811+0000","last_undegraded":"2023-11-16T16:47:43.622811+0000","last_fullsized":"2023-11-16T16:47:43.622811+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:50:37.989145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970985+0000","last_change":"2023-11-16T16:47:17.638622+0000","last_active":"2023-11-16T16:47:43.970985+0000","last_peered":"2023-11-16T16:47:43.970985+0000","last_clean":"2023-11-16T16:47:43.970985+0000","last_became_active":"2023-11-16T16:47:17.638303+0000","last_became_peered":"2023-11-16T16:47:17.638303+0000","last_unstale":"2023-11-16T16:47:43.970985+0000","last_undegraded":"2023-11-16T16:47:43.970985+0000","last_fullsized":"2023-11-16T16:47:43.970985+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:03:52.117172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626598+0000","last_change":"2023-11-16T16:47:18.173474+0000","last_active":"2023-11-16T16:47:43.626598+0000","last_peered":"2023-11-16T16:47:43.626598+0000","last_clean":"2023-11-16T16:47:43.626598+0000","last_became_active":"2023-11-16T16:47:18.173140+0000","last_became_peered":"2023-11-16T16:47:18.173140+0000","last_unstale":"2023-11-16T16:47:43.626598+0000","last_undegraded":"2023-11-16T16:47:43.626598+0000","last_fullsized":"2023-11-16T16:47:43.626598+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:13:23.724052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629397+0000","last_change":"2023-11-16T16:47:22.655806+0000","last_active":"2023-11-16T16:47:43.629397+0000","last_peered":"2023-11-16T16:47:43.629397+0000","last_clean":"2023-11-16T16:47:43.629397+0000","last_became_active":"2023-11-16T16:47:22.655385+0000","last_became_peered":"2023-11-16T16:47:22.655385+0000","last_unstale":"2023-11-16T16:47:43.629397+0000","last_undegraded":"2023-11-16T16:47:43.629397+0000","last_fullsized":"2023-11-16T16:47:43.629397+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:31:41.160819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622449+0000","last_change":"2023-11-16T16:47:17.631673+0000","last_active":"2023-11-16T16:47:43.622449+0000","last_peered":"2023-11-16T16:47:43.622449+0000","last_clean":"2023-11-16T16:47:43.622449+0000","last_became_active":"2023-11-16T16:47:17.631356+0000","last_became_peered":"2023-11-16T16:47:17.631356+0000","last_unstale":"2023-11-16T16:47:43.622449+0000","last_undegraded":"2023-11-16T16:47:43.622449+0000","last_fullsized":"2023-11-16T16:47:43.622449+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:29:12.540977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970538+0000","last_change":"2023-11-16T16:47:18.186706+0000","last_active":"2023-11-16T16:47:43.970538+0000","last_peered":"2023-11-16T16:47:43.970538+0000","last_clean":"2023-11-16T16:47:43.970538+0000","last_became_active":"2023-11-16T16:47:18.186360+0000","last_became_peered":"2023-11-16T16:47:18.186360+0000","last_unstale":"2023-11-16T16:47:43.970538+0000","last_undegraded":"2023-11-16T16:47:43.970538+0000","last_fullsized":"2023-11-16T16:47:43.970538+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:32:31.165136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629693+0000","last_change":"2023-11-16T16:47:22.719065+0000","last_active":"2023-11-16T16:47:43.629693+0000","last_peered":"2023-11-16T16:47:43.629693+0000","last_clean":"2023-11-16T16:47:43.629693+0000","last_became_active":"2023-11-16T16:47:22.718662+0000","last_became_peered":"2023-11-16T16:47:22.718662+0000","last_unstale":"2023-11-16T16:47:43.629693+0000","last_undegraded":"2023-11-16T16:47:43.629693+0000","last_fullsized":"2023-11-16T16:47:43.629693+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:56:10.235168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"82'2","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.638200+0000","last_change":"2023-11-16T16:47:43.637790+0000","last_active":"2023-11-16T16:47:43.638200+0000","last_peered":"2023-11-16T16:47:43.638200+0000","last_clean":"2023-11-16T16:47:43.638200+0000","last_became_active":"2023-11-16T16:47:43.632225+0000","last_became_peered":"2023-11-16T16:47:43.632225+0000","last_unstale":"2023-11-16T16:47:43.638200+0000","last_undegraded":"2023-11-16T16:47:43.638200+0000","last_fullsized":"2023-11-16T16:47:43.638200+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:31:01.698264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622703+0000","last_change":"2023-11-16T16:47:18.180999+0000","last_active":"2023-11-16T16:47:43.622703+0000","last_peered":"2023-11-16T16:47:43.622703+0000","last_clean":"2023-11-16T16:47:43.622703+0000","last_became_active":"2023-11-16T16:47:18.180724+0000","last_became_peered":"2023-11-16T16:47:18.180724+0000","last_unstale":"2023-11-16T16:47:43.622703+0000","last_undegraded":"2023-11-16T16:47:43.622703+0000","last_fullsized":"2023-11-16T16:47:43.622703+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:38:44.147293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947186+0000","last_change":"2023-11-16T16:47:22.665607+0000","last_active":"2023-11-16T16:47:43.947186+0000","last_peered":"2023-11-16T16:47:43.947186+0000","last_clean":"2023-11-16T16:47:43.947186+0000","last_became_active":"2023-11-16T16:47:22.665273+0000","last_became_peered":"2023-11-16T16:47:22.665273+0000","last_unstale":"2023-11-16T16:47:43.947186+0000","last_undegraded":"2023-11-16T16:47:43.947186+0000","last_fullsized":"2023-11-16T16:47:43.947186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:08:37.804057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622937+0000","last_change":"2023-11-16T16:47:17.632614+0000","last_active":"2023-11-16T16:47:43.622937+0000","last_peered":"2023-11-16T16:47:43.622937+0000","last_clean":"2023-11-16T16:47:43.622937+0000","last_became_active":"2023-11-16T16:47:17.632227+0000","last_became_peered":"2023-11-16T16:47:17.632227+0000","last_unstale":"2023-11-16T16:47:43.622937+0000","last_undegraded":"2023-11-16T16:47:43.622937+0000","last_fullsized":"2023-11-16T16:47:43.622937+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:58:13.465619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623388+0000","last_change":"2023-11-16T16:47:18.177607+0000","last_active":"2023-11-16T16:47:43.623388+0000","last_peered":"2023-11-16T16:47:43.623388+0000","last_clean":"2023-11-16T16:47:43.623388+0000","last_became_active":"2023-11-16T16:47:18.177266+0000","last_became_peered":"2023-11-16T16:47:18.177266+0000","last_unstale":"2023-11-16T16:47:43.623388+0000","last_undegraded":"2023-11-16T16:47:43.623388+0000","last_fullsized":"2023-11-16T16:47:43.623388+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:03:08.676930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625888+0000","last_change":"2023-11-16T16:47:22.659987+0000","last_active":"2023-11-16T16:47:43.625888+0000","last_peered":"2023-11-16T16:47:43.625888+0000","last_clean":"2023-11-16T16:47:43.625888+0000","last_became_active":"2023-11-16T16:47:22.658725+0000","last_became_peered":"2023-11-16T16:47:22.658725+0000","last_unstale":"2023-11-16T16:47:43.625888+0000","last_undegraded":"2023-11-16T16:47:43.625888+0000","last_fullsized":"2023-11-16T16:47:43.625888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:51:27.380649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630637+0000","last_change":"2023-11-16T16:47:17.640696+0000","last_active":"2023-11-16T16:47:43.630637+0000","last_peered":"2023-11-16T16:47:43.630637+0000","last_clean":"2023-11-16T16:47:43.630637+0000","last_became_active":"2023-11-16T16:47:17.640428+0000","last_became_peered":"2023-11-16T16:47:17.640428+0000","last_unstale":"2023-11-16T16:47:43.630637+0000","last_undegraded":"2023-11-16T16:47:43.630637+0000","last_fullsized":"2023-11-16T16:47:43.630637+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:47:12.331771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.972273+0000","last_change":"2023-11-16T16:47:43.972273+0000","last_active":"2023-11-16T16:47:43.972273+0000","last_peered":"2023-11-16T16:47:43.972273+0000","last_clean":"2023-11-16T16:47:43.972273+0000","last_became_active":"2023-11-16T16:47:43.971981+0000","last_became_peered":"2023-11-16T16:47:43.971981+0000","last_unstale":"2023-11-16T16:47:43.972273+0000","last_undegraded":"2023-11-16T16:47:43.972273+0000","last_fullsized":"2023-11-16T16:47:43.972273+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:38:01.080979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629465+0000","last_change":"2023-11-16T16:47:22.649436+0000","last_active":"2023-11-16T16:47:43.629465+0000","last_peered":"2023-11-16T16:47:43.629465+0000","last_clean":"2023-11-16T16:47:43.629465+0000","last_became_active":"2023-11-16T16:47:22.649034+0000","last_became_peered":"2023-11-16T16:47:22.649034+0000","last_unstale":"2023-11-16T16:47:43.629465+0000","last_undegraded":"2023-11-16T16:47:43.629465+0000","last_fullsized":"2023-11-16T16:47:43.629465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:50:06.438075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.631007+0000","last_change":"2023-11-16T16:47:17.634582+0000","last_active":"2023-11-16T16:47:43.631007+0000","last_peered":"2023-11-16T16:47:43.631007+0000","last_clean":"2023-11-16T16:47:43.631007+0000","last_became_active":"2023-11-16T16:47:17.634172+0000","last_became_peered":"2023-11-16T16:47:17.634172+0000","last_unstale":"2023-11-16T16:47:43.631007+0000","last_undegraded":"2023-11-16T16:47:43.631007+0000","last_fullsized":"2023-11-16T16:47:43.631007+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:50:55.582584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623419+0000","last_change":"2023-11-16T16:47:18.180466+0000","last_active":"2023-11-16T16:47:43.623419+0000","last_peered":"2023-11-16T16:47:43.623419+0000","last_clean":"2023-11-16T16:47:43.623419+0000","last_became_active":"2023-11-16T16:47:18.180166+0000","last_became_peered":"2023-11-16T16:47:18.180166+0000","last_unstale":"2023-11-16T16:47:43.623419+0000","last_undegraded":"2023-11-16T16:47:43.623419+0000","last_fullsized":"2023-11-16T16:47:43.623419+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:02:56.576134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626739+0000","last_change":"2023-11-16T16:47:18.179196+0000","last_active":"2023-11-16T16:47:43.626739+0000","last_peered":"2023-11-16T16:47:43.626739+0000","last_clean":"2023-11-16T16:47:43.626739+0000","last_became_active":"2023-11-16T16:47:18.178827+0000","last_became_peered":"2023-11-16T16:47:18.178827+0000","last_unstale":"2023-11-16T16:47:43.626739+0000","last_undegraded":"2023-11-16T16:47:43.626739+0000","last_fullsized":"2023-11-16T16:47:43.626739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:07:11.423464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622851+0000","last_change":"2023-11-16T16:47:18.184630+0000","last_active":"2023-11-16T16:47:43.622851+0000","last_peered":"2023-11-16T16:47:43.622851+0000","last_clean":"2023-11-16T16:47:43.622851+0000","last_became_active":"2023-11-16T16:47:18.184314+0000","last_became_peered":"2023-11-16T16:47:18.184314+0000","last_unstale":"2023-11-16T16:47:43.622851+0000","last_undegraded":"2023-11-16T16:47:43.622851+0000","last_fullsized":"2023-11-16T16:47:43.622851+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:56:13.651803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622558+0000","last_change":"2023-11-16T16:47:18.184255+0000","last_active":"2023-11-16T16:47:43.622558+0000","last_peered":"2023-11-16T16:47:43.622558+0000","last_clean":"2023-11-16T16:47:43.622558+0000","last_became_active":"2023-11-16T16:47:18.183936+0000","last_became_peered":"2023-11-16T16:47:18.183936+0000","last_unstale":"2023-11-16T16:47:43.622558+0000","last_undegraded":"2023-11-16T16:47:43.622558+0000","last_fullsized":"2023-11-16T16:47:43.622558+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:59:41.136565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626136+0000","last_change":"2023-11-16T16:47:18.196331+0000","last_active":"2023-11-16T16:47:43.626136+0000","last_peered":"2023-11-16T16:47:43.626136+0000","last_clean":"2023-11-16T16:47:43.626136+0000","last_became_active":"2023-11-16T16:47:18.195954+0000","last_became_peered":"2023-11-16T16:47:18.195954+0000","last_unstale":"2023-11-16T16:47:43.626136+0000","last_undegraded":"2023-11-16T16:47:43.626136+0000","last_fullsized":"2023-11-16T16:47:43.626136+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:31:23.208122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629719+0000","last_change":"2023-11-16T16:47:18.190537+0000","last_active":"2023-11-16T16:47:43.629719+0000","last_peered":"2023-11-16T16:47:43.629719+0000","last_clean":"2023-11-16T16:47:43.629719+0000","last_became_active":"2023-11-16T16:47:18.190118+0000","last_became_peered":"2023-11-16T16:47:18.190118+0000","last_unstale":"2023-11-16T16:47:43.629719+0000","last_undegraded":"2023-11-16T16:47:43.629719+0000","last_fullsized":"2023-11-16T16:47:43.629719+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:39:38.125056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625721+0000","last_change":"2023-11-16T16:47:18.179215+0000","last_active":"2023-11-16T16:47:43.625721+0000","last_peered":"2023-11-16T16:47:43.625721+0000","last_clean":"2023-11-16T16:47:43.625721+0000","last_became_active":"2023-11-16T16:47:18.178759+0000","last_became_peered":"2023-11-16T16:47:18.178759+0000","last_unstale":"2023-11-16T16:47:43.625721+0000","last_undegraded":"2023-11-16T16:47:43.625721+0000","last_fullsized":"2023-11-16T16:47:43.625721+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:06:54.886510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623599+0000","last_change":"2023-11-16T16:47:18.179588+0000","last_active":"2023-11-16T16:47:43.623599+0000","last_peered":"2023-11-16T16:47:43.623599+0000","last_clean":"2023-11-16T16:47:43.623599+0000","last_became_active":"2023-11-16T16:47:18.179294+0000","last_became_peered":"2023-11-16T16:47:18.179294+0000","last_unstale":"2023-11-16T16:47:43.623599+0000","last_undegraded":"2023-11-16T16:47:43.623599+0000","last_fullsized":"2023-11-16T16:47:43.623599+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:02:55.079560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.971128+0000","last_change":"2023-11-16T16:47:18.181587+0000","last_active":"2023-11-16T16:47:43.971128+0000","last_peered":"2023-11-16T16:47:43.971128+0000","last_clean":"2023-11-16T16:47:43.971128+0000","last_became_active":"2023-11-16T16:47:18.181203+0000","last_became_peered":"2023-11-16T16:47:18.181203+0000","last_unstale":"2023-11-16T16:47:43.971128+0000","last_undegraded":"2023-11-16T16:47:43.971128+0000","last_fullsized":"2023-11-16T16:47:43.971128+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:13:49.378115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629816+0000","last_change":"2023-11-16T16:47:18.182690+0000","last_active":"2023-11-16T16:47:43.629816+0000","last_peered":"2023-11-16T16:47:43.629816+0000","last_clean":"2023-11-16T16:47:43.629816+0000","last_became_active":"2023-11-16T16:47:18.182361+0000","last_became_peered":"2023-11-16T16:47:18.182361+0000","last_unstale":"2023-11-16T16:47:43.629816+0000","last_undegraded":"2023-11-16T16:47:43.629816+0000","last_fullsized":"2023-11-16T16:47:43.629816+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:55:41.987600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970993+0000","last_change":"2023-11-16T16:47:18.184599+0000","last_active":"2023-11-16T16:47:43.970993+0000","last_peered":"2023-11-16T16:47:43.970993+0000","last_clean":"2023-11-16T16:47:43.970993+0000","last_became_active":"2023-11-16T16:47:18.184288+0000","last_became_peered":"2023-11-16T16:47:18.184288+0000","last_unstale":"2023-11-16T16:47:43.970993+0000","last_undegraded":"2023-11-16T16:47:43.970993+0000","last_fullsized":"2023-11-16T16:47:43.970993+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:43:20.190571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947382+0000","last_change":"2023-11-16T16:47:18.187267+0000","last_active":"2023-11-16T16:47:43.947382+0000","last_peered":"2023-11-16T16:47:43.947382+0000","last_clean":"2023-11-16T16:47:43.947382+0000","last_became_active":"2023-11-16T16:47:18.186984+0000","last_became_peered":"2023-11-16T16:47:18.186984+0000","last_unstale":"2023-11-16T16:47:43.947382+0000","last_undegraded":"2023-11-16T16:47:43.947382+0000","last_fullsized":"2023-11-16T16:47:43.947382+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:30:20.891991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630154+0000","last_change":"2023-11-16T16:47:18.191566+0000","last_active":"2023-11-16T16:47:43.630154+0000","last_peered":"2023-11-16T16:47:43.630154+0000","last_clean":"2023-11-16T16:47:43.630154+0000","last_became_active":"2023-11-16T16:47:18.185342+0000","last_became_peered":"2023-11-16T16:47:18.185342+0000","last_unstale":"2023-11-16T16:47:43.630154+0000","last_undegraded":"2023-11-16T16:47:43.630154+0000","last_fullsized":"2023-11-16T16:47:43.630154+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:55:29.393440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629636+0000","last_change":"2023-11-16T16:47:43.629636+0000","last_active":"2023-11-16T16:47:43.629636+0000","last_peered":"2023-11-16T16:47:43.629636+0000","last_clean":"2023-11-16T16:47:43.629636+0000","last_became_active":"2023-11-16T16:47:43.629179+0000","last_became_peered":"2023-11-16T16:47:43.629179+0000","last_unstale":"2023-11-16T16:47:43.629636+0000","last_undegraded":"2023-11-16T16:47:43.629636+0000","last_fullsized":"2023-11-16T16:47:43.629636+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:21:10.581239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281399,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61706,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":106,"num_read_kb":117,"num_write":76,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":185118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449117,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6140,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739012,"statfs":{"total":95995035648,"available":95988748288,"internally_reserved":0,"allocated":970752,"data_stored":487258,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51000000000000001,"15min":0.51400000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.55700000000000005,"15min":0.55100000000000005},"min":{"1min":0.38800000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.79000000000000004}]},{"osd":1,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58899999999999997,"15min":0.58599999999999997},"min":{"1min":0.40600000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81000000000000005,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.52800000000000002}]},{"osd":2,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45700000000000002,"15min":0.45500000000000002},"min":{"1min":0.313,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.46400000000000002,"15min":0.46500000000000002},"min":{"1min":0.34899999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.51000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.63,"15min":0.626},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.621,"15min":0.625},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.77000000000000002,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.72799999999999998}]},{"osd":5,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.33500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.77100000000000002,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.59599999999999997,"15min":0.59799999999999998},"min":{"1min":0.378,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.87,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.75600000000000001}]},{"osd":6,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.67800000000000005,"15min":0.68999999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.89900000000000002,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.75700000000000001,"15min":0.72999999999999998},"min":{"1min":0.496,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":4.2510000000000003,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":0.745}]},{"osd":7,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70999999999999996,"15min":0.71099999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.90800000000000003,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.72699999999999998,"15min":0.73399999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.86199999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.66700000000000004}]},{"osd":8,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.80900000000000005}]},{"osd":9,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.76800000000000002,"15min":0.76300000000000001},"min":{"1min":0.69199999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":0.92200000000000004,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.77100000000000002,"15min":0.77400000000000002},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":0.98199999999999998,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.92000000000000004}]},{"osd":10,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76200000000000001,"15min":0.76700000000000002},"min":{"1min":0.58099999999999996,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.93500000000000005,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.59899999999999998,"5min":0.498,"15min":0.498},"max":{"1min":0.90400000000000003,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.85199999999999998}]}]},{"osd":10,"up_from":71,"seq":304942678049,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6516,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738636,"statfs":{"total":95995035648,"available":95988363264,"internally_reserved":0,"allocated":1093632,"data_stored":1134610,"data_compressed":53424,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.72799999999999998}]},{"osd":1,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.47099999999999997,"15min":0.48099999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.502,"15min":0.51400000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.622,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.39000000000000001}]},{"osd":2,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65600000000000003,"15min":0.66800000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.90500000000000003,"5min":1.577,"15min":1.577},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65400000000000003,"15min":0.66300000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.83199999999999996,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.54000000000000004}]},{"osd":3,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64500000000000002,"15min":0.65200000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.67000000000000004,"15min":0.68000000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.84599999999999997,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.55200000000000005}]},{"osd":4,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.58199999999999996,"15min":0.58999999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.505},{"interface":"front","average":{"1min":0.496,"5min":0.56200000000000006,"15min":0.57299999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.46500000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.69699999999999995,"15min":0.70399999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.90600000000000003,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.91400000000000003,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.75800000000000001,"15min":0.76700000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77600000000000002,"15min":0.77900000000000003},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.68899999999999995}]},{"osd":7,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.69999999999999996}]},{"osd":8,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.84399999999999997,"15min":0.86499999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.052,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.77800000000000002,"15min":0.79100000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.93200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.625}]},{"osd":9,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.86099999999999999,"15min":0.84699999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":4.0129999999999999,"5min":4.0129999999999999,"15min":4.0129999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.80600000000000005,"15min":0.82599999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.86799999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.67300000000000004}]},{"osd":11,"last update":"Thu Nov 16 16:48:30 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.80700000000000005,"15min":0.82599999999999996},"min":{"1min":0.49199999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.83699999999999997,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.79900000000000004,"15min":0.80900000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.91200000000000003,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.74399999999999999}]}]},{"osd":9,"up_from":66,"seq":283467841573,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6272,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":974848,"data_stored":491914,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.56200000000000006,"15min":0.57599999999999996},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.68300000000000005,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.59399999999999997,"15min":0.61199999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.75700000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.497}]},{"osd":1,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.82199999999999995,"15min":0.83699999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.98599999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.622},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.77300000000000002,"15min":0.79100000000000004},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.83899999999999997,"5min":1.071,"15min":1.071},"last":0.64400000000000002}]},{"osd":2,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.82699999999999996,"15min":0.84199999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.96999999999999997,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.76800000000000002,"15min":0.78200000000000003},"min":{"1min":0.55300000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.93500000000000005,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.67600000000000005}]},{"osd":3,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.78900000000000003,"15min":0.81499999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94399999999999995,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.57099999999999995}]},{"osd":4,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.83999999999999997,"15min":0.86599999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.89800000000000002,"5min":1.298,"15min":1.298},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.76400000000000001,"15min":0.78200000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.85599999999999998,"5min":1.181,"15min":1.181},"last":0.63400000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.81100000000000005,"15min":0.83099999999999996},"min":{"1min":0.47999999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0860000000000001,"5min":1.268,"15min":1.268},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.93999999999999995,"15min":0.96599999999999997},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.032,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.75}]},{"osd":6,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":1.054},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.95599999999999996}]},{"osd":7,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.89400000000000002,"15min":0.91800000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.89200000000000002,"5min":1.405,"15min":1.405},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.81699999999999995,"5min":1.0409999999999999,"15min":1.079},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0740000000000001,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.65400000000000003}]},{"osd":8,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.94799999999999995,"15min":0.97599999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0269999999999999,"5min":1.363,"15min":1.363},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.95399999999999996,"15min":0.97999999999999998},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96999999999999997,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.79800000000000004}]},{"osd":10,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.92800000000000005,"15min":0.95199999999999996},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.016,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.85799999999999998,"5min":1.0109999999999999,"15min":1.0369999999999999},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.0960000000000001,"5min":1.714,"15min":1.714},"last":0.78100000000000003}]},{"osd":11,"last update":"Thu Nov 16 16:48:22 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.88},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.82499999999999996}]}]},{"osd":8,"up_from":58,"seq":249108103210,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6460,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738692,"statfs":{"total":95995035648,"available":95988420608,"internally_reserved":0,"allocated":970752,"data_stored":487818,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.83999999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.38,"5min":0.313,"15min":0.313},"max":{"1min":0.78600000000000003,"5min":1.113,"15min":1.113},"last":0.63600000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.61699999999999999,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84799999999999998,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61399999999999999,"15min":0.60699999999999998},"min":{"1min":0.40600000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79200000000000004,"5min":1.177,"15min":1.177},"last":0.625}]},{"osd":2,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.34799999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.60999999999999999,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.47199999999999998,"15min":0.46500000000000002},"min":{"1min":0.379,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63500000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.436}]},{"osd":3,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.49299999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.141,"15min":1.141},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.65900000000000003,"15min":0.64900000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.55000000000000004}]},{"osd":4,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.63200000000000001,"15min":0.626},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.91100000000000003,"5min":0.999,"15min":0.999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69299999999999995,"15min":0.68700000000000006},"min":{"1min":0.58199999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.81599999999999995,"5min":1.242,"15min":1.242},"last":0.68899999999999995}]},{"osd":5,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55000000000000004,"15min":0.53400000000000003},"min":{"1min":0.40600000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71799999999999997,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.57399999999999995,"15min":0.56200000000000006},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75700000000000001,"5min":1.224,"15min":1.224},"last":0.59499999999999997}]},{"osd":6,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.77300000000000002,"15min":0.73699999999999999},"min":{"1min":0.501,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":4.7309999999999999,"5min":4.7309999999999999,"15min":4.7309999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.68600000000000005,"15min":0.67800000000000005},"min":{"1min":0.52500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83599999999999997,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.57999999999999996}]},{"osd":7,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.70199999999999996,"15min":0.68999999999999995},"min":{"1min":0.59199999999999997,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88500000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.752,"5min":0.73199999999999998,"15min":0.72399999999999998},"min":{"1min":0.59299999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95299999999999996,"5min":1.198,"15min":1.198},"last":0.74399999999999999}]},{"osd":9,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77200000000000002,"15min":0.77100000000000002},"min":{"1min":0.52200000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92000000000000004,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.81200000000000006,"15min":0.81399999999999995},"min":{"1min":0.55900000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0069999999999999,"5min":1.298,"15min":1.298},"last":0.78800000000000003}]},{"osd":10,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.78000000000000003,"15min":0.77600000000000002},"min":{"1min":0.63700000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.022,"5min":1.222,"15min":1.222},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.82399999999999995,"15min":0.82299999999999995},"min":{"1min":0.65000000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.2350000000000001,"5min":1.274,"15min":1.274},"last":0.75600000000000001}]},{"osd":11,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.77500000000000002}]}]},{"osd":7,"up_from":51,"seq":219043332142,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6576,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738576,"statfs":{"total":95995035648,"available":95988301824,"internally_reserved":0,"allocated":1024000,"data_stored":1082282,"data_compressed":53424,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.377,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81799999999999995,"5min":0.875,"15min":0.875},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.42299999999999999,"15min":0.40899999999999997},"min":{"1min":0.28699999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.60099999999999998,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.44900000000000001,"15min":0.44},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.59799999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.628}]},{"osd":2,"last update":"Thu Nov 16 16:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.59099999999999997,"15min":0.57999999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81200000000000006,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.59699999999999998,"15min":0.58099999999999996},"min":{"1min":0.46500000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78400000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.745}]},{"osd":3,"last update":"Thu Nov 16 16:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57199999999999995,"15min":0.55400000000000005},"min":{"1min":0.39600000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.38800000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.84799999999999998,"5min":0.873,"15min":0.873},"last":0.66700000000000004}]},{"osd":4,"last update":"Thu Nov 16 16:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33700000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67200000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.72899999999999998,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.52500000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.44500000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.84199999999999997,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65000000000000002,"15min":0.63400000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.89400000000000002,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.82899999999999996}]},{"osd":6,"last update":"Thu Nov 16 16:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.68799999999999994,"15min":0.67700000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.90700000000000003,"5min":1.004,"15min":1.004},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66900000000000004,"15min":0.66800000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.85799999999999998,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.81200000000000006}]},{"osd":8,"last update":"Thu Nov 16 16:48:30 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.69699999999999995,"15min":0.67700000000000005},"min":{"1min":0.50600000000000001,"5min":0.436,"15min":0.436},"max":{"1min":0.95699999999999996,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69999999999999996,"15min":0.70199999999999996},"min":{"1min":0.47299999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.878,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.70999999999999996}]},{"osd":9,"last update":"Thu Nov 16 16:47:52 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.77200000000000002,"15min":0.78400000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.92800000000000005,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.80000000000000004,"15min":0.80800000000000005},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.93100000000000005,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.90400000000000003}]},{"osd":10,"last update":"Thu Nov 16 16:48:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.74299999999999999,"15min":0.755},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86599999999999999,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.77900000000000003,"15min":0.78900000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.89200000000000002,"5min":1.01,"15min":1.01},"last":0.86099999999999999}]},{"osd":11,"last update":"Thu Nov 16 16:48:23 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.84699999999999998}]}]},{"osd":6,"up_from":46,"seq":197568495667,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6388,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738764,"statfs":{"total":95995035648,"available":95988494336,"internally_reserved":0,"allocated":962560,"data_stored":483140,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.38800000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.434},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.43099999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84599999999999997,"5min":1.345,"15min":1.345},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64900000000000002,"15min":0.64800000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.98199999999999998,"5min":1.111,"15min":1.111},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69699999999999995,"15min":0.70399999999999996},"min":{"1min":0.505,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0229999999999999,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.71799999999999997,"15min":0.70799999999999996},"min":{"1min":0.54900000000000004,"5min":0.436,"15min":0.436},"max":{"1min":0.998,"5min":1.407,"15min":1.407},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.71899999999999997,"15min":0.72499999999999998},"min":{"1min":0.48799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.98699999999999999,"5min":1.171,"15min":1.171},"last":0.70199999999999996}]},{"osd":3,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.72799999999999998,"15min":0.73999999999999999},"min":{"1min":0.49099999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0189999999999999,"5min":1.022,"15min":1.022},"last":0.61699999999999999}]},{"osd":4,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.746,"15min":0.73199999999999998},"min":{"1min":0.56899999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.755,"15min":0.76000000000000001},"min":{"1min":0.58499999999999996,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.048,"5min":1.083,"15min":1.083},"last":0.65100000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79000000000000004,"15min":0.80000000000000004},"min":{"1min":0.61099999999999999,"5min":0.503,"15min":0.503},"max":{"1min":1.0940000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.88300000000000001,"15min":0.89700000000000002},"min":{"1min":0.747,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.2749999999999999,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.873}]},{"osd":7,"last update":"Thu Nov 16 16:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.81899999999999995,"15min":0.79800000000000004},"min":{"1min":0.64300000000000002,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.1100000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.87,"5min":0.78000000000000003,"15min":0.753},"min":{"1min":0.59399999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0820000000000001,"5min":1.256,"15min":1.256},"last":0.73999999999999999}]},{"osd":8,"last update":"Thu Nov 16 16:48:27 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.84899999999999998,"15min":0.84799999999999998},"min":{"1min":0.66200000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.754},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.90200000000000002,"15min":0.89200000000000002},"min":{"1min":0.65800000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.284,"5min":1.823,"15min":1.823},"last":0.94499999999999995}]},{"osd":9,"last update":"Thu Nov 16 16:47:50 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.96499999999999997,"15min":0.96699999999999997},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.238,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":1.052},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.84699999999999998,"15min":0.83899999999999997},"min":{"1min":0.61899999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.069,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.81899999999999995}]},{"osd":10,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.91500000000000004,"15min":0.92900000000000005},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.097,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.98799999999999999,"15min":0.98599999999999999},"min":{"1min":0.85599999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.206,"5min":1.393,"15min":1.393},"last":0.98399999999999999}]},{"osd":11,"last update":"Thu Nov 16 16:48:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":1.018},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.85199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7480,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737672,"statfs":{"total":95995035648,"available":95987376128,"internally_reserved":0,"allocated":1032192,"data_stored":544048,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55100000000000005,"15min":0.52500000000000002},"min":{"1min":0.39600000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.77500000000000002,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.54000000000000004,"15min":0.51400000000000001},"min":{"1min":0.40500000000000003,"5min":0.315,"15min":0.315},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.57899999999999996}]},{"osd":2,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.55900000000000005,"15min":0.53600000000000003},"min":{"1min":0.41099999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.54700000000000004,"15min":0.51100000000000001},"min":{"1min":0.434,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.86199999999999999,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.59999999999999998}]},{"osd":3,"last update":"Thu Nov 16 16:47:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55000000000000004,"15min":0.53400000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.58899999999999997,"15min":0.57299999999999995},"min":{"1min":0.41199999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.63100000000000001}]},{"osd":4,"last update":"Thu Nov 16 16:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.437,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.45400000000000001,"15min":0.438},"min":{"1min":0.29099999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.38800000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:48:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.60499999999999998,"15min":0.57399999999999995},"min":{"1min":0.50800000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.629,"15min":0.60699999999999998},"min":{"1min":0.51400000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.54400000000000004}]},{"osd":6,"last update":"Thu Nov 16 16:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.42599999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82999999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.65800000000000003,"15min":0.64200000000000002},"min":{"1min":0.48899999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.68000000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:48:07 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55300000000000005,"15min":0.52100000000000002},"min":{"1min":0.36099999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.84099999999999997,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.58399999999999996,"15min":0.55900000000000005},"min":{"1min":0.379,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.93500000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.70699999999999996}]},{"osd":8,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.67600000000000005,"15min":0.64200000000000002},"min":{"1min":0.63100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.97099999999999997,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.68799999999999994,"15min":0.65600000000000003},"min":{"1min":0.55200000000000005,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.755}]},{"osd":9,"last update":"Thu Nov 16 16:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.73699999999999999,"15min":0.72799999999999998},"min":{"1min":0.57199999999999995,"5min":0.5,"15min":0.5},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.755,"15min":0.749},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.74299999999999999}]},{"osd":10,"last update":"Thu Nov 16 16:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.77500000000000002,"15min":0.78000000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.97599999999999998,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78400000000000003,"15min":0.78300000000000003},"min":{"1min":0.54000000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.80900000000000005}]},{"osd":11,"last update":"Thu Nov 16 16:48:20 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.83699999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7288,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737864,"statfs":{"total":95995035648,"available":95987572736,"internally_reserved":0,"allocated":966656,"data_stored":483230,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 16:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57099999999999995,"15min":0.54300000000000004},"min":{"1min":0.375,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.88400000000000001,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.752,"5min":0.627,"15min":0.56899999999999995},"min":{"1min":0.38100000000000001,"5min":0.316,"15min":0.316},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.65600000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.70299999999999996,"15min":0.60199999999999998},"min":{"1min":0.49399999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.90800000000000003,"5min":5.4969999999999999,"15min":5.4969999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.66400000000000003,"15min":0.61599999999999999},"min":{"1min":0.41799999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.88600000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.55300000000000005}]},{"osd":3,"last update":"Thu Nov 16 16:48:21 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.46800000000000003}]},{"osd":4,"last update":"Thu Nov 16 16:48:01 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.66100000000000003,"15min":0.59299999999999997},"min":{"1min":0.441,"5min":0.434,"15min":0.434},"max":{"1min":1.048,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.73699999999999999,"15min":0.78400000000000003},"min":{"1min":0.52000000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93700000000000006,"5min":5.4409999999999998,"15min":5.4409999999999998},"last":0.53700000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:48:21 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.76000000000000001,"15min":0.70899999999999996},"min":{"1min":0.60899999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.1589999999999998,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.74399999999999999,"15min":0.72399999999999998},"min":{"1min":0.59599999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.4420000000000002,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":0.68999999999999995}]},{"osd":6,"last update":"Thu Nov 16 16:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.70299999999999996,"15min":0.68700000000000006},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.98599999999999999,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.69499999999999995,"15min":0.66600000000000004},"min":{"1min":0.42499999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.036,"5min":1.093,"15min":1.093},"last":0.622}]},{"osd":7,"last update":"Thu Nov 16 16:48:26 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.8089999999999999,"5min":1.8089999999999999,"15min":1.8089999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":3.1179999999999999,"5min":3.1179999999999999,"15min":3.1179999999999999},"last":0.63700000000000001}]},{"osd":8,"last update":"Thu Nov 16 16:48:26 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.876,"15min":0.88200000000000001},"min":{"1min":0.59399999999999997,"5min":0.498,"15min":0.498},"max":{"1min":2.891,"5min":2.891,"15min":2.891},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.90800000000000003,"15min":0.91900000000000004},"min":{"1min":0.71399999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":1.8759999999999999},"last":0.80100000000000005}]},{"osd":9,"last update":"Thu Nov 16 16:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.98399999999999999},"min":{"1min":0.64100000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1930000000000001,"5min":1.278,"15min":1.278},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.91000000000000003,"15min":0.92100000000000004},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.117,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.79200000000000004}]},{"osd":10,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.91100000000000003,"15min":0.92200000000000004},"min":{"1min":0.63900000000000001,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.2330000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.88100000000000001,"5min":1.0640000000000001,"15min":1.0940000000000001},"min":{"1min":0.67300000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.1679999999999999,"5min":3.694,"15min":3.694},"last":0.86799999999999999}]},{"osd":11,"last update":"Thu Nov 16 16:48:31 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":0.90900000000000003,"15min":0.88700000000000001},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.1909999999999998,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.91300000000000003,"15min":0.88900000000000001},"min":{"1min":0.65500000000000003,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":3.0489999999999999,"5min":3.0489999999999999,"15min":3.0489999999999999},"last":0.81299999999999994}]}]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7544,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737608,"statfs":{"total":95995035648,"available":95987310592,"internally_reserved":0,"allocated":1032192,"data_stored":543809,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.53400000000000003,"15min":0.46999999999999997},"min":{"1min":0.41599999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52100000000000002,"15min":0.495},"min":{"1min":0.33100000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.76200000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:48:12 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.54000000000000004,"15min":0.49399999999999999},"min":{"1min":0.36099999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.56899999999999995,"15min":0.504},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.495}]},{"osd":3,"last update":"Thu Nov 16 16:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.67100000000000004,"15min":0.71799999999999997},"min":{"1min":0.442,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.95499999999999996,"5min":3.98,"15min":3.98},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.61899999999999999,"15min":0.59299999999999997},"min":{"1min":0.45100000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.83699999999999997}]},{"osd":4,"last update":"Thu Nov 16 16:48:19 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.55100000000000005}]},{"osd":5,"last update":"Thu Nov 16 16:48:22 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.48099999999999998,"15min":0.46300000000000002},"min":{"1min":0.374,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.78500000000000003,"5min":0.871,"15min":0.871},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.50800000000000001,"15min":0.48199999999999998},"min":{"1min":0.40000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70999999999999996,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.66700000000000004}]},{"osd":6,"last update":"Thu Nov 16 16:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.59199999999999997,"15min":0.58199999999999996},"min":{"1min":0.44400000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.81000000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:48:22 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.85499999999999998}]},{"osd":8,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.61499999999999999,"15min":0.60099999999999998},"min":{"1min":0.47799999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.58599999999999997,"15min":0.57599999999999996},"min":{"1min":0.47599999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.79400000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.88500000000000001}]},{"osd":9,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.71899999999999997,"15min":0.71699999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.91300000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.73099999999999998,"15min":0.72899999999999998},"min":{"1min":0.51200000000000001,"5min":0.503,"15min":0.503},"max":{"1min":0.91500000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.95699999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:48:12 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.69599999999999995,"15min":0.68400000000000005},"min":{"1min":0.48799999999999999,"5min":0.441,"15min":0.441},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.70199999999999996,"15min":0.69199999999999995},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.96799999999999997}]},{"osd":11,"last update":"Thu Nov 16 16:48:19 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.93100000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182464,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6716,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738436,"statfs":{"total":95995035648,"available":95988158464,"internally_reserved":0,"allocated":970752,"data_stored":483244,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:35 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.45500000000000002,"15min":0.42599999999999999},"min":{"1min":0.40500000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57099999999999995,"5min":1.085,"15min":1.085},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54500000000000004,"15min":0.49299999999999999},"min":{"1min":0.36799999999999999,"5min":0.313,"15min":0.313},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.53900000000000003}]},{"osd":1,"last update":"Thu Nov 16 16:47:35 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.60499999999999998,"15min":0.57399999999999995},"min":{"1min":0.41899999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.96999999999999997,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.63200000000000001,"15min":0.59499999999999997},"min":{"1min":0.45400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.006,"5min":1.286,"15min":1.286},"last":0.44400000000000001}]},{"osd":2,"last update":"Thu Nov 16 16:47:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.63300000000000001,"15min":0.58699999999999997},"min":{"1min":0.53200000000000003,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.92500000000000004,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.64500000000000002,"15min":0.60099999999999998},"min":{"1min":0.48499999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.71399999999999997}]},{"osd":4,"last update":"Thu Nov 16 16:48:01 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.67600000000000005,"15min":0.61599999999999999},"min":{"1min":0.47299999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.8420000000000001,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.70199999999999996,"15min":0.66800000000000004},"min":{"1min":0.52200000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.95399999999999996,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.80700000000000005}]},{"osd":5,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71899999999999997,"15min":0.69299999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.004,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.76000000000000001,"15min":0.68600000000000005},"min":{"1min":0.49399999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.2330000000000001,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":0.752}]},{"osd":6,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.69399999999999995,"15min":0.65900000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0669999999999999,"5min":1.323,"15min":1.323},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.70599999999999996,"15min":0.68200000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.02,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.41999999999999998}]},{"osd":7,"last update":"Thu Nov 16 16:48:03 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.82799999999999996,"15min":0.79800000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":5.1070000000000002,"5min":5.1070000000000002,"15min":5.1070000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.86899999999999999,"15min":0.85299999999999998},"min":{"1min":0.629,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.60599999999999998}]},{"osd":8,"last update":"Thu Nov 16 16:46:23 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.94199999999999995}]},{"osd":9,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.90400000000000003,"15min":0.91700000000000004},"min":{"1min":0.66700000000000004,"5min":0.501,"15min":0.501},"max":{"1min":1.0900000000000001,"5min":1.395,"15min":1.395},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.97199999999999998,"15min":0.97699999999999998},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.76800000000000002}]},{"osd":10,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.86599999999999999,"15min":0.878},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.115,"5min":1.304,"15min":1.304},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.94999999999999996,"15min":0.94899999999999995},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.673,"5min":2.673,"15min":2.673},"last":0.67600000000000005}]},{"osd":11,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.78000000000000003}]}]},{"osd":4,"up_from":33,"seq":141733920827,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6524,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738628,"statfs":{"total":95995035648,"available":95988355072,"internally_reserved":0,"allocated":970752,"data_stored":487258,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.48899999999999999,"15min":0.46600000000000003},"min":{"1min":0.439,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.51100000000000001,"15min":0.48599999999999999},"min":{"1min":0.47399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.63500000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.36099999999999999,"15min":0.32400000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.54400000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.38200000000000001,"15min":0.34899999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.502}]},{"osd":2,"last update":"Thu Nov 16 16:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.56200000000000006,"15min":0.51500000000000001},"min":{"1min":0.33400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.55200000000000005,"15min":0.50700000000000001},"min":{"1min":0.42399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.70099999999999996}]},{"osd":3,"last update":"Thu Nov 16 16:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.56699999999999995,"15min":0.52700000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.415,"5min":0.72499999999999998,"15min":0.58199999999999996},"min":{"1min":0.42799999999999999,"5min":0.374,"15min":0.374},"max":{"1min":14.795999999999999,"5min":14.795999999999999,"15min":14.795999999999999},"last":0.59299999999999997}]},{"osd":5,"last update":"Thu Nov 16 16:48:17 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.59599999999999997,"15min":0.54700000000000004},"min":{"1min":0.39900000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.58399999999999996,"15min":0.53200000000000003},"min":{"1min":0.441,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81899999999999995,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.66200000000000003}]},{"osd":6,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.61299999999999999,"15min":0.58199999999999996},"min":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.65700000000000003,"15min":0.61899999999999999},"min":{"1min":0.45100000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":0.55100000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:48:10 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.55000000000000004,"15min":0.52400000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59799999999999998,"15min":0.57099999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.78100000000000003}]},{"osd":8,"last update":"Thu Nov 16 16:48:26 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.72099999999999997,"15min":0.71299999999999997},"min":{"1min":0.498,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.83899999999999997,"5min":2.7650000000000001,"15min":2.7650000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.73599999999999999,"15min":0.745},"min":{"1min":0.44,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.86399999999999999,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.83399999999999996}]},{"osd":9,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.79600000000000004,"15min":0.80500000000000005},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.77900000000000003,"15min":0.78100000000000003},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.80900000000000005}]},{"osd":10,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69499999999999995,"15min":0.69999999999999996},"min":{"1min":0.47599999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.86499999999999999,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.71799999999999997,"15min":0.72199999999999998},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.73899999999999999}]},{"osd":11,"last update":"Thu Nov 16 16:48:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.82399999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757305,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6708,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738444,"statfs":{"total":95995035648,"available":95988166656,"internally_reserved":0,"allocated":1028096,"data_stored":1077786,"data_compressed":53424,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:19 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52100000000000002,"15min":0.51500000000000001},"min":{"1min":0.34599999999999997,"5min":0.318,"15min":0.318},"max":{"1min":0.751,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.44400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76000000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.48899999999999999}]},{"osd":1,"last update":"Thu Nov 16 16:48:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.56899999999999995,"15min":0.54700000000000004},"min":{"1min":0.38500000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77900000000000003,"5min":1.167,"15min":1.167},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.55700000000000005,"15min":0.51700000000000002},"min":{"1min":0.441,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81799999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Nov 16 16:48:19 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.38900000000000001,"15min":0.372},"min":{"1min":0.32400000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60299999999999998,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.41599999999999998,"15min":0.39100000000000001},"min":{"1min":0.33300000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.63500000000000001,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.38}]},{"osd":3,"last update":"Thu Nov 16 16:48:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.55500000000000005,"15min":0.53500000000000003},"min":{"1min":0.40699999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84299999999999997,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.436},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59299999999999997,"15min":0.56999999999999995},"min":{"1min":0.42899999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78600000000000003,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.55600000000000005}]},{"osd":4,"last update":"Thu Nov 16 16:48:19 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.57399999999999995,"15min":0.55300000000000005},"min":{"1min":0.39400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81299999999999994,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.55800000000000005,"15min":0.52800000000000002},"min":{"1min":0.42099999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.80400000000000005,"5min":0.879,"15min":0.879},"last":0.58499999999999996}]},{"osd":6,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.63200000000000001,"15min":0.63},"min":{"1min":0.436,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91200000000000003,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.53300000000000003}]},{"osd":7,"last update":"Thu Nov 16 16:48:06 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84099999999999997,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.748},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84799999999999998,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.61499999999999999}]},{"osd":8,"last update":"Thu Nov 16 16:48:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.42299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80100000000000005,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.58799999999999997},"min":{"1min":0.47799999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.59899999999999998}]},{"osd":9,"last update":"Thu Nov 16 16:47:50 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.748,"15min":0.76100000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.86799999999999999,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.749,"15min":0.75900000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.871,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.72799999999999998}]},{"osd":10,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.70599999999999996,"15min":0.71299999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88800000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.76300000000000001,"15min":0.755},"min":{"1min":0.51900000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":2.492,"5min":2.492,"15min":2.492},"last":0.68600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":60942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T16:48:39.898 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph pg dump --format=json 2023-11-16T16:48:40.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:40 smithi077 ceph-mon[131204]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:48:40.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:40 smithi044 ceph-mon[128750]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:48:40.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:40 smithi176 ceph-mon[131630]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:48:40.762 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:41.358 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:41 smithi044 ceph-mon[128750]: pgmap v318: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:41.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:41 smithi077 ceph-mon[131204]: pgmap v318: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:41.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:41 smithi176 ceph-mon[131630]: pgmap v318: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:42.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.296+0000 7f8f00fd0700 1 -- 172.21.15.44:0/318204299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8efc1055e0 msgr2=0x7f8efc10d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:42.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.296+0000 7f8f00fd0700 1 --2- 172.21.15.44:0/318204299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8efc1055e0 0x7f8efc10d160 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8ef400b0d0 tx=0x7f8ef400b3e0 comp rx=0 tx=0).stop 2023-11-16T16:48:42.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.296+0000 7f8f00fd0700 1 -- 172.21.15.44:0/318204299 shutdown_connections 2023-11-16T16:48:42.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.296+0000 7f8f00fd0700 1 --2- 172.21.15.44:0/318204299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8efc1055e0 0x7f8efc10d160 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:42.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.296+0000 7f8f00fd0700 1 --2- 172.21.15.44:0/318204299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8efc100540 0x7f8efc100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:42.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.296+0000 7f8f00fd0700 1 --2- 172.21.15.44:0/318204299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8efc100ee0 0x7f8efc104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:42.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.296+0000 7f8f00fd0700 1 -- 172.21.15.44:0/318204299 >> 172.21.15.44:0/318204299 conn(0x7f8efc0fbdc0 msgr2=0x7f8efc0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:42.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.297+0000 7f8f00fd0700 1 -- 172.21.15.44:0/318204299 shutdown_connections 2023-11-16T16:48:42.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.297+0000 7f8f00fd0700 1 -- 172.21.15.44:0/318204299 wait complete. 2023-11-16T16:48:42.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.298+0000 7f8f00fd0700 1 Processor -- start 2023-11-16T16:48:42.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.298+0000 7f8f00fd0700 1 -- start start 2023-11-16T16:48:42.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.298+0000 7f8f00fd0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8efc100540 0x7f8efc198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:42.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.298+0000 7f8f00fd0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8efc100ee0 0x7f8efc198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:42.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.298+0000 7f8f00fd0700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8efc1055e0 0x7f8efc19c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:42.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.298+0000 7f8f00fd0700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc19c600 con 0x7f8efc1055e0 2023-11-16T16:48:42.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.298+0000 7f8f00fd0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc19c770 con 0x7f8efc100ee0 2023-11-16T16:48:42.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.298+0000 7f8f00fd0700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc19c930 con 0x7f8efc100540 2023-11-16T16:48:42.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.298+0000 7f8efad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8efc1055e0 0x7f8efc19c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:42.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.298+0000 7f8ef9d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8efc100ee0 0x7f8efc198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:42.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.298+0000 7f8efad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8efc1055e0 0x7f8efc19c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44510/0 (socket says 172.21.15.44:44510) 2023-11-16T16:48:42.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.298+0000 7f8efad9d700 1 -- 172.21.15.44:0/196952347 learned_addr learned my addr 172.21.15.44:0/196952347 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:42.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.299+0000 7f8efa59c700 1 --2- 172.21.15.44:0/196952347 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8efc100540 0x7f8efc198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:42.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.299+0000 7f8efad9d700 1 -- 172.21.15.44:0/196952347 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8efc100540 msgr2=0x7f8efc198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:42.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.299+0000 7f8efad9d700 1 --2- 172.21.15.44:0/196952347 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8efc100540 0x7f8efc198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:42.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.299+0000 7f8efad9d700 1 -- 172.21.15.44:0/196952347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8efc100ee0 msgr2=0x7f8efc198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:42.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.299+0000 7f8efad9d700 1 --2- 172.21.15.44:0/196952347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8efc100ee0 0x7f8efc198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:42.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.299+0000 7f8efad9d700 1 -- 172.21.15.44:0/196952347 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef4009d00 con 0x7f8efc1055e0 2023-11-16T16:48:42.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.299+0000 7f8ef9d9b700 1 --2- 172.21.15.44:0/196952347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8efc100ee0 0x7f8efc198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:42.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.299+0000 7f8efad9d700 1 --2- 172.21.15.44:0/196952347 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8efc1055e0 0x7f8efc19c000 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4007870 tx=0x7f8ef40078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:42.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.299+0000 7f8eeb7fe700 1 -- 172.21.15.44:0/196952347 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef400e040 con 0x7f8efc1055e0 2023-11-16T16:48:42.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.299+0000 7f8eeb7fe700 1 -- 172.21.15.44:0/196952347 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ef4005070 con 0x7f8efc1055e0 2023-11-16T16:48:42.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.299+0000 7f8f00fd0700 1 -- 172.21.15.44:0/196952347 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8efc110320 con 0x7f8efc1055e0 2023-11-16T16:48:42.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.300+0000 7f8eeb7fe700 1 -- 172.21.15.44:0/196952347 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef401b6c0 con 0x7f8efc1055e0 2023-11-16T16:48:42.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.300+0000 7f8f00fd0700 1 -- 172.21.15.44:0/196952347 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8efc110190 con 0x7f8efc1055e0 2023-11-16T16:48:42.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.301+0000 7f8f00fd0700 1 -- 172.21.15.44:0/196952347 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8efc1104b0 con 0x7f8efc1055e0 2023-11-16T16:48:42.305 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.303+0000 7f8eeb7fe700 1 -- 172.21.15.44:0/196952347 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8ef40051e0 con 0x7f8efc1055e0 2023-11-16T16:48:42.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.304+0000 7f8eeb7fe700 1 --2- 172.21.15.44:0/196952347 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8ee40a11e0 0x7f8ee40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:42.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.304+0000 7f8eeb7fe700 1 -- 172.21.15.44:0/196952347 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8ef40cde30 con 0x7f8efc1055e0 2023-11-16T16:48:42.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.305+0000 7f8efa59c700 1 --2- 172.21.15.44:0/196952347 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8ee40a11e0 0x7f8ee40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:42.307 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.305+0000 7f8efa59c700 1 --2- 172.21.15.44:0/196952347 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8ee40a11e0 0x7f8ee40a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8eec009c60 tx=0x7f8eec009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:42.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.308+0000 7f8eeb7fe700 1 -- 172.21.15.44:0/196952347 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8ef40971a0 con 0x7f8efc1055e0 2023-11-16T16:48:42.512 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.511+0000 7f8f00fd0700 1 -- 172.21.15.44:0/196952347 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8efc10bd10 con 0x7f8ee40a11e0 2023-11-16T16:48:42.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.524+0000 7f8eeb7fe700 1 -- 172.21.15.44:0/196952347 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427264 (secure 0 0 0) 0x7f8efc10bd10 con 0x7f8ee40a11e0 2023-11-16T16:48:42.533 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:48:42.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.531+0000 7f8f00fd0700 1 -- 172.21.15.44:0/196952347 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8ee40a11e0 msgr2=0x7f8ee40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.531+0000 7f8f00fd0700 1 --2- 172.21.15.44:0/196952347 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8ee40a11e0 0x7f8ee40a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8eec009c60 tx=0x7f8eec009260 comp rx=0 tx=0).stop 2023-11-16T16:48:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.531+0000 7f8f00fd0700 1 -- 172.21.15.44:0/196952347 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8efc1055e0 msgr2=0x7f8efc19c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.531+0000 7f8f00fd0700 1 --2- 172.21.15.44:0/196952347 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8efc1055e0 0x7f8efc19c000 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4007870 tx=0x7f8ef40078a0 comp rx=0 tx=0).stop 2023-11-16T16:48:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.531+0000 7f8f00fd0700 1 -- 172.21.15.44:0/196952347 shutdown_connections 2023-11-16T16:48:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.531+0000 7f8f00fd0700 1 --2- 172.21.15.44:0/196952347 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8ee40a11e0 0x7f8ee40a3690 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.531+0000 7f8f00fd0700 1 --2- 172.21.15.44:0/196952347 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8efc1055e0 0x7f8efc19c000 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.531+0000 7f8f00fd0700 1 --2- 172.21.15.44:0/196952347 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8efc100540 0x7f8efc198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.531+0000 7f8f00fd0700 1 --2- 172.21.15.44:0/196952347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8efc100ee0 0x7f8efc198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.531+0000 7f8f00fd0700 1 -- 172.21.15.44:0/196952347 >> 172.21.15.44:0/196952347 conn(0x7f8efc0fbdc0 msgr2=0x7f8efc10b600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.531+0000 7f8f00fd0700 1 -- 172.21.15.44:0/196952347 shutdown_connections 2023-11-16T16:48:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:42.531+0000 7f8f00fd0700 1 -- 172.21.15.44:0/196952347 wait complete. 2023-11-16T16:48:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2023-11-16T16:48:42.960 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2023-11-16T16:48:40.553850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652320,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":252,"num_read_kb":241,"num_write":259,"num_write_kb":3028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80612,"kb_used_data":11716,"kb_used_omap":0,"kb_used_meta":68800,"kb_avail":1124861212,"statfs":{"total":1151940427776,"available":1151857881088,"internally_reserved":0,"allocated":11997184,"data_stored":7786397,"data_compressed":318969,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":0,"internal_metadata":70451200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003599"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625866+0000","last_change":"2023-11-16T16:47:18.183024+0000","last_active":"2023-11-16T16:47:43.625866+0000","last_peered":"2023-11-16T16:47:43.625866+0000","last_clean":"2023-11-16T16:47:43.625866+0000","last_became_active":"2023-11-16T16:47:18.182387+0000","last_became_peered":"2023-11-16T16:47:18.182387+0000","last_unstale":"2023-11-16T16:47:43.625866+0000","last_undegraded":"2023-11-16T16:47:43.625866+0000","last_fullsized":"2023-11-16T16:47:43.625866+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:33:54.818555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626848+0000","last_change":"2023-11-16T16:47:18.200135+0000","last_active":"2023-11-16T16:47:43.626848+0000","last_peered":"2023-11-16T16:47:43.626848+0000","last_clean":"2023-11-16T16:47:43.626848+0000","last_became_active":"2023-11-16T16:47:18.194732+0000","last_became_peered":"2023-11-16T16:47:18.194732+0000","last_unstale":"2023-11-16T16:47:43.626848+0000","last_undegraded":"2023-11-16T16:47:43.626848+0000","last_fullsized":"2023-11-16T16:47:43.626848+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:35:44.790080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626424+0000","last_change":"2023-11-16T16:47:18.184127+0000","last_active":"2023-11-16T16:47:43.626424+0000","last_peered":"2023-11-16T16:47:43.626424+0000","last_clean":"2023-11-16T16:47:43.626424+0000","last_became_active":"2023-11-16T16:47:18.183163+0000","last_became_peered":"2023-11-16T16:47:18.183163+0000","last_unstale":"2023-11-16T16:47:43.626424+0000","last_undegraded":"2023-11-16T16:47:43.626424+0000","last_fullsized":"2023-11-16T16:47:43.626424+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:22:40.700926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629755+0000","last_change":"2023-11-16T16:47:18.177476+0000","last_active":"2023-11-16T16:47:43.629755+0000","last_peered":"2023-11-16T16:47:43.629755+0000","last_clean":"2023-11-16T16:47:43.629755+0000","last_became_active":"2023-11-16T16:47:18.177178+0000","last_became_peered":"2023-11-16T16:47:18.177178+0000","last_unstale":"2023-11-16T16:47:43.629755+0000","last_undegraded":"2023-11-16T16:47:43.629755+0000","last_fullsized":"2023-11-16T16:47:43.629755+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:40:05.452664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626685+0000","last_change":"2023-11-16T16:47:18.178511+0000","last_active":"2023-11-16T16:47:43.626685+0000","last_peered":"2023-11-16T16:47:43.626685+0000","last_clean":"2023-11-16T16:47:43.626685+0000","last_became_active":"2023-11-16T16:47:18.178198+0000","last_became_peered":"2023-11-16T16:47:18.178198+0000","last_unstale":"2023-11-16T16:47:43.626685+0000","last_undegraded":"2023-11-16T16:47:43.626685+0000","last_fullsized":"2023-11-16T16:47:43.626685+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:53:35.196107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626326+0000","last_change":"2023-11-16T16:47:18.186552+0000","last_active":"2023-11-16T16:47:43.626326+0000","last_peered":"2023-11-16T16:47:43.626326+0000","last_clean":"2023-11-16T16:47:43.626326+0000","last_became_active":"2023-11-16T16:47:18.186238+0000","last_became_peered":"2023-11-16T16:47:18.186238+0000","last_unstale":"2023-11-16T16:47:43.626326+0000","last_undegraded":"2023-11-16T16:47:43.626326+0000","last_fullsized":"2023-11-16T16:47:43.626326+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:36:52.844919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622387+0000","last_change":"2023-11-16T16:47:18.179839+0000","last_active":"2023-11-16T16:47:43.622387+0000","last_peered":"2023-11-16T16:47:43.622387+0000","last_clean":"2023-11-16T16:47:43.622387+0000","last_became_active":"2023-11-16T16:47:18.179060+0000","last_became_peered":"2023-11-16T16:47:18.179060+0000","last_unstale":"2023-11-16T16:47:43.622387+0000","last_undegraded":"2023-11-16T16:47:43.622387+0000","last_fullsized":"2023-11-16T16:47:43.622387+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:55:30.972787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.633015+0000","last_change":"2023-11-16T16:47:43.631738+0000","last_active":"2023-11-16T16:47:43.633015+0000","last_peered":"2023-11-16T16:47:43.633015+0000","last_clean":"2023-11-16T16:47:43.633015+0000","last_became_active":"2023-11-16T16:47:43.631146+0000","last_became_peered":"2023-11-16T16:47:43.631146+0000","last_unstale":"2023-11-16T16:47:43.633015+0000","last_undegraded":"2023-11-16T16:47:43.633015+0000","last_fullsized":"2023-11-16T16:47:43.633015+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:32:26.552018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947385+0000","last_change":"2023-11-16T16:47:18.177834+0000","last_active":"2023-11-16T16:47:43.947385+0000","last_peered":"2023-11-16T16:47:43.947385+0000","last_clean":"2023-11-16T16:47:43.947385+0000","last_became_active":"2023-11-16T16:47:18.176299+0000","last_became_peered":"2023-11-16T16:47:18.176299+0000","last_unstale":"2023-11-16T16:47:43.947385+0000","last_undegraded":"2023-11-16T16:47:43.947385+0000","last_fullsized":"2023-11-16T16:47:43.947385+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:18:34.674950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626777+0000","last_change":"2023-11-16T16:47:18.179394+0000","last_active":"2023-11-16T16:47:43.626777+0000","last_peered":"2023-11-16T16:47:43.626777+0000","last_clean":"2023-11-16T16:47:43.626777+0000","last_became_active":"2023-11-16T16:47:18.179119+0000","last_became_peered":"2023-11-16T16:47:18.179119+0000","last_unstale":"2023-11-16T16:47:43.626777+0000","last_undegraded":"2023-11-16T16:47:43.626777+0000","last_fullsized":"2023-11-16T16:47:43.626777+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:32:22.770618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623315+0000","last_change":"2023-11-16T16:47:18.183585+0000","last_active":"2023-11-16T16:47:43.623315+0000","last_peered":"2023-11-16T16:47:43.623315+0000","last_clean":"2023-11-16T16:47:43.623315+0000","last_became_active":"2023-11-16T16:47:18.183209+0000","last_became_peered":"2023-11-16T16:47:18.183209+0000","last_unstale":"2023-11-16T16:47:43.623315+0000","last_undegraded":"2023-11-16T16:47:43.623315+0000","last_fullsized":"2023-11-16T16:47:43.623315+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:22:47.894810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629596+0000","last_change":"2023-11-16T16:47:18.186503+0000","last_active":"2023-11-16T16:47:43.629596+0000","last_peered":"2023-11-16T16:47:43.629596+0000","last_clean":"2023-11-16T16:47:43.629596+0000","last_became_active":"2023-11-16T16:47:18.186193+0000","last_became_peered":"2023-11-16T16:47:18.186193+0000","last_unstale":"2023-11-16T16:47:43.629596+0000","last_undegraded":"2023-11-16T16:47:43.629596+0000","last_fullsized":"2023-11-16T16:47:43.629596+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:59:03.486858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629884+0000","last_change":"2023-11-16T16:47:18.193842+0000","last_active":"2023-11-16T16:47:43.629884+0000","last_peered":"2023-11-16T16:47:43.629884+0000","last_clean":"2023-11-16T16:47:43.629884+0000","last_became_active":"2023-11-16T16:47:18.192309+0000","last_became_peered":"2023-11-16T16:47:18.192309+0000","last_unstale":"2023-11-16T16:47:43.629884+0000","last_undegraded":"2023-11-16T16:47:43.629884+0000","last_fullsized":"2023-11-16T16:47:43.629884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:53:33.884565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625812+0000","last_change":"2023-11-16T16:47:18.192359+0000","last_active":"2023-11-16T16:47:43.625812+0000","last_peered":"2023-11-16T16:47:43.625812+0000","last_clean":"2023-11-16T16:47:43.625812+0000","last_became_active":"2023-11-16T16:47:18.189765+0000","last_became_peered":"2023-11-16T16:47:18.189765+0000","last_unstale":"2023-11-16T16:47:43.625812+0000","last_undegraded":"2023-11-16T16:47:43.625812+0000","last_fullsized":"2023-11-16T16:47:43.625812+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:17:15.440725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625875+0000","last_change":"2023-11-16T16:47:18.178386+0000","last_active":"2023-11-16T16:47:43.625875+0000","last_peered":"2023-11-16T16:47:43.625875+0000","last_clean":"2023-11-16T16:47:43.625875+0000","last_became_active":"2023-11-16T16:47:18.178041+0000","last_became_peered":"2023-11-16T16:47:18.178041+0000","last_unstale":"2023-11-16T16:47:43.625875+0000","last_undegraded":"2023-11-16T16:47:43.625875+0000","last_fullsized":"2023-11-16T16:47:43.625875+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:27:47.278353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.946667+0000","last_change":"2023-11-16T16:47:18.174424+0000","last_active":"2023-11-16T16:47:43.946667+0000","last_peered":"2023-11-16T16:47:43.946667+0000","last_clean":"2023-11-16T16:47:43.946667+0000","last_became_active":"2023-11-16T16:47:18.174078+0000","last_became_peered":"2023-11-16T16:47:18.174078+0000","last_unstale":"2023-11-16T16:47:43.946667+0000","last_undegraded":"2023-11-16T16:47:43.946667+0000","last_fullsized":"2023-11-16T16:47:43.946667+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:14:13.491783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625739+0000","last_change":"2023-11-16T16:47:18.183161+0000","last_active":"2023-11-16T16:47:43.625739+0000","last_peered":"2023-11-16T16:47:43.625739+0000","last_clean":"2023-11-16T16:47:43.625739+0000","last_became_active":"2023-11-16T16:47:18.182806+0000","last_became_peered":"2023-11-16T16:47:18.182806+0000","last_unstale":"2023-11-16T16:47:43.625739+0000","last_undegraded":"2023-11-16T16:47:43.625739+0000","last_fullsized":"2023-11-16T16:47:43.625739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:50:04.072542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622713+0000","last_change":"2023-11-16T16:47:18.187225+0000","last_active":"2023-11-16T16:47:43.622713+0000","last_peered":"2023-11-16T16:47:43.622713+0000","last_clean":"2023-11-16T16:47:43.622713+0000","last_became_active":"2023-11-16T16:47:18.186858+0000","last_became_peered":"2023-11-16T16:47:18.186858+0000","last_unstale":"2023-11-16T16:47:43.622713+0000","last_undegraded":"2023-11-16T16:47:43.622713+0000","last_fullsized":"2023-11-16T16:47:43.622713+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:42:28.736883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623323+0000","last_change":"2023-11-16T16:47:18.190947+0000","last_active":"2023-11-16T16:47:43.623323+0000","last_peered":"2023-11-16T16:47:43.623323+0000","last_clean":"2023-11-16T16:47:43.623323+0000","last_became_active":"2023-11-16T16:47:18.190737+0000","last_became_peered":"2023-11-16T16:47:18.190737+0000","last_unstale":"2023-11-16T16:47:43.623323+0000","last_undegraded":"2023-11-16T16:47:43.623323+0000","last_fullsized":"2023-11-16T16:47:43.623323+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:34:59.150388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622328+0000","last_change":"2023-11-16T16:47:22.659730+0000","last_active":"2023-11-16T16:47:43.622328+0000","last_peered":"2023-11-16T16:47:43.622328+0000","last_clean":"2023-11-16T16:47:43.622328+0000","last_became_active":"2023-11-16T16:47:22.659313+0000","last_became_peered":"2023-11-16T16:47:22.659313+0000","last_unstale":"2023-11-16T16:47:43.622328+0000","last_undegraded":"2023-11-16T16:47:43.622328+0000","last_fullsized":"2023-11-16T16:47:43.622328+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:05:05.336507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'4","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622965+0000","last_change":"2023-11-16T16:47:17.629748+0000","last_active":"2023-11-16T16:47:43.622965+0000","last_peered":"2023-11-16T16:47:43.622965+0000","last_clean":"2023-11-16T16:47:43.622965+0000","last_became_active":"2023-11-16T16:47:17.629284+0000","last_became_peered":"2023-11-16T16:47:17.629284+0000","last_unstale":"2023-11-16T16:47:43.622965+0000","last_undegraded":"2023-11-16T16:47:43.622965+0000","last_fullsized":"2023-11-16T16:47:43.622965+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:41:29.574724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970501+0000","last_change":"2023-11-16T16:47:18.176239+0000","last_active":"2023-11-16T16:47:43.970501+0000","last_peered":"2023-11-16T16:47:43.970501+0000","last_clean":"2023-11-16T16:47:43.970501+0000","last_became_active":"2023-11-16T16:47:18.175879+0000","last_became_peered":"2023-11-16T16:47:18.175879+0000","last_unstale":"2023-11-16T16:47:43.970501+0000","last_undegraded":"2023-11-16T16:47:43.970501+0000","last_fullsized":"2023-11-16T16:47:43.970501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:32:09.544453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.628988+0000","last_change":"2023-11-16T16:47:22.659679+0000","last_active":"2023-11-16T16:47:43.628988+0000","last_peered":"2023-11-16T16:47:43.628988+0000","last_clean":"2023-11-16T16:47:43.628988+0000","last_became_active":"2023-11-16T16:47:22.658946+0000","last_became_peered":"2023-11-16T16:47:22.658946+0000","last_unstale":"2023-11-16T16:47:43.628988+0000","last_undegraded":"2023-11-16T16:47:43.628988+0000","last_fullsized":"2023-11-16T16:47:43.628988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:33:44.392111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"82'1","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.971259+0000","last_change":"2023-11-16T16:47:17.643944+0000","last_active":"2023-11-16T16:47:43.971259+0000","last_peered":"2023-11-16T16:47:43.971259+0000","last_clean":"2023-11-16T16:47:43.971259+0000","last_became_active":"2023-11-16T16:47:17.643470+0000","last_became_peered":"2023-11-16T16:47:17.643470+0000","last_unstale":"2023-11-16T16:47:43.971259+0000","last_undegraded":"2023-11-16T16:47:43.971259+0000","last_fullsized":"2023-11-16T16:47:43.971259+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:28:22.870692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.627110+0000","last_change":"2023-11-16T16:47:18.198139+0000","last_active":"2023-11-16T16:47:43.627110+0000","last_peered":"2023-11-16T16:47:43.627110+0000","last_clean":"2023-11-16T16:47:43.627110+0000","last_became_active":"2023-11-16T16:47:18.197762+0000","last_became_peered":"2023-11-16T16:47:18.197762+0000","last_unstale":"2023-11-16T16:47:43.627110+0000","last_undegraded":"2023-11-16T16:47:43.627110+0000","last_fullsized":"2023-11-16T16:47:43.627110+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:27:32.243043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629208+0000","last_change":"2023-11-16T16:47:22.655810+0000","last_active":"2023-11-16T16:47:43.629208+0000","last_peered":"2023-11-16T16:47:43.629208+0000","last_clean":"2023-11-16T16:47:43.629208+0000","last_became_active":"2023-11-16T16:47:22.655388+0000","last_became_peered":"2023-11-16T16:47:22.655388+0000","last_unstale":"2023-11-16T16:47:43.629208+0000","last_undegraded":"2023-11-16T16:47:43.629208+0000","last_fullsized":"2023-11-16T16:47:43.629208+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:05:46.905125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'2","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.635266+0000","last_change":"2023-11-16T16:47:43.635266+0000","last_active":"2023-11-16T16:47:43.635266+0000","last_peered":"2023-11-16T16:47:43.635266+0000","last_clean":"2023-11-16T16:47:43.635266+0000","last_became_active":"2023-11-16T16:47:43.630583+0000","last_became_peered":"2023-11-16T16:47:43.630583+0000","last_unstale":"2023-11-16T16:47:43.635266+0000","last_undegraded":"2023-11-16T16:47:43.635266+0000","last_fullsized":"2023-11-16T16:47:43.635266+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:51:43.792892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.946938+0000","last_change":"2023-11-16T16:47:18.186341+0000","last_active":"2023-11-16T16:47:43.946938+0000","last_peered":"2023-11-16T16:47:43.946938+0000","last_clean":"2023-11-16T16:47:43.946938+0000","last_became_active":"2023-11-16T16:47:18.186066+0000","last_became_peered":"2023-11-16T16:47:18.186066+0000","last_unstale":"2023-11-16T16:47:43.946938+0000","last_undegraded":"2023-11-16T16:47:43.946938+0000","last_fullsized":"2023-11-16T16:47:43.946938+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:22:46.055671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629005+0000","last_change":"2023-11-16T16:47:22.657227+0000","last_active":"2023-11-16T16:47:43.629005+0000","last_peered":"2023-11-16T16:47:43.629005+0000","last_clean":"2023-11-16T16:47:43.629005+0000","last_became_active":"2023-11-16T16:47:22.656835+0000","last_became_peered":"2023-11-16T16:47:22.656835+0000","last_unstale":"2023-11-16T16:47:43.629005+0000","last_undegraded":"2023-11-16T16:47:43.629005+0000","last_fullsized":"2023-11-16T16:47:43.629005+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:07.552894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625950+0000","last_change":"2023-11-16T16:47:17.630194+0000","last_active":"2023-11-16T16:47:43.625950+0000","last_peered":"2023-11-16T16:47:43.625950+0000","last_clean":"2023-11-16T16:47:43.625950+0000","last_became_active":"2023-11-16T16:47:17.629886+0000","last_became_peered":"2023-11-16T16:47:17.629886+0000","last_unstale":"2023-11-16T16:47:43.625950+0000","last_undegraded":"2023-11-16T16:47:43.625950+0000","last_fullsized":"2023-11-16T16:47:43.625950+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:57:09.970289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630557+0000","last_change":"2023-11-16T16:47:18.181596+0000","last_active":"2023-11-16T16:47:43.630557+0000","last_peered":"2023-11-16T16:47:43.630557+0000","last_clean":"2023-11-16T16:47:43.630557+0000","last_became_active":"2023-11-16T16:47:18.181287+0000","last_became_peered":"2023-11-16T16:47:18.181287+0000","last_unstale":"2023-11-16T16:47:43.630557+0000","last_undegraded":"2023-11-16T16:47:43.630557+0000","last_fullsized":"2023-11-16T16:47:43.630557+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:31:55.798497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622763+0000","last_change":"2023-11-16T16:47:22.649197+0000","last_active":"2023-11-16T16:47:43.622763+0000","last_peered":"2023-11-16T16:47:43.622763+0000","last_clean":"2023-11-16T16:47:43.622763+0000","last_became_active":"2023-11-16T16:47:22.648664+0000","last_became_peered":"2023-11-16T16:47:22.648664+0000","last_unstale":"2023-11-16T16:47:43.622763+0000","last_undegraded":"2023-11-16T16:47:43.622763+0000","last_fullsized":"2023-11-16T16:47:43.622763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:56:11.119940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"82'1","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630141+0000","last_change":"2023-11-16T16:47:17.627672+0000","last_active":"2023-11-16T16:47:43.630141+0000","last_peered":"2023-11-16T16:47:43.630141+0000","last_clean":"2023-11-16T16:47:43.630141+0000","last_became_active":"2023-11-16T16:47:17.627383+0000","last_became_peered":"2023-11-16T16:47:17.627383+0000","last_unstale":"2023-11-16T16:47:43.630141+0000","last_undegraded":"2023-11-16T16:47:43.630141+0000","last_fullsized":"2023-11-16T16:47:43.630141+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:35:57.379840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625043+0000","last_change":"2023-11-16T16:47:18.180983+0000","last_active":"2023-11-16T16:47:43.625043+0000","last_peered":"2023-11-16T16:47:43.625043+0000","last_clean":"2023-11-16T16:47:43.625043+0000","last_became_active":"2023-11-16T16:47:18.180702+0000","last_became_peered":"2023-11-16T16:47:18.180702+0000","last_unstale":"2023-11-16T16:47:43.625043+0000","last_undegraded":"2023-11-16T16:47:43.625043+0000","last_fullsized":"2023-11-16T16:47:43.625043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:20:27.895648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622657+0000","last_change":"2023-11-16T16:47:22.659898+0000","last_active":"2023-11-16T16:47:43.622657+0000","last_peered":"2023-11-16T16:47:43.622657+0000","last_clean":"2023-11-16T16:47:43.622657+0000","last_became_active":"2023-11-16T16:47:22.659326+0000","last_became_peered":"2023-11-16T16:47:22.659326+0000","last_unstale":"2023-11-16T16:47:43.622657+0000","last_undegraded":"2023-11-16T16:47:43.622657+0000","last_fullsized":"2023-11-16T16:47:43.622657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:57:36.289645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"82'1","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.949231+0000","last_change":"2023-11-16T16:47:43.949231+0000","last_active":"2023-11-16T16:47:43.949231+0000","last_peered":"2023-11-16T16:47:43.949231+0000","last_clean":"2023-11-16T16:47:43.949231+0000","last_became_active":"2023-11-16T16:47:43.948766+0000","last_became_peered":"2023-11-16T16:47:43.948766+0000","last_unstale":"2023-11-16T16:47:43.949231+0000","last_undegraded":"2023-11-16T16:47:43.949231+0000","last_fullsized":"2023-11-16T16:47:43.949231+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:46:57.086811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630999+0000","last_change":"2023-11-16T16:47:18.185263+0000","last_active":"2023-11-16T16:47:43.630999+0000","last_peered":"2023-11-16T16:47:43.630999+0000","last_clean":"2023-11-16T16:47:43.630999+0000","last_became_active":"2023-11-16T16:47:18.184910+0000","last_became_peered":"2023-11-16T16:47:18.184910+0000","last_unstale":"2023-11-16T16:47:43.630999+0000","last_undegraded":"2023-11-16T16:47:43.630999+0000","last_fullsized":"2023-11-16T16:47:43.630999+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:25:21.463648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625581+0000","last_change":"2023-11-16T16:47:22.659711+0000","last_active":"2023-11-16T16:47:43.625581+0000","last_peered":"2023-11-16T16:47:43.625581+0000","last_clean":"2023-11-16T16:47:43.625581+0000","last_became_active":"2023-11-16T16:47:22.659087+0000","last_became_peered":"2023-11-16T16:47:22.659087+0000","last_unstale":"2023-11-16T16:47:43.625581+0000","last_undegraded":"2023-11-16T16:47:43.625581+0000","last_fullsized":"2023-11-16T16:47:43.625581+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:13:26.339037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629045+0000","last_change":"2023-11-16T16:47:17.629703+0000","last_active":"2023-11-16T16:47:43.629045+0000","last_peered":"2023-11-16T16:47:43.629045+0000","last_clean":"2023-11-16T16:47:43.629045+0000","last_became_active":"2023-11-16T16:47:17.629245+0000","last_became_peered":"2023-11-16T16:47:17.629245+0000","last_unstale":"2023-11-16T16:47:43.629045+0000","last_undegraded":"2023-11-16T16:47:43.629045+0000","last_fullsized":"2023-11-16T16:47:43.629045+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:59:34.790629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970314+0000","last_change":"2023-11-16T16:47:18.187149+0000","last_active":"2023-11-16T16:47:43.970314+0000","last_peered":"2023-11-16T16:47:43.970314+0000","last_clean":"2023-11-16T16:47:43.970314+0000","last_became_active":"2023-11-16T16:47:18.185717+0000","last_became_peered":"2023-11-16T16:47:18.185717+0000","last_unstale":"2023-11-16T16:47:43.970314+0000","last_undegraded":"2023-11-16T16:47:43.970314+0000","last_fullsized":"2023-11-16T16:47:43.970314+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:23:04.064643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626668+0000","last_change":"2023-11-16T16:47:22.663025+0000","last_active":"2023-11-16T16:47:43.626668+0000","last_peered":"2023-11-16T16:47:43.626668+0000","last_clean":"2023-11-16T16:47:43.626668+0000","last_became_active":"2023-11-16T16:47:22.662596+0000","last_became_peered":"2023-11-16T16:47:22.662596+0000","last_unstale":"2023-11-16T16:47:43.626668+0000","last_undegraded":"2023-11-16T16:47:43.626668+0000","last_fullsized":"2023-11-16T16:47:43.626668+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:47:26.815530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629668+0000","last_change":"2023-11-16T16:47:17.626357+0000","last_active":"2023-11-16T16:47:43.629668+0000","last_peered":"2023-11-16T16:47:43.629668+0000","last_clean":"2023-11-16T16:47:43.629668+0000","last_became_active":"2023-11-16T16:47:17.626029+0000","last_became_peered":"2023-11-16T16:47:17.626029+0000","last_unstale":"2023-11-16T16:47:43.629668+0000","last_undegraded":"2023-11-16T16:47:43.629668+0000","last_fullsized":"2023-11-16T16:47:43.629668+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:55:28.659340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.621841+0000","last_change":"2023-11-16T16:47:18.177153+0000","last_active":"2023-11-16T16:47:43.621841+0000","last_peered":"2023-11-16T16:47:43.621841+0000","last_clean":"2023-11-16T16:47:43.621841+0000","last_became_active":"2023-11-16T16:47:18.176751+0000","last_became_peered":"2023-11-16T16:47:18.176751+0000","last_unstale":"2023-11-16T16:47:43.621841+0000","last_undegraded":"2023-11-16T16:47:43.621841+0000","last_fullsized":"2023-11-16T16:47:43.621841+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:17:58.058194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970584+0000","last_change":"2023-11-16T16:47:22.652718+0000","last_active":"2023-11-16T16:47:43.970584+0000","last_peered":"2023-11-16T16:47:43.970584+0000","last_clean":"2023-11-16T16:47:43.970584+0000","last_became_active":"2023-11-16T16:47:22.652278+0000","last_became_peered":"2023-11-16T16:47:22.652278+0000","last_unstale":"2023-11-16T16:47:43.970584+0000","last_undegraded":"2023-11-16T16:47:43.970584+0000","last_fullsized":"2023-11-16T16:47:43.970584+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:43:41.546583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626473+0000","last_change":"2023-11-16T16:47:17.627308+0000","last_active":"2023-11-16T16:47:43.626473+0000","last_peered":"2023-11-16T16:47:43.626473+0000","last_clean":"2023-11-16T16:47:43.626473+0000","last_became_active":"2023-11-16T16:47:17.626992+0000","last_became_peered":"2023-11-16T16:47:17.626992+0000","last_unstale":"2023-11-16T16:47:43.626473+0000","last_undegraded":"2023-11-16T16:47:43.626473+0000","last_fullsized":"2023-11-16T16:47:43.626473+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:41:51.914757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947323+0000","last_change":"2023-11-16T16:47:18.188015+0000","last_active":"2023-11-16T16:47:43.947323+0000","last_peered":"2023-11-16T16:47:43.947323+0000","last_clean":"2023-11-16T16:47:43.947323+0000","last_became_active":"2023-11-16T16:47:18.187730+0000","last_became_peered":"2023-11-16T16:47:18.187730+0000","last_unstale":"2023-11-16T16:47:43.947323+0000","last_undegraded":"2023-11-16T16:47:43.947323+0000","last_fullsized":"2023-11-16T16:47:43.947323+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:39:02.651682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.621932+0000","last_change":"2023-11-16T16:47:22.655656+0000","last_active":"2023-11-16T16:47:43.621932+0000","last_peered":"2023-11-16T16:47:43.621932+0000","last_clean":"2023-11-16T16:47:43.621932+0000","last_became_active":"2023-11-16T16:47:22.655133+0000","last_became_peered":"2023-11-16T16:47:22.655133+0000","last_unstale":"2023-11-16T16:47:43.621932+0000","last_undegraded":"2023-11-16T16:47:43.621932+0000","last_fullsized":"2023-11-16T16:47:43.621932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:19:21.932819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947947+0000","last_change":"2023-11-16T16:47:43.947947+0000","last_active":"2023-11-16T16:47:43.947947+0000","last_peered":"2023-11-16T16:47:43.947947+0000","last_clean":"2023-11-16T16:47:43.947947+0000","last_became_active":"2023-11-16T16:47:43.947478+0000","last_became_peered":"2023-11-16T16:47:43.947478+0000","last_unstale":"2023-11-16T16:47:43.947947+0000","last_undegraded":"2023-11-16T16:47:43.947947+0000","last_fullsized":"2023-11-16T16:47:43.947947+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:40:14.254873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.632657+0000","last_change":"2023-11-16T16:47:43.632657+0000","last_active":"2023-11-16T16:47:43.632657+0000","last_peered":"2023-11-16T16:47:43.632657+0000","last_clean":"2023-11-16T16:47:43.632657+0000","last_became_active":"2023-11-16T16:47:43.631573+0000","last_became_peered":"2023-11-16T16:47:43.631573+0000","last_unstale":"2023-11-16T16:47:43.632657+0000","last_undegraded":"2023-11-16T16:47:43.632657+0000","last_fullsized":"2023-11-16T16:47:43.632657+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:45:08.317787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629418+0000","last_change":"2023-11-16T16:47:22.661639+0000","last_active":"2023-11-16T16:47:43.629418+0000","last_peered":"2023-11-16T16:47:43.629418+0000","last_clean":"2023-11-16T16:47:43.629418+0000","last_became_active":"2023-11-16T16:47:22.661230+0000","last_became_peered":"2023-11-16T16:47:22.661230+0000","last_unstale":"2023-11-16T16:47:43.629418+0000","last_undegraded":"2023-11-16T16:47:43.629418+0000","last_fullsized":"2023-11-16T16:47:43.629418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:04:16.909613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"82'3","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630143+0000","last_change":"2023-11-16T16:47:17.638209+0000","last_active":"2023-11-16T16:47:43.630143+0000","last_peered":"2023-11-16T16:47:43.630143+0000","last_clean":"2023-11-16T16:47:43.630143+0000","last_became_active":"2023-11-16T16:47:17.637957+0000","last_became_peered":"2023-11-16T16:47:17.637957+0000","last_unstale":"2023-11-16T16:47:43.630143+0000","last_undegraded":"2023-11-16T16:47:43.630143+0000","last_fullsized":"2023-11-16T16:47:43.630143+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:08:43.599916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947142+0000","last_change":"2023-11-16T16:47:18.182693+0000","last_active":"2023-11-16T16:47:43.947142+0000","last_peered":"2023-11-16T16:47:43.947142+0000","last_clean":"2023-11-16T16:47:43.947142+0000","last_became_active":"2023-11-16T16:47:18.182370+0000","last_became_peered":"2023-11-16T16:47:18.182370+0000","last_unstale":"2023-11-16T16:47:43.947142+0000","last_undegraded":"2023-11-16T16:47:43.947142+0000","last_fullsized":"2023-11-16T16:47:43.947142+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:38:49.650460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.621838+0000","last_change":"2023-11-16T16:47:22.649348+0000","last_active":"2023-11-16T16:47:43.621838+0000","last_peered":"2023-11-16T16:47:43.621838+0000","last_clean":"2023-11-16T16:47:43.621838+0000","last_became_active":"2023-11-16T16:47:22.648860+0000","last_became_peered":"2023-11-16T16:47:22.648860+0000","last_unstale":"2023-11-16T16:47:43.621838+0000","last_undegraded":"2023-11-16T16:47:43.621838+0000","last_fullsized":"2023-11-16T16:47:43.621838+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:45:18.960009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'31","reported_seq":167,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:48:39.252245+0000","last_change":"2023-11-16T16:47:17.619417+0000","last_active":"2023-11-16T16:48:39.252245+0000","last_peered":"2023-11-16T16:48:39.252245+0000","last_clean":"2023-11-16T16:48:39.252245+0000","last_became_active":"2023-11-16T16:47:17.619086+0000","last_became_peered":"2023-11-16T16:47:17.619086+0000","last_unstale":"2023-11-16T16:48:39.252245+0000","last_undegraded":"2023-11-16T16:48:39.252245+0000","last_fullsized":"2023-11-16T16:48:39.252245+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:27:30.044686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60908,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":110,"num_read_kb":116,"num_write":31,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625947+0000","last_change":"2023-11-16T16:47:18.177537+0000","last_active":"2023-11-16T16:47:43.625947+0000","last_peered":"2023-11-16T16:47:43.625947+0000","last_clean":"2023-11-16T16:47:43.625947+0000","last_became_active":"2023-11-16T16:47:18.177112+0000","last_became_peered":"2023-11-16T16:47:18.177112+0000","last_unstale":"2023-11-16T16:47:43.625947+0000","last_undegraded":"2023-11-16T16:47:43.625947+0000","last_fullsized":"2023-11-16T16:47:43.625947+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:17:18.541523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630846+0000","last_change":"2023-11-16T16:47:22.658086+0000","last_active":"2023-11-16T16:47:43.630846+0000","last_peered":"2023-11-16T16:47:43.630846+0000","last_clean":"2023-11-16T16:47:43.630846+0000","last_became_active":"2023-11-16T16:47:22.657725+0000","last_became_peered":"2023-11-16T16:47:22.657725+0000","last_unstale":"2023-11-16T16:47:43.630846+0000","last_undegraded":"2023-11-16T16:47:43.630846+0000","last_fullsized":"2023-11-16T16:47:43.630846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:53:51.729635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"82'2","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622837+0000","last_change":"2023-11-16T16:47:17.625405+0000","last_active":"2023-11-16T16:47:43.622837+0000","last_peered":"2023-11-16T16:47:43.622837+0000","last_clean":"2023-11-16T16:47:43.622837+0000","last_became_active":"2023-11-16T16:47:17.625026+0000","last_became_peered":"2023-11-16T16:47:17.625026+0000","last_unstale":"2023-11-16T16:47:43.622837+0000","last_undegraded":"2023-11-16T16:47:43.622837+0000","last_fullsized":"2023-11-16T16:47:43.622837+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:45:14.505117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629372+0000","last_change":"2023-11-16T16:47:18.196345+0000","last_active":"2023-11-16T16:47:43.629372+0000","last_peered":"2023-11-16T16:47:43.629372+0000","last_clean":"2023-11-16T16:47:43.629372+0000","last_became_active":"2023-11-16T16:47:18.196040+0000","last_became_peered":"2023-11-16T16:47:18.196040+0000","last_unstale":"2023-11-16T16:47:43.629372+0000","last_undegraded":"2023-11-16T16:47:43.629372+0000","last_fullsized":"2023-11-16T16:47:43.629372+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:07:42.452837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.624832+0000","last_change":"2023-11-16T16:47:22.660955+0000","last_active":"2023-11-16T16:47:43.624832+0000","last_peered":"2023-11-16T16:47:43.624832+0000","last_clean":"2023-11-16T16:47:43.624832+0000","last_became_active":"2023-11-16T16:47:22.660574+0000","last_became_peered":"2023-11-16T16:47:22.660574+0000","last_unstale":"2023-11-16T16:47:43.624832+0000","last_undegraded":"2023-11-16T16:47:43.624832+0000","last_fullsized":"2023-11-16T16:47:43.624832+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:41:43.745744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630317+0000","last_change":"2023-11-16T16:47:17.630481+0000","last_active":"2023-11-16T16:47:43.630317+0000","last_peered":"2023-11-16T16:47:43.630317+0000","last_clean":"2023-11-16T16:47:43.630317+0000","last_became_active":"2023-11-16T16:47:17.630190+0000","last_became_peered":"2023-11-16T16:47:17.630190+0000","last_unstale":"2023-11-16T16:47:43.630317+0000","last_undegraded":"2023-11-16T16:47:43.630317+0000","last_fullsized":"2023-11-16T16:47:43.630317+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:03:46.177268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622893+0000","last_change":"2023-11-16T16:47:18.179833+0000","last_active":"2023-11-16T16:47:43.622893+0000","last_peered":"2023-11-16T16:47:43.622893+0000","last_clean":"2023-11-16T16:47:43.622893+0000","last_became_active":"2023-11-16T16:47:18.179498+0000","last_became_peered":"2023-11-16T16:47:18.179498+0000","last_unstale":"2023-11-16T16:47:43.622893+0000","last_undegraded":"2023-11-16T16:47:43.622893+0000","last_fullsized":"2023-11-16T16:47:43.622893+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:29:41.782619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622702+0000","last_change":"2023-11-16T16:47:22.656312+0000","last_active":"2023-11-16T16:47:43.622702+0000","last_peered":"2023-11-16T16:47:43.622702+0000","last_clean":"2023-11-16T16:47:43.622702+0000","last_became_active":"2023-11-16T16:47:22.655870+0000","last_became_peered":"2023-11-16T16:47:22.655870+0000","last_unstale":"2023-11-16T16:47:43.622702+0000","last_undegraded":"2023-11-16T16:47:43.622702+0000","last_fullsized":"2023-11-16T16:47:43.622702+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:42:39.568364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629185+0000","last_change":"2023-11-16T16:47:17.622648+0000","last_active":"2023-11-16T16:47:43.629185+0000","last_peered":"2023-11-16T16:47:43.629185+0000","last_clean":"2023-11-16T16:47:43.629185+0000","last_became_active":"2023-11-16T16:47:17.622329+0000","last_became_peered":"2023-11-16T16:47:17.622329+0000","last_unstale":"2023-11-16T16:47:43.629185+0000","last_undegraded":"2023-11-16T16:47:43.629185+0000","last_fullsized":"2023-11-16T16:47:43.629185+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:01:26.211530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622424+0000","last_change":"2023-11-16T16:47:18.179712+0000","last_active":"2023-11-16T16:47:43.622424+0000","last_peered":"2023-11-16T16:47:43.622424+0000","last_clean":"2023-11-16T16:47:43.622424+0000","last_became_active":"2023-11-16T16:47:18.179392+0000","last_became_peered":"2023-11-16T16:47:18.179392+0000","last_unstale":"2023-11-16T16:47:43.622424+0000","last_undegraded":"2023-11-16T16:47:43.622424+0000","last_fullsized":"2023-11-16T16:47:43.622424+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:56:44.447970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.631211+0000","last_change":"2023-11-16T16:47:22.665088+0000","last_active":"2023-11-16T16:47:43.631211+0000","last_peered":"2023-11-16T16:47:43.631211+0000","last_clean":"2023-11-16T16:47:43.631211+0000","last_became_active":"2023-11-16T16:47:22.662802+0000","last_became_peered":"2023-11-16T16:47:22.662802+0000","last_unstale":"2023-11-16T16:47:43.631211+0000","last_undegraded":"2023-11-16T16:47:43.631211+0000","last_fullsized":"2023-11-16T16:47:43.631211+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:51:55.859193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"82'8","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.624624+0000","last_change":"2023-11-16T16:47:17.639246+0000","last_active":"2023-11-16T16:47:43.624624+0000","last_peered":"2023-11-16T16:47:43.624624+0000","last_clean":"2023-11-16T16:47:43.624624+0000","last_became_active":"2023-11-16T16:47:17.638928+0000","last_became_peered":"2023-11-16T16:47:17.638928+0000","last_unstale":"2023-11-16T16:47:43.624624+0000","last_undegraded":"2023-11-16T16:47:43.624624+0000","last_fullsized":"2023-11-16T16:47:43.624624+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:18:41.175512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629991+0000","last_change":"2023-11-16T16:47:18.182609+0000","last_active":"2023-11-16T16:47:43.629991+0000","last_peered":"2023-11-16T16:47:43.629991+0000","last_clean":"2023-11-16T16:47:43.629991+0000","last_became_active":"2023-11-16T16:47:18.182260+0000","last_became_peered":"2023-11-16T16:47:18.182260+0000","last_unstale":"2023-11-16T16:47:43.629991+0000","last_undegraded":"2023-11-16T16:47:43.629991+0000","last_fullsized":"2023-11-16T16:47:43.629991+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:14:14.972367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629794+0000","last_change":"2023-11-16T16:47:22.647586+0000","last_active":"2023-11-16T16:47:43.629794+0000","last_peered":"2023-11-16T16:47:43.629794+0000","last_clean":"2023-11-16T16:47:43.629794+0000","last_became_active":"2023-11-16T16:47:22.647187+0000","last_became_peered":"2023-11-16T16:47:22.647187+0000","last_unstale":"2023-11-16T16:47:43.629794+0000","last_undegraded":"2023-11-16T16:47:43.629794+0000","last_fullsized":"2023-11-16T16:47:43.629794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:01:47.702590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.595597+0000","last_change":"2023-11-16T16:47:43.933501+0000","last_active":"2023-11-16T16:47:44.595597+0000","last_peered":"2023-11-16T16:47:44.595597+0000","last_clean":"2023-11-16T16:47:44.595597+0000","last_became_active":"2023-11-16T16:47:43.631746+0000","last_became_peered":"2023-11-16T16:47:43.631746+0000","last_unstale":"2023-11-16T16:47:44.595597+0000","last_undegraded":"2023-11-16T16:47:44.595597+0000","last_fullsized":"2023-11-16T16:47:44.595597+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:11:10.737388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625043+0000","last_change":"2023-11-16T16:47:18.175210+0000","last_active":"2023-11-16T16:47:43.625043+0000","last_peered":"2023-11-16T16:47:43.625043+0000","last_clean":"2023-11-16T16:47:43.625043+0000","last_became_active":"2023-11-16T16:47:18.174926+0000","last_became_peered":"2023-11-16T16:47:18.174926+0000","last_unstale":"2023-11-16T16:47:43.625043+0000","last_undegraded":"2023-11-16T16:47:43.625043+0000","last_fullsized":"2023-11-16T16:47:43.625043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:31:59.526898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.624874+0000","last_change":"2023-11-16T16:47:22.719361+0000","last_active":"2023-11-16T16:47:43.624874+0000","last_peered":"2023-11-16T16:47:43.624874+0000","last_clean":"2023-11-16T16:47:43.624874+0000","last_became_active":"2023-11-16T16:47:22.718936+0000","last_became_peered":"2023-11-16T16:47:22.718936+0000","last_unstale":"2023-11-16T16:47:43.624874+0000","last_undegraded":"2023-11-16T16:47:43.624874+0000","last_fullsized":"2023-11-16T16:47:43.624874+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:18:52.706386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622234+0000","last_change":"2023-11-16T16:47:17.632623+0000","last_active":"2023-11-16T16:47:43.622234+0000","last_peered":"2023-11-16T16:47:43.622234+0000","last_clean":"2023-11-16T16:47:43.622234+0000","last_became_active":"2023-11-16T16:47:17.631323+0000","last_became_peered":"2023-11-16T16:47:17.631323+0000","last_unstale":"2023-11-16T16:47:43.622234+0000","last_undegraded":"2023-11-16T16:47:43.622234+0000","last_fullsized":"2023-11-16T16:47:43.622234+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:39:00.464312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626526+0000","last_change":"2023-11-16T16:47:18.190779+0000","last_active":"2023-11-16T16:47:43.626526+0000","last_peered":"2023-11-16T16:47:43.626526+0000","last_clean":"2023-11-16T16:47:43.626526+0000","last_became_active":"2023-11-16T16:47:18.190440+0000","last_became_peered":"2023-11-16T16:47:18.190440+0000","last_unstale":"2023-11-16T16:47:43.626526+0000","last_undegraded":"2023-11-16T16:47:43.626526+0000","last_fullsized":"2023-11-16T16:47:43.626526+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:41:07.686317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630403+0000","last_change":"2023-11-16T16:47:22.664847+0000","last_active":"2023-11-16T16:47:43.630403+0000","last_peered":"2023-11-16T16:47:43.630403+0000","last_clean":"2023-11-16T16:47:43.630403+0000","last_became_active":"2023-11-16T16:47:22.659382+0000","last_became_peered":"2023-11-16T16:47:22.659382+0000","last_unstale":"2023-11-16T16:47:43.630403+0000","last_undegraded":"2023-11-16T16:47:43.630403+0000","last_fullsized":"2023-11-16T16:47:43.630403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:11:46.749829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622597+0000","last_change":"2023-11-16T16:47:17.630956+0000","last_active":"2023-11-16T16:47:43.622597+0000","last_peered":"2023-11-16T16:47:43.622597+0000","last_clean":"2023-11-16T16:47:43.622597+0000","last_became_active":"2023-11-16T16:47:17.629864+0000","last_became_peered":"2023-11-16T16:47:17.629864+0000","last_unstale":"2023-11-16T16:47:43.622597+0000","last_undegraded":"2023-11-16T16:47:43.622597+0000","last_fullsized":"2023-11-16T16:47:43.622597+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:01:12.432152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626055+0000","last_change":"2023-11-16T16:47:18.183375+0000","last_active":"2023-11-16T16:47:43.626055+0000","last_peered":"2023-11-16T16:47:43.626055+0000","last_clean":"2023-11-16T16:47:43.626055+0000","last_became_active":"2023-11-16T16:47:18.183050+0000","last_became_peered":"2023-11-16T16:47:18.183050+0000","last_unstale":"2023-11-16T16:47:43.626055+0000","last_undegraded":"2023-11-16T16:47:43.626055+0000","last_fullsized":"2023-11-16T16:47:43.626055+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:27:13.855960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630861+0000","last_change":"2023-11-16T16:47:22.719109+0000","last_active":"2023-11-16T16:47:43.630861+0000","last_peered":"2023-11-16T16:47:43.630861+0000","last_clean":"2023-11-16T16:47:43.630861+0000","last_became_active":"2023-11-16T16:47:22.718680+0000","last_became_peered":"2023-11-16T16:47:22.718680+0000","last_unstale":"2023-11-16T16:47:43.630861+0000","last_undegraded":"2023-11-16T16:47:43.630861+0000","last_fullsized":"2023-11-16T16:47:43.630861+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:36:16.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629990+0000","last_change":"2023-11-16T16:47:18.181632+0000","last_active":"2023-11-16T16:47:43.629990+0000","last_peered":"2023-11-16T16:47:43.629990+0000","last_clean":"2023-11-16T16:47:43.629990+0000","last_became_active":"2023-11-16T16:47:18.181323+0000","last_became_peered":"2023-11-16T16:47:18.181323+0000","last_unstale":"2023-11-16T16:47:43.629990+0000","last_undegraded":"2023-11-16T16:47:43.629990+0000","last_fullsized":"2023-11-16T16:47:43.629990+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:29:17.626827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629025+0000","last_change":"2023-11-16T16:47:17.637269+0000","last_active":"2023-11-16T16:47:43.629025+0000","last_peered":"2023-11-16T16:47:43.629025+0000","last_clean":"2023-11-16T16:47:43.629025+0000","last_became_active":"2023-11-16T16:47:17.636881+0000","last_became_peered":"2023-11-16T16:47:17.636881+0000","last_unstale":"2023-11-16T16:47:43.629025+0000","last_undegraded":"2023-11-16T16:47:43.629025+0000","last_fullsized":"2023-11-16T16:47:43.629025+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:43.798391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623215+0000","last_change":"2023-11-16T16:47:22.659528+0000","last_active":"2023-11-16T16:47:43.623215+0000","last_peered":"2023-11-16T16:47:43.623215+0000","last_clean":"2023-11-16T16:47:43.623215+0000","last_became_active":"2023-11-16T16:47:22.659076+0000","last_became_peered":"2023-11-16T16:47:22.659076+0000","last_unstale":"2023-11-16T16:47:43.623215+0000","last_undegraded":"2023-11-16T16:47:43.623215+0000","last_fullsized":"2023-11-16T16:47:43.623215+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:06:05.562183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970448+0000","last_change":"2023-11-16T16:47:18.182102+0000","last_active":"2023-11-16T16:47:43.970448+0000","last_peered":"2023-11-16T16:47:43.970448+0000","last_clean":"2023-11-16T16:47:43.970448+0000","last_became_active":"2023-11-16T16:47:18.181528+0000","last_became_peered":"2023-11-16T16:47:18.181528+0000","last_unstale":"2023-11-16T16:47:43.970448+0000","last_undegraded":"2023-11-16T16:47:43.970448+0000","last_fullsized":"2023-11-16T16:47:43.970448+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:28:03.200017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629674+0000","last_change":"2023-11-16T16:47:17.626655+0000","last_active":"2023-11-16T16:47:43.629674+0000","last_peered":"2023-11-16T16:47:43.629674+0000","last_clean":"2023-11-16T16:47:43.629674+0000","last_became_active":"2023-11-16T16:47:17.626329+0000","last_became_peered":"2023-11-16T16:47:17.626329+0000","last_unstale":"2023-11-16T16:47:43.629674+0000","last_undegraded":"2023-11-16T16:47:43.629674+0000","last_fullsized":"2023-11-16T16:47:43.629674+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:30:26.199727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622227+0000","last_change":"2023-11-16T16:46:11.943398+0000","last_active":"2023-11-16T16:47:43.622227+0000","last_peered":"2023-11-16T16:47:43.622227+0000","last_clean":"2023-11-16T16:47:43.622227+0000","last_became_active":"2023-11-16T16:46:11.942862+0000","last_became_peered":"2023-11-16T16:46:11.942862+0000","last_unstale":"2023-11-16T16:47:43.622227+0000","last_undegraded":"2023-11-16T16:47:43.622227+0000","last_fullsized":"2023-11-16T16:47:43.622227+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:43:03.109797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:43:03.109797+0000","last_clean_scrub_stamp":"2023-11-16T16:43:03.109797+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:14:00.412865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630080+0000","last_change":"2023-11-16T16:47:22.719963+0000","last_active":"2023-11-16T16:47:43.630080+0000","last_peered":"2023-11-16T16:47:43.630080+0000","last_clean":"2023-11-16T16:47:43.630080+0000","last_became_active":"2023-11-16T16:47:22.719506+0000","last_became_peered":"2023-11-16T16:47:22.719506+0000","last_unstale":"2023-11-16T16:47:43.630080+0000","last_undegraded":"2023-11-16T16:47:43.630080+0000","last_fullsized":"2023-11-16T16:47:43.630080+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:27:40.361521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.628810+0000","last_change":"2023-11-16T16:47:18.184630+0000","last_active":"2023-11-16T16:47:43.628810+0000","last_peered":"2023-11-16T16:47:43.628810+0000","last_clean":"2023-11-16T16:47:43.628810+0000","last_became_active":"2023-11-16T16:47:18.181978+0000","last_became_peered":"2023-11-16T16:47:18.181978+0000","last_unstale":"2023-11-16T16:47:43.628810+0000","last_undegraded":"2023-11-16T16:47:43.628810+0000","last_fullsized":"2023-11-16T16:47:43.628810+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:22:39.294919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.635963+0000","last_change":"2023-11-16T16:47:43.635365+0000","last_active":"2023-11-16T16:47:43.635963+0000","last_peered":"2023-11-16T16:47:43.635963+0000","last_clean":"2023-11-16T16:47:43.635963+0000","last_became_active":"2023-11-16T16:47:43.634974+0000","last_became_peered":"2023-11-16T16:47:43.634974+0000","last_unstale":"2023-11-16T16:47:43.635963+0000","last_undegraded":"2023-11-16T16:47:43.635963+0000","last_fullsized":"2023-11-16T16:47:43.635963+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:40:21.789398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625072+0000","last_change":"2023-11-16T16:47:22.657133+0000","last_active":"2023-11-16T16:47:43.625072+0000","last_peered":"2023-11-16T16:47:43.625072+0000","last_clean":"2023-11-16T16:47:43.625072+0000","last_became_active":"2023-11-16T16:47:22.656557+0000","last_became_peered":"2023-11-16T16:47:22.656557+0000","last_unstale":"2023-11-16T16:47:43.625072+0000","last_undegraded":"2023-11-16T16:47:43.625072+0000","last_fullsized":"2023-11-16T16:47:43.625072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:06:56.883403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626111+0000","last_change":"2023-11-16T16:47:17.639107+0000","last_active":"2023-11-16T16:47:43.626111+0000","last_peered":"2023-11-16T16:47:43.626111+0000","last_clean":"2023-11-16T16:47:43.626111+0000","last_became_active":"2023-11-16T16:47:17.638796+0000","last_became_peered":"2023-11-16T16:47:17.638796+0000","last_unstale":"2023-11-16T16:47:43.626111+0000","last_undegraded":"2023-11-16T16:47:43.626111+0000","last_fullsized":"2023-11-16T16:47:43.626111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:15:27.137682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629965+0000","last_change":"2023-11-16T16:47:18.186761+0000","last_active":"2023-11-16T16:47:43.629965+0000","last_peered":"2023-11-16T16:47:43.629965+0000","last_clean":"2023-11-16T16:47:43.629965+0000","last_became_active":"2023-11-16T16:47:18.185594+0000","last_became_peered":"2023-11-16T16:47:18.185594+0000","last_unstale":"2023-11-16T16:47:43.629965+0000","last_undegraded":"2023-11-16T16:47:43.629965+0000","last_fullsized":"2023-11-16T16:47:43.629965+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:30:05.305376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970230+0000","last_change":"2023-11-16T16:47:22.719534+0000","last_active":"2023-11-16T16:47:43.970230+0000","last_peered":"2023-11-16T16:47:43.970230+0000","last_clean":"2023-11-16T16:47:43.970230+0000","last_became_active":"2023-11-16T16:47:22.718738+0000","last_became_peered":"2023-11-16T16:47:22.718738+0000","last_unstale":"2023-11-16T16:47:43.970230+0000","last_undegraded":"2023-11-16T16:47:43.970230+0000","last_fullsized":"2023-11-16T16:47:43.970230+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:14:30.912366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'4","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:57.089897+0000","last_change":"2023-11-16T16:47:17.630318+0000","last_active":"2023-11-16T16:47:57.089897+0000","last_peered":"2023-11-16T16:47:57.089897+0000","last_clean":"2023-11-16T16:47:57.089897+0000","last_became_active":"2023-11-16T16:47:17.629035+0000","last_became_peered":"2023-11-16T16:47:17.629035+0000","last_unstale":"2023-11-16T16:47:57.089897+0000","last_undegraded":"2023-11-16T16:47:57.089897+0000","last_fullsized":"2023-11-16T16:47:57.089897+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:44:38.576547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970772+0000","last_change":"2023-11-16T16:47:18.185034+0000","last_active":"2023-11-16T16:47:43.970772+0000","last_peered":"2023-11-16T16:47:43.970772+0000","last_clean":"2023-11-16T16:47:43.970772+0000","last_became_active":"2023-11-16T16:47:18.181961+0000","last_became_peered":"2023-11-16T16:47:18.181961+0000","last_unstale":"2023-11-16T16:47:43.970772+0000","last_undegraded":"2023-11-16T16:47:43.970772+0000","last_fullsized":"2023-11-16T16:47:43.970772+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:38:55.393578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622320+0000","last_change":"2023-11-16T16:47:22.649952+0000","last_active":"2023-11-16T16:47:43.622320+0000","last_peered":"2023-11-16T16:47:43.622320+0000","last_clean":"2023-11-16T16:47:43.622320+0000","last_became_active":"2023-11-16T16:47:22.649481+0000","last_became_peered":"2023-11-16T16:47:22.649481+0000","last_unstale":"2023-11-16T16:47:43.622320+0000","last_undegraded":"2023-11-16T16:47:43.622320+0000","last_fullsized":"2023-11-16T16:47:43.622320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:36:07.109965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"82'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629922+0000","last_change":"2023-11-16T16:47:17.631982+0000","last_active":"2023-11-16T16:47:43.629922+0000","last_peered":"2023-11-16T16:47:43.629922+0000","last_clean":"2023-11-16T16:47:43.629922+0000","last_became_active":"2023-11-16T16:47:17.631552+0000","last_became_peered":"2023-11-16T16:47:17.631552+0000","last_unstale":"2023-11-16T16:47:43.629922+0000","last_undegraded":"2023-11-16T16:47:43.629922+0000","last_fullsized":"2023-11-16T16:47:43.629922+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:35:56.360671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626666+0000","last_change":"2023-11-16T16:47:18.197325+0000","last_active":"2023-11-16T16:47:43.626666+0000","last_peered":"2023-11-16T16:47:43.626666+0000","last_clean":"2023-11-16T16:47:43.626666+0000","last_became_active":"2023-11-16T16:47:18.190602+0000","last_became_peered":"2023-11-16T16:47:18.190602+0000","last_unstale":"2023-11-16T16:47:43.626666+0000","last_undegraded":"2023-11-16T16:47:43.626666+0000","last_fullsized":"2023-11-16T16:47:43.626666+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:28:40.186656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629554+0000","last_change":"2023-11-16T16:47:22.720023+0000","last_active":"2023-11-16T16:47:43.629554+0000","last_peered":"2023-11-16T16:47:43.629554+0000","last_clean":"2023-11-16T16:47:43.629554+0000","last_became_active":"2023-11-16T16:47:22.719561+0000","last_became_peered":"2023-11-16T16:47:22.719561+0000","last_unstale":"2023-11-16T16:47:43.629554+0000","last_undegraded":"2023-11-16T16:47:43.629554+0000","last_fullsized":"2023-11-16T16:47:43.629554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:40:41.808974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.946763+0000","last_change":"2023-11-16T16:47:17.629379+0000","last_active":"2023-11-16T16:47:43.946763+0000","last_peered":"2023-11-16T16:47:43.946763+0000","last_clean":"2023-11-16T16:47:43.946763+0000","last_became_active":"2023-11-16T16:47:17.629025+0000","last_became_peered":"2023-11-16T16:47:17.629025+0000","last_unstale":"2023-11-16T16:47:43.946763+0000","last_undegraded":"2023-11-16T16:47:43.946763+0000","last_fullsized":"2023-11-16T16:47:43.946763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:23:29.948397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626265+0000","last_change":"2023-11-16T16:47:18.185418+0000","last_active":"2023-11-16T16:47:43.626265+0000","last_peered":"2023-11-16T16:47:43.626265+0000","last_clean":"2023-11-16T16:47:43.626265+0000","last_became_active":"2023-11-16T16:47:18.185092+0000","last_became_peered":"2023-11-16T16:47:18.185092+0000","last_unstale":"2023-11-16T16:47:43.626265+0000","last_undegraded":"2023-11-16T16:47:43.626265+0000","last_fullsized":"2023-11-16T16:47:43.626265+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:02:15.891150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622811+0000","last_change":"2023-11-16T16:47:22.650701+0000","last_active":"2023-11-16T16:47:43.622811+0000","last_peered":"2023-11-16T16:47:43.622811+0000","last_clean":"2023-11-16T16:47:43.622811+0000","last_became_active":"2023-11-16T16:47:22.648710+0000","last_became_peered":"2023-11-16T16:47:22.648710+0000","last_unstale":"2023-11-16T16:47:43.622811+0000","last_undegraded":"2023-11-16T16:47:43.622811+0000","last_fullsized":"2023-11-16T16:47:43.622811+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:50:37.989145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970985+0000","last_change":"2023-11-16T16:47:17.638622+0000","last_active":"2023-11-16T16:47:43.970985+0000","last_peered":"2023-11-16T16:47:43.970985+0000","last_clean":"2023-11-16T16:47:43.970985+0000","last_became_active":"2023-11-16T16:47:17.638303+0000","last_became_peered":"2023-11-16T16:47:17.638303+0000","last_unstale":"2023-11-16T16:47:43.970985+0000","last_undegraded":"2023-11-16T16:47:43.970985+0000","last_fullsized":"2023-11-16T16:47:43.970985+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:03:52.117172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626598+0000","last_change":"2023-11-16T16:47:18.173474+0000","last_active":"2023-11-16T16:47:43.626598+0000","last_peered":"2023-11-16T16:47:43.626598+0000","last_clean":"2023-11-16T16:47:43.626598+0000","last_became_active":"2023-11-16T16:47:18.173140+0000","last_became_peered":"2023-11-16T16:47:18.173140+0000","last_unstale":"2023-11-16T16:47:43.626598+0000","last_undegraded":"2023-11-16T16:47:43.626598+0000","last_fullsized":"2023-11-16T16:47:43.626598+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:13:23.724052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629397+0000","last_change":"2023-11-16T16:47:22.655806+0000","last_active":"2023-11-16T16:47:43.629397+0000","last_peered":"2023-11-16T16:47:43.629397+0000","last_clean":"2023-11-16T16:47:43.629397+0000","last_became_active":"2023-11-16T16:47:22.655385+0000","last_became_peered":"2023-11-16T16:47:22.655385+0000","last_unstale":"2023-11-16T16:47:43.629397+0000","last_undegraded":"2023-11-16T16:47:43.629397+0000","last_fullsized":"2023-11-16T16:47:43.629397+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:31:41.160819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622449+0000","last_change":"2023-11-16T16:47:17.631673+0000","last_active":"2023-11-16T16:47:43.622449+0000","last_peered":"2023-11-16T16:47:43.622449+0000","last_clean":"2023-11-16T16:47:43.622449+0000","last_became_active":"2023-11-16T16:47:17.631356+0000","last_became_peered":"2023-11-16T16:47:17.631356+0000","last_unstale":"2023-11-16T16:47:43.622449+0000","last_undegraded":"2023-11-16T16:47:43.622449+0000","last_fullsized":"2023-11-16T16:47:43.622449+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:29:12.540977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970538+0000","last_change":"2023-11-16T16:47:18.186706+0000","last_active":"2023-11-16T16:47:43.970538+0000","last_peered":"2023-11-16T16:47:43.970538+0000","last_clean":"2023-11-16T16:47:43.970538+0000","last_became_active":"2023-11-16T16:47:18.186360+0000","last_became_peered":"2023-11-16T16:47:18.186360+0000","last_unstale":"2023-11-16T16:47:43.970538+0000","last_undegraded":"2023-11-16T16:47:43.970538+0000","last_fullsized":"2023-11-16T16:47:43.970538+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:32:31.165136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629693+0000","last_change":"2023-11-16T16:47:22.719065+0000","last_active":"2023-11-16T16:47:43.629693+0000","last_peered":"2023-11-16T16:47:43.629693+0000","last_clean":"2023-11-16T16:47:43.629693+0000","last_became_active":"2023-11-16T16:47:22.718662+0000","last_became_peered":"2023-11-16T16:47:22.718662+0000","last_unstale":"2023-11-16T16:47:43.629693+0000","last_undegraded":"2023-11-16T16:47:43.629693+0000","last_fullsized":"2023-11-16T16:47:43.629693+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:56:10.235168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"82'2","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.638200+0000","last_change":"2023-11-16T16:47:43.637790+0000","last_active":"2023-11-16T16:47:43.638200+0000","last_peered":"2023-11-16T16:47:43.638200+0000","last_clean":"2023-11-16T16:47:43.638200+0000","last_became_active":"2023-11-16T16:47:43.632225+0000","last_became_peered":"2023-11-16T16:47:43.632225+0000","last_unstale":"2023-11-16T16:47:43.638200+0000","last_undegraded":"2023-11-16T16:47:43.638200+0000","last_fullsized":"2023-11-16T16:47:43.638200+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:31:01.698264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622703+0000","last_change":"2023-11-16T16:47:18.180999+0000","last_active":"2023-11-16T16:47:43.622703+0000","last_peered":"2023-11-16T16:47:43.622703+0000","last_clean":"2023-11-16T16:47:43.622703+0000","last_became_active":"2023-11-16T16:47:18.180724+0000","last_became_peered":"2023-11-16T16:47:18.180724+0000","last_unstale":"2023-11-16T16:47:43.622703+0000","last_undegraded":"2023-11-16T16:47:43.622703+0000","last_fullsized":"2023-11-16T16:47:43.622703+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:38:44.147293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947186+0000","last_change":"2023-11-16T16:47:22.665607+0000","last_active":"2023-11-16T16:47:43.947186+0000","last_peered":"2023-11-16T16:47:43.947186+0000","last_clean":"2023-11-16T16:47:43.947186+0000","last_became_active":"2023-11-16T16:47:22.665273+0000","last_became_peered":"2023-11-16T16:47:22.665273+0000","last_unstale":"2023-11-16T16:47:43.947186+0000","last_undegraded":"2023-11-16T16:47:43.947186+0000","last_fullsized":"2023-11-16T16:47:43.947186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:08:37.804057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622937+0000","last_change":"2023-11-16T16:47:17.632614+0000","last_active":"2023-11-16T16:47:43.622937+0000","last_peered":"2023-11-16T16:47:43.622937+0000","last_clean":"2023-11-16T16:47:43.622937+0000","last_became_active":"2023-11-16T16:47:17.632227+0000","last_became_peered":"2023-11-16T16:47:17.632227+0000","last_unstale":"2023-11-16T16:47:43.622937+0000","last_undegraded":"2023-11-16T16:47:43.622937+0000","last_fullsized":"2023-11-16T16:47:43.622937+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:58:13.465619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623388+0000","last_change":"2023-11-16T16:47:18.177607+0000","last_active":"2023-11-16T16:47:43.623388+0000","last_peered":"2023-11-16T16:47:43.623388+0000","last_clean":"2023-11-16T16:47:43.623388+0000","last_became_active":"2023-11-16T16:47:18.177266+0000","last_became_peered":"2023-11-16T16:47:18.177266+0000","last_unstale":"2023-11-16T16:47:43.623388+0000","last_undegraded":"2023-11-16T16:47:43.623388+0000","last_fullsized":"2023-11-16T16:47:43.623388+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:03:08.676930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625888+0000","last_change":"2023-11-16T16:47:22.659987+0000","last_active":"2023-11-16T16:47:43.625888+0000","last_peered":"2023-11-16T16:47:43.625888+0000","last_clean":"2023-11-16T16:47:43.625888+0000","last_became_active":"2023-11-16T16:47:22.658725+0000","last_became_peered":"2023-11-16T16:47:22.658725+0000","last_unstale":"2023-11-16T16:47:43.625888+0000","last_undegraded":"2023-11-16T16:47:43.625888+0000","last_fullsized":"2023-11-16T16:47:43.625888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:51:27.380649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630637+0000","last_change":"2023-11-16T16:47:17.640696+0000","last_active":"2023-11-16T16:47:43.630637+0000","last_peered":"2023-11-16T16:47:43.630637+0000","last_clean":"2023-11-16T16:47:43.630637+0000","last_became_active":"2023-11-16T16:47:17.640428+0000","last_became_peered":"2023-11-16T16:47:17.640428+0000","last_unstale":"2023-11-16T16:47:43.630637+0000","last_undegraded":"2023-11-16T16:47:43.630637+0000","last_fullsized":"2023-11-16T16:47:43.630637+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:47:12.331771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.972273+0000","last_change":"2023-11-16T16:47:43.972273+0000","last_active":"2023-11-16T16:47:43.972273+0000","last_peered":"2023-11-16T16:47:43.972273+0000","last_clean":"2023-11-16T16:47:43.972273+0000","last_became_active":"2023-11-16T16:47:43.971981+0000","last_became_peered":"2023-11-16T16:47:43.971981+0000","last_unstale":"2023-11-16T16:47:43.972273+0000","last_undegraded":"2023-11-16T16:47:43.972273+0000","last_fullsized":"2023-11-16T16:47:43.972273+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:38:01.080979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629465+0000","last_change":"2023-11-16T16:47:22.649436+0000","last_active":"2023-11-16T16:47:43.629465+0000","last_peered":"2023-11-16T16:47:43.629465+0000","last_clean":"2023-11-16T16:47:43.629465+0000","last_became_active":"2023-11-16T16:47:22.649034+0000","last_became_peered":"2023-11-16T16:47:22.649034+0000","last_unstale":"2023-11-16T16:47:43.629465+0000","last_undegraded":"2023-11-16T16:47:43.629465+0000","last_fullsized":"2023-11-16T16:47:43.629465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:50:06.438075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.631007+0000","last_change":"2023-11-16T16:47:17.634582+0000","last_active":"2023-11-16T16:47:43.631007+0000","last_peered":"2023-11-16T16:47:43.631007+0000","last_clean":"2023-11-16T16:47:43.631007+0000","last_became_active":"2023-11-16T16:47:17.634172+0000","last_became_peered":"2023-11-16T16:47:17.634172+0000","last_unstale":"2023-11-16T16:47:43.631007+0000","last_undegraded":"2023-11-16T16:47:43.631007+0000","last_fullsized":"2023-11-16T16:47:43.631007+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:50:55.582584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623419+0000","last_change":"2023-11-16T16:47:18.180466+0000","last_active":"2023-11-16T16:47:43.623419+0000","last_peered":"2023-11-16T16:47:43.623419+0000","last_clean":"2023-11-16T16:47:43.623419+0000","last_became_active":"2023-11-16T16:47:18.180166+0000","last_became_peered":"2023-11-16T16:47:18.180166+0000","last_unstale":"2023-11-16T16:47:43.623419+0000","last_undegraded":"2023-11-16T16:47:43.623419+0000","last_fullsized":"2023-11-16T16:47:43.623419+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:02:56.576134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626739+0000","last_change":"2023-11-16T16:47:18.179196+0000","last_active":"2023-11-16T16:47:43.626739+0000","last_peered":"2023-11-16T16:47:43.626739+0000","last_clean":"2023-11-16T16:47:43.626739+0000","last_became_active":"2023-11-16T16:47:18.178827+0000","last_became_peered":"2023-11-16T16:47:18.178827+0000","last_unstale":"2023-11-16T16:47:43.626739+0000","last_undegraded":"2023-11-16T16:47:43.626739+0000","last_fullsized":"2023-11-16T16:47:43.626739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:07:11.423464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622851+0000","last_change":"2023-11-16T16:47:18.184630+0000","last_active":"2023-11-16T16:47:43.622851+0000","last_peered":"2023-11-16T16:47:43.622851+0000","last_clean":"2023-11-16T16:47:43.622851+0000","last_became_active":"2023-11-16T16:47:18.184314+0000","last_became_peered":"2023-11-16T16:47:18.184314+0000","last_unstale":"2023-11-16T16:47:43.622851+0000","last_undegraded":"2023-11-16T16:47:43.622851+0000","last_fullsized":"2023-11-16T16:47:43.622851+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:56:13.651803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.622558+0000","last_change":"2023-11-16T16:47:18.184255+0000","last_active":"2023-11-16T16:47:43.622558+0000","last_peered":"2023-11-16T16:47:43.622558+0000","last_clean":"2023-11-16T16:47:43.622558+0000","last_became_active":"2023-11-16T16:47:18.183936+0000","last_became_peered":"2023-11-16T16:47:18.183936+0000","last_unstale":"2023-11-16T16:47:43.622558+0000","last_undegraded":"2023-11-16T16:47:43.622558+0000","last_fullsized":"2023-11-16T16:47:43.622558+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:59:41.136565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.626136+0000","last_change":"2023-11-16T16:47:18.196331+0000","last_active":"2023-11-16T16:47:43.626136+0000","last_peered":"2023-11-16T16:47:43.626136+0000","last_clean":"2023-11-16T16:47:43.626136+0000","last_became_active":"2023-11-16T16:47:18.195954+0000","last_became_peered":"2023-11-16T16:47:18.195954+0000","last_unstale":"2023-11-16T16:47:43.626136+0000","last_undegraded":"2023-11-16T16:47:43.626136+0000","last_fullsized":"2023-11-16T16:47:43.626136+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:31:23.208122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629719+0000","last_change":"2023-11-16T16:47:18.190537+0000","last_active":"2023-11-16T16:47:43.629719+0000","last_peered":"2023-11-16T16:47:43.629719+0000","last_clean":"2023-11-16T16:47:43.629719+0000","last_became_active":"2023-11-16T16:47:18.190118+0000","last_became_peered":"2023-11-16T16:47:18.190118+0000","last_unstale":"2023-11-16T16:47:43.629719+0000","last_undegraded":"2023-11-16T16:47:43.629719+0000","last_fullsized":"2023-11-16T16:47:43.629719+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:39:38.125056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.625721+0000","last_change":"2023-11-16T16:47:18.179215+0000","last_active":"2023-11-16T16:47:43.625721+0000","last_peered":"2023-11-16T16:47:43.625721+0000","last_clean":"2023-11-16T16:47:43.625721+0000","last_became_active":"2023-11-16T16:47:18.178759+0000","last_became_peered":"2023-11-16T16:47:18.178759+0000","last_unstale":"2023-11-16T16:47:43.625721+0000","last_undegraded":"2023-11-16T16:47:43.625721+0000","last_fullsized":"2023-11-16T16:47:43.625721+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:06:54.886510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.623599+0000","last_change":"2023-11-16T16:47:18.179588+0000","last_active":"2023-11-16T16:47:43.623599+0000","last_peered":"2023-11-16T16:47:43.623599+0000","last_clean":"2023-11-16T16:47:43.623599+0000","last_became_active":"2023-11-16T16:47:18.179294+0000","last_became_peered":"2023-11-16T16:47:18.179294+0000","last_unstale":"2023-11-16T16:47:43.623599+0000","last_undegraded":"2023-11-16T16:47:43.623599+0000","last_fullsized":"2023-11-16T16:47:43.623599+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:02:55.079560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.971128+0000","last_change":"2023-11-16T16:47:18.181587+0000","last_active":"2023-11-16T16:47:43.971128+0000","last_peered":"2023-11-16T16:47:43.971128+0000","last_clean":"2023-11-16T16:47:43.971128+0000","last_became_active":"2023-11-16T16:47:18.181203+0000","last_became_peered":"2023-11-16T16:47:18.181203+0000","last_unstale":"2023-11-16T16:47:43.971128+0000","last_undegraded":"2023-11-16T16:47:43.971128+0000","last_fullsized":"2023-11-16T16:47:43.971128+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:13:49.378115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629816+0000","last_change":"2023-11-16T16:47:18.182690+0000","last_active":"2023-11-16T16:47:43.629816+0000","last_peered":"2023-11-16T16:47:43.629816+0000","last_clean":"2023-11-16T16:47:43.629816+0000","last_became_active":"2023-11-16T16:47:18.182361+0000","last_became_peered":"2023-11-16T16:47:18.182361+0000","last_unstale":"2023-11-16T16:47:43.629816+0000","last_undegraded":"2023-11-16T16:47:43.629816+0000","last_fullsized":"2023-11-16T16:47:43.629816+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:55:41.987600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.970993+0000","last_change":"2023-11-16T16:47:18.184599+0000","last_active":"2023-11-16T16:47:43.970993+0000","last_peered":"2023-11-16T16:47:43.970993+0000","last_clean":"2023-11-16T16:47:43.970993+0000","last_became_active":"2023-11-16T16:47:18.184288+0000","last_became_peered":"2023-11-16T16:47:18.184288+0000","last_unstale":"2023-11-16T16:47:43.970993+0000","last_undegraded":"2023-11-16T16:47:43.970993+0000","last_fullsized":"2023-11-16T16:47:43.970993+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:43:20.190571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.947382+0000","last_change":"2023-11-16T16:47:18.187267+0000","last_active":"2023-11-16T16:47:43.947382+0000","last_peered":"2023-11-16T16:47:43.947382+0000","last_clean":"2023-11-16T16:47:43.947382+0000","last_became_active":"2023-11-16T16:47:18.186984+0000","last_became_peered":"2023-11-16T16:47:18.186984+0000","last_unstale":"2023-11-16T16:47:43.947382+0000","last_undegraded":"2023-11-16T16:47:43.947382+0000","last_fullsized":"2023-11-16T16:47:43.947382+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:30:20.891991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.630154+0000","last_change":"2023-11-16T16:47:18.191566+0000","last_active":"2023-11-16T16:47:43.630154+0000","last_peered":"2023-11-16T16:47:43.630154+0000","last_clean":"2023-11-16T16:47:43.630154+0000","last_became_active":"2023-11-16T16:47:18.185342+0000","last_became_peered":"2023-11-16T16:47:18.185342+0000","last_unstale":"2023-11-16T16:47:43.630154+0000","last_undegraded":"2023-11-16T16:47:43.630154+0000","last_fullsized":"2023-11-16T16:47:43.630154+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:55:29.393440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-11-16T16:47:43.629636+0000","last_change":"2023-11-16T16:47:43.629636+0000","last_active":"2023-11-16T16:47:43.629636+0000","last_peered":"2023-11-16T16:47:43.629636+0000","last_clean":"2023-11-16T16:47:43.629636+0000","last_became_active":"2023-11-16T16:47:43.629179+0000","last_became_peered":"2023-11-16T16:47:43.629179+0000","last_unstale":"2023-11-16T16:47:43.629636+0000","last_undegraded":"2023-11-16T16:47:43.629636+0000","last_fullsized":"2023-11-16T16:47:43.629636+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:21:10.581239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281399,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61706,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":116,"num_read_kb":122,"num_write":76,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":185118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449118,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6140,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739012,"statfs":{"total":95995035648,"available":95988748288,"internally_reserved":0,"allocated":970752,"data_stored":487258,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51000000000000001,"15min":0.51400000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.55700000000000005,"15min":0.55100000000000005},"min":{"1min":0.38800000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58899999999999997,"15min":0.58599999999999997},"min":{"1min":0.40600000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81000000000000005,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.50900000000000001}]},{"osd":2,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45700000000000002,"15min":0.45500000000000002},"min":{"1min":0.313,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.38},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.46400000000000002,"15min":0.46500000000000002},"min":{"1min":0.34899999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.441}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.63,"15min":0.626},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.621,"15min":0.625},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.77000000000000002,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.64500000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.33500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.77100000000000002,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.59599999999999997,"15min":0.59799999999999998},"min":{"1min":0.378,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.87,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.48599999999999999}]},{"osd":6,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.67800000000000005,"15min":0.68999999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.89900000000000002,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.75700000000000001,"15min":0.72999999999999998},"min":{"1min":0.496,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":4.2510000000000003,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":0.61299999999999999}]},{"osd":7,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70999999999999996,"15min":0.71099999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.90800000000000003,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.72699999999999998,"15min":0.73399999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.86199999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.56499999999999995}]},{"osd":8,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.52300000000000002}]},{"osd":9,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.76800000000000002,"15min":0.76300000000000001},"min":{"1min":0.69199999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":0.92200000000000004,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.77100000000000002,"15min":0.77400000000000002},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":0.98199999999999998,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.57799999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76200000000000001,"15min":0.76700000000000002},"min":{"1min":0.58099999999999996,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.93500000000000005,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.59899999999999998,"5min":0.498,"15min":0.498},"max":{"1min":0.90400000000000003,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.72999999999999998}]}]},{"osd":10,"up_from":71,"seq":304942678050,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6516,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738636,"statfs":{"total":95995035648,"available":95988363264,"internally_reserved":0,"allocated":1093632,"data_stored":1134610,"data_compressed":53424,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.42299999999999999}]},{"osd":1,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.47099999999999997,"15min":0.48099999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.502,"15min":0.51400000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.622,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.378}]},{"osd":2,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65600000000000003,"15min":0.66800000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.90500000000000003,"5min":1.577,"15min":1.577},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65400000000000003,"15min":0.66300000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.83199999999999996,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.54800000000000004}]},{"osd":3,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64500000000000002,"15min":0.65200000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.67000000000000004,"15min":0.68000000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.84599999999999997,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.53500000000000003}]},{"osd":4,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.58199999999999996,"15min":0.58999999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.496},{"interface":"front","average":{"1min":0.496,"5min":0.56200000000000006,"15min":0.57299999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.56100000000000005}]},{"osd":5,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.69699999999999995,"15min":0.70399999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.90600000000000003,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.625},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.91400000000000003,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.69199999999999995}]},{"osd":6,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.75800000000000001,"15min":0.76700000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77600000000000002,"15min":0.77900000000000003},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.68300000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.69999999999999996}]},{"osd":8,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.84399999999999997,"15min":0.86499999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.052,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.77800000000000002,"15min":0.79100000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.93200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.66800000000000004}]},{"osd":9,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.86099999999999999,"15min":0.84699999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":4.0129999999999999,"5min":4.0129999999999999,"15min":4.0129999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.80600000000000005,"15min":0.82599999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.86799999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.751}]},{"osd":11,"last update":"Thu Nov 16 16:48:30 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.80700000000000005,"15min":0.82599999999999996},"min":{"1min":0.49199999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.83699999999999997,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.79900000000000004,"15min":0.80900000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.91200000000000003,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.63800000000000001}]}]},{"osd":9,"up_from":66,"seq":283467841573,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6272,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":974848,"data_stored":491914,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.56200000000000006,"15min":0.57599999999999996},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.68300000000000005,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.59399999999999997,"15min":0.61199999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.75700000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.497}]},{"osd":1,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.82199999999999995,"15min":0.83699999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.98599999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.622},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.77300000000000002,"15min":0.79100000000000004},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.83899999999999997,"5min":1.071,"15min":1.071},"last":0.64400000000000002}]},{"osd":2,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.82699999999999996,"15min":0.84199999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.96999999999999997,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.76800000000000002,"15min":0.78200000000000003},"min":{"1min":0.55300000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.93500000000000005,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.67600000000000005}]},{"osd":3,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.78900000000000003,"15min":0.81499999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94399999999999995,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.57099999999999995}]},{"osd":4,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.83999999999999997,"15min":0.86599999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.89800000000000002,"5min":1.298,"15min":1.298},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.76400000000000001,"15min":0.78200000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.85599999999999998,"5min":1.181,"15min":1.181},"last":0.63400000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.81100000000000005,"15min":0.83099999999999996},"min":{"1min":0.47999999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0860000000000001,"5min":1.268,"15min":1.268},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.93999999999999995,"15min":0.96599999999999997},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.032,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.75}]},{"osd":6,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":1.054},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.95599999999999996}]},{"osd":7,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.89400000000000002,"15min":0.91800000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.89200000000000002,"5min":1.405,"15min":1.405},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.81699999999999995,"5min":1.0409999999999999,"15min":1.079},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0740000000000001,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.65400000000000003}]},{"osd":8,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.94799999999999995,"15min":0.97599999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0269999999999999,"5min":1.363,"15min":1.363},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.95399999999999996,"15min":0.97999999999999998},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96999999999999997,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.79800000000000004}]},{"osd":10,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.92800000000000005,"15min":0.95199999999999996},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.016,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.85799999999999998,"5min":1.0109999999999999,"15min":1.0369999999999999},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.0960000000000001,"5min":1.714,"15min":1.714},"last":0.78100000000000003}]},{"osd":11,"last update":"Thu Nov 16 16:48:22 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.88},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.82499999999999996}]}]},{"osd":8,"up_from":58,"seq":249108103211,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6460,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738692,"statfs":{"total":95995035648,"available":95988420608,"internally_reserved":0,"allocated":970752,"data_stored":487818,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.83999999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.38,"5min":0.313,"15min":0.313},"max":{"1min":0.78600000000000003,"5min":1.113,"15min":1.113},"last":0.66500000000000004}]},{"osd":1,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.61699999999999999,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84799999999999998,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61399999999999999,"15min":0.60699999999999998},"min":{"1min":0.40600000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79200000000000004,"5min":1.177,"15min":1.177},"last":0.68300000000000005}]},{"osd":2,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.34799999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.60999999999999999,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.47199999999999998,"15min":0.46500000000000002},"min":{"1min":0.379,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63500000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53800000000000003}]},{"osd":3,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.49299999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.141,"15min":1.141},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.65900000000000003,"15min":0.64900000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.64400000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.63200000000000001,"15min":0.626},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.91100000000000003,"5min":0.999,"15min":0.999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69299999999999995,"15min":0.68700000000000006},"min":{"1min":0.58199999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.81599999999999995,"5min":1.242,"15min":1.242},"last":0.59899999999999998}]},{"osd":5,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55000000000000004,"15min":0.53400000000000003},"min":{"1min":0.40600000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71799999999999997,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.501},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.57399999999999995,"15min":0.56200000000000006},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75700000000000001,"5min":1.224,"15min":1.224},"last":0.624}]},{"osd":6,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.77300000000000002,"15min":0.73699999999999999},"min":{"1min":0.501,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":4.7309999999999999,"5min":4.7309999999999999,"15min":4.7309999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.68600000000000005,"15min":0.67800000000000005},"min":{"1min":0.52500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83599999999999997,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.88800000000000001}]},{"osd":7,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.70199999999999996,"15min":0.68999999999999995},"min":{"1min":0.59199999999999997,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88500000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.73199999999999998,"15min":0.72399999999999998},"min":{"1min":0.59299999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95299999999999996,"5min":1.198,"15min":1.198},"last":0.78000000000000003}]},{"osd":9,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77200000000000002,"15min":0.77100000000000002},"min":{"1min":0.52200000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92000000000000004,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.81200000000000006,"15min":0.81399999999999995},"min":{"1min":0.55900000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0069999999999999,"5min":1.298,"15min":1.298},"last":0.69799999999999995}]},{"osd":10,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.78000000000000003,"15min":0.77600000000000002},"min":{"1min":0.63700000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.022,"5min":1.222,"15min":1.222},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.82399999999999995,"15min":0.82299999999999995},"min":{"1min":0.65000000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.2350000000000001,"5min":1.274,"15min":1.274},"last":0.747}]},{"osd":11,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.85099999999999998}]}]},{"osd":7,"up_from":51,"seq":219043332143,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6576,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738576,"statfs":{"total":95995035648,"available":95988301824,"internally_reserved":0,"allocated":1024000,"data_stored":1082282,"data_compressed":53424,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.377,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81799999999999995,"5min":0.875,"15min":0.875},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.42299999999999999,"15min":0.40899999999999997},"min":{"1min":0.28699999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.60099999999999998,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.44900000000000001,"15min":0.44},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.59799999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Nov 16 16:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.59099999999999997,"15min":0.57999999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81200000000000006,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.59699999999999998,"15min":0.58099999999999996},"min":{"1min":0.46500000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78400000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.55100000000000005}]},{"osd":3,"last update":"Thu Nov 16 16:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57199999999999995,"15min":0.55400000000000005},"min":{"1min":0.39600000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.501},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.38800000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.84799999999999998,"5min":0.873,"15min":0.873},"last":0.53400000000000003}]},{"osd":4,"last update":"Thu Nov 16 16:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33700000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67200000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.72899999999999998,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.42199999999999999}]},{"osd":5,"last update":"Thu Nov 16 16:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.44500000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.84199999999999997,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65000000000000002,"15min":0.63400000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.89400000000000002,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.58299999999999996}]},{"osd":6,"last update":"Thu Nov 16 16:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.68799999999999994,"15min":0.67700000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.90700000000000003,"5min":1.004,"15min":1.004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66900000000000004,"15min":0.66800000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.85799999999999998,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.60099999999999998}]},{"osd":8,"last update":"Thu Nov 16 16:48:30 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.69699999999999995,"15min":0.67700000000000005},"min":{"1min":0.50600000000000001,"5min":0.436,"15min":0.436},"max":{"1min":0.95699999999999996,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69999999999999996,"15min":0.70199999999999996},"min":{"1min":0.47299999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.878,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.68400000000000005}]},{"osd":9,"last update":"Thu Nov 16 16:47:52 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.77200000000000002,"15min":0.78400000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.92800000000000005,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.748,"5min":0.80000000000000004,"15min":0.80800000000000005},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.93100000000000005,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.66400000000000003}]},{"osd":10,"last update":"Thu Nov 16 16:48:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.74299999999999999,"15min":0.755},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86599999999999999,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.77900000000000003,"15min":0.78900000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.89200000000000002,"5min":1.01,"15min":1.01},"last":0.621}]},{"osd":11,"last update":"Thu Nov 16 16:48:23 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.78000000000000003}]}]},{"osd":6,"up_from":46,"seq":197568495668,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6388,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738764,"statfs":{"total":95995035648,"available":95988494336,"internally_reserved":0,"allocated":962560,"data_stored":483140,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.38800000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.505},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.43099999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84599999999999997,"5min":1.345,"15min":1.345},"last":0.46400000000000002}]},{"osd":1,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64900000000000002,"15min":0.64800000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.98199999999999998,"5min":1.111,"15min":1.111},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69699999999999995,"15min":0.70399999999999996},"min":{"1min":0.505,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0229999999999999,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.64800000000000002}]},{"osd":2,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.71799999999999997,"15min":0.70799999999999996},"min":{"1min":0.54900000000000004,"5min":0.436,"15min":0.436},"max":{"1min":0.998,"5min":1.407,"15min":1.407},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.71899999999999997,"15min":0.72499999999999998},"min":{"1min":0.48799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.98699999999999999,"5min":1.171,"15min":1.171},"last":0.89600000000000002}]},{"osd":3,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.72799999999999998,"15min":0.73999999999999999},"min":{"1min":0.49099999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0189999999999999,"5min":1.022,"15min":1.022},"last":0.60199999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.746,"15min":0.73199999999999998},"min":{"1min":0.56899999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.755,"15min":0.76000000000000001},"min":{"1min":0.58499999999999996,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.048,"5min":1.083,"15min":1.083},"last":0.66100000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79000000000000004,"15min":0.80000000000000004},"min":{"1min":0.61099999999999999,"5min":0.503,"15min":0.503},"max":{"1min":1.0940000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.88300000000000001,"15min":0.89700000000000002},"min":{"1min":0.747,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.2749999999999999,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.95099999999999996}]},{"osd":7,"last update":"Thu Nov 16 16:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.81899999999999995,"15min":0.79800000000000004},"min":{"1min":0.64300000000000002,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.1100000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.87,"5min":0.78000000000000003,"15min":0.753},"min":{"1min":0.59399999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0820000000000001,"5min":1.256,"15min":1.256},"last":0.96399999999999997}]},{"osd":8,"last update":"Thu Nov 16 16:48:27 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.84899999999999998,"15min":0.84799999999999998},"min":{"1min":0.66200000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.90200000000000002,"15min":0.89200000000000002},"min":{"1min":0.65800000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.284,"5min":1.823,"15min":1.823},"last":1.0509999999999999}]},{"osd":9,"last update":"Thu Nov 16 16:47:50 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.96499999999999997,"15min":0.96699999999999997},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.238,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.84699999999999998,"15min":0.83899999999999997},"min":{"1min":0.61899999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.069,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.93400000000000005}]},{"osd":10,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.91500000000000004,"15min":0.92900000000000005},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.097,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.98799999999999999,"15min":0.98599999999999999},"min":{"1min":0.85599999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.206,"5min":1.393,"15min":1.393},"last":1.02}]},{"osd":11,"last update":"Thu Nov 16 16:48:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":1.083},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.98099999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7480,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737672,"statfs":{"total":95995035648,"available":95987376128,"internally_reserved":0,"allocated":1032192,"data_stored":544048,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55100000000000005,"15min":0.52500000000000002},"min":{"1min":0.39600000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.77500000000000002,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.54000000000000004,"15min":0.51400000000000001},"min":{"1min":0.40500000000000003,"5min":0.315,"15min":0.315},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.57899999999999996}]},{"osd":2,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.55900000000000005,"15min":0.53600000000000003},"min":{"1min":0.41099999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.54700000000000004,"15min":0.51100000000000001},"min":{"1min":0.434,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.86199999999999999,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.59999999999999998}]},{"osd":3,"last update":"Thu Nov 16 16:47:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55000000000000004,"15min":0.53400000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.58899999999999997,"15min":0.57299999999999995},"min":{"1min":0.41199999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.63100000000000001}]},{"osd":4,"last update":"Thu Nov 16 16:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.437,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.45400000000000001,"15min":0.438},"min":{"1min":0.29099999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.38800000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:48:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.60499999999999998,"15min":0.57399999999999995},"min":{"1min":0.50800000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.629,"15min":0.60699999999999998},"min":{"1min":0.51400000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.54400000000000004}]},{"osd":6,"last update":"Thu Nov 16 16:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.42599999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82999999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.65800000000000003,"15min":0.64200000000000002},"min":{"1min":0.48899999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.68000000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:48:07 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55300000000000005,"15min":0.52100000000000002},"min":{"1min":0.36099999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.84099999999999997,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.58399999999999996,"15min":0.55900000000000005},"min":{"1min":0.379,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.93500000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.70699999999999996}]},{"osd":8,"last update":"Thu Nov 16 16:48:24 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.67600000000000005,"15min":0.64200000000000002},"min":{"1min":0.63100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.97099999999999997,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.68799999999999994,"15min":0.65600000000000003},"min":{"1min":0.55200000000000005,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.755}]},{"osd":9,"last update":"Thu Nov 16 16:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.73699999999999999,"15min":0.72799999999999998},"min":{"1min":0.57199999999999995,"5min":0.5,"15min":0.5},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.755,"15min":0.749},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.74299999999999999}]},{"osd":10,"last update":"Thu Nov 16 16:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.77500000000000002,"15min":0.78000000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.97599999999999998,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78400000000000003,"15min":0.78300000000000003},"min":{"1min":0.54000000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.80900000000000005}]},{"osd":11,"last update":"Thu Nov 16 16:48:20 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.83699999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7288,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737864,"statfs":{"total":95995035648,"available":95987572736,"internally_reserved":0,"allocated":966656,"data_stored":483230,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 16:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57099999999999995,"15min":0.54300000000000004},"min":{"1min":0.375,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.88400000000000001,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.627,"15min":0.56899999999999995},"min":{"1min":0.38100000000000001,"5min":0.316,"15min":0.316},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.64600000000000002}]},{"osd":2,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.70299999999999996,"15min":0.60199999999999998},"min":{"1min":0.49399999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.90800000000000003,"5min":5.4969999999999999,"15min":5.4969999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.66400000000000003,"15min":0.61599999999999999},"min":{"1min":0.41799999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.88600000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.53000000000000003}]},{"osd":3,"last update":"Thu Nov 16 16:48:21 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.501},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.47199999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:48:01 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.66100000000000003,"15min":0.59299999999999997},"min":{"1min":0.441,"5min":0.434,"15min":0.434},"max":{"1min":1.048,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.73699999999999999,"15min":0.78400000000000003},"min":{"1min":0.52000000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93700000000000006,"5min":5.4409999999999998,"15min":5.4409999999999998},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Nov 16 16:48:21 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.76000000000000001,"15min":0.70899999999999996},"min":{"1min":0.60899999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.1589999999999998,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.74399999999999999,"15min":0.72399999999999998},"min":{"1min":0.59599999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.4420000000000002,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":0.63200000000000001}]},{"osd":6,"last update":"Thu Nov 16 16:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.70299999999999996,"15min":0.68700000000000006},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.98599999999999999,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.69499999999999995,"15min":0.66600000000000004},"min":{"1min":0.42499999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.036,"5min":1.093,"15min":1.093},"last":0.72199999999999998}]},{"osd":7,"last update":"Thu Nov 16 16:48:26 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.8089999999999999,"5min":1.8089999999999999,"15min":1.8089999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":3.1179999999999999,"5min":3.1179999999999999,"15min":3.1179999999999999},"last":0.83499999999999996}]},{"osd":8,"last update":"Thu Nov 16 16:48:26 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.876,"15min":0.88200000000000001},"min":{"1min":0.59399999999999997,"5min":0.498,"15min":0.498},"max":{"1min":2.891,"5min":2.891,"15min":2.891},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.90800000000000003,"15min":0.91900000000000004},"min":{"1min":0.71399999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":1.8759999999999999},"last":0.84999999999999998}]},{"osd":9,"last update":"Thu Nov 16 16:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.98399999999999999},"min":{"1min":0.64100000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1930000000000001,"5min":1.278,"15min":1.278},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.91000000000000003,"15min":0.92100000000000004},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.117,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.76300000000000001}]},{"osd":10,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.91100000000000003,"15min":0.92200000000000004},"min":{"1min":0.63900000000000001,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.2330000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":1.0640000000000001,"15min":1.0940000000000001},"min":{"1min":0.67300000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.1679999999999999,"5min":3.694,"15min":3.694},"last":0.877}]},{"osd":11,"last update":"Thu Nov 16 16:48:31 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":0.90900000000000003,"15min":0.88700000000000001},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.1909999999999998,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.91300000000000003,"15min":0.88900000000000001},"min":{"1min":0.65500000000000003,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":3.0489999999999999,"5min":3.0489999999999999,"15min":3.0489999999999999},"last":0.88800000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7544,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737608,"statfs":{"total":95995035648,"available":95987310592,"internally_reserved":0,"allocated":1032192,"data_stored":543809,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.53400000000000003,"15min":0.46999999999999997},"min":{"1min":0.41599999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52100000000000002,"15min":0.495},"min":{"1min":0.33100000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.76200000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:48:12 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.54000000000000004,"15min":0.49399999999999999},"min":{"1min":0.36099999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.56899999999999995,"15min":0.504},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.495}]},{"osd":3,"last update":"Thu Nov 16 16:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.67100000000000004,"15min":0.71799999999999997},"min":{"1min":0.442,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.95499999999999996,"5min":3.98,"15min":3.98},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.61899999999999999,"15min":0.59299999999999997},"min":{"1min":0.45100000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.83699999999999997}]},{"osd":4,"last update":"Thu Nov 16 16:48:19 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.55100000000000005}]},{"osd":5,"last update":"Thu Nov 16 16:48:22 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.48099999999999998,"15min":0.46300000000000002},"min":{"1min":0.374,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.78500000000000003,"5min":0.871,"15min":0.871},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.50800000000000001,"15min":0.48199999999999998},"min":{"1min":0.40000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70999999999999996,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.66700000000000004}]},{"osd":6,"last update":"Thu Nov 16 16:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.59199999999999997,"15min":0.58199999999999996},"min":{"1min":0.44400000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.81000000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:48:22 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.85499999999999998}]},{"osd":8,"last update":"Thu Nov 16 16:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.61499999999999999,"15min":0.60099999999999998},"min":{"1min":0.47799999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.58599999999999997,"15min":0.57599999999999996},"min":{"1min":0.47599999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.79400000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.88500000000000001}]},{"osd":9,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.71899999999999997,"15min":0.71699999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.91300000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.73099999999999998,"15min":0.72899999999999998},"min":{"1min":0.51200000000000001,"5min":0.503,"15min":0.503},"max":{"1min":0.91500000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.95699999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:48:12 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.69599999999999995,"15min":0.68400000000000005},"min":{"1min":0.48799999999999999,"5min":0.441,"15min":0.441},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.70199999999999996,"15min":0.69199999999999995},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.96799999999999997}]},{"osd":11,"last update":"Thu Nov 16 16:48:19 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.93100000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182464,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6716,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738436,"statfs":{"total":95995035648,"available":95988158464,"internally_reserved":0,"allocated":970752,"data_stored":483244,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:35 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.45500000000000002,"15min":0.42599999999999999},"min":{"1min":0.40500000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57099999999999995,"5min":1.085,"15min":1.085},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54500000000000004,"15min":0.49299999999999999},"min":{"1min":0.36799999999999999,"5min":0.313,"15min":0.313},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.53900000000000003}]},{"osd":1,"last update":"Thu Nov 16 16:47:35 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.60499999999999998,"15min":0.57399999999999995},"min":{"1min":0.41899999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.96999999999999997,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.63200000000000001,"15min":0.59499999999999997},"min":{"1min":0.45400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.006,"5min":1.286,"15min":1.286},"last":0.44400000000000001}]},{"osd":2,"last update":"Thu Nov 16 16:47:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.63300000000000001,"15min":0.58699999999999997},"min":{"1min":0.53200000000000003,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.92500000000000004,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.64500000000000002,"15min":0.60099999999999998},"min":{"1min":0.48499999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.71399999999999997}]},{"osd":4,"last update":"Thu Nov 16 16:48:01 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.67600000000000005,"15min":0.61599999999999999},"min":{"1min":0.47299999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.8420000000000001,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.70199999999999996,"15min":0.66800000000000004},"min":{"1min":0.52200000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.95399999999999996,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.80700000000000005}]},{"osd":5,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71899999999999997,"15min":0.69299999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.004,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.76000000000000001,"15min":0.68600000000000005},"min":{"1min":0.49399999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.2330000000000001,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":0.752}]},{"osd":6,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.69399999999999995,"15min":0.65900000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0669999999999999,"5min":1.323,"15min":1.323},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.70599999999999996,"15min":0.68200000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.02,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.41999999999999998}]},{"osd":7,"last update":"Thu Nov 16 16:48:03 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.82799999999999996,"15min":0.79800000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":5.1070000000000002,"5min":5.1070000000000002,"15min":5.1070000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.86899999999999999,"15min":0.85299999999999998},"min":{"1min":0.629,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.60599999999999998}]},{"osd":8,"last update":"Thu Nov 16 16:46:23 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.94199999999999995}]},{"osd":9,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.90400000000000003,"15min":0.91700000000000004},"min":{"1min":0.66700000000000004,"5min":0.501,"15min":0.501},"max":{"1min":1.0900000000000001,"5min":1.395,"15min":1.395},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.97199999999999998,"15min":0.97699999999999998},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.76800000000000002}]},{"osd":10,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.86599999999999999,"15min":0.878},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.115,"5min":1.304,"15min":1.304},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.94999999999999996,"15min":0.94899999999999995},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.673,"5min":2.673,"15min":2.673},"last":0.67600000000000005}]},{"osd":11,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.78000000000000003}]}]},{"osd":4,"up_from":33,"seq":141733920827,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6524,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738628,"statfs":{"total":95995035648,"available":95988355072,"internally_reserved":0,"allocated":970752,"data_stored":487258,"data_compressed":17633,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.48899999999999999,"15min":0.46600000000000003},"min":{"1min":0.439,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.51100000000000001,"15min":0.48599999999999999},"min":{"1min":0.47399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.63500000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.36099999999999999,"15min":0.32400000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.54400000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.38200000000000001,"15min":0.34899999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.502}]},{"osd":2,"last update":"Thu Nov 16 16:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.56200000000000006,"15min":0.51500000000000001},"min":{"1min":0.33400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.55200000000000005,"15min":0.50700000000000001},"min":{"1min":0.42399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.70099999999999996}]},{"osd":3,"last update":"Thu Nov 16 16:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.56699999999999995,"15min":0.52700000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.415,"5min":0.72499999999999998,"15min":0.58199999999999996},"min":{"1min":0.42799999999999999,"5min":0.374,"15min":0.374},"max":{"1min":14.795999999999999,"5min":14.795999999999999,"15min":14.795999999999999},"last":0.59299999999999997}]},{"osd":5,"last update":"Thu Nov 16 16:48:17 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.59599999999999997,"15min":0.54700000000000004},"min":{"1min":0.39900000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.58399999999999996,"15min":0.53200000000000003},"min":{"1min":0.441,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81899999999999995,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.66200000000000003}]},{"osd":6,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.61299999999999999,"15min":0.58199999999999996},"min":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.65700000000000003,"15min":0.61899999999999999},"min":{"1min":0.45100000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":0.55100000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:48:10 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.55000000000000004,"15min":0.52400000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59799999999999998,"15min":0.57099999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.78100000000000003}]},{"osd":8,"last update":"Thu Nov 16 16:48:26 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.72099999999999997,"15min":0.71299999999999997},"min":{"1min":0.498,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.83899999999999997,"5min":2.7650000000000001,"15min":2.7650000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.73599999999999999,"15min":0.745},"min":{"1min":0.44,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.86399999999999999,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.83399999999999996}]},{"osd":9,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.79600000000000004,"15min":0.80500000000000005},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.77900000000000003,"15min":0.78100000000000003},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.80900000000000005}]},{"osd":10,"last update":"Thu Nov 16 16:48:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69499999999999995,"15min":0.69999999999999996},"min":{"1min":0.47599999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.86499999999999999,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.71799999999999997,"15min":0.72199999999999998},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.73899999999999999}]},{"osd":11,"last update":"Thu Nov 16 16:48:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.82399999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757305,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6708,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738444,"statfs":{"total":95995035648,"available":95988166656,"internally_reserved":0,"allocated":1028096,"data_stored":1077786,"data_compressed":53424,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:19 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52100000000000002,"15min":0.51500000000000001},"min":{"1min":0.34599999999999997,"5min":0.318,"15min":0.318},"max":{"1min":0.751,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.44400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76000000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.48899999999999999}]},{"osd":1,"last update":"Thu Nov 16 16:48:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.56899999999999995,"15min":0.54700000000000004},"min":{"1min":0.38500000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77900000000000003,"5min":1.167,"15min":1.167},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.55700000000000005,"15min":0.51700000000000002},"min":{"1min":0.441,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81799999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Nov 16 16:48:19 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.38900000000000001,"15min":0.372},"min":{"1min":0.32400000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60299999999999998,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.41599999999999998,"15min":0.39100000000000001},"min":{"1min":0.33300000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.63500000000000001,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.38}]},{"osd":3,"last update":"Thu Nov 16 16:48:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.55500000000000005,"15min":0.53500000000000003},"min":{"1min":0.40699999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84299999999999997,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.436},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59299999999999997,"15min":0.56999999999999995},"min":{"1min":0.42899999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78600000000000003,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.55600000000000005}]},{"osd":4,"last update":"Thu Nov 16 16:48:19 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.57399999999999995,"15min":0.55300000000000005},"min":{"1min":0.39400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81299999999999994,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.55800000000000005,"15min":0.52800000000000002},"min":{"1min":0.42099999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.80400000000000005,"5min":0.879,"15min":0.879},"last":0.58499999999999996}]},{"osd":6,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.63200000000000001,"15min":0.63},"min":{"1min":0.436,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91200000000000003,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.53300000000000003}]},{"osd":7,"last update":"Thu Nov 16 16:48:06 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84099999999999997,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.748},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84799999999999998,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.61499999999999999}]},{"osd":8,"last update":"Thu Nov 16 16:48:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.42299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80100000000000005,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.58799999999999997},"min":{"1min":0.47799999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.59899999999999998}]},{"osd":9,"last update":"Thu Nov 16 16:47:50 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.748,"15min":0.76100000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.86799999999999999,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.749,"15min":0.75900000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.871,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.72799999999999998}]},{"osd":10,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.70599999999999996,"15min":0.71299999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88800000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.76300000000000001,"15min":0.755},"min":{"1min":0.51900000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":2.492,"5min":2.492,"15min":2.492},"last":0.68600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":60942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T16:48:42.967 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-11-16T16:48:42.967 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-16T16:48:42.967 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-11-16T16:48:42.968 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph health --format=json 2023-11-16T16:48:43.834 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:43.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:43 smithi044 ceph-mon[128750]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:48:43.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:43 smithi044 ceph-mon[128750]: pgmap v319: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:43.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2023-11-16T16:48:43.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2023-11-16T16:48:43.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2023-11-16T16:48:43.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2023-11-16T16:48:43.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:43 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2023-11-16T16:48:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:43 smithi176 ceph-mon[131630]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:48:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:43 smithi176 ceph-mon[131630]: pgmap v319: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2023-11-16T16:48:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2023-11-16T16:48:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2023-11-16T16:48:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2023-11-16T16:48:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:43 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2023-11-16T16:48:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:43 smithi077 ceph-mon[131204]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:48:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:43 smithi077 ceph-mon[131204]: pgmap v319: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2023-11-16T16:48:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2023-11-16T16:48:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2023-11-16T16:48:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2023-11-16T16:48:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:43 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2023-11-16T16:48:44.699 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:44 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2023-11-16T16:48:44.699 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:44 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2023-11-16T16:48:44.699 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:44 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2023-11-16T16:48:44.699 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:44 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2023-11-16T16:48:44.699 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:44 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2023-11-16T16:48:44.700 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:44 smithi044 ceph-mon[128750]: osdmap e90: 12 total, 12 up, 12 in 2023-11-16T16:48:44.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:44 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2023-11-16T16:48:44.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:44 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2023-11-16T16:48:44.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:44 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2023-11-16T16:48:44.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:44 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2023-11-16T16:48:44.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:44 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2023-11-16T16:48:44.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:44 smithi176 ceph-mon[131630]: osdmap e90: 12 total, 12 up, 12 in 2023-11-16T16:48:44.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:44 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2023-11-16T16:48:44.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:44 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2023-11-16T16:48:44.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:44 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2023-11-16T16:48:44.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:44 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2023-11-16T16:48:44.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:44 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2023-11-16T16:48:44.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:44 smithi077 ceph-mon[131204]: osdmap e90: 12 total, 12 up, 12 in 2023-11-16T16:48:45.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.370+0000 7fc9fc533700 1 -- 172.21.15.44:0/831749405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9f41055e0 msgr2=0x7fc9f410d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:45.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.370+0000 7fc9fc533700 1 --2- 172.21.15.44:0/831749405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9f41055e0 0x7fc9f410d160 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00b0d0 tx=0x7fc9ec00b3e0 comp rx=0 tx=0).stop 2023-11-16T16:48:45.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.370+0000 7fc9fc533700 1 -- 172.21.15.44:0/831749405 shutdown_connections 2023-11-16T16:48:45.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.370+0000 7fc9fc533700 1 --2- 172.21.15.44:0/831749405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9f41055e0 0x7fc9f410d160 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:45.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.370+0000 7fc9fc533700 1 --2- 172.21.15.44:0/831749405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9f4100540 0x7fc9f4100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:45.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.370+0000 7fc9fc533700 1 --2- 172.21.15.44:0/831749405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f4100ee0 0x7fc9f4104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:45.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.370+0000 7fc9fc533700 1 -- 172.21.15.44:0/831749405 >> 172.21.15.44:0/831749405 conn(0x7fc9f40fbdc0 msgr2=0x7fc9f40fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:45.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.370+0000 7fc9fc533700 1 -- 172.21.15.44:0/831749405 shutdown_connections 2023-11-16T16:48:45.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.370+0000 7fc9fc533700 1 -- 172.21.15.44:0/831749405 wait complete. 2023-11-16T16:48:45.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.371+0000 7fc9fc533700 1 Processor -- start 2023-11-16T16:48:45.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.371+0000 7fc9fc533700 1 -- start start 2023-11-16T16:48:45.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.372+0000 7fc9fc533700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f4100540 0x7fc9f41964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:45.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.372+0000 7fc9fc533700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9f4100ee0 0x7fc9f41969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.372+0000 7fc9fc533700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9f41055e0 0x7fc9f419ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.372+0000 7fc9fc533700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f419b470 con 0x7fc9f4100ee0 2023-11-16T16:48:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.372+0000 7fc9fc533700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f419b5e0 con 0x7fc9f4100540 2023-11-16T16:48:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.372+0000 7fc9fc533700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f419b7a0 con 0x7fc9f41055e0 2023-11-16T16:48:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.372+0000 7fc9fa2cf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f4100540 0x7fc9f41964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.372+0000 7fc9f9ace700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9f4100ee0 0x7fc9f41969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.372+0000 7fc9fa2cf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f4100540 0x7fc9f41964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:40098/0 (socket says 172.21.15.44:40098) 2023-11-16T16:48:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.372+0000 7fc9fa2cf700 1 -- 172.21.15.44:0/844805291 learned_addr learned my addr 172.21.15.44:0/844805291 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.372+0000 7fc9faad0700 1 --2- 172.21.15.44:0/844805291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9f41055e0 0x7fc9f419ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.372+0000 7fc9fa2cf700 1 -- 172.21.15.44:0/844805291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9f41055e0 msgr2=0x7fc9f419ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.372+0000 7fc9fa2cf700 1 --2- 172.21.15.44:0/844805291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9f41055e0 0x7fc9f419ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:45.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.372+0000 7fc9fa2cf700 1 -- 172.21.15.44:0/844805291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9f4100ee0 msgr2=0x7fc9f41969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:45.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.372+0000 7fc9fa2cf700 1 --2- 172.21.15.44:0/844805291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9f4100ee0 0x7fc9f41969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:45.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.372+0000 7fc9fa2cf700 1 -- 172.21.15.44:0/844805291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ec009d00 con 0x7fc9f4100540 2023-11-16T16:48:45.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.373+0000 7fc9fa2cf700 1 --2- 172.21.15.44:0/844805291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f4100540 0x7fc9f41964b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000e9a0 tx=0x7fc9f000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:45.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.373+0000 7fc9eb7fe700 1 -- 172.21.15.44:0/844805291 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0009790 con 0x7fc9f4100540 2023-11-16T16:48:45.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.373+0000 7fc9fc533700 1 -- 172.21.15.44:0/844805291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f4110370 con 0x7fc9f4100540 2023-11-16T16:48:45.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.373+0000 7fc9eb7fe700 1 -- 172.21.15.44:0/844805291 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9f0010430 con 0x7fc9f4100540 2023-11-16T16:48:45.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.373+0000 7fc9eb7fe700 1 -- 172.21.15.44:0/844805291 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0018680 con 0x7fc9f4100540 2023-11-16T16:48:45.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.373+0000 7fc9fc533700 1 -- 172.21.15.44:0/844805291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f41101e0 con 0x7fc9f4100540 2023-11-16T16:48:45.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.374+0000 7fc9fc533700 1 -- 172.21.15.44:0/844805291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f4110500 con 0x7fc9f4100540 2023-11-16T16:48:45.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.376+0000 7fc9eb7fe700 1 -- 172.21.15.44:0/844805291 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc9f00105a0 con 0x7fc9f4100540 2023-11-16T16:48:45.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.378+0000 7fc9eb7fe700 1 --2- 172.21.15.44:0/844805291 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc9e00a11e0 0x7fc9e00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:45.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.378+0000 7fc9eb7fe700 1 -- 172.21.15.44:0/844805291 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fc9f0014070 con 0x7fc9f4100540 2023-11-16T16:48:45.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.378+0000 7fc9f9ace700 1 --2- 172.21.15.44:0/844805291 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc9e00a11e0 0x7fc9e00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:45.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.378+0000 7fc9f9ace700 1 --2- 172.21.15.44:0/844805291 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc9e00a11e0 0x7fc9e00a3690 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4197ad0 tx=0x7fc9e4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:45.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.380+0000 7fc9eb7fe700 1 -- 172.21.15.44:0/844805291 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc9f0096440 con 0x7fc9f4100540 2023-11-16T16:48:45.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.631+0000 7fc9fc533700 1 -- 172.21.15.44:0/844805291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc9f4066ea0 con 0x7fc9f4100540 2023-11-16T16:48:45.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.632+0000 7fc9eb7fe700 1 -- 172.21.15.44:0/844805291 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fc9f0099a60 con 0x7fc9f4100540 2023-11-16T16:48:45.634 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:48:45.634 INFO:teuthology.orchestra.run.smithi044.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-16T16:48:45.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.634+0000 7fc9fc533700 1 -- 172.21.15.44:0/844805291 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc9e00a11e0 msgr2=0x7fc9e00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.634+0000 7fc9fc533700 1 --2- 172.21.15.44:0/844805291 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc9e00a11e0 0x7fc9e00a3690 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4197ad0 tx=0x7fc9e4009260 comp rx=0 tx=0).stop 2023-11-16T16:48:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.634+0000 7fc9fc533700 1 -- 172.21.15.44:0/844805291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f4100540 msgr2=0x7fc9f41964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.634+0000 7fc9fc533700 1 --2- 172.21.15.44:0/844805291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f4100540 0x7fc9f41964b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000e9a0 tx=0x7fc9f000ecb0 comp rx=0 tx=0).stop 2023-11-16T16:48:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.635+0000 7fc9fc533700 1 -- 172.21.15.44:0/844805291 shutdown_connections 2023-11-16T16:48:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.635+0000 7fc9fc533700 1 --2- 172.21.15.44:0/844805291 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc9e00a11e0 0x7fc9e00a3690 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.635+0000 7fc9fc533700 1 --2- 172.21.15.44:0/844805291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9f41055e0 0x7fc9f419ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.635+0000 7fc9fc533700 1 --2- 172.21.15.44:0/844805291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f4100540 0x7fc9f41964b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.635+0000 7fc9fc533700 1 --2- 172.21.15.44:0/844805291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9f4100ee0 0x7fc9f41969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.635+0000 7fc9fc533700 1 -- 172.21.15.44:0/844805291 >> 172.21.15.44:0/844805291 conn(0x7fc9f40fbdc0 msgr2=0x7fc9f40fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.635+0000 7fc9fc533700 1 -- 172.21.15.44:0/844805291 shutdown_connections 2023-11-16T16:48:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:45.635+0000 7fc9fc533700 1 -- 172.21.15.44:0/844805291 wait complete. 2023-11-16T16:48:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:45 smithi176 ceph-mon[131630]: pgmap v321: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2023-11-16T16:48:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:45 smithi176 ceph-mon[131630]: osdmap e91: 12 total, 12 up, 12 in 2023-11-16T16:48:45.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:45 smithi077 ceph-mon[131204]: pgmap v321: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2023-11-16T16:48:45.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:45 smithi077 ceph-mon[131204]: osdmap e91: 12 total, 12 up, 12 in 2023-11-16T16:48:46.065 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:45 smithi044 ceph-mon[128750]: pgmap v321: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2023-11-16T16:48:46.066 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:45 smithi044 ceph-mon[128750]: osdmap e91: 12 total, 12 up, 12 in 2023-11-16T16:48:46.134 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-11-16T16:48:46.135 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:48:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.347+0000 7f51203ff700 1 -- 172.21.15.44:0/2965840709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51181014a0 msgr2=0x7f51181018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.347+0000 7f51203ff700 1 --2- 172.21.15.44:0/2965840709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51181014a0 0x7f51181018f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5108009990 tx=0x7f5108009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.347+0000 7f51203ff700 1 -- 172.21.15.44:0/2965840709 shutdown_connections 2023-11-16T16:48:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.347+0000 7f51203ff700 1 --2- 172.21.15.44:0/2965840709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5118101e30 0x7f51181086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.347+0000 7f51203ff700 1 --2- 172.21.15.44:0/2965840709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51181002b0 0x7f51181006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.347+0000 7f51203ff700 1 --2- 172.21.15.44:0/2965840709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51181014a0 0x7f51181018f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.347+0000 7f51203ff700 1 -- 172.21.15.44:0/2965840709 >> 172.21.15.44:0/2965840709 conn(0x7f51180fb890 msgr2=0x7f51180fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.347+0000 7f51203ff700 1 -- 172.21.15.44:0/2965840709 shutdown_connections 2023-11-16T16:48:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.347+0000 7f51203ff700 1 -- 172.21.15.44:0/2965840709 wait complete. 2023-11-16T16:48:46.350 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.348+0000 7f51203ff700 1 Processor -- start 2023-11-16T16:48:46.350 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.349+0000 7f51203ff700 1 -- start start 2023-11-16T16:48:46.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.349+0000 7f51203ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51181002b0 0x7f51180748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:46.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.349+0000 7f51203ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51181014a0 0x7f5118072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:46.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.349+0000 7f51203ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5118101e30 0x7f5118073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:46.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.349+0000 7f51203ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5118073bc0 con 0x7f51181014a0 2023-11-16T16:48:46.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.349+0000 7f51203ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5118073d30 con 0x7f5118101e30 2023-11-16T16:48:46.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.349+0000 7f51203ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511819ffe0 con 0x7f51181002b0 2023-11-16T16:48:46.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.349+0000 7f511d99a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51181014a0 0x7f5118072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:46.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.349+0000 7f511d99a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51181014a0 0x7f5118072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39118/0 (socket says 172.21.15.44:39118) 2023-11-16T16:48:46.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.349+0000 7f511e19b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51181002b0 0x7f51180748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:46.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.349+0000 7f511d99a700 1 -- 172.21.15.44:0/887781835 learned_addr learned my addr 172.21.15.44:0/887781835 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:46.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.349+0000 7f511e99c700 1 --2- 172.21.15.44:0/887781835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5118101e30 0x7f5118073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:46.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.350+0000 7f511d99a700 1 -- 172.21.15.44:0/887781835 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51181002b0 msgr2=0x7f51180748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:46.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.350+0000 7f511d99a700 1 --2- 172.21.15.44:0/887781835 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51181002b0 0x7f51180748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:46.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.350+0000 7f511d99a700 1 -- 172.21.15.44:0/887781835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5118101e30 msgr2=0x7f5118073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:46.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.350+0000 7f511d99a700 1 --2- 172.21.15.44:0/887781835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5118101e30 0x7f5118073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:46.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.350+0000 7f511d99a700 1 -- 172.21.15.44:0/887781835 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5114009520 con 0x7f51181014a0 2023-11-16T16:48:46.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.350+0000 7f511d99a700 1 --2- 172.21.15.44:0/887781835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51181014a0 0x7f5118072ee0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f510800b680 tx=0x7f5108005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:46.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.350+0000 7f510f7fe700 1 -- 172.21.15.44:0/887781835 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510801d070 con 0x7f51181014a0 2023-11-16T16:48:46.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.350+0000 7f51203ff700 1 -- 172.21.15.44:0/887781835 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5110009c40 con 0x7f51181014a0 2023-11-16T16:48:46.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.350+0000 7f510f7fe700 1 -- 172.21.15.44:0/887781835 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5108022470 con 0x7f51181014a0 2023-11-16T16:48:46.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.350+0000 7f510f7fe700 1 -- 172.21.15.44:0/887781835 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108017650 con 0x7f51181014a0 2023-11-16T16:48:46.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.350+0000 7f51203ff700 1 -- 172.21.15.44:0/887781835 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51080095f0 con 0x7f51181014a0 2023-11-16T16:48:46.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.351+0000 7f51203ff700 1 -- 172.21.15.44:0/887781835 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51181078a0 con 0x7f51181014a0 2023-11-16T16:48:46.355 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.353+0000 7f510f7fe700 1 -- 172.21.15.44:0/887781835 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f51080225e0 con 0x7f51181014a0 2023-11-16T16:48:46.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.355+0000 7f510f7fe700 1 --2- 172.21.15.44:0/887781835 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f51040a11e0 0x7f51040a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:46.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.355+0000 7f510f7fe700 1 -- 172.21.15.44:0/887781835 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f51080ce2d0 con 0x7f51181014a0 2023-11-16T16:48:46.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.355+0000 7f511e19b700 1 --2- 172.21.15.44:0/887781835 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f51040a11e0 0x7f51040a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:46.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.356+0000 7f511e19b700 1 --2- 172.21.15.44:0/887781835 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f51040a11e0 0x7f51040a3690 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5118101300 tx=0x7f5114009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:46.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.359+0000 7f510f7fe700 1 -- 172.21.15.44:0/887781835 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5108097550 con 0x7f51181014a0 2023-11-16T16:48:46.571 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.570+0000 7f51203ff700 1 -- 172.21.15.44:0/887781835 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f511802cd50 con 0x7f51181014a0 2023-11-16T16:48:46.573 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.571+0000 7f510f7fe700 1 -- 172.21.15.44:0/887781835 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24685 (secure 0 0 0) 0x7f510809ab70 con 0x7f51181014a0 2023-11-16T16:48:46.573 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:48:46.574 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:48:44.613706+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:48:46.576 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.574+0000 7f51203ff700 1 -- 172.21.15.44:0/887781835 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f51040a11e0 msgr2=0x7f51040a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:46.576 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.574+0000 7f51203ff700 1 --2- 172.21.15.44:0/887781835 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f51040a11e0 0x7f51040a3690 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5118101300 tx=0x7f5114009260 comp rx=0 tx=0).stop 2023-11-16T16:48:46.576 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.574+0000 7f51203ff700 1 -- 172.21.15.44:0/887781835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51181014a0 msgr2=0x7f5118072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:46.576 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.574+0000 7f51203ff700 1 --2- 172.21.15.44:0/887781835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51181014a0 0x7f5118072ee0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f510800b680 tx=0x7f5108005740 comp rx=0 tx=0).stop 2023-11-16T16:48:46.576 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.574+0000 7f51203ff700 1 -- 172.21.15.44:0/887781835 shutdown_connections 2023-11-16T16:48:46.576 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.574+0000 7f51203ff700 1 --2- 172.21.15.44:0/887781835 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f51040a11e0 0x7f51040a3690 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:46.576 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.574+0000 7f51203ff700 1 --2- 172.21.15.44:0/887781835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5118101e30 0x7f5118073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:46.576 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.574+0000 7f51203ff700 1 --2- 172.21.15.44:0/887781835 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51181002b0 0x7f51180748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:46.576 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.574+0000 7f51203ff700 1 --2- 172.21.15.44:0/887781835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51181014a0 0x7f5118072ee0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:46.576 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.574+0000 7f51203ff700 1 -- 172.21.15.44:0/887781835 >> 172.21.15.44:0/887781835 conn(0x7f51180fb890 msgr2=0x7f51181049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:46.576 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.574+0000 7f51203ff700 1 -- 172.21.15.44:0/887781835 shutdown_connections 2023-11-16T16:48:46.576 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.574+0000 7f51203ff700 1 -- 172.21.15.44:0/887781835 wait complete. 2023-11-16T16:48:46.595 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:03.059622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:16.211826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:17.601591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:21.136084+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T16:48:46.596 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:48:46.806 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.805+0000 7f21367ff700 1 -- 172.21.15.44:0/929346652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21301002c0 msgr2=0x7f21301006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:46.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.805+0000 7f21367ff700 1 --2- 172.21.15.44:0/929346652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21301002c0 0x7f21301006d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2120009990 tx=0x7f2120009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:46.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.805+0000 7f21367ff700 1 -- 172.21.15.44:0/929346652 shutdown_connections 2023-11-16T16:48:46.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.805+0000 7f21367ff700 1 --2- 172.21.15.44:0/929346652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2130101e40 0x7f2130108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:46.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.805+0000 7f21367ff700 1 --2- 172.21.15.44:0/929346652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21301002c0 0x7f21301006d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:46.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.805+0000 7f21367ff700 1 --2- 172.21.15.44:0/929346652 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21301014b0 0x7f2130101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:46.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.805+0000 7f21367ff700 1 -- 172.21.15.44:0/929346652 >> 172.21.15.44:0/929346652 conn(0x7f21300fb8a0 msgr2=0x7f21300fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:46.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.805+0000 7f21367ff700 1 -- 172.21.15.44:0/929346652 shutdown_connections 2023-11-16T16:48:46.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.805+0000 7f21367ff700 1 -- 172.21.15.44:0/929346652 wait complete. 2023-11-16T16:48:46.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.806+0000 7f21367ff700 1 Processor -- start 2023-11-16T16:48:46.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.806+0000 7f21367ff700 1 -- start start 2023-11-16T16:48:46.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.806+0000 7f21367ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21301002c0 0x7f2130198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:46.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.806+0000 7f21367ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21301014b0 0x7f21301987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:46.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.806+0000 7f21367ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2130101e40 0x7f213019f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:46.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.806+0000 7f21367ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213019fe10 con 0x7f2130101e40 2023-11-16T16:48:46.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.806+0000 7f21367ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213019ff50 con 0x7f21301014b0 2023-11-16T16:48:46.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.806+0000 7f21367ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21301a00e0 con 0x7f21301002c0 2023-11-16T16:48:46.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.807+0000 7f2134d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2130101e40 0x7f213019f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:46.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.807+0000 7f212f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21301014b0 0x7f21301987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:46.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.807+0000 7f2134d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2130101e40 0x7f213019f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39136/0 (socket says 172.21.15.44:39136) 2023-11-16T16:48:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.807+0000 7f2134d9c700 1 -- 172.21.15.44:0/3992778129 learned_addr learned my addr 172.21.15.44:0/3992778129 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.807+0000 7f212ffff700 1 --2- 172.21.15.44:0/3992778129 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21301002c0 0x7f2130198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.807+0000 7f2134d9c700 1 -- 172.21.15.44:0/3992778129 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21301002c0 msgr2=0x7f2130198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.807+0000 7f2134d9c700 1 --2- 172.21.15.44:0/3992778129 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21301002c0 0x7f2130198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.807+0000 7f2134d9c700 1 -- 172.21.15.44:0/3992778129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21301014b0 msgr2=0x7f21301987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.807+0000 7f2134d9c700 1 --2- 172.21.15.44:0/3992778129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21301014b0 0x7f21301987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.807+0000 7f2134d9c700 1 -- 172.21.15.44:0/3992778129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2128009c40 con 0x7f2130101e40 2023-11-16T16:48:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.807+0000 7f2134d9c700 1 --2- 172.21.15.44:0/3992778129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2130101e40 0x7f213019f840 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f212800c370 tx=0x7f212800c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:46.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.808+0000 7f212d7fa700 1 -- 172.21.15.44:0/3992778129 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2128017070 con 0x7f2130101e40 2023-11-16T16:48:46.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.808+0000 7f212d7fa700 1 -- 172.21.15.44:0/3992778129 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f212800f040 con 0x7f2130101e40 2023-11-16T16:48:46.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.808+0000 7f21367ff700 1 -- 172.21.15.44:0/3992778129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21200095f0 con 0x7f2130101e40 2023-11-16T16:48:46.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.808+0000 7f212d7fa700 1 -- 172.21.15.44:0/3992778129 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212800e050 con 0x7f2130101e40 2023-11-16T16:48:46.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.808+0000 7f21367ff700 1 -- 172.21.15.44:0/3992778129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f213010bc90 con 0x7f2130101e40 2023-11-16T16:48:46.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.809+0000 7f21367ff700 1 -- 172.21.15.44:0/3992778129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21301929f0 con 0x7f2130101e40 2023-11-16T16:48:46.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.811+0000 7f212d7fa700 1 -- 172.21.15.44:0/3992778129 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f212800cb60 con 0x7f2130101e40 2023-11-16T16:48:46.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.812+0000 7f212d7fa700 1 --2- 172.21.15.44:0/3992778129 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f21180a1260 0x7f21180a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:46.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.812+0000 7f212d7fa700 1 -- 172.21.15.44:0/3992778129 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f21280cc750 con 0x7f2130101e40 2023-11-16T16:48:46.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.812+0000 7f212ffff700 1 --2- 172.21.15.44:0/3992778129 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f21180a1260 0x7f21180a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:46.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.813+0000 7f212ffff700 1 --2- 172.21.15.44:0/3992778129 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f21180a1260 0x7f21180a3710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f212000b2e0 tx=0x7f2120005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:46.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:46.816+0000 7f212d7fa700 1 -- 172.21.15.44:0/3992778129 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2128095940 con 0x7f2130101e40 2023-11-16T16:48:46.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:46 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/844805291' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-11-16T16:48:46.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:46 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/887781835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:46.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:46 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/844805291' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-11-16T16:48:46.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:46 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/887781835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:46.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:46 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/844805291' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-11-16T16:48:46.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:46 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/887781835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:47.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.029+0000 7f21367ff700 1 -- 172.21.15.44:0/3992778129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2130067b00 con 0x7f2130101e40 2023-11-16T16:48:47.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.030+0000 7f212d7fa700 1 -- 172.21.15.44:0/3992778129 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2128098f60 con 0x7f2130101e40 2023-11-16T16:48:47.031 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2023-11-16T16:48:47.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.032+0000 7f21367ff700 1 -- 172.21.15.44:0/3992778129 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f21180a1260 msgr2=0x7f21180a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:47.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.032+0000 7f21367ff700 1 --2- 172.21.15.44:0/3992778129 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f21180a1260 0x7f21180a3710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f212000b2e0 tx=0x7f2120005fb0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.032+0000 7f21367ff700 1 -- 172.21.15.44:0/3992778129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2130101e40 msgr2=0x7f213019f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:47.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.032+0000 7f21367ff700 1 --2- 172.21.15.44:0/3992778129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2130101e40 0x7f213019f840 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f212800c370 tx=0x7f212800c680 comp rx=0 tx=0).stop 2023-11-16T16:48:47.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.033+0000 7f21367ff700 1 -- 172.21.15.44:0/3992778129 shutdown_connections 2023-11-16T16:48:47.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.033+0000 7f21367ff700 1 --2- 172.21.15.44:0/3992778129 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f21180a1260 0x7f21180a3710 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.033+0000 7f21367ff700 1 --2- 172.21.15.44:0/3992778129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2130101e40 0x7f213019f840 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.033+0000 7f21367ff700 1 --2- 172.21.15.44:0/3992778129 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21301002c0 0x7f2130198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.033+0000 7f21367ff700 1 --2- 172.21.15.44:0/3992778129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21301014b0 0x7f21301987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.033+0000 7f21367ff700 1 -- 172.21.15.44:0/3992778129 >> 172.21.15.44:0/3992778129 conn(0x7f21300fb8a0 msgr2=0x7f2130106d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:47.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.033+0000 7f21367ff700 1 -- 172.21.15.44:0/3992778129 shutdown_connections 2023-11-16T16:48:47.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.033+0000 7f21367ff700 1 -- 172.21.15.44:0/3992778129 wait complete. 2023-11-16T16:48:47.054 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T16:48:47.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.259+0000 7f8fd25ff700 1 -- 172.21.15.44:0/4178652769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fcc1002d0 msgr2=0x7f8fcc1006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:47.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.259+0000 7f8fd25ff700 1 --2- 172.21.15.44:0/4178652769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fcc1002d0 0x7f8fcc1006e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc009990 tx=0x7f8fbc009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.259+0000 7f8fd25ff700 1 -- 172.21.15.44:0/4178652769 shutdown_connections 2023-11-16T16:48:47.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.259+0000 7f8fd25ff700 1 --2- 172.21.15.44:0/4178652769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8fcc101e50 0x7f8fcc108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.259+0000 7f8fd25ff700 1 --2- 172.21.15.44:0/4178652769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fcc1002d0 0x7f8fcc1006e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.259+0000 7f8fd25ff700 1 --2- 172.21.15.44:0/4178652769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fcc1014c0 0x7f8fcc101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.259+0000 7f8fd25ff700 1 -- 172.21.15.44:0/4178652769 >> 172.21.15.44:0/4178652769 conn(0x7f8fcc0fb8b0 msgr2=0x7f8fcc0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:47.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.259+0000 7f8fd25ff700 1 -- 172.21.15.44:0/4178652769 shutdown_connections 2023-11-16T16:48:47.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.259+0000 7f8fd25ff700 1 -- 172.21.15.44:0/4178652769 wait complete. 2023-11-16T16:48:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.260+0000 7f8fd25ff700 1 Processor -- start 2023-11-16T16:48:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.260+0000 7f8fd25ff700 1 -- start start 2023-11-16T16:48:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fd25ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8fcc1002d0 0x7f8fcc198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fd25ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fcc1014c0 0x7f8fcc198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fd25ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fcc101e50 0x7f8fcc19f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fd25ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc19fe80 con 0x7f8fcc101e50 2023-11-16T16:48:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fd25ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc19ffc0 con 0x7f8fcc1014c0 2023-11-16T16:48:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fd25ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc1a0150 con 0x7f8fcc1002d0 2023-11-16T16:48:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fd0b9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fcc101e50 0x7f8fcc19f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fcb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fcc1014c0 0x7f8fcc198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fd0b9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fcc101e50 0x7f8fcc19f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39148/0 (socket says 172.21.15.44:39148) 2023-11-16T16:48:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fd0b9c700 1 -- 172.21.15.44:0/1801720456 learned_addr learned my addr 172.21.15.44:0/1801720456 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fcbfff700 1 --2- 172.21.15.44:0/1801720456 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8fcc1002d0 0x7f8fcc198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fd0b9c700 1 -- 172.21.15.44:0/1801720456 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8fcc1002d0 msgr2=0x7f8fcc198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fd0b9c700 1 --2- 172.21.15.44:0/1801720456 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8fcc1002d0 0x7f8fcc198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fd0b9c700 1 -- 172.21.15.44:0/1801720456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fcc1014c0 msgr2=0x7f8fcc198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fd0b9c700 1 --2- 172.21.15.44:0/1801720456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fcc1014c0 0x7f8fcc198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fd0b9c700 1 -- 172.21.15.44:0/1801720456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc0095f0 con 0x7f8fcc101e50 2023-11-16T16:48:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.261+0000 7f8fcb7fe700 1 --2- 172.21.15.44:0/1801720456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fcc1014c0 0x7f8fcc198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.262+0000 7f8fd0b9c700 1 --2- 172.21.15.44:0/1801720456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fcc101e50 0x7f8fcc19f8b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4007ee0 tx=0x7f8fc400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.262+0000 7f8fc97fa700 1 -- 172.21.15.44:0/1801720456 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc400dd20 con 0x7f8fcc101e50 2023-11-16T16:48:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.262+0000 7f8fd25ff700 1 -- 172.21.15.44:0/1801720456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc10bca0 con 0x7f8fcc101e50 2023-11-16T16:48:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.262+0000 7f8fc97fa700 1 -- 172.21.15.44:0/1801720456 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8fc400f040 con 0x7f8fcc101e50 2023-11-16T16:48:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.262+0000 7f8fc97fa700 1 -- 172.21.15.44:0/1801720456 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40048a0 con 0x7f8fcc101e50 2023-11-16T16:48:47.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.262+0000 7f8fd25ff700 1 -- 172.21.15.44:0/1801720456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fcc10bb10 con 0x7f8fcc101e50 2023-11-16T16:48:47.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.263+0000 7f8fd25ff700 1 -- 172.21.15.44:0/1801720456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc1a0840 con 0x7f8fcc101e50 2023-11-16T16:48:47.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.265+0000 7f8fc97fa700 1 -- 172.21.15.44:0/1801720456 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8fc4003a00 con 0x7f8fcc101e50 2023-11-16T16:48:47.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.266+0000 7f8fc97fa700 1 --2- 172.21.15.44:0/1801720456 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8fb40a1260 0x7f8fb40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:47.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.267+0000 7f8fc97fa700 1 -- 172.21.15.44:0/1801720456 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f8fc40cc670 con 0x7f8fcc101e50 2023-11-16T16:48:47.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.267+0000 7f8fcbfff700 1 --2- 172.21.15.44:0/1801720456 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8fb40a1260 0x7f8fb40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:47.269 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.267+0000 7f8fcbfff700 1 --2- 172.21.15.44:0/1801720456 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8fb40a1260 0x7f8fb40a3710 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00b2e0 tx=0x7f8fbc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:47.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.270+0000 7f8fc97fa700 1 -- 172.21.15.44:0/1801720456 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8fc404e730 con 0x7f8fcc101e50 2023-11-16T16:48:47.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.481+0000 7f8fd25ff700 1 -- 172.21.15.44:0/1801720456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8fcc1a0df0 con 0x7f8fcc101e50 2023-11-16T16:48:47.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.481+0000 7f8fc97fa700 1 -- 172.21.15.44:0/1801720456 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8fc4095860 con 0x7f8fcc101e50 2023-11-16T16:48:47.483 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:48:47.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.484+0000 7f8fd25ff700 1 -- 172.21.15.44:0/1801720456 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8fb40a1260 msgr2=0x7f8fb40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:47.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.484+0000 7f8fd25ff700 1 --2- 172.21.15.44:0/1801720456 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8fb40a1260 0x7f8fb40a3710 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00b2e0 tx=0x7f8fbc005fb0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.484+0000 7f8fd25ff700 1 -- 172.21.15.44:0/1801720456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fcc101e50 msgr2=0x7f8fcc19f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:47.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.484+0000 7f8fd25ff700 1 --2- 172.21.15.44:0/1801720456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fcc101e50 0x7f8fcc19f8b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4007ee0 tx=0x7f8fc400d3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.484+0000 7f8fd25ff700 1 -- 172.21.15.44:0/1801720456 shutdown_connections 2023-11-16T16:48:47.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.484+0000 7f8fd25ff700 1 --2- 172.21.15.44:0/1801720456 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8fb40a1260 0x7f8fb40a3710 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.484+0000 7f8fd25ff700 1 --2- 172.21.15.44:0/1801720456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fcc101e50 0x7f8fcc19f8b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.484+0000 7f8fd25ff700 1 --2- 172.21.15.44:0/1801720456 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8fcc1002d0 0x7f8fcc198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.484+0000 7f8fd25ff700 1 --2- 172.21.15.44:0/1801720456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fcc1014c0 0x7f8fcc198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.484+0000 7f8fd25ff700 1 -- 172.21.15.44:0/1801720456 >> 172.21.15.44:0/1801720456 conn(0x7f8fcc0fb8b0 msgr2=0x7f8fcc0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:47.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.484+0000 7f8fd25ff700 1 -- 172.21.15.44:0/1801720456 shutdown_connections 2023-11-16T16:48:47.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.484+0000 7f8fd25ff700 1 -- 172.21.15.44:0/1801720456 wait complete. 2023-11-16T16:48:47.506 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T16:48:47.715 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.713+0000 7f1ac81ff700 1 -- 172.21.15.44:0/3672702027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ac0101e40 msgr2=0x7f1ac0108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.713+0000 7f1ac81ff700 1 --2- 172.21.15.44:0/3672702027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ac0101e40 0x7f1ac0108700 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8009f90 tx=0x7f1ab800b330 comp rx=0 tx=0).stop 2023-11-16T16:48:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.714+0000 7f1ac81ff700 1 -- 172.21.15.44:0/3672702027 shutdown_connections 2023-11-16T16:48:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.714+0000 7f1ac81ff700 1 --2- 172.21.15.44:0/3672702027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ac0101e40 0x7f1ac0108700 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.714+0000 7f1ac81ff700 1 --2- 172.21.15.44:0/3672702027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ac01002c0 0x7f1ac01006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.714+0000 7f1ac81ff700 1 --2- 172.21.15.44:0/3672702027 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ac01014b0 0x7f1ac0101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.714+0000 7f1ac81ff700 1 -- 172.21.15.44:0/3672702027 >> 172.21.15.44:0/3672702027 conn(0x7f1ac00fb8a0 msgr2=0x7f1ac00fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.714+0000 7f1ac81ff700 1 -- 172.21.15.44:0/3672702027 shutdown_connections 2023-11-16T16:48:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.714+0000 7f1ac81ff700 1 -- 172.21.15.44:0/3672702027 wait complete. 2023-11-16T16:48:47.718 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.715+0000 7f1ac81ff700 1 Processor -- start 2023-11-16T16:48:47.718 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.715+0000 7f1ac81ff700 1 -- start start 2023-11-16T16:48:47.718 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.715+0000 7f1ac81ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ac01002c0 0x7f1ac0198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:47.718 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.715+0000 7f1ac81ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ac01014b0 0x7f1ac0198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:47.718 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.715+0000 7f1ac81ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ac0101e40 0x7f1ac019f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:47.718 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.715+0000 7f1ac81ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac019feb0 con 0x7f1ac01014b0 2023-11-16T16:48:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.715+0000 7f1ac81ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac019fff0 con 0x7f1ac0101e40 2023-11-16T16:48:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.715+0000 7f1ac81ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac01a0180 con 0x7f1ac01002c0 2023-11-16T16:48:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.715+0000 7f1ac579a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ac01014b0 0x7f1ac0198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.715+0000 7f1ac579a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ac01014b0 0x7f1ac0198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39164/0 (socket says 172.21.15.44:39164) 2023-11-16T16:48:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.715+0000 7f1ac5f9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ac01002c0 0x7f1ac0198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.715+0000 7f1ac579a700 1 -- 172.21.15.44:0/2942960470 learned_addr learned my addr 172.21.15.44:0/2942960470 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.716+0000 7f1ac679c700 1 --2- 172.21.15.44:0/2942960470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ac0101e40 0x7f1ac019f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.716+0000 7f1ac579a700 1 -- 172.21.15.44:0/2942960470 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ac01002c0 msgr2=0x7f1ac0198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.716+0000 7f1ac579a700 1 --2- 172.21.15.44:0/2942960470 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ac01002c0 0x7f1ac0198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.716+0000 7f1ac579a700 1 -- 172.21.15.44:0/2942960470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ac0101e40 msgr2=0x7f1ac019f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.716+0000 7f1ac579a700 1 --2- 172.21.15.44:0/2942960470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ac0101e40 0x7f1ac019f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.716+0000 7f1ac579a700 1 -- 172.21.15.44:0/2942960470 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc0095f0 con 0x7f1ac01014b0 2023-11-16T16:48:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.716+0000 7f1ac579a700 1 --2- 172.21.15.44:0/2942960470 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ac01014b0 0x7f1ac0198870 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000e9a0 tx=0x7f1ab000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.716+0000 7f1ab6ffd700 1 -- 172.21.15.44:0/2942960470 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0009790 con 0x7f1ac01014b0 2023-11-16T16:48:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.716+0000 7f1ab6ffd700 1 -- 172.21.15.44:0/2942960470 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ab0010430 con 0x7f1ac01014b0 2023-11-16T16:48:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.716+0000 7f1ab6ffd700 1 -- 172.21.15.44:0/2942960470 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0018680 con 0x7f1ac01014b0 2023-11-16T16:48:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.716+0000 7f1ac81ff700 1 -- 172.21.15.44:0/2942960470 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab8009c40 con 0x7f1ac01014b0 2023-11-16T16:48:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.717+0000 7f1ac81ff700 1 -- 172.21.15.44:0/2942960470 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac010bc90 con 0x7f1ac01014b0 2023-11-16T16:48:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.717+0000 7f1ac81ff700 1 -- 172.21.15.44:0/2942960470 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac0067b00 con 0x7f1ac01014b0 2023-11-16T16:48:47.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.719+0000 7f1ab6ffd700 1 -- 172.21.15.44:0/2942960470 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1ab00105a0 con 0x7f1ac01014b0 2023-11-16T16:48:47.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.721+0000 7f1ab6ffd700 1 --2- 172.21.15.44:0/2942960470 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1aac0a1260 0x7f1aac0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:47.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.721+0000 7f1ab6ffd700 1 -- 172.21.15.44:0/2942960470 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f1ab0014070 con 0x7f1ac01014b0 2023-11-16T16:48:47.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.721+0000 7f1ac5f9b700 1 --2- 172.21.15.44:0/2942960470 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1aac0a1260 0x7f1aac0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:47.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.722+0000 7f1ac5f9b700 1 --2- 172.21.15.44:0/2942960470 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1aac0a1260 0x7f1aac0a3710 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0101310 tx=0x7f1abc009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:47.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.724+0000 7f1ab6ffd700 1 -- 172.21.15.44:0/2942960470 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1ab0051730 con 0x7f1ac01014b0 2023-11-16T16:48:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:47 smithi044 ceph-mon[128750]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:47 smithi044 ceph-mon[128750]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2023-11-16T16:48:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:47 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3992778129' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:48:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:47 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1801720456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:48:47.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:47 smithi176 ceph-mon[131630]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:47.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:47 smithi176 ceph-mon[131630]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2023-11-16T16:48:47.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:47 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3992778129' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:48:47.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:47 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1801720456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:48:47.934 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.933+0000 7f1ac81ff700 1 -- 172.21.15.44:0/2942960470 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1ac01a0c40 con 0x7f1ac01014b0 2023-11-16T16:48:47.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.933+0000 7f1ab6ffd700 1 -- 172.21.15.44:0/2942960470 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1ab0096200 con 0x7f1ac01014b0 2023-11-16T16:48:47.935 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2023-11-16T16:48:47.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.936+0000 7f1ac81ff700 1 -- 172.21.15.44:0/2942960470 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1aac0a1260 msgr2=0x7f1aac0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:47.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.936+0000 7f1ac81ff700 1 --2- 172.21.15.44:0/2942960470 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1aac0a1260 0x7f1aac0a3710 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0101310 tx=0x7f1abc009310 comp rx=0 tx=0).stop 2023-11-16T16:48:47.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.936+0000 7f1ac81ff700 1 -- 172.21.15.44:0/2942960470 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ac01014b0 msgr2=0x7f1ac0198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:47.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.936+0000 7f1ac81ff700 1 --2- 172.21.15.44:0/2942960470 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ac01014b0 0x7f1ac0198870 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000e9a0 tx=0x7f1ab000ecb0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.936+0000 7f1ac81ff700 1 -- 172.21.15.44:0/2942960470 shutdown_connections 2023-11-16T16:48:47.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.936+0000 7f1ac81ff700 1 --2- 172.21.15.44:0/2942960470 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f1aac0a1260 0x7f1aac0a3710 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.936+0000 7f1ac81ff700 1 --2- 172.21.15.44:0/2942960470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ac0101e40 0x7f1ac019f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.936+0000 7f1ac81ff700 1 --2- 172.21.15.44:0/2942960470 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ac01002c0 0x7f1ac0198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.936+0000 7f1ac81ff700 1 --2- 172.21.15.44:0/2942960470 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ac01014b0 0x7f1ac0198870 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:47.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.936+0000 7f1ac81ff700 1 -- 172.21.15.44:0/2942960470 >> 172.21.15.44:0/2942960470 conn(0x7f1ac00fb8a0 msgr2=0x7f1ac00fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:47.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.936+0000 7f1ac81ff700 1 -- 172.21.15.44:0/2942960470 shutdown_connections 2023-11-16T16:48:47.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:47.936+0000 7f1ac81ff700 1 -- 172.21.15.44:0/2942960470 wait complete. 2023-11-16T16:48:47.956 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T16:48:47.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:47 smithi077 ceph-mon[131204]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:47.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:47 smithi077 ceph-mon[131204]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2023-11-16T16:48:47.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:47 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3992778129' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:48:47.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:47 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1801720456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:48:48.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.162+0000 7f917f3ff700 1 -- 172.21.15.44:0/1696518207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91781014c0 msgr2=0x7f9178101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:48.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.162+0000 7f917f3ff700 1 --2- 172.21.15.44:0/1696518207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91781014c0 0x7f9178101910 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9168009990 tx=0x7f9168009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.162+0000 7f917f3ff700 1 -- 172.21.15.44:0/1696518207 shutdown_connections 2023-11-16T16:48:48.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.162+0000 7f917f3ff700 1 --2- 172.21.15.44:0/1696518207 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9178101e50 0x7f9178108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.162+0000 7f917f3ff700 1 --2- 172.21.15.44:0/1696518207 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91781002d0 0x7f91781006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.162+0000 7f917f3ff700 1 --2- 172.21.15.44:0/1696518207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91781014c0 0x7f9178101910 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.165 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.162+0000 7f917f3ff700 1 -- 172.21.15.44:0/1696518207 >> 172.21.15.44:0/1696518207 conn(0x7f91780fb8b0 msgr2=0x7f91780fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:48.165 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.162+0000 7f917f3ff700 1 -- 172.21.15.44:0/1696518207 shutdown_connections 2023-11-16T16:48:48.165 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.162+0000 7f917f3ff700 1 -- 172.21.15.44:0/1696518207 wait complete. 2023-11-16T16:48:48.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.163+0000 7f917f3ff700 1 Processor -- start 2023-11-16T16:48:48.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.163+0000 7f917f3ff700 1 -- start start 2023-11-16T16:48:48.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.164+0000 7f917f3ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91781002d0 0x7f91780748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:48.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.164+0000 7f917f3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91781014c0 0x7f9178072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:48.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.164+0000 7f917f3ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9178101e50 0x7f9178073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:48.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.164+0000 7f917f3ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9178073bc0 con 0x7f91781002d0 2023-11-16T16:48:48.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.164+0000 7f917f3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9178073d30 con 0x7f91781014c0 2023-11-16T16:48:48.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.164+0000 7f917f3ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91781a00e0 con 0x7f9178101e50 2023-11-16T16:48:48.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.164+0000 7f917d19b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91781002d0 0x7f91780748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:48.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.164+0000 7f917d19b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91781002d0 0x7f91780748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39172/0 (socket says 172.21.15.44:39172) 2023-11-16T16:48:48.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.164+0000 7f917d19b700 1 -- 172.21.15.44:0/307431850 learned_addr learned my addr 172.21.15.44:0/307431850 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:48.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.164+0000 7f917c99a700 1 --2- 172.21.15.44:0/307431850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91781014c0 0x7f9178072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:48.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.164+0000 7f917d19b700 1 -- 172.21.15.44:0/307431850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9178101e50 msgr2=0x7f9178073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:48.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.164+0000 7f917d19b700 1 --2- 172.21.15.44:0/307431850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9178101e50 0x7f9178073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.164+0000 7f917d19b700 1 -- 172.21.15.44:0/307431850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91781014c0 msgr2=0x7f9178072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:48.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.164+0000 7f917d19b700 1 --2- 172.21.15.44:0/307431850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91781014c0 0x7f9178072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.165+0000 7f917d19b700 1 -- 172.21.15.44:0/307431850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91680095f0 con 0x7f91781002d0 2023-11-16T16:48:48.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.165+0000 7f917d19b700 1 --2- 172.21.15.44:0/307431850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91781002d0 0x7f91780748a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f917400e970 tx=0x7f917400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:48.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.165+0000 7f916e7fc700 1 -- 172.21.15.44:0/307431850 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9174009790 con 0x7f91781002d0 2023-11-16T16:48:48.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.165+0000 7f917f3ff700 1 -- 172.21.15.44:0/307431850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f917810bc80 con 0x7f91781002d0 2023-11-16T16:48:48.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.165+0000 7f916e7fc700 1 -- 172.21.15.44:0/307431850 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f917400eec0 con 0x7f91781002d0 2023-11-16T16:48:48.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.165+0000 7f916e7fc700 1 -- 172.21.15.44:0/307431850 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9174003cf0 con 0x7f91781002d0 2023-11-16T16:48:48.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.165+0000 7f917f3ff700 1 -- 172.21.15.44:0/307431850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f917810baf0 con 0x7f91781002d0 2023-11-16T16:48:48.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.167+0000 7f917f3ff700 1 -- 172.21.15.44:0/307431850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91781a0880 con 0x7f91781002d0 2023-11-16T16:48:48.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.171+0000 7f916e7fc700 1 -- 172.21.15.44:0/307431850 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f917400ccd0 con 0x7f91781002d0 2023-11-16T16:48:48.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.173+0000 7f916e7fc700 1 --2- 172.21.15.44:0/307431850 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f91640a11e0 0x7f91640a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:48.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.173+0000 7f916e7fc700 1 -- 172.21.15.44:0/307431850 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f9174014070 con 0x7f91781002d0 2023-11-16T16:48:48.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.173+0000 7f917c99a700 1 --2- 172.21.15.44:0/307431850 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f91640a11e0 0x7f91640a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:48.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.173+0000 7f917c99a700 1 --2- 172.21.15.44:0/307431850 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f91640a11e0 0x7f91640a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f916800b2e0 tx=0x7f9168005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:48.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.174+0000 7f916e7fc700 1 -- 172.21.15.44:0/307431850 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9174096420 con 0x7f91781002d0 2023-11-16T16:48:48.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.383+0000 7f917f3ff700 1 -- 172.21.15.44:0/307431850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9178107200 con 0x7f91781002d0 2023-11-16T16:48:48.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.384+0000 7f916e7fc700 1 -- 172.21.15.44:0/307431850 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9174099a40 con 0x7f91781002d0 2023-11-16T16:48:48.385 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:48:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.386+0000 7f917f3ff700 1 -- 172.21.15.44:0/307431850 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f91640a11e0 msgr2=0x7f91640a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.386+0000 7f917f3ff700 1 --2- 172.21.15.44:0/307431850 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f91640a11e0 0x7f91640a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f916800b2e0 tx=0x7f9168005fb0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.386+0000 7f917f3ff700 1 -- 172.21.15.44:0/307431850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91781002d0 msgr2=0x7f91780748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.386+0000 7f917f3ff700 1 --2- 172.21.15.44:0/307431850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91781002d0 0x7f91780748a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f917400e970 tx=0x7f917400ec80 comp rx=0 tx=0).stop 2023-11-16T16:48:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.386+0000 7f917f3ff700 1 -- 172.21.15.44:0/307431850 shutdown_connections 2023-11-16T16:48:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.386+0000 7f917f3ff700 1 --2- 172.21.15.44:0/307431850 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f91640a11e0 0x7f91640a3690 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.386+0000 7f917f3ff700 1 --2- 172.21.15.44:0/307431850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9178101e50 0x7f9178073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.386+0000 7f917f3ff700 1 --2- 172.21.15.44:0/307431850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91781002d0 0x7f91780748a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.386+0000 7f917f3ff700 1 --2- 172.21.15.44:0/307431850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91781014c0 0x7f9178072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.386+0000 7f917f3ff700 1 -- 172.21.15.44:0/307431850 >> 172.21.15.44:0/307431850 conn(0x7f91780fb8b0 msgr2=0x7f91780fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.386+0000 7f917f3ff700 1 -- 172.21.15.44:0/307431850 shutdown_connections 2023-11-16T16:48:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.386+0000 7f917f3ff700 1 -- 172.21.15.44:0/307431850 wait complete. 2023-11-16T16:48:48.406 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:48:48.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.609+0000 7f6e731ff700 1 -- 172.21.15.44:0/442223777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e6c101480 msgr2=0x7f6e6c1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:48.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.609+0000 7f6e731ff700 1 --2- 172.21.15.44:0/442223777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e6c101480 0x7f6e6c1018d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6e58009990 tx=0x7f6e58009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.609+0000 7f6e731ff700 1 -- 172.21.15.44:0/442223777 shutdown_connections 2023-11-16T16:48:48.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.609+0000 7f6e731ff700 1 --2- 172.21.15.44:0/442223777 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e6c101e10 0x7f6e6c1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.609+0000 7f6e731ff700 1 --2- 172.21.15.44:0/442223777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c100290 0x7f6e6c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.609+0000 7f6e731ff700 1 --2- 172.21.15.44:0/442223777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e6c101480 0x7f6e6c1018d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.609+0000 7f6e731ff700 1 -- 172.21.15.44:0/442223777 >> 172.21.15.44:0/442223777 conn(0x7f6e6c0fb870 msgr2=0x7f6e6c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:48.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.609+0000 7f6e731ff700 1 -- 172.21.15.44:0/442223777 shutdown_connections 2023-11-16T16:48:48.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.609+0000 7f6e731ff700 1 -- 172.21.15.44:0/442223777 wait complete. 2023-11-16T16:48:48.613 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.610+0000 7f6e731ff700 1 Processor -- start 2023-11-16T16:48:48.613 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.610+0000 7f6e731ff700 1 -- start start 2023-11-16T16:48:48.613 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.610+0000 7f6e731ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e6c100290 0x7f6e6c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.610+0000 7f6e731ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e6c101480 0x7f6e6c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.610+0000 7f6e731ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c101e10 0x7f6e6c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.611+0000 7f6e731ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c073bc0 con 0x7f6e6c100290 2023-11-16T16:48:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.611+0000 7f6e731ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c073d30 con 0x7f6e6c101e10 2023-11-16T16:48:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.611+0000 7f6e731ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c1a0030 con 0x7f6e6c101480 2023-11-16T16:48:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.611+0000 7f6e70f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e6c100290 0x7f6e6c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.611+0000 7f6e70f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e6c100290 0x7f6e6c0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39178/0 (socket says 172.21.15.44:39178) 2023-11-16T16:48:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.611+0000 7f6e70f9b700 1 -- 172.21.15.44:0/2032241741 learned_addr learned my addr 172.21.15.44:0/2032241741 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.611+0000 7f6e7179c700 1 --2- 172.21.15.44:0/2032241741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c101e10 0x7f6e6c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.611+0000 7f6e6bfff700 1 --2- 172.21.15.44:0/2032241741 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e6c101480 0x7f6e6c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.611+0000 7f6e70f9b700 1 -- 172.21.15.44:0/2032241741 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e6c101480 msgr2=0x7f6e6c072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:48.614 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.611+0000 7f6e70f9b700 1 --2- 172.21.15.44:0/2032241741 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e6c101480 0x7f6e6c072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.611+0000 7f6e70f9b700 1 -- 172.21.15.44:0/2032241741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c101e10 msgr2=0x7f6e6c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:48.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.611+0000 7f6e70f9b700 1 --2- 172.21.15.44:0/2032241741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c101e10 0x7f6e6c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.611+0000 7f6e70f9b700 1 -- 172.21.15.44:0/2032241741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e580095f0 con 0x7f6e6c100290 2023-11-16T16:48:48.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.611+0000 7f6e7179c700 1 --2- 172.21.15.44:0/2032241741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c101e10 0x7f6e6c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:48:48.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.611+0000 7f6e70f9b700 1 --2- 172.21.15.44:0/2032241741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e6c100290 0x7f6e6c0748a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000e970 tx=0x7f6e6000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:48.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.612+0000 7f6e69ffb700 1 -- 172.21.15.44:0/2032241741 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60009790 con 0x7f6e6c100290 2023-11-16T16:48:48.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.612+0000 7f6e69ffb700 1 -- 172.21.15.44:0/2032241741 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e600044e0 con 0x7f6e6c100290 2023-11-16T16:48:48.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.612+0000 7f6e731ff700 1 -- 172.21.15.44:0/2032241741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6c10bc50 con 0x7f6e6c100290 2023-11-16T16:48:48.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.612+0000 7f6e69ffb700 1 -- 172.21.15.44:0/2032241741 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60003e30 con 0x7f6e6c100290 2023-11-16T16:48:48.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.612+0000 7f6e731ff700 1 -- 172.21.15.44:0/2032241741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6c10bac0 con 0x7f6e6c100290 2023-11-16T16:48:48.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.613+0000 7f6e731ff700 1 -- 172.21.15.44:0/2032241741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6c1a07d0 con 0x7f6e6c100290 2023-11-16T16:48:48.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.615+0000 7f6e69ffb700 1 -- 172.21.15.44:0/2032241741 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6e60004000 con 0x7f6e6c100290 2023-11-16T16:48:48.618 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.616+0000 7f6e69ffb700 1 --2- 172.21.15.44:0/2032241741 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6e5c0a1260 0x7f6e5c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:48.618 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.618+0000 7f6e69ffb700 1 -- 172.21.15.44:0/2032241741 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f6e60014070 con 0x7f6e6c100290 2023-11-16T16:48:48.618 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.618+0000 7f6e6bfff700 1 --2- 172.21.15.44:0/2032241741 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6e5c0a1260 0x7f6e5c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:48.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.618+0000 7f6e6bfff700 1 --2- 172.21.15.44:0/2032241741 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6e5c0a1260 0x7f6e5c0a3710 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6e5800b2e0 tx=0x7f6e58005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:48.621 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.620+0000 7f6e69ffb700 1 -- 172.21.15.44:0/2032241741 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6e600962f0 con 0x7f6e6c100290 2023-11-16T16:48:48.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:48 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2942960470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:48:48.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:48 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/307431850' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:48:48.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.878+0000 7f6e731ff700 1 -- 172.21.15.44:0/2032241741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e6c1071c0 con 0x7f6e6c100290 2023-11-16T16:48:48.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.878+0000 7f6e69ffb700 1 -- 172.21.15.44:0/2032241741 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f6e60099910 con 0x7f6e6c100290 2023-11-16T16:48:48.879 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:48:48.879 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:48:48.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.880+0000 7f6e731ff700 1 -- 172.21.15.44:0/2032241741 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6e5c0a1260 msgr2=0x7f6e5c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:48.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.880+0000 7f6e731ff700 1 --2- 172.21.15.44:0/2032241741 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6e5c0a1260 0x7f6e5c0a3710 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6e5800b2e0 tx=0x7f6e58005fb0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.880+0000 7f6e731ff700 1 -- 172.21.15.44:0/2032241741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e6c100290 msgr2=0x7f6e6c0748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:48.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.880+0000 7f6e731ff700 1 --2- 172.21.15.44:0/2032241741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e6c100290 0x7f6e6c0748a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000e970 tx=0x7f6e6000ed30 comp rx=0 tx=0).stop 2023-11-16T16:48:48.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.880+0000 7f6e731ff700 1 -- 172.21.15.44:0/2032241741 shutdown_connections 2023-11-16T16:48:48.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.880+0000 7f6e731ff700 1 --2- 172.21.15.44:0/2032241741 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6e5c0a1260 0x7f6e5c0a3710 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.880+0000 7f6e731ff700 1 --2- 172.21.15.44:0/2032241741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c101e10 0x7f6e6c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.880+0000 7f6e731ff700 1 --2- 172.21.15.44:0/2032241741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e6c100290 0x7f6e6c0748a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.880+0000 7f6e731ff700 1 --2- 172.21.15.44:0/2032241741 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e6c101480 0x7f6e6c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.880+0000 7f6e731ff700 1 -- 172.21.15.44:0/2032241741 >> 172.21.15.44:0/2032241741 conn(0x7f6e6c0fb870 msgr2=0x7f6e6c0fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:48.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.880+0000 7f6e731ff700 1 -- 172.21.15.44:0/2032241741 shutdown_connections 2023-11-16T16:48:48.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:48.880+0000 7f6e731ff700 1 -- 172.21.15.44:0/2032241741 wait complete. 2023-11-16T16:48:48.883 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:48:48.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:48 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2942960470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:48:48.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:48 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/307431850' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:48:48.901 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:48:48.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:48 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2942960470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:48:48.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:48 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/307431850' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:48:49.104 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.103+0000 7f6a7d39d700 1 -- 172.21.15.44:0/814870668 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a78101340 msgr2=0x7f6a781017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:49.104 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.103+0000 7f6a7d39d700 1 --2- 172.21.15.44:0/814870668 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a78101340 0x7f6a781017b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009990 tx=0x7f6a6c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.104 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.103+0000 7f6a7d39d700 1 -- 172.21.15.44:0/814870668 shutdown_connections 2023-11-16T16:48:49.104 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.103+0000 7f6a7d39d700 1 --2- 172.21.15.44:0/814870668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a78101cf0 0x7f6a78108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.105 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.103+0000 7f6a7d39d700 1 --2- 172.21.15.44:0/814870668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a781000b0 0x7f6a781004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.105 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.103+0000 7f6a7d39d700 1 --2- 172.21.15.44:0/814870668 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a78101340 0x7f6a781017b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.105 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.103+0000 7f6a7d39d700 1 -- 172.21.15.44:0/814870668 >> 172.21.15.44:0/814870668 conn(0x7f6a780fb6d0 msgr2=0x7f6a780fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:49.105 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.103+0000 7f6a7d39d700 1 -- 172.21.15.44:0/814870668 shutdown_connections 2023-11-16T16:48:49.105 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.103+0000 7f6a7d39d700 1 -- 172.21.15.44:0/814870668 wait complete. 2023-11-16T16:48:49.107 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.104+0000 7f6a7d39d700 1 Processor -- start 2023-11-16T16:48:49.107 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.105+0000 7f6a7d39d700 1 -- start start 2023-11-16T16:48:49.107 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.105+0000 7f6a7d39d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a781000b0 0x7f6a78198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:49.107 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.105+0000 7f6a7d39d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a78101340 0x7f6a78198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:49.107 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.105+0000 7f6a7d39d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a78101cf0 0x7f6a7819f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:49.107 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.105+0000 7f6a7d39d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7819fba0 con 0x7f6a78101340 2023-11-16T16:48:49.107 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.105+0000 7f6a7d39d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7819fce0 con 0x7f6a78101cf0 2023-11-16T16:48:49.107 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.105+0000 7f6a7d39d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7819fe70 con 0x7f6a781000b0 2023-11-16T16:48:49.107 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.105+0000 7f6a777fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a78101340 0x7f6a78198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:49.107 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.105+0000 7f6a777fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a78101340 0x7f6a78198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39188/0 (socket says 172.21.15.44:39188) 2023-11-16T16:48:49.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.105+0000 7f6a777fe700 1 -- 172.21.15.44:0/1530285764 learned_addr learned my addr 172.21.15.44:0/1530285764 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:49.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.105+0000 7f6a77fff700 1 --2- 172.21.15.44:0/1530285764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a781000b0 0x7f6a78198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:49.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.105+0000 7f6a7cb9c700 1 --2- 172.21.15.44:0/1530285764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a78101cf0 0x7f6a7819f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:49.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.106+0000 7f6a777fe700 1 -- 172.21.15.44:0/1530285764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a781000b0 msgr2=0x7f6a78198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:49.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.106+0000 7f6a777fe700 1 --2- 172.21.15.44:0/1530285764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a781000b0 0x7f6a78198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.106+0000 7f6a777fe700 1 -- 172.21.15.44:0/1530285764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a78101cf0 msgr2=0x7f6a7819f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:49.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.106+0000 7f6a777fe700 1 --2- 172.21.15.44:0/1530285764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a78101cf0 0x7f6a7819f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.106+0000 7f6a777fe700 1 -- 172.21.15.44:0/1530285764 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c0095f0 con 0x7f6a78101340 2023-11-16T16:48:49.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.106+0000 7f6a777fe700 1 --2- 172.21.15.44:0/1530285764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a78101340 0x7f6a78198560 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c004740 tx=0x7f6a6c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:49.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.106+0000 7f6a757fa700 1 -- 172.21.15.44:0/1530285764 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c01d070 con 0x7f6a78101340 2023-11-16T16:48:49.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.106+0000 7f6a757fa700 1 -- 172.21.15.44:0/1530285764 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a6c004830 con 0x7f6a78101340 2023-11-16T16:48:49.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.106+0000 7f6a7d39d700 1 -- 172.21.15.44:0/1530285764 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7810baa0 con 0x7f6a78101340 2023-11-16T16:48:49.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.106+0000 7f6a757fa700 1 -- 172.21.15.44:0/1530285764 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c0053c0 con 0x7f6a78101340 2023-11-16T16:48:49.109 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.106+0000 7f6a7d39d700 1 -- 172.21.15.44:0/1530285764 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7810b910 con 0x7f6a78101340 2023-11-16T16:48:49.109 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.107+0000 7f6a7d39d700 1 -- 172.21.15.44:0/1530285764 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a781926f0 con 0x7f6a78101340 2023-11-16T16:48:49.110 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.109+0000 7f6a757fa700 1 -- 172.21.15.44:0/1530285764 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6a6c005560 con 0x7f6a78101340 2023-11-16T16:48:49.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.111+0000 7f6a757fa700 1 --2- 172.21.15.44:0/1530285764 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6a600a1260 0x7f6a600a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:49.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.111+0000 7f6a757fa700 1 -- 172.21.15.44:0/1530285764 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f6a6c0cdf90 con 0x7f6a78101340 2023-11-16T16:48:49.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.111+0000 7f6a77fff700 1 --2- 172.21.15.44:0/1530285764 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6a600a1260 0x7f6a600a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:49.112 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.111+0000 7f6a77fff700 1 --2- 172.21.15.44:0/1530285764 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6a600a1260 0x7f6a600a3710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009c60 tx=0x7f6a68009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:49.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.114+0000 7f6a757fa700 1 -- 172.21.15.44:0/1530285764 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6a6c05a730 con 0x7f6a78101340 2023-11-16T16:48:49.324 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.324+0000 7f6a7d39d700 1 -- 172.21.15.44:0/1530285764 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a78067980 con 0x7f6a78101340 2023-11-16T16:48:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.326+0000 7f6a757fa700 1 -- 172.21.15.44:0/1530285764 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24685 (secure 0 0 0) 0x7f6a6c097210 con 0x7f6a78101340 2023-11-16T16:48:49.327 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:48:49.327 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:48:44.613706+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:48:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.328+0000 7f6a7d39d700 1 -- 172.21.15.44:0/1530285764 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6a600a1260 msgr2=0x7f6a600a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.328+0000 7f6a7d39d700 1 --2- 172.21.15.44:0/1530285764 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6a600a1260 0x7f6a600a3710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009c60 tx=0x7f6a68009260 comp rx=0 tx=0).stop 2023-11-16T16:48:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.328+0000 7f6a7d39d700 1 -- 172.21.15.44:0/1530285764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a78101340 msgr2=0x7f6a78198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.328+0000 7f6a7d39d700 1 --2- 172.21.15.44:0/1530285764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a78101340 0x7f6a78198560 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c004740 tx=0x7f6a6c009580 comp rx=0 tx=0).stop 2023-11-16T16:48:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.328+0000 7f6a7d39d700 1 -- 172.21.15.44:0/1530285764 shutdown_connections 2023-11-16T16:48:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.328+0000 7f6a7d39d700 1 --2- 172.21.15.44:0/1530285764 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f6a600a1260 0x7f6a600a3710 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.328+0000 7f6a7d39d700 1 --2- 172.21.15.44:0/1530285764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a78101cf0 0x7f6a7819f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.328+0000 7f6a7d39d700 1 --2- 172.21.15.44:0/1530285764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a781000b0 0x7f6a78198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.328+0000 7f6a7d39d700 1 --2- 172.21.15.44:0/1530285764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a78101340 0x7f6a78198560 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.328+0000 7f6a7d39d700 1 -- 172.21.15.44:0/1530285764 >> 172.21.15.44:0/1530285764 conn(0x7f6a780fb6d0 msgr2=0x7f6a78106c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.328+0000 7f6a7d39d700 1 -- 172.21.15.44:0/1530285764 shutdown_connections 2023-11-16T16:48:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.329+0000 7f6a7d39d700 1 -- 172.21.15.44:0/1530285764 wait complete. 2023-11-16T16:48:49.349 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:03.059622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:16.211826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:17.601591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:21.136084+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T16:48:49.349 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:48:49.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.555+0000 7fc47c1ff700 1 -- 172.21.15.44:0/3323591096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc474101e50 msgr2=0x7fc474108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:49.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.555+0000 7fc47c1ff700 1 --2- 172.21.15.44:0/3323591096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc474101e50 0x7fc474108710 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc46c009f90 tx=0x7fc46c00b330 comp rx=0 tx=0).stop 2023-11-16T16:48:49.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.555+0000 7fc47c1ff700 1 -- 172.21.15.44:0/3323591096 shutdown_connections 2023-11-16T16:48:49.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.555+0000 7fc47c1ff700 1 --2- 172.21.15.44:0/3323591096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc474101e50 0x7fc474108710 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.555+0000 7fc47c1ff700 1 --2- 172.21.15.44:0/3323591096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4741002d0 0x7fc4741006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.555+0000 7fc47c1ff700 1 --2- 172.21.15.44:0/3323591096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4741014c0 0x7fc474101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.555+0000 7fc47c1ff700 1 -- 172.21.15.44:0/3323591096 >> 172.21.15.44:0/3323591096 conn(0x7fc4740fb8b0 msgr2=0x7fc4740fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:49.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.555+0000 7fc47c1ff700 1 -- 172.21.15.44:0/3323591096 shutdown_connections 2023-11-16T16:48:49.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.555+0000 7fc47c1ff700 1 -- 172.21.15.44:0/3323591096 wait complete. 2023-11-16T16:48:49.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.556+0000 7fc47c1ff700 1 Processor -- start 2023-11-16T16:48:49.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.557+0000 7fc47c1ff700 1 -- start start 2023-11-16T16:48:49.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.557+0000 7fc47c1ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4741002d0 0x7fc474196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:49.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.557+0000 7fc47c1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4741014c0 0x7fc474196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:49.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.557+0000 7fc47c1ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc474101e50 0x7fc47419d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:49.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.557+0000 7fc47c1ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47419dc90 con 0x7fc474101e50 2023-11-16T16:48:49.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.557+0000 7fc47c1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47419ddd0 con 0x7fc4741014c0 2023-11-16T16:48:49.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.557+0000 7fc47c1ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47419df60 con 0x7fc4741002d0 2023-11-16T16:48:49.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.557+0000 7fc47a79c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc474101e50 0x7fc47419d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:49.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.557+0000 7fc47979a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4741014c0 0x7fc474196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:49.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.557+0000 7fc47a79c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc474101e50 0x7fc47419d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39214/0 (socket says 172.21.15.44:39214) 2023-11-16T16:48:49.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.557+0000 7fc47a79c700 1 -- 172.21.15.44:0/3491733398 learned_addr learned my addr 172.21.15.44:0/3491733398 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:49.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.557+0000 7fc479f9b700 1 --2- 172.21.15.44:0/3491733398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4741002d0 0x7fc474196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:49.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.558+0000 7fc47a79c700 1 -- 172.21.15.44:0/3491733398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4741002d0 msgr2=0x7fc474196110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:49.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.558+0000 7fc47a79c700 1 --2- 172.21.15.44:0/3491733398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4741002d0 0x7fc474196110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.558+0000 7fc47a79c700 1 -- 172.21.15.44:0/3491733398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4741014c0 msgr2=0x7fc474196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:49.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.558+0000 7fc47a79c700 1 --2- 172.21.15.44:0/3491733398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4741014c0 0x7fc474196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.558+0000 7fc47a79c700 1 -- 172.21.15.44:0/3491733398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46c009c40 con 0x7fc474101e50 2023-11-16T16:48:49.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.558+0000 7fc47979a700 1 --2- 172.21.15.44:0/3491733398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4741014c0 0x7fc474196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:49.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.558+0000 7fc47a79c700 1 --2- 172.21.15.44:0/3491733398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc474101e50 0x7fc47419d6c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc46c007930 tx=0x7fc46c004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:49.561 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.558+0000 7fc46affd700 1 -- 172.21.15.44:0/3491733398 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c00e040 con 0x7fc474101e50 2023-11-16T16:48:49.561 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.558+0000 7fc46affd700 1 -- 172.21.15.44:0/3491733398 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc46c00bda0 con 0x7fc474101e50 2023-11-16T16:48:49.561 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.558+0000 7fc47c1ff700 1 -- 172.21.15.44:0/3491733398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47410bc60 con 0x7fc474101e50 2023-11-16T16:48:49.561 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.558+0000 7fc46affd700 1 -- 172.21.15.44:0/3491733398 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c0052a0 con 0x7fc474101e50 2023-11-16T16:48:49.561 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.558+0000 7fc47c1ff700 1 -- 172.21.15.44:0/3491733398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47410bad0 con 0x7fc474101e50 2023-11-16T16:48:49.561 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.560+0000 7fc47c1ff700 1 -- 172.21.15.44:0/3491733398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4741077f0 con 0x7fc474101e50 2023-11-16T16:48:49.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.562+0000 7fc46affd700 1 -- 172.21.15.44:0/3491733398 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc46c01d470 con 0x7fc474101e50 2023-11-16T16:48:49.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.564+0000 7fc46affd700 1 --2- 172.21.15.44:0/3491733398 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc4600a1260 0x7fc4600a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:49.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.564+0000 7fc46affd700 1 -- 172.21.15.44:0/3491733398 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fc46c0ce340 con 0x7fc474101e50 2023-11-16T16:48:49.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.564+0000 7fc479f9b700 1 --2- 172.21.15.44:0/3491733398 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc4600a1260 0x7fc4600a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:49.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.565+0000 7fc479f9b700 1 --2- 172.21.15.44:0/3491733398 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc4600a1260 0x7fc4600a3710 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc474101320 tx=0x7fc470009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:49.568 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.568+0000 7fc46affd700 1 -- 172.21.15.44:0/3491733398 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc46c0976c0 con 0x7fc474101e50 2023-11-16T16:48:49.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.780+0000 7fc47c1ff700 1 -- 172.21.15.44:0/3491733398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc474061700 con 0x7fc474101e50 2023-11-16T16:48:49.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.780+0000 7fc46affd700 1 -- 172.21.15.44:0/3491733398 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc46c02e350 con 0x7fc474101e50 2023-11-16T16:48:49.781 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2023-11-16T16:48:49.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.783+0000 7fc47c1ff700 1 -- 172.21.15.44:0/3491733398 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc4600a1260 msgr2=0x7fc4600a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:49.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.783+0000 7fc47c1ff700 1 --2- 172.21.15.44:0/3491733398 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc4600a1260 0x7fc4600a3710 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc474101320 tx=0x7fc470009260 comp rx=0 tx=0).stop 2023-11-16T16:48:49.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.783+0000 7fc47c1ff700 1 -- 172.21.15.44:0/3491733398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc474101e50 msgr2=0x7fc47419d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:49.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.783+0000 7fc47c1ff700 1 --2- 172.21.15.44:0/3491733398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc474101e50 0x7fc47419d6c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc46c007930 tx=0x7fc46c004ce0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.783+0000 7fc47c1ff700 1 -- 172.21.15.44:0/3491733398 shutdown_connections 2023-11-16T16:48:49.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.783+0000 7fc47c1ff700 1 --2- 172.21.15.44:0/3491733398 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc4600a1260 0x7fc4600a3710 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.783+0000 7fc47c1ff700 1 --2- 172.21.15.44:0/3491733398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc474101e50 0x7fc47419d6c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.783+0000 7fc47c1ff700 1 --2- 172.21.15.44:0/3491733398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4741002d0 0x7fc474196110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.783+0000 7fc47c1ff700 1 --2- 172.21.15.44:0/3491733398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4741014c0 0x7fc474196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:49.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.783+0000 7fc47c1ff700 1 -- 172.21.15.44:0/3491733398 >> 172.21.15.44:0/3491733398 conn(0x7fc4740fb8b0 msgr2=0x7fc474104930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:49.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.783+0000 7fc47c1ff700 1 -- 172.21.15.44:0/3491733398 shutdown_connections 2023-11-16T16:48:49.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:49.783+0000 7fc47c1ff700 1 -- 172.21.15.44:0/3491733398 wait complete. 2023-11-16T16:48:49.803 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T16:48:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:49 smithi176 ceph-mon[131630]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2023-11-16T16:48:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:49 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2032241741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:49 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1530285764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:49.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:49 smithi077 ceph-mon[131204]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2023-11-16T16:48:49.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:49 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2032241741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:49.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:49 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1530285764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:50.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.008+0000 7f99123ff700 1 -- 172.21.15.44:0/1656124617 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f990c074b30 msgr2=0x7f990c072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:50.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.008+0000 7f99123ff700 1 --2- 172.21.15.44:0/1656124617 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f990c074b30 0x7f990c072f80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f98fc009990 tx=0x7f98fc009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.008+0000 7f99123ff700 1 -- 172.21.15.44:0/1656124617 shutdown_connections 2023-11-16T16:48:50.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.008+0000 7f99123ff700 1 --2- 172.21.15.44:0/1656124617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f990c108180 0x7f990c10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.008+0000 7f99123ff700 1 --2- 172.21.15.44:0/1656124617 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f990c074b30 0x7f990c072f80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.008+0000 7f99123ff700 1 --2- 172.21.15.44:0/1656124617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f990c073550 0x7f990c0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.010 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.008+0000 7f99123ff700 1 -- 172.21.15.44:0/1656124617 >> 172.21.15.44:0/1656124617 conn(0x7f990c0fb8a0 msgr2=0x7f990c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:50.010 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.008+0000 7f99123ff700 1 -- 172.21.15.44:0/1656124617 shutdown_connections 2023-11-16T16:48:50.010 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.008+0000 7f99123ff700 1 -- 172.21.15.44:0/1656124617 wait complete. 2023-11-16T16:48:50.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.009+0000 7f99123ff700 1 Processor -- start 2023-11-16T16:48:50.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.010+0000 7f99123ff700 1 -- start start 2023-11-16T16:48:50.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.010+0000 7f99123ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f990c073550 0x7f990c198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:50.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.010+0000 7f99123ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f990c074b30 0x7f990c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:50.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.010+0000 7f99123ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f990c108180 0x7f990c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:50.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.010+0000 7f99123ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c19fd80 con 0x7f990c073550 2023-11-16T16:48:50.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.010+0000 7f99123ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c19fec0 con 0x7f990c108180 2023-11-16T16:48:50.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.010+0000 7f99123ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c1a0050 con 0x7f990c074b30 2023-11-16T16:48:50.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.010+0000 7f990bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f990c073550 0x7f990c198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:50.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.010+0000 7f990bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f990c073550 0x7f990c198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39244/0 (socket says 172.21.15.44:39244) 2023-11-16T16:48:50.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.010+0000 7f990bfff700 1 -- 172.21.15.44:0/2748792310 learned_addr learned my addr 172.21.15.44:0/2748792310 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:50.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.010+0000 7f990b7fe700 1 --2- 172.21.15.44:0/2748792310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f990c074b30 0x7f990c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:50.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.010+0000 7f991099c700 1 --2- 172.21.15.44:0/2748792310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f990c108180 0x7f990c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:50.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.011+0000 7f990bfff700 1 -- 172.21.15.44:0/2748792310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f990c074b30 msgr2=0x7f990c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:50.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.011+0000 7f990bfff700 1 --2- 172.21.15.44:0/2748792310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f990c074b30 0x7f990c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.011+0000 7f990bfff700 1 -- 172.21.15.44:0/2748792310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f990c108180 msgr2=0x7f990c19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:50.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.011+0000 7f990bfff700 1 --2- 172.21.15.44:0/2748792310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f990c108180 0x7f990c19f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.011+0000 7f990bfff700 1 -- 172.21.15.44:0/2748792310 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc0095f0 con 0x7f990c073550 2023-11-16T16:48:50.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.011+0000 7f991099c700 1 --2- 172.21.15.44:0/2748792310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f990c108180 0x7f990c19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:48:50.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.011+0000 7f990bfff700 1 --2- 172.21.15.44:0/2748792310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f990c073550 0x7f990c198200 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f98fc004710 tx=0x7f98fc009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:50.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.011+0000 7f99097fa700 1 -- 172.21.15.44:0/2748792310 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc01d070 con 0x7f990c073550 2023-11-16T16:48:50.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.011+0000 7f99123ff700 1 -- 172.21.15.44:0/2748792310 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f990c10de80 con 0x7f990c073550 2023-11-16T16:48:50.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.011+0000 7f99097fa700 1 -- 172.21.15.44:0/2748792310 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98fc021410 con 0x7f990c073550 2023-11-16T16:48:50.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.011+0000 7f99097fa700 1 -- 172.21.15.44:0/2748792310 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc017650 con 0x7f990c073550 2023-11-16T16:48:50.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.011+0000 7f99123ff700 1 -- 172.21.15.44:0/2748792310 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990c10dcf0 con 0x7f990c073550 2023-11-16T16:48:50.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.013+0000 7f99123ff700 1 -- 172.21.15.44:0/2748792310 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990c106830 con 0x7f990c073550 2023-11-16T16:48:50.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.014+0000 7f99097fa700 1 -- 172.21.15.44:0/2748792310 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f98fc021580 con 0x7f990c073550 2023-11-16T16:48:50.017 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.016+0000 7f99097fa700 1 --2- 172.21.15.44:0/2748792310 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f98f40a11e0 0x7f98f40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:50.017 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.017+0000 7f99097fa700 1 -- 172.21.15.44:0/2748792310 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f98fc0ce320 con 0x7f990c073550 2023-11-16T16:48:50.017 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.017+0000 7f990b7fe700 1 --2- 172.21.15.44:0/2748792310 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f98f40a11e0 0x7f98f40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:50.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.017+0000 7f990b7fe700 1 --2- 172.21.15.44:0/2748792310 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f98f40a11e0 0x7f98f40a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f990c074850 tx=0x7f9900009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:50.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.020+0000 7f99097fa700 1 -- 172.21.15.44:0/2748792310 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f98fc0975a0 con 0x7f990c073550 2023-11-16T16:48:50.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:49 smithi044 ceph-mon[128750]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2023-11-16T16:48:50.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:49 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2032241741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:50.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:49 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1530285764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.233+0000 7f99123ff700 1 -- 172.21.15.44:0/2748792310 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f990c067b00 con 0x7f990c073550 2023-11-16T16:48:50.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.233+0000 7f99097fa700 1 -- 172.21.15.44:0/2748792310 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f98fc09abc0 con 0x7f990c073550 2023-11-16T16:48:50.234 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:48:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.236+0000 7f99123ff700 1 -- 172.21.15.44:0/2748792310 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f98f40a11e0 msgr2=0x7f98f40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.236+0000 7f99123ff700 1 --2- 172.21.15.44:0/2748792310 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f98f40a11e0 0x7f98f40a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f990c074850 tx=0x7f9900009260 comp rx=0 tx=0).stop 2023-11-16T16:48:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.236+0000 7f99123ff700 1 -- 172.21.15.44:0/2748792310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f990c073550 msgr2=0x7f990c198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.236+0000 7f99123ff700 1 --2- 172.21.15.44:0/2748792310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f990c073550 0x7f990c198200 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f98fc004710 tx=0x7f98fc009580 comp rx=0 tx=0).stop 2023-11-16T16:48:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.236+0000 7f99123ff700 1 -- 172.21.15.44:0/2748792310 shutdown_connections 2023-11-16T16:48:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.236+0000 7f99123ff700 1 --2- 172.21.15.44:0/2748792310 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f98f40a11e0 0x7f98f40a3690 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.236+0000 7f99123ff700 1 --2- 172.21.15.44:0/2748792310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f990c108180 0x7f990c19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.236+0000 7f99123ff700 1 --2- 172.21.15.44:0/2748792310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f990c073550 0x7f990c198200 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.236+0000 7f99123ff700 1 --2- 172.21.15.44:0/2748792310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f990c074b30 0x7f990c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.236+0000 7f99123ff700 1 -- 172.21.15.44:0/2748792310 >> 172.21.15.44:0/2748792310 conn(0x7f990c0fb8a0 msgr2=0x7f990c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.236+0000 7f99123ff700 1 -- 172.21.15.44:0/2748792310 shutdown_connections 2023-11-16T16:48:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.236+0000 7f99123ff700 1 -- 172.21.15.44:0/2748792310 wait complete. 2023-11-16T16:48:50.256 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T16:48:50.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.462+0000 7fb73c9ff700 1 -- 172.21.15.44:0/3894603205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7381014b0 msgr2=0x7fb738101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:50.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.462+0000 7fb73c9ff700 1 --2- 172.21.15.44:0/3894603205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7381014b0 0x7fb738101900 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb720009990 tx=0x7fb720009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.462+0000 7fb73c9ff700 1 -- 172.21.15.44:0/3894603205 shutdown_connections 2023-11-16T16:48:50.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.462+0000 7fb73c9ff700 1 --2- 172.21.15.44:0/3894603205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb738101e40 0x7fb738108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.462+0000 7fb73c9ff700 1 --2- 172.21.15.44:0/3894603205 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7381002c0 0x7fb7381006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.464 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.462+0000 7fb73c9ff700 1 --2- 172.21.15.44:0/3894603205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7381014b0 0x7fb738101900 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.464 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.462+0000 7fb73c9ff700 1 -- 172.21.15.44:0/3894603205 >> 172.21.15.44:0/3894603205 conn(0x7fb7380fb8a0 msgr2=0x7fb7380fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:50.464 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.462+0000 7fb73c9ff700 1 -- 172.21.15.44:0/3894603205 shutdown_connections 2023-11-16T16:48:50.464 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.462+0000 7fb73c9ff700 1 -- 172.21.15.44:0/3894603205 wait complete. 2023-11-16T16:48:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.463+0000 7fb73c9ff700 1 Processor -- start 2023-11-16T16:48:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.464+0000 7fb73c9ff700 1 -- start start 2023-11-16T16:48:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.464+0000 7fb73c9ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7381002c0 0x7fb7380748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.464+0000 7fb73c9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7381014b0 0x7fb738072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.464+0000 7fb73c9ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb738101e40 0x7fb738073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.464+0000 7fb73c9ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb738073bc0 con 0x7fb738101e40 2023-11-16T16:48:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.464+0000 7fb73c9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb738073d30 con 0x7fb7381014b0 2023-11-16T16:48:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.464+0000 7fb73c9ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7381a0120 con 0x7fb7381002c0 2023-11-16T16:48:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.464+0000 7fb73659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7381002c0 0x7fb7380748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:50.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.464+0000 7fb73659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7381002c0 0x7fb7380748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:54676/0 (socket says 172.21.15.44:54676) 2023-11-16T16:48:50.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.464+0000 7fb73659c700 1 -- 172.21.15.44:0/1761268074 learned_addr learned my addr 172.21.15.44:0/1761268074 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:50.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.464+0000 7fb736d9d700 1 --2- 172.21.15.44:0/1761268074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb738101e40 0x7fb738073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:50.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.464+0000 7fb72ffff700 1 --2- 172.21.15.44:0/1761268074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7381014b0 0x7fb738072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:50.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.465+0000 7fb73659c700 1 -- 172.21.15.44:0/1761268074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7381014b0 msgr2=0x7fb738072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:50.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.465+0000 7fb73659c700 1 --2- 172.21.15.44:0/1761268074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7381014b0 0x7fb738072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.465+0000 7fb73659c700 1 -- 172.21.15.44:0/1761268074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb738101e40 msgr2=0x7fb738073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:50.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.465+0000 7fb73659c700 1 --2- 172.21.15.44:0/1761268074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb738101e40 0x7fb738073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.465+0000 7fb73659c700 1 -- 172.21.15.44:0/1761268074 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7200095f0 con 0x7fb7381002c0 2023-11-16T16:48:50.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.465+0000 7fb73659c700 1 --2- 172.21.15.44:0/1761268074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7381002c0 0x7fb7380748a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb72800e970 tx=0x7fb72800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:50.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.465+0000 7fb72f7fe700 1 -- 172.21.15.44:0/1761268074 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb728009790 con 0x7fb7381002c0 2023-11-16T16:48:50.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.465+0000 7fb73c9ff700 1 -- 172.21.15.44:0/1761268074 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73810bc70 con 0x7fb7381002c0 2023-11-16T16:48:50.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.465+0000 7fb72f7fe700 1 -- 172.21.15.44:0/1761268074 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb72800c7e0 con 0x7fb7381002c0 2023-11-16T16:48:50.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.465+0000 7fb72f7fe700 1 -- 172.21.15.44:0/1761268074 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb728003c70 con 0x7fb7381002c0 2023-11-16T16:48:50.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.465+0000 7fb73c9ff700 1 -- 172.21.15.44:0/1761268074 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73810bae0 con 0x7fb7381002c0 2023-11-16T16:48:50.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.466+0000 7fb73c9ff700 1 -- 172.21.15.44:0/1761268074 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb738107860 con 0x7fb7381002c0 2023-11-16T16:48:50.469 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.468+0000 7fb72f7fe700 1 -- 172.21.15.44:0/1761268074 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb72800cc50 con 0x7fb7381002c0 2023-11-16T16:48:50.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.470+0000 7fb72f7fe700 1 --2- 172.21.15.44:0/1761268074 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb7240a11e0 0x7fb7240a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:50.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.470+0000 7fb72f7fe700 1 -- 172.21.15.44:0/1761268074 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fb728014070 con 0x7fb7381002c0 2023-11-16T16:48:50.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.470+0000 7fb72ffff700 1 --2- 172.21.15.44:0/1761268074 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb7240a11e0 0x7fb7240a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:50.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.471+0000 7fb72ffff700 1 --2- 172.21.15.44:0/1761268074 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb7240a11e0 0x7fb7240a3690 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb72000b2e0 tx=0x7fb720005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:50.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.472+0000 7fb72f7fe700 1 -- 172.21.15.44:0/1761268074 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb728096380 con 0x7fb7381002c0 2023-11-16T16:48:50.685 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.684+0000 7fb73c9ff700 1 -- 172.21.15.44:0/1761268074 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb738061ad0 con 0x7fb7381002c0 2023-11-16T16:48:50.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.685+0000 7fb72f7fe700 1 -- 172.21.15.44:0/1761268074 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb728003dd0 con 0x7fb7381002c0 2023-11-16T16:48:50.686 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2023-11-16T16:48:50.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.688+0000 7fb73c9ff700 1 -- 172.21.15.44:0/1761268074 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb7240a11e0 msgr2=0x7fb7240a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:50.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.688+0000 7fb73c9ff700 1 --2- 172.21.15.44:0/1761268074 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb7240a11e0 0x7fb7240a3690 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb72000b2e0 tx=0x7fb720005fb0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.688+0000 7fb73c9ff700 1 -- 172.21.15.44:0/1761268074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7381002c0 msgr2=0x7fb7380748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:50.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.688+0000 7fb73c9ff700 1 --2- 172.21.15.44:0/1761268074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7381002c0 0x7fb7380748a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb72800e970 tx=0x7fb72800ec80 comp rx=0 tx=0).stop 2023-11-16T16:48:50.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.688+0000 7fb73c9ff700 1 -- 172.21.15.44:0/1761268074 shutdown_connections 2023-11-16T16:48:50.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.688+0000 7fb73c9ff700 1 --2- 172.21.15.44:0/1761268074 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb7240a11e0 0x7fb7240a3690 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.688+0000 7fb73c9ff700 1 --2- 172.21.15.44:0/1761268074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb738101e40 0x7fb738073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.688+0000 7fb73c9ff700 1 --2- 172.21.15.44:0/1761268074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7381002c0 0x7fb7380748a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.688+0000 7fb73c9ff700 1 --2- 172.21.15.44:0/1761268074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7381014b0 0x7fb738072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.688+0000 7fb73c9ff700 1 -- 172.21.15.44:0/1761268074 >> 172.21.15.44:0/1761268074 conn(0x7fb7380fb8a0 msgr2=0x7fb7381049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:50.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.688+0000 7fb73c9ff700 1 -- 172.21.15.44:0/1761268074 shutdown_connections 2023-11-16T16:48:50.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.688+0000 7fb73c9ff700 1 -- 172.21.15.44:0/1761268074 wait complete. 2023-11-16T16:48:50.708 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T16:48:50.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:50 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3491733398' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:48:50.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:50 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2748792310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:48:50.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.914+0000 7f2ef5dff700 1 -- 172.21.15.44:0/1375997065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef0074b30 msgr2=0x7f2ef0072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:50.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.914+0000 7f2ef5dff700 1 --2- 172.21.15.44:0/1375997065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef0074b30 0x7f2ef0072f80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0009990 tx=0x7f2ee0009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.914+0000 7f2ef5dff700 1 -- 172.21.15.44:0/1375997065 shutdown_connections 2023-11-16T16:48:50.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.914+0000 7f2ef5dff700 1 --2- 172.21.15.44:0/1375997065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ef0108180 0x7f2ef010a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.914+0000 7f2ef5dff700 1 --2- 172.21.15.44:0/1375997065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef0074b30 0x7f2ef0072f80 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.914+0000 7f2ef5dff700 1 --2- 172.21.15.44:0/1375997065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ef0073550 0x7f2ef00739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.914+0000 7f2ef5dff700 1 -- 172.21.15.44:0/1375997065 >> 172.21.15.44:0/1375997065 conn(0x7f2ef00fb8a0 msgr2=0x7f2ef00fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:50.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.914+0000 7f2ef5dff700 1 -- 172.21.15.44:0/1375997065 shutdown_connections 2023-11-16T16:48:50.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.914+0000 7f2ef5dff700 1 -- 172.21.15.44:0/1375997065 wait complete. 2023-11-16T16:48:50.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.915+0000 7f2ef5dff700 1 Processor -- start 2023-11-16T16:48:50.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.915+0000 7f2ef5dff700 1 -- start start 2023-11-16T16:48:50.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.916+0000 7f2ef5dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ef0073550 0x7f2ef0198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:50.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.916+0000 7f2ef5dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ef0074b30 0x7f2ef01987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:50.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.916+0000 7f2ef5dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef0108180 0x7f2ef019f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:50.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.916+0000 7f2ef5dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef019fe10 con 0x7f2ef0108180 2023-11-16T16:48:50.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.916+0000 7f2ef5dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef019ff50 con 0x7f2ef0073550 2023-11-16T16:48:50.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.916+0000 7f2ef5dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef01a00e0 con 0x7f2ef0074b30 2023-11-16T16:48:50.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.916+0000 7f2eeffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef0108180 0x7f2ef019f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:50.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.916+0000 7f2eeeffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ef0074b30 0x7f2ef01987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:50.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.916+0000 7f2eeffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef0108180 0x7f2ef019f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39286/0 (socket says 172.21.15.44:39286) 2023-11-16T16:48:50.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.916+0000 7f2eeffff700 1 -- 172.21.15.44:0/1288406963 learned_addr learned my addr 172.21.15.44:0/1288406963 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:50.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.916+0000 7f2eeeffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ef0074b30 0x7f2ef01987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:54686/0 (socket says 172.21.15.44:54686) 2023-11-16T16:48:50.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.916+0000 7f2eef7fe700 1 --2- 172.21.15.44:0/1288406963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ef0073550 0x7f2ef0198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:50.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.917+0000 7f2eeffff700 1 -- 172.21.15.44:0/1288406963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ef0074b30 msgr2=0x7f2ef01987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:50.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.917+0000 7f2eeffff700 1 --2- 172.21.15.44:0/1288406963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ef0074b30 0x7f2ef01987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.917+0000 7f2eeffff700 1 -- 172.21.15.44:0/1288406963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ef0073550 msgr2=0x7f2ef0198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:50.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.917+0000 7f2eeffff700 1 --2- 172.21.15.44:0/1288406963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ef0073550 0x7f2ef0198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:50.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.917+0000 7f2eeffff700 1 -- 172.21.15.44:0/1288406963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee00095f0 con 0x7f2ef0108180 2023-11-16T16:48:50.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.917+0000 7f2eeeffd700 1 --2- 172.21.15.44:0/1288406963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ef0074b30 0x7f2ef01987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:50.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.917+0000 7f2eeffff700 1 --2- 172.21.15.44:0/1288406963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef0108180 0x7f2ef019f840 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8007ee0 tx=0x7f2ee800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:50.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.917+0000 7f2eecff9700 1 -- 172.21.15.44:0/1288406963 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8017070 con 0x7f2ef0108180 2023-11-16T16:48:50.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.917+0000 7f2ef5dff700 1 -- 172.21.15.44:0/1288406963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef010de80 con 0x7f2ef0108180 2023-11-16T16:48:50.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.917+0000 7f2eecff9700 1 -- 172.21.15.44:0/1288406963 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ee800f040 con 0x7f2ef0108180 2023-11-16T16:48:50.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.917+0000 7f2eecff9700 1 -- 172.21.15.44:0/1288406963 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8005550 con 0x7f2ef0108180 2023-11-16T16:48:50.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.917+0000 7f2ef5dff700 1 -- 172.21.15.44:0/1288406963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef010dcf0 con 0x7f2ef0108180 2023-11-16T16:48:50.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.918+0000 7f2ef5dff700 1 -- 172.21.15.44:0/1288406963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef0106830 con 0x7f2ef0108180 2023-11-16T16:48:50.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.920+0000 7f2eecff9700 1 -- 172.21.15.44:0/1288406963 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2ee800db20 con 0x7f2ef0108180 2023-11-16T16:48:50.923 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.923+0000 7f2eecff9700 1 --2- 172.21.15.44:0/1288406963 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ed80a1260 0x7f2ed80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:50.923 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.923+0000 7f2eecff9700 1 -- 172.21.15.44:0/1288406963 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f2ee80cc530 con 0x7f2ef0108180 2023-11-16T16:48:50.923 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.923+0000 7f2eef7fe700 1 --2- 172.21.15.44:0/1288406963 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ed80a1260 0x7f2ed80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:50.924 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.924+0000 7f2eef7fe700 1 --2- 172.21.15.44:0/1288406963 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ed80a1260 0x7f2ed80a3710 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0000c00 tx=0x7f2ee000b260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:50.927 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:50.927+0000 7f2eecff9700 1 -- 172.21.15.44:0/1288406963 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2ee8095720 con 0x7f2ef0108180 2023-11-16T16:48:50.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:50 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3491733398' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:48:50.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:50 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2748792310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:48:51.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:50 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3491733398' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:48:51.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:50 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2748792310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:48:51.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.137+0000 7f2ef5dff700 1 -- 172.21.15.44:0/1288406963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2ef0061700 con 0x7f2ef0108180 2023-11-16T16:48:51.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.137+0000 7f2eecff9700 1 -- 172.21.15.44:0/1288406963 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2ee8098d40 con 0x7f2ef0108180 2023-11-16T16:48:51.138 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:48:51.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.139+0000 7f2ef5dff700 1 -- 172.21.15.44:0/1288406963 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ed80a1260 msgr2=0x7f2ed80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:51.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.139+0000 7f2ef5dff700 1 --2- 172.21.15.44:0/1288406963 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ed80a1260 0x7f2ed80a3710 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0000c00 tx=0x7f2ee000b260 comp rx=0 tx=0).stop 2023-11-16T16:48:51.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.139+0000 7f2ef5dff700 1 -- 172.21.15.44:0/1288406963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef0108180 msgr2=0x7f2ef019f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:51.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.139+0000 7f2ef5dff700 1 --2- 172.21.15.44:0/1288406963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef0108180 0x7f2ef019f840 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8007ee0 tx=0x7f2ee800d3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.140+0000 7f2ef5dff700 1 -- 172.21.15.44:0/1288406963 shutdown_connections 2023-11-16T16:48:51.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.140+0000 7f2ef5dff700 1 --2- 172.21.15.44:0/1288406963 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ed80a1260 0x7f2ed80a3710 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.140+0000 7f2ef5dff700 1 --2- 172.21.15.44:0/1288406963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef0108180 0x7f2ef019f840 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.140+0000 7f2ef5dff700 1 --2- 172.21.15.44:0/1288406963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ef0073550 0x7f2ef0198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.140+0000 7f2ef5dff700 1 --2- 172.21.15.44:0/1288406963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ef0074b30 0x7f2ef01987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.140+0000 7f2ef5dff700 1 -- 172.21.15.44:0/1288406963 >> 172.21.15.44:0/1288406963 conn(0x7f2ef00fb8a0 msgr2=0x7f2ef00fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:51.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.140+0000 7f2ef5dff700 1 -- 172.21.15.44:0/1288406963 shutdown_connections 2023-11-16T16:48:51.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.140+0000 7f2ef5dff700 1 -- 172.21.15.44:0/1288406963 wait complete. 2023-11-16T16:48:51.160 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:48:51.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.362+0000 7efea03ff700 1 -- 172.21.15.44:0/566671655 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe981002d0 msgr2=0x7efe981006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:51.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.362+0000 7efea03ff700 1 --2- 172.21.15.44:0/566671655 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe981002d0 0x7efe981006e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efe94009990 tx=0x7efe94009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.362+0000 7efea03ff700 1 -- 172.21.15.44:0/566671655 shutdown_connections 2023-11-16T16:48:51.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.362+0000 7efea03ff700 1 --2- 172.21.15.44:0/566671655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe98101f10 0x7efe981087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.362+0000 7efea03ff700 1 --2- 172.21.15.44:0/566671655 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe981002d0 0x7efe981006e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.362+0000 7efea03ff700 1 --2- 172.21.15.44:0/566671655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe98101560 0x7efe981019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.362+0000 7efea03ff700 1 -- 172.21.15.44:0/566671655 >> 172.21.15.44:0/566671655 conn(0x7efe980fb930 msgr2=0x7efe980fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:51.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.362+0000 7efea03ff700 1 -- 172.21.15.44:0/566671655 shutdown_connections 2023-11-16T16:48:51.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.362+0000 7efea03ff700 1 -- 172.21.15.44:0/566671655 wait complete. 2023-11-16T16:48:51.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.363+0000 7efea03ff700 1 Processor -- start 2023-11-16T16:48:51.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.363+0000 7efea03ff700 1 -- start start 2023-11-16T16:48:51.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.364+0000 7efea03ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe981002d0 0x7efe98195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:51.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.364+0000 7efea03ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe98101560 0x7efe981964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:51.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.364+0000 7efea03ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe98101f10 0x7efe9819d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:51.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.364+0000 7efea03ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9819db10 con 0x7efe98101560 2023-11-16T16:48:51.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.364+0000 7efea03ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9819dc50 con 0x7efe98101f10 2023-11-16T16:48:51.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.364+0000 7efea03ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9819dde0 con 0x7efe981002d0 2023-11-16T16:48:51.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.364+0000 7efe9e19b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe981002d0 0x7efe98195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:51.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.364+0000 7efe9d99a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe98101560 0x7efe981964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:51.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.364+0000 7efe9e19b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe981002d0 0x7efe98195f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:54702/0 (socket says 172.21.15.44:54702) 2023-11-16T16:48:51.366 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.364+0000 7efe9e19b700 1 -- 172.21.15.44:0/3649567705 learned_addr learned my addr 172.21.15.44:0/3649567705 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:51.366 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.364+0000 7efe9e99c700 1 --2- 172.21.15.44:0/3649567705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe98101f10 0x7efe9819d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:51.367 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.364+0000 7efe9e19b700 1 -- 172.21.15.44:0/3649567705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe98101f10 msgr2=0x7efe9819d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:51.367 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.364+0000 7efe9e19b700 1 --2- 172.21.15.44:0/3649567705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe98101f10 0x7efe9819d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.367 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.364+0000 7efe9e19b700 1 -- 172.21.15.44:0/3649567705 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe98101560 msgr2=0x7efe981964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:51.367 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.365+0000 7efe9e19b700 1 --2- 172.21.15.44:0/3649567705 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe98101560 0x7efe981964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.367 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.365+0000 7efe9e19b700 1 -- 172.21.15.44:0/3649567705 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe940095f0 con 0x7efe981002d0 2023-11-16T16:48:51.367 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.365+0000 7efe9d99a700 1 --2- 172.21.15.44:0/3649567705 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe98101560 0x7efe981964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:51.367 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.365+0000 7efe9e19b700 1 --2- 172.21.15.44:0/3649567705 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe981002d0 0x7efe98195f90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efe9400b680 tx=0x7efe94005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:51.367 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.365+0000 7efe8f7fe700 1 -- 172.21.15.44:0/3649567705 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9401d070 con 0x7efe981002d0 2023-11-16T16:48:51.367 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.365+0000 7efe8f7fe700 1 -- 172.21.15.44:0/3649567705 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe94022470 con 0x7efe981002d0 2023-11-16T16:48:51.367 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.365+0000 7efea03ff700 1 -- 172.21.15.44:0/3649567705 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9810bcc0 con 0x7efe981002d0 2023-11-16T16:48:51.368 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.365+0000 7efe8f7fe700 1 -- 172.21.15.44:0/3649567705 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe94017650 con 0x7efe981002d0 2023-11-16T16:48:51.368 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.365+0000 7efea03ff700 1 -- 172.21.15.44:0/3649567705 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9810bb30 con 0x7efe981002d0 2023-11-16T16:48:51.368 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.366+0000 7efea03ff700 1 -- 172.21.15.44:0/3649567705 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe98107910 con 0x7efe981002d0 2023-11-16T16:48:51.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.368+0000 7efe8f7fe700 1 -- 172.21.15.44:0/3649567705 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7efe940225e0 con 0x7efe981002d0 2023-11-16T16:48:51.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.370+0000 7efe8f7fe700 1 --2- 172.21.15.44:0/3649567705 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efe840a11e0 0x7efe840a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:51.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.370+0000 7efe8f7fe700 1 -- 172.21.15.44:0/3649567705 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7efe940cea10 con 0x7efe981002d0 2023-11-16T16:48:51.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.370+0000 7efe9d99a700 1 --2- 172.21.15.44:0/3649567705 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efe840a11e0 0x7efe840a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:51.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.371+0000 7efe9d99a700 1 --2- 172.21.15.44:0/3649567705 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efe840a11e0 0x7efe840a3690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efe88011490 tx=0x7efe88009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:51.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.372+0000 7efe8f7fe700 1 -- 172.21.15.44:0/3649567705 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efe94097db0 con 0x7efe981002d0 2023-11-16T16:48:51.629 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.628+0000 7efea03ff700 1 -- 172.21.15.44:0/3649567705 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe9802cfe0 con 0x7efe981002d0 2023-11-16T16:48:51.630 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.629+0000 7efe8f7fe700 1 -- 172.21.15.44:0/3649567705 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7efe9409b3d0 con 0x7efe981002d0 2023-11-16T16:48:51.630 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:48:51.630 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:48:51.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.631+0000 7efea03ff700 1 -- 172.21.15.44:0/3649567705 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efe840a11e0 msgr2=0x7efe840a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:51.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.631+0000 7efea03ff700 1 --2- 172.21.15.44:0/3649567705 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efe840a11e0 0x7efe840a3690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efe88011490 tx=0x7efe88009260 comp rx=0 tx=0).stop 2023-11-16T16:48:51.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.631+0000 7efea03ff700 1 -- 172.21.15.44:0/3649567705 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe981002d0 msgr2=0x7efe98195f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:51.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.631+0000 7efea03ff700 1 --2- 172.21.15.44:0/3649567705 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe981002d0 0x7efe98195f90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efe9400b680 tx=0x7efe94005740 comp rx=0 tx=0).stop 2023-11-16T16:48:51.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.631+0000 7efea03ff700 1 -- 172.21.15.44:0/3649567705 shutdown_connections 2023-11-16T16:48:51.633 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.631+0000 7efea03ff700 1 --2- 172.21.15.44:0/3649567705 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7efe840a11e0 0x7efe840a3690 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.633 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.631+0000 7efea03ff700 1 --2- 172.21.15.44:0/3649567705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe98101f10 0x7efe9819d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.633 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.631+0000 7efea03ff700 1 --2- 172.21.15.44:0/3649567705 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe981002d0 0x7efe98195f90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.633 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.631+0000 7efea03ff700 1 --2- 172.21.15.44:0/3649567705 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe98101560 0x7efe981964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.633 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.631+0000 7efea03ff700 1 -- 172.21.15.44:0/3649567705 >> 172.21.15.44:0/3649567705 conn(0x7efe980fb930 msgr2=0x7efe98104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:51.633 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.632+0000 7efea03ff700 1 -- 172.21.15.44:0/3649567705 shutdown_connections 2023-11-16T16:48:51.633 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.632+0000 7efea03ff700 1 -- 172.21.15.44:0/3649567705 wait complete. 2023-11-16T16:48:51.633 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:48:51.652 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:48:51.856 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.855+0000 7fe5d2dff700 1 -- 172.21.15.44:0/3592061519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5cc101e30 msgr2=0x7fe5cc1086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:51.856 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.855+0000 7fe5d2dff700 1 --2- 172.21.15.44:0/3592061519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5cc101e30 0x7fe5cc1086f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400b0a0 tx=0x7fe5c400b3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.856 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.855+0000 7fe5d2dff700 1 -- 172.21.15.44:0/3592061519 shutdown_connections 2023-11-16T16:48:51.856 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.855+0000 7fe5d2dff700 1 --2- 172.21.15.44:0/3592061519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5cc101e30 0x7fe5cc1086f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.856 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.855+0000 7fe5d2dff700 1 --2- 172.21.15.44:0/3592061519 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5cc1002b0 0x7fe5cc1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.856 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.855+0000 7fe5d2dff700 1 --2- 172.21.15.44:0/3592061519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5cc1014a0 0x7fe5cc1018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.856 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.855+0000 7fe5d2dff700 1 -- 172.21.15.44:0/3592061519 >> 172.21.15.44:0/3592061519 conn(0x7fe5cc0fb890 msgr2=0x7fe5cc0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:51.856 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.855+0000 7fe5d2dff700 1 -- 172.21.15.44:0/3592061519 shutdown_connections 2023-11-16T16:48:51.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.855+0000 7fe5d2dff700 1 -- 172.21.15.44:0/3592061519 wait complete. 2023-11-16T16:48:51.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.856+0000 7fe5d2dff700 1 Processor -- start 2023-11-16T16:48:51.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.856+0000 7fe5d2dff700 1 -- start start 2023-11-16T16:48:51.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.857+0000 7fe5d2dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5cc1002b0 0x7fe5cc198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:51.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.857+0000 7fe5d2dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5cc1014a0 0x7fe5cc198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:51.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.857+0000 7fe5d2dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5cc101e30 0x7fe5cc19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:51.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.857+0000 7fe5d2dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc19fdd0 con 0x7fe5cc101e30 2023-11-16T16:48:51.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.857+0000 7fe5d2dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc19ff10 con 0x7fe5cc1014a0 2023-11-16T16:48:51.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.857+0000 7fe5d2dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc1a00a0 con 0x7fe5cc1002b0 2023-11-16T16:48:51.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.857+0000 7fe5cbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5cc1014a0 0x7fe5cc198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:51.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.857+0000 7fe5d139c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5cc101e30 0x7fe5cc19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:51.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.857+0000 7fe5cbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5cc1014a0 0x7fe5cc198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:40364/0 (socket says 172.21.15.44:40364) 2023-11-16T16:48:51.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.857+0000 7fe5cbfff700 1 -- 172.21.15.44:0/2355181663 learned_addr learned my addr 172.21.15.44:0/2355181663 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:51.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.857+0000 7fe5d0b9b700 1 --2- 172.21.15.44:0/2355181663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5cc1002b0 0x7fe5cc198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:51.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.857+0000 7fe5d139c700 1 -- 172.21.15.44:0/2355181663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5cc1002b0 msgr2=0x7fe5cc198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:51.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.857+0000 7fe5d139c700 1 --2- 172.21.15.44:0/2355181663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5cc1002b0 0x7fe5cc198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.857+0000 7fe5d139c700 1 -- 172.21.15.44:0/2355181663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5cc1014a0 msgr2=0x7fe5cc198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:51.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.857+0000 7fe5d139c700 1 --2- 172.21.15.44:0/2355181663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5cc1014a0 0x7fe5cc198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.857+0000 7fe5d139c700 1 -- 172.21.15.44:0/2355181663 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c0009520 con 0x7fe5cc101e30 2023-11-16T16:48:51.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.858+0000 7fe5d139c700 1 --2- 172.21.15.44:0/2355181663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5cc101e30 0x7fe5cc19f800 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400b0a0 tx=0x7fe5c4007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:51.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.858+0000 7fe5c9ffb700 1 -- 172.21.15.44:0/2355181663 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c400e040 con 0x7fe5cc101e30 2023-11-16T16:48:51.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.858+0000 7fe5c9ffb700 1 -- 172.21.15.44:0/2355181663 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5c400bde0 con 0x7fe5cc101e30 2023-11-16T16:48:51.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.858+0000 7fe5d2dff700 1 -- 172.21.15.44:0/2355181663 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c4009d00 con 0x7fe5cc101e30 2023-11-16T16:48:51.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.858+0000 7fe5c9ffb700 1 -- 172.21.15.44:0/2355181663 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c401b6b0 con 0x7fe5cc101e30 2023-11-16T16:48:51.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.858+0000 7fe5d2dff700 1 -- 172.21.15.44:0/2355181663 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5cc10bc40 con 0x7fe5cc101e30 2023-11-16T16:48:51.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.859+0000 7fe5d2dff700 1 -- 172.21.15.44:0/2355181663 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5cc067a60 con 0x7fe5cc101e30 2023-11-16T16:48:51.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.861+0000 7fe5c9ffb700 1 -- 172.21.15.44:0/2355181663 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe5c401d480 con 0x7fe5cc101e30 2023-11-16T16:48:51.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.862+0000 7fe5c9ffb700 1 --2- 172.21.15.44:0/2355181663 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5bc0a1260 0x7fe5bc0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:51.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.863+0000 7fe5c9ffb700 1 -- 172.21.15.44:0/2355181663 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fe5c40ce3c0 con 0x7fe5cc101e30 2023-11-16T16:48:51.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.863+0000 7fe5d0b9b700 1 --2- 172.21.15.44:0/2355181663 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5bc0a1260 0x7fe5bc0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:51.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.863+0000 7fe5d0b9b700 1 --2- 172.21.15.44:0/2355181663 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5bc0a1260 0x7fe5bc0a3710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc101300 tx=0x7fe5c0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:51.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:51.866+0000 7fe5c9ffb700 1 -- 172.21.15.44:0/2355181663 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe5c4097640 con 0x7fe5cc101e30 2023-11-16T16:48:51.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:51 smithi176 ceph-mon[131630]: pgmap v325: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:51.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:51 smithi176 ceph-mon[131630]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2023-11-16T16:48:51.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:51 smithi176 ceph-mon[131630]: Cluster is now healthy 2023-11-16T16:48:51.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:51 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1761268074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:48:51.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:51 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1288406963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:48:51.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:51 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3649567705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:51.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:51 smithi077 ceph-mon[131204]: pgmap v325: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:51.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:51 smithi077 ceph-mon[131204]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2023-11-16T16:48:51.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:51 smithi077 ceph-mon[131204]: Cluster is now healthy 2023-11-16T16:48:51.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:51 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1761268074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:48:51.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:51 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1288406963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:48:51.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:51 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3649567705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:52.076 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.075+0000 7fe5d2dff700 1 -- 172.21.15.44:0/2355181663 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5cc1a0c00 con 0x7fe5cc101e30 2023-11-16T16:48:52.078 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.077+0000 7fe5c9ffb700 1 -- 172.21.15.44:0/2355181663 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24685 (secure 0 0 0) 0x7fe5c409ac60 con 0x7fe5cc101e30 2023-11-16T16:48:52.078 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:48:52.079 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:48:44.613706+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:48:52.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.080+0000 7fe5d2dff700 1 -- 172.21.15.44:0/2355181663 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5bc0a1260 msgr2=0x7fe5bc0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.080+0000 7fe5d2dff700 1 --2- 172.21.15.44:0/2355181663 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5bc0a1260 0x7fe5bc0a3710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc101300 tx=0x7fe5c0009260 comp rx=0 tx=0).stop 2023-11-16T16:48:52.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.080+0000 7fe5d2dff700 1 -- 172.21.15.44:0/2355181663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5cc101e30 msgr2=0x7fe5cc19f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.080+0000 7fe5d2dff700 1 --2- 172.21.15.44:0/2355181663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5cc101e30 0x7fe5cc19f800 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400b0a0 tx=0x7fe5c4007f30 comp rx=0 tx=0).stop 2023-11-16T16:48:52.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.080+0000 7fe5d2dff700 1 -- 172.21.15.44:0/2355181663 shutdown_connections 2023-11-16T16:48:52.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.080+0000 7fe5d2dff700 1 --2- 172.21.15.44:0/2355181663 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5bc0a1260 0x7fe5bc0a3710 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.080+0000 7fe5d2dff700 1 --2- 172.21.15.44:0/2355181663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5cc101e30 0x7fe5cc19f800 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.080+0000 7fe5d2dff700 1 --2- 172.21.15.44:0/2355181663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5cc1002b0 0x7fe5cc198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.080+0000 7fe5d2dff700 1 --2- 172.21.15.44:0/2355181663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5cc1014a0 0x7fe5cc198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.080+0000 7fe5d2dff700 1 -- 172.21.15.44:0/2355181663 >> 172.21.15.44:0/2355181663 conn(0x7fe5cc0fb890 msgr2=0x7fe5cc106d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:52.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.080+0000 7fe5d2dff700 1 -- 172.21.15.44:0/2355181663 shutdown_connections 2023-11-16T16:48:52.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.080+0000 7fe5d2dff700 1 -- 172.21.15.44:0/2355181663 wait complete. 2023-11-16T16:48:52.097 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:51 smithi044 ceph-mon[128750]: pgmap v325: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:48:52.097 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:51 smithi044 ceph-mon[128750]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2023-11-16T16:48:52.098 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:51 smithi044 ceph-mon[128750]: Cluster is now healthy 2023-11-16T16:48:52.098 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:51 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1761268074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:48:52.098 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:51 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1288406963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:48:52.098 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:51 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3649567705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:52.100 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:48:52.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.305+0000 7f88003ff700 1 -- 172.21.15.44:0/2898336944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87f8101e10 msgr2=0x7f87f81086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.305+0000 7f88003ff700 1 --2- 172.21.15.44:0/2898336944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87f8101e10 0x7f87f81086d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f87f000b0a0 tx=0x7f87f000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.305+0000 7f88003ff700 1 -- 172.21.15.44:0/2898336944 shutdown_connections 2023-11-16T16:48:52.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.305+0000 7f88003ff700 1 --2- 172.21.15.44:0/2898336944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87f8101e10 0x7f87f81086d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.305+0000 7f88003ff700 1 --2- 172.21.15.44:0/2898336944 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87f8100290 0x7f87f81006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.305+0000 7f88003ff700 1 --2- 172.21.15.44:0/2898336944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87f8101480 0x7f87f81018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.305+0000 7f88003ff700 1 -- 172.21.15.44:0/2898336944 >> 172.21.15.44:0/2898336944 conn(0x7f87f80fb870 msgr2=0x7f87f80fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:52.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.305+0000 7f88003ff700 1 -- 172.21.15.44:0/2898336944 shutdown_connections 2023-11-16T16:48:52.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.305+0000 7f88003ff700 1 -- 172.21.15.44:0/2898336944 wait complete. 2023-11-16T16:48:52.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.306+0000 7f88003ff700 1 Processor -- start 2023-11-16T16:48:52.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.306+0000 7f88003ff700 1 -- start start 2023-11-16T16:48:52.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.306+0000 7f88003ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87f8100290 0x7f87f80748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.306+0000 7f88003ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87f8101480 0x7f87f8072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.306+0000 7f88003ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87f8101e10 0x7f87f8073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.307+0000 7f88003ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f8073bc0 con 0x7f87f8100290 2023-11-16T16:48:52.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.307+0000 7f88003ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f8073d30 con 0x7f87f8101480 2023-11-16T16:48:52.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.307+0000 7f88003ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f81a00f0 con 0x7f87f8101e10 2023-11-16T16:48:52.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.307+0000 7f87fe19b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87f8100290 0x7f87f80748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.307+0000 7f87fd99a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87f8101480 0x7f87f8072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.307+0000 7f87fe19b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87f8100290 0x7f87f80748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39344/0 (socket says 172.21.15.44:39344) 2023-11-16T16:48:52.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.307+0000 7f87fd99a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87f8101480 0x7f87f8072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:40372/0 (socket says 172.21.15.44:40372) 2023-11-16T16:48:52.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.307+0000 7f87fe19b700 1 -- 172.21.15.44:0/1766119481 learned_addr learned my addr 172.21.15.44:0/1766119481 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:52.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.307+0000 7f87fe99c700 1 --2- 172.21.15.44:0/1766119481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87f8101e10 0x7f87f8073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.307+0000 7f87fe19b700 1 -- 172.21.15.44:0/1766119481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87f8101e10 msgr2=0x7f87f8073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.307+0000 7f87fe19b700 1 --2- 172.21.15.44:0/1766119481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87f8101e10 0x7f87f8073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.307+0000 7f87fe19b700 1 -- 172.21.15.44:0/1766119481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87f8101480 msgr2=0x7f87f8072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.307+0000 7f87fe19b700 1 --2- 172.21.15.44:0/1766119481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87f8101480 0x7f87f8072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.307+0000 7f87fe19b700 1 -- 172.21.15.44:0/1766119481 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f0009d00 con 0x7f87f8100290 2023-11-16T16:48:52.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.307+0000 7f87fd99a700 1 --2- 172.21.15.44:0/1766119481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87f8101480 0x7f87f8072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:52.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.307+0000 7f87fe19b700 1 --2- 172.21.15.44:0/1766119481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87f8100290 0x7f87f80748a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f87f400e970 tx=0x7f87f400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:52.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.308+0000 7f87ef7fe700 1 -- 172.21.15.44:0/1766119481 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f4009790 con 0x7f87f8100290 2023-11-16T16:48:52.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.308+0000 7f87ef7fe700 1 -- 172.21.15.44:0/1766119481 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87f400c7e0 con 0x7f87f8100290 2023-11-16T16:48:52.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.308+0000 7f88003ff700 1 -- 172.21.15.44:0/1766119481 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f810bc70 con 0x7f87f8100290 2023-11-16T16:48:52.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.308+0000 7f87ef7fe700 1 -- 172.21.15.44:0/1766119481 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f4003c70 con 0x7f87f8100290 2023-11-16T16:48:52.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.308+0000 7f88003ff700 1 -- 172.21.15.44:0/1766119481 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f810bae0 con 0x7f87f8100290 2023-11-16T16:48:52.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.309+0000 7f88003ff700 1 -- 172.21.15.44:0/1766119481 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f8107770 con 0x7f87f8100290 2023-11-16T16:48:52.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.311+0000 7f87ef7fe700 1 -- 172.21.15.44:0/1766119481 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f87f400cc50 con 0x7f87f8100290 2023-11-16T16:48:52.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.312+0000 7f87ef7fe700 1 --2- 172.21.15.44:0/1766119481 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f87e40a11e0 0x7f87e40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.312+0000 7f87ef7fe700 1 -- 172.21.15.44:0/1766119481 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f87f4014070 con 0x7f87f8100290 2023-11-16T16:48:52.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.312+0000 7f87fd99a700 1 --2- 172.21.15.44:0/1766119481 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f87e40a11e0 0x7f87e40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.313+0000 7f87fd99a700 1 --2- 172.21.15.44:0/1766119481 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f87e40a11e0 0x7f87e40a3690 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f87e8011490 tx=0x7f87e8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:52.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.316+0000 7f87ef7fe700 1 -- 172.21.15.44:0/1766119481 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f87f4096240 con 0x7f87f8100290 2023-11-16T16:48:52.571 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.570+0000 7f88003ff700 1 -- 172.21.15.44:0/1766119481 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87f802cfe0 con 0x7f87f8100290 2023-11-16T16:48:52.571 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.571+0000 7f87ef7fe700 1 -- 172.21.15.44:0/1766119481 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f87f4003dd0 con 0x7f87f8100290 2023-11-16T16:48:52.572 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:48:52.572 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:48:52.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.574+0000 7f88003ff700 1 -- 172.21.15.44:0/1766119481 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f87e40a11e0 msgr2=0x7f87e40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.574+0000 7f88003ff700 1 --2- 172.21.15.44:0/1766119481 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f87e40a11e0 0x7f87e40a3690 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f87e8011490 tx=0x7f87e8009260 comp rx=0 tx=0).stop 2023-11-16T16:48:52.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.574+0000 7f88003ff700 1 -- 172.21.15.44:0/1766119481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87f8100290 msgr2=0x7f87f80748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.574+0000 7f88003ff700 1 --2- 172.21.15.44:0/1766119481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87f8100290 0x7f87f80748a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f87f400e970 tx=0x7f87f400ec80 comp rx=0 tx=0).stop 2023-11-16T16:48:52.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.574+0000 7f88003ff700 1 -- 172.21.15.44:0/1766119481 shutdown_connections 2023-11-16T16:48:52.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.574+0000 7f88003ff700 1 --2- 172.21.15.44:0/1766119481 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f87e40a11e0 0x7f87e40a3690 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.574+0000 7f88003ff700 1 --2- 172.21.15.44:0/1766119481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87f8101e10 0x7f87f8073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.574+0000 7f88003ff700 1 --2- 172.21.15.44:0/1766119481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87f8100290 0x7f87f80748a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.574+0000 7f88003ff700 1 --2- 172.21.15.44:0/1766119481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87f8101480 0x7f87f8072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.574+0000 7f88003ff700 1 -- 172.21.15.44:0/1766119481 >> 172.21.15.44:0/1766119481 conn(0x7f87f80fb870 msgr2=0x7f87f81048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:52.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.574+0000 7f88003ff700 1 -- 172.21.15.44:0/1766119481 shutdown_connections 2023-11-16T16:48:52.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.574+0000 7f88003ff700 1 -- 172.21.15.44:0/1766119481 wait complete. 2023-11-16T16:48:52.576 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:48:52.594 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-11-16T16:47:18.150861+0000', 'modified': '2023-11-16T16:47:50.656307+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24551}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24538': {'gid': 24538, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.176:6839/2828920421', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6838', 'nonce': 2828920421}, {'type': 'v1', 'addr': '172.21.15.176:6839', 'nonce': 2828920421}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24551': {'gid': 24551, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6839/1948116745', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6838', 'nonce': 1948116745}, {'type': 'v1', 'addr': '172.21.15.77:6839', 'nonce': 1948116745}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-16T16:48:52.594 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-16T16:48:52.594 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-16T16:48:52.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.801+0000 7f03059ff700 1 -- 172.21.15.44:0/409743560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0300101420 msgr2=0x7f0300101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.801+0000 7f03059ff700 1 --2- 172.21.15.44:0/409743560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0300101420 0x7f0300101870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f02e80098f0 tx=0x7f02e8009c00 comp rx=0 tx=0).stop 2023-11-16T16:48:52.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.801+0000 7f03059ff700 1 -- 172.21.15.44:0/409743560 shutdown_connections 2023-11-16T16:48:52.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.801+0000 7f03059ff700 1 --2- 172.21.15.44:0/409743560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0300101db0 0x7f0300108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.801+0000 7f03059ff700 1 --2- 172.21.15.44:0/409743560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0300100230 0x7f0300100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.801+0000 7f03059ff700 1 --2- 172.21.15.44:0/409743560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0300101420 0x7f0300101870 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.801+0000 7f03059ff700 1 -- 172.21.15.44:0/409743560 >> 172.21.15.44:0/409743560 conn(0x7f03000fb810 msgr2=0x7f03000fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:52.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.801+0000 7f03059ff700 1 -- 172.21.15.44:0/409743560 shutdown_connections 2023-11-16T16:48:52.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.801+0000 7f03059ff700 1 -- 172.21.15.44:0/409743560 wait complete. 2023-11-16T16:48:52.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.802+0000 7f03059ff700 1 Processor -- start 2023-11-16T16:48:52.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.802+0000 7f03059ff700 1 -- start start 2023-11-16T16:48:52.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.802+0000 7f03059ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0300100230 0x7f0300195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.802+0000 7f03059ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0300101420 0x7f03001964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.802+0000 7f03059ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0300101db0 0x7f030019d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.802+0000 7f03059ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030019db30 con 0x7f0300101db0 2023-11-16T16:48:52.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.802+0000 7f03059ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030019dc70 con 0x7f0300100230 2023-11-16T16:48:52.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.802+0000 7f03059ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030019de00 con 0x7f0300101420 2023-11-16T16:48:52.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.803+0000 7f02ff7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0300101db0 0x7f030019d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.803+0000 7f02feffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0300100230 0x7f0300195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.803+0000 7f02ff7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0300101db0 0x7f030019d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39376/0 (socket says 172.21.15.44:39376) 2023-11-16T16:48:52.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.803+0000 7f02feffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0300100230 0x7f0300195fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:40378/0 (socket says 172.21.15.44:40378) 2023-11-16T16:48:52.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.803+0000 7f02ff7fe700 1 -- 172.21.15.44:0/1966095607 learned_addr learned my addr 172.21.15.44:0/1966095607 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:52.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.803+0000 7f02f7fff700 1 --2- 172.21.15.44:0/1966095607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0300101420 0x7f03001964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.803+0000 7f02feffd700 1 -- 172.21.15.44:0/1966095607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0300101420 msgr2=0x7f03001964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.803+0000 7f02feffd700 1 --2- 172.21.15.44:0/1966095607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0300101420 0x7f03001964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.803+0000 7f02feffd700 1 -- 172.21.15.44:0/1966095607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0300101db0 msgr2=0x7f030019d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.803+0000 7f02feffd700 1 --2- 172.21.15.44:0/1966095607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0300101db0 0x7f030019d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.803+0000 7f02feffd700 1 -- 172.21.15.44:0/1966095607 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e8009520 con 0x7f0300100230 2023-11-16T16:48:52.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.803+0000 7f02ff7fe700 1 --2- 172.21.15.44:0/1966095607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0300101db0 0x7f030019d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:52.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.803+0000 7f02feffd700 1 --2- 172.21.15.44:0/1966095607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0300100230 0x7f0300195fb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009c20 tx=0x7f02f0009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:52.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.804+0000 7f03049fd700 1 -- 172.21.15.44:0/1966095607 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f0015070 con 0x7f0300100230 2023-11-16T16:48:52.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.804+0000 7f03049fd700 1 -- 172.21.15.44:0/1966095607 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02f0004020 con 0x7f0300100230 2023-11-16T16:48:52.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.804+0000 7f03059ff700 1 -- 172.21.15.44:0/1966095607 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f030010bb80 con 0x7f0300100230 2023-11-16T16:48:52.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.804+0000 7f03049fd700 1 -- 172.21.15.44:0/1966095607 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f0003bc0 con 0x7f0300100230 2023-11-16T16:48:52.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.804+0000 7f03059ff700 1 -- 172.21.15.44:0/1966095607 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f030010b9f0 con 0x7f0300100230 2023-11-16T16:48:52.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.807+0000 7f03049fd700 1 -- 172.21.15.44:0/1966095607 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f02f0004190 con 0x7f0300100230 2023-11-16T16:48:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.808+0000 7f0319258b80 1 Processor -- start 2023-11-16T16:48:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.809+0000 7f0319258b80 1 -- start start 2023-11-16T16:48:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.809+0000 7f0319258b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d0bb0 0x5583694d0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.809+0000 7f0319258b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 0x5583694d1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.809+0000 7f0319258b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d20c0 0x5583694d2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.809+0000 7f0319258b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583693d6010 con 0x5583694d0bb0 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.809+0000 7f0319258b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x558369428850 con 0x5583694d20c0 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.809+0000 7f0319258b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x558369421ce0 con 0x5583694d1500 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.809+0000 7f02ff7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d0bb0 0x5583694d0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.810+0000 7f02feffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 0x5583694d1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.810+0000 7f03049fd700 1 --2- 172.21.15.44:0/1966095607 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f02ec0a11e0 0x7f02ec0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.810+0000 7f02ff7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d0bb0 0x5583694d0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39390/0 (socket says 172.21.15.44:39390) 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.810+0000 7f02ff7fe700 1 -- 172.21.15.44:0/2335812528 learned_addr learned my addr 172.21.15.44:0/2335812528 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.810+0000 7f02f7fff700 1 --2- 172.21.15.44:0/2335812528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d20c0 0x5583694d2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.810+0000 7f03049fd700 1 -- 172.21.15.44:0/1966095607 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f02f00d07c0 con 0x7f0300100230 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.810+0000 7f02ff7fe700 1 --2- 172.21.15.44:0/1966095607 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f02ec0a11e0 0x7f02ec0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.810+0000 7f02feffd700 1 -- 172.21.15.44:0/2335812528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d20c0 msgr2=0x5583694d2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.810+0000 7f02feffd700 1 --2- 172.21.15.44:0/2335812528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d20c0 0x5583694d2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.810+0000 7f02feffd700 1 -- 172.21.15.44:0/2335812528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d0bb0 msgr2=0x5583694d0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.810+0000 7f02feffd700 1 --2- 172.21.15.44:0/2335812528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d0bb0 0x5583694d0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.810+0000 7f02feffd700 1 -- 172.21.15.44:0/2335812528 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583692a35e0 con 0x5583694d1500 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.810+0000 7f02ff7fe700 1 --2- 172.21.15.44:0/2335812528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d0bb0 0x5583694d0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.810+0000 7f02feffd700 1 --2- 172.21.15.44:0/2335812528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 0x5583694d1ae0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f02f00cdd70 tx=0x7f02f009d570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=867a269f6114dc71 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.811+0000 7f02ff7fe700 1 --2- 172.21.15.44:0/1966095607 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f02ec0a11e0 0x7f02ec0a3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f02f8009610 tx=0x7f02f8004b40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.811+0000 7f02f5ffb700 1 -- 172.21.15.44:0/2335812528 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f00a16a0 con 0x5583694d1500 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.811+0000 7f02f5ffb700 1 -- 172.21.15.44:0/2335812528 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02f00a5ac0 con 0x5583694d1500 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.811+0000 7f02f5ffb700 1 -- 172.21.15.44:0/2335812528 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f00d0470 con 0x5583694d1500 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.811+0000 7f0319258b80 1 -- 172.21.15.44:0/2335812528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 msgr2=0x5583694d1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.811+0000 7f0319258b80 1 --2- 172.21.15.44:0/2335812528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 0x5583694d1ae0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f02f00cdd70 tx=0x7f02f009d570 comp rx=0 tx=0).stop 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.811+0000 7f0319258b80 1 -- 172.21.15.44:0/2335812528 shutdown_connections 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.811+0000 7f0319258b80 1 --2- 172.21.15.44:0/2335812528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d20c0 0x5583694d2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.811+0000 7f0319258b80 1 --2- 172.21.15.44:0/2335812528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d0bb0 0x5583694d0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.811+0000 7f0319258b80 1 --2- 172.21.15.44:0/2335812528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 0x5583694d1ae0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.811+0000 7f0319258b80 1 -- 172.21.15.44:0/2335812528 >> 172.21.15.44:0/2335812528 conn(0x5583694c6310 msgr2=0x5583694cb620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.811+0000 7f0319258b80 1 -- 172.21.15.44:0/2335812528 shutdown_connections 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.811+0000 7f0319258b80 1 -- 172.21.15.44:0/2335812528 wait complete. 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.812+0000 7f0319258b80 1 -- start start 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.812+0000 7f0319258b80 1 Processor -- start 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f0319258b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d0bb0 0x5583694da4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f0319258b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 0x5583694daa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f0319258b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d20c0 0x5583694e1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f0319258b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583693c3890 con 0x5583694d0bb0 2023-11-16T16:48:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f0319258b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583693c3530 con 0x5583694d20c0 2023-11-16T16:48:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f0319258b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583692a58d0 con 0x5583694d1500 2023-11-16T16:48:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f0319258b80 10 client.0 _ll_register_callbacks cb 0x5583693e1730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:48:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f02feffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d0bb0 0x5583694da4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f02feffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d0bb0 0x5583694da4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39396/0 (socket says 172.21.15.44:39396) 2023-11-16T16:48:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f02feffd700 1 -- 172.21.15.44:0/3162288298 learned_addr learned my addr 172.21.15.44:0/3162288298 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f02f7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 0x5583694daa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f02ff7fe700 1 --2- 172.21.15.44:0/3162288298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d20c0 0x5583694e1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f02f7fff700 1 -- 172.21.15.44:0/3162288298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d20c0 msgr2=0x5583694e1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f02f7fff700 1 --2- 172.21.15.44:0/3162288298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d20c0 0x5583694e1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f02f7fff700 1 -- 172.21.15.44:0/3162288298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d0bb0 msgr2=0x5583694da4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:52.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f02f7fff700 1 --2- 172.21.15.44:0/3162288298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d0bb0 0x5583694da4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f02f7fff700 1 -- 172.21.15.44:0/3162288298 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f030010b9f0 con 0x5583694d1500 2023-11-16T16:48:52.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.813+0000 7f02feffd700 1 --2- 172.21.15.44:0/3162288298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d0bb0 0x5583694da4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:52.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.814+0000 7f02f7fff700 1 --2- 172.21.15.44:0/3162288298 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 0x5583694daa10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009230 tx=0x7f02e800b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32594b983f7f9a87 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:52.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.814+0000 7f02deffd700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:48:52.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.814+0000 7f02deffd700 1 -- 172.21.15.44:0/3162288298 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e8027070 con 0x5583694d1500 2023-11-16T16:48:52.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.814+0000 7f0319258b80 1 -- 172.21.15.44:0/3162288298 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5583694c3ad0 con 0x5583694d1500 2023-11-16T16:48:52.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.814+0000 7f02deffd700 1 -- 172.21.15.44:0/3162288298 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f02e800bb10 con 0x5583694d1500 2023-11-16T16:48:52.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.814+0000 7f02deffd700 1 -- 172.21.15.44:0/3162288298 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02e8005040 con 0x5583694d1500 2023-11-16T16:48:52.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.814+0000 7f0319258b80 10 client.25117 fetch_fsmap learned FSMap version 21 2023-11-16T16:48:52.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.814+0000 7f0319258b80 1 -- 172.21.15.44:0/3162288298 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5583693e8140 con 0x5583694d1500 2023-11-16T16:48:52.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.814+0000 7f02deffd700 1 -- 172.21.15.44:0/3162288298 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5583694c3ad0 con 0x5583694d1500 2023-11-16T16:48:52.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.815+0000 7f02deffd700 1 -- 172.21.15.44:0/3162288298 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f02e802a020 con 0x5583694d1500 2023-11-16T16:48:52.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.815+0000 7f0319258b80 10 client.25117 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:48:52.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.815+0000 7f0319258b80 10 client.25117 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T16:48:52.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.815+0000 7f0319258b80 20 client.25117 populate_metadata read hostname 'smithi044' 2023-11-16T16:48:52.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.815+0000 7f0319258b80 1 --2- 172.21.15.44:0/3162288298 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5583694e5740 0x5583694e4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.815+0000 7f0319258b80 4 client.25117 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:48:52.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.815+0000 7f0319258b80 1 -- 172.21.15.44:0/3162288298 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5583693e4050 con 0x5583694e5740 2023-11-16T16:48:52.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.815+0000 7f02feffd700 1 --2- 172.21.15.44:0/3162288298 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5583694e5740 0x5583694e4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.816+0000 7f02feffd700 1 --2- 172.21.15.44:0/3162288298 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5583694e5740 0x5583694e4b50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df957bf49fa7a9a7 server_cookie=d570d0bf6d71c148 in_seq=0 out_seq=0 2023-11-16T16:48:52.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.816+0000 7f02deffd700 10 client.25117 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:48:52.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.816+0000 7f02deffd700 1 -- 172.21.15.44:0/3162288298 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5583693e4050 con 0x5583694e5740 2023-11-16T16:48:52.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.816+0000 7f02deffd700 10 client.25117 handle_command_reply: tid=0 2023-11-16T16:48:52.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.816+0000 7f0319258b80 1 client.25117 shutdown 2023-11-16T16:48:52.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.817+0000 7f0319258b80 1 -- 172.21.15.44:0/3162288298 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 msgr2=0x5583694daa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.817+0000 7f0319258b80 1 --2- 172.21.15.44:0/3162288298 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 0x5583694daa10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009230 tx=0x7f02e800b790 comp rx=0 tx=0).stop 2023-11-16T16:48:52.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.817+0000 7f0319258b80 1 -- 172.21.15.44:0/3162288298 shutdown_connections 2023-11-16T16:48:52.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.817+0000 7f0319258b80 1 --2- 172.21.15.44:0/3162288298 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5583694e5740 0x5583694e4b50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.817+0000 7f0319258b80 1 --2- 172.21.15.44:0/3162288298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d20c0 0x5583694e1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.817+0000 7f0319258b80 1 --2- 172.21.15.44:0/3162288298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d0bb0 0x5583694da4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.817+0000 7f02deffd700 0 client.25117 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:48:52.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.817+0000 7f0319258b80 1 --2- 172.21.15.44:0/3162288298 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 0x5583694daa10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.817+0000 7f0319258b80 1 -- 172.21.15.44:0/3162288298 >> 172.21.15.44:0/3162288298 conn(0x5583694c6310 msgr2=0x5583694cad90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:52.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.817+0000 7f0319258b80 1 -- 172.21.15.44:0/3162288298 shutdown_connections 2023-11-16T16:48:52.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.817+0000 7f0319258b80 1 -- 172.21.15.44:0/3162288298 wait complete. 2023-11-16T16:48:52.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.817+0000 7f0319258b80 20 client.25117 trim_cache size 0 max 16384 2023-11-16T16:48:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.835+0000 7f0319258b80 1 Processor -- start 2023-11-16T16:48:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.836+0000 7f0319258b80 1 -- start start 2023-11-16T16:48:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.836+0000 7f0319258b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d0bb0 0x5583694e4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.836+0000 7f0319258b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 0x5583694e1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.836+0000 7f0319258b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d20c0 0x5583694d4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.836+0000 7f0319258b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583694db6c0 con 0x5583694d20c0 2023-11-16T16:48:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.836+0000 7f0319258b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55836932ded0 con 0x5583694d0bb0 2023-11-16T16:48:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.836+0000 7f0319258b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583693dffa0 con 0x5583694d1500 2023-11-16T16:48:52.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.836+0000 7f02ff7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d20c0 0x5583694d4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.836+0000 7f02feffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d0bb0 0x5583694e4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.836+0000 7f02ff7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d20c0 0x5583694d4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39404/0 (socket says 172.21.15.44:39404) 2023-11-16T16:48:52.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.836+0000 7f02ff7fe700 1 -- 172.21.15.44:0/2708947460 learned_addr learned my addr 172.21.15.44:0/2708947460 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:52.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.836+0000 7f02feffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d0bb0 0x5583694e4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:40404/0 (socket says 172.21.15.44:40404) 2023-11-16T16:48:52.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.836+0000 7f02f7fff700 1 --2- 172.21.15.44:0/2708947460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 0x5583694e1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.837+0000 7f02feffd700 1 -- 172.21.15.44:0/2708947460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 msgr2=0x5583694e1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:52.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.837+0000 7f02feffd700 1 --2- 172.21.15.44:0/2708947460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 0x5583694e1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.837+0000 7f02feffd700 1 -- 172.21.15.44:0/2708947460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d20c0 msgr2=0x5583694d4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:52.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.837+0000 7f02feffd700 1 --2- 172.21.15.44:0/2708947460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d20c0 0x5583694d4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.837+0000 7f02feffd700 1 -- 172.21.15.44:0/2708947460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583693e7e90 con 0x5583694d0bb0 2023-11-16T16:48:52.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.837+0000 7f02feffd700 1 --2- 172.21.15.44:0/2708947460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d0bb0 0x5583694e4b50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009230 tx=0x7f02f0003680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92174e38a7c5e73a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:52.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.837+0000 7f02f67fc700 1 -- 172.21.15.44:0/2708947460 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f00a1720 con 0x5583694d0bb0 2023-11-16T16:48:52.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.837+0000 7f02f67fc700 1 -- 172.21.15.44:0/2708947460 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02f00b0150 con 0x5583694d0bb0 2023-11-16T16:48:52.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.837+0000 7f02f67fc700 1 -- 172.21.15.44:0/2708947460 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f009d8d0 con 0x5583694d0bb0 2023-11-16T16:48:52.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.838+0000 7f0319258b80 1 -- 172.21.15.44:0/2708947460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d0bb0 msgr2=0x5583694e4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.838+0000 7f0319258b80 1 --2- 172.21.15.44:0/2708947460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d0bb0 0x5583694e4b50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009230 tx=0x7f02f0003680 comp rx=0 tx=0).stop 2023-11-16T16:48:52.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.838+0000 7f0319258b80 1 -- 172.21.15.44:0/2708947460 shutdown_connections 2023-11-16T16:48:52.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.838+0000 7f0319258b80 1 --2- 172.21.15.44:0/2708947460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d20c0 0x5583694d4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.838+0000 7f0319258b80 1 --2- 172.21.15.44:0/2708947460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d0bb0 0x5583694e4b50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.838+0000 7f0319258b80 1 --2- 172.21.15.44:0/2708947460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d1500 0x5583694e1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.838+0000 7f0319258b80 1 -- 172.21.15.44:0/2708947460 >> 172.21.15.44:0/2708947460 conn(0x5583694c6310 msgr2=0x5583694d91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:52.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.838+0000 7f0319258b80 1 -- 172.21.15.44:0/2708947460 shutdown_connections 2023-11-16T16:48:52.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.838+0000 7f0319258b80 1 -- 172.21.15.44:0/2708947460 wait complete. 2023-11-16T16:48:52.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.838+0000 7f0319258b80 1 -- start start 2023-11-16T16:48:52.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.838+0000 7f0319258b80 1 Processor -- start 2023-11-16T16:48:52.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f0319258b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d0bb0 0x5583694dbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f0319258b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d1500 0x5583694dc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f0319258b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d20c0 0x5583694e14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f0319258b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583694e1a10 con 0x5583694d20c0 2023-11-16T16:48:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f0319258b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583694e1b80 con 0x5583694d1500 2023-11-16T16:48:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f0319258b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583694e1d40 con 0x5583694d0bb0 2023-11-16T16:48:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f0319258b80 10 client.0 _ll_register_callbacks cb 0x7f02e800f790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:48:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f02ff7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d20c0 0x5583694e14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f02f7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d1500 0x5583694dc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f02feffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d0bb0 0x5583694dbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f02ff7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d20c0 0x5583694e14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39418/0 (socket says 172.21.15.44:39418) 2023-11-16T16:48:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f02ff7fe700 1 -- 172.21.15.44:0/3842030682 learned_addr learned my addr 172.21.15.44:0/3842030682 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f02f7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d1500 0x5583694dc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:40410/0 (socket says 172.21.15.44:40410) 2023-11-16T16:48:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f02ff7fe700 1 -- 172.21.15.44:0/3842030682 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d0bb0 msgr2=0x5583694dbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:52.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f02ff7fe700 1 --2- 172.21.15.44:0/3842030682 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d0bb0 0x5583694dbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f02ff7fe700 1 -- 172.21.15.44:0/3842030682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d1500 msgr2=0x5583694dc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:52.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f02ff7fe700 1 --2- 172.21.15.44:0/3842030682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d1500 0x5583694dc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.839+0000 7f02ff7fe700 1 -- 172.21.15.44:0/3842030682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5583692a35e0 con 0x5583694d20c0 2023-11-16T16:48:52.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.840+0000 7f02ff7fe700 1 --2- 172.21.15.44:0/3842030682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d20c0 0x5583694e14d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f02f801a7c0 tx=0x7f02f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72d61cf342b6b6ed server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:52.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.840+0000 7f02deffd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:48:52.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.840+0000 7f02deffd700 1 -- 172.21.15.44:0/3842030682 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f800e020 con 0x5583694d20c0 2023-11-16T16:48:52.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.840+0000 7f0319258b80 1 -- 172.21.15.44:0/3842030682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f02f00a5ac0 con 0x5583694d20c0 2023-11-16T16:48:52.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.840+0000 7f02deffd700 1 -- 172.21.15.44:0/3842030682 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f02f801f040 con 0x5583694d20c0 2023-11-16T16:48:52.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.840+0000 7f02deffd700 1 -- 172.21.15.44:0/3842030682 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02f8004db0 con 0x5583694d20c0 2023-11-16T16:48:52.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.840+0000 7f0319258b80 10 client.15105 fetch_fsmap learned FSMap version 21 2023-11-16T16:48:52.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.840+0000 7f0319258b80 1 -- 172.21.15.44:0/3842030682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x558369238c10 con 0x5583694d20c0 2023-11-16T16:48:52.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.841+0000 7f02deffd700 1 -- 172.21.15.44:0/3842030682 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02f00a5ac0 con 0x5583694d20c0 2023-11-16T16:48:52.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.841+0000 7f02deffd700 1 -- 172.21.15.44:0/3842030682 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f02f80052c0 con 0x5583694d20c0 2023-11-16T16:48:52.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.841+0000 7f0319258b80 10 client.15105 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:48:52.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.841+0000 7f0319258b80 10 client.15105 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T16:48:52.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.841+0000 7f0319258b80 20 client.15105 populate_metadata read hostname 'smithi044' 2023-11-16T16:48:52.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.841+0000 7f0319258b80 1 --2- 172.21.15.44:0/3842030682 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55836953e1a0 0x55836953ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.841+0000 7f0319258b80 4 client.15105 mds_command: new command op to 24551 tid=0[{"prefix": "status"}] 2023-11-16T16:48:52.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.841+0000 7f0319258b80 1 -- 172.21.15.44:0/3842030682 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5583693f0610 con 0x55836953e1a0 2023-11-16T16:48:52.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.841+0000 7f02feffd700 1 --2- 172.21.15.44:0/3842030682 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55836953e1a0 0x55836953ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.842+0000 7f02deffd700 10 client.15105 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:48:52.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.842+0000 7f02feffd700 1 --2- 172.21.15.44:0/3842030682 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55836953e1a0 0x55836953ab10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=403922c6cd50e95a server_cookie=c6a360a285b072b7 in_seq=0 out_seq=0 2023-11-16T16:48:52.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.842+0000 7f02deffd700 1 -- 172.21.15.44:0/3842030682 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5583693f0610 con 0x55836953e1a0 2023-11-16T16:48:52.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.842+0000 7f02deffd700 10 client.15105 handle_command_reply: tid=0 2023-11-16T16:48:52.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.842+0000 7f0319258b80 1 client.15105 shutdown 2023-11-16T16:48:52.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.842+0000 7f0319258b80 1 -- 172.21.15.44:0/3842030682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d20c0 msgr2=0x5583694e14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.842+0000 7f0319258b80 1 --2- 172.21.15.44:0/3842030682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d20c0 0x5583694e14d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f02f801a7c0 tx=0x7f02f8004ad0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.842+0000 7f0319258b80 1 -- 172.21.15.44:0/3842030682 shutdown_connections 2023-11-16T16:48:52.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.842+0000 7f0319258b80 1 --2- 172.21.15.44:0/3842030682 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55836953e1a0 0x55836953ab10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.842+0000 7f0319258b80 1 --2- 172.21.15.44:0/3842030682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5583694d20c0 0x5583694e14d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.842+0000 7f0319258b80 1 --2- 172.21.15.44:0/3842030682 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5583694d0bb0 0x5583694dbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.842+0000 7f0319258b80 1 --2- 172.21.15.44:0/3842030682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5583694d1500 0x5583694dc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.842+0000 7f0319258b80 1 -- 172.21.15.44:0/3842030682 >> 172.21.15.44:0/3842030682 conn(0x5583694c6310 msgr2=0x5583694e4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:52.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.842+0000 7f02deffd700 0 client.15105 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:48:52.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.842+0000 7f0319258b80 1 -- 172.21.15.44:0/3842030682 shutdown_connections 2023-11-16T16:48:52.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.842+0000 7f0319258b80 1 -- 172.21.15.44:0/3842030682 wait complete. 2023-11-16T16:48:52.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.843+0000 7f0319258b80 20 client.15105 trim_cache size 0 max 16384 2023-11-16T16:48:52.849 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:48:52.849 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "7f2b12a8-849e-11ee-8dbe-212e2dc638e7", 2023-11-16T16:48:52.849 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2023-11-16T16:48:52.849 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24551, 2023-11-16T16:48:52.849 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2023-11-16T16:48:52.849 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2023-11-16T16:48:52.850 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2023-11-16T16:48:52.850 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 94.686659863000003, 2023-11-16T16:48:52.850 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 21, 2023-11-16T16:48:52.850 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 91, 2023-11-16T16:48:52.850 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 89, 2023-11-16T16:48:52.850 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 100.756947668 2023-11-16T16:48:52.850 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:48:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.843+0000 7f03059ff700 1 -- 172.21.15.44:0/1966095607 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f02ec0a11e0 msgr2=0x7f02ec0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.843+0000 7f03059ff700 1 --2- 172.21.15.44:0/1966095607 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f02ec0a11e0 0x7f02ec0a3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f02f8009610 tx=0x7f02f8004b40 comp rx=0 tx=0).stop 2023-11-16T16:48:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.843+0000 7f03059ff700 1 -- 172.21.15.44:0/1966095607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0300100230 msgr2=0x7f0300195fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.843+0000 7f03059ff700 1 --2- 172.21.15.44:0/1966095607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0300100230 0x7f0300195fb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009c20 tx=0x7f02f0009f30 comp rx=0 tx=0).stop 2023-11-16T16:48:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.843+0000 7f03059ff700 1 -- 172.21.15.44:0/1966095607 shutdown_connections 2023-11-16T16:48:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.843+0000 7f03059ff700 1 --2- 172.21.15.44:0/1966095607 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f02ec0a11e0 0x7f02ec0a3690 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.843+0000 7f03059ff700 1 --2- 172.21.15.44:0/1966095607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0300101db0 0x7f030019d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.843+0000 7f03059ff700 1 --2- 172.21.15.44:0/1966095607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0300100230 0x7f0300195fb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.843+0000 7f03059ff700 1 --2- 172.21.15.44:0/1966095607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0300101420 0x7f03001964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.843+0000 7f03059ff700 1 -- 172.21.15.44:0/1966095607 >> 172.21.15.44:0/1966095607 conn(0x7f03000fb810 msgr2=0x7f0300104900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:52.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.843+0000 7f03059ff700 1 -- 172.21.15.44:0/1966095607 shutdown_connections 2023-11-16T16:48:52.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:52.843+0000 7f03059ff700 1 -- 172.21.15.44:0/1966095607 wait complete. 2023-11-16T16:48:52.860 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:52 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2355181663' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:52.860 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:52 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1766119481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:52.862 INFO:tasks.cephadm:Setup complete, yielding 2023-11-16T16:48:52.863 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-11-16T16:48:52.872 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2023-11-16T16:48:52.872 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch status' 2023-11-16T16:48:52.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:52 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2355181663' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:52.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:52 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1766119481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:52.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:52 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2355181663' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:52.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:52 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1766119481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:53.734 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:53.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:53 smithi176 ceph-mon[131630]: pgmap v326: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2023-11-16T16:48:53.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:53 smithi077 ceph-mon[131204]: pgmap v326: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2023-11-16T16:48:53.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:53 smithi044 ceph-mon[128750]: pgmap v326: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2023-11-16T16:48:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.240+0000 7fac02998700 1 -- 172.21.15.44:0/2061280393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabfc1002f0 msgr2=0x7fabfc100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.240+0000 7fac02998700 1 --2- 172.21.15.44:0/2061280393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabfc1002f0 0x7fabfc100700 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fabec009990 tx=0x7fabec009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.240+0000 7fac02998700 1 -- 172.21.15.44:0/2061280393 shutdown_connections 2023-11-16T16:48:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.240+0000 7fac02998700 1 --2- 172.21.15.44:0/2061280393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabfc101e80 0x7fabfc108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.240+0000 7fac02998700 1 --2- 172.21.15.44:0/2061280393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabfc1002f0 0x7fabfc100700 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.240+0000 7fac02998700 1 --2- 172.21.15.44:0/2061280393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabfc1014f0 0x7fabfc101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.240+0000 7fac02998700 1 -- 172.21.15.44:0/2061280393 >> 172.21.15.44:0/2061280393 conn(0x7fabfc0fb900 msgr2=0x7fabfc0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.240+0000 7fac02998700 1 -- 172.21.15.44:0/2061280393 shutdown_connections 2023-11-16T16:48:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.240+0000 7fac02998700 1 -- 172.21.15.44:0/2061280393 wait complete. 2023-11-16T16:48:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.241+0000 7fac02998700 1 Processor -- start 2023-11-16T16:48:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.241+0000 7fac02998700 1 -- start start 2023-11-16T16:48:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.242+0000 7fac02998700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabfc1002f0 0x7fabfc1960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.242+0000 7fac02998700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabfc1014f0 0x7fabfc1965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.242+0000 7fac02998700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabfc101e80 0x7fabfc19d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.242+0000 7fac02998700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc19dc30 con 0x7fabfc1014f0 2023-11-16T16:48:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.242+0000 7fac02998700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc19dd70 con 0x7fabfc101e80 2023-11-16T16:48:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.242+0000 7fac02998700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc19df00 con 0x7fabfc1002f0 2023-11-16T16:48:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.242+0000 7fabfb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabfc1014f0 0x7fabfc1965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.242+0000 7fabfb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabfc1014f0 0x7fabfc1965e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33932/0 (socket says 172.21.15.44:33932) 2023-11-16T16:48:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.242+0000 7fabfb7fe700 1 -- 172.21.15.44:0/424841973 learned_addr learned my addr 172.21.15.44:0/424841973 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.242+0000 7fac00f35700 1 --2- 172.21.15.44:0/424841973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabfc101e80 0x7fabfc19d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:55.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.242+0000 7fabfbfff700 1 --2- 172.21.15.44:0/424841973 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabfc1002f0 0x7fabfc1960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:55.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.242+0000 7fabfb7fe700 1 -- 172.21.15.44:0/424841973 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabfc1002f0 msgr2=0x7fabfc1960a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:55.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.242+0000 7fabfb7fe700 1 --2- 172.21.15.44:0/424841973 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabfc1002f0 0x7fabfc1960a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.242+0000 7fabfb7fe700 1 -- 172.21.15.44:0/424841973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabfc101e80 msgr2=0x7fabfc19d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:55.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.242+0000 7fabfb7fe700 1 --2- 172.21.15.44:0/424841973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabfc101e80 0x7fabfc19d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.242+0000 7fabfb7fe700 1 -- 172.21.15.44:0/424841973 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabec0095f0 con 0x7fabfc1014f0 2023-11-16T16:48:55.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.243+0000 7fabfb7fe700 1 --2- 172.21.15.44:0/424841973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabfc1014f0 0x7fabfc1965e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fabf000e9d0 tx=0x7fabf000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:55.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.243+0000 7fabf97fa700 1 -- 172.21.15.44:0/424841973 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf0009790 con 0x7fabfc1014f0 2023-11-16T16:48:55.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.243+0000 7fabf97fa700 1 -- 172.21.15.44:0/424841973 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabf000ee70 con 0x7fabfc1014f0 2023-11-16T16:48:55.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.243+0000 7fac02998700 1 -- 172.21.15.44:0/424841973 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabfc10bcb0 con 0x7fabfc1014f0 2023-11-16T16:48:55.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.243+0000 7fabf97fa700 1 -- 172.21.15.44:0/424841973 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf0010ea0 con 0x7fabfc1014f0 2023-11-16T16:48:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.243+0000 7fac02998700 1 -- 172.21.15.44:0/424841973 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabfc10bb20 con 0x7fabfc1014f0 2023-11-16T16:48:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.244+0000 7fac02998700 1 -- 172.21.15.44:0/424841973 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabfc107930 con 0x7fabfc1014f0 2023-11-16T16:48:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.246+0000 7fabf97fa700 1 -- 172.21.15.44:0/424841973 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fabf000cbd0 con 0x7fabfc1014f0 2023-11-16T16:48:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.247+0000 7fabf97fa700 1 --2- 172.21.15.44:0/424841973 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fabe40a11e0 0x7fabe40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.247+0000 7fabf97fa700 1 -- 172.21.15.44:0/424841973 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fabf0014070 con 0x7fabfc1014f0 2023-11-16T16:48:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.248+0000 7fabfbfff700 1 --2- 172.21.15.44:0/424841973 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fabe40a11e0 0x7fabe40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.248+0000 7fabfbfff700 1 --2- 172.21.15.44:0/424841973 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fabe40a11e0 0x7fabe40a3690 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fabec00b2e0 tx=0x7fabec005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.251+0000 7fabf97fa700 1 -- 172.21.15.44:0/424841973 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fabf0051730 con 0x7fabfc1014f0 2023-11-16T16:48:55.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.476+0000 7fac02998700 1 -- 172.21.15.44:0/424841973 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fabfc061c70 con 0x7fabe40a11e0 2023-11-16T16:48:55.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.477+0000 7fabf97fa700 1 -- 172.21.15.44:0/424841973 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fabfc061c70 con 0x7fabe40a11e0 2023-11-16T16:48:55.478 INFO:teuthology.orchestra.run.smithi044.stdout:Backend: cephadm 2023-11-16T16:48:55.478 INFO:teuthology.orchestra.run.smithi044.stdout:Available: Yes 2023-11-16T16:48:55.478 INFO:teuthology.orchestra.run.smithi044.stdout:Paused: No 2023-11-16T16:48:55.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.479+0000 7fac02998700 1 -- 172.21.15.44:0/424841973 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fabe40a11e0 msgr2=0x7fabe40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:55.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.479+0000 7fac02998700 1 --2- 172.21.15.44:0/424841973 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fabe40a11e0 0x7fabe40a3690 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fabec00b2e0 tx=0x7fabec005fb0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.479+0000 7fac02998700 1 -- 172.21.15.44:0/424841973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabfc1014f0 msgr2=0x7fabfc1965e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.479+0000 7fac02998700 1 --2- 172.21.15.44:0/424841973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabfc1014f0 0x7fabfc1965e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fabf000e9d0 tx=0x7fabf000ece0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.480+0000 7fac02998700 1 -- 172.21.15.44:0/424841973 shutdown_connections 2023-11-16T16:48:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.480+0000 7fac02998700 1 --2- 172.21.15.44:0/424841973 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fabe40a11e0 0x7fabe40a3690 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.480+0000 7fac02998700 1 --2- 172.21.15.44:0/424841973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabfc101e80 0x7fabfc19d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.480+0000 7fac02998700 1 --2- 172.21.15.44:0/424841973 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabfc1002f0 0x7fabfc1960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.480+0000 7fac02998700 1 --2- 172.21.15.44:0/424841973 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabfc1014f0 0x7fabfc1965e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.480+0000 7fac02998700 1 -- 172.21.15.44:0/424841973 >> 172.21.15.44:0/424841973 conn(0x7fabfc0fb900 msgr2=0x7fabfc104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.480+0000 7fac02998700 1 -- 172.21.15.44:0/424841973 shutdown_connections 2023-11-16T16:48:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:55.480+0000 7fac02998700 1 -- 172.21.15.44:0/424841973 wait complete. 2023-11-16T16:48:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:55 smithi044 ceph-mon[128750]: pgmap v327: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2023-11-16T16:48:55.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:55 smithi176 ceph-mon[131630]: pgmap v327: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2023-11-16T16:48:55.936 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch ps' 2023-11-16T16:48:55.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:55 smithi077 ceph-mon[131204]: pgmap v327: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2023-11-16T16:48:56.798 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:56.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:56 smithi077 ceph-mon[131204]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:48:57.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:56 smithi044 ceph-mon[128750]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:48:57.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:56 smithi176 ceph-mon[131630]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:48:57.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:57 smithi077 ceph-mon[131204]: pgmap v328: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 857 B/s rd, 1 op/s 2023-11-16T16:48:58.090 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:57 smithi044 ceph-mon[128750]: pgmap v328: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 857 B/s rd, 1 op/s 2023-11-16T16:48:58.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:57 smithi176 ceph-mon[131630]: pgmap v328: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 857 B/s rd, 1 op/s 2023-11-16T16:48:58.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.339+0000 7fe5bcc2d700 1 -- 172.21.15.44:0/2372472410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5b8101e80 msgr2=0x7fe5b8108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.339+0000 7fe5bcc2d700 1 --2- 172.21.15.44:0/2372472410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5b8101e80 0x7fe5b8108740 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe5b000b0a0 tx=0x7fe5b000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.339+0000 7fe5bcc2d700 1 -- 172.21.15.44:0/2372472410 shutdown_connections 2023-11-16T16:48:58.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.339+0000 7fe5bcc2d700 1 --2- 172.21.15.44:0/2372472410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5b8101e80 0x7fe5b8108740 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.339+0000 7fe5bcc2d700 1 --2- 172.21.15.44:0/2372472410 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5b81002f0 0x7fe5b8100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.339+0000 7fe5bcc2d700 1 --2- 172.21.15.44:0/2372472410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5b81014f0 0x7fe5b8101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.341 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.339+0000 7fe5bcc2d700 1 -- 172.21.15.44:0/2372472410 >> 172.21.15.44:0/2372472410 conn(0x7fe5b80fb900 msgr2=0x7fe5b80fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:58.341 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.339+0000 7fe5bcc2d700 1 -- 172.21.15.44:0/2372472410 shutdown_connections 2023-11-16T16:48:58.341 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.339+0000 7fe5bcc2d700 1 -- 172.21.15.44:0/2372472410 wait complete. 2023-11-16T16:48:58.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.340+0000 7fe5bcc2d700 1 Processor -- start 2023-11-16T16:48:58.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.340+0000 7fe5bcc2d700 1 -- start start 2023-11-16T16:48:58.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.341+0000 7fe5bcc2d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5b81002f0 0x7fe5b8198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.341+0000 7fe5bcc2d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5b81014f0 0x7fe5b8198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.341+0000 7fe5bcc2d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5b8101e80 0x7fe5b819f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.341+0000 7fe5bcc2d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b819fee0 con 0x7fe5b81002f0 2023-11-16T16:48:58.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.341+0000 7fe5bcc2d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b81a0020 con 0x7fe5b81014f0 2023-11-16T16:48:58.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.341+0000 7fe5bcc2d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b81a01b0 con 0x7fe5b8101e80 2023-11-16T16:48:58.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.341+0000 7fe5b659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5b81002f0 0x7fe5b8198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.341+0000 7fe5b659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5b81002f0 0x7fe5b8198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33946/0 (socket says 172.21.15.44:33946) 2023-11-16T16:48:58.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.341+0000 7fe5b659c700 1 -- 172.21.15.44:0/2530600018 learned_addr learned my addr 172.21.15.44:0/2530600018 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:48:58.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.341+0000 7fe5b6d9d700 1 --2- 172.21.15.44:0/2530600018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5b8101e80 0x7fe5b819f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.341+0000 7fe5b5d9b700 1 --2- 172.21.15.44:0/2530600018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5b81014f0 0x7fe5b8198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.341+0000 7fe5b659c700 1 -- 172.21.15.44:0/2530600018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5b8101e80 msgr2=0x7fe5b819f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.341+0000 7fe5b659c700 1 --2- 172.21.15.44:0/2530600018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5b8101e80 0x7fe5b819f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.341+0000 7fe5b659c700 1 -- 172.21.15.44:0/2530600018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5b81014f0 msgr2=0x7fe5b8198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.341+0000 7fe5b659c700 1 --2- 172.21.15.44:0/2530600018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5b81014f0 0x7fe5b8198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.341+0000 7fe5b659c700 1 -- 172.21.15.44:0/2530600018 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b0009d00 con 0x7fe5b81002f0 2023-11-16T16:48:58.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.342+0000 7fe5b659c700 1 --2- 172.21.15.44:0/2530600018 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5b81002f0 0x7fe5b8198350 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe5a800d9b0 tx=0x7fe5a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:58.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.342+0000 7fe5a77fe700 1 -- 172.21.15.44:0/2530600018 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a8004020 con 0x7fe5b81002f0 2023-11-16T16:48:58.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.342+0000 7fe5bcc2d700 1 -- 172.21.15.44:0/2530600018 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b810bcb0 con 0x7fe5b81002f0 2023-11-16T16:48:58.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.342+0000 7fe5a77fe700 1 -- 172.21.15.44:0/2530600018 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5a8009730 con 0x7fe5b81002f0 2023-11-16T16:48:58.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.342+0000 7fe5a77fe700 1 -- 172.21.15.44:0/2530600018 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a8003930 con 0x7fe5b81002f0 2023-11-16T16:48:58.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.342+0000 7fe5bcc2d700 1 -- 172.21.15.44:0/2530600018 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b810bb20 con 0x7fe5b81002f0 2023-11-16T16:48:58.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.343+0000 7fe5bcc2d700 1 -- 172.21.15.44:0/2530600018 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b8066ea0 con 0x7fe5b81002f0 2023-11-16T16:48:58.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.345+0000 7fe5a77fe700 1 -- 172.21.15.44:0/2530600018 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe5a80098a0 con 0x7fe5b81002f0 2023-11-16T16:48:58.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.346+0000 7fe5a77fe700 1 --2- 172.21.15.44:0/2530600018 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5a00a11e0 0x7fe5a00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.347+0000 7fe5a77fe700 1 -- 172.21.15.44:0/2530600018 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fe5a80cc7d0 con 0x7fe5b81002f0 2023-11-16T16:48:58.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.347+0000 7fe5b5d9b700 1 --2- 172.21.15.44:0/2530600018 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5a00a11e0 0x7fe5a00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.347+0000 7fe5b5d9b700 1 --2- 172.21.15.44:0/2530600018 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5a00a11e0 0x7fe5a00a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac009c60 tx=0x7fe5ac009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:58.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.351+0000 7fe5a77fe700 1 -- 172.21.15.44:0/2530600018 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe5a80959c0 con 0x7fe5b81002f0 2023-11-16T16:48:58.569 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.568+0000 7fe5bcc2d700 1 -- 172.21.15.44:0/2530600018 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5b81a0d60 con 0x7fe5a00a11e0 2023-11-16T16:48:58.582 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.581+0000 7fe5a77fe700 1 -- 172.21.15.44:0/2530600018 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7fe5b81a0d60 con 0x7fe5a00a11e0 2023-11-16T16:48:58.583 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-11-16T16:48:58.583 INFO:teuthology.orchestra.run.smithi044.stdout:mds.a smithi044 running (109s) 26s ago 108s 13.7M - 18.0.0-3905-g5e4a743a d5b1f852d682 33941a07c921 2023-11-16T16:48:58.583 INFO:teuthology.orchestra.run.smithi044.stdout:mds.b smithi077 running (106s) 35s ago 106s 28.9M - 18.0.0-3905-g5e4a743a d5b1f852d682 a51f50877448 2023-11-16T16:48:58.583 INFO:teuthology.orchestra.run.smithi044.stdout:mds.c smithi176 running (119s) 35s ago 119s 17.5M - 18.0.0-3905-g5e4a743a d5b1f852d682 d553725e39cf 2023-11-16T16:48:58.583 INFO:teuthology.orchestra.run.smithi044.stdout:mds.d smithi044 running (2m) 26s ago 2m 13.8M - 18.0.0-3905-g5e4a743a d5b1f852d682 0f118334eb98 2023-11-16T16:48:58.583 INFO:teuthology.orchestra.run.smithi044.stdout:mds.e smithi077 running (2m) 35s ago 2m 17.5M - 18.0.0-3905-g5e4a743a d5b1f852d682 dafc8b7cc14e 2023-11-16T16:48:58.583 INFO:teuthology.orchestra.run.smithi044.stdout:mds.f smithi176 running (2m) 35s ago 2m 15.1M - 18.0.0-3905-g5e4a743a d5b1f852d682 402c4a9c283b 2023-11-16T16:48:58.583 INFO:teuthology.orchestra.run.smithi044.stdout:mds.g smithi044 running (112s) 26s ago 111s 13.7M - 18.0.0-3905-g5e4a743a d5b1f852d682 ecc4738c1543 2023-11-16T16:48:58.583 INFO:teuthology.orchestra.run.smithi044.stdout:mds.h smithi077 running (117s) 35s ago 116s 14.3M - 18.0.0-3905-g5e4a743a d5b1f852d682 5b9e8bd097f3 2023-11-16T16:48:58.584 INFO:teuthology.orchestra.run.smithi044.stdout:mds.i smithi176 running (115s) 35s ago 114s 19.9M - 18.0.0-3905-g5e4a743a d5b1f852d682 1c4a3f323f7b 2023-11-16T16:48:58.584 INFO:teuthology.orchestra.run.smithi044.stdout:mds.j smithi044 running (2m) 26s ago 2m 13.8M - 18.0.0-3905-g5e4a743a d5b1f852d682 bfaef8baf56b 2023-11-16T16:48:58.584 INFO:teuthology.orchestra.run.smithi044.stdout:mds.k smithi077 running (2m) 35s ago 2m 15.1M - 18.0.0-3905-g5e4a743a d5b1f852d682 ffa8502df4d4 2023-11-16T16:48:58.584 INFO:teuthology.orchestra.run.smithi044.stdout:mds.l smithi176 running (2m) 35s ago 2m 17.3M - 18.0.0-3905-g5e4a743a d5b1f852d682 21103a09a097 2023-11-16T16:48:58.584 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi044 *:9283,8765 running (9m) 26s ago 9m 521M - 18.0.0-3905-g5e4a743a d5b1f852d682 f85ee589f567 2023-11-16T16:48:58.584 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi077 *:8443,8765 running (7m) 35s ago 7m 446M - 18.0.0-3905-g5e4a743a d5b1f852d682 4241444eddf8 2023-11-16T16:48:58.584 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.z smithi176 *:8443,8765 running (7m) 35s ago 7m 448M - 18.0.0-3905-g5e4a743a d5b1f852d682 1fc33ae431a2 2023-11-16T16:48:58.584 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (9m) 26s ago 9m 93.4M 2048M 18.0.0-3905-g5e4a743a d5b1f852d682 4abeb7b1fb73 2023-11-16T16:48:58.584 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi077 running (7m) 35s ago 7m 63.5M 2048M 18.0.0-3905-g5e4a743a d5b1f852d682 90599dd7dadd 2023-11-16T16:48:58.584 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi176 running (7m) 35s ago 7m 66.3M 2048M 18.0.0-3905-g5e4a743a d5b1f852d682 bf242255d40b 2023-11-16T16:48:58.585 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (6m) 26s ago 6m 70.9M 4319M 18.0.0-3905-g5e4a743a d5b1f852d682 26b10277c44d 2023-11-16T16:48:58.585 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi077 running (6m) 35s ago 6m 79.4M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 15c3facf14b8 2023-11-16T16:48:58.585 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi176 running (6m) 35s ago 6m 73.3M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 8a8e01abe60b 2023-11-16T16:48:58.585 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (5m) 26s ago 5m 65.3M 4319M 18.0.0-3905-g5e4a743a d5b1f852d682 3cdb17eae898 2023-11-16T16:48:58.585 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi077 running (5m) 35s ago 5m 65.2M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 035c16f470cf 2023-11-16T16:48:58.585 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi176 running (4m) 35s ago 4m 71.4M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 7eb67e2de181 2023-11-16T16:48:58.585 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi044 running (4m) 26s ago 4m 61.7M 4319M 18.0.0-3905-g5e4a743a d5b1f852d682 888fff08ef16 2023-11-16T16:48:58.585 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi077 running (4m) 35s ago 4m 65.2M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 8d54d466de8e 2023-11-16T16:48:58.585 INFO:teuthology.orchestra.run.smithi044.stdout:osd.8 smithi176 running (3m) 35s ago 3m 60.8M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 3e12fed54480 2023-11-16T16:48:58.585 INFO:teuthology.orchestra.run.smithi044.stdout:osd.9 smithi044 running (3m) 26s ago 3m 59.2M 4319M 18.0.0-3905-g5e4a743a d5b1f852d682 fcc174151972 2023-11-16T16:48:58.585 INFO:teuthology.orchestra.run.smithi044.stdout:osd.10 smithi077 running (2m) 35s ago 2m 61.3M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 2effabbabbec 2023-11-16T16:48:58.585 INFO:teuthology.orchestra.run.smithi044.stdout:osd.11 smithi176 running (2m) 35s ago 2m 57.3M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 9581d8da7a6b 2023-11-16T16:48:58.586 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.584+0000 7fe5bcc2d700 1 -- 172.21.15.44:0/2530600018 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5a00a11e0 msgr2=0x7fe5a00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.586 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.584+0000 7fe5bcc2d700 1 --2- 172.21.15.44:0/2530600018 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5a00a11e0 0x7fe5a00a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac009c60 tx=0x7fe5ac009260 comp rx=0 tx=0).stop 2023-11-16T16:48:58.586 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.584+0000 7fe5bcc2d700 1 -- 172.21.15.44:0/2530600018 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5b81002f0 msgr2=0x7fe5b8198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.587 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.584+0000 7fe5bcc2d700 1 --2- 172.21.15.44:0/2530600018 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5b81002f0 0x7fe5b8198350 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe5a800d9b0 tx=0x7fe5a800dcc0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.587 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.584+0000 7fe5bcc2d700 1 -- 172.21.15.44:0/2530600018 shutdown_connections 2023-11-16T16:48:58.587 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.584+0000 7fe5bcc2d700 1 --2- 172.21.15.44:0/2530600018 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5a00a11e0 0x7fe5a00a3690 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.587 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.584+0000 7fe5bcc2d700 1 --2- 172.21.15.44:0/2530600018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5b8101e80 0x7fe5b819f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.587 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.584+0000 7fe5bcc2d700 1 --2- 172.21.15.44:0/2530600018 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5b81002f0 0x7fe5b8198350 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.587 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.584+0000 7fe5bcc2d700 1 --2- 172.21.15.44:0/2530600018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5b81014f0 0x7fe5b8198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.587 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.584+0000 7fe5bcc2d700 1 -- 172.21.15.44:0/2530600018 >> 172.21.15.44:0/2530600018 conn(0x7fe5b80fb900 msgr2=0x7fe5b80fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:58.587 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.584+0000 7fe5bcc2d700 1 -- 172.21.15.44:0/2530600018 shutdown_connections 2023-11-16T16:48:58.587 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:48:58.584+0000 7fe5bcc2d700 1 -- 172.21.15.44:0/2530600018 wait complete. 2023-11-16T16:48:59.202 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch ls' 2023-11-16T16:48:59.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:59 smithi077 ceph-mon[131204]: pgmap v329: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:59.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:48:59 smithi077 ceph-mon[131204]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:48:59.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:59 smithi044 ceph-mon[128750]: pgmap v329: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:59.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:48:59 smithi044 ceph-mon[128750]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:48:59.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:59 smithi176 ceph-mon[131630]: pgmap v329: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:48:59.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:48:59 smithi176 ceph-mon[131630]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:00.066 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:49:01.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.587+0000 7f3c862ce700 1 -- 172.21.15.44:0/1875789566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c801014f0 msgr2=0x7f3c80101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:01.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.587+0000 7f3c862ce700 1 --2- 172.21.15.44:0/1875789566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c801014f0 0x7f3c80101940 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3c74009990 tx=0x7f3c74009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:01.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.588+0000 7f3c862ce700 1 -- 172.21.15.44:0/1875789566 shutdown_connections 2023-11-16T16:49:01.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.588+0000 7f3c862ce700 1 --2- 172.21.15.44:0/1875789566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c80101e80 0x7f3c80108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:01.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.588+0000 7f3c862ce700 1 --2- 172.21.15.44:0/1875789566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c801002f0 0x7f3c80100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:01.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.588+0000 7f3c862ce700 1 --2- 172.21.15.44:0/1875789566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c801014f0 0x7f3c80101940 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:01.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.588+0000 7f3c862ce700 1 -- 172.21.15.44:0/1875789566 >> 172.21.15.44:0/1875789566 conn(0x7f3c800fb900 msgr2=0x7f3c800fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:01.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.588+0000 7f3c862ce700 1 -- 172.21.15.44:0/1875789566 shutdown_connections 2023-11-16T16:49:01.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.588+0000 7f3c862ce700 1 -- 172.21.15.44:0/1875789566 wait complete. 2023-11-16T16:49:01.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.589+0000 7f3c862ce700 1 Processor -- start 2023-11-16T16:49:01.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.589+0000 7f3c862ce700 1 -- start start 2023-11-16T16:49:01.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.589+0000 7f3c862ce700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c801002f0 0x7f3c80198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:01.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.589+0000 7f3c862ce700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c801014f0 0x7f3c80198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:01.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.589+0000 7f3c862ce700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c80101e80 0x7f3c8019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:01.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.589+0000 7f3c862ce700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8019fda0 con 0x7f3c80101e80 2023-11-16T16:49:01.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.589+0000 7f3c862ce700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8019fee0 con 0x7f3c801014f0 2023-11-16T16:49:01.592 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.589+0000 7f3c862ce700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c801a0070 con 0x7f3c801002f0 2023-11-16T16:49:01.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.590+0000 7f3c7f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c801014f0 0x7f3c80198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:01.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.590+0000 7f3c8486b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c80101e80 0x7f3c8019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:01.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.590+0000 7f3c7f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c801014f0 0x7f3c80198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:41062/0 (socket says 172.21.15.44:41062) 2023-11-16T16:49:01.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.590+0000 7f3c7f7fe700 1 -- 172.21.15.44:0/2310376727 learned_addr learned my addr 172.21.15.44:0/2310376727 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:01.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.590+0000 7f3c7ffff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c801002f0 0x7f3c80198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:01.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.590+0000 7f3c8486b700 1 -- 172.21.15.44:0/2310376727 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c801002f0 msgr2=0x7f3c80198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:01.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.590+0000 7f3c8486b700 1 --2- 172.21.15.44:0/2310376727 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c801002f0 0x7f3c80198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:01.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.590+0000 7f3c8486b700 1 -- 172.21.15.44:0/2310376727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c801014f0 msgr2=0x7f3c80198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:01.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.590+0000 7f3c8486b700 1 --2- 172.21.15.44:0/2310376727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c801014f0 0x7f3c80198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:01.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.590+0000 7f3c8486b700 1 -- 172.21.15.44:0/2310376727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c740095f0 con 0x7f3c80101e80 2023-11-16T16:49:01.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.590+0000 7f3c7f7fe700 1 --2- 172.21.15.44:0/2310376727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c801014f0 0x7f3c80198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:01.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.590+0000 7f3c8486b700 1 --2- 172.21.15.44:0/2310376727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c80101e80 0x7f3c8019f7d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009f60 tx=0x7f3c78009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:01.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.591+0000 7f3c7d7fa700 1 -- 172.21.15.44:0/2310376727 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c78018070 con 0x7f3c80101e80 2023-11-16T16:49:01.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.591+0000 7f3c862ce700 1 -- 172.21.15.44:0/2310376727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8010bc50 con 0x7f3c80101e80 2023-11-16T16:49:01.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.591+0000 7f3c7d7fa700 1 -- 172.21.15.44:0/2310376727 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c7800f040 con 0x7f3c80101e80 2023-11-16T16:49:01.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.591+0000 7f3c7d7fa700 1 -- 172.21.15.44:0/2310376727 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7800ccc0 con 0x7f3c80101e80 2023-11-16T16:49:01.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.591+0000 7f3c862ce700 1 -- 172.21.15.44:0/2310376727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8010bac0 con 0x7f3c80101e80 2023-11-16T16:49:01.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.592+0000 7f3c862ce700 1 -- 172.21.15.44:0/2310376727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c801a06f0 con 0x7f3c80101e80 2023-11-16T16:49:01.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.594+0000 7f3c7d7fa700 1 -- 172.21.15.44:0/2310376727 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3c78003a00 con 0x7f3c80101e80 2023-11-16T16:49:01.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.595+0000 7f3c7d7fa700 1 --2- 172.21.15.44:0/2310376727 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3c680a1260 0x7f3c680a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:01.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.596+0000 7f3c7d7fa700 1 -- 172.21.15.44:0/2310376727 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f3c780cc870 con 0x7f3c80101e80 2023-11-16T16:49:01.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.596+0000 7f3c7ffff700 1 --2- 172.21.15.44:0/2310376727 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3c680a1260 0x7f3c680a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:01.598 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.596+0000 7f3c7ffff700 1 --2- 172.21.15.44:0/2310376727 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3c680a1260 0x7f3c680a3710 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3c80101350 tx=0x7f3c70007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:01.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.599+0000 7f3c7d7fa700 1 -- 172.21.15.44:0/2310376727 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3c78095af0 con 0x7f3c80101e80 2023-11-16T16:49:01.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.818+0000 7f3c862ce700 1 -- 172.21.15.44:0/2310376727 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c80107e90 con 0x7f3c680a1260 2023-11-16T16:49:01.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.823+0000 7f3c7d7fa700 1 -- 172.21.15.44:0/2310376727 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f3c80107e90 con 0x7f3c680a1260 2023-11-16T16:49:01.825 INFO:teuthology.orchestra.run.smithi044.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-11-16T16:49:01.825 INFO:teuthology.orchestra.run.smithi044.stdout:mds.all 12/12 38s ago 2m smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi176=c;smithi176=f;smithi176=i;smithi176=l;count:12 2023-11-16T16:49:01.826 INFO:teuthology.orchestra.run.smithi044.stdout:mgr 3/3 38s ago 7m smithi044=x;smithi077=y;smithi176=z;count:3 2023-11-16T16:49:01.826 INFO:teuthology.orchestra.run.smithi044.stdout:mon 3/3 38s ago 7m smithi044:172.21.15.44=a;smithi077:172.21.15.77=b;smithi176:172.21.15.176=c;count:3 2023-11-16T16:49:01.826 INFO:teuthology.orchestra.run.smithi044.stdout:osd 12 38s ago - 2023-11-16T16:49:01.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.826+0000 7f3c862ce700 1 -- 172.21.15.44:0/2310376727 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3c680a1260 msgr2=0x7f3c680a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:01.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.826+0000 7f3c862ce700 1 --2- 172.21.15.44:0/2310376727 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3c680a1260 0x7f3c680a3710 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3c80101350 tx=0x7f3c70007400 comp rx=0 tx=0).stop 2023-11-16T16:49:01.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.826+0000 7f3c862ce700 1 -- 172.21.15.44:0/2310376727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c80101e80 msgr2=0x7f3c8019f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:01.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.826+0000 7f3c862ce700 1 --2- 172.21.15.44:0/2310376727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c80101e80 0x7f3c8019f7d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009f60 tx=0x7f3c78009f90 comp rx=0 tx=0).stop 2023-11-16T16:49:01.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.826+0000 7f3c862ce700 1 -- 172.21.15.44:0/2310376727 shutdown_connections 2023-11-16T16:49:01.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.826+0000 7f3c862ce700 1 --2- 172.21.15.44:0/2310376727 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3c680a1260 0x7f3c680a3710 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:01.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.826+0000 7f3c862ce700 1 --2- 172.21.15.44:0/2310376727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c80101e80 0x7f3c8019f7d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:01.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.826+0000 7f3c862ce700 1 --2- 172.21.15.44:0/2310376727 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c801002f0 0x7f3c80198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:01.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.826+0000 7f3c862ce700 1 --2- 172.21.15.44:0/2310376727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c801014f0 0x7f3c80198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:01.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.826+0000 7f3c862ce700 1 -- 172.21.15.44:0/2310376727 >> 172.21.15.44:0/2310376727 conn(0x7f3c800fb900 msgr2=0x7f3c800fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:01.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.826+0000 7f3c862ce700 1 -- 172.21.15.44:0/2310376727 shutdown_connections 2023-11-16T16:49:01.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:01.826+0000 7f3c862ce700 1 -- 172.21.15.44:0/2310376727 wait complete. 2023-11-16T16:49:01.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:01 smithi044 ceph-mon[128750]: pgmap v330: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:49:01.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:01 smithi176 ceph-mon[131630]: pgmap v330: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:49:01.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:01 smithi077 ceph-mon[131204]: pgmap v330: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:49:02.266 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch host ls' 2023-11-16T16:49:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:02 smithi176 ceph-mon[131630]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:02.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:02 smithi077 ceph-mon[131204]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:03.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:02 smithi044 ceph-mon[128750]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:03.134 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:49:03.739 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:03 smithi044 ceph-mon[128750]: pgmap v331: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:03.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:03 smithi176 ceph-mon[131630]: pgmap v331: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:03.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:03 smithi077 ceph-mon[131204]: pgmap v331: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:04.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.771+0000 7fd0d4684700 1 -- 172.21.15.44:0/2885371712 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0cc1014f0 msgr2=0x7fd0cc101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:04.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.771+0000 7fd0d4684700 1 --2- 172.21.15.44:0/2885371712 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0cc1014f0 0x7fd0cc101940 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc009990 tx=0x7fd0bc009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:04.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.772+0000 7fd0d4684700 1 -- 172.21.15.44:0/2885371712 shutdown_connections 2023-11-16T16:49:04.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.772+0000 7fd0d4684700 1 --2- 172.21.15.44:0/2885371712 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0cc101e80 0x7fd0cc108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:04.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.772+0000 7fd0d4684700 1 --2- 172.21.15.44:0/2885371712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0cc1002f0 0x7fd0cc100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:04.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.772+0000 7fd0d4684700 1 --2- 172.21.15.44:0/2885371712 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0cc1014f0 0x7fd0cc101940 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:04.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.772+0000 7fd0d4684700 1 -- 172.21.15.44:0/2885371712 >> 172.21.15.44:0/2885371712 conn(0x7fd0cc0fb900 msgr2=0x7fd0cc0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:04.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.772+0000 7fd0d4684700 1 -- 172.21.15.44:0/2885371712 shutdown_connections 2023-11-16T16:49:04.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.772+0000 7fd0d4684700 1 -- 172.21.15.44:0/2885371712 wait complete. 2023-11-16T16:49:04.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.773+0000 7fd0d4684700 1 Processor -- start 2023-11-16T16:49:04.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.773+0000 7fd0d4684700 1 -- start start 2023-11-16T16:49:04.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.774+0000 7fd0d4684700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0cc1002f0 0x7fd0cc196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:04.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.774+0000 7fd0d4684700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0cc1014f0 0x7fd0cc196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:04.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.774+0000 7fd0d4684700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0cc101e80 0x7fd0cc19d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:04.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.774+0000 7fd0d4684700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc19dbc0 con 0x7fd0cc1014f0 2023-11-16T16:49:04.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.774+0000 7fd0d4684700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc19dd00 con 0x7fd0cc1002f0 2023-11-16T16:49:04.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.774+0000 7fd0d4684700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc19de90 con 0x7fd0cc101e80 2023-11-16T16:49:04.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.774+0000 7fd0d2420700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0cc1002f0 0x7fd0cc196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:04.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.774+0000 7fd0d2420700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0cc1002f0 0x7fd0cc196030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:41078/0 (socket says 172.21.15.44:41078) 2023-11-16T16:49:04.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.774+0000 7fd0d2420700 1 -- 172.21.15.44:0/1164239801 learned_addr learned my addr 172.21.15.44:0/1164239801 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:04.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.774+0000 7fd0d1c1f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0cc1014f0 0x7fd0cc196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:04.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.774+0000 7fd0d2c21700 1 --2- 172.21.15.44:0/1164239801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0cc101e80 0x7fd0cc19d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:04.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.775+0000 7fd0d1c1f700 1 -- 172.21.15.44:0/1164239801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0cc101e80 msgr2=0x7fd0cc19d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:04.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.775+0000 7fd0d1c1f700 1 --2- 172.21.15.44:0/1164239801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0cc101e80 0x7fd0cc19d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:04.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.775+0000 7fd0d1c1f700 1 -- 172.21.15.44:0/1164239801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0cc1002f0 msgr2=0x7fd0cc196030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:04.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.775+0000 7fd0d1c1f700 1 --2- 172.21.15.44:0/1164239801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0cc1002f0 0x7fd0cc196030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:04.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.775+0000 7fd0d1c1f700 1 -- 172.21.15.44:0/1164239801 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc0095f0 con 0x7fd0cc1014f0 2023-11-16T16:49:04.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.775+0000 7fd0d2420700 1 --2- 172.21.15.44:0/1164239801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0cc1002f0 0x7fd0cc196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:04.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.775+0000 7fd0d1c1f700 1 --2- 172.21.15.44:0/1164239801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0cc1014f0 0x7fd0cc196570 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc000c00 tx=0x7fd0bc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:04.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.775+0000 7fd0c37fe700 1 -- 172.21.15.44:0/1164239801 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc01d070 con 0x7fd0cc1014f0 2023-11-16T16:49:04.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.775+0000 7fd0c37fe700 1 -- 172.21.15.44:0/1164239801 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0bc005030 con 0x7fd0cc1014f0 2023-11-16T16:49:04.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.775+0000 7fd0d4684700 1 -- 172.21.15.44:0/1164239801 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0cc10bc50 con 0x7fd0cc1014f0 2023-11-16T16:49:04.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.775+0000 7fd0c37fe700 1 -- 172.21.15.44:0/1164239801 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc004a20 con 0x7fd0cc1014f0 2023-11-16T16:49:04.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.775+0000 7fd0d4684700 1 -- 172.21.15.44:0/1164239801 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0cc10bac0 con 0x7fd0cc1014f0 2023-11-16T16:49:04.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.777+0000 7fd0d4684700 1 -- 172.21.15.44:0/1164239801 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0cc19e450 con 0x7fd0cc1014f0 2023-11-16T16:49:04.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.778+0000 7fd0c37fe700 1 -- 172.21.15.44:0/1164239801 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd0bc0051a0 con 0x7fd0cc1014f0 2023-11-16T16:49:04.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.780+0000 7fd0c37fe700 1 --2- 172.21.15.44:0/1164239801 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd0b80a1260 0x7fd0b80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:04.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.780+0000 7fd0c37fe700 1 -- 172.21.15.44:0/1164239801 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fd0bc0cdd00 con 0x7fd0cc1014f0 2023-11-16T16:49:04.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.780+0000 7fd0d2420700 1 --2- 172.21.15.44:0/1164239801 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd0b80a1260 0x7fd0b80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:04.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.781+0000 7fd0d2420700 1 --2- 172.21.15.44:0/1164239801 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd0b80a1260 0x7fd0b80a3710 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd0c80105c0 tx=0x7fd0c800b4a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:04.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.784+0000 7fd0c37fe700 1 -- 172.21.15.44:0/1164239801 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd0bc096f80 con 0x7fd0cc1014f0 2023-11-16T16:49:04.998 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.997+0000 7fd0d4684700 1 -- 172.21.15.44:0/1164239801 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0cc060f30 con 0x7fd0b80a1260 2023-11-16T16:49:05.000 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:04.998+0000 7fd0c37fe700 1 -- 172.21.15.44:0/1164239801 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fd0cc060f30 con 0x7fd0b80a1260 2023-11-16T16:49:05.000 INFO:teuthology.orchestra.run.smithi044.stdout:HOST ADDR LABELS STATUS 2023-11-16T16:49:05.000 INFO:teuthology.orchestra.run.smithi044.stdout:smithi044 172.21.15.44 2023-11-16T16:49:05.000 INFO:teuthology.orchestra.run.smithi044.stdout:smithi077 172.21.15.77 2023-11-16T16:49:05.000 INFO:teuthology.orchestra.run.smithi044.stdout:smithi176 172.21.15.176 2023-11-16T16:49:05.000 INFO:teuthology.orchestra.run.smithi044.stdout:3 hosts in cluster 2023-11-16T16:49:05.002 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:05.001+0000 7fd0d4684700 1 -- 172.21.15.44:0/1164239801 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd0b80a1260 msgr2=0x7fd0b80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:05.001+0000 7fd0d4684700 1 --2- 172.21.15.44:0/1164239801 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd0b80a1260 0x7fd0b80a3710 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd0c80105c0 tx=0x7fd0c800b4a0 comp rx=0 tx=0).stop 2023-11-16T16:49:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:05.001+0000 7fd0d4684700 1 -- 172.21.15.44:0/1164239801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0cc1014f0 msgr2=0x7fd0cc196570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:05.001+0000 7fd0d4684700 1 --2- 172.21.15.44:0/1164239801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0cc1014f0 0x7fd0cc196570 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc000c00 tx=0x7fd0bc00f690 comp rx=0 tx=0).stop 2023-11-16T16:49:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:05.001+0000 7fd0d4684700 1 -- 172.21.15.44:0/1164239801 shutdown_connections 2023-11-16T16:49:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:05.001+0000 7fd0d4684700 1 --2- 172.21.15.44:0/1164239801 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fd0b80a1260 0x7fd0b80a3710 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:05.001+0000 7fd0d4684700 1 --2- 172.21.15.44:0/1164239801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0cc101e80 0x7fd0cc19d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:05.001+0000 7fd0d4684700 1 --2- 172.21.15.44:0/1164239801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0cc1002f0 0x7fd0cc196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:05.001+0000 7fd0d4684700 1 --2- 172.21.15.44:0/1164239801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0cc1014f0 0x7fd0cc196570 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:05.001+0000 7fd0d4684700 1 -- 172.21.15.44:0/1164239801 >> 172.21.15.44:0/1164239801 conn(0x7fd0cc0fb900 msgr2=0x7fd0cc0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:05.001+0000 7fd0d4684700 1 -- 172.21.15.44:0/1164239801 shutdown_connections 2023-11-16T16:49:05.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:05.001+0000 7fd0d4684700 1 -- 172.21.15.44:0/1164239801 wait complete. 2023-11-16T16:49:05.434 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch device ls' 2023-11-16T16:49:05.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:05 smithi044 ceph-mon[128750]: pgmap v332: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:05.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:05 smithi176 ceph-mon[131630]: pgmap v332: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:05.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:05 smithi077 ceph-mon[131204]: pgmap v332: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:06.298 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:49:06.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:06 smithi044 ceph-mon[128750]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:06.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:06 smithi176 ceph-mon[131630]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:06.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:06 smithi077 ceph-mon[131204]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:07.708 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.706+0000 7f707c2f7700 1 -- 172.21.15.44:0/2731370735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70741002f0 msgr2=0x7f7074100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:07.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.706+0000 7f707c2f7700 1 --2- 172.21.15.44:0/2731370735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70741002f0 0x7f7074100700 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7070009990 tx=0x7f7070009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:07.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.706+0000 7f707c2f7700 1 -- 172.21.15.44:0/2731370735 shutdown_connections 2023-11-16T16:49:07.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.706+0000 7f707c2f7700 1 --2- 172.21.15.44:0/2731370735 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7074101e80 0x7f7074108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:07.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.706+0000 7f707c2f7700 1 --2- 172.21.15.44:0/2731370735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70741002f0 0x7f7074100700 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:07.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.706+0000 7f707c2f7700 1 --2- 172.21.15.44:0/2731370735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70741014f0 0x7f7074101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:07.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.706+0000 7f707c2f7700 1 -- 172.21.15.44:0/2731370735 >> 172.21.15.44:0/2731370735 conn(0x7f70740fb900 msgr2=0x7f70740fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:07.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.706+0000 7f707c2f7700 1 -- 172.21.15.44:0/2731370735 shutdown_connections 2023-11-16T16:49:07.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.706+0000 7f707c2f7700 1 -- 172.21.15.44:0/2731370735 wait complete. 2023-11-16T16:49:07.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.707+0000 7f707c2f7700 1 Processor -- start 2023-11-16T16:49:07.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.708+0000 7f707c2f7700 1 -- start start 2023-11-16T16:49:07.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.708+0000 7f707c2f7700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70741002f0 0x7f7074074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:07.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.708+0000 7f707c2f7700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70741014f0 0x7f7074072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:07.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.708+0000 7f707c2f7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7074101e80 0x7f7074073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:07.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.708+0000 7f707c2f7700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7074073ba0 con 0x7f70741014f0 2023-11-16T16:49:07.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.708+0000 7f707c2f7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7074073d10 con 0x7f7074101e80 2023-11-16T16:49:07.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.708+0000 7f707c2f7700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70741a0100 con 0x7f70741002f0 2023-11-16T16:49:07.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.708+0000 7f7079892700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70741014f0 0x7f7074072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:07.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.708+0000 7f7079892700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70741014f0 0x7f7074072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40594/0 (socket says 172.21.15.44:40594) 2023-11-16T16:49:07.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.708+0000 7f7079892700 1 -- 172.21.15.44:0/4218779876 learned_addr learned my addr 172.21.15.44:0/4218779876 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:07.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.708+0000 7f707a093700 1 --2- 172.21.15.44:0/4218779876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70741002f0 0x7f7074074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:07.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.708+0000 7f707a894700 1 --2- 172.21.15.44:0/4218779876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7074101e80 0x7f7074073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:07.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.709+0000 7f7079892700 1 -- 172.21.15.44:0/4218779876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70741002f0 msgr2=0x7f7074074890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:07.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.709+0000 7f7079892700 1 --2- 172.21.15.44:0/4218779876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70741002f0 0x7f7074074890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:07.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.709+0000 7f7079892700 1 -- 172.21.15.44:0/4218779876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7074101e80 msgr2=0x7f7074073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:07.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.709+0000 7f7079892700 1 --2- 172.21.15.44:0/4218779876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7074101e80 0x7f7074073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:07.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.709+0000 7f7079892700 1 -- 172.21.15.44:0/4218779876 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7064009520 con 0x7f70741014f0 2023-11-16T16:49:07.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.709+0000 7f7079892700 1 --2- 172.21.15.44:0/4218779876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70741014f0 0x7f7074072ee0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f706400eb30 tx=0x7f706400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:07.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.709+0000 7f706b7fe700 1 -- 172.21.15.44:0/4218779876 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064009880 con 0x7f70741014f0 2023-11-16T16:49:07.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.709+0000 7f706b7fe700 1 -- 172.21.15.44:0/4218779876 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f70640044e0 con 0x7f70741014f0 2023-11-16T16:49:07.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.709+0000 7f706b7fe700 1 -- 172.21.15.44:0/4218779876 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064003d80 con 0x7f70741014f0 2023-11-16T16:49:07.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.709+0000 7f707c2f7700 1 -- 172.21.15.44:0/4218779876 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70700095f0 con 0x7f70741014f0 2023-11-16T16:49:07.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.710+0000 7f707c2f7700 1 -- 172.21.15.44:0/4218779876 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f707410bcb0 con 0x7f70741014f0 2023-11-16T16:49:07.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.711+0000 7f707c2f7700 1 -- 172.21.15.44:0/4218779876 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7074066ea0 con 0x7f70741014f0 2023-11-16T16:49:07.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.713+0000 7f706b7fe700 1 -- 172.21.15.44:0/4218779876 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7064003f20 con 0x7f70741014f0 2023-11-16T16:49:07.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.715+0000 7f706b7fe700 1 --2- 172.21.15.44:0/4218779876 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f70600a1260 0x7f70600a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:07.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.715+0000 7f706b7fe700 1 -- 172.21.15.44:0/4218779876 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f7064014070 con 0x7f70741014f0 2023-11-16T16:49:07.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.715+0000 7f707a093700 1 --2- 172.21.15.44:0/4218779876 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f70600a1260 0x7f70600a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:07.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.716+0000 7f707a093700 1 --2- 172.21.15.44:0/4218779876 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f70600a1260 0x7f70600a3710 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f707000b2e0 tx=0x7f7070005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:07.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.719+0000 7f706b7fe700 1 -- 172.21.15.44:0/4218779876 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7064096560 con 0x7f70741014f0 2023-11-16T16:49:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:07 smithi044 ceph-mon[128750]: pgmap v333: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:49:07.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:07 smithi176 ceph-mon[131630]: pgmap v333: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:49:07.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.935+0000 7f707c2f7700 1 -- 172.21.15.44:0/4218779876 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f70741a0c10 con 0x7f70600a1260 2023-11-16T16:49:07.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.937+0000 7f706b7fe700 1 -- 172.21.15.44:0/4218779876 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7f70741a0c10 con 0x7f70600a1260 2023-11-16T16:49:07.939 INFO:teuthology.orchestra.run.smithi044.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-11-16T16:49:07.939 INFO:teuthology.orchestra.run.smithi044.stdout:smithi044 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331001K400BGN 372G No 3m ago LVM detected, locked 2023-11-16T16:49:07.939 INFO:teuthology.orchestra.run.smithi044.stdout:smithi077 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN 372G No 3m ago LVM detected, locked 2023-11-16T16:49:07.939 INFO:teuthology.orchestra.run.smithi044.stdout:smithi176 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT62330099400BGN 372G No 2m ago LVM detected, locked 2023-11-16T16:49:07.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.939+0000 7f707c2f7700 1 -- 172.21.15.44:0/4218779876 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f70600a1260 msgr2=0x7f70600a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:07.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.939+0000 7f707c2f7700 1 --2- 172.21.15.44:0/4218779876 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f70600a1260 0x7f70600a3710 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f707000b2e0 tx=0x7f7070005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:07.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.939+0000 7f707c2f7700 1 -- 172.21.15.44:0/4218779876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70741014f0 msgr2=0x7f7074072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:07.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.939+0000 7f707c2f7700 1 --2- 172.21.15.44:0/4218779876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70741014f0 0x7f7074072ee0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f706400eb30 tx=0x7f706400ee40 comp rx=0 tx=0).stop 2023-11-16T16:49:07.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.939+0000 7f707c2f7700 1 -- 172.21.15.44:0/4218779876 shutdown_connections 2023-11-16T16:49:07.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.939+0000 7f707c2f7700 1 --2- 172.21.15.44:0/4218779876 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f70600a1260 0x7f70600a3710 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:07.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.939+0000 7f707c2f7700 1 --2- 172.21.15.44:0/4218779876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7074101e80 0x7f7074073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:07.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.939+0000 7f707c2f7700 1 --2- 172.21.15.44:0/4218779876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70741002f0 0x7f7074074890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:07.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.939+0000 7f707c2f7700 1 --2- 172.21.15.44:0/4218779876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70741014f0 0x7f7074072ee0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:07.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.939+0000 7f707c2f7700 1 -- 172.21.15.44:0/4218779876 >> 172.21.15.44:0/4218779876 conn(0x7f70740fb900 msgr2=0x7f70740fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:07.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.939+0000 7f707c2f7700 1 -- 172.21.15.44:0/4218779876 shutdown_connections 2023-11-16T16:49:07.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:07.940+0000 7f707c2f7700 1 -- 172.21.15.44:0/4218779876 wait complete. 2023-11-16T16:49:07.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:07 smithi077 ceph-mon[131204]: pgmap v333: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:49:08.394 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-11-16T16:49:08.403 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2023-11-16T16:49:08.403 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph fs dump' 2023-11-16T16:49:09.270 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:49:09.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:09 smithi077 ceph-mon[131204]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:09.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:09 smithi077 ceph-mon[131204]: pgmap v334: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:09.561 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:09 smithi044 ceph-mon[128750]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:09.561 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:09 smithi044 ceph-mon[128750]: pgmap v334: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:09.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:09 smithi176 ceph-mon[131630]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:09.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:09 smithi176 ceph-mon[131630]: pgmap v334: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:10.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.804+0000 7f56e5f94700 1 -- 172.21.15.44:0/3231733020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56e0101e80 msgr2=0x7f56e0108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:10.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.804+0000 7f56e5f94700 1 --2- 172.21.15.44:0/3231733020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56e0101e80 0x7f56e0108740 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f56d800b0a0 tx=0x7f56d800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:10.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.804+0000 7f56e5f94700 1 -- 172.21.15.44:0/3231733020 shutdown_connections 2023-11-16T16:49:10.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.804+0000 7f56e5f94700 1 --2- 172.21.15.44:0/3231733020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56e0101e80 0x7f56e0108740 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:10.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.804+0000 7f56e5f94700 1 --2- 172.21.15.44:0/3231733020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56e01002f0 0x7f56e0100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:10.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.804+0000 7f56e5f94700 1 --2- 172.21.15.44:0/3231733020 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e01014f0 0x7f56e0101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:10.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.804+0000 7f56e5f94700 1 -- 172.21.15.44:0/3231733020 >> 172.21.15.44:0/3231733020 conn(0x7f56e00fb900 msgr2=0x7f56e00fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:10.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.804+0000 7f56e5f94700 1 -- 172.21.15.44:0/3231733020 shutdown_connections 2023-11-16T16:49:10.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.805+0000 7f56e5f94700 1 -- 172.21.15.44:0/3231733020 wait complete. 2023-11-16T16:49:10.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.805+0000 7f56e5f94700 1 Processor -- start 2023-11-16T16:49:10.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.806+0000 7f56e5f94700 1 -- start start 2023-11-16T16:49:10.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.806+0000 7f56e5f94700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e01002f0 0x7f56e0198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:10.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.806+0000 7f56e5f94700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56e01014f0 0x7f56e0198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:10.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.806+0000 7f56e5f94700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56e0101e80 0x7f56e019f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:10.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.806+0000 7f56e5f94700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e019fed0 con 0x7f56e01014f0 2023-11-16T16:49:10.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.806+0000 7f56e5f94700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e01a0010 con 0x7f56e0101e80 2023-11-16T16:49:10.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.806+0000 7f56e5f94700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e01a01a0 con 0x7f56e01002f0 2023-11-16T16:49:10.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.806+0000 7f56deffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56e01014f0 0x7f56e0198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:10.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.806+0000 7f56dffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56e0101e80 0x7f56e019f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:10.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.806+0000 7f56df7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e01002f0 0x7f56e0198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:10.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.806+0000 7f56dffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56e0101e80 0x7f56e019f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:40692/0 (socket says 172.21.15.44:40692) 2023-11-16T16:49:10.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.806+0000 7f56df7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e01002f0 0x7f56e0198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:47544/0 (socket says 172.21.15.44:47544) 2023-11-16T16:49:10.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.806+0000 7f56dffff700 1 -- 172.21.15.44:0/2677811718 learned_addr learned my addr 172.21.15.44:0/2677811718 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:10.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.807+0000 7f56dffff700 1 -- 172.21.15.44:0/2677811718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e01002f0 msgr2=0x7f56e0198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:10.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.807+0000 7f56dffff700 1 --2- 172.21.15.44:0/2677811718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e01002f0 0x7f56e0198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:10.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.807+0000 7f56dffff700 1 -- 172.21.15.44:0/2677811718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56e01014f0 msgr2=0x7f56e0198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:10.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.807+0000 7f56dffff700 1 --2- 172.21.15.44:0/2677811718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56e01014f0 0x7f56e0198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:10.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.807+0000 7f56dffff700 1 -- 172.21.15.44:0/2677811718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d8009d00 con 0x7f56e0101e80 2023-11-16T16:49:10.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.807+0000 7f56df7fe700 1 --2- 172.21.15.44:0/2677811718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e01002f0 0x7f56e0198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:10.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.807+0000 7f56deffd700 1 --2- 172.21.15.44:0/2677811718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56e01014f0 0x7f56e0198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:10.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.807+0000 7f56dffff700 1 --2- 172.21.15.44:0/2677811718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56e0101e80 0x7f56e019f900 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f56d800b070 tx=0x7f56d8004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:10.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.807+0000 7f56dcff9700 1 -- 172.21.15.44:0/2677811718 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d800e040 con 0x7f56e0101e80 2023-11-16T16:49:10.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.807+0000 7f56e5f94700 1 -- 172.21.15.44:0/2677811718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e010bcb0 con 0x7f56e0101e80 2023-11-16T16:49:10.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.807+0000 7f56dcff9700 1 -- 172.21.15.44:0/2677811718 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56d8004dd0 con 0x7f56e0101e80 2023-11-16T16:49:10.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.807+0000 7f56dcff9700 1 -- 172.21.15.44:0/2677811718 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d801b6b0 con 0x7f56e0101e80 2023-11-16T16:49:10.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.807+0000 7f56e5f94700 1 -- 172.21.15.44:0/2677811718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e010bb20 con 0x7f56e0101e80 2023-11-16T16:49:10.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.808+0000 7f56e5f94700 1 -- 172.21.15.44:0/2677811718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e01a07e0 con 0x7f56e0101e80 2023-11-16T16:49:10.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.810+0000 7f56dcff9700 1 -- 172.21.15.44:0/2677811718 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f56d80041b0 con 0x7f56e0101e80 2023-11-16T16:49:10.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.812+0000 7f56dcff9700 1 --2- 172.21.15.44:0/2677811718 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f56c80a1260 0x7f56c80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:10.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.812+0000 7f56dcff9700 1 -- 172.21.15.44:0/2677811718 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f56d80cdf70 con 0x7f56e0101e80 2023-11-16T16:49:10.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.812+0000 7f56df7fe700 1 --2- 172.21.15.44:0/2677811718 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f56c80a1260 0x7f56c80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:10.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.812+0000 7f56df7fe700 1 --2- 172.21.15.44:0/2677811718 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f56c80a1260 0x7f56c80a3710 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f56e0101350 tx=0x7f56d0007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:10.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:10.814+0000 7f56dcff9700 1 -- 172.21.15.44:0/2677811718 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f56d80971f0 con 0x7f56e0101e80 2023-11-16T16:49:11.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:11.065+0000 7f56e5f94700 1 -- 172.21.15.44:0/2677811718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f56e01a0d90 con 0x7f56e0101e80 2023-11-16T16:49:11.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:11.065+0000 7f56dcff9700 1 -- 172.21.15.44:0/2677811718 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+2861 (secure 0 0 0) 0x7f56d80048b0 con 0x7f56e0101e80 2023-11-16T16:49:11.067 INFO:teuthology.orchestra.run.smithi044.stdout:e21 2023-11-16T16:49:11.067 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T16:49:11.068 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T16:49:11.068 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2023-11-16T16:49:11.068 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:11.068 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2023-11-16T16:49:11.068 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2023-11-16T16:49:11.068 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 21 2023-11-16T16:49:11.068 INFO:teuthology.orchestra.run.smithi044.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-11-16T16:49:11.068 INFO:teuthology.orchestra.run.smithi044.stdout:created 2023-11-16T16:47:18.150861+0000 2023-11-16T16:49:11.068 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2023-11-16T16:47:50.656307+0000 2023-11-16T16:49:11.068 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2023-11-16T16:49:11.068 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2023-11-16T16:49:11.068 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 300 2023-11-16T16:49:11.068 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2023-11-16T16:49:11.069 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2023-11-16T16:49:11.069 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2023-11-16T16:49:11.069 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2023-11-16T16:49:11.069 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2023-11-16T16:49:11.069 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2023-11-16T16:49:11.069 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T16:49:11.069 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2023-11-16T16:49:11.069 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2023-11-16T16:49:11.069 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=24551} 2023-11-16T16:49:11.069 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2023-11-16T16:49:11.069 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2023-11-16T16:49:11.069 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2023-11-16T16:49:11.070 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3,4] 2023-11-16T16:49:11.070 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2023-11-16T16:49:11.070 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2023-11-16T16:49:11.070 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2023-11-16T16:49:11.070 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2023-11-16T16:49:11.070 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 0 2023-11-16T16:49:11.070 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.b{0:24551} state up:active seq 3 addr [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:11.070 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.i{0:24538} state up:standby-replay seq 1 addr [v2:172.21.15.176:6838/2828920421,v1:172.21.15.176:6839/2828920421] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:11.070 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:11.070 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:11.070 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2023-11-16T16:49:11.070 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:11.070 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.j{-1:14460} state up:standby seq 1 addr [v2:172.21.15.44:6834/3970258683,v1:172.21.15.44:6835/3970258683] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:11.071 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.d{-1:14472} state up:standby seq 1 addr [v2:172.21.15.44:6836/159805710,v1:172.21.15.44:6837/159805710] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:11.071 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.g{-1:14523} state up:standby seq 1 addr [v2:172.21.15.44:6838/1792053180,v1:172.21.15.44:6839/1792053180] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:11.071 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.a{-1:14535} state up:standby seq 1 addr [v2:172.21.15.44:6840/2080079420,v1:172.21.15.44:6841/2080079420] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:11.071 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.k{-1:24440} state up:standby seq 1 addr [v2:172.21.15.77:6832/2897858995,v1:172.21.15.77:6833/2897858995] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:11.071 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.l{-1:24442} state up:standby seq 1 addr [v2:172.21.15.176:6832/1226611,v1:172.21.15.176:6833/1226611] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:11.071 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.e{-1:24476} state up:standby seq 1 addr [v2:172.21.15.77:6834/1355935860,v1:172.21.15.77:6835/1355935860] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:11.071 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.f{-1:24505} state up:standby seq 1 addr [v2:172.21.15.176:6834/1665062606,v1:172.21.15.176:6835/1665062606] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:11.071 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.h{-1:24506} state up:standby seq 1 addr [v2:172.21.15.77:6836/440017436,v1:172.21.15.77:6837/440017436] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:11.071 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.c{-1:24517} state up:standby seq 1 addr [v2:172.21.15.176:6836/3753678422,v1:172.21.15.176:6837/3753678422] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:11.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:11.068+0000 7f56e5f94700 1 -- 172.21.15.44:0/2677811718 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f56c80a1260 msgr2=0x7f56c80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:11.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:11.068+0000 7f56e5f94700 1 --2- 172.21.15.44:0/2677811718 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f56c80a1260 0x7f56c80a3710 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f56e0101350 tx=0x7f56d0007400 comp rx=0 tx=0).stop 2023-11-16T16:49:11.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:11.068+0000 7f56e5f94700 1 -- 172.21.15.44:0/2677811718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56e0101e80 msgr2=0x7f56e019f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:11.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:11.068+0000 7f56e5f94700 1 --2- 172.21.15.44:0/2677811718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56e0101e80 0x7f56e019f900 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f56d800b070 tx=0x7f56d8004a50 comp rx=0 tx=0).stop 2023-11-16T16:49:11.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:11.068+0000 7f56e5f94700 1 -- 172.21.15.44:0/2677811718 shutdown_connections 2023-11-16T16:49:11.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:11.068+0000 7f56e5f94700 1 --2- 172.21.15.44:0/2677811718 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f56c80a1260 0x7f56c80a3710 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:11.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:11.068+0000 7f56e5f94700 1 --2- 172.21.15.44:0/2677811718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56e0101e80 0x7f56e019f900 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:11.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:11.068+0000 7f56e5f94700 1 --2- 172.21.15.44:0/2677811718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e01002f0 0x7f56e0198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:11.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:11.068+0000 7f56e5f94700 1 --2- 172.21.15.44:0/2677811718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56e01014f0 0x7f56e0198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:11.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:11.068+0000 7f56e5f94700 1 -- 172.21.15.44:0/2677811718 >> 172.21.15.44:0/2677811718 conn(0x7f56e00fb900 msgr2=0x7f56e0106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:11.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:11.068+0000 7f56e5f94700 1 -- 172.21.15.44:0/2677811718 shutdown_connections 2023-11-16T16:49:11.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:11.068+0000 7f56e5f94700 1 -- 172.21.15.44:0/2677811718 wait complete. 2023-11-16T16:49:11.073 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:49:11.469 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph osd dump' 2023-11-16T16:49:11.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:11 smithi044 ceph-mon[128750]: pgmap v335: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:49:11.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:11 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2677811718' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-11-16T16:49:11.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:11 smithi176 ceph-mon[131630]: pgmap v335: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:49:11.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:11 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2677811718' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-11-16T16:49:11.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:11 smithi077 ceph-mon[131204]: pgmap v335: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:49:11.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:11 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2677811718' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-11-16T16:49:12.336 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:49:13.865 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.863+0000 7f01d711e700 1 -- 172.21.15.44:0/3327872038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01d01014f0 msgr2=0x7f01d0101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:13.865 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.863+0000 7f01d711e700 1 --2- 172.21.15.44:0/3327872038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01d01014f0 0x7f01d0101940 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f01bc009990 tx=0x7f01bc009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:13.865 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.863+0000 7f01d711e700 1 -- 172.21.15.44:0/3327872038 shutdown_connections 2023-11-16T16:49:13.865 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.863+0000 7f01d711e700 1 --2- 172.21.15.44:0/3327872038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01d0101e80 0x7f01d0108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:13.865 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.863+0000 7f01d711e700 1 --2- 172.21.15.44:0/3327872038 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01d01002f0 0x7f01d0100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:13.865 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.863+0000 7f01d711e700 1 --2- 172.21.15.44:0/3327872038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01d01014f0 0x7f01d0101940 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:13.865 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.863+0000 7f01d711e700 1 -- 172.21.15.44:0/3327872038 >> 172.21.15.44:0/3327872038 conn(0x7f01d00fb900 msgr2=0x7f01d00fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:13.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.863+0000 7f01d711e700 1 -- 172.21.15.44:0/3327872038 shutdown_connections 2023-11-16T16:49:13.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.863+0000 7f01d711e700 1 -- 172.21.15.44:0/3327872038 wait complete. 2023-11-16T16:49:13.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.864+0000 7f01d711e700 1 Processor -- start 2023-11-16T16:49:13.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.865+0000 7f01d711e700 1 -- start start 2023-11-16T16:49:13.867 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.865+0000 7f01d711e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01d01002f0 0x7f01d0074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:13.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.865+0000 7f01d711e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01d01014f0 0x7f01d0072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:13.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.865+0000 7f01d711e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01d0101e80 0x7f01d0073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:13.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.865+0000 7f01d711e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0073ba0 con 0x7f01d01014f0 2023-11-16T16:49:13.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.865+0000 7f01d711e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0073d10 con 0x7f01d01002f0 2023-11-16T16:49:13.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.865+0000 7f01d711e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d01a00d0 con 0x7f01d0101e80 2023-11-16T16:49:13.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.865+0000 7f01cffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01d01014f0 0x7f01d0072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:13.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.865+0000 7f01cffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01d01014f0 0x7f01d0072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40626/0 (socket says 172.21.15.44:40626) 2023-11-16T16:49:13.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.865+0000 7f01cffff700 1 -- 172.21.15.44:0/349507324 learned_addr learned my addr 172.21.15.44:0/349507324 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:13.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.865+0000 7f01d4eba700 1 --2- 172.21.15.44:0/349507324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01d01002f0 0x7f01d0074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:13.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.866+0000 7f01d56bb700 1 --2- 172.21.15.44:0/349507324 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01d0101e80 0x7f01d0073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:13.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.866+0000 7f01cffff700 1 -- 172.21.15.44:0/349507324 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01d0101e80 msgr2=0x7f01d0073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:13.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.866+0000 7f01cffff700 1 --2- 172.21.15.44:0/349507324 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01d0101e80 0x7f01d0073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:13.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.866+0000 7f01cffff700 1 -- 172.21.15.44:0/349507324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01d01002f0 msgr2=0x7f01d0074890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:13.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.866+0000 7f01cffff700 1 --2- 172.21.15.44:0/349507324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01d01002f0 0x7f01d0074890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:13.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.866+0000 7f01cffff700 1 -- 172.21.15.44:0/349507324 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01bc0095f0 con 0x7f01d01014f0 2023-11-16T16:49:13.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.866+0000 7f01cffff700 1 --2- 172.21.15.44:0/349507324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01d01014f0 0x7f01d0072ee0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00b680 tx=0x7f01bc005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:13.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:13 smithi044 ceph-mon[128750]: pgmap v336: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:13.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.866+0000 7f01cdffb700 1 -- 172.21.15.44:0/349507324 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01bc01d070 con 0x7f01d01014f0 2023-11-16T16:49:13.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.866+0000 7f01cdffb700 1 -- 172.21.15.44:0/349507324 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01bc022470 con 0x7f01d01014f0 2023-11-16T16:49:13.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.866+0000 7f01cdffb700 1 -- 172.21.15.44:0/349507324 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01bc017650 con 0x7f01d01014f0 2023-11-16T16:49:13.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.866+0000 7f01d711e700 1 -- 172.21.15.44:0/349507324 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d010bc90 con 0x7f01d01014f0 2023-11-16T16:49:13.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.867+0000 7f01d711e700 1 -- 172.21.15.44:0/349507324 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d010bb00 con 0x7f01d01014f0 2023-11-16T16:49:13.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.868+0000 7f01d711e700 1 -- 172.21.15.44:0/349507324 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d01078f0 con 0x7f01d01014f0 2023-11-16T16:49:13.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.869+0000 7f01cdffb700 1 -- 172.21.15.44:0/349507324 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f01bc0225e0 con 0x7f01d01014f0 2023-11-16T16:49:13.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.871+0000 7f01cdffb700 1 --2- 172.21.15.44:0/349507324 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f01c00a11d0 0x7f01c00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:13.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.871+0000 7f01cdffb700 1 -- 172.21.15.44:0/349507324 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f01bc0ce130 con 0x7f01d01014f0 2023-11-16T16:49:13.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.871+0000 7f01d4eba700 1 --2- 172.21.15.44:0/349507324 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f01c00a11d0 0x7f01c00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:13.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.872+0000 7f01d4eba700 1 --2- 172.21.15.44:0/349507324 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f01c00a11d0 0x7f01c00a3680 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f01d0101350 tx=0x7f01c4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:13.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:13.875+0000 7f01cdffb700 1 -- 172.21.15.44:0/349507324 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f01bc097510 con 0x7f01d01014f0 2023-11-16T16:49:13.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:13 smithi176 ceph-mon[131630]: pgmap v336: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:13.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:13 smithi077 ceph-mon[131204]: pgmap v336: 129 pgs: 129 active+clean; 637 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:14.082 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.081+0000 7f01d711e700 1 -- 172.21.15.44:0/349507324 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f01d002cfe0 con 0x7f01d01014f0 2023-11-16T16:49:14.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.082+0000 7f01cdffb700 1 -- 172.21.15.44:0/349507324 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v91) v1 ==== 56+0+5718 (secure 0 0 0) 0x7f01bc09ab30 con 0x7f01d01014f0 2023-11-16T16:49:14.084 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 91 2023-11-16T16:49:14.084 INFO:teuthology.orchestra.run.smithi044.stdout:fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:49:14.084 INFO:teuthology.orchestra.run.smithi044.stdout:created 2023-11-16T16:39:41.796646+0000 2023-11-16T16:49:14.084 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2023-11-16T16:48:44.613706+0000 2023-11-16T16:49:14.084 INFO:teuthology.orchestra.run.smithi044.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-11-16T16:49:14.084 INFO:teuthology.orchestra.run.smithi044.stdout:crush_version 27 2023-11-16T16:49:14.084 INFO:teuthology.orchestra.run.smithi044.stdout:full_ratio 0.95 2023-11-16T16:49:14.084 INFO:teuthology.orchestra.run.smithi044.stdout:backfillfull_ratio 0.9 2023-11-16T16:49:14.084 INFO:teuthology.orchestra.run.smithi044.stdout:nearfull_ratio 0.85 2023-11-16T16:49:14.084 INFO:teuthology.orchestra.run.smithi044.stdout:require_min_compat_client luminous 2023-11-16T16:49:14.084 INFO:teuthology.orchestra.run.smithi044.stdout:min_compat_client luminous 2023-11-16T16:49:14.085 INFO:teuthology.orchestra.run.smithi044.stdout:require_osd_release reef 2023-11-16T16:49:14.085 INFO:teuthology.orchestra.run.smithi044.stdout:stretch_mode_enabled false 2023-11-16T16:49:14.085 INFO:teuthology.orchestra.run.smithi044.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-11-16T16:49:14.085 INFO:teuthology.orchestra.run.smithi044.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2023-11-16T16:49:14.085 INFO:teuthology.orchestra.run.smithi044.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 83 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2023-11-16T16:49:14.085 INFO:teuthology.orchestra.run.smithi044.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 87 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2023-11-16T16:49:14.085 INFO:teuthology.orchestra.run.smithi044.stdout:max_osd 12 2023-11-16T16:49:14.085 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 up in weight 1 up_from 8 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.44:6802/2561264157,v1:172.21.15.44:6803/2561264157] [v2:172.21.15.44:6804/2561264157,v1:172.21.15.44:6805/2561264157] exists,up a062dd5b-f36d-4686-86b7-b93b81bf317d 2023-11-16T16:49:14.085 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 up in weight 1 up_from 13 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6800/1505198975,v1:172.21.15.77:6801/1505198975] [v2:172.21.15.77:6802/1505198975,v1:172.21.15.77:6803/1505198975] exists,up 48ba3e8d-abd4-4253-a4aa-bc3d123a8bec 2023-11-16T16:49:14.085 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.176:6800/1877349342,v1:172.21.15.176:6801/1877349342] [v2:172.21.15.176:6802/1877349342,v1:172.21.15.176:6803/1877349342] exists,up 98830193-aa89-42b5-a96b-ea72147286c1 2023-11-16T16:49:14.085 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 up in weight 1 up_from 25 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.44:6810/1041267584,v1:172.21.15.44:6811/1041267584] [v2:172.21.15.44:6812/1041267584,v1:172.21.15.44:6813/1041267584] exists,up ba9c2299-3850-4b08-b361-6ed673629019 2023-11-16T16:49:14.085 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 up in weight 1 up_from 33 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6808/2688407497,v1:172.21.15.77:6809/2688407497] [v2:172.21.15.77:6810/2688407497,v1:172.21.15.77:6811/2688407497] exists,up 6a7ee641-fb5c-4a14-83b5-07c5b5842cfc 2023-11-16T16:49:14.085 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 up in weight 1 up_from 38 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.176:6808/609622683,v1:172.21.15.176:6809/609622683] [v2:172.21.15.176:6810/609622683,v1:172.21.15.176:6811/609622683] exists,up d16999a0-6bd4-4447-b88e-3271c8ab904a 2023-11-16T16:49:14.085 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 up in weight 1 up_from 46 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.44:6818/2634951849,v1:172.21.15.44:6819/2634951849] [v2:172.21.15.44:6820/2634951849,v1:172.21.15.44:6821/2634951849] exists,up a31ca92c-d241-4731-8e79-a2aa0fda2c42 2023-11-16T16:49:14.086 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6816/1651452313,v1:172.21.15.77:6817/1651452313] [v2:172.21.15.77:6818/1651452313,v1:172.21.15.77:6819/1651452313] exists,up bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969 2023-11-16T16:49:14.086 INFO:teuthology.orchestra.run.smithi044.stdout:osd.8 up in weight 1 up_from 58 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.176:6816/832350338,v1:172.21.15.176:6817/832350338] [v2:172.21.15.176:6818/832350338,v1:172.21.15.176:6819/832350338] exists,up ba086b54-d6b6-4563-ade8-59c99f2ca473 2023-11-16T16:49:14.086 INFO:teuthology.orchestra.run.smithi044.stdout:osd.9 up in weight 1 up_from 66 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.44:6826/1897410398,v1:172.21.15.44:6827/1897410398] [v2:172.21.15.44:6828/1897410398,v1:172.21.15.44:6829/1897410398] exists,up 88e6c45c-9bac-49a2-b71d-2a6932bae792 2023-11-16T16:49:14.086 INFO:teuthology.orchestra.run.smithi044.stdout:osd.10 up in weight 1 up_from 71 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6824/3918521093,v1:172.21.15.77:6825/3918521093] [v2:172.21.15.77:6826/3918521093,v1:172.21.15.77:6827/3918521093] exists,up a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41 2023-11-16T16:49:14.086 INFO:teuthology.orchestra.run.smithi044.stdout:osd.11 up in weight 1 up_from 78 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.176:6824/1098488101,v1:172.21.15.176:6825/1098488101] [v2:172.21.15.176:6826/1098488101,v1:172.21.15.176:6827/1098488101] exists,up ca5bd483-91ad-4385-922d-129f21cf955a 2023-11-16T16:49:14.086 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.0 [7,11] 2023-11-16T16:49:14.086 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.b [2,6] 2023-11-16T16:49:14.086 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.f [8,11] 2023-11-16T16:49:14.086 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.16 [2,4] 2023-11-16T16:49:14.086 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.1a [8,1] 2023-11-16T16:49:14.086 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.1d [2,4] 2023-11-16T16:49:14.086 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.c [5,6] 2023-11-16T16:49:14.087 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.13 [4,9] 2023-11-16T16:49:14.087 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.17 [4,8] 2023-11-16T16:49:14.087 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.2c [4,9] 2023-11-16T16:49:14.087 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.38 [3,9] 2023-11-16T16:49:14.087 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 4.6 [5,8] 2023-11-16T16:49:14.087 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 4.f [9,10] 2023-11-16T16:49:14.087 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 4.14 [9,8] 2023-11-16T16:49:14.087 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 4.1c [9,8] 2023-11-16T16:49:14.087 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/1018560948 expires 2023-11-17T16:40:42.449496+0000 2023-11-16T16:49:14.087 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/1599940692 expires 2023-11-17T16:40:42.449496+0000 2023-11-16T16:49:14.087 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/3409868354 expires 2023-11-17T16:40:42.449496+0000 2023-11-16T16:49:14.087 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6801/4234382393 expires 2023-11-17T16:40:42.449496+0000 2023-11-16T16:49:14.087 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6801/1852059921 expires 2023-11-17T16:40:15.683996+0000 2023-11-16T16:49:14.087 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/3135397911 expires 2023-11-17T16:40:15.683996+0000 2023-11-16T16:49:14.088 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6800/1852059921 expires 2023-11-17T16:40:15.683996+0000 2023-11-16T16:49:14.088 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/3599504154 expires 2023-11-17T16:40:15.683996+0000 2023-11-16T16:49:14.088 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/264068429 expires 2023-11-17T16:40:15.683996+0000 2023-11-16T16:49:14.088 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6800/4234382393 expires 2023-11-17T16:40:42.449496+0000 2023-11-16T16:49:14.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.084+0000 7f01d711e700 1 -- 172.21.15.44:0/349507324 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f01c00a11d0 msgr2=0x7f01c00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:14.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.084+0000 7f01d711e700 1 --2- 172.21.15.44:0/349507324 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f01c00a11d0 0x7f01c00a3680 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f01d0101350 tx=0x7f01c4009260 comp rx=0 tx=0).stop 2023-11-16T16:49:14.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.084+0000 7f01d711e700 1 -- 172.21.15.44:0/349507324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01d01014f0 msgr2=0x7f01d0072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:14.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.084+0000 7f01d711e700 1 --2- 172.21.15.44:0/349507324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01d01014f0 0x7f01d0072ee0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00b680 tx=0x7f01bc005740 comp rx=0 tx=0).stop 2023-11-16T16:49:14.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.084+0000 7f01d711e700 1 -- 172.21.15.44:0/349507324 shutdown_connections 2023-11-16T16:49:14.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.084+0000 7f01d711e700 1 --2- 172.21.15.44:0/349507324 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f01c00a11d0 0x7f01c00a3680 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.084+0000 7f01d711e700 1 --2- 172.21.15.44:0/349507324 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01d0101e80 0x7f01d0073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.084+0000 7f01d711e700 1 --2- 172.21.15.44:0/349507324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01d01002f0 0x7f01d0074890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.084+0000 7f01d711e700 1 --2- 172.21.15.44:0/349507324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01d01014f0 0x7f01d0072ee0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.084+0000 7f01d711e700 1 -- 172.21.15.44:0/349507324 >> 172.21.15.44:0/349507324 conn(0x7f01d00fb900 msgr2=0x7f01d0104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:14.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.084+0000 7f01d711e700 1 -- 172.21.15.44:0/349507324 shutdown_connections 2023-11-16T16:49:14.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.084+0000 7f01d711e700 1 -- 172.21.15.44:0/349507324 wait complete. 2023-11-16T16:49:14.536 INFO:teuthology.run_tasks:Running task fs.ready... 2023-11-16T16:49:14.547 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:49:14.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.761+0000 7f3e3e5ff700 1 -- 172.21.15.44:0/2090919044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e3810c860 msgr2=0x7f3e3810ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:14.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.761+0000 7f3e3e5ff700 1 --2- 172.21.15.44:0/2090919044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e3810c860 0x7f3e3810ec40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3e3000b0a0 tx=0x7f3e3000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.761+0000 7f3e3e5ff700 1 -- 172.21.15.44:0/2090919044 shutdown_connections 2023-11-16T16:49:14.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.761+0000 7f3e3e5ff700 1 --2- 172.21.15.44:0/2090919044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e3810c860 0x7f3e3810ec40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.761+0000 7f3e3e5ff700 1 --2- 172.21.15.44:0/2090919044 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e38074b30 0x7f3e38072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.761+0000 7f3e3e5ff700 1 --2- 172.21.15.44:0/2090919044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e38073550 0x7f3e380739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.764 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.761+0000 7f3e3e5ff700 1 -- 172.21.15.44:0/2090919044 >> 172.21.15.44:0/2090919044 conn(0x7f3e380fb870 msgr2=0x7f3e380fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:14.764 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.761+0000 7f3e3e5ff700 1 -- 172.21.15.44:0/2090919044 shutdown_connections 2023-11-16T16:49:14.764 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.761+0000 7f3e3e5ff700 1 -- 172.21.15.44:0/2090919044 wait complete. 2023-11-16T16:49:14.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.762+0000 7f3e3e5ff700 1 Processor -- start 2023-11-16T16:49:14.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.762+0000 7f3e3e5ff700 1 -- start start 2023-11-16T16:49:14.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.762+0000 7f3e3e5ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e38073550 0x7f3e3819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:14.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.762+0000 7f3e3e5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e38074b30 0x7f3e3819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:14.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.762+0000 7f3e3e5ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e3810c860 0x7f3e381a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:14.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.762+0000 7f3e3e5ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e381a4220 con 0x7f3e38073550 2023-11-16T16:49:14.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.762+0000 7f3e3e5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e381a4360 con 0x7f3e38074b30 2023-11-16T16:49:14.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.762+0000 7f3e3e5ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e381a44f0 con 0x7f3e3810c860 2023-11-16T16:49:14.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.763+0000 7f3e37fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e38073550 0x7f3e3819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:14.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.763+0000 7f3e377fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e38074b30 0x7f3e3819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:14.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.763+0000 7f3e37fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e38073550 0x7f3e3819c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40634/0 (socket says 172.21.15.44:40634) 2023-11-16T16:49:14.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.763+0000 7f3e37fff700 1 -- 172.21.15.44:0/1317894209 learned_addr learned my addr 172.21.15.44:0/1317894209 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:14.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.763+0000 7f3e3cb9c700 1 --2- 172.21.15.44:0/1317894209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e3810c860 0x7f3e381a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:14.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.763+0000 7f3e377fe700 1 -- 172.21.15.44:0/1317894209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e3810c860 msgr2=0x7f3e381a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:14.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.763+0000 7f3e377fe700 1 --2- 172.21.15.44:0/1317894209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e3810c860 0x7f3e381a3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.763+0000 7f3e377fe700 1 -- 172.21.15.44:0/1317894209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e38073550 msgr2=0x7f3e3819c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:14.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.763+0000 7f3e377fe700 1 --2- 172.21.15.44:0/1317894209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e38073550 0x7f3e3819c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.763+0000 7f3e377fe700 1 -- 172.21.15.44:0/1317894209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e30009d00 con 0x7f3e38074b30 2023-11-16T16:49:14.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.763+0000 7f3e377fe700 1 --2- 172.21.15.44:0/1317894209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e38074b30 0x7f3e3819cbe0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00e9c0 tx=0x7f3e2c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:14.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.764+0000 7f3e357fa700 1 -- 172.21.15.44:0/1317894209 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c004220 con 0x7f3e38074b30 2023-11-16T16:49:14.769 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.764+0000 7f3e3e5ff700 1 -- 172.21.15.44:0/1317894209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e38112270 con 0x7f3e38074b30 2023-11-16T16:49:14.769 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.764+0000 7f3e357fa700 1 -- 172.21.15.44:0/1317894209 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e2c004380 con 0x7f3e38074b30 2023-11-16T16:49:14.769 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.764+0000 7f3e357fa700 1 -- 172.21.15.44:0/1317894209 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c003be0 con 0x7f3e38074b30 2023-11-16T16:49:14.769 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.764+0000 7f3e3e5ff700 1 -- 172.21.15.44:0/1317894209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e381120e0 con 0x7f3e38074b30 2023-11-16T16:49:14.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.765+0000 7f3e3e5ff700 1 -- 172.21.15.44:0/1317894209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e381023b0 con 0x7f3e38074b30 2023-11-16T16:49:14.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.767+0000 7f3e357fa700 1 -- 172.21.15.44:0/1317894209 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3e2c003d40 con 0x7f3e38074b30 2023-11-16T16:49:14.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.768+0000 7f3e357fa700 1 --2- 172.21.15.44:0/1317894209 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3e200a1150 0x7f3e200a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:14.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.768+0000 7f3e357fa700 1 -- 172.21.15.44:0/1317894209 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f3e2c014070 con 0x7f3e38074b30 2023-11-16T16:49:14.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.768+0000 7f3e37fff700 1 --2- 172.21.15.44:0/1317894209 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3e200a1150 0x7f3e200a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:14.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.769+0000 7f3e37fff700 1 --2- 172.21.15.44:0/1317894209 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3e200a1150 0x7f3e200a3600 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3e28005950 tx=0x7f3e28009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:14.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.770+0000 7f3e357fa700 1 -- 172.21.15.44:0/1317894209 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3e2c0968d0 con 0x7f3e38074b30 2023-11-16T16:49:14.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:14 smithi044 ceph-mon[128750]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-11-16T16:49:14.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:14 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/349507324' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-11-16T16:49:14.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:14 smithi176 ceph-mon[131630]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-11-16T16:49:14.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:14 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/349507324' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-11-16T16:49:14.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:14 smithi077 ceph-mon[131204]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-11-16T16:49:14.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:14 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/349507324' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-11-16T16:49:14.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.979+0000 7f3e3e5ff700 1 -- 172.21.15.44:0/1317894209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e3802cfe0 con 0x7f3e38074b30 2023-11-16T16:49:14.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.981+0000 7f3e357fa700 1 -- 172.21.15.44:0/1317894209 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24685 (secure 0 0 0) 0x7f3e2c01f340 con 0x7f3e38074b30 2023-11-16T16:49:14.983 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:14.984 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:48:44.613706+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:49:14.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.983+0000 7f3e3e5ff700 1 -- 172.21.15.44:0/1317894209 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3e200a1150 msgr2=0x7f3e200a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:14.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.983+0000 7f3e3e5ff700 1 --2- 172.21.15.44:0/1317894209 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3e200a1150 0x7f3e200a3600 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3e28005950 tx=0x7f3e28009260 comp rx=0 tx=0).stop 2023-11-16T16:49:14.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.984+0000 7f3e3e5ff700 1 -- 172.21.15.44:0/1317894209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e38074b30 msgr2=0x7f3e3819cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:14.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.984+0000 7f3e3e5ff700 1 --2- 172.21.15.44:0/1317894209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e38074b30 0x7f3e3819cbe0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00e9c0 tx=0x7f3e2c00ecd0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.984+0000 7f3e3e5ff700 1 -- 172.21.15.44:0/1317894209 shutdown_connections 2023-11-16T16:49:14.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.984+0000 7f3e3e5ff700 1 --2- 172.21.15.44:0/1317894209 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f3e200a1150 0x7f3e200a3600 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.984+0000 7f3e3e5ff700 1 --2- 172.21.15.44:0/1317894209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e3810c860 0x7f3e381a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.984+0000 7f3e3e5ff700 1 --2- 172.21.15.44:0/1317894209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e38073550 0x7f3e3819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.984+0000 7f3e3e5ff700 1 --2- 172.21.15.44:0/1317894209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e38074b30 0x7f3e3819cbe0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.984+0000 7f3e3e5ff700 1 -- 172.21.15.44:0/1317894209 >> 172.21.15.44:0/1317894209 conn(0x7f3e380fb870 msgr2=0x7f3e380fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:14.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.984+0000 7f3e3e5ff700 1 -- 172.21.15.44:0/1317894209 shutdown_connections 2023-11-16T16:49:14.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:14.984+0000 7f3e3e5ff700 1 -- 172.21.15.44:0/1317894209 wait complete. 2023-11-16T16:49:15.005 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:03.059622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:16.211826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:17.601591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:21.136084+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T16:49:15.005 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:49:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.210+0000 7f63bb59e700 1 -- 172.21.15.44:0/3859023175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63bc101cc0 msgr2=0x7f63bc108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.210+0000 7f63bb59e700 1 --2- 172.21.15.44:0/3859023175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63bc101cc0 0x7f63bc108560 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f63b400b0a0 tx=0x7f63b400b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.210+0000 7f63bb59e700 1 -- 172.21.15.44:0/3859023175 shutdown_connections 2023-11-16T16:49:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.210+0000 7f63bb59e700 1 --2- 172.21.15.44:0/3859023175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63bc101cc0 0x7f63bc108560 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.210+0000 7f63bb59e700 1 --2- 172.21.15.44:0/3859023175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63bc100080 0x7f63bc100490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.210+0000 7f63bb59e700 1 --2- 172.21.15.44:0/3859023175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63bc101310 0x7f63bc101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.210+0000 7f63bb59e700 1 -- 172.21.15.44:0/3859023175 >> 172.21.15.44:0/3859023175 conn(0x7f63bc0fb6a0 msgr2=0x7f63bc0fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:15.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.210+0000 7f63bb59e700 1 -- 172.21.15.44:0/3859023175 shutdown_connections 2023-11-16T16:49:15.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.210+0000 7f63bb59e700 1 -- 172.21.15.44:0/3859023175 wait complete. 2023-11-16T16:49:15.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.212+0000 7f63bb59e700 1 Processor -- start 2023-11-16T16:49:15.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.212+0000 7f63bb59e700 1 -- start start 2023-11-16T16:49:15.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63bb59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63bc100080 0x7f63bc1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:15.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63bb59e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63bc101310 0x7f63bc198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:15.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63bb59e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63bc101cc0 0x7f63bc19f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:15.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63bb59e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc19fc50 con 0x7f63bc101cc0 2023-11-16T16:49:15.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63bb59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc19fd90 con 0x7f63bc100080 2023-11-16T16:49:15.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63bb59e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc19ff20 con 0x7f63bc101310 2023-11-16T16:49:15.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63bad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63bc101cc0 0x7f63bc19f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:15.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63bad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63bc101cc0 0x7f63bc19f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51432/0 (socket says 172.21.15.44:51432) 2023-11-16T16:49:15.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63bad9d700 1 -- 172.21.15.44:0/3285676291 learned_addr learned my addr 172.21.15.44:0/3285676291 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:15.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63b9d9b700 1 --2- 172.21.15.44:0/3285676291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63bc101310 0x7f63bc198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:15.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63ba59c700 1 --2- 172.21.15.44:0/3285676291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63bc100080 0x7f63bc1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:15.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63bad9d700 1 -- 172.21.15.44:0/3285676291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63bc101310 msgr2=0x7f63bc198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:15.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63bad9d700 1 --2- 172.21.15.44:0/3285676291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63bc101310 0x7f63bc198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63bad9d700 1 -- 172.21.15.44:0/3285676291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63bc100080 msgr2=0x7f63bc1980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:15.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63bad9d700 1 --2- 172.21.15.44:0/3285676291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63bc100080 0x7f63bc1980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63bad9d700 1 -- 172.21.15.44:0/3285676291 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b4009d00 con 0x7f63bc101cc0 2023-11-16T16:49:15.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.213+0000 7f63ba59c700 1 --2- 172.21.15.44:0/3285676291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63bc100080 0x7f63bc1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:49:15.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.214+0000 7f63bad9d700 1 --2- 172.21.15.44:0/3285676291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63bc101cc0 0x7f63bc19f680 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f63b4009fd0 tx=0x7f63b400bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:15.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.214+0000 7f63ab7fe700 1 -- 172.21.15.44:0/3285676291 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b400e040 con 0x7f63bc101cc0 2023-11-16T16:49:15.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.214+0000 7f63ab7fe700 1 -- 172.21.15.44:0/3285676291 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63b40094a0 con 0x7f63bc101cc0 2023-11-16T16:49:15.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.214+0000 7f63bb59e700 1 -- 172.21.15.44:0/3285676291 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63bc10ba80 con 0x7f63bc101cc0 2023-11-16T16:49:15.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.214+0000 7f63ab7fe700 1 -- 172.21.15.44:0/3285676291 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b401b6a0 con 0x7f63bc101cc0 2023-11-16T16:49:15.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.214+0000 7f63bb59e700 1 -- 172.21.15.44:0/3285676291 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63bc10b8f0 con 0x7f63bc101cc0 2023-11-16T16:49:15.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.215+0000 7f63bb59e700 1 -- 172.21.15.44:0/3285676291 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63bc067980 con 0x7f63bc101cc0 2023-11-16T16:49:15.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.217+0000 7f63ab7fe700 1 -- 172.21.15.44:0/3285676291 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f63b401d470 con 0x7f63bc101cc0 2023-11-16T16:49:15.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.218+0000 7f63ab7fe700 1 --2- 172.21.15.44:0/3285676291 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f63a40a1150 0x7f63a40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:15.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.218+0000 7f63ab7fe700 1 -- 172.21.15.44:0/3285676291 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f63b40ce0b0 con 0x7f63bc101cc0 2023-11-16T16:49:15.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.219+0000 7f63ba59c700 1 --2- 172.21.15.44:0/3285676291 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f63a40a1150 0x7f63a40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:15.221 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.219+0000 7f63ba59c700 1 --2- 172.21.15.44:0/3285676291 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f63a40a1150 0x7f63a40a3600 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f63ac009cb0 tx=0x7f63ac009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:15.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.222+0000 7f63ab7fe700 1 -- 172.21.15.44:0/3285676291 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f63b4097490 con 0x7f63bc101cc0 2023-11-16T16:49:15.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.429+0000 7f63bb59e700 1 -- 172.21.15.44:0/3285676291 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f63bc1a0950 con 0x7f63bc101cc0 2023-11-16T16:49:15.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.430+0000 7f63ab7fe700 1 -- 172.21.15.44:0/3285676291 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f63b409aab0 con 0x7f63bc101cc0 2023-11-16T16:49:15.432 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2023-11-16T16:49:15.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.433+0000 7f63bb59e700 1 -- 172.21.15.44:0/3285676291 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f63a40a1150 msgr2=0x7f63a40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:15.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.433+0000 7f63bb59e700 1 --2- 172.21.15.44:0/3285676291 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f63a40a1150 0x7f63a40a3600 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f63ac009cb0 tx=0x7f63ac009260 comp rx=0 tx=0).stop 2023-11-16T16:49:15.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.433+0000 7f63bb59e700 1 -- 172.21.15.44:0/3285676291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63bc101cc0 msgr2=0x7f63bc19f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:15.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.433+0000 7f63bb59e700 1 --2- 172.21.15.44:0/3285676291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63bc101cc0 0x7f63bc19f680 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f63b4009fd0 tx=0x7f63b400bce0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.433+0000 7f63bb59e700 1 -- 172.21.15.44:0/3285676291 shutdown_connections 2023-11-16T16:49:15.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.433+0000 7f63bb59e700 1 --2- 172.21.15.44:0/3285676291 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f63a40a1150 0x7f63a40a3600 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.433+0000 7f63bb59e700 1 --2- 172.21.15.44:0/3285676291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63bc101cc0 0x7f63bc19f680 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.433+0000 7f63bb59e700 1 --2- 172.21.15.44:0/3285676291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63bc100080 0x7f63bc1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.433+0000 7f63bb59e700 1 --2- 172.21.15.44:0/3285676291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63bc101310 0x7f63bc198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.433+0000 7f63bb59e700 1 -- 172.21.15.44:0/3285676291 >> 172.21.15.44:0/3285676291 conn(0x7f63bc0fb6a0 msgr2=0x7f63bc0fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:15.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.433+0000 7f63bb59e700 1 -- 172.21.15.44:0/3285676291 shutdown_connections 2023-11-16T16:49:15.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.433+0000 7f63bb59e700 1 -- 172.21.15.44:0/3285676291 wait complete. 2023-11-16T16:49:15.453 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T16:49:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.660+0000 7fa6999ff700 1 -- 172.21.15.44:0/1065445139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa694101e50 msgr2=0x7fa694108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.660+0000 7fa6999ff700 1 --2- 172.21.15.44:0/1065445139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa694101e50 0x7fa694108710 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa68c00b0a0 tx=0x7fa68c00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.660+0000 7fa6999ff700 1 -- 172.21.15.44:0/1065445139 shutdown_connections 2023-11-16T16:49:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.660+0000 7fa6999ff700 1 --2- 172.21.15.44:0/1065445139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa694101e50 0x7fa694108710 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.660+0000 7fa6999ff700 1 --2- 172.21.15.44:0/1065445139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6941002d0 0x7fa6941006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.660+0000 7fa6999ff700 1 --2- 172.21.15.44:0/1065445139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6941014c0 0x7fa694101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.660+0000 7fa6999ff700 1 -- 172.21.15.44:0/1065445139 >> 172.21.15.44:0/1065445139 conn(0x7fa6940fb8b0 msgr2=0x7fa6940fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.660+0000 7fa6999ff700 1 -- 172.21.15.44:0/1065445139 shutdown_connections 2023-11-16T16:49:15.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.660+0000 7fa6999ff700 1 -- 172.21.15.44:0/1065445139 wait complete. 2023-11-16T16:49:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.661+0000 7fa6999ff700 1 Processor -- start 2023-11-16T16:49:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.662+0000 7fa6999ff700 1 -- start start 2023-11-16T16:49:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.662+0000 7fa6999ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6941002d0 0x7fa694196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.662+0000 7fa6999ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6941014c0 0x7fa694196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:15.664 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.662+0000 7fa6999ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa694101e50 0x7fa69419d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.662+0000 7fa6999ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69419dc90 con 0x7fa6941002d0 2023-11-16T16:49:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.662+0000 7fa6999ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69419ddd0 con 0x7fa6941014c0 2023-11-16T16:49:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.662+0000 7fa6999ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69419df60 con 0x7fa694101e50 2023-11-16T16:49:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.662+0000 7fa692ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6941002d0 0x7fa694196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.662+0000 7fa692ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6941002d0 0x7fa694196110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51446/0 (socket says 172.21.15.44:51446) 2023-11-16T16:49:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.662+0000 7fa692ffd700 1 -- 172.21.15.44:0/3821708373 learned_addr learned my addr 172.21.15.44:0/3821708373 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.662+0000 7fa68bfff700 1 --2- 172.21.15.44:0/3821708373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6941014c0 0x7fa694196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.662+0000 7fa6937fe700 1 --2- 172.21.15.44:0/3821708373 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa694101e50 0x7fa69419d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.663+0000 7fa692ffd700 1 -- 172.21.15.44:0/3821708373 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa694101e50 msgr2=0x7fa69419d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.663+0000 7fa692ffd700 1 --2- 172.21.15.44:0/3821708373 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa694101e50 0x7fa69419d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.663+0000 7fa692ffd700 1 -- 172.21.15.44:0/3821708373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6941014c0 msgr2=0x7fa694196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.663+0000 7fa692ffd700 1 --2- 172.21.15.44:0/3821708373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6941014c0 0x7fa694196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.663+0000 7fa692ffd700 1 -- 172.21.15.44:0/3821708373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68c009d00 con 0x7fa6941002d0 2023-11-16T16:49:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.663+0000 7fa692ffd700 1 --2- 172.21.15.44:0/3821708373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6941002d0 0x7fa694196110 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa68400e970 tx=0x7fa68400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.663+0000 7fa6989fd700 1 -- 172.21.15.44:0/3821708373 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa684009790 con 0x7fa6941002d0 2023-11-16T16:49:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.663+0000 7fa6989fd700 1 -- 172.21.15.44:0/3821708373 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa68400c7e0 con 0x7fa6941002d0 2023-11-16T16:49:15.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.663+0000 7fa6989fd700 1 -- 172.21.15.44:0/3821708373 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa684003c70 con 0x7fa6941002d0 2023-11-16T16:49:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.663+0000 7fa6999ff700 1 -- 172.21.15.44:0/3821708373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa69410bc50 con 0x7fa6941002d0 2023-11-16T16:49:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.663+0000 7fa6999ff700 1 -- 172.21.15.44:0/3821708373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa69410bac0 con 0x7fa6941002d0 2023-11-16T16:49:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.665+0000 7fa6999ff700 1 -- 172.21.15.44:0/3821708373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa69419e660 con 0x7fa6941002d0 2023-11-16T16:49:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.666+0000 7fa6989fd700 1 -- 172.21.15.44:0/3821708373 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa68400cc50 con 0x7fa6941002d0 2023-11-16T16:49:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.669+0000 7fa6989fd700 1 --2- 172.21.15.44:0/3821708373 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa6800a1150 0x7fa6800a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.669+0000 7fa6989fd700 1 -- 172.21.15.44:0/3821708373 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fa684014070 con 0x7fa6941002d0 2023-11-16T16:49:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.669+0000 7fa68bfff700 1 --2- 172.21.15.44:0/3821708373 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa6800a1150 0x7fa6800a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:15.671 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.670+0000 7fa68bfff700 1 --2- 172.21.15.44:0/3821708373 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa6800a1150 0x7fa6800a3600 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa67c009c60 tx=0x7fa67c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:15.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.672+0000 7fa6989fd700 1 -- 172.21.15.44:0/3821708373 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa6840961f0 con 0x7fa6941002d0 2023-11-16T16:49:15.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:15 smithi044 ceph-mon[128750]: pgmap v337: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:15.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:15 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1317894209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:15.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:15 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3285676291' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:15.881 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.880+0000 7fa6999ff700 1 -- 172.21.15.44:0/3821708373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa694107150 con 0x7fa6941002d0 2023-11-16T16:49:15.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.881+0000 7fa6989fd700 1 -- 172.21.15.44:0/3821708373 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa684003dd0 con 0x7fa6941002d0 2023-11-16T16:49:15.882 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:49:15.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.883+0000 7fa6999ff700 1 -- 172.21.15.44:0/3821708373 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa6800a1150 msgr2=0x7fa6800a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:15.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.883+0000 7fa6999ff700 1 --2- 172.21.15.44:0/3821708373 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa6800a1150 0x7fa6800a3600 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa67c009c60 tx=0x7fa67c009260 comp rx=0 tx=0).stop 2023-11-16T16:49:15.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.883+0000 7fa6999ff700 1 -- 172.21.15.44:0/3821708373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6941002d0 msgr2=0x7fa694196110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:15.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.883+0000 7fa6999ff700 1 --2- 172.21.15.44:0/3821708373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6941002d0 0x7fa694196110 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa68400e970 tx=0x7fa68400ec80 comp rx=0 tx=0).stop 2023-11-16T16:49:15.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.883+0000 7fa6999ff700 1 -- 172.21.15.44:0/3821708373 shutdown_connections 2023-11-16T16:49:15.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.883+0000 7fa6999ff700 1 --2- 172.21.15.44:0/3821708373 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa6800a1150 0x7fa6800a3600 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.883+0000 7fa6999ff700 1 --2- 172.21.15.44:0/3821708373 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa694101e50 0x7fa69419d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.883+0000 7fa6999ff700 1 --2- 172.21.15.44:0/3821708373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6941002d0 0x7fa694196110 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.883+0000 7fa6999ff700 1 --2- 172.21.15.44:0/3821708373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6941014c0 0x7fa694196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:15.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.883+0000 7fa6999ff700 1 -- 172.21.15.44:0/3821708373 >> 172.21.15.44:0/3821708373 conn(0x7fa6940fb8b0 msgr2=0x7fa6940fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:15.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.883+0000 7fa6999ff700 1 -- 172.21.15.44:0/3821708373 shutdown_connections 2023-11-16T16:49:15.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:15.883+0000 7fa6999ff700 1 -- 172.21.15.44:0/3821708373 wait complete. 2023-11-16T16:49:15.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:15 smithi176 ceph-mon[131630]: pgmap v337: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:15.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:15 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1317894209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:15 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3285676291' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:15.904 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T16:49:15.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:15 smithi077 ceph-mon[131204]: pgmap v337: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:15.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:15 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1317894209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:15.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:15 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3285676291' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:16.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.109+0000 7f2aeadff700 1 -- 172.21.15.44:0/514925921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ae4073550 msgr2=0x7f2ae40739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:16.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.109+0000 7f2aeadff700 1 --2- 172.21.15.44:0/514925921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ae4073550 0x7f2ae40739c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2ad00098f0 tx=0x7f2ad0009c00 comp rx=0 tx=0).stop 2023-11-16T16:49:16.112 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.110+0000 7f2aeadff700 1 -- 172.21.15.44:0/514925921 shutdown_connections 2023-11-16T16:49:16.112 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.110+0000 7f2aeadff700 1 --2- 172.21.15.44:0/514925921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae410c890 0x7f2ae410ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.112 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.110+0000 7f2aeadff700 1 --2- 172.21.15.44:0/514925921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ae4074b30 0x7f2ae4072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.112 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.110+0000 7f2aeadff700 1 --2- 172.21.15.44:0/514925921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ae4073550 0x7f2ae40739c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.112 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.110+0000 7f2aeadff700 1 -- 172.21.15.44:0/514925921 >> 172.21.15.44:0/514925921 conn(0x7f2ae40fb8a0 msgr2=0x7f2ae40fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:16.112 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.110+0000 7f2aeadff700 1 -- 172.21.15.44:0/514925921 shutdown_connections 2023-11-16T16:49:16.112 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.110+0000 7f2aeadff700 1 -- 172.21.15.44:0/514925921 wait complete. 2023-11-16T16:49:16.113 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.111+0000 7f2aeadff700 1 Processor -- start 2023-11-16T16:49:16.113 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.111+0000 7f2aeadff700 1 -- start start 2023-11-16T16:49:16.113 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.111+0000 7f2aeadff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ae4073550 0x7f2ae419c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:16.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.111+0000 7f2aeadff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae4074b30 0x7f2ae419cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:16.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.111+0000 7f2aeadff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ae410c890 0x7f2ae41a3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:16.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.111+0000 7f2aeadff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae41a4250 con 0x7f2ae4073550 2023-11-16T16:49:16.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.111+0000 7f2aeadff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae41a4390 con 0x7f2ae410c890 2023-11-16T16:49:16.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.111+0000 7f2aeadff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae41a4520 con 0x7f2ae4074b30 2023-11-16T16:49:16.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.111+0000 7f2ae8b9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ae4073550 0x7f2ae419c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:16.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.111+0000 7f2ae8b9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ae4073550 0x7f2ae419c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51462/0 (socket says 172.21.15.44:51462) 2023-11-16T16:49:16.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.112+0000 7f2ae8b9b700 1 -- 172.21.15.44:0/2872371955 learned_addr learned my addr 172.21.15.44:0/2872371955 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:16.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.111+0000 7f2ae939c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ae410c890 0x7f2ae41a3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:16.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.112+0000 7f2ae3fff700 1 --2- 172.21.15.44:0/2872371955 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae4074b30 0x7f2ae419cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:16.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.112+0000 7f2ae939c700 1 -- 172.21.15.44:0/2872371955 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae4074b30 msgr2=0x7f2ae419cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:16.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.112+0000 7f2ae939c700 1 --2- 172.21.15.44:0/2872371955 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae4074b30 0x7f2ae419cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.112+0000 7f2ae939c700 1 -- 172.21.15.44:0/2872371955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ae4073550 msgr2=0x7f2ae419c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:16.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.112+0000 7f2ae939c700 1 --2- 172.21.15.44:0/2872371955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ae4073550 0x7f2ae419c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.112+0000 7f2ae939c700 1 -- 172.21.15.44:0/2872371955 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad80095f0 con 0x7f2ae410c890 2023-11-16T16:49:16.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.112+0000 7f2ae8b9b700 1 --2- 172.21.15.44:0/2872371955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ae4073550 0x7f2ae419c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:16.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.112+0000 7f2ae939c700 1 --2- 172.21.15.44:0/2872371955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ae410c890 0x7f2ae41a3c80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2adc00efc0 tx=0x7f2adc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:16.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.112+0000 7f2ae1ffb700 1 -- 172.21.15.44:0/2872371955 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc018070 con 0x7f2ae410c890 2023-11-16T16:49:16.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.112+0000 7f2aeadff700 1 -- 172.21.15.44:0/2872371955 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad0009520 con 0x7f2ae410c890 2023-11-16T16:49:16.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.112+0000 7f2ae1ffb700 1 -- 172.21.15.44:0/2872371955 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2adc00f040 con 0x7f2ae410c890 2023-11-16T16:49:16.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.112+0000 7f2ae1ffb700 1 -- 172.21.15.44:0/2872371955 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc00cd40 con 0x7f2ae410c890 2023-11-16T16:49:16.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.113+0000 7f2aeadff700 1 -- 172.21.15.44:0/2872371955 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae41122a0 con 0x7f2ae410c890 2023-11-16T16:49:16.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.113+0000 7f2aeadff700 1 -- 172.21.15.44:0/2872371955 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae4112430 con 0x7f2ae410c890 2023-11-16T16:49:16.117 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.116+0000 7f2ae1ffb700 1 -- 172.21.15.44:0/2872371955 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2adc003a00 con 0x7f2ae410c890 2023-11-16T16:49:16.118 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.117+0000 7f2ae1ffb700 1 --2- 172.21.15.44:0/2872371955 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ad40a1150 0x7f2ad40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:16.119 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.117+0000 7f2ae1ffb700 1 -- 172.21.15.44:0/2872371955 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f2adc0cc9a0 con 0x7f2ae410c890 2023-11-16T16:49:16.119 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.117+0000 7f2ae8b9b700 1 --2- 172.21.15.44:0/2872371955 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ad40a1150 0x7f2ad40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:16.119 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.118+0000 7f2ae8b9b700 1 --2- 172.21.15.44:0/2872371955 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ad40a1150 0x7f2ad40a3600 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2ad800f710 tx=0x7f2ad8009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:16.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.119+0000 7f2ae1ffb700 1 -- 172.21.15.44:0/2872371955 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2adc095d80 con 0x7f2ae410c890 2023-11-16T16:49:16.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.331+0000 7f2aeadff700 1 -- 172.21.15.44:0/2872371955 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2ae41023e0 con 0x7f2ae410c890 2023-11-16T16:49:16.333 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.332+0000 7f2ae1ffb700 1 -- 172.21.15.44:0/2872371955 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2adc0993a0 con 0x7f2ae410c890 2023-11-16T16:49:16.334 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2023-11-16T16:49:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.334+0000 7f2aeadff700 1 -- 172.21.15.44:0/2872371955 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ad40a1150 msgr2=0x7f2ad40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.334+0000 7f2aeadff700 1 --2- 172.21.15.44:0/2872371955 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ad40a1150 0x7f2ad40a3600 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2ad800f710 tx=0x7f2ad8009310 comp rx=0 tx=0).stop 2023-11-16T16:49:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.334+0000 7f2aeadff700 1 -- 172.21.15.44:0/2872371955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ae410c890 msgr2=0x7f2ae41a3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.334+0000 7f2aeadff700 1 --2- 172.21.15.44:0/2872371955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ae410c890 0x7f2ae41a3c80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2adc00efc0 tx=0x7f2adc00c3a0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.334+0000 7f2aeadff700 1 -- 172.21.15.44:0/2872371955 shutdown_connections 2023-11-16T16:49:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.334+0000 7f2aeadff700 1 --2- 172.21.15.44:0/2872371955 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ad40a1150 0x7f2ad40a3600 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.334+0000 7f2aeadff700 1 --2- 172.21.15.44:0/2872371955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ae410c890 0x7f2ae41a3c80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.334+0000 7f2aeadff700 1 --2- 172.21.15.44:0/2872371955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ae4073550 0x7f2ae419c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.334+0000 7f2aeadff700 1 --2- 172.21.15.44:0/2872371955 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae4074b30 0x7f2ae419cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.334+0000 7f2aeadff700 1 -- 172.21.15.44:0/2872371955 >> 172.21.15.44:0/2872371955 conn(0x7f2ae40fb8a0 msgr2=0x7f2ae40fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:16.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.334+0000 7f2aeadff700 1 -- 172.21.15.44:0/2872371955 shutdown_connections 2023-11-16T16:49:16.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.334+0000 7f2aeadff700 1 -- 172.21.15.44:0/2872371955 wait complete. 2023-11-16T16:49:16.355 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T16:49:16.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.560+0000 7fc9b63ff700 1 -- 172.21.15.44:0/230992974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9b01002c0 msgr2=0x7fc9b01006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:16.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.560+0000 7fc9b63ff700 1 --2- 172.21.15.44:0/230992974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9b01002c0 0x7fc9b01006d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009990 tx=0x7fc9a0009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.560+0000 7fc9b63ff700 1 -- 172.21.15.44:0/230992974 shutdown_connections 2023-11-16T16:49:16.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.560+0000 7fc9b63ff700 1 --2- 172.21.15.44:0/230992974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9b0101e40 0x7fc9b0108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.560+0000 7fc9b63ff700 1 --2- 172.21.15.44:0/230992974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9b01002c0 0x7fc9b01006d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.560+0000 7fc9b63ff700 1 --2- 172.21.15.44:0/230992974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9b01014b0 0x7fc9b0101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.560+0000 7fc9b63ff700 1 -- 172.21.15.44:0/230992974 >> 172.21.15.44:0/230992974 conn(0x7fc9b00fb8a0 msgr2=0x7fc9b00fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:16.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.560+0000 7fc9b63ff700 1 -- 172.21.15.44:0/230992974 shutdown_connections 2023-11-16T16:49:16.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.560+0000 7fc9b63ff700 1 -- 172.21.15.44:0/230992974 wait complete. 2023-11-16T16:49:16.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.561+0000 7fc9b63ff700 1 Processor -- start 2023-11-16T16:49:16.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.562+0000 7fc9b63ff700 1 -- start start 2023-11-16T16:49:16.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.562+0000 7fc9b63ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9b01002c0 0x7fc9b0198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:16.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.562+0000 7fc9b63ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9b01014b0 0x7fc9b01987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:16.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.562+0000 7fc9b63ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9b0101e40 0x7fc9b019f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:16.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.562+0000 7fc9b63ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b019fe10 con 0x7fc9b0101e40 2023-11-16T16:49:16.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.562+0000 7fc9b63ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b019ff50 con 0x7fc9b01002c0 2023-11-16T16:49:16.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.562+0000 7fc9b63ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b01a00e0 con 0x7fc9b01014b0 2023-11-16T16:49:16.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.562+0000 7fc9b499c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9b0101e40 0x7fc9b019f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:16.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.562+0000 7fc9af7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9b01014b0 0x7fc9b01987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:16.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.562+0000 7fc9affff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9b01002c0 0x7fc9b0198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:16.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.562+0000 7fc9af7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9b01014b0 0x7fc9b01987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:42270/0 (socket says 172.21.15.44:42270) 2023-11-16T16:49:16.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.562+0000 7fc9af7fe700 1 -- 172.21.15.44:0/2371688526 learned_addr learned my addr 172.21.15.44:0/2371688526 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:16.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.562+0000 7fc9affff700 1 --2- 172.21.15.44:0/2371688526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9b01002c0 0x7fc9b0198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:52906/0 (socket says 172.21.15.44:52906) 2023-11-16T16:49:16.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.563+0000 7fc9af7fe700 1 -- 172.21.15.44:0/2371688526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9b01002c0 msgr2=0x7fc9b0198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:16.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.563+0000 7fc9af7fe700 1 --2- 172.21.15.44:0/2371688526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9b01002c0 0x7fc9b0198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.563+0000 7fc9af7fe700 1 -- 172.21.15.44:0/2371688526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9b0101e40 msgr2=0x7fc9b019f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:16.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.563+0000 7fc9af7fe700 1 --2- 172.21.15.44:0/2371688526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9b0101e40 0x7fc9b019f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.563+0000 7fc9af7fe700 1 -- 172.21.15.44:0/2371688526 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a00095f0 con 0x7fc9b01014b0 2023-11-16T16:49:16.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.563+0000 7fc9b499c700 1 --2- 172.21.15.44:0/2371688526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9b0101e40 0x7fc9b019f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:16.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.563+0000 7fc9af7fe700 1 --2- 172.21.15.44:0/2371688526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9b01014b0 0x7fc9b01987d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400e9a0 tx=0x7fc9a400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:16.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.563+0000 7fc9ad7fa700 1 -- 172.21.15.44:0/2371688526 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4009790 con 0x7fc9b01014b0 2023-11-16T16:49:16.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.563+0000 7fc9ad7fa700 1 -- 172.21.15.44:0/2371688526 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9a4010430 con 0x7fc9b01014b0 2023-11-16T16:49:16.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.563+0000 7fc9b63ff700 1 -- 172.21.15.44:0/2371688526 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b010bc90 con 0x7fc9b01014b0 2023-11-16T16:49:16.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.563+0000 7fc9ad7fa700 1 -- 172.21.15.44:0/2371688526 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4018680 con 0x7fc9b01014b0 2023-11-16T16:49:16.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.563+0000 7fc9b63ff700 1 -- 172.21.15.44:0/2371688526 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b010bb00 con 0x7fc9b01014b0 2023-11-16T16:49:16.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.564+0000 7fc9b63ff700 1 -- 172.21.15.44:0/2371688526 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b01929f0 con 0x7fc9b01014b0 2023-11-16T16:49:16.568 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.567+0000 7fc9ad7fa700 1 -- 172.21.15.44:0/2371688526 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc9a40105a0 con 0x7fc9b01014b0 2023-11-16T16:49:16.569 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.568+0000 7fc9ad7fa700 1 --2- 172.21.15.44:0/2371688526 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc9980a1150 0x7fc9980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:16.570 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.568+0000 7fc9ad7fa700 1 -- 172.21.15.44:0/2371688526 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fc9a4014070 con 0x7fc9b01014b0 2023-11-16T16:49:16.570 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.568+0000 7fc9affff700 1 --2- 172.21.15.44:0/2371688526 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc9980a1150 0x7fc9980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:16.570 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.569+0000 7fc9affff700 1 --2- 172.21.15.44:0/2371688526 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc9980a1150 0x7fc9980a3600 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc9a000b2e0 tx=0x7fc9a0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:16.571 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.570+0000 7fc9ad7fa700 1 -- 172.21.15.44:0/2371688526 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc9a4096400 con 0x7fc9b01014b0 2023-11-16T16:49:16.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.782+0000 7fc9b63ff700 1 -- 172.21.15.44:0/2371688526 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc9b0061700 con 0x7fc9b01014b0 2023-11-16T16:49:16.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.783+0000 7fc9ad7fa700 1 -- 172.21.15.44:0/2371688526 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc9a40260a0 con 0x7fc9b01014b0 2023-11-16T16:49:16.784 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:49:16.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.785+0000 7fc9b63ff700 1 -- 172.21.15.44:0/2371688526 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc9980a1150 msgr2=0x7fc9980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:16.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.785+0000 7fc9b63ff700 1 --2- 172.21.15.44:0/2371688526 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc9980a1150 0x7fc9980a3600 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc9a000b2e0 tx=0x7fc9a0005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.785+0000 7fc9b63ff700 1 -- 172.21.15.44:0/2371688526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9b01014b0 msgr2=0x7fc9b01987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:16.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.785+0000 7fc9b63ff700 1 --2- 172.21.15.44:0/2371688526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9b01014b0 0x7fc9b01987d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400e9a0 tx=0x7fc9a400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.785+0000 7fc9b63ff700 1 -- 172.21.15.44:0/2371688526 shutdown_connections 2023-11-16T16:49:16.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.785+0000 7fc9b63ff700 1 --2- 172.21.15.44:0/2371688526 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc9980a1150 0x7fc9980a3600 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.785+0000 7fc9b63ff700 1 --2- 172.21.15.44:0/2371688526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9b0101e40 0x7fc9b019f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.785+0000 7fc9b63ff700 1 --2- 172.21.15.44:0/2371688526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9b01002c0 0x7fc9b0198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.785+0000 7fc9b63ff700 1 --2- 172.21.15.44:0/2371688526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9b01014b0 0x7fc9b01987d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:16.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.785+0000 7fc9b63ff700 1 -- 172.21.15.44:0/2371688526 >> 172.21.15.44:0/2371688526 conn(0x7fc9b00fb8a0 msgr2=0x7fc9b0106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:16.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.786+0000 7fc9b63ff700 1 -- 172.21.15.44:0/2371688526 shutdown_connections 2023-11-16T16:49:16.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:16.786+0000 7fc9b63ff700 1 -- 172.21.15.44:0/2371688526 wait complete. 2023-11-16T16:49:16.806 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:49:16.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:16 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3821708373' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:16.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:16 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2872371955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:16.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:16 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3821708373' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:16.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:16 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2872371955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:17.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.009+0000 7f9e059ff700 1 -- 172.21.15.44:0/648232223 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e00100290 msgr2=0x7f9e001006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.009+0000 7f9e059ff700 1 --2- 172.21.15.44:0/648232223 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e00100290 0x7f9e001006a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9df00098f0 tx=0x7f9df0009c00 comp rx=0 tx=0).stop 2023-11-16T16:49:17.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.009+0000 7f9e059ff700 1 -- 172.21.15.44:0/648232223 shutdown_connections 2023-11-16T16:49:17.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.009+0000 7f9e059ff700 1 --2- 172.21.15.44:0/648232223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e00101e10 0x7f9e001086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.009+0000 7f9e059ff700 1 --2- 172.21.15.44:0/648232223 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e00100290 0x7f9e001006a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.009+0000 7f9e059ff700 1 --2- 172.21.15.44:0/648232223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e00101480 0x7f9e001018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.009+0000 7f9e059ff700 1 -- 172.21.15.44:0/648232223 >> 172.21.15.44:0/648232223 conn(0x7f9e000fb870 msgr2=0x7f9e000fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:17.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.009+0000 7f9e059ff700 1 -- 172.21.15.44:0/648232223 shutdown_connections 2023-11-16T16:49:17.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.009+0000 7f9e059ff700 1 -- 172.21.15.44:0/648232223 wait complete. 2023-11-16T16:49:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.010+0000 7f9e059ff700 1 Processor -- start 2023-11-16T16:49:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.010+0000 7f9e059ff700 1 -- start start 2023-11-16T16:49:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.011+0000 7f9e059ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e00100290 0x7f9e00198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.011+0000 7f9e059ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e00101480 0x7f9e00198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.011+0000 7f9e059ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e00101e10 0x7f9e0019f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.011+0000 7f9e059ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0019fca0 con 0x7f9e00101480 2023-11-16T16:49:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.011+0000 7f9e059ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0019fde0 con 0x7f9e00101e10 2023-11-16T16:49:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.011+0000 7f9e059ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0019ff70 con 0x7f9e00100290 2023-11-16T16:49:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.011+0000 7f9df7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e00101480 0x7f9e00198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:17.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.011+0000 7f9df7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e00101480 0x7f9e00198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51504/0 (socket says 172.21.15.44:51504) 2023-11-16T16:49:17.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.011+0000 7f9df7fff700 1 -- 172.21.15.44:0/4164166261 learned_addr learned my addr 172.21.15.44:0/4164166261 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:17.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.011+0000 7f9dff7fe700 1 --2- 172.21.15.44:0/4164166261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e00101e10 0x7f9e0019f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:17.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.011+0000 7f9dfeffd700 1 --2- 172.21.15.44:0/4164166261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e00100290 0x7f9e00198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:17.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.011+0000 7f9df7fff700 1 -- 172.21.15.44:0/4164166261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e00100290 msgr2=0x7f9e00198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.011+0000 7f9df7fff700 1 --2- 172.21.15.44:0/4164166261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e00100290 0x7f9e00198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.011+0000 7f9df7fff700 1 -- 172.21.15.44:0/4164166261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e00101e10 msgr2=0x7f9e0019f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.011+0000 7f9df7fff700 1 --2- 172.21.15.44:0/4164166261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e00101e10 0x7f9e0019f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.011+0000 7f9df7fff700 1 -- 172.21.15.44:0/4164166261 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df0009520 con 0x7f9e00101480 2023-11-16T16:49:17.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.012+0000 7f9df7fff700 1 --2- 172.21.15.44:0/4164166261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e00101480 0x7f9e00198660 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9de800e9c0 tx=0x7f9de800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:17.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.012+0000 7f9e049fd700 1 -- 172.21.15.44:0/4164166261 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de8004a30 con 0x7f9e00101480 2023-11-16T16:49:17.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.012+0000 7f9e059ff700 1 -- 172.21.15.44:0/4164166261 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0010bbe0 con 0x7f9e00101480 2023-11-16T16:49:17.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.012+0000 7f9e049fd700 1 -- 172.21.15.44:0/4164166261 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9de800cc80 con 0x7f9e00101480 2023-11-16T16:49:17.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.012+0000 7f9e059ff700 1 -- 172.21.15.44:0/4164166261 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0010ba50 con 0x7f9e00101480 2023-11-16T16:49:17.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.012+0000 7f9e049fd700 1 -- 172.21.15.44:0/4164166261 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de8005480 con 0x7f9e00101480 2023-11-16T16:49:17.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.013+0000 7f9e059ff700 1 -- 172.21.15.44:0/4164166261 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e00192870 con 0x7f9e00101480 2023-11-16T16:49:17.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.015+0000 7f9e049fd700 1 -- 172.21.15.44:0/4164166261 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9de80055e0 con 0x7f9e00101480 2023-11-16T16:49:17.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.017+0000 7f9e049fd700 1 --2- 172.21.15.44:0/4164166261 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9dec0a1150 0x7f9dec0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:17.022 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.017+0000 7f9e049fd700 1 -- 172.21.15.44:0/4164166261 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f9de8014070 con 0x7f9e00101480 2023-11-16T16:49:17.022 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.017+0000 7f9dfeffd700 1 --2- 172.21.15.44:0/4164166261 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9dec0a1150 0x7f9dec0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:17.022 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.018+0000 7f9dfeffd700 1 --2- 172.21.15.44:0/4164166261 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9dec0a1150 0x7f9dec0a3600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9df0009fd0 tx=0x7f9df0019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:17.022 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.020+0000 7f9e049fd700 1 -- 172.21.15.44:0/4164166261 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9de8096270 con 0x7f9e00101480 2023-11-16T16:49:17.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:16 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3821708373' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:17.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:16 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2872371955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:17.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.276+0000 7f9e059ff700 1 -- 172.21.15.44:0/4164166261 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e0002cfe0 con 0x7f9e00101480 2023-11-16T16:49:17.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.277+0000 7f9e049fd700 1 -- 172.21.15.44:0/4164166261 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f9de8016330 con 0x7f9e00101480 2023-11-16T16:49:17.279 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:17.279 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:49:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.279+0000 7f9e059ff700 1 -- 172.21.15.44:0/4164166261 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9dec0a1150 msgr2=0x7f9dec0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.279+0000 7f9e059ff700 1 --2- 172.21.15.44:0/4164166261 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9dec0a1150 0x7f9dec0a3600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9df0009fd0 tx=0x7f9df0019040 comp rx=0 tx=0).stop 2023-11-16T16:49:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.279+0000 7f9e059ff700 1 -- 172.21.15.44:0/4164166261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e00101480 msgr2=0x7f9e00198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.279+0000 7f9e059ff700 1 --2- 172.21.15.44:0/4164166261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e00101480 0x7f9e00198660 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9de800e9c0 tx=0x7f9de800ecd0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.279+0000 7f9e059ff700 1 -- 172.21.15.44:0/4164166261 shutdown_connections 2023-11-16T16:49:17.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.279+0000 7f9e059ff700 1 --2- 172.21.15.44:0/4164166261 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9dec0a1150 0x7f9dec0a3600 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.279+0000 7f9e059ff700 1 --2- 172.21.15.44:0/4164166261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e00101e10 0x7f9e0019f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.279+0000 7f9e059ff700 1 --2- 172.21.15.44:0/4164166261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e00100290 0x7f9e00198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.279+0000 7f9e059ff700 1 --2- 172.21.15.44:0/4164166261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e00101480 0x7f9e00198660 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.279+0000 7f9e059ff700 1 -- 172.21.15.44:0/4164166261 >> 172.21.15.44:0/4164166261 conn(0x7f9e000fb870 msgr2=0x7f9e00106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:17.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.279+0000 7f9e059ff700 1 -- 172.21.15.44:0/4164166261 shutdown_connections 2023-11-16T16:49:17.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.279+0000 7f9e059ff700 1 -- 172.21.15.44:0/4164166261 wait complete. 2023-11-16T16:49:17.283 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:49:17.300 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:49:17.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.502+0000 7f2a847ff700 1 -- 172.21.15.44:0/2577038984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c100290 msgr2=0x7f2a7c1006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.502+0000 7f2a847ff700 1 --2- 172.21.15.44:0/2577038984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c100290 0x7f2a7c1006a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2a78009990 tx=0x7f2a78009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.503+0000 7f2a847ff700 1 -- 172.21.15.44:0/2577038984 shutdown_connections 2023-11-16T16:49:17.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.503+0000 7f2a847ff700 1 --2- 172.21.15.44:0/2577038984 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a7c101e10 0x7f2a7c1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.503+0000 7f2a847ff700 1 --2- 172.21.15.44:0/2577038984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c100290 0x7f2a7c1006a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.503+0000 7f2a847ff700 1 --2- 172.21.15.44:0/2577038984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7c101480 0x7f2a7c1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.503+0000 7f2a847ff700 1 -- 172.21.15.44:0/2577038984 >> 172.21.15.44:0/2577038984 conn(0x7f2a7c0fb870 msgr2=0x7f2a7c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:17.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.503+0000 7f2a847ff700 1 -- 172.21.15.44:0/2577038984 shutdown_connections 2023-11-16T16:49:17.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.503+0000 7f2a847ff700 1 -- 172.21.15.44:0/2577038984 wait complete. 2023-11-16T16:49:17.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.504+0000 7f2a847ff700 1 Processor -- start 2023-11-16T16:49:17.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.504+0000 7f2a847ff700 1 -- start start 2023-11-16T16:49:17.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.504+0000 7f2a847ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a7c100290 0x7f2a7c1981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:17.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.504+0000 7f2a847ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7c101480 0x7f2a7c198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:17.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.504+0000 7f2a847ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c101e10 0x7f2a7c19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:17.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.504+0000 7f2a847ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c19fd40 con 0x7f2a7c101e10 2023-11-16T16:49:17.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.504+0000 7f2a847ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c19fe80 con 0x7f2a7c101480 2023-11-16T16:49:17.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.504+0000 7f2a847ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c1a0010 con 0x7f2a7c100290 2023-11-16T16:49:17.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.505+0000 7f2a82d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c101e10 0x7f2a7c19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:17.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.505+0000 7f2a82d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c101e10 0x7f2a7c19f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51522/0 (socket says 172.21.15.44:51522) 2023-11-16T16:49:17.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.505+0000 7f2a82d9c700 1 -- 172.21.15.44:0/603118946 learned_addr learned my addr 172.21.15.44:0/603118946 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:17.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.505+0000 7f2a81d9a700 1 --2- 172.21.15.44:0/603118946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7c101480 0x7f2a7c198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:17.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.505+0000 7f2a8259b700 1 --2- 172.21.15.44:0/603118946 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a7c100290 0x7f2a7c1981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:17.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.505+0000 7f2a82d9c700 1 -- 172.21.15.44:0/603118946 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a7c100290 msgr2=0x7f2a7c1981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.505+0000 7f2a82d9c700 1 --2- 172.21.15.44:0/603118946 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a7c100290 0x7f2a7c1981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.505+0000 7f2a82d9c700 1 -- 172.21.15.44:0/603118946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7c101480 msgr2=0x7f2a7c198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.505+0000 7f2a82d9c700 1 --2- 172.21.15.44:0/603118946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7c101480 0x7f2a7c198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.505+0000 7f2a82d9c700 1 -- 172.21.15.44:0/603118946 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a6c009520 con 0x7f2a7c101e10 2023-11-16T16:49:17.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.505+0000 7f2a82d9c700 1 --2- 172.21.15.44:0/603118946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c101e10 0x7f2a7c19f770 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2a7400ef30 tx=0x7f2a7400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:17.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.505+0000 7f2a737fe700 1 -- 172.21.15.44:0/603118946 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a74018070 con 0x7f2a7c101e10 2023-11-16T16:49:17.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.505+0000 7f2a737fe700 1 -- 172.21.15.44:0/603118946 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a7400f040 con 0x7f2a7c101e10 2023-11-16T16:49:17.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.505+0000 7f2a847ff700 1 -- 172.21.15.44:0/603118946 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a780095f0 con 0x7f2a7c101e10 2023-11-16T16:49:17.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.505+0000 7f2a737fe700 1 -- 172.21.15.44:0/603118946 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7400cd00 con 0x7f2a7c101e10 2023-11-16T16:49:17.512 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.506+0000 7f2a847ff700 1 -- 172.21.15.44:0/603118946 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7c10bc40 con 0x7f2a7c101e10 2023-11-16T16:49:17.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.507+0000 7f2a847ff700 1 -- 172.21.15.44:0/603118946 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7c1928d0 con 0x7f2a7c101e10 2023-11-16T16:49:17.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.508+0000 7f2a737fe700 1 -- 172.21.15.44:0/603118946 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2a74003a00 con 0x7f2a7c101e10 2023-11-16T16:49:17.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.510+0000 7f2a737fe700 1 --2- 172.21.15.44:0/603118946 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2a680a11d0 0x7f2a680a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:17.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.510+0000 7f2a737fe700 1 -- 172.21.15.44:0/603118946 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f2a740cc7b0 con 0x7f2a7c101e10 2023-11-16T16:49:17.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.510+0000 7f2a8259b700 1 --2- 172.21.15.44:0/603118946 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2a680a11d0 0x7f2a680a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:17.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.511+0000 7f2a8259b700 1 --2- 172.21.15.44:0/603118946 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2a680a11d0 0x7f2a680a3680 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2a7800b2e0 tx=0x7f2a78005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:17.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.513+0000 7f2a737fe700 1 -- 172.21.15.44:0/603118946 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2a74095b90 con 0x7f2a7c101e10 2023-11-16T16:49:17.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.728+0000 7f2a847ff700 1 -- 172.21.15.44:0/603118946 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a7c02cfe0 con 0x7f2a7c101e10 2023-11-16T16:49:17.732 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.730+0000 7f2a737fe700 1 -- 172.21.15.44:0/603118946 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24685 (secure 0 0 0) 0x7f2a740991b0 con 0x7f2a7c101e10 2023-11-16T16:49:17.732 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:17.733 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:48:44.613706+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.732+0000 7f2a847ff700 1 -- 172.21.15.44:0/603118946 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2a680a11d0 msgr2=0x7f2a680a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.732+0000 7f2a847ff700 1 --2- 172.21.15.44:0/603118946 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2a680a11d0 0x7f2a680a3680 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2a7800b2e0 tx=0x7f2a78005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.732+0000 7f2a847ff700 1 -- 172.21.15.44:0/603118946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c101e10 msgr2=0x7f2a7c19f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.732+0000 7f2a847ff700 1 --2- 172.21.15.44:0/603118946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c101e10 0x7f2a7c19f770 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2a7400ef30 tx=0x7f2a7400c3a0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.732+0000 7f2a847ff700 1 -- 172.21.15.44:0/603118946 shutdown_connections 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.732+0000 7f2a847ff700 1 --2- 172.21.15.44:0/603118946 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2a680a11d0 0x7f2a680a3680 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.732+0000 7f2a847ff700 1 --2- 172.21.15.44:0/603118946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a7c101e10 0x7f2a7c19f770 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.732+0000 7f2a847ff700 1 --2- 172.21.15.44:0/603118946 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a7c100290 0x7f2a7c1981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.732+0000 7f2a847ff700 1 --2- 172.21.15.44:0/603118946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7c101480 0x7f2a7c198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.732+0000 7f2a847ff700 1 -- 172.21.15.44:0/603118946 >> 172.21.15.44:0/603118946 conn(0x7f2a7c0fb870 msgr2=0x7f2a7c106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.733+0000 7f2a847ff700 1 -- 172.21.15.44:0/603118946 shutdown_connections 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.733+0000 7f2a847ff700 1 -- 172.21.15.44:0/603118946 wait complete. 2023-11-16T16:49:17.754 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:03.059622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:16.211826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:17.601591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:21.136084+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T16:49:17.755 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:49:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:17 smithi176 ceph-mon[131630]: pgmap v338: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:49:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:17 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2371688526' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:17 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/4164166261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:17.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.960+0000 7fc1b03ff700 1 -- 172.21.15.44:0/317738682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1a81002c0 msgr2=0x7fc1a81006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.960+0000 7fc1b03ff700 1 --2- 172.21.15.44:0/317738682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1a81002c0 0x7fc1a81006d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc1a40098f0 tx=0x7fc1a4009c00 comp rx=0 tx=0).stop 2023-11-16T16:49:17.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.960+0000 7fc1b03ff700 1 -- 172.21.15.44:0/317738682 shutdown_connections 2023-11-16T16:49:17.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.960+0000 7fc1b03ff700 1 --2- 172.21.15.44:0/317738682 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc1a8101e40 0x7fc1a8108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.960+0000 7fc1b03ff700 1 --2- 172.21.15.44:0/317738682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1a81002c0 0x7fc1a81006d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.960+0000 7fc1b03ff700 1 --2- 172.21.15.44:0/317738682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a81014b0 0x7fc1a8101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.960+0000 7fc1b03ff700 1 -- 172.21.15.44:0/317738682 >> 172.21.15.44:0/317738682 conn(0x7fc1a80fb8a0 msgr2=0x7fc1a80fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:17.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.960+0000 7fc1b03ff700 1 -- 172.21.15.44:0/317738682 shutdown_connections 2023-11-16T16:49:17.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.960+0000 7fc1b03ff700 1 -- 172.21.15.44:0/317738682 wait complete. 2023-11-16T16:49:17.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.961+0000 7fc1b03ff700 1 Processor -- start 2023-11-16T16:49:17.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.961+0000 7fc1b03ff700 1 -- start start 2023-11-16T16:49:17.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.961+0000 7fc1b03ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc1a81002c0 0x7fc1a80748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:17.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.961+0000 7fc1b03ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a81014b0 0x7fc1a8072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:17.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1b03ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1a8101e40 0x7fc1a8073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:17.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1b03ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a8073bc0 con 0x7fc1a81014b0 2023-11-16T16:49:17.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1b03ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a8073d30 con 0x7fc1a8101e40 2023-11-16T16:49:17.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1b03ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a81a0140 con 0x7fc1a81002c0 2023-11-16T16:49:17.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1ad99a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a81014b0 0x7fc1a8072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:17.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1ae99c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1a8101e40 0x7fc1a8073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:17.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1ad99a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a81014b0 0x7fc1a8072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51542/0 (socket says 172.21.15.44:51542) 2023-11-16T16:49:17.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1ad99a700 1 -- 172.21.15.44:0/3597517351 learned_addr learned my addr 172.21.15.44:0/3597517351 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:17.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1ae19b700 1 --2- 172.21.15.44:0/3597517351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc1a81002c0 0x7fc1a80748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:17.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1ae99c700 1 -- 172.21.15.44:0/3597517351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc1a81002c0 msgr2=0x7fc1a80748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1ae99c700 1 --2- 172.21.15.44:0/3597517351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc1a81002c0 0x7fc1a80748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1ae99c700 1 -- 172.21.15.44:0/3597517351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a81014b0 msgr2=0x7fc1a8072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1ae99c700 1 --2- 172.21.15.44:0/3597517351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a81014b0 0x7fc1a8072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1ae99c700 1 -- 172.21.15.44:0/3597517351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a4009520 con 0x7fc1a8101e40 2023-11-16T16:49:17.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1ad99a700 1 --2- 172.21.15.44:0/3597517351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a81014b0 0x7fc1a8072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:17.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1ae19b700 1 --2- 172.21.15.44:0/3597517351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc1a81002c0 0x7fc1a80748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:49:17.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.962+0000 7fc1ae99c700 1 --2- 172.21.15.44:0/3597517351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1a8101e40 0x7fc1a8073420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000efc0 tx=0x7fc1a000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:17.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.963+0000 7fc19f7fe700 1 -- 172.21.15.44:0/3597517351 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0018070 con 0x7fc1a8101e40 2023-11-16T16:49:17.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.963+0000 7fc19f7fe700 1 -- 172.21.15.44:0/3597517351 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc1a000f040 con 0x7fc1a8101e40 2023-11-16T16:49:17.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.963+0000 7fc1b03ff700 1 -- 172.21.15.44:0/3597517351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a810bc90 con 0x7fc1a8101e40 2023-11-16T16:49:17.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.963+0000 7fc19f7fe700 1 -- 172.21.15.44:0/3597517351 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a000cdf0 con 0x7fc1a8101e40 2023-11-16T16:49:17.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.963+0000 7fc1b03ff700 1 -- 172.21.15.44:0/3597517351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a810bb00 con 0x7fc1a8101e40 2023-11-16T16:49:17.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.964+0000 7fc1b03ff700 1 -- 172.21.15.44:0/3597517351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a81a07d0 con 0x7fc1a8101e40 2023-11-16T16:49:17.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.966+0000 7fc19f7fe700 1 -- 172.21.15.44:0/3597517351 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc1a0007500 con 0x7fc1a8101e40 2023-11-16T16:49:17.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.967+0000 7fc19f7fe700 1 --2- 172.21.15.44:0/3597517351 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc1940a1150 0x7fc1940a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:17.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.967+0000 7fc19f7fe700 1 -- 172.21.15.44:0/3597517351 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fc1a00cc920 con 0x7fc1a8101e40 2023-11-16T16:49:17.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.967+0000 7fc1ae19b700 1 --2- 172.21.15.44:0/3597517351 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc1940a1150 0x7fc1940a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:17.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.968+0000 7fc1ae19b700 1 --2- 172.21.15.44:0/3597517351 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc1940a1150 0x7fc1940a3600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc1a40098c0 tx=0x7fc1a4019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:17.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:17.971+0000 7fc19f7fe700 1 -- 172.21.15.44:0/3597517351 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc1a0095d00 con 0x7fc1a8101e40 2023-11-16T16:49:17.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:17 smithi077 ceph-mon[131204]: pgmap v338: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:49:17.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:17 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2371688526' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:17.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:17 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/4164166261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:18.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:17 smithi044 ceph-mon[128750]: pgmap v338: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:49:18.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:17 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2371688526' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:18.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:17 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/4164166261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:18.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.178+0000 7fc1b03ff700 1 -- 172.21.15.44:0/3597517351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc1a81071e0 con 0x7fc1a8101e40 2023-11-16T16:49:18.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.178+0000 7fc19f7fe700 1 -- 172.21.15.44:0/3597517351 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc1a0099320 con 0x7fc1a8101e40 2023-11-16T16:49:18.180 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2023-11-16T16:49:18.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.181+0000 7fc1b03ff700 1 -- 172.21.15.44:0/3597517351 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc1940a1150 msgr2=0x7fc1940a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:18.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.181+0000 7fc1b03ff700 1 --2- 172.21.15.44:0/3597517351 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc1940a1150 0x7fc1940a3600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc1a40098c0 tx=0x7fc1a4019040 comp rx=0 tx=0).stop 2023-11-16T16:49:18.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.181+0000 7fc1b03ff700 1 -- 172.21.15.44:0/3597517351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1a8101e40 msgr2=0x7fc1a8073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:18.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.181+0000 7fc1b03ff700 1 --2- 172.21.15.44:0/3597517351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1a8101e40 0x7fc1a8073420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000efc0 tx=0x7fc1a000c3a0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.181+0000 7fc1b03ff700 1 -- 172.21.15.44:0/3597517351 shutdown_connections 2023-11-16T16:49:18.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.181+0000 7fc1b03ff700 1 --2- 172.21.15.44:0/3597517351 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fc1940a1150 0x7fc1940a3600 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.181+0000 7fc1b03ff700 1 --2- 172.21.15.44:0/3597517351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1a8101e40 0x7fc1a8073420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.181+0000 7fc1b03ff700 1 --2- 172.21.15.44:0/3597517351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc1a81002c0 0x7fc1a80748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.181+0000 7fc1b03ff700 1 --2- 172.21.15.44:0/3597517351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a81014b0 0x7fc1a8072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.181+0000 7fc1b03ff700 1 -- 172.21.15.44:0/3597517351 >> 172.21.15.44:0/3597517351 conn(0x7fc1a80fb8a0 msgr2=0x7fc1a80fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:18.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.181+0000 7fc1b03ff700 1 -- 172.21.15.44:0/3597517351 shutdown_connections 2023-11-16T16:49:18.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.181+0000 7fc1b03ff700 1 -- 172.21.15.44:0/3597517351 wait complete. 2023-11-16T16:49:18.204 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T16:49:18.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.410+0000 7fa8e4dff700 1 -- 172.21.15.44:0/3232506605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e0101e40 msgr2=0x7fa8e0108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:18.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.410+0000 7fa8e4dff700 1 --2- 172.21.15.44:0/3232506605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e0101e40 0x7fa8e0108700 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800b0a0 tx=0x7fa8d800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.410+0000 7fa8e4dff700 1 -- 172.21.15.44:0/3232506605 shutdown_connections 2023-11-16T16:49:18.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.410+0000 7fa8e4dff700 1 --2- 172.21.15.44:0/3232506605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e0101e40 0x7fa8e0108700 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.410+0000 7fa8e4dff700 1 --2- 172.21.15.44:0/3232506605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8e01002c0 0x7fa8e01006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.413 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.410+0000 7fa8e4dff700 1 --2- 172.21.15.44:0/3232506605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e01014b0 0x7fa8e0101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.413 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.410+0000 7fa8e4dff700 1 -- 172.21.15.44:0/3232506605 >> 172.21.15.44:0/3232506605 conn(0x7fa8e00fb8a0 msgr2=0x7fa8e00fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:18.413 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.410+0000 7fa8e4dff700 1 -- 172.21.15.44:0/3232506605 shutdown_connections 2023-11-16T16:49:18.413 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.411+0000 7fa8e4dff700 1 -- 172.21.15.44:0/3232506605 wait complete. 2023-11-16T16:49:18.414 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.411+0000 7fa8e4dff700 1 Processor -- start 2023-11-16T16:49:18.414 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.412+0000 7fa8e4dff700 1 -- start start 2023-11-16T16:49:18.414 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.412+0000 7fa8e4dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e01002c0 0x7fa8e0198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:18.414 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.412+0000 7fa8e4dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e01014b0 0x7fa8e0198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:18.414 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.412+0000 7fa8e4dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8e0101e40 0x7fa8e019f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:18.414 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.412+0000 7fa8e4dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e019fec0 con 0x7fa8e01014b0 2023-11-16T16:49:18.414 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.412+0000 7fa8e4dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e01a0000 con 0x7fa8e01002c0 2023-11-16T16:49:18.414 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.412+0000 7fa8e4dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e01a0190 con 0x7fa8e0101e40 2023-11-16T16:49:18.415 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.412+0000 7fa8d7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e01014b0 0x7fa8e0198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:18.415 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.412+0000 7fa8d7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e01014b0 0x7fa8e0198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51564/0 (socket says 172.21.15.44:51564) 2023-11-16T16:49:18.415 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.412+0000 7fa8d7fff700 1 -- 172.21.15.44:0/3701736595 learned_addr learned my addr 172.21.15.44:0/3701736595 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:18.415 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.412+0000 7fa8de59c700 1 --2- 172.21.15.44:0/3701736595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e01002c0 0x7fa8e0198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:18.415 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.412+0000 7fa8ded9d700 1 --2- 172.21.15.44:0/3701736595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8e0101e40 0x7fa8e019f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:18.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.413+0000 7fa8d7fff700 1 -- 172.21.15.44:0/3701736595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8e0101e40 msgr2=0x7fa8e019f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:18.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.413+0000 7fa8d7fff700 1 --2- 172.21.15.44:0/3701736595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8e0101e40 0x7fa8e019f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.413+0000 7fa8d7fff700 1 -- 172.21.15.44:0/3701736595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e01002c0 msgr2=0x7fa8e0198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:18.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.413+0000 7fa8d7fff700 1 --2- 172.21.15.44:0/3701736595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e01002c0 0x7fa8e0198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.413+0000 7fa8d7fff700 1 -- 172.21.15.44:0/3701736595 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d8009d00 con 0x7fa8e01014b0 2023-11-16T16:49:18.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.413+0000 7fa8de59c700 1 --2- 172.21.15.44:0/3701736595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e01002c0 0x7fa8e0198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:18.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.413+0000 7fa8d7fff700 1 --2- 172.21.15.44:0/3701736595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e01014b0 0x7fa8e0198880 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800e9a0 tx=0x7fa8c800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:18.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.413+0000 7fa8d77fe700 1 -- 172.21.15.44:0/3701736595 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8009790 con 0x7fa8e01014b0 2023-11-16T16:49:18.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.413+0000 7fa8d77fe700 1 -- 172.21.15.44:0/3701736595 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa8c8010430 con 0x7fa8e01014b0 2023-11-16T16:49:18.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.413+0000 7fa8e4dff700 1 -- 172.21.15.44:0/3701736595 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e010bc90 con 0x7fa8e01014b0 2023-11-16T16:49:18.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.413+0000 7fa8d77fe700 1 -- 172.21.15.44:0/3701736595 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8018680 con 0x7fa8e01014b0 2023-11-16T16:49:18.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.413+0000 7fa8e4dff700 1 -- 172.21.15.44:0/3701736595 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e010bb00 con 0x7fa8e01014b0 2023-11-16T16:49:18.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.414+0000 7fa8e4dff700 1 -- 172.21.15.44:0/3701736595 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e0067b00 con 0x7fa8e01014b0 2023-11-16T16:49:18.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.417+0000 7fa8d77fe700 1 -- 172.21.15.44:0/3701736595 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa8c80105a0 con 0x7fa8e01014b0 2023-11-16T16:49:18.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.418+0000 7fa8d77fe700 1 --2- 172.21.15.44:0/3701736595 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa8cc0a11d0 0x7fa8cc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:18.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.418+0000 7fa8d77fe700 1 -- 172.21.15.44:0/3701736595 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fa8c8014070 con 0x7fa8e01014b0 2023-11-16T16:49:18.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.418+0000 7fa8de59c700 1 --2- 172.21.15.44:0/3701736595 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa8cc0a11d0 0x7fa8cc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:18.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.419+0000 7fa8de59c700 1 --2- 172.21.15.44:0/3701736595 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa8cc0a11d0 0x7fa8cc0a3680 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0101310 tx=0x7fa8d0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:18.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.422+0000 7fa8d77fe700 1 -- 172.21.15.44:0/3701736595 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa8c8051730 con 0x7fa8e01014b0 2023-11-16T16:49:18.633 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.631+0000 7fa8e4dff700 1 -- 172.21.15.44:0/3701736595 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa8e01a0db0 con 0x7fa8e01014b0 2023-11-16T16:49:18.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.632+0000 7fa8d77fe700 1 -- 172.21.15.44:0/3701736595 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa8c80961d0 con 0x7fa8e01014b0 2023-11-16T16:49:18.634 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:49:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.634+0000 7fa8e4dff700 1 -- 172.21.15.44:0/3701736595 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa8cc0a11d0 msgr2=0x7fa8cc0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.634+0000 7fa8e4dff700 1 --2- 172.21.15.44:0/3701736595 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa8cc0a11d0 0x7fa8cc0a3680 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0101310 tx=0x7fa8d0009260 comp rx=0 tx=0).stop 2023-11-16T16:49:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.634+0000 7fa8e4dff700 1 -- 172.21.15.44:0/3701736595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e01014b0 msgr2=0x7fa8e0198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.634+0000 7fa8e4dff700 1 --2- 172.21.15.44:0/3701736595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e01014b0 0x7fa8e0198880 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800e9a0 tx=0x7fa8c800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.634+0000 7fa8e4dff700 1 -- 172.21.15.44:0/3701736595 shutdown_connections 2023-11-16T16:49:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.634+0000 7fa8e4dff700 1 --2- 172.21.15.44:0/3701736595 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fa8cc0a11d0 0x7fa8cc0a3680 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.634+0000 7fa8e4dff700 1 --2- 172.21.15.44:0/3701736595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8e0101e40 0x7fa8e019f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.634+0000 7fa8e4dff700 1 --2- 172.21.15.44:0/3701736595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e01002c0 0x7fa8e0198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.634+0000 7fa8e4dff700 1 --2- 172.21.15.44:0/3701736595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e01014b0 0x7fa8e0198880 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.634+0000 7fa8e4dff700 1 -- 172.21.15.44:0/3701736595 >> 172.21.15.44:0/3701736595 conn(0x7fa8e00fb8a0 msgr2=0x7fa8e0106d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.635+0000 7fa8e4dff700 1 -- 172.21.15.44:0/3701736595 shutdown_connections 2023-11-16T16:49:18.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.635+0000 7fa8e4dff700 1 -- 172.21.15.44:0/3701736595 wait complete. 2023-11-16T16:49:18.656 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T16:49:18.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.862+0000 7f0fee5ff700 1 -- 172.21.15.44:0/1511954678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fe81002c0 msgr2=0x7f0fe81006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:18.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.862+0000 7f0fee5ff700 1 --2- 172.21.15.44:0/1511954678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fe81002c0 0x7f0fe81006d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8009990 tx=0x7f0fd8009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.862+0000 7f0fee5ff700 1 -- 172.21.15.44:0/1511954678 shutdown_connections 2023-11-16T16:49:18.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.862+0000 7f0fee5ff700 1 --2- 172.21.15.44:0/1511954678 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fe8101e40 0x7f0fe8108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.862+0000 7f0fee5ff700 1 --2- 172.21.15.44:0/1511954678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fe81002c0 0x7f0fe81006d0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.862+0000 7f0fee5ff700 1 --2- 172.21.15.44:0/1511954678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe81014b0 0x7f0fe8101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.865 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.862+0000 7f0fee5ff700 1 -- 172.21.15.44:0/1511954678 >> 172.21.15.44:0/1511954678 conn(0x7f0fe80fb8a0 msgr2=0x7f0fe80fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:18.865 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.862+0000 7f0fee5ff700 1 -- 172.21.15.44:0/1511954678 shutdown_connections 2023-11-16T16:49:18.865 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.862+0000 7f0fee5ff700 1 -- 172.21.15.44:0/1511954678 wait complete. 2023-11-16T16:49:18.867 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.863+0000 7f0fee5ff700 1 Processor -- start 2023-11-16T16:49:18.867 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.863+0000 7f0fee5ff700 1 -- start start 2023-11-16T16:49:18.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fee5ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fe81002c0 0x7f0fe80748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:18.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fee5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe81014b0 0x7f0fe8072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:18.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fee5ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fe8101e40 0x7f0fe8073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:18.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fee5ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8073bc0 con 0x7f0fe81002c0 2023-11-16T16:49:18.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fee5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8073d30 con 0x7f0fe81014b0 2023-11-16T16:49:18.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fee5ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe81a00f0 con 0x7f0fe8101e40 2023-11-16T16:49:18.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fe7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fe81002c0 0x7f0fe80748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:18.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fe7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fe81002c0 0x7f0fe80748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51590/0 (socket says 172.21.15.44:51590) 2023-11-16T16:49:18.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fe7fff700 1 -- 172.21.15.44:0/3074605046 learned_addr learned my addr 172.21.15.44:0/3074605046 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:18.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fe77fe700 1 --2- 172.21.15.44:0/3074605046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe81014b0 0x7f0fe8072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:18.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fecb9c700 1 --2- 172.21.15.44:0/3074605046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fe8101e40 0x7f0fe8073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:18.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fe7fff700 1 -- 172.21.15.44:0/3074605046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fe8101e40 msgr2=0x7f0fe8073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:18.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fe7fff700 1 --2- 172.21.15.44:0/3074605046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fe8101e40 0x7f0fe8073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fe7fff700 1 -- 172.21.15.44:0/3074605046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe81014b0 msgr2=0x7f0fe8072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:18.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fe7fff700 1 --2- 172.21.15.44:0/3074605046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe81014b0 0x7f0fe8072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:18.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fe7fff700 1 -- 172.21.15.44:0/3074605046 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc009520 con 0x7f0fe81002c0 2023-11-16T16:49:18.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.864+0000 7f0fe7fff700 1 --2- 172.21.15.44:0/3074605046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fe81002c0 0x7f0fe80748a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8004730 tx=0x7f0fd8004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:18.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.865+0000 7f0fe57fa700 1 -- 172.21.15.44:0/3074605046 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd801d070 con 0x7f0fe81002c0 2023-11-16T16:49:18.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.865+0000 7f0fe57fa700 1 -- 172.21.15.44:0/3074605046 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fd8021410 con 0x7f0fe81002c0 2023-11-16T16:49:18.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.865+0000 7f0fe57fa700 1 -- 172.21.15.44:0/3074605046 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8017650 con 0x7f0fe81002c0 2023-11-16T16:49:18.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.865+0000 7f0fee5ff700 1 -- 172.21.15.44:0/3074605046 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd80095f0 con 0x7f0fe81002c0 2023-11-16T16:49:18.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.865+0000 7f0fee5ff700 1 -- 172.21.15.44:0/3074605046 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe810bc90 con 0x7f0fe81002c0 2023-11-16T16:49:18.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.866+0000 7f0fee5ff700 1 -- 172.21.15.44:0/3074605046 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe810be20 con 0x7f0fe81002c0 2023-11-16T16:49:18.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.868+0000 7f0fe57fa700 1 -- 172.21.15.44:0/3074605046 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0fd8021810 con 0x7f0fe81002c0 2023-11-16T16:49:18.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.869+0000 7f0fe57fa700 1 --2- 172.21.15.44:0/3074605046 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0fd00a11d0 0x7f0fd00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:18.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.869+0000 7f0fe57fa700 1 -- 172.21.15.44:0/3074605046 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f0fd80ce160 con 0x7f0fe81002c0 2023-11-16T16:49:18.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.870+0000 7f0fe77fe700 1 --2- 172.21.15.44:0/3074605046 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0fd00a11d0 0x7f0fd00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:18.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.870+0000 7f0fe77fe700 1 --2- 172.21.15.44:0/3074605046 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0fd00a11d0 0x7f0fd00a3680 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009df0 tx=0x7f0fdc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:18.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:18.873+0000 7f0fe57fa700 1 -- 172.21.15.44:0/3074605046 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0fd8097540 con 0x7f0fe81002c0 2023-11-16T16:49:18.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:18 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/603118946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:18.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:18 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3597517351' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:18.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:18 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3701736595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:18.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:18 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/603118946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:18.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:18 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3597517351' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:18.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:18 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3701736595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:19.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.079+0000 7f0fee5ff700 1 -- 172.21.15.44:0/3074605046 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0fe802cfe0 con 0x7f0fe81002c0 2023-11-16T16:49:19.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.079+0000 7f0fe57fa700 1 -- 172.21.15.44:0/3074605046 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0fd809ab60 con 0x7f0fe81002c0 2023-11-16T16:49:19.081 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2023-11-16T16:49:19.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.082+0000 7f0fee5ff700 1 -- 172.21.15.44:0/3074605046 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0fd00a11d0 msgr2=0x7f0fd00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:19.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.082+0000 7f0fee5ff700 1 --2- 172.21.15.44:0/3074605046 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0fd00a11d0 0x7f0fd00a3680 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009df0 tx=0x7f0fdc009260 comp rx=0 tx=0).stop 2023-11-16T16:49:19.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.082+0000 7f0fee5ff700 1 -- 172.21.15.44:0/3074605046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fe81002c0 msgr2=0x7f0fe80748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:19.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.082+0000 7f0fee5ff700 1 --2- 172.21.15.44:0/3074605046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fe81002c0 0x7f0fe80748a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8004730 tx=0x7f0fd8004760 comp rx=0 tx=0).stop 2023-11-16T16:49:19.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.082+0000 7f0fee5ff700 1 -- 172.21.15.44:0/3074605046 shutdown_connections 2023-11-16T16:49:19.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.082+0000 7f0fee5ff700 1 --2- 172.21.15.44:0/3074605046 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0fd00a11d0 0x7f0fd00a3680 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.082+0000 7f0fee5ff700 1 --2- 172.21.15.44:0/3074605046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fe8101e40 0x7f0fe8073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.082+0000 7f0fee5ff700 1 --2- 172.21.15.44:0/3074605046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fe81002c0 0x7f0fe80748a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.082+0000 7f0fee5ff700 1 --2- 172.21.15.44:0/3074605046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe81014b0 0x7f0fe8072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.082+0000 7f0fee5ff700 1 -- 172.21.15.44:0/3074605046 >> 172.21.15.44:0/3074605046 conn(0x7f0fe80fb8a0 msgr2=0x7f0fe8104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:19.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.082+0000 7f0fee5ff700 1 -- 172.21.15.44:0/3074605046 shutdown_connections 2023-11-16T16:49:19.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.082+0000 7f0fee5ff700 1 -- 172.21.15.44:0/3074605046 wait complete. 2023-11-16T16:49:19.101 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:18 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/603118946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:19.101 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:18 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3597517351' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:19.101 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:18 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3701736595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:19.103 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T16:49:19.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.308+0000 7f7be3fff700 1 -- 172.21.15.44:0/289779715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7be40734c0 msgr2=0x7f7be4073930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:19.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.308+0000 7f7be3fff700 1 --2- 172.21.15.44:0/289779715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7be40734c0 0x7f7be4073930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009990 tx=0x7f7bd8009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.309+0000 7f7be3fff700 1 -- 172.21.15.44:0/289779715 shutdown_connections 2023-11-16T16:49:19.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.309+0000 7f7be3fff700 1 --2- 172.21.15.44:0/289779715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7be4107ed0 0x7f7be410a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.309+0000 7f7be3fff700 1 --2- 172.21.15.44:0/289779715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7be4074a90 0x7f7be4072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.309+0000 7f7be3fff700 1 --2- 172.21.15.44:0/289779715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7be40734c0 0x7f7be4073930 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.309+0000 7f7be3fff700 1 -- 172.21.15.44:0/289779715 >> 172.21.15.44:0/289779715 conn(0x7f7be40fb650 msgr2=0x7f7be40fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:19.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.309+0000 7f7be3fff700 1 -- 172.21.15.44:0/289779715 shutdown_connections 2023-11-16T16:49:19.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.309+0000 7f7be3fff700 1 -- 172.21.15.44:0/289779715 wait complete. 2023-11-16T16:49:19.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.310+0000 7f7be3fff700 1 Processor -- start 2023-11-16T16:49:19.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.310+0000 7f7be3fff700 1 -- start start 2023-11-16T16:49:19.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.310+0000 7f7be3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7be40734c0 0x7f7be4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:19.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.310+0000 7f7be3fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7be4074a90 0x7f7be4198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:19.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.310+0000 7f7be3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7be4107ed0 0x7f7be419f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:19.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.310+0000 7f7be3fff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be419fb70 con 0x7f7be4107ed0 2023-11-16T16:49:19.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.310+0000 7f7be3fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be419fcb0 con 0x7f7be40734c0 2023-11-16T16:49:19.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.310+0000 7f7be3fff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be419fe40 con 0x7f7be4074a90 2023-11-16T16:49:19.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.310+0000 7f7be37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7be4107ed0 0x7f7be419f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:19.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.310+0000 7f7be2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7be40734c0 0x7f7be4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:19.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.310+0000 7f7be2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7be40734c0 0x7f7be4197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:53016/0 (socket says 172.21.15.44:53016) 2023-11-16T16:49:19.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.310+0000 7f7be2ffd700 1 -- 172.21.15.44:0/2496566965 learned_addr learned my addr 172.21.15.44:0/2496566965 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:19.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.310+0000 7f7be37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7be4107ed0 0x7f7be419f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51600/0 (socket says 172.21.15.44:51600) 2023-11-16T16:49:19.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.311+0000 7f7be27fc700 1 --2- 172.21.15.44:0/2496566965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7be4074a90 0x7f7be4198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:19.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.311+0000 7f7be37fe700 1 -- 172.21.15.44:0/2496566965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7be4074a90 msgr2=0x7f7be4198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:19.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.311+0000 7f7be37fe700 1 --2- 172.21.15.44:0/2496566965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7be4074a90 0x7f7be4198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.311+0000 7f7be37fe700 1 -- 172.21.15.44:0/2496566965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7be40734c0 msgr2=0x7f7be4197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:19.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.311+0000 7f7be37fe700 1 --2- 172.21.15.44:0/2496566965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7be40734c0 0x7f7be4197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.311+0000 7f7be37fe700 1 -- 172.21.15.44:0/2496566965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd80095f0 con 0x7f7be4107ed0 2023-11-16T16:49:19.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.311+0000 7f7be27fc700 1 --2- 172.21.15.44:0/2496566965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7be4074a90 0x7f7be4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:49:19.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.311+0000 7f7be2ffd700 1 --2- 172.21.15.44:0/2496566965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7be40734c0 0x7f7be4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:19.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.311+0000 7f7be37fe700 1 --2- 172.21.15.44:0/2496566965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7be4107ed0 0x7f7be419f5a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc00efc0 tx=0x7f7bdc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:19.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.311+0000 7f7bcbfff700 1 -- 172.21.15.44:0/2496566965 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc018070 con 0x7f7be4107ed0 2023-11-16T16:49:19.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.311+0000 7f7bcbfff700 1 -- 172.21.15.44:0/2496566965 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7bdc00f040 con 0x7f7be4107ed0 2023-11-16T16:49:19.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.311+0000 7f7be3fff700 1 -- 172.21.15.44:0/2496566965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be410dc30 con 0x7f7be4107ed0 2023-11-16T16:49:19.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.311+0000 7f7bcbfff700 1 -- 172.21.15.44:0/2496566965 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc00cdf0 con 0x7f7be4107ed0 2023-11-16T16:49:19.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.312+0000 7f7be3fff700 1 -- 172.21.15.44:0/2496566965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be410daa0 con 0x7f7be4107ed0 2023-11-16T16:49:19.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.313+0000 7f7be3fff700 1 -- 172.21.15.44:0/2496566965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be4106660 con 0x7f7be4107ed0 2023-11-16T16:49:19.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.314+0000 7f7bcbfff700 1 -- 172.21.15.44:0/2496566965 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7bdc007500 con 0x7f7be4107ed0 2023-11-16T16:49:19.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.316+0000 7f7bcbfff700 1 --2- 172.21.15.44:0/2496566965 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7bcc0a1150 0x7f7bcc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:19.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.316+0000 7f7bcbfff700 1 -- 172.21.15.44:0/2496566965 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f7bdc0cc920 con 0x7f7be4107ed0 2023-11-16T16:49:19.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.316+0000 7f7be2ffd700 1 --2- 172.21.15.44:0/2496566965 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7bcc0a1150 0x7f7bcc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:19.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.317+0000 7f7be2ffd700 1 --2- 172.21.15.44:0/2496566965 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7bcc0a1150 0x7f7bcc0a3600 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4005950 tx=0x7f7bd4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:19.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.320+0000 7f7bcbfff700 1 -- 172.21.15.44:0/2496566965 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7bdc095d00 con 0x7f7be4107ed0 2023-11-16T16:49:19.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.531+0000 7f7be3fff700 1 -- 172.21.15.44:0/2496566965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f7be4067980 con 0x7f7be4107ed0 2023-11-16T16:49:19.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.531+0000 7f7bcbfff700 1 -- 172.21.15.44:0/2496566965 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7bdc099320 con 0x7f7be4107ed0 2023-11-16T16:49:19.533 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:49:19.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.533+0000 7f7be3fff700 1 -- 172.21.15.44:0/2496566965 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7bcc0a1150 msgr2=0x7f7bcc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:19.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.533+0000 7f7be3fff700 1 --2- 172.21.15.44:0/2496566965 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7bcc0a1150 0x7f7bcc0a3600 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4005950 tx=0x7f7bd4009260 comp rx=0 tx=0).stop 2023-11-16T16:49:19.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.533+0000 7f7be3fff700 1 -- 172.21.15.44:0/2496566965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7be4107ed0 msgr2=0x7f7be419f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:19.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.533+0000 7f7be3fff700 1 --2- 172.21.15.44:0/2496566965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7be4107ed0 0x7f7be419f5a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc00efc0 tx=0x7f7bdc00c3a0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.533+0000 7f7be3fff700 1 -- 172.21.15.44:0/2496566965 shutdown_connections 2023-11-16T16:49:19.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.533+0000 7f7be3fff700 1 --2- 172.21.15.44:0/2496566965 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7bcc0a1150 0x7f7bcc0a3600 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.533+0000 7f7be3fff700 1 --2- 172.21.15.44:0/2496566965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7be4107ed0 0x7f7be419f5a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.533+0000 7f7be3fff700 1 --2- 172.21.15.44:0/2496566965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7be40734c0 0x7f7be4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.533+0000 7f7be3fff700 1 --2- 172.21.15.44:0/2496566965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7be4074a90 0x7f7be4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.533+0000 7f7be3fff700 1 -- 172.21.15.44:0/2496566965 >> 172.21.15.44:0/2496566965 conn(0x7f7be40fb650 msgr2=0x7f7be40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:19.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.534+0000 7f7be3fff700 1 -- 172.21.15.44:0/2496566965 shutdown_connections 2023-11-16T16:49:19.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.534+0000 7f7be3fff700 1 -- 172.21.15.44:0/2496566965 wait complete. 2023-11-16T16:49:19.554 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:49:19.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.759+0000 7f2aaf1ff700 1 -- 172.21.15.44:0/4165927190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8101f10 msgr2=0x7f2aa81087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:19.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.759+0000 7f2aaf1ff700 1 --2- 172.21.15.44:0/4165927190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8101f10 0x7f2aa81087b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000b0a0 tx=0x7f2aa000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.759+0000 7f2aaf1ff700 1 -- 172.21.15.44:0/4165927190 shutdown_connections 2023-11-16T16:49:19.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.759+0000 7f2aaf1ff700 1 --2- 172.21.15.44:0/4165927190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8101f10 0x7f2aa81087b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.759+0000 7f2aaf1ff700 1 --2- 172.21.15.44:0/4165927190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa81002d0 0x7f2aa81006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.759+0000 7f2aaf1ff700 1 --2- 172.21.15.44:0/4165927190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2aa8101560 0x7f2aa81019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.759+0000 7f2aaf1ff700 1 -- 172.21.15.44:0/4165927190 >> 172.21.15.44:0/4165927190 conn(0x7f2aa80fb930 msgr2=0x7f2aa80fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:19.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.759+0000 7f2aaf1ff700 1 -- 172.21.15.44:0/4165927190 shutdown_connections 2023-11-16T16:49:19.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.759+0000 7f2aaf1ff700 1 -- 172.21.15.44:0/4165927190 wait complete. 2023-11-16T16:49:19.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.760+0000 7f2aaf1ff700 1 Processor -- start 2023-11-16T16:49:19.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.760+0000 7f2aaf1ff700 1 -- start start 2023-11-16T16:49:19.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aaf1ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa81002d0 0x7f2aa8198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:19.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aaf1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa8101560 0x7f2aa81987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:19.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aaf1ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2aa8101f10 0x7f2aa819f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:19.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aaf1ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa819fe00 con 0x7f2aa81002d0 2023-11-16T16:49:19.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aaf1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa819ff40 con 0x7f2aa8101560 2023-11-16T16:49:19.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aaf1ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa81a00d0 con 0x7f2aa8101f10 2023-11-16T16:49:19.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aacf9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa81002d0 0x7f2aa8198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:19.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aacf9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa81002d0 0x7f2aa8198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51606/0 (socket says 172.21.15.44:51606) 2023-11-16T16:49:19.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aacf9b700 1 -- 172.21.15.44:0/1280237037 learned_addr learned my addr 172.21.15.44:0/1280237037 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:19.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aa7fff700 1 --2- 172.21.15.44:0/1280237037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa8101560 0x7f2aa81987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:19.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aad79c700 1 --2- 172.21.15.44:0/1280237037 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2aa8101f10 0x7f2aa819f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:19.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aacf9b700 1 -- 172.21.15.44:0/1280237037 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2aa8101f10 msgr2=0x7f2aa819f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:19.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aacf9b700 1 --2- 172.21.15.44:0/1280237037 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2aa8101f10 0x7f2aa819f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aacf9b700 1 -- 172.21.15.44:0/1280237037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa8101560 msgr2=0x7f2aa81987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:19.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aacf9b700 1 --2- 172.21.15.44:0/1280237037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa8101560 0x7f2aa81987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:19.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aacf9b700 1 -- 172.21.15.44:0/1280237037 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c009520 con 0x7f2aa81002d0 2023-11-16T16:49:19.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.761+0000 7f2aa7fff700 1 --2- 172.21.15.44:0/1280237037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa8101560 0x7f2aa81987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:19.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.762+0000 7f2aacf9b700 1 --2- 172.21.15.44:0/1280237037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa81002d0 0x7f2aa8198280 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c00eb30 tx=0x7f2a9c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:19.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.762+0000 7f2aa5ffb700 1 -- 172.21.15.44:0/1280237037 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c009880 con 0x7f2aa81002d0 2023-11-16T16:49:19.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.762+0000 7f2aa5ffb700 1 -- 172.21.15.44:0/1280237037 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a9c0044e0 con 0x7f2aa81002d0 2023-11-16T16:49:19.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.762+0000 7f2aa5ffb700 1 -- 172.21.15.44:0/1280237037 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c003d80 con 0x7f2aa81002d0 2023-11-16T16:49:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.762+0000 7f2aaf1ff700 1 -- 172.21.15.44:0/1280237037 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a94009520 con 0x7f2aa81002d0 2023-11-16T16:49:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.762+0000 7f2aaf1ff700 1 -- 172.21.15.44:0/1280237037 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa0009d00 con 0x7f2aa81002d0 2023-11-16T16:49:19.769 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.763+0000 7f2aaf1ff700 1 -- 172.21.15.44:0/1280237037 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa810bb30 con 0x7f2aa81002d0 2023-11-16T16:49:19.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.765+0000 7f2aa5ffb700 1 -- 172.21.15.44:0/1280237037 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2a9c003f20 con 0x7f2aa81002d0 2023-11-16T16:49:19.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.768+0000 7f2aa5ffb700 1 --2- 172.21.15.44:0/1280237037 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2a980a11d0 0x7f2a980a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:19.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.768+0000 7f2aa5ffb700 1 -- 172.21.15.44:0/1280237037 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f2a9c014070 con 0x7f2aa81002d0 2023-11-16T16:49:19.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.768+0000 7f2aa7fff700 1 --2- 172.21.15.44:0/1280237037 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2a980a11d0 0x7f2a980a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:19.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.768+0000 7f2aa7fff700 1 --2- 172.21.15.44:0/1280237037 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2a980a11d0 0x7f2a980a3680 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2a94009fd0 tx=0x7f2a94009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:19.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:19.771+0000 7f2aa5ffb700 1 -- 172.21.15.44:0/1280237037 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2a9c096530 con 0x7f2aa81002d0 2023-11-16T16:49:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:19 smithi044 ceph-mon[128750]: pgmap v339: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2023-11-16T16:49:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:19 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3074605046' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:19 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2496566965' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:19.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:19 smithi077 ceph-mon[131204]: pgmap v339: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2023-11-16T16:49:19.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:19 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3074605046' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:19.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:19 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2496566965' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:20.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.025+0000 7f2aaf1ff700 1 -- 172.21.15.44:0/1280237037 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2aa810bcc0 con 0x7f2aa81002d0 2023-11-16T16:49:20.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.026+0000 7f2aa5ffb700 1 -- 172.21.15.44:0/1280237037 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f2a9c099b50 con 0x7f2aa81002d0 2023-11-16T16:49:20.028 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:20.029 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:49:20.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.028+0000 7f2aaf1ff700 1 -- 172.21.15.44:0/1280237037 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2a980a11d0 msgr2=0x7f2a980a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.028+0000 7f2aaf1ff700 1 --2- 172.21.15.44:0/1280237037 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2a980a11d0 0x7f2a980a3680 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2a94009fd0 tx=0x7f2a94009260 comp rx=0 tx=0).stop 2023-11-16T16:49:20.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.028+0000 7f2aaf1ff700 1 -- 172.21.15.44:0/1280237037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa81002d0 msgr2=0x7f2aa8198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.028+0000 7f2aaf1ff700 1 --2- 172.21.15.44:0/1280237037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa81002d0 0x7f2aa8198280 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c00eb30 tx=0x7f2a9c00ee40 comp rx=0 tx=0).stop 2023-11-16T16:49:20.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.029+0000 7f2aaf1ff700 1 -- 172.21.15.44:0/1280237037 shutdown_connections 2023-11-16T16:49:20.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.029+0000 7f2aaf1ff700 1 --2- 172.21.15.44:0/1280237037 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2a980a11d0 0x7f2a980a3680 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.029+0000 7f2aaf1ff700 1 --2- 172.21.15.44:0/1280237037 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2aa8101f10 0x7f2aa819f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.029+0000 7f2aaf1ff700 1 --2- 172.21.15.44:0/1280237037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa81002d0 0x7f2aa8198280 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.029+0000 7f2aaf1ff700 1 --2- 172.21.15.44:0/1280237037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa8101560 0x7f2aa81987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.029+0000 7f2aaf1ff700 1 -- 172.21.15.44:0/1280237037 >> 172.21.15.44:0/1280237037 conn(0x7f2aa80fb930 msgr2=0x7f2aa8106dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:20.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.029+0000 7f2aaf1ff700 1 -- 172.21.15.44:0/1280237037 shutdown_connections 2023-11-16T16:49:20.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.029+0000 7f2aaf1ff700 1 -- 172.21.15.44:0/1280237037 wait complete. 2023-11-16T16:49:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:49:20.050 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:49:20.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:19 smithi176 ceph-mon[131630]: pgmap v339: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2023-11-16T16:49:20.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:19 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3074605046' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:20.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:19 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2496566965' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:20.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.252+0000 7fb7efdff700 1 -- 172.21.15.44:0/999504876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7e8100290 msgr2=0x7fb7e81006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.252+0000 7fb7efdff700 1 --2- 172.21.15.44:0/999504876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7e8100290 0x7fb7e81006a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb7e40098f0 tx=0x7fb7e4009c00 comp rx=0 tx=0).stop 2023-11-16T16:49:20.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.252+0000 7fb7efdff700 1 -- 172.21.15.44:0/999504876 shutdown_connections 2023-11-16T16:49:20.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.252+0000 7fb7efdff700 1 --2- 172.21.15.44:0/999504876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7e8101e10 0x7fb7e81086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.252+0000 7fb7efdff700 1 --2- 172.21.15.44:0/999504876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7e8100290 0x7fb7e81006a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.252+0000 7fb7efdff700 1 --2- 172.21.15.44:0/999504876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7e8101480 0x7fb7e81018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.252+0000 7fb7efdff700 1 -- 172.21.15.44:0/999504876 >> 172.21.15.44:0/999504876 conn(0x7fb7e80fb870 msgr2=0x7fb7e80fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:20.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.252+0000 7fb7efdff700 1 -- 172.21.15.44:0/999504876 shutdown_connections 2023-11-16T16:49:20.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.252+0000 7fb7efdff700 1 -- 172.21.15.44:0/999504876 wait complete. 2023-11-16T16:49:20.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.253+0000 7fb7efdff700 1 Processor -- start 2023-11-16T16:49:20.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.254+0000 7fb7efdff700 1 -- start start 2023-11-16T16:49:20.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.254+0000 7fb7efdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7e8100290 0x7fb7e8198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.254+0000 7fb7efdff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7e8101480 0x7fb7e81986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.254+0000 7fb7efdff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7e8101e10 0x7fb7e819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.254+0000 7fb7efdff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e819fd00 con 0x7fb7e8101480 2023-11-16T16:49:20.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.254+0000 7fb7efdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e819fe40 con 0x7fb7e8100290 2023-11-16T16:49:20.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.254+0000 7fb7efdff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e819ffd0 con 0x7fb7e8101e10 2023-11-16T16:49:20.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.254+0000 7fb7ed39a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7e8101480 0x7fb7e81986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.254+0000 7fb7edb9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7e8100290 0x7fb7e8198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.254+0000 7fb7ed39a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7e8101480 0x7fb7e81986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51628/0 (socket says 172.21.15.44:51628) 2023-11-16T16:49:20.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.254+0000 7fb7ed39a700 1 -- 172.21.15.44:0/2860136410 learned_addr learned my addr 172.21.15.44:0/2860136410 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:20.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.254+0000 7fb7ee39c700 1 --2- 172.21.15.44:0/2860136410 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7e8101e10 0x7fb7e819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.255+0000 7fb7edb9b700 1 -- 172.21.15.44:0/2860136410 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7e8101e10 msgr2=0x7fb7e819f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.255+0000 7fb7edb9b700 1 --2- 172.21.15.44:0/2860136410 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7e8101e10 0x7fb7e819f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.255+0000 7fb7edb9b700 1 -- 172.21.15.44:0/2860136410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7e8101480 msgr2=0x7fb7e81986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.255+0000 7fb7edb9b700 1 --2- 172.21.15.44:0/2860136410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7e8101480 0x7fb7e81986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.255+0000 7fb7edb9b700 1 -- 172.21.15.44:0/2860136410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e0009c40 con 0x7fb7e8100290 2023-11-16T16:49:20.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.255+0000 7fb7ed39a700 1 --2- 172.21.15.44:0/2860136410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7e8101480 0x7fb7e81986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:20.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.255+0000 7fb7edb9b700 1 --2- 172.21.15.44:0/2860136410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7e8100290 0x7fb7e8198180 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4000c00 tx=0x7fb7e400b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:20.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.255+0000 7fb7deffd700 1 -- 172.21.15.44:0/2860136410 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e401c070 con 0x7fb7e8100290 2023-11-16T16:49:20.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.255+0000 7fb7deffd700 1 -- 172.21.15.44:0/2860136410 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7e400f7d0 con 0x7fb7e8100290 2023-11-16T16:49:20.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.255+0000 7fb7efdff700 1 -- 172.21.15.44:0/2860136410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e4009520 con 0x7fb7e8100290 2023-11-16T16:49:20.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.255+0000 7fb7deffd700 1 -- 172.21.15.44:0/2860136410 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e40039b0 con 0x7fb7e8100290 2023-11-16T16:49:20.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.255+0000 7fb7efdff700 1 -- 172.21.15.44:0/2860136410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e810bba0 con 0x7fb7e8100290 2023-11-16T16:49:20.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.257+0000 7fb7efdff700 1 -- 172.21.15.44:0/2860136410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e810bd30 con 0x7fb7e8100290 2023-11-16T16:49:20.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.258+0000 7fb7deffd700 1 -- 172.21.15.44:0/2860136410 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb7e4020070 con 0x7fb7e8100290 2023-11-16T16:49:20.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.260+0000 7fb7deffd700 1 --2- 172.21.15.44:0/2860136410 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb7d40a1150 0x7fb7d40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.260+0000 7fb7deffd700 1 -- 172.21.15.44:0/2860136410 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fb7e40cdec0 con 0x7fb7e8100290 2023-11-16T16:49:20.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.260+0000 7fb7ed39a700 1 --2- 172.21.15.44:0/2860136410 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb7d40a1150 0x7fb7d40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.263 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.260+0000 7fb7ed39a700 1 --2- 172.21.15.44:0/2860136410 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb7d40a1150 0x7fb7d40a3600 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8005950 tx=0x7fb7d8009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:20.263 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.262+0000 7fb7deffd700 1 -- 172.21.15.44:0/2860136410 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb7e40972a0 con 0x7fb7e8100290 2023-11-16T16:49:20.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.475+0000 7fb7efdff700 1 -- 172.21.15.44:0/2860136410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7e802d010 con 0x7fb7e8100290 2023-11-16T16:49:20.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.477+0000 7fb7deffd700 1 -- 172.21.15.44:0/2860136410 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24685 (secure 0 0 0) 0x7fb7e409a8c0 con 0x7fb7e8100290 2023-11-16T16:49:20.480 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:20.480 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:48:44.613706+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:49:20.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.480+0000 7fb7efdff700 1 -- 172.21.15.44:0/2860136410 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb7d40a1150 msgr2=0x7fb7d40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.480+0000 7fb7efdff700 1 --2- 172.21.15.44:0/2860136410 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb7d40a1150 0x7fb7d40a3600 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8005950 tx=0x7fb7d8009310 comp rx=0 tx=0).stop 2023-11-16T16:49:20.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.480+0000 7fb7efdff700 1 -- 172.21.15.44:0/2860136410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7e8100290 msgr2=0x7fb7e8198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.480+0000 7fb7efdff700 1 --2- 172.21.15.44:0/2860136410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7e8100290 0x7fb7e8198180 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4000c00 tx=0x7fb7e400b790 comp rx=0 tx=0).stop 2023-11-16T16:49:20.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.480+0000 7fb7efdff700 1 -- 172.21.15.44:0/2860136410 shutdown_connections 2023-11-16T16:49:20.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.480+0000 7fb7efdff700 1 --2- 172.21.15.44:0/2860136410 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb7d40a1150 0x7fb7d40a3600 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.480+0000 7fb7efdff700 1 --2- 172.21.15.44:0/2860136410 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7e8101e10 0x7fb7e819f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.480+0000 7fb7efdff700 1 --2- 172.21.15.44:0/2860136410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7e8100290 0x7fb7e8198180 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.480+0000 7fb7efdff700 1 --2- 172.21.15.44:0/2860136410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7e8101480 0x7fb7e81986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.480+0000 7fb7efdff700 1 -- 172.21.15.44:0/2860136410 >> 172.21.15.44:0/2860136410 conn(0x7fb7e80fb870 msgr2=0x7fb7e8106df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:20.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.480+0000 7fb7efdff700 1 -- 172.21.15.44:0/2860136410 shutdown_connections 2023-11-16T16:49:20.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.480+0000 7fb7efdff700 1 -- 172.21.15.44:0/2860136410 wait complete. 2023-11-16T16:49:20.501 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-11-16T16:47:18.150861+0000', 'modified': '2023-11-16T16:47:50.656307+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24551}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24538': {'gid': 24538, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.176:6839/2828920421', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6838', 'nonce': 2828920421}, {'type': 'v1', 'addr': '172.21.15.176:6839', 'nonce': 2828920421}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24551': {'gid': 24551, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6839/1948116745', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6838', 'nonce': 1948116745}, {'type': 'v1', 'addr': '172.21.15.77:6839', 'nonce': 1948116745}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-16T16:49:20.501 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-16T16:49:20.502 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-16T16:49:20.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.707+0000 7fb5711ff700 1 -- 172.21.15.44:0/3922732729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb56c074b30 msgr2=0x7fb56c072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.707+0000 7fb5711ff700 1 --2- 172.21.15.44:0/3922732729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb56c074b30 0x7fb56c072f80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb55c0098f0 tx=0x7fb55c009c00 comp rx=0 tx=0).stop 2023-11-16T16:49:20.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.707+0000 7fb5711ff700 1 -- 172.21.15.44:0/3922732729 shutdown_connections 2023-11-16T16:49:20.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.707+0000 7fb5711ff700 1 --2- 172.21.15.44:0/3922732729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb56c1080f0 0x7fb56c10a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.707+0000 7fb5711ff700 1 --2- 172.21.15.44:0/3922732729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb56c074b30 0x7fb56c072f80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.707+0000 7fb5711ff700 1 --2- 172.21.15.44:0/3922732729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb56c073550 0x7fb56c0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.707+0000 7fb5711ff700 1 -- 172.21.15.44:0/3922732729 >> 172.21.15.44:0/3922732729 conn(0x7fb56c0fb810 msgr2=0x7fb56c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:20.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.707+0000 7fb5711ff700 1 -- 172.21.15.44:0/3922732729 shutdown_connections 2023-11-16T16:49:20.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.707+0000 7fb5711ff700 1 -- 172.21.15.44:0/3922732729 wait complete. 2023-11-16T16:49:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.708+0000 7fb5711ff700 1 Processor -- start 2023-11-16T16:49:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.708+0000 7fb5711ff700 1 -- start start 2023-11-16T16:49:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.708+0000 7fb5711ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb56c073550 0x7fb56c198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.708+0000 7fb5711ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb56c074b30 0x7fb56c1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.708+0000 7fb5711ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb56c1080f0 0x7fb56c19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.708+0000 7fb5711ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c19fdf0 con 0x7fb56c1080f0 2023-11-16T16:49:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.708+0000 7fb5711ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c19ff30 con 0x7fb56c073550 2023-11-16T16:49:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.708+0000 7fb5711ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c1a00c0 con 0x7fb56c074b30 2023-11-16T16:49:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.709+0000 7fb56b59e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb56c1080f0 0x7fb56c19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.709+0000 7fb56ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb56c073550 0x7fb56c198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.709+0000 7fb56ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb56c073550 0x7fb56c198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:53050/0 (socket says 172.21.15.44:53050) 2023-11-16T16:49:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.709+0000 7fb56ad9d700 1 -- 172.21.15.44:0/2677841528 learned_addr learned my addr 172.21.15.44:0/2677841528 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.709+0000 7fb563fff700 1 --2- 172.21.15.44:0/2677841528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb56c074b30 0x7fb56c1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.709+0000 7fb56ad9d700 1 -- 172.21.15.44:0/2677841528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb56c074b30 msgr2=0x7fb56c1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.709+0000 7fb56ad9d700 1 --2- 172.21.15.44:0/2677841528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb56c074b30 0x7fb56c1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.709+0000 7fb56ad9d700 1 -- 172.21.15.44:0/2677841528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb56c1080f0 msgr2=0x7fb56c19f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.709+0000 7fb56ad9d700 1 --2- 172.21.15.44:0/2677841528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb56c1080f0 0x7fb56c19f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.709+0000 7fb56ad9d700 1 -- 172.21.15.44:0/2677841528 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55c009520 con 0x7fb56c073550 2023-11-16T16:49:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.709+0000 7fb56b59e700 1 --2- 172.21.15.44:0/2677841528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb56c1080f0 0x7fb56c19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.709+0000 7fb56ad9d700 1 --2- 172.21.15.44:0/2677841528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb56c073550 0x7fb56c198270 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009fd0 tx=0x7fb55c00b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.710+0000 7fb568d99700 1 -- 172.21.15.44:0/2677841528 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c01c070 con 0x7fb56c073550 2023-11-16T16:49:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.710+0000 7fb5711ff700 1 -- 172.21.15.44:0/2677841528 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56c10de00 con 0x7fb56c073550 2023-11-16T16:49:20.715 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.710+0000 7fb568d99700 1 -- 172.21.15.44:0/2677841528 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb55c00f7d0 con 0x7fb56c073550 2023-11-16T16:49:20.715 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.710+0000 7fb568d99700 1 -- 172.21.15.44:0/2677841528 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c003a40 con 0x7fb56c073550 2023-11-16T16:49:20.715 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.710+0000 7fb5711ff700 1 -- 172.21.15.44:0/2677841528 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56c10dc70 con 0x7fb56c073550 2023-11-16T16:49:20.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.713+0000 7fb568d99700 1 -- 172.21.15.44:0/2677841528 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb55c020030 con 0x7fb56c073550 2023-11-16T16:49:20.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.714+0000 7fb584afdb80 1 Processor -- start 2023-11-16T16:49:20.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.714+0000 7fb584afdb80 1 -- start start 2023-11-16T16:49:20.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.714+0000 7fb568d99700 1 --2- 172.21.15.44:0/2677841528 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb5580a11d0 0x7fb5580a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.714+0000 7fb584afdb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda01bb0 0x55dbeda01fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.714+0000 7fb584afdb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 0x55dbeda02ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.714+0000 7fb584afdb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 0x55dbeda03530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.714+0000 7fb568d99700 1 -- 172.21.15.44:0/2677841528 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fb55c0cdc00 con 0x7fb56c073550 2023-11-16T16:49:20.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.714+0000 7fb584afdb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbed907010 con 0x55dbeda030c0 2023-11-16T16:49:20.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.714+0000 7fb584afdb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbed959850 con 0x55dbeda01bb0 2023-11-16T16:49:20.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.714+0000 7fb584afdb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbed952ce0 con 0x55dbeda02500 2023-11-16T16:49:20.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.715+0000 7fb56b59e700 1 --2- 172.21.15.44:0/2677841528 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb5580a11d0 0x7fb5580a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.715+0000 7fb56b59e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 0x55dbeda03530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.715+0000 7fb56b59e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 0x55dbeda03530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51658/0 (socket says 172.21.15.44:51658) 2023-11-16T16:49:20.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.715+0000 7fb56b59e700 1 -- 172.21.15.44:0/3301476148 learned_addr learned my addr 172.21.15.44:0/3301476148 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:20.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.715+0000 7fb563fff700 1 --2- 172.21.15.44:0/3301476148 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 0x55dbeda02ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.715+0000 7fb56ad9d700 1 --2- 172.21.15.44:0/3301476148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda01bb0 0x55dbeda01fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.715+0000 7fb56ad9d700 1 -- 172.21.15.44:0/3301476148 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 msgr2=0x55dbeda02ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:20.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.715+0000 7fb56ad9d700 1 --2- 172.21.15.44:0/3301476148 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 0x55dbeda02ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.715+0000 7fb56ad9d700 1 -- 172.21.15.44:0/3301476148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 msgr2=0x55dbeda03530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:20.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.715+0000 7fb56ad9d700 1 --2- 172.21.15.44:0/3301476148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 0x55dbeda03530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.715+0000 7fb56ad9d700 1 -- 172.21.15.44:0/3301476148 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dbed7d45e0 con 0x55dbeda01bb0 2023-11-16T16:49:20.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.715+0000 7fb563fff700 1 --2- 172.21.15.44:0/3301476148 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 0x55dbeda02ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:20.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.715+0000 7fb56b59e700 1 --2- 172.21.15.44:0/3301476148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 0x55dbeda03530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:20.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.716+0000 7fb56b59e700 1 --2- 172.21.15.44:0/2677841528 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb5580a11d0 0x7fb5580a3680 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb564008cc0 tx=0x7fb5640093b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:20.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.716+0000 7fb56ad9d700 1 --2- 172.21.15.44:0/3301476148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda01bb0 0x55dbeda01fc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb55c0cb300 tx=0x7fb55c09ab40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3a5bfd79f187670 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:20.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.716+0000 7fb5617fa700 1 -- 172.21.15.44:0/3301476148 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c09ecd0 con 0x55dbeda01bb0 2023-11-16T16:49:20.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.716+0000 7fb5617fa700 1 -- 172.21.15.44:0/3301476148 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb55c0a3060 con 0x55dbeda01bb0 2023-11-16T16:49:20.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.716+0000 7fb5617fa700 1 -- 172.21.15.44:0/3301476148 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c09af00 con 0x55dbeda01bb0 2023-11-16T16:49:20.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.716+0000 7fb584afdb80 1 -- 172.21.15.44:0/3301476148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda01bb0 msgr2=0x55dbeda01fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.716+0000 7fb584afdb80 1 --2- 172.21.15.44:0/3301476148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda01bb0 0x55dbeda01fc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb55c0cb300 tx=0x7fb55c09ab40 comp rx=0 tx=0).stop 2023-11-16T16:49:20.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.717+0000 7fb584afdb80 1 -- 172.21.15.44:0/3301476148 shutdown_connections 2023-11-16T16:49:20.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.717+0000 7fb584afdb80 1 --2- 172.21.15.44:0/3301476148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 0x55dbeda03530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.717+0000 7fb584afdb80 1 --2- 172.21.15.44:0/3301476148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda01bb0 0x55dbeda01fc0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.717+0000 7fb584afdb80 1 --2- 172.21.15.44:0/3301476148 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 0x55dbeda02ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.717+0000 7fb584afdb80 1 -- 172.21.15.44:0/3301476148 >> 172.21.15.44:0/3301476148 conn(0x55dbed9f7310 msgr2=0x55dbed9fc620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:20.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.717+0000 7fb584afdb80 1 -- 172.21.15.44:0/3301476148 shutdown_connections 2023-11-16T16:49:20.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.717+0000 7fb584afdb80 1 -- 172.21.15.44:0/3301476148 wait complete. 2023-11-16T16:49:20.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.717+0000 7fb584afdb80 1 -- start start 2023-11-16T16:49:20.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.717+0000 7fb584afdb80 1 Processor -- start 2023-11-16T16:49:20.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.718+0000 7fb584afdb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda01bb0 0x55dbeda0b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.718+0000 7fb584afdb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 0x55dbeda0b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.718+0000 7fb584afdb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda030c0 0x55dbeda12700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.718+0000 7fb584afdb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbed8f4890 con 0x55dbeda01bb0 2023-11-16T16:49:20.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.718+0000 7fb584afdb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbed8f4530 con 0x55dbeda030c0 2023-11-16T16:49:20.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.718+0000 7fb584afdb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbed7d68d0 con 0x55dbeda02500 2023-11-16T16:49:20.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.718+0000 7fb584afdb80 10 client.0 _ll_register_callbacks cb 0x55dbed912730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:49:20.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.718+0000 7fb563fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 0x55dbeda0b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.718+0000 7fb56ad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda01bb0 0x55dbeda0b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.718+0000 7fb563fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 0x55dbeda0b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:42454/0 (socket says 172.21.15.44:42454) 2023-11-16T16:49:20.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.718+0000 7fb563fff700 1 -- 172.21.15.44:0/2040716545 learned_addr learned my addr 172.21.15.44:0/2040716545 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:20.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.718+0000 7fb56b59e700 1 --2- 172.21.15.44:0/2040716545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda030c0 0x55dbeda12700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.719+0000 7fb56ad9d700 1 -- 172.21.15.44:0/2040716545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 msgr2=0x55dbeda0b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:20.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.719+0000 7fb56ad9d700 1 --2- 172.21.15.44:0/2040716545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 0x55dbeda0b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.719+0000 7fb56ad9d700 1 -- 172.21.15.44:0/2040716545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda030c0 msgr2=0x55dbeda12700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:20.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.719+0000 7fb56ad9d700 1 --2- 172.21.15.44:0/2040716545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda030c0 0x55dbeda12700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.719+0000 7fb56ad9d700 1 -- 172.21.15.44:0/2040716545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56c10dc70 con 0x55dbeda01bb0 2023-11-16T16:49:20.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.719+0000 7fb563fff700 1 --2- 172.21.15.44:0/2040716545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 0x55dbeda0b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:20.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.719+0000 7fb56ad9d700 1 --2- 172.21.15.44:0/2040716545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda01bb0 0x55dbeda0b180 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb55c0cb300 tx=0x7fb55c09adb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709038fed6dcb26b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:20.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.719+0000 7fb5527fc700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:49:20.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.719+0000 7fb5527fc700 1 -- 172.21.15.44:0/2040716545 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c098de0 con 0x55dbeda01bb0 2023-11-16T16:49:20.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.719+0000 7fb5527fc700 1 -- 172.21.15.44:0/2040716545 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fb55c09ecd0 con 0x55dbeda01bb0 2023-11-16T16:49:20.732 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.719+0000 7fb584afdb80 1 -- 172.21.15.44:0/2040716545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dbed9f4ad0 con 0x55dbeda01bb0 2023-11-16T16:49:20.732 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.719+0000 7fb5527fc700 1 -- 172.21.15.44:0/2040716545 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb55c0b3230 con 0x55dbeda01bb0 2023-11-16T16:49:20.732 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.719+0000 7fb584afdb80 10 client.15237 fetch_fsmap learned FSMap version 21 2023-11-16T16:49:20.732 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.719+0000 7fb584afdb80 1 -- 172.21.15.44:0/2040716545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55dbed919140 con 0x55dbeda01bb0 2023-11-16T16:49:20.732 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.719+0000 7fb5527fc700 1 -- 172.21.15.44:0/2040716545 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dbed9f4ad0 con 0x55dbeda01bb0 2023-11-16T16:49:20.732 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.720+0000 7fb5527fc700 1 -- 172.21.15.44:0/2040716545 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb55c09b000 con 0x55dbeda01bb0 2023-11-16T16:49:20.732 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.720+0000 7fb584afdb80 10 client.15237 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:49:20.732 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.720+0000 7fb584afdb80 10 client.15237 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T16:49:20.732 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.720+0000 7fb584afdb80 20 client.15237 populate_metadata read hostname 'smithi044' 2023-11-16T16:49:20.732 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.720+0000 7fb584afdb80 1 --2- 172.21.15.44:0/2040716545 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55dbeda16550 0x55dbeda15630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.733 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.720+0000 7fb584afdb80 4 client.15237 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:49:20.733 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.720+0000 7fb584afdb80 1 -- 172.21.15.44:0/2040716545 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dbed915050 con 0x55dbeda16550 2023-11-16T16:49:20.733 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.720+0000 7fb563fff700 1 --2- 172.21.15.44:0/2040716545 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55dbeda16550 0x55dbeda15630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.733 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.721+0000 7fb563fff700 1 --2- 172.21.15.44:0/2040716545 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55dbeda16550 0x55dbeda15630 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bbdb16bf6a1e81a server_cookie=15fe367ff15f9936 in_seq=0 out_seq=0 2023-11-16T16:49:20.733 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.721+0000 7fb5527fc700 10 client.15237 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:49:20.733 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.721+0000 7fb5527fc700 1 -- 172.21.15.44:0/2040716545 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55dbed915050 con 0x55dbeda16550 2023-11-16T16:49:20.733 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.721+0000 7fb5527fc700 10 client.15237 handle_command_reply: tid=0 2023-11-16T16:49:20.733 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.721+0000 7fb584afdb80 1 client.15237 shutdown 2023-11-16T16:49:20.734 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.722+0000 7fb584afdb80 1 -- 172.21.15.44:0/2040716545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda01bb0 msgr2=0x55dbeda0b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.734 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.722+0000 7fb584afdb80 1 --2- 172.21.15.44:0/2040716545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda01bb0 0x55dbeda0b180 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb55c0cb300 tx=0x7fb55c09adb0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.734 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.722+0000 7fb584afdb80 1 -- 172.21.15.44:0/2040716545 shutdown_connections 2023-11-16T16:49:20.734 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.722+0000 7fb584afdb80 1 --2- 172.21.15.44:0/2040716545 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55dbeda16550 0x55dbeda15630 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.734 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.722+0000 7fb584afdb80 1 --2- 172.21.15.44:0/2040716545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda030c0 0x55dbeda12700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.734 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.722+0000 7fb584afdb80 1 --2- 172.21.15.44:0/2040716545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda01bb0 0x55dbeda0b180 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.734 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.722+0000 7fb584afdb80 1 --2- 172.21.15.44:0/2040716545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 0x55dbeda0b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.734 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.722+0000 7fb584afdb80 1 -- 172.21.15.44:0/2040716545 >> 172.21.15.44:0/2040716545 conn(0x55dbed9f7310 msgr2=0x55dbed9fbd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.722+0000 7fb5527fc700 0 client.15237 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:49:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.722+0000 7fb584afdb80 1 -- 172.21.15.44:0/2040716545 shutdown_connections 2023-11-16T16:49:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.722+0000 7fb584afdb80 1 -- 172.21.15.44:0/2040716545 wait complete. 2023-11-16T16:49:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.722+0000 7fb584afdb80 20 client.15237 trim_cache size 0 max 16384 2023-11-16T16:49:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.740+0000 7fb584afdb80 1 Processor -- start 2023-11-16T16:49:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.740+0000 7fb584afdb80 1 -- start start 2023-11-16T16:49:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.741+0000 7fb584afdb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda01bb0 0x55dbeda093d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.741+0000 7fb584afdb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda02500 0x55dbeda09910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.741+0000 7fb584afdb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 0x55dbeda126c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.741+0000 7fb584afdb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbed8f4530 con 0x55dbeda030c0 2023-11-16T16:49:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.741+0000 7fb584afdb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbed907010 con 0x55dbeda02500 2023-11-16T16:49:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.741+0000 7fb584afdb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbed952ce0 con 0x55dbeda01bb0 2023-11-16T16:49:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.741+0000 7fb56b59e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 0x55dbeda126c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.741+0000 7fb56b59e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 0x55dbeda126c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51670/0 (socket says 172.21.15.44:51670) 2023-11-16T16:49:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.741+0000 7fb56b59e700 1 -- 172.21.15.44:0/2051753635 learned_addr learned my addr 172.21.15.44:0/2051753635 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.741+0000 7fb56ad9d700 1 --2- 172.21.15.44:0/2051753635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda01bb0 0x55dbeda093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.741+0000 7fb563fff700 1 --2- 172.21.15.44:0/2051753635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda02500 0x55dbeda09910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.741+0000 7fb56b59e700 1 -- 172.21.15.44:0/2051753635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda01bb0 msgr2=0x55dbeda093d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.741+0000 7fb56b59e700 1 --2- 172.21.15.44:0/2051753635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda01bb0 0x55dbeda093d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.741+0000 7fb56b59e700 1 -- 172.21.15.44:0/2051753635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda02500 msgr2=0x55dbeda09910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.741+0000 7fb56b59e700 1 --2- 172.21.15.44:0/2051753635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda02500 0x55dbeda09910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.741+0000 7fb56b59e700 1 -- 172.21.15.44:0/2051753635 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dbed918e90 con 0x55dbeda030c0 2023-11-16T16:49:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.742+0000 7fb56b59e700 1 --2- 172.21.15.44:0/2051753635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 0x55dbeda126c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb564008e00 tx=0x7fb56401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5afb1c46cc0fe572 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.742+0000 7fb561ffb700 1 -- 172.21.15.44:0/2051753635 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56401bac0 con 0x55dbeda030c0 2023-11-16T16:49:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.742+0000 7fb561ffb700 1 -- 172.21.15.44:0/2051753635 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb56400f040 con 0x55dbeda030c0 2023-11-16T16:49:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.742+0000 7fb561ffb700 1 -- 172.21.15.44:0/2051753635 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb564026e60 con 0x55dbeda030c0 2023-11-16T16:49:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.742+0000 7fb584afdb80 1 -- 172.21.15.44:0/2051753635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 msgr2=0x55dbeda126c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.742+0000 7fb584afdb80 1 --2- 172.21.15.44:0/2051753635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 0x55dbeda126c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb564008e00 tx=0x7fb56401d000 comp rx=0 tx=0).stop 2023-11-16T16:49:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.743+0000 7fb584afdb80 1 -- 172.21.15.44:0/2051753635 shutdown_connections 2023-11-16T16:49:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.743+0000 7fb584afdb80 1 --2- 172.21.15.44:0/2051753635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 0x55dbeda126c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.743+0000 7fb584afdb80 1 --2- 172.21.15.44:0/2051753635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda01bb0 0x55dbeda093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.743+0000 7fb584afdb80 1 --2- 172.21.15.44:0/2051753635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda02500 0x55dbeda09910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.743+0000 7fb584afdb80 1 -- 172.21.15.44:0/2051753635 >> 172.21.15.44:0/2051753635 conn(0x55dbed9f7310 msgr2=0x55dbeda15630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.743+0000 7fb584afdb80 1 -- 172.21.15.44:0/2051753635 shutdown_connections 2023-11-16T16:49:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.743+0000 7fb584afdb80 1 -- 172.21.15.44:0/2051753635 wait complete. 2023-11-16T16:49:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.743+0000 7fb584afdb80 1 -- start start 2023-11-16T16:49:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.743+0000 7fb584afdb80 1 Processor -- start 2023-11-16T16:49:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb584afdb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda01bb0 0x55dbeda0d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb584afdb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 0x55dbeda0da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb584afdb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 0x55dbeda69da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb584afdb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbeda17890 con 0x55dbeda030c0 2023-11-16T16:49:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb584afdb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbeda0dff0 con 0x55dbeda01bb0 2023-11-16T16:49:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb584afdb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbeda6a2e0 con 0x55dbeda02500 2023-11-16T16:49:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb584afdb80 10 client.0 _ll_register_callbacks cb 0x7fb564008810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:49:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb56b59e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 0x55dbeda69da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb56ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda01bb0 0x55dbeda0d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb56ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda01bb0 0x55dbeda0d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:53082/0 (socket says 172.21.15.44:53082) 2023-11-16T16:49:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb56ad9d700 1 -- 172.21.15.44:0/2517258090 learned_addr learned my addr 172.21.15.44:0/2517258090 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb563fff700 1 --2- 172.21.15.44:0/2517258090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 0x55dbeda0da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb56ad9d700 1 -- 172.21.15.44:0/2517258090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 msgr2=0x55dbeda0da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:20.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb56ad9d700 1 --2- 172.21.15.44:0/2517258090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 0x55dbeda0da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb56ad9d700 1 -- 172.21.15.44:0/2517258090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 msgr2=0x55dbeda69da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:20.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb56ad9d700 1 --2- 172.21.15.44:0/2517258090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 0x55dbeda69da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.744+0000 7fb56ad9d700 1 -- 172.21.15.44:0/2517258090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dbed769c10 con 0x55dbeda01bb0 2023-11-16T16:49:20.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.745+0000 7fb56ad9d700 1 --2- 172.21.15.44:0/2517258090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda01bb0 0x55dbeda0d500 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00c010 tx=0x7fb55c003dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a19738b917d2bd7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:20.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.745+0000 7fb5527fc700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:49:20.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.745+0000 7fb5527fc700 1 -- 172.21.15.44:0/2517258090 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c099100 con 0x55dbeda01bb0 2023-11-16T16:49:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.745+0000 7fb5527fc700 1 -- 172.21.15.44:0/2517258090 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fb55c098de0 con 0x55dbeda01bb0 2023-11-16T16:49:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.745+0000 7fb584afdb80 1 -- 172.21.15.44:0/2517258090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb55c0a3060 con 0x55dbeda01bb0 2023-11-16T16:49:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.745+0000 7fb5527fc700 1 -- 172.21.15.44:0/2517258090 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb55c09a7e0 con 0x55dbeda01bb0 2023-11-16T16:49:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.745+0000 7fb584afdb80 10 client.25259 fetch_fsmap learned FSMap version 21 2023-11-16T16:49:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.745+0000 7fb584afdb80 1 -- 172.21.15.44:0/2517258090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55dbed780d90 con 0x55dbeda01bb0 2023-11-16T16:49:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.745+0000 7fb5527fc700 1 -- 172.21.15.44:0/2517258090 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb55c0a3060 con 0x55dbeda01bb0 2023-11-16T16:49:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.746+0000 7fb5527fc700 1 -- 172.21.15.44:0/2517258090 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb55c09a190 con 0x55dbeda01bb0 2023-11-16T16:49:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.746+0000 7fb584afdb80 10 client.25259 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:49:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.746+0000 7fb584afdb80 10 client.25259 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T16:49:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.746+0000 7fb584afdb80 20 client.25259 populate_metadata read hostname 'smithi044' 2023-11-16T16:49:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.746+0000 7fb584afdb80 1 --2- 172.21.15.44:0/2517258090 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55dbeda6e800 0x55dbeda6b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.746+0000 7fb584afdb80 4 client.25259 mds_command: new command op to 24551 tid=0[{"prefix": "status"}] 2023-11-16T16:49:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.746+0000 7fb584afdb80 1 -- 172.21.15.44:0/2517258090 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55dbed921610 con 0x55dbeda6e800 2023-11-16T16:49:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.746+0000 7fb563fff700 1 --2- 172.21.15.44:0/2517258090 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55dbeda6e800 0x55dbeda6b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.746+0000 7fb563fff700 1 --2- 172.21.15.44:0/2517258090 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55dbeda6e800 0x55dbeda6b420 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6fc599866cda80f server_cookie=e79440eace1682f6 in_seq=0 out_seq=0 2023-11-16T16:49:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.746+0000 7fb5527fc700 10 client.25259 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:49:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.747+0000 7fb5527fc700 1 -- 172.21.15.44:0/2517258090 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55dbed921610 con 0x55dbeda6e800 2023-11-16T16:49:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.747+0000 7fb5527fc700 10 client.25259 handle_command_reply: tid=0 2023-11-16T16:49:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.747+0000 7fb584afdb80 1 client.25259 shutdown 2023-11-16T16:49:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.747+0000 7fb584afdb80 1 -- 172.21.15.44:0/2517258090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda01bb0 msgr2=0x55dbeda0d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.747+0000 7fb584afdb80 1 --2- 172.21.15.44:0/2517258090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda01bb0 0x55dbeda0d500 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00c010 tx=0x7fb55c003dd0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.747+0000 7fb584afdb80 1 -- 172.21.15.44:0/2517258090 shutdown_connections 2023-11-16T16:49:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.747+0000 7fb584afdb80 1 --2- 172.21.15.44:0/2517258090 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55dbeda6e800 0x55dbeda6b420 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.747+0000 7fb584afdb80 1 --2- 172.21.15.44:0/2517258090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbeda030c0 0x55dbeda69da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.747+0000 7fb5527fc700 0 client.25259 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:49:20.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.747+0000 7fb584afdb80 1 --2- 172.21.15.44:0/2517258090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dbeda01bb0 0x55dbeda0d500 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.747+0000 7fb584afdb80 1 --2- 172.21.15.44:0/2517258090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dbeda02500 0x55dbeda0da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.747+0000 7fb584afdb80 1 -- 172.21.15.44:0/2517258090 >> 172.21.15.44:0/2517258090 conn(0x55dbed9f7310 msgr2=0x55dbeda05f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:20.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.747+0000 7fb584afdb80 1 -- 172.21.15.44:0/2517258090 shutdown_connections 2023-11-16T16:49:20.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.747+0000 7fb584afdb80 1 -- 172.21.15.44:0/2517258090 wait complete. 2023-11-16T16:49:20.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.747+0000 7fb584afdb80 20 client.25259 trim_cache size 0 max 16384 2023-11-16T16:49:20.758 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:49:20.759 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "7f2b12a8-849e-11ee-8dbe-212e2dc638e7", 2023-11-16T16:49:20.759 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2023-11-16T16:49:20.759 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24551, 2023-11-16T16:49:20.759 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2023-11-16T16:49:20.759 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2023-11-16T16:49:20.759 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2023-11-16T16:49:20.759 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 122.592426764, 2023-11-16T16:49:20.760 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 21, 2023-11-16T16:49:20.760 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 91, 2023-11-16T16:49:20.760 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 89, 2023-11-16T16:49:20.760 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 128.66271598700001 2023-11-16T16:49:20.760 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:49:20.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.748+0000 7fb5711ff700 1 -- 172.21.15.44:0/2677841528 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb5580a11d0 msgr2=0x7fb5580a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.748+0000 7fb5711ff700 1 --2- 172.21.15.44:0/2677841528 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb5580a11d0 0x7fb5580a3680 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb564008cc0 tx=0x7fb5640093b0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.748+0000 7fb5711ff700 1 -- 172.21.15.44:0/2677841528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb56c073550 msgr2=0x7fb56c198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.748+0000 7fb5711ff700 1 --2- 172.21.15.44:0/2677841528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb56c073550 0x7fb56c198270 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009fd0 tx=0x7fb55c00b790 comp rx=0 tx=0).stop 2023-11-16T16:49:20.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.748+0000 7fb5711ff700 1 -- 172.21.15.44:0/2677841528 shutdown_connections 2023-11-16T16:49:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.748+0000 7fb5711ff700 1 --2- 172.21.15.44:0/2677841528 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb5580a11d0 0x7fb5580a3680 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.748+0000 7fb5711ff700 1 --2- 172.21.15.44:0/2677841528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb56c1080f0 0x7fb56c19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.748+0000 7fb5711ff700 1 --2- 172.21.15.44:0/2677841528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb56c073550 0x7fb56c198270 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.748+0000 7fb5711ff700 1 --2- 172.21.15.44:0/2677841528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb56c074b30 0x7fb56c1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.748+0000 7fb5711ff700 1 -- 172.21.15.44:0/2677841528 >> 172.21.15.44:0/2677841528 conn(0x7fb56c0fb810 msgr2=0x7fb56c0fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.748+0000 7fb5711ff700 1 -- 172.21.15.44:0/2677841528 shutdown_connections 2023-11-16T16:49:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:20.748+0000 7fb5711ff700 1 -- 172.21.15.44:0/2677841528 wait complete. 2023-11-16T16:49:20.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:20 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1280237037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:20.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:20 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2860136410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:20.768 INFO:teuthology.run_tasks:Running task kclient... 2023-11-16T16:49:20.782 INFO:tasks.kclient:Mounting kernel clients... 2023-11-16T16:49:20.782 INFO:tasks.kclient:config is {'client.0': {}} 2023-11-16T16:49:20.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:20.782 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2023-11-16T16:49:20.827 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-11-16T16:49:20.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:20.827 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-16T16:49:20.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:20.938 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link delete ceph-brx 2023-11-16T16:49:20.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:20 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1280237037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:20.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:20 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2860136410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:21.045 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2023-11-16T16:49:21.045 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-16T16:49:21.045 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-16T16:49:21.045 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph fs ls 2023-11-16T16:49:21.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:20 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1280237037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:21.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:20 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2860136410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:21.907 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:49:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:21 smithi077 ceph-mon[131204]: pgmap v340: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:49:22.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:21 smithi044 ceph-mon[128750]: pgmap v340: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:49:22.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:21 smithi176 ceph-mon[131630]: pgmap v340: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:49:23.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.342+0000 7faf3db27700 1 -- 172.21.15.44:0/3313677336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf381055e0 msgr2=0x7faf3810d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:23.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.342+0000 7faf3db27700 1 --2- 172.21.15.44:0/3313677336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf381055e0 0x7faf3810d160 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7faf3000b0d0 tx=0x7faf3000b3e0 comp rx=0 tx=0).stop 2023-11-16T16:49:23.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.342+0000 7faf3db27700 1 -- 172.21.15.44:0/3313677336 shutdown_connections 2023-11-16T16:49:23.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.342+0000 7faf3db27700 1 --2- 172.21.15.44:0/3313677336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf381055e0 0x7faf3810d160 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:23.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.342+0000 7faf3db27700 1 --2- 172.21.15.44:0/3313677336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf38100540 0x7faf38100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:23.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.342+0000 7faf3db27700 1 --2- 172.21.15.44:0/3313677336 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf38100ee0 0x7faf38104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:23.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.342+0000 7faf3db27700 1 -- 172.21.15.44:0/3313677336 >> 172.21.15.44:0/3313677336 conn(0x7faf380fbdc0 msgr2=0x7faf380fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:23.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.342+0000 7faf3db27700 1 -- 172.21.15.44:0/3313677336 shutdown_connections 2023-11-16T16:49:23.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.343+0000 7faf3db27700 1 -- 172.21.15.44:0/3313677336 wait complete. 2023-11-16T16:49:23.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.343+0000 7faf3db27700 1 Processor -- start 2023-11-16T16:49:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.344+0000 7faf3db27700 1 -- start start 2023-11-16T16:49:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.344+0000 7faf3db27700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf38100540 0x7faf381986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.344+0000 7faf3db27700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf38100ee0 0x7faf38198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.344+0000 7faf3db27700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf381055e0 0x7faf3819bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.344+0000 7faf3db27700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3819c5d0 con 0x7faf38100540 2023-11-16T16:49:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.344+0000 7faf3db27700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3819c740 con 0x7faf381055e0 2023-11-16T16:49:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.344+0000 7faf3db27700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3819c900 con 0x7faf38100ee0 2023-11-16T16:49:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.344+0000 7faf377fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf38100540 0x7faf381986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.344+0000 7faf377fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf38100540 0x7faf381986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51688/0 (socket says 172.21.15.44:51688) 2023-11-16T16:49:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.344+0000 7faf377fe700 1 -- 172.21.15.44:0/1781170445 learned_addr learned my addr 172.21.15.44:0/1781170445 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.345+0000 7faf37fff700 1 --2- 172.21.15.44:0/1781170445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf381055e0 0x7faf3819bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.345+0000 7faf36ffd700 1 --2- 172.21.15.44:0/1781170445 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf38100ee0 0x7faf38198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:23.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.345+0000 7faf377fe700 1 -- 172.21.15.44:0/1781170445 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf38100ee0 msgr2=0x7faf38198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.345+0000 7faf377fe700 1 --2- 172.21.15.44:0/1781170445 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf38100ee0 0x7faf38198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.345+0000 7faf377fe700 1 -- 172.21.15.44:0/1781170445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf381055e0 msgr2=0x7faf3819bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.345+0000 7faf377fe700 1 --2- 172.21.15.44:0/1781170445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf381055e0 0x7faf3819bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.345+0000 7faf377fe700 1 -- 172.21.15.44:0/1781170445 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf28009520 con 0x7faf38100540 2023-11-16T16:49:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.345+0000 7faf377fe700 1 --2- 172.21.15.44:0/1781170445 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf38100540 0x7faf381986d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7faf2800eb30 tx=0x7faf2800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.345+0000 7faf34ff9700 1 -- 172.21.15.44:0/1781170445 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28009880 con 0x7faf38100540 2023-11-16T16:49:23.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.345+0000 7faf34ff9700 1 -- 172.21.15.44:0/1781170445 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faf280044e0 con 0x7faf38100540 2023-11-16T16:49:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.345+0000 7faf34ff9700 1 -- 172.21.15.44:0/1781170445 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28003d80 con 0x7faf38100540 2023-11-16T16:49:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.345+0000 7faf3db27700 1 -- 172.21.15.44:0/1781170445 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf30009d00 con 0x7faf38100540 2023-11-16T16:49:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.346+0000 7faf3db27700 1 -- 172.21.15.44:0/1781170445 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf38110370 con 0x7faf38100540 2023-11-16T16:49:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.347+0000 7faf3db27700 1 -- 172.21.15.44:0/1781170445 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf381101e0 con 0x7faf38100540 2023-11-16T16:49:23.350 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.349+0000 7faf34ff9700 1 -- 172.21.15.44:0/1781170445 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7faf28003f00 con 0x7faf38100540 2023-11-16T16:49:23.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.350+0000 7faf34ff9700 1 --2- 172.21.15.44:0/1781170445 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7faf200a1150 0x7faf200a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:23.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.350+0000 7faf34ff9700 1 -- 172.21.15.44:0/1781170445 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7faf28014070 con 0x7faf38100540 2023-11-16T16:49:23.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.350+0000 7faf36ffd700 1 --2- 172.21.15.44:0/1781170445 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7faf200a1150 0x7faf200a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:23.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.351+0000 7faf36ffd700 1 --2- 172.21.15.44:0/1781170445 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7faf200a1150 0x7faf200a3600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7faf38199cf0 tx=0x7faf2c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:23.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.354+0000 7faf34ff9700 1 -- 172.21.15.44:0/1781170445 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7faf280965e0 con 0x7faf38100540 2023-11-16T16:49:23.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.607+0000 7faf3db27700 1 -- 172.21.15.44:0/1781170445 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7faf38066ea0 con 0x7faf38100540 2023-11-16T16:49:32.979 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.608+0000 7faf34ff9700 1 -- 172.21.15.44:0/1781170445 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) v1 ==== 53+0+88 (secure 0 0 0) 0x7faf28099c00 con 0x7faf38100540 2023-11-16T16:49:32.979 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-16T16:49:32.980 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:23 smithi176 ceph-mon[131630]: pgmap v341: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:49:32.980 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:23 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1781170445' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T16:49:32.980 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:25 smithi176 ceph-mon[131630]: pgmap v342: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:49:32.980 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:27 smithi176 ceph-mon[131630]: pgmap v343: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:49:32.980 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:29 smithi176 ceph-mon[131630]: pgmap v344: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2023-11-16T16:49:32.980 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:31 smithi176 ceph-mon[131630]: pgmap v345: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2023-11-16T16:49:32.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:23 smithi077 ceph-mon[131204]: pgmap v341: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:49:32.981 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:23 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1781170445' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T16:49:32.981 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:25 smithi077 ceph-mon[131204]: pgmap v342: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:49:32.981 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:27 smithi077 ceph-mon[131204]: pgmap v343: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:49:32.981 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:29 smithi077 ceph-mon[131204]: pgmap v344: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2023-11-16T16:49:32.981 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:31 smithi077 ceph-mon[131204]: pgmap v345: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2023-11-16T16:49:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.610+0000 7faf3db27700 1 -- 172.21.15.44:0/1781170445 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7faf200a1150 msgr2=0x7faf200a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:32.982 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.610+0000 7faf3db27700 1 --2- 172.21.15.44:0/1781170445 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7faf200a1150 0x7faf200a3600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7faf38199cf0 tx=0x7faf2c00b410 comp rx=0 tx=0).stop 2023-11-16T16:49:32.982 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.610+0000 7faf3db27700 1 -- 172.21.15.44:0/1781170445 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf38100540 msgr2=0x7faf381986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:32.982 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.610+0000 7faf3db27700 1 --2- 172.21.15.44:0/1781170445 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf38100540 0x7faf381986d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7faf2800eb30 tx=0x7faf2800ee40 comp rx=0 tx=0).stop 2023-11-16T16:49:32.982 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.610+0000 7faf3db27700 1 -- 172.21.15.44:0/1781170445 shutdown_connections 2023-11-16T16:49:32.982 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.610+0000 7faf3db27700 1 --2- 172.21.15.44:0/1781170445 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7faf200a1150 0x7faf200a3600 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:32.982 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.610+0000 7faf3db27700 1 --2- 172.21.15.44:0/1781170445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf381055e0 0x7faf3819bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:32.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.610+0000 7faf3db27700 1 --2- 172.21.15.44:0/1781170445 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf38100540 0x7faf381986d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:32.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.610+0000 7faf3db27700 1 --2- 172.21.15.44:0/1781170445 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf38100ee0 0x7faf38198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:32.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.610+0000 7faf3db27700 1 -- 172.21.15.44:0/1781170445 >> 172.21.15.44:0/1781170445 conn(0x7faf380fbdc0 msgr2=0x7faf3810b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:32.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.610+0000 7faf3db27700 1 -- 172.21.15.44:0/1781170445 shutdown_connections 2023-11-16T16:49:32.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:23.611+0000 7faf3db27700 1 -- 172.21.15.44:0/1781170445 wait complete. 2023-11-16T16:49:32.984 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:23 smithi044 ceph-mon[128750]: pgmap v341: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:49:32.984 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:23 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1781170445' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T16:49:32.985 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-16T16:49:32.985 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-16T16:49:32.985 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2023-11-16T16:49:32.985 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-16T16:49:32.986 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-16T16:49:32.986 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-16T16:49:32.986 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-16T16:49:32.986 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-16T16:49:32.986 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-16T16:49:32.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:32.987 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2023-11-16T16:49:32.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:25 smithi044 ceph-mon[128750]: pgmap v342: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:49:32.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:27 smithi044 ceph-mon[128750]: pgmap v343: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:49:32.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:29 smithi044 ceph-mon[128750]: pgmap v344: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2023-11-16T16:49:32.988 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:31 smithi044 ceph-mon[128750]: pgmap v345: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2023-11-16T16:49:33.038 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-16T16:49:33.038 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-16T16:49:33.038 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2023-11-16T16:49:33.038 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2023-11-16T16:49:33.038 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2023-11-16T16:49:33.038 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2023-11-16T16:49:33.039 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T16:49:33.039 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2023-11-16T16:49:33.039 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2023-11-16T16:49:33.039 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T16:49:33.039 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2023-11-16T16:49:33.039 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2023-11-16T16:49:33.039 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-16T16:49:33.039 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2023-11-16T16:49:33.039 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2023-11-16T16:49:33.039 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-16T16:49:33.039 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 41429sec preferred_lft 41429sec 2023-11-16T16:49:33.039 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link 2023-11-16T16:49:33.039 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2023-11-16T16:49:33.040 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T16:49:33.040 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2023-11-16T16:49:33.040 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2023-11-16T16:49:33.040 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-16T16:49:33.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:33.041 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec bash -c ' 2023-11-16T16:49:33.041 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2023-11-16T16:49:33.041 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add name ceph-brx type bridge 2023-11-16T16:49:33.041 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr flush dev ceph-brx 2023-11-16T16:49:33.041 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set ceph-brx up 2023-11-16T16:49:33.041 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-16T16:49:33.041 DEBUG:teuthology.orchestra.run.smithi044:> ') 2023-11-16T16:49:33.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:33.260 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-16T16:49:33.332 INFO:teuthology.orchestra.run.smithi044.stdout:1 2023-11-16T16:49:33.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:33.336 DEBUG:teuthology.orchestra.run.smithi044:> route 2023-11-16T16:49:33.382 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2023-11-16T16:49:33.383 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-16T16:49:33.383 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-16T16:49:33.383 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-16T16:49:33.383 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-16T16:49:33.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:33.384 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec bash -c ' 2023-11-16T16:49:33.384 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2023-11-16T16:49:33.384 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-16T16:49:33.384 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-16T16:49:33.384 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-16T16:49:33.384 DEBUG:teuthology.orchestra.run.smithi044:> ') 2023-11-16T16:49:33.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:33.562 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2023-11-16T16:49:33.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:33.606 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list-id 2023-11-16T16:49:33.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:33.692 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec bash -c ' 2023-11-16T16:49:33.692 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2023-11-16T16:49:33.692 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-16T16:49:33.692 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-11-16T16:49:33.692 DEBUG:teuthology.orchestra.run.smithi044:> ') 2023-11-16T16:49:33.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:33 smithi044 ceph-mon[128750]: pgmap v346: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2023-11-16T16:49:33.878 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-11-16T16:49:33.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:33.878 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec bash -c ' 2023-11-16T16:49:33.878 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2023-11-16T16:49:33.879 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-11-16T16:49:33.879 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-16T16:49:33.879 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-11-16T16:49:33.879 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-11-16T16:49:33.879 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-11-16T16:49:33.879 DEBUG:teuthology.orchestra.run.smithi044:> ') 2023-11-16T16:49:33.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:33 smithi176 ceph-mon[131630]: pgmap v346: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2023-11-16T16:49:33.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:33 smithi077 ceph-mon[131204]: pgmap v346: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2023-11-16T16:49:34.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:34.148 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec bash -c ' 2023-11-16T16:49:34.148 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2023-11-16T16:49:34.148 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2023-11-16T16:49:34.148 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set dev brx.0 master ceph-brx 2023-11-16T16:49:34.148 DEBUG:teuthology.orchestra.run.smithi044:> ') 2023-11-16T16:49:34.331 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-16T16:49:34.331 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-16T16:49:34.379 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-16T16:49:34.380 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-16T16:49:34.380 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-16T16:49:34.426 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2023-11-16T16:49:34.426 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2023-11-16T16:49:34.426 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/_nogroup/sv_1/fc1dc7c3-3f9b-4705-9c25-f5705d7e927f 2023-11-16T16:49:34.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:34.426 DEBUG:teuthology.orchestra.run.smithi044:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/_nogroup/sv_1/fc1dc7c3-3f9b-4705-9c25-f5705d7e927f /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2023-11-16T16:49:34.581 INFO:teuthology.orchestra.run.smithi044.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2023-11-16T16:49:34.581 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2023-11-16T16:49:34.784 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:34 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:34.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:34 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:34.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:34 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:35.197 INFO:teuthology.orchestra.run.smithi044.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2023-11-16T16:49:35.197 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2023-11-16T16:49:35.197 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: resolved to: "172.21.15.44:3300,172.21.15.77:3300,172.21.15.176:3300" 2023-11-16T16:49:35.197 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: trying mount with new device syntax: 0@7f2b12a8-849e-11ee-8dbe-212e2dc638e7.cephfs=/volumes/_nogroup/sv_1/fc1dc7c3-3f9b-4705-9c25-f5705d7e927f 2023-11-16T16:49:35.197 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync,name=0,key=0,mon_addr=172.21.15.44:3300/172.21.15.77:3300/172.21.15.176:3300" will pass to kernel 2023-11-16T16:49:35.205 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-16T16:49:35.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:35.205 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-16T16:49:35.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:35.280 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-16T16:49:35.374 INFO:teuthology.orchestra.run.smithi044.stdout:client25268 2023-11-16T16:49:35.374 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:49:35.374 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T16:49:35.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:35.419 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-16T16:49:35.553 INFO:teuthology.orchestra.run.smithi044.stdout:client25268 2023-11-16T16:49:35.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:35.553 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-16T16:49:35.647 INFO:teuthology.orchestra.run.smithi044.stdout:client25268 2023-11-16T16:49:35.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:35.647 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7.client25268/status') 2023-11-16T16:49:35.765 INFO:teuthology.orchestra.run.smithi044.stdout:instance: client.25268 (3)172.21.15.44:0/3283777617 2023-11-16T16:49:35.765 INFO:teuthology.orchestra.run.smithi044.stdout:blocklisted: false 2023-11-16T16:49:35.766 INFO:teuthology.orchestra.run.smithi044.stderr:0+1 records in 2023-11-16T16:49:35.766 INFO:teuthology.orchestra.run.smithi044.stderr:0+1 records out 2023-11-16T16:49:35.766 INFO:teuthology.orchestra.run.smithi044.stderr:71 bytes copied, 9.9178e-05 s, 716 kB/s 2023-11-16T16:49:35.769 INFO:teuthology.run_tasks:Running task check-counter... 2023-11-16T16:49:35.781 INFO:teuthology.run_tasks:Running task fwd_scrub... 2023-11-16T16:49:35.790 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:49:35.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:35 smithi176 ceph-mon[131630]: pgmap v347: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:35 smithi077 ceph-mon[131204]: pgmap v347: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:36.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.013+0000 7fe68d5ff700 1 -- 172.21.15.44:0/930760978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe688073550 msgr2=0x7fe6880739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.013+0000 7fe68d5ff700 1 --2- 172.21.15.44:0/930760978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe688073550 0x7fe6880739c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009990 tx=0x7fe67c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.014+0000 7fe68d5ff700 1 -- 172.21.15.44:0/930760978 shutdown_connections 2023-11-16T16:49:36.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.014+0000 7fe68d5ff700 1 --2- 172.21.15.44:0/930760978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe68810c860 0x7fe68810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.014+0000 7fe68d5ff700 1 --2- 172.21.15.44:0/930760978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe688074b30 0x7fe688072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.014+0000 7fe68d5ff700 1 --2- 172.21.15.44:0/930760978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe688073550 0x7fe6880739c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.014+0000 7fe68d5ff700 1 -- 172.21.15.44:0/930760978 >> 172.21.15.44:0/930760978 conn(0x7fe6880fb870 msgr2=0x7fe6880fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:36.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.014+0000 7fe68d5ff700 1 -- 172.21.15.44:0/930760978 shutdown_connections 2023-11-16T16:49:36.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.014+0000 7fe68d5ff700 1 -- 172.21.15.44:0/930760978 wait complete. 2023-11-16T16:49:36.017 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.015+0000 7fe68d5ff700 1 Processor -- start 2023-11-16T16:49:36.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.015+0000 7fe68d5ff700 1 -- start start 2023-11-16T16:49:36.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.015+0000 7fe68d5ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe688073550 0x7fe68819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.015+0000 7fe68d5ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe688074b30 0x7fe68819cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.015+0000 7fe68d5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe68810c860 0x7fe6881a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.015+0000 7fe68d5ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6881a4170 con 0x7fe688073550 2023-11-16T16:49:36.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.015+0000 7fe68d5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6881a42b0 con 0x7fe68810c860 2023-11-16T16:49:36.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.015+0000 7fe68d5ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6881a4440 con 0x7fe688074b30 2023-11-16T16:49:36.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.016+0000 7fe6867fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe688074b30 0x7fe68819cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.016+0000 7fe6877fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe68810c860 0x7fe6881a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.016+0000 7fe6867fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe688074b30 0x7fe68819cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:37718/0 (socket says 172.21.15.44:37718) 2023-11-16T16:49:36.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.016+0000 7fe686ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe688073550 0x7fe68819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.016+0000 7fe6867fc700 1 -- 172.21.15.44:0/2340284144 learned_addr learned my addr 172.21.15.44:0/2340284144 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.016+0000 7fe6877fe700 1 -- 172.21.15.44:0/2340284144 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe688074b30 msgr2=0x7fe68819cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.016+0000 7fe6877fe700 1 --2- 172.21.15.44:0/2340284144 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe688074b30 0x7fe68819cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.016+0000 7fe6877fe700 1 -- 172.21.15.44:0/2340284144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe688073550 msgr2=0x7fe68819c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.016+0000 7fe6877fe700 1 --2- 172.21.15.44:0/2340284144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe688073550 0x7fe68819c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.016+0000 7fe6877fe700 1 -- 172.21.15.44:0/2340284144 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe680009c40 con 0x7fe68810c860 2023-11-16T16:49:36.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.016+0000 7fe6877fe700 1 --2- 172.21.15.44:0/2340284144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe68810c860 0x7fe6881a3ba0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe68000efb0 tx=0x7fe68000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:36.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.017+0000 7fe66ffff700 1 -- 172.21.15.44:0/2340284144 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe680003a00 con 0x7fe68810c860 2023-11-16T16:49:36.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.017+0000 7fe66ffff700 1 -- 172.21.15.44:0/2340284144 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe68000f040 con 0x7fe68810c860 2023-11-16T16:49:36.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.017+0000 7fe68d5ff700 1 -- 172.21.15.44:0/2340284144 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67c0095f0 con 0x7fe68810c860 2023-11-16T16:49:36.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.017+0000 7fe66ffff700 1 -- 172.21.15.44:0/2340284144 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe680004540 con 0x7fe68810c860 2023-11-16T16:49:36.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.017+0000 7fe68d5ff700 1 -- 172.21.15.44:0/2340284144 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe688112270 con 0x7fe68810c860 2023-11-16T16:49:36.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.018+0000 7fe68d5ff700 1 -- 172.21.15.44:0/2340284144 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe688112400 con 0x7fe68810c860 2023-11-16T16:49:36.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.020+0000 7fe66ffff700 1 -- 172.21.15.44:0/2340284144 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe680003bf0 con 0x7fe68810c860 2023-11-16T16:49:36.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.021+0000 7fe66ffff700 1 --2- 172.21.15.44:0/2340284144 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe6700a1150 0x7fe6700a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.021+0000 7fe66ffff700 1 -- 172.21.15.44:0/2340284144 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fe6800cc8a0 con 0x7fe68810c860 2023-11-16T16:49:36.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.021+0000 7fe686ffd700 1 --2- 172.21.15.44:0/2340284144 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe6700a1150 0x7fe6700a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.022+0000 7fe686ffd700 1 --2- 172.21.15.44:0/2340284144 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe6700a1150 0x7fe6700a3600 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe688074f00 tx=0x7fe678009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:36.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.023+0000 7fe66ffff700 1 -- 172.21.15.44:0/2340284144 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe680095c80 con 0x7fe68810c860 2023-11-16T16:49:36.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:35 smithi044 ceph-mon[128750]: pgmap v347: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:36.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.241+0000 7fe68d5ff700 1 -- 172.21.15.44:0/2340284144 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe68802cfe0 con 0x7fe68810c860 2023-11-16T16:49:36.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.243+0000 7fe66ffff700 1 -- 172.21.15.44:0/2340284144 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24685 (secure 0 0 0) 0x7fe6800992a0 con 0x7fe68810c860 2023-11-16T16:49:36.245 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:36.246 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:48:44.613706+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:49:36.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.246+0000 7fe68d5ff700 1 -- 172.21.15.44:0/2340284144 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe6700a1150 msgr2=0x7fe6700a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.246+0000 7fe68d5ff700 1 --2- 172.21.15.44:0/2340284144 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe6700a1150 0x7fe6700a3600 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe688074f00 tx=0x7fe678009260 comp rx=0 tx=0).stop 2023-11-16T16:49:36.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.246+0000 7fe68d5ff700 1 -- 172.21.15.44:0/2340284144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe68810c860 msgr2=0x7fe6881a3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.246+0000 7fe68d5ff700 1 --2- 172.21.15.44:0/2340284144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe68810c860 0x7fe6881a3ba0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe68000efb0 tx=0x7fe68000c3a0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.246+0000 7fe68d5ff700 1 -- 172.21.15.44:0/2340284144 shutdown_connections 2023-11-16T16:49:36.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.246+0000 7fe68d5ff700 1 --2- 172.21.15.44:0/2340284144 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe6700a1150 0x7fe6700a3600 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.246+0000 7fe68d5ff700 1 --2- 172.21.15.44:0/2340284144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe68810c860 0x7fe6881a3ba0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.246+0000 7fe68d5ff700 1 --2- 172.21.15.44:0/2340284144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe688073550 0x7fe68819c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.246+0000 7fe68d5ff700 1 --2- 172.21.15.44:0/2340284144 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe688074b30 0x7fe68819cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.246+0000 7fe68d5ff700 1 -- 172.21.15.44:0/2340284144 >> 172.21.15.44:0/2340284144 conn(0x7fe6880fb870 msgr2=0x7fe6880fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:36.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.246+0000 7fe68d5ff700 1 -- 172.21.15.44:0/2340284144 shutdown_connections 2023-11-16T16:49:36.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.246+0000 7fe68d5ff700 1 -- 172.21.15.44:0/2340284144 wait complete. 2023-11-16T16:49:36.268 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:03.059622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:16.211826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:17.601591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:21.136084+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T16:49:36.269 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:49:36.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.481+0000 7fe9f25ff700 1 -- 172.21.15.44:0/3590308975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9ec071ce0 msgr2=0x7fe9ec072130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.481+0000 7fe9f25ff700 1 --2- 172.21.15.44:0/3590308975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9ec071ce0 0x7fe9ec072130 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0009990 tx=0x7fe9e0009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.481+0000 7fe9f25ff700 1 -- 172.21.15.44:0/3590308975 shutdown_connections 2023-11-16T16:49:36.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.481+0000 7fe9f25ff700 1 --2- 172.21.15.44:0/3590308975 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9ec072670 0x7fe9ec111050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.481+0000 7fe9f25ff700 1 --2- 172.21.15.44:0/3590308975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec107d60 0x7fe9ec108170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.481+0000 7fe9f25ff700 1 --2- 172.21.15.44:0/3590308975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9ec071ce0 0x7fe9ec072130 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.481+0000 7fe9f25ff700 1 -- 172.21.15.44:0/3590308975 >> 172.21.15.44:0/3590308975 conn(0x7fe9ec06d0d0 msgr2=0x7fe9ec06f520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:36.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.481+0000 7fe9f25ff700 1 -- 172.21.15.44:0/3590308975 shutdown_connections 2023-11-16T16:49:36.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.481+0000 7fe9f25ff700 1 -- 172.21.15.44:0/3590308975 wait complete. 2023-11-16T16:49:36.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.482+0000 7fe9f25ff700 1 Processor -- start 2023-11-16T16:49:36.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.483+0000 7fe9f25ff700 1 -- start start 2023-11-16T16:49:36.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.483+0000 7fe9f25ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec071ce0 0x7fe9ec10a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.483+0000 7fe9f25ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9ec072670 0x7fe9ec108dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.483+0000 7fe9f25ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9ec107d60 0x7fe9ec109300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.483+0000 7fe9f25ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec109aa0 con 0x7fe9ec072670 2023-11-16T16:49:36.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.483+0000 7fe9f25ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec109c10 con 0x7fe9ec071ce0 2023-11-16T16:49:36.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.483+0000 7fe9f25ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec1a88b0 con 0x7fe9ec107d60 2023-11-16T16:49:36.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.484+0000 7fe9ebfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec071ce0 0x7fe9ec10a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.484+0000 7fe9ebfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec071ce0 0x7fe9ec10a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:49668/0 (socket says 172.21.15.44:49668) 2023-11-16T16:49:36.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.484+0000 7fe9ebfff700 1 -- 172.21.15.44:0/3222330704 learned_addr learned my addr 172.21.15.44:0/3222330704 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:36.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.484+0000 7fe9eb7fe700 1 --2- 172.21.15.44:0/3222330704 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9ec072670 0x7fe9ec108dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.484+0000 7fe9f0b9c700 1 --2- 172.21.15.44:0/3222330704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9ec107d60 0x7fe9ec109300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.484+0000 7fe9ebfff700 1 -- 172.21.15.44:0/3222330704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9ec107d60 msgr2=0x7fe9ec109300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.484+0000 7fe9ebfff700 1 --2- 172.21.15.44:0/3222330704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9ec107d60 0x7fe9ec109300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.484+0000 7fe9ebfff700 1 -- 172.21.15.44:0/3222330704 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9ec072670 msgr2=0x7fe9ec108dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.484+0000 7fe9ebfff700 1 --2- 172.21.15.44:0/3222330704 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9ec072670 0x7fe9ec108dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.484+0000 7fe9ebfff700 1 -- 172.21.15.44:0/3222330704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e00095f0 con 0x7fe9ec071ce0 2023-11-16T16:49:36.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.484+0000 7fe9ebfff700 1 --2- 172.21.15.44:0/3222330704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec071ce0 0x7fe9ec10a780 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00e970 tx=0x7fe9dc00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:36.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.485+0000 7fe9e97fa700 1 -- 172.21.15.44:0/3222330704 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc009790 con 0x7fe9ec071ce0 2023-11-16T16:49:36.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.485+0000 7fe9f25ff700 1 -- 172.21.15.44:0/3222330704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ec114450 con 0x7fe9ec071ce0 2023-11-16T16:49:36.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.485+0000 7fe9e97fa700 1 -- 172.21.15.44:0/3222330704 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9dc00c7e0 con 0x7fe9ec071ce0 2023-11-16T16:49:36.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.485+0000 7fe9e97fa700 1 -- 172.21.15.44:0/3222330704 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc003c70 con 0x7fe9ec071ce0 2023-11-16T16:49:36.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.485+0000 7fe9f25ff700 1 -- 172.21.15.44:0/3222330704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ec1142c0 con 0x7fe9ec071ce0 2023-11-16T16:49:36.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.486+0000 7fe9f25ff700 1 -- 172.21.15.44:0/3222330704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ec1a8fa0 con 0x7fe9ec071ce0 2023-11-16T16:49:36.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.488+0000 7fe9e97fa700 1 -- 172.21.15.44:0/3222330704 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe9dc00cc50 con 0x7fe9ec071ce0 2023-11-16T16:49:36.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.489+0000 7fe9e97fa700 1 --2- 172.21.15.44:0/3222330704 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe9d40a1150 0x7fe9d40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.489+0000 7fe9e97fa700 1 -- 172.21.15.44:0/3222330704 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fe9dc014070 con 0x7fe9ec071ce0 2023-11-16T16:49:36.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.490+0000 7fe9eb7fe700 1 --2- 172.21.15.44:0/3222330704 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe9d40a1150 0x7fe9d40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.490+0000 7fe9eb7fe700 1 --2- 172.21.15.44:0/3222330704 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe9d40a1150 0x7fe9d40a3600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe9e000b2e0 tx=0x7fe9e0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:36.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.493+0000 7fe9e97fa700 1 -- 172.21.15.44:0/3222330704 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe9dc096210 con 0x7fe9ec071ce0 2023-11-16T16:49:36.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.734+0000 7fe9f25ff700 1 -- 172.21.15.44:0/3222330704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe9ec10fb40 con 0x7fe9ec071ce0 2023-11-16T16:49:36.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.734+0000 7fe9e97fa700 1 -- 172.21.15.44:0/3222330704 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe9dc003dd0 con 0x7fe9ec071ce0 2023-11-16T16:49:36.736 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2023-11-16T16:49:36.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.737+0000 7fe9f25ff700 1 -- 172.21.15.44:0/3222330704 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe9d40a1150 msgr2=0x7fe9d40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.737+0000 7fe9f25ff700 1 --2- 172.21.15.44:0/3222330704 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe9d40a1150 0x7fe9d40a3600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe9e000b2e0 tx=0x7fe9e0005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.737+0000 7fe9f25ff700 1 -- 172.21.15.44:0/3222330704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec071ce0 msgr2=0x7fe9ec10a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.737+0000 7fe9f25ff700 1 --2- 172.21.15.44:0/3222330704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec071ce0 0x7fe9ec10a780 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00e970 tx=0x7fe9dc00ec80 comp rx=0 tx=0).stop 2023-11-16T16:49:36.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.737+0000 7fe9f25ff700 1 -- 172.21.15.44:0/3222330704 shutdown_connections 2023-11-16T16:49:36.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.737+0000 7fe9f25ff700 1 --2- 172.21.15.44:0/3222330704 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe9d40a1150 0x7fe9d40a3600 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.737+0000 7fe9f25ff700 1 --2- 172.21.15.44:0/3222330704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9ec107d60 0x7fe9ec109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.737+0000 7fe9f25ff700 1 --2- 172.21.15.44:0/3222330704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec071ce0 0x7fe9ec10a780 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.737+0000 7fe9f25ff700 1 --2- 172.21.15.44:0/3222330704 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9ec072670 0x7fe9ec108dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.737+0000 7fe9f25ff700 1 -- 172.21.15.44:0/3222330704 >> 172.21.15.44:0/3222330704 conn(0x7fe9ec06d0d0 msgr2=0x7fe9ec06f360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:36.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.737+0000 7fe9f25ff700 1 -- 172.21.15.44:0/3222330704 shutdown_connections 2023-11-16T16:49:36.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.737+0000 7fe9f25ff700 1 -- 172.21.15.44:0/3222330704 wait complete. 2023-11-16T16:49:36.758 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T16:49:36.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:36 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2340284144' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:36.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.969+0000 7f8f38fff700 1 -- 172.21.15.44:0/2002717412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f34101e60 msgr2=0x7f8f34108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.969+0000 7f8f38fff700 1 --2- 172.21.15.44:0/2002717412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f34101e60 0x7f8f34108720 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c009f90 tx=0x7f8f2c00b330 comp rx=0 tx=0).stop 2023-11-16T16:49:36.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.969+0000 7f8f38fff700 1 -- 172.21.15.44:0/2002717412 shutdown_connections 2023-11-16T16:49:36.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.969+0000 7f8f38fff700 1 --2- 172.21.15.44:0/2002717412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f34101e60 0x7f8f34108720 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.969+0000 7f8f38fff700 1 --2- 172.21.15.44:0/2002717412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f341002e0 0x7f8f341006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.969+0000 7f8f38fff700 1 --2- 172.21.15.44:0/2002717412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f341014d0 0x7f8f34101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.969+0000 7f8f38fff700 1 -- 172.21.15.44:0/2002717412 >> 172.21.15.44:0/2002717412 conn(0x7f8f340fb8e0 msgr2=0x7f8f340fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:36.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.969+0000 7f8f38fff700 1 -- 172.21.15.44:0/2002717412 shutdown_connections 2023-11-16T16:49:36.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.969+0000 7f8f38fff700 1 -- 172.21.15.44:0/2002717412 wait complete. 2023-11-16T16:49:36.977 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.970+0000 7f8f38fff700 1 Processor -- start 2023-11-16T16:49:36.977 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.970+0000 7f8f38fff700 1 -- start start 2023-11-16T16:49:36.977 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.970+0000 7f8f38fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f341002e0 0x7f8f34196150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.977 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.970+0000 7f8f38fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f341014d0 0x7f8f34196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.977 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.970+0000 7f8f38fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f34101e60 0x7f8f3419d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.970+0000 7f8f38fff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3419dcd0 con 0x7f8f34101e60 2023-11-16T16:49:36.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.970+0000 7f8f38fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3419de10 con 0x7f8f341002e0 2023-11-16T16:49:36.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.970+0000 7f8f38fff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3419dfa0 con 0x7f8f341014d0 2023-11-16T16:49:36.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.971+0000 7f8f2bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f341014d0 0x7f8f34196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.971+0000 7f8f3259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f341002e0 0x7f8f34196150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.971+0000 7f8f32d9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f34101e60 0x7f8f3419d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.979 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.971+0000 7f8f2bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f341014d0 0x7f8f34196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:37756/0 (socket says 172.21.15.44:37756) 2023-11-16T16:49:36.979 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.971+0000 7f8f2bfff700 1 -- 172.21.15.44:0/3873652227 learned_addr learned my addr 172.21.15.44:0/3873652227 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:36.979 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.971+0000 7f8f3259c700 1 -- 172.21.15.44:0/3873652227 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f341014d0 msgr2=0x7f8f34196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.979 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.971+0000 7f8f3259c700 1 --2- 172.21.15.44:0/3873652227 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f341014d0 0x7f8f34196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.979 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.971+0000 7f8f3259c700 1 -- 172.21.15.44:0/3873652227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f34101e60 msgr2=0x7f8f3419d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.979 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.971+0000 7f8f3259c700 1 --2- 172.21.15.44:0/3873652227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f34101e60 0x7f8f3419d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.971+0000 7f8f3259c700 1 -- 172.21.15.44:0/3873652227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2c009c40 con 0x7f8f341002e0 2023-11-16T16:49:36.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.971+0000 7f8f32d9d700 1 --2- 172.21.15.44:0/3873652227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f34101e60 0x7f8f3419d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:36.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.971+0000 7f8f3259c700 1 --2- 172.21.15.44:0/3873652227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f341002e0 0x7f8f34196150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8f24005fd0 tx=0x7f8f2400eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:36.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.971+0000 7f8f2b7fe700 1 -- 172.21.15.44:0/3873652227 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f24004220 con 0x7f8f341002e0 2023-11-16T16:49:36.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.971+0000 7f8f2b7fe700 1 -- 172.21.15.44:0/3873652227 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f24004380 con 0x7f8f341002e0 2023-11-16T16:49:36.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.972+0000 7f8f38fff700 1 -- 172.21.15.44:0/3873652227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f3410bcb0 con 0x7f8f341002e0 2023-11-16T16:49:36.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.972+0000 7f8f2b7fe700 1 -- 172.21.15.44:0/3873652227 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f24003be0 con 0x7f8f341002e0 2023-11-16T16:49:36.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.972+0000 7f8f38fff700 1 -- 172.21.15.44:0/3873652227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f3410bb20 con 0x7f8f341002e0 2023-11-16T16:49:36.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.973+0000 7f8f38fff700 1 -- 172.21.15.44:0/3873652227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f34107170 con 0x7f8f341002e0 2023-11-16T16:49:36.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.974+0000 7f8f2b7fe700 1 -- 172.21.15.44:0/3873652227 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8f24003d40 con 0x7f8f341002e0 2023-11-16T16:49:36.982 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:36 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2340284144' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:36.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.976+0000 7f8f2b7fe700 1 --2- 172.21.15.44:0/3873652227 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8f200a1150 0x7f8f200a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.976+0000 7f8f2b7fe700 1 -- 172.21.15.44:0/3873652227 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f8f24014070 con 0x7f8f341002e0 2023-11-16T16:49:36.984 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.976+0000 7f8f2bfff700 1 --2- 172.21.15.44:0/3873652227 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8f200a1150 0x7f8f200a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.984 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.977+0000 7f8f2bfff700 1 --2- 172.21.15.44:0/3873652227 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8f200a1150 0x7f8f200a3600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c010e40 tx=0x7f8f1c012040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:36.984 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:36.978+0000 7f8f2b7fe700 1 -- 172.21.15.44:0/3873652227 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8f240967f0 con 0x7f8f341002e0 2023-11-16T16:49:37.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:36 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2340284144' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:37.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.204+0000 7f8f38fff700 1 -- 172.21.15.44:0/3873652227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8f34061700 con 0x7f8f341002e0 2023-11-16T16:49:37.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.205+0000 7f8f2b7fe700 1 -- 172.21.15.44:0/3873652227 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8f24099e10 con 0x7f8f341002e0 2023-11-16T16:49:37.207 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:49:37.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.207+0000 7f8f38fff700 1 -- 172.21.15.44:0/3873652227 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8f200a1150 msgr2=0x7f8f200a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.208+0000 7f8f38fff700 1 --2- 172.21.15.44:0/3873652227 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8f200a1150 0x7f8f200a3600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c010e40 tx=0x7f8f1c012040 comp rx=0 tx=0).stop 2023-11-16T16:49:37.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.208+0000 7f8f38fff700 1 -- 172.21.15.44:0/3873652227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f341002e0 msgr2=0x7f8f34196150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.208+0000 7f8f38fff700 1 --2- 172.21.15.44:0/3873652227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f341002e0 0x7f8f34196150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8f24005fd0 tx=0x7f8f2400eca0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.208+0000 7f8f38fff700 1 -- 172.21.15.44:0/3873652227 shutdown_connections 2023-11-16T16:49:37.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.208+0000 7f8f38fff700 1 --2- 172.21.15.44:0/3873652227 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f8f200a1150 0x7f8f200a3600 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.208+0000 7f8f38fff700 1 --2- 172.21.15.44:0/3873652227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f34101e60 0x7f8f3419d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.208+0000 7f8f38fff700 1 --2- 172.21.15.44:0/3873652227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f341002e0 0x7f8f34196150 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.208+0000 7f8f38fff700 1 --2- 172.21.15.44:0/3873652227 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f341014d0 0x7f8f34196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.208+0000 7f8f38fff700 1 -- 172.21.15.44:0/3873652227 >> 172.21.15.44:0/3873652227 conn(0x7f8f340fb8e0 msgr2=0x7f8f340fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.208+0000 7f8f38fff700 1 -- 172.21.15.44:0/3873652227 shutdown_connections 2023-11-16T16:49:37.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.208+0000 7f8f38fff700 1 -- 172.21.15.44:0/3873652227 wait complete. 2023-11-16T16:49:37.229 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T16:49:37.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.446+0000 7f92d41ff700 1 -- 172.21.15.44:0/219640513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92cc1002e0 msgr2=0x7f92cc1006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.446+0000 7f92d41ff700 1 --2- 172.21.15.44:0/219640513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92cc1002e0 0x7f92cc1006f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f92c8009990 tx=0x7f92c8009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.446+0000 7f92d41ff700 1 -- 172.21.15.44:0/219640513 shutdown_connections 2023-11-16T16:49:37.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.446+0000 7f92d41ff700 1 --2- 172.21.15.44:0/219640513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92cc101e60 0x7f92cc108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.446+0000 7f92d41ff700 1 --2- 172.21.15.44:0/219640513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92cc1002e0 0x7f92cc1006f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.446+0000 7f92d41ff700 1 --2- 172.21.15.44:0/219640513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92cc1014d0 0x7f92cc101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.446+0000 7f92d41ff700 1 -- 172.21.15.44:0/219640513 >> 172.21.15.44:0/219640513 conn(0x7f92cc0fb8e0 msgr2=0x7f92cc0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.446+0000 7f92d41ff700 1 -- 172.21.15.44:0/219640513 shutdown_connections 2023-11-16T16:49:37.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.446+0000 7f92d41ff700 1 -- 172.21.15.44:0/219640513 wait complete. 2023-11-16T16:49:37.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.447+0000 7f92d41ff700 1 Processor -- start 2023-11-16T16:49:37.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.447+0000 7f92d41ff700 1 -- start start 2023-11-16T16:49:37.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.447+0000 7f92d41ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92cc1002e0 0x7f92cc196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.447+0000 7f92d41ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92cc1014d0 0x7f92cc1965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.448+0000 7f92d41ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92cc101e60 0x7f92cc19d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.448+0000 7f92d41ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc19dbf0 con 0x7f92cc1014d0 2023-11-16T16:49:37.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.448+0000 7f92d41ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc19dd30 con 0x7f92cc101e60 2023-11-16T16:49:37.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.448+0000 7f92d41ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc19dec0 con 0x7f92cc1002e0 2023-11-16T16:49:37.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.448+0000 7f92d179a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92cc1014d0 0x7f92cc1965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.448+0000 7f92d179a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92cc1014d0 0x7f92cc1965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60910/0 (socket says 172.21.15.44:60910) 2023-11-16T16:49:37.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.448+0000 7f92d179a700 1 -- 172.21.15.44:0/2177490179 learned_addr learned my addr 172.21.15.44:0/2177490179 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:37.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.448+0000 7f92d1f9b700 1 --2- 172.21.15.44:0/2177490179 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92cc1002e0 0x7f92cc196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.448+0000 7f92d279c700 1 --2- 172.21.15.44:0/2177490179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92cc101e60 0x7f92cc19d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.448+0000 7f92d179a700 1 -- 172.21.15.44:0/2177490179 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92cc1002e0 msgr2=0x7f92cc196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.448+0000 7f92d179a700 1 --2- 172.21.15.44:0/2177490179 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92cc1002e0 0x7f92cc196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.448+0000 7f92d179a700 1 -- 172.21.15.44:0/2177490179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92cc101e60 msgr2=0x7f92cc19d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.448+0000 7f92d179a700 1 --2- 172.21.15.44:0/2177490179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92cc101e60 0x7f92cc19d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.448+0000 7f92d179a700 1 -- 172.21.15.44:0/2177490179 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c80095f0 con 0x7f92cc1014d0 2023-11-16T16:49:37.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.448+0000 7f92d179a700 1 --2- 172.21.15.44:0/2177490179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92cc1014d0 0x7f92cc1965b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00e9a0 tx=0x7f92bc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.449+0000 7f92c2ffd700 1 -- 172.21.15.44:0/2177490179 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc009790 con 0x7f92cc1014d0 2023-11-16T16:49:37.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.449+0000 7f92c2ffd700 1 -- 172.21.15.44:0/2177490179 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f92bc010430 con 0x7f92cc1014d0 2023-11-16T16:49:37.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.449+0000 7f92d41ff700 1 -- 172.21.15.44:0/2177490179 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92cc10bcb0 con 0x7f92cc1014d0 2023-11-16T16:49:37.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.449+0000 7f92c2ffd700 1 -- 172.21.15.44:0/2177490179 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc018680 con 0x7f92cc1014d0 2023-11-16T16:49:37.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.449+0000 7f92d41ff700 1 -- 172.21.15.44:0/2177490179 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92cc10bb20 con 0x7f92cc1014d0 2023-11-16T16:49:37.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.450+0000 7f92d41ff700 1 -- 172.21.15.44:0/2177490179 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92cc107210 con 0x7f92cc1014d0 2023-11-16T16:49:37.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.452+0000 7f92c2ffd700 1 -- 172.21.15.44:0/2177490179 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f92bc0105a0 con 0x7f92cc1014d0 2023-11-16T16:49:37.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.453+0000 7f92c2ffd700 1 --2- 172.21.15.44:0/2177490179 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f92b80a1150 0x7f92b80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.453+0000 7f92c2ffd700 1 -- 172.21.15.44:0/2177490179 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f92bc014070 con 0x7f92cc1014d0 2023-11-16T16:49:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.454+0000 7f92d1f9b700 1 --2- 172.21.15.44:0/2177490179 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f92b80a1150 0x7f92b80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.454+0000 7f92d1f9b700 1 --2- 172.21.15.44:0/2177490179 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f92b80a1150 0x7f92b80a3600 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f92c800b2e0 tx=0x7f92c8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.458+0000 7f92c2ffd700 1 -- 172.21.15.44:0/2177490179 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f92bc051730 con 0x7f92cc1014d0 2023-11-16T16:49:37.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.690+0000 7f92d41ff700 1 -- 172.21.15.44:0/2177490179 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f92cc061cc0 con 0x7f92cc1014d0 2023-11-16T16:49:37.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.691+0000 7f92c2ffd700 1 -- 172.21.15.44:0/2177490179 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f92bc0d2020 con 0x7f92cc1014d0 2023-11-16T16:49:37.693 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2023-11-16T16:49:37.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.693+0000 7f92d41ff700 1 -- 172.21.15.44:0/2177490179 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f92b80a1150 msgr2=0x7f92b80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.693+0000 7f92d41ff700 1 --2- 172.21.15.44:0/2177490179 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f92b80a1150 0x7f92b80a3600 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f92c800b2e0 tx=0x7f92c8005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.693+0000 7f92d41ff700 1 -- 172.21.15.44:0/2177490179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92cc1014d0 msgr2=0x7f92cc1965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.693+0000 7f92d41ff700 1 --2- 172.21.15.44:0/2177490179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92cc1014d0 0x7f92cc1965b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00e9a0 tx=0x7f92bc00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.693+0000 7f92d41ff700 1 -- 172.21.15.44:0/2177490179 shutdown_connections 2023-11-16T16:49:37.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.693+0000 7f92d41ff700 1 --2- 172.21.15.44:0/2177490179 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f92b80a1150 0x7f92b80a3600 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.693+0000 7f92d41ff700 1 --2- 172.21.15.44:0/2177490179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92cc101e60 0x7f92cc19d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.693+0000 7f92d41ff700 1 --2- 172.21.15.44:0/2177490179 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92cc1002e0 0x7f92cc196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.693+0000 7f92d41ff700 1 --2- 172.21.15.44:0/2177490179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92cc1014d0 0x7f92cc1965b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.693+0000 7f92d41ff700 1 -- 172.21.15.44:0/2177490179 >> 172.21.15.44:0/2177490179 conn(0x7f92cc0fb8e0 msgr2=0x7f92cc0fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.694+0000 7f92d41ff700 1 -- 172.21.15.44:0/2177490179 shutdown_connections 2023-11-16T16:49:37.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.694+0000 7f92d41ff700 1 -- 172.21.15.44:0/2177490179 wait complete. 2023-11-16T16:49:37.716 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T16:49:37.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.933+0000 7fba62dff700 1 -- 172.21.15.44:0/3146680010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba5c1014c0 msgr2=0x7fba5c101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.933+0000 7fba62dff700 1 --2- 172.21.15.44:0/3146680010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba5c1014c0 0x7fba5c101910 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fba480098f0 tx=0x7fba48009c00 comp rx=0 tx=0).stop 2023-11-16T16:49:37.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.933+0000 7fba62dff700 1 -- 172.21.15.44:0/3146680010 shutdown_connections 2023-11-16T16:49:37.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.933+0000 7fba62dff700 1 --2- 172.21.15.44:0/3146680010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5c101e50 0x7fba5c108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.933+0000 7fba62dff700 1 --2- 172.21.15.44:0/3146680010 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba5c1002d0 0x7fba5c1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.933+0000 7fba62dff700 1 --2- 172.21.15.44:0/3146680010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba5c1014c0 0x7fba5c101910 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.933+0000 7fba62dff700 1 -- 172.21.15.44:0/3146680010 >> 172.21.15.44:0/3146680010 conn(0x7fba5c0fb8b0 msgr2=0x7fba5c0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.933+0000 7fba62dff700 1 -- 172.21.15.44:0/3146680010 shutdown_connections 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.933+0000 7fba62dff700 1 -- 172.21.15.44:0/3146680010 wait complete. 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.934+0000 7fba62dff700 1 Processor -- start 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.934+0000 7fba62dff700 1 -- start start 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.935+0000 7fba62dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba5c1002d0 0x7fba5c198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.935+0000 7fba62dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba5c1014c0 0x7fba5c1988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.935+0000 7fba62dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5c101e50 0x7fba5c19f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.935+0000 7fba62dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c19fee0 con 0x7fba5c101e50 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.935+0000 7fba62dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c1a0020 con 0x7fba5c1002d0 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.935+0000 7fba62dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c1a01b0 con 0x7fba5c1014c0 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.935+0000 7fba60b9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba5c1002d0 0x7fba5c198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.935+0000 7fba6139c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5c101e50 0x7fba5c19f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.935+0000 7fba60b9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba5c1002d0 0x7fba5c198360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:49712/0 (socket says 172.21.15.44:49712) 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.935+0000 7fba60b9b700 1 -- 172.21.15.44:0/469067028 learned_addr learned my addr 172.21.15.44:0/469067028 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.935+0000 7fba5bfff700 1 --2- 172.21.15.44:0/469067028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba5c1014c0 0x7fba5c1988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.935+0000 7fba60b9b700 1 -- 172.21.15.44:0/469067028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba5c1014c0 msgr2=0x7fba5c1988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.935+0000 7fba60b9b700 1 --2- 172.21.15.44:0/469067028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba5c1014c0 0x7fba5c1988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.935+0000 7fba60b9b700 1 -- 172.21.15.44:0/469067028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5c101e50 msgr2=0x7fba5c19f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.935+0000 7fba60b9b700 1 --2- 172.21.15.44:0/469067028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5c101e50 0x7fba5c19f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.935+0000 7fba60b9b700 1 -- 172.21.15.44:0/469067028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba48009520 con 0x7fba5c1002d0 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.936+0000 7fba60b9b700 1 --2- 172.21.15.44:0/469067028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba5c1002d0 0x7fba5c198360 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fba5000e970 tx=0x7fba5000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.936+0000 7fba59ffb700 1 -- 172.21.15.44:0/469067028 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba50009790 con 0x7fba5c1002d0 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.936+0000 7fba59ffb700 1 -- 172.21.15.44:0/469067028 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba5000c7e0 con 0x7fba5c1002d0 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.936+0000 7fba62dff700 1 -- 172.21.15.44:0/469067028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5c10bc80 con 0x7fba5c1002d0 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.936+0000 7fba59ffb700 1 -- 172.21.15.44:0/469067028 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba50003c70 con 0x7fba5c1002d0 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.936+0000 7fba62dff700 1 -- 172.21.15.44:0/469067028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5c10baf0 con 0x7fba5c1002d0 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.937+0000 7fba62dff700 1 -- 172.21.15.44:0/469067028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5c1929e0 con 0x7fba5c1002d0 2023-11-16T16:49:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.939+0000 7fba59ffb700 1 -- 172.21.15.44:0/469067028 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fba5000cc50 con 0x7fba5c1002d0 2023-11-16T16:49:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.940+0000 7fba59ffb700 1 --2- 172.21.15.44:0/469067028 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba4c0a1150 0x7fba4c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.940+0000 7fba59ffb700 1 -- 172.21.15.44:0/469067028 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fba50014070 con 0x7fba5c1002d0 2023-11-16T16:49:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.941+0000 7fba5bfff700 1 --2- 172.21.15.44:0/469067028 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba4c0a1150 0x7fba4c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.941+0000 7fba5bfff700 1 --2- 172.21.15.44:0/469067028 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba4c0a1150 0x7fba4c0a3600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fba48005e90 tx=0x7fba48019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:37.944+0000 7fba59ffb700 1 -- 172.21.15.44:0/469067028 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fba50096210 con 0x7fba5c1002d0 2023-11-16T16:49:37.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:37 smithi077 ceph-mon[131204]: pgmap v348: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:49:37.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:37 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:37 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:37 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:37 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:37 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3222330704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:37 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3873652227' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:38.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:37 smithi044 ceph-mon[128750]: pgmap v348: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:49:38.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:38.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:38.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:38.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:37 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:38.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:37 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3222330704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:38.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:37 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3873652227' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:38.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:37 smithi176 ceph-mon[131630]: pgmap v348: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:49:38.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:38.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:38.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:38.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:37 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:38.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:37 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3222330704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:38.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:37 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3873652227' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.156+0000 7fba62dff700 1 -- 172.21.15.44:0/469067028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fba5c067b00 con 0x7fba5c1002d0 2023-11-16T16:49:38.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.156+0000 7fba59ffb700 1 -- 172.21.15.44:0/469067028 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7fba50003dd0 con 0x7fba5c1002d0 2023-11-16T16:49:38.158 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:49:38.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.159+0000 7fba62dff700 1 -- 172.21.15.44:0/469067028 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba4c0a1150 msgr2=0x7fba4c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:38.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.159+0000 7fba62dff700 1 --2- 172.21.15.44:0/469067028 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba4c0a1150 0x7fba4c0a3600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fba48005e90 tx=0x7fba48019040 comp rx=0 tx=0).stop 2023-11-16T16:49:38.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.159+0000 7fba62dff700 1 -- 172.21.15.44:0/469067028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba5c1002d0 msgr2=0x7fba5c198360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:38.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.159+0000 7fba62dff700 1 --2- 172.21.15.44:0/469067028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba5c1002d0 0x7fba5c198360 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fba5000e970 tx=0x7fba5000ec80 comp rx=0 tx=0).stop 2023-11-16T16:49:38.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.159+0000 7fba62dff700 1 -- 172.21.15.44:0/469067028 shutdown_connections 2023-11-16T16:49:38.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.159+0000 7fba62dff700 1 --2- 172.21.15.44:0/469067028 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fba4c0a1150 0x7fba4c0a3600 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.159+0000 7fba62dff700 1 --2- 172.21.15.44:0/469067028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba5c101e50 0x7fba5c19f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.159+0000 7fba62dff700 1 --2- 172.21.15.44:0/469067028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba5c1002d0 0x7fba5c198360 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.159+0000 7fba62dff700 1 --2- 172.21.15.44:0/469067028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba5c1014c0 0x7fba5c1988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.159+0000 7fba62dff700 1 -- 172.21.15.44:0/469067028 >> 172.21.15.44:0/469067028 conn(0x7fba5c0fb8b0 msgr2=0x7fba5c106e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:38.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.159+0000 7fba62dff700 1 -- 172.21.15.44:0/469067028 shutdown_connections 2023-11-16T16:49:38.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.159+0000 7fba62dff700 1 -- 172.21.15.44:0/469067028 wait complete. 2023-11-16T16:49:38.181 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:49:38.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.387+0000 7fb08a5ff700 1 -- 172.21.15.44:0/1162160501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb084101e10 msgr2=0x7fb0841086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:38.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.387+0000 7fb08a5ff700 1 --2- 172.21.15.44:0/1162160501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb084101e10 0x7fb0841086d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb07c00b0a0 tx=0x7fb07c00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.388+0000 7fb08a5ff700 1 -- 172.21.15.44:0/1162160501 shutdown_connections 2023-11-16T16:49:38.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.388+0000 7fb08a5ff700 1 --2- 172.21.15.44:0/1162160501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb084101e10 0x7fb0841086d0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.388+0000 7fb08a5ff700 1 --2- 172.21.15.44:0/1162160501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb084100290 0x7fb0841006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.388+0000 7fb08a5ff700 1 --2- 172.21.15.44:0/1162160501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb084101480 0x7fb0841018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.388+0000 7fb08a5ff700 1 -- 172.21.15.44:0/1162160501 >> 172.21.15.44:0/1162160501 conn(0x7fb0840fb870 msgr2=0x7fb0840fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:38.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.388+0000 7fb08a5ff700 1 -- 172.21.15.44:0/1162160501 shutdown_connections 2023-11-16T16:49:38.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.388+0000 7fb08a5ff700 1 -- 172.21.15.44:0/1162160501 wait complete. 2023-11-16T16:49:38.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.389+0000 7fb08a5ff700 1 Processor -- start 2023-11-16T16:49:38.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.389+0000 7fb08a5ff700 1 -- start start 2023-11-16T16:49:38.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.389+0000 7fb08a5ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb084100290 0x7fb084196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:38.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.389+0000 7fb08a5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb084101480 0x7fb0841965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:38.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.389+0000 7fb08a5ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb084101e10 0x7fb08419d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:38.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.389+0000 7fb08a5ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08419dbe0 con 0x7fb084101e10 2023-11-16T16:49:38.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.389+0000 7fb08a5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08419dd20 con 0x7fb084101480 2023-11-16T16:49:38.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.389+0000 7fb08a5ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08419deb0 con 0x7fb084100290 2023-11-16T16:49:38.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.390+0000 7fb083fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb084100290 0x7fb084196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:38.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.390+0000 7fb088b9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb084101e10 0x7fb08419d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:38.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.390+0000 7fb083fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb084100290 0x7fb084196060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:37808/0 (socket says 172.21.15.44:37808) 2023-11-16T16:49:38.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.390+0000 7fb083fff700 1 -- 172.21.15.44:0/4044734639 learned_addr learned my addr 172.21.15.44:0/4044734639 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:38.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.390+0000 7fb0837fe700 1 --2- 172.21.15.44:0/4044734639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb084101480 0x7fb0841965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:38.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.390+0000 7fb083fff700 1 -- 172.21.15.44:0/4044734639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb084101480 msgr2=0x7fb0841965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:38.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.390+0000 7fb083fff700 1 --2- 172.21.15.44:0/4044734639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb084101480 0x7fb0841965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.390+0000 7fb083fff700 1 -- 172.21.15.44:0/4044734639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb084101e10 msgr2=0x7fb08419d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:38.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.390+0000 7fb083fff700 1 --2- 172.21.15.44:0/4044734639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb084101e10 0x7fb08419d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.390+0000 7fb083fff700 1 -- 172.21.15.44:0/4044734639 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07c009d00 con 0x7fb084100290 2023-11-16T16:49:38.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.390+0000 7fb083fff700 1 --2- 172.21.15.44:0/4044734639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb084100290 0x7fb084196060 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb07400e970 tx=0x7fb07400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:38.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.391+0000 7fb0817fa700 1 -- 172.21.15.44:0/4044734639 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb074009790 con 0x7fb084100290 2023-11-16T16:49:38.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.391+0000 7fb08a5ff700 1 -- 172.21.15.44:0/4044734639 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb08410bc70 con 0x7fb084100290 2023-11-16T16:49:38.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.391+0000 7fb0817fa700 1 -- 172.21.15.44:0/4044734639 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb07400c7e0 con 0x7fb084100290 2023-11-16T16:49:38.394 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.391+0000 7fb0817fa700 1 -- 172.21.15.44:0/4044734639 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb074003b60 con 0x7fb084100290 2023-11-16T16:49:38.394 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.391+0000 7fb08a5ff700 1 -- 172.21.15.44:0/4044734639 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb08410bae0 con 0x7fb084100290 2023-11-16T16:49:38.394 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.392+0000 7fb08a5ff700 1 -- 172.21.15.44:0/4044734639 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0841077e0 con 0x7fb084100290 2023-11-16T16:49:38.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.394+0000 7fb0817fa700 1 -- 172.21.15.44:0/4044734639 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb074003d40 con 0x7fb084100290 2023-11-16T16:49:38.398 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.396+0000 7fb0817fa700 1 --2- 172.21.15.44:0/4044734639 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb06c0a11d0 0x7fb06c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:38.398 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.397+0000 7fb0817fa700 1 -- 172.21.15.44:0/4044734639 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fb074014070 con 0x7fb084100290 2023-11-16T16:49:38.398 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.397+0000 7fb0837fe700 1 --2- 172.21.15.44:0/4044734639 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb06c0a11d0 0x7fb06c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:38.399 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.397+0000 7fb0837fe700 1 --2- 172.21.15.44:0/4044734639 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb06c0a11d0 0x7fb06c0a3680 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb078009c60 tx=0x7fb078009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:38.399 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.398+0000 7fb0817fa700 1 -- 172.21.15.44:0/4044734639 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb0740967a0 con 0x7fb084100290 2023-11-16T16:49:38.613 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.612+0000 7fb08a5ff700 1 -- 172.21.15.44:0/4044734639 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb08402cfe0 con 0x7fb084100290 2023-11-16T16:49:38.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.613+0000 7fb0817fa700 1 -- 172.21.15.44:0/4044734639 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24685 (secure 0 0 0) 0x7fb074099dc0 con 0x7fb084100290 2023-11-16T16:49:38.616 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:38.616 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:48:44.613706+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:49:38.618 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.616+0000 7fb08a5ff700 1 -- 172.21.15.44:0/4044734639 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb06c0a11d0 msgr2=0x7fb06c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:38.618 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.616+0000 7fb08a5ff700 1 --2- 172.21.15.44:0/4044734639 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb06c0a11d0 0x7fb06c0a3680 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb078009c60 tx=0x7fb078009260 comp rx=0 tx=0).stop 2023-11-16T16:49:38.618 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.616+0000 7fb08a5ff700 1 -- 172.21.15.44:0/4044734639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb084100290 msgr2=0x7fb084196060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:38.618 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.616+0000 7fb08a5ff700 1 --2- 172.21.15.44:0/4044734639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb084100290 0x7fb084196060 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb07400e970 tx=0x7fb07400ec80 comp rx=0 tx=0).stop 2023-11-16T16:49:38.618 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.616+0000 7fb08a5ff700 1 -- 172.21.15.44:0/4044734639 shutdown_connections 2023-11-16T16:49:38.618 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.616+0000 7fb08a5ff700 1 --2- 172.21.15.44:0/4044734639 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb06c0a11d0 0x7fb06c0a3680 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.618 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.616+0000 7fb08a5ff700 1 --2- 172.21.15.44:0/4044734639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb084101e10 0x7fb08419d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.618 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.616+0000 7fb08a5ff700 1 --2- 172.21.15.44:0/4044734639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb084100290 0x7fb084196060 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.618 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.616+0000 7fb08a5ff700 1 --2- 172.21.15.44:0/4044734639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb084101480 0x7fb0841965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.616+0000 7fb08a5ff700 1 -- 172.21.15.44:0/4044734639 >> 172.21.15.44:0/4044734639 conn(0x7fb0840fb870 msgr2=0x7fb084104920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:38.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.616+0000 7fb08a5ff700 1 -- 172.21.15.44:0/4044734639 shutdown_connections 2023-11-16T16:49:38.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.617+0000 7fb08a5ff700 1 -- 172.21.15.44:0/4044734639 wait complete. 2023-11-16T16:49:38.639 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:03.059622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:16.211826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:17.601591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:21.136084+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T16:49:38.640 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:49:38.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.870+0000 7f62fa7ff700 1 -- 172.21.15.44:0/2913708353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f4101e80 msgr2=0x7f62f4108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:38.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.870+0000 7f62fa7ff700 1 --2- 172.21.15.44:0/2913708353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f4101e80 0x7f62f4108740 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f62ec00b0a0 tx=0x7f62ec00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.870+0000 7f62fa7ff700 1 -- 172.21.15.44:0/2913708353 shutdown_connections 2023-11-16T16:49:38.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.870+0000 7f62fa7ff700 1 --2- 172.21.15.44:0/2913708353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f4101e80 0x7f62f4108740 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.870+0000 7f62fa7ff700 1 --2- 172.21.15.44:0/2913708353 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62f4100300 0x7f62f4100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.870+0000 7f62fa7ff700 1 --2- 172.21.15.44:0/2913708353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62f41014f0 0x7f62f4101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.870+0000 7f62fa7ff700 1 -- 172.21.15.44:0/2913708353 >> 172.21.15.44:0/2913708353 conn(0x7f62f40fb900 msgr2=0x7f62f40fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:38.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.870+0000 7f62fa7ff700 1 -- 172.21.15.44:0/2913708353 shutdown_connections 2023-11-16T16:49:38.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.870+0000 7f62fa7ff700 1 -- 172.21.15.44:0/2913708353 wait complete. 2023-11-16T16:49:38.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.871+0000 7f62fa7ff700 1 Processor -- start 2023-11-16T16:49:38.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.872+0000 7f62fa7ff700 1 -- start start 2023-11-16T16:49:38.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.872+0000 7f62fa7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f4100300 0x7f62f41983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:38.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.872+0000 7f62fa7ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62f41014f0 0x7f62f4198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:38.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.872+0000 7f62fa7ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62f4101e80 0x7f62f419f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:38.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.872+0000 7f62fa7ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f419ff60 con 0x7f62f41014f0 2023-11-16T16:49:38.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.872+0000 7f62fa7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41a00a0 con 0x7f62f4100300 2023-11-16T16:49:38.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.872+0000 7f62fa7ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41a0230 con 0x7f62f4101e80 2023-11-16T16:49:38.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.872+0000 7f62f3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f4100300 0x7f62f41983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:38.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.872+0000 7f62f37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62f41014f0 0x7f62f4198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:38.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.872+0000 7f62f3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f4100300 0x7f62f41983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:49740/0 (socket says 172.21.15.44:49740) 2023-11-16T16:49:38.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.872+0000 7f62f3fff700 1 -- 172.21.15.44:0/2333656773 learned_addr learned my addr 172.21.15.44:0/2333656773 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:38.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.873+0000 7f62f8d9c700 1 --2- 172.21.15.44:0/2333656773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62f4101e80 0x7f62f419f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:38.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.873+0000 7f62f3fff700 1 -- 172.21.15.44:0/2333656773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62f4101e80 msgr2=0x7f62f419f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:38.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.873+0000 7f62f3fff700 1 --2- 172.21.15.44:0/2333656773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62f4101e80 0x7f62f419f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.873+0000 7f62f3fff700 1 -- 172.21.15.44:0/2333656773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62f41014f0 msgr2=0x7f62f4198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:38.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.873+0000 7f62f3fff700 1 --2- 172.21.15.44:0/2333656773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62f41014f0 0x7f62f4198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:38.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.873+0000 7f62f3fff700 1 -- 172.21.15.44:0/2333656773 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec009d00 con 0x7f62f4100300 2023-11-16T16:49:38.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.873+0000 7f62f37fe700 1 --2- 172.21.15.44:0/2333656773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62f41014f0 0x7f62f4198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:38.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.873+0000 7f62f3fff700 1 --2- 172.21.15.44:0/2333656773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f4100300 0x7f62f41983e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f62e400d980 tx=0x7f62e400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:38.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.873+0000 7f62f17fa700 1 -- 172.21.15.44:0/2333656773 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4004020 con 0x7f62f4100300 2023-11-16T16:49:38.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.873+0000 7f62f17fa700 1 -- 172.21.15.44:0/2333656773 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62e400f690 con 0x7f62f4100300 2023-11-16T16:49:38.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.873+0000 7f62fa7ff700 1 -- 172.21.15.44:0/2333656773 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f410bcd0 con 0x7f62f4100300 2023-11-16T16:49:38.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.873+0000 7f62f17fa700 1 -- 172.21.15.44:0/2333656773 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4003a60 con 0x7f62f4100300 2023-11-16T16:49:38.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.874+0000 7f62fa7ff700 1 -- 172.21.15.44:0/2333656773 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f410bb40 con 0x7f62f4100300 2023-11-16T16:49:38.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.874+0000 7f62fa7ff700 1 -- 172.21.15.44:0/2333656773 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f41a0980 con 0x7f62f4100300 2023-11-16T16:49:38.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.877+0000 7f62f17fa700 1 -- 172.21.15.44:0/2333656773 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f62e400fb00 con 0x7f62f4100300 2023-11-16T16:49:38.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.878+0000 7f62f17fa700 1 --2- 172.21.15.44:0/2333656773 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f62dc0a1150 0x7f62dc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:38.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.878+0000 7f62f17fa700 1 -- 172.21.15.44:0/2333656773 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f62e40ccc40 con 0x7f62f4100300 2023-11-16T16:49:38.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.879+0000 7f62f37fe700 1 --2- 172.21.15.44:0/2333656773 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f62dc0a1150 0x7f62dc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:38.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.879+0000 7f62f37fe700 1 --2- 172.21.15.44:0/2333656773 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f62dc0a1150 0x7f62dc0a3600 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f62e8009fd0 tx=0x7f62e8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:38.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:38.881+0000 7f62f17fa700 1 -- 172.21.15.44:0/2333656773 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f62e4095f90 con 0x7f62f4100300 2023-11-16T16:49:38.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:38 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2177490179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:38.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:38 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:38.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:38 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:38.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:38 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/469067028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:38.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:38 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/4044734639' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:39.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.090+0000 7f62fa7ff700 1 -- 172.21.15.44:0/2333656773 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f62f41a1020 con 0x7f62f4100300 2023-11-16T16:49:39.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.091+0000 7f62f17fa700 1 -- 172.21.15.44:0/2333656773 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f62e40995b0 con 0x7f62f4100300 2023-11-16T16:49:39.093 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2023-11-16T16:49:39.095 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.093+0000 7f62fa7ff700 1 -- 172.21.15.44:0/2333656773 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f62dc0a1150 msgr2=0x7f62dc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:39.095 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.093+0000 7f62fa7ff700 1 --2- 172.21.15.44:0/2333656773 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f62dc0a1150 0x7f62dc0a3600 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f62e8009fd0 tx=0x7f62e8009260 comp rx=0 tx=0).stop 2023-11-16T16:49:39.095 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.094+0000 7f62fa7ff700 1 -- 172.21.15.44:0/2333656773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f4100300 msgr2=0x7f62f41983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:39.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.094+0000 7f62fa7ff700 1 --2- 172.21.15.44:0/2333656773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f4100300 0x7f62f41983e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f62e400d980 tx=0x7f62e400dc90 comp rx=0 tx=0).stop 2023-11-16T16:49:39.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.094+0000 7f62fa7ff700 1 -- 172.21.15.44:0/2333656773 shutdown_connections 2023-11-16T16:49:39.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.094+0000 7f62fa7ff700 1 --2- 172.21.15.44:0/2333656773 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f62dc0a1150 0x7f62dc0a3600 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.094+0000 7f62fa7ff700 1 --2- 172.21.15.44:0/2333656773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62f4101e80 0x7f62f419f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.094+0000 7f62fa7ff700 1 --2- 172.21.15.44:0/2333656773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f4100300 0x7f62f41983e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.094+0000 7f62fa7ff700 1 --2- 172.21.15.44:0/2333656773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62f41014f0 0x7f62f4198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.094+0000 7f62fa7ff700 1 -- 172.21.15.44:0/2333656773 >> 172.21.15.44:0/2333656773 conn(0x7f62f40fb900 msgr2=0x7f62f40fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:39.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.094+0000 7f62fa7ff700 1 -- 172.21.15.44:0/2333656773 shutdown_connections 2023-11-16T16:49:39.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.094+0000 7f62fa7ff700 1 -- 172.21.15.44:0/2333656773 wait complete. 2023-11-16T16:49:39.112 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:38 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2177490179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:39.112 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:38 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:39.112 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:38 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:39.112 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:38 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/469067028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:39.112 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:38 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/4044734639' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:39.115 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T16:49:39.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:38 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2177490179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:39.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:38 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:39.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:38 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:39.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:38 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/469067028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:39.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:38 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/4044734639' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:39.324 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.321+0000 7f0eca9ff700 1 -- 172.21.15.44:0/1403644493 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ec41002c0 msgr2=0x7f0ec41006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:39.324 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.321+0000 7f0eca9ff700 1 --2- 172.21.15.44:0/1403644493 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ec41002c0 0x7f0ec41006d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4009990 tx=0x7f0eb4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.324 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.321+0000 7f0eca9ff700 1 -- 172.21.15.44:0/1403644493 shutdown_connections 2023-11-16T16:49:39.324 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.321+0000 7f0eca9ff700 1 --2- 172.21.15.44:0/1403644493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec4101e40 0x7f0ec4108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.324 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.321+0000 7f0eca9ff700 1 --2- 172.21.15.44:0/1403644493 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ec41002c0 0x7f0ec41006d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.325 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.321+0000 7f0eca9ff700 1 --2- 172.21.15.44:0/1403644493 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ec41014b0 0x7f0ec4101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.325 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.321+0000 7f0eca9ff700 1 -- 172.21.15.44:0/1403644493 >> 172.21.15.44:0/1403644493 conn(0x7f0ec40fb8a0 msgr2=0x7f0ec40fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:39.325 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.321+0000 7f0eca9ff700 1 -- 172.21.15.44:0/1403644493 shutdown_connections 2023-11-16T16:49:39.325 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.321+0000 7f0eca9ff700 1 -- 172.21.15.44:0/1403644493 wait complete. 2023-11-16T16:49:39.325 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.322+0000 7f0eca9ff700 1 Processor -- start 2023-11-16T16:49:39.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.322+0000 7f0eca9ff700 1 -- start start 2023-11-16T16:49:39.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.322+0000 7f0eca9ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ec41002c0 0x7f0ec4198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:39.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.322+0000 7f0eca9ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ec41014b0 0x7f0ec4198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:39.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.323+0000 7f0eca9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec4101e40 0x7f0ec419f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:39.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.323+0000 7f0eca9ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec419fe90 con 0x7f0ec41002c0 2023-11-16T16:49:39.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.323+0000 7f0eca9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec419ffd0 con 0x7f0ec4101e40 2023-11-16T16:49:39.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.323+0000 7f0eca9ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec41a0160 con 0x7f0ec41014b0 2023-11-16T16:49:39.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.323+0000 7f0ec3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ec41002c0 0x7f0ec4198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:39.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.323+0000 7f0ec3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ec41002c0 0x7f0ec4198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60976/0 (socket says 172.21.15.44:60976) 2023-11-16T16:49:39.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.323+0000 7f0ec3fff700 1 -- 172.21.15.44:0/2394539273 learned_addr learned my addr 172.21.15.44:0/2394539273 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:39.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.323+0000 7f0ebbfff700 1 --2- 172.21.15.44:0/2394539273 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ec41014b0 0x7f0ec4198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:39.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.323+0000 7f0ec8f9c700 1 --2- 172.21.15.44:0/2394539273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec4101e40 0x7f0ec419f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:39.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.323+0000 7f0ec3fff700 1 -- 172.21.15.44:0/2394539273 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ec41014b0 msgr2=0x7f0ec4198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:39.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.323+0000 7f0ec3fff700 1 --2- 172.21.15.44:0/2394539273 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ec41014b0 0x7f0ec4198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.323+0000 7f0ec3fff700 1 -- 172.21.15.44:0/2394539273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec4101e40 msgr2=0x7f0ec419f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:39.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.323+0000 7f0ec3fff700 1 --2- 172.21.15.44:0/2394539273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec4101e40 0x7f0ec419f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.323+0000 7f0ec3fff700 1 -- 172.21.15.44:0/2394539273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ebc009c40 con 0x7f0ec41002c0 2023-11-16T16:49:39.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.323+0000 7f0ec3fff700 1 --2- 172.21.15.44:0/2394539273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ec41002c0 0x7f0ec4198310 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4004730 tx=0x7f0eb4004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:39.333 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.324+0000 7f0ec1ffb700 1 -- 172.21.15.44:0/2394539273 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb401d070 con 0x7f0ec41002c0 2023-11-16T16:49:39.333 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.324+0000 7f0ec1ffb700 1 -- 172.21.15.44:0/2394539273 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0eb4021410 con 0x7f0ec41002c0 2023-11-16T16:49:39.333 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.324+0000 7f0eca9ff700 1 -- 172.21.15.44:0/2394539273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb40095f0 con 0x7f0ec41002c0 2023-11-16T16:49:39.333 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.324+0000 7f0ec1ffb700 1 -- 172.21.15.44:0/2394539273 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4017650 con 0x7f0ec41002c0 2023-11-16T16:49:39.333 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.324+0000 7f0eca9ff700 1 -- 172.21.15.44:0/2394539273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec410bc90 con 0x7f0ec41002c0 2023-11-16T16:49:39.333 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.325+0000 7f0eca9ff700 1 -- 172.21.15.44:0/2394539273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec41929f0 con 0x7f0ec41002c0 2023-11-16T16:49:39.333 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.327+0000 7f0ec1ffb700 1 -- 172.21.15.44:0/2394539273 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0eb4021810 con 0x7f0ec41002c0 2023-11-16T16:49:39.333 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.328+0000 7f0ec1ffb700 1 --2- 172.21.15.44:0/2394539273 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0eb00a1150 0x7f0eb00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:39.334 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.328+0000 7f0ec1ffb700 1 -- 172.21.15.44:0/2394539273 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f0eb40ce120 con 0x7f0ec41002c0 2023-11-16T16:49:39.334 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.328+0000 7f0ebbfff700 1 --2- 172.21.15.44:0/2394539273 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0eb00a1150 0x7f0eb00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:39.334 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.329+0000 7f0ebbfff700 1 --2- 172.21.15.44:0/2394539273 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0eb00a1150 0x7f0eb00a3600 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0eac009c60 tx=0x7f0eac009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:39.335 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.332+0000 7f0ec1ffb700 1 -- 172.21.15.44:0/2394539273 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0eb4097500 con 0x7f0ec41002c0 2023-11-16T16:49:39.542 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.540+0000 7f0eca9ff700 1 -- 172.21.15.44:0/2394539273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0ec4060f30 con 0x7f0ec41002c0 2023-11-16T16:49:39.542 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.541+0000 7f0ec1ffb700 1 -- 172.21.15.44:0/2394539273 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0eb409ab20 con 0x7f0ec41002c0 2023-11-16T16:49:39.542 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:49:39.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.543+0000 7f0eca9ff700 1 -- 172.21.15.44:0/2394539273 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0eb00a1150 msgr2=0x7f0eb00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:39.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.543+0000 7f0eca9ff700 1 --2- 172.21.15.44:0/2394539273 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0eb00a1150 0x7f0eb00a3600 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0eac009c60 tx=0x7f0eac009260 comp rx=0 tx=0).stop 2023-11-16T16:49:39.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.543+0000 7f0eca9ff700 1 -- 172.21.15.44:0/2394539273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ec41002c0 msgr2=0x7f0ec4198310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:39.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.543+0000 7f0eca9ff700 1 --2- 172.21.15.44:0/2394539273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ec41002c0 0x7f0ec4198310 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4004730 tx=0x7f0eb4004760 comp rx=0 tx=0).stop 2023-11-16T16:49:39.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.543+0000 7f0eca9ff700 1 -- 172.21.15.44:0/2394539273 shutdown_connections 2023-11-16T16:49:39.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.543+0000 7f0eca9ff700 1 --2- 172.21.15.44:0/2394539273 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0eb00a1150 0x7f0eb00a3600 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.543+0000 7f0eca9ff700 1 --2- 172.21.15.44:0/2394539273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec4101e40 0x7f0ec419f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.546 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.543+0000 7f0eca9ff700 1 --2- 172.21.15.44:0/2394539273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ec41002c0 0x7f0ec4198310 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.546 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.543+0000 7f0eca9ff700 1 --2- 172.21.15.44:0/2394539273 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ec41014b0 0x7f0ec4198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.546 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.543+0000 7f0eca9ff700 1 -- 172.21.15.44:0/2394539273 >> 172.21.15.44:0/2394539273 conn(0x7f0ec40fb8a0 msgr2=0x7f0ec4106df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:39.546 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.543+0000 7f0eca9ff700 1 -- 172.21.15.44:0/2394539273 shutdown_connections 2023-11-16T16:49:39.546 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.543+0000 7f0eca9ff700 1 -- 172.21.15.44:0/2394539273 wait complete. 2023-11-16T16:49:39.564 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T16:49:39.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.770+0000 7f7e4e3ff700 1 -- 172.21.15.44:0/4228042737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e48101e40 msgr2=0x7f7e48108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:39.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.770+0000 7f7e4e3ff700 1 --2- 172.21.15.44:0/4228042737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e48101e40 0x7f7e48108700 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7e4000b0a0 tx=0x7f7e4000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.771+0000 7f7e4e3ff700 1 -- 172.21.15.44:0/4228042737 shutdown_connections 2023-11-16T16:49:39.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.771+0000 7f7e4e3ff700 1 --2- 172.21.15.44:0/4228042737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e48101e40 0x7f7e48108700 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.771+0000 7f7e4e3ff700 1 --2- 172.21.15.44:0/4228042737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e481002c0 0x7f7e481006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.771+0000 7f7e4e3ff700 1 --2- 172.21.15.44:0/4228042737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e481014b0 0x7f7e48101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.771+0000 7f7e4e3ff700 1 -- 172.21.15.44:0/4228042737 >> 172.21.15.44:0/4228042737 conn(0x7f7e480fb8a0 msgr2=0x7f7e480fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:39.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.771+0000 7f7e4e3ff700 1 -- 172.21.15.44:0/4228042737 shutdown_connections 2023-11-16T16:49:39.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.771+0000 7f7e4e3ff700 1 -- 172.21.15.44:0/4228042737 wait complete. 2023-11-16T16:49:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.772+0000 7f7e4e3ff700 1 Processor -- start 2023-11-16T16:49:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.772+0000 7f7e4e3ff700 1 -- start start 2023-11-16T16:49:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.772+0000 7f7e4e3ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e481002c0 0x7f7e48198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.772+0000 7f7e4e3ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e481014b0 0x7f7e48198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.772+0000 7f7e4e3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e48101e40 0x7f7e4819f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.772+0000 7f7e4e3ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4819fec0 con 0x7f7e481014b0 2023-11-16T16:49:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.772+0000 7f7e4e3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e481a0000 con 0x7f7e48101e40 2023-11-16T16:49:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.772+0000 7f7e4e3ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e481a0190 con 0x7f7e481002c0 2023-11-16T16:49:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.772+0000 7f7e477fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e481014b0 0x7f7e48198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.772+0000 7f7e477fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e481014b0 0x7f7e48198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60998/0 (socket says 172.21.15.44:60998) 2023-11-16T16:49:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.772+0000 7f7e477fe700 1 -- 172.21.15.44:0/1630963698 learned_addr learned my addr 172.21.15.44:0/1630963698 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.773+0000 7f7e4c99c700 1 --2- 172.21.15.44:0/1630963698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e48101e40 0x7f7e4819f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.773+0000 7f7e47fff700 1 --2- 172.21.15.44:0/1630963698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e481002c0 0x7f7e48198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.773+0000 7f7e477fe700 1 -- 172.21.15.44:0/1630963698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e481002c0 msgr2=0x7f7e48198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.773+0000 7f7e477fe700 1 --2- 172.21.15.44:0/1630963698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e481002c0 0x7f7e48198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.773+0000 7f7e477fe700 1 -- 172.21.15.44:0/1630963698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e48101e40 msgr2=0x7f7e4819f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.773+0000 7f7e477fe700 1 --2- 172.21.15.44:0/1630963698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e48101e40 0x7f7e4819f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.773+0000 7f7e477fe700 1 -- 172.21.15.44:0/1630963698 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e40009d00 con 0x7f7e481014b0 2023-11-16T16:49:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.773+0000 7f7e477fe700 1 --2- 172.21.15.44:0/1630963698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e481014b0 0x7f7e48198880 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c00e9a0 tx=0x7f7e3c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:39.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.773+0000 7f7e457fa700 1 -- 172.21.15.44:0/1630963698 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c009790 con 0x7f7e481014b0 2023-11-16T16:49:39.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.773+0000 7f7e457fa700 1 -- 172.21.15.44:0/1630963698 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e3c010430 con 0x7f7e481014b0 2023-11-16T16:49:39.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.773+0000 7f7e457fa700 1 -- 172.21.15.44:0/1630963698 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c018680 con 0x7f7e481014b0 2023-11-16T16:49:39.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.773+0000 7f7e4e3ff700 1 -- 172.21.15.44:0/1630963698 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e4810bc90 con 0x7f7e481014b0 2023-11-16T16:49:39.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.774+0000 7f7e4e3ff700 1 -- 172.21.15.44:0/1630963698 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e4810bb00 con 0x7f7e481014b0 2023-11-16T16:49:39.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.775+0000 7f7e4e3ff700 1 -- 172.21.15.44:0/1630963698 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e481a0830 con 0x7f7e481014b0 2023-11-16T16:49:39.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.777+0000 7f7e457fa700 1 -- 172.21.15.44:0/1630963698 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7e3c0105a0 con 0x7f7e481014b0 2023-11-16T16:49:39.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.778+0000 7f7e457fa700 1 --2- 172.21.15.44:0/1630963698 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7e300a1150 0x7f7e300a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:39.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.778+0000 7f7e457fa700 1 -- 172.21.15.44:0/1630963698 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f7e3c014070 con 0x7f7e481014b0 2023-11-16T16:49:39.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.778+0000 7f7e47fff700 1 --2- 172.21.15.44:0/1630963698 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7e300a1150 0x7f7e300a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:39.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.779+0000 7f7e47fff700 1 --2- 172.21.15.44:0/1630963698 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7e300a1150 0x7f7e300a3600 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7e48101310 tx=0x7f7e38009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:39.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.782+0000 7f7e457fa700 1 -- 172.21.15.44:0/1630963698 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7e3c051730 con 0x7f7e481014b0 2023-11-16T16:49:39.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:39 smithi044 ceph-mon[128750]: pgmap v349: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:39.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:39 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:49:39.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:39 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:49:39.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:39 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:49:39.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:39 smithi044 ceph-mon[128750]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:39.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:39 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:49:39.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:39 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:49:39.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:39 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T16:49:39.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:39 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2333656773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:39.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:39 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2394539273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:39.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:39 smithi077 ceph-mon[131204]: pgmap v349: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:40.028 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:49:40.028 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:49:40.028 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:39 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:49:40.028 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:39 smithi077 ceph-mon[131204]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:40.028 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:39 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:49:40.029 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:39 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:49:40.029 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:39 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T16:49:40.029 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:39 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2333656773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:40.029 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:39 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2394539273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:40.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.993+0000 7f7e4e3ff700 1 -- 172.21.15.44:0/1630963698 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7e481a0de0 con 0x7f7e481014b0 2023-11-16T16:49:40.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.994+0000 7f7e457fa700 1 -- 172.21.15.44:0/1630963698 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7e3c0d2020 con 0x7f7e481014b0 2023-11-16T16:49:40.030 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2023-11-16T16:49:40.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.996+0000 7f7e4e3ff700 1 -- 172.21.15.44:0/1630963698 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7e300a1150 msgr2=0x7f7e300a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.996+0000 7f7e4e3ff700 1 --2- 172.21.15.44:0/1630963698 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7e300a1150 0x7f7e300a3600 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7e48101310 tx=0x7f7e38009260 comp rx=0 tx=0).stop 2023-11-16T16:49:40.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.996+0000 7f7e4e3ff700 1 -- 172.21.15.44:0/1630963698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e481014b0 msgr2=0x7f7e48198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.996+0000 7f7e4e3ff700 1 --2- 172.21.15.44:0/1630963698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e481014b0 0x7f7e48198880 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c00e9a0 tx=0x7f7e3c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.996+0000 7f7e4e3ff700 1 -- 172.21.15.44:0/1630963698 shutdown_connections 2023-11-16T16:49:40.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.996+0000 7f7e4e3ff700 1 --2- 172.21.15.44:0/1630963698 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f7e300a1150 0x7f7e300a3600 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.996+0000 7f7e4e3ff700 1 --2- 172.21.15.44:0/1630963698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e48101e40 0x7f7e4819f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.996+0000 7f7e4e3ff700 1 --2- 172.21.15.44:0/1630963698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e481002c0 0x7f7e48198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.996+0000 7f7e4e3ff700 1 --2- 172.21.15.44:0/1630963698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e481014b0 0x7f7e48198880 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.996+0000 7f7e4e3ff700 1 -- 172.21.15.44:0/1630963698 >> 172.21.15.44:0/1630963698 conn(0x7f7e480fb8a0 msgr2=0x7f7e48106d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:40.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.996+0000 7f7e4e3ff700 1 -- 172.21.15.44:0/1630963698 shutdown_connections 2023-11-16T16:49:40.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:39.997+0000 7f7e4e3ff700 1 -- 172.21.15.44:0/1630963698 wait complete. 2023-11-16T16:49:40.032 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T16:49:40.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:39 smithi176 ceph-mon[131630]: pgmap v349: 129 pgs: 129 active+clean; 637 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:49:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:39 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:49:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:39 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:49:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:39 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:49:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:39 smithi176 ceph-mon[131630]: from='mgr.14150 172.21.15.44:0/1841572393' entity='mgr.x' 2023-11-16T16:49:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:39 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:49:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:39 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:49:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:39 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T16:49:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:39 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2333656773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:39 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2394539273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:40.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.238+0000 7f95179ff700 1 -- 172.21.15.44:0/394294741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95101002c0 msgr2=0x7f95101006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.238+0000 7f95179ff700 1 --2- 172.21.15.44:0/394294741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95101002c0 0x7f95101006d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f950c009990 tx=0x7f950c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.238+0000 7f95179ff700 1 -- 172.21.15.44:0/394294741 shutdown_connections 2023-11-16T16:49:40.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.238+0000 7f95179ff700 1 --2- 172.21.15.44:0/394294741 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9510101e40 0x7f9510108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.238+0000 7f95179ff700 1 --2- 172.21.15.44:0/394294741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95101002c0 0x7f95101006d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.238+0000 7f95179ff700 1 --2- 172.21.15.44:0/394294741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95101014b0 0x7f9510101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.238+0000 7f95179ff700 1 -- 172.21.15.44:0/394294741 >> 172.21.15.44:0/394294741 conn(0x7f95100fb8a0 msgr2=0x7f95100fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:40.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.238+0000 7f95179ff700 1 -- 172.21.15.44:0/394294741 shutdown_connections 2023-11-16T16:49:40.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.238+0000 7f95179ff700 1 -- 172.21.15.44:0/394294741 wait complete. 2023-11-16T16:49:40.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.239+0000 7f95179ff700 1 Processor -- start 2023-11-16T16:49:40.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.239+0000 7f95179ff700 1 -- start start 2023-11-16T16:49:40.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.239+0000 7f95179ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95101002c0 0x7f95101982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:40.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.239+0000 7f95179ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95101014b0 0x7f95101987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:40.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f95179ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9510101e40 0x7f951019f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:40.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f95179ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951019fe30 con 0x7f95101002c0 2023-11-16T16:49:40.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f95179ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951019ff70 con 0x7f95101014b0 2023-11-16T16:49:40.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f95179ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95101a0100 con 0x7f9510101e40 2023-11-16T16:49:40.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f951579b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95101002c0 0x7f95101982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:40.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f9514f9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95101014b0 0x7f95101987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:40.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f9514f9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95101014b0 0x7f95101987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:49802/0 (socket says 172.21.15.44:49802) 2023-11-16T16:49:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f951579b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95101002c0 0x7f95101982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32772/0 (socket says 172.21.15.44:32772) 2023-11-16T16:49:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f9514f9a700 1 -- 172.21.15.44:0/503880560 learned_addr learned my addr 172.21.15.44:0/503880560 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f9515f9c700 1 --2- 172.21.15.44:0/503880560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9510101e40 0x7f951019f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f9514f9a700 1 -- 172.21.15.44:0/503880560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9510101e40 msgr2=0x7f951019f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f9514f9a700 1 --2- 172.21.15.44:0/503880560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9510101e40 0x7f951019f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f9514f9a700 1 -- 172.21.15.44:0/503880560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95101002c0 msgr2=0x7f95101982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f9514f9a700 1 --2- 172.21.15.44:0/503880560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95101002c0 0x7f95101982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f9514f9a700 1 -- 172.21.15.44:0/503880560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c0095f0 con 0x7f95101014b0 2023-11-16T16:49:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f9515f9c700 1 --2- 172.21.15.44:0/503880560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9510101e40 0x7f951019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:49:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.240+0000 7f951579b700 1 --2- 172.21.15.44:0/503880560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95101002c0 0x7f95101982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.241+0000 7f9514f9a700 1 --2- 172.21.15.44:0/503880560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95101014b0 0x7f95101987f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f950000e9a0 tx=0x7f950000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.241+0000 7f95067fc700 1 -- 172.21.15.44:0/503880560 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500009790 con 0x7f95101014b0 2023-11-16T16:49:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.241+0000 7f95067fc700 1 -- 172.21.15.44:0/503880560 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f950000cbd0 con 0x7f95101014b0 2023-11-16T16:49:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.241+0000 7f95179ff700 1 -- 172.21.15.44:0/503880560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951010bc90 con 0x7f95101014b0 2023-11-16T16:49:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.241+0000 7f95067fc700 1 -- 172.21.15.44:0/503880560 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500018680 con 0x7f95101014b0 2023-11-16T16:49:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.241+0000 7f95179ff700 1 -- 172.21.15.44:0/503880560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951010bb00 con 0x7f95101014b0 2023-11-16T16:49:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.242+0000 7f95179ff700 1 -- 172.21.15.44:0/503880560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95101929f0 con 0x7f95101014b0 2023-11-16T16:49:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.244+0000 7f95067fc700 1 -- 172.21.15.44:0/503880560 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f950000cd40 con 0x7f95101014b0 2023-11-16T16:49:40.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.245+0000 7f95067fc700 1 --2- 172.21.15.44:0/503880560 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f94fc0a1150 0x7f94fc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:40.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.245+0000 7f95067fc700 1 -- 172.21.15.44:0/503880560 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f9500014070 con 0x7f95101014b0 2023-11-16T16:49:40.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.245+0000 7f951579b700 1 --2- 172.21.15.44:0/503880560 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f94fc0a1150 0x7f94fc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:40.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.246+0000 7f951579b700 1 --2- 172.21.15.44:0/503880560 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f94fc0a1150 0x7f94fc0a3600 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f950c00b2e0 tx=0x7f950c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:40.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.249+0000 7f95067fc700 1 -- 172.21.15.44:0/503880560 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9500051730 con 0x7f95101014b0 2023-11-16T16:49:40.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.462+0000 7f95179ff700 1 -- 172.21.15.44:0/503880560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9510061950 con 0x7f95101014b0 2023-11-16T16:49:40.464 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.463+0000 7f95067fc700 1 -- 172.21.15.44:0/503880560 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f95000962b0 con 0x7f95101014b0 2023-11-16T16:49:40.465 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:49:40.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.465+0000 7f95179ff700 1 -- 172.21.15.44:0/503880560 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f94fc0a1150 msgr2=0x7f94fc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.465+0000 7f95179ff700 1 --2- 172.21.15.44:0/503880560 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f94fc0a1150 0x7f94fc0a3600 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f950c00b2e0 tx=0x7f950c005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.465+0000 7f95179ff700 1 -- 172.21.15.44:0/503880560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95101014b0 msgr2=0x7f95101987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.465+0000 7f95179ff700 1 --2- 172.21.15.44:0/503880560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95101014b0 0x7f95101987f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f950000e9a0 tx=0x7f950000ecb0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.466+0000 7f95179ff700 1 -- 172.21.15.44:0/503880560 shutdown_connections 2023-11-16T16:49:40.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.466+0000 7f95179ff700 1 --2- 172.21.15.44:0/503880560 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f94fc0a1150 0x7f94fc0a3600 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.466+0000 7f95179ff700 1 --2- 172.21.15.44:0/503880560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9510101e40 0x7f951019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.466+0000 7f95179ff700 1 --2- 172.21.15.44:0/503880560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95101002c0 0x7f95101982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.466+0000 7f95179ff700 1 --2- 172.21.15.44:0/503880560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95101014b0 0x7f95101987f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.466+0000 7f95179ff700 1 -- 172.21.15.44:0/503880560 >> 172.21.15.44:0/503880560 conn(0x7f95100fb8a0 msgr2=0x7f9510106d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:40.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.466+0000 7f95179ff700 1 -- 172.21.15.44:0/503880560 shutdown_connections 2023-11-16T16:49:40.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.466+0000 7f95179ff700 1 -- 172.21.15.44:0/503880560 wait complete. 2023-11-16T16:49:40.486 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2023-11-16T16:49:40.487 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:49:40.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.691+0000 7f95847ff700 1 -- 172.21.15.44:0/595215007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f957c074b30 msgr2=0x7f957c072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.691+0000 7f95847ff700 1 --2- 172.21.15.44:0/595215007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f957c074b30 0x7f957c072f80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9578009990 tx=0x7f9578009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.691+0000 7f95847ff700 1 -- 172.21.15.44:0/595215007 shutdown_connections 2023-11-16T16:49:40.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.691+0000 7f95847ff700 1 --2- 172.21.15.44:0/595215007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f957c10c860 0x7f957c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.694 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.691+0000 7f95847ff700 1 --2- 172.21.15.44:0/595215007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f957c074b30 0x7f957c072f80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.694 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.691+0000 7f95847ff700 1 --2- 172.21.15.44:0/595215007 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f957c073550 0x7f957c0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.694 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.691+0000 7f95847ff700 1 -- 172.21.15.44:0/595215007 >> 172.21.15.44:0/595215007 conn(0x7f957c0fb870 msgr2=0x7f957c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:40.694 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.691+0000 7f95847ff700 1 -- 172.21.15.44:0/595215007 shutdown_connections 2023-11-16T16:49:40.694 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.692+0000 7f95847ff700 1 -- 172.21.15.44:0/595215007 wait complete. 2023-11-16T16:49:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.692+0000 7f95847ff700 1 Processor -- start 2023-11-16T16:49:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.693+0000 7f95847ff700 1 -- start start 2023-11-16T16:49:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.693+0000 7f95847ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f957c073550 0x7f957c19c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.693+0000 7f95847ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f957c074b30 0x7f957c19cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.693+0000 7f95847ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f957c10c860 0x7f957c1a3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.693+0000 7f95847ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c1a4140 con 0x7f957c10c860 2023-11-16T16:49:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.693+0000 7f95847ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c1a4280 con 0x7f957c074b30 2023-11-16T16:49:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.693+0000 7f95847ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c1a4410 con 0x7f957c073550 2023-11-16T16:49:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.693+0000 7f958259b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f957c073550 0x7f957c19c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.693+0000 7f958259b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f957c073550 0x7f957c19c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:37922/0 (socket says 172.21.15.44:37922) 2023-11-16T16:49:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.693+0000 7f958259b700 1 -- 172.21.15.44:0/3352392595 learned_addr learned my addr 172.21.15.44:0/3352392595 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.693+0000 7f9581d9a700 1 --2- 172.21.15.44:0/3352392595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f957c074b30 0x7f957c19cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.693+0000 7f9582d9c700 1 --2- 172.21.15.44:0/3352392595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f957c10c860 0x7f957c1a3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.694+0000 7f958259b700 1 -- 172.21.15.44:0/3352392595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f957c074b30 msgr2=0x7f957c19cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.694+0000 7f958259b700 1 --2- 172.21.15.44:0/3352392595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f957c074b30 0x7f957c19cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.694+0000 7f958259b700 1 -- 172.21.15.44:0/3352392595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f957c10c860 msgr2=0x7f957c1a3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.694+0000 7f958259b700 1 --2- 172.21.15.44:0/3352392595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f957c10c860 0x7f957c1a3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.694+0000 7f958259b700 1 -- 172.21.15.44:0/3352392595 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956c009520 con 0x7f957c073550 2023-11-16T16:49:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.694+0000 7f958259b700 1 --2- 172.21.15.44:0/3352392595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f957c073550 0x7f957c19c5c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9578005720 tx=0x7f9578009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.694+0000 7f95737fe700 1 -- 172.21.15.44:0/3352392595 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957801d070 con 0x7f957c073550 2023-11-16T16:49:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.694+0000 7f95847ff700 1 -- 172.21.15.44:0/3352392595 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95780095f0 con 0x7f957c073550 2023-11-16T16:49:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.694+0000 7f95737fe700 1 -- 172.21.15.44:0/3352392595 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9578021410 con 0x7f957c073550 2023-11-16T16:49:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.694+0000 7f95737fe700 1 -- 172.21.15.44:0/3352392595 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9578017650 con 0x7f957c073550 2023-11-16T16:49:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.694+0000 7f95847ff700 1 -- 172.21.15.44:0/3352392595 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f957c112250 con 0x7f957c073550 2023-11-16T16:49:40.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.695+0000 7f95847ff700 1 -- 172.21.15.44:0/3352392595 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f957c1120c0 con 0x7f957c073550 2023-11-16T16:49:40.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.697+0000 7f95737fe700 1 -- 172.21.15.44:0/3352392595 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9578021580 con 0x7f957c073550 2023-11-16T16:49:40.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.698+0000 7f95737fe700 1 --2- 172.21.15.44:0/3352392595 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f95680a1150 0x7f95680a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:40.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.699+0000 7f95737fe700 1 -- 172.21.15.44:0/3352392595 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f95780cdeb0 con 0x7f957c073550 2023-11-16T16:49:40.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.699+0000 7f9581d9a700 1 --2- 172.21.15.44:0/3352392595 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f95680a1150 0x7f95680a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:40.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.699+0000 7f9581d9a700 1 --2- 172.21.15.44:0/3352392595 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f95680a1150 0x7f95680a3600 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f957c074850 tx=0x7f956c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:40.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.703+0000 7f95737fe700 1 -- 172.21.15.44:0/3352392595 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9578097290 con 0x7f957c073550 2023-11-16T16:49:40.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:40 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1630963698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:40.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:40 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/503880560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:40.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.958+0000 7f95847ff700 1 -- 172.21.15.44:0/3352392595 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f957c1123e0 con 0x7f957c073550 2023-11-16T16:49:40.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.959+0000 7f95737fe700 1 -- 172.21.15.44:0/3352392595 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f957809a8b0 con 0x7f957c073550 2023-11-16T16:49:40.961 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:40.961 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:49:40.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.961+0000 7f95847ff700 1 -- 172.21.15.44:0/3352392595 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f95680a1150 msgr2=0x7f95680a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.961+0000 7f95847ff700 1 --2- 172.21.15.44:0/3352392595 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f95680a1150 0x7f95680a3600 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f957c074850 tx=0x7f956c009260 comp rx=0 tx=0).stop 2023-11-16T16:49:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.961+0000 7f95847ff700 1 -- 172.21.15.44:0/3352392595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f957c073550 msgr2=0x7f957c19c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.961+0000 7f95847ff700 1 --2- 172.21.15.44:0/3352392595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f957c073550 0x7f957c19c5c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9578005720 tx=0x7f9578009580 comp rx=0 tx=0).stop 2023-11-16T16:49:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.962+0000 7f95847ff700 1 -- 172.21.15.44:0/3352392595 shutdown_connections 2023-11-16T16:49:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.962+0000 7f95847ff700 1 --2- 172.21.15.44:0/3352392595 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f95680a1150 0x7f95680a3600 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.962+0000 7f95847ff700 1 --2- 172.21.15.44:0/3352392595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f957c10c860 0x7f957c1a3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.962+0000 7f95847ff700 1 --2- 172.21.15.44:0/3352392595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f957c073550 0x7f957c19c5c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.962+0000 7f95847ff700 1 --2- 172.21.15.44:0/3352392595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f957c074b30 0x7f957c19cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.962+0000 7f95847ff700 1 -- 172.21.15.44:0/3352392595 >> 172.21.15.44:0/3352392595 conn(0x7f957c0fb870 msgr2=0x7f957c0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.962+0000 7f95847ff700 1 -- 172.21.15.44:0/3352392595 shutdown_connections 2023-11-16T16:49:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:40.962+0000 7f95847ff700 1 -- 172.21.15.44:0/3352392595 wait complete. 2023-11-16T16:49:40.965 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:49:40.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:40 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1630963698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:40.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:40 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/503880560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:40.983 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2023-11-16T16:49:40.983 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2023-11-16T16:49:40.983 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T16:49:41.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:40 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1630963698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:41.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:40 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/503880560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:41.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.186+0000 7f0d297ff700 1 -- 172.21.15.44:0/1192929046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d241002b0 msgr2=0x7f0d241006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:41.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.186+0000 7f0d297ff700 1 --2- 172.21.15.44:0/1192929046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d241002b0 0x7f0d241006c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009990 tx=0x7f0d14009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.187+0000 7f0d297ff700 1 -- 172.21.15.44:0/1192929046 shutdown_connections 2023-11-16T16:49:41.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.187+0000 7f0d297ff700 1 --2- 172.21.15.44:0/1192929046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d24101e30 0x7f0d241086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.187+0000 7f0d297ff700 1 --2- 172.21.15.44:0/1192929046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d241002b0 0x7f0d241006c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.187+0000 7f0d297ff700 1 --2- 172.21.15.44:0/1192929046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d241014a0 0x7f0d241018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.187+0000 7f0d297ff700 1 -- 172.21.15.44:0/1192929046 >> 172.21.15.44:0/1192929046 conn(0x7f0d240fb890 msgr2=0x7f0d240fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:41.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.187+0000 7f0d297ff700 1 -- 172.21.15.44:0/1192929046 shutdown_connections 2023-11-16T16:49:41.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.187+0000 7f0d297ff700 1 -- 172.21.15.44:0/1192929046 wait complete. 2023-11-16T16:49:41.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.188+0000 7f0d297ff700 1 Processor -- start 2023-11-16T16:49:41.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.188+0000 7f0d297ff700 1 -- start start 2023-11-16T16:49:41.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.188+0000 7f0d297ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d241002b0 0x7f0d24198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:41.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.188+0000 7f0d297ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d241014a0 0x7f0d24198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:41.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.188+0000 7f0d297ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d24101e30 0x7f0d2419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:41.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.188+0000 7f0d297ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2419fd80 con 0x7f0d24101e30 2023-11-16T16:49:41.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.188+0000 7f0d297ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2419fec0 con 0x7f0d241014a0 2023-11-16T16:49:41.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.188+0000 7f0d297ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d241a0050 con 0x7f0d241002b0 2023-11-16T16:49:41.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.188+0000 7f0d1bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d241014a0 0x7f0d24198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:41.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.188+0000 7f0d237fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d24101e30 0x7f0d2419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:41.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.188+0000 7f0d1bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d241014a0 0x7f0d24198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:49838/0 (socket says 172.21.15.44:49838) 2023-11-16T16:49:41.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.188+0000 7f0d22ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d241002b0 0x7f0d24198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:41.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.188+0000 7f0d1bfff700 1 -- 172.21.15.44:0/1293468809 learned_addr learned my addr 172.21.15.44:0/1293468809 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:41.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.189+0000 7f0d1bfff700 1 -- 172.21.15.44:0/1293468809 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d241002b0 msgr2=0x7f0d24198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:41.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.189+0000 7f0d1bfff700 1 --2- 172.21.15.44:0/1293468809 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d241002b0 0x7f0d24198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.189+0000 7f0d1bfff700 1 -- 172.21.15.44:0/1293468809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d24101e30 msgr2=0x7f0d2419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:41.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.189+0000 7f0d1bfff700 1 --2- 172.21.15.44:0/1293468809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d24101e30 0x7f0d2419f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.189+0000 7f0d1bfff700 1 -- 172.21.15.44:0/1293468809 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d140095f0 con 0x7f0d241014a0 2023-11-16T16:49:41.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.189+0000 7f0d237fe700 1 --2- 172.21.15.44:0/1293468809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d24101e30 0x7f0d2419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:41.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.189+0000 7f0d1bfff700 1 --2- 172.21.15.44:0/1293468809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d241014a0 0x7f0d24198740 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00e9a0 tx=0x7f0d0c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:41.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.189+0000 7f0d20ff9700 1 -- 172.21.15.44:0/1293468809 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c009790 con 0x7f0d241014a0 2023-11-16T16:49:41.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.189+0000 7f0d297ff700 1 -- 172.21.15.44:0/1293468809 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d2410bc80 con 0x7f0d241014a0 2023-11-16T16:49:41.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.189+0000 7f0d20ff9700 1 -- 172.21.15.44:0/1293468809 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d0c010430 con 0x7f0d241014a0 2023-11-16T16:49:41.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.190+0000 7f0d20ff9700 1 -- 172.21.15.44:0/1293468809 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c018680 con 0x7f0d241014a0 2023-11-16T16:49:41.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.190+0000 7f0d297ff700 1 -- 172.21.15.44:0/1293468809 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d2410baf0 con 0x7f0d241014a0 2023-11-16T16:49:41.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.190+0000 7f0d297ff700 1 -- 172.21.15.44:0/1293468809 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d241a0740 con 0x7f0d241014a0 2023-11-16T16:49:41.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.193+0000 7f0d20ff9700 1 -- 172.21.15.44:0/1293468809 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0d0c0105a0 con 0x7f0d241014a0 2023-11-16T16:49:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.194+0000 7f0d20ff9700 1 --2- 172.21.15.44:0/1293468809 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0d100a1150 0x7f0d100a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.194+0000 7f0d20ff9700 1 -- 172.21.15.44:0/1293468809 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f0d0c014070 con 0x7f0d241014a0 2023-11-16T16:49:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.194+0000 7f0d22ffd700 1 --2- 172.21.15.44:0/1293468809 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0d100a1150 0x7f0d100a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.195+0000 7f0d22ffd700 1 --2- 172.21.15.44:0/1293468809 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0d100a1150 0x7f0d100a3600 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0d1400b2e0 tx=0x7f0d14005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.196+0000 7f0d20ff9700 1 -- 172.21.15.44:0/1293468809 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0d0c096410 con 0x7f0d241014a0 2023-11-16T16:49:41.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.403+0000 7f0d297ff700 1 -- 172.21.15.44:0/1293468809 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d24108450 con 0x7f0d100a1150 2023-11-16T16:49:41.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.417+0000 7f0d20ff9700 1 -- 172.21.15.44:0/1293468809 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426969 (secure 0 0 0) 0x7f0d24108450 con 0x7f0d100a1150 2023-11-16T16:49:41.425 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:41.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.423+0000 7f0d297ff700 1 -- 172.21.15.44:0/1293468809 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0d100a1150 msgr2=0x7f0d100a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:41.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.423+0000 7f0d297ff700 1 --2- 172.21.15.44:0/1293468809 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0d100a1150 0x7f0d100a3600 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0d1400b2e0 tx=0x7f0d14005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.423+0000 7f0d297ff700 1 -- 172.21.15.44:0/1293468809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d241014a0 msgr2=0x7f0d24198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:41.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.423+0000 7f0d297ff700 1 --2- 172.21.15.44:0/1293468809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d241014a0 0x7f0d24198740 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00e9a0 tx=0x7f0d0c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.423+0000 7f0d297ff700 1 -- 172.21.15.44:0/1293468809 shutdown_connections 2023-11-16T16:49:41.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.423+0000 7f0d297ff700 1 --2- 172.21.15.44:0/1293468809 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f0d100a1150 0x7f0d100a3600 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.423+0000 7f0d297ff700 1 --2- 172.21.15.44:0/1293468809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d24101e30 0x7f0d2419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.423+0000 7f0d297ff700 1 --2- 172.21.15.44:0/1293468809 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d241002b0 0x7f0d24198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.423+0000 7f0d297ff700 1 --2- 172.21.15.44:0/1293468809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d241014a0 0x7f0d24198740 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.423+0000 7f0d297ff700 1 -- 172.21.15.44:0/1293468809 >> 172.21.15.44:0/1293468809 conn(0x7f0d240fb890 msgr2=0x7f0d24106d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:41.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.423+0000 7f0d297ff700 1 -- 172.21.15.44:0/1293468809 shutdown_connections 2023-11-16T16:49:41.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.423+0000 7f0d297ff700 1 -- 172.21.15.44:0/1293468809 wait complete. 2023-11-16T16:49:41.430 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2023-11-16T16:49:41.444 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2023-11-16T16:49:40.570933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658264,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":375,"num_read_kb":309,"num_write":267,"num_write_kb":3039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":713,"ondisk_log_size":713,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":82132,"kb_used_data":12020,"kb_used_omap":0,"kb_used_meta":70016,"kb_avail":1124859692,"statfs":{"total":1151940427776,"available":1151856324608,"internally_reserved":0,"allocated":12308480,"data_stored":8034530,"data_compressed":390621,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":0,"internal_metadata":71696384},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5695,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":22,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003677"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.591350+0000","last_change":"2023-11-16T16:47:18.183024+0000","last_active":"2023-11-16T16:49:12.591350+0000","last_peered":"2023-11-16T16:49:12.591350+0000","last_clean":"2023-11-16T16:49:12.591350+0000","last_became_active":"2023-11-16T16:47:18.182387+0000","last_became_peered":"2023-11-16T16:47:18.182387+0000","last_unstale":"2023-11-16T16:49:12.591350+0000","last_undegraded":"2023-11-16T16:49:12.591350+0000","last_fullsized":"2023-11-16T16:49:12.591350+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:08:25.731718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.589033+0000","last_change":"2023-11-16T16:47:18.200135+0000","last_active":"2023-11-16T16:49:12.589033+0000","last_peered":"2023-11-16T16:49:12.589033+0000","last_clean":"2023-11-16T16:49:12.589033+0000","last_became_active":"2023-11-16T16:47:18.194732+0000","last_became_peered":"2023-11-16T16:47:18.194732+0000","last_unstale":"2023-11-16T16:49:12.589033+0000","last_undegraded":"2023-11-16T16:49:12.589033+0000","last_fullsized":"2023-11-16T16:49:12.589033+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:36:09.213648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.589098+0000","last_change":"2023-11-16T16:47:18.184127+0000","last_active":"2023-11-16T16:49:12.589098+0000","last_peered":"2023-11-16T16:49:12.589098+0000","last_clean":"2023-11-16T16:49:12.589098+0000","last_became_active":"2023-11-16T16:47:18.183163+0000","last_became_peered":"2023-11-16T16:47:18.183163+0000","last_unstale":"2023-11-16T16:49:12.589098+0000","last_undegraded":"2023-11-16T16:49:12.589098+0000","last_fullsized":"2023-11-16T16:49:12.589098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:18:16.732196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.577897+0000","last_change":"2023-11-16T16:47:18.177476+0000","last_active":"2023-11-16T16:49:12.577897+0000","last_peered":"2023-11-16T16:49:12.577897+0000","last_clean":"2023-11-16T16:49:12.577897+0000","last_became_active":"2023-11-16T16:47:18.177178+0000","last_became_peered":"2023-11-16T16:47:18.177178+0000","last_unstale":"2023-11-16T16:49:12.577897+0000","last_undegraded":"2023-11-16T16:49:12.577897+0000","last_fullsized":"2023-11-16T16:49:12.577897+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:34:33.524556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.578631+0000","last_change":"2023-11-16T16:47:18.178511+0000","last_active":"2023-11-16T16:49:12.578631+0000","last_peered":"2023-11-16T16:49:12.578631+0000","last_clean":"2023-11-16T16:49:12.578631+0000","last_became_active":"2023-11-16T16:47:18.178198+0000","last_became_peered":"2023-11-16T16:47:18.178198+0000","last_unstale":"2023-11-16T16:49:12.578631+0000","last_undegraded":"2023-11-16T16:49:12.578631+0000","last_fullsized":"2023-11-16T16:49:12.578631+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:04:36.059447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.577044+0000","last_change":"2023-11-16T16:47:18.186552+0000","last_active":"2023-11-16T16:49:12.577044+0000","last_peered":"2023-11-16T16:49:12.577044+0000","last_clean":"2023-11-16T16:49:12.577044+0000","last_became_active":"2023-11-16T16:47:18.186238+0000","last_became_peered":"2023-11-16T16:47:18.186238+0000","last_unstale":"2023-11-16T16:49:12.577044+0000","last_undegraded":"2023-11-16T16:49:12.577044+0000","last_fullsized":"2023-11-16T16:49:12.577044+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:58:35.571757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.566038+0000","last_change":"2023-11-16T16:47:18.179839+0000","last_active":"2023-11-16T16:49:12.566038+0000","last_peered":"2023-11-16T16:49:12.566038+0000","last_clean":"2023-11-16T16:49:12.566038+0000","last_became_active":"2023-11-16T16:47:18.179060+0000","last_became_peered":"2023-11-16T16:47:18.179060+0000","last_unstale":"2023-11-16T16:49:12.566038+0000","last_undegraded":"2023-11-16T16:49:12.566038+0000","last_fullsized":"2023-11-16T16:49:12.566038+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:04:42.573359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.576400+0000","last_change":"2023-11-16T16:47:43.631738+0000","last_active":"2023-11-16T16:49:12.576400+0000","last_peered":"2023-11-16T16:49:12.576400+0000","last_clean":"2023-11-16T16:49:12.576400+0000","last_became_active":"2023-11-16T16:47:43.631146+0000","last_became_peered":"2023-11-16T16:47:43.631146+0000","last_unstale":"2023-11-16T16:49:12.576400+0000","last_undegraded":"2023-11-16T16:49:12.576400+0000","last_fullsized":"2023-11-16T16:49:12.576400+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:58:11.158742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579232+0000","last_change":"2023-11-16T16:47:18.177834+0000","last_active":"2023-11-16T16:49:12.579232+0000","last_peered":"2023-11-16T16:49:12.579232+0000","last_clean":"2023-11-16T16:49:12.579232+0000","last_became_active":"2023-11-16T16:47:18.176299+0000","last_became_peered":"2023-11-16T16:47:18.176299+0000","last_unstale":"2023-11-16T16:49:12.579232+0000","last_undegraded":"2023-11-16T16:49:12.579232+0000","last_fullsized":"2023-11-16T16:49:12.579232+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:27:07.129418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.571605+0000","last_change":"2023-11-16T16:47:18.179394+0000","last_active":"2023-11-16T16:49:12.571605+0000","last_peered":"2023-11-16T16:49:12.571605+0000","last_clean":"2023-11-16T16:49:12.571605+0000","last_became_active":"2023-11-16T16:47:18.179119+0000","last_became_peered":"2023-11-16T16:47:18.179119+0000","last_unstale":"2023-11-16T16:49:12.571605+0000","last_undegraded":"2023-11-16T16:49:12.571605+0000","last_fullsized":"2023-11-16T16:49:12.571605+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:50:38.327278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579257+0000","last_change":"2023-11-16T16:47:18.183585+0000","last_active":"2023-11-16T16:49:12.579257+0000","last_peered":"2023-11-16T16:49:12.579257+0000","last_clean":"2023-11-16T16:49:12.579257+0000","last_became_active":"2023-11-16T16:47:18.183209+0000","last_became_peered":"2023-11-16T16:47:18.183209+0000","last_unstale":"2023-11-16T16:49:12.579257+0000","last_undegraded":"2023-11-16T16:49:12.579257+0000","last_fullsized":"2023-11-16T16:49:12.579257+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:35:11.334704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.570790+0000","last_change":"2023-11-16T16:47:18.186503+0000","last_active":"2023-11-16T16:49:12.570790+0000","last_peered":"2023-11-16T16:49:12.570790+0000","last_clean":"2023-11-16T16:49:12.570790+0000","last_became_active":"2023-11-16T16:47:18.186193+0000","last_became_peered":"2023-11-16T16:47:18.186193+0000","last_unstale":"2023-11-16T16:49:12.570790+0000","last_undegraded":"2023-11-16T16:49:12.570790+0000","last_fullsized":"2023-11-16T16:49:12.570790+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:08:26.864946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.584559+0000","last_change":"2023-11-16T16:47:18.193842+0000","last_active":"2023-11-16T16:49:12.584559+0000","last_peered":"2023-11-16T16:49:12.584559+0000","last_clean":"2023-11-16T16:49:12.584559+0000","last_became_active":"2023-11-16T16:47:18.192309+0000","last_became_peered":"2023-11-16T16:47:18.192309+0000","last_unstale":"2023-11-16T16:49:12.584559+0000","last_undegraded":"2023-11-16T16:49:12.584559+0000","last_fullsized":"2023-11-16T16:49:12.584559+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:47:16.555679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.585449+0000","last_change":"2023-11-16T16:47:18.192359+0000","last_active":"2023-11-16T16:49:12.585449+0000","last_peered":"2023-11-16T16:49:12.585449+0000","last_clean":"2023-11-16T16:49:12.585449+0000","last_became_active":"2023-11-16T16:47:18.189765+0000","last_became_peered":"2023-11-16T16:47:18.189765+0000","last_unstale":"2023-11-16T16:49:12.585449+0000","last_undegraded":"2023-11-16T16:49:12.585449+0000","last_fullsized":"2023-11-16T16:49:12.585449+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:56:51.066489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.588434+0000","last_change":"2023-11-16T16:47:18.178386+0000","last_active":"2023-11-16T16:49:12.588434+0000","last_peered":"2023-11-16T16:49:12.588434+0000","last_clean":"2023-11-16T16:49:12.588434+0000","last_became_active":"2023-11-16T16:47:18.178041+0000","last_became_peered":"2023-11-16T16:47:18.178041+0000","last_unstale":"2023-11-16T16:49:12.588434+0000","last_undegraded":"2023-11-16T16:49:12.588434+0000","last_fullsized":"2023-11-16T16:49:12.588434+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:08:49.324272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.591918+0000","last_change":"2023-11-16T16:47:18.174424+0000","last_active":"2023-11-16T16:49:12.591918+0000","last_peered":"2023-11-16T16:49:12.591918+0000","last_clean":"2023-11-16T16:49:12.591918+0000","last_became_active":"2023-11-16T16:47:18.174078+0000","last_became_peered":"2023-11-16T16:47:18.174078+0000","last_unstale":"2023-11-16T16:49:12.591918+0000","last_undegraded":"2023-11-16T16:49:12.591918+0000","last_fullsized":"2023-11-16T16:49:12.591918+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:23:55.960648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.591349+0000","last_change":"2023-11-16T16:47:18.183161+0000","last_active":"2023-11-16T16:49:12.591349+0000","last_peered":"2023-11-16T16:49:12.591349+0000","last_clean":"2023-11-16T16:49:12.591349+0000","last_became_active":"2023-11-16T16:47:18.182806+0000","last_became_peered":"2023-11-16T16:47:18.182806+0000","last_unstale":"2023-11-16T16:49:12.591349+0000","last_undegraded":"2023-11-16T16:49:12.591349+0000","last_fullsized":"2023-11-16T16:49:12.591349+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:17:48.777833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.580719+0000","last_change":"2023-11-16T16:47:18.187225+0000","last_active":"2023-11-16T16:49:12.580719+0000","last_peered":"2023-11-16T16:49:12.580719+0000","last_clean":"2023-11-16T16:49:12.580719+0000","last_became_active":"2023-11-16T16:47:18.186858+0000","last_became_peered":"2023-11-16T16:47:18.186858+0000","last_unstale":"2023-11-16T16:49:12.580719+0000","last_undegraded":"2023-11-16T16:49:12.580719+0000","last_fullsized":"2023-11-16T16:49:12.580719+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:03:50.025202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579231+0000","last_change":"2023-11-16T16:47:18.190947+0000","last_active":"2023-11-16T16:49:12.579231+0000","last_peered":"2023-11-16T16:49:12.579231+0000","last_clean":"2023-11-16T16:49:12.579231+0000","last_became_active":"2023-11-16T16:47:18.190737+0000","last_became_peered":"2023-11-16T16:47:18.190737+0000","last_unstale":"2023-11-16T16:49:12.579231+0000","last_undegraded":"2023-11-16T16:49:12.579231+0000","last_fullsized":"2023-11-16T16:49:12.579231+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:50:36.934811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623109+0000","last_change":"2023-11-16T16:47:22.659730+0000","last_active":"2023-11-16T16:48:44.623109+0000","last_peered":"2023-11-16T16:48:44.623109+0000","last_clean":"2023-11-16T16:48:44.623109+0000","last_became_active":"2023-11-16T16:47:22.659313+0000","last_became_peered":"2023-11-16T16:47:22.659313+0000","last_unstale":"2023-11-16T16:48:44.623109+0000","last_undegraded":"2023-11-16T16:48:44.623109+0000","last_fullsized":"2023-11-16T16:48:44.623109+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:12:16.481784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'4","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626280+0000","last_change":"2023-11-16T16:47:17.629748+0000","last_active":"2023-11-16T16:48:44.626280+0000","last_peered":"2023-11-16T16:48:44.626280+0000","last_clean":"2023-11-16T16:48:44.626280+0000","last_became_active":"2023-11-16T16:47:17.629284+0000","last_became_peered":"2023-11-16T16:47:17.629284+0000","last_unstale":"2023-11-16T16:48:44.626280+0000","last_undegraded":"2023-11-16T16:48:44.626280+0000","last_fullsized":"2023-11-16T16:48:44.626280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:07:47.593078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.577615+0000","last_change":"2023-11-16T16:47:18.176239+0000","last_active":"2023-11-16T16:49:12.577615+0000","last_peered":"2023-11-16T16:49:12.577615+0000","last_clean":"2023-11-16T16:49:12.577615+0000","last_became_active":"2023-11-16T16:47:18.175879+0000","last_became_peered":"2023-11-16T16:47:18.175879+0000","last_unstale":"2023-11-16T16:49:12.577615+0000","last_undegraded":"2023-11-16T16:49:12.577615+0000","last_fullsized":"2023-11-16T16:49:12.577615+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:31:16.561539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.622732+0000","last_change":"2023-11-16T16:47:22.659679+0000","last_active":"2023-11-16T16:48:44.622732+0000","last_peered":"2023-11-16T16:48:44.622732+0000","last_clean":"2023-11-16T16:48:44.622732+0000","last_became_active":"2023-11-16T16:47:22.658946+0000","last_became_peered":"2023-11-16T16:47:22.658946+0000","last_unstale":"2023-11-16T16:48:44.622732+0000","last_undegraded":"2023-11-16T16:48:44.622732+0000","last_fullsized":"2023-11-16T16:48:44.622732+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:32:34.649897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"82'1","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.912310+0000","last_change":"2023-11-16T16:47:17.643944+0000","last_active":"2023-11-16T16:48:44.912310+0000","last_peered":"2023-11-16T16:48:44.912310+0000","last_clean":"2023-11-16T16:48:44.912310+0000","last_became_active":"2023-11-16T16:47:17.643470+0000","last_became_peered":"2023-11-16T16:47:17.643470+0000","last_unstale":"2023-11-16T16:48:44.912310+0000","last_undegraded":"2023-11-16T16:48:44.912310+0000","last_fullsized":"2023-11-16T16:48:44.912310+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:54:38.723005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.588194+0000","last_change":"2023-11-16T16:47:18.198139+0000","last_active":"2023-11-16T16:49:12.588194+0000","last_peered":"2023-11-16T16:49:12.588194+0000","last_clean":"2023-11-16T16:49:12.588194+0000","last_became_active":"2023-11-16T16:47:18.197762+0000","last_became_peered":"2023-11-16T16:47:18.197762+0000","last_unstale":"2023-11-16T16:49:12.588194+0000","last_undegraded":"2023-11-16T16:49:12.588194+0000","last_fullsized":"2023-11-16T16:49:12.588194+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:24:38.383735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623023+0000","last_change":"2023-11-16T16:47:22.655810+0000","last_active":"2023-11-16T16:48:44.623023+0000","last_peered":"2023-11-16T16:48:44.623023+0000","last_clean":"2023-11-16T16:48:44.623023+0000","last_became_active":"2023-11-16T16:47:22.655388+0000","last_became_peered":"2023-11-16T16:47:22.655388+0000","last_unstale":"2023-11-16T16:48:44.623023+0000","last_undegraded":"2023-11-16T16:48:44.623023+0000","last_fullsized":"2023-11-16T16:48:44.623023+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:44:39.423524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'2","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.630121+0000","last_change":"2023-11-16T16:47:43.635266+0000","last_active":"2023-11-16T16:48:44.630121+0000","last_peered":"2023-11-16T16:48:44.630121+0000","last_clean":"2023-11-16T16:48:44.630121+0000","last_became_active":"2023-11-16T16:47:43.630583+0000","last_became_peered":"2023-11-16T16:47:43.630583+0000","last_unstale":"2023-11-16T16:48:44.630121+0000","last_undegraded":"2023-11-16T16:48:44.630121+0000","last_fullsized":"2023-11-16T16:48:44.630121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:09:31.118965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.585032+0000","last_change":"2023-11-16T16:47:18.186341+0000","last_active":"2023-11-16T16:49:12.585032+0000","last_peered":"2023-11-16T16:49:12.585032+0000","last_clean":"2023-11-16T16:49:12.585032+0000","last_became_active":"2023-11-16T16:47:18.186066+0000","last_became_peered":"2023-11-16T16:47:18.186066+0000","last_unstale":"2023-11-16T16:49:12.585032+0000","last_undegraded":"2023-11-16T16:49:12.585032+0000","last_fullsized":"2023-11-16T16:49:12.585032+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:35:29.489855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.622861+0000","last_change":"2023-11-16T16:47:22.657227+0000","last_active":"2023-11-16T16:48:44.622861+0000","last_peered":"2023-11-16T16:48:44.622861+0000","last_clean":"2023-11-16T16:48:44.622861+0000","last_became_active":"2023-11-16T16:47:22.656835+0000","last_became_peered":"2023-11-16T16:47:22.656835+0000","last_unstale":"2023-11-16T16:48:44.622861+0000","last_undegraded":"2023-11-16T16:48:44.622861+0000","last_fullsized":"2023-11-16T16:48:44.622861+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:55:50.404613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.630547+0000","last_change":"2023-11-16T16:47:17.630194+0000","last_active":"2023-11-16T16:48:44.630547+0000","last_peered":"2023-11-16T16:48:44.630547+0000","last_clean":"2023-11-16T16:48:44.630547+0000","last_became_active":"2023-11-16T16:47:17.629886+0000","last_became_peered":"2023-11-16T16:47:17.629886+0000","last_unstale":"2023-11-16T16:48:44.630547+0000","last_undegraded":"2023-11-16T16:48:44.630547+0000","last_fullsized":"2023-11-16T16:48:44.630547+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:58:10.365614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.573343+0000","last_change":"2023-11-16T16:47:18.181596+0000","last_active":"2023-11-16T16:49:12.573343+0000","last_peered":"2023-11-16T16:49:12.573343+0000","last_clean":"2023-11-16T16:49:12.573343+0000","last_became_active":"2023-11-16T16:47:18.181287+0000","last_became_peered":"2023-11-16T16:47:18.181287+0000","last_unstale":"2023-11-16T16:49:12.573343+0000","last_undegraded":"2023-11-16T16:49:12.573343+0000","last_fullsized":"2023-11-16T16:49:12.573343+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:10:47.679818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623684+0000","last_change":"2023-11-16T16:47:22.649197+0000","last_active":"2023-11-16T16:48:44.623684+0000","last_peered":"2023-11-16T16:48:44.623684+0000","last_clean":"2023-11-16T16:48:44.623684+0000","last_became_active":"2023-11-16T16:47:22.648664+0000","last_became_peered":"2023-11-16T16:47:22.648664+0000","last_unstale":"2023-11-16T16:48:44.623684+0000","last_undegraded":"2023-11-16T16:48:44.623684+0000","last_fullsized":"2023-11-16T16:48:44.623684+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:52:46.052156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"82'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911165+0000","last_change":"2023-11-16T16:47:17.627672+0000","last_active":"2023-11-16T16:48:44.911165+0000","last_peered":"2023-11-16T16:48:44.911165+0000","last_clean":"2023-11-16T16:48:44.911165+0000","last_became_active":"2023-11-16T16:47:17.627383+0000","last_became_peered":"2023-11-16T16:47:17.627383+0000","last_unstale":"2023-11-16T16:48:44.911165+0000","last_undegraded":"2023-11-16T16:48:44.911165+0000","last_fullsized":"2023-11-16T16:48:44.911165+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:51:59.811190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.560868+0000","last_change":"2023-11-16T16:47:18.180983+0000","last_active":"2023-11-16T16:49:12.560868+0000","last_peered":"2023-11-16T16:49:12.560868+0000","last_clean":"2023-11-16T16:49:12.560868+0000","last_became_active":"2023-11-16T16:47:18.180702+0000","last_became_peered":"2023-11-16T16:47:18.180702+0000","last_unstale":"2023-11-16T16:49:12.560868+0000","last_undegraded":"2023-11-16T16:49:12.560868+0000","last_fullsized":"2023-11-16T16:49:12.560868+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:02:50.226183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.913692+0000","last_change":"2023-11-16T16:48:44.913692+0000","last_active":"2023-11-16T16:48:44.913692+0000","last_peered":"2023-11-16T16:48:44.913692+0000","last_clean":"2023-11-16T16:48:44.913692+0000","last_became_active":"2023-11-16T16:48:44.913204+0000","last_became_peered":"2023-11-16T16:48:44.913204+0000","last_unstale":"2023-11-16T16:48:44.913692+0000","last_undegraded":"2023-11-16T16:48:44.913692+0000","last_fullsized":"2023-11-16T16:48:44.913692+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:04:35.705849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"82'1","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626774+0000","last_change":"2023-11-16T16:47:43.949231+0000","last_active":"2023-11-16T16:48:44.626774+0000","last_peered":"2023-11-16T16:48:44.626774+0000","last_clean":"2023-11-16T16:48:44.626774+0000","last_became_active":"2023-11-16T16:47:43.948766+0000","last_became_peered":"2023-11-16T16:47:43.948766+0000","last_unstale":"2023-11-16T16:48:44.626774+0000","last_undegraded":"2023-11-16T16:48:44.626774+0000","last_fullsized":"2023-11-16T16:48:44.626774+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:10:43.974784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579822+0000","last_change":"2023-11-16T16:47:18.185263+0000","last_active":"2023-11-16T16:49:12.579822+0000","last_peered":"2023-11-16T16:49:12.579822+0000","last_clean":"2023-11-16T16:49:12.579822+0000","last_became_active":"2023-11-16T16:47:18.184910+0000","last_became_peered":"2023-11-16T16:47:18.184910+0000","last_unstale":"2023-11-16T16:49:12.579822+0000","last_undegraded":"2023-11-16T16:49:12.579822+0000","last_fullsized":"2023-11-16T16:49:12.579822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:33:21.979376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628573+0000","last_change":"2023-11-16T16:47:22.659711+0000","last_active":"2023-11-16T16:48:44.628573+0000","last_peered":"2023-11-16T16:48:44.628573+0000","last_clean":"2023-11-16T16:48:44.628573+0000","last_became_active":"2023-11-16T16:47:22.659087+0000","last_became_peered":"2023-11-16T16:47:22.659087+0000","last_unstale":"2023-11-16T16:48:44.628573+0000","last_undegraded":"2023-11-16T16:48:44.628573+0000","last_fullsized":"2023-11-16T16:48:44.628573+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:21:48.086846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910749+0000","last_change":"2023-11-16T16:47:17.629703+0000","last_active":"2023-11-16T16:48:44.910749+0000","last_peered":"2023-11-16T16:48:44.910749+0000","last_clean":"2023-11-16T16:48:44.910749+0000","last_became_active":"2023-11-16T16:47:17.629245+0000","last_became_peered":"2023-11-16T16:47:17.629245+0000","last_unstale":"2023-11-16T16:48:44.910749+0000","last_undegraded":"2023-11-16T16:48:44.910749+0000","last_fullsized":"2023-11-16T16:48:44.910749+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:40:19.828594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.584225+0000","last_change":"2023-11-16T16:47:18.187149+0000","last_active":"2023-11-16T16:49:12.584225+0000","last_peered":"2023-11-16T16:49:12.584225+0000","last_clean":"2023-11-16T16:49:12.584225+0000","last_became_active":"2023-11-16T16:47:18.185717+0000","last_became_peered":"2023-11-16T16:47:18.185717+0000","last_unstale":"2023-11-16T16:49:12.584225+0000","last_undegraded":"2023-11-16T16:49:12.584225+0000","last_fullsized":"2023-11-16T16:49:12.584225+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:47:48.993299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.913952+0000","last_change":"2023-11-16T16:47:22.663025+0000","last_active":"2023-11-16T16:48:44.913952+0000","last_peered":"2023-11-16T16:48:44.913952+0000","last_clean":"2023-11-16T16:48:44.913952+0000","last_became_active":"2023-11-16T16:47:22.662596+0000","last_became_peered":"2023-11-16T16:47:22.662596+0000","last_unstale":"2023-11-16T16:48:44.913952+0000","last_undegraded":"2023-11-16T16:48:44.913952+0000","last_fullsized":"2023-11-16T16:48:44.913952+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:01:20.821553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910950+0000","last_change":"2023-11-16T16:47:17.626357+0000","last_active":"2023-11-16T16:48:44.910950+0000","last_peered":"2023-11-16T16:48:44.910950+0000","last_clean":"2023-11-16T16:48:44.910950+0000","last_became_active":"2023-11-16T16:47:17.626029+0000","last_became_peered":"2023-11-16T16:47:17.626029+0000","last_unstale":"2023-11-16T16:48:44.910950+0000","last_undegraded":"2023-11-16T16:48:44.910950+0000","last_fullsized":"2023-11-16T16:48:44.910950+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:55:41.716453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579606+0000","last_change":"2023-11-16T16:47:18.177153+0000","last_active":"2023-11-16T16:49:12.579606+0000","last_peered":"2023-11-16T16:49:12.579606+0000","last_clean":"2023-11-16T16:49:12.579606+0000","last_became_active":"2023-11-16T16:47:18.176751+0000","last_became_peered":"2023-11-16T16:47:18.176751+0000","last_unstale":"2023-11-16T16:49:12.579606+0000","last_undegraded":"2023-11-16T16:49:12.579606+0000","last_fullsized":"2023-11-16T16:49:12.579606+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:21:22.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910720+0000","last_change":"2023-11-16T16:47:22.652718+0000","last_active":"2023-11-16T16:48:44.910720+0000","last_peered":"2023-11-16T16:48:44.910720+0000","last_clean":"2023-11-16T16:48:44.910720+0000","last_became_active":"2023-11-16T16:47:22.652278+0000","last_became_peered":"2023-11-16T16:47:22.652278+0000","last_unstale":"2023-11-16T16:48:44.910720+0000","last_undegraded":"2023-11-16T16:48:44.910720+0000","last_fullsized":"2023-11-16T16:48:44.910720+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:34:31.220318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.627489+0000","last_change":"2023-11-16T16:47:17.627308+0000","last_active":"2023-11-16T16:48:44.627489+0000","last_peered":"2023-11-16T16:48:44.627489+0000","last_clean":"2023-11-16T16:48:44.627489+0000","last_became_active":"2023-11-16T16:47:17.626992+0000","last_became_peered":"2023-11-16T16:47:17.626992+0000","last_unstale":"2023-11-16T16:48:44.627489+0000","last_undegraded":"2023-11-16T16:48:44.627489+0000","last_fullsized":"2023-11-16T16:48:44.627489+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:01:05.366137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579997+0000","last_change":"2023-11-16T16:47:18.188015+0000","last_active":"2023-11-16T16:49:12.579997+0000","last_peered":"2023-11-16T16:49:12.579997+0000","last_clean":"2023-11-16T16:49:12.579997+0000","last_became_active":"2023-11-16T16:47:18.187730+0000","last_became_peered":"2023-11-16T16:47:18.187730+0000","last_unstale":"2023-11-16T16:49:12.579997+0000","last_undegraded":"2023-11-16T16:49:12.579997+0000","last_fullsized":"2023-11-16T16:49:12.579997+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:43:39.411790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.625358+0000","last_change":"2023-11-16T16:47:22.655656+0000","last_active":"2023-11-16T16:48:44.625358+0000","last_peered":"2023-11-16T16:48:44.625358+0000","last_clean":"2023-11-16T16:48:44.625358+0000","last_became_active":"2023-11-16T16:47:22.655133+0000","last_became_peered":"2023-11-16T16:47:22.655133+0000","last_unstale":"2023-11-16T16:48:44.625358+0000","last_undegraded":"2023-11-16T16:48:44.625358+0000","last_fullsized":"2023-11-16T16:48:44.625358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:57:06.052661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626104+0000","last_change":"2023-11-16T16:47:43.947947+0000","last_active":"2023-11-16T16:48:44.626104+0000","last_peered":"2023-11-16T16:48:44.626104+0000","last_clean":"2023-11-16T16:48:44.626104+0000","last_became_active":"2023-11-16T16:47:43.947478+0000","last_became_peered":"2023-11-16T16:47:43.947478+0000","last_unstale":"2023-11-16T16:48:44.626104+0000","last_undegraded":"2023-11-16T16:48:44.626104+0000","last_fullsized":"2023-11-16T16:48:44.626104+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:31:17.754660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.590686+0000","last_change":"2023-11-16T16:47:43.632657+0000","last_active":"2023-11-16T16:49:12.590686+0000","last_peered":"2023-11-16T16:49:12.590686+0000","last_clean":"2023-11-16T16:49:12.590686+0000","last_became_active":"2023-11-16T16:47:43.631573+0000","last_became_peered":"2023-11-16T16:47:43.631573+0000","last_unstale":"2023-11-16T16:49:12.590686+0000","last_undegraded":"2023-11-16T16:49:12.590686+0000","last_fullsized":"2023-11-16T16:49:12.590686+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:22:12.311807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623185+0000","last_change":"2023-11-16T16:47:22.661639+0000","last_active":"2023-11-16T16:48:44.623185+0000","last_peered":"2023-11-16T16:48:44.623185+0000","last_clean":"2023-11-16T16:48:44.623185+0000","last_became_active":"2023-11-16T16:47:22.661230+0000","last_became_peered":"2023-11-16T16:47:22.661230+0000","last_unstale":"2023-11-16T16:48:44.623185+0000","last_undegraded":"2023-11-16T16:48:44.623185+0000","last_fullsized":"2023-11-16T16:48:44.623185+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:25:46.012584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"82'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911151+0000","last_change":"2023-11-16T16:47:17.638209+0000","last_active":"2023-11-16T16:48:44.911151+0000","last_peered":"2023-11-16T16:48:44.911151+0000","last_clean":"2023-11-16T16:48:44.911151+0000","last_became_active":"2023-11-16T16:47:17.637957+0000","last_became_peered":"2023-11-16T16:47:17.637957+0000","last_unstale":"2023-11-16T16:48:44.911151+0000","last_undegraded":"2023-11-16T16:48:44.911151+0000","last_fullsized":"2023-11-16T16:48:44.911151+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:46:12.089473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.580155+0000","last_change":"2023-11-16T16:47:18.182693+0000","last_active":"2023-11-16T16:49:12.580155+0000","last_peered":"2023-11-16T16:49:12.580155+0000","last_clean":"2023-11-16T16:49:12.580155+0000","last_became_active":"2023-11-16T16:47:18.182370+0000","last_became_peered":"2023-11-16T16:47:18.182370+0000","last_unstale":"2023-11-16T16:49:12.580155+0000","last_undegraded":"2023-11-16T16:49:12.580155+0000","last_fullsized":"2023-11-16T16:49:12.580155+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:52.028156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.625577+0000","last_change":"2023-11-16T16:47:22.649348+0000","last_active":"2023-11-16T16:48:44.625577+0000","last_peered":"2023-11-16T16:48:44.625577+0000","last_clean":"2023-11-16T16:48:44.625577+0000","last_became_active":"2023-11-16T16:47:22.648860+0000","last_became_peered":"2023-11-16T16:47:22.648860+0000","last_unstale":"2023-11-16T16:48:44.625577+0000","last_undegraded":"2023-11-16T16:48:44.625577+0000","last_fullsized":"2023-11-16T16:48:44.625577+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:13:04.049776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'37","reported_seq":301,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:39.352456+0000","last_change":"2023-11-16T16:47:17.619417+0000","last_active":"2023-11-16T16:49:39.352456+0000","last_peered":"2023-11-16T16:49:39.352456+0000","last_clean":"2023-11-16T16:49:39.352456+0000","last_became_active":"2023-11-16T16:47:17.619086+0000","last_became_peered":"2023-11-16T16:47:17.619086+0000","last_unstale":"2023-11-16T16:49:39.352456+0000","last_undegraded":"2023-11-16T16:49:39.352456+0000","last_fullsized":"2023-11-16T16:49:39.352456+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:47:53.380456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66852,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":233,"num_read_kb":184,"num_write":37,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.566501+0000","last_change":"2023-11-16T16:47:18.177537+0000","last_active":"2023-11-16T16:49:12.566501+0000","last_peered":"2023-11-16T16:49:12.566501+0000","last_clean":"2023-11-16T16:49:12.566501+0000","last_became_active":"2023-11-16T16:47:18.177112+0000","last_became_peered":"2023-11-16T16:47:18.177112+0000","last_unstale":"2023-11-16T16:49:12.566501+0000","last_undegraded":"2023-11-16T16:49:12.566501+0000","last_fullsized":"2023-11-16T16:49:12.566501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:46:36.459769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911616+0000","last_change":"2023-11-16T16:47:22.658086+0000","last_active":"2023-11-16T16:48:44.911616+0000","last_peered":"2023-11-16T16:48:44.911616+0000","last_clean":"2023-11-16T16:48:44.911616+0000","last_became_active":"2023-11-16T16:47:22.657725+0000","last_became_peered":"2023-11-16T16:47:22.657725+0000","last_unstale":"2023-11-16T16:48:44.911616+0000","last_undegraded":"2023-11-16T16:48:44.911616+0000","last_fullsized":"2023-11-16T16:48:44.911616+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:20:10.492766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"82'2","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623489+0000","last_change":"2023-11-16T16:47:17.625405+0000","last_active":"2023-11-16T16:48:44.623489+0000","last_peered":"2023-11-16T16:48:44.623489+0000","last_clean":"2023-11-16T16:48:44.623489+0000","last_became_active":"2023-11-16T16:47:17.625026+0000","last_became_peered":"2023-11-16T16:47:17.625026+0000","last_unstale":"2023-11-16T16:48:44.623489+0000","last_undegraded":"2023-11-16T16:48:44.623489+0000","last_fullsized":"2023-11-16T16:48:44.623489+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:07:58.847671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.582795+0000","last_change":"2023-11-16T16:47:18.196345+0000","last_active":"2023-11-16T16:49:12.582795+0000","last_peered":"2023-11-16T16:49:12.582795+0000","last_clean":"2023-11-16T16:49:12.582795+0000","last_became_active":"2023-11-16T16:47:18.196040+0000","last_became_peered":"2023-11-16T16:47:18.196040+0000","last_unstale":"2023-11-16T16:49:12.582795+0000","last_undegraded":"2023-11-16T16:49:12.582795+0000","last_fullsized":"2023-11-16T16:49:12.582795+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:07:23.062566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.914613+0000","last_change":"2023-11-16T16:48:44.914613+0000","last_active":"2023-11-16T16:48:44.914613+0000","last_peered":"2023-11-16T16:48:44.914613+0000","last_clean":"2023-11-16T16:48:44.914613+0000","last_became_active":"2023-11-16T16:48:44.914216+0000","last_became_peered":"2023-11-16T16:48:44.914216+0000","last_unstale":"2023-11-16T16:48:44.914613+0000","last_undegraded":"2023-11-16T16:48:44.914613+0000","last_fullsized":"2023-11-16T16:48:44.914613+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:40:27.544928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911496+0000","last_change":"2023-11-16T16:47:17.630481+0000","last_active":"2023-11-16T16:48:44.911496+0000","last_peered":"2023-11-16T16:48:44.911496+0000","last_clean":"2023-11-16T16:48:44.911496+0000","last_became_active":"2023-11-16T16:47:17.630190+0000","last_became_peered":"2023-11-16T16:47:17.630190+0000","last_unstale":"2023-11-16T16:48:44.911496+0000","last_undegraded":"2023-11-16T16:48:44.911496+0000","last_fullsized":"2023-11-16T16:48:44.911496+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:22:25.493507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.585638+0000","last_change":"2023-11-16T16:48:44.913061+0000","last_active":"2023-11-16T16:49:12.585638+0000","last_peered":"2023-11-16T16:49:12.585638+0000","last_clean":"2023-11-16T16:49:12.585638+0000","last_became_active":"2023-11-16T16:48:44.912616+0000","last_became_peered":"2023-11-16T16:48:44.912616+0000","last_unstale":"2023-11-16T16:49:12.585638+0000","last_undegraded":"2023-11-16T16:49:12.585638+0000","last_fullsized":"2023-11-16T16:49:12.585638+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:12:42.146437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626024+0000","last_change":"2023-11-16T16:47:22.656312+0000","last_active":"2023-11-16T16:48:44.626024+0000","last_peered":"2023-11-16T16:48:44.626024+0000","last_clean":"2023-11-16T16:48:44.626024+0000","last_became_active":"2023-11-16T16:47:22.655870+0000","last_became_peered":"2023-11-16T16:47:22.655870+0000","last_unstale":"2023-11-16T16:48:44.626024+0000","last_undegraded":"2023-11-16T16:48:44.626024+0000","last_fullsized":"2023-11-16T16:48:44.626024+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:26:18.515418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.622957+0000","last_change":"2023-11-16T16:47:17.622648+0000","last_active":"2023-11-16T16:48:44.622957+0000","last_peered":"2023-11-16T16:48:44.622957+0000","last_clean":"2023-11-16T16:48:44.622957+0000","last_became_active":"2023-11-16T16:47:17.622329+0000","last_became_peered":"2023-11-16T16:47:17.622329+0000","last_unstale":"2023-11-16T16:48:44.622957+0000","last_undegraded":"2023-11-16T16:48:44.622957+0000","last_fullsized":"2023-11-16T16:48:44.622957+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:04:26.204200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.569410+0000","last_change":"2023-11-16T16:47:18.179712+0000","last_active":"2023-11-16T16:49:12.569410+0000","last_peered":"2023-11-16T16:49:12.569410+0000","last_clean":"2023-11-16T16:49:12.569410+0000","last_became_active":"2023-11-16T16:47:18.179392+0000","last_became_peered":"2023-11-16T16:47:18.179392+0000","last_unstale":"2023-11-16T16:49:12.569410+0000","last_undegraded":"2023-11-16T16:49:12.569410+0000","last_fullsized":"2023-11-16T16:49:12.569410+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:14:52.593975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.912858+0000","last_change":"2023-11-16T16:47:22.665088+0000","last_active":"2023-11-16T16:48:44.912858+0000","last_peered":"2023-11-16T16:48:44.912858+0000","last_clean":"2023-11-16T16:48:44.912858+0000","last_became_active":"2023-11-16T16:47:22.662802+0000","last_became_peered":"2023-11-16T16:47:22.662802+0000","last_unstale":"2023-11-16T16:48:44.912858+0000","last_undegraded":"2023-11-16T16:48:44.912858+0000","last_fullsized":"2023-11-16T16:48:44.912858+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:30:36.233143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"82'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628107+0000","last_change":"2023-11-16T16:47:17.639246+0000","last_active":"2023-11-16T16:48:44.628107+0000","last_peered":"2023-11-16T16:48:44.628107+0000","last_clean":"2023-11-16T16:48:44.628107+0000","last_became_active":"2023-11-16T16:47:17.638928+0000","last_became_peered":"2023-11-16T16:47:17.638928+0000","last_unstale":"2023-11-16T16:48:44.628107+0000","last_undegraded":"2023-11-16T16:48:44.628107+0000","last_fullsized":"2023-11-16T16:48:44.628107+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:18:59.651350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.568411+0000","last_change":"2023-11-16T16:47:18.182609+0000","last_active":"2023-11-16T16:49:12.568411+0000","last_peered":"2023-11-16T16:49:12.568411+0000","last_clean":"2023-11-16T16:49:12.568411+0000","last_became_active":"2023-11-16T16:47:18.182260+0000","last_became_peered":"2023-11-16T16:47:18.182260+0000","last_unstale":"2023-11-16T16:49:12.568411+0000","last_undegraded":"2023-11-16T16:49:12.568411+0000","last_fullsized":"2023-11-16T16:49:12.568411+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:31:25.530504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911115+0000","last_change":"2023-11-16T16:47:22.647586+0000","last_active":"2023-11-16T16:48:44.911115+0000","last_peered":"2023-11-16T16:48:44.911115+0000","last_clean":"2023-11-16T16:48:44.911115+0000","last_became_active":"2023-11-16T16:47:22.647187+0000","last_became_peered":"2023-11-16T16:47:22.647187+0000","last_unstale":"2023-11-16T16:48:44.911115+0000","last_undegraded":"2023-11-16T16:48:44.911115+0000","last_fullsized":"2023-11-16T16:48:44.911115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:29:19.788377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628288+0000","last_change":"2023-11-16T16:47:43.933501+0000","last_active":"2023-11-16T16:48:44.628288+0000","last_peered":"2023-11-16T16:48:44.628288+0000","last_clean":"2023-11-16T16:48:44.628288+0000","last_became_active":"2023-11-16T16:47:43.631746+0000","last_became_peered":"2023-11-16T16:47:43.631746+0000","last_unstale":"2023-11-16T16:48:44.628288+0000","last_undegraded":"2023-11-16T16:48:44.628288+0000","last_fullsized":"2023-11-16T16:48:44.628288+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:01:43.174333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.567503+0000","last_change":"2023-11-16T16:47:18.175210+0000","last_active":"2023-11-16T16:49:12.567503+0000","last_peered":"2023-11-16T16:49:12.567503+0000","last_clean":"2023-11-16T16:49:12.567503+0000","last_became_active":"2023-11-16T16:47:18.174926+0000","last_became_peered":"2023-11-16T16:47:18.174926+0000","last_unstale":"2023-11-16T16:49:12.567503+0000","last_undegraded":"2023-11-16T16:49:12.567503+0000","last_fullsized":"2023-11-16T16:49:12.567503+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:51:21.318906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628055+0000","last_change":"2023-11-16T16:47:22.719361+0000","last_active":"2023-11-16T16:48:44.628055+0000","last_peered":"2023-11-16T16:48:44.628055+0000","last_clean":"2023-11-16T16:48:44.628055+0000","last_became_active":"2023-11-16T16:47:22.718936+0000","last_became_peered":"2023-11-16T16:47:22.718936+0000","last_unstale":"2023-11-16T16:48:44.628055+0000","last_undegraded":"2023-11-16T16:48:44.628055+0000","last_fullsized":"2023-11-16T16:48:44.628055+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:07:12.278332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626068+0000","last_change":"2023-11-16T16:47:17.632623+0000","last_active":"2023-11-16T16:48:44.626068+0000","last_peered":"2023-11-16T16:48:44.626068+0000","last_clean":"2023-11-16T16:48:44.626068+0000","last_became_active":"2023-11-16T16:47:17.631323+0000","last_became_peered":"2023-11-16T16:47:17.631323+0000","last_unstale":"2023-11-16T16:48:44.626068+0000","last_undegraded":"2023-11-16T16:48:44.626068+0000","last_fullsized":"2023-11-16T16:48:44.626068+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:15:53.760791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.584748+0000","last_change":"2023-11-16T16:47:18.190779+0000","last_active":"2023-11-16T16:49:12.584748+0000","last_peered":"2023-11-16T16:49:12.584748+0000","last_clean":"2023-11-16T16:49:12.584748+0000","last_became_active":"2023-11-16T16:47:18.190440+0000","last_became_peered":"2023-11-16T16:47:18.190440+0000","last_unstale":"2023-11-16T16:49:12.584748+0000","last_undegraded":"2023-11-16T16:49:12.584748+0000","last_fullsized":"2023-11-16T16:49:12.584748+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:42:49.621136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911253+0000","last_change":"2023-11-16T16:47:22.664847+0000","last_active":"2023-11-16T16:48:44.911253+0000","last_peered":"2023-11-16T16:48:44.911253+0000","last_clean":"2023-11-16T16:48:44.911253+0000","last_became_active":"2023-11-16T16:47:22.659382+0000","last_became_peered":"2023-11-16T16:47:22.659382+0000","last_unstale":"2023-11-16T16:48:44.911253+0000","last_undegraded":"2023-11-16T16:48:44.911253+0000","last_fullsized":"2023-11-16T16:48:44.911253+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:25:48.404036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.625730+0000","last_change":"2023-11-16T16:47:17.630956+0000","last_active":"2023-11-16T16:48:44.625730+0000","last_peered":"2023-11-16T16:48:44.625730+0000","last_clean":"2023-11-16T16:48:44.625730+0000","last_became_active":"2023-11-16T16:47:17.629864+0000","last_became_peered":"2023-11-16T16:47:17.629864+0000","last_unstale":"2023-11-16T16:48:44.625730+0000","last_undegraded":"2023-11-16T16:48:44.625730+0000","last_fullsized":"2023-11-16T16:48:44.625730+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:34:09.169015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.581549+0000","last_change":"2023-11-16T16:47:18.183375+0000","last_active":"2023-11-16T16:49:12.581549+0000","last_peered":"2023-11-16T16:49:12.581549+0000","last_clean":"2023-11-16T16:49:12.581549+0000","last_became_active":"2023-11-16T16:47:18.183050+0000","last_became_peered":"2023-11-16T16:47:18.183050+0000","last_unstale":"2023-11-16T16:49:12.581549+0000","last_undegraded":"2023-11-16T16:49:12.581549+0000","last_fullsized":"2023-11-16T16:49:12.581549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:13:59.684725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911824+0000","last_change":"2023-11-16T16:47:22.719109+0000","last_active":"2023-11-16T16:48:44.911824+0000","last_peered":"2023-11-16T16:48:44.911824+0000","last_clean":"2023-11-16T16:48:44.911824+0000","last_became_active":"2023-11-16T16:47:22.718680+0000","last_became_peered":"2023-11-16T16:47:22.718680+0000","last_unstale":"2023-11-16T16:48:44.911824+0000","last_undegraded":"2023-11-16T16:48:44.911824+0000","last_fullsized":"2023-11-16T16:48:44.911824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:23:29.879756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.569880+0000","last_change":"2023-11-16T16:47:18.181632+0000","last_active":"2023-11-16T16:49:12.569880+0000","last_peered":"2023-11-16T16:49:12.569880+0000","last_clean":"2023-11-16T16:49:12.569880+0000","last_became_active":"2023-11-16T16:47:18.181323+0000","last_became_peered":"2023-11-16T16:47:18.181323+0000","last_unstale":"2023-11-16T16:49:12.569880+0000","last_undegraded":"2023-11-16T16:49:12.569880+0000","last_fullsized":"2023-11-16T16:49:12.569880+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:02:57.803797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910665+0000","last_change":"2023-11-16T16:47:17.637269+0000","last_active":"2023-11-16T16:48:44.910665+0000","last_peered":"2023-11-16T16:48:44.910665+0000","last_clean":"2023-11-16T16:48:44.910665+0000","last_became_active":"2023-11-16T16:47:17.636881+0000","last_became_peered":"2023-11-16T16:47:17.636881+0000","last_unstale":"2023-11-16T16:48:44.910665+0000","last_undegraded":"2023-11-16T16:48:44.910665+0000","last_fullsized":"2023-11-16T16:48:44.910665+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:15:29.077370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626370+0000","last_change":"2023-11-16T16:47:22.659528+0000","last_active":"2023-11-16T16:48:44.626370+0000","last_peered":"2023-11-16T16:48:44.626370+0000","last_clean":"2023-11-16T16:48:44.626370+0000","last_became_active":"2023-11-16T16:47:22.659076+0000","last_became_peered":"2023-11-16T16:47:22.659076+0000","last_unstale":"2023-11-16T16:48:44.626370+0000","last_undegraded":"2023-11-16T16:48:44.626370+0000","last_fullsized":"2023-11-16T16:48:44.626370+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:22:26.166083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579137+0000","last_change":"2023-11-16T16:47:18.182102+0000","last_active":"2023-11-16T16:49:12.579137+0000","last_peered":"2023-11-16T16:49:12.579137+0000","last_clean":"2023-11-16T16:49:12.579137+0000","last_became_active":"2023-11-16T16:47:18.181528+0000","last_became_peered":"2023-11-16T16:47:18.181528+0000","last_unstale":"2023-11-16T16:49:12.579137+0000","last_undegraded":"2023-11-16T16:49:12.579137+0000","last_fullsized":"2023-11-16T16:49:12.579137+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:07:02.787319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623090+0000","last_change":"2023-11-16T16:47:17.626655+0000","last_active":"2023-11-16T16:48:44.623090+0000","last_peered":"2023-11-16T16:48:44.623090+0000","last_clean":"2023-11-16T16:48:44.623090+0000","last_became_active":"2023-11-16T16:47:17.626329+0000","last_became_peered":"2023-11-16T16:47:17.626329+0000","last_unstale":"2023-11-16T16:48:44.623090+0000","last_undegraded":"2023-11-16T16:48:44.623090+0000","last_fullsized":"2023-11-16T16:48:44.623090+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:22:13.860674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":357,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.625369+0000","last_change":"2023-11-16T16:46:11.943398+0000","last_active":"2023-11-16T16:48:44.625369+0000","last_peered":"2023-11-16T16:48:44.625369+0000","last_clean":"2023-11-16T16:48:44.625369+0000","last_became_active":"2023-11-16T16:46:11.942862+0000","last_became_peered":"2023-11-16T16:46:11.942862+0000","last_unstale":"2023-11-16T16:48:44.625369+0000","last_undegraded":"2023-11-16T16:48:44.625369+0000","last_fullsized":"2023-11-16T16:48:44.625369+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:43:03.109797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:43:03.109797+0000","last_clean_scrub_stamp":"2023-11-16T16:43:03.109797+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:27:18.973668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.915360+0000","last_change":"2023-11-16T16:48:44.914761+0000","last_active":"2023-11-16T16:48:44.915360+0000","last_peered":"2023-11-16T16:48:44.915360+0000","last_clean":"2023-11-16T16:48:44.915360+0000","last_became_active":"2023-11-16T16:48:44.914365+0000","last_became_peered":"2023-11-16T16:48:44.914365+0000","last_unstale":"2023-11-16T16:48:44.915360+0000","last_undegraded":"2023-11-16T16:48:44.915360+0000","last_fullsized":"2023-11-16T16:48:44.915360+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:54:07.544631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579927+0000","last_change":"2023-11-16T16:47:18.184630+0000","last_active":"2023-11-16T16:49:12.579927+0000","last_peered":"2023-11-16T16:49:12.579927+0000","last_clean":"2023-11-16T16:49:12.579927+0000","last_became_active":"2023-11-16T16:47:18.181978+0000","last_became_peered":"2023-11-16T16:47:18.181978+0000","last_unstale":"2023-11-16T16:49:12.579927+0000","last_undegraded":"2023-11-16T16:49:12.579927+0000","last_fullsized":"2023-11-16T16:49:12.579927+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:18:09.046316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628159+0000","last_change":"2023-11-16T16:47:43.635365+0000","last_active":"2023-11-16T16:48:44.628159+0000","last_peered":"2023-11-16T16:48:44.628159+0000","last_clean":"2023-11-16T16:48:44.628159+0000","last_became_active":"2023-11-16T16:47:43.634974+0000","last_became_peered":"2023-11-16T16:47:43.634974+0000","last_unstale":"2023-11-16T16:48:44.628159+0000","last_undegraded":"2023-11-16T16:48:44.628159+0000","last_fullsized":"2023-11-16T16:48:44.628159+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:34:17.420977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.627888+0000","last_change":"2023-11-16T16:47:22.657133+0000","last_active":"2023-11-16T16:48:44.627888+0000","last_peered":"2023-11-16T16:48:44.627888+0000","last_clean":"2023-11-16T16:48:44.627888+0000","last_became_active":"2023-11-16T16:47:22.656557+0000","last_became_peered":"2023-11-16T16:47:22.656557+0000","last_unstale":"2023-11-16T16:48:44.627888+0000","last_undegraded":"2023-11-16T16:48:44.627888+0000","last_fullsized":"2023-11-16T16:48:44.627888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:50:31.162768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.629935+0000","last_change":"2023-11-16T16:47:17.639107+0000","last_active":"2023-11-16T16:48:44.629935+0000","last_peered":"2023-11-16T16:48:44.629935+0000","last_clean":"2023-11-16T16:48:44.629935+0000","last_became_active":"2023-11-16T16:47:17.638796+0000","last_became_peered":"2023-11-16T16:47:17.638796+0000","last_unstale":"2023-11-16T16:48:44.629935+0000","last_undegraded":"2023-11-16T16:48:44.629935+0000","last_fullsized":"2023-11-16T16:48:44.629935+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:55:19.889558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.581882+0000","last_change":"2023-11-16T16:47:18.186761+0000","last_active":"2023-11-16T16:49:12.581882+0000","last_peered":"2023-11-16T16:49:12.581882+0000","last_clean":"2023-11-16T16:49:12.581882+0000","last_became_active":"2023-11-16T16:47:18.185594+0000","last_became_peered":"2023-11-16T16:47:18.185594+0000","last_unstale":"2023-11-16T16:49:12.581882+0000","last_undegraded":"2023-11-16T16:49:12.581882+0000","last_fullsized":"2023-11-16T16:49:12.581882+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:29:09.490006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.912099+0000","last_change":"2023-11-16T16:47:22.719534+0000","last_active":"2023-11-16T16:48:44.912099+0000","last_peered":"2023-11-16T16:48:44.912099+0000","last_clean":"2023-11-16T16:48:44.912099+0000","last_became_active":"2023-11-16T16:47:22.718738+0000","last_became_peered":"2023-11-16T16:47:22.718738+0000","last_unstale":"2023-11-16T16:48:44.912099+0000","last_undegraded":"2023-11-16T16:48:44.912099+0000","last_fullsized":"2023-11-16T16:48:44.912099+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:48:26.395670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'5","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:17.091818+0000","last_change":"2023-11-16T16:47:17.630318+0000","last_active":"2023-11-16T16:49:17.091818+0000","last_peered":"2023-11-16T16:49:17.091818+0000","last_clean":"2023-11-16T16:49:17.091818+0000","last_became_active":"2023-11-16T16:47:17.629035+0000","last_became_peered":"2023-11-16T16:47:17.629035+0000","last_unstale":"2023-11-16T16:49:17.091818+0000","last_undegraded":"2023-11-16T16:49:17.091818+0000","last_fullsized":"2023-11-16T16:49:17.091818+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:34:15.724433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.574220+0000","last_change":"2023-11-16T16:47:18.185034+0000","last_active":"2023-11-16T16:49:12.574220+0000","last_peered":"2023-11-16T16:49:12.574220+0000","last_clean":"2023-11-16T16:49:12.574220+0000","last_became_active":"2023-11-16T16:47:18.181961+0000","last_became_peered":"2023-11-16T16:47:18.181961+0000","last_unstale":"2023-11-16T16:49:12.574220+0000","last_undegraded":"2023-11-16T16:49:12.574220+0000","last_fullsized":"2023-11-16T16:49:12.574220+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:51:00.509275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623131+0000","last_change":"2023-11-16T16:47:22.649952+0000","last_active":"2023-11-16T16:48:44.623131+0000","last_peered":"2023-11-16T16:48:44.623131+0000","last_clean":"2023-11-16T16:48:44.623131+0000","last_became_active":"2023-11-16T16:47:22.649481+0000","last_became_peered":"2023-11-16T16:47:22.649481+0000","last_unstale":"2023-11-16T16:48:44.623131+0000","last_undegraded":"2023-11-16T16:48:44.623131+0000","last_fullsized":"2023-11-16T16:48:44.623131+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:52:18.082684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"82'2","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623572+0000","last_change":"2023-11-16T16:47:17.631982+0000","last_active":"2023-11-16T16:48:44.623572+0000","last_peered":"2023-11-16T16:48:44.623572+0000","last_clean":"2023-11-16T16:48:44.623572+0000","last_became_active":"2023-11-16T16:47:17.631552+0000","last_became_peered":"2023-11-16T16:47:17.631552+0000","last_unstale":"2023-11-16T16:48:44.623572+0000","last_undegraded":"2023-11-16T16:48:44.623572+0000","last_fullsized":"2023-11-16T16:48:44.623572+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:23:15.968583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.589149+0000","last_change":"2023-11-16T16:47:18.197325+0000","last_active":"2023-11-16T16:49:12.589149+0000","last_peered":"2023-11-16T16:49:12.589149+0000","last_clean":"2023-11-16T16:49:12.589149+0000","last_became_active":"2023-11-16T16:47:18.190602+0000","last_became_peered":"2023-11-16T16:47:18.190602+0000","last_unstale":"2023-11-16T16:49:12.589149+0000","last_undegraded":"2023-11-16T16:49:12.589149+0000","last_fullsized":"2023-11-16T16:49:12.589149+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:21:06.387535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623556+0000","last_change":"2023-11-16T16:47:22.720023+0000","last_active":"2023-11-16T16:48:44.623556+0000","last_peered":"2023-11-16T16:48:44.623556+0000","last_clean":"2023-11-16T16:48:44.623556+0000","last_became_active":"2023-11-16T16:47:22.719561+0000","last_became_peered":"2023-11-16T16:47:22.719561+0000","last_unstale":"2023-11-16T16:48:44.623556+0000","last_undegraded":"2023-11-16T16:48:44.623556+0000","last_fullsized":"2023-11-16T16:48:44.623556+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:06:42.918943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.647594+0000","last_change":"2023-11-16T16:47:17.629379+0000","last_active":"2023-11-16T16:48:44.647594+0000","last_peered":"2023-11-16T16:48:44.647594+0000","last_clean":"2023-11-16T16:48:44.647594+0000","last_became_active":"2023-11-16T16:47:17.629025+0000","last_became_peered":"2023-11-16T16:47:17.629025+0000","last_unstale":"2023-11-16T16:48:44.647594+0000","last_undegraded":"2023-11-16T16:48:44.647594+0000","last_fullsized":"2023-11-16T16:48:44.647594+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:38:07.996802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.566023+0000","last_change":"2023-11-16T16:47:18.185418+0000","last_active":"2023-11-16T16:49:12.566023+0000","last_peered":"2023-11-16T16:49:12.566023+0000","last_clean":"2023-11-16T16:49:12.566023+0000","last_became_active":"2023-11-16T16:47:18.185092+0000","last_became_peered":"2023-11-16T16:47:18.185092+0000","last_unstale":"2023-11-16T16:49:12.566023+0000","last_undegraded":"2023-11-16T16:49:12.566023+0000","last_fullsized":"2023-11-16T16:49:12.566023+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:57:19.899302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.913330+0000","last_change":"2023-11-16T16:48:44.913330+0000","last_active":"2023-11-16T16:48:44.913330+0000","last_peered":"2023-11-16T16:48:44.913330+0000","last_clean":"2023-11-16T16:48:44.913330+0000","last_became_active":"2023-11-16T16:48:44.912707+0000","last_became_peered":"2023-11-16T16:48:44.912707+0000","last_unstale":"2023-11-16T16:48:44.913330+0000","last_undegraded":"2023-11-16T16:48:44.913330+0000","last_fullsized":"2023-11-16T16:48:44.913330+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:12:18.512977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910905+0000","last_change":"2023-11-16T16:47:17.638622+0000","last_active":"2023-11-16T16:48:44.910905+0000","last_peered":"2023-11-16T16:48:44.910905+0000","last_clean":"2023-11-16T16:48:44.910905+0000","last_became_active":"2023-11-16T16:47:17.638303+0000","last_became_peered":"2023-11-16T16:47:17.638303+0000","last_unstale":"2023-11-16T16:48:44.910905+0000","last_undegraded":"2023-11-16T16:48:44.910905+0000","last_fullsized":"2023-11-16T16:48:44.910905+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:21:41.090636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.590794+0000","last_change":"2023-11-16T16:47:18.173474+0000","last_active":"2023-11-16T16:49:12.590794+0000","last_peered":"2023-11-16T16:49:12.590794+0000","last_clean":"2023-11-16T16:49:12.590794+0000","last_became_active":"2023-11-16T16:47:18.173140+0000","last_became_peered":"2023-11-16T16:47:18.173140+0000","last_unstale":"2023-11-16T16:49:12.590794+0000","last_undegraded":"2023-11-16T16:49:12.590794+0000","last_fullsized":"2023-11-16T16:49:12.590794+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:30:41.318669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623128+0000","last_change":"2023-11-16T16:47:22.655806+0000","last_active":"2023-11-16T16:48:44.623128+0000","last_peered":"2023-11-16T16:48:44.623128+0000","last_clean":"2023-11-16T16:48:44.623128+0000","last_became_active":"2023-11-16T16:47:22.655385+0000","last_became_peered":"2023-11-16T16:47:22.655385+0000","last_unstale":"2023-11-16T16:48:44.623128+0000","last_undegraded":"2023-11-16T16:48:44.623128+0000","last_fullsized":"2023-11-16T16:48:44.623128+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:15:10.616521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623263+0000","last_change":"2023-11-16T16:47:17.631673+0000","last_active":"2023-11-16T16:48:44.623263+0000","last_peered":"2023-11-16T16:48:44.623263+0000","last_clean":"2023-11-16T16:48:44.623263+0000","last_became_active":"2023-11-16T16:47:17.631356+0000","last_became_peered":"2023-11-16T16:47:17.631356+0000","last_unstale":"2023-11-16T16:48:44.623263+0000","last_undegraded":"2023-11-16T16:48:44.623263+0000","last_fullsized":"2023-11-16T16:48:44.623263+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:05:33.004512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.589316+0000","last_change":"2023-11-16T16:47:18.186706+0000","last_active":"2023-11-16T16:49:12.589316+0000","last_peered":"2023-11-16T16:49:12.589316+0000","last_clean":"2023-11-16T16:49:12.589316+0000","last_became_active":"2023-11-16T16:47:18.186360+0000","last_became_peered":"2023-11-16T16:47:18.186360+0000","last_unstale":"2023-11-16T16:49:12.589316+0000","last_undegraded":"2023-11-16T16:49:12.589316+0000","last_fullsized":"2023-11-16T16:49:12.589316+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:46:42.071656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623436+0000","last_change":"2023-11-16T16:47:22.719065+0000","last_active":"2023-11-16T16:48:44.623436+0000","last_peered":"2023-11-16T16:48:44.623436+0000","last_clean":"2023-11-16T16:48:44.623436+0000","last_became_active":"2023-11-16T16:47:22.718662+0000","last_became_peered":"2023-11-16T16:47:22.718662+0000","last_unstale":"2023-11-16T16:48:44.623436+0000","last_undegraded":"2023-11-16T16:48:44.623436+0000","last_fullsized":"2023-11-16T16:48:44.623436+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:39:06.356934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"82'2","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628612+0000","last_change":"2023-11-16T16:47:43.637790+0000","last_active":"2023-11-16T16:48:44.628612+0000","last_peered":"2023-11-16T16:48:44.628612+0000","last_clean":"2023-11-16T16:48:44.628612+0000","last_became_active":"2023-11-16T16:47:43.632225+0000","last_became_peered":"2023-11-16T16:47:43.632225+0000","last_unstale":"2023-11-16T16:48:44.628612+0000","last_undegraded":"2023-11-16T16:48:44.628612+0000","last_fullsized":"2023-11-16T16:48:44.628612+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:37:02.441726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.584025+0000","last_change":"2023-11-16T16:47:18.180999+0000","last_active":"2023-11-16T16:49:12.584025+0000","last_peered":"2023-11-16T16:49:12.584025+0000","last_clean":"2023-11-16T16:49:12.584025+0000","last_became_active":"2023-11-16T16:47:18.180724+0000","last_became_peered":"2023-11-16T16:47:18.180724+0000","last_unstale":"2023-11-16T16:49:12.584025+0000","last_undegraded":"2023-11-16T16:49:12.584025+0000","last_fullsized":"2023-11-16T16:49:12.584025+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:21:21.954717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.647762+0000","last_change":"2023-11-16T16:47:22.665607+0000","last_active":"2023-11-16T16:48:44.647762+0000","last_peered":"2023-11-16T16:48:44.647762+0000","last_clean":"2023-11-16T16:48:44.647762+0000","last_became_active":"2023-11-16T16:47:22.665273+0000","last_became_peered":"2023-11-16T16:47:22.665273+0000","last_unstale":"2023-11-16T16:48:44.647762+0000","last_undegraded":"2023-11-16T16:48:44.647762+0000","last_fullsized":"2023-11-16T16:48:44.647762+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:31:43.368003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623867+0000","last_change":"2023-11-16T16:47:17.632614+0000","last_active":"2023-11-16T16:48:44.623867+0000","last_peered":"2023-11-16T16:48:44.623867+0000","last_clean":"2023-11-16T16:48:44.623867+0000","last_became_active":"2023-11-16T16:47:17.632227+0000","last_became_peered":"2023-11-16T16:47:17.632227+0000","last_unstale":"2023-11-16T16:48:44.623867+0000","last_undegraded":"2023-11-16T16:48:44.623867+0000","last_fullsized":"2023-11-16T16:48:44.623867+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:52:45.369452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579477+0000","last_change":"2023-11-16T16:47:18.177607+0000","last_active":"2023-11-16T16:49:12.579477+0000","last_peered":"2023-11-16T16:49:12.579477+0000","last_clean":"2023-11-16T16:49:12.579477+0000","last_became_active":"2023-11-16T16:47:18.177266+0000","last_became_peered":"2023-11-16T16:47:18.177266+0000","last_unstale":"2023-11-16T16:49:12.579477+0000","last_undegraded":"2023-11-16T16:49:12.579477+0000","last_fullsized":"2023-11-16T16:49:12.579477+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:45:38.139269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910928+0000","last_change":"2023-11-16T16:47:22.659987+0000","last_active":"2023-11-16T16:48:44.910928+0000","last_peered":"2023-11-16T16:48:44.910928+0000","last_clean":"2023-11-16T16:48:44.910928+0000","last_became_active":"2023-11-16T16:47:22.658725+0000","last_became_peered":"2023-11-16T16:47:22.658725+0000","last_unstale":"2023-11-16T16:48:44.910928+0000","last_undegraded":"2023-11-16T16:48:44.910928+0000","last_fullsized":"2023-11-16T16:48:44.910928+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:09:57.462778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911469+0000","last_change":"2023-11-16T16:47:17.640696+0000","last_active":"2023-11-16T16:48:44.911469+0000","last_peered":"2023-11-16T16:48:44.911469+0000","last_clean":"2023-11-16T16:48:44.911469+0000","last_became_active":"2023-11-16T16:47:17.640428+0000","last_became_peered":"2023-11-16T16:47:17.640428+0000","last_unstale":"2023-11-16T16:48:44.911469+0000","last_undegraded":"2023-11-16T16:48:44.911469+0000","last_fullsized":"2023-11-16T16:48:44.911469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:06:54.539557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.583537+0000","last_change":"2023-11-16T16:47:43.972273+0000","last_active":"2023-11-16T16:49:12.583537+0000","last_peered":"2023-11-16T16:49:12.583537+0000","last_clean":"2023-11-16T16:49:12.583537+0000","last_became_active":"2023-11-16T16:47:43.971981+0000","last_became_peered":"2023-11-16T16:47:43.971981+0000","last_unstale":"2023-11-16T16:49:12.583537+0000","last_undegraded":"2023-11-16T16:49:12.583537+0000","last_fullsized":"2023-11-16T16:49:12.583537+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:07:07.722793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910594+0000","last_change":"2023-11-16T16:47:22.649436+0000","last_active":"2023-11-16T16:48:44.910594+0000","last_peered":"2023-11-16T16:48:44.910594+0000","last_clean":"2023-11-16T16:48:44.910594+0000","last_became_active":"2023-11-16T16:47:22.649034+0000","last_became_peered":"2023-11-16T16:47:22.649034+0000","last_unstale":"2023-11-16T16:48:44.910594+0000","last_undegraded":"2023-11-16T16:48:44.910594+0000","last_fullsized":"2023-11-16T16:48:44.910594+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:46:22.957484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.912764+0000","last_change":"2023-11-16T16:47:17.634582+0000","last_active":"2023-11-16T16:48:44.912764+0000","last_peered":"2023-11-16T16:48:44.912764+0000","last_clean":"2023-11-16T16:48:44.912764+0000","last_became_active":"2023-11-16T16:47:17.634172+0000","last_became_peered":"2023-11-16T16:47:17.634172+0000","last_unstale":"2023-11-16T16:48:44.912764+0000","last_undegraded":"2023-11-16T16:48:44.912764+0000","last_fullsized":"2023-11-16T16:48:44.912764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:43:12.528888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.569892+0000","last_change":"2023-11-16T16:47:18.180466+0000","last_active":"2023-11-16T16:49:12.569892+0000","last_peered":"2023-11-16T16:49:12.569892+0000","last_clean":"2023-11-16T16:49:12.569892+0000","last_became_active":"2023-11-16T16:47:18.180166+0000","last_became_peered":"2023-11-16T16:47:18.180166+0000","last_unstale":"2023-11-16T16:49:12.569892+0000","last_undegraded":"2023-11-16T16:49:12.569892+0000","last_fullsized":"2023-11-16T16:49:12.569892+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:57:25.704442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.570479+0000","last_change":"2023-11-16T16:47:18.179196+0000","last_active":"2023-11-16T16:49:12.570479+0000","last_peered":"2023-11-16T16:49:12.570479+0000","last_clean":"2023-11-16T16:49:12.570479+0000","last_became_active":"2023-11-16T16:47:18.178827+0000","last_became_peered":"2023-11-16T16:47:18.178827+0000","last_unstale":"2023-11-16T16:49:12.570479+0000","last_undegraded":"2023-11-16T16:49:12.570479+0000","last_fullsized":"2023-11-16T16:49:12.570479+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:21:55.520738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.578735+0000","last_change":"2023-11-16T16:47:18.184630+0000","last_active":"2023-11-16T16:49:12.578735+0000","last_peered":"2023-11-16T16:49:12.578735+0000","last_clean":"2023-11-16T16:49:12.578735+0000","last_became_active":"2023-11-16T16:47:18.184314+0000","last_became_peered":"2023-11-16T16:47:18.184314+0000","last_unstale":"2023-11-16T16:49:12.578735+0000","last_undegraded":"2023-11-16T16:49:12.578735+0000","last_fullsized":"2023-11-16T16:49:12.578735+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:20:06.066404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.581521+0000","last_change":"2023-11-16T16:47:18.184255+0000","last_active":"2023-11-16T16:49:12.581521+0000","last_peered":"2023-11-16T16:49:12.581521+0000","last_clean":"2023-11-16T16:49:12.581521+0000","last_became_active":"2023-11-16T16:47:18.183936+0000","last_became_peered":"2023-11-16T16:47:18.183936+0000","last_unstale":"2023-11-16T16:49:12.581521+0000","last_undegraded":"2023-11-16T16:49:12.581521+0000","last_fullsized":"2023-11-16T16:49:12.581521+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:40:04.120978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.588698+0000","last_change":"2023-11-16T16:47:18.196331+0000","last_active":"2023-11-16T16:49:12.588698+0000","last_peered":"2023-11-16T16:49:12.588698+0000","last_clean":"2023-11-16T16:49:12.588698+0000","last_became_active":"2023-11-16T16:47:18.195954+0000","last_became_peered":"2023-11-16T16:47:18.195954+0000","last_unstale":"2023-11-16T16:49:12.588698+0000","last_undegraded":"2023-11-16T16:49:12.588698+0000","last_fullsized":"2023-11-16T16:49:12.588698+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:48:28.247082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.591835+0000","last_change":"2023-11-16T16:47:18.190537+0000","last_active":"2023-11-16T16:49:12.591835+0000","last_peered":"2023-11-16T16:49:12.591835+0000","last_clean":"2023-11-16T16:49:12.591835+0000","last_became_active":"2023-11-16T16:47:18.190118+0000","last_became_peered":"2023-11-16T16:47:18.190118+0000","last_unstale":"2023-11-16T16:49:12.591835+0000","last_undegraded":"2023-11-16T16:49:12.591835+0000","last_fullsized":"2023-11-16T16:49:12.591835+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:53:21.438815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.566299+0000","last_change":"2023-11-16T16:47:18.179215+0000","last_active":"2023-11-16T16:49:12.566299+0000","last_peered":"2023-11-16T16:49:12.566299+0000","last_clean":"2023-11-16T16:49:12.566299+0000","last_became_active":"2023-11-16T16:47:18.178759+0000","last_became_peered":"2023-11-16T16:47:18.178759+0000","last_unstale":"2023-11-16T16:49:12.566299+0000","last_undegraded":"2023-11-16T16:49:12.566299+0000","last_fullsized":"2023-11-16T16:49:12.566299+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:56:47.424865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.581180+0000","last_change":"2023-11-16T16:47:18.179588+0000","last_active":"2023-11-16T16:49:12.581180+0000","last_peered":"2023-11-16T16:49:12.581180+0000","last_clean":"2023-11-16T16:49:12.581180+0000","last_became_active":"2023-11-16T16:47:18.179294+0000","last_became_peered":"2023-11-16T16:47:18.179294+0000","last_unstale":"2023-11-16T16:49:12.581180+0000","last_undegraded":"2023-11-16T16:49:12.581180+0000","last_fullsized":"2023-11-16T16:49:12.581180+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:21:17.462988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.572426+0000","last_change":"2023-11-16T16:47:18.181587+0000","last_active":"2023-11-16T16:49:12.572426+0000","last_peered":"2023-11-16T16:49:12.572426+0000","last_clean":"2023-11-16T16:49:12.572426+0000","last_became_active":"2023-11-16T16:47:18.181203+0000","last_became_peered":"2023-11-16T16:47:18.181203+0000","last_unstale":"2023-11-16T16:49:12.572426+0000","last_undegraded":"2023-11-16T16:49:12.572426+0000","last_fullsized":"2023-11-16T16:49:12.572426+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:41:14.468221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579568+0000","last_change":"2023-11-16T16:47:18.182690+0000","last_active":"2023-11-16T16:49:12.579568+0000","last_peered":"2023-11-16T16:49:12.579568+0000","last_clean":"2023-11-16T16:49:12.579568+0000","last_became_active":"2023-11-16T16:47:18.182361+0000","last_became_peered":"2023-11-16T16:47:18.182361+0000","last_unstale":"2023-11-16T16:49:12.579568+0000","last_undegraded":"2023-11-16T16:49:12.579568+0000","last_fullsized":"2023-11-16T16:49:12.579568+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:24:27.460239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.578706+0000","last_change":"2023-11-16T16:47:18.184599+0000","last_active":"2023-11-16T16:49:12.578706+0000","last_peered":"2023-11-16T16:49:12.578706+0000","last_clean":"2023-11-16T16:49:12.578706+0000","last_became_active":"2023-11-16T16:47:18.184288+0000","last_became_peered":"2023-11-16T16:47:18.184288+0000","last_unstale":"2023-11-16T16:49:12.578706+0000","last_undegraded":"2023-11-16T16:49:12.578706+0000","last_fullsized":"2023-11-16T16:49:12.578706+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:09:35.265387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.575511+0000","last_change":"2023-11-16T16:47:18.187267+0000","last_active":"2023-11-16T16:49:12.575511+0000","last_peered":"2023-11-16T16:49:12.575511+0000","last_clean":"2023-11-16T16:49:12.575511+0000","last_became_active":"2023-11-16T16:47:18.186984+0000","last_became_peered":"2023-11-16T16:47:18.186984+0000","last_unstale":"2023-11-16T16:49:12.575511+0000","last_undegraded":"2023-11-16T16:49:12.575511+0000","last_fullsized":"2023-11-16T16:49:12.575511+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:22:44.826690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.592202+0000","last_change":"2023-11-16T16:47:18.191566+0000","last_active":"2023-11-16T16:49:12.592202+0000","last_peered":"2023-11-16T16:49:12.592202+0000","last_clean":"2023-11-16T16:49:12.592202+0000","last_became_active":"2023-11-16T16:47:18.185342+0000","last_became_peered":"2023-11-16T16:47:18.185342+0000","last_unstale":"2023-11-16T16:49:12.592202+0000","last_undegraded":"2023-11-16T16:49:12.592202+0000","last_fullsized":"2023-11-16T16:49:12.592202+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:53:50.072132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.565174+0000","last_change":"2023-11-16T16:47:43.629636+0000","last_active":"2023-11-16T16:49:12.565174+0000","last_peered":"2023-11-16T16:49:12.565174+0000","last_clean":"2023-11-16T16:49:12.565174+0000","last_became_active":"2023-11-16T16:47:43.629179+0000","last_became_peered":"2023-11-16T16:47:43.629179+0000","last_unstale":"2023-11-16T16:49:12.565174+0000","last_undegraded":"2023-11-16T16:49:12.565174+0000","last_fullsized":"2023-11-16T16:49:12.565174+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:20:45.082147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281399,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67650,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":239,"num_read_kb":190,"num_write":84,"num_write_kb":99,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":197571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449130,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6228,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738924,"statfs":{"total":95995035648,"available":95988658176,"internally_reserved":0,"allocated":995328,"data_stored":506898,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.41999999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.77400000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.57499999999999996,"15min":0.55700000000000005},"min":{"1min":0.46200000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79900000000000004,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.60799999999999998}]},{"osd":1,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59199999999999997,"15min":0.58699999999999997},"min":{"1min":0.505,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74099999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.50900000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.751,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.72799999999999998}]},{"osd":2,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.47499999999999998,"15min":0.46100000000000002},"min":{"1min":0.36799999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.47899999999999998,"15min":0.46999999999999997},"min":{"1min":0.39100000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.57299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.63500000000000001,"15min":0.627},"min":{"1min":0.52400000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.625,"15min":0.626},"min":{"1min":0.46100000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.41399999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.79500000000000004,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.38600000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.89200000000000002,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.46899999999999997}]},{"osd":6,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.67800000000000005,"15min":0.68999999999999995},"min":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.84299999999999997,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.623},{"interface":"front","average":{"1min":0.754,"5min":0.76500000000000001,"15min":0.73299999999999998},"min":{"1min":0.54300000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.93000000000000005,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":0.68400000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.56399999999999995,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95599999999999996,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.73299999999999998,"15min":0.73599999999999999},"min":{"1min":0.56499999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.95699999999999996,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.69499999999999995}]},{"osd":8,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.72299999999999998,"15min":0.71699999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.93000000000000005,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.70499999999999996}]},{"osd":9,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.78500000000000003,"15min":0.76900000000000002},"min":{"1min":0.69999999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.042,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.98599999999999999,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.752}]},{"osd":10,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.76700000000000002,"15min":0.76900000000000002},"min":{"1min":0.59099999999999997,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.94999999999999996,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.78200000000000003,"15min":0.77000000000000002},"min":{"1min":0.69799999999999995,"5min":0.498,"15min":0.498},"max":{"1min":1.006,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.80200000000000005}]}]},{"osd":10,"up_from":71,"seq":304942678062,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6676,"kb_used_data":1100,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738476,"statfs":{"total":95995035648,"available":95988199424,"internally_reserved":0,"allocated":1126400,"data_stored":1160194,"data_compressed":59395,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:49:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.72899999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72799999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.26900000000000002}]},{"osd":1,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.26200000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60299999999999998,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.49099999999999999,"15min":0.51100000000000001},"min":{"1min":0.28299999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70799999999999996,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.247}]},{"osd":2,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.63400000000000001,"15min":0.66100000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.70499999999999996,"5min":1.577,"15min":1.577},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.63800000000000001,"15min":0.65800000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78300000000000003,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.29799999999999999}]},{"osd":3,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.623,"15min":0.64500000000000002},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80000000000000004,"5min":1.464,"15min":1.464},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.65400000000000003,"15min":0.67500000000000004},"min":{"1min":0.40400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76800000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.374}]},{"osd":4,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56399999999999995,"15min":0.58399999999999996},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73899999999999999,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54700000000000004,"15min":0.56799999999999995},"min":{"1min":0.372,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.752,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.30399999999999999}]},{"osd":5,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.68300000000000005,"15min":0.69899999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82899999999999996,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.68700000000000006,"15min":0.69899999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.91300000000000003,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.37}]},{"osd":6,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.73399999999999999,"15min":0.75900000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86299999999999999,"5min":1.524,"15min":1.524},"last":0.432},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76000000000000001,"15min":0.77400000000000002},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.85999999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.5}]},{"osd":7,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.69999999999999996}]},{"osd":8,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.81299999999999994,"15min":0.85499999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.91400000000000003,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.755,"15min":0.78300000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.93999999999999995,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.438}]},{"osd":9,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.84399999999999997,"15min":0.84099999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.92700000000000005,"5min":4.0129999999999999,"15min":4.0129999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.82199999999999995,"15min":0.83099999999999996},"min":{"1min":0.56299999999999994,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":4.5380000000000003,"5min":4.5380000000000003,"15min":4.5380000000000003},"last":0.46300000000000002}]},{"osd":11,"last update":"Thu Nov 16 16:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.78400000000000003,"15min":0.81799999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.96599999999999997,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.79100000000000004,"15min":0.80700000000000005},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.95599999999999996,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.48099999999999998}]}]},{"osd":9,"up_from":66,"seq":283467841585,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6424,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738728,"statfs":{"total":95995035648,"available":95988457472,"internally_reserved":0,"allocated":999424,"data_stored":511554,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.54400000000000004,"15min":0.56999999999999995},"min":{"1min":0.28399999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.57199999999999995,"15min":0.60399999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.40300000000000002}]},{"osd":1,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.78600000000000003,"15min":0.82499999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.89500000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.747,"15min":0.78200000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.53700000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.80100000000000005,"15min":0.83299999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0640000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.753,"15min":0.77700000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59099999999999997,"15min":0.61499999999999999},"min":{"1min":0.40899999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.746,"15min":0.80000000000000004},"min":{"1min":0.42299999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.2210000000000001,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.61099999999999999}]},{"osd":4,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.79600000000000004,"15min":0.85099999999999998},"min":{"1min":0.48799999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.167,"5min":1.298,"15min":1.298},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.73699999999999999,"15min":0.77300000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.94199999999999995,"5min":1.181,"15min":1.181},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.78800000000000003,"15min":0.82399999999999995},"min":{"1min":0.51200000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.081,"5min":1.268,"15min":1.268},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.90000000000000002,"15min":0.95299999999999996},"min":{"1min":0.621,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1040000000000001,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.76700000000000002}]},{"osd":6,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":1.054},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.95599999999999996}]},{"osd":7,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.84899999999999998,"15min":0.90300000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.99199999999999999,"5min":1.405,"15min":1.405},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.98099999999999998,"15min":1.0589999999999999},"min":{"1min":0.59799999999999998,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.133,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.748}]},{"osd":8,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.90200000000000002,"15min":0.96099999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.1970000000000001,"5min":1.363,"15min":1.363},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.90400000000000003,"15min":0.96399999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.149,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.68700000000000006}]},{"osd":10,"last update":"Thu Nov 16 16:49:15 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.88800000000000001,"15min":0.93899999999999995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.036,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.96499999999999997,"15min":1.0209999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1599999999999999,"5min":1.714,"15min":1.714},"last":0.79600000000000004}]},{"osd":11,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.80300000000000005,"15min":0.80800000000000005},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.038,"5min":1.115,"15min":1.115},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.89100000000000001,"15min":0.90600000000000003},"min":{"1min":0.67800000000000005,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.0600000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.82099999999999995}]}]},{"osd":8,"up_from":58,"seq":249108103223,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6548,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738604,"statfs":{"total":95995035648,"available":95988330496,"internally_reserved":0,"allocated":995328,"data_stored":507458,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.34300000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.79600000000000004,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.39200000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.86299999999999999,"5min":1.113,"15min":1.113},"last":0.436}]},{"osd":1,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63200000000000001,"15min":0.60399999999999998},"min":{"1min":0.51600000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76600000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.623,"15min":0.60999999999999999},"min":{"1min":0.40999999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.85099999999999998,"5min":1.177,"15min":1.177},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47299999999999998,"15min":0.45700000000000002},"min":{"1min":0.29699999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.73499999999999999,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47499999999999998,"15min":0.46600000000000003},"min":{"1min":0.38700000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.40000000000000002}]},{"osd":3,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.629,"15min":0.63},"min":{"1min":0.42799999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.90600000000000003,"5min":1.141,"15min":1.141},"last":0.496},{"interface":"front","average":{"1min":0.623,"5min":0.65500000000000003,"15min":0.64700000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.91800000000000004,"5min":1.502,"15min":1.502},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.64100000000000001,"15min":0.629},"min":{"1min":0.50700000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79800000000000004,"5min":0.999,"15min":0.999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69799999999999995,"15min":0.68799999999999994},"min":{"1min":0.54300000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83199999999999996,"5min":1.242,"15min":1.242},"last":0.48099999999999998}]},{"osd":5,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55800000000000005,"15min":0.53600000000000003},"min":{"1min":0.45900000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83699999999999997,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57599999999999996,"15min":0.56299999999999994},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68000000000000005,"5min":1.224,"15min":1.224},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.77600000000000002,"15min":0.73799999999999999},"min":{"1min":0.54900000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.89000000000000001,"5min":4.7309999999999999,"15min":4.7309999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.69599999999999995,"15min":0.68100000000000005},"min":{"1min":0.54200000000000004,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0149999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.56200000000000006}]},{"osd":7,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.71599999999999997,"15min":0.69499999999999995},"min":{"1min":0.60499999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90300000000000002,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.73899999999999999,"15min":0.72599999999999998},"min":{"1min":0.59699999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95599999999999996,"5min":1.198,"15min":1.198},"last":0.622}]},{"osd":9,"last update":"Thu Nov 16 16:48:51 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.76900000000000002,"15min":0.77000000000000002},"min":{"1min":0.52700000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.89100000000000001,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.80300000000000005,"15min":0.81100000000000005},"min":{"1min":0.60299999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.88500000000000001,"5min":1.298,"15min":1.298},"last":0.65400000000000003}]},{"osd":10,"last update":"Thu Nov 16 16:49:11 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78500000000000003,"15min":0.77700000000000002},"min":{"1min":0.70199999999999996,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.97699999999999998,"5min":1.222,"15min":1.222},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.81599999999999995,"15min":0.81999999999999995},"min":{"1min":0.66100000000000003,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.91900000000000004,"5min":1.274,"15min":1.274},"last":0.52200000000000002}]},{"osd":11,"last update":"Thu Nov 16 16:49:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.77300000000000002,"15min":0.77500000000000002},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.94499999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.77800000000000002,"15min":0.78400000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.875,"5min":1.05,"15min":1.05},"last":0.64400000000000002}]}]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6664,"kb_used_data":1024,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738488,"statfs":{"total":95995035648,"available":95988211712,"internally_reserved":0,"allocated":1048576,"data_stored":1101922,"data_compressed":59395,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.377,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81799999999999995,"5min":0.875,"15min":0.875},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.42899999999999999,"15min":0.41099999999999998},"min":{"1min":0.28499999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59499999999999997,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.45100000000000001,"15min":0.441},"min":{"1min":0.29899999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.628,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.60299999999999998}]},{"osd":2,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.58099999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.76600000000000001,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.61199999999999999,"15min":0.58599999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81799999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.629}]},{"osd":3,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.58799999999999997,"15min":0.55900000000000005},"min":{"1min":0.46999999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.44,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.76600000000000001}]},{"osd":4,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.375,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67900000000000005,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.51900000000000002,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.69799999999999995,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.48799999999999999}]},{"osd":5,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64900000000000002,"15min":0.65100000000000002},"min":{"1min":0.45500000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81899999999999995,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.621},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.66500000000000004,"15min":0.63900000000000001},"min":{"1min":0.45700000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.88100000000000001,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.73099999999999998}]},{"osd":6,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.69599999999999995,"15min":0.68000000000000005},"min":{"1min":0.50700000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46999999999999997,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.93500000000000005,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.64800000000000002}]},{"osd":8,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.72099999999999997,"15min":0.68500000000000005},"min":{"1min":0.64200000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.92900000000000005,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.70699999999999996,"15min":0.70399999999999996},"min":{"1min":0.58499999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.95299999999999996,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.67600000000000005}]},{"osd":9,"last update":"Thu Nov 16 16:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.75600000000000001,"15min":0.77900000000000003},"min":{"1min":0.53800000000000003,"5min":0.442,"15min":0.442},"max":{"1min":0.88900000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78400000000000003,"15min":0.80300000000000005},"min":{"1min":0.61599999999999999,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.94999999999999996,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.88}]},{"osd":10,"last update":"Thu Nov 16 16:49:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.72699999999999998,"15min":0.75},"min":{"1min":0.48799999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94199999999999995,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.76800000000000002,"15min":0.78500000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93500000000000005,"5min":1.01,"15min":1.01},"last":0.83399999999999996}]},{"osd":11,"last update":"Thu Nov 16 16:49:23 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81200000000000006,"15min":0.81000000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.76400000000000001,"15min":0.76100000000000001},"min":{"1min":0.56799999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.71999999999999997}]}]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6540,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738612,"statfs":{"total":95995035648,"available":95988338688,"internally_reserved":0,"allocated":987136,"data_stored":502780,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.752},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.375,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.96499999999999997}]},{"osd":1,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.66900000000000004,"15min":0.65500000000000003},"min":{"1min":0.42899999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.68999999999999995,"15min":0.70199999999999996},"min":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0569999999999999,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72699999999999998,"15min":0.71099999999999997},"min":{"1min":0.50800000000000001,"5min":0.436,"15min":0.436},"max":{"1min":0.92800000000000005,"5min":1.407,"15min":1.407},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.72399999999999998},"min":{"1min":0.505,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0189999999999999,"5min":1.171,"15min":1.171},"last":0.61499999999999999}]},{"osd":3,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.71799999999999997,"15min":0.73599999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.755,"15min":0.73499999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.1020000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.74399999999999999,"15min":0.75600000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.77800000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.79500000000000004,"15min":0.80100000000000005},"min":{"1min":0.52300000000000002,"5min":0.503,"15min":0.503},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.873,"15min":0.89300000000000002},"min":{"1min":0.57899999999999996,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.083,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.90800000000000003}]},{"osd":7,"last update":"Thu Nov 16 16:49:02 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.82199999999999995,"15min":0.79900000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.1080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.78500000000000003,"15min":0.755},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0780000000000001,"5min":1.256,"15min":1.256},"last":0.80100000000000005}]},{"osd":8,"last update":"Thu Nov 16 16:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.83099999999999996,"15min":0.84199999999999997},"min":{"1min":0.55200000000000005,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.998,"5min":1.284,"15min":1.284},"last":0.629},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.89000000000000001,"15min":0.88800000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0580000000000001,"5min":1.823,"15min":1.823},"last":0.92700000000000005}]},{"osd":9,"last update":"Thu Nov 16 16:48:51 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.94499999999999995,"15min":0.96099999999999997},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.208,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.83999999999999997,"15min":0.83699999999999997},"min":{"1min":0.623,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.82299999999999995}]},{"osd":10,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.89500000000000002,"15min":0.92200000000000004},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.2370000000000001,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.871,"5min":0.96499999999999997,"15min":0.97799999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.02,"5min":1.393,"15min":1.393},"last":1.0029999999999999}]},{"osd":11,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.022,"15min":1.036},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.91900000000000004,"15min":0.93500000000000005},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":0.98099999999999998,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.874}]}]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7632,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737520,"statfs":{"total":95995035648,"available":95987220480,"internally_reserved":0,"allocated":1056768,"data_stored":564253,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:52 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.52900000000000003},"min":{"1min":0.47899999999999998,"5min":0.32900000000000001,"15min":0.317},"max":{"1min":0.91900000000000004,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.56000000000000005,"15min":0.52100000000000002},"min":{"1min":0.36199999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.82599999999999996,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:49:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.58499999999999996,"15min":0.54500000000000004},"min":{"1min":0.46500000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.83299999999999996,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.57999999999999996,"15min":0.52200000000000002},"min":{"1min":0.48099999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.84099999999999997,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.71599999999999997}]},{"osd":3,"last update":"Thu Nov 16 16:48:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56599999999999995,"15min":0.53900000000000003},"min":{"1min":0.44800000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.93100000000000005,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.621,"15min":0.58299999999999996},"min":{"1min":0.495,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.61399999999999999}]},{"osd":4,"last update":"Thu Nov 16 16:48:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.45700000000000002,"15min":0.42699999999999999},"min":{"1min":0.34799999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.72999999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.46700000000000003,"15min":0.442},"min":{"1min":0.37,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76900000000000002,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.371}]},{"osd":5,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.63700000000000001,"15min":0.58499999999999996},"min":{"1min":0.57399999999999995,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.876,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.65700000000000003,"15min":0.61599999999999999},"min":{"1min":0.54400000000000004,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90100000000000002,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.75900000000000001}]},{"osd":6,"last update":"Thu Nov 16 16:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.66900000000000004,"15min":0.64600000000000002},"min":{"1min":0.51400000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.68000000000000005,"15min":0.65000000000000002},"min":{"1min":0.59499999999999997,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.87,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.65400000000000003}]},{"osd":7,"last update":"Thu Nov 16 16:49:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.57599999999999996,"15min":0.52900000000000003},"min":{"1min":0.38500000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.61199999999999999,"15min":0.56799999999999995},"min":{"1min":0.41899999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.80300000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.74399999999999999}]},{"osd":8,"last update":"Thu Nov 16 16:49:28 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.70999999999999996,"15min":0.65400000000000003},"min":{"1min":0.50700000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.90600000000000003,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.71999999999999997,"15min":0.66600000000000004},"min":{"1min":0.57399999999999995,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.97799999999999998,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.79900000000000004}]},{"osd":9,"last update":"Thu Nov 16 16:48:58 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.749,"15min":0.73199999999999998},"min":{"1min":0.54000000000000004,"5min":0.5,"15min":0.5},"max":{"1min":0.98099999999999998,"5min":1.276,"15min":1.276},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.76300000000000001,"15min":0.751},"min":{"1min":0.63800000000000001,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.92600000000000005,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.70399999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:49:11 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.77700000000000002,"15min":0.78100000000000003},"min":{"1min":0.65700000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.91100000000000003,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.78500000000000003,"15min":0.78300000000000003},"min":{"1min":0.58299999999999996,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.81299999999999994}]},{"osd":11,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85599999999999998,"15min":0.85999999999999999},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":0.96799999999999997,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.88800000000000001,"15min":0.89200000000000002},"min":{"1min":0.73299999999999998,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.0069999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.84099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7440,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737712,"statfs":{"total":95995035648,"available":95987417088,"internally_reserved":0,"allocated":991232,"data_stored":502870,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 16:48:53 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.61299999999999999,"15min":0.55700000000000005},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66000000000000003,"15min":0.57999999999999996},"min":{"1min":0.432,"5min":0.316,"15min":0.316},"max":{"1min":1.7789999999999999,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.61399999999999999}]},{"osd":2,"last update":"Thu Nov 16 16:49:12 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.75900000000000001,"15min":0.621},"min":{"1min":0.57199999999999995,"5min":0.40600000000000003,"15min":0.372},"max":{"1min":2.0529999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.70399999999999996,"15min":0.63},"min":{"1min":0.53000000000000003,"5min":0.41799999999999998,"15min":0.35099999999999998},"max":{"1min":2.1000000000000001,"5min":2.1000000000000001,"15min":2.1000000000000001},"last":0.66500000000000004}]},{"osd":3,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83499999999999996,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.63300000000000001,"15min":0.63500000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.901,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.46000000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:49:04 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.71299999999999997,"15min":0.61099999999999999},"min":{"1min":0.59899999999999998,"5min":0.434,"15min":0.434},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.748,"15min":0.78800000000000003},"min":{"1min":0.53700000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.016,"5min":5.4409999999999998,"15min":5.4409999999999998},"last":0.64900000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.79200000000000004,"15min":0.71999999999999997},"min":{"1min":0.55000000000000004,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.9390000000000001,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.74399999999999999,"15min":0.72399999999999998},"min":{"1min":0.50900000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.97599999999999998,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":0.55900000000000005}]},{"osd":6,"last update":"Thu Nov 16 16:48:48 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.73199999999999998,"15min":0.69599999999999995},"min":{"1min":0.51000000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.76100000000000001,"15min":0.68799999999999994},"min":{"1min":0.622,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":0.72499999999999998}]},{"osd":7,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.86699999999999999,"15min":0.872},"min":{"1min":0.59699999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.054,"5min":1.8089999999999999,"15min":1.8089999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.87,"15min":0.88300000000000001},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2330000000000001,"5min":3.1179999999999999,"15min":3.1179999999999999},"last":0.51000000000000001}]},{"osd":8,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.874,"15min":0.88200000000000001},"min":{"1min":0.66600000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.2829999999999999,"5min":2.891,"15min":2.891},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.90100000000000002,"15min":0.91700000000000004},"min":{"1min":0.63200000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.1499999999999999,"5min":1.8759999999999999,"15min":1.8759999999999999},"last":0.64000000000000001}]},{"osd":9,"last update":"Thu Nov 16 16:48:53 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.97399999999999998,"15min":0.98499999999999999},"min":{"1min":0.67600000000000005,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.69899999999999995,"5min":0.495,"15min":0.495},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.78100000000000003}]},{"osd":10,"last update":"Thu Nov 16 16:49:12 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.70099999999999996,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":3.0859999999999999,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.04,"15min":1.0860000000000001},"min":{"1min":0.68200000000000005,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.9079999999999999,"5min":3.694,"15min":3.694},"last":0.67700000000000005}]},{"osd":11,"last update":"Thu Nov 16 16:49:32 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91600000000000004,"15min":0.88900000000000001},"min":{"1min":0.751,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.1859999999999999,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":0.749},{"interface":"front","average":{"1min":1.0069999999999999,"5min":0.93899999999999995,"15min":0.89800000000000002},"min":{"1min":0.75,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.4370000000000001,"5min":3.0489999999999999,"15min":3.0489999999999999},"last":0.80500000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7640,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737512,"statfs":{"total":95995035648,"available":95987212288,"internally_reserved":0,"allocated":1064960,"data_stored":569393,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56499999999999995,"15min":0.47999999999999998},"min":{"1min":0.44800000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.91400000000000003,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.504},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53500000000000003,"15min":0.499},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.45300000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56399999999999995,"15min":0.502},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.83299999999999996,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.60999999999999999,"15min":0.51800000000000002},"min":{"1min":0.33600000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85799999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.439}]},{"osd":3,"last update":"Thu Nov 16 16:49:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.64900000000000002,"15min":0.71099999999999997},"min":{"1min":0.438,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":3.98},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63400000000000001,"15min":0.59799999999999998},"min":{"1min":0.47499999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95899999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.51400000000000001}]},{"osd":4,"last update":"Thu Nov 16 16:49:19 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.70999999999999996,"15min":0.70599999999999996},"min":{"1min":0.501,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.67100000000000004,"15min":0.67400000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.35999999999999999}]},{"osd":5,"last update":"Thu Nov 16 16:49:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.50700000000000001,"15min":0.47099999999999997},"min":{"1min":0.39200000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76100000000000001,"5min":0.871,"15min":0.871},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.52800000000000002,"15min":0.48899999999999999},"min":{"1min":0.45400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70699999999999996,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.42299999999999999}]},{"osd":6,"last update":"Thu Nov 16 16:48:47 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.629,"15min":0.59499999999999997},"min":{"1min":0.442,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63700000000000001,"15min":0.61499999999999999},"min":{"1min":0.46600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85799999999999998,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.497}]},{"osd":7,"last update":"Thu Nov 16 16:49:24 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.40699999999999997,"5min":0.376,"15min":0.376},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.80000000000000004,"15min":0.79400000000000004},"min":{"1min":0.627,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.627}]},{"osd":8,"last update":"Thu Nov 16 16:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.64500000000000002,"15min":0.61099999999999999},"min":{"1min":0.44800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60099999999999998,"15min":0.58099999999999996},"min":{"1min":0.434,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.45900000000000002}]},{"osd":9,"last update":"Thu Nov 16 16:48:53 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.73299999999999998,"15min":0.72099999999999997},"min":{"1min":0.58799999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.022,"5min":0.79000000000000004,"15min":0.749},"min":{"1min":0.52100000000000002,"5min":0.503,"15min":0.503},"max":{"1min":5.2510000000000003,"5min":5.2510000000000003,"15min":5.2510000000000003},"last":0.57799999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.72199999999999998,"15min":0.69299999999999995},"min":{"1min":0.49299999999999999,"5min":0.441,"15min":0.441},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.71999999999999997,"15min":0.69799999999999995},"min":{"1min":0.56299999999999994,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.52100000000000002}]},{"osd":11,"last update":"Thu Nov 16 16:49:19 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.81399999999999995,"15min":0.80800000000000005},"min":{"1min":0.65200000000000002,"5min":0.499,"15min":0.499},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.745,"15min":0.73799999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.48599999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6804,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738348,"statfs":{"total":95995035648,"available":95988068352,"internally_reserved":0,"allocated":995328,"data_stored":502884,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.48199999999999998,"15min":0.435},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.077,"5min":1.085,"15min":1.085},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.58399999999999996,"15min":0.50600000000000001},"min":{"1min":0.39600000000000002,"5min":0.313,"15min":0.313},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":0.58499999999999996}]},{"osd":1,"last update":"Thu Nov 16 16:48:38 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":0.73699999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.379,"15min":0.379},"max":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":10.266999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.66000000000000003,"15min":0.60399999999999998},"min":{"1min":0.44400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.2050000000000001,"5min":1.286,"15min":1.286},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:48:38 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66500000000000004,"15min":0.59699999999999998},"min":{"1min":0.51200000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.02,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.68700000000000006,"15min":0.61499999999999999},"min":{"1min":0.42299999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.4350000000000001,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Nov 16 16:49:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.71599999999999997,"15min":0.63},"min":{"1min":0.50900000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.25,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.73299999999999998,"15min":0.67900000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.88300000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:49:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.71899999999999997,"15min":0.69299999999999995},"min":{"1min":0.45400000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.879,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.62},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.76800000000000002,"15min":0.68799999999999994},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86399999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":0.56200000000000006}]},{"osd":6,"last update":"Thu Nov 16 16:48:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.71099999999999997,"15min":0.66500000000000004},"min":{"1min":0.47999999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.29,"5min":1.323,"15min":1.323},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70499999999999996,"15min":0.68200000000000005},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.04,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.752}]},{"osd":7,"last update":"Thu Nov 16 16:49:05 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.84299999999999997,"15min":0.80300000000000005},"min":{"1min":0.50700000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.637,"5min":5.1070000000000002,"15min":5.1070000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1,"15min":0.89700000000000002},"min":{"1min":0.60599999999999998,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":10.305999999999999,"5min":10.305999999999999,"15min":10.305999999999999},"last":0.92000000000000004}]},{"osd":8,"last update":"Thu Nov 16 16:46:23 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.94199999999999995}]},{"osd":9,"last update":"Thu Nov 16 16:48:51 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.88400000000000001,"15min":0.91000000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":2.5920000000000001,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.94599999999999995,"15min":0.96799999999999997},"min":{"1min":0.54600000000000004,"5min":0.501,"15min":0.501},"max":{"1min":1.417,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.99299999999999999}]},{"osd":10,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":0.96799999999999997,"15min":0.91200000000000003},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":10.287000000000001,"5min":10.287000000000001,"15min":10.287000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.91800000000000004,"15min":0.93799999999999994},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.012,"5min":2.673,"15min":2.673},"last":0.70799999999999996}]},{"osd":11,"last update":"Thu Nov 16 16:49:24 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.92500000000000004,"15min":0.93500000000000005},"min":{"1min":0.64000000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.1000000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":1.028},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.82999999999999996,"15min":0.82799999999999996},"min":{"1min":0.626,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.78900000000000003}]}]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6676,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738476,"statfs":{"total":95995035648,"available":95988199424,"internally_reserved":0,"allocated":995328,"data_stored":506898,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.48899999999999999,"15min":0.46600000000000003},"min":{"1min":0.439,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.51100000000000001,"15min":0.48599999999999999},"min":{"1min":0.47399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.63500000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.38,"15min":0.33000000000000002},"min":{"1min":0.26500000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57799999999999996,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40000000000000002,"15min":0.35499999999999998},"min":{"1min":0.28100000000000003,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.56299999999999994,"5min":1.212,"15min":1.212},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58499999999999996,"15min":0.52300000000000002},"min":{"1min":0.371,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80800000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.57999999999999996,"15min":0.51600000000000001},"min":{"1min":0.435,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.83799999999999997,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.53100000000000003}]},{"osd":3,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.58899999999999997,"15min":0.53500000000000003},"min":{"1min":0.40899999999999997,"5min":0.307,"15min":0.307},"max":{"1min":0.85699999999999998,"5min":1.349,"15min":1.349},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.745,"15min":0.58899999999999997},"min":{"1min":0.39500000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.84599999999999997,"5min":14.795999999999999,"15min":14.795999999999999},"last":0.40000000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:49:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61299999999999999,"15min":0.55300000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.84899999999999998,"5min":1.462,"15min":1.462},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60499999999999998,"15min":0.53900000000000003},"min":{"1min":0.442,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.496}]},{"osd":6,"last update":"Thu Nov 16 16:48:52 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.624,"15min":0.58599999999999997},"min":{"1min":0.44500000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.85899999999999999,"5min":1.323,"15min":1.323},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.66200000000000003,"15min":0.62},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89400000000000002,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":0.54800000000000004}]},{"osd":7,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56200000000000006,"15min":0.52800000000000002},"min":{"1min":0.38900000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88500000000000001,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":0.443},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.60599999999999998,"15min":0.57399999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81999999999999995,"5min":1.764,"15min":1.764},"last":0.46100000000000002}]},{"osd":8,"last update":"Thu Nov 16 16:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.71999999999999997,"15min":0.71299999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88500000000000001,"5min":2.7650000000000001,"15min":2.7650000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73299999999999998,"15min":0.74399999999999999},"min":{"1min":0.51200000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.872,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.51000000000000001}]},{"osd":9,"last update":"Thu Nov 16 16:48:52 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.77700000000000002,"15min":0.79900000000000004},"min":{"1min":0.48599999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.90900000000000003,"5min":1.246,"15min":1.246},"last":0.621},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.76500000000000001,"15min":0.77700000000000002},"min":{"1min":0.51200000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.88900000000000001,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.57699999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.69099999999999995,"15min":0.69899999999999995},"min":{"1min":0.48699999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.89000000000000001,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.70199999999999996,"15min":0.71599999999999997},"min":{"1min":0.497,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90200000000000002,"5min":1.532,"15min":1.532},"last":0.56999999999999995}]},{"osd":11,"last update":"Thu Nov 16 16:49:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.79800000000000004,"15min":0.80500000000000005},"min":{"1min":0.54500000000000004,"5min":0.501,"15min":0.501},"max":{"1min":0.93400000000000005,"5min":1.502,"15min":1.502},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.82099999999999995,"15min":0.82699999999999996},"min":{"1min":0.57399999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.96499999999999997,"5min":1.425,"15min":1.425},"last":0.59299999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6860,"kb_used_data":1028,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738292,"statfs":{"total":95995035648,"available":95988011008,"internally_reserved":0,"allocated":1052672,"data_stored":1097426,"data_compressed":59395,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:49:20 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.51100000000000001,"15min":0.51200000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.60199999999999998,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54500000000000004,"15min":0.56000000000000005},"min":{"1min":0.35799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63200000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Nov 16 16:49:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.58999999999999997,"15min":0.55400000000000005},"min":{"1min":0.435,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.82299999999999995,"5min":1.167,"15min":1.167},"last":0.377},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.58399999999999996,"15min":0.52600000000000002},"min":{"1min":0.36299999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.75600000000000001,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.30199999999999999}]},{"osd":2,"last update":"Thu Nov 16 16:49:20 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40400000000000003,"15min":0.377},"min":{"1min":0.33900000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57699999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39700000000000002},"min":{"1min":0.33000000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.68200000000000005,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.33500000000000002}]},{"osd":3,"last update":"Thu Nov 16 16:49:20 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55300000000000005,"15min":0.53500000000000003},"min":{"1min":0.41199999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.65800000000000003,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.497},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59999999999999998,"15min":0.57199999999999995},"min":{"1min":0.46100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.71499999999999997,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.46300000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:49:20 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.59699999999999998,"15min":0.56100000000000005},"min":{"1min":0.54600000000000004,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85799999999999998,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57899999999999996,"15min":0.53500000000000003},"min":{"1min":0.44700000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.84299999999999997,"5min":0.879,"15min":0.879},"last":0.314}]},{"osd":6,"last update":"Thu Nov 16 16:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.438,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86699999999999999,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.64500000000000002,"15min":0.63300000000000001},"min":{"1min":0.53000000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.92800000000000005,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.60899999999999999}]},{"osd":7,"last update":"Thu Nov 16 16:49:08 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.63300000000000001,"15min":0.61099999999999999},"min":{"1min":0.49199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.86699999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.436},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.65500000000000003,"15min":0.63700000000000001},"min":{"1min":0.58399999999999996,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.95499999999999996,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.46899999999999997}]},{"osd":8,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56100000000000005,"15min":0.54400000000000004},"min":{"1min":0.436,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80400000000000005,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.62,"15min":0.59599999999999997},"min":{"1min":0.51800000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.90300000000000002,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.48099999999999998}]},{"osd":9,"last update":"Thu Nov 16 16:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.747,"15min":0.76100000000000001},"min":{"1min":0.61599999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.91700000000000004,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.745,"15min":0.75700000000000001},"min":{"1min":0.60399999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.90700000000000003,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.64200000000000002}]},{"osd":10,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.71299999999999997},"min":{"1min":0.55600000000000005,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.89500000000000002,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.76900000000000002,"15min":0.75700000000000001},"min":{"1min":0.68600000000000005,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.995,"5min":2.492,"15min":2.492},"last":0.57599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":66886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":67046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T16:49:41.452 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T16:49:41.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.690+0000 7fb4761ff700 1 -- 172.21.15.44:0/2238790508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb470101e10 msgr2=0x7fb4701086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:41.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.690+0000 7fb4761ff700 1 --2- 172.21.15.44:0/2238790508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb470101e10 0x7fb4701086d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb46800b0a0 tx=0x7fb46800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.690+0000 7fb4761ff700 1 -- 172.21.15.44:0/2238790508 shutdown_connections 2023-11-16T16:49:41.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.690+0000 7fb4761ff700 1 --2- 172.21.15.44:0/2238790508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb470101e10 0x7fb4701086d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.690+0000 7fb4761ff700 1 --2- 172.21.15.44:0/2238790508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb470100290 0x7fb4701006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.690+0000 7fb4761ff700 1 --2- 172.21.15.44:0/2238790508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb470101480 0x7fb4701018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.690+0000 7fb4761ff700 1 -- 172.21.15.44:0/2238790508 >> 172.21.15.44:0/2238790508 conn(0x7fb4700fb870 msgr2=0x7fb4700fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:41.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.690+0000 7fb4761ff700 1 -- 172.21.15.44:0/2238790508 shutdown_connections 2023-11-16T16:49:41.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.690+0000 7fb4761ff700 1 -- 172.21.15.44:0/2238790508 wait complete. 2023-11-16T16:49:41.694 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.691+0000 7fb4761ff700 1 Processor -- start 2023-11-16T16:49:41.695 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.691+0000 7fb4761ff700 1 -- start start 2023-11-16T16:49:41.695 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.691+0000 7fb4761ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb470100290 0x7fb4701982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:41.695 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.691+0000 7fb4761ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb470101480 0x7fb4701987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:41.695 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.692+0000 7fb4761ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb470101e10 0x7fb47019f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:41.695 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.692+0000 7fb4761ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47019fe20 con 0x7fb470101e10 2023-11-16T16:49:41.695 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.692+0000 7fb4761ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47019ff60 con 0x7fb470101480 2023-11-16T16:49:41.695 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.692+0000 7fb4761ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4701a00f0 con 0x7fb470100290 2023-11-16T16:49:41.695 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.692+0000 7fb46ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb470101e10 0x7fb47019f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:41.695 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.692+0000 7fb46ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb470101e10 0x7fb47019f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32842/0 (socket says 172.21.15.44:32842) 2023-11-16T16:49:41.695 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.692+0000 7fb46ffff700 1 -- 172.21.15.44:0/3368261311 learned_addr learned my addr 172.21.15.44:0/3368261311 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:41.695 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.692+0000 7fb46f7fe700 1 --2- 172.21.15.44:0/3368261311 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb470100290 0x7fb4701982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:41.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.692+0000 7fb46effd700 1 --2- 172.21.15.44:0/3368261311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb470101480 0x7fb4701987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:41.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.692+0000 7fb46ffff700 1 -- 172.21.15.44:0/3368261311 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb470100290 msgr2=0x7fb4701982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:41.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.692+0000 7fb46ffff700 1 --2- 172.21.15.44:0/3368261311 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb470100290 0x7fb4701982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.692+0000 7fb46ffff700 1 -- 172.21.15.44:0/3368261311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb470101480 msgr2=0x7fb4701987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:41.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.692+0000 7fb46ffff700 1 --2- 172.21.15.44:0/3368261311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb470101480 0x7fb4701987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.692+0000 7fb46ffff700 1 -- 172.21.15.44:0/3368261311 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb468009d00 con 0x7fb470101e10 2023-11-16T16:49:41.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.693+0000 7fb46ffff700 1 --2- 172.21.15.44:0/3368261311 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb470101e10 0x7fb47019f850 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb46800d010 tx=0x7fb46800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:41.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.693+0000 7fb46cff9700 1 -- 172.21.15.44:0/3368261311 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46800e040 con 0x7fb470101e10 2023-11-16T16:49:41.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.693+0000 7fb4761ff700 1 -- 172.21.15.44:0/3368261311 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47010bc70 con 0x7fb470101e10 2023-11-16T16:49:41.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.693+0000 7fb46cff9700 1 -- 172.21.15.44:0/3368261311 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb46800bdb0 con 0x7fb470101e10 2023-11-16T16:49:41.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.693+0000 7fb46cff9700 1 -- 172.21.15.44:0/3368261311 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb468005260 con 0x7fb470101e10 2023-11-16T16:49:41.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.693+0000 7fb4761ff700 1 -- 172.21.15.44:0/3368261311 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47010bae0 con 0x7fb470101e10 2023-11-16T16:49:41.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.694+0000 7fb4761ff700 1 -- 172.21.15.44:0/3368261311 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb470067a60 con 0x7fb470101e10 2023-11-16T16:49:41.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.696+0000 7fb46cff9700 1 -- 172.21.15.44:0/3368261311 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb46801d480 con 0x7fb470101e10 2023-11-16T16:49:41.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.697+0000 7fb46cff9700 1 --2- 172.21.15.44:0/3368261311 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb4580a1150 0x7fb4580a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:41.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.697+0000 7fb46cff9700 1 -- 172.21.15.44:0/3368261311 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fb4680cde90 con 0x7fb470101e10 2023-11-16T16:49:41.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.698+0000 7fb46f7fe700 1 --2- 172.21.15.44:0/3368261311 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb4580a1150 0x7fb4580a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:41.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.698+0000 7fb46f7fe700 1 --2- 172.21.15.44:0/3368261311 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb4580a1150 0x7fb4580a3600 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb4701012e0 tx=0x7fb460009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:41.703 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.701+0000 7fb46cff9700 1 -- 172.21.15.44:0/3368261311 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb468097270 con 0x7fb470101e10 2023-11-16T16:49:41.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:41 smithi044 ceph-mon[128750]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 682 B/s wr, 2 op/s 2023-11-16T16:49:41.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:41 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3352392595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:41.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.906+0000 7fb4761ff700 1 -- 172.21.15.44:0/3368261311 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb470108430 con 0x7fb4580a1150 2023-11-16T16:49:41.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.919+0000 7fb46cff9700 1 -- 172.21.15.44:0/3368261311 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426969 (secure 0 0 0) 0x7fb470108430 con 0x7fb4580a1150 2023-11-16T16:49:41.926 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:41.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.924+0000 7fb4761ff700 1 -- 172.21.15.44:0/3368261311 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb4580a1150 msgr2=0x7fb4580a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:41.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.924+0000 7fb4761ff700 1 --2- 172.21.15.44:0/3368261311 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb4580a1150 0x7fb4580a3600 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb4701012e0 tx=0x7fb460009260 comp rx=0 tx=0).stop 2023-11-16T16:49:41.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.924+0000 7fb4761ff700 1 -- 172.21.15.44:0/3368261311 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb470101e10 msgr2=0x7fb47019f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:41.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.924+0000 7fb4761ff700 1 --2- 172.21.15.44:0/3368261311 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb470101e10 0x7fb47019f850 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb46800d010 tx=0x7fb46800bc80 comp rx=0 tx=0).stop 2023-11-16T16:49:41.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.924+0000 7fb4761ff700 1 -- 172.21.15.44:0/3368261311 shutdown_connections 2023-11-16T16:49:41.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.924+0000 7fb4761ff700 1 --2- 172.21.15.44:0/3368261311 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fb4580a1150 0x7fb4580a3600 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.924+0000 7fb4761ff700 1 --2- 172.21.15.44:0/3368261311 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb470101e10 0x7fb47019f850 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.924+0000 7fb4761ff700 1 --2- 172.21.15.44:0/3368261311 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb470100290 0x7fb4701982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.924+0000 7fb4761ff700 1 --2- 172.21.15.44:0/3368261311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb470101480 0x7fb4701987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:41.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.924+0000 7fb4761ff700 1 -- 172.21.15.44:0/3368261311 >> 172.21.15.44:0/3368261311 conn(0x7fb4700fb870 msgr2=0x7fb470106d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:41.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.924+0000 7fb4761ff700 1 -- 172.21.15.44:0/3368261311 shutdown_connections 2023-11-16T16:49:41.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:41.924+0000 7fb4761ff700 1 -- 172.21.15.44:0/3368261311 wait complete. 2023-11-16T16:49:41.930 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2023-11-16T16:49:41.945 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2023-11-16T16:49:40.570933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658264,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":375,"num_read_kb":309,"num_write":267,"num_write_kb":3039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":713,"ondisk_log_size":713,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":82132,"kb_used_data":12020,"kb_used_omap":0,"kb_used_meta":70016,"kb_avail":1124859692,"statfs":{"total":1151940427776,"available":1151856324608,"internally_reserved":0,"allocated":12308480,"data_stored":8034530,"data_compressed":390621,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":0,"internal_metadata":71696384},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5695,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":22,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003677"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.591350+0000","last_change":"2023-11-16T16:47:18.183024+0000","last_active":"2023-11-16T16:49:12.591350+0000","last_peered":"2023-11-16T16:49:12.591350+0000","last_clean":"2023-11-16T16:49:12.591350+0000","last_became_active":"2023-11-16T16:47:18.182387+0000","last_became_peered":"2023-11-16T16:47:18.182387+0000","last_unstale":"2023-11-16T16:49:12.591350+0000","last_undegraded":"2023-11-16T16:49:12.591350+0000","last_fullsized":"2023-11-16T16:49:12.591350+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:08:25.731718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.589033+0000","last_change":"2023-11-16T16:47:18.200135+0000","last_active":"2023-11-16T16:49:12.589033+0000","last_peered":"2023-11-16T16:49:12.589033+0000","last_clean":"2023-11-16T16:49:12.589033+0000","last_became_active":"2023-11-16T16:47:18.194732+0000","last_became_peered":"2023-11-16T16:47:18.194732+0000","last_unstale":"2023-11-16T16:49:12.589033+0000","last_undegraded":"2023-11-16T16:49:12.589033+0000","last_fullsized":"2023-11-16T16:49:12.589033+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:36:09.213648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.589098+0000","last_change":"2023-11-16T16:47:18.184127+0000","last_active":"2023-11-16T16:49:12.589098+0000","last_peered":"2023-11-16T16:49:12.589098+0000","last_clean":"2023-11-16T16:49:12.589098+0000","last_became_active":"2023-11-16T16:47:18.183163+0000","last_became_peered":"2023-11-16T16:47:18.183163+0000","last_unstale":"2023-11-16T16:49:12.589098+0000","last_undegraded":"2023-11-16T16:49:12.589098+0000","last_fullsized":"2023-11-16T16:49:12.589098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:18:16.732196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.577897+0000","last_change":"2023-11-16T16:47:18.177476+0000","last_active":"2023-11-16T16:49:12.577897+0000","last_peered":"2023-11-16T16:49:12.577897+0000","last_clean":"2023-11-16T16:49:12.577897+0000","last_became_active":"2023-11-16T16:47:18.177178+0000","last_became_peered":"2023-11-16T16:47:18.177178+0000","last_unstale":"2023-11-16T16:49:12.577897+0000","last_undegraded":"2023-11-16T16:49:12.577897+0000","last_fullsized":"2023-11-16T16:49:12.577897+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:34:33.524556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.578631+0000","last_change":"2023-11-16T16:47:18.178511+0000","last_active":"2023-11-16T16:49:12.578631+0000","last_peered":"2023-11-16T16:49:12.578631+0000","last_clean":"2023-11-16T16:49:12.578631+0000","last_became_active":"2023-11-16T16:47:18.178198+0000","last_became_peered":"2023-11-16T16:47:18.178198+0000","last_unstale":"2023-11-16T16:49:12.578631+0000","last_undegraded":"2023-11-16T16:49:12.578631+0000","last_fullsized":"2023-11-16T16:49:12.578631+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:04:36.059447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.577044+0000","last_change":"2023-11-16T16:47:18.186552+0000","last_active":"2023-11-16T16:49:12.577044+0000","last_peered":"2023-11-16T16:49:12.577044+0000","last_clean":"2023-11-16T16:49:12.577044+0000","last_became_active":"2023-11-16T16:47:18.186238+0000","last_became_peered":"2023-11-16T16:47:18.186238+0000","last_unstale":"2023-11-16T16:49:12.577044+0000","last_undegraded":"2023-11-16T16:49:12.577044+0000","last_fullsized":"2023-11-16T16:49:12.577044+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:58:35.571757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.566038+0000","last_change":"2023-11-16T16:47:18.179839+0000","last_active":"2023-11-16T16:49:12.566038+0000","last_peered":"2023-11-16T16:49:12.566038+0000","last_clean":"2023-11-16T16:49:12.566038+0000","last_became_active":"2023-11-16T16:47:18.179060+0000","last_became_peered":"2023-11-16T16:47:18.179060+0000","last_unstale":"2023-11-16T16:49:12.566038+0000","last_undegraded":"2023-11-16T16:49:12.566038+0000","last_fullsized":"2023-11-16T16:49:12.566038+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:04:42.573359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.576400+0000","last_change":"2023-11-16T16:47:43.631738+0000","last_active":"2023-11-16T16:49:12.576400+0000","last_peered":"2023-11-16T16:49:12.576400+0000","last_clean":"2023-11-16T16:49:12.576400+0000","last_became_active":"2023-11-16T16:47:43.631146+0000","last_became_peered":"2023-11-16T16:47:43.631146+0000","last_unstale":"2023-11-16T16:49:12.576400+0000","last_undegraded":"2023-11-16T16:49:12.576400+0000","last_fullsized":"2023-11-16T16:49:12.576400+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:58:11.158742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579232+0000","last_change":"2023-11-16T16:47:18.177834+0000","last_active":"2023-11-16T16:49:12.579232+0000","last_peered":"2023-11-16T16:49:12.579232+0000","last_clean":"2023-11-16T16:49:12.579232+0000","last_became_active":"2023-11-16T16:47:18.176299+0000","last_became_peered":"2023-11-16T16:47:18.176299+0000","last_unstale":"2023-11-16T16:49:12.579232+0000","last_undegraded":"2023-11-16T16:49:12.579232+0000","last_fullsized":"2023-11-16T16:49:12.579232+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:27:07.129418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.571605+0000","last_change":"2023-11-16T16:47:18.179394+0000","last_active":"2023-11-16T16:49:12.571605+0000","last_peered":"2023-11-16T16:49:12.571605+0000","last_clean":"2023-11-16T16:49:12.571605+0000","last_became_active":"2023-11-16T16:47:18.179119+0000","last_became_peered":"2023-11-16T16:47:18.179119+0000","last_unstale":"2023-11-16T16:49:12.571605+0000","last_undegraded":"2023-11-16T16:49:12.571605+0000","last_fullsized":"2023-11-16T16:49:12.571605+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:50:38.327278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579257+0000","last_change":"2023-11-16T16:47:18.183585+0000","last_active":"2023-11-16T16:49:12.579257+0000","last_peered":"2023-11-16T16:49:12.579257+0000","last_clean":"2023-11-16T16:49:12.579257+0000","last_became_active":"2023-11-16T16:47:18.183209+0000","last_became_peered":"2023-11-16T16:47:18.183209+0000","last_unstale":"2023-11-16T16:49:12.579257+0000","last_undegraded":"2023-11-16T16:49:12.579257+0000","last_fullsized":"2023-11-16T16:49:12.579257+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:35:11.334704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.570790+0000","last_change":"2023-11-16T16:47:18.186503+0000","last_active":"2023-11-16T16:49:12.570790+0000","last_peered":"2023-11-16T16:49:12.570790+0000","last_clean":"2023-11-16T16:49:12.570790+0000","last_became_active":"2023-11-16T16:47:18.186193+0000","last_became_peered":"2023-11-16T16:47:18.186193+0000","last_unstale":"2023-11-16T16:49:12.570790+0000","last_undegraded":"2023-11-16T16:49:12.570790+0000","last_fullsized":"2023-11-16T16:49:12.570790+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:08:26.864946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.584559+0000","last_change":"2023-11-16T16:47:18.193842+0000","last_active":"2023-11-16T16:49:12.584559+0000","last_peered":"2023-11-16T16:49:12.584559+0000","last_clean":"2023-11-16T16:49:12.584559+0000","last_became_active":"2023-11-16T16:47:18.192309+0000","last_became_peered":"2023-11-16T16:47:18.192309+0000","last_unstale":"2023-11-16T16:49:12.584559+0000","last_undegraded":"2023-11-16T16:49:12.584559+0000","last_fullsized":"2023-11-16T16:49:12.584559+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:47:16.555679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.585449+0000","last_change":"2023-11-16T16:47:18.192359+0000","last_active":"2023-11-16T16:49:12.585449+0000","last_peered":"2023-11-16T16:49:12.585449+0000","last_clean":"2023-11-16T16:49:12.585449+0000","last_became_active":"2023-11-16T16:47:18.189765+0000","last_became_peered":"2023-11-16T16:47:18.189765+0000","last_unstale":"2023-11-16T16:49:12.585449+0000","last_undegraded":"2023-11-16T16:49:12.585449+0000","last_fullsized":"2023-11-16T16:49:12.585449+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:56:51.066489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.588434+0000","last_change":"2023-11-16T16:47:18.178386+0000","last_active":"2023-11-16T16:49:12.588434+0000","last_peered":"2023-11-16T16:49:12.588434+0000","last_clean":"2023-11-16T16:49:12.588434+0000","last_became_active":"2023-11-16T16:47:18.178041+0000","last_became_peered":"2023-11-16T16:47:18.178041+0000","last_unstale":"2023-11-16T16:49:12.588434+0000","last_undegraded":"2023-11-16T16:49:12.588434+0000","last_fullsized":"2023-11-16T16:49:12.588434+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:08:49.324272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.591918+0000","last_change":"2023-11-16T16:47:18.174424+0000","last_active":"2023-11-16T16:49:12.591918+0000","last_peered":"2023-11-16T16:49:12.591918+0000","last_clean":"2023-11-16T16:49:12.591918+0000","last_became_active":"2023-11-16T16:47:18.174078+0000","last_became_peered":"2023-11-16T16:47:18.174078+0000","last_unstale":"2023-11-16T16:49:12.591918+0000","last_undegraded":"2023-11-16T16:49:12.591918+0000","last_fullsized":"2023-11-16T16:49:12.591918+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:23:55.960648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.591349+0000","last_change":"2023-11-16T16:47:18.183161+0000","last_active":"2023-11-16T16:49:12.591349+0000","last_peered":"2023-11-16T16:49:12.591349+0000","last_clean":"2023-11-16T16:49:12.591349+0000","last_became_active":"2023-11-16T16:47:18.182806+0000","last_became_peered":"2023-11-16T16:47:18.182806+0000","last_unstale":"2023-11-16T16:49:12.591349+0000","last_undegraded":"2023-11-16T16:49:12.591349+0000","last_fullsized":"2023-11-16T16:49:12.591349+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:17:48.777833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.580719+0000","last_change":"2023-11-16T16:47:18.187225+0000","last_active":"2023-11-16T16:49:12.580719+0000","last_peered":"2023-11-16T16:49:12.580719+0000","last_clean":"2023-11-16T16:49:12.580719+0000","last_became_active":"2023-11-16T16:47:18.186858+0000","last_became_peered":"2023-11-16T16:47:18.186858+0000","last_unstale":"2023-11-16T16:49:12.580719+0000","last_undegraded":"2023-11-16T16:49:12.580719+0000","last_fullsized":"2023-11-16T16:49:12.580719+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:03:50.025202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579231+0000","last_change":"2023-11-16T16:47:18.190947+0000","last_active":"2023-11-16T16:49:12.579231+0000","last_peered":"2023-11-16T16:49:12.579231+0000","last_clean":"2023-11-16T16:49:12.579231+0000","last_became_active":"2023-11-16T16:47:18.190737+0000","last_became_peered":"2023-11-16T16:47:18.190737+0000","last_unstale":"2023-11-16T16:49:12.579231+0000","last_undegraded":"2023-11-16T16:49:12.579231+0000","last_fullsized":"2023-11-16T16:49:12.579231+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:50:36.934811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623109+0000","last_change":"2023-11-16T16:47:22.659730+0000","last_active":"2023-11-16T16:48:44.623109+0000","last_peered":"2023-11-16T16:48:44.623109+0000","last_clean":"2023-11-16T16:48:44.623109+0000","last_became_active":"2023-11-16T16:47:22.659313+0000","last_became_peered":"2023-11-16T16:47:22.659313+0000","last_unstale":"2023-11-16T16:48:44.623109+0000","last_undegraded":"2023-11-16T16:48:44.623109+0000","last_fullsized":"2023-11-16T16:48:44.623109+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:12:16.481784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'4","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626280+0000","last_change":"2023-11-16T16:47:17.629748+0000","last_active":"2023-11-16T16:48:44.626280+0000","last_peered":"2023-11-16T16:48:44.626280+0000","last_clean":"2023-11-16T16:48:44.626280+0000","last_became_active":"2023-11-16T16:47:17.629284+0000","last_became_peered":"2023-11-16T16:47:17.629284+0000","last_unstale":"2023-11-16T16:48:44.626280+0000","last_undegraded":"2023-11-16T16:48:44.626280+0000","last_fullsized":"2023-11-16T16:48:44.626280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:07:47.593078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.577615+0000","last_change":"2023-11-16T16:47:18.176239+0000","last_active":"2023-11-16T16:49:12.577615+0000","last_peered":"2023-11-16T16:49:12.577615+0000","last_clean":"2023-11-16T16:49:12.577615+0000","last_became_active":"2023-11-16T16:47:18.175879+0000","last_became_peered":"2023-11-16T16:47:18.175879+0000","last_unstale":"2023-11-16T16:49:12.577615+0000","last_undegraded":"2023-11-16T16:49:12.577615+0000","last_fullsized":"2023-11-16T16:49:12.577615+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:31:16.561539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.622732+0000","last_change":"2023-11-16T16:47:22.659679+0000","last_active":"2023-11-16T16:48:44.622732+0000","last_peered":"2023-11-16T16:48:44.622732+0000","last_clean":"2023-11-16T16:48:44.622732+0000","last_became_active":"2023-11-16T16:47:22.658946+0000","last_became_peered":"2023-11-16T16:47:22.658946+0000","last_unstale":"2023-11-16T16:48:44.622732+0000","last_undegraded":"2023-11-16T16:48:44.622732+0000","last_fullsized":"2023-11-16T16:48:44.622732+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:32:34.649897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"82'1","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.912310+0000","last_change":"2023-11-16T16:47:17.643944+0000","last_active":"2023-11-16T16:48:44.912310+0000","last_peered":"2023-11-16T16:48:44.912310+0000","last_clean":"2023-11-16T16:48:44.912310+0000","last_became_active":"2023-11-16T16:47:17.643470+0000","last_became_peered":"2023-11-16T16:47:17.643470+0000","last_unstale":"2023-11-16T16:48:44.912310+0000","last_undegraded":"2023-11-16T16:48:44.912310+0000","last_fullsized":"2023-11-16T16:48:44.912310+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:54:38.723005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.588194+0000","last_change":"2023-11-16T16:47:18.198139+0000","last_active":"2023-11-16T16:49:12.588194+0000","last_peered":"2023-11-16T16:49:12.588194+0000","last_clean":"2023-11-16T16:49:12.588194+0000","last_became_active":"2023-11-16T16:47:18.197762+0000","last_became_peered":"2023-11-16T16:47:18.197762+0000","last_unstale":"2023-11-16T16:49:12.588194+0000","last_undegraded":"2023-11-16T16:49:12.588194+0000","last_fullsized":"2023-11-16T16:49:12.588194+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:24:38.383735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623023+0000","last_change":"2023-11-16T16:47:22.655810+0000","last_active":"2023-11-16T16:48:44.623023+0000","last_peered":"2023-11-16T16:48:44.623023+0000","last_clean":"2023-11-16T16:48:44.623023+0000","last_became_active":"2023-11-16T16:47:22.655388+0000","last_became_peered":"2023-11-16T16:47:22.655388+0000","last_unstale":"2023-11-16T16:48:44.623023+0000","last_undegraded":"2023-11-16T16:48:44.623023+0000","last_fullsized":"2023-11-16T16:48:44.623023+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:44:39.423524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'2","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.630121+0000","last_change":"2023-11-16T16:47:43.635266+0000","last_active":"2023-11-16T16:48:44.630121+0000","last_peered":"2023-11-16T16:48:44.630121+0000","last_clean":"2023-11-16T16:48:44.630121+0000","last_became_active":"2023-11-16T16:47:43.630583+0000","last_became_peered":"2023-11-16T16:47:43.630583+0000","last_unstale":"2023-11-16T16:48:44.630121+0000","last_undegraded":"2023-11-16T16:48:44.630121+0000","last_fullsized":"2023-11-16T16:48:44.630121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:09:31.118965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.585032+0000","last_change":"2023-11-16T16:47:18.186341+0000","last_active":"2023-11-16T16:49:12.585032+0000","last_peered":"2023-11-16T16:49:12.585032+0000","last_clean":"2023-11-16T16:49:12.585032+0000","last_became_active":"2023-11-16T16:47:18.186066+0000","last_became_peered":"2023-11-16T16:47:18.186066+0000","last_unstale":"2023-11-16T16:49:12.585032+0000","last_undegraded":"2023-11-16T16:49:12.585032+0000","last_fullsized":"2023-11-16T16:49:12.585032+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:35:29.489855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.622861+0000","last_change":"2023-11-16T16:47:22.657227+0000","last_active":"2023-11-16T16:48:44.622861+0000","last_peered":"2023-11-16T16:48:44.622861+0000","last_clean":"2023-11-16T16:48:44.622861+0000","last_became_active":"2023-11-16T16:47:22.656835+0000","last_became_peered":"2023-11-16T16:47:22.656835+0000","last_unstale":"2023-11-16T16:48:44.622861+0000","last_undegraded":"2023-11-16T16:48:44.622861+0000","last_fullsized":"2023-11-16T16:48:44.622861+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:55:50.404613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.630547+0000","last_change":"2023-11-16T16:47:17.630194+0000","last_active":"2023-11-16T16:48:44.630547+0000","last_peered":"2023-11-16T16:48:44.630547+0000","last_clean":"2023-11-16T16:48:44.630547+0000","last_became_active":"2023-11-16T16:47:17.629886+0000","last_became_peered":"2023-11-16T16:47:17.629886+0000","last_unstale":"2023-11-16T16:48:44.630547+0000","last_undegraded":"2023-11-16T16:48:44.630547+0000","last_fullsized":"2023-11-16T16:48:44.630547+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:58:10.365614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.573343+0000","last_change":"2023-11-16T16:47:18.181596+0000","last_active":"2023-11-16T16:49:12.573343+0000","last_peered":"2023-11-16T16:49:12.573343+0000","last_clean":"2023-11-16T16:49:12.573343+0000","last_became_active":"2023-11-16T16:47:18.181287+0000","last_became_peered":"2023-11-16T16:47:18.181287+0000","last_unstale":"2023-11-16T16:49:12.573343+0000","last_undegraded":"2023-11-16T16:49:12.573343+0000","last_fullsized":"2023-11-16T16:49:12.573343+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:10:47.679818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623684+0000","last_change":"2023-11-16T16:47:22.649197+0000","last_active":"2023-11-16T16:48:44.623684+0000","last_peered":"2023-11-16T16:48:44.623684+0000","last_clean":"2023-11-16T16:48:44.623684+0000","last_became_active":"2023-11-16T16:47:22.648664+0000","last_became_peered":"2023-11-16T16:47:22.648664+0000","last_unstale":"2023-11-16T16:48:44.623684+0000","last_undegraded":"2023-11-16T16:48:44.623684+0000","last_fullsized":"2023-11-16T16:48:44.623684+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:52:46.052156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"82'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911165+0000","last_change":"2023-11-16T16:47:17.627672+0000","last_active":"2023-11-16T16:48:44.911165+0000","last_peered":"2023-11-16T16:48:44.911165+0000","last_clean":"2023-11-16T16:48:44.911165+0000","last_became_active":"2023-11-16T16:47:17.627383+0000","last_became_peered":"2023-11-16T16:47:17.627383+0000","last_unstale":"2023-11-16T16:48:44.911165+0000","last_undegraded":"2023-11-16T16:48:44.911165+0000","last_fullsized":"2023-11-16T16:48:44.911165+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:51:59.811190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.560868+0000","last_change":"2023-11-16T16:47:18.180983+0000","last_active":"2023-11-16T16:49:12.560868+0000","last_peered":"2023-11-16T16:49:12.560868+0000","last_clean":"2023-11-16T16:49:12.560868+0000","last_became_active":"2023-11-16T16:47:18.180702+0000","last_became_peered":"2023-11-16T16:47:18.180702+0000","last_unstale":"2023-11-16T16:49:12.560868+0000","last_undegraded":"2023-11-16T16:49:12.560868+0000","last_fullsized":"2023-11-16T16:49:12.560868+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:02:50.226183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.913692+0000","last_change":"2023-11-16T16:48:44.913692+0000","last_active":"2023-11-16T16:48:44.913692+0000","last_peered":"2023-11-16T16:48:44.913692+0000","last_clean":"2023-11-16T16:48:44.913692+0000","last_became_active":"2023-11-16T16:48:44.913204+0000","last_became_peered":"2023-11-16T16:48:44.913204+0000","last_unstale":"2023-11-16T16:48:44.913692+0000","last_undegraded":"2023-11-16T16:48:44.913692+0000","last_fullsized":"2023-11-16T16:48:44.913692+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:04:35.705849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"82'1","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626774+0000","last_change":"2023-11-16T16:47:43.949231+0000","last_active":"2023-11-16T16:48:44.626774+0000","last_peered":"2023-11-16T16:48:44.626774+0000","last_clean":"2023-11-16T16:48:44.626774+0000","last_became_active":"2023-11-16T16:47:43.948766+0000","last_became_peered":"2023-11-16T16:47:43.948766+0000","last_unstale":"2023-11-16T16:48:44.626774+0000","last_undegraded":"2023-11-16T16:48:44.626774+0000","last_fullsized":"2023-11-16T16:48:44.626774+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:10:43.974784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579822+0000","last_change":"2023-11-16T16:47:18.185263+0000","last_active":"2023-11-16T16:49:12.579822+0000","last_peered":"2023-11-16T16:49:12.579822+0000","last_clean":"2023-11-16T16:49:12.579822+0000","last_became_active":"2023-11-16T16:47:18.184910+0000","last_became_peered":"2023-11-16T16:47:18.184910+0000","last_unstale":"2023-11-16T16:49:12.579822+0000","last_undegraded":"2023-11-16T16:49:12.579822+0000","last_fullsized":"2023-11-16T16:49:12.579822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:33:21.979376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628573+0000","last_change":"2023-11-16T16:47:22.659711+0000","last_active":"2023-11-16T16:48:44.628573+0000","last_peered":"2023-11-16T16:48:44.628573+0000","last_clean":"2023-11-16T16:48:44.628573+0000","last_became_active":"2023-11-16T16:47:22.659087+0000","last_became_peered":"2023-11-16T16:47:22.659087+0000","last_unstale":"2023-11-16T16:48:44.628573+0000","last_undegraded":"2023-11-16T16:48:44.628573+0000","last_fullsized":"2023-11-16T16:48:44.628573+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:21:48.086846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910749+0000","last_change":"2023-11-16T16:47:17.629703+0000","last_active":"2023-11-16T16:48:44.910749+0000","last_peered":"2023-11-16T16:48:44.910749+0000","last_clean":"2023-11-16T16:48:44.910749+0000","last_became_active":"2023-11-16T16:47:17.629245+0000","last_became_peered":"2023-11-16T16:47:17.629245+0000","last_unstale":"2023-11-16T16:48:44.910749+0000","last_undegraded":"2023-11-16T16:48:44.910749+0000","last_fullsized":"2023-11-16T16:48:44.910749+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:40:19.828594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.584225+0000","last_change":"2023-11-16T16:47:18.187149+0000","last_active":"2023-11-16T16:49:12.584225+0000","last_peered":"2023-11-16T16:49:12.584225+0000","last_clean":"2023-11-16T16:49:12.584225+0000","last_became_active":"2023-11-16T16:47:18.185717+0000","last_became_peered":"2023-11-16T16:47:18.185717+0000","last_unstale":"2023-11-16T16:49:12.584225+0000","last_undegraded":"2023-11-16T16:49:12.584225+0000","last_fullsized":"2023-11-16T16:49:12.584225+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:47:48.993299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.913952+0000","last_change":"2023-11-16T16:47:22.663025+0000","last_active":"2023-11-16T16:48:44.913952+0000","last_peered":"2023-11-16T16:48:44.913952+0000","last_clean":"2023-11-16T16:48:44.913952+0000","last_became_active":"2023-11-16T16:47:22.662596+0000","last_became_peered":"2023-11-16T16:47:22.662596+0000","last_unstale":"2023-11-16T16:48:44.913952+0000","last_undegraded":"2023-11-16T16:48:44.913952+0000","last_fullsized":"2023-11-16T16:48:44.913952+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:01:20.821553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910950+0000","last_change":"2023-11-16T16:47:17.626357+0000","last_active":"2023-11-16T16:48:44.910950+0000","last_peered":"2023-11-16T16:48:44.910950+0000","last_clean":"2023-11-16T16:48:44.910950+0000","last_became_active":"2023-11-16T16:47:17.626029+0000","last_became_peered":"2023-11-16T16:47:17.626029+0000","last_unstale":"2023-11-16T16:48:44.910950+0000","last_undegraded":"2023-11-16T16:48:44.910950+0000","last_fullsized":"2023-11-16T16:48:44.910950+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:55:41.716453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579606+0000","last_change":"2023-11-16T16:47:18.177153+0000","last_active":"2023-11-16T16:49:12.579606+0000","last_peered":"2023-11-16T16:49:12.579606+0000","last_clean":"2023-11-16T16:49:12.579606+0000","last_became_active":"2023-11-16T16:47:18.176751+0000","last_became_peered":"2023-11-16T16:47:18.176751+0000","last_unstale":"2023-11-16T16:49:12.579606+0000","last_undegraded":"2023-11-16T16:49:12.579606+0000","last_fullsized":"2023-11-16T16:49:12.579606+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:21:22.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910720+0000","last_change":"2023-11-16T16:47:22.652718+0000","last_active":"2023-11-16T16:48:44.910720+0000","last_peered":"2023-11-16T16:48:44.910720+0000","last_clean":"2023-11-16T16:48:44.910720+0000","last_became_active":"2023-11-16T16:47:22.652278+0000","last_became_peered":"2023-11-16T16:47:22.652278+0000","last_unstale":"2023-11-16T16:48:44.910720+0000","last_undegraded":"2023-11-16T16:48:44.910720+0000","last_fullsized":"2023-11-16T16:48:44.910720+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:34:31.220318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.627489+0000","last_change":"2023-11-16T16:47:17.627308+0000","last_active":"2023-11-16T16:48:44.627489+0000","last_peered":"2023-11-16T16:48:44.627489+0000","last_clean":"2023-11-16T16:48:44.627489+0000","last_became_active":"2023-11-16T16:47:17.626992+0000","last_became_peered":"2023-11-16T16:47:17.626992+0000","last_unstale":"2023-11-16T16:48:44.627489+0000","last_undegraded":"2023-11-16T16:48:44.627489+0000","last_fullsized":"2023-11-16T16:48:44.627489+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:01:05.366137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579997+0000","last_change":"2023-11-16T16:47:18.188015+0000","last_active":"2023-11-16T16:49:12.579997+0000","last_peered":"2023-11-16T16:49:12.579997+0000","last_clean":"2023-11-16T16:49:12.579997+0000","last_became_active":"2023-11-16T16:47:18.187730+0000","last_became_peered":"2023-11-16T16:47:18.187730+0000","last_unstale":"2023-11-16T16:49:12.579997+0000","last_undegraded":"2023-11-16T16:49:12.579997+0000","last_fullsized":"2023-11-16T16:49:12.579997+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:43:39.411790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.625358+0000","last_change":"2023-11-16T16:47:22.655656+0000","last_active":"2023-11-16T16:48:44.625358+0000","last_peered":"2023-11-16T16:48:44.625358+0000","last_clean":"2023-11-16T16:48:44.625358+0000","last_became_active":"2023-11-16T16:47:22.655133+0000","last_became_peered":"2023-11-16T16:47:22.655133+0000","last_unstale":"2023-11-16T16:48:44.625358+0000","last_undegraded":"2023-11-16T16:48:44.625358+0000","last_fullsized":"2023-11-16T16:48:44.625358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:57:06.052661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626104+0000","last_change":"2023-11-16T16:47:43.947947+0000","last_active":"2023-11-16T16:48:44.626104+0000","last_peered":"2023-11-16T16:48:44.626104+0000","last_clean":"2023-11-16T16:48:44.626104+0000","last_became_active":"2023-11-16T16:47:43.947478+0000","last_became_peered":"2023-11-16T16:47:43.947478+0000","last_unstale":"2023-11-16T16:48:44.626104+0000","last_undegraded":"2023-11-16T16:48:44.626104+0000","last_fullsized":"2023-11-16T16:48:44.626104+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:31:17.754660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.590686+0000","last_change":"2023-11-16T16:47:43.632657+0000","last_active":"2023-11-16T16:49:12.590686+0000","last_peered":"2023-11-16T16:49:12.590686+0000","last_clean":"2023-11-16T16:49:12.590686+0000","last_became_active":"2023-11-16T16:47:43.631573+0000","last_became_peered":"2023-11-16T16:47:43.631573+0000","last_unstale":"2023-11-16T16:49:12.590686+0000","last_undegraded":"2023-11-16T16:49:12.590686+0000","last_fullsized":"2023-11-16T16:49:12.590686+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:22:12.311807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623185+0000","last_change":"2023-11-16T16:47:22.661639+0000","last_active":"2023-11-16T16:48:44.623185+0000","last_peered":"2023-11-16T16:48:44.623185+0000","last_clean":"2023-11-16T16:48:44.623185+0000","last_became_active":"2023-11-16T16:47:22.661230+0000","last_became_peered":"2023-11-16T16:47:22.661230+0000","last_unstale":"2023-11-16T16:48:44.623185+0000","last_undegraded":"2023-11-16T16:48:44.623185+0000","last_fullsized":"2023-11-16T16:48:44.623185+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:25:46.012584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"82'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911151+0000","last_change":"2023-11-16T16:47:17.638209+0000","last_active":"2023-11-16T16:48:44.911151+0000","last_peered":"2023-11-16T16:48:44.911151+0000","last_clean":"2023-11-16T16:48:44.911151+0000","last_became_active":"2023-11-16T16:47:17.637957+0000","last_became_peered":"2023-11-16T16:47:17.637957+0000","last_unstale":"2023-11-16T16:48:44.911151+0000","last_undegraded":"2023-11-16T16:48:44.911151+0000","last_fullsized":"2023-11-16T16:48:44.911151+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:46:12.089473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.580155+0000","last_change":"2023-11-16T16:47:18.182693+0000","last_active":"2023-11-16T16:49:12.580155+0000","last_peered":"2023-11-16T16:49:12.580155+0000","last_clean":"2023-11-16T16:49:12.580155+0000","last_became_active":"2023-11-16T16:47:18.182370+0000","last_became_peered":"2023-11-16T16:47:18.182370+0000","last_unstale":"2023-11-16T16:49:12.580155+0000","last_undegraded":"2023-11-16T16:49:12.580155+0000","last_fullsized":"2023-11-16T16:49:12.580155+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:52.028156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.625577+0000","last_change":"2023-11-16T16:47:22.649348+0000","last_active":"2023-11-16T16:48:44.625577+0000","last_peered":"2023-11-16T16:48:44.625577+0000","last_clean":"2023-11-16T16:48:44.625577+0000","last_became_active":"2023-11-16T16:47:22.648860+0000","last_became_peered":"2023-11-16T16:47:22.648860+0000","last_unstale":"2023-11-16T16:48:44.625577+0000","last_undegraded":"2023-11-16T16:48:44.625577+0000","last_fullsized":"2023-11-16T16:48:44.625577+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:13:04.049776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'37","reported_seq":301,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:39.352456+0000","last_change":"2023-11-16T16:47:17.619417+0000","last_active":"2023-11-16T16:49:39.352456+0000","last_peered":"2023-11-16T16:49:39.352456+0000","last_clean":"2023-11-16T16:49:39.352456+0000","last_became_active":"2023-11-16T16:47:17.619086+0000","last_became_peered":"2023-11-16T16:47:17.619086+0000","last_unstale":"2023-11-16T16:49:39.352456+0000","last_undegraded":"2023-11-16T16:49:39.352456+0000","last_fullsized":"2023-11-16T16:49:39.352456+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:47:53.380456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66852,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":233,"num_read_kb":184,"num_write":37,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.566501+0000","last_change":"2023-11-16T16:47:18.177537+0000","last_active":"2023-11-16T16:49:12.566501+0000","last_peered":"2023-11-16T16:49:12.566501+0000","last_clean":"2023-11-16T16:49:12.566501+0000","last_became_active":"2023-11-16T16:47:18.177112+0000","last_became_peered":"2023-11-16T16:47:18.177112+0000","last_unstale":"2023-11-16T16:49:12.566501+0000","last_undegraded":"2023-11-16T16:49:12.566501+0000","last_fullsized":"2023-11-16T16:49:12.566501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:46:36.459769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911616+0000","last_change":"2023-11-16T16:47:22.658086+0000","last_active":"2023-11-16T16:48:44.911616+0000","last_peered":"2023-11-16T16:48:44.911616+0000","last_clean":"2023-11-16T16:48:44.911616+0000","last_became_active":"2023-11-16T16:47:22.657725+0000","last_became_peered":"2023-11-16T16:47:22.657725+0000","last_unstale":"2023-11-16T16:48:44.911616+0000","last_undegraded":"2023-11-16T16:48:44.911616+0000","last_fullsized":"2023-11-16T16:48:44.911616+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:20:10.492766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"82'2","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623489+0000","last_change":"2023-11-16T16:47:17.625405+0000","last_active":"2023-11-16T16:48:44.623489+0000","last_peered":"2023-11-16T16:48:44.623489+0000","last_clean":"2023-11-16T16:48:44.623489+0000","last_became_active":"2023-11-16T16:47:17.625026+0000","last_became_peered":"2023-11-16T16:47:17.625026+0000","last_unstale":"2023-11-16T16:48:44.623489+0000","last_undegraded":"2023-11-16T16:48:44.623489+0000","last_fullsized":"2023-11-16T16:48:44.623489+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:07:58.847671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.582795+0000","last_change":"2023-11-16T16:47:18.196345+0000","last_active":"2023-11-16T16:49:12.582795+0000","last_peered":"2023-11-16T16:49:12.582795+0000","last_clean":"2023-11-16T16:49:12.582795+0000","last_became_active":"2023-11-16T16:47:18.196040+0000","last_became_peered":"2023-11-16T16:47:18.196040+0000","last_unstale":"2023-11-16T16:49:12.582795+0000","last_undegraded":"2023-11-16T16:49:12.582795+0000","last_fullsized":"2023-11-16T16:49:12.582795+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:07:23.062566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.914613+0000","last_change":"2023-11-16T16:48:44.914613+0000","last_active":"2023-11-16T16:48:44.914613+0000","last_peered":"2023-11-16T16:48:44.914613+0000","last_clean":"2023-11-16T16:48:44.914613+0000","last_became_active":"2023-11-16T16:48:44.914216+0000","last_became_peered":"2023-11-16T16:48:44.914216+0000","last_unstale":"2023-11-16T16:48:44.914613+0000","last_undegraded":"2023-11-16T16:48:44.914613+0000","last_fullsized":"2023-11-16T16:48:44.914613+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:40:27.544928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911496+0000","last_change":"2023-11-16T16:47:17.630481+0000","last_active":"2023-11-16T16:48:44.911496+0000","last_peered":"2023-11-16T16:48:44.911496+0000","last_clean":"2023-11-16T16:48:44.911496+0000","last_became_active":"2023-11-16T16:47:17.630190+0000","last_became_peered":"2023-11-16T16:47:17.630190+0000","last_unstale":"2023-11-16T16:48:44.911496+0000","last_undegraded":"2023-11-16T16:48:44.911496+0000","last_fullsized":"2023-11-16T16:48:44.911496+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:22:25.493507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.585638+0000","last_change":"2023-11-16T16:48:44.913061+0000","last_active":"2023-11-16T16:49:12.585638+0000","last_peered":"2023-11-16T16:49:12.585638+0000","last_clean":"2023-11-16T16:49:12.585638+0000","last_became_active":"2023-11-16T16:48:44.912616+0000","last_became_peered":"2023-11-16T16:48:44.912616+0000","last_unstale":"2023-11-16T16:49:12.585638+0000","last_undegraded":"2023-11-16T16:49:12.585638+0000","last_fullsized":"2023-11-16T16:49:12.585638+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:12:42.146437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626024+0000","last_change":"2023-11-16T16:47:22.656312+0000","last_active":"2023-11-16T16:48:44.626024+0000","last_peered":"2023-11-16T16:48:44.626024+0000","last_clean":"2023-11-16T16:48:44.626024+0000","last_became_active":"2023-11-16T16:47:22.655870+0000","last_became_peered":"2023-11-16T16:47:22.655870+0000","last_unstale":"2023-11-16T16:48:44.626024+0000","last_undegraded":"2023-11-16T16:48:44.626024+0000","last_fullsized":"2023-11-16T16:48:44.626024+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:26:18.515418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.622957+0000","last_change":"2023-11-16T16:47:17.622648+0000","last_active":"2023-11-16T16:48:44.622957+0000","last_peered":"2023-11-16T16:48:44.622957+0000","last_clean":"2023-11-16T16:48:44.622957+0000","last_became_active":"2023-11-16T16:47:17.622329+0000","last_became_peered":"2023-11-16T16:47:17.622329+0000","last_unstale":"2023-11-16T16:48:44.622957+0000","last_undegraded":"2023-11-16T16:48:44.622957+0000","last_fullsized":"2023-11-16T16:48:44.622957+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:04:26.204200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.569410+0000","last_change":"2023-11-16T16:47:18.179712+0000","last_active":"2023-11-16T16:49:12.569410+0000","last_peered":"2023-11-16T16:49:12.569410+0000","last_clean":"2023-11-16T16:49:12.569410+0000","last_became_active":"2023-11-16T16:47:18.179392+0000","last_became_peered":"2023-11-16T16:47:18.179392+0000","last_unstale":"2023-11-16T16:49:12.569410+0000","last_undegraded":"2023-11-16T16:49:12.569410+0000","last_fullsized":"2023-11-16T16:49:12.569410+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:14:52.593975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.912858+0000","last_change":"2023-11-16T16:47:22.665088+0000","last_active":"2023-11-16T16:48:44.912858+0000","last_peered":"2023-11-16T16:48:44.912858+0000","last_clean":"2023-11-16T16:48:44.912858+0000","last_became_active":"2023-11-16T16:47:22.662802+0000","last_became_peered":"2023-11-16T16:47:22.662802+0000","last_unstale":"2023-11-16T16:48:44.912858+0000","last_undegraded":"2023-11-16T16:48:44.912858+0000","last_fullsized":"2023-11-16T16:48:44.912858+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:30:36.233143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"82'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628107+0000","last_change":"2023-11-16T16:47:17.639246+0000","last_active":"2023-11-16T16:48:44.628107+0000","last_peered":"2023-11-16T16:48:44.628107+0000","last_clean":"2023-11-16T16:48:44.628107+0000","last_became_active":"2023-11-16T16:47:17.638928+0000","last_became_peered":"2023-11-16T16:47:17.638928+0000","last_unstale":"2023-11-16T16:48:44.628107+0000","last_undegraded":"2023-11-16T16:48:44.628107+0000","last_fullsized":"2023-11-16T16:48:44.628107+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:18:59.651350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.568411+0000","last_change":"2023-11-16T16:47:18.182609+0000","last_active":"2023-11-16T16:49:12.568411+0000","last_peered":"2023-11-16T16:49:12.568411+0000","last_clean":"2023-11-16T16:49:12.568411+0000","last_became_active":"2023-11-16T16:47:18.182260+0000","last_became_peered":"2023-11-16T16:47:18.182260+0000","last_unstale":"2023-11-16T16:49:12.568411+0000","last_undegraded":"2023-11-16T16:49:12.568411+0000","last_fullsized":"2023-11-16T16:49:12.568411+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:31:25.530504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911115+0000","last_change":"2023-11-16T16:47:22.647586+0000","last_active":"2023-11-16T16:48:44.911115+0000","last_peered":"2023-11-16T16:48:44.911115+0000","last_clean":"2023-11-16T16:48:44.911115+0000","last_became_active":"2023-11-16T16:47:22.647187+0000","last_became_peered":"2023-11-16T16:47:22.647187+0000","last_unstale":"2023-11-16T16:48:44.911115+0000","last_undegraded":"2023-11-16T16:48:44.911115+0000","last_fullsized":"2023-11-16T16:48:44.911115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:29:19.788377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628288+0000","last_change":"2023-11-16T16:47:43.933501+0000","last_active":"2023-11-16T16:48:44.628288+0000","last_peered":"2023-11-16T16:48:44.628288+0000","last_clean":"2023-11-16T16:48:44.628288+0000","last_became_active":"2023-11-16T16:47:43.631746+0000","last_became_peered":"2023-11-16T16:47:43.631746+0000","last_unstale":"2023-11-16T16:48:44.628288+0000","last_undegraded":"2023-11-16T16:48:44.628288+0000","last_fullsized":"2023-11-16T16:48:44.628288+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:01:43.174333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.567503+0000","last_change":"2023-11-16T16:47:18.175210+0000","last_active":"2023-11-16T16:49:12.567503+0000","last_peered":"2023-11-16T16:49:12.567503+0000","last_clean":"2023-11-16T16:49:12.567503+0000","last_became_active":"2023-11-16T16:47:18.174926+0000","last_became_peered":"2023-11-16T16:47:18.174926+0000","last_unstale":"2023-11-16T16:49:12.567503+0000","last_undegraded":"2023-11-16T16:49:12.567503+0000","last_fullsized":"2023-11-16T16:49:12.567503+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:51:21.318906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628055+0000","last_change":"2023-11-16T16:47:22.719361+0000","last_active":"2023-11-16T16:48:44.628055+0000","last_peered":"2023-11-16T16:48:44.628055+0000","last_clean":"2023-11-16T16:48:44.628055+0000","last_became_active":"2023-11-16T16:47:22.718936+0000","last_became_peered":"2023-11-16T16:47:22.718936+0000","last_unstale":"2023-11-16T16:48:44.628055+0000","last_undegraded":"2023-11-16T16:48:44.628055+0000","last_fullsized":"2023-11-16T16:48:44.628055+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:07:12.278332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626068+0000","last_change":"2023-11-16T16:47:17.632623+0000","last_active":"2023-11-16T16:48:44.626068+0000","last_peered":"2023-11-16T16:48:44.626068+0000","last_clean":"2023-11-16T16:48:44.626068+0000","last_became_active":"2023-11-16T16:47:17.631323+0000","last_became_peered":"2023-11-16T16:47:17.631323+0000","last_unstale":"2023-11-16T16:48:44.626068+0000","last_undegraded":"2023-11-16T16:48:44.626068+0000","last_fullsized":"2023-11-16T16:48:44.626068+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:15:53.760791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.584748+0000","last_change":"2023-11-16T16:47:18.190779+0000","last_active":"2023-11-16T16:49:12.584748+0000","last_peered":"2023-11-16T16:49:12.584748+0000","last_clean":"2023-11-16T16:49:12.584748+0000","last_became_active":"2023-11-16T16:47:18.190440+0000","last_became_peered":"2023-11-16T16:47:18.190440+0000","last_unstale":"2023-11-16T16:49:12.584748+0000","last_undegraded":"2023-11-16T16:49:12.584748+0000","last_fullsized":"2023-11-16T16:49:12.584748+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:42:49.621136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911253+0000","last_change":"2023-11-16T16:47:22.664847+0000","last_active":"2023-11-16T16:48:44.911253+0000","last_peered":"2023-11-16T16:48:44.911253+0000","last_clean":"2023-11-16T16:48:44.911253+0000","last_became_active":"2023-11-16T16:47:22.659382+0000","last_became_peered":"2023-11-16T16:47:22.659382+0000","last_unstale":"2023-11-16T16:48:44.911253+0000","last_undegraded":"2023-11-16T16:48:44.911253+0000","last_fullsized":"2023-11-16T16:48:44.911253+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:25:48.404036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.625730+0000","last_change":"2023-11-16T16:47:17.630956+0000","last_active":"2023-11-16T16:48:44.625730+0000","last_peered":"2023-11-16T16:48:44.625730+0000","last_clean":"2023-11-16T16:48:44.625730+0000","last_became_active":"2023-11-16T16:47:17.629864+0000","last_became_peered":"2023-11-16T16:47:17.629864+0000","last_unstale":"2023-11-16T16:48:44.625730+0000","last_undegraded":"2023-11-16T16:48:44.625730+0000","last_fullsized":"2023-11-16T16:48:44.625730+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:34:09.169015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.581549+0000","last_change":"2023-11-16T16:47:18.183375+0000","last_active":"2023-11-16T16:49:12.581549+0000","last_peered":"2023-11-16T16:49:12.581549+0000","last_clean":"2023-11-16T16:49:12.581549+0000","last_became_active":"2023-11-16T16:47:18.183050+0000","last_became_peered":"2023-11-16T16:47:18.183050+0000","last_unstale":"2023-11-16T16:49:12.581549+0000","last_undegraded":"2023-11-16T16:49:12.581549+0000","last_fullsized":"2023-11-16T16:49:12.581549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:13:59.684725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911824+0000","last_change":"2023-11-16T16:47:22.719109+0000","last_active":"2023-11-16T16:48:44.911824+0000","last_peered":"2023-11-16T16:48:44.911824+0000","last_clean":"2023-11-16T16:48:44.911824+0000","last_became_active":"2023-11-16T16:47:22.718680+0000","last_became_peered":"2023-11-16T16:47:22.718680+0000","last_unstale":"2023-11-16T16:48:44.911824+0000","last_undegraded":"2023-11-16T16:48:44.911824+0000","last_fullsized":"2023-11-16T16:48:44.911824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:23:29.879756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.569880+0000","last_change":"2023-11-16T16:47:18.181632+0000","last_active":"2023-11-16T16:49:12.569880+0000","last_peered":"2023-11-16T16:49:12.569880+0000","last_clean":"2023-11-16T16:49:12.569880+0000","last_became_active":"2023-11-16T16:47:18.181323+0000","last_became_peered":"2023-11-16T16:47:18.181323+0000","last_unstale":"2023-11-16T16:49:12.569880+0000","last_undegraded":"2023-11-16T16:49:12.569880+0000","last_fullsized":"2023-11-16T16:49:12.569880+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:02:57.803797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910665+0000","last_change":"2023-11-16T16:47:17.637269+0000","last_active":"2023-11-16T16:48:44.910665+0000","last_peered":"2023-11-16T16:48:44.910665+0000","last_clean":"2023-11-16T16:48:44.910665+0000","last_became_active":"2023-11-16T16:47:17.636881+0000","last_became_peered":"2023-11-16T16:47:17.636881+0000","last_unstale":"2023-11-16T16:48:44.910665+0000","last_undegraded":"2023-11-16T16:48:44.910665+0000","last_fullsized":"2023-11-16T16:48:44.910665+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:15:29.077370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626370+0000","last_change":"2023-11-16T16:47:22.659528+0000","last_active":"2023-11-16T16:48:44.626370+0000","last_peered":"2023-11-16T16:48:44.626370+0000","last_clean":"2023-11-16T16:48:44.626370+0000","last_became_active":"2023-11-16T16:47:22.659076+0000","last_became_peered":"2023-11-16T16:47:22.659076+0000","last_unstale":"2023-11-16T16:48:44.626370+0000","last_undegraded":"2023-11-16T16:48:44.626370+0000","last_fullsized":"2023-11-16T16:48:44.626370+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:22:26.166083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579137+0000","last_change":"2023-11-16T16:47:18.182102+0000","last_active":"2023-11-16T16:49:12.579137+0000","last_peered":"2023-11-16T16:49:12.579137+0000","last_clean":"2023-11-16T16:49:12.579137+0000","last_became_active":"2023-11-16T16:47:18.181528+0000","last_became_peered":"2023-11-16T16:47:18.181528+0000","last_unstale":"2023-11-16T16:49:12.579137+0000","last_undegraded":"2023-11-16T16:49:12.579137+0000","last_fullsized":"2023-11-16T16:49:12.579137+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:07:02.787319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623090+0000","last_change":"2023-11-16T16:47:17.626655+0000","last_active":"2023-11-16T16:48:44.623090+0000","last_peered":"2023-11-16T16:48:44.623090+0000","last_clean":"2023-11-16T16:48:44.623090+0000","last_became_active":"2023-11-16T16:47:17.626329+0000","last_became_peered":"2023-11-16T16:47:17.626329+0000","last_unstale":"2023-11-16T16:48:44.623090+0000","last_undegraded":"2023-11-16T16:48:44.623090+0000","last_fullsized":"2023-11-16T16:48:44.623090+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:22:13.860674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":357,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.625369+0000","last_change":"2023-11-16T16:46:11.943398+0000","last_active":"2023-11-16T16:48:44.625369+0000","last_peered":"2023-11-16T16:48:44.625369+0000","last_clean":"2023-11-16T16:48:44.625369+0000","last_became_active":"2023-11-16T16:46:11.942862+0000","last_became_peered":"2023-11-16T16:46:11.942862+0000","last_unstale":"2023-11-16T16:48:44.625369+0000","last_undegraded":"2023-11-16T16:48:44.625369+0000","last_fullsized":"2023-11-16T16:48:44.625369+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:43:03.109797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:43:03.109797+0000","last_clean_scrub_stamp":"2023-11-16T16:43:03.109797+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:27:18.973668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.915360+0000","last_change":"2023-11-16T16:48:44.914761+0000","last_active":"2023-11-16T16:48:44.915360+0000","last_peered":"2023-11-16T16:48:44.915360+0000","last_clean":"2023-11-16T16:48:44.915360+0000","last_became_active":"2023-11-16T16:48:44.914365+0000","last_became_peered":"2023-11-16T16:48:44.914365+0000","last_unstale":"2023-11-16T16:48:44.915360+0000","last_undegraded":"2023-11-16T16:48:44.915360+0000","last_fullsized":"2023-11-16T16:48:44.915360+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:54:07.544631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579927+0000","last_change":"2023-11-16T16:47:18.184630+0000","last_active":"2023-11-16T16:49:12.579927+0000","last_peered":"2023-11-16T16:49:12.579927+0000","last_clean":"2023-11-16T16:49:12.579927+0000","last_became_active":"2023-11-16T16:47:18.181978+0000","last_became_peered":"2023-11-16T16:47:18.181978+0000","last_unstale":"2023-11-16T16:49:12.579927+0000","last_undegraded":"2023-11-16T16:49:12.579927+0000","last_fullsized":"2023-11-16T16:49:12.579927+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:18:09.046316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628159+0000","last_change":"2023-11-16T16:47:43.635365+0000","last_active":"2023-11-16T16:48:44.628159+0000","last_peered":"2023-11-16T16:48:44.628159+0000","last_clean":"2023-11-16T16:48:44.628159+0000","last_became_active":"2023-11-16T16:47:43.634974+0000","last_became_peered":"2023-11-16T16:47:43.634974+0000","last_unstale":"2023-11-16T16:48:44.628159+0000","last_undegraded":"2023-11-16T16:48:44.628159+0000","last_fullsized":"2023-11-16T16:48:44.628159+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:34:17.420977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.627888+0000","last_change":"2023-11-16T16:47:22.657133+0000","last_active":"2023-11-16T16:48:44.627888+0000","last_peered":"2023-11-16T16:48:44.627888+0000","last_clean":"2023-11-16T16:48:44.627888+0000","last_became_active":"2023-11-16T16:47:22.656557+0000","last_became_peered":"2023-11-16T16:47:22.656557+0000","last_unstale":"2023-11-16T16:48:44.627888+0000","last_undegraded":"2023-11-16T16:48:44.627888+0000","last_fullsized":"2023-11-16T16:48:44.627888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:50:31.162768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.629935+0000","last_change":"2023-11-16T16:47:17.639107+0000","last_active":"2023-11-16T16:48:44.629935+0000","last_peered":"2023-11-16T16:48:44.629935+0000","last_clean":"2023-11-16T16:48:44.629935+0000","last_became_active":"2023-11-16T16:47:17.638796+0000","last_became_peered":"2023-11-16T16:47:17.638796+0000","last_unstale":"2023-11-16T16:48:44.629935+0000","last_undegraded":"2023-11-16T16:48:44.629935+0000","last_fullsized":"2023-11-16T16:48:44.629935+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:55:19.889558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.581882+0000","last_change":"2023-11-16T16:47:18.186761+0000","last_active":"2023-11-16T16:49:12.581882+0000","last_peered":"2023-11-16T16:49:12.581882+0000","last_clean":"2023-11-16T16:49:12.581882+0000","last_became_active":"2023-11-16T16:47:18.185594+0000","last_became_peered":"2023-11-16T16:47:18.185594+0000","last_unstale":"2023-11-16T16:49:12.581882+0000","last_undegraded":"2023-11-16T16:49:12.581882+0000","last_fullsized":"2023-11-16T16:49:12.581882+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:29:09.490006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.912099+0000","last_change":"2023-11-16T16:47:22.719534+0000","last_active":"2023-11-16T16:48:44.912099+0000","last_peered":"2023-11-16T16:48:44.912099+0000","last_clean":"2023-11-16T16:48:44.912099+0000","last_became_active":"2023-11-16T16:47:22.718738+0000","last_became_peered":"2023-11-16T16:47:22.718738+0000","last_unstale":"2023-11-16T16:48:44.912099+0000","last_undegraded":"2023-11-16T16:48:44.912099+0000","last_fullsized":"2023-11-16T16:48:44.912099+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:48:26.395670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'5","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:17.091818+0000","last_change":"2023-11-16T16:47:17.630318+0000","last_active":"2023-11-16T16:49:17.091818+0000","last_peered":"2023-11-16T16:49:17.091818+0000","last_clean":"2023-11-16T16:49:17.091818+0000","last_became_active":"2023-11-16T16:47:17.629035+0000","last_became_peered":"2023-11-16T16:47:17.629035+0000","last_unstale":"2023-11-16T16:49:17.091818+0000","last_undegraded":"2023-11-16T16:49:17.091818+0000","last_fullsized":"2023-11-16T16:49:17.091818+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:34:15.724433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.574220+0000","last_change":"2023-11-16T16:47:18.185034+0000","last_active":"2023-11-16T16:49:12.574220+0000","last_peered":"2023-11-16T16:49:12.574220+0000","last_clean":"2023-11-16T16:49:12.574220+0000","last_became_active":"2023-11-16T16:47:18.181961+0000","last_became_peered":"2023-11-16T16:47:18.181961+0000","last_unstale":"2023-11-16T16:49:12.574220+0000","last_undegraded":"2023-11-16T16:49:12.574220+0000","last_fullsized":"2023-11-16T16:49:12.574220+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:51:00.509275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623131+0000","last_change":"2023-11-16T16:47:22.649952+0000","last_active":"2023-11-16T16:48:44.623131+0000","last_peered":"2023-11-16T16:48:44.623131+0000","last_clean":"2023-11-16T16:48:44.623131+0000","last_became_active":"2023-11-16T16:47:22.649481+0000","last_became_peered":"2023-11-16T16:47:22.649481+0000","last_unstale":"2023-11-16T16:48:44.623131+0000","last_undegraded":"2023-11-16T16:48:44.623131+0000","last_fullsized":"2023-11-16T16:48:44.623131+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:52:18.082684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"82'2","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623572+0000","last_change":"2023-11-16T16:47:17.631982+0000","last_active":"2023-11-16T16:48:44.623572+0000","last_peered":"2023-11-16T16:48:44.623572+0000","last_clean":"2023-11-16T16:48:44.623572+0000","last_became_active":"2023-11-16T16:47:17.631552+0000","last_became_peered":"2023-11-16T16:47:17.631552+0000","last_unstale":"2023-11-16T16:48:44.623572+0000","last_undegraded":"2023-11-16T16:48:44.623572+0000","last_fullsized":"2023-11-16T16:48:44.623572+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:23:15.968583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.589149+0000","last_change":"2023-11-16T16:47:18.197325+0000","last_active":"2023-11-16T16:49:12.589149+0000","last_peered":"2023-11-16T16:49:12.589149+0000","last_clean":"2023-11-16T16:49:12.589149+0000","last_became_active":"2023-11-16T16:47:18.190602+0000","last_became_peered":"2023-11-16T16:47:18.190602+0000","last_unstale":"2023-11-16T16:49:12.589149+0000","last_undegraded":"2023-11-16T16:49:12.589149+0000","last_fullsized":"2023-11-16T16:49:12.589149+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:21:06.387535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623556+0000","last_change":"2023-11-16T16:47:22.720023+0000","last_active":"2023-11-16T16:48:44.623556+0000","last_peered":"2023-11-16T16:48:44.623556+0000","last_clean":"2023-11-16T16:48:44.623556+0000","last_became_active":"2023-11-16T16:47:22.719561+0000","last_became_peered":"2023-11-16T16:47:22.719561+0000","last_unstale":"2023-11-16T16:48:44.623556+0000","last_undegraded":"2023-11-16T16:48:44.623556+0000","last_fullsized":"2023-11-16T16:48:44.623556+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:06:42.918943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.647594+0000","last_change":"2023-11-16T16:47:17.629379+0000","last_active":"2023-11-16T16:48:44.647594+0000","last_peered":"2023-11-16T16:48:44.647594+0000","last_clean":"2023-11-16T16:48:44.647594+0000","last_became_active":"2023-11-16T16:47:17.629025+0000","last_became_peered":"2023-11-16T16:47:17.629025+0000","last_unstale":"2023-11-16T16:48:44.647594+0000","last_undegraded":"2023-11-16T16:48:44.647594+0000","last_fullsized":"2023-11-16T16:48:44.647594+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:38:07.996802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.566023+0000","last_change":"2023-11-16T16:47:18.185418+0000","last_active":"2023-11-16T16:49:12.566023+0000","last_peered":"2023-11-16T16:49:12.566023+0000","last_clean":"2023-11-16T16:49:12.566023+0000","last_became_active":"2023-11-16T16:47:18.185092+0000","last_became_peered":"2023-11-16T16:47:18.185092+0000","last_unstale":"2023-11-16T16:49:12.566023+0000","last_undegraded":"2023-11-16T16:49:12.566023+0000","last_fullsized":"2023-11-16T16:49:12.566023+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:57:19.899302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.913330+0000","last_change":"2023-11-16T16:48:44.913330+0000","last_active":"2023-11-16T16:48:44.913330+0000","last_peered":"2023-11-16T16:48:44.913330+0000","last_clean":"2023-11-16T16:48:44.913330+0000","last_became_active":"2023-11-16T16:48:44.912707+0000","last_became_peered":"2023-11-16T16:48:44.912707+0000","last_unstale":"2023-11-16T16:48:44.913330+0000","last_undegraded":"2023-11-16T16:48:44.913330+0000","last_fullsized":"2023-11-16T16:48:44.913330+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:12:18.512977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910905+0000","last_change":"2023-11-16T16:47:17.638622+0000","last_active":"2023-11-16T16:48:44.910905+0000","last_peered":"2023-11-16T16:48:44.910905+0000","last_clean":"2023-11-16T16:48:44.910905+0000","last_became_active":"2023-11-16T16:47:17.638303+0000","last_became_peered":"2023-11-16T16:47:17.638303+0000","last_unstale":"2023-11-16T16:48:44.910905+0000","last_undegraded":"2023-11-16T16:48:44.910905+0000","last_fullsized":"2023-11-16T16:48:44.910905+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:21:41.090636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.590794+0000","last_change":"2023-11-16T16:47:18.173474+0000","last_active":"2023-11-16T16:49:12.590794+0000","last_peered":"2023-11-16T16:49:12.590794+0000","last_clean":"2023-11-16T16:49:12.590794+0000","last_became_active":"2023-11-16T16:47:18.173140+0000","last_became_peered":"2023-11-16T16:47:18.173140+0000","last_unstale":"2023-11-16T16:49:12.590794+0000","last_undegraded":"2023-11-16T16:49:12.590794+0000","last_fullsized":"2023-11-16T16:49:12.590794+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:30:41.318669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623128+0000","last_change":"2023-11-16T16:47:22.655806+0000","last_active":"2023-11-16T16:48:44.623128+0000","last_peered":"2023-11-16T16:48:44.623128+0000","last_clean":"2023-11-16T16:48:44.623128+0000","last_became_active":"2023-11-16T16:47:22.655385+0000","last_became_peered":"2023-11-16T16:47:22.655385+0000","last_unstale":"2023-11-16T16:48:44.623128+0000","last_undegraded":"2023-11-16T16:48:44.623128+0000","last_fullsized":"2023-11-16T16:48:44.623128+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:15:10.616521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623263+0000","last_change":"2023-11-16T16:47:17.631673+0000","last_active":"2023-11-16T16:48:44.623263+0000","last_peered":"2023-11-16T16:48:44.623263+0000","last_clean":"2023-11-16T16:48:44.623263+0000","last_became_active":"2023-11-16T16:47:17.631356+0000","last_became_peered":"2023-11-16T16:47:17.631356+0000","last_unstale":"2023-11-16T16:48:44.623263+0000","last_undegraded":"2023-11-16T16:48:44.623263+0000","last_fullsized":"2023-11-16T16:48:44.623263+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:05:33.004512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.589316+0000","last_change":"2023-11-16T16:47:18.186706+0000","last_active":"2023-11-16T16:49:12.589316+0000","last_peered":"2023-11-16T16:49:12.589316+0000","last_clean":"2023-11-16T16:49:12.589316+0000","last_became_active":"2023-11-16T16:47:18.186360+0000","last_became_peered":"2023-11-16T16:47:18.186360+0000","last_unstale":"2023-11-16T16:49:12.589316+0000","last_undegraded":"2023-11-16T16:49:12.589316+0000","last_fullsized":"2023-11-16T16:49:12.589316+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:46:42.071656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623436+0000","last_change":"2023-11-16T16:47:22.719065+0000","last_active":"2023-11-16T16:48:44.623436+0000","last_peered":"2023-11-16T16:48:44.623436+0000","last_clean":"2023-11-16T16:48:44.623436+0000","last_became_active":"2023-11-16T16:47:22.718662+0000","last_became_peered":"2023-11-16T16:47:22.718662+0000","last_unstale":"2023-11-16T16:48:44.623436+0000","last_undegraded":"2023-11-16T16:48:44.623436+0000","last_fullsized":"2023-11-16T16:48:44.623436+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:39:06.356934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"82'2","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628612+0000","last_change":"2023-11-16T16:47:43.637790+0000","last_active":"2023-11-16T16:48:44.628612+0000","last_peered":"2023-11-16T16:48:44.628612+0000","last_clean":"2023-11-16T16:48:44.628612+0000","last_became_active":"2023-11-16T16:47:43.632225+0000","last_became_peered":"2023-11-16T16:47:43.632225+0000","last_unstale":"2023-11-16T16:48:44.628612+0000","last_undegraded":"2023-11-16T16:48:44.628612+0000","last_fullsized":"2023-11-16T16:48:44.628612+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:37:02.441726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.584025+0000","last_change":"2023-11-16T16:47:18.180999+0000","last_active":"2023-11-16T16:49:12.584025+0000","last_peered":"2023-11-16T16:49:12.584025+0000","last_clean":"2023-11-16T16:49:12.584025+0000","last_became_active":"2023-11-16T16:47:18.180724+0000","last_became_peered":"2023-11-16T16:47:18.180724+0000","last_unstale":"2023-11-16T16:49:12.584025+0000","last_undegraded":"2023-11-16T16:49:12.584025+0000","last_fullsized":"2023-11-16T16:49:12.584025+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:21:21.954717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.647762+0000","last_change":"2023-11-16T16:47:22.665607+0000","last_active":"2023-11-16T16:48:44.647762+0000","last_peered":"2023-11-16T16:48:44.647762+0000","last_clean":"2023-11-16T16:48:44.647762+0000","last_became_active":"2023-11-16T16:47:22.665273+0000","last_became_peered":"2023-11-16T16:47:22.665273+0000","last_unstale":"2023-11-16T16:48:44.647762+0000","last_undegraded":"2023-11-16T16:48:44.647762+0000","last_fullsized":"2023-11-16T16:48:44.647762+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:31:43.368003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623867+0000","last_change":"2023-11-16T16:47:17.632614+0000","last_active":"2023-11-16T16:48:44.623867+0000","last_peered":"2023-11-16T16:48:44.623867+0000","last_clean":"2023-11-16T16:48:44.623867+0000","last_became_active":"2023-11-16T16:47:17.632227+0000","last_became_peered":"2023-11-16T16:47:17.632227+0000","last_unstale":"2023-11-16T16:48:44.623867+0000","last_undegraded":"2023-11-16T16:48:44.623867+0000","last_fullsized":"2023-11-16T16:48:44.623867+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:52:45.369452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579477+0000","last_change":"2023-11-16T16:47:18.177607+0000","last_active":"2023-11-16T16:49:12.579477+0000","last_peered":"2023-11-16T16:49:12.579477+0000","last_clean":"2023-11-16T16:49:12.579477+0000","last_became_active":"2023-11-16T16:47:18.177266+0000","last_became_peered":"2023-11-16T16:47:18.177266+0000","last_unstale":"2023-11-16T16:49:12.579477+0000","last_undegraded":"2023-11-16T16:49:12.579477+0000","last_fullsized":"2023-11-16T16:49:12.579477+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:45:38.139269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910928+0000","last_change":"2023-11-16T16:47:22.659987+0000","last_active":"2023-11-16T16:48:44.910928+0000","last_peered":"2023-11-16T16:48:44.910928+0000","last_clean":"2023-11-16T16:48:44.910928+0000","last_became_active":"2023-11-16T16:47:22.658725+0000","last_became_peered":"2023-11-16T16:47:22.658725+0000","last_unstale":"2023-11-16T16:48:44.910928+0000","last_undegraded":"2023-11-16T16:48:44.910928+0000","last_fullsized":"2023-11-16T16:48:44.910928+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:09:57.462778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911469+0000","last_change":"2023-11-16T16:47:17.640696+0000","last_active":"2023-11-16T16:48:44.911469+0000","last_peered":"2023-11-16T16:48:44.911469+0000","last_clean":"2023-11-16T16:48:44.911469+0000","last_became_active":"2023-11-16T16:47:17.640428+0000","last_became_peered":"2023-11-16T16:47:17.640428+0000","last_unstale":"2023-11-16T16:48:44.911469+0000","last_undegraded":"2023-11-16T16:48:44.911469+0000","last_fullsized":"2023-11-16T16:48:44.911469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:06:54.539557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.583537+0000","last_change":"2023-11-16T16:47:43.972273+0000","last_active":"2023-11-16T16:49:12.583537+0000","last_peered":"2023-11-16T16:49:12.583537+0000","last_clean":"2023-11-16T16:49:12.583537+0000","last_became_active":"2023-11-16T16:47:43.971981+0000","last_became_peered":"2023-11-16T16:47:43.971981+0000","last_unstale":"2023-11-16T16:49:12.583537+0000","last_undegraded":"2023-11-16T16:49:12.583537+0000","last_fullsized":"2023-11-16T16:49:12.583537+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:07:07.722793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910594+0000","last_change":"2023-11-16T16:47:22.649436+0000","last_active":"2023-11-16T16:48:44.910594+0000","last_peered":"2023-11-16T16:48:44.910594+0000","last_clean":"2023-11-16T16:48:44.910594+0000","last_became_active":"2023-11-16T16:47:22.649034+0000","last_became_peered":"2023-11-16T16:47:22.649034+0000","last_unstale":"2023-11-16T16:48:44.910594+0000","last_undegraded":"2023-11-16T16:48:44.910594+0000","last_fullsized":"2023-11-16T16:48:44.910594+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:46:22.957484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.912764+0000","last_change":"2023-11-16T16:47:17.634582+0000","last_active":"2023-11-16T16:48:44.912764+0000","last_peered":"2023-11-16T16:48:44.912764+0000","last_clean":"2023-11-16T16:48:44.912764+0000","last_became_active":"2023-11-16T16:47:17.634172+0000","last_became_peered":"2023-11-16T16:47:17.634172+0000","last_unstale":"2023-11-16T16:48:44.912764+0000","last_undegraded":"2023-11-16T16:48:44.912764+0000","last_fullsized":"2023-11-16T16:48:44.912764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:43:12.528888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.569892+0000","last_change":"2023-11-16T16:47:18.180466+0000","last_active":"2023-11-16T16:49:12.569892+0000","last_peered":"2023-11-16T16:49:12.569892+0000","last_clean":"2023-11-16T16:49:12.569892+0000","last_became_active":"2023-11-16T16:47:18.180166+0000","last_became_peered":"2023-11-16T16:47:18.180166+0000","last_unstale":"2023-11-16T16:49:12.569892+0000","last_undegraded":"2023-11-16T16:49:12.569892+0000","last_fullsized":"2023-11-16T16:49:12.569892+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:57:25.704442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.570479+0000","last_change":"2023-11-16T16:47:18.179196+0000","last_active":"2023-11-16T16:49:12.570479+0000","last_peered":"2023-11-16T16:49:12.570479+0000","last_clean":"2023-11-16T16:49:12.570479+0000","last_became_active":"2023-11-16T16:47:18.178827+0000","last_became_peered":"2023-11-16T16:47:18.178827+0000","last_unstale":"2023-11-16T16:49:12.570479+0000","last_undegraded":"2023-11-16T16:49:12.570479+0000","last_fullsized":"2023-11-16T16:49:12.570479+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:21:55.520738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.578735+0000","last_change":"2023-11-16T16:47:18.184630+0000","last_active":"2023-11-16T16:49:12.578735+0000","last_peered":"2023-11-16T16:49:12.578735+0000","last_clean":"2023-11-16T16:49:12.578735+0000","last_became_active":"2023-11-16T16:47:18.184314+0000","last_became_peered":"2023-11-16T16:47:18.184314+0000","last_unstale":"2023-11-16T16:49:12.578735+0000","last_undegraded":"2023-11-16T16:49:12.578735+0000","last_fullsized":"2023-11-16T16:49:12.578735+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:20:06.066404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.581521+0000","last_change":"2023-11-16T16:47:18.184255+0000","last_active":"2023-11-16T16:49:12.581521+0000","last_peered":"2023-11-16T16:49:12.581521+0000","last_clean":"2023-11-16T16:49:12.581521+0000","last_became_active":"2023-11-16T16:47:18.183936+0000","last_became_peered":"2023-11-16T16:47:18.183936+0000","last_unstale":"2023-11-16T16:49:12.581521+0000","last_undegraded":"2023-11-16T16:49:12.581521+0000","last_fullsized":"2023-11-16T16:49:12.581521+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:40:04.120978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.588698+0000","last_change":"2023-11-16T16:47:18.196331+0000","last_active":"2023-11-16T16:49:12.588698+0000","last_peered":"2023-11-16T16:49:12.588698+0000","last_clean":"2023-11-16T16:49:12.588698+0000","last_became_active":"2023-11-16T16:47:18.195954+0000","last_became_peered":"2023-11-16T16:47:18.195954+0000","last_unstale":"2023-11-16T16:49:12.588698+0000","last_undegraded":"2023-11-16T16:49:12.588698+0000","last_fullsized":"2023-11-16T16:49:12.588698+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:48:28.247082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.591835+0000","last_change":"2023-11-16T16:47:18.190537+0000","last_active":"2023-11-16T16:49:12.591835+0000","last_peered":"2023-11-16T16:49:12.591835+0000","last_clean":"2023-11-16T16:49:12.591835+0000","last_became_active":"2023-11-16T16:47:18.190118+0000","last_became_peered":"2023-11-16T16:47:18.190118+0000","last_unstale":"2023-11-16T16:49:12.591835+0000","last_undegraded":"2023-11-16T16:49:12.591835+0000","last_fullsized":"2023-11-16T16:49:12.591835+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:53:21.438815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.566299+0000","last_change":"2023-11-16T16:47:18.179215+0000","last_active":"2023-11-16T16:49:12.566299+0000","last_peered":"2023-11-16T16:49:12.566299+0000","last_clean":"2023-11-16T16:49:12.566299+0000","last_became_active":"2023-11-16T16:47:18.178759+0000","last_became_peered":"2023-11-16T16:47:18.178759+0000","last_unstale":"2023-11-16T16:49:12.566299+0000","last_undegraded":"2023-11-16T16:49:12.566299+0000","last_fullsized":"2023-11-16T16:49:12.566299+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:56:47.424865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.581180+0000","last_change":"2023-11-16T16:47:18.179588+0000","last_active":"2023-11-16T16:49:12.581180+0000","last_peered":"2023-11-16T16:49:12.581180+0000","last_clean":"2023-11-16T16:49:12.581180+0000","last_became_active":"2023-11-16T16:47:18.179294+0000","last_became_peered":"2023-11-16T16:47:18.179294+0000","last_unstale":"2023-11-16T16:49:12.581180+0000","last_undegraded":"2023-11-16T16:49:12.581180+0000","last_fullsized":"2023-11-16T16:49:12.581180+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:21:17.462988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.572426+0000","last_change":"2023-11-16T16:47:18.181587+0000","last_active":"2023-11-16T16:49:12.572426+0000","last_peered":"2023-11-16T16:49:12.572426+0000","last_clean":"2023-11-16T16:49:12.572426+0000","last_became_active":"2023-11-16T16:47:18.181203+0000","last_became_peered":"2023-11-16T16:47:18.181203+0000","last_unstale":"2023-11-16T16:49:12.572426+0000","last_undegraded":"2023-11-16T16:49:12.572426+0000","last_fullsized":"2023-11-16T16:49:12.572426+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:41:14.468221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579568+0000","last_change":"2023-11-16T16:47:18.182690+0000","last_active":"2023-11-16T16:49:12.579568+0000","last_peered":"2023-11-16T16:49:12.579568+0000","last_clean":"2023-11-16T16:49:12.579568+0000","last_became_active":"2023-11-16T16:47:18.182361+0000","last_became_peered":"2023-11-16T16:47:18.182361+0000","last_unstale":"2023-11-16T16:49:12.579568+0000","last_undegraded":"2023-11-16T16:49:12.579568+0000","last_fullsized":"2023-11-16T16:49:12.579568+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:24:27.460239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.578706+0000","last_change":"2023-11-16T16:47:18.184599+0000","last_active":"2023-11-16T16:49:12.578706+0000","last_peered":"2023-11-16T16:49:12.578706+0000","last_clean":"2023-11-16T16:49:12.578706+0000","last_became_active":"2023-11-16T16:47:18.184288+0000","last_became_peered":"2023-11-16T16:47:18.184288+0000","last_unstale":"2023-11-16T16:49:12.578706+0000","last_undegraded":"2023-11-16T16:49:12.578706+0000","last_fullsized":"2023-11-16T16:49:12.578706+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:09:35.265387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.575511+0000","last_change":"2023-11-16T16:47:18.187267+0000","last_active":"2023-11-16T16:49:12.575511+0000","last_peered":"2023-11-16T16:49:12.575511+0000","last_clean":"2023-11-16T16:49:12.575511+0000","last_became_active":"2023-11-16T16:47:18.186984+0000","last_became_peered":"2023-11-16T16:47:18.186984+0000","last_unstale":"2023-11-16T16:49:12.575511+0000","last_undegraded":"2023-11-16T16:49:12.575511+0000","last_fullsized":"2023-11-16T16:49:12.575511+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:22:44.826690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.592202+0000","last_change":"2023-11-16T16:47:18.191566+0000","last_active":"2023-11-16T16:49:12.592202+0000","last_peered":"2023-11-16T16:49:12.592202+0000","last_clean":"2023-11-16T16:49:12.592202+0000","last_became_active":"2023-11-16T16:47:18.185342+0000","last_became_peered":"2023-11-16T16:47:18.185342+0000","last_unstale":"2023-11-16T16:49:12.592202+0000","last_undegraded":"2023-11-16T16:49:12.592202+0000","last_fullsized":"2023-11-16T16:49:12.592202+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:53:50.072132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.565174+0000","last_change":"2023-11-16T16:47:43.629636+0000","last_active":"2023-11-16T16:49:12.565174+0000","last_peered":"2023-11-16T16:49:12.565174+0000","last_clean":"2023-11-16T16:49:12.565174+0000","last_became_active":"2023-11-16T16:47:43.629179+0000","last_became_peered":"2023-11-16T16:47:43.629179+0000","last_unstale":"2023-11-16T16:49:12.565174+0000","last_undegraded":"2023-11-16T16:49:12.565174+0000","last_fullsized":"2023-11-16T16:49:12.565174+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:20:45.082147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281399,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67650,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":239,"num_read_kb":190,"num_write":84,"num_write_kb":99,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":197571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449130,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6228,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738924,"statfs":{"total":95995035648,"available":95988658176,"internally_reserved":0,"allocated":995328,"data_stored":506898,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.41999999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.77400000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.57499999999999996,"15min":0.55700000000000005},"min":{"1min":0.46200000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79900000000000004,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.60799999999999998}]},{"osd":1,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59199999999999997,"15min":0.58699999999999997},"min":{"1min":0.505,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74099999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.50900000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.751,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.72799999999999998}]},{"osd":2,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.47499999999999998,"15min":0.46100000000000002},"min":{"1min":0.36799999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.47899999999999998,"15min":0.46999999999999997},"min":{"1min":0.39100000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.57299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.63500000000000001,"15min":0.627},"min":{"1min":0.52400000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.625,"15min":0.626},"min":{"1min":0.46100000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.41399999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.79500000000000004,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.38600000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.89200000000000002,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.46899999999999997}]},{"osd":6,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.67800000000000005,"15min":0.68999999999999995},"min":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.84299999999999997,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.623},{"interface":"front","average":{"1min":0.754,"5min":0.76500000000000001,"15min":0.73299999999999998},"min":{"1min":0.54300000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.93000000000000005,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":0.68400000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.56399999999999995,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95599999999999996,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.73299999999999998,"15min":0.73599999999999999},"min":{"1min":0.56499999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.95699999999999996,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.69499999999999995}]},{"osd":8,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.72299999999999998,"15min":0.71699999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.93000000000000005,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.70499999999999996}]},{"osd":9,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.78500000000000003,"15min":0.76900000000000002},"min":{"1min":0.69999999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.042,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.98599999999999999,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.752}]},{"osd":10,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.76700000000000002,"15min":0.76900000000000002},"min":{"1min":0.59099999999999997,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.94999999999999996,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.78200000000000003,"15min":0.77000000000000002},"min":{"1min":0.69799999999999995,"5min":0.498,"15min":0.498},"max":{"1min":1.006,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.80200000000000005}]}]},{"osd":10,"up_from":71,"seq":304942678062,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6676,"kb_used_data":1100,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738476,"statfs":{"total":95995035648,"available":95988199424,"internally_reserved":0,"allocated":1126400,"data_stored":1160194,"data_compressed":59395,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:49:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.72899999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72799999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.26900000000000002}]},{"osd":1,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.26200000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60299999999999998,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.49099999999999999,"15min":0.51100000000000001},"min":{"1min":0.28299999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70799999999999996,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.247}]},{"osd":2,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.63400000000000001,"15min":0.66100000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.70499999999999996,"5min":1.577,"15min":1.577},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.63800000000000001,"15min":0.65800000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78300000000000003,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.29799999999999999}]},{"osd":3,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.623,"15min":0.64500000000000002},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80000000000000004,"5min":1.464,"15min":1.464},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.65400000000000003,"15min":0.67500000000000004},"min":{"1min":0.40400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76800000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.374}]},{"osd":4,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56399999999999995,"15min":0.58399999999999996},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73899999999999999,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54700000000000004,"15min":0.56799999999999995},"min":{"1min":0.372,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.752,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.30399999999999999}]},{"osd":5,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.68300000000000005,"15min":0.69899999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82899999999999996,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.68700000000000006,"15min":0.69899999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.91300000000000003,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.37}]},{"osd":6,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.73399999999999999,"15min":0.75900000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86299999999999999,"5min":1.524,"15min":1.524},"last":0.432},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76000000000000001,"15min":0.77400000000000002},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.85999999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.5}]},{"osd":7,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.69999999999999996}]},{"osd":8,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.81299999999999994,"15min":0.85499999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.91400000000000003,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.755,"15min":0.78300000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.93999999999999995,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.438}]},{"osd":9,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.84399999999999997,"15min":0.84099999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.92700000000000005,"5min":4.0129999999999999,"15min":4.0129999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.82199999999999995,"15min":0.83099999999999996},"min":{"1min":0.56299999999999994,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":4.5380000000000003,"5min":4.5380000000000003,"15min":4.5380000000000003},"last":0.46300000000000002}]},{"osd":11,"last update":"Thu Nov 16 16:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.78400000000000003,"15min":0.81799999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.96599999999999997,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.79100000000000004,"15min":0.80700000000000005},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.95599999999999996,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.48099999999999998}]}]},{"osd":9,"up_from":66,"seq":283467841585,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6424,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738728,"statfs":{"total":95995035648,"available":95988457472,"internally_reserved":0,"allocated":999424,"data_stored":511554,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.54400000000000004,"15min":0.56999999999999995},"min":{"1min":0.28399999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.57199999999999995,"15min":0.60399999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.40300000000000002}]},{"osd":1,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.78600000000000003,"15min":0.82499999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.89500000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.747,"15min":0.78200000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.53700000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.80100000000000005,"15min":0.83299999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0640000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.753,"15min":0.77700000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59099999999999997,"15min":0.61499999999999999},"min":{"1min":0.40899999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.746,"15min":0.80000000000000004},"min":{"1min":0.42299999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.2210000000000001,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.61099999999999999}]},{"osd":4,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.79600000000000004,"15min":0.85099999999999998},"min":{"1min":0.48799999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.167,"5min":1.298,"15min":1.298},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.73699999999999999,"15min":0.77300000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.94199999999999995,"5min":1.181,"15min":1.181},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.78800000000000003,"15min":0.82399999999999995},"min":{"1min":0.51200000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.081,"5min":1.268,"15min":1.268},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.90000000000000002,"15min":0.95299999999999996},"min":{"1min":0.621,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1040000000000001,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.76700000000000002}]},{"osd":6,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":1.054},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.95599999999999996}]},{"osd":7,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.84899999999999998,"15min":0.90300000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.99199999999999999,"5min":1.405,"15min":1.405},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.98099999999999998,"15min":1.0589999999999999},"min":{"1min":0.59799999999999998,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.133,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.748}]},{"osd":8,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.90200000000000002,"15min":0.96099999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.1970000000000001,"5min":1.363,"15min":1.363},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.90400000000000003,"15min":0.96399999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.149,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.68700000000000006}]},{"osd":10,"last update":"Thu Nov 16 16:49:15 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.88800000000000001,"15min":0.93899999999999995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.036,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.96499999999999997,"15min":1.0209999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1599999999999999,"5min":1.714,"15min":1.714},"last":0.79600000000000004}]},{"osd":11,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.80300000000000005,"15min":0.80800000000000005},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.038,"5min":1.115,"15min":1.115},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.89100000000000001,"15min":0.90600000000000003},"min":{"1min":0.67800000000000005,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.0600000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.82099999999999995}]}]},{"osd":8,"up_from":58,"seq":249108103223,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6548,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738604,"statfs":{"total":95995035648,"available":95988330496,"internally_reserved":0,"allocated":995328,"data_stored":507458,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.34300000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.79600000000000004,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.39200000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.86299999999999999,"5min":1.113,"15min":1.113},"last":0.436}]},{"osd":1,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63200000000000001,"15min":0.60399999999999998},"min":{"1min":0.51600000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76600000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.623,"15min":0.60999999999999999},"min":{"1min":0.40999999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.85099999999999998,"5min":1.177,"15min":1.177},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47299999999999998,"15min":0.45700000000000002},"min":{"1min":0.29699999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.73499999999999999,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47499999999999998,"15min":0.46600000000000003},"min":{"1min":0.38700000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.40000000000000002}]},{"osd":3,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.629,"15min":0.63},"min":{"1min":0.42799999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.90600000000000003,"5min":1.141,"15min":1.141},"last":0.496},{"interface":"front","average":{"1min":0.623,"5min":0.65500000000000003,"15min":0.64700000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.91800000000000004,"5min":1.502,"15min":1.502},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.64100000000000001,"15min":0.629},"min":{"1min":0.50700000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79800000000000004,"5min":0.999,"15min":0.999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69799999999999995,"15min":0.68799999999999994},"min":{"1min":0.54300000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83199999999999996,"5min":1.242,"15min":1.242},"last":0.48099999999999998}]},{"osd":5,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55800000000000005,"15min":0.53600000000000003},"min":{"1min":0.45900000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83699999999999997,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57599999999999996,"15min":0.56299999999999994},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68000000000000005,"5min":1.224,"15min":1.224},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.77600000000000002,"15min":0.73799999999999999},"min":{"1min":0.54900000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.89000000000000001,"5min":4.7309999999999999,"15min":4.7309999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.69599999999999995,"15min":0.68100000000000005},"min":{"1min":0.54200000000000004,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0149999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.56200000000000006}]},{"osd":7,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.71599999999999997,"15min":0.69499999999999995},"min":{"1min":0.60499999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90300000000000002,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.73899999999999999,"15min":0.72599999999999998},"min":{"1min":0.59699999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95599999999999996,"5min":1.198,"15min":1.198},"last":0.622}]},{"osd":9,"last update":"Thu Nov 16 16:48:51 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.76900000000000002,"15min":0.77000000000000002},"min":{"1min":0.52700000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.89100000000000001,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.80300000000000005,"15min":0.81100000000000005},"min":{"1min":0.60299999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.88500000000000001,"5min":1.298,"15min":1.298},"last":0.65400000000000003}]},{"osd":10,"last update":"Thu Nov 16 16:49:11 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78500000000000003,"15min":0.77700000000000002},"min":{"1min":0.70199999999999996,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.97699999999999998,"5min":1.222,"15min":1.222},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.81599999999999995,"15min":0.81999999999999995},"min":{"1min":0.66100000000000003,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.91900000000000004,"5min":1.274,"15min":1.274},"last":0.52200000000000002}]},{"osd":11,"last update":"Thu Nov 16 16:49:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.77300000000000002,"15min":0.77500000000000002},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.94499999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.77800000000000002,"15min":0.78400000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.875,"5min":1.05,"15min":1.05},"last":0.64400000000000002}]}]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6664,"kb_used_data":1024,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738488,"statfs":{"total":95995035648,"available":95988211712,"internally_reserved":0,"allocated":1048576,"data_stored":1101922,"data_compressed":59395,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.377,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81799999999999995,"5min":0.875,"15min":0.875},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.42899999999999999,"15min":0.41099999999999998},"min":{"1min":0.28499999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59499999999999997,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.45100000000000001,"15min":0.441},"min":{"1min":0.29899999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.628,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.60299999999999998}]},{"osd":2,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.58099999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.76600000000000001,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.61199999999999999,"15min":0.58599999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81799999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.629}]},{"osd":3,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.58799999999999997,"15min":0.55900000000000005},"min":{"1min":0.46999999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.44,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.76600000000000001}]},{"osd":4,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.375,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67900000000000005,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.51900000000000002,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.69799999999999995,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.48799999999999999}]},{"osd":5,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64900000000000002,"15min":0.65100000000000002},"min":{"1min":0.45500000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81899999999999995,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.621},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.66500000000000004,"15min":0.63900000000000001},"min":{"1min":0.45700000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.88100000000000001,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.73099999999999998}]},{"osd":6,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.69599999999999995,"15min":0.68000000000000005},"min":{"1min":0.50700000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46999999999999997,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.93500000000000005,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.64800000000000002}]},{"osd":8,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.72099999999999997,"15min":0.68500000000000005},"min":{"1min":0.64200000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.92900000000000005,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.70699999999999996,"15min":0.70399999999999996},"min":{"1min":0.58499999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.95299999999999996,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.67600000000000005}]},{"osd":9,"last update":"Thu Nov 16 16:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.75600000000000001,"15min":0.77900000000000003},"min":{"1min":0.53800000000000003,"5min":0.442,"15min":0.442},"max":{"1min":0.88900000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78400000000000003,"15min":0.80300000000000005},"min":{"1min":0.61599999999999999,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.94999999999999996,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.88}]},{"osd":10,"last update":"Thu Nov 16 16:49:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.72699999999999998,"15min":0.75},"min":{"1min":0.48799999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94199999999999995,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.76800000000000002,"15min":0.78500000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93500000000000005,"5min":1.01,"15min":1.01},"last":0.83399999999999996}]},{"osd":11,"last update":"Thu Nov 16 16:49:23 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81200000000000006,"15min":0.81000000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.76400000000000001,"15min":0.76100000000000001},"min":{"1min":0.56799999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.71999999999999997}]}]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6540,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738612,"statfs":{"total":95995035648,"available":95988338688,"internally_reserved":0,"allocated":987136,"data_stored":502780,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.752},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.375,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.96499999999999997}]},{"osd":1,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.66900000000000004,"15min":0.65500000000000003},"min":{"1min":0.42899999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.68999999999999995,"15min":0.70199999999999996},"min":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0569999999999999,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72699999999999998,"15min":0.71099999999999997},"min":{"1min":0.50800000000000001,"5min":0.436,"15min":0.436},"max":{"1min":0.92800000000000005,"5min":1.407,"15min":1.407},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.72399999999999998},"min":{"1min":0.505,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0189999999999999,"5min":1.171,"15min":1.171},"last":0.61499999999999999}]},{"osd":3,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.71799999999999997,"15min":0.73599999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.755,"15min":0.73499999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.1020000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.74399999999999999,"15min":0.75600000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.77800000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.79500000000000004,"15min":0.80100000000000005},"min":{"1min":0.52300000000000002,"5min":0.503,"15min":0.503},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.873,"15min":0.89300000000000002},"min":{"1min":0.57899999999999996,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.083,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.90800000000000003}]},{"osd":7,"last update":"Thu Nov 16 16:49:02 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.82199999999999995,"15min":0.79900000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.1080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.78500000000000003,"15min":0.755},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0780000000000001,"5min":1.256,"15min":1.256},"last":0.80100000000000005}]},{"osd":8,"last update":"Thu Nov 16 16:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.83099999999999996,"15min":0.84199999999999997},"min":{"1min":0.55200000000000005,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.998,"5min":1.284,"15min":1.284},"last":0.629},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.89000000000000001,"15min":0.88800000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0580000000000001,"5min":1.823,"15min":1.823},"last":0.92700000000000005}]},{"osd":9,"last update":"Thu Nov 16 16:48:51 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.94499999999999995,"15min":0.96099999999999997},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.208,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.83999999999999997,"15min":0.83699999999999997},"min":{"1min":0.623,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.82299999999999995}]},{"osd":10,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.89500000000000002,"15min":0.92200000000000004},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.2370000000000001,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.871,"5min":0.96499999999999997,"15min":0.97799999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.02,"5min":1.393,"15min":1.393},"last":1.0029999999999999}]},{"osd":11,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.022,"15min":1.036},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.91900000000000004,"15min":0.93500000000000005},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":0.98099999999999998,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.874}]}]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7632,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737520,"statfs":{"total":95995035648,"available":95987220480,"internally_reserved":0,"allocated":1056768,"data_stored":564253,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:52 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.52900000000000003},"min":{"1min":0.47899999999999998,"5min":0.32900000000000001,"15min":0.317},"max":{"1min":0.91900000000000004,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.56000000000000005,"15min":0.52100000000000002},"min":{"1min":0.36199999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.82599999999999996,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:49:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.58499999999999996,"15min":0.54500000000000004},"min":{"1min":0.46500000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.83299999999999996,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.57999999999999996,"15min":0.52200000000000002},"min":{"1min":0.48099999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.84099999999999997,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.71599999999999997}]},{"osd":3,"last update":"Thu Nov 16 16:48:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56599999999999995,"15min":0.53900000000000003},"min":{"1min":0.44800000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.93100000000000005,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.621,"15min":0.58299999999999996},"min":{"1min":0.495,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.61399999999999999}]},{"osd":4,"last update":"Thu Nov 16 16:48:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.45700000000000002,"15min":0.42699999999999999},"min":{"1min":0.34799999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.72999999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.46700000000000003,"15min":0.442},"min":{"1min":0.37,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76900000000000002,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.371}]},{"osd":5,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.63700000000000001,"15min":0.58499999999999996},"min":{"1min":0.57399999999999995,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.876,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.65700000000000003,"15min":0.61599999999999999},"min":{"1min":0.54400000000000004,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90100000000000002,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.75900000000000001}]},{"osd":6,"last update":"Thu Nov 16 16:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.66900000000000004,"15min":0.64600000000000002},"min":{"1min":0.51400000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.68000000000000005,"15min":0.65000000000000002},"min":{"1min":0.59499999999999997,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.87,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.65400000000000003}]},{"osd":7,"last update":"Thu Nov 16 16:49:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.57599999999999996,"15min":0.52900000000000003},"min":{"1min":0.38500000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.61199999999999999,"15min":0.56799999999999995},"min":{"1min":0.41899999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.80300000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.74399999999999999}]},{"osd":8,"last update":"Thu Nov 16 16:49:28 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.70999999999999996,"15min":0.65400000000000003},"min":{"1min":0.50700000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.90600000000000003,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.71999999999999997,"15min":0.66600000000000004},"min":{"1min":0.57399999999999995,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.97799999999999998,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.79900000000000004}]},{"osd":9,"last update":"Thu Nov 16 16:48:58 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.749,"15min":0.73199999999999998},"min":{"1min":0.54000000000000004,"5min":0.5,"15min":0.5},"max":{"1min":0.98099999999999998,"5min":1.276,"15min":1.276},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.76300000000000001,"15min":0.751},"min":{"1min":0.63800000000000001,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.92600000000000005,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.70399999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:49:11 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.77700000000000002,"15min":0.78100000000000003},"min":{"1min":0.65700000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.91100000000000003,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.78500000000000003,"15min":0.78300000000000003},"min":{"1min":0.58299999999999996,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.81299999999999994}]},{"osd":11,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85599999999999998,"15min":0.85999999999999999},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":0.96799999999999997,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.88800000000000001,"15min":0.89200000000000002},"min":{"1min":0.73299999999999998,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.0069999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.84099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7440,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737712,"statfs":{"total":95995035648,"available":95987417088,"internally_reserved":0,"allocated":991232,"data_stored":502870,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 16:48:53 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.61299999999999999,"15min":0.55700000000000005},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66000000000000003,"15min":0.57999999999999996},"min":{"1min":0.432,"5min":0.316,"15min":0.316},"max":{"1min":1.7789999999999999,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.61399999999999999}]},{"osd":2,"last update":"Thu Nov 16 16:49:12 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.75900000000000001,"15min":0.621},"min":{"1min":0.57199999999999995,"5min":0.40600000000000003,"15min":0.372},"max":{"1min":2.0529999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.70399999999999996,"15min":0.63},"min":{"1min":0.53000000000000003,"5min":0.41799999999999998,"15min":0.35099999999999998},"max":{"1min":2.1000000000000001,"5min":2.1000000000000001,"15min":2.1000000000000001},"last":0.66500000000000004}]},{"osd":3,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83499999999999996,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.63300000000000001,"15min":0.63500000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.901,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.46000000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:49:04 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.71299999999999997,"15min":0.61099999999999999},"min":{"1min":0.59899999999999998,"5min":0.434,"15min":0.434},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.748,"15min":0.78800000000000003},"min":{"1min":0.53700000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.016,"5min":5.4409999999999998,"15min":5.4409999999999998},"last":0.64900000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.79200000000000004,"15min":0.71999999999999997},"min":{"1min":0.55000000000000004,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.9390000000000001,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.74399999999999999,"15min":0.72399999999999998},"min":{"1min":0.50900000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.97599999999999998,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":0.55900000000000005}]},{"osd":6,"last update":"Thu Nov 16 16:48:48 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.73199999999999998,"15min":0.69599999999999995},"min":{"1min":0.51000000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.76100000000000001,"15min":0.68799999999999994},"min":{"1min":0.622,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":0.72499999999999998}]},{"osd":7,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.86699999999999999,"15min":0.872},"min":{"1min":0.59699999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.054,"5min":1.8089999999999999,"15min":1.8089999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.87,"15min":0.88300000000000001},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2330000000000001,"5min":3.1179999999999999,"15min":3.1179999999999999},"last":0.51000000000000001}]},{"osd":8,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.874,"15min":0.88200000000000001},"min":{"1min":0.66600000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.2829999999999999,"5min":2.891,"15min":2.891},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.90100000000000002,"15min":0.91700000000000004},"min":{"1min":0.63200000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.1499999999999999,"5min":1.8759999999999999,"15min":1.8759999999999999},"last":0.64000000000000001}]},{"osd":9,"last update":"Thu Nov 16 16:48:53 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.97399999999999998,"15min":0.98499999999999999},"min":{"1min":0.67600000000000005,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.69899999999999995,"5min":0.495,"15min":0.495},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.78100000000000003}]},{"osd":10,"last update":"Thu Nov 16 16:49:12 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.70099999999999996,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":3.0859999999999999,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.04,"15min":1.0860000000000001},"min":{"1min":0.68200000000000005,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.9079999999999999,"5min":3.694,"15min":3.694},"last":0.67700000000000005}]},{"osd":11,"last update":"Thu Nov 16 16:49:32 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91600000000000004,"15min":0.88900000000000001},"min":{"1min":0.751,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.1859999999999999,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":0.749},{"interface":"front","average":{"1min":1.0069999999999999,"5min":0.93899999999999995,"15min":0.89800000000000002},"min":{"1min":0.75,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.4370000000000001,"5min":3.0489999999999999,"15min":3.0489999999999999},"last":0.80500000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7640,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737512,"statfs":{"total":95995035648,"available":95987212288,"internally_reserved":0,"allocated":1064960,"data_stored":569393,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56499999999999995,"15min":0.47999999999999998},"min":{"1min":0.44800000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.91400000000000003,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.504},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53500000000000003,"15min":0.499},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.45300000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56399999999999995,"15min":0.502},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.83299999999999996,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.60999999999999999,"15min":0.51800000000000002},"min":{"1min":0.33600000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85799999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.439}]},{"osd":3,"last update":"Thu Nov 16 16:49:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.64900000000000002,"15min":0.71099999999999997},"min":{"1min":0.438,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":3.98},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63400000000000001,"15min":0.59799999999999998},"min":{"1min":0.47499999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95899999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.51400000000000001}]},{"osd":4,"last update":"Thu Nov 16 16:49:19 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.70999999999999996,"15min":0.70599999999999996},"min":{"1min":0.501,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.67100000000000004,"15min":0.67400000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.35999999999999999}]},{"osd":5,"last update":"Thu Nov 16 16:49:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.50700000000000001,"15min":0.47099999999999997},"min":{"1min":0.39200000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76100000000000001,"5min":0.871,"15min":0.871},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.52800000000000002,"15min":0.48899999999999999},"min":{"1min":0.45400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70699999999999996,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.42299999999999999}]},{"osd":6,"last update":"Thu Nov 16 16:48:47 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.629,"15min":0.59499999999999997},"min":{"1min":0.442,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63700000000000001,"15min":0.61499999999999999},"min":{"1min":0.46600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85799999999999998,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.497}]},{"osd":7,"last update":"Thu Nov 16 16:49:24 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.40699999999999997,"5min":0.376,"15min":0.376},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.80000000000000004,"15min":0.79400000000000004},"min":{"1min":0.627,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.627}]},{"osd":8,"last update":"Thu Nov 16 16:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.64500000000000002,"15min":0.61099999999999999},"min":{"1min":0.44800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60099999999999998,"15min":0.58099999999999996},"min":{"1min":0.434,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.45900000000000002}]},{"osd":9,"last update":"Thu Nov 16 16:48:53 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.73299999999999998,"15min":0.72099999999999997},"min":{"1min":0.58799999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.022,"5min":0.79000000000000004,"15min":0.749},"min":{"1min":0.52100000000000002,"5min":0.503,"15min":0.503},"max":{"1min":5.2510000000000003,"5min":5.2510000000000003,"15min":5.2510000000000003},"last":0.57799999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.72199999999999998,"15min":0.69299999999999995},"min":{"1min":0.49299999999999999,"5min":0.441,"15min":0.441},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.71999999999999997,"15min":0.69799999999999995},"min":{"1min":0.56299999999999994,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.52100000000000002}]},{"osd":11,"last update":"Thu Nov 16 16:49:19 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.81399999999999995,"15min":0.80800000000000005},"min":{"1min":0.65200000000000002,"5min":0.499,"15min":0.499},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.745,"15min":0.73799999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.48599999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6804,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738348,"statfs":{"total":95995035648,"available":95988068352,"internally_reserved":0,"allocated":995328,"data_stored":502884,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.48199999999999998,"15min":0.435},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.077,"5min":1.085,"15min":1.085},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.58399999999999996,"15min":0.50600000000000001},"min":{"1min":0.39600000000000002,"5min":0.313,"15min":0.313},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":0.58499999999999996}]},{"osd":1,"last update":"Thu Nov 16 16:48:38 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":0.73699999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.379,"15min":0.379},"max":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":10.266999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.66000000000000003,"15min":0.60399999999999998},"min":{"1min":0.44400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.2050000000000001,"5min":1.286,"15min":1.286},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:48:38 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66500000000000004,"15min":0.59699999999999998},"min":{"1min":0.51200000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.02,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.68700000000000006,"15min":0.61499999999999999},"min":{"1min":0.42299999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.4350000000000001,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Nov 16 16:49:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.71599999999999997,"15min":0.63},"min":{"1min":0.50900000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.25,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.73299999999999998,"15min":0.67900000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.88300000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:49:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.71899999999999997,"15min":0.69299999999999995},"min":{"1min":0.45400000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.879,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.62},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.76800000000000002,"15min":0.68799999999999994},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86399999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":0.56200000000000006}]},{"osd":6,"last update":"Thu Nov 16 16:48:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.71099999999999997,"15min":0.66500000000000004},"min":{"1min":0.47999999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.29,"5min":1.323,"15min":1.323},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70499999999999996,"15min":0.68200000000000005},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.04,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.752}]},{"osd":7,"last update":"Thu Nov 16 16:49:05 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.84299999999999997,"15min":0.80300000000000005},"min":{"1min":0.50700000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.637,"5min":5.1070000000000002,"15min":5.1070000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1,"15min":0.89700000000000002},"min":{"1min":0.60599999999999998,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":10.305999999999999,"5min":10.305999999999999,"15min":10.305999999999999},"last":0.92000000000000004}]},{"osd":8,"last update":"Thu Nov 16 16:46:23 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.94199999999999995}]},{"osd":9,"last update":"Thu Nov 16 16:48:51 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.88400000000000001,"15min":0.91000000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":2.5920000000000001,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.94599999999999995,"15min":0.96799999999999997},"min":{"1min":0.54600000000000004,"5min":0.501,"15min":0.501},"max":{"1min":1.417,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.99299999999999999}]},{"osd":10,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":0.96799999999999997,"15min":0.91200000000000003},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":10.287000000000001,"5min":10.287000000000001,"15min":10.287000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.91800000000000004,"15min":0.93799999999999994},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.012,"5min":2.673,"15min":2.673},"last":0.70799999999999996}]},{"osd":11,"last update":"Thu Nov 16 16:49:24 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.92500000000000004,"15min":0.93500000000000005},"min":{"1min":0.64000000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.1000000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":1.028},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.82999999999999996,"15min":0.82799999999999996},"min":{"1min":0.626,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.78900000000000003}]}]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6676,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738476,"statfs":{"total":95995035648,"available":95988199424,"internally_reserved":0,"allocated":995328,"data_stored":506898,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.48899999999999999,"15min":0.46600000000000003},"min":{"1min":0.439,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.51100000000000001,"15min":0.48599999999999999},"min":{"1min":0.47399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.63500000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.38,"15min":0.33000000000000002},"min":{"1min":0.26500000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57799999999999996,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40000000000000002,"15min":0.35499999999999998},"min":{"1min":0.28100000000000003,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.56299999999999994,"5min":1.212,"15min":1.212},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58499999999999996,"15min":0.52300000000000002},"min":{"1min":0.371,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80800000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.57999999999999996,"15min":0.51600000000000001},"min":{"1min":0.435,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.83799999999999997,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.53100000000000003}]},{"osd":3,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.58899999999999997,"15min":0.53500000000000003},"min":{"1min":0.40899999999999997,"5min":0.307,"15min":0.307},"max":{"1min":0.85699999999999998,"5min":1.349,"15min":1.349},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.745,"15min":0.58899999999999997},"min":{"1min":0.39500000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.84599999999999997,"5min":14.795999999999999,"15min":14.795999999999999},"last":0.40000000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:49:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61299999999999999,"15min":0.55300000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.84899999999999998,"5min":1.462,"15min":1.462},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60499999999999998,"15min":0.53900000000000003},"min":{"1min":0.442,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.496}]},{"osd":6,"last update":"Thu Nov 16 16:48:52 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.624,"15min":0.58599999999999997},"min":{"1min":0.44500000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.85899999999999999,"5min":1.323,"15min":1.323},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.66200000000000003,"15min":0.62},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89400000000000002,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":0.54800000000000004}]},{"osd":7,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56200000000000006,"15min":0.52800000000000002},"min":{"1min":0.38900000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88500000000000001,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":0.443},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.60599999999999998,"15min":0.57399999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81999999999999995,"5min":1.764,"15min":1.764},"last":0.46100000000000002}]},{"osd":8,"last update":"Thu Nov 16 16:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.71999999999999997,"15min":0.71299999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88500000000000001,"5min":2.7650000000000001,"15min":2.7650000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73299999999999998,"15min":0.74399999999999999},"min":{"1min":0.51200000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.872,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.51000000000000001}]},{"osd":9,"last update":"Thu Nov 16 16:48:52 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.77700000000000002,"15min":0.79900000000000004},"min":{"1min":0.48599999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.90900000000000003,"5min":1.246,"15min":1.246},"last":0.621},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.76500000000000001,"15min":0.77700000000000002},"min":{"1min":0.51200000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.88900000000000001,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.57699999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.69099999999999995,"15min":0.69899999999999995},"min":{"1min":0.48699999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.89000000000000001,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.70199999999999996,"15min":0.71599999999999997},"min":{"1min":0.497,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90200000000000002,"5min":1.532,"15min":1.532},"last":0.56999999999999995}]},{"osd":11,"last update":"Thu Nov 16 16:49:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.79800000000000004,"15min":0.80500000000000005},"min":{"1min":0.54500000000000004,"5min":0.501,"15min":0.501},"max":{"1min":0.93400000000000005,"5min":1.502,"15min":1.502},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.82099999999999995,"15min":0.82699999999999996},"min":{"1min":0.57399999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.96499999999999997,"5min":1.425,"15min":1.425},"last":0.59299999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6860,"kb_used_data":1028,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738292,"statfs":{"total":95995035648,"available":95988011008,"internally_reserved":0,"allocated":1052672,"data_stored":1097426,"data_compressed":59395,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:49:20 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.51100000000000001,"15min":0.51200000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.60199999999999998,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54500000000000004,"15min":0.56000000000000005},"min":{"1min":0.35799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63200000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Nov 16 16:49:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.58999999999999997,"15min":0.55400000000000005},"min":{"1min":0.435,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.82299999999999995,"5min":1.167,"15min":1.167},"last":0.377},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.58399999999999996,"15min":0.52600000000000002},"min":{"1min":0.36299999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.75600000000000001,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.30199999999999999}]},{"osd":2,"last update":"Thu Nov 16 16:49:20 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40400000000000003,"15min":0.377},"min":{"1min":0.33900000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57699999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39700000000000002},"min":{"1min":0.33000000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.68200000000000005,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.33500000000000002}]},{"osd":3,"last update":"Thu Nov 16 16:49:20 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55300000000000005,"15min":0.53500000000000003},"min":{"1min":0.41199999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.65800000000000003,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.497},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59999999999999998,"15min":0.57199999999999995},"min":{"1min":0.46100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.71499999999999997,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.46300000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:49:20 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.59699999999999998,"15min":0.56100000000000005},"min":{"1min":0.54600000000000004,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85799999999999998,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57899999999999996,"15min":0.53500000000000003},"min":{"1min":0.44700000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.84299999999999997,"5min":0.879,"15min":0.879},"last":0.314}]},{"osd":6,"last update":"Thu Nov 16 16:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.438,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86699999999999999,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.64500000000000002,"15min":0.63300000000000001},"min":{"1min":0.53000000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.92800000000000005,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.60899999999999999}]},{"osd":7,"last update":"Thu Nov 16 16:49:08 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.63300000000000001,"15min":0.61099999999999999},"min":{"1min":0.49199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.86699999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.436},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.65500000000000003,"15min":0.63700000000000001},"min":{"1min":0.58399999999999996,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.95499999999999996,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.46899999999999997}]},{"osd":8,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56100000000000005,"15min":0.54400000000000004},"min":{"1min":0.436,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80400000000000005,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.62,"15min":0.59599999999999997},"min":{"1min":0.51800000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.90300000000000002,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.48099999999999998}]},{"osd":9,"last update":"Thu Nov 16 16:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.747,"15min":0.76100000000000001},"min":{"1min":0.61599999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.91700000000000004,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.745,"15min":0.75700000000000001},"min":{"1min":0.60399999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.90700000000000003,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.64200000000000002}]},{"osd":10,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.71299999999999997},"min":{"1min":0.55600000000000005,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.89500000000000002,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.76900000000000002,"15min":0.75700000000000001},"min":{"1min":0.68600000000000005,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.995,"5min":2.492,"15min":2.492},"last":0.57599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":66886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":67046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T16:49:41.952 INFO:tasks.fwd_scrub.ceph_manager:clean! 2023-11-16T16:49:41.952 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T16:49:41.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:41 smithi077 ceph-mon[131204]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 682 B/s wr, 2 op/s 2023-11-16T16:49:41.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:41 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3352392595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:41 smithi176 ceph-mon[131630]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 682 B/s wr, 2 op/s 2023-11-16T16:49:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:41 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3352392595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:42.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.189+0000 7f9d9699d700 1 -- 172.21.15.44:0/3370195441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d90101310 msgr2=0x7f9d90103790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:42.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.189+0000 7f9d9699d700 1 --2- 172.21.15.44:0/3370195441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d90101310 0x7f9d90103790 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9d800098f0 tx=0x7f9d80009c00 comp rx=0 tx=0).stop 2023-11-16T16:49:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.190+0000 7f9d9699d700 1 -- 172.21.15.44:0/3370195441 shutdown_connections 2023-11-16T16:49:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.190+0000 7f9d9699d700 1 --2- 172.21.15.44:0/3370195441 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d90103cd0 0x7f9d90104150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.190+0000 7f9d9699d700 1 --2- 172.21.15.44:0/3370195441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d900fe9b0 0x7f9d90100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.190+0000 7f9d9699d700 1 --2- 172.21.15.44:0/3370195441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d90101310 0x7f9d90103790 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.190+0000 7f9d9699d700 1 -- 172.21.15.44:0/3370195441 >> 172.21.15.44:0/3370195441 conn(0x7f9d900fa620 msgr2=0x7f9d900fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.190+0000 7f9d9699d700 1 -- 172.21.15.44:0/3370195441 shutdown_connections 2023-11-16T16:49:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.190+0000 7f9d9699d700 1 -- 172.21.15.44:0/3370195441 wait complete. 2023-11-16T16:49:42.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.191+0000 7f9d9699d700 1 Processor -- start 2023-11-16T16:49:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.191+0000 7f9d9699d700 1 -- start start 2023-11-16T16:49:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.191+0000 7f9d9699d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d900fe9b0 0x7f9d9019c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.191+0000 7f9d9699d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d90101310 0x7f9d9019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.191+0000 7f9d9699d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d90103cd0 0x7f9d901a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.191+0000 7f9d9699d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d901a3f80 con 0x7f9d90101310 2023-11-16T16:49:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.191+0000 7f9d9699d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d901a40c0 con 0x7f9d900fe9b0 2023-11-16T16:49:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.191+0000 7f9d9699d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d901a4250 con 0x7f9d90103cd0 2023-11-16T16:49:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.191+0000 7f9d9599b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d900fe9b0 0x7f9d9019c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.191+0000 7f9d9519a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d90101310 0x7f9d9019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.191+0000 7f9d9599b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d900fe9b0 0x7f9d9019c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:49872/0 (socket says 172.21.15.44:49872) 2023-11-16T16:49:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.191+0000 7f9d9599b700 1 -- 172.21.15.44:0/3796321254 learned_addr learned my addr 172.21.15.44:0/3796321254 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.192+0000 7f9d9619c700 1 --2- 172.21.15.44:0/3796321254 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d90103cd0 0x7f9d901a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.192+0000 7f9d9599b700 1 -- 172.21.15.44:0/3796321254 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d90103cd0 msgr2=0x7f9d901a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.192+0000 7f9d9599b700 1 --2- 172.21.15.44:0/3796321254 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d90103cd0 0x7f9d901a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.192+0000 7f9d9599b700 1 -- 172.21.15.44:0/3796321254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d90101310 msgr2=0x7f9d9019c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.192+0000 7f9d9599b700 1 --2- 172.21.15.44:0/3796321254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d90101310 0x7f9d9019c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.192+0000 7f9d9599b700 1 -- 172.21.15.44:0/3796321254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d80009520 con 0x7f9d900fe9b0 2023-11-16T16:49:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.192+0000 7f9d9599b700 1 --2- 172.21.15.44:0/3796321254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d900fe9b0 0x7f9d9019c400 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00e970 tx=0x7f9d8c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.193+0000 7f9d86ffd700 1 -- 172.21.15.44:0/3796321254 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c009790 con 0x7f9d900fe9b0 2023-11-16T16:49:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.193+0000 7f9d86ffd700 1 -- 172.21.15.44:0/3796321254 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d8c00c7e0 con 0x7f9d900fe9b0 2023-11-16T16:49:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.193+0000 7f9d86ffd700 1 -- 172.21.15.44:0/3796321254 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c003c70 con 0x7f9d900fe9b0 2023-11-16T16:49:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.193+0000 7f9d9699d700 1 -- 172.21.15.44:0/3796321254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d90112080 con 0x7f9d900fe9b0 2023-11-16T16:49:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.193+0000 7f9d9699d700 1 -- 172.21.15.44:0/3796321254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d90111ef0 con 0x7f9d900fe9b0 2023-11-16T16:49:42.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.194+0000 7f9d9699d700 1 -- 172.21.15.44:0/3796321254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d900fc190 con 0x7f9d900fe9b0 2023-11-16T16:49:42.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.196+0000 7f9d86ffd700 1 -- 172.21.15.44:0/3796321254 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9d8c00cc50 con 0x7f9d900fe9b0 2023-11-16T16:49:42.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.197+0000 7f9d86ffd700 1 --2- 172.21.15.44:0/3796321254 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9d7c0a11d0 0x7f9d7c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:42.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.198+0000 7f9d86ffd700 1 -- 172.21.15.44:0/3796321254 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f9d8c014070 con 0x7f9d900fe9b0 2023-11-16T16:49:42.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.198+0000 7f9d9519a700 1 --2- 172.21.15.44:0/3796321254 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9d7c0a11d0 0x7f9d7c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:42.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.198+0000 7f9d9519a700 1 --2- 172.21.15.44:0/3796321254 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9d7c0a11d0 0x7f9d7c0a3680 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9d80005e90 tx=0x7f9d80019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:42.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.201+0000 7f9d86ffd700 1 -- 172.21.15.44:0/3796321254 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9d8c096170 con 0x7f9d900fe9b0 2023-11-16T16:49:42.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.405+0000 7f9d9699d700 1 -- 172.21.15.44:0/3796321254 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9d90061cb0 con 0x7f9d7c0a11d0 2023-11-16T16:49:42.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.418+0000 7f9d86ffd700 1 -- 172.21.15.44:0/3796321254 <== mgr.14150 v2:172.21.15.44:6800/304422349 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426969 (secure 0 0 0) 0x7f9d90061cb0 con 0x7f9d7c0a11d0 2023-11-16T16:49:42.426 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:42.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.424+0000 7f9d9699d700 1 -- 172.21.15.44:0/3796321254 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9d7c0a11d0 msgr2=0x7f9d7c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:42.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.424+0000 7f9d9699d700 1 --2- 172.21.15.44:0/3796321254 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9d7c0a11d0 0x7f9d7c0a3680 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9d80005e90 tx=0x7f9d80019040 comp rx=0 tx=0).stop 2023-11-16T16:49:42.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.424+0000 7f9d9699d700 1 -- 172.21.15.44:0/3796321254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d900fe9b0 msgr2=0x7f9d9019c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:42.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.424+0000 7f9d9699d700 1 --2- 172.21.15.44:0/3796321254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d900fe9b0 0x7f9d9019c400 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00e970 tx=0x7f9d8c00ec80 comp rx=0 tx=0).stop 2023-11-16T16:49:42.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.424+0000 7f9d9699d700 1 -- 172.21.15.44:0/3796321254 shutdown_connections 2023-11-16T16:49:42.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.424+0000 7f9d9699d700 1 --2- 172.21.15.44:0/3796321254 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9d7c0a11d0 0x7f9d7c0a3680 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.424+0000 7f9d9699d700 1 --2- 172.21.15.44:0/3796321254 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d90103cd0 0x7f9d901a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.424+0000 7f9d9699d700 1 --2- 172.21.15.44:0/3796321254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d900fe9b0 0x7f9d9019c400 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.424+0000 7f9d9699d700 1 --2- 172.21.15.44:0/3796321254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d90101310 0x7f9d9019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.424+0000 7f9d9699d700 1 -- 172.21.15.44:0/3796321254 >> 172.21.15.44:0/3796321254 conn(0x7f9d900fa620 msgr2=0x7f9d900ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:42.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.424+0000 7f9d9699d700 1 -- 172.21.15.44:0/3796321254 shutdown_connections 2023-11-16T16:49:42.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.424+0000 7f9d9699d700 1 -- 172.21.15.44:0/3796321254 wait complete. 2023-11-16T16:49:42.431 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2023-11-16T16:49:42.445 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2023-11-16T16:49:40.570933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658264,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":375,"num_read_kb":309,"num_write":267,"num_write_kb":3039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":713,"ondisk_log_size":713,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":82132,"kb_used_data":12020,"kb_used_omap":0,"kb_used_meta":70016,"kb_avail":1124859692,"statfs":{"total":1151940427776,"available":1151856324608,"internally_reserved":0,"allocated":12308480,"data_stored":8034530,"data_compressed":390621,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":0,"internal_metadata":71696384},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5695,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":22,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003677"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.591350+0000","last_change":"2023-11-16T16:47:18.183024+0000","last_active":"2023-11-16T16:49:12.591350+0000","last_peered":"2023-11-16T16:49:12.591350+0000","last_clean":"2023-11-16T16:49:12.591350+0000","last_became_active":"2023-11-16T16:47:18.182387+0000","last_became_peered":"2023-11-16T16:47:18.182387+0000","last_unstale":"2023-11-16T16:49:12.591350+0000","last_undegraded":"2023-11-16T16:49:12.591350+0000","last_fullsized":"2023-11-16T16:49:12.591350+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:08:25.731718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.589033+0000","last_change":"2023-11-16T16:47:18.200135+0000","last_active":"2023-11-16T16:49:12.589033+0000","last_peered":"2023-11-16T16:49:12.589033+0000","last_clean":"2023-11-16T16:49:12.589033+0000","last_became_active":"2023-11-16T16:47:18.194732+0000","last_became_peered":"2023-11-16T16:47:18.194732+0000","last_unstale":"2023-11-16T16:49:12.589033+0000","last_undegraded":"2023-11-16T16:49:12.589033+0000","last_fullsized":"2023-11-16T16:49:12.589033+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:36:09.213648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.589098+0000","last_change":"2023-11-16T16:47:18.184127+0000","last_active":"2023-11-16T16:49:12.589098+0000","last_peered":"2023-11-16T16:49:12.589098+0000","last_clean":"2023-11-16T16:49:12.589098+0000","last_became_active":"2023-11-16T16:47:18.183163+0000","last_became_peered":"2023-11-16T16:47:18.183163+0000","last_unstale":"2023-11-16T16:49:12.589098+0000","last_undegraded":"2023-11-16T16:49:12.589098+0000","last_fullsized":"2023-11-16T16:49:12.589098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:18:16.732196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.577897+0000","last_change":"2023-11-16T16:47:18.177476+0000","last_active":"2023-11-16T16:49:12.577897+0000","last_peered":"2023-11-16T16:49:12.577897+0000","last_clean":"2023-11-16T16:49:12.577897+0000","last_became_active":"2023-11-16T16:47:18.177178+0000","last_became_peered":"2023-11-16T16:47:18.177178+0000","last_unstale":"2023-11-16T16:49:12.577897+0000","last_undegraded":"2023-11-16T16:49:12.577897+0000","last_fullsized":"2023-11-16T16:49:12.577897+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:34:33.524556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.578631+0000","last_change":"2023-11-16T16:47:18.178511+0000","last_active":"2023-11-16T16:49:12.578631+0000","last_peered":"2023-11-16T16:49:12.578631+0000","last_clean":"2023-11-16T16:49:12.578631+0000","last_became_active":"2023-11-16T16:47:18.178198+0000","last_became_peered":"2023-11-16T16:47:18.178198+0000","last_unstale":"2023-11-16T16:49:12.578631+0000","last_undegraded":"2023-11-16T16:49:12.578631+0000","last_fullsized":"2023-11-16T16:49:12.578631+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:04:36.059447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.577044+0000","last_change":"2023-11-16T16:47:18.186552+0000","last_active":"2023-11-16T16:49:12.577044+0000","last_peered":"2023-11-16T16:49:12.577044+0000","last_clean":"2023-11-16T16:49:12.577044+0000","last_became_active":"2023-11-16T16:47:18.186238+0000","last_became_peered":"2023-11-16T16:47:18.186238+0000","last_unstale":"2023-11-16T16:49:12.577044+0000","last_undegraded":"2023-11-16T16:49:12.577044+0000","last_fullsized":"2023-11-16T16:49:12.577044+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:58:35.571757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.566038+0000","last_change":"2023-11-16T16:47:18.179839+0000","last_active":"2023-11-16T16:49:12.566038+0000","last_peered":"2023-11-16T16:49:12.566038+0000","last_clean":"2023-11-16T16:49:12.566038+0000","last_became_active":"2023-11-16T16:47:18.179060+0000","last_became_peered":"2023-11-16T16:47:18.179060+0000","last_unstale":"2023-11-16T16:49:12.566038+0000","last_undegraded":"2023-11-16T16:49:12.566038+0000","last_fullsized":"2023-11-16T16:49:12.566038+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:04:42.573359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.576400+0000","last_change":"2023-11-16T16:47:43.631738+0000","last_active":"2023-11-16T16:49:12.576400+0000","last_peered":"2023-11-16T16:49:12.576400+0000","last_clean":"2023-11-16T16:49:12.576400+0000","last_became_active":"2023-11-16T16:47:43.631146+0000","last_became_peered":"2023-11-16T16:47:43.631146+0000","last_unstale":"2023-11-16T16:49:12.576400+0000","last_undegraded":"2023-11-16T16:49:12.576400+0000","last_fullsized":"2023-11-16T16:49:12.576400+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:58:11.158742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579232+0000","last_change":"2023-11-16T16:47:18.177834+0000","last_active":"2023-11-16T16:49:12.579232+0000","last_peered":"2023-11-16T16:49:12.579232+0000","last_clean":"2023-11-16T16:49:12.579232+0000","last_became_active":"2023-11-16T16:47:18.176299+0000","last_became_peered":"2023-11-16T16:47:18.176299+0000","last_unstale":"2023-11-16T16:49:12.579232+0000","last_undegraded":"2023-11-16T16:49:12.579232+0000","last_fullsized":"2023-11-16T16:49:12.579232+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:27:07.129418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.571605+0000","last_change":"2023-11-16T16:47:18.179394+0000","last_active":"2023-11-16T16:49:12.571605+0000","last_peered":"2023-11-16T16:49:12.571605+0000","last_clean":"2023-11-16T16:49:12.571605+0000","last_became_active":"2023-11-16T16:47:18.179119+0000","last_became_peered":"2023-11-16T16:47:18.179119+0000","last_unstale":"2023-11-16T16:49:12.571605+0000","last_undegraded":"2023-11-16T16:49:12.571605+0000","last_fullsized":"2023-11-16T16:49:12.571605+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:50:38.327278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579257+0000","last_change":"2023-11-16T16:47:18.183585+0000","last_active":"2023-11-16T16:49:12.579257+0000","last_peered":"2023-11-16T16:49:12.579257+0000","last_clean":"2023-11-16T16:49:12.579257+0000","last_became_active":"2023-11-16T16:47:18.183209+0000","last_became_peered":"2023-11-16T16:47:18.183209+0000","last_unstale":"2023-11-16T16:49:12.579257+0000","last_undegraded":"2023-11-16T16:49:12.579257+0000","last_fullsized":"2023-11-16T16:49:12.579257+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:35:11.334704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.570790+0000","last_change":"2023-11-16T16:47:18.186503+0000","last_active":"2023-11-16T16:49:12.570790+0000","last_peered":"2023-11-16T16:49:12.570790+0000","last_clean":"2023-11-16T16:49:12.570790+0000","last_became_active":"2023-11-16T16:47:18.186193+0000","last_became_peered":"2023-11-16T16:47:18.186193+0000","last_unstale":"2023-11-16T16:49:12.570790+0000","last_undegraded":"2023-11-16T16:49:12.570790+0000","last_fullsized":"2023-11-16T16:49:12.570790+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:08:26.864946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.584559+0000","last_change":"2023-11-16T16:47:18.193842+0000","last_active":"2023-11-16T16:49:12.584559+0000","last_peered":"2023-11-16T16:49:12.584559+0000","last_clean":"2023-11-16T16:49:12.584559+0000","last_became_active":"2023-11-16T16:47:18.192309+0000","last_became_peered":"2023-11-16T16:47:18.192309+0000","last_unstale":"2023-11-16T16:49:12.584559+0000","last_undegraded":"2023-11-16T16:49:12.584559+0000","last_fullsized":"2023-11-16T16:49:12.584559+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:47:16.555679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.585449+0000","last_change":"2023-11-16T16:47:18.192359+0000","last_active":"2023-11-16T16:49:12.585449+0000","last_peered":"2023-11-16T16:49:12.585449+0000","last_clean":"2023-11-16T16:49:12.585449+0000","last_became_active":"2023-11-16T16:47:18.189765+0000","last_became_peered":"2023-11-16T16:47:18.189765+0000","last_unstale":"2023-11-16T16:49:12.585449+0000","last_undegraded":"2023-11-16T16:49:12.585449+0000","last_fullsized":"2023-11-16T16:49:12.585449+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:56:51.066489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.588434+0000","last_change":"2023-11-16T16:47:18.178386+0000","last_active":"2023-11-16T16:49:12.588434+0000","last_peered":"2023-11-16T16:49:12.588434+0000","last_clean":"2023-11-16T16:49:12.588434+0000","last_became_active":"2023-11-16T16:47:18.178041+0000","last_became_peered":"2023-11-16T16:47:18.178041+0000","last_unstale":"2023-11-16T16:49:12.588434+0000","last_undegraded":"2023-11-16T16:49:12.588434+0000","last_fullsized":"2023-11-16T16:49:12.588434+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:08:49.324272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.591918+0000","last_change":"2023-11-16T16:47:18.174424+0000","last_active":"2023-11-16T16:49:12.591918+0000","last_peered":"2023-11-16T16:49:12.591918+0000","last_clean":"2023-11-16T16:49:12.591918+0000","last_became_active":"2023-11-16T16:47:18.174078+0000","last_became_peered":"2023-11-16T16:47:18.174078+0000","last_unstale":"2023-11-16T16:49:12.591918+0000","last_undegraded":"2023-11-16T16:49:12.591918+0000","last_fullsized":"2023-11-16T16:49:12.591918+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:23:55.960648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.591349+0000","last_change":"2023-11-16T16:47:18.183161+0000","last_active":"2023-11-16T16:49:12.591349+0000","last_peered":"2023-11-16T16:49:12.591349+0000","last_clean":"2023-11-16T16:49:12.591349+0000","last_became_active":"2023-11-16T16:47:18.182806+0000","last_became_peered":"2023-11-16T16:47:18.182806+0000","last_unstale":"2023-11-16T16:49:12.591349+0000","last_undegraded":"2023-11-16T16:49:12.591349+0000","last_fullsized":"2023-11-16T16:49:12.591349+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:17:48.777833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.580719+0000","last_change":"2023-11-16T16:47:18.187225+0000","last_active":"2023-11-16T16:49:12.580719+0000","last_peered":"2023-11-16T16:49:12.580719+0000","last_clean":"2023-11-16T16:49:12.580719+0000","last_became_active":"2023-11-16T16:47:18.186858+0000","last_became_peered":"2023-11-16T16:47:18.186858+0000","last_unstale":"2023-11-16T16:49:12.580719+0000","last_undegraded":"2023-11-16T16:49:12.580719+0000","last_fullsized":"2023-11-16T16:49:12.580719+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:03:50.025202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579231+0000","last_change":"2023-11-16T16:47:18.190947+0000","last_active":"2023-11-16T16:49:12.579231+0000","last_peered":"2023-11-16T16:49:12.579231+0000","last_clean":"2023-11-16T16:49:12.579231+0000","last_became_active":"2023-11-16T16:47:18.190737+0000","last_became_peered":"2023-11-16T16:47:18.190737+0000","last_unstale":"2023-11-16T16:49:12.579231+0000","last_undegraded":"2023-11-16T16:49:12.579231+0000","last_fullsized":"2023-11-16T16:49:12.579231+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:50:36.934811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623109+0000","last_change":"2023-11-16T16:47:22.659730+0000","last_active":"2023-11-16T16:48:44.623109+0000","last_peered":"2023-11-16T16:48:44.623109+0000","last_clean":"2023-11-16T16:48:44.623109+0000","last_became_active":"2023-11-16T16:47:22.659313+0000","last_became_peered":"2023-11-16T16:47:22.659313+0000","last_unstale":"2023-11-16T16:48:44.623109+0000","last_undegraded":"2023-11-16T16:48:44.623109+0000","last_fullsized":"2023-11-16T16:48:44.623109+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:12:16.481784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'4","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626280+0000","last_change":"2023-11-16T16:47:17.629748+0000","last_active":"2023-11-16T16:48:44.626280+0000","last_peered":"2023-11-16T16:48:44.626280+0000","last_clean":"2023-11-16T16:48:44.626280+0000","last_became_active":"2023-11-16T16:47:17.629284+0000","last_became_peered":"2023-11-16T16:47:17.629284+0000","last_unstale":"2023-11-16T16:48:44.626280+0000","last_undegraded":"2023-11-16T16:48:44.626280+0000","last_fullsized":"2023-11-16T16:48:44.626280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:07:47.593078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.577615+0000","last_change":"2023-11-16T16:47:18.176239+0000","last_active":"2023-11-16T16:49:12.577615+0000","last_peered":"2023-11-16T16:49:12.577615+0000","last_clean":"2023-11-16T16:49:12.577615+0000","last_became_active":"2023-11-16T16:47:18.175879+0000","last_became_peered":"2023-11-16T16:47:18.175879+0000","last_unstale":"2023-11-16T16:49:12.577615+0000","last_undegraded":"2023-11-16T16:49:12.577615+0000","last_fullsized":"2023-11-16T16:49:12.577615+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:31:16.561539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.622732+0000","last_change":"2023-11-16T16:47:22.659679+0000","last_active":"2023-11-16T16:48:44.622732+0000","last_peered":"2023-11-16T16:48:44.622732+0000","last_clean":"2023-11-16T16:48:44.622732+0000","last_became_active":"2023-11-16T16:47:22.658946+0000","last_became_peered":"2023-11-16T16:47:22.658946+0000","last_unstale":"2023-11-16T16:48:44.622732+0000","last_undegraded":"2023-11-16T16:48:44.622732+0000","last_fullsized":"2023-11-16T16:48:44.622732+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:32:34.649897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"82'1","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.912310+0000","last_change":"2023-11-16T16:47:17.643944+0000","last_active":"2023-11-16T16:48:44.912310+0000","last_peered":"2023-11-16T16:48:44.912310+0000","last_clean":"2023-11-16T16:48:44.912310+0000","last_became_active":"2023-11-16T16:47:17.643470+0000","last_became_peered":"2023-11-16T16:47:17.643470+0000","last_unstale":"2023-11-16T16:48:44.912310+0000","last_undegraded":"2023-11-16T16:48:44.912310+0000","last_fullsized":"2023-11-16T16:48:44.912310+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:54:38.723005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.588194+0000","last_change":"2023-11-16T16:47:18.198139+0000","last_active":"2023-11-16T16:49:12.588194+0000","last_peered":"2023-11-16T16:49:12.588194+0000","last_clean":"2023-11-16T16:49:12.588194+0000","last_became_active":"2023-11-16T16:47:18.197762+0000","last_became_peered":"2023-11-16T16:47:18.197762+0000","last_unstale":"2023-11-16T16:49:12.588194+0000","last_undegraded":"2023-11-16T16:49:12.588194+0000","last_fullsized":"2023-11-16T16:49:12.588194+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:24:38.383735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623023+0000","last_change":"2023-11-16T16:47:22.655810+0000","last_active":"2023-11-16T16:48:44.623023+0000","last_peered":"2023-11-16T16:48:44.623023+0000","last_clean":"2023-11-16T16:48:44.623023+0000","last_became_active":"2023-11-16T16:47:22.655388+0000","last_became_peered":"2023-11-16T16:47:22.655388+0000","last_unstale":"2023-11-16T16:48:44.623023+0000","last_undegraded":"2023-11-16T16:48:44.623023+0000","last_fullsized":"2023-11-16T16:48:44.623023+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:44:39.423524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'2","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.630121+0000","last_change":"2023-11-16T16:47:43.635266+0000","last_active":"2023-11-16T16:48:44.630121+0000","last_peered":"2023-11-16T16:48:44.630121+0000","last_clean":"2023-11-16T16:48:44.630121+0000","last_became_active":"2023-11-16T16:47:43.630583+0000","last_became_peered":"2023-11-16T16:47:43.630583+0000","last_unstale":"2023-11-16T16:48:44.630121+0000","last_undegraded":"2023-11-16T16:48:44.630121+0000","last_fullsized":"2023-11-16T16:48:44.630121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:09:31.118965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.585032+0000","last_change":"2023-11-16T16:47:18.186341+0000","last_active":"2023-11-16T16:49:12.585032+0000","last_peered":"2023-11-16T16:49:12.585032+0000","last_clean":"2023-11-16T16:49:12.585032+0000","last_became_active":"2023-11-16T16:47:18.186066+0000","last_became_peered":"2023-11-16T16:47:18.186066+0000","last_unstale":"2023-11-16T16:49:12.585032+0000","last_undegraded":"2023-11-16T16:49:12.585032+0000","last_fullsized":"2023-11-16T16:49:12.585032+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:35:29.489855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.622861+0000","last_change":"2023-11-16T16:47:22.657227+0000","last_active":"2023-11-16T16:48:44.622861+0000","last_peered":"2023-11-16T16:48:44.622861+0000","last_clean":"2023-11-16T16:48:44.622861+0000","last_became_active":"2023-11-16T16:47:22.656835+0000","last_became_peered":"2023-11-16T16:47:22.656835+0000","last_unstale":"2023-11-16T16:48:44.622861+0000","last_undegraded":"2023-11-16T16:48:44.622861+0000","last_fullsized":"2023-11-16T16:48:44.622861+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:55:50.404613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.630547+0000","last_change":"2023-11-16T16:47:17.630194+0000","last_active":"2023-11-16T16:48:44.630547+0000","last_peered":"2023-11-16T16:48:44.630547+0000","last_clean":"2023-11-16T16:48:44.630547+0000","last_became_active":"2023-11-16T16:47:17.629886+0000","last_became_peered":"2023-11-16T16:47:17.629886+0000","last_unstale":"2023-11-16T16:48:44.630547+0000","last_undegraded":"2023-11-16T16:48:44.630547+0000","last_fullsized":"2023-11-16T16:48:44.630547+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:58:10.365614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.573343+0000","last_change":"2023-11-16T16:47:18.181596+0000","last_active":"2023-11-16T16:49:12.573343+0000","last_peered":"2023-11-16T16:49:12.573343+0000","last_clean":"2023-11-16T16:49:12.573343+0000","last_became_active":"2023-11-16T16:47:18.181287+0000","last_became_peered":"2023-11-16T16:47:18.181287+0000","last_unstale":"2023-11-16T16:49:12.573343+0000","last_undegraded":"2023-11-16T16:49:12.573343+0000","last_fullsized":"2023-11-16T16:49:12.573343+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:10:47.679818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623684+0000","last_change":"2023-11-16T16:47:22.649197+0000","last_active":"2023-11-16T16:48:44.623684+0000","last_peered":"2023-11-16T16:48:44.623684+0000","last_clean":"2023-11-16T16:48:44.623684+0000","last_became_active":"2023-11-16T16:47:22.648664+0000","last_became_peered":"2023-11-16T16:47:22.648664+0000","last_unstale":"2023-11-16T16:48:44.623684+0000","last_undegraded":"2023-11-16T16:48:44.623684+0000","last_fullsized":"2023-11-16T16:48:44.623684+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:52:46.052156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"82'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911165+0000","last_change":"2023-11-16T16:47:17.627672+0000","last_active":"2023-11-16T16:48:44.911165+0000","last_peered":"2023-11-16T16:48:44.911165+0000","last_clean":"2023-11-16T16:48:44.911165+0000","last_became_active":"2023-11-16T16:47:17.627383+0000","last_became_peered":"2023-11-16T16:47:17.627383+0000","last_unstale":"2023-11-16T16:48:44.911165+0000","last_undegraded":"2023-11-16T16:48:44.911165+0000","last_fullsized":"2023-11-16T16:48:44.911165+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:51:59.811190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.560868+0000","last_change":"2023-11-16T16:47:18.180983+0000","last_active":"2023-11-16T16:49:12.560868+0000","last_peered":"2023-11-16T16:49:12.560868+0000","last_clean":"2023-11-16T16:49:12.560868+0000","last_became_active":"2023-11-16T16:47:18.180702+0000","last_became_peered":"2023-11-16T16:47:18.180702+0000","last_unstale":"2023-11-16T16:49:12.560868+0000","last_undegraded":"2023-11-16T16:49:12.560868+0000","last_fullsized":"2023-11-16T16:49:12.560868+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:02:50.226183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.913692+0000","last_change":"2023-11-16T16:48:44.913692+0000","last_active":"2023-11-16T16:48:44.913692+0000","last_peered":"2023-11-16T16:48:44.913692+0000","last_clean":"2023-11-16T16:48:44.913692+0000","last_became_active":"2023-11-16T16:48:44.913204+0000","last_became_peered":"2023-11-16T16:48:44.913204+0000","last_unstale":"2023-11-16T16:48:44.913692+0000","last_undegraded":"2023-11-16T16:48:44.913692+0000","last_fullsized":"2023-11-16T16:48:44.913692+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:04:35.705849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"82'1","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626774+0000","last_change":"2023-11-16T16:47:43.949231+0000","last_active":"2023-11-16T16:48:44.626774+0000","last_peered":"2023-11-16T16:48:44.626774+0000","last_clean":"2023-11-16T16:48:44.626774+0000","last_became_active":"2023-11-16T16:47:43.948766+0000","last_became_peered":"2023-11-16T16:47:43.948766+0000","last_unstale":"2023-11-16T16:48:44.626774+0000","last_undegraded":"2023-11-16T16:48:44.626774+0000","last_fullsized":"2023-11-16T16:48:44.626774+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:10:43.974784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579822+0000","last_change":"2023-11-16T16:47:18.185263+0000","last_active":"2023-11-16T16:49:12.579822+0000","last_peered":"2023-11-16T16:49:12.579822+0000","last_clean":"2023-11-16T16:49:12.579822+0000","last_became_active":"2023-11-16T16:47:18.184910+0000","last_became_peered":"2023-11-16T16:47:18.184910+0000","last_unstale":"2023-11-16T16:49:12.579822+0000","last_undegraded":"2023-11-16T16:49:12.579822+0000","last_fullsized":"2023-11-16T16:49:12.579822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:33:21.979376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628573+0000","last_change":"2023-11-16T16:47:22.659711+0000","last_active":"2023-11-16T16:48:44.628573+0000","last_peered":"2023-11-16T16:48:44.628573+0000","last_clean":"2023-11-16T16:48:44.628573+0000","last_became_active":"2023-11-16T16:47:22.659087+0000","last_became_peered":"2023-11-16T16:47:22.659087+0000","last_unstale":"2023-11-16T16:48:44.628573+0000","last_undegraded":"2023-11-16T16:48:44.628573+0000","last_fullsized":"2023-11-16T16:48:44.628573+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:21:48.086846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910749+0000","last_change":"2023-11-16T16:47:17.629703+0000","last_active":"2023-11-16T16:48:44.910749+0000","last_peered":"2023-11-16T16:48:44.910749+0000","last_clean":"2023-11-16T16:48:44.910749+0000","last_became_active":"2023-11-16T16:47:17.629245+0000","last_became_peered":"2023-11-16T16:47:17.629245+0000","last_unstale":"2023-11-16T16:48:44.910749+0000","last_undegraded":"2023-11-16T16:48:44.910749+0000","last_fullsized":"2023-11-16T16:48:44.910749+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:40:19.828594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.584225+0000","last_change":"2023-11-16T16:47:18.187149+0000","last_active":"2023-11-16T16:49:12.584225+0000","last_peered":"2023-11-16T16:49:12.584225+0000","last_clean":"2023-11-16T16:49:12.584225+0000","last_became_active":"2023-11-16T16:47:18.185717+0000","last_became_peered":"2023-11-16T16:47:18.185717+0000","last_unstale":"2023-11-16T16:49:12.584225+0000","last_undegraded":"2023-11-16T16:49:12.584225+0000","last_fullsized":"2023-11-16T16:49:12.584225+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:47:48.993299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.913952+0000","last_change":"2023-11-16T16:47:22.663025+0000","last_active":"2023-11-16T16:48:44.913952+0000","last_peered":"2023-11-16T16:48:44.913952+0000","last_clean":"2023-11-16T16:48:44.913952+0000","last_became_active":"2023-11-16T16:47:22.662596+0000","last_became_peered":"2023-11-16T16:47:22.662596+0000","last_unstale":"2023-11-16T16:48:44.913952+0000","last_undegraded":"2023-11-16T16:48:44.913952+0000","last_fullsized":"2023-11-16T16:48:44.913952+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:01:20.821553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910950+0000","last_change":"2023-11-16T16:47:17.626357+0000","last_active":"2023-11-16T16:48:44.910950+0000","last_peered":"2023-11-16T16:48:44.910950+0000","last_clean":"2023-11-16T16:48:44.910950+0000","last_became_active":"2023-11-16T16:47:17.626029+0000","last_became_peered":"2023-11-16T16:47:17.626029+0000","last_unstale":"2023-11-16T16:48:44.910950+0000","last_undegraded":"2023-11-16T16:48:44.910950+0000","last_fullsized":"2023-11-16T16:48:44.910950+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:55:41.716453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579606+0000","last_change":"2023-11-16T16:47:18.177153+0000","last_active":"2023-11-16T16:49:12.579606+0000","last_peered":"2023-11-16T16:49:12.579606+0000","last_clean":"2023-11-16T16:49:12.579606+0000","last_became_active":"2023-11-16T16:47:18.176751+0000","last_became_peered":"2023-11-16T16:47:18.176751+0000","last_unstale":"2023-11-16T16:49:12.579606+0000","last_undegraded":"2023-11-16T16:49:12.579606+0000","last_fullsized":"2023-11-16T16:49:12.579606+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:21:22.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910720+0000","last_change":"2023-11-16T16:47:22.652718+0000","last_active":"2023-11-16T16:48:44.910720+0000","last_peered":"2023-11-16T16:48:44.910720+0000","last_clean":"2023-11-16T16:48:44.910720+0000","last_became_active":"2023-11-16T16:47:22.652278+0000","last_became_peered":"2023-11-16T16:47:22.652278+0000","last_unstale":"2023-11-16T16:48:44.910720+0000","last_undegraded":"2023-11-16T16:48:44.910720+0000","last_fullsized":"2023-11-16T16:48:44.910720+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:34:31.220318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.627489+0000","last_change":"2023-11-16T16:47:17.627308+0000","last_active":"2023-11-16T16:48:44.627489+0000","last_peered":"2023-11-16T16:48:44.627489+0000","last_clean":"2023-11-16T16:48:44.627489+0000","last_became_active":"2023-11-16T16:47:17.626992+0000","last_became_peered":"2023-11-16T16:47:17.626992+0000","last_unstale":"2023-11-16T16:48:44.627489+0000","last_undegraded":"2023-11-16T16:48:44.627489+0000","last_fullsized":"2023-11-16T16:48:44.627489+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:01:05.366137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579997+0000","last_change":"2023-11-16T16:47:18.188015+0000","last_active":"2023-11-16T16:49:12.579997+0000","last_peered":"2023-11-16T16:49:12.579997+0000","last_clean":"2023-11-16T16:49:12.579997+0000","last_became_active":"2023-11-16T16:47:18.187730+0000","last_became_peered":"2023-11-16T16:47:18.187730+0000","last_unstale":"2023-11-16T16:49:12.579997+0000","last_undegraded":"2023-11-16T16:49:12.579997+0000","last_fullsized":"2023-11-16T16:49:12.579997+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:43:39.411790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.625358+0000","last_change":"2023-11-16T16:47:22.655656+0000","last_active":"2023-11-16T16:48:44.625358+0000","last_peered":"2023-11-16T16:48:44.625358+0000","last_clean":"2023-11-16T16:48:44.625358+0000","last_became_active":"2023-11-16T16:47:22.655133+0000","last_became_peered":"2023-11-16T16:47:22.655133+0000","last_unstale":"2023-11-16T16:48:44.625358+0000","last_undegraded":"2023-11-16T16:48:44.625358+0000","last_fullsized":"2023-11-16T16:48:44.625358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:57:06.052661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626104+0000","last_change":"2023-11-16T16:47:43.947947+0000","last_active":"2023-11-16T16:48:44.626104+0000","last_peered":"2023-11-16T16:48:44.626104+0000","last_clean":"2023-11-16T16:48:44.626104+0000","last_became_active":"2023-11-16T16:47:43.947478+0000","last_became_peered":"2023-11-16T16:47:43.947478+0000","last_unstale":"2023-11-16T16:48:44.626104+0000","last_undegraded":"2023-11-16T16:48:44.626104+0000","last_fullsized":"2023-11-16T16:48:44.626104+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:31:17.754660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.590686+0000","last_change":"2023-11-16T16:47:43.632657+0000","last_active":"2023-11-16T16:49:12.590686+0000","last_peered":"2023-11-16T16:49:12.590686+0000","last_clean":"2023-11-16T16:49:12.590686+0000","last_became_active":"2023-11-16T16:47:43.631573+0000","last_became_peered":"2023-11-16T16:47:43.631573+0000","last_unstale":"2023-11-16T16:49:12.590686+0000","last_undegraded":"2023-11-16T16:49:12.590686+0000","last_fullsized":"2023-11-16T16:49:12.590686+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:22:12.311807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623185+0000","last_change":"2023-11-16T16:47:22.661639+0000","last_active":"2023-11-16T16:48:44.623185+0000","last_peered":"2023-11-16T16:48:44.623185+0000","last_clean":"2023-11-16T16:48:44.623185+0000","last_became_active":"2023-11-16T16:47:22.661230+0000","last_became_peered":"2023-11-16T16:47:22.661230+0000","last_unstale":"2023-11-16T16:48:44.623185+0000","last_undegraded":"2023-11-16T16:48:44.623185+0000","last_fullsized":"2023-11-16T16:48:44.623185+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:25:46.012584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"82'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911151+0000","last_change":"2023-11-16T16:47:17.638209+0000","last_active":"2023-11-16T16:48:44.911151+0000","last_peered":"2023-11-16T16:48:44.911151+0000","last_clean":"2023-11-16T16:48:44.911151+0000","last_became_active":"2023-11-16T16:47:17.637957+0000","last_became_peered":"2023-11-16T16:47:17.637957+0000","last_unstale":"2023-11-16T16:48:44.911151+0000","last_undegraded":"2023-11-16T16:48:44.911151+0000","last_fullsized":"2023-11-16T16:48:44.911151+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:46:12.089473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.580155+0000","last_change":"2023-11-16T16:47:18.182693+0000","last_active":"2023-11-16T16:49:12.580155+0000","last_peered":"2023-11-16T16:49:12.580155+0000","last_clean":"2023-11-16T16:49:12.580155+0000","last_became_active":"2023-11-16T16:47:18.182370+0000","last_became_peered":"2023-11-16T16:47:18.182370+0000","last_unstale":"2023-11-16T16:49:12.580155+0000","last_undegraded":"2023-11-16T16:49:12.580155+0000","last_fullsized":"2023-11-16T16:49:12.580155+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:52.028156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.625577+0000","last_change":"2023-11-16T16:47:22.649348+0000","last_active":"2023-11-16T16:48:44.625577+0000","last_peered":"2023-11-16T16:48:44.625577+0000","last_clean":"2023-11-16T16:48:44.625577+0000","last_became_active":"2023-11-16T16:47:22.648860+0000","last_became_peered":"2023-11-16T16:47:22.648860+0000","last_unstale":"2023-11-16T16:48:44.625577+0000","last_undegraded":"2023-11-16T16:48:44.625577+0000","last_fullsized":"2023-11-16T16:48:44.625577+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:13:04.049776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'37","reported_seq":301,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:39.352456+0000","last_change":"2023-11-16T16:47:17.619417+0000","last_active":"2023-11-16T16:49:39.352456+0000","last_peered":"2023-11-16T16:49:39.352456+0000","last_clean":"2023-11-16T16:49:39.352456+0000","last_became_active":"2023-11-16T16:47:17.619086+0000","last_became_peered":"2023-11-16T16:47:17.619086+0000","last_unstale":"2023-11-16T16:49:39.352456+0000","last_undegraded":"2023-11-16T16:49:39.352456+0000","last_fullsized":"2023-11-16T16:49:39.352456+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:47:53.380456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66852,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":233,"num_read_kb":184,"num_write":37,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.566501+0000","last_change":"2023-11-16T16:47:18.177537+0000","last_active":"2023-11-16T16:49:12.566501+0000","last_peered":"2023-11-16T16:49:12.566501+0000","last_clean":"2023-11-16T16:49:12.566501+0000","last_became_active":"2023-11-16T16:47:18.177112+0000","last_became_peered":"2023-11-16T16:47:18.177112+0000","last_unstale":"2023-11-16T16:49:12.566501+0000","last_undegraded":"2023-11-16T16:49:12.566501+0000","last_fullsized":"2023-11-16T16:49:12.566501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:46:36.459769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911616+0000","last_change":"2023-11-16T16:47:22.658086+0000","last_active":"2023-11-16T16:48:44.911616+0000","last_peered":"2023-11-16T16:48:44.911616+0000","last_clean":"2023-11-16T16:48:44.911616+0000","last_became_active":"2023-11-16T16:47:22.657725+0000","last_became_peered":"2023-11-16T16:47:22.657725+0000","last_unstale":"2023-11-16T16:48:44.911616+0000","last_undegraded":"2023-11-16T16:48:44.911616+0000","last_fullsized":"2023-11-16T16:48:44.911616+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:20:10.492766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"82'2","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623489+0000","last_change":"2023-11-16T16:47:17.625405+0000","last_active":"2023-11-16T16:48:44.623489+0000","last_peered":"2023-11-16T16:48:44.623489+0000","last_clean":"2023-11-16T16:48:44.623489+0000","last_became_active":"2023-11-16T16:47:17.625026+0000","last_became_peered":"2023-11-16T16:47:17.625026+0000","last_unstale":"2023-11-16T16:48:44.623489+0000","last_undegraded":"2023-11-16T16:48:44.623489+0000","last_fullsized":"2023-11-16T16:48:44.623489+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:07:58.847671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.582795+0000","last_change":"2023-11-16T16:47:18.196345+0000","last_active":"2023-11-16T16:49:12.582795+0000","last_peered":"2023-11-16T16:49:12.582795+0000","last_clean":"2023-11-16T16:49:12.582795+0000","last_became_active":"2023-11-16T16:47:18.196040+0000","last_became_peered":"2023-11-16T16:47:18.196040+0000","last_unstale":"2023-11-16T16:49:12.582795+0000","last_undegraded":"2023-11-16T16:49:12.582795+0000","last_fullsized":"2023-11-16T16:49:12.582795+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:07:23.062566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.914613+0000","last_change":"2023-11-16T16:48:44.914613+0000","last_active":"2023-11-16T16:48:44.914613+0000","last_peered":"2023-11-16T16:48:44.914613+0000","last_clean":"2023-11-16T16:48:44.914613+0000","last_became_active":"2023-11-16T16:48:44.914216+0000","last_became_peered":"2023-11-16T16:48:44.914216+0000","last_unstale":"2023-11-16T16:48:44.914613+0000","last_undegraded":"2023-11-16T16:48:44.914613+0000","last_fullsized":"2023-11-16T16:48:44.914613+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:40:27.544928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911496+0000","last_change":"2023-11-16T16:47:17.630481+0000","last_active":"2023-11-16T16:48:44.911496+0000","last_peered":"2023-11-16T16:48:44.911496+0000","last_clean":"2023-11-16T16:48:44.911496+0000","last_became_active":"2023-11-16T16:47:17.630190+0000","last_became_peered":"2023-11-16T16:47:17.630190+0000","last_unstale":"2023-11-16T16:48:44.911496+0000","last_undegraded":"2023-11-16T16:48:44.911496+0000","last_fullsized":"2023-11-16T16:48:44.911496+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:22:25.493507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.585638+0000","last_change":"2023-11-16T16:48:44.913061+0000","last_active":"2023-11-16T16:49:12.585638+0000","last_peered":"2023-11-16T16:49:12.585638+0000","last_clean":"2023-11-16T16:49:12.585638+0000","last_became_active":"2023-11-16T16:48:44.912616+0000","last_became_peered":"2023-11-16T16:48:44.912616+0000","last_unstale":"2023-11-16T16:49:12.585638+0000","last_undegraded":"2023-11-16T16:49:12.585638+0000","last_fullsized":"2023-11-16T16:49:12.585638+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:12:42.146437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626024+0000","last_change":"2023-11-16T16:47:22.656312+0000","last_active":"2023-11-16T16:48:44.626024+0000","last_peered":"2023-11-16T16:48:44.626024+0000","last_clean":"2023-11-16T16:48:44.626024+0000","last_became_active":"2023-11-16T16:47:22.655870+0000","last_became_peered":"2023-11-16T16:47:22.655870+0000","last_unstale":"2023-11-16T16:48:44.626024+0000","last_undegraded":"2023-11-16T16:48:44.626024+0000","last_fullsized":"2023-11-16T16:48:44.626024+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:26:18.515418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.622957+0000","last_change":"2023-11-16T16:47:17.622648+0000","last_active":"2023-11-16T16:48:44.622957+0000","last_peered":"2023-11-16T16:48:44.622957+0000","last_clean":"2023-11-16T16:48:44.622957+0000","last_became_active":"2023-11-16T16:47:17.622329+0000","last_became_peered":"2023-11-16T16:47:17.622329+0000","last_unstale":"2023-11-16T16:48:44.622957+0000","last_undegraded":"2023-11-16T16:48:44.622957+0000","last_fullsized":"2023-11-16T16:48:44.622957+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:04:26.204200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.569410+0000","last_change":"2023-11-16T16:47:18.179712+0000","last_active":"2023-11-16T16:49:12.569410+0000","last_peered":"2023-11-16T16:49:12.569410+0000","last_clean":"2023-11-16T16:49:12.569410+0000","last_became_active":"2023-11-16T16:47:18.179392+0000","last_became_peered":"2023-11-16T16:47:18.179392+0000","last_unstale":"2023-11-16T16:49:12.569410+0000","last_undegraded":"2023-11-16T16:49:12.569410+0000","last_fullsized":"2023-11-16T16:49:12.569410+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:14:52.593975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.912858+0000","last_change":"2023-11-16T16:47:22.665088+0000","last_active":"2023-11-16T16:48:44.912858+0000","last_peered":"2023-11-16T16:48:44.912858+0000","last_clean":"2023-11-16T16:48:44.912858+0000","last_became_active":"2023-11-16T16:47:22.662802+0000","last_became_peered":"2023-11-16T16:47:22.662802+0000","last_unstale":"2023-11-16T16:48:44.912858+0000","last_undegraded":"2023-11-16T16:48:44.912858+0000","last_fullsized":"2023-11-16T16:48:44.912858+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:30:36.233143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"82'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628107+0000","last_change":"2023-11-16T16:47:17.639246+0000","last_active":"2023-11-16T16:48:44.628107+0000","last_peered":"2023-11-16T16:48:44.628107+0000","last_clean":"2023-11-16T16:48:44.628107+0000","last_became_active":"2023-11-16T16:47:17.638928+0000","last_became_peered":"2023-11-16T16:47:17.638928+0000","last_unstale":"2023-11-16T16:48:44.628107+0000","last_undegraded":"2023-11-16T16:48:44.628107+0000","last_fullsized":"2023-11-16T16:48:44.628107+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:18:59.651350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.568411+0000","last_change":"2023-11-16T16:47:18.182609+0000","last_active":"2023-11-16T16:49:12.568411+0000","last_peered":"2023-11-16T16:49:12.568411+0000","last_clean":"2023-11-16T16:49:12.568411+0000","last_became_active":"2023-11-16T16:47:18.182260+0000","last_became_peered":"2023-11-16T16:47:18.182260+0000","last_unstale":"2023-11-16T16:49:12.568411+0000","last_undegraded":"2023-11-16T16:49:12.568411+0000","last_fullsized":"2023-11-16T16:49:12.568411+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:31:25.530504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911115+0000","last_change":"2023-11-16T16:47:22.647586+0000","last_active":"2023-11-16T16:48:44.911115+0000","last_peered":"2023-11-16T16:48:44.911115+0000","last_clean":"2023-11-16T16:48:44.911115+0000","last_became_active":"2023-11-16T16:47:22.647187+0000","last_became_peered":"2023-11-16T16:47:22.647187+0000","last_unstale":"2023-11-16T16:48:44.911115+0000","last_undegraded":"2023-11-16T16:48:44.911115+0000","last_fullsized":"2023-11-16T16:48:44.911115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:29:19.788377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628288+0000","last_change":"2023-11-16T16:47:43.933501+0000","last_active":"2023-11-16T16:48:44.628288+0000","last_peered":"2023-11-16T16:48:44.628288+0000","last_clean":"2023-11-16T16:48:44.628288+0000","last_became_active":"2023-11-16T16:47:43.631746+0000","last_became_peered":"2023-11-16T16:47:43.631746+0000","last_unstale":"2023-11-16T16:48:44.628288+0000","last_undegraded":"2023-11-16T16:48:44.628288+0000","last_fullsized":"2023-11-16T16:48:44.628288+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:01:43.174333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.567503+0000","last_change":"2023-11-16T16:47:18.175210+0000","last_active":"2023-11-16T16:49:12.567503+0000","last_peered":"2023-11-16T16:49:12.567503+0000","last_clean":"2023-11-16T16:49:12.567503+0000","last_became_active":"2023-11-16T16:47:18.174926+0000","last_became_peered":"2023-11-16T16:47:18.174926+0000","last_unstale":"2023-11-16T16:49:12.567503+0000","last_undegraded":"2023-11-16T16:49:12.567503+0000","last_fullsized":"2023-11-16T16:49:12.567503+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:51:21.318906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628055+0000","last_change":"2023-11-16T16:47:22.719361+0000","last_active":"2023-11-16T16:48:44.628055+0000","last_peered":"2023-11-16T16:48:44.628055+0000","last_clean":"2023-11-16T16:48:44.628055+0000","last_became_active":"2023-11-16T16:47:22.718936+0000","last_became_peered":"2023-11-16T16:47:22.718936+0000","last_unstale":"2023-11-16T16:48:44.628055+0000","last_undegraded":"2023-11-16T16:48:44.628055+0000","last_fullsized":"2023-11-16T16:48:44.628055+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:07:12.278332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626068+0000","last_change":"2023-11-16T16:47:17.632623+0000","last_active":"2023-11-16T16:48:44.626068+0000","last_peered":"2023-11-16T16:48:44.626068+0000","last_clean":"2023-11-16T16:48:44.626068+0000","last_became_active":"2023-11-16T16:47:17.631323+0000","last_became_peered":"2023-11-16T16:47:17.631323+0000","last_unstale":"2023-11-16T16:48:44.626068+0000","last_undegraded":"2023-11-16T16:48:44.626068+0000","last_fullsized":"2023-11-16T16:48:44.626068+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:15:53.760791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.584748+0000","last_change":"2023-11-16T16:47:18.190779+0000","last_active":"2023-11-16T16:49:12.584748+0000","last_peered":"2023-11-16T16:49:12.584748+0000","last_clean":"2023-11-16T16:49:12.584748+0000","last_became_active":"2023-11-16T16:47:18.190440+0000","last_became_peered":"2023-11-16T16:47:18.190440+0000","last_unstale":"2023-11-16T16:49:12.584748+0000","last_undegraded":"2023-11-16T16:49:12.584748+0000","last_fullsized":"2023-11-16T16:49:12.584748+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:42:49.621136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911253+0000","last_change":"2023-11-16T16:47:22.664847+0000","last_active":"2023-11-16T16:48:44.911253+0000","last_peered":"2023-11-16T16:48:44.911253+0000","last_clean":"2023-11-16T16:48:44.911253+0000","last_became_active":"2023-11-16T16:47:22.659382+0000","last_became_peered":"2023-11-16T16:47:22.659382+0000","last_unstale":"2023-11-16T16:48:44.911253+0000","last_undegraded":"2023-11-16T16:48:44.911253+0000","last_fullsized":"2023-11-16T16:48:44.911253+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:25:48.404036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.625730+0000","last_change":"2023-11-16T16:47:17.630956+0000","last_active":"2023-11-16T16:48:44.625730+0000","last_peered":"2023-11-16T16:48:44.625730+0000","last_clean":"2023-11-16T16:48:44.625730+0000","last_became_active":"2023-11-16T16:47:17.629864+0000","last_became_peered":"2023-11-16T16:47:17.629864+0000","last_unstale":"2023-11-16T16:48:44.625730+0000","last_undegraded":"2023-11-16T16:48:44.625730+0000","last_fullsized":"2023-11-16T16:48:44.625730+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:34:09.169015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.581549+0000","last_change":"2023-11-16T16:47:18.183375+0000","last_active":"2023-11-16T16:49:12.581549+0000","last_peered":"2023-11-16T16:49:12.581549+0000","last_clean":"2023-11-16T16:49:12.581549+0000","last_became_active":"2023-11-16T16:47:18.183050+0000","last_became_peered":"2023-11-16T16:47:18.183050+0000","last_unstale":"2023-11-16T16:49:12.581549+0000","last_undegraded":"2023-11-16T16:49:12.581549+0000","last_fullsized":"2023-11-16T16:49:12.581549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:13:59.684725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911824+0000","last_change":"2023-11-16T16:47:22.719109+0000","last_active":"2023-11-16T16:48:44.911824+0000","last_peered":"2023-11-16T16:48:44.911824+0000","last_clean":"2023-11-16T16:48:44.911824+0000","last_became_active":"2023-11-16T16:47:22.718680+0000","last_became_peered":"2023-11-16T16:47:22.718680+0000","last_unstale":"2023-11-16T16:48:44.911824+0000","last_undegraded":"2023-11-16T16:48:44.911824+0000","last_fullsized":"2023-11-16T16:48:44.911824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:23:29.879756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.569880+0000","last_change":"2023-11-16T16:47:18.181632+0000","last_active":"2023-11-16T16:49:12.569880+0000","last_peered":"2023-11-16T16:49:12.569880+0000","last_clean":"2023-11-16T16:49:12.569880+0000","last_became_active":"2023-11-16T16:47:18.181323+0000","last_became_peered":"2023-11-16T16:47:18.181323+0000","last_unstale":"2023-11-16T16:49:12.569880+0000","last_undegraded":"2023-11-16T16:49:12.569880+0000","last_fullsized":"2023-11-16T16:49:12.569880+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:02:57.803797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910665+0000","last_change":"2023-11-16T16:47:17.637269+0000","last_active":"2023-11-16T16:48:44.910665+0000","last_peered":"2023-11-16T16:48:44.910665+0000","last_clean":"2023-11-16T16:48:44.910665+0000","last_became_active":"2023-11-16T16:47:17.636881+0000","last_became_peered":"2023-11-16T16:47:17.636881+0000","last_unstale":"2023-11-16T16:48:44.910665+0000","last_undegraded":"2023-11-16T16:48:44.910665+0000","last_fullsized":"2023-11-16T16:48:44.910665+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:15:29.077370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.626370+0000","last_change":"2023-11-16T16:47:22.659528+0000","last_active":"2023-11-16T16:48:44.626370+0000","last_peered":"2023-11-16T16:48:44.626370+0000","last_clean":"2023-11-16T16:48:44.626370+0000","last_became_active":"2023-11-16T16:47:22.659076+0000","last_became_peered":"2023-11-16T16:47:22.659076+0000","last_unstale":"2023-11-16T16:48:44.626370+0000","last_undegraded":"2023-11-16T16:48:44.626370+0000","last_fullsized":"2023-11-16T16:48:44.626370+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:22:26.166083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579137+0000","last_change":"2023-11-16T16:47:18.182102+0000","last_active":"2023-11-16T16:49:12.579137+0000","last_peered":"2023-11-16T16:49:12.579137+0000","last_clean":"2023-11-16T16:49:12.579137+0000","last_became_active":"2023-11-16T16:47:18.181528+0000","last_became_peered":"2023-11-16T16:47:18.181528+0000","last_unstale":"2023-11-16T16:49:12.579137+0000","last_undegraded":"2023-11-16T16:49:12.579137+0000","last_fullsized":"2023-11-16T16:49:12.579137+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:07:02.787319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623090+0000","last_change":"2023-11-16T16:47:17.626655+0000","last_active":"2023-11-16T16:48:44.623090+0000","last_peered":"2023-11-16T16:48:44.623090+0000","last_clean":"2023-11-16T16:48:44.623090+0000","last_became_active":"2023-11-16T16:47:17.626329+0000","last_became_peered":"2023-11-16T16:47:17.626329+0000","last_unstale":"2023-11-16T16:48:44.623090+0000","last_undegraded":"2023-11-16T16:48:44.623090+0000","last_fullsized":"2023-11-16T16:48:44.623090+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:22:13.860674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":357,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.625369+0000","last_change":"2023-11-16T16:46:11.943398+0000","last_active":"2023-11-16T16:48:44.625369+0000","last_peered":"2023-11-16T16:48:44.625369+0000","last_clean":"2023-11-16T16:48:44.625369+0000","last_became_active":"2023-11-16T16:46:11.942862+0000","last_became_peered":"2023-11-16T16:46:11.942862+0000","last_unstale":"2023-11-16T16:48:44.625369+0000","last_undegraded":"2023-11-16T16:48:44.625369+0000","last_fullsized":"2023-11-16T16:48:44.625369+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:43:03.109797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:43:03.109797+0000","last_clean_scrub_stamp":"2023-11-16T16:43:03.109797+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:27:18.973668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.915360+0000","last_change":"2023-11-16T16:48:44.914761+0000","last_active":"2023-11-16T16:48:44.915360+0000","last_peered":"2023-11-16T16:48:44.915360+0000","last_clean":"2023-11-16T16:48:44.915360+0000","last_became_active":"2023-11-16T16:48:44.914365+0000","last_became_peered":"2023-11-16T16:48:44.914365+0000","last_unstale":"2023-11-16T16:48:44.915360+0000","last_undegraded":"2023-11-16T16:48:44.915360+0000","last_fullsized":"2023-11-16T16:48:44.915360+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:54:07.544631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579927+0000","last_change":"2023-11-16T16:47:18.184630+0000","last_active":"2023-11-16T16:49:12.579927+0000","last_peered":"2023-11-16T16:49:12.579927+0000","last_clean":"2023-11-16T16:49:12.579927+0000","last_became_active":"2023-11-16T16:47:18.181978+0000","last_became_peered":"2023-11-16T16:47:18.181978+0000","last_unstale":"2023-11-16T16:49:12.579927+0000","last_undegraded":"2023-11-16T16:49:12.579927+0000","last_fullsized":"2023-11-16T16:49:12.579927+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:18:09.046316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628159+0000","last_change":"2023-11-16T16:47:43.635365+0000","last_active":"2023-11-16T16:48:44.628159+0000","last_peered":"2023-11-16T16:48:44.628159+0000","last_clean":"2023-11-16T16:48:44.628159+0000","last_became_active":"2023-11-16T16:47:43.634974+0000","last_became_peered":"2023-11-16T16:47:43.634974+0000","last_unstale":"2023-11-16T16:48:44.628159+0000","last_undegraded":"2023-11-16T16:48:44.628159+0000","last_fullsized":"2023-11-16T16:48:44.628159+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:34:17.420977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.627888+0000","last_change":"2023-11-16T16:47:22.657133+0000","last_active":"2023-11-16T16:48:44.627888+0000","last_peered":"2023-11-16T16:48:44.627888+0000","last_clean":"2023-11-16T16:48:44.627888+0000","last_became_active":"2023-11-16T16:47:22.656557+0000","last_became_peered":"2023-11-16T16:47:22.656557+0000","last_unstale":"2023-11-16T16:48:44.627888+0000","last_undegraded":"2023-11-16T16:48:44.627888+0000","last_fullsized":"2023-11-16T16:48:44.627888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:50:31.162768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.629935+0000","last_change":"2023-11-16T16:47:17.639107+0000","last_active":"2023-11-16T16:48:44.629935+0000","last_peered":"2023-11-16T16:48:44.629935+0000","last_clean":"2023-11-16T16:48:44.629935+0000","last_became_active":"2023-11-16T16:47:17.638796+0000","last_became_peered":"2023-11-16T16:47:17.638796+0000","last_unstale":"2023-11-16T16:48:44.629935+0000","last_undegraded":"2023-11-16T16:48:44.629935+0000","last_fullsized":"2023-11-16T16:48:44.629935+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:55:19.889558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.581882+0000","last_change":"2023-11-16T16:47:18.186761+0000","last_active":"2023-11-16T16:49:12.581882+0000","last_peered":"2023-11-16T16:49:12.581882+0000","last_clean":"2023-11-16T16:49:12.581882+0000","last_became_active":"2023-11-16T16:47:18.185594+0000","last_became_peered":"2023-11-16T16:47:18.185594+0000","last_unstale":"2023-11-16T16:49:12.581882+0000","last_undegraded":"2023-11-16T16:49:12.581882+0000","last_fullsized":"2023-11-16T16:49:12.581882+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:29:09.490006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.912099+0000","last_change":"2023-11-16T16:47:22.719534+0000","last_active":"2023-11-16T16:48:44.912099+0000","last_peered":"2023-11-16T16:48:44.912099+0000","last_clean":"2023-11-16T16:48:44.912099+0000","last_became_active":"2023-11-16T16:47:22.718738+0000","last_became_peered":"2023-11-16T16:47:22.718738+0000","last_unstale":"2023-11-16T16:48:44.912099+0000","last_undegraded":"2023-11-16T16:48:44.912099+0000","last_fullsized":"2023-11-16T16:48:44.912099+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:48:26.395670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'5","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:17.091818+0000","last_change":"2023-11-16T16:47:17.630318+0000","last_active":"2023-11-16T16:49:17.091818+0000","last_peered":"2023-11-16T16:49:17.091818+0000","last_clean":"2023-11-16T16:49:17.091818+0000","last_became_active":"2023-11-16T16:47:17.629035+0000","last_became_peered":"2023-11-16T16:47:17.629035+0000","last_unstale":"2023-11-16T16:49:17.091818+0000","last_undegraded":"2023-11-16T16:49:17.091818+0000","last_fullsized":"2023-11-16T16:49:17.091818+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:34:15.724433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.574220+0000","last_change":"2023-11-16T16:47:18.185034+0000","last_active":"2023-11-16T16:49:12.574220+0000","last_peered":"2023-11-16T16:49:12.574220+0000","last_clean":"2023-11-16T16:49:12.574220+0000","last_became_active":"2023-11-16T16:47:18.181961+0000","last_became_peered":"2023-11-16T16:47:18.181961+0000","last_unstale":"2023-11-16T16:49:12.574220+0000","last_undegraded":"2023-11-16T16:49:12.574220+0000","last_fullsized":"2023-11-16T16:49:12.574220+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:51:00.509275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623131+0000","last_change":"2023-11-16T16:47:22.649952+0000","last_active":"2023-11-16T16:48:44.623131+0000","last_peered":"2023-11-16T16:48:44.623131+0000","last_clean":"2023-11-16T16:48:44.623131+0000","last_became_active":"2023-11-16T16:47:22.649481+0000","last_became_peered":"2023-11-16T16:47:22.649481+0000","last_unstale":"2023-11-16T16:48:44.623131+0000","last_undegraded":"2023-11-16T16:48:44.623131+0000","last_fullsized":"2023-11-16T16:48:44.623131+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:52:18.082684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"82'2","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623572+0000","last_change":"2023-11-16T16:47:17.631982+0000","last_active":"2023-11-16T16:48:44.623572+0000","last_peered":"2023-11-16T16:48:44.623572+0000","last_clean":"2023-11-16T16:48:44.623572+0000","last_became_active":"2023-11-16T16:47:17.631552+0000","last_became_peered":"2023-11-16T16:47:17.631552+0000","last_unstale":"2023-11-16T16:48:44.623572+0000","last_undegraded":"2023-11-16T16:48:44.623572+0000","last_fullsized":"2023-11-16T16:48:44.623572+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:23:15.968583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.589149+0000","last_change":"2023-11-16T16:47:18.197325+0000","last_active":"2023-11-16T16:49:12.589149+0000","last_peered":"2023-11-16T16:49:12.589149+0000","last_clean":"2023-11-16T16:49:12.589149+0000","last_became_active":"2023-11-16T16:47:18.190602+0000","last_became_peered":"2023-11-16T16:47:18.190602+0000","last_unstale":"2023-11-16T16:49:12.589149+0000","last_undegraded":"2023-11-16T16:49:12.589149+0000","last_fullsized":"2023-11-16T16:49:12.589149+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:21:06.387535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623556+0000","last_change":"2023-11-16T16:47:22.720023+0000","last_active":"2023-11-16T16:48:44.623556+0000","last_peered":"2023-11-16T16:48:44.623556+0000","last_clean":"2023-11-16T16:48:44.623556+0000","last_became_active":"2023-11-16T16:47:22.719561+0000","last_became_peered":"2023-11-16T16:47:22.719561+0000","last_unstale":"2023-11-16T16:48:44.623556+0000","last_undegraded":"2023-11-16T16:48:44.623556+0000","last_fullsized":"2023-11-16T16:48:44.623556+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:06:42.918943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.647594+0000","last_change":"2023-11-16T16:47:17.629379+0000","last_active":"2023-11-16T16:48:44.647594+0000","last_peered":"2023-11-16T16:48:44.647594+0000","last_clean":"2023-11-16T16:48:44.647594+0000","last_became_active":"2023-11-16T16:47:17.629025+0000","last_became_peered":"2023-11-16T16:47:17.629025+0000","last_unstale":"2023-11-16T16:48:44.647594+0000","last_undegraded":"2023-11-16T16:48:44.647594+0000","last_fullsized":"2023-11-16T16:48:44.647594+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:38:07.996802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.566023+0000","last_change":"2023-11-16T16:47:18.185418+0000","last_active":"2023-11-16T16:49:12.566023+0000","last_peered":"2023-11-16T16:49:12.566023+0000","last_clean":"2023-11-16T16:49:12.566023+0000","last_became_active":"2023-11-16T16:47:18.185092+0000","last_became_peered":"2023-11-16T16:47:18.185092+0000","last_unstale":"2023-11-16T16:49:12.566023+0000","last_undegraded":"2023-11-16T16:49:12.566023+0000","last_fullsized":"2023-11-16T16:49:12.566023+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:57:19.899302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.913330+0000","last_change":"2023-11-16T16:48:44.913330+0000","last_active":"2023-11-16T16:48:44.913330+0000","last_peered":"2023-11-16T16:48:44.913330+0000","last_clean":"2023-11-16T16:48:44.913330+0000","last_became_active":"2023-11-16T16:48:44.912707+0000","last_became_peered":"2023-11-16T16:48:44.912707+0000","last_unstale":"2023-11-16T16:48:44.913330+0000","last_undegraded":"2023-11-16T16:48:44.913330+0000","last_fullsized":"2023-11-16T16:48:44.913330+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:12:18.512977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910905+0000","last_change":"2023-11-16T16:47:17.638622+0000","last_active":"2023-11-16T16:48:44.910905+0000","last_peered":"2023-11-16T16:48:44.910905+0000","last_clean":"2023-11-16T16:48:44.910905+0000","last_became_active":"2023-11-16T16:47:17.638303+0000","last_became_peered":"2023-11-16T16:47:17.638303+0000","last_unstale":"2023-11-16T16:48:44.910905+0000","last_undegraded":"2023-11-16T16:48:44.910905+0000","last_fullsized":"2023-11-16T16:48:44.910905+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:21:41.090636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.590794+0000","last_change":"2023-11-16T16:47:18.173474+0000","last_active":"2023-11-16T16:49:12.590794+0000","last_peered":"2023-11-16T16:49:12.590794+0000","last_clean":"2023-11-16T16:49:12.590794+0000","last_became_active":"2023-11-16T16:47:18.173140+0000","last_became_peered":"2023-11-16T16:47:18.173140+0000","last_unstale":"2023-11-16T16:49:12.590794+0000","last_undegraded":"2023-11-16T16:49:12.590794+0000","last_fullsized":"2023-11-16T16:49:12.590794+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:30:41.318669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623128+0000","last_change":"2023-11-16T16:47:22.655806+0000","last_active":"2023-11-16T16:48:44.623128+0000","last_peered":"2023-11-16T16:48:44.623128+0000","last_clean":"2023-11-16T16:48:44.623128+0000","last_became_active":"2023-11-16T16:47:22.655385+0000","last_became_peered":"2023-11-16T16:47:22.655385+0000","last_unstale":"2023-11-16T16:48:44.623128+0000","last_undegraded":"2023-11-16T16:48:44.623128+0000","last_fullsized":"2023-11-16T16:48:44.623128+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:15:10.616521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623263+0000","last_change":"2023-11-16T16:47:17.631673+0000","last_active":"2023-11-16T16:48:44.623263+0000","last_peered":"2023-11-16T16:48:44.623263+0000","last_clean":"2023-11-16T16:48:44.623263+0000","last_became_active":"2023-11-16T16:47:17.631356+0000","last_became_peered":"2023-11-16T16:47:17.631356+0000","last_unstale":"2023-11-16T16:48:44.623263+0000","last_undegraded":"2023-11-16T16:48:44.623263+0000","last_fullsized":"2023-11-16T16:48:44.623263+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:05:33.004512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.589316+0000","last_change":"2023-11-16T16:47:18.186706+0000","last_active":"2023-11-16T16:49:12.589316+0000","last_peered":"2023-11-16T16:49:12.589316+0000","last_clean":"2023-11-16T16:49:12.589316+0000","last_became_active":"2023-11-16T16:47:18.186360+0000","last_became_peered":"2023-11-16T16:47:18.186360+0000","last_unstale":"2023-11-16T16:49:12.589316+0000","last_undegraded":"2023-11-16T16:49:12.589316+0000","last_fullsized":"2023-11-16T16:49:12.589316+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:46:42.071656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623436+0000","last_change":"2023-11-16T16:47:22.719065+0000","last_active":"2023-11-16T16:48:44.623436+0000","last_peered":"2023-11-16T16:48:44.623436+0000","last_clean":"2023-11-16T16:48:44.623436+0000","last_became_active":"2023-11-16T16:47:22.718662+0000","last_became_peered":"2023-11-16T16:47:22.718662+0000","last_unstale":"2023-11-16T16:48:44.623436+0000","last_undegraded":"2023-11-16T16:48:44.623436+0000","last_fullsized":"2023-11-16T16:48:44.623436+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:39:06.356934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"82'2","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.628612+0000","last_change":"2023-11-16T16:47:43.637790+0000","last_active":"2023-11-16T16:48:44.628612+0000","last_peered":"2023-11-16T16:48:44.628612+0000","last_clean":"2023-11-16T16:48:44.628612+0000","last_became_active":"2023-11-16T16:47:43.632225+0000","last_became_peered":"2023-11-16T16:47:43.632225+0000","last_unstale":"2023-11-16T16:48:44.628612+0000","last_undegraded":"2023-11-16T16:48:44.628612+0000","last_fullsized":"2023-11-16T16:48:44.628612+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:37:02.441726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.584025+0000","last_change":"2023-11-16T16:47:18.180999+0000","last_active":"2023-11-16T16:49:12.584025+0000","last_peered":"2023-11-16T16:49:12.584025+0000","last_clean":"2023-11-16T16:49:12.584025+0000","last_became_active":"2023-11-16T16:47:18.180724+0000","last_became_peered":"2023-11-16T16:47:18.180724+0000","last_unstale":"2023-11-16T16:49:12.584025+0000","last_undegraded":"2023-11-16T16:49:12.584025+0000","last_fullsized":"2023-11-16T16:49:12.584025+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:21:21.954717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.647762+0000","last_change":"2023-11-16T16:47:22.665607+0000","last_active":"2023-11-16T16:48:44.647762+0000","last_peered":"2023-11-16T16:48:44.647762+0000","last_clean":"2023-11-16T16:48:44.647762+0000","last_became_active":"2023-11-16T16:47:22.665273+0000","last_became_peered":"2023-11-16T16:47:22.665273+0000","last_unstale":"2023-11-16T16:48:44.647762+0000","last_undegraded":"2023-11-16T16:48:44.647762+0000","last_fullsized":"2023-11-16T16:48:44.647762+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:31:43.368003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.623867+0000","last_change":"2023-11-16T16:47:17.632614+0000","last_active":"2023-11-16T16:48:44.623867+0000","last_peered":"2023-11-16T16:48:44.623867+0000","last_clean":"2023-11-16T16:48:44.623867+0000","last_became_active":"2023-11-16T16:47:17.632227+0000","last_became_peered":"2023-11-16T16:47:17.632227+0000","last_unstale":"2023-11-16T16:48:44.623867+0000","last_undegraded":"2023-11-16T16:48:44.623867+0000","last_fullsized":"2023-11-16T16:48:44.623867+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:52:45.369452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579477+0000","last_change":"2023-11-16T16:47:18.177607+0000","last_active":"2023-11-16T16:49:12.579477+0000","last_peered":"2023-11-16T16:49:12.579477+0000","last_clean":"2023-11-16T16:49:12.579477+0000","last_became_active":"2023-11-16T16:47:18.177266+0000","last_became_peered":"2023-11-16T16:47:18.177266+0000","last_unstale":"2023-11-16T16:49:12.579477+0000","last_undegraded":"2023-11-16T16:49:12.579477+0000","last_fullsized":"2023-11-16T16:49:12.579477+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:45:38.139269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910928+0000","last_change":"2023-11-16T16:47:22.659987+0000","last_active":"2023-11-16T16:48:44.910928+0000","last_peered":"2023-11-16T16:48:44.910928+0000","last_clean":"2023-11-16T16:48:44.910928+0000","last_became_active":"2023-11-16T16:47:22.658725+0000","last_became_peered":"2023-11-16T16:47:22.658725+0000","last_unstale":"2023-11-16T16:48:44.910928+0000","last_undegraded":"2023-11-16T16:48:44.910928+0000","last_fullsized":"2023-11-16T16:48:44.910928+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:09:57.462778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.911469+0000","last_change":"2023-11-16T16:47:17.640696+0000","last_active":"2023-11-16T16:48:44.911469+0000","last_peered":"2023-11-16T16:48:44.911469+0000","last_clean":"2023-11-16T16:48:44.911469+0000","last_became_active":"2023-11-16T16:47:17.640428+0000","last_became_peered":"2023-11-16T16:47:17.640428+0000","last_unstale":"2023-11-16T16:48:44.911469+0000","last_undegraded":"2023-11-16T16:48:44.911469+0000","last_fullsized":"2023-11-16T16:48:44.911469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:06:54.539557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.583537+0000","last_change":"2023-11-16T16:47:43.972273+0000","last_active":"2023-11-16T16:49:12.583537+0000","last_peered":"2023-11-16T16:49:12.583537+0000","last_clean":"2023-11-16T16:49:12.583537+0000","last_became_active":"2023-11-16T16:47:43.971981+0000","last_became_peered":"2023-11-16T16:47:43.971981+0000","last_unstale":"2023-11-16T16:49:12.583537+0000","last_undegraded":"2023-11-16T16:49:12.583537+0000","last_fullsized":"2023-11-16T16:49:12.583537+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:07:07.722793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.910594+0000","last_change":"2023-11-16T16:47:22.649436+0000","last_active":"2023-11-16T16:48:44.910594+0000","last_peered":"2023-11-16T16:48:44.910594+0000","last_clean":"2023-11-16T16:48:44.910594+0000","last_became_active":"2023-11-16T16:47:22.649034+0000","last_became_peered":"2023-11-16T16:47:22.649034+0000","last_unstale":"2023-11-16T16:48:44.910594+0000","last_undegraded":"2023-11-16T16:48:44.910594+0000","last_fullsized":"2023-11-16T16:48:44.910594+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:21.621065+0000","last_clean_scrub_stamp":"2023-11-16T16:47:21.621065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:46:22.957484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:44.912764+0000","last_change":"2023-11-16T16:47:17.634582+0000","last_active":"2023-11-16T16:48:44.912764+0000","last_peered":"2023-11-16T16:48:44.912764+0000","last_clean":"2023-11-16T16:48:44.912764+0000","last_became_active":"2023-11-16T16:47:17.634172+0000","last_became_peered":"2023-11-16T16:47:17.634172+0000","last_unstale":"2023-11-16T16:48:44.912764+0000","last_undegraded":"2023-11-16T16:48:44.912764+0000","last_fullsized":"2023-11-16T16:48:44.912764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:16.582784+0000","last_clean_scrub_stamp":"2023-11-16T16:47:16.582784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:43:12.528888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.569892+0000","last_change":"2023-11-16T16:47:18.180466+0000","last_active":"2023-11-16T16:49:12.569892+0000","last_peered":"2023-11-16T16:49:12.569892+0000","last_clean":"2023-11-16T16:49:12.569892+0000","last_became_active":"2023-11-16T16:47:18.180166+0000","last_became_peered":"2023-11-16T16:47:18.180166+0000","last_unstale":"2023-11-16T16:49:12.569892+0000","last_undegraded":"2023-11-16T16:49:12.569892+0000","last_fullsized":"2023-11-16T16:49:12.569892+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:57:25.704442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.570479+0000","last_change":"2023-11-16T16:47:18.179196+0000","last_active":"2023-11-16T16:49:12.570479+0000","last_peered":"2023-11-16T16:49:12.570479+0000","last_clean":"2023-11-16T16:49:12.570479+0000","last_became_active":"2023-11-16T16:47:18.178827+0000","last_became_peered":"2023-11-16T16:47:18.178827+0000","last_unstale":"2023-11-16T16:49:12.570479+0000","last_undegraded":"2023-11-16T16:49:12.570479+0000","last_fullsized":"2023-11-16T16:49:12.570479+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:21:55.520738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.578735+0000","last_change":"2023-11-16T16:47:18.184630+0000","last_active":"2023-11-16T16:49:12.578735+0000","last_peered":"2023-11-16T16:49:12.578735+0000","last_clean":"2023-11-16T16:49:12.578735+0000","last_became_active":"2023-11-16T16:47:18.184314+0000","last_became_peered":"2023-11-16T16:47:18.184314+0000","last_unstale":"2023-11-16T16:49:12.578735+0000","last_undegraded":"2023-11-16T16:49:12.578735+0000","last_fullsized":"2023-11-16T16:49:12.578735+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:20:06.066404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.581521+0000","last_change":"2023-11-16T16:47:18.184255+0000","last_active":"2023-11-16T16:49:12.581521+0000","last_peered":"2023-11-16T16:49:12.581521+0000","last_clean":"2023-11-16T16:49:12.581521+0000","last_became_active":"2023-11-16T16:47:18.183936+0000","last_became_peered":"2023-11-16T16:47:18.183936+0000","last_unstale":"2023-11-16T16:49:12.581521+0000","last_undegraded":"2023-11-16T16:49:12.581521+0000","last_fullsized":"2023-11-16T16:49:12.581521+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:40:04.120978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.588698+0000","last_change":"2023-11-16T16:47:18.196331+0000","last_active":"2023-11-16T16:49:12.588698+0000","last_peered":"2023-11-16T16:49:12.588698+0000","last_clean":"2023-11-16T16:49:12.588698+0000","last_became_active":"2023-11-16T16:47:18.195954+0000","last_became_peered":"2023-11-16T16:47:18.195954+0000","last_unstale":"2023-11-16T16:49:12.588698+0000","last_undegraded":"2023-11-16T16:49:12.588698+0000","last_fullsized":"2023-11-16T16:49:12.588698+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:48:28.247082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.591835+0000","last_change":"2023-11-16T16:47:18.190537+0000","last_active":"2023-11-16T16:49:12.591835+0000","last_peered":"2023-11-16T16:49:12.591835+0000","last_clean":"2023-11-16T16:49:12.591835+0000","last_became_active":"2023-11-16T16:47:18.190118+0000","last_became_peered":"2023-11-16T16:47:18.190118+0000","last_unstale":"2023-11-16T16:49:12.591835+0000","last_undegraded":"2023-11-16T16:49:12.591835+0000","last_fullsized":"2023-11-16T16:49:12.591835+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:53:21.438815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.566299+0000","last_change":"2023-11-16T16:47:18.179215+0000","last_active":"2023-11-16T16:49:12.566299+0000","last_peered":"2023-11-16T16:49:12.566299+0000","last_clean":"2023-11-16T16:49:12.566299+0000","last_became_active":"2023-11-16T16:47:18.178759+0000","last_became_peered":"2023-11-16T16:47:18.178759+0000","last_unstale":"2023-11-16T16:49:12.566299+0000","last_undegraded":"2023-11-16T16:49:12.566299+0000","last_fullsized":"2023-11-16T16:49:12.566299+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:56:47.424865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.581180+0000","last_change":"2023-11-16T16:47:18.179588+0000","last_active":"2023-11-16T16:49:12.581180+0000","last_peered":"2023-11-16T16:49:12.581180+0000","last_clean":"2023-11-16T16:49:12.581180+0000","last_became_active":"2023-11-16T16:47:18.179294+0000","last_became_peered":"2023-11-16T16:47:18.179294+0000","last_unstale":"2023-11-16T16:49:12.581180+0000","last_undegraded":"2023-11-16T16:49:12.581180+0000","last_fullsized":"2023-11-16T16:49:12.581180+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:21:17.462988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.572426+0000","last_change":"2023-11-16T16:47:18.181587+0000","last_active":"2023-11-16T16:49:12.572426+0000","last_peered":"2023-11-16T16:49:12.572426+0000","last_clean":"2023-11-16T16:49:12.572426+0000","last_became_active":"2023-11-16T16:47:18.181203+0000","last_became_peered":"2023-11-16T16:47:18.181203+0000","last_unstale":"2023-11-16T16:49:12.572426+0000","last_undegraded":"2023-11-16T16:49:12.572426+0000","last_fullsized":"2023-11-16T16:49:12.572426+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:41:14.468221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.579568+0000","last_change":"2023-11-16T16:47:18.182690+0000","last_active":"2023-11-16T16:49:12.579568+0000","last_peered":"2023-11-16T16:49:12.579568+0000","last_clean":"2023-11-16T16:49:12.579568+0000","last_became_active":"2023-11-16T16:47:18.182361+0000","last_became_peered":"2023-11-16T16:47:18.182361+0000","last_unstale":"2023-11-16T16:49:12.579568+0000","last_undegraded":"2023-11-16T16:49:12.579568+0000","last_fullsized":"2023-11-16T16:49:12.579568+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:24:27.460239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.578706+0000","last_change":"2023-11-16T16:47:18.184599+0000","last_active":"2023-11-16T16:49:12.578706+0000","last_peered":"2023-11-16T16:49:12.578706+0000","last_clean":"2023-11-16T16:49:12.578706+0000","last_became_active":"2023-11-16T16:47:18.184288+0000","last_became_peered":"2023-11-16T16:47:18.184288+0000","last_unstale":"2023-11-16T16:49:12.578706+0000","last_undegraded":"2023-11-16T16:49:12.578706+0000","last_fullsized":"2023-11-16T16:49:12.578706+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:09:35.265387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.575511+0000","last_change":"2023-11-16T16:47:18.187267+0000","last_active":"2023-11-16T16:49:12.575511+0000","last_peered":"2023-11-16T16:49:12.575511+0000","last_clean":"2023-11-16T16:49:12.575511+0000","last_became_active":"2023-11-16T16:47:18.186984+0000","last_became_peered":"2023-11-16T16:47:18.186984+0000","last_unstale":"2023-11-16T16:49:12.575511+0000","last_undegraded":"2023-11-16T16:49:12.575511+0000","last_fullsized":"2023-11-16T16:49:12.575511+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:22:44.826690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.592202+0000","last_change":"2023-11-16T16:47:18.191566+0000","last_active":"2023-11-16T16:49:12.592202+0000","last_peered":"2023-11-16T16:49:12.592202+0000","last_clean":"2023-11-16T16:49:12.592202+0000","last_became_active":"2023-11-16T16:47:18.185342+0000","last_became_peered":"2023-11-16T16:47:18.185342+0000","last_unstale":"2023-11-16T16:49:12.592202+0000","last_undegraded":"2023-11-16T16:49:12.592202+0000","last_fullsized":"2023-11-16T16:49:12.592202+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:53:50.072132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:49:12.565174+0000","last_change":"2023-11-16T16:47:43.629636+0000","last_active":"2023-11-16T16:49:12.565174+0000","last_peered":"2023-11-16T16:49:12.565174+0000","last_clean":"2023-11-16T16:49:12.565174+0000","last_became_active":"2023-11-16T16:47:43.629179+0000","last_became_peered":"2023-11-16T16:47:43.629179+0000","last_unstale":"2023-11-16T16:49:12.565174+0000","last_undegraded":"2023-11-16T16:49:12.565174+0000","last_fullsized":"2023-11-16T16:49:12.565174+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:17.602263+0000","last_clean_scrub_stamp":"2023-11-16T16:47:17.602263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:20:45.082147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281399,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67650,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":239,"num_read_kb":190,"num_write":84,"num_write_kb":99,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":197571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449130,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6228,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738924,"statfs":{"total":95995035648,"available":95988658176,"internally_reserved":0,"allocated":995328,"data_stored":506898,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.41999999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.77400000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.57499999999999996,"15min":0.55700000000000005},"min":{"1min":0.46200000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79900000000000004,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.60799999999999998}]},{"osd":1,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59199999999999997,"15min":0.58699999999999997},"min":{"1min":0.505,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74099999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.50900000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.751,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.72799999999999998}]},{"osd":2,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.47499999999999998,"15min":0.46100000000000002},"min":{"1min":0.36799999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.47899999999999998,"15min":0.46999999999999997},"min":{"1min":0.39100000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.57299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.63500000000000001,"15min":0.627},"min":{"1min":0.52400000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.625,"15min":0.626},"min":{"1min":0.46100000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.41399999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.79500000000000004,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.38600000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.89200000000000002,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.46899999999999997}]},{"osd":6,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.67800000000000005,"15min":0.68999999999999995},"min":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.84299999999999997,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.623},{"interface":"front","average":{"1min":0.754,"5min":0.76500000000000001,"15min":0.73299999999999998},"min":{"1min":0.54300000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.93000000000000005,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":0.68400000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.56399999999999995,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95599999999999996,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.73299999999999998,"15min":0.73599999999999999},"min":{"1min":0.56499999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.95699999999999996,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.69499999999999995}]},{"osd":8,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.72299999999999998,"15min":0.71699999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.93000000000000005,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.70499999999999996}]},{"osd":9,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.78500000000000003,"15min":0.76900000000000002},"min":{"1min":0.69999999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.042,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.98599999999999999,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.752}]},{"osd":10,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.76700000000000002,"15min":0.76900000000000002},"min":{"1min":0.59099999999999997,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.94999999999999996,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.78200000000000003,"15min":0.77000000000000002},"min":{"1min":0.69799999999999995,"5min":0.498,"15min":0.498},"max":{"1min":1.006,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.80200000000000005}]}]},{"osd":10,"up_from":71,"seq":304942678062,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6676,"kb_used_data":1100,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738476,"statfs":{"total":95995035648,"available":95988199424,"internally_reserved":0,"allocated":1126400,"data_stored":1160194,"data_compressed":59395,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:49:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.72899999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72799999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.26900000000000002}]},{"osd":1,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.26200000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60299999999999998,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.49099999999999999,"15min":0.51100000000000001},"min":{"1min":0.28299999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70799999999999996,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.247}]},{"osd":2,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.63400000000000001,"15min":0.66100000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.70499999999999996,"5min":1.577,"15min":1.577},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.63800000000000001,"15min":0.65800000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78300000000000003,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.29799999999999999}]},{"osd":3,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.623,"15min":0.64500000000000002},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80000000000000004,"5min":1.464,"15min":1.464},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.65400000000000003,"15min":0.67500000000000004},"min":{"1min":0.40400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76800000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.374}]},{"osd":4,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56399999999999995,"15min":0.58399999999999996},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73899999999999999,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54700000000000004,"15min":0.56799999999999995},"min":{"1min":0.372,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.752,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.30399999999999999}]},{"osd":5,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.68300000000000005,"15min":0.69899999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82899999999999996,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.68700000000000006,"15min":0.69899999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.91300000000000003,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.37}]},{"osd":6,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.73399999999999999,"15min":0.75900000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86299999999999999,"5min":1.524,"15min":1.524},"last":0.432},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76000000000000001,"15min":0.77400000000000002},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.85999999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.5}]},{"osd":7,"last update":"Thu Nov 16 16:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.69999999999999996}]},{"osd":8,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.81299999999999994,"15min":0.85499999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.91400000000000003,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.755,"15min":0.78300000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.93999999999999995,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.438}]},{"osd":9,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.84399999999999997,"15min":0.84099999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.92700000000000005,"5min":4.0129999999999999,"15min":4.0129999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.82199999999999995,"15min":0.83099999999999996},"min":{"1min":0.56299999999999994,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":4.5380000000000003,"5min":4.5380000000000003,"15min":4.5380000000000003},"last":0.46300000000000002}]},{"osd":11,"last update":"Thu Nov 16 16:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.78400000000000003,"15min":0.81799999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.96599999999999997,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.79100000000000004,"15min":0.80700000000000005},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.95599999999999996,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.48099999999999998}]}]},{"osd":9,"up_from":66,"seq":283467841585,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6424,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738728,"statfs":{"total":95995035648,"available":95988457472,"internally_reserved":0,"allocated":999424,"data_stored":511554,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.54400000000000004,"15min":0.56999999999999995},"min":{"1min":0.28399999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.57199999999999995,"15min":0.60399999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.40300000000000002}]},{"osd":1,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.78600000000000003,"15min":0.82499999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.89500000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.747,"15min":0.78200000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.53700000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.80100000000000005,"15min":0.83299999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0640000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.753,"15min":0.77700000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59099999999999997,"15min":0.61499999999999999},"min":{"1min":0.40899999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.746,"15min":0.80000000000000004},"min":{"1min":0.42299999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.2210000000000001,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.61099999999999999}]},{"osd":4,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.79600000000000004,"15min":0.85099999999999998},"min":{"1min":0.48799999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.167,"5min":1.298,"15min":1.298},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.73699999999999999,"15min":0.77300000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.94199999999999995,"5min":1.181,"15min":1.181},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.78800000000000003,"15min":0.82399999999999995},"min":{"1min":0.51200000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.081,"5min":1.268,"15min":1.268},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.90000000000000002,"15min":0.95299999999999996},"min":{"1min":0.621,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1040000000000001,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.76700000000000002}]},{"osd":6,"last update":"Thu Nov 16 16:46:50 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":1.054},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.95599999999999996}]},{"osd":7,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.84899999999999998,"15min":0.90300000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.99199999999999999,"5min":1.405,"15min":1.405},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.98099999999999998,"15min":1.0589999999999999},"min":{"1min":0.59799999999999998,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.133,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.748}]},{"osd":8,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.90200000000000002,"15min":0.96099999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.1970000000000001,"5min":1.363,"15min":1.363},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.90400000000000003,"15min":0.96399999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.149,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.68700000000000006}]},{"osd":10,"last update":"Thu Nov 16 16:49:15 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.88800000000000001,"15min":0.93899999999999995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.036,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.96499999999999997,"15min":1.0209999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1599999999999999,"5min":1.714,"15min":1.714},"last":0.79600000000000004}]},{"osd":11,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.80300000000000005,"15min":0.80800000000000005},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.038,"5min":1.115,"15min":1.115},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.89100000000000001,"15min":0.90600000000000003},"min":{"1min":0.67800000000000005,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.0600000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.82099999999999995}]}]},{"osd":8,"up_from":58,"seq":249108103223,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6548,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738604,"statfs":{"total":95995035648,"available":95988330496,"internally_reserved":0,"allocated":995328,"data_stored":507458,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.34300000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.79600000000000004,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.39200000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.86299999999999999,"5min":1.113,"15min":1.113},"last":0.436}]},{"osd":1,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63200000000000001,"15min":0.60399999999999998},"min":{"1min":0.51600000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76600000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.623,"15min":0.60999999999999999},"min":{"1min":0.40999999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.85099999999999998,"5min":1.177,"15min":1.177},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47299999999999998,"15min":0.45700000000000002},"min":{"1min":0.29699999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.73499999999999999,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47499999999999998,"15min":0.46600000000000003},"min":{"1min":0.38700000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.40000000000000002}]},{"osd":3,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.629,"15min":0.63},"min":{"1min":0.42799999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.90600000000000003,"5min":1.141,"15min":1.141},"last":0.496},{"interface":"front","average":{"1min":0.623,"5min":0.65500000000000003,"15min":0.64700000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.91800000000000004,"5min":1.502,"15min":1.502},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.64100000000000001,"15min":0.629},"min":{"1min":0.50700000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79800000000000004,"5min":0.999,"15min":0.999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69799999999999995,"15min":0.68799999999999994},"min":{"1min":0.54300000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83199999999999996,"5min":1.242,"15min":1.242},"last":0.48099999999999998}]},{"osd":5,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55800000000000005,"15min":0.53600000000000003},"min":{"1min":0.45900000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83699999999999997,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57599999999999996,"15min":0.56299999999999994},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68000000000000005,"5min":1.224,"15min":1.224},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.77600000000000002,"15min":0.73799999999999999},"min":{"1min":0.54900000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.89000000000000001,"5min":4.7309999999999999,"15min":4.7309999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.69599999999999995,"15min":0.68100000000000005},"min":{"1min":0.54200000000000004,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0149999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.56200000000000006}]},{"osd":7,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.71599999999999997,"15min":0.69499999999999995},"min":{"1min":0.60499999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90300000000000002,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.73899999999999999,"15min":0.72599999999999998},"min":{"1min":0.59699999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95599999999999996,"5min":1.198,"15min":1.198},"last":0.622}]},{"osd":9,"last update":"Thu Nov 16 16:48:51 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.76900000000000002,"15min":0.77000000000000002},"min":{"1min":0.52700000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.89100000000000001,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.80300000000000005,"15min":0.81100000000000005},"min":{"1min":0.60299999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.88500000000000001,"5min":1.298,"15min":1.298},"last":0.65400000000000003}]},{"osd":10,"last update":"Thu Nov 16 16:49:11 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78500000000000003,"15min":0.77700000000000002},"min":{"1min":0.70199999999999996,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.97699999999999998,"5min":1.222,"15min":1.222},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.81599999999999995,"15min":0.81999999999999995},"min":{"1min":0.66100000000000003,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.91900000000000004,"5min":1.274,"15min":1.274},"last":0.52200000000000002}]},{"osd":11,"last update":"Thu Nov 16 16:49:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.77300000000000002,"15min":0.77500000000000002},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.94499999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.77800000000000002,"15min":0.78400000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.875,"5min":1.05,"15min":1.05},"last":0.64400000000000002}]}]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6664,"kb_used_data":1024,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738488,"statfs":{"total":95995035648,"available":95988211712,"internally_reserved":0,"allocated":1048576,"data_stored":1101922,"data_compressed":59395,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.377,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81799999999999995,"5min":0.875,"15min":0.875},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.42899999999999999,"15min":0.41099999999999998},"min":{"1min":0.28499999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59499999999999997,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.45100000000000001,"15min":0.441},"min":{"1min":0.29899999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.628,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.60299999999999998}]},{"osd":2,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.58099999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.76600000000000001,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.61199999999999999,"15min":0.58599999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81799999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.629}]},{"osd":3,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.58799999999999997,"15min":0.55900000000000005},"min":{"1min":0.46999999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.44,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.76600000000000001}]},{"osd":4,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.375,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67900000000000005,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.51900000000000002,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.69799999999999995,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.48799999999999999}]},{"osd":5,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64900000000000002,"15min":0.65100000000000002},"min":{"1min":0.45500000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81899999999999995,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.621},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.66500000000000004,"15min":0.63900000000000001},"min":{"1min":0.45700000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.88100000000000001,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.73099999999999998}]},{"osd":6,"last update":"Thu Nov 16 16:49:07 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.69599999999999995,"15min":0.68000000000000005},"min":{"1min":0.50700000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46999999999999997,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.93500000000000005,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.64800000000000002}]},{"osd":8,"last update":"Thu Nov 16 16:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.72099999999999997,"15min":0.68500000000000005},"min":{"1min":0.64200000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.92900000000000005,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.70699999999999996,"15min":0.70399999999999996},"min":{"1min":0.58499999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.95299999999999996,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.67600000000000005}]},{"osd":9,"last update":"Thu Nov 16 16:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.75600000000000001,"15min":0.77900000000000003},"min":{"1min":0.53800000000000003,"5min":0.442,"15min":0.442},"max":{"1min":0.88900000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78400000000000003,"15min":0.80300000000000005},"min":{"1min":0.61599999999999999,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.94999999999999996,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.88}]},{"osd":10,"last update":"Thu Nov 16 16:49:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.72699999999999998,"15min":0.75},"min":{"1min":0.48799999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94199999999999995,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.76800000000000002,"15min":0.78500000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93500000000000005,"5min":1.01,"15min":1.01},"last":0.83399999999999996}]},{"osd":11,"last update":"Thu Nov 16 16:49:23 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81200000000000006,"15min":0.81000000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.76400000000000001,"15min":0.76100000000000001},"min":{"1min":0.56799999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.71999999999999997}]}]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6540,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738612,"statfs":{"total":95995035648,"available":95988338688,"internally_reserved":0,"allocated":987136,"data_stored":502780,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.752},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.375,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.96499999999999997}]},{"osd":1,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.66900000000000004,"15min":0.65500000000000003},"min":{"1min":0.42899999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.68999999999999995,"15min":0.70199999999999996},"min":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0569999999999999,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72699999999999998,"15min":0.71099999999999997},"min":{"1min":0.50800000000000001,"5min":0.436,"15min":0.436},"max":{"1min":0.92800000000000005,"5min":1.407,"15min":1.407},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.72399999999999998},"min":{"1min":0.505,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0189999999999999,"5min":1.171,"15min":1.171},"last":0.61499999999999999}]},{"osd":3,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.71799999999999997,"15min":0.73599999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.755,"15min":0.73499999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.1020000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.74399999999999999,"15min":0.75600000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.77800000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.79500000000000004,"15min":0.80100000000000005},"min":{"1min":0.52300000000000002,"5min":0.503,"15min":0.503},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.873,"15min":0.89300000000000002},"min":{"1min":0.57899999999999996,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.083,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.90800000000000003}]},{"osd":7,"last update":"Thu Nov 16 16:49:02 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.82199999999999995,"15min":0.79900000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.1080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.78500000000000003,"15min":0.755},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0780000000000001,"5min":1.256,"15min":1.256},"last":0.80100000000000005}]},{"osd":8,"last update":"Thu Nov 16 16:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.83099999999999996,"15min":0.84199999999999997},"min":{"1min":0.55200000000000005,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.998,"5min":1.284,"15min":1.284},"last":0.629},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.89000000000000001,"15min":0.88800000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0580000000000001,"5min":1.823,"15min":1.823},"last":0.92700000000000005}]},{"osd":9,"last update":"Thu Nov 16 16:48:51 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.94499999999999995,"15min":0.96099999999999997},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.208,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.83999999999999997,"15min":0.83699999999999997},"min":{"1min":0.623,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.82299999999999995}]},{"osd":10,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.89500000000000002,"15min":0.92200000000000004},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.2370000000000001,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.871,"5min":0.96499999999999997,"15min":0.97799999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.02,"5min":1.393,"15min":1.393},"last":1.0029999999999999}]},{"osd":11,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.022,"15min":1.036},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.91900000000000004,"15min":0.93500000000000005},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":0.98099999999999998,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.874}]}]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7632,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737520,"statfs":{"total":95995035648,"available":95987220480,"internally_reserved":0,"allocated":1056768,"data_stored":564253,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:52 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.52900000000000003},"min":{"1min":0.47899999999999998,"5min":0.32900000000000001,"15min":0.317},"max":{"1min":0.91900000000000004,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.56000000000000005,"15min":0.52100000000000002},"min":{"1min":0.36199999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.82599999999999996,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:49:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.58499999999999996,"15min":0.54500000000000004},"min":{"1min":0.46500000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.83299999999999996,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.57999999999999996,"15min":0.52200000000000002},"min":{"1min":0.48099999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.84099999999999997,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.71599999999999997}]},{"osd":3,"last update":"Thu Nov 16 16:48:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56599999999999995,"15min":0.53900000000000003},"min":{"1min":0.44800000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.93100000000000005,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.621,"15min":0.58299999999999996},"min":{"1min":0.495,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.61399999999999999}]},{"osd":4,"last update":"Thu Nov 16 16:48:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.45700000000000002,"15min":0.42699999999999999},"min":{"1min":0.34799999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.72999999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.46700000000000003,"15min":0.442},"min":{"1min":0.37,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76900000000000002,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.371}]},{"osd":5,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.63700000000000001,"15min":0.58499999999999996},"min":{"1min":0.57399999999999995,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.876,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.65700000000000003,"15min":0.61599999999999999},"min":{"1min":0.54400000000000004,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90100000000000002,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.75900000000000001}]},{"osd":6,"last update":"Thu Nov 16 16:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.66900000000000004,"15min":0.64600000000000002},"min":{"1min":0.51400000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.68000000000000005,"15min":0.65000000000000002},"min":{"1min":0.59499999999999997,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.87,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.65400000000000003}]},{"osd":7,"last update":"Thu Nov 16 16:49:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.57599999999999996,"15min":0.52900000000000003},"min":{"1min":0.38500000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.61199999999999999,"15min":0.56799999999999995},"min":{"1min":0.41899999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.80300000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.74399999999999999}]},{"osd":8,"last update":"Thu Nov 16 16:49:28 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.70999999999999996,"15min":0.65400000000000003},"min":{"1min":0.50700000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.90600000000000003,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.71999999999999997,"15min":0.66600000000000004},"min":{"1min":0.57399999999999995,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.97799999999999998,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.79900000000000004}]},{"osd":9,"last update":"Thu Nov 16 16:48:58 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.749,"15min":0.73199999999999998},"min":{"1min":0.54000000000000004,"5min":0.5,"15min":0.5},"max":{"1min":0.98099999999999998,"5min":1.276,"15min":1.276},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.76300000000000001,"15min":0.751},"min":{"1min":0.63800000000000001,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.92600000000000005,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.70399999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:49:11 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.77700000000000002,"15min":0.78100000000000003},"min":{"1min":0.65700000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.91100000000000003,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.78500000000000003,"15min":0.78300000000000003},"min":{"1min":0.58299999999999996,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.81299999999999994}]},{"osd":11,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85599999999999998,"15min":0.85999999999999999},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":0.96799999999999997,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.88800000000000001,"15min":0.89200000000000002},"min":{"1min":0.73299999999999998,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.0069999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.84099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7440,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737712,"statfs":{"total":95995035648,"available":95987417088,"internally_reserved":0,"allocated":991232,"data_stored":502870,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 16:48:53 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.61299999999999999,"15min":0.55700000000000005},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66000000000000003,"15min":0.57999999999999996},"min":{"1min":0.432,"5min":0.316,"15min":0.316},"max":{"1min":1.7789999999999999,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.61399999999999999}]},{"osd":2,"last update":"Thu Nov 16 16:49:12 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.75900000000000001,"15min":0.621},"min":{"1min":0.57199999999999995,"5min":0.40600000000000003,"15min":0.372},"max":{"1min":2.0529999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.70399999999999996,"15min":0.63},"min":{"1min":0.53000000000000003,"5min":0.41799999999999998,"15min":0.35099999999999998},"max":{"1min":2.1000000000000001,"5min":2.1000000000000001,"15min":2.1000000000000001},"last":0.66500000000000004}]},{"osd":3,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83499999999999996,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.63300000000000001,"15min":0.63500000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.901,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.46000000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:49:04 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.71299999999999997,"15min":0.61099999999999999},"min":{"1min":0.59899999999999998,"5min":0.434,"15min":0.434},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.748,"15min":0.78800000000000003},"min":{"1min":0.53700000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.016,"5min":5.4409999999999998,"15min":5.4409999999999998},"last":0.64900000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:49:22 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.79200000000000004,"15min":0.71999999999999997},"min":{"1min":0.55000000000000004,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.9390000000000001,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.74399999999999999,"15min":0.72399999999999998},"min":{"1min":0.50900000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.97599999999999998,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":0.55900000000000005}]},{"osd":6,"last update":"Thu Nov 16 16:48:48 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.73199999999999998,"15min":0.69599999999999995},"min":{"1min":0.51000000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.76100000000000001,"15min":0.68799999999999994},"min":{"1min":0.622,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":0.72499999999999998}]},{"osd":7,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.86699999999999999,"15min":0.872},"min":{"1min":0.59699999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.054,"5min":1.8089999999999999,"15min":1.8089999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.87,"15min":0.88300000000000001},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2330000000000001,"5min":3.1179999999999999,"15min":3.1179999999999999},"last":0.51000000000000001}]},{"osd":8,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.874,"15min":0.88200000000000001},"min":{"1min":0.66600000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.2829999999999999,"5min":2.891,"15min":2.891},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.90100000000000002,"15min":0.91700000000000004},"min":{"1min":0.63200000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.1499999999999999,"5min":1.8759999999999999,"15min":1.8759999999999999},"last":0.64000000000000001}]},{"osd":9,"last update":"Thu Nov 16 16:48:53 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.97399999999999998,"15min":0.98499999999999999},"min":{"1min":0.67600000000000005,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.69899999999999995,"5min":0.495,"15min":0.495},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.78100000000000003}]},{"osd":10,"last update":"Thu Nov 16 16:49:12 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.70099999999999996,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":3.0859999999999999,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.04,"15min":1.0860000000000001},"min":{"1min":0.68200000000000005,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.9079999999999999,"5min":3.694,"15min":3.694},"last":0.67700000000000005}]},{"osd":11,"last update":"Thu Nov 16 16:49:32 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91600000000000004,"15min":0.88900000000000001},"min":{"1min":0.751,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.1859999999999999,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":0.749},{"interface":"front","average":{"1min":1.0069999999999999,"5min":0.93899999999999995,"15min":0.89800000000000002},"min":{"1min":0.75,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.4370000000000001,"5min":3.0489999999999999,"15min":3.0489999999999999},"last":0.80500000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7640,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737512,"statfs":{"total":95995035648,"available":95987212288,"internally_reserved":0,"allocated":1064960,"data_stored":569393,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56499999999999995,"15min":0.47999999999999998},"min":{"1min":0.44800000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.91400000000000003,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.504},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53500000000000003,"15min":0.499},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.45300000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56399999999999995,"15min":0.502},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.83299999999999996,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.60999999999999999,"15min":0.51800000000000002},"min":{"1min":0.33600000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85799999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.439}]},{"osd":3,"last update":"Thu Nov 16 16:49:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.64900000000000002,"15min":0.71099999999999997},"min":{"1min":0.438,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":3.98},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63400000000000001,"15min":0.59799999999999998},"min":{"1min":0.47499999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95899999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.51400000000000001}]},{"osd":4,"last update":"Thu Nov 16 16:49:19 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.70999999999999996,"15min":0.70599999999999996},"min":{"1min":0.501,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.67100000000000004,"15min":0.67400000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.35999999999999999}]},{"osd":5,"last update":"Thu Nov 16 16:49:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.50700000000000001,"15min":0.47099999999999997},"min":{"1min":0.39200000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76100000000000001,"5min":0.871,"15min":0.871},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.52800000000000002,"15min":0.48899999999999999},"min":{"1min":0.45400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70699999999999996,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.42299999999999999}]},{"osd":6,"last update":"Thu Nov 16 16:48:47 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.629,"15min":0.59499999999999997},"min":{"1min":0.442,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63700000000000001,"15min":0.61499999999999999},"min":{"1min":0.46600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85799999999999998,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.497}]},{"osd":7,"last update":"Thu Nov 16 16:49:24 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.40699999999999997,"5min":0.376,"15min":0.376},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.80000000000000004,"15min":0.79400000000000004},"min":{"1min":0.627,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.627}]},{"osd":8,"last update":"Thu Nov 16 16:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.64500000000000002,"15min":0.61099999999999999},"min":{"1min":0.44800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60099999999999998,"15min":0.58099999999999996},"min":{"1min":0.434,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.45900000000000002}]},{"osd":9,"last update":"Thu Nov 16 16:48:53 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.73299999999999998,"15min":0.72099999999999997},"min":{"1min":0.58799999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.022,"5min":0.79000000000000004,"15min":0.749},"min":{"1min":0.52100000000000002,"5min":0.503,"15min":0.503},"max":{"1min":5.2510000000000003,"5min":5.2510000000000003,"15min":5.2510000000000003},"last":0.57799999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.72199999999999998,"15min":0.69299999999999995},"min":{"1min":0.49299999999999999,"5min":0.441,"15min":0.441},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.71999999999999997,"15min":0.69799999999999995},"min":{"1min":0.56299999999999994,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.52100000000000002}]},{"osd":11,"last update":"Thu Nov 16 16:49:19 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.81399999999999995,"15min":0.80800000000000005},"min":{"1min":0.65200000000000002,"5min":0.499,"15min":0.499},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.745,"15min":0.73799999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.48599999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6804,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738348,"statfs":{"total":95995035648,"available":95988068352,"internally_reserved":0,"allocated":995328,"data_stored":502884,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.48199999999999998,"15min":0.435},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.077,"5min":1.085,"15min":1.085},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.58399999999999996,"15min":0.50600000000000001},"min":{"1min":0.39600000000000002,"5min":0.313,"15min":0.313},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":0.58499999999999996}]},{"osd":1,"last update":"Thu Nov 16 16:48:38 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":0.73699999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.379,"15min":0.379},"max":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":10.266999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.66000000000000003,"15min":0.60399999999999998},"min":{"1min":0.44400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.2050000000000001,"5min":1.286,"15min":1.286},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:48:38 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66500000000000004,"15min":0.59699999999999998},"min":{"1min":0.51200000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.02,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.68700000000000006,"15min":0.61499999999999999},"min":{"1min":0.42299999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.4350000000000001,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Nov 16 16:49:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.71599999999999997,"15min":0.63},"min":{"1min":0.50900000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.25,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.73299999999999998,"15min":0.67900000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.88300000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:49:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.71899999999999997,"15min":0.69299999999999995},"min":{"1min":0.45400000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.879,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.62},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.76800000000000002,"15min":0.68799999999999994},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86399999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":0.56200000000000006}]},{"osd":6,"last update":"Thu Nov 16 16:48:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.71099999999999997,"15min":0.66500000000000004},"min":{"1min":0.47999999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.29,"5min":1.323,"15min":1.323},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70499999999999996,"15min":0.68200000000000005},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.04,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.752}]},{"osd":7,"last update":"Thu Nov 16 16:49:05 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.84299999999999997,"15min":0.80300000000000005},"min":{"1min":0.50700000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.637,"5min":5.1070000000000002,"15min":5.1070000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1,"15min":0.89700000000000002},"min":{"1min":0.60599999999999998,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":10.305999999999999,"5min":10.305999999999999,"15min":10.305999999999999},"last":0.92000000000000004}]},{"osd":8,"last update":"Thu Nov 16 16:46:23 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.94199999999999995}]},{"osd":9,"last update":"Thu Nov 16 16:48:51 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.88400000000000001,"15min":0.91000000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":2.5920000000000001,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.94599999999999995,"15min":0.96799999999999997},"min":{"1min":0.54600000000000004,"5min":0.501,"15min":0.501},"max":{"1min":1.417,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.99299999999999999}]},{"osd":10,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":0.96799999999999997,"15min":0.91200000000000003},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":10.287000000000001,"5min":10.287000000000001,"15min":10.287000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.91800000000000004,"15min":0.93799999999999994},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.012,"5min":2.673,"15min":2.673},"last":0.70799999999999996}]},{"osd":11,"last update":"Thu Nov 16 16:49:24 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.92500000000000004,"15min":0.93500000000000005},"min":{"1min":0.64000000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.1000000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":1.028},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.82999999999999996,"15min":0.82799999999999996},"min":{"1min":0.626,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.78900000000000003}]}]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6676,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738476,"statfs":{"total":95995035648,"available":95988199424,"internally_reserved":0,"allocated":995328,"data_stored":506898,"data_compressed":23604,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.48899999999999999,"15min":0.46600000000000003},"min":{"1min":0.439,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.51100000000000001,"15min":0.48599999999999999},"min":{"1min":0.47399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.63500000000000001}]},{"osd":1,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.38,"15min":0.33000000000000002},"min":{"1min":0.26500000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57799999999999996,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40000000000000002,"15min":0.35499999999999998},"min":{"1min":0.28100000000000003,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.56299999999999994,"5min":1.212,"15min":1.212},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58499999999999996,"15min":0.52300000000000002},"min":{"1min":0.371,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80800000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.57999999999999996,"15min":0.51600000000000001},"min":{"1min":0.435,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.83799999999999997,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.53100000000000003}]},{"osd":3,"last update":"Thu Nov 16 16:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.58899999999999997,"15min":0.53500000000000003},"min":{"1min":0.40899999999999997,"5min":0.307,"15min":0.307},"max":{"1min":0.85699999999999998,"5min":1.349,"15min":1.349},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.745,"15min":0.58899999999999997},"min":{"1min":0.39500000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.84599999999999997,"5min":14.795999999999999,"15min":14.795999999999999},"last":0.40000000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:49:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61299999999999999,"15min":0.55300000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.84899999999999998,"5min":1.462,"15min":1.462},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60499999999999998,"15min":0.53900000000000003},"min":{"1min":0.442,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.496}]},{"osd":6,"last update":"Thu Nov 16 16:48:52 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.624,"15min":0.58599999999999997},"min":{"1min":0.44500000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.85899999999999999,"5min":1.323,"15min":1.323},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.66200000000000003,"15min":0.62},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89400000000000002,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":0.54800000000000004}]},{"osd":7,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56200000000000006,"15min":0.52800000000000002},"min":{"1min":0.38900000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88500000000000001,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":0.443},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.60599999999999998,"15min":0.57399999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81999999999999995,"5min":1.764,"15min":1.764},"last":0.46100000000000002}]},{"osd":8,"last update":"Thu Nov 16 16:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.71999999999999997,"15min":0.71299999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88500000000000001,"5min":2.7650000000000001,"15min":2.7650000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73299999999999998,"15min":0.74399999999999999},"min":{"1min":0.51200000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.872,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.51000000000000001}]},{"osd":9,"last update":"Thu Nov 16 16:48:52 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.77700000000000002,"15min":0.79900000000000004},"min":{"1min":0.48599999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.90900000000000003,"5min":1.246,"15min":1.246},"last":0.621},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.76500000000000001,"15min":0.77700000000000002},"min":{"1min":0.51200000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.88900000000000001,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.57699999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.69099999999999995,"15min":0.69899999999999995},"min":{"1min":0.48699999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.89000000000000001,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.70199999999999996,"15min":0.71599999999999997},"min":{"1min":0.497,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90200000000000002,"5min":1.532,"15min":1.532},"last":0.56999999999999995}]},{"osd":11,"last update":"Thu Nov 16 16:49:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.79800000000000004,"15min":0.80500000000000005},"min":{"1min":0.54500000000000004,"5min":0.501,"15min":0.501},"max":{"1min":0.93400000000000005,"5min":1.502,"15min":1.502},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.82099999999999995,"15min":0.82699999999999996},"min":{"1min":0.57399999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.96499999999999997,"5min":1.425,"15min":1.425},"last":0.59299999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6860,"kb_used_data":1028,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738292,"statfs":{"total":95995035648,"available":95988011008,"internally_reserved":0,"allocated":1052672,"data_stored":1097426,"data_compressed":59395,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:49:20 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.51100000000000001,"15min":0.51200000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.60199999999999998,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54500000000000004,"15min":0.56000000000000005},"min":{"1min":0.35799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63200000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Nov 16 16:49:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.58999999999999997,"15min":0.55400000000000005},"min":{"1min":0.435,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.82299999999999995,"5min":1.167,"15min":1.167},"last":0.377},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.58399999999999996,"15min":0.52600000000000002},"min":{"1min":0.36299999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.75600000000000001,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.30199999999999999}]},{"osd":2,"last update":"Thu Nov 16 16:49:20 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40400000000000003,"15min":0.377},"min":{"1min":0.33900000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57699999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39700000000000002},"min":{"1min":0.33000000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.68200000000000005,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.33500000000000002}]},{"osd":3,"last update":"Thu Nov 16 16:49:20 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55300000000000005,"15min":0.53500000000000003},"min":{"1min":0.41199999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.65800000000000003,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.497},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59999999999999998,"15min":0.57199999999999995},"min":{"1min":0.46100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.71499999999999997,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.46300000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:49:20 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.59699999999999998,"15min":0.56100000000000005},"min":{"1min":0.54600000000000004,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85799999999999998,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57899999999999996,"15min":0.53500000000000003},"min":{"1min":0.44700000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.84299999999999997,"5min":0.879,"15min":0.879},"last":0.314}]},{"osd":6,"last update":"Thu Nov 16 16:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.438,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86699999999999999,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.64500000000000002,"15min":0.63300000000000001},"min":{"1min":0.53000000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.92800000000000005,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.60899999999999999}]},{"osd":7,"last update":"Thu Nov 16 16:49:08 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.63300000000000001,"15min":0.61099999999999999},"min":{"1min":0.49199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.86699999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.436},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.65500000000000003,"15min":0.63700000000000001},"min":{"1min":0.58399999999999996,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.95499999999999996,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.46899999999999997}]},{"osd":8,"last update":"Thu Nov 16 16:49:27 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56100000000000005,"15min":0.54400000000000004},"min":{"1min":0.436,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80400000000000005,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.62,"15min":0.59599999999999997},"min":{"1min":0.51800000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.90300000000000002,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.48099999999999998}]},{"osd":9,"last update":"Thu Nov 16 16:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.747,"15min":0.76100000000000001},"min":{"1min":0.61599999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.91700000000000004,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.745,"15min":0.75700000000000001},"min":{"1min":0.60399999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.90700000000000003,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.64200000000000002}]},{"osd":10,"last update":"Thu Nov 16 16:49:14 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.71299999999999997},"min":{"1min":0.55600000000000005,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.89500000000000002,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.76900000000000002,"15min":0.75700000000000001},"min":{"1min":0.68600000000000005,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.995,"5min":2.492,"15min":2.492},"last":0.57599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34148,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35791,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":66886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":67046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T16:49:42.452 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:49:42.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.690+0000 7f9fa39ff700 1 -- 172.21.15.44:0/994206318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c103640 msgr2=0x7f9f9c105a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:42.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.690+0000 7f9fa39ff700 1 --2- 172.21.15.44:0/994206318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c103640 0x7f9f9c105a20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009990 tx=0x7f9f8c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.691+0000 7f9fa39ff700 1 -- 172.21.15.44:0/994206318 shutdown_connections 2023-11-16T16:49:42.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.691+0000 7f9fa39ff700 1 --2- 172.21.15.44:0/994206318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f9c105f60 0x7f9f9c1083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.691+0000 7f9fa39ff700 1 --2- 172.21.15.44:0/994206318 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f9c100d20 0x7f9f9c103100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.691+0000 7f9fa39ff700 1 --2- 172.21.15.44:0/994206318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c103640 0x7f9f9c105a20 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.691+0000 7f9fa39ff700 1 -- 172.21.15.44:0/994206318 >> 172.21.15.44:0/994206318 conn(0x7f9f9c0fa7a0 msgr2=0x7f9f9c0fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:42.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.691+0000 7f9fa39ff700 1 -- 172.21.15.44:0/994206318 shutdown_connections 2023-11-16T16:49:42.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.691+0000 7f9fa39ff700 1 -- 172.21.15.44:0/994206318 wait complete. 2023-11-16T16:49:42.695 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.692+0000 7f9fa39ff700 1 Processor -- start 2023-11-16T16:49:42.695 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.692+0000 7f9fa39ff700 1 -- start start 2023-11-16T16:49:42.695 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.692+0000 7f9fa39ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f9c100d20 0x7f9f9c195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:42.695 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.692+0000 7f9fa39ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c103640 0x7f9f9c196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:42.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.692+0000 7f9fa39ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f9c105f60 0x7f9f9c19d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:42.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.692+0000 7f9fa39ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c19db50 con 0x7f9f9c103640 2023-11-16T16:49:42.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.692+0000 7f9fa39ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c19dc90 con 0x7f9f9c105f60 2023-11-16T16:49:42.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.692+0000 7f9fa39ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c19de20 con 0x7f9f9c100d20 2023-11-16T16:49:42.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.692+0000 7f9fa0f9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c103640 0x7f9f9c196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:42.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.692+0000 7f9fa0f9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c103640 0x7f9f9c196510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32866/0 (socket says 172.21.15.44:32866) 2023-11-16T16:49:42.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.692+0000 7f9fa0f9a700 1 -- 172.21.15.44:0/1518476530 learned_addr learned my addr 172.21.15.44:0/1518476530 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:42.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.693+0000 7f9fa179b700 1 --2- 172.21.15.44:0/1518476530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f9c100d20 0x7f9f9c195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:42.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.693+0000 7f9fa1f9c700 1 --2- 172.21.15.44:0/1518476530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f9c105f60 0x7f9f9c19d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:42.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.693+0000 7f9fa0f9a700 1 -- 172.21.15.44:0/1518476530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f9c100d20 msgr2=0x7f9f9c195fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:42.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.693+0000 7f9fa0f9a700 1 --2- 172.21.15.44:0/1518476530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f9c100d20 0x7f9f9c195fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.693+0000 7f9fa0f9a700 1 -- 172.21.15.44:0/1518476530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f9c105f60 msgr2=0x7f9f9c19d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:42.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.693+0000 7f9fa0f9a700 1 --2- 172.21.15.44:0/1518476530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f9c105f60 0x7f9f9c19d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.693+0000 7f9fa0f9a700 1 -- 172.21.15.44:0/1518476530 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c0095f0 con 0x7f9f9c103640 2023-11-16T16:49:42.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.693+0000 7f9fa0f9a700 1 --2- 172.21.15.44:0/1518476530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c103640 0x7f9f9c196510 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c00b680 tx=0x7f9f8c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:42.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.693+0000 7f9f927fc700 1 -- 172.21.15.44:0/1518476530 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c01d070 con 0x7f9f9c103640 2023-11-16T16:49:42.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.693+0000 7f9f927fc700 1 -- 172.21.15.44:0/1518476530 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f8c021410 con 0x7f9f9c103640 2023-11-16T16:49:42.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.693+0000 7f9fa39ff700 1 -- 172.21.15.44:0/1518476530 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9c10bc80 con 0x7f9f9c103640 2023-11-16T16:49:42.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.693+0000 7f9f927fc700 1 -- 172.21.15.44:0/1518476530 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c017650 con 0x7f9f9c103640 2023-11-16T16:49:42.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.694+0000 7f9fa39ff700 1 -- 172.21.15.44:0/1518476530 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9c10baf0 con 0x7f9f9c103640 2023-11-16T16:49:42.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.695+0000 7f9fa39ff700 1 -- 172.21.15.44:0/1518476530 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9c0fc310 con 0x7f9f9c103640 2023-11-16T16:49:42.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.697+0000 7f9f927fc700 1 -- 172.21.15.44:0/1518476530 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9f8c0217f0 con 0x7f9f9c103640 2023-11-16T16:49:42.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.698+0000 7f9f927fc700 1 --2- 172.21.15.44:0/1518476530 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9f880a1150 0x7f9f880a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:42.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.698+0000 7f9f927fc700 1 -- 172.21.15.44:0/1518476530 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f9f8c0ce7f0 con 0x7f9f9c103640 2023-11-16T16:49:42.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.698+0000 7f9fa179b700 1 --2- 172.21.15.44:0/1518476530 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9f880a1150 0x7f9f880a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:42.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.699+0000 7f9fa179b700 1 --2- 172.21.15.44:0/1518476530 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9f880a1150 0x7f9f880a3600 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9f98009c60 tx=0x7f9f98009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:42.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.700+0000 7f9f927fc700 1 -- 172.21.15.44:0/1518476530 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9f8c097bd0 con 0x7f9f9c103640 2023-11-16T16:49:42.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:42 smithi044 ceph-mon[128750]: from='client.25340 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:49:42.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.911+0000 7f9fa39ff700 1 -- 172.21.15.44:0/1518476530 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f9c067eb0 con 0x7f9f9c103640 2023-11-16T16:49:42.914 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.913+0000 7f9f927fc700 1 -- 172.21.15.44:0/1518476530 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24685 (secure 0 0 0) 0x7f9f8c09b1f0 con 0x7f9f9c103640 2023-11-16T16:49:42.915 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:42.915 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:48:44.613706+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:49:42.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.915+0000 7f9fa39ff700 1 -- 172.21.15.44:0/1518476530 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9f880a1150 msgr2=0x7f9f880a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:42.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.915+0000 7f9fa39ff700 1 --2- 172.21.15.44:0/1518476530 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9f880a1150 0x7f9f880a3600 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9f98009c60 tx=0x7f9f98009260 comp rx=0 tx=0).stop 2023-11-16T16:49:42.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.915+0000 7f9fa39ff700 1 -- 172.21.15.44:0/1518476530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c103640 msgr2=0x7f9f9c196510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:42.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.915+0000 7f9fa39ff700 1 --2- 172.21.15.44:0/1518476530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c103640 0x7f9f9c196510 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c00b680 tx=0x7f9f8c005740 comp rx=0 tx=0).stop 2023-11-16T16:49:42.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.915+0000 7f9fa39ff700 1 -- 172.21.15.44:0/1518476530 shutdown_connections 2023-11-16T16:49:42.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.915+0000 7f9fa39ff700 1 --2- 172.21.15.44:0/1518476530 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f9f880a1150 0x7f9f880a3600 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.915+0000 7f9fa39ff700 1 --2- 172.21.15.44:0/1518476530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f9c105f60 0x7f9f9c19d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.915+0000 7f9fa39ff700 1 --2- 172.21.15.44:0/1518476530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f9c100d20 0x7f9f9c195fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.915+0000 7f9fa39ff700 1 --2- 172.21.15.44:0/1518476530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f9c103640 0x7f9f9c196510 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:42.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.915+0000 7f9fa39ff700 1 -- 172.21.15.44:0/1518476530 >> 172.21.15.44:0/1518476530 conn(0x7f9f9c0fa7a0 msgr2=0x7f9f9c101980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:42.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.915+0000 7f9fa39ff700 1 -- 172.21.15.44:0/1518476530 shutdown_connections 2023-11-16T16:49:42.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:42.916+0000 7f9fa39ff700 1 -- 172.21.15.44:0/1518476530 wait complete. 2023-11-16T16:49:42.936 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:03.059622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:16.211826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:17.601591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:21.136084+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T16:49:42.937 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:49:42.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:42 smithi077 ceph-mon[131204]: from='client.25340 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:49:43.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.143+0000 7fed74dff700 1 -- 172.21.15.44:0/1622205211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed701014c0 msgr2=0x7fed70101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:43.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.143+0000 7fed74dff700 1 --2- 172.21.15.44:0/1622205211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed701014c0 0x7fed70101910 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fed580098f0 tx=0x7fed58009c00 comp rx=0 tx=0).stop 2023-11-16T16:49:43.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.143+0000 7fed74dff700 1 -- 172.21.15.44:0/1622205211 shutdown_connections 2023-11-16T16:49:43.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.143+0000 7fed74dff700 1 --2- 172.21.15.44:0/1622205211 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed70101e50 0x7fed70108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.143+0000 7fed74dff700 1 --2- 172.21.15.44:0/1622205211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed701002d0 0x7fed701006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.143+0000 7fed74dff700 1 --2- 172.21.15.44:0/1622205211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed701014c0 0x7fed70101910 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.143+0000 7fed74dff700 1 -- 172.21.15.44:0/1622205211 >> 172.21.15.44:0/1622205211 conn(0x7fed700fb8b0 msgr2=0x7fed700fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:43.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.143+0000 7fed74dff700 1 -- 172.21.15.44:0/1622205211 shutdown_connections 2023-11-16T16:49:43.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.143+0000 7fed74dff700 1 -- 172.21.15.44:0/1622205211 wait complete. 2023-11-16T16:49:43.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:42 smithi176 ceph-mon[131630]: from='client.25340 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:49:43.148 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.144+0000 7fed74dff700 1 Processor -- start 2023-11-16T16:49:43.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.144+0000 7fed74dff700 1 -- start start 2023-11-16T16:49:43.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.144+0000 7fed74dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed701002d0 0x7fed70198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:43.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.144+0000 7fed74dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed701014c0 0x7fed701987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:43.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.144+0000 7fed74dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed70101e50 0x7fed7019f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:43.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.144+0000 7fed74dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7019fde0 con 0x7fed70101e50 2023-11-16T16:49:43.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.144+0000 7fed74dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7019ff20 con 0x7fed701002d0 2023-11-16T16:49:43.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.145+0000 7fed74dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed701a00b0 con 0x7fed701014c0 2023-11-16T16:49:43.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.145+0000 7fed6ed9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed70101e50 0x7fed7019f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:43.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.145+0000 7fed6ed9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed70101e50 0x7fed7019f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32878/0 (socket says 172.21.15.44:32878) 2023-11-16T16:49:43.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.145+0000 7fed67fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed701014c0 0x7fed701987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:43.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.145+0000 7fed6ed9d700 1 -- 172.21.15.44:0/1171758026 learned_addr learned my addr 172.21.15.44:0/1171758026 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:43.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.145+0000 7fed6e59c700 1 --2- 172.21.15.44:0/1171758026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed701002d0 0x7fed70198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:43.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.145+0000 7fed6ed9d700 1 -- 172.21.15.44:0/1171758026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed701014c0 msgr2=0x7fed701987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:43.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.145+0000 7fed6ed9d700 1 --2- 172.21.15.44:0/1171758026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed701014c0 0x7fed701987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.145+0000 7fed6ed9d700 1 -- 172.21.15.44:0/1171758026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed701002d0 msgr2=0x7fed70198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:43.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.145+0000 7fed6ed9d700 1 --2- 172.21.15.44:0/1171758026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed701002d0 0x7fed70198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.145+0000 7fed6ed9d700 1 -- 172.21.15.44:0/1171758026 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed68009d00 con 0x7fed70101e50 2023-11-16T16:49:43.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.146+0000 7fed6ed9d700 1 --2- 172.21.15.44:0/1171758026 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed70101e50 0x7fed7019f810 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fed6800e3c0 tx=0x7fed6800e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:43.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.146+0000 7fed677fe700 1 -- 172.21.15.44:0/1171758026 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed680090d0 con 0x7fed70101e50 2023-11-16T16:49:43.151 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.146+0000 7fed74dff700 1 -- 172.21.15.44:0/1171758026 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed58009520 con 0x7fed70101e50 2023-11-16T16:49:43.151 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.146+0000 7fed677fe700 1 -- 172.21.15.44:0/1171758026 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed6800f040 con 0x7fed70101e50 2023-11-16T16:49:43.151 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.146+0000 7fed677fe700 1 -- 172.21.15.44:0/1171758026 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed68007c20 con 0x7fed70101e50 2023-11-16T16:49:43.151 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.146+0000 7fed74dff700 1 -- 172.21.15.44:0/1171758026 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7010bbe0 con 0x7fed70101e50 2023-11-16T16:49:43.151 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.147+0000 7fed74dff700 1 -- 172.21.15.44:0/1171758026 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed7010bd70 con 0x7fed70101e50 2023-11-16T16:49:43.151 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.149+0000 7fed677fe700 1 -- 172.21.15.44:0/1171758026 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fed68003a00 con 0x7fed70101e50 2023-11-16T16:49:43.152 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.150+0000 7fed677fe700 1 --2- 172.21.15.44:0/1171758026 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fed5c0a1150 0x7fed5c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:43.152 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.150+0000 7fed677fe700 1 -- 172.21.15.44:0/1171758026 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fed680ccc60 con 0x7fed70101e50 2023-11-16T16:49:43.152 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.151+0000 7fed6e59c700 1 --2- 172.21.15.44:0/1171758026 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fed5c0a1150 0x7fed5c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:43.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.151+0000 7fed6e59c700 1 --2- 172.21.15.44:0/1171758026 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fed5c0a1150 0x7fed5c0a3600 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fed70101320 tx=0x7fed60009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:43.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.154+0000 7fed677fe700 1 -- 172.21.15.44:0/1171758026 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fed68096040 con 0x7fed70101e50 2023-11-16T16:49:43.368 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.366+0000 7fed74dff700 1 -- 172.21.15.44:0/1171758026 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fed700617a0 con 0x7fed70101e50 2023-11-16T16:49:43.368 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.367+0000 7fed677fe700 1 -- 172.21.15.44:0/1171758026 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fed68099660 con 0x7fed70101e50 2023-11-16T16:49:43.369 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2023-11-16T16:49:43.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.369+0000 7fed74dff700 1 -- 172.21.15.44:0/1171758026 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fed5c0a1150 msgr2=0x7fed5c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:43.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.369+0000 7fed74dff700 1 --2- 172.21.15.44:0/1171758026 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fed5c0a1150 0x7fed5c0a3600 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fed70101320 tx=0x7fed60009260 comp rx=0 tx=0).stop 2023-11-16T16:49:43.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.369+0000 7fed74dff700 1 -- 172.21.15.44:0/1171758026 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed70101e50 msgr2=0x7fed7019f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:43.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.369+0000 7fed74dff700 1 --2- 172.21.15.44:0/1171758026 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed70101e50 0x7fed7019f810 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fed6800e3c0 tx=0x7fed6800e6d0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.369+0000 7fed74dff700 1 -- 172.21.15.44:0/1171758026 shutdown_connections 2023-11-16T16:49:43.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.369+0000 7fed74dff700 1 --2- 172.21.15.44:0/1171758026 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fed5c0a1150 0x7fed5c0a3600 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.369+0000 7fed74dff700 1 --2- 172.21.15.44:0/1171758026 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed70101e50 0x7fed7019f810 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.369+0000 7fed74dff700 1 --2- 172.21.15.44:0/1171758026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed701002d0 0x7fed70198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.369+0000 7fed74dff700 1 --2- 172.21.15.44:0/1171758026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed701014c0 0x7fed701987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.369+0000 7fed74dff700 1 -- 172.21.15.44:0/1171758026 >> 172.21.15.44:0/1171758026 conn(0x7fed700fb8b0 msgr2=0x7fed70106e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:43.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.369+0000 7fed74dff700 1 -- 172.21.15.44:0/1171758026 shutdown_connections 2023-11-16T16:49:43.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.369+0000 7fed74dff700 1 -- 172.21.15.44:0/1171758026 wait complete. 2023-11-16T16:49:43.391 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T16:49:43.600 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.598+0000 7fe22b7ff700 1 -- 172.21.15.44:0/4182774934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2241014b0 msgr2=0x7fe224101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:43.600 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.598+0000 7fe22b7ff700 1 --2- 172.21.15.44:0/4182774934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2241014b0 0x7fe224101900 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe214009990 tx=0x7fe214009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.600 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.598+0000 7fe22b7ff700 1 -- 172.21.15.44:0/4182774934 shutdown_connections 2023-11-16T16:49:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.598+0000 7fe22b7ff700 1 --2- 172.21.15.44:0/4182774934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe224101e40 0x7fe224108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.598+0000 7fe22b7ff700 1 --2- 172.21.15.44:0/4182774934 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2241002c0 0x7fe2241006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.598+0000 7fe22b7ff700 1 --2- 172.21.15.44:0/4182774934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2241014b0 0x7fe224101900 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.598+0000 7fe22b7ff700 1 -- 172.21.15.44:0/4182774934 >> 172.21.15.44:0/4182774934 conn(0x7fe2240fb8a0 msgr2=0x7fe2240fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.598+0000 7fe22b7ff700 1 -- 172.21.15.44:0/4182774934 shutdown_connections 2023-11-16T16:49:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.598+0000 7fe22b7ff700 1 -- 172.21.15.44:0/4182774934 wait complete. 2023-11-16T16:49:43.604 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.599+0000 7fe22b7ff700 1 Processor -- start 2023-11-16T16:49:43.604 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.599+0000 7fe22b7ff700 1 -- start start 2023-11-16T16:49:43.604 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.600+0000 7fe22b7ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2241002c0 0x7fe224198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:43.604 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.600+0000 7fe22b7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2241014b0 0x7fe224198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:43.604 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.600+0000 7fe22b7ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe224101e40 0x7fe22419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:43.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.600+0000 7fe22b7ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22419fdc0 con 0x7fe2241002c0 2023-11-16T16:49:43.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.600+0000 7fe22b7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22419ff00 con 0x7fe2241014b0 2023-11-16T16:49:43.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.600+0000 7fe22b7ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2241a0090 con 0x7fe224101e40 2023-11-16T16:49:43.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.600+0000 7fe22959b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2241002c0 0x7fe224198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:43.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.600+0000 7fe22959b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2241002c0 0x7fe224198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32908/0 (socket says 172.21.15.44:32908) 2023-11-16T16:49:43.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.600+0000 7fe22959b700 1 -- 172.21.15.44:0/875234620 learned_addr learned my addr 172.21.15.44:0/875234620 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:43.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.600+0000 7fe228d9a700 1 --2- 172.21.15.44:0/875234620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2241014b0 0x7fe224198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.600+0000 7fe229d9c700 1 --2- 172.21.15.44:0/875234620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe224101e40 0x7fe22419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.600+0000 7fe22959b700 1 -- 172.21.15.44:0/875234620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe224101e40 msgr2=0x7fe22419f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.600+0000 7fe22959b700 1 --2- 172.21.15.44:0/875234620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe224101e40 0x7fe22419f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.600+0000 7fe22959b700 1 -- 172.21.15.44:0/875234620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2241014b0 msgr2=0x7fe224198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.600+0000 7fe22959b700 1 --2- 172.21.15.44:0/875234620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2241014b0 0x7fe224198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.600+0000 7fe22959b700 1 -- 172.21.15.44:0/875234620 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2140095f0 con 0x7fe2241002c0 2023-11-16T16:49:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.601+0000 7fe22959b700 1 --2- 172.21.15.44:0/875234620 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2241002c0 0x7fe224198240 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe22000d980 tx=0x7fe22000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.601+0000 7fe21a7fc700 1 -- 172.21.15.44:0/875234620 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22000b950 con 0x7fe2241002c0 2023-11-16T16:49:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.601+0000 7fe21a7fc700 1 -- 172.21.15.44:0/875234620 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe220007620 con 0x7fe2241002c0 2023-11-16T16:49:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.601+0000 7fe22b7ff700 1 -- 172.21.15.44:0/875234620 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22410bc70 con 0x7fe2241002c0 2023-11-16T16:49:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.601+0000 7fe21a7fc700 1 -- 172.21.15.44:0/875234620 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22000f3c0 con 0x7fe2241002c0 2023-11-16T16:49:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.601+0000 7fe22b7ff700 1 -- 172.21.15.44:0/875234620 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22410bae0 con 0x7fe2241002c0 2023-11-16T16:49:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.602+0000 7fe22b7ff700 1 -- 172.21.15.44:0/875234620 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe224067b00 con 0x7fe2241002c0 2023-11-16T16:49:43.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.604+0000 7fe21a7fc700 1 -- 172.21.15.44:0/875234620 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe220010640 con 0x7fe2241002c0 2023-11-16T16:49:43.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.605+0000 7fe21a7fc700 1 --2- 172.21.15.44:0/875234620 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe2100a1150 0x7fe2100a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:43.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.606+0000 7fe21a7fc700 1 -- 172.21.15.44:0/875234620 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fe2200ccbf0 con 0x7fe2241002c0 2023-11-16T16:49:43.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.606+0000 7fe228d9a700 1 --2- 172.21.15.44:0/875234620 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe2100a1150 0x7fe2100a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:43.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.606+0000 7fe228d9a700 1 --2- 172.21.15.44:0/875234620 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe2100a1150 0x7fe2100a3600 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe21400b5c0 tx=0x7fe214005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:43.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.609+0000 7fe21a7fc700 1 -- 172.21.15.44:0/875234620 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe220095f40 con 0x7fe2241002c0 2023-11-16T16:49:43.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.817+0000 7fe22b7ff700 1 -- 172.21.15.44:0/875234620 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe2241a0ba0 con 0x7fe2241002c0 2023-11-16T16:49:43.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.818+0000 7fe21a7fc700 1 -- 172.21.15.44:0/875234620 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe220099560 con 0x7fe2241002c0 2023-11-16T16:49:43.820 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:49:43.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.820+0000 7fe22b7ff700 1 -- 172.21.15.44:0/875234620 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe2100a1150 msgr2=0x7fe2100a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:43.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.820+0000 7fe22b7ff700 1 --2- 172.21.15.44:0/875234620 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe2100a1150 0x7fe2100a3600 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe21400b5c0 tx=0x7fe214005fd0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.820+0000 7fe22b7ff700 1 -- 172.21.15.44:0/875234620 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2241002c0 msgr2=0x7fe224198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:43.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.820+0000 7fe22b7ff700 1 --2- 172.21.15.44:0/875234620 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2241002c0 0x7fe224198240 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe22000d980 tx=0x7fe22000dc90 comp rx=0 tx=0).stop 2023-11-16T16:49:43.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.820+0000 7fe22b7ff700 1 -- 172.21.15.44:0/875234620 shutdown_connections 2023-11-16T16:49:43.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.820+0000 7fe22b7ff700 1 --2- 172.21.15.44:0/875234620 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe2100a1150 0x7fe2100a3600 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.820+0000 7fe22b7ff700 1 --2- 172.21.15.44:0/875234620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe224101e40 0x7fe22419f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.820+0000 7fe22b7ff700 1 --2- 172.21.15.44:0/875234620 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2241002c0 0x7fe224198240 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.820+0000 7fe22b7ff700 1 --2- 172.21.15.44:0/875234620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2241014b0 0x7fe224198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:43.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.820+0000 7fe22b7ff700 1 -- 172.21.15.44:0/875234620 >> 172.21.15.44:0/875234620 conn(0x7fe2240fb8a0 msgr2=0x7fe224106d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:43.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.820+0000 7fe22b7ff700 1 -- 172.21.15.44:0/875234620 shutdown_connections 2023-11-16T16:49:43.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:43.820+0000 7fe22b7ff700 1 -- 172.21.15.44:0/875234620 wait complete. 2023-11-16T16:49:43.841 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T16:49:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:43 smithi077 ceph-mon[131204]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:49:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:43 smithi077 ceph-mon[131204]: from='client.25352 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:49:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:43 smithi077 ceph-mon[131204]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2023-11-16T16:49:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:43 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1518476530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:43 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1171758026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:44.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.047+0000 7f48789ff700 1 -- 172.21.15.44:0/854492301 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48741002d0 msgr2=0x7f48741006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:44.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.047+0000 7f48789ff700 1 --2- 172.21.15.44:0/854492301 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48741002d0 0x7f48741006e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4864009990 tx=0x7f4864009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.047+0000 7f48789ff700 1 -- 172.21.15.44:0/854492301 shutdown_connections 2023-11-16T16:49:44.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.047+0000 7f48789ff700 1 --2- 172.21.15.44:0/854492301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4874101e50 0x7f4874108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.047+0000 7f48789ff700 1 --2- 172.21.15.44:0/854492301 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48741002d0 0x7f48741006e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.047+0000 7f48789ff700 1 --2- 172.21.15.44:0/854492301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48741014c0 0x7f4874101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.047+0000 7f48789ff700 1 -- 172.21.15.44:0/854492301 >> 172.21.15.44:0/854492301 conn(0x7f48740fb8b0 msgr2=0x7f48740fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:44.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.047+0000 7f48789ff700 1 -- 172.21.15.44:0/854492301 shutdown_connections 2023-11-16T16:49:44.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.047+0000 7f48789ff700 1 -- 172.21.15.44:0/854492301 wait complete. 2023-11-16T16:49:44.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.049+0000 7f48789ff700 1 Processor -- start 2023-11-16T16:49:44.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.049+0000 7f48789ff700 1 -- start start 2023-11-16T16:49:44.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.049+0000 7f48789ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48741002d0 0x7f4874196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:44.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.049+0000 7f48789ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48741014c0 0x7f48741965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:44.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.049+0000 7f48789ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4874101e50 0x7f487419d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:44.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.049+0000 7f48789ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487419dbf0 con 0x7f48741014c0 2023-11-16T16:49:44.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.049+0000 7f48789ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487419dd30 con 0x7f4874101e50 2023-11-16T16:49:44.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.049+0000 7f48789ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487419dec0 con 0x7f48741002d0 2023-11-16T16:49:44.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.050+0000 7f486bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48741014c0 0x7f48741965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:44.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.050+0000 7f486bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48741014c0 0x7f48741965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32916/0 (socket says 172.21.15.44:32916) 2023-11-16T16:49:44.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.050+0000 7f486bfff700 1 -- 172.21.15.44:0/3457088797 learned_addr learned my addr 172.21.15.44:0/3457088797 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:44.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.050+0000 7f487259c700 1 --2- 172.21.15.44:0/3457088797 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48741002d0 0x7f4874196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:44.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.050+0000 7f4872d9d700 1 --2- 172.21.15.44:0/3457088797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4874101e50 0x7f487419d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:44.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.050+0000 7f486bfff700 1 -- 172.21.15.44:0/3457088797 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48741002d0 msgr2=0x7f4874196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:44.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.050+0000 7f486bfff700 1 --2- 172.21.15.44:0/3457088797 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48741002d0 0x7f4874196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.050+0000 7f486bfff700 1 -- 172.21.15.44:0/3457088797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4874101e50 msgr2=0x7f487419d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:44.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.050+0000 7f486bfff700 1 --2- 172.21.15.44:0/3457088797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4874101e50 0x7f487419d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.050+0000 7f486bfff700 1 -- 172.21.15.44:0/3457088797 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48640095f0 con 0x7f48741014c0 2023-11-16T16:49:44.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.050+0000 7f486bfff700 1 --2- 172.21.15.44:0/3457088797 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48741014c0 0x7f48741965b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f485c00e9a0 tx=0x7f485c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:44.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.051+0000 7f486b7fe700 1 -- 172.21.15.44:0/3457088797 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c009790 con 0x7f48741014c0 2023-11-16T16:49:44.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.051+0000 7f48789ff700 1 -- 172.21.15.44:0/3457088797 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f487410bc20 con 0x7f48741014c0 2023-11-16T16:49:44.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.051+0000 7f486b7fe700 1 -- 172.21.15.44:0/3457088797 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f485c00cbd0 con 0x7f48741014c0 2023-11-16T16:49:44.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.051+0000 7f486b7fe700 1 -- 172.21.15.44:0/3457088797 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c018680 con 0x7f48741014c0 2023-11-16T16:49:44.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.051+0000 7f48789ff700 1 -- 172.21.15.44:0/3457088797 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487410ba90 con 0x7f48741014c0 2023-11-16T16:49:44.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.052+0000 7f48789ff700 1 -- 172.21.15.44:0/3457088797 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f487419e5b0 con 0x7f48741014c0 2023-11-16T16:49:44.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.054+0000 7f486b7fe700 1 -- 172.21.15.44:0/3457088797 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f485c00cd40 con 0x7f48741014c0 2023-11-16T16:49:44.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.055+0000 7f486b7fe700 1 --2- 172.21.15.44:0/3457088797 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f48600a1150 0x7f48600a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:44.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.055+0000 7f486b7fe700 1 -- 172.21.15.44:0/3457088797 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f485c014070 con 0x7f48741014c0 2023-11-16T16:49:44.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.056+0000 7f487259c700 1 --2- 172.21.15.44:0/3457088797 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f48600a1150 0x7f48600a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:44.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.056+0000 7f487259c700 1 --2- 172.21.15.44:0/3457088797 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f48600a1150 0x7f48600a3600 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f486400b2e0 tx=0x7f4864005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:44.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.059+0000 7f486b7fe700 1 -- 172.21.15.44:0/3457088797 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f485c051730 con 0x7f48741014c0 2023-11-16T16:49:44.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:43 smithi044 ceph-mon[128750]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:49:44.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:43 smithi044 ceph-mon[128750]: from='client.25352 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:49:44.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:43 smithi044 ceph-mon[128750]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2023-11-16T16:49:44.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:43 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1518476530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:44.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:43 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1171758026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:44.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:43 smithi176 ceph-mon[131630]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:49:44.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:43 smithi176 ceph-mon[131630]: from='client.25352 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:49:44.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:43 smithi176 ceph-mon[131630]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2023-11-16T16:49:44.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:43 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1518476530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:44.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:43 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1171758026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:44.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.269+0000 7f48789ff700 1 -- 172.21.15.44:0/3457088797 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4874107200 con 0x7f48741014c0 2023-11-16T16:49:44.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.269+0000 7f486b7fe700 1 -- 172.21.15.44:0/3457088797 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f485c0962b0 con 0x7f48741014c0 2023-11-16T16:49:44.272 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2023-11-16T16:49:44.274 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.272+0000 7f48789ff700 1 -- 172.21.15.44:0/3457088797 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f48600a1150 msgr2=0x7f48600a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:44.274 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.272+0000 7f48789ff700 1 --2- 172.21.15.44:0/3457088797 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f48600a1150 0x7f48600a3600 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f486400b2e0 tx=0x7f4864005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.275 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.272+0000 7f48789ff700 1 -- 172.21.15.44:0/3457088797 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48741014c0 msgr2=0x7f48741965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:44.275 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.272+0000 7f48789ff700 1 --2- 172.21.15.44:0/3457088797 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48741014c0 0x7f48741965b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f485c00e9a0 tx=0x7f485c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.275 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.272+0000 7f48789ff700 1 -- 172.21.15.44:0/3457088797 shutdown_connections 2023-11-16T16:49:44.275 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.272+0000 7f48789ff700 1 --2- 172.21.15.44:0/3457088797 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f48600a1150 0x7f48600a3600 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.276 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.272+0000 7f48789ff700 1 --2- 172.21.15.44:0/3457088797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4874101e50 0x7f487419d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.276 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.272+0000 7f48789ff700 1 --2- 172.21.15.44:0/3457088797 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48741002d0 0x7f4874196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.276 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.272+0000 7f48789ff700 1 --2- 172.21.15.44:0/3457088797 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48741014c0 0x7f48741965b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.276 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.272+0000 7f48789ff700 1 -- 172.21.15.44:0/3457088797 >> 172.21.15.44:0/3457088797 conn(0x7f48740fb8b0 msgr2=0x7f48740fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:44.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.272+0000 7f48789ff700 1 -- 172.21.15.44:0/3457088797 shutdown_connections 2023-11-16T16:49:44.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.272+0000 7f48789ff700 1 -- 172.21.15.44:0/3457088797 wait complete. 2023-11-16T16:49:44.293 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T16:49:44.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.499+0000 7fe9163ff700 1 -- 172.21.15.44:0/425279349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9101002c0 msgr2=0x7fe9101006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:44.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.499+0000 7fe9163ff700 1 --2- 172.21.15.44:0/425279349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9101002c0 0x7fe9101006d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe900009990 tx=0x7fe900009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.499+0000 7fe9163ff700 1 -- 172.21.15.44:0/425279349 shutdown_connections 2023-11-16T16:49:44.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.499+0000 7fe9163ff700 1 --2- 172.21.15.44:0/425279349 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe910101e40 0x7fe910108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.499+0000 7fe9163ff700 1 --2- 172.21.15.44:0/425279349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9101002c0 0x7fe9101006d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.499+0000 7fe9163ff700 1 --2- 172.21.15.44:0/425279349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9101014b0 0x7fe910101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.499+0000 7fe9163ff700 1 -- 172.21.15.44:0/425279349 >> 172.21.15.44:0/425279349 conn(0x7fe9100fb8a0 msgr2=0x7fe9100fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:44.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.499+0000 7fe9163ff700 1 -- 172.21.15.44:0/425279349 shutdown_connections 2023-11-16T16:49:44.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.499+0000 7fe9163ff700 1 -- 172.21.15.44:0/425279349 wait complete. 2023-11-16T16:49:44.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.500+0000 7fe9163ff700 1 Processor -- start 2023-11-16T16:49:44.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.500+0000 7fe9163ff700 1 -- start start 2023-11-16T16:49:44.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.500+0000 7fe9163ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9101002c0 0x7fe910198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:44.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.500+0000 7fe9163ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9101014b0 0x7fe9101987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.500+0000 7fe9163ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe910101e40 0x7fe91019f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.500+0000 7fe9163ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91019fe00 con 0x7fe9101014b0 2023-11-16T16:49:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.500+0000 7fe9163ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91019ff40 con 0x7fe910101e40 2023-11-16T16:49:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.500+0000 7fe9163ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9101a00d0 con 0x7fe9101002c0 2023-11-16T16:49:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.501+0000 7fe90f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9101014b0 0x7fe9101987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.501+0000 7fe90f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9101014b0 0x7fe9101987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32932/0 (socket says 172.21.15.44:32932) 2023-11-16T16:49:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.501+0000 7fe90f7fe700 1 -- 172.21.15.44:0/3701383675 learned_addr learned my addr 172.21.15.44:0/3701383675 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.501+0000 7fe90ffff700 1 --2- 172.21.15.44:0/3701383675 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9101002c0 0x7fe910198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.501+0000 7fe91499c700 1 --2- 172.21.15.44:0/3701383675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe910101e40 0x7fe91019f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.501+0000 7fe90f7fe700 1 -- 172.21.15.44:0/3701383675 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9101002c0 msgr2=0x7fe910198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.501+0000 7fe90f7fe700 1 --2- 172.21.15.44:0/3701383675 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9101002c0 0x7fe910198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.501+0000 7fe90f7fe700 1 -- 172.21.15.44:0/3701383675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe910101e40 msgr2=0x7fe91019f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.501+0000 7fe90f7fe700 1 --2- 172.21.15.44:0/3701383675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe910101e40 0x7fe91019f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.501+0000 7fe90f7fe700 1 -- 172.21.15.44:0/3701383675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9000095f0 con 0x7fe9101014b0 2023-11-16T16:49:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.501+0000 7fe90f7fe700 1 --2- 172.21.15.44:0/3701383675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9101014b0 0x7fe9101987c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe90400e9a0 tx=0x7fe90400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.502+0000 7fe90d7fa700 1 -- 172.21.15.44:0/3701383675 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe904009790 con 0x7fe9101014b0 2023-11-16T16:49:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.502+0000 7fe90d7fa700 1 -- 172.21.15.44:0/3701383675 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe904010430 con 0x7fe9101014b0 2023-11-16T16:49:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.502+0000 7fe9163ff700 1 -- 172.21.15.44:0/3701383675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe91010bc90 con 0x7fe9101014b0 2023-11-16T16:49:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.502+0000 7fe90d7fa700 1 -- 172.21.15.44:0/3701383675 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe904018680 con 0x7fe9101014b0 2023-11-16T16:49:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.502+0000 7fe9163ff700 1 -- 172.21.15.44:0/3701383675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91010bb00 con 0x7fe9101014b0 2023-11-16T16:49:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.503+0000 7fe9163ff700 1 -- 172.21.15.44:0/3701383675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9101929f0 con 0x7fe9101014b0 2023-11-16T16:49:44.512 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.505+0000 7fe90d7fa700 1 -- 172.21.15.44:0/3701383675 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe9040105a0 con 0x7fe9101014b0 2023-11-16T16:49:44.512 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.506+0000 7fe90d7fa700 1 --2- 172.21.15.44:0/3701383675 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe8f80a11d0 0x7fe8f80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:44.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.506+0000 7fe90d7fa700 1 -- 172.21.15.44:0/3701383675 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fe904014070 con 0x7fe9101014b0 2023-11-16T16:49:44.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.506+0000 7fe90ffff700 1 --2- 172.21.15.44:0/3701383675 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe8f80a11d0 0x7fe8f80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:44.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.507+0000 7fe90ffff700 1 --2- 172.21.15.44:0/3701383675 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe8f80a11d0 0x7fe8f80a3680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe90000b2e0 tx=0x7fe900005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:44.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.510+0000 7fe90d7fa700 1 -- 172.21.15.44:0/3701383675 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe904051730 con 0x7fe9101014b0 2023-11-16T16:49:44.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.724+0000 7fe9163ff700 1 -- 172.21.15.44:0/3701383675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe910061c40 con 0x7fe9101014b0 2023-11-16T16:49:44.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.725+0000 7fe90d7fa700 1 -- 172.21.15.44:0/3701383675 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe9040d3080 con 0x7fe9101014b0 2023-11-16T16:49:44.728 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T16:49:44.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.727+0000 7fe9163ff700 1 -- 172.21.15.44:0/3701383675 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe8f80a11d0 msgr2=0x7fe8f80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:44.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.727+0000 7fe9163ff700 1 --2- 172.21.15.44:0/3701383675 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe8f80a11d0 0x7fe8f80a3680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe90000b2e0 tx=0x7fe900005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.727+0000 7fe9163ff700 1 -- 172.21.15.44:0/3701383675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9101014b0 msgr2=0x7fe9101987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:44.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.727+0000 7fe9163ff700 1 --2- 172.21.15.44:0/3701383675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9101014b0 0x7fe9101987c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe90400e9a0 tx=0x7fe90400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.727+0000 7fe9163ff700 1 -- 172.21.15.44:0/3701383675 shutdown_connections 2023-11-16T16:49:44.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.727+0000 7fe9163ff700 1 --2- 172.21.15.44:0/3701383675 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe8f80a11d0 0x7fe8f80a3680 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.727+0000 7fe9163ff700 1 --2- 172.21.15.44:0/3701383675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe910101e40 0x7fe91019f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.727+0000 7fe9163ff700 1 --2- 172.21.15.44:0/3701383675 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9101002c0 0x7fe910198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.732 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.727+0000 7fe9163ff700 1 --2- 172.21.15.44:0/3701383675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9101014b0 0x7fe9101987c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.732 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.727+0000 7fe9163ff700 1 -- 172.21.15.44:0/3701383675 >> 172.21.15.44:0/3701383675 conn(0x7fe9100fb8a0 msgr2=0x7fe910106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:44.732 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.728+0000 7fe9163ff700 1 -- 172.21.15.44:0/3701383675 shutdown_connections 2023-11-16T16:49:44.732 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.728+0000 7fe9163ff700 1 -- 172.21.15.44:0/3701383675 wait complete. 2023-11-16T16:49:44.749 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:49:44.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.952+0000 7f26c87ff700 1 -- 172.21.15.44:0/3792311275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26c0101e10 msgr2=0x7f26c01086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:44.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.952+0000 7f26c87ff700 1 --2- 172.21.15.44:0/3792311275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26c0101e10 0x7f26c01086d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f26b800b0a0 tx=0x7f26b800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.953+0000 7f26c87ff700 1 -- 172.21.15.44:0/3792311275 shutdown_connections 2023-11-16T16:49:44.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.953+0000 7f26c87ff700 1 --2- 172.21.15.44:0/3792311275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26c0101e10 0x7f26c01086d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.953+0000 7f26c87ff700 1 --2- 172.21.15.44:0/3792311275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c0100290 0x7f26c01006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.953+0000 7f26c87ff700 1 --2- 172.21.15.44:0/3792311275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c0101480 0x7f26c01018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.953+0000 7f26c87ff700 1 -- 172.21.15.44:0/3792311275 >> 172.21.15.44:0/3792311275 conn(0x7f26c00fb870 msgr2=0x7f26c00fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:44.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.953+0000 7f26c87ff700 1 -- 172.21.15.44:0/3792311275 shutdown_connections 2023-11-16T16:49:44.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.953+0000 7f26c87ff700 1 -- 172.21.15.44:0/3792311275 wait complete. 2023-11-16T16:49:44.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.954+0000 7f26c87ff700 1 Processor -- start 2023-11-16T16:49:44.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.954+0000 7f26c87ff700 1 -- start start 2023-11-16T16:49:44.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.954+0000 7f26c87ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c0100290 0x7f26c0198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:44.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.954+0000 7f26c87ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26c0101480 0x7f26c01987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:44.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.954+0000 7f26c87ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c0101e10 0x7f26c019f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:44.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.954+0000 7f26c87ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c019fe10 con 0x7f26c0101480 2023-11-16T16:49:44.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.954+0000 7f26c87ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c019ff50 con 0x7f26c0100290 2023-11-16T16:49:44.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.954+0000 7f26c87ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c01a00e0 con 0x7f26c0101e10 2023-11-16T16:49:44.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.954+0000 7f26c5d9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26c0101480 0x7f26c01987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:44.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.955+0000 7f26c6d9c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c0101e10 0x7f26c019f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:44.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.955+0000 7f26c6d9c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c0101e10 0x7f26c019f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:51416/0 (socket says 172.21.15.44:51416) 2023-11-16T16:49:44.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.955+0000 7f26c6d9c700 1 -- 172.21.15.44:0/3170013193 learned_addr learned my addr 172.21.15.44:0/3170013193 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:44.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.955+0000 7f26c659b700 1 --2- 172.21.15.44:0/3170013193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c0100290 0x7f26c0198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:44.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.955+0000 7f26c6d9c700 1 -- 172.21.15.44:0/3170013193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c0100290 msgr2=0x7f26c0198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:44.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.955+0000 7f26c6d9c700 1 --2- 172.21.15.44:0/3170013193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c0100290 0x7f26c0198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.955+0000 7f26c6d9c700 1 -- 172.21.15.44:0/3170013193 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26c0101480 msgr2=0x7f26c01987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:44.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.955+0000 7f26c6d9c700 1 --2- 172.21.15.44:0/3170013193 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26c0101480 0x7f26c01987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.955+0000 7f26c6d9c700 1 -- 172.21.15.44:0/3170013193 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b8009d00 con 0x7f26c0101e10 2023-11-16T16:49:44.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.955+0000 7f26c6d9c700 1 --2- 172.21.15.44:0/3170013193 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c0101e10 0x7f26c019f840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f26b800d010 tx=0x7f26b800bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:44.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.955+0000 7f26b77fe700 1 -- 172.21.15.44:0/3170013193 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b800e040 con 0x7f26c0101e10 2023-11-16T16:49:44.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.956+0000 7f26b77fe700 1 -- 172.21.15.44:0/3170013193 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f26b800bdb0 con 0x7f26c0101e10 2023-11-16T16:49:44.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.956+0000 7f26c87ff700 1 -- 172.21.15.44:0/3170013193 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c010bc70 con 0x7f26c0101e10 2023-11-16T16:49:44.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.956+0000 7f26b77fe700 1 -- 172.21.15.44:0/3170013193 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b8005260 con 0x7f26c0101e10 2023-11-16T16:49:44.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.956+0000 7f26c87ff700 1 -- 172.21.15.44:0/3170013193 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c010bae0 con 0x7f26c0101e10 2023-11-16T16:49:44.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.957+0000 7f26c87ff700 1 -- 172.21.15.44:0/3170013193 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c0067a60 con 0x7f26c0101e10 2023-11-16T16:49:44.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.959+0000 7f26b77fe700 1 -- 172.21.15.44:0/3170013193 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f26b802e020 con 0x7f26c0101e10 2023-11-16T16:49:44.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.960+0000 7f26b77fe700 1 --2- 172.21.15.44:0/3170013193 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f26ac0a1150 0x7f26ac0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:44.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.960+0000 7f26b77fe700 1 -- 172.21.15.44:0/3170013193 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f26b80ce170 con 0x7f26c0101e10 2023-11-16T16:49:44.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.960+0000 7f26c659b700 1 --2- 172.21.15.44:0/3170013193 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f26ac0a1150 0x7f26ac0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:44.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.961+0000 7f26c659b700 1 --2- 172.21.15.44:0/3170013193 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f26ac0a1150 0x7f26ac0a3600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f26c01012e0 tx=0x7f26bc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:44.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:44.962+0000 7f26b77fe700 1 -- 172.21.15.44:0/3170013193 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f26b8097550 con 0x7f26c0101e10 2023-11-16T16:49:44.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:44 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/875234620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:44.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:44 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3457088797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:45.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:44 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/875234620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:45.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:44 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3457088797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:45.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:44 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/875234620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:45.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:44 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3457088797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.215+0000 7f26c87ff700 1 -- 172.21.15.44:0/3170013193 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26c01a0c00 con 0x7f26c0101e10 2023-11-16T16:49:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.216+0000 7f26b77fe700 1 -- 172.21.15.44:0/3170013193 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f26b809ab70 con 0x7f26c0101e10 2023-11-16T16:49:45.219 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:45.219 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:49:45.221 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.219+0000 7f26c87ff700 1 -- 172.21.15.44:0/3170013193 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f26ac0a1150 msgr2=0x7f26ac0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.221 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.219+0000 7f26c87ff700 1 --2- 172.21.15.44:0/3170013193 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f26ac0a1150 0x7f26ac0a3600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f26c01012e0 tx=0x7f26bc009260 comp rx=0 tx=0).stop 2023-11-16T16:49:45.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.219+0000 7f26c87ff700 1 -- 172.21.15.44:0/3170013193 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c0101e10 msgr2=0x7f26c019f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.219+0000 7f26c87ff700 1 --2- 172.21.15.44:0/3170013193 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c0101e10 0x7f26c019f840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f26b800d010 tx=0x7f26b800bc80 comp rx=0 tx=0).stop 2023-11-16T16:49:45.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.219+0000 7f26c87ff700 1 -- 172.21.15.44:0/3170013193 shutdown_connections 2023-11-16T16:49:45.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.219+0000 7f26c87ff700 1 --2- 172.21.15.44:0/3170013193 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f26ac0a1150 0x7f26ac0a3600 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.219+0000 7f26c87ff700 1 --2- 172.21.15.44:0/3170013193 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c0101e10 0x7f26c019f840 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.219+0000 7f26c87ff700 1 --2- 172.21.15.44:0/3170013193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c0100290 0x7f26c0198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.219+0000 7f26c87ff700 1 --2- 172.21.15.44:0/3170013193 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26c0101480 0x7f26c01987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.219+0000 7f26c87ff700 1 -- 172.21.15.44:0/3170013193 >> 172.21.15.44:0/3170013193 conn(0x7f26c00fb870 msgr2=0x7f26c00fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:45.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.219+0000 7f26c87ff700 1 -- 172.21.15.44:0/3170013193 shutdown_connections 2023-11-16T16:49:45.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.219+0000 7f26c87ff700 1 -- 172.21.15.44:0/3170013193 wait complete. 2023-11-16T16:49:45.223 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:49:45.240 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:49:45.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.444+0000 7f4cacbff700 1 -- 172.21.15.44:0/731370193 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ca8101480 msgr2=0x7f4ca81018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.444+0000 7f4cacbff700 1 --2- 172.21.15.44:0/731370193 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ca8101480 0x7f4ca81018d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009990 tx=0x7f4c9c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.444+0000 7f4cacbff700 1 -- 172.21.15.44:0/731370193 shutdown_connections 2023-11-16T16:49:45.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.444+0000 7f4cacbff700 1 --2- 172.21.15.44:0/731370193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ca8101e10 0x7f4ca81086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.444+0000 7f4cacbff700 1 --2- 172.21.15.44:0/731370193 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ca8100290 0x7f4ca81006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.444+0000 7f4cacbff700 1 --2- 172.21.15.44:0/731370193 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ca8101480 0x7f4ca81018d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.444+0000 7f4cacbff700 1 -- 172.21.15.44:0/731370193 >> 172.21.15.44:0/731370193 conn(0x7f4ca80fb870 msgr2=0x7f4ca80fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:45.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.444+0000 7f4cacbff700 1 -- 172.21.15.44:0/731370193 shutdown_connections 2023-11-16T16:49:45.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.444+0000 7f4cacbff700 1 -- 172.21.15.44:0/731370193 wait complete. 2023-11-16T16:49:45.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.445+0000 7f4cacbff700 1 Processor -- start 2023-11-16T16:49:45.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.445+0000 7f4cacbff700 1 -- start start 2023-11-16T16:49:45.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.445+0000 7f4cacbff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ca8100290 0x7f4ca80748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.445+0000 7f4cacbff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ca8101480 0x7f4ca8072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.445+0000 7f4cacbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ca8101e10 0x7f4ca8073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.446+0000 7f4cacbff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca8073bc0 con 0x7f4ca8101480 2023-11-16T16:49:45.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.446+0000 7f4cacbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca8073d30 con 0x7f4ca8101e10 2023-11-16T16:49:45.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.446+0000 7f4cacbff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca819fff0 con 0x7f4ca8100290 2023-11-16T16:49:45.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.446+0000 7f4ca5d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ca8101480 0x7f4ca8072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.446+0000 7f4ca5d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ca8101480 0x7f4ca8072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56450/0 (socket says 172.21.15.44:56450) 2023-11-16T16:49:45.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.446+0000 7f4ca5d9b700 1 -- 172.21.15.44:0/1175923456 learned_addr learned my addr 172.21.15.44:0/1175923456 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:45.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.446+0000 7f4ca659c700 1 --2- 172.21.15.44:0/1175923456 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ca8100290 0x7f4ca80748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.446+0000 7f4ca6d9d700 1 --2- 172.21.15.44:0/1175923456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ca8101e10 0x7f4ca8073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.446+0000 7f4ca5d9b700 1 -- 172.21.15.44:0/1175923456 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ca8100290 msgr2=0x7f4ca80748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.446+0000 7f4ca5d9b700 1 --2- 172.21.15.44:0/1175923456 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ca8100290 0x7f4ca80748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.446+0000 7f4ca5d9b700 1 -- 172.21.15.44:0/1175923456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ca8101e10 msgr2=0x7f4ca8073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.446+0000 7f4ca5d9b700 1 --2- 172.21.15.44:0/1175923456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ca8101e10 0x7f4ca8073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.446+0000 7f4ca5d9b700 1 -- 172.21.15.44:0/1175923456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9c0095f0 con 0x7f4ca8101480 2023-11-16T16:49:45.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.446+0000 7f4ca5d9b700 1 --2- 172.21.15.44:0/1175923456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ca8101480 0x7f4ca8072ee0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c000c00 tx=0x7f4c9c0047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:45.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.447+0000 7f4c977fe700 1 -- 172.21.15.44:0/1175923456 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c01d070 con 0x7f4ca8101480 2023-11-16T16:49:45.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.447+0000 7f4c977fe700 1 -- 172.21.15.44:0/1175923456 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c9c022470 con 0x7f4ca8101480 2023-11-16T16:49:45.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.447+0000 7f4cacbff700 1 -- 172.21.15.44:0/1175923456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca810bc00 con 0x7f4ca8101480 2023-11-16T16:49:45.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.447+0000 7f4c977fe700 1 -- 172.21.15.44:0/1175923456 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c017650 con 0x7f4ca8101480 2023-11-16T16:49:45.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.447+0000 7f4cacbff700 1 -- 172.21.15.44:0/1175923456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca810ba70 con 0x7f4ca8101480 2023-11-16T16:49:45.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.448+0000 7f4cacbff700 1 -- 172.21.15.44:0/1175923456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca8107210 con 0x7f4ca8101480 2023-11-16T16:49:45.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.450+0000 7f4c977fe700 1 -- 172.21.15.44:0/1175923456 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4c9c0225e0 con 0x7f4ca8101480 2023-11-16T16:49:45.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.451+0000 7f4c977fe700 1 --2- 172.21.15.44:0/1175923456 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4c900a11d0 0x7f4c900a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.451+0000 7f4c977fe700 1 -- 172.21.15.44:0/1175923456 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f4c9c0ce200 con 0x7f4ca8101480 2023-11-16T16:49:45.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.452+0000 7f4ca659c700 1 --2- 172.21.15.44:0/1175923456 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4c900a11d0 0x7f4c900a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.452+0000 7f4ca659c700 1 --2- 172.21.15.44:0/1175923456 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4c900a11d0 0x7f4c900a3680 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4ca81012e0 tx=0x7f4c98009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:45.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.455+0000 7f4c977fe700 1 -- 172.21.15.44:0/1175923456 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4c9c0975e0 con 0x7f4ca8101480 2023-11-16T16:49:45.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.665+0000 7f4cacbff700 1 -- 172.21.15.44:0/1175923456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ca802cfe0 con 0x7f4ca8101480 2023-11-16T16:49:45.669 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.667+0000 7f4c977fe700 1 -- 172.21.15.44:0/1175923456 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24685 (secure 0 0 0) 0x7f4c9c09ac00 con 0x7f4ca8101480 2023-11-16T16:49:45.669 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:49:45.670 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T16:48:44.613706+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:49:45.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.669+0000 7f4cacbff700 1 -- 172.21.15.44:0/1175923456 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4c900a11d0 msgr2=0x7f4c900a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.669+0000 7f4cacbff700 1 --2- 172.21.15.44:0/1175923456 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4c900a11d0 0x7f4c900a3680 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4ca81012e0 tx=0x7f4c98009260 comp rx=0 tx=0).stop 2023-11-16T16:49:45.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.669+0000 7f4cacbff700 1 -- 172.21.15.44:0/1175923456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ca8101480 msgr2=0x7f4ca8072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.669+0000 7f4cacbff700 1 --2- 172.21.15.44:0/1175923456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ca8101480 0x7f4ca8072ee0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c000c00 tx=0x7f4c9c0047a0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.670+0000 7f4cacbff700 1 -- 172.21.15.44:0/1175923456 shutdown_connections 2023-11-16T16:49:45.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.670+0000 7f4cacbff700 1 --2- 172.21.15.44:0/1175923456 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f4c900a11d0 0x7f4c900a3680 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.670+0000 7f4cacbff700 1 --2- 172.21.15.44:0/1175923456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ca8101e10 0x7f4ca8073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.670+0000 7f4cacbff700 1 --2- 172.21.15.44:0/1175923456 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ca8100290 0x7f4ca80748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.670+0000 7f4cacbff700 1 --2- 172.21.15.44:0/1175923456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ca8101480 0x7f4ca8072ee0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.670+0000 7f4cacbff700 1 -- 172.21.15.44:0/1175923456 >> 172.21.15.44:0/1175923456 conn(0x7f4ca80fb870 msgr2=0x7f4ca80fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:45.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.670+0000 7f4cacbff700 1 -- 172.21.15.44:0/1175923456 shutdown_connections 2023-11-16T16:49:45.673 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.670+0000 7f4cacbff700 1 -- 172.21.15.44:0/1175923456 wait complete. 2023-11-16T16:49:45.691 DEBUG:tasks.fwd_scrub:Yielding 2023-11-16T16:49:45.692 INFO:teuthology.run_tasks:Running task exec... 2023-11-16T16:49:45.703 INFO:teuthology.task.exec:Executing custom commands... 2023-11-16T16:49:45.703 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2023-11-16T16:49:45.704 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2023-11-16T16:49:45.704 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2023-11-16T16:49:45.704 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:49:45.705 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:49:45.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.909+0000 7f52bb1ff700 1 -- 172.21.15.44:0/2587834815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52b4074ab0 msgr2=0x7f52b4072f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.909+0000 7f52bb1ff700 1 --2- 172.21.15.44:0/2587834815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52b4074ab0 0x7f52b4072f00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f52a80098f0 tx=0x7f52a8009c00 comp rx=0 tx=0).stop 2023-11-16T16:49:45.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.909+0000 7f52bb1ff700 1 -- 172.21.15.44:0/2587834815 shutdown_connections 2023-11-16T16:49:45.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.909+0000 7f52bb1ff700 1 --2- 172.21.15.44:0/2587834815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52b41080f0 0x7f52b410a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.909+0000 7f52bb1ff700 1 --2- 172.21.15.44:0/2587834815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52b4074ab0 0x7f52b4072f00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.909+0000 7f52bb1ff700 1 --2- 172.21.15.44:0/2587834815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52b40734d0 0x7f52b4073940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.909+0000 7f52bb1ff700 1 -- 172.21.15.44:0/2587834815 >> 172.21.15.44:0/2587834815 conn(0x7f52b40fb840 msgr2=0x7f52b40fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:45.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.909+0000 7f52bb1ff700 1 -- 172.21.15.44:0/2587834815 shutdown_connections 2023-11-16T16:49:45.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.909+0000 7f52bb1ff700 1 -- 172.21.15.44:0/2587834815 wait complete. 2023-11-16T16:49:45.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.910+0000 7f52bb1ff700 1 Processor -- start 2023-11-16T16:49:45.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.911+0000 7f52bb1ff700 1 -- start start 2023-11-16T16:49:45.914 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.911+0000 7f52bb1ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52b40734d0 0x7f52b4198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.914 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.911+0000 7f52bb1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52b4074ab0 0x7f52b41987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.914 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.911+0000 7f52bb1ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52b41080f0 0x7f52b419f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.911+0000 7f52bb1ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b419fdf0 con 0x7f52b41080f0 2023-11-16T16:49:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.911+0000 7f52bb1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b419ff30 con 0x7f52b4074ab0 2023-11-16T16:49:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.911+0000 7f52bb1ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b41a00c0 con 0x7f52b40734d0 2023-11-16T16:49:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.911+0000 7f52b3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52b4074ab0 0x7f52b41987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.911+0000 7f52b979c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52b41080f0 0x7f52b419f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.911+0000 7f52b8f9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52b40734d0 0x7f52b4198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.911+0000 7f52b3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52b4074ab0 0x7f52b41987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:57588/0 (socket says 172.21.15.44:57588) 2023-11-16T16:49:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.912+0000 7f52b3fff700 1 -- 172.21.15.44:0/3429799178 learned_addr learned my addr 172.21.15.44:0/3429799178 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.912+0000 7f52b3fff700 1 -- 172.21.15.44:0/3429799178 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52b40734d0 msgr2=0x7f52b4198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.912+0000 7f52b3fff700 1 --2- 172.21.15.44:0/3429799178 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52b40734d0 0x7f52b4198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.912+0000 7f52b3fff700 1 -- 172.21.15.44:0/3429799178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52b41080f0 msgr2=0x7f52b419f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.912+0000 7f52b3fff700 1 --2- 172.21.15.44:0/3429799178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52b41080f0 0x7f52b419f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.912+0000 7f52b3fff700 1 -- 172.21.15.44:0/3429799178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a8009520 con 0x7f52b4074ab0 2023-11-16T16:49:45.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.912+0000 7f52b979c700 1 --2- 172.21.15.44:0/3429799178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52b41080f0 0x7f52b419f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:45.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.912+0000 7f52b3fff700 1 --2- 172.21.15.44:0/3429799178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52b4074ab0 0x7f52b41987b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f52a000e9a0 tx=0x7f52a000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:45.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.913+0000 7f52b1ffb700 1 -- 172.21.15.44:0/3429799178 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a0009790 con 0x7f52b4074ab0 2023-11-16T16:49:45.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.913+0000 7f52bb1ff700 1 -- 172.21.15.44:0/3429799178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b410dde0 con 0x7f52b4074ab0 2023-11-16T16:49:45.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.913+0000 7f52b1ffb700 1 -- 172.21.15.44:0/3429799178 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f52a0010430 con 0x7f52b4074ab0 2023-11-16T16:49:45.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.913+0000 7f52b1ffb700 1 -- 172.21.15.44:0/3429799178 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a0018680 con 0x7f52b4074ab0 2023-11-16T16:49:45.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.913+0000 7f52bb1ff700 1 -- 172.21.15.44:0/3429799178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b410dc50 con 0x7f52b4074ab0 2023-11-16T16:49:45.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.914+0000 7f52bb1ff700 1 -- 172.21.15.44:0/3429799178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b4106830 con 0x7f52b4074ab0 2023-11-16T16:49:45.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.916+0000 7f52b1ffb700 1 -- 172.21.15.44:0/3429799178 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f52a00105a0 con 0x7f52b4074ab0 2023-11-16T16:49:45.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.917+0000 7f52b1ffb700 1 --2- 172.21.15.44:0/3429799178 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f52a40a11d0 0x7f52a40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.917+0000 7f52b1ffb700 1 -- 172.21.15.44:0/3429799178 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f52a0014070 con 0x7f52b4074ab0 2023-11-16T16:49:45.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.917+0000 7f52b8f9b700 1 --2- 172.21.15.44:0/3429799178 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f52a40a11d0 0x7f52a40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.918+0000 7f52b8f9b700 1 --2- 172.21.15.44:0/3429799178 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f52a40a11d0 0x7f52a40a3680 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f52a8005e90 tx=0x7f52a8019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:45.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.920+0000 7f5921dff700 1 -- 172.21.15.44:0/1463950795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f591c103660 msgr2=0x7f591c105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.920+0000 7f5921dff700 1 --2- 172.21.15.44:0/1463950795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f591c103660 0x7f591c105a40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5910009990 tx=0x7f5910009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.920+0000 7f5921dff700 1 -- 172.21.15.44:0/1463950795 shutdown_connections 2023-11-16T16:49:45.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.920+0000 7f5921dff700 1 --2- 172.21.15.44:0/1463950795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f591c105f80 0x7f591c1083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.920+0000 7f5921dff700 1 --2- 172.21.15.44:0/1463950795 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f591c100d40 0x7f591c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.920+0000 7f5921dff700 1 --2- 172.21.15.44:0/1463950795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f591c103660 0x7f591c105a40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.920+0000 7f5921dff700 1 -- 172.21.15.44:0/1463950795 >> 172.21.15.44:0/1463950795 conn(0x7f591c0fa7c0 msgr2=0x7f591c0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:45.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.920+0000 7f5921dff700 1 -- 172.21.15.44:0/1463950795 shutdown_connections 2023-11-16T16:49:45.923 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.920+0000 7f5921dff700 1 -- 172.21.15.44:0/1463950795 wait complete. 2023-11-16T16:49:45.923 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.921+0000 7f5921dff700 1 Processor -- start 2023-11-16T16:49:45.923 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.921+0000 7f5921dff700 1 -- start start 2023-11-16T16:49:45.924 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.921+0000 7f52b1ffb700 1 -- 172.21.15.44:0/3429799178 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f52a00960a0 con 0x7f52b4074ab0 2023-11-16T16:49:45.925 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.921+0000 7f5921dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f591c100d40 0x7f591c198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.925 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.921+0000 7f5921dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f591c103660 0x7f591c1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.925 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.921+0000 7f5921dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f591c105f80 0x7f591c19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.921+0000 7f5921dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c19fdf0 con 0x7f591c100d40 2023-11-16T16:49:45.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.921+0000 7f5921dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c19ff30 con 0x7f591c103660 2023-11-16T16:49:45.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.921+0000 7f5921dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c1a00c0 con 0x7f591c105f80 2023-11-16T16:49:45.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.922+0000 7f591b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f591c100d40 0x7f591c198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.922+0000 7f591b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f591c100d40 0x7f591c198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56486/0 (socket says 172.21.15.44:56486) 2023-11-16T16:49:45.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.922+0000 7f591b7fe700 1 -- 172.21.15.44:0/1288529078 learned_addr learned my addr 172.21.15.44:0/1288529078 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:45.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.922+0000 7f591affd700 1 --2- 172.21.15.44:0/1288529078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f591c103660 0x7f591c1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.922+0000 7f591bfff700 1 --2- 172.21.15.44:0/1288529078 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f591c105f80 0x7f591c19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.922+0000 7f591b7fe700 1 -- 172.21.15.44:0/1288529078 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f591c105f80 msgr2=0x7f591c19f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.922+0000 7f591b7fe700 1 --2- 172.21.15.44:0/1288529078 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f591c105f80 0x7f591c19f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.922+0000 7f591b7fe700 1 -- 172.21.15.44:0/1288529078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f591c103660 msgr2=0x7f591c1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.927 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.922+0000 7f591b7fe700 1 --2- 172.21.15.44:0/1288529078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f591c103660 0x7f591c1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.927 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.922+0000 7f591b7fe700 1 -- 172.21.15.44:0/1288529078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59100095f0 con 0x7f591c100d40 2023-11-16T16:49:45.927 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.922+0000 7f591bfff700 1 --2- 172.21.15.44:0/1288529078 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f591c105f80 0x7f591c19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:49:45.927 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.922+0000 7f591b7fe700 1 --2- 172.21.15.44:0/1288529078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f591c100d40 0x7f591c198270 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f590c00e9a0 tx=0x7f590c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:45.927 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.923+0000 7f5918ff9700 1 -- 172.21.15.44:0/1288529078 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c009790 con 0x7f591c100d40 2023-11-16T16:49:45.927 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.923+0000 7f5918ff9700 1 -- 172.21.15.44:0/1288529078 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f590c00cbd0 con 0x7f591c100d40 2023-11-16T16:49:45.927 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.923+0000 7f5921dff700 1 -- 172.21.15.44:0/1288529078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591c10bc80 con 0x7f591c100d40 2023-11-16T16:49:45.927 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.923+0000 7f5918ff9700 1 -- 172.21.15.44:0/1288529078 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c018680 con 0x7f591c100d40 2023-11-16T16:49:45.927 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.923+0000 7f5921dff700 1 -- 172.21.15.44:0/1288529078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f591c10baf0 con 0x7f591c100d40 2023-11-16T16:49:45.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.927+0000 7f5935633b80 1 Processor -- start 2023-11-16T16:49:45.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.928+0000 7f5935633b80 1 -- start start 2023-11-16T16:49:45.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.928+0000 7f5918ff9700 1 -- 172.21.15.44:0/1288529078 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f590c00cd40 con 0x7f591c100d40 2023-11-16T16:49:45.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.928+0000 7f5935633b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222c520 0x55855222c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.928+0000 7f5935633b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222d3b0 0x55855222d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.928+0000 7f5935633b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222dd60 0x558552234600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.928+0000 7f5935633b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585521290e0 con 0x55855222c520 2023-11-16T16:49:45.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.928+0000 7f5935633b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55855217b920 con 0x55855222d3b0 2023-11-16T16:49:45.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.928+0000 7f5935633b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x558552174db0 con 0x55855222dd60 2023-11-16T16:49:45.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.928+0000 7f591bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222c520 0x55855222c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.928+0000 7f591bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222c520 0x55855222c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56494/0 (socket says 172.21.15.44:56494) 2023-11-16T16:49:45.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.928+0000 7f591bfff700 1 -- 172.21.15.44:0/855508498 learned_addr learned my addr 172.21.15.44:0/855508498 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:45.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.928+0000 7f591b7fe700 1 --2- 172.21.15.44:0/855508498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222d3b0 0x55855222d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.928+0000 7f591affd700 1 --2- 172.21.15.44:0/855508498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222dd60 0x558552234600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.929+0000 7f591bfff700 1 -- 172.21.15.44:0/855508498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222dd60 msgr2=0x558552234600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:45.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.929+0000 7f591bfff700 1 --2- 172.21.15.44:0/855508498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222dd60 0x558552234600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.929+0000 7f591bfff700 1 -- 172.21.15.44:0/855508498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222d3b0 msgr2=0x55855222d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:45.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.929+0000 7f591bfff700 1 --2- 172.21.15.44:0/855508498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222d3b0 0x55855222d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.929+0000 7f591bfff700 1 -- 172.21.15.44:0/855508498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558551ff66b0 con 0x55855222c520 2023-11-16T16:49:45.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.929+0000 7f591bfff700 1 --2- 172.21.15.44:0/855508498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222c520 0x55855222c930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5914009e50 tx=0x7f59140077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c4b1027971cb15 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:45.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.930+0000 7f5901ffb700 1 -- 172.21.15.44:0/855508498 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5914014610 con 0x55855222c520 2023-11-16T16:49:45.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.930+0000 7f5901ffb700 1 -- 172.21.15.44:0/855508498 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f591400f050 con 0x55855222c520 2023-11-16T16:49:45.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.930+0000 7f5901ffb700 1 -- 172.21.15.44:0/855508498 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5914013a70 con 0x55855222c520 2023-11-16T16:49:45.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.930+0000 7f5918ff9700 1 --2- 172.21.15.44:0/1288529078 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f59040a11d0 0x7f59040a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.934 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.930+0000 7f5918ff9700 1 -- 172.21.15.44:0/1288529078 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f590c014070 con 0x7f591c100d40 2023-11-16T16:49:45.934 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.930+0000 7f591b7fe700 1 --2- 172.21.15.44:0/1288529078 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f59040a11d0 0x7f59040a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.931+0000 7f591b7fe700 1 --2- 172.21.15.44:0/1288529078 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f59040a11d0 0x7f59040a3680 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f590c099bf0 tx=0x7f590c0d0760 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:45.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.931+0000 7f5935633b80 1 -- 172.21.15.44:0/855508498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222c520 msgr2=0x55855222c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.931+0000 7f5935633b80 1 --2- 172.21.15.44:0/855508498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222c520 0x55855222c930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5914009e50 tx=0x7f59140077e0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.931+0000 7f5935633b80 1 -- 172.21.15.44:0/855508498 shutdown_connections 2023-11-16T16:49:45.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.931+0000 7f5935633b80 1 --2- 172.21.15.44:0/855508498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222dd60 0x558552234600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.931+0000 7f5935633b80 1 --2- 172.21.15.44:0/855508498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222c520 0x55855222c930 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.931+0000 7f5935633b80 1 --2- 172.21.15.44:0/855508498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222d3b0 0x55855222d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.931+0000 7f5935633b80 1 -- 172.21.15.44:0/855508498 >> 172.21.15.44:0/855508498 conn(0x5585522193e0 msgr2=0x55855221e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.931+0000 7f5935633b80 1 -- 172.21.15.44:0/855508498 shutdown_connections 2023-11-16T16:49:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.931+0000 7f5935633b80 1 -- 172.21.15.44:0/855508498 wait complete. 2023-11-16T16:49:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.931+0000 7f5935633b80 1 -- start start 2023-11-16T16:49:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.932+0000 7f5935633b80 1 Processor -- start 2023-11-16T16:49:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.932+0000 7f5935633b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222c520 0x558552235e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.932+0000 7f5935633b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222d3b0 0x5585522363c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.932+0000 7f5935633b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222dd60 0x55855223d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.932+0000 7f5935633b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x558552116960 con 0x55855222c520 2023-11-16T16:49:45.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.932+0000 7f5935633b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x558552116600 con 0x55855222dd60 2023-11-16T16:49:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.932+0000 7f5935633b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x558551ff89a0 con 0x55855222d3b0 2023-11-16T16:49:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.932+0000 7f5935633b80 10 client.0 _ll_register_callbacks cb 0x558552134800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:49:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.932+0000 7f591bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222c520 0x558552235e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.932+0000 7f591bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222c520 0x558552235e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56508/0 (socket says 172.21.15.44:56508) 2023-11-16T16:49:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.932+0000 7f591bfff700 1 -- 172.21.15.44:0/1537224024 learned_addr learned my addr 172.21.15.44:0/1537224024 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.933+0000 7f591affd700 1 --2- 172.21.15.44:0/1537224024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222d3b0 0x5585522363c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.933+0000 7f591bfff700 1 --2- 172.21.15.44:0/1537224024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222dd60 0x55855223d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.933+0000 7f591bfff700 1 -- 172.21.15.44:0/1537224024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222d3b0 msgr2=0x5585522363c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.933+0000 7f591bfff700 1 --2- 172.21.15.44:0/1537224024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222d3b0 0x5585522363c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.933+0000 7f591bfff700 1 -- 172.21.15.44:0/1537224024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222dd60 msgr2=0x55855223d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.933+0000 7f591bfff700 1 --2- 172.21.15.44:0/1537224024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222dd60 0x55855223d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.933+0000 7f591bfff700 1 -- 172.21.15.44:0/1537224024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5914009a80 con 0x55855222c520 2023-11-16T16:49:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.933+0000 7f591bfff700 1 --2- 172.21.15.44:0/1537224024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222c520 0x558552235e80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5914004fd0 tx=0x7f5914015fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb7d019bca08929 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.933+0000 7f58faffd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:49:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.933+0000 7f58faffd700 1 -- 172.21.15.44:0/1537224024 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5914013eb0 con 0x55855222c520 2023-11-16T16:49:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.933+0000 7f5935633b80 1 -- 172.21.15.44:0/1537224024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558552216ba0 con 0x55855222c520 2023-11-16T16:49:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.934+0000 7f58faffd700 1 -- 172.21.15.44:0/1537224024 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f5914003a00 con 0x55855222c520 2023-11-16T16:49:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.934+0000 7f58faffd700 1 -- 172.21.15.44:0/1537224024 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5914003bc0 con 0x55855222c520 2023-11-16T16:49:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.934+0000 7f5935633b80 10 client.15393 fetch_fsmap learned FSMap version 21 2023-11-16T16:49:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.934+0000 7f5935633b80 1 -- 172.21.15.44:0/1537224024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55855213b210 con 0x55855222c520 2023-11-16T16:49:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.934+0000 7f58faffd700 1 -- 172.21.15.44:0/1537224024 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558552216ba0 con 0x55855222c520 2023-11-16T16:49:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.934+0000 7f58faffd700 1 -- 172.21.15.44:0/1537224024 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5914014410 con 0x55855222c520 2023-11-16T16:49:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.934+0000 7f5935633b80 10 client.15393 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:49:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.934+0000 7f5935633b80 10 client.15393 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:49:45.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.934+0000 7f5935633b80 20 client.15393 populate_metadata read hostname 'smithi044' 2023-11-16T16:49:45.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.934+0000 7f5935633b80 1 --2- 172.21.15.44:0/1537224024 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558552241b80 0x558552243f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.934+0000 7f5935633b80 4 client.15393 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:49:45.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.934+0000 7f5935633b80 1 -- 172.21.15.44:0/1537224024 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558552137120 con 0x558552241b80 2023-11-16T16:49:45.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.935+0000 7f591affd700 1 --2- 172.21.15.44:0/1537224024 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558552241b80 0x558552243f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.935+0000 7f591affd700 1 --2- 172.21.15.44:0/1537224024 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558552241b80 0x558552243f90 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a4de69977869819 server_cookie=bd9f46875126d84 in_seq=0 out_seq=0 2023-11-16T16:49:45.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.935+0000 7f58faffd700 10 client.15393 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:49:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.936+0000 7f58faffd700 1 -- 172.21.15.44:0/1537224024 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x558552137120 con 0x558552241b80 2023-11-16T16:49:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.936+0000 7f58faffd700 10 client.15393 handle_command_reply: tid=0 2023-11-16T16:49:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.936+0000 7f5935633b80 1 client.15393 shutdown 2023-11-16T16:49:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.936+0000 7f5935633b80 1 -- 172.21.15.44:0/1537224024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222c520 msgr2=0x558552235e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.936+0000 7f5935633b80 1 --2- 172.21.15.44:0/1537224024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222c520 0x558552235e80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5914004fd0 tx=0x7f5914015fe0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.936+0000 7f5935633b80 1 -- 172.21.15.44:0/1537224024 shutdown_connections 2023-11-16T16:49:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.936+0000 7f5935633b80 1 --2- 172.21.15.44:0/1537224024 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558552241b80 0x558552243f90 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.936+0000 7f5935633b80 1 --2- 172.21.15.44:0/1537224024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222dd60 0x55855223d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.936+0000 7f5935633b80 1 --2- 172.21.15.44:0/1537224024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222c520 0x558552235e80 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.936+0000 7f5935633b80 1 --2- 172.21.15.44:0/1537224024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222d3b0 0x5585522363c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.936+0000 7f58faffd700 0 client.15393 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:49:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.936+0000 7f5935633b80 1 -- 172.21.15.44:0/1537224024 >> 172.21.15.44:0/1537224024 conn(0x5585522193e0 msgr2=0x55855221de60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.937+0000 7f5935633b80 1 -- 172.21.15.44:0/1537224024 shutdown_connections 2023-11-16T16:49:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.937+0000 7f5935633b80 1 -- 172.21.15.44:0/1537224024 wait complete. 2023-11-16T16:49:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.937+0000 7f5935633b80 20 client.15393 trim_cache size 0 max 16384 2023-11-16T16:49:45.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.955+0000 7f5935633b80 1 Processor -- start 2023-11-16T16:49:45.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.955+0000 7f5935633b80 1 -- start start 2023-11-16T16:49:45.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.955+0000 7f5935633b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222c520 0x5585522488b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.955+0000 7f5935633b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222d3b0 0x55855223d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.955+0000 7f5935633b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222dd60 0x558552230ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.955+0000 7f5935633b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55855222cd90 con 0x55855222d3b0 2023-11-16T16:49:45.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.955+0000 7f5935633b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585522191f0 con 0x55855222c520 2023-11-16T16:49:45.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.955+0000 7f5935633b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x558552241670 con 0x55855222dd60 2023-11-16T16:49:45.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.955+0000 7f591bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222c520 0x5585522488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.955+0000 7f591bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222c520 0x5585522488b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:57636/0 (socket says 172.21.15.44:57636) 2023-11-16T16:49:45.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.955+0000 7f591affd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222d3b0 0x55855223d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.955+0000 7f591bfff700 1 -- 172.21.15.44:0/921817765 learned_addr learned my addr 172.21.15.44:0/921817765 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:45.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.956+0000 7f591bfff700 1 --2- 172.21.15.44:0/921817765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222dd60 0x558552230ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.956+0000 7f591bfff700 1 -- 172.21.15.44:0/921817765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222dd60 msgr2=0x558552230ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:45.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.956+0000 7f591bfff700 1 --2- 172.21.15.44:0/921817765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222dd60 0x558552230ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.956+0000 7f591bfff700 1 -- 172.21.15.44:0/921817765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222d3b0 msgr2=0x55855223d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.956+0000 7f591bfff700 1 --2- 172.21.15.44:0/921817765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222d3b0 0x55855223d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.956+0000 7f591bfff700 1 -- 172.21.15.44:0/921817765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55855221ab80 con 0x55855222c520 2023-11-16T16:49:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.956+0000 7f591bfff700 1 --2- 172.21.15.44:0/921817765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222c520 0x5585522488b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5914014820 tx=0x7f59140061a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea4351b9aa4b4e8d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.956+0000 7f59027fc700 1 -- 172.21.15.44:0/921817765 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5914007b30 con 0x55855222c520 2023-11-16T16:49:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.956+0000 7f59027fc700 1 -- 172.21.15.44:0/921817765 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5914014a90 con 0x55855222c520 2023-11-16T16:49:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.956+0000 7f59027fc700 1 -- 172.21.15.44:0/921817765 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5914004030 con 0x55855222c520 2023-11-16T16:49:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.957+0000 7f5935633b80 1 -- 172.21.15.44:0/921817765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222c520 msgr2=0x5585522488b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.957+0000 7f5935633b80 1 --2- 172.21.15.44:0/921817765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222c520 0x5585522488b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5914014820 tx=0x7f59140061a0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.957+0000 7f5935633b80 1 -- 172.21.15.44:0/921817765 shutdown_connections 2023-11-16T16:49:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.957+0000 7f5935633b80 1 --2- 172.21.15.44:0/921817765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222dd60 0x558552230ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.957+0000 7f5935633b80 1 --2- 172.21.15.44:0/921817765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222c520 0x5585522488b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.957+0000 7f5935633b80 1 --2- 172.21.15.44:0/921817765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222d3b0 0x55855223d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.957+0000 7f5935633b80 1 -- 172.21.15.44:0/921817765 >> 172.21.15.44:0/921817765 conn(0x5585522193e0 msgr2=0x55855221e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:45.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.957+0000 7f5935633b80 1 -- 172.21.15.44:0/921817765 shutdown_connections 2023-11-16T16:49:45.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.957+0000 7f5935633b80 1 -- 172.21.15.44:0/921817765 wait complete. 2023-11-16T16:49:45.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.957+0000 7f5935633b80 1 -- start start 2023-11-16T16:49:45.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.958+0000 7f5935633b80 1 Processor -- start 2023-11-16T16:49:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.958+0000 7f5935633b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222c520 0x55855223cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.958+0000 7f5935633b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222d3b0 0x55855223d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.958+0000 7f5935633b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222dd60 0x55855223b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.958+0000 7f5935633b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55855223bae0 con 0x55855222dd60 2023-11-16T16:49:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.958+0000 7f5935633b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55855223bc50 con 0x55855222c520 2023-11-16T16:49:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.958+0000 7f5935633b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55855223be10 con 0x55855222d3b0 2023-11-16T16:49:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.958+0000 7f5935633b80 10 client.0 _ll_register_callbacks cb 0x7f590c0d0630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:49:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.958+0000 7f591affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222d3b0 0x55855223d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.959+0000 7f591affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222d3b0 0x55855223d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:51494/0 (socket says 172.21.15.44:51494) 2023-11-16T16:49:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.959+0000 7f591affd700 1 -- 172.21.15.44:0/2278725947 learned_addr learned my addr 172.21.15.44:0/2278725947 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.959+0000 7f591bfff700 1 --2- 172.21.15.44:0/2278725947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222dd60 0x55855223b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.959+0000 7f591bfff700 1 --2- 172.21.15.44:0/2278725947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222c520 0x55855223cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.959+0000 7f591affd700 1 -- 172.21.15.44:0/2278725947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222c520 msgr2=0x55855223cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.959+0000 7f591affd700 1 --2- 172.21.15.44:0/2278725947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222c520 0x55855223cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.959+0000 7f591affd700 1 -- 172.21.15.44:0/2278725947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222dd60 msgr2=0x55855223b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.959+0000 7f591affd700 1 --2- 172.21.15.44:0/2278725947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222dd60 0x55855223b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.959+0000 7f591affd700 1 -- 172.21.15.44:0/2278725947 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55855213b210 con 0x55855222d3b0 2023-11-16T16:49:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.959+0000 7f591bfff700 1 --2- 172.21.15.44:0/2278725947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222dd60 0x55855223b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:49:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.959+0000 7f591affd700 1 --2- 172.21.15.44:0/2278725947 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222d3b0 0x55855223d330 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5910000c00 tx=0x7f591000ba60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ddd706eb28d1670 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.959+0000 7f58faffd700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:49:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.959+0000 7f58faffd700 1 -- 172.21.15.44:0/2278725947 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591001d040 con 0x55855222d3b0 2023-11-16T16:49:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.960+0000 7f5935633b80 1 -- 172.21.15.44:0/2278725947 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f591400f050 con 0x55855222d3b0 2023-11-16T16:49:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.960+0000 7f58faffd700 1 -- 172.21.15.44:0/2278725947 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f59100290b0 con 0x55855222d3b0 2023-11-16T16:49:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.960+0000 7f58faffd700 1 -- 172.21.15.44:0/2278725947 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5910029270 con 0x55855222d3b0 2023-11-16T16:49:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.960+0000 7f5935633b80 10 client.25414 fetch_fsmap learned FSMap version 21 2023-11-16T16:49:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.960+0000 7f5935633b80 1 -- 172.21.15.44:0/2278725947 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x558551fa2e60 con 0x55855222d3b0 2023-11-16T16:49:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.961+0000 7f58faffd700 1 -- 172.21.15.44:0/2278725947 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f591400f050 con 0x55855222d3b0 2023-11-16T16:49:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.961+0000 7f58faffd700 1 -- 172.21.15.44:0/2278725947 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5910004590 con 0x55855222d3b0 2023-11-16T16:49:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.961+0000 7f5935633b80 10 client.25414 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:49:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.961+0000 7f5935633b80 10 client.25414 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:49:45.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.961+0000 7f5935633b80 20 client.25414 populate_metadata read hostname 'smithi044' 2023-11-16T16:49:45.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.961+0000 7f5935633b80 1 --2- 172.21.15.44:0/2278725947 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558552296cb0 0x5585522990c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.961+0000 7f5935633b80 4 client.25414 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:49:45.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.961+0000 7f5935633b80 1 -- 172.21.15.44:0/2278725947 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5585521436e0 con 0x558552296cb0 2023-11-16T16:49:45.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.961+0000 7f591bfff700 1 --2- 172.21.15.44:0/2278725947 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558552296cb0 0x5585522990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.962+0000 7f58faffd700 10 client.25414 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:49:45.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.962+0000 7f591bfff700 1 --2- 172.21.15.44:0/2278725947 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558552296cb0 0x5585522990c0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce8844c76e995c82 server_cookie=dbfa31e51dde4ef9 in_seq=0 out_seq=0 2023-11-16T16:49:45.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.963+0000 7f58faffd700 1 -- 172.21.15.44:0/2278725947 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5585521436e0 con 0x558552296cb0 2023-11-16T16:49:45.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.963+0000 7f58faffd700 10 client.25414 handle_command_reply: tid=0 2023-11-16T16:49:45.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.963+0000 7f5935633b80 1 client.25414 shutdown 2023-11-16T16:49:45.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.963+0000 7f5935633b80 1 -- 172.21.15.44:0/2278725947 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222d3b0 msgr2=0x55855223d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.963+0000 7f5935633b80 1 --2- 172.21.15.44:0/2278725947 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222d3b0 0x55855223d330 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5910000c00 tx=0x7f591000ba60 comp rx=0 tx=0).stop 2023-11-16T16:49:45.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.963+0000 7f5935633b80 1 -- 172.21.15.44:0/2278725947 shutdown_connections 2023-11-16T16:49:45.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.963+0000 7f5935633b80 1 --2- 172.21.15.44:0/2278725947 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558552296cb0 0x5585522990c0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.963+0000 7f5935633b80 1 --2- 172.21.15.44:0/2278725947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55855222dd60 0x55855223b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.963+0000 7f5935633b80 1 --2- 172.21.15.44:0/2278725947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55855222c520 0x55855223cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.968 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.963+0000 7f5935633b80 1 --2- 172.21.15.44:0/2278725947 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55855222d3b0 0x55855223d330 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.968 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.963+0000 7f58faffd700 0 client.25414 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:49:45.968 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.963+0000 7f5935633b80 1 -- 172.21.15.44:0/2278725947 >> 172.21.15.44:0/2278725947 conn(0x5585522193e0 msgr2=0x558552230ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:45.968 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.963+0000 7f5935633b80 1 -- 172.21.15.44:0/2278725947 shutdown_connections 2023-11-16T16:49:45.968 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.963+0000 7f5935633b80 1 -- 172.21.15.44:0/2278725947 wait complete. 2023-11-16T16:49:45.968 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.964+0000 7f5935633b80 20 client.25414 trim_cache size 0 max 16384 2023-11-16T16:49:45.968 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:49:45.968 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:49:45.968 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "29a7248d-7d7a-4679-813d-cf52d3e1d1ef", 2023-11-16T16:49:45.968 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:49:45.969 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:49:45.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.964+0000 7f5921dff700 1 -- 172.21.15.44:0/1288529078 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f59040a11d0 msgr2=0x7f59040a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.964+0000 7f5921dff700 1 --2- 172.21.15.44:0/1288529078 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f59040a11d0 0x7f59040a3680 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f590c099bf0 tx=0x7f590c0d0760 comp rx=0 tx=0).stop 2023-11-16T16:49:45.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.964+0000 7f5921dff700 1 -- 172.21.15.44:0/1288529078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f591c100d40 msgr2=0x7f591c198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.964+0000 7f5921dff700 1 --2- 172.21.15.44:0/1288529078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f591c100d40 0x7f591c198270 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f590c00e9a0 tx=0x7f590c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.964+0000 7f5921dff700 1 -- 172.21.15.44:0/1288529078 shutdown_connections 2023-11-16T16:49:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.964+0000 7f5921dff700 1 --2- 172.21.15.44:0/1288529078 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f59040a11d0 0x7f59040a3680 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.964+0000 7f5921dff700 1 --2- 172.21.15.44:0/1288529078 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f591c105f80 0x7f591c19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.964+0000 7f5921dff700 1 --2- 172.21.15.44:0/1288529078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f591c100d40 0x7f591c198270 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.964+0000 7f5921dff700 1 --2- 172.21.15.44:0/1288529078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f591c103660 0x7f591c1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.964+0000 7f5921dff700 1 -- 172.21.15.44:0/1288529078 >> 172.21.15.44:0/1288529078 conn(0x7f591c0fa7c0 msgr2=0x7f591c1019a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.964+0000 7f5921dff700 1 -- 172.21.15.44:0/1288529078 shutdown_connections 2023-11-16T16:49:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:45.964+0000 7f5921dff700 1 -- 172.21.15.44:0/1288529078 wait complete. 2023-11-16T16:49:45.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:45 smithi077 ceph-mon[131204]: pgmap v352: 129 pgs: 129 active+clean; 643 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2023-11-16T16:49:45.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:45 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3701383675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:45.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:45 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3170013193' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:45.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:45 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1175923456' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:45.982 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:45 smithi044 ceph-mon[128750]: pgmap v352: 129 pgs: 129 active+clean; 643 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2023-11-16T16:49:45.982 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:45 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3701383675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:45.982 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:45 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3170013193' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:45.982 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:45 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1175923456' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:45.984 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:49:46.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:45 smithi176 ceph-mon[131630]: pgmap v352: 129 pgs: 129 active+clean; 643 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2023-11-16T16:49:46.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:45 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3701383675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:46.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:45 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3170013193' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:46.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:45 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1175923456' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:46.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.190+0000 7f2ee13ff700 1 -- 172.21.15.44:0/2007043967 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2edc102510 msgr2=0x7f2edc102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.190+0000 7f2ee13ff700 1 --2- 172.21.15.44:0/2007043967 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2edc102510 0x7f2edc102920 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009990 tx=0x7f2ecc009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.190+0000 7f2ee13ff700 1 -- 172.21.15.44:0/2007043967 shutdown_connections 2023-11-16T16:49:46.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.190+0000 7f2ee13ff700 1 --2- 172.21.15.44:0/2007043967 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2edc104090 0x7f2edc10a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.190+0000 7f2ee13ff700 1 --2- 172.21.15.44:0/2007043967 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2edc102510 0x7f2edc102920 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.190+0000 7f2ee13ff700 1 --2- 172.21.15.44:0/2007043967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2edc103700 0x7f2edc103b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.190+0000 7f2ee13ff700 1 -- 172.21.15.44:0/2007043967 >> 172.21.15.44:0/2007043967 conn(0x7f2edc0fdb30 msgr2=0x7f2edc0fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:46.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.190+0000 7f2ee13ff700 1 -- 172.21.15.44:0/2007043967 shutdown_connections 2023-11-16T16:49:46.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.191+0000 7f2ee13ff700 1 -- 172.21.15.44:0/2007043967 wait complete. 2023-11-16T16:49:46.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.191+0000 7f2ee13ff700 1 Processor -- start 2023-11-16T16:49:46.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.192+0000 7f2ee13ff700 1 -- start start 2023-11-16T16:49:46.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.192+0000 7f2ee13ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2edc102510 0x7f2edc198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.192+0000 7f2ee13ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2edc103700 0x7f2edc1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.192+0000 7f2ee13ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2edc104090 0x7f2edc19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.192+0000 7f2ee13ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc19fdf0 con 0x7f2edc103700 2023-11-16T16:49:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.192+0000 7f2ee13ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc19ff30 con 0x7f2edc104090 2023-11-16T16:49:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.192+0000 7f2ee13ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc1a00c0 con 0x7f2edc102510 2023-11-16T16:49:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.192+0000 7f2eda7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2edc103700 0x7f2edc1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.192+0000 7f2eda7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2edc103700 0x7f2edc1987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56552/0 (socket says 172.21.15.44:56552) 2023-11-16T16:49:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.192+0000 7f2eda7fc700 1 -- 172.21.15.44:0/2314593675 learned_addr learned my addr 172.21.15.44:0/2314593675 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.192+0000 7f2edaffd700 1 --2- 172.21.15.44:0/2314593675 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2edc102510 0x7f2edc198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.192+0000 7f2edb7fe700 1 --2- 172.21.15.44:0/2314593675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2edc104090 0x7f2edc19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.193+0000 7f2eda7fc700 1 -- 172.21.15.44:0/2314593675 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2edc102510 msgr2=0x7f2edc198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.193+0000 7f2eda7fc700 1 --2- 172.21.15.44:0/2314593675 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2edc102510 0x7f2edc198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.193+0000 7f2eda7fc700 1 -- 172.21.15.44:0/2314593675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2edc104090 msgr2=0x7f2edc19f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.193+0000 7f2eda7fc700 1 --2- 172.21.15.44:0/2314593675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2edc104090 0x7f2edc19f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.193+0000 7f2eda7fc700 1 -- 172.21.15.44:0/2314593675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ecc0095f0 con 0x7f2edc103700 2023-11-16T16:49:46.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.193+0000 7f2eda7fc700 1 --2- 172.21.15.44:0/2314593675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2edc103700 0x7f2edc1987b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000e9a0 tx=0x7f2ed000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:46.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.193+0000 7f2ec3fff700 1 -- 172.21.15.44:0/2314593675 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0009790 con 0x7f2edc103700 2023-11-16T16:49:46.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.193+0000 7f2ec3fff700 1 -- 172.21.15.44:0/2314593675 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ed0010430 con 0x7f2edc103700 2023-11-16T16:49:46.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.193+0000 7f2ec3fff700 1 -- 172.21.15.44:0/2314593675 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0018680 con 0x7f2edc103700 2023-11-16T16:49:46.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.193+0000 7f2ee13ff700 1 -- 172.21.15.44:0/2314593675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2edc10ded0 con 0x7f2edc103700 2023-11-16T16:49:46.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.193+0000 7f2ee13ff700 1 -- 172.21.15.44:0/2314593675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2edc10dd40 con 0x7f2edc103700 2023-11-16T16:49:46.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.197+0000 7f2ec3fff700 1 -- 172.21.15.44:0/2314593675 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2ed00105a0 con 0x7f2edc103700 2023-11-16T16:49:46.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.199+0000 7f2ef4e1ab80 1 Processor -- start 2023-11-16T16:49:46.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.199+0000 7f2ef4e1ab80 1 -- start start 2023-11-16T16:49:46.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.199+0000 7f2ec3fff700 1 --2- 172.21.15.44:0/2314593675 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ec40a11d0 0x7f2ec40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.199+0000 7f2ec3fff700 1 -- 172.21.15.44:0/2314593675 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f2ed0014070 con 0x7f2edc103700 2023-11-16T16:49:46.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.199+0000 7f2ef4e1ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a6c00 0x55b3208a7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.199+0000 7f2ef4e1ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a7550 0x55b3208a7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.199+0000 7f2ef4e1ab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a8110 0x55b3208a8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.199+0000 7f2ef4e1ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3207ac060 con 0x55b3208a6c00 2023-11-16T16:49:46.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.199+0000 7f2ef4e1ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3207fe8a0 con 0x55b3208a7550 2023-11-16T16:49:46.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.199+0000 7f2ef4e1ab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3207f7d30 con 0x55b3208a8110 2023-11-16T16:49:46.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.200+0000 7f2edb7fe700 1 --2- 172.21.15.44:0/2314593675 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ec40a11d0 0x7f2ec40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.200+0000 7f2edaffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a6c00 0x55b3208a7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.200+0000 7f2edaffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a6c00 0x55b3208a7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56556/0 (socket says 172.21.15.44:56556) 2023-11-16T16:49:46.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.200+0000 7f2edaffd700 1 -- 172.21.15.44:0/2781462865 learned_addr learned my addr 172.21.15.44:0/2781462865 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:46.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.200+0000 7f2eda7fc700 1 --2- 172.21.15.44:0/2781462865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a7550 0x55b3208a7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.200+0000 7f2edb7fe700 1 --2- 172.21.15.44:0/2781462865 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a8110 0x55b3208a8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.200+0000 7f2edaffd700 1 -- 172.21.15.44:0/2781462865 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a8110 msgr2=0x55b3208a8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:46.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.200+0000 7f2edaffd700 1 --2- 172.21.15.44:0/2781462865 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a8110 0x55b3208a8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.204 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.200+0000 7f2edaffd700 1 -- 172.21.15.44:0/2781462865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a7550 msgr2=0x55b3208a7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:46.204 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.200+0000 7f2edaffd700 1 --2- 172.21.15.44:0/2781462865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a7550 0x55b3208a7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.204 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.200+0000 7f2edaffd700 1 -- 172.21.15.44:0/2781462865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b320679630 con 0x55b3208a6c00 2023-11-16T16:49:46.204 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.200+0000 7f2edb7fe700 1 --2- 172.21.15.44:0/2781462865 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a8110 0x55b3208a8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:49:46.204 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.200+0000 7f2edb7fe700 1 --2- 172.21.15.44:0/2314593675 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ec40a11d0 0x7f2ec40a3680 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4003f20 tx=0x7f2ed4009ba0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:46.204 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.201+0000 7f2edaffd700 1 --2- 172.21.15.44:0/2781462865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a6c00 0x55b3208a7010 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2edc103560 tx=0x7f2ecc005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ba496ebd706fbb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:46.204 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.201+0000 7f2ec17fa700 1 -- 172.21.15.44:0/2781462865 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc01d070 con 0x55b3208a6c00 2023-11-16T16:49:46.204 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.201+0000 7f2ec17fa700 1 -- 172.21.15.44:0/2781462865 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ecc01e040 con 0x55b3208a6c00 2023-11-16T16:49:46.204 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.201+0000 7f2ec17fa700 1 -- 172.21.15.44:0/2781462865 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc004920 con 0x55b3208a6c00 2023-11-16T16:49:46.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.202+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/2781462865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a6c00 msgr2=0x55b3208a7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.202+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/2781462865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a6c00 0x55b3208a7010 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2edc103560 tx=0x7f2ecc005bc0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.202+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/2781462865 shutdown_connections 2023-11-16T16:49:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.202+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/2781462865 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a8110 0x55b3208a8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.202+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/2781462865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a6c00 0x55b3208a7010 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.202+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/2781462865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a7550 0x55b3208a7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.202+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/2781462865 >> 172.21.15.44:0/2781462865 conn(0x55b32089c360 msgr2=0x55b3208a1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.202+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/2781462865 shutdown_connections 2023-11-16T16:49:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.202+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/2781462865 wait complete. 2023-11-16T16:49:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.202+0000 7f2ef4e1ab80 1 -- start start 2023-11-16T16:49:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.203+0000 7f2ef4e1ab80 1 Processor -- start 2023-11-16T16:49:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.203+0000 7f2ef4e1ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a6c00 0x55b3208b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.203+0000 7f2ef4e1ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a7550 0x55b3208b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.203+0000 7f2ef4e1ab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a8110 0x55b3208b7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.203+0000 7f2ef4e1ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3207998e0 con 0x55b3208a6c00 2023-11-16T16:49:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.203+0000 7f2ef4e1ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b320799580 con 0x55b3208a7550 2023-11-16T16:49:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.203+0000 7f2ef4e1ab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b32067b920 con 0x55b3208a8110 2023-11-16T16:49:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.203+0000 7f2ef4e1ab80 10 client.0 _ll_register_callbacks cb 0x55b3207b7780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:49:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.204+0000 7f2edaffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a6c00 0x55b3208b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.204+0000 7f2eda7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a7550 0x55b3208b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.204+0000 7f2edaffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a6c00 0x55b3208b0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56560/0 (socket says 172.21.15.44:56560) 2023-11-16T16:49:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.204+0000 7f2edaffd700 1 -- 172.21.15.44:0/1654925128 learned_addr learned my addr 172.21.15.44:0/1654925128 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.204+0000 7f2edb7fe700 1 --2- 172.21.15.44:0/1654925128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a8110 0x55b3208b7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.204+0000 7f2edaffd700 1 -- 172.21.15.44:0/1654925128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a8110 msgr2=0x55b3208b7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.204+0000 7f2edaffd700 1 --2- 172.21.15.44:0/1654925128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a8110 0x55b3208b7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.204+0000 7f2edaffd700 1 -- 172.21.15.44:0/1654925128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a7550 msgr2=0x55b3208b0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.204+0000 7f2edaffd700 1 --2- 172.21.15.44:0/1654925128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a7550 0x55b3208b0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.204+0000 7f2edaffd700 1 -- 172.21.15.44:0/1654925128 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2edc10b280 con 0x55b3208a6c00 2023-11-16T16:49:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.204+0000 7f2eda7fc700 1 --2- 172.21.15.44:0/1654925128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a7550 0x55b3208b0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.204+0000 7f2edaffd700 1 --2- 172.21.15.44:0/1654925128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a6c00 0x55b3208b0550 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009990 tx=0x7f2ecc00ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3c302ac00d53aa server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.204+0000 7f2ebeff5700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:49:46.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.204+0000 7f52bb1ff700 1 -- 172.21.15.44:0/3429799178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f52b4061c40 con 0x7f52b4074ab0 2023-11-16T16:49:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.205+0000 7f2ebeff5700 1 -- 172.21.15.44:0/1654925128 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc01d030 con 0x55b3208a6c00 2023-11-16T16:49:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.205+0000 7f2ebeff5700 1 -- 172.21.15.44:0/1654925128 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f2ecc0048a0 con 0x55b3208a6c00 2023-11-16T16:49:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.205+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/1654925128 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b320899b20 con 0x55b3208a6c00 2023-11-16T16:49:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.205+0000 7f2ebeff5700 1 -- 172.21.15.44:0/1654925128 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ecc004a80 con 0x55b3208a6c00 2023-11-16T16:49:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.205+0000 7f2ef4e1ab80 10 client.15411 fetch_fsmap learned FSMap version 21 2023-11-16T16:49:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.205+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/1654925128 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55b3207be190 con 0x55b3208a6c00 2023-11-16T16:49:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.205+0000 7f2ebeff5700 1 -- 172.21.15.44:0/1654925128 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b320899b20 con 0x55b3208a6c00 2023-11-16T16:49:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.205+0000 7f2ebeff5700 1 -- 172.21.15.44:0/1654925128 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2ecc00fcc0 con 0x55b3208a6c00 2023-11-16T16:49:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.205+0000 7f2ef4e1ab80 10 client.15411 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:49:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.205+0000 7f2ef4e1ab80 10 client.15411 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:49:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.205+0000 7f2ef4e1ab80 20 client.15411 populate_metadata read hostname 'smithi044' 2023-11-16T16:49:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.206+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/1654925128 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b3208bbd20 0x55b3208be130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.206+0000 7f2ef4e1ab80 4 client.15411 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:49:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.206+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/1654925128 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b3207ba0a0 con 0x55b3208bbd20 2023-11-16T16:49:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.206+0000 7f2eda7fc700 1 --2- 172.21.15.44:0/1654925128 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b3208bbd20 0x55b3208be130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.206+0000 7f2eda7fc700 1 --2- 172.21.15.44:0/1654925128 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b3208bbd20 0x55b3208be130 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bedd10824b5c5e31 server_cookie=1576abf76e4db3b6 in_seq=0 out_seq=0 2023-11-16T16:49:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.206+0000 7f2ebeff5700 10 client.15411 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:49:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.207+0000 7f2ebeff5700 1 -- 172.21.15.44:0/1654925128 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55b3207ba0a0 con 0x55b3208bbd20 2023-11-16T16:49:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.207+0000 7f2ebeff5700 10 client.15411 handle_command_reply: tid=0 2023-11-16T16:49:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.207+0000 7f2ef4e1ab80 1 client.15411 shutdown 2023-11-16T16:49:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.207+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/1654925128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a6c00 msgr2=0x55b3208b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.207+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/1654925128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a6c00 0x55b3208b0550 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009990 tx=0x7f2ecc00ba20 comp rx=0 tx=0).stop 2023-11-16T16:49:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.207+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/1654925128 shutdown_connections 2023-11-16T16:49:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.207+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/1654925128 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b3208bbd20 0x55b3208be130 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.207+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/1654925128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a8110 0x55b3208b7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.207+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/1654925128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a6c00 0x55b3208b0550 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.207+0000 7f2ebeff5700 0 client.15411 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:49:46.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.207+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/1654925128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a7550 0x55b3208b0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.207+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/1654925128 >> 172.21.15.44:0/1654925128 conn(0x55b32089c360 msgr2=0x55b3208a0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:46.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.208+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/1654925128 shutdown_connections 2023-11-16T16:49:46.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.208+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/1654925128 wait complete. 2023-11-16T16:49:46.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.208+0000 7f2ef4e1ab80 20 client.15411 trim_cache size 0 max 16384 2023-11-16T16:49:46.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.225+0000 7f2ef4e1ab80 1 Processor -- start 2023-11-16T16:49:46.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.225+0000 7f2ef4e1ab80 1 -- start start 2023-11-16T16:49:46.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.226+0000 7f2ef4e1ab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a6c00 0x55b3208b83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.226+0000 7f2ef4e1ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a7550 0x55b3208b7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.226+0000 7f2ef4e1ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a8110 0x55b3208b0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.226+0000 7f2ef4e1ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3208a79e0 con 0x55b3208a7550 2023-11-16T16:49:46.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.226+0000 7f2ef4e1ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3208ae2b0 con 0x55b3208a8110 2023-11-16T16:49:46.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.226+0000 7f2ef4e1ab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3208abb70 con 0x55b3208a6c00 2023-11-16T16:49:46.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.226+0000 7f2eda7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a7550 0x55b3208b7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.226+0000 7f2eda7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a7550 0x55b3208b7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56564/0 (socket says 172.21.15.44:56564) 2023-11-16T16:49:46.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.226+0000 7f2eda7fc700 1 -- 172.21.15.44:0/2757971671 learned_addr learned my addr 172.21.15.44:0/2757971671 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:46.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.226+0000 7f2edb7fe700 1 --2- 172.21.15.44:0/2757971671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a8110 0x55b3208b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.226+0000 7f2edaffd700 1 --2- 172.21.15.44:0/2757971671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a6c00 0x55b3208b83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.226+0000 7f2eda7fc700 1 -- 172.21.15.44:0/2757971671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a6c00 msgr2=0x55b3208b83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:46.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.226+0000 7f2eda7fc700 1 --2- 172.21.15.44:0/2757971671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a6c00 0x55b3208b83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.226+0000 7f2eda7fc700 1 -- 172.21.15.44:0/2757971671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a8110 msgr2=0x55b3208b0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:46.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.226+0000 7f2eda7fc700 1 --2- 172.21.15.44:0/2757971671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a8110 0x55b3208b0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.226+0000 7f2eda7fc700 1 -- 172.21.15.44:0/2757971671 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b3207bdee0 con 0x55b3208a7550 2023-11-16T16:49:46.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.227+0000 7f2eda7fc700 1 --2- 172.21.15.44:0/2757971671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a7550 0x55b3208b7a90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000e970 tx=0x7f2ed00cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddcdad6d5d369e28 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:46.230 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.227+0000 7f2ec1ffb700 1 -- 172.21.15.44:0/2757971671 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed00b8270 con 0x55b3208a7550 2023-11-16T16:49:46.230 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.227+0000 7f2ec1ffb700 1 -- 172.21.15.44:0/2757971671 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ed00b1070 con 0x55b3208a7550 2023-11-16T16:49:46.230 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.227+0000 7f2ec1ffb700 1 -- 172.21.15.44:0/2757971671 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed00a1e50 con 0x55b3208a7550 2023-11-16T16:49:46.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.227+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/2757971671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a7550 msgr2=0x55b3208b7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.227+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/2757971671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a7550 0x55b3208b7a90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000e970 tx=0x7f2ed00cc040 comp rx=0 tx=0).stop 2023-11-16T16:49:46.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.228+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/2757971671 shutdown_connections 2023-11-16T16:49:46.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.228+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/2757971671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a8110 0x55b3208b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.228+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/2757971671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a6c00 0x55b3208b83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.228+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/2757971671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a7550 0x55b3208b7a90 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.228+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/2757971671 >> 172.21.15.44:0/2757971671 conn(0x55b32089c360 msgr2=0x55b3208af290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:46.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.228+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/2757971671 shutdown_connections 2023-11-16T16:49:46.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.228+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/2757971671 wait complete. 2023-11-16T16:49:46.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.228+0000 7f2ef4e1ab80 1 -- start start 2023-11-16T16:49:46.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.228+0000 7f2ef4e1ab80 1 Processor -- start 2023-11-16T16:49:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2ef4e1ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a6c00 0x55b3208b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2ef4e1ab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a7550 0x55b3208b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2ef4e1ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a8110 0x55b3208b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2ef4e1ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3208b7610 con 0x55b3208a8110 2023-11-16T16:49:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2ef4e1ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3208b7780 con 0x55b3208a6c00 2023-11-16T16:49:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2ef4e1ab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3208b7940 con 0x55b3208a7550 2023-11-16T16:49:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2ef4e1ab80 10 client.0 _ll_register_callbacks cb 0x7f2ed0005760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:49:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2edb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a8110 0x55b3208b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2edb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a8110 0x55b3208b70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56572/0 (socket says 172.21.15.44:56572) 2023-11-16T16:49:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2edb7fe700 1 -- 172.21.15.44:0/1726390485 learned_addr learned my addr 172.21.15.44:0/1726390485 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2eda7fc700 1 --2- 172.21.15.44:0/1726390485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a7550 0x55b3208b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2edaffd700 1 --2- 172.21.15.44:0/1726390485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a6c00 0x55b3208b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2edb7fe700 1 -- 172.21.15.44:0/1726390485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a7550 msgr2=0x55b3208b40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2edb7fe700 1 --2- 172.21.15.44:0/1726390485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a7550 0x55b3208b40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2edb7fe700 1 -- 172.21.15.44:0/1726390485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a6c00 msgr2=0x55b3208b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2edb7fe700 1 --2- 172.21.15.44:0/1726390485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a6c00 0x55b3208b3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.229+0000 7f2edb7fe700 1 -- 172.21.15.44:0/1726390485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2edc10dd40 con 0x55b3208a8110 2023-11-16T16:49:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.230+0000 7f2edb7fe700 1 --2- 172.21.15.44:0/1726390485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a8110 0x55b3208b70d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4007ab0 tx=0x7f2ed4004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b224d1b45606e78 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.230+0000 7f2ebeff5700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:49:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.230+0000 7f2ebeff5700 1 -- 172.21.15.44:0/1726390485 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed401f070 con 0x55b3208a8110 2023-11-16T16:49:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.230+0000 7f2ebeff5700 1 -- 172.21.15.44:0/1726390485 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f2ed4017a80 con 0x55b3208a8110 2023-11-16T16:49:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.230+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/1726390485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2ecc01e040 con 0x55b3208a8110 2023-11-16T16:49:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.230+0000 7f2ebeff5700 1 -- 172.21.15.44:0/1726390485 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ed4017c40 con 0x55b3208a8110 2023-11-16T16:49:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.230+0000 7f2ef4e1ab80 10 client.15417 fetch_fsmap learned FSMap version 21 2023-11-16T16:49:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.230+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/1726390485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55b32060ec60 con 0x55b3208a8110 2023-11-16T16:49:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.230+0000 7f2ebeff5700 1 -- 172.21.15.44:0/1726390485 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ecc01e040 con 0x55b3208a8110 2023-11-16T16:49:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.231+0000 7f2ebeff5700 1 -- 172.21.15.44:0/1726390485 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2ed4018410 con 0x55b3208a8110 2023-11-16T16:49:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.231+0000 7f2ef4e1ab80 10 client.15417 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:49:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.231+0000 7f2ef4e1ab80 10 client.15417 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:49:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.231+0000 7f2ef4e1ab80 20 client.15417 populate_metadata read hostname 'smithi044' 2023-11-16T16:49:46.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.231+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/1726390485 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b320910f80 0x55b320913390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.231+0000 7f2ef4e1ab80 4 client.15417 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:49:46.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.231+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/1726390485 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b3207c6660 con 0x55b320910f80 2023-11-16T16:49:46.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.231+0000 7f2edaffd700 1 --2- 172.21.15.44:0/1726390485 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b320910f80 0x55b320913390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2edaffd700 1 --2- 172.21.15.44:0/1726390485 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b320910f80 0x55b320913390 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54f5a6f7437447ca server_cookie=aba232d29ce72597 in_seq=0 out_seq=0 2023-11-16T16:49:46.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2ebeff5700 10 client.15417 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:49:46.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2ebeff5700 1 -- 172.21.15.44:0/1726390485 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55b3207c6660 con 0x55b320910f80 2023-11-16T16:49:46.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2ebeff5700 10 client.15417 handle_command_reply: tid=0 2023-11-16T16:49:46.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2ef4e1ab80 1 client.15417 shutdown 2023-11-16T16:49:46.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/1726390485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a8110 msgr2=0x55b3208b70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/1726390485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a8110 0x55b3208b70d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4007ab0 tx=0x7f2ed4004b40 comp rx=0 tx=0).stop 2023-11-16T16:49:46.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/1726390485 shutdown_connections 2023-11-16T16:49:46.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/1726390485 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b320910f80 0x55b320913390 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/1726390485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b3208a8110 0x55b3208b70d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2ebeff5700 0 client.15417 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:49:46.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/1726390485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3208a6c00 0x55b3208b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2ef4e1ab80 1 --2- 172.21.15.44:0/1726390485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b3208a7550 0x55b3208b40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/1726390485 >> 172.21.15.44:0/1726390485 conn(0x55b32089c360 msgr2=0x55b3208af290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:46.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/1726390485 shutdown_connections 2023-11-16T16:49:46.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2ef4e1ab80 1 -- 172.21.15.44:0/1726390485 wait complete. 2023-11-16T16:49:46.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.232+0000 7f2ef4e1ab80 20 client.15417 trim_cache size 0 max 16384 2023-11-16T16:49:46.238 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:49:46.238 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2023-11-16T16:49:46.238 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2023-11-16T16:49:46.239 INFO:teuthology.orchestra.run.smithi044.stdout: "29a7248d-7d7a-4679-813d-cf52d3e1d1ef": { 2023-11-16T16:49:46.239 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2023-11-16T16:49:46.239 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "29a7248d-7d7a-4679-813d-cf52d3e1d1ef", 2023-11-16T16:49:46.239 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2023-11-16T16:49:46.239 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:49:46.239 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:49:46.239 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:49:46.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.233+0000 7f2ee13ff700 1 -- 172.21.15.44:0/2314593675 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ec40a11d0 msgr2=0x7f2ec40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.233+0000 7f2ee13ff700 1 --2- 172.21.15.44:0/2314593675 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ec40a11d0 0x7f2ec40a3680 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4003f20 tx=0x7f2ed4009ba0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.233+0000 7f2ee13ff700 1 -- 172.21.15.44:0/2314593675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2edc103700 msgr2=0x7f2edc1987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.233+0000 7f2ee13ff700 1 --2- 172.21.15.44:0/2314593675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2edc103700 0x7f2edc1987b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000e9a0 tx=0x7f2ed000ecb0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.233+0000 7f2ee13ff700 1 -- 172.21.15.44:0/2314593675 shutdown_connections 2023-11-16T16:49:46.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.233+0000 7f2ee13ff700 1 --2- 172.21.15.44:0/2314593675 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f2ec40a11d0 0x7f2ec40a3680 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.233+0000 7f2ee13ff700 1 --2- 172.21.15.44:0/2314593675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2edc104090 0x7f2edc19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.233+0000 7f2ee13ff700 1 --2- 172.21.15.44:0/2314593675 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2edc102510 0x7f2edc198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.233+0000 7f2ee13ff700 1 --2- 172.21.15.44:0/2314593675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2edc103700 0x7f2edc1987b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.233+0000 7f2ee13ff700 1 -- 172.21.15.44:0/2314593675 >> 172.21.15.44:0/2314593675 conn(0x7f2edc0fdb30 msgr2=0x7f2edc106b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:46.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.233+0000 7f2ee13ff700 1 -- 172.21.15.44:0/2314593675 shutdown_connections 2023-11-16T16:49:46.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.233+0000 7f2ee13ff700 1 -- 172.21.15.44:0/2314593675 wait complete. 2023-11-16T16:49:46.252 INFO:tasks.cephfs.filesystem:scrub status for tag:29a7248d-7d7a-4679-813d-cf52d3e1d1ef - {'path': '/', 'tag': '29a7248d-7d7a-4679-813d-cf52d3e1d1ef', 'options': 'recursive,force'} 2023-11-16T16:49:46.707 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.706+0000 7f52b1ffb700 1 -- 172.21.15.44:0/3429799178 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 19) v1 ==== 135732+0+0 (secure 0 0 0) 0x7f52a0010880 con 0x7f52b4074ab0 2023-11-16T16:49:46.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.708+0000 7f52b1ffb700 1 -- 172.21.15.44:0/3429799178 <== mon.2 v2:172.21.15.77:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f52a0088590 con 0x7f52b4074ab0 2023-11-16T16:49:46.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.711+0000 7f52bb1ff700 1 -- 172.21.15.44:0/3429799178 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f52a40a11d0 msgr2=0x7f52a40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.711+0000 7f52bb1ff700 1 --2- 172.21.15.44:0/3429799178 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f52a40a11d0 0x7f52a40a3680 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f52a8005e90 tx=0x7f52a8019040 comp rx=0 tx=0).stop 2023-11-16T16:49:46.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.711+0000 7f52bb1ff700 1 -- 172.21.15.44:0/3429799178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52b4074ab0 msgr2=0x7f52b41987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.711+0000 7f52bb1ff700 1 --2- 172.21.15.44:0/3429799178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52b4074ab0 0x7f52b41987b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f52a000e9a0 tx=0x7f52a000ecb0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.711+0000 7f52bb1ff700 1 -- 172.21.15.44:0/3429799178 shutdown_connections 2023-11-16T16:49:46.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.711+0000 7f52bb1ff700 1 --2- 172.21.15.44:0/3429799178 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f52a40a11d0 0x7f52a40a3680 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.711+0000 7f52bb1ff700 1 --2- 172.21.15.44:0/3429799178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52b41080f0 0x7f52b419f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.711+0000 7f52bb1ff700 1 --2- 172.21.15.44:0/3429799178 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52b40734d0 0x7f52b4198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.711+0000 7f52bb1ff700 1 --2- 172.21.15.44:0/3429799178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52b4074ab0 0x7f52b41987b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.711+0000 7f52bb1ff700 1 -- 172.21.15.44:0/3429799178 >> 172.21.15.44:0/3429799178 conn(0x7f52b40fb840 msgr2=0x7f52b40fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:46.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.711+0000 7f52bb1ff700 1 -- 172.21.15.44:0/3429799178 shutdown_connections 2023-11-16T16:49:46.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.711+0000 7f52bb1ff700 1 -- 172.21.15.44:0/3429799178 wait complete. 2023-11-16T16:49:46.734 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2023-11-16T16:49:46.957 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:46 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:49:46.957 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:46 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:49:46.957 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:46 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:49:46.957 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:46 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:49:46.957 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:46 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3429799178' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-11-16T16:49:46.957 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:46 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-11-16T16:49:46.957 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:46 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: ignoring --setuser ceph since I am not root 2023-11-16T16:49:46.957 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:46 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: ignoring --setgroup ceph since I am not root 2023-11-16T16:49:46.957 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:46 smithi077 ceph-mgr[132689]: -- 172.21.15.77:0/1182618024 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b839efd520 con 0x55b839ed2c00 2023-11-16T16:49:46.957 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:46 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:46.874+0000 7fd83359a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T16:49:46.957 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:46 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:46.954+0000 7fd83359a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T16:49:46.963 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:46 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:49:46.963 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:46 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:49:46.963 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:46 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:49:46.963 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:46 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:49:46.963 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:46 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3429799178' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-11-16T16:49:46.963 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:46 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-11-16T16:49:46.963 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:46 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: ignoring --setuser ceph since I am not root 2023-11-16T16:49:46.963 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:46 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: ignoring --setgroup ceph since I am not root 2023-11-16T16:49:46.963 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:46 smithi176 ceph-mgr[133522]: -- 172.21.15.176:0/3338890196 <== mon.1 v2:172.21.15.176:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562ee4981520 con 0x562ee4957000 2023-11-16T16:49:46.963 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:46 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:46.882+0000 7f4d5b494200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T16:49:46.963 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:46 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:46.961+0000 7f4d5b494200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T16:49:46.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.984+0000 7ff6655ff700 1 -- 172.21.15.44:0/2723501962 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff660100240 msgr2=0x7ff660100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.984+0000 7ff6655ff700 1 --2- 172.21.15.44:0/2723501962 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff660100240 0x7ff660100650 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff6500098f0 tx=0x7ff650009c00 comp rx=0 tx=0).stop 2023-11-16T16:49:46.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.984+0000 7ff6655ff700 1 -- 172.21.15.44:0/2723501962 shutdown_connections 2023-11-16T16:49:46.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.984+0000 7ff6655ff700 1 --2- 172.21.15.44:0/2723501962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff660101dc0 0x7ff660108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.984+0000 7ff6655ff700 1 --2- 172.21.15.44:0/2723501962 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff660100240 0x7ff660100650 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.984+0000 7ff6655ff700 1 --2- 172.21.15.44:0/2723501962 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff660101430 0x7ff660101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.984+0000 7ff6655ff700 1 -- 172.21.15.44:0/2723501962 >> 172.21.15.44:0/2723501962 conn(0x7ff6600fb840 msgr2=0x7ff6600fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:46.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.984+0000 7ff6655ff700 1 -- 172.21.15.44:0/2723501962 shutdown_connections 2023-11-16T16:49:46.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.984+0000 7ff6655ff700 1 -- 172.21.15.44:0/2723501962 wait complete. 2023-11-16T16:49:46.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.986+0000 7ff6655ff700 1 Processor -- start 2023-11-16T16:49:46.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.986+0000 7ff6655ff700 1 -- start start 2023-11-16T16:49:46.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.986+0000 7ff6655ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff660100240 0x7ff660196130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.986+0000 7ff6655ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff660101430 0x7ff660196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.986+0000 7ff6655ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff660101dc0 0x7ff66019d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.986+0000 7ff6655ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66019dcb0 con 0x7ff660101dc0 2023-11-16T16:49:46.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.986+0000 7ff6655ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66019ddf0 con 0x7ff660100240 2023-11-16T16:49:46.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.986+0000 7ff6655ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66019df80 con 0x7ff660101430 2023-11-16T16:49:46.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.987+0000 7ff65effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff660100240 0x7ff660196130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.987+0000 7ff65effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff660100240 0x7ff660196130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:57744/0 (socket says 172.21.15.44:57744) 2023-11-16T16:49:46.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.987+0000 7ff65effd700 1 -- 172.21.15.44:0/3115939855 learned_addr learned my addr 172.21.15.44:0/3115939855 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:46.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.987+0000 7ff65e7fc700 1 --2- 172.21.15.44:0/3115939855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff660101430 0x7ff660196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.987+0000 7ff65f7fe700 1 --2- 172.21.15.44:0/3115939855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff660101dc0 0x7ff66019d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.987+0000 7ff65effd700 1 -- 172.21.15.44:0/3115939855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff660101430 msgr2=0x7ff660196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.987+0000 7ff65effd700 1 --2- 172.21.15.44:0/3115939855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff660101430 0x7ff660196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.987+0000 7ff65effd700 1 -- 172.21.15.44:0/3115939855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff660101dc0 msgr2=0x7ff66019d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.987+0000 7ff65effd700 1 --2- 172.21.15.44:0/3115939855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff660101dc0 0x7ff66019d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.987+0000 7ff65effd700 1 -- 172.21.15.44:0/3115939855 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff658009c40 con 0x7ff660100240 2023-11-16T16:49:46.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.987+0000 7ff65effd700 1 --2- 172.21.15.44:0/3115939855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff660100240 0x7ff660196130 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff650005e90 tx=0x7ff65000b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:46.991 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.988+0000 7ff647fff700 1 -- 172.21.15.44:0/3115939855 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65001c070 con 0x7ff660100240 2023-11-16T16:49:46.991 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.988+0000 7ff647fff700 1 -- 172.21.15.44:0/3115939855 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff65000f7f0 con 0x7ff660100240 2023-11-16T16:49:46.991 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.988+0000 7ff6655ff700 1 -- 172.21.15.44:0/3115939855 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff650009520 con 0x7ff660100240 2023-11-16T16:49:46.991 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.988+0000 7ff647fff700 1 -- 172.21.15.44:0/3115939855 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff650003ac0 con 0x7ff660100240 2023-11-16T16:49:46.991 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.988+0000 7ff6655ff700 1 -- 172.21.15.44:0/3115939855 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff66010bbf0 con 0x7ff660100240 2023-11-16T16:49:46.991 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.989+0000 7ff6655ff700 1 -- 172.21.15.44:0/3115939855 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6601078c0 con 0x7ff660100240 2023-11-16T16:49:46.992 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.991+0000 7ff647fff700 1 -- 172.21.15.44:0/3115939855 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 135732+0+0 (secure 0 0 0) 0x7ff650020030 con 0x7ff660100240 2023-11-16T16:49:46.994 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.992+0000 7ff647fff700 1 --2- 172.21.15.44:0/3115939855 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff6480a11a0 0x7ff6480a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.994 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.992+0000 7ff65e7fc700 1 -- 172.21.15.44:0/3115939855 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff6480a11a0 msgr2=0x7ff6480a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/304422349 2023-11-16T16:49:46.994 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.992+0000 7ff647fff700 1 -- 172.21.15.44:0/3115939855 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7ff6500ce5f0 con 0x7ff660100240 2023-11-16T16:49:46.994 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.992+0000 7ff65e7fc700 1 --2- 172.21.15.44:0/3115939855 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff6480a11a0 0x7ff6480a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:49:46.995 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:46.994+0000 7ff647fff700 1 -- 172.21.15.44:0/3115939855 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff6500979d0 con 0x7ff660100240 2023-11-16T16:49:47.024 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:46 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: ignoring --setuser ceph since I am not root 2023-11-16T16:49:47.024 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:46 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: ignoring --setgroup ceph since I am not root 2023-11-16T16:49:47.024 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:46 smithi044 ceph-mgr[129024]: -- 172.21.15.44:0/496442621 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559d1bd9b520 con 0x559d1bd72800 2023-11-16T16:49:47.024 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:47 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:47.022+0000 7fc1cc28b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T16:49:47.024 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:46 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:49:47.024 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:46 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:49:47.024 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:46 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:49:47.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:46 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:49:47.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:46 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3429799178' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-11-16T16:49:47.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:46 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-11-16T16:49:47.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.193+0000 7ff65e7fc700 1 -- 172.21.15.44:0/3115939855 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff6480a11a0 msgr2=0x7ff6480a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/304422349 2023-11-16T16:49:47.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.193+0000 7ff65e7fc700 1 --2- 172.21.15.44:0/3115939855 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff6480a11a0 0x7ff6480a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T16:49:47.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.208+0000 7ff6655ff700 1 -- 172.21.15.44:0/3115939855 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7ff66002cfe0 con 0x7ff660100240 2023-11-16T16:49:47.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.213+0000 7ff647fff700 1 -- 172.21.15.44:0/3115939855 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff65009aff0 con 0x7ff660100240 2023-11-16T16:49:47.219 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.217+0000 7ff6655ff700 1 -- 172.21.15.44:0/3115939855 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff6480a11a0 msgr2=0x7ff6480a3650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:49:47.219 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.217+0000 7ff6655ff700 1 --2- 172.21.15.44:0/3115939855 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff6480a11a0 0x7ff6480a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.219 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.217+0000 7ff6655ff700 1 -- 172.21.15.44:0/3115939855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff660100240 msgr2=0x7ff660196130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:47.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.217+0000 7ff6655ff700 1 --2- 172.21.15.44:0/3115939855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff660100240 0x7ff660196130 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff650005e90 tx=0x7ff65000b7e0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.217+0000 7ff6655ff700 1 -- 172.21.15.44:0/3115939855 shutdown_connections 2023-11-16T16:49:47.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.217+0000 7ff6655ff700 1 --2- 172.21.15.44:0/3115939855 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7ff6480a11a0 0x7ff6480a3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.217+0000 7ff6655ff700 1 --2- 172.21.15.44:0/3115939855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff660101dc0 0x7ff66019d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.217+0000 7ff6655ff700 1 --2- 172.21.15.44:0/3115939855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff660100240 0x7ff660196130 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.217+0000 7ff6655ff700 1 --2- 172.21.15.44:0/3115939855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff660101430 0x7ff660196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.217+0000 7ff6655ff700 1 -- 172.21.15.44:0/3115939855 >> 172.21.15.44:0/3115939855 conn(0x7ff6600fb840 msgr2=0x7ff660104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:47.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.217+0000 7ff6655ff700 1 -- 172.21.15.44:0/3115939855 shutdown_connections 2023-11-16T16:49:47.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.217+0000 7ff6655ff700 1 -- 172.21.15.44:0/3115939855 wait complete. 2023-11-16T16:49:47.226 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:47 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:47.045+0000 7fd83359a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T16:49:47.226 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:47 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:47.137+0000 7fd83359a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T16:49:47.246 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2023-11-16T16:49:47.314 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:47 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:47.130+0000 7fc1cc28b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T16:49:47.314 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:47 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:47.264+0000 7fc1cc28b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T16:49:47.315 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:47.052+0000 7f4d5b494200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T16:49:47.315 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:47.146+0000 7f4d5b494200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T16:49:47.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.457+0000 7fe57dbff700 1 -- 172.21.15.44:0/3237024383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe578100240 msgr2=0x7fe578100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:47.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.457+0000 7fe57dbff700 1 --2- 172.21.15.44:0/3237024383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe578100240 0x7fe578100650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe568009990 tx=0x7fe568009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.457+0000 7fe57dbff700 1 -- 172.21.15.44:0/3237024383 shutdown_connections 2023-11-16T16:49:47.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.457+0000 7fe57dbff700 1 --2- 172.21.15.44:0/3237024383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe578101dc0 0x7fe578108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.457+0000 7fe57dbff700 1 --2- 172.21.15.44:0/3237024383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe578100240 0x7fe578100650 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.457+0000 7fe57dbff700 1 --2- 172.21.15.44:0/3237024383 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe578101430 0x7fe578101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.457+0000 7fe57dbff700 1 -- 172.21.15.44:0/3237024383 >> 172.21.15.44:0/3237024383 conn(0x7fe5780fb840 msgr2=0x7fe5780fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:47.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.457+0000 7fe57dbff700 1 -- 172.21.15.44:0/3237024383 shutdown_connections 2023-11-16T16:49:47.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.457+0000 7fe57dbff700 1 -- 172.21.15.44:0/3237024383 wait complete. 2023-11-16T16:49:47.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.458+0000 7fe57dbff700 1 Processor -- start 2023-11-16T16:49:47.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.458+0000 7fe57dbff700 1 -- start start 2023-11-16T16:49:47.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe57dbff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe578100240 0x7fe578198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:47.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe57dbff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe578101430 0x7fe578198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:47.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe57dbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe578101dc0 0x7fe57819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:47.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe57dbff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57819fda0 con 0x7fe578100240 2023-11-16T16:49:47.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe57dbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57819fee0 con 0x7fe578101dc0 2023-11-16T16:49:47.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe57dbff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5781a0070 con 0x7fe578101430 2023-11-16T16:49:47.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe576ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe578101430 0x7fe578198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:47.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe5777fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe578100240 0x7fe578198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:47.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe577fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe578101dc0 0x7fe57819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:47.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe576ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe578101430 0x7fe578198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:51590/0 (socket says 172.21.15.44:51590) 2023-11-16T16:49:47.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe576ffd700 1 -- 172.21.15.44:0/3298135034 learned_addr learned my addr 172.21.15.44:0/3298135034 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:47.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe5777fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe578100240 0x7fe578198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56624/0 (socket says 172.21.15.44:56624) 2023-11-16T16:49:47.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe576ffd700 1 -- 172.21.15.44:0/3298135034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe578101dc0 msgr2=0x7fe57819f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:47.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe576ffd700 1 --2- 172.21.15.44:0/3298135034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe578101dc0 0x7fe57819f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe576ffd700 1 -- 172.21.15.44:0/3298135034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe578100240 msgr2=0x7fe578198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:47.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe576ffd700 1 --2- 172.21.15.44:0/3298135034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe578100240 0x7fe578198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe576ffd700 1 -- 172.21.15.44:0/3298135034 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5680095f0 con 0x7fe578101430 2023-11-16T16:49:47.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe577fff700 1 --2- 172.21.15.44:0/3298135034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe578101dc0 0x7fe57819f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:47.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.459+0000 7fe5777fe700 1 --2- 172.21.15.44:0/3298135034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe578100240 0x7fe578198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:47.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.460+0000 7fe576ffd700 1 --2- 172.21.15.44:0/3298135034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe578101430 0x7fe578198760 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe56c00e9a0 tx=0x7fe56c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:47.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.460+0000 7fe574ff9700 1 -- 172.21.15.44:0/3298135034 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c009790 con 0x7fe578101430 2023-11-16T16:49:47.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.460+0000 7fe574ff9700 1 -- 172.21.15.44:0/3298135034 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe56c010430 con 0x7fe578101430 2023-11-16T16:49:47.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.460+0000 7fe57dbff700 1 -- 172.21.15.44:0/3298135034 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57810bbf0 con 0x7fe578101430 2023-11-16T16:49:47.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.460+0000 7fe574ff9700 1 -- 172.21.15.44:0/3298135034 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c018680 con 0x7fe578101430 2023-11-16T16:49:47.464 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.460+0000 7fe57dbff700 1 -- 172.21.15.44:0/3298135034 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57810ba60 con 0x7fe578101430 2023-11-16T16:49:47.464 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.461+0000 7fe57dbff700 1 -- 172.21.15.44:0/3298135034 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe578192950 con 0x7fe578101430 2023-11-16T16:49:47.464 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.463+0000 7fe574ff9700 1 -- 172.21.15.44:0/3298135034 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 19) v1 ==== 135732+0+0 (secure 0 0 0) 0x7fe56c0105a0 con 0x7fe578101430 2023-11-16T16:49:47.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.464+0000 7fe574ff9700 1 --2- 172.21.15.44:0/3298135034 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5600a11a0 0x7fe5600a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:47.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.464+0000 7fe5777fe700 1 -- 172.21.15.44:0/3298135034 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5600a11a0 msgr2=0x7fe5600a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/304422349 2023-11-16T16:49:47.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.464+0000 7fe574ff9700 1 -- 172.21.15.44:0/3298135034 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fe56c014070 con 0x7fe578101430 2023-11-16T16:49:47.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.464+0000 7fe5777fe700 1 --2- 172.21.15.44:0/3298135034 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5600a11a0 0x7fe5600a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:49:47.469 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.468+0000 7fe574ff9700 1 -- 172.21.15.44:0/3298135034 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe56c051730 con 0x7fe578101430 2023-11-16T16:49:47.612 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:47 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:47.306+0000 7fd83359a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T16:49:47.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:47 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:47.397+0000 7fc1cc28b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T16:49:47.624 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:47.314+0000 7f4d5b494200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T16:49:47.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.665+0000 7fe5777fe700 1 -- 172.21.15.44:0/3298135034 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5600a11a0 msgr2=0x7fe5600a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/304422349 2023-11-16T16:49:47.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.665+0000 7fe5777fe700 1 --2- 172.21.15.44:0/3298135034 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5600a11a0 0x7fe5600a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T16:49:47.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.685+0000 7fe57dbff700 1 -- 172.21.15.44:0/3298135034 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fe57802cfe0 con 0x7fe578101430 2023-11-16T16:49:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.693+0000 7fe574ff9700 1 -- 172.21.15.44:0/3298135034 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe56c099d10 con 0x7fe578101430 2023-11-16T16:49:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.697+0000 7fe57dbff700 1 -- 172.21.15.44:0/3298135034 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5600a11a0 msgr2=0x7fe5600a3650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:49:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.697+0000 7fe57dbff700 1 --2- 172.21.15.44:0/3298135034 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5600a11a0 0x7fe5600a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.697+0000 7fe57dbff700 1 -- 172.21.15.44:0/3298135034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe578101430 msgr2=0x7fe578198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.697+0000 7fe57dbff700 1 --2- 172.21.15.44:0/3298135034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe578101430 0x7fe578198760 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe56c00e9a0 tx=0x7fe56c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.697+0000 7fe57dbff700 1 -- 172.21.15.44:0/3298135034 shutdown_connections 2023-11-16T16:49:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.697+0000 7fe57dbff700 1 --2- 172.21.15.44:0/3298135034 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7fe5600a11a0 0x7fe5600a3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.697+0000 7fe57dbff700 1 --2- 172.21.15.44:0/3298135034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe578101dc0 0x7fe57819f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.697+0000 7fe57dbff700 1 --2- 172.21.15.44:0/3298135034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe578100240 0x7fe578198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.697+0000 7fe57dbff700 1 --2- 172.21.15.44:0/3298135034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe578101430 0x7fe578198760 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.697+0000 7fe57dbff700 1 -- 172.21.15.44:0/3298135034 >> 172.21.15.44:0/3298135034 conn(0x7fe5780fb840 msgr2=0x7fe578106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:47.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.697+0000 7fe57dbff700 1 -- 172.21.15.44:0/3298135034 shutdown_connections 2023-11-16T16:49:47.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.697+0000 7fe57dbff700 1 -- 172.21.15.44:0/3298135034 wait complete. 2023-11-16T16:49:47.721 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M' 2023-11-16T16:49:47.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:47 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-11-16T16:49:47.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:47 smithi176 ceph-mon[131630]: mgrmap e19: x(active, since 9m), standbys: z, y 2023-11-16T16:49:47.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:47 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:49:47.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:47 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' 2023-11-16T16:49:47.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:47 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' 2023-11-16T16:49:47.895 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:47.623+0000 7f4d5b494200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T16:49:47.895 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:47.714+0000 7f4d5b494200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T16:49:47.895 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:47.843+0000 7f4d5b494200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T16:49:47.924 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:47 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-11-16T16:49:47.924 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:47 smithi077 ceph-mon[131204]: mgrmap e19: x(active, since 9m), standbys: z, y 2023-11-16T16:49:47.924 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:47 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:49:47.924 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:47 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' 2023-11-16T16:49:47.924 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:47 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' 2023-11-16T16:49:47.924 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:47 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:47.610+0000 7fd83359a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T16:49:47.925 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:47 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:47.699+0000 7fd83359a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T16:49:47.925 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:47 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:47.828+0000 7fd83359a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T16:49:47.925 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:47 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:47.922+0000 7fd83359a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T16:49:47.925 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.923+0000 7f492bbff700 1 -- 172.21.15.44:0/4064017502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4924101dc0 msgr2=0x7f4924108680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:47.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.923+0000 7f492bbff700 1 --2- 172.21.15.44:0/4064017502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4924101dc0 0x7f4924108680 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f491c00b0a0 tx=0x7f491c00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.923+0000 7f492bbff700 1 -- 172.21.15.44:0/4064017502 shutdown_connections 2023-11-16T16:49:47.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.923+0000 7f492bbff700 1 --2- 172.21.15.44:0/4064017502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4924101dc0 0x7f4924108680 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.923+0000 7f492bbff700 1 --2- 172.21.15.44:0/4064017502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4924100240 0x7f4924100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.923+0000 7f492bbff700 1 --2- 172.21.15.44:0/4064017502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4924101430 0x7f4924101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.923+0000 7f492bbff700 1 -- 172.21.15.44:0/4064017502 >> 172.21.15.44:0/4064017502 conn(0x7f49240fb840 msgr2=0x7f49240fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:47.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.924+0000 7f492bbff700 1 -- 172.21.15.44:0/4064017502 shutdown_connections 2023-11-16T16:49:47.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.924+0000 7f492bbff700 1 -- 172.21.15.44:0/4064017502 wait complete. 2023-11-16T16:49:47.928 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.925+0000 7f492bbff700 1 Processor -- start 2023-11-16T16:49:47.928 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.925+0000 7f492bbff700 1 -- start start 2023-11-16T16:49:47.928 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.925+0000 7f492bbff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4924100240 0x7f49241960f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:47.928 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.925+0000 7f492bbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4924101430 0x7f4924196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:47.928 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.925+0000 7f492bbff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4924101dc0 0x7f492419d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:47.928 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.925+0000 7f492bbff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492419dc70 con 0x7f4924100240 2023-11-16T16:49:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.925+0000 7f492bbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492419ddb0 con 0x7f4924101430 2023-11-16T16:49:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.925+0000 7f492bbff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492419df40 con 0x7f4924101dc0 2023-11-16T16:49:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.925+0000 7f492999b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4924100240 0x7f49241960f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.925+0000 7f492919a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4924101430 0x7f4924196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.925+0000 7f492a19c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4924101dc0 0x7f492419d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.925+0000 7f492999b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4924100240 0x7f49241960f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56642/0 (socket says 172.21.15.44:56642) 2023-11-16T16:49:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.925+0000 7f492919a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4924101430 0x7f4924196630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:57778/0 (socket says 172.21.15.44:57778) 2023-11-16T16:49:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.925+0000 7f492999b700 1 -- 172.21.15.44:0/2585948165 learned_addr learned my addr 172.21.15.44:0/2585948165 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.926+0000 7f492919a700 1 -- 172.21.15.44:0/2585948165 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4924101dc0 msgr2=0x7f492419d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.926+0000 7f492919a700 1 --2- 172.21.15.44:0/2585948165 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4924101dc0 0x7f492419d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.926+0000 7f492919a700 1 -- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4924100240 msgr2=0x7f49241960f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.926+0000 7f492919a700 1 --2- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4924100240 0x7f49241960f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.926+0000 7f492919a700 1 -- 172.21.15.44:0/2585948165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491c009d00 con 0x7f4924101430 2023-11-16T16:49:47.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.926+0000 7f492919a700 1 --2- 172.21.15.44:0/2585948165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4924101430 0x7f4924196630 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f491400e9f0 tx=0x7f491400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:47.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.926+0000 7f491affd700 1 -- 172.21.15.44:0/2585948165 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491400c9c0 con 0x7f4924101430 2023-11-16T16:49:47.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.926+0000 7f491affd700 1 -- 172.21.15.44:0/2585948165 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4914010430 con 0x7f4924101430 2023-11-16T16:49:47.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.926+0000 7f491affd700 1 -- 172.21.15.44:0/2585948165 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914018680 con 0x7f4924101430 2023-11-16T16:49:47.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.926+0000 7f492bbff700 1 -- 172.21.15.44:0/2585948165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492410bc20 con 0x7f4924101430 2023-11-16T16:49:47.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.927+0000 7f492bbff700 1 -- 172.21.15.44:0/2585948165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492410ba90 con 0x7f4924101430 2023-11-16T16:49:47.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.928+0000 7f492bbff700 1 -- 172.21.15.44:0/2585948165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4924073140 con 0x7f4924101430 2023-11-16T16:49:47.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.929+0000 7f491affd700 1 -- 172.21.15.44:0/2585948165 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 135732+0+0 (secure 0 0 0) 0x7f4914010860 con 0x7f4924101430 2023-11-16T16:49:47.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.931+0000 7f491affd700 1 --2- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49100a1220 0x7f49100a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:47.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.931+0000 7f492999b700 1 -- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49100a1220 msgr2=0x7f49100a36d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/304422349 2023-11-16T16:49:47.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.931+0000 7f491affd700 1 -- 172.21.15.44:0/2585948165 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f4914014070 con 0x7f4924101430 2023-11-16T16:49:47.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.931+0000 7f492999b700 1 --2- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49100a1220 0x7f49100a36d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:49:47.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:47.935+0000 7f491affd700 1 -- 172.21.15.44:0/2585948165 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4914051730 con 0x7f4924101430 2023-11-16T16:49:48.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:47 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-11-16T16:49:48.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:47 smithi044 ceph-mon[128750]: mgrmap e19: x(active, since 9m), standbys: z, y 2023-11-16T16:49:48.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:47 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:49:48.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:47 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' 2023-11-16T16:49:48.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:47 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' 2023-11-16T16:49:48.055 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:47 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:47.631+0000 7fc1cc28b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T16:49:48.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:48.132+0000 7f492999b700 1 -- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49100a1220 msgr2=0x7f49100a36d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/304422349 2023-11-16T16:49:48.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:48.132+0000 7f492999b700 1 --2- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49100a1220 0x7f49100a36d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T16:49:48.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:48.171+0000 7f492bbff700 1 -- 172.21.15.44:0/2585948165 --> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f4924073f20 con 0x7f49100a1220 2023-11-16T16:49:48.345 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:48 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:48.052+0000 7fc1cc28b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T16:49:48.345 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:48 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:48.173+0000 7fc1cc28b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T16:49:48.359 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:47.938+0000 7f4d5b494200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T16:49:48.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:48.533+0000 7f492999b700 1 -- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49100a1220 msgr2=0x7f49100a36d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/304422349 2023-11-16T16:49:48.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:48.533+0000 7f492999b700 1 --2- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49100a1220 0x7f49100a36d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-16T16:49:48.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:48 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:48.343+0000 7fc1cc28b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T16:49:48.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:48 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:48.472+0000 7fc1cc28b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T16:49:48.644 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:48 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:48.358+0000 7f4d5b494200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T16:49:48.726 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:48 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:48.343+0000 7fd83359a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T16:49:49.226 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:48 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:48.905+0000 7fd83359a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T16:49:49.226 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:49.075+0000 7fd83359a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T16:49:49.335 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:49.334+0000 7f492999b700 1 -- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49100a1220 msgr2=0x7f49100a36d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/304422349 2023-11-16T16:49:49.335 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:49.334+0000 7f492999b700 1 --2- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49100a1220 0x7f49100a36d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-16T16:49:49.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:49 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:49.034+0000 7fc1cc28b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T16:49:49.394 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:48 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:48.925+0000 7f4d5b494200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T16:49:49.395 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:49 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:49.096+0000 7f4d5b494200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T16:49:50.096 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:49.834+0000 7fd83359a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:49:50.096 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:49.925+0000 7fd83359a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T16:49:50.096 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:50.093+0000 7fd83359a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:49:50.118 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:49 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:49.811+0000 7fc1cc28b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T16:49:50.118 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:50 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:50.028+0000 7fc1cc28b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T16:49:50.126 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:49 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:49.863+0000 7f4d5b494200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:49:50.126 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:49 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:49.955+0000 7f4d5b494200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T16:49:50.394 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:50 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:50.125+0000 7f4d5b494200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:49:50.726 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:50.371+0000 7fd83359a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T16:49:50.726 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:50.594+0000 7fd83359a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T16:49:50.776 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:50 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:50.402+0000 7f4d5b494200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T16:49:50.776 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:50 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:50.628+0000 7f4d5b494200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T16:49:50.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:50.936+0000 7f492999b700 1 -- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49100a1220 msgr2=0x7f49100a36d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/304422349 2023-11-16T16:49:50.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:50.936+0000 7f492999b700 1 --2- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49100a1220 0x7f49100a36d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-16T16:49:51.002 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:50.741+0000 7fd83359a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T16:49:51.036 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:50 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:50.775+0000 7f4d5b494200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T16:49:51.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:51.021+0000 7fc1cc28b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:49:51.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:51.145+0000 7fc1cc28b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T16:49:51.394 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:51 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:51.035+0000 7f4d5b494200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T16:49:51.476 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:51 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:51.000+0000 7fd83359a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T16:49:51.759 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:51.378+0000 7fc1cc28b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:49:51.976 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:51 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:51.693+0000 7fd83359a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T16:49:51.976 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:51 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:51.822+0000 7fd83359a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T16:49:52.048 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:51.757+0000 7fc1cc28b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T16:49:52.058 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:51 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:51.737+0000 7f4d5b494200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T16:49:52.058 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:51 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:51.868+0000 7f4d5b494200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T16:49:52.341 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:52 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:52.012+0000 7fd83359a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T16:49:52.341 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:52 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:52.125+0000 7fd83359a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T16:49:52.341 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:52 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:52.339+0000 7fd83359a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T16:49:52.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:52 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:52.046+0000 7fc1cc28b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T16:49:52.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:52 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:52.248+0000 7fc1cc28b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T16:49:52.385 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:52 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:52.057+0000 7f4d5b494200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T16:49:52.386 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:52 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:52.170+0000 7f4d5b494200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T16:49:52.644 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:52 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:52.384+0000 7f4d5b494200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T16:49:52.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:52 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:52.607+0000 7fc1cc28b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T16:49:53.476 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 16:49:53 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y[132665]: 2023-11-16T16:49:53.188+0000 7fd83359a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T16:49:53.644 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 16:49:53 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z[133479]: 2023-11-16T16:49:53.239+0000 7f4d5b494200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T16:49:53.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:53 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:53.585+0000 7fc1cc28b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T16:49:53.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:53 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:53.767+0000 7fc1cc28b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T16:49:54.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:54.140+0000 7f492999b700 1 -- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49100a1220 msgr2=0x7f49100a36d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/304422349 2023-11-16T16:49:54.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:54.140+0000 7f492999b700 1 --2- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49100a1220 0x7f49100a36d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-16T16:49:54.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:54.207+0000 7f491affd700 1 -- 172.21.15.44:0/2585948165 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 20) v1 ==== 135732+0+0 (secure 0 0 0) 0x7f4914096550 con 0x7f4924101430 2023-11-16T16:49:54.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:54 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:54.027+0000 7fc1cc28b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T16:49:54.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:54 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:54.183+0000 7fc1cc28b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T16:49:54.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:54 smithi044 ceph-mon[128750]: Standby manager daemon y restarted 2023-11-16T16:49:54.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:54 smithi044 ceph-mon[128750]: Standby manager daemon y started 2023-11-16T16:49:54.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:54 smithi044 ceph-mon[128750]: from='mgr.? 172.21.15.77:0/1174593880' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T16:49:54.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:54 smithi044 ceph-mon[128750]: from='mgr.? 172.21.15.77:0/1174593880' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:49:54.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:54 smithi044 ceph-mon[128750]: from='mgr.? 172.21.15.77:0/1174593880' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T16:49:54.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:54 smithi044 ceph-mon[128750]: from='mgr.? 172.21.15.77:0/1174593880' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:49:54.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:54 smithi044 ceph-mon[128750]: Standby manager daemon z restarted 2023-11-16T16:49:54.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:54 smithi044 ceph-mon[128750]: Standby manager daemon z started 2023-11-16T16:49:54.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:54 smithi044 ceph-mon[128750]: from='mgr.? 172.21.15.176:0/2647272864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T16:49:54.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:54 smithi044 ceph-mon[128750]: from='mgr.? 172.21.15.176:0/2647272864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:49:54.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:54 smithi044 ceph-mon[128750]: from='mgr.? 172.21.15.176:0/2647272864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T16:49:54.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:54 smithi044 ceph-mon[128750]: from='mgr.? 172.21.15.176:0/2647272864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:49:54.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:54 smithi077 ceph-mon[131204]: Standby manager daemon y restarted 2023-11-16T16:49:54.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:54 smithi077 ceph-mon[131204]: Standby manager daemon y started 2023-11-16T16:49:54.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:54 smithi077 ceph-mon[131204]: from='mgr.? 172.21.15.77:0/1174593880' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T16:49:54.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:54 smithi077 ceph-mon[131204]: from='mgr.? 172.21.15.77:0/1174593880' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:49:54.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:54 smithi077 ceph-mon[131204]: from='mgr.? 172.21.15.77:0/1174593880' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T16:49:54.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:54 smithi077 ceph-mon[131204]: from='mgr.? 172.21.15.77:0/1174593880' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:49:54.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:54 smithi077 ceph-mon[131204]: Standby manager daemon z restarted 2023-11-16T16:49:54.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:54 smithi077 ceph-mon[131204]: Standby manager daemon z started 2023-11-16T16:49:54.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:54 smithi077 ceph-mon[131204]: from='mgr.? 172.21.15.176:0/2647272864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T16:49:54.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:54 smithi077 ceph-mon[131204]: from='mgr.? 172.21.15.176:0/2647272864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:49:54.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:54 smithi077 ceph-mon[131204]: from='mgr.? 172.21.15.176:0/2647272864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T16:49:54.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:54 smithi077 ceph-mon[131204]: from='mgr.? 172.21.15.176:0/2647272864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:49:54.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:54 smithi176 ceph-mon[131630]: Standby manager daemon y restarted 2023-11-16T16:49:54.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:54 smithi176 ceph-mon[131630]: Standby manager daemon y started 2023-11-16T16:49:54.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:54 smithi176 ceph-mon[131630]: from='mgr.? 172.21.15.77:0/1174593880' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T16:49:54.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:54 smithi176 ceph-mon[131630]: from='mgr.? 172.21.15.77:0/1174593880' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:49:54.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:54 smithi176 ceph-mon[131630]: from='mgr.? 172.21.15.77:0/1174593880' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T16:49:54.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:54 smithi176 ceph-mon[131630]: from='mgr.? 172.21.15.77:0/1174593880' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:49:54.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:54 smithi176 ceph-mon[131630]: Standby manager daemon z restarted 2023-11-16T16:49:54.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:54 smithi176 ceph-mon[131630]: Standby manager daemon z started 2023-11-16T16:49:54.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:54 smithi176 ceph-mon[131630]: from='mgr.? 172.21.15.176:0/2647272864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T16:49:54.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:54 smithi176 ceph-mon[131630]: from='mgr.? 172.21.15.176:0/2647272864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:49:54.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:54 smithi176 ceph-mon[131630]: from='mgr.? 172.21.15.176:0/2647272864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T16:49:54.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:54 smithi176 ceph-mon[131630]: from='mgr.? 172.21.15.176:0/2647272864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:49:54.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:54 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:54.479+0000 7fc1cc28b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T16:49:55.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:55 smithi077 ceph-mon[131204]: mgrmap e20: x(active, since 9m), standbys: z, y 2023-11-16T16:49:55.605 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:55 smithi044 ceph-mon[128750]: mgrmap e20: x(active, since 9m), standbys: z, y 2023-11-16T16:49:55.623 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:55.623+0000 7f491affd700 1 -- 172.21.15.44:0/2585948165 <== mon.2 v2:172.21.15.77:3300/0 8 ==== mgrmap(e 21) v1 ==== 135400+0+0 (secure 0 0 0) 0x7f49140ca5b0 con 0x7f4924101430 2023-11-16T16:49:55.623 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:55.623+0000 7f491affd700 1 -- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49100a1220 msgr2=0x7f49100a36d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:49:55.624 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:55.623+0000 7f491affd700 1 --2- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/304422349,v1:172.21.15.44:6801/304422349] conn(0x7f49100a1220 0x7f49100a36d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:55.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:55 smithi176 ceph-mon[131630]: mgrmap e20: x(active, since 9m), standbys: z, y 2023-11-16T16:49:55.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:55.604+0000 7fc1cc28b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T16:49:55.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:55.841+0000 7fc1286e3700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:55.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:55.841+0000 7fc1286e3700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:55.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:55.841+0000 7fc1286e3700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:55.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:55.841+0000 7fc1286e3700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:55.868 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:55.841+0000 7fc1286e3700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:55.842+0000 7fc12b6e9700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:55.842+0000 7fc12b6e9700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:55.842+0000 7fc12b6e9700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:55.842+0000 7fc12b6e9700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:49:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:49:55.842+0000 7fc12b6e9700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:56.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: Active manager daemon x restarted 2023-11-16T16:49:56.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: Activating manager daemon x 2023-11-16T16:49:56.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: osdmap e92: 12 total, 12 up, 12 in 2023-11-16T16:49:56.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: mgrmap e21: x(active, starting, since 0.0119786s), standbys: z, y 2023-11-16T16:49:56.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:49:56.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:49:56.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:49:56.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T16:49:56.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T16:49:56.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T16:49:56.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T16:49:56.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T16:49:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T16:49:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T16:49:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T16:49:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T16:49:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T16:49:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T16:49:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T16:49:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T16:49:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T16:49:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T16:49:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:49:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:49:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:49:56.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:49:56.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:49:56.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:49:56.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:49:56.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:49:56.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:49:56.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:49:56.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:49:56.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:49:56.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T16:49:56.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T16:49:56.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T16:49:56.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: Manager daemon x is now available 2023-11-16T16:49:56.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:56.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:56.372 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T16:49:56.372 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:56 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T16:49:56.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: Active manager daemon x restarted 2023-11-16T16:49:56.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: Activating manager daemon x 2023-11-16T16:49:56.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: osdmap e92: 12 total, 12 up, 12 in 2023-11-16T16:49:56.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: mgrmap e21: x(active, starting, since 0.0119786s), standbys: z, y 2023-11-16T16:49:56.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:49:56.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:49:56.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:49:56.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T16:49:56.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T16:49:56.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T16:49:56.477 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T16:49:56.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T16:49:56.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T16:49:56.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T16:49:56.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T16:49:56.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T16:49:56.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T16:49:56.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T16:49:56.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T16:49:56.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T16:49:56.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T16:49:56.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T16:49:56.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:49:56.478 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:49:56.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:49:56.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:49:56.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:49:56.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:49:56.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:49:56.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:49:56.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:49:56.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:49:56.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:49:56.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:49:56.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T16:49:56.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T16:49:56.479 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T16:49:56.480 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: Manager daemon x is now available 2023-11-16T16:49:56.480 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:56.480 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:56.480 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T16:49:56.480 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:56 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T16:49:56.627 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.627+0000 7f491affd700 1 -- 172.21.15.44:0/2585948165 <== mon.2 v2:172.21.15.77:3300/0 9 ==== mgrmap(e 22) v1 ==== 135527+0+0 (secure 0 0 0) 0x7f49140462f0 con 0x7f4924101430 2023-11-16T16:49:56.628 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.627+0000 7f491affd700 1 --2- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4910027f50 0x7f4910028320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:56.628 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.627+0000 7f491affd700 1 -- 172.21.15.44:0/2585948165 --> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f4924073f20 con 0x7f4910027f50 2023-11-16T16:49:56.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.633+0000 7f492999b700 1 --2- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4910027f50 0x7f4910028320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:56.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.638+0000 7f492999b700 1 --2- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4910027f50 0x7f4910028320 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4920005950 tx=0x7f4920009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:56.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: Active manager daemon x restarted 2023-11-16T16:49:56.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: Activating manager daemon x 2023-11-16T16:49:56.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: osdmap e92: 12 total, 12 up, 12 in 2023-11-16T16:49:56.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: mgrmap e21: x(active, starting, since 0.0119786s), standbys: z, y 2023-11-16T16:49:56.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:49:56.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:49:56.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:49:56.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T16:49:56.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T16:49:56.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T16:49:56.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T16:49:56.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T16:49:56.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T16:49:56.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T16:49:56.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T16:49:56.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T16:49:56.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T16:49:56.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T16:49:56.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T16:49:56.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T16:49:56.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T16:49:56.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T16:49:56.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:49:56.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:49:56.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:49:56.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:49:56.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:49:56.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:49:56.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:49:56.647 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:49:56.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:49:56.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:49:56.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:49:56.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:49:56.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T16:49:56.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T16:49:56.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T16:49:56.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: Manager daemon x is now available 2023-11-16T16:49:56.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:56.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:56.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T16:49:56.648 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:56 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T16:49:56.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.699+0000 7f491affd700 1 -- 172.21.15.44:0/2585948165 <== mgr.25447 v2:172.21.15.44:6800/3382499657 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4924073f20 con 0x7f4910027f50 2023-11-16T16:49:56.699 INFO:teuthology.orchestra.run.smithi044.stdout:Schedule set for path / 2023-11-16T16:49:56.702 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.701+0000 7f492bbff700 1 -- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4910027f50 msgr2=0x7f4910028320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:56.702 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.701+0000 7f492bbff700 1 --2- 172.21.15.44:0/2585948165 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4910027f50 0x7f4910028320 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4920005950 tx=0x7f4920009260 comp rx=0 tx=0).stop 2023-11-16T16:49:56.702 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.701+0000 7f492bbff700 1 -- 172.21.15.44:0/2585948165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4924101430 msgr2=0x7f4924196630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:56.702 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.701+0000 7f492bbff700 1 --2- 172.21.15.44:0/2585948165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4924101430 0x7f4924196630 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f491400e9f0 tx=0x7f491400ed00 comp rx=0 tx=0).stop 2023-11-16T16:49:56.703 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.701+0000 7f492a19c700 1 -- 172.21.15.44:0/2585948165 reap_dead start 2023-11-16T16:49:56.703 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.701+0000 7f492bbff700 1 -- 172.21.15.44:0/2585948165 shutdown_connections 2023-11-16T16:49:56.703 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.701+0000 7f492bbff700 1 -- 172.21.15.44:0/2585948165 >> 172.21.15.44:0/2585948165 conn(0x7f49240fb840 msgr2=0x7f49240fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:56.703 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.701+0000 7f492bbff700 1 -- 172.21.15.44:0/2585948165 shutdown_connections 2023-11-16T16:49:56.703 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.701+0000 7f492bbff700 1 -- 172.21.15.44:0/2585948165 wait complete. 2023-11-16T16:49:56.723 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h' 2023-11-16T16:49:56.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.943+0000 7f6c7fbff700 1 -- 172.21.15.44:0/2407698185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c78100cc0 msgr2=0x7f6c781030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:56.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.943+0000 7f6c7fbff700 1 --2- 172.21.15.44:0/2407698185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c78100cc0 0x7f6c781030a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009990 tx=0x7f6c74009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.943+0000 7f6c7fbff700 1 -- 172.21.15.44:0/2407698185 shutdown_connections 2023-11-16T16:49:56.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.943+0000 7f6c7fbff700 1 --2- 172.21.15.44:0/2407698185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c78105f00 0x7f6c78108360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.943+0000 7f6c7fbff700 1 --2- 172.21.15.44:0/2407698185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c78100cc0 0x7f6c781030a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.943+0000 7f6c7fbff700 1 --2- 172.21.15.44:0/2407698185 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c781035e0 0x7f6c781059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.943+0000 7f6c7fbff700 1 -- 172.21.15.44:0/2407698185 >> 172.21.15.44:0/2407698185 conn(0x7f6c780fa720 msgr2=0x7f6c780fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:56.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.943+0000 7f6c7fbff700 1 -- 172.21.15.44:0/2407698185 shutdown_connections 2023-11-16T16:49:56.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.943+0000 7f6c7fbff700 1 -- 172.21.15.44:0/2407698185 wait complete. 2023-11-16T16:49:56.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.944+0000 7f6c7fbff700 1 Processor -- start 2023-11-16T16:49:56.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.945+0000 7f6c7fbff700 1 -- start start 2023-11-16T16:49:56.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.945+0000 7f6c7fbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c78100cc0 0x7f6c78100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:56.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.945+0000 7f6c7fbff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c781035e0 0x7f6c780fec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:56.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.945+0000 7f6c7fbff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c78105f00 0x7f6c780ff1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:56.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.945+0000 7f6c7fbff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c780ff950 con 0x7f6c78105f00 2023-11-16T16:49:56.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.945+0000 7f6c7fbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c780ffac0 con 0x7f6c78100cc0 2023-11-16T16:49:56.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.945+0000 7f6c7fbff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c781a0030 con 0x7f6c781035e0 2023-11-16T16:49:56.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.945+0000 7f6c7d99b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c78100cc0 0x7f6c78100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:56.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.945+0000 7f6c7d99b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c78100cc0 0x7f6c78100630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:52332/0 (socket says 172.21.15.44:52332) 2023-11-16T16:49:56.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.945+0000 7f6c7d99b700 1 -- 172.21.15.44:0/2773882700 learned_addr learned my addr 172.21.15.44:0/2773882700 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:56.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.945+0000 7f6c7d19a700 1 --2- 172.21.15.44:0/2773882700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c781035e0 0x7f6c780fec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:56.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.945+0000 7f6c7e19c700 1 --2- 172.21.15.44:0/2773882700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c78105f00 0x7f6c780ff1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:56.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.946+0000 7f6c7d99b700 1 -- 172.21.15.44:0/2773882700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c781035e0 msgr2=0x7f6c780fec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:56.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.946+0000 7f6c7d99b700 1 --2- 172.21.15.44:0/2773882700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c781035e0 0x7f6c780fec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.946+0000 7f6c7d99b700 1 -- 172.21.15.44:0/2773882700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c78105f00 msgr2=0x7f6c780ff1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:56.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.946+0000 7f6c7d99b700 1 --2- 172.21.15.44:0/2773882700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c78105f00 0x7f6c780ff1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.946+0000 7f6c7d99b700 1 -- 172.21.15.44:0/2773882700 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c740095f0 con 0x7f6c78100cc0 2023-11-16T16:49:56.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.946+0000 7f6c7d19a700 1 --2- 172.21.15.44:0/2773882700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c781035e0 0x7f6c780fec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:49:56.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.946+0000 7f6c7e19c700 1 --2- 172.21.15.44:0/2773882700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c78105f00 0x7f6c780ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:49:56.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.946+0000 7f6c7d99b700 1 --2- 172.21.15.44:0/2773882700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c78100cc0 0x7f6c78100630 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400b680 tx=0x7f6c74005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:56.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.946+0000 7f6c6effd700 1 -- 172.21.15.44:0/2773882700 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7401d070 con 0x7f6c78100cc0 2023-11-16T16:49:56.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.946+0000 7f6c7fbff700 1 -- 172.21.15.44:0/2773882700 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7810bbf0 con 0x7f6c78100cc0 2023-11-16T16:49:56.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.946+0000 7f6c7fbff700 1 -- 172.21.15.44:0/2773882700 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7810ba60 con 0x7f6c78100cc0 2023-11-16T16:49:56.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.946+0000 7f6c6effd700 1 -- 172.21.15.44:0/2773882700 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c7400fbc0 con 0x7f6c78100cc0 2023-11-16T16:49:56.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.946+0000 7f6c6effd700 1 -- 172.21.15.44:0/2773882700 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74017650 con 0x7f6c78100cc0 2023-11-16T16:49:56.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.948+0000 7f6c7fbff700 1 -- 172.21.15.44:0/2773882700 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5c005320 con 0x7f6c78100cc0 2023-11-16T16:49:56.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.949+0000 7f6c6effd700 1 -- 172.21.15.44:0/2773882700 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 22) v1 ==== 135527+0+0 (secure 0 0 0) 0x7f6c7400fd30 con 0x7f6c78100cc0 2023-11-16T16:49:56.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.950+0000 7f6c6effd700 1 --2- 172.21.15.44:0/2773882700 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6c640a0f50 0x7f6c640a3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:56.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.951+0000 7f6c6effd700 1 -- 172.21.15.44:0/2773882700 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f6c740cddd0 con 0x7f6c78100cc0 2023-11-16T16:49:56.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.951+0000 7f6c7d19a700 1 --2- 172.21.15.44:0/2773882700 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6c640a0f50 0x7f6c640a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:56.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.952+0000 7f6c7d19a700 1 --2- 172.21.15.44:0/2773882700 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6c640a0f50 0x7f6c640a3400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009cb0 tx=0x7f6c68009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:56.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:56.954+0000 7f6c6effd700 1 -- 172.21.15.44:0/2773882700 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6c7405c730 con 0x7f6c78100cc0 2023-11-16T16:49:57.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.206+0000 7f6c7fbff700 1 -- 172.21.15.44:0/2773882700 --> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c5c000c90 con 0x7f6c640a0f50 2023-11-16T16:49:57.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.261+0000 7f6c6effd700 1 -- 172.21.15.44:0/2773882700 <== mgr.25447 v2:172.21.15.44:6800/3382499657 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f6c5c000c90 con 0x7f6c640a0f50 2023-11-16T16:49:57.262 INFO:teuthology.orchestra.run.smithi044.stdout:Retention added to path / 2023-11-16T16:49:57.264 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.263+0000 7f6c7fbff700 1 -- 172.21.15.44:0/2773882700 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6c640a0f50 msgr2=0x7f6c640a3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:57.264 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.263+0000 7f6c7fbff700 1 --2- 172.21.15.44:0/2773882700 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6c640a0f50 0x7f6c640a3400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009cb0 tx=0x7f6c68009260 comp rx=0 tx=0).stop 2023-11-16T16:49:57.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.263+0000 7f6c7fbff700 1 -- 172.21.15.44:0/2773882700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c78100cc0 msgr2=0x7f6c78100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:57.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.263+0000 7f6c7fbff700 1 --2- 172.21.15.44:0/2773882700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c78100cc0 0x7f6c78100630 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400b680 tx=0x7f6c74005ef0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.263+0000 7f6c7fbff700 1 -- 172.21.15.44:0/2773882700 shutdown_connections 2023-11-16T16:49:57.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.263+0000 7f6c7fbff700 1 --2- 172.21.15.44:0/2773882700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c78105f00 0x7f6c780ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.263+0000 7f6c7fbff700 1 --2- 172.21.15.44:0/2773882700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c78100cc0 0x7f6c78100630 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.263+0000 7f6c7fbff700 1 --2- 172.21.15.44:0/2773882700 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6c640a0f50 0x7f6c640a3400 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.263+0000 7f6c7fbff700 1 --2- 172.21.15.44:0/2773882700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c781035e0 0x7f6c780fec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.263+0000 7f6c7fbff700 1 -- 172.21.15.44:0/2773882700 >> 172.21.15.44:0/2773882700 conn(0x7f6c780fa720 msgr2=0x7f6c78102120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:57.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.264+0000 7f6c7fbff700 1 -- 172.21.15.44:0/2773882700 shutdown_connections 2023-11-16T16:49:57.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.264+0000 7f6c7fbff700 1 -- 172.21.15.44:0/2773882700 wait complete. 2023-11-16T16:49:57.283 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2023-11-16T16:49:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.502+0000 7f6ab23ff700 1 -- 172.21.15.44:0/821498443 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6aac100cc0 msgr2=0x7f6aac1030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.502+0000 7f6ab23ff700 1 --2- 172.21.15.44:0/821498443 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6aac100cc0 0x7f6aac1030a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c009990 tx=0x7f6a9c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.502+0000 7f6ab23ff700 1 -- 172.21.15.44:0/821498443 shutdown_connections 2023-11-16T16:49:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.502+0000 7f6ab23ff700 1 --2- 172.21.15.44:0/821498443 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6aac105f00 0x7f6aac108360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.502+0000 7f6ab23ff700 1 --2- 172.21.15.44:0/821498443 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6aac100cc0 0x7f6aac1030a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.502+0000 7f6ab23ff700 1 --2- 172.21.15.44:0/821498443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6aac1035e0 0x7f6aac1059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.502+0000 7f6ab23ff700 1 -- 172.21.15.44:0/821498443 >> 172.21.15.44:0/821498443 conn(0x7f6aac0fa720 msgr2=0x7f6aac0fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.502+0000 7f6ab23ff700 1 -- 172.21.15.44:0/821498443 shutdown_connections 2023-11-16T16:49:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.502+0000 7f6ab23ff700 1 -- 172.21.15.44:0/821498443 wait complete. 2023-11-16T16:49:57.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.503+0000 7f6ab23ff700 1 Processor -- start 2023-11-16T16:49:57.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.504+0000 7f6ab23ff700 1 -- start start 2023-11-16T16:49:57.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.504+0000 7f6ab23ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6aac100cc0 0x7f6aac196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:57.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.504+0000 7f6ab23ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6aac1035e0 0x7f6aac196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:57.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.504+0000 7f6ab23ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6aac105f00 0x7f6aac19d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:57.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.504+0000 7f6ab23ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac19db90 con 0x7f6aac100cc0 2023-11-16T16:49:57.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.504+0000 7f6ab23ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac19dcd0 con 0x7f6aac1035e0 2023-11-16T16:49:57.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.504+0000 7f6ab23ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac19de60 con 0x7f6aac105f00 2023-11-16T16:49:57.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.504+0000 7f6aab7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6aac1035e0 0x7f6aac196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:57.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.505+0000 7f6aab7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6aac1035e0 0x7f6aac196550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:52344/0 (socket says 172.21.15.44:52344) 2023-11-16T16:49:57.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.505+0000 7f6aab7fe700 1 -- 172.21.15.44:0/2565217526 learned_addr learned my addr 172.21.15.44:0/2565217526 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:57.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.505+0000 7f6ab099c700 1 --2- 172.21.15.44:0/2565217526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6aac105f00 0x7f6aac19d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:57.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.505+0000 7f6aabfff700 1 --2- 172.21.15.44:0/2565217526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6aac100cc0 0x7f6aac196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:57.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.505+0000 7f6aab7fe700 1 -- 172.21.15.44:0/2565217526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6aac105f00 msgr2=0x7f6aac19d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:57.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.505+0000 7f6aab7fe700 1 --2- 172.21.15.44:0/2565217526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6aac105f00 0x7f6aac19d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.505+0000 7f6aab7fe700 1 -- 172.21.15.44:0/2565217526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6aac100cc0 msgr2=0x7f6aac196010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:57.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.505+0000 7f6aab7fe700 1 --2- 172.21.15.44:0/2565217526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6aac100cc0 0x7f6aac196010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.505+0000 7f6aab7fe700 1 -- 172.21.15.44:0/2565217526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9c0095f0 con 0x7f6aac1035e0 2023-11-16T16:49:57.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.505+0000 7f6aab7fe700 1 --2- 172.21.15.44:0/2565217526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6aac1035e0 0x7f6aac196550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6aa000e9a0 tx=0x7f6aa000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:57.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.505+0000 7f6aa97fa700 1 -- 172.21.15.44:0/2565217526 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0009790 con 0x7f6aac1035e0 2023-11-16T16:49:57.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.505+0000 7f6aa97fa700 1 -- 172.21.15.44:0/2565217526 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6aa0010430 con 0x7f6aac1035e0 2023-11-16T16:49:57.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.505+0000 7f6aa97fa700 1 -- 172.21.15.44:0/2565217526 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0018680 con 0x7f6aac1035e0 2023-11-16T16:49:57.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.505+0000 7f6ab23ff700 1 -- 172.21.15.44:0/2565217526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aac10bc50 con 0x7f6aac1035e0 2023-11-16T16:49:57.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.506+0000 7f6ab23ff700 1 -- 172.21.15.44:0/2565217526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aac10bac0 con 0x7f6aac1035e0 2023-11-16T16:49:57.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.507+0000 7f6ab23ff700 1 -- 172.21.15.44:0/2565217526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aac0fc290 con 0x7f6aac1035e0 2023-11-16T16:49:57.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.509+0000 7f6aa97fa700 1 -- 172.21.15.44:0/2565217526 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 22) v1 ==== 135527+0+0 (secure 0 0 0) 0x7f6aa00105a0 con 0x7f6aac1035e0 2023-11-16T16:49:57.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.510+0000 7f6aa97fa700 1 --2- 172.21.15.44:0/2565217526 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6a940a0f50 0x7f6a940a3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:57.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.510+0000 7f6aa97fa700 1 -- 172.21.15.44:0/2565217526 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f6aa0014070 con 0x7f6aac1035e0 2023-11-16T16:49:57.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.510+0000 7f6aabfff700 1 --2- 172.21.15.44:0/2565217526 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6a940a0f50 0x7f6a940a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:57.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.511+0000 7f6aabfff700 1 --2- 172.21.15.44:0/2565217526 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6a940a0f50 0x7f6a940a3400 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c00b2e0 tx=0x7f6a9c005fb0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:57.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.513+0000 7f6aa97fa700 1 -- 172.21.15.44:0/2565217526 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6aa0051730 con 0x7f6aac1035e0 2023-11-16T16:49:57.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.636+0000 7f6aa97fa700 1 -- 172.21.15.44:0/2565217526 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 23) v1 ==== 135891+0+0 (secure 0 0 0) 0x7f6aa00ced50 con 0x7f6aac1035e0 2023-11-16T16:49:57.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.767+0000 7f6ab23ff700 1 -- 172.21.15.44:0/2565217526 --> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f6aac061c10 con 0x7f6a940a0f50 2023-11-16T16:49:57.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.775+0000 7f6aa97fa700 1 -- 172.21.15.44:0/2565217526 <== mgr.25447 v2:172.21.15.44:6800/3382499657 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+283 (secure 0 0 0) 0x7f6aac061c10 con 0x7f6a940a0f50 2023-11-16T16:49:57.776 INFO:teuthology.orchestra.run.smithi044.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2023-11-16T00:00:00", "created": "2023-11-16T16:49:56", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2023-11-16T16:49:57.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.777+0000 7f6ab23ff700 1 -- 172.21.15.44:0/2565217526 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6a940a0f50 msgr2=0x7f6a940a3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:57.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.777+0000 7f6ab23ff700 1 --2- 172.21.15.44:0/2565217526 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6a940a0f50 0x7f6a940a3400 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c00b2e0 tx=0x7f6a9c005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.777+0000 7f6ab23ff700 1 -- 172.21.15.44:0/2565217526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6aac1035e0 msgr2=0x7f6aac196550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:57.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.777+0000 7f6ab23ff700 1 --2- 172.21.15.44:0/2565217526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6aac1035e0 0x7f6aac196550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6aa000e9a0 tx=0x7f6aa000ecb0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.778+0000 7f6ab23ff700 1 -- 172.21.15.44:0/2565217526 shutdown_connections 2023-11-16T16:49:57.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.778+0000 7f6ab23ff700 1 --2- 172.21.15.44:0/2565217526 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6a940a0f50 0x7f6a940a3400 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.778+0000 7f6ab23ff700 1 --2- 172.21.15.44:0/2565217526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6aac105f00 0x7f6aac19d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.778+0000 7f6ab23ff700 1 --2- 172.21.15.44:0/2565217526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6aac100cc0 0x7f6aac196010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.778+0000 7f6ab23ff700 1 --2- 172.21.15.44:0/2565217526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6aac1035e0 0x7f6aac196550 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.778+0000 7f6ab23ff700 1 -- 172.21.15.44:0/2565217526 >> 172.21.15.44:0/2565217526 conn(0x7f6aac0fa720 msgr2=0x7f6aac101920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:57.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.778+0000 7f6ab23ff700 1 -- 172.21.15.44:0/2565217526 shutdown_connections 2023-11-16T16:49:57.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:57.778+0000 7f6ab23ff700 1 -- 172.21.15.44:0/2565217526 wait complete. 2023-11-16T16:49:57.799 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2023-11-16T16:49:57.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:57 smithi176 ceph-mon[131630]: [16/Nov/2023:16:49:56] ENGINE Bus STARTING 2023-11-16T16:49:57.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:57 smithi176 ceph-mon[131630]: [16/Nov/2023:16:49:56] ENGINE Serving on http://172.21.15.44:8765 2023-11-16T16:49:57.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:57 smithi176 ceph-mon[131630]: [16/Nov/2023:16:49:56] ENGINE Serving on https://172.21.15.44:7150 2023-11-16T16:49:57.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:57 smithi176 ceph-mon[131630]: [16/Nov/2023:16:49:56] ENGINE Bus STARTED 2023-11-16T16:49:57.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:57 smithi176 ceph-mon[131630]: mgrmap e22: x(active, since 1.01911s), standbys: z, y 2023-11-16T16:49:57.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:57 smithi176 ceph-mon[131630]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:57.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:57 smithi176 ceph-mon[131630]: pgmap v3: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:49:57.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:57 smithi077 ceph-mon[131204]: [16/Nov/2023:16:49:56] ENGINE Bus STARTING 2023-11-16T16:49:57.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:57 smithi077 ceph-mon[131204]: [16/Nov/2023:16:49:56] ENGINE Serving on http://172.21.15.44:8765 2023-11-16T16:49:57.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:57 smithi077 ceph-mon[131204]: [16/Nov/2023:16:49:56] ENGINE Serving on https://172.21.15.44:7150 2023-11-16T16:49:57.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:57 smithi077 ceph-mon[131204]: [16/Nov/2023:16:49:56] ENGINE Bus STARTED 2023-11-16T16:49:57.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:57 smithi077 ceph-mon[131204]: mgrmap e22: x(active, since 1.01911s), standbys: z, y 2023-11-16T16:49:57.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:57 smithi077 ceph-mon[131204]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:57.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:57 smithi077 ceph-mon[131204]: pgmap v3: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:49:58.010 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.009+0000 7fe8a7dff700 1 -- 172.21.15.44:0/1381969267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a0101dc0 msgr2=0x7fe8a0108680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:58.010 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.009+0000 7fe8a7dff700 1 --2- 172.21.15.44:0/1381969267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a0101dc0 0x7fe8a0108680 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe89800b0a0 tx=0x7fe89800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:58.010 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.009+0000 7fe8a7dff700 1 -- 172.21.15.44:0/1381969267 shutdown_connections 2023-11-16T16:49:58.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.009+0000 7fe8a7dff700 1 --2- 172.21.15.44:0/1381969267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a0101dc0 0x7fe8a0108680 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:58.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.009+0000 7fe8a7dff700 1 --2- 172.21.15.44:0/1381969267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8a0100240 0x7fe8a0100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:58.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.009+0000 7fe8a7dff700 1 --2- 172.21.15.44:0/1381969267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a0101430 0x7fe8a0101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:58.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.009+0000 7fe8a7dff700 1 -- 172.21.15.44:0/1381969267 >> 172.21.15.44:0/1381969267 conn(0x7fe8a00fb840 msgr2=0x7fe8a00fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:58.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.010+0000 7fe8a7dff700 1 -- 172.21.15.44:0/1381969267 shutdown_connections 2023-11-16T16:49:58.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.010+0000 7fe8a7dff700 1 -- 172.21.15.44:0/1381969267 wait complete. 2023-11-16T16:49:58.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.011+0000 7fe8a7dff700 1 Processor -- start 2023-11-16T16:49:58.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.011+0000 7fe8a7dff700 1 -- start start 2023-11-16T16:49:58.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.011+0000 7fe8a7dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8a0100240 0x7fe8a00748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:58.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.011+0000 7fe8a7dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a0101430 0x7fe8a0072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:58.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.011+0000 7fe8a7dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a0101dc0 0x7fe8a0073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:58.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.011+0000 7fe8a7dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a0073bc0 con 0x7fe8a0101430 2023-11-16T16:49:58.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.011+0000 7fe8a7dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a0073d30 con 0x7fe8a0101dc0 2023-11-16T16:49:58.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.011+0000 7fe8a7dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a01a00d0 con 0x7fe8a0100240 2023-11-16T16:49:58.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.011+0000 7fe8a539a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a0101430 0x7fe8a0072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:58.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.012+0000 7fe8a5b9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8a0100240 0x7fe8a00748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:58.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.012+0000 7fe8a5b9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8a0100240 0x7fe8a00748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:58138/0 (socket says 172.21.15.44:58138) 2023-11-16T16:49:58.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.012+0000 7fe8a5b9b700 1 -- 172.21.15.44:0/1978023720 learned_addr learned my addr 172.21.15.44:0/1978023720 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:49:58.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.012+0000 7fe8a639c700 1 --2- 172.21.15.44:0/1978023720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a0101dc0 0x7fe8a0073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:58.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.012+0000 7fe8a539a700 1 -- 172.21.15.44:0/1978023720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8a0100240 msgr2=0x7fe8a00748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:58.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.012+0000 7fe8a539a700 1 --2- 172.21.15.44:0/1978023720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8a0100240 0x7fe8a00748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:58.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.012+0000 7fe8a539a700 1 -- 172.21.15.44:0/1978023720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a0101dc0 msgr2=0x7fe8a0073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:58.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.012+0000 7fe8a539a700 1 --2- 172.21.15.44:0/1978023720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a0101dc0 0x7fe8a0073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:58.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.012+0000 7fe8a539a700 1 -- 172.21.15.44:0/1978023720 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe890009520 con 0x7fe8a0101430 2023-11-16T16:49:58.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.012+0000 7fe8a5b9b700 1 --2- 172.21.15.44:0/1978023720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8a0100240 0x7fe8a00748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:58.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.012+0000 7fe8a539a700 1 --2- 172.21.15.44:0/1978023720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a0101430 0x7fe8a0072ee0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe89000e9f0 tx=0x7fe89000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:58.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.012+0000 7fe896ffd700 1 -- 172.21.15.44:0/1978023720 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89000cb70 con 0x7fe8a0101430 2023-11-16T16:49:58.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.013+0000 7fe896ffd700 1 -- 172.21.15.44:0/1978023720 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe89000ee90 con 0x7fe8a0101430 2023-11-16T16:49:58.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.013+0000 7fe8a7dff700 1 -- 172.21.15.44:0/1978023720 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe898009d00 con 0x7fe8a0101430 2023-11-16T16:49:58.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.013+0000 7fe896ffd700 1 -- 172.21.15.44:0/1978023720 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890003cf0 con 0x7fe8a0101430 2023-11-16T16:49:58.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.013+0000 7fe8a7dff700 1 -- 172.21.15.44:0/1978023720 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a010bbb0 con 0x7fe8a0101430 2023-11-16T16:49:58.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.014+0000 7fe8a7dff700 1 -- 172.21.15.44:0/1978023720 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a0067c10 con 0x7fe8a0101430 2023-11-16T16:49:58.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.016+0000 7fe896ffd700 1 -- 172.21.15.44:0/1978023720 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) v1 ==== 135891+0+0 (secure 0 0 0) 0x7fe89000ccd0 con 0x7fe8a0101430 2023-11-16T16:49:58.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.017+0000 7fe896ffd700 1 --2- 172.21.15.44:0/1978023720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe88c0a1350 0x7fe88c0a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:58.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.017+0000 7fe896ffd700 1 -- 172.21.15.44:0/1978023720 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fe890014070 con 0x7fe8a0101430 2023-11-16T16:49:58.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.017+0000 7fe8a5b9b700 1 --2- 172.21.15.44:0/1978023720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe88c0a1350 0x7fe88c0a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:58.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.018+0000 7fe8a5b9b700 1 --2- 172.21.15.44:0/1978023720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe88c0a1350 0x7fe88c0a3800 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe89c005950 tx=0x7fe89c009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:58.022 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.021+0000 7fe896ffd700 1 -- 172.21.15.44:0/1978023720 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe890096410 con 0x7fe8a0101430 2023-11-16T16:49:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:57 smithi044 ceph-mon[128750]: [16/Nov/2023:16:49:56] ENGINE Bus STARTING 2023-11-16T16:49:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:57 smithi044 ceph-mon[128750]: [16/Nov/2023:16:49:56] ENGINE Serving on http://172.21.15.44:8765 2023-11-16T16:49:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:57 smithi044 ceph-mon[128750]: [16/Nov/2023:16:49:56] ENGINE Serving on https://172.21.15.44:7150 2023-11-16T16:49:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:57 smithi044 ceph-mon[128750]: [16/Nov/2023:16:49:56] ENGINE Bus STARTED 2023-11-16T16:49:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:57 smithi044 ceph-mon[128750]: mgrmap e22: x(active, since 1.01911s), standbys: z, y 2023-11-16T16:49:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:57 smithi044 ceph-mon[128750]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:58.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:57 smithi044 ceph-mon[128750]: pgmap v3: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:49:58.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.281+0000 7fe8a7dff700 1 -- 172.21.15.44:0/1978023720 --> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fe8a01a0c40 con 0x7fe88c0a1350 2023-11-16T16:49:58.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.288+0000 7fe896ffd700 1 -- 172.21.15.44:0/1978023720 <== mgr.25447 v2:172.21.15.44:6800/3382499657 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fe8a01a0c40 con 0x7fe88c0a1350 2023-11-16T16:49:58.289 INFO:teuthology.orchestra.run.smithi044.stdout:/ 1M 6M3h 2023-11-16T16:49:58.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.291+0000 7fe8a7dff700 1 -- 172.21.15.44:0/1978023720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe88c0a1350 msgr2=0x7fe88c0a3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:58.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.291+0000 7fe8a7dff700 1 --2- 172.21.15.44:0/1978023720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe88c0a1350 0x7fe88c0a3800 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe89c005950 tx=0x7fe89c009260 comp rx=0 tx=0).stop 2023-11-16T16:49:58.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.291+0000 7fe8a7dff700 1 -- 172.21.15.44:0/1978023720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a0101430 msgr2=0x7fe8a0072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:58.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.291+0000 7fe8a7dff700 1 --2- 172.21.15.44:0/1978023720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a0101430 0x7fe8a0072ee0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe89000e9f0 tx=0x7fe89000ed00 comp rx=0 tx=0).stop 2023-11-16T16:49:58.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.292+0000 7fe8a7dff700 1 -- 172.21.15.44:0/1978023720 shutdown_connections 2023-11-16T16:49:58.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.292+0000 7fe8a7dff700 1 --2- 172.21.15.44:0/1978023720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a0101dc0 0x7fe8a0073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:58.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.292+0000 7fe8a7dff700 1 --2- 172.21.15.44:0/1978023720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe88c0a1350 0x7fe88c0a3800 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:58.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.292+0000 7fe8a7dff700 1 --2- 172.21.15.44:0/1978023720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8a0100240 0x7fe8a00748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:58.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.292+0000 7fe8a7dff700 1 --2- 172.21.15.44:0/1978023720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a0101430 0x7fe8a0072ee0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:58.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.292+0000 7fe8a7dff700 1 -- 172.21.15.44:0/1978023720 >> 172.21.15.44:0/1978023720 conn(0x7fe8a00fb840 msgr2=0x7fe8a00fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:58.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.292+0000 7fe8a7dff700 1 -- 172.21.15.44:0/1978023720 shutdown_connections 2023-11-16T16:49:58.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:49:58.292+0000 7fe8a7dff700 1 -- 172.21.15.44:0/1978023720 wait complete. 2023-11-16T16:49:58.312 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2023-11-16T16:49:58.399 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2023-11-16T16:49:58.409 INFO:teuthology.run_tasks:Running task background_exec... 2023-11-16T16:49:58.418 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2023-11-16T16:49:58.418 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2023-11-16T16:49:58.424 INFO:teuthology.run_tasks:Running task workunit... 2023-11-16T16:49:58.436 INFO:tasks.workunit:Pulling workunits from ref 5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:49:58.436 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-11-16T16:49:58.436 INFO:tasks.workunit:timeout=3h 2023-11-16T16:49:58.436 INFO:tasks.workunit:cleanup=True 2023-11-16T16:49:58.437 DEBUG:teuthology.orchestra.run.smithi044:> stat -- /home/ubuntu/cephtest/mnt.0 2023-11-16T16:49:58.487 INFO:teuthology.orchestra.run.smithi044.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-11-16T16:49:58.487 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-11-16T16:49:58.487 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 79h/121d Inode: 1099511628283 Links: 2 2023-11-16T16:49:58.487 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:49:58.487 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:cephfs_t:s0 2023-11-16T16:49:58.487 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2023-11-16 16:47:52.173762376 +0000 2023-11-16T16:49:58.487 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2023-11-16 16:47:52.173762376 +0000 2023-11-16T16:49:58.487 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2023-11-16 16:49:35.273463039 +0000 2023-11-16T16:49:58.488 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2023-11-16 16:47:52.173762376 +0000 2023-11-16T16:49:58.488 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-11-16T16:49:58.488 DEBUG:teuthology.orchestra.run.smithi044:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-11-16T16:49:58.568 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:49:58.616 INFO:tasks.workunit.client.0.smithi044.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-11-16T16:49:58.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:58 smithi044 ceph-mon[128750]: from='client.25514 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:58.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:58 smithi044 ceph-mon[128750]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:49:58.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:58 smithi044 ceph-mon[128750]: mgrmap e23: x(active, since 2s), standbys: z, y 2023-11-16T16:49:58.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:58 smithi044 ceph-mon[128750]: from='client.25520 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:58.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:58 smithi176 ceph-mon[131630]: from='client.25514 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:58.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:58 smithi176 ceph-mon[131630]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:49:58.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:58 smithi176 ceph-mon[131630]: mgrmap e23: x(active, since 2s), standbys: z, y 2023-11-16T16:49:58.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:58 smithi176 ceph-mon[131630]: from='client.25520 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:58.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:58 smithi077 ceph-mon[131204]: from='client.25514 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:58.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:58 smithi077 ceph-mon[131204]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:49:58.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:58 smithi077 ceph-mon[131204]: mgrmap e23: x(active, since 2s), standbys: z, y 2023-11-16T16:49:58.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:58 smithi077 ceph-mon[131204]: from='client.25520 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:59.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:59 smithi176 ceph-mon[131630]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:59.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:59 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:49:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:59 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:49:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:59 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:49:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:49:59 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:49:59.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:59 smithi077 ceph-mon[131204]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:59.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:59 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:49:59.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:59 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:49:59.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:59 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:49:59.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:49:59 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:59 smithi044 ceph-mon[128750]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:50:00.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:59 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:59 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:59 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:49:59 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:50:00.272+0000 7fc109365700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:50:00.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:50:00.272+0000 7fc109365700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:50:00.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:50:00.272+0000 7fc109365700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:50:00.618 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:50:00.272+0000 7fc109365700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:50:00.619 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T16:50:00.272+0000 7fc109365700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:50:00.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:00 smithi176 ceph-mon[131630]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:50:00.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:00 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:00 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:00 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:00 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:00 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:00 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:00 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:00 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:00 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:00 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:00 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:00 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:00 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:00 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:00 smithi176 ceph-mon[131630]: overall HEALTH_OK 2023-11-16T16:50:00.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:00 smithi077 ceph-mon[131204]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:50:00.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:00 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:00 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:00 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:00 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:00 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:00 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:00 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:00 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:00 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:00 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:00 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:00 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:00 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:00 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:00 smithi077 ceph-mon[131204]: overall HEALTH_OK 2023-11-16T16:50:01.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-mon[128750]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:50:01.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:01.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:01.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:01.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:01.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:01.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:01.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:01.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:01.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:01.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:01.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:01.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:01.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:01.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:01.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:00 smithi044 ceph-mon[128750]: overall HEALTH_OK 2023-11-16T16:50:01.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:01 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi176 to 196.2M 2023-11-16T16:50:01.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:01 smithi176 ceph-mon[131630]: Unable to set osd_memory_target on smithi176 to 205772492: error parsing value: Value '205772492' is below minimum 939524096 2023-11-16T16:50:01.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:01 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi077 to 196.2M 2023-11-16T16:50:01.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:01 smithi176 ceph-mon[131630]: Unable to set osd_memory_target on smithi077 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T16:50:01.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:01 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi176 to 196.2M 2023-11-16T16:50:01.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:01 smithi077 ceph-mon[131204]: Unable to set osd_memory_target on smithi176 to 205772492: error parsing value: Value '205772492' is below minimum 939524096 2023-11-16T16:50:01.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:01 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi077 to 196.2M 2023-11-16T16:50:01.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:01 smithi077 ceph-mon[131204]: Unable to set osd_memory_target on smithi077 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T16:50:02.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:01 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi176 to 196.2M 2023-11-16T16:50:02.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:01 smithi044 ceph-mon[128750]: Unable to set osd_memory_target on smithi176 to 205772492: error parsing value: Value '205772492' is below minimum 939524096 2023-11-16T16:50:02.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:01 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi077 to 196.2M 2023-11-16T16:50:02.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:01 smithi044 ceph-mon[128750]: Unable to set osd_memory_target on smithi077 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T16:50:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:50:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: mgrmap e24: x(active, since 6s), standbys: z, y 2023-11-16T16:50:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi044 to 223.0M 2023-11-16T16:50:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: Unable to set osd_memory_target on smithi044 to 233877504: error parsing value: Value '233877504' is below minimum 939524096 2023-11-16T16:50:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:50:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:50:02.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:50:02.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: Updating smithi044:/etc/ceph/ceph.conf 2023-11-16T16:50:02.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: Updating smithi077:/etc/ceph/ceph.conf 2023-11-16T16:50:02.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: Updating smithi176:/etc/ceph/ceph.conf 2023-11-16T16:50:02.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:02.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:02 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:02.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:50:02.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: mgrmap e24: x(active, since 6s), standbys: z, y 2023-11-16T16:50:02.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:02.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:02.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:02.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:02.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:02.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:02.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi044 to 223.0M 2023-11-16T16:50:02.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: Unable to set osd_memory_target on smithi044 to 233877504: error parsing value: Value '233877504' is below minimum 939524096 2023-11-16T16:50:02.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:50:02.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:50:02.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:50:02.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: Updating smithi044:/etc/ceph/ceph.conf 2023-11-16T16:50:02.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: Updating smithi077:/etc/ceph/ceph.conf 2023-11-16T16:50:02.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: Updating smithi176:/etc/ceph/ceph.conf 2023-11-16T16:50:02.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:02.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:02 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: mgrmap e24: x(active, since 6s), standbys: z, y 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi044 to 223.0M 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: Unable to set osd_memory_target on smithi044 to 233877504: error parsing value: Value '233877504' is below minimum 939524096 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: Updating smithi044:/etc/ceph/ceph.conf 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: Updating smithi077:/etc/ceph/ceph.conf 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: Updating smithi176:/etc/ceph/ceph.conf 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:02 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:03.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:03 smithi077 ceph-mon[131204]: Updating smithi176:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:03.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:03 smithi077 ceph-mon[131204]: Updating smithi077:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:03 smithi077 ceph-mon[131204]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:03 smithi077 ceph-mon[131204]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:03 smithi077 ceph-mon[131204]: Updating smithi044:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:03 smithi077 ceph-mon[131204]: Updating smithi176:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:03 smithi077 ceph-mon[131204]: Updating smithi077:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:03 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:03 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:03.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:03 smithi077 ceph-mon[131204]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:04.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:03 smithi044 ceph-mon[128750]: Updating smithi176:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:04.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:03 smithi044 ceph-mon[128750]: Updating smithi077:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:04.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:03 smithi044 ceph-mon[128750]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:04.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:03 smithi044 ceph-mon[128750]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:04.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:03 smithi044 ceph-mon[128750]: Updating smithi044:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:04.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:03 smithi044 ceph-mon[128750]: Updating smithi176:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:04.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:03 smithi044 ceph-mon[128750]: Updating smithi077:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:04.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:03 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:04.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:03 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:04.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:03 smithi044 ceph-mon[128750]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:04.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:03 smithi176 ceph-mon[131630]: Updating smithi176:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:04.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:03 smithi176 ceph-mon[131630]: Updating smithi077:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:04.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:03 smithi176 ceph-mon[131630]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:04.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:03 smithi176 ceph-mon[131630]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:04.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:03 smithi176 ceph-mon[131630]: Updating smithi044:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:04.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:03 smithi176 ceph-mon[131630]: Updating smithi176:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:04.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:03 smithi176 ceph-mon[131630]: Updating smithi077:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:04.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:03 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:04.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:03 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:04.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:03 smithi176 ceph-mon[131630]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:05.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:04 smithi044 ceph-mon[128750]: Updating smithi044:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:05.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:04 smithi044 ceph-mon[128750]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 170 KiB/s wr, 68 op/s 2023-11-16T16:50:05.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:04 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:05.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:04 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:05.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:04 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:05.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:04 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:50:05.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:04 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:50:05.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:04 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:50:05.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:04 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:05.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:04 smithi176 ceph-mon[131630]: Updating smithi044:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:05.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:04 smithi176 ceph-mon[131630]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 170 KiB/s wr, 68 op/s 2023-11-16T16:50:05.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:04 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:05.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:04 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:05.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:04 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:05.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:04 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:50:05.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:04 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:50:05.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:04 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:50:05.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:04 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:05.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:04 smithi077 ceph-mon[131204]: Updating smithi044:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:05.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:04 smithi077 ceph-mon[131204]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 170 KiB/s wr, 68 op/s 2023-11-16T16:50:05.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:04 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:05.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:04 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:05.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:04 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:05.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:04 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:50:05.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:04 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:50:05.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:04 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:50:05.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:04 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:50:06.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:05 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T16:50:06.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:05 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:50:06.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:05 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:50:06.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:05 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:50:06.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:05 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T16:50:06.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:05 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:50:06.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:05 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:50:06.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:05 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T16:50:06.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:05 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:50:06.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:05 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:50:06.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:05 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:50:06.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:05 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T16:50:06.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:05 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:50:06.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:05 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:50:06.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:05 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T16:50:06.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:05 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:50:06.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:05 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:50:06.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:05 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:50:06.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:05 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T16:50:06.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:05 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:50:06.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:05 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:50:07.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:06 smithi044 ceph-mon[128750]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 132 KiB/s wr, 53 op/s 2023-11-16T16:50:07.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:06 smithi176 ceph-mon[131630]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 132 KiB/s wr, 53 op/s 2023-11-16T16:50:07.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:06 smithi077 ceph-mon[131204]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 132 KiB/s wr, 53 op/s 2023-11-16T16:50:08.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:07 smithi044 ceph-mon[128750]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 109 KiB/s wr, 45 op/s 2023-11-16T16:50:08.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:07 smithi176 ceph-mon[131630]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 109 KiB/s wr, 45 op/s 2023-11-16T16:50:08.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:07 smithi077 ceph-mon[131204]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 109 KiB/s wr, 45 op/s 2023-11-16T16:50:10.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:10 smithi077 ceph-mon[131204]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 100 KiB/s wr, 41 op/s 2023-11-16T16:50:11.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:10 smithi044 ceph-mon[128750]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 100 KiB/s wr, 41 op/s 2023-11-16T16:50:11.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:10 smithi176 ceph-mon[131630]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 100 KiB/s wr, 41 op/s 2023-11-16T16:50:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.693+0000 7fbd297ff700 1 -- 172.21.15.44:0/529492604 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd240746a0 msgr2=0x7fbd24072d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.693+0000 7fbd297ff700 1 --2- 172.21.15.44:0/529492604 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd240746a0 0x7fbd24072d20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009990 tx=0x7fbd14009ca0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.693+0000 7fbd297ff700 1 -- 172.21.15.44:0/529492604 shutdown_connections 2023-11-16T16:50:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.693+0000 7fbd297ff700 1 --2- 172.21.15.44:0/529492604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd2410ce50 0x7fbd2410f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.693+0000 7fbd297ff700 1 --2- 172.21.15.44:0/529492604 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd240746a0 0x7fbd24072d20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.693+0000 7fbd297ff700 1 --2- 172.21.15.44:0/529492604 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd24074b00 0x7fbd24073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.693+0000 7fbd297ff700 1 -- 172.21.15.44:0/529492604 >> 172.21.15.44:0/529492604 conn(0x7fbd24077550 msgr2=0x7fbd24077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.693+0000 7fbd297ff700 1 -- 172.21.15.44:0/529492604 shutdown_connections 2023-11-16T16:50:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.693+0000 7fbd297ff700 1 -- 172.21.15.44:0/529492604 wait complete. 2023-11-16T16:50:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.694+0000 7fbd297ff700 1 Processor -- start 2023-11-16T16:50:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.695+0000 7fbd297ff700 1 -- start start 2023-11-16T16:50:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.695+0000 7fbd297ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd240746a0 0x7fbd241985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.695+0000 7fbd297ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd24074b00 0x7fbd24198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.695+0000 7fbd297ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd2410ce50 0x7fbd2419cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.695+0000 7fbd297ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2419d5c0 con 0x7fbd24074b00 2023-11-16T16:50:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.695+0000 7fbd297ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2419d730 con 0x7fbd2410ce50 2023-11-16T16:50:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.695+0000 7fbd297ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2419d8f0 con 0x7fbd240746a0 2023-11-16T16:50:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.695+0000 7fbd227fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd24074b00 0x7fbd24198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.695+0000 7fbd22ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd240746a0 0x7fbd241985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.695+0000 7fbd237fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd2410ce50 0x7fbd2419cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.695+0000 7fbd227fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd24074b00 0x7fbd24198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40438/0 (socket says 172.21.15.44:40438) 2023-11-16T16:50:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.695+0000 7fbd237fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd2410ce50 0x7fbd2419cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:37548/0 (socket says 172.21.15.44:37548) 2023-11-16T16:50:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.695+0000 7fbd227fc700 1 -- 172.21.15.44:0/2291005626 learned_addr learned my addr 172.21.15.44:0/2291005626 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.695+0000 7fbd22ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd240746a0 0x7fbd241985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:37932/0 (socket says 172.21.15.44:37932) 2023-11-16T16:50:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.696+0000 7fbd227fc700 1 -- 172.21.15.44:0/2291005626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd240746a0 msgr2=0x7fbd241985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.696+0000 7fbd227fc700 1 --2- 172.21.15.44:0/2291005626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd240746a0 0x7fbd241985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.696+0000 7fbd227fc700 1 -- 172.21.15.44:0/2291005626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd2410ce50 msgr2=0x7fbd2419cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.696+0000 7fbd227fc700 1 --2- 172.21.15.44:0/2291005626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd2410ce50 0x7fbd2419cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.696+0000 7fbd227fc700 1 -- 172.21.15.44:0/2291005626 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd18009520 con 0x7fbd24074b00 2023-11-16T16:50:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.696+0000 7fbd237fe700 1 --2- 172.21.15.44:0/2291005626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd2410ce50 0x7fbd2419cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.696+0000 7fbd227fc700 1 --2- 172.21.15.44:0/2291005626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd24074b00 0x7fbd24198b20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800eb30 tx=0x7fbd1800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.696+0000 7fbd0bfff700 1 -- 172.21.15.44:0/2291005626 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd18009880 con 0x7fbd24074b00 2023-11-16T16:50:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.696+0000 7fbd0bfff700 1 -- 172.21.15.44:0/2291005626 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd180044e0 con 0x7fbd24074b00 2023-11-16T16:50:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.696+0000 7fbd0bfff700 1 -- 172.21.15.44:0/2291005626 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd18003d80 con 0x7fbd24074b00 2023-11-16T16:50:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.696+0000 7fbd297ff700 1 -- 172.21.15.44:0/2291005626 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd140095f0 con 0x7fbd24074b00 2023-11-16T16:50:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.697+0000 7fbd297ff700 1 -- 172.21.15.44:0/2291005626 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd24112590 con 0x7fbd24074b00 2023-11-16T16:50:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.700+0000 7fbd0bfff700 1 -- 172.21.15.44:0/2291005626 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7fbd18003f20 con 0x7fbd24074b00 2023-11-16T16:50:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.701+0000 7fbd3d167b80 1 Processor -- start 2023-11-16T16:50:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.701+0000 7fbd0bfff700 1 --2- 172.21.15.44:0/2291005626 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbd0c0a1460 0x7fbd0c0a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.702+0000 7fbd3d167b80 1 -- start start 2023-11-16T16:50:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.702+0000 7fbd0bfff700 1 -- 172.21.15.44:0/2291005626 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fbd18014070 con 0x7fbd24074b00 2023-11-16T16:50:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.702+0000 7fbd237fe700 1 --2- 172.21.15.44:0/2291005626 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbd0c0a1460 0x7fbd0c0a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.702+0000 7fbd3d167b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0b636b0 0x5609d0d0d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.702+0000 7fbd3d167b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0d720 0x5609d0d07560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.702+0000 7fbd3d167b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0d0db90 0x5609d0d07b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.702+0000 7fbd3d167b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609d0c0d000 con 0x5609d0d0d720 2023-11-16T16:50:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.702+0000 7fbd3d167b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609d0c5f840 con 0x5609d0d0db90 2023-11-16T16:50:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.702+0000 7fbd3d167b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609d0c58cd0 con 0x5609d0b636b0 2023-11-16T16:50:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.702+0000 7fbd227fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0d720 0x5609d0d07560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.702+0000 7fbd22ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0b636b0 0x5609d0d0d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.702+0000 7fbd227fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0d720 0x5609d0d07560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40448/0 (socket says 172.21.15.44:40448) 2023-11-16T16:50:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.702+0000 7fbd227fc700 1 -- 172.21.15.44:0/982705895 learned_addr learned my addr 172.21.15.44:0/982705895 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.703+0000 7fbd237fe700 1 --2- 172.21.15.44:0/982705895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0d0db90 0x5609d0d07b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.703+0000 7fbd237fe700 1 --2- 172.21.15.44:0/2291005626 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbd0c0a1460 0x7fbd0c0a3910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbd24069fd0 tx=0x7fbd1c00b040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.703+0000 7fbd22ffd700 1 -- 172.21.15.44:0/982705895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0d0db90 msgr2=0x5609d0d07b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.703+0000 7fbd22ffd700 1 --2- 172.21.15.44:0/982705895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0d0db90 0x5609d0d07b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.703+0000 7fbd22ffd700 1 -- 172.21.15.44:0/982705895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0d720 msgr2=0x5609d0d07560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.703+0000 7fbd22ffd700 1 --2- 172.21.15.44:0/982705895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0d720 0x5609d0d07560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.703+0000 7fbd22ffd700 1 -- 172.21.15.44:0/982705895 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609d0ada5d0 con 0x5609d0b636b0 2023-11-16T16:50:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.703+0000 7fbd237fe700 1 --2- 172.21.15.44:0/982705895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0d0db90 0x5609d0d07b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.703+0000 7fbd227fc700 1 --2- 172.21.15.44:0/982705895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0d720 0x5609d0d07560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.703+0000 7fbd22ffd700 1 --2- 172.21.15.44:0/982705895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0b636b0 0x5609d0d0d1e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbd14000c00 tx=0x7fbd14005ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afab90cd92b3c549 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.703+0000 7fbd097fa700 1 -- 172.21.15.44:0/982705895 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1401d070 con 0x5609d0b636b0 2023-11-16T16:50:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.704+0000 7fbd097fa700 1 -- 172.21.15.44:0/982705895 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd1401e040 con 0x5609d0b636b0 2023-11-16T16:50:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.704+0000 7fbd097fa700 1 -- 172.21.15.44:0/982705895 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd14004740 con 0x5609d0b636b0 2023-11-16T16:50:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.704+0000 7fbd3d167b80 1 -- 172.21.15.44:0/982705895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0b636b0 msgr2=0x5609d0d0d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.704+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/982705895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0b636b0 0x5609d0d0d1e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbd14000c00 tx=0x7fbd14005ea0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.704+0000 7fbd3d167b80 1 -- 172.21.15.44:0/982705895 shutdown_connections 2023-11-16T16:50:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.704+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/982705895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0d0db90 0x5609d0d07b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.704+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/982705895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0b636b0 0x5609d0d0d1e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.704+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/982705895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0d720 0x5609d0d07560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.704+0000 7fbd3d167b80 1 -- 172.21.15.44:0/982705895 >> 172.21.15.44:0/982705895 conn(0x5609d0cfcf50 msgr2=0x5609d0d02270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.704+0000 7fbd3d167b80 1 -- 172.21.15.44:0/982705895 shutdown_connections 2023-11-16T16:50:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.705+0000 7fbd3d167b80 1 -- 172.21.15.44:0/982705895 wait complete. 2023-11-16T16:50:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.705+0000 7fbd3d167b80 1 -- start start 2023-11-16T16:50:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.705+0000 7fbd3d167b80 1 Processor -- start 2023-11-16T16:50:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd3d167b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0b636b0 0x5609d0d139c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd3d167b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0d720 0x5609d0d10c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd3d167b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0d0db90 0x5609d0d11220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd3d167b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609d0bfa880 con 0x5609d0d0d720 2023-11-16T16:50:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd3d167b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609d0bfa520 con 0x5609d0d0db90 2023-11-16T16:50:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd3d167b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609d0adc8c0 con 0x5609d0b636b0 2023-11-16T16:50:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd3d167b80 10 client.0 _ll_register_callbacks cb 0x5609d0c18720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd227fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0d720 0x5609d0d10c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd227fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0d720 0x5609d0d10c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40458/0 (socket says 172.21.15.44:40458) 2023-11-16T16:50:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd227fc700 1 -- 172.21.15.44:0/1795208252 learned_addr learned my addr 172.21.15.44:0/1795208252 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd22ffd700 1 --2- 172.21.15.44:0/1795208252 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0b636b0 0x5609d0d139c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd237fe700 1 --2- 172.21.15.44:0/1795208252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0d0db90 0x5609d0d11220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd227fc700 1 -- 172.21.15.44:0/1795208252 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0b636b0 msgr2=0x5609d0d139c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd227fc700 1 --2- 172.21.15.44:0/1795208252 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0b636b0 0x5609d0d139c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd227fc700 1 -- 172.21.15.44:0/1795208252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0d0db90 msgr2=0x5609d0d11220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd227fc700 1 --2- 172.21.15.44:0/1795208252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0d0db90 0x5609d0d11220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd227fc700 1 -- 172.21.15.44:0/1795208252 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd2410fb60 con 0x5609d0d0d720 2023-11-16T16:50:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.706+0000 7fbd22ffd700 1 --2- 172.21.15.44:0/1795208252 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0b636b0 0x5609d0d139c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.707+0000 7fbd227fc700 1 --2- 172.21.15.44:0/1795208252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0d720 0x5609d0d10c40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fbd1809a030 tx=0x7fbd1801e3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1717faf9725c91f6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.707+0000 7fbd06ff5700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:50:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.707+0000 7fbd06ff5700 1 -- 172.21.15.44:0/1795208252 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1809e160 con 0x5609d0d0d720 2023-11-16T16:50:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.707+0000 7fbd3d167b80 1 -- 172.21.15.44:0/1795208252 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5609d0cfa710 con 0x5609d0d0d720 2023-11-16T16:50:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.707+0000 7fbd06ff5700 1 -- 172.21.15.44:0/1795208252 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fbd1809a330 con 0x5609d0d0d720 2023-11-16T16:50:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.707+0000 7fbd06ff5700 1 -- 172.21.15.44:0/1795208252 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd1809e360 con 0x5609d0d0d720 2023-11-16T16:50:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.707+0000 7fbd3d167b80 10 client.15525 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.707+0000 7fbd3d167b80 1 -- 172.21.15.44:0/1795208252 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5609d0c1f130 con 0x5609d0d0d720 2023-11-16T16:50:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.707+0000 7fbd06ff5700 1 -- 172.21.15.44:0/1795208252 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5609d0cfa710 con 0x5609d0d0d720 2023-11-16T16:50:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.708+0000 7fbd06ff5700 1 -- 172.21.15.44:0/1795208252 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbd180a22f0 con 0x5609d0d0d720 2023-11-16T16:50:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.708+0000 7fbd3d167b80 10 client.15525 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.708+0000 7fbd3d167b80 10 client.15525 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.708+0000 7fbd3d167b80 20 client.15525 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.708+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/1795208252 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5609d0d20ee0 0x5609d0d232f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.708+0000 7fbd3d167b80 4 client.15525 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.708+0000 7fbd3d167b80 1 -- 172.21.15.44:0/1795208252 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5609d0c1b040 con 0x5609d0d20ee0 2023-11-16T16:50:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.708+0000 7fbd22ffd700 1 --2- 172.21.15.44:0/1795208252 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5609d0d20ee0 0x5609d0d232f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.709+0000 7fbd06ff5700 10 client.15525 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.709+0000 7fbd22ffd700 1 --2- 172.21.15.44:0/1795208252 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5609d0d20ee0 0x5609d0d232f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15562d59441e3fe9 server_cookie=653eef4c78a701b6 in_seq=0 out_seq=0 2023-11-16T16:50:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.709+0000 7fbd06ff5700 1 -- 172.21.15.44:0/1795208252 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5609d0c1b040 con 0x5609d0d20ee0 2023-11-16T16:50:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.710+0000 7fbd06ff5700 10 client.15525 handle_command_reply: tid=0 2023-11-16T16:50:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.710+0000 7fbd3d167b80 1 client.15525 shutdown 2023-11-16T16:50:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.710+0000 7fbd3d167b80 1 -- 172.21.15.44:0/1795208252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0d720 msgr2=0x5609d0d10c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.710+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/1795208252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0d720 0x5609d0d10c40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fbd1809a030 tx=0x7fbd1801e3d0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.710+0000 7fbd3d167b80 1 -- 172.21.15.44:0/1795208252 shutdown_connections 2023-11-16T16:50:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.710+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/1795208252 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5609d0d20ee0 0x5609d0d232f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.710+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/1795208252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0d0db90 0x5609d0d11220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.710+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/1795208252 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0b636b0 0x5609d0d139c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.710+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/1795208252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0d720 0x5609d0d10c40 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.710+0000 7fbd06ff5700 0 client.15525 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.710+0000 7fbd3d167b80 1 -- 172.21.15.44:0/1795208252 >> 172.21.15.44:0/1795208252 conn(0x5609d0cfcf50 msgr2=0x5609d0d017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.710+0000 7fbd3d167b80 1 -- 172.21.15.44:0/1795208252 shutdown_connections 2023-11-16T16:50:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.710+0000 7fbd3d167b80 1 -- 172.21.15.44:0/1795208252 wait complete. 2023-11-16T16:50:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.710+0000 7fbd3d167b80 20 client.15525 trim_cache size 0 max 16384 2023-11-16T16:50:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.726+0000 7fbd3d167b80 1 Processor -- start 2023-11-16T16:50:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.726+0000 7fbd3d167b80 1 -- start start 2023-11-16T16:50:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.726+0000 7fbd3d167b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0b636b0 0x5609d0d17fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.726+0000 7fbd3d167b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0d0d720 0x5609d0d13560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.726+0000 7fbd3d167b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0db90 0x5609d0d01f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.726+0000 7fbd3d167b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609d0d0f220 con 0x5609d0d0db90 2023-11-16T16:50:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.726+0000 7fbd3d167b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609d0d0b970 con 0x5609d0b636b0 2023-11-16T16:50:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.726+0000 7fbd3d167b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609d0d02f40 con 0x5609d0d0d720 2023-11-16T16:50:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.727+0000 7fbd237fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0db90 0x5609d0d01f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.727+0000 7fbd22ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0b636b0 0x5609d0d17fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.727+0000 7fbd237fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0db90 0x5609d0d01f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40474/0 (socket says 172.21.15.44:40474) 2023-11-16T16:50:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.727+0000 7fbd237fe700 1 -- 172.21.15.44:0/2369939722 learned_addr learned my addr 172.21.15.44:0/2369939722 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.727+0000 7fbd227fc700 1 --2- 172.21.15.44:0/2369939722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0d0d720 0x5609d0d13560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.727+0000 7fbd22ffd700 1 -- 172.21.15.44:0/2369939722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0d0d720 msgr2=0x5609d0d13560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.727+0000 7fbd22ffd700 1 --2- 172.21.15.44:0/2369939722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0d0d720 0x5609d0d13560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.727+0000 7fbd22ffd700 1 -- 172.21.15.44:0/2369939722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0db90 msgr2=0x5609d0d01f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.727+0000 7fbd22ffd700 1 --2- 172.21.15.44:0/2369939722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0db90 0x5609d0d01f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.727+0000 7fbd22ffd700 1 -- 172.21.15.44:0/2369939722 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609d0c1ee80 con 0x5609d0b636b0 2023-11-16T16:50:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.727+0000 7fbd22ffd700 1 --2- 172.21.15.44:0/2369939722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0b636b0 0x5609d0d17fe0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbd1400b2e0 tx=0x7fbd14005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7aba10ba305a0b8b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.727+0000 7fbd09ffb700 1 -- 172.21.15.44:0/2369939722 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1401d040 con 0x5609d0b636b0 2023-11-16T16:50:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.728+0000 7fbd09ffb700 1 -- 172.21.15.44:0/2369939722 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd1400bed0 con 0x5609d0b636b0 2023-11-16T16:50:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.728+0000 7fbd09ffb700 1 -- 172.21.15.44:0/2369939722 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd14033b30 con 0x5609d0b636b0 2023-11-16T16:50:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.728+0000 7fbd3d167b80 1 -- 172.21.15.44:0/2369939722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0b636b0 msgr2=0x5609d0d17fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.728+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/2369939722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0b636b0 0x5609d0d17fe0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbd1400b2e0 tx=0x7fbd14005fd0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.728+0000 7fbd3d167b80 1 -- 172.21.15.44:0/2369939722 shutdown_connections 2023-11-16T16:50:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.728+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/2369939722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0d0db90 0x5609d0d01f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.728+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/2369939722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0b636b0 0x5609d0d17fe0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.728+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/2369939722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0d0d720 0x5609d0d13560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.728+0000 7fbd3d167b80 1 -- 172.21.15.44:0/2369939722 >> 172.21.15.44:0/2369939722 conn(0x5609d0cfcf50 msgr2=0x5609d0d0ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.728+0000 7fbd3d167b80 1 -- 172.21.15.44:0/2369939722 shutdown_connections 2023-11-16T16:50:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.728+0000 7fbd3d167b80 1 -- 172.21.15.44:0/2369939722 wait complete. 2023-11-16T16:50:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.729+0000 7fbd3d167b80 1 -- start start 2023-11-16T16:50:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.729+0000 7fbd3d167b80 1 Processor -- start 2023-11-16T16:50:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.729+0000 7fbd3d167b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0b636b0 0x5609d0d08c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.729+0000 7fbd3d167b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0d0d720 0x5609d0d091a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.729+0000 7fbd3d167b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0d0db90 0x5609d0d096e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.729+0000 7fbd3d167b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609d0d09c20 con 0x5609d0b636b0 2023-11-16T16:50:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.729+0000 7fbd3d167b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609d0d09d90 con 0x5609d0d0d720 2023-11-16T16:50:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.729+0000 7fbd3d167b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609d0d09f50 con 0x5609d0d0db90 2023-11-16T16:50:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.729+0000 7fbd3d167b80 10 client.0 _ll_register_callbacks cb 0x5609d0c18720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.730+0000 7fbd22ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0b636b0 0x5609d0d08c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.730+0000 7fbd22ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0b636b0 0x5609d0d08c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40486/0 (socket says 172.21.15.44:40486) 2023-11-16T16:50:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.730+0000 7fbd22ffd700 1 -- 172.21.15.44:0/2056863082 learned_addr learned my addr 172.21.15.44:0/2056863082 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.730+0000 7fbd227fc700 1 --2- 172.21.15.44:0/2056863082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0d0d720 0x5609d0d091a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.730+0000 7fbd237fe700 1 --2- 172.21.15.44:0/2056863082 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0d0db90 0x5609d0d096e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.730+0000 7fbd22ffd700 1 -- 172.21.15.44:0/2056863082 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0d0db90 msgr2=0x5609d0d096e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.730+0000 7fbd22ffd700 1 --2- 172.21.15.44:0/2056863082 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0d0db90 0x5609d0d096e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.730+0000 7fbd22ffd700 1 -- 172.21.15.44:0/2056863082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0d0d720 msgr2=0x5609d0d091a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.730+0000 7fbd22ffd700 1 --2- 172.21.15.44:0/2056863082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0d0d720 0x5609d0d091a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.730+0000 7fbd22ffd700 1 -- 172.21.15.44:0/2056863082 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5609d0ada5d0 con 0x5609d0b636b0 2023-11-16T16:50:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.730+0000 7fbd22ffd700 1 --2- 172.21.15.44:0/2056863082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0b636b0 0x5609d0d08c60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009ce0 tx=0x7fbd14005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1994de3b92c10b97 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.730+0000 7fbd06ff5700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:50:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.731+0000 7fbd06ff5700 1 -- 172.21.15.44:0/2056863082 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd14005250 con 0x5609d0b636b0 2023-11-16T16:50:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.731+0000 7fbd06ff5700 1 -- 172.21.15.44:0/2056863082 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fbd140053b0 con 0x5609d0b636b0 2023-11-16T16:50:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.731+0000 7fbd3d167b80 1 -- 172.21.15.44:0/2056863082 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd1401e040 con 0x5609d0b636b0 2023-11-16T16:50:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.731+0000 7fbd06ff5700 1 -- 172.21.15.44:0/2056863082 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd14005570 con 0x5609d0b636b0 2023-11-16T16:50:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.731+0000 7fbd3d167b80 10 client.15531 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.731+0000 7fbd3d167b80 1 -- 172.21.15.44:0/2056863082 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5609d0a6fc00 con 0x5609d0b636b0 2023-11-16T16:50:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.731+0000 7fbd06ff5700 1 -- 172.21.15.44:0/2056863082 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd14046080 con 0x5609d0b636b0 2023-11-16T16:50:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.731+0000 7fbd06ff5700 1 -- 172.21.15.44:0/2056863082 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbd140048a0 con 0x5609d0b636b0 2023-11-16T16:50:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.731+0000 7fbd3d167b80 10 client.15531 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.731+0000 7fbd3d167b80 10 client.15531 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.731+0000 7fbd3d167b80 20 client.15531 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.731+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/2056863082 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5609d0d752f0 0x5609d0d776d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.731+0000 7fbd3d167b80 4 client.15531 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:50:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.731+0000 7fbd3d167b80 1 -- 172.21.15.44:0/2056863082 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5609d0c27600 con 0x5609d0d752f0 2023-11-16T16:50:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.732+0000 7fbd227fc700 1 --2- 172.21.15.44:0/2056863082 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5609d0d752f0 0x5609d0d776d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.732+0000 7fbd227fc700 1 --2- 172.21.15.44:0/2056863082 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5609d0d752f0 0x5609d0d776d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79589217e65e47a7 server_cookie=991b6a20493db77d in_seq=0 out_seq=0 2023-11-16T16:50:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.732+0000 7fbd06ff5700 10 client.15531 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.745+0000 7fbd06ff5700 1 -- 172.21.15.44:0/2056863082 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5609d0c27600 con 0x5609d0d752f0 2023-11-16T16:50:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.745+0000 7fbd06ff5700 10 client.15531 handle_command_reply: tid=0 2023-11-16T16:50:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.745+0000 7fbd3d167b80 1 client.15531 shutdown 2023-11-16T16:50:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.746+0000 7fbd3d167b80 1 -- 172.21.15.44:0/2056863082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0b636b0 msgr2=0x5609d0d08c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.746+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/2056863082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0b636b0 0x5609d0d08c60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009ce0 tx=0x7fbd14005970 comp rx=0 tx=0).stop 2023-11-16T16:50:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.746+0000 7fbd3d167b80 1 -- 172.21.15.44:0/2056863082 shutdown_connections 2023-11-16T16:50:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.746+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/2056863082 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5609d0d752f0 0x5609d0d776d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.746+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/2056863082 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5609d0d0db90 0x5609d0d096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.746+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/2056863082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609d0b636b0 0x5609d0d08c60 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.746+0000 7fbd06ff5700 0 client.15531 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.746+0000 7fbd3d167b80 1 --2- 172.21.15.44:0/2056863082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d0d0d720 0x5609d0d091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.746+0000 7fbd3d167b80 1 -- 172.21.15.44:0/2056863082 >> 172.21.15.44:0/2056863082 conn(0x5609d0cfcf50 msgr2=0x5609d0d0df60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:11.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.746+0000 7fbd3d167b80 1 -- 172.21.15.44:0/2056863082 shutdown_connections 2023-11-16T16:50:11.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.746+0000 7fbd3d167b80 1 -- 172.21.15.44:0/2056863082 wait complete. 2023-11-16T16:50:11.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.746+0000 7fbd3d167b80 20 client.15531 trim_cache size 0 max 16384 2023-11-16T16:50:11.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:50:11.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:50:11.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:50:11.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:50:11.750 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.747+0000 7fbd297ff700 1 -- 172.21.15.44:0/2291005626 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbd0c0a1460 msgr2=0x7fbd0c0a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:11.750 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.747+0000 7fbd297ff700 1 --2- 172.21.15.44:0/2291005626 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbd0c0a1460 0x7fbd0c0a3910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbd24069fd0 tx=0x7fbd1c00b040 comp rx=0 tx=0).stop 2023-11-16T16:50:11.750 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.747+0000 7fbd297ff700 1 -- 172.21.15.44:0/2291005626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd24074b00 msgr2=0x7fbd24198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:11.750 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.747+0000 7fbd297ff700 1 --2- 172.21.15.44:0/2291005626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd24074b00 0x7fbd24198b20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800eb30 tx=0x7fbd1800ee40 comp rx=0 tx=0).stop 2023-11-16T16:50:11.750 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.747+0000 7fbd297ff700 1 -- 172.21.15.44:0/2291005626 shutdown_connections 2023-11-16T16:50:11.750 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.747+0000 7fbd297ff700 1 --2- 172.21.15.44:0/2291005626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd2410ce50 0x7fbd2419cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.747+0000 7fbd297ff700 1 --2- 172.21.15.44:0/2291005626 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbd0c0a1460 0x7fbd0c0a3910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.747+0000 7fbd297ff700 1 --2- 172.21.15.44:0/2291005626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd240746a0 0x7fbd241985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.747+0000 7fbd297ff700 1 --2- 172.21.15.44:0/2291005626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd24074b00 0x7fbd24198b20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.747+0000 7fbd297ff700 1 -- 172.21.15.44:0/2291005626 >> 172.21.15.44:0/2291005626 conn(0x7fbd24077550 msgr2=0x7fbd24104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.747+0000 7fbd297ff700 1 -- 172.21.15.44:0/2291005626 shutdown_connections 2023-11-16T16:50:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:11.747+0000 7fbd297ff700 1 -- 172.21.15.44:0/2291005626 wait complete. 2023-11-16T16:50:12.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:12 smithi077 ceph-mon[131204]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 100 KiB/s wr, 41 op/s 2023-11-16T16:50:13.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:12 smithi044 ceph-mon[128750]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 100 KiB/s wr, 41 op/s 2023-11-16T16:50:13.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:12 smithi176 ceph-mon[131630]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 100 KiB/s wr, 41 op/s 2023-11-16T16:50:14.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:14 smithi077 ceph-mon[131204]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 100 KiB/s wr, 42 op/s 2023-11-16T16:50:14.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:14 smithi044 ceph-mon[128750]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 100 KiB/s wr, 42 op/s 2023-11-16T16:50:14.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:14 smithi176 ceph-mon[131630]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 100 KiB/s wr, 42 op/s 2023-11-16T16:50:16.253 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:50:16.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.472+0000 7f9e543ff700 1 -- 172.21.15.44:0/3085568473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e4c101660 msgr2=0x7f9e4c104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:16.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.472+0000 7f9e543ff700 1 --2- 172.21.15.44:0/3085568473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e4c101660 0x7f9e4c104ef0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c0098f0 tx=0x7f9e3c009c00 comp rx=0 tx=0).stop 2023-11-16T16:50:16.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.472+0000 7f9e543ff700 1 -- 172.21.15.44:0/3085568473 shutdown_connections 2023-11-16T16:50:16.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.472+0000 7f9e543ff700 1 --2- 172.21.15.44:0/3085568473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e4c105620 0x7f9e4c10d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.472+0000 7f9e543ff700 1 --2- 172.21.15.44:0/3085568473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e4c100cc0 0x7f9e4c101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.472+0000 7f9e543ff700 1 --2- 172.21.15.44:0/3085568473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e4c101660 0x7f9e4c104ef0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.472+0000 7f9e543ff700 1 -- 172.21.15.44:0/3085568473 >> 172.21.15.44:0/3085568473 conn(0x7f9e4c077560 msgr2=0x7f9e4c077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.472+0000 7f9e543ff700 1 -- 172.21.15.44:0/3085568473 shutdown_connections 2023-11-16T16:50:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.472+0000 7f9e543ff700 1 -- 172.21.15.44:0/3085568473 wait complete. 2023-11-16T16:50:16.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.473+0000 7f9e543ff700 1 Processor -- start 2023-11-16T16:50:16.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.473+0000 7f9e543ff700 1 -- start start 2023-11-16T16:50:16.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e543ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e4c100cc0 0x7f9e4c196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e543ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e4c101660 0x7f9e4c196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e543ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e4c105620 0x7f9e4c19aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e543ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c19b510 con 0x7f9e4c105620 2023-11-16T16:50:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e543ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c19b680 con 0x7f9e4c100cc0 2023-11-16T16:50:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e543ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c19b840 con 0x7f9e4c101660 2023-11-16T16:50:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e5219b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e4c100cc0 0x7f9e4c196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e5219b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e4c100cc0 0x7f9e4c196530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:36260/0 (socket says 172.21.15.44:36260) 2023-11-16T16:50:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e5219b700 1 -- 172.21.15.44:0/3618061208 learned_addr learned my addr 172.21.15.44:0/3618061208 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e5199a700 1 --2- 172.21.15.44:0/3618061208 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e4c101660 0x7f9e4c196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e5299c700 1 --2- 172.21.15.44:0/3618061208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e4c105620 0x7f9e4c19aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e5219b700 1 -- 172.21.15.44:0/3618061208 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e4c101660 msgr2=0x7f9e4c196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e5219b700 1 --2- 172.21.15.44:0/3618061208 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e4c101660 0x7f9e4c196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e5219b700 1 -- 172.21.15.44:0/3618061208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e4c105620 msgr2=0x7f9e4c19aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:16.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e5219b700 1 --2- 172.21.15.44:0/3618061208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e4c105620 0x7f9e4c19aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e5219b700 1 -- 172.21.15.44:0/3618061208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3c009520 con 0x7f9e4c100cc0 2023-11-16T16:50:16.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.474+0000 7f9e5299c700 1 --2- 172.21.15.44:0/3618061208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e4c105620 0x7f9e4c19aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:16.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.475+0000 7f9e5219b700 1 --2- 172.21.15.44:0/3618061208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e4c100cc0 0x7f9e4c196530 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9e4800e9c0 tx=0x7f9e4800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:16.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.475+0000 7f9e437fe700 1 -- 172.21.15.44:0/3618061208 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4800cbd0 con 0x7f9e4c100cc0 2023-11-16T16:50:16.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.475+0000 7f9e543ff700 1 -- 172.21.15.44:0/3618061208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4c110370 con 0x7f9e4c100cc0 2023-11-16T16:50:16.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.475+0000 7f9e437fe700 1 -- 172.21.15.44:0/3618061208 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e4800eeb0 con 0x7f9e4c100cc0 2023-11-16T16:50:16.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.475+0000 7f9e437fe700 1 -- 172.21.15.44:0/3618061208 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48005580 con 0x7f9e4c100cc0 2023-11-16T16:50:16.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.475+0000 7f9e543ff700 1 -- 172.21.15.44:0/3618061208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e4c1101e0 con 0x7f9e4c100cc0 2023-11-16T16:50:16.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.478+0000 7f9e437fe700 1 -- 172.21.15.44:0/3618061208 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f9e48004750 con 0x7f9e4c100cc0 2023-11-16T16:50:16.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.480+0000 7f9e67c5cb80 1 Processor -- start 2023-11-16T16:50:16.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.480+0000 7f9e67c5cb80 1 -- start start 2023-11-16T16:50:16.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e67c5cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb3368df70 0x55eb3368e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e67c5cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368e880 0x55eb33692780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e67c5cb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb33692cc0 0x55eb33693110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e67c5cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb3358f060 con 0x55eb3368e880 2023-11-16T16:50:16.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e67c5cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb335e18a0 con 0x55eb3368df70 2023-11-16T16:50:16.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e67c5cb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb335dad30 con 0x55eb33692cc0 2023-11-16T16:50:16.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e437fe700 1 --2- 172.21.15.44:0/3618061208 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9e380a13e0 0x7f9e380a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e5219b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368e880 0x55eb33692780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e5299c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb3368df70 0x55eb3368e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e5219b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368e880 0x55eb33692780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47706/0 (socket says 172.21.15.44:47706) 2023-11-16T16:50:16.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e5219b700 1 -- 172.21.15.44:0/422440514 learned_addr learned my addr 172.21.15.44:0/422440514 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:16.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e437fe700 1 -- 172.21.15.44:0/3618061208 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f9e48014070 con 0x7f9e4c100cc0 2023-11-16T16:50:16.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e5199a700 1 --2- 172.21.15.44:0/422440514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb33692cc0 0x55eb33693110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e5299c700 1 --2- 172.21.15.44:0/3618061208 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9e380a13e0 0x7f9e380a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e5219b700 1 -- 172.21.15.44:0/422440514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb33692cc0 msgr2=0x55eb33693110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:16.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e5219b700 1 --2- 172.21.15.44:0/422440514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb33692cc0 0x55eb33693110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e5219b700 1 -- 172.21.15.44:0/422440514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb3368df70 msgr2=0x55eb3368e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:16.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e5219b700 1 --2- 172.21.15.44:0/422440514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb3368df70 0x55eb3368e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e5219b700 1 -- 172.21.15.44:0/422440514 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb3345c630 con 0x55eb3368e880 2023-11-16T16:50:16.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e5199a700 1 --2- 172.21.15.44:0/422440514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb33692cc0 0x55eb33693110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:16.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.481+0000 7f9e5299c700 1 --2- 172.21.15.44:0/422440514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb3368df70 0x55eb3368e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:16.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.482+0000 7f9e5219b700 1 --2- 172.21.15.44:0/422440514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368e880 0x55eb33692780 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9e4809a1c0 tx=0x7f9e4806c2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fffc1df9023b2f56 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:16.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.482+0000 7f9e5299c700 1 --2- 172.21.15.44:0/3618061208 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9e380a13e0 0x7f9e380a3890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9e44012b90 tx=0x7f9e4400b040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:16.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.482+0000 7f9e40bfe700 1 -- 172.21.15.44:0/422440514 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4809a230 con 0x55eb3368e880 2023-11-16T16:50:16.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.482+0000 7f9e40bfe700 1 -- 172.21.15.44:0/422440514 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e4809e260 con 0x55eb3368e880 2023-11-16T16:50:16.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.482+0000 7f9e40bfe700 1 -- 172.21.15.44:0/422440514 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e480cd5c0 con 0x55eb3368e880 2023-11-16T16:50:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.483+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/422440514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368e880 msgr2=0x55eb33692780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.483+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/422440514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368e880 0x55eb33692780 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9e4809a1c0 tx=0x7f9e4806c2a0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.483+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/422440514 shutdown_connections 2023-11-16T16:50:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.483+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/422440514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb33692cc0 0x55eb33693110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.483+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/422440514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb3368df70 0x55eb3368e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.483+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/422440514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368e880 0x55eb33692780 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.483+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/422440514 >> 172.21.15.44:0/422440514 conn(0x55eb3367f360 msgr2=0x55eb33684b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.483+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/422440514 shutdown_connections 2023-11-16T16:50:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.483+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/422440514 wait complete. 2023-11-16T16:50:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.483+0000 7f9e67c5cb80 1 -- start start 2023-11-16T16:50:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.483+0000 7f9e67c5cb80 1 Processor -- start 2023-11-16T16:50:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.484+0000 7f9e67c5cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368df70 0x55eb3369a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.484+0000 7f9e67c5cb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 0x55eb33697690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.484+0000 7f9e67c5cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb33692cc0 0x55eb33697cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.484+0000 7f9e67c5cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb3357c8e0 con 0x55eb3368df70 2023-11-16T16:50:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.484+0000 7f9e67c5cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb3357c580 con 0x55eb33692cc0 2023-11-16T16:50:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.484+0000 7f9e67c5cb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb3345e920 con 0x55eb3368e880 2023-11-16T16:50:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.484+0000 7f9e67c5cb80 10 client.0 _ll_register_callbacks cb 0x55eb3359a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.484+0000 7f9e5219b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368df70 0x55eb3369a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.484+0000 7f9e5299c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb33692cc0 0x55eb33697cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.484+0000 7f9e5299c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb33692cc0 0x55eb33697cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:36288/0 (socket says 172.21.15.44:36288) 2023-11-16T16:50:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.484+0000 7f9e5299c700 1 -- 172.21.15.44:0/382240925 learned_addr learned my addr 172.21.15.44:0/382240925 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.484+0000 7f9e5199a700 1 --2- 172.21.15.44:0/382240925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 0x55eb33697690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.485+0000 7f9e5219b700 1 -- 172.21.15.44:0/382240925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 msgr2=0x55eb33697690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.485+0000 7f9e5219b700 1 --2- 172.21.15.44:0/382240925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 0x55eb33697690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.485+0000 7f9e5219b700 1 -- 172.21.15.44:0/382240925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb33692cc0 msgr2=0x55eb33697cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.485+0000 7f9e5219b700 1 --2- 172.21.15.44:0/382240925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb33692cc0 0x55eb33697cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.485+0000 7f9e5219b700 1 -- 172.21.15.44:0/382240925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e4c1101e0 con 0x55eb3368df70 2023-11-16T16:50:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.485+0000 7f9e5199a700 1 --2- 172.21.15.44:0/382240925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 0x55eb33697690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.485+0000 7f9e5219b700 1 --2- 172.21.15.44:0/382240925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368df70 0x55eb3369a480 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9e48018000 tx=0x7f9e480a26a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6162b2d9a841f54 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.485+0000 7f9e35ffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:50:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.485+0000 7f9e35ffb700 1 -- 172.21.15.44:0/382240925 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48020cb0 con 0x55eb3368df70 2023-11-16T16:50:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.485+0000 7f9e35ffb700 1 -- 172.21.15.44:0/382240925 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f9e48094e40 con 0x55eb3368df70 2023-11-16T16:50:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.485+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/382240925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eb3367cb20 con 0x55eb3368df70 2023-11-16T16:50:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.485+0000 7f9e35ffb700 1 -- 172.21.15.44:0/382240925 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e4809a230 con 0x55eb3368df70 2023-11-16T16:50:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.485+0000 7f9e67c5cb80 10 client.15543 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.485+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/382240925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55eb335a1190 con 0x55eb3368df70 2023-11-16T16:50:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.485+0000 7f9e35ffb700 1 -- 172.21.15.44:0/382240925 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55eb3367cb20 con 0x55eb3368df70 2023-11-16T16:50:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.486+0000 7f9e35ffb700 1 -- 172.21.15.44:0/382240925 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9e480a21f0 con 0x55eb3368df70 2023-11-16T16:50:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.486+0000 7f9e67c5cb80 10 client.15543 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.486+0000 7f9e67c5cb80 10 client.15543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.486+0000 7f9e67c5cb80 20 client.15543 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.486+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/382240925 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55eb336a7700 0x55eb336a9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.486+0000 7f9e67c5cb80 4 client.15543 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.486+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/382240925 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eb3359d0a0 con 0x55eb336a7700 2023-11-16T16:50:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.486+0000 7f9e5199a700 1 --2- 172.21.15.44:0/382240925 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55eb336a7700 0x55eb336a9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.487+0000 7f9e5199a700 1 --2- 172.21.15.44:0/382240925 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55eb336a7700 0x55eb336a9ae0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21e76abab882ef7c server_cookie=c6f96c0ba6667a69 in_seq=0 out_seq=0 2023-11-16T16:50:16.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.487+0000 7f9e35ffb700 10 client.15543 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:16.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.488+0000 7f9e35ffb700 1 -- 172.21.15.44:0/382240925 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55eb3359d0a0 con 0x55eb336a7700 2023-11-16T16:50:16.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.488+0000 7f9e35ffb700 10 client.15543 handle_command_reply: tid=0 2023-11-16T16:50:16.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.488+0000 7f9e67c5cb80 1 client.15543 shutdown 2023-11-16T16:50:16.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.488+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/382240925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368df70 msgr2=0x55eb3369a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:16.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.488+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/382240925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368df70 0x55eb3369a480 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9e48018000 tx=0x7f9e480a26a0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.488+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/382240925 shutdown_connections 2023-11-16T16:50:16.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.488+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/382240925 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55eb336a7700 0x55eb336a9ae0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.488+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/382240925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb33692cc0 0x55eb33697cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.488+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/382240925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368df70 0x55eb3369a480 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.488+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/382240925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 0x55eb33697690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.488+0000 7f9e35ffb700 0 client.15543 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:16.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.488+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/382240925 >> 172.21.15.44:0/382240925 conn(0x55eb3367f360 msgr2=0x55eb33694170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:16.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.488+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/382240925 shutdown_connections 2023-11-16T16:50:16.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.488+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/382240925 wait complete. 2023-11-16T16:50:16.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.488+0000 7f9e67c5cb80 20 client.15543 trim_cache size 0 max 16384 2023-11-16T16:50:16.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.506+0000 7f9e67c5cb80 1 Processor -- start 2023-11-16T16:50:16.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.506+0000 7f9e67c5cb80 1 -- start start 2023-11-16T16:50:16.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.506+0000 7f9e67c5cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb3368df70 0x55eb3369ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.506+0000 7f9e67c5cb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 0x55eb33690cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.506+0000 7f9e67c5cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb33692cc0 0x55eb33699fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.506+0000 7f9e67c5cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb3367eee0 con 0x55eb33692cc0 2023-11-16T16:50:16.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.506+0000 7f9e67c5cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb3369aa60 con 0x55eb3368df70 2023-11-16T16:50:16.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.506+0000 7f9e67c5cb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb33695bf0 con 0x55eb3368e880 2023-11-16T16:50:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.507+0000 7f9e5299c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb33692cc0 0x55eb33699fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.507+0000 7f9e5299c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb33692cc0 0x55eb33699fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47728/0 (socket says 172.21.15.44:47728) 2023-11-16T16:50:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.507+0000 7f9e5299c700 1 -- 172.21.15.44:0/1674053469 learned_addr learned my addr 172.21.15.44:0/1674053469 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.507+0000 7f9e5219b700 1 --2- 172.21.15.44:0/1674053469 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb3368df70 0x55eb3369ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.507+0000 7f9e5199a700 1 --2- 172.21.15.44:0/1674053469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 0x55eb33690cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.507+0000 7f9e5299c700 1 -- 172.21.15.44:0/1674053469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 msgr2=0x55eb33690cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.507+0000 7f9e5299c700 1 --2- 172.21.15.44:0/1674053469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 0x55eb33690cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.507+0000 7f9e5299c700 1 -- 172.21.15.44:0/1674053469 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb3368df70 msgr2=0x55eb3369ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.507+0000 7f9e5299c700 1 --2- 172.21.15.44:0/1674053469 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb3368df70 0x55eb3369ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.507+0000 7f9e5299c700 1 -- 172.21.15.44:0/1674053469 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb33680b00 con 0x55eb33692cc0 2023-11-16T16:50:16.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.507+0000 7f9e5299c700 1 --2- 172.21.15.44:0/1674053469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb33692cc0 0x55eb33699fb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f9e44009370 tx=0x7f9e44009070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3620777b42877705 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:16.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.508+0000 7f9e413ff700 1 -- 172.21.15.44:0/1674053469 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e44021040 con 0x55eb33692cc0 2023-11-16T16:50:16.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.508+0000 7f9e413ff700 1 -- 172.21.15.44:0/1674053469 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e440093a0 con 0x55eb33692cc0 2023-11-16T16:50:16.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.508+0000 7f9e413ff700 1 -- 172.21.15.44:0/1674053469 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4400e040 con 0x55eb33692cc0 2023-11-16T16:50:16.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.508+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/1674053469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb33692cc0 msgr2=0x55eb33699fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:16.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.508+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/1674053469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb33692cc0 0x55eb33699fb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f9e44009370 tx=0x7f9e44009070 comp rx=0 tx=0).stop 2023-11-16T16:50:16.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.508+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/1674053469 shutdown_connections 2023-11-16T16:50:16.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.508+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/1674053469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb33692cc0 0x55eb33699fb0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.508+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/1674053469 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb3368df70 0x55eb3369ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.508+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/1674053469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 0x55eb33690cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.508+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/1674053469 >> 172.21.15.44:0/1674053469 conn(0x55eb3367f360 msgr2=0x55eb33683550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:16.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.508+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/1674053469 shutdown_connections 2023-11-16T16:50:16.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.508+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/1674053469 wait complete. 2023-11-16T16:50:16.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.509+0000 7f9e67c5cb80 1 -- start start 2023-11-16T16:50:16.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.509+0000 7f9e67c5cb80 1 Processor -- start 2023-11-16T16:50:16.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.509+0000 7f9e67c5cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368df70 0x55eb3369e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.509+0000 7f9e67c5cb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 0x55eb3369eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.509+0000 7f9e67c5cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb33692cc0 0x55eb3369cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.509+0000 7f9e67c5cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb3369d5f0 con 0x55eb3368df70 2023-11-16T16:50:16.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.509+0000 7f9e67c5cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb3369d760 con 0x55eb33692cc0 2023-11-16T16:50:16.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.509+0000 7f9e67c5cb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb3369d920 con 0x55eb3368e880 2023-11-16T16:50:16.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.509+0000 7f9e67c5cb80 10 client.0 _ll_register_callbacks cb 0x55eb3359a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:16.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.510+0000 7f9e5219b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368df70 0x55eb3369e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.510+0000 7f9e5219b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368df70 0x55eb3369e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47732/0 (socket says 172.21.15.44:47732) 2023-11-16T16:50:16.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.510+0000 7f9e5219b700 1 -- 172.21.15.44:0/3181235547 learned_addr learned my addr 172.21.15.44:0/3181235547 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:16.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.510+0000 7f9e5199a700 1 --2- 172.21.15.44:0/3181235547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 0x55eb3369eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.510+0000 7f9e5299c700 1 --2- 172.21.15.44:0/3181235547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb33692cc0 0x55eb3369cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.510+0000 7f9e5219b700 1 -- 172.21.15.44:0/3181235547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 msgr2=0x55eb3369eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:16.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.510+0000 7f9e5219b700 1 --2- 172.21.15.44:0/3181235547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 0x55eb3369eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.510+0000 7f9e5219b700 1 -- 172.21.15.44:0/3181235547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb33692cc0 msgr2=0x55eb3369cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:16.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.510+0000 7f9e5219b700 1 --2- 172.21.15.44:0/3181235547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb33692cc0 0x55eb3369cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.510+0000 7f9e5219b700 1 -- 172.21.15.44:0/3181235547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e440090e0 con 0x55eb3368df70 2023-11-16T16:50:16.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.510+0000 7f9e5199a700 1 --2- 172.21.15.44:0/3181235547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 0x55eb3369eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:16.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.510+0000 7f9e5219b700 1 --2- 172.21.15.44:0/3181235547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368df70 0x55eb3369e9a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9e4809e1d0 tx=0x7f9e480b8250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83a4ff0f7c72f02 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:16.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.510+0000 7f9e35ffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:50:16.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.510+0000 7f9e35ffb700 1 -- 172.21.15.44:0/3181235547 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48020cb0 con 0x55eb3368df70 2023-11-16T16:50:16.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.510+0000 7f9e35ffb700 1 -- 172.21.15.44:0/3181235547 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f9e48094e40 con 0x55eb3368df70 2023-11-16T16:50:16.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.510+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/3181235547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eb335a9660 con 0x55eb3368df70 2023-11-16T16:50:16.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.511+0000 7f9e35ffb700 1 -- 172.21.15.44:0/3181235547 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e48096570 con 0x55eb3368df70 2023-11-16T16:50:16.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.511+0000 7f9e67c5cb80 10 client.15549 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:16.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.511+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/3181235547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55eb33408de0 con 0x55eb3368df70 2023-11-16T16:50:16.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.511+0000 7f9e35ffb700 1 -- 172.21.15.44:0/3181235547 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e48062020 con 0x55eb3368df70 2023-11-16T16:50:16.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.511+0000 7f9e35ffb700 1 -- 172.21.15.44:0/3181235547 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9e4809a520 con 0x55eb3368df70 2023-11-16T16:50:16.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.511+0000 7f9e67c5cb80 10 client.15549 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:16.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.511+0000 7f9e67c5cb80 10 client.15549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:16.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.511+0000 7f9e67c5cb80 20 client.15549 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:16.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.511+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/3181235547 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55eb336fc2b0 0x55eb336fe690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.511+0000 7f9e67c5cb80 4 client.15549 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:50:16.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.511+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/3181235547 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55eb33429f70 con 0x55eb336fc2b0 2023-11-16T16:50:16.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.511+0000 7f9e5199a700 1 --2- 172.21.15.44:0/3181235547 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55eb336fc2b0 0x55eb336fe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.512+0000 7f9e5199a700 1 --2- 172.21.15.44:0/3181235547 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55eb336fc2b0 0x55eb336fe690 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7807d9a8b6dcbb3 server_cookie=42bc6529b23d147 in_seq=0 out_seq=0 2023-11-16T16:50:16.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.512+0000 7f9e35ffb700 10 client.15549 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:16.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.512+0000 7f9e35ffb700 1 -- 172.21.15.44:0/3181235547 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55eb33429f70 con 0x55eb336fc2b0 2023-11-16T16:50:16.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.512+0000 7f9e35ffb700 10 client.15549 handle_command_reply: tid=0 2023-11-16T16:50:16.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.512+0000 7f9e67c5cb80 1 client.15549 shutdown 2023-11-16T16:50:16.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.513+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/3181235547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368df70 msgr2=0x55eb3369e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:16.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.513+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/3181235547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368df70 0x55eb3369e9a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9e4809e1d0 tx=0x7f9e480b8250 comp rx=0 tx=0).stop 2023-11-16T16:50:16.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.513+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/3181235547 shutdown_connections 2023-11-16T16:50:16.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.513+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/3181235547 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55eb336fc2b0 0x55eb336fe690 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.513+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/3181235547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eb33692cc0 0x55eb3369cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.513+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/3181235547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3368df70 0x55eb3369e9a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.513+0000 7f9e35ffb700 0 client.15549 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:16.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.513+0000 7f9e67c5cb80 1 --2- 172.21.15.44:0/3181235547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55eb3368e880 0x55eb3369eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.513+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/3181235547 >> 172.21.15.44:0/3181235547 conn(0x55eb3367f360 msgr2=0x55eb33690cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:16.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.513+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/3181235547 shutdown_connections 2023-11-16T16:50:16.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.513+0000 7f9e67c5cb80 1 -- 172.21.15.44:0/3181235547 wait complete. 2023-11-16T16:50:16.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.513+0000 7f9e67c5cb80 20 client.15549 trim_cache size 0 max 16384 2023-11-16T16:50:16.520 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:50:16.520 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T16:50:16.520 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T16:50:16.520 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:50:16.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.513+0000 7f9e543ff700 1 -- 172.21.15.44:0/3618061208 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9e380a13e0 msgr2=0x7f9e380a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:16.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.513+0000 7f9e543ff700 1 --2- 172.21.15.44:0/3618061208 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9e380a13e0 0x7f9e380a3890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9e44012b90 tx=0x7f9e4400b040 comp rx=0 tx=0).stop 2023-11-16T16:50:16.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.513+0000 7f9e543ff700 1 -- 172.21.15.44:0/3618061208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e4c100cc0 msgr2=0x7f9e4c196530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:16.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.514+0000 7f9e543ff700 1 --2- 172.21.15.44:0/3618061208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e4c100cc0 0x7f9e4c196530 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9e4800e9c0 tx=0x7f9e4800ecd0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.514+0000 7f9e543ff700 1 -- 172.21.15.44:0/3618061208 shutdown_connections 2023-11-16T16:50:16.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.514+0000 7f9e543ff700 1 --2- 172.21.15.44:0/3618061208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e4c105620 0x7f9e4c19aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.514+0000 7f9e543ff700 1 --2- 172.21.15.44:0/3618061208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e4c100cc0 0x7f9e4c196530 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.514+0000 7f9e543ff700 1 --2- 172.21.15.44:0/3618061208 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9e380a13e0 0x7f9e380a3890 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.522 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.514+0000 7f9e543ff700 1 --2- 172.21.15.44:0/3618061208 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e4c101660 0x7f9e4c196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.522 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.514+0000 7f9e543ff700 1 -- 172.21.15.44:0/3618061208 >> 172.21.15.44:0/3618061208 conn(0x7f9e4c077560 msgr2=0x7f9e4c0ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:16.522 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.514+0000 7f9e543ff700 1 -- 172.21.15.44:0/3618061208 shutdown_connections 2023-11-16T16:50:16.522 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.514+0000 7f9e543ff700 1 -- 172.21.15.44:0/3618061208 wait complete. 2023-11-16T16:50:16.533 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T16:50:16.534 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:50:16.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.740+0000 7f098b5ff700 1 -- 172.21.15.44:0/3637382362 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0984100ef0 msgr2=0x7f0984104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:16.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.740+0000 7f098b5ff700 1 --2- 172.21.15.44:0/3637382362 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0984100ef0 0x7f0984104e60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f09740099c0 tx=0x7f0974009cd0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.740+0000 7f098b5ff700 1 -- 172.21.15.44:0/3637382362 shutdown_connections 2023-11-16T16:50:16.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.740+0000 7f098b5ff700 1 --2- 172.21.15.44:0/3637382362 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0984105540 0x7f098410d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.740+0000 7f098b5ff700 1 --2- 172.21.15.44:0/3637382362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0984100550 0x7f0984100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.740+0000 7f098b5ff700 1 --2- 172.21.15.44:0/3637382362 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0984100ef0 0x7f0984104e60 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.740+0000 7f098b5ff700 1 -- 172.21.15.44:0/3637382362 >> 172.21.15.44:0/3637382362 conn(0x7f0984077020 msgr2=0x7f0984077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:16.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.740+0000 7f098b5ff700 1 -- 172.21.15.44:0/3637382362 shutdown_connections 2023-11-16T16:50:16.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.740+0000 7f098b5ff700 1 -- 172.21.15.44:0/3637382362 wait complete. 2023-11-16T16:50:16.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.741+0000 7f098b5ff700 1 Processor -- start 2023-11-16T16:50:16.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.741+0000 7f098b5ff700 1 -- start start 2023-11-16T16:50:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.742+0000 7f098b5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0984100550 0x7f0984198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.742+0000 7f098b5ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0984100ef0 0x7f0984198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.742+0000 7f098b5ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0984105540 0x7f098419cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.742+0000 7f098b5ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098419d5e0 con 0x7f0984100ef0 2023-11-16T16:50:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.742+0000 7f098b5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098419d750 con 0x7f0984100550 2023-11-16T16:50:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.742+0000 7f098b5ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098419d910 con 0x7f0984105540 2023-11-16T16:50:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.742+0000 7f098939b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0984100550 0x7f0984198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.742+0000 7f098939b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0984100550 0x7f0984198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:36336/0 (socket says 172.21.15.44:36336) 2023-11-16T16:50:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.742+0000 7f0988b9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0984100ef0 0x7f0984198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.742+0000 7f098939b700 1 -- 172.21.15.44:0/608643892 learned_addr learned my addr 172.21.15.44:0/608643892 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.742+0000 7f0989b9c700 1 --2- 172.21.15.44:0/608643892 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0984105540 0x7f098419cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.743+0000 7f098939b700 1 -- 172.21.15.44:0/608643892 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0984105540 msgr2=0x7f098419cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:16.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.743+0000 7f098939b700 1 --2- 172.21.15.44:0/608643892 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0984105540 0x7f098419cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.743+0000 7f098939b700 1 -- 172.21.15.44:0/608643892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0984100ef0 msgr2=0x7f0984198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:16.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.743+0000 7f098939b700 1 --2- 172.21.15.44:0/608643892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0984100ef0 0x7f0984198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:16.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.743+0000 7f098939b700 1 -- 172.21.15.44:0/608643892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09740095f0 con 0x7f0984100550 2023-11-16T16:50:16.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.743+0000 7f0988b9a700 1 --2- 172.21.15.44:0/608643892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0984100ef0 0x7f0984198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:16.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.743+0000 7f098939b700 1 --2- 172.21.15.44:0/608643892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0984100550 0x7f0984198600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f098000e9a0 tx=0x7f098000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.743+0000 7f097a7fc700 1 -- 172.21.15.44:0/608643892 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0980009790 con 0x7f0984100550 2023-11-16T16:50:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.743+0000 7f098b5ff700 1 -- 172.21.15.44:0/608643892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0984110310 con 0x7f0984100550 2023-11-16T16:50:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.743+0000 7f097a7fc700 1 -- 172.21.15.44:0/608643892 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f098000cbd0 con 0x7f0984100550 2023-11-16T16:50:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.743+0000 7f097a7fc700 1 -- 172.21.15.44:0/608643892 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0980018680 con 0x7f0984100550 2023-11-16T16:50:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.743+0000 7f098b5ff700 1 -- 172.21.15.44:0/608643892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0984110180 con 0x7f0984100550 2023-11-16T16:50:16.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.744+0000 7f098b5ff700 1 -- 172.21.15.44:0/608643892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09841104a0 con 0x7f0984100550 2023-11-16T16:50:16.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.746+0000 7f097a7fc700 1 -- 172.21.15.44:0/608643892 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f098000cd40 con 0x7f0984100550 2023-11-16T16:50:16.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.748+0000 7f097a7fc700 1 --2- 172.21.15.44:0/608643892 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f09700a13e0 0x7f09700a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:16.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.748+0000 7f097a7fc700 1 -- 172.21.15.44:0/608643892 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f0980014070 con 0x7f0984100550 2023-11-16T16:50:16.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.748+0000 7f0988b9a700 1 --2- 172.21.15.44:0/608643892 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f09700a13e0 0x7f09700a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:16.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.748+0000 7f0988b9a700 1 --2- 172.21.15.44:0/608643892 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f09700a13e0 0x7f09700a3890 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f09740098e0 tx=0x7f0974000bc0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:16.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:16.750+0000 7f097a7fc700 1 -- 172.21.15.44:0/608643892 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f09800964c0 con 0x7f0984100550 2023-11-16T16:50:16.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:16 smithi044 ceph-mon[128750]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 853 B/s wr, 2 op/s 2023-11-16T16:50:16.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:16 smithi077 ceph-mon[131204]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 853 B/s wr, 2 op/s 2023-11-16T16:50:17.010 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.008+0000 7f098b5ff700 1 -- 172.21.15.44:0/608643892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0984067a60 con 0x7f0984100550 2023-11-16T16:50:17.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.009+0000 7f097a7fc700 1 -- 172.21.15.44:0/608643892 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f0980099ae0 con 0x7f0984100550 2023-11-16T16:50:17.011 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:50:17.011 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:50:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.012+0000 7f098b5ff700 1 -- 172.21.15.44:0/608643892 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f09700a13e0 msgr2=0x7f09700a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.012+0000 7f098b5ff700 1 --2- 172.21.15.44:0/608643892 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f09700a13e0 0x7f09700a3890 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f09740098e0 tx=0x7f0974000bc0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.012+0000 7f098b5ff700 1 -- 172.21.15.44:0/608643892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0984100550 msgr2=0x7f0984198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.012+0000 7f098b5ff700 1 --2- 172.21.15.44:0/608643892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0984100550 0x7f0984198600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f098000e9a0 tx=0x7f098000ecb0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.012+0000 7f098b5ff700 1 -- 172.21.15.44:0/608643892 shutdown_connections 2023-11-16T16:50:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.012+0000 7f098b5ff700 1 --2- 172.21.15.44:0/608643892 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f09700a13e0 0x7f09700a3890 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.012+0000 7f098b5ff700 1 --2- 172.21.15.44:0/608643892 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0984105540 0x7f098419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.012+0000 7f098b5ff700 1 --2- 172.21.15.44:0/608643892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0984100550 0x7f0984198600 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.012+0000 7f098b5ff700 1 --2- 172.21.15.44:0/608643892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0984100ef0 0x7f0984198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.012+0000 7f098b5ff700 1 -- 172.21.15.44:0/608643892 >> 172.21.15.44:0/608643892 conn(0x7f0984077020 msgr2=0x7f09840fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:17.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.012+0000 7f098b5ff700 1 -- 172.21.15.44:0/608643892 shutdown_connections 2023-11-16T16:50:17.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.012+0000 7f098b5ff700 1 -- 172.21.15.44:0/608643892 wait complete. 2023-11-16T16:50:17.015 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:50:17.033 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T16:50:17.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:16 smithi176 ceph-mon[131630]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 853 B/s wr, 2 op/s 2023-11-16T16:50:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.238+0000 7f353dbff700 1 -- 172.21.15.44:0/2981880744 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3538101660 msgr2=0x7f3538104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.238+0000 7f353dbff700 1 --2- 172.21.15.44:0/2981880744 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3538101660 0x7f3538104ef0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f35200099c0 tx=0x7f3520009cd0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.238+0000 7f353dbff700 1 -- 172.21.15.44:0/2981880744 shutdown_connections 2023-11-16T16:50:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.238+0000 7f353dbff700 1 --2- 172.21.15.44:0/2981880744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3538105620 0x7f353810d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.238+0000 7f353dbff700 1 --2- 172.21.15.44:0/2981880744 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3538100cc0 0x7f3538101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.238+0000 7f353dbff700 1 --2- 172.21.15.44:0/2981880744 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3538101660 0x7f3538104ef0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.238+0000 7f353dbff700 1 -- 172.21.15.44:0/2981880744 >> 172.21.15.44:0/2981880744 conn(0x7f3538077560 msgr2=0x7f3538077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.238+0000 7f353dbff700 1 -- 172.21.15.44:0/2981880744 shutdown_connections 2023-11-16T16:50:17.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.238+0000 7f353dbff700 1 -- 172.21.15.44:0/2981880744 wait complete. 2023-11-16T16:50:17.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.240+0000 7f353dbff700 1 Processor -- start 2023-11-16T16:50:17.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.240+0000 7f353dbff700 1 -- start start 2023-11-16T16:50:17.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.240+0000 7f353dbff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3538100cc0 0x7f35381964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.240+0000 7f353dbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3538101660 0x7f35381969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.240+0000 7f353dbff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3538105620 0x7f353819ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.240+0000 7f353dbff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353819b490 con 0x7f3538105620 2023-11-16T16:50:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.240+0000 7f353dbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353819b600 con 0x7f3538101660 2023-11-16T16:50:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.240+0000 7f353dbff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353819b7c0 con 0x7f3538100cc0 2023-11-16T16:50:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.241+0000 7f35377fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3538100cc0 0x7f35381964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.241+0000 7f35377fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3538100cc0 0x7f35381964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:59002/0 (socket says 172.21.15.44:59002) 2023-11-16T16:50:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.241+0000 7f35377fe700 1 -- 172.21.15.44:0/3206573437 learned_addr learned my addr 172.21.15.44:0/3206573437 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.241+0000 7f3537fff700 1 --2- 172.21.15.44:0/3206573437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3538105620 0x7f353819ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.241+0000 7f352ffff700 1 --2- 172.21.15.44:0/3206573437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3538101660 0x7f35381969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.241+0000 7f3537fff700 1 -- 172.21.15.44:0/3206573437 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3538100cc0 msgr2=0x7f35381964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.241+0000 7f3537fff700 1 --2- 172.21.15.44:0/3206573437 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3538100cc0 0x7f35381964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.241+0000 7f3537fff700 1 -- 172.21.15.44:0/3206573437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3538101660 msgr2=0x7f35381969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.241+0000 7f3537fff700 1 --2- 172.21.15.44:0/3206573437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3538101660 0x7f35381969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.241+0000 7f3537fff700 1 -- 172.21.15.44:0/3206573437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35200095f0 con 0x7f3538105620 2023-11-16T16:50:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.241+0000 7f35377fe700 1 --2- 172.21.15.44:0/3206573437 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3538100cc0 0x7f35381964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.241+0000 7f3537fff700 1 --2- 172.21.15.44:0/3206573437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3538105620 0x7f353819ae50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3530007f10 tx=0x7f353000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.242+0000 7f35357fa700 1 -- 172.21.15.44:0/3206573437 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3530004db0 con 0x7f3538105620 2023-11-16T16:50:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.242+0000 7f35357fa700 1 -- 172.21.15.44:0/3206573437 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f353000f040 con 0x7f3538105620 2023-11-16T16:50:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.242+0000 7f353dbff700 1 -- 172.21.15.44:0/3206573437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3538110370 con 0x7f3538105620 2023-11-16T16:50:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.242+0000 7f35357fa700 1 -- 172.21.15.44:0/3206573437 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3530005990 con 0x7f3538105620 2023-11-16T16:50:17.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.242+0000 7f353dbff700 1 -- 172.21.15.44:0/3206573437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35381101e0 con 0x7f3538105620 2023-11-16T16:50:17.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.245+0000 7f35357fa700 1 -- 172.21.15.44:0/3206573437 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f353000db20 con 0x7f3538105620 2023-11-16T16:50:17.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.246+0000 7f3551570b80 1 Processor -- start 2023-11-16T16:50:17.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.246+0000 7f3551570b80 1 -- start start 2023-11-16T16:50:17.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.247+0000 7f3551570b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7ef70 0x557104f7f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.247+0000 7f3551570b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7f880 0x557104f83780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.247+0000 7f3551570b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f83cc0 0x557104f84110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.247+0000 7f3551570b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x557104e80060 con 0x557104f7f880 2023-11-16T16:50:17.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.247+0000 7f3551570b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x557104ed28a0 con 0x557104f83cc0 2023-11-16T16:50:17.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.247+0000 7f3551570b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x557104ecbd30 con 0x557104f7ef70 2023-11-16T16:50:17.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.247+0000 7f35357fa700 1 --2- 172.21.15.44:0/3206573437 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f35240a13e0 0x7f35240a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.247+0000 7f35357fa700 1 -- 172.21.15.44:0/3206573437 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f35300ccb90 con 0x7f3538105620 2023-11-16T16:50:17.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.247+0000 7f35377fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7f880 0x557104f83780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.247+0000 7f352ffff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7ef70 0x557104f7f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.247+0000 7f35377fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7f880 0x557104f83780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47776/0 (socket says 172.21.15.44:47776) 2023-11-16T16:50:17.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.247+0000 7f35377fe700 1 -- 172.21.15.44:0/3909085709 learned_addr learned my addr 172.21.15.44:0/3909085709 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:17.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.247+0000 7f3537fff700 1 --2- 172.21.15.44:0/3206573437 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f35240a13e0 0x7f35240a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.247+0000 7f352ffff700 1 --2- 172.21.15.44:0/3909085709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f83cc0 0x557104f84110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.248+0000 7f35377fe700 1 -- 172.21.15.44:0/3909085709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7ef70 msgr2=0x557104f7f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:17.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.248+0000 7f35377fe700 1 --2- 172.21.15.44:0/3909085709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7ef70 0x557104f7f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.248+0000 7f35377fe700 1 -- 172.21.15.44:0/3909085709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f83cc0 msgr2=0x557104f84110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:17.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.248+0000 7f35377fe700 1 --2- 172.21.15.44:0/3909085709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f83cc0 0x557104f84110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.248+0000 7f35377fe700 1 -- 172.21.15.44:0/3909085709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557104d4d630 con 0x557104f7f880 2023-11-16T16:50:17.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.248+0000 7f3537fff700 1 --2- 172.21.15.44:0/3206573437 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f35240a13e0 0x7f35240a3890 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f35300c9de0 tx=0x7f3530009f90 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:17.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.248+0000 7f35377fe700 1 --2- 172.21.15.44:0/3909085709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7f880 0x557104f83780 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f35280117e0 tx=0x7f3528011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c34f0879e64326f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:17.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.248+0000 7f352dffb700 1 -- 172.21.15.44:0/3909085709 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352800f480 con 0x557104f7f880 2023-11-16T16:50:17.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.248+0000 7f352dffb700 1 -- 172.21.15.44:0/3909085709 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f352800faa0 con 0x557104f7f880 2023-11-16T16:50:17.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.248+0000 7f352dffb700 1 -- 172.21.15.44:0/3909085709 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3528019990 con 0x557104f7f880 2023-11-16T16:50:17.256 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.249+0000 7f3551570b80 1 -- 172.21.15.44:0/3909085709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7f880 msgr2=0x557104f83780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.249+0000 7f3551570b80 1 --2- 172.21.15.44:0/3909085709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7f880 0x557104f83780 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f35280117e0 tx=0x7f3528011af0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.249+0000 7f3551570b80 1 -- 172.21.15.44:0/3909085709 shutdown_connections 2023-11-16T16:50:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.249+0000 7f3551570b80 1 --2- 172.21.15.44:0/3909085709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f83cc0 0x557104f84110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.249+0000 7f3551570b80 1 --2- 172.21.15.44:0/3909085709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7ef70 0x557104f7f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.249+0000 7f3551570b80 1 --2- 172.21.15.44:0/3909085709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7f880 0x557104f83780 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.249+0000 7f3551570b80 1 -- 172.21.15.44:0/3909085709 >> 172.21.15.44:0/3909085709 conn(0x557104f70360 msgr2=0x557104f75b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.249+0000 7f3551570b80 1 -- 172.21.15.44:0/3909085709 shutdown_connections 2023-11-16T16:50:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.249+0000 7f3551570b80 1 -- 172.21.15.44:0/3909085709 wait complete. 2023-11-16T16:50:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.249+0000 7f3551570b80 1 -- start start 2023-11-16T16:50:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.250+0000 7f3551570b80 1 Processor -- start 2023-11-16T16:50:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.250+0000 7f3551570b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7ef70 0x557104f8b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.250+0000 7f3551570b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7f880 0x557104f88600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.250+0000 7f3551570b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f83cc0 0x557104f88ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.250+0000 7f3551570b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x557104e6d8e0 con 0x557104f7ef70 2023-11-16T16:50:17.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.250+0000 7f3551570b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x557104e6d580 con 0x557104f83cc0 2023-11-16T16:50:17.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.250+0000 7f3551570b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x557104d4f920 con 0x557104f7f880 2023-11-16T16:50:17.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.250+0000 7f3551570b80 10 client.0 _ll_register_callbacks cb 0x557104e8b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:17.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.250+0000 7f352ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7ef70 0x557104f8b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.251+0000 7f352ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7ef70 0x557104f8b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47780/0 (socket says 172.21.15.44:47780) 2023-11-16T16:50:17.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.251+0000 7f352ffff700 1 -- 172.21.15.44:0/3467163066 learned_addr learned my addr 172.21.15.44:0/3467163066 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:17.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.251+0000 7f35377fe700 1 --2- 172.21.15.44:0/3467163066 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7f880 0x557104f88600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.251+0000 7f352ffff700 1 --2- 172.21.15.44:0/3467163066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f83cc0 0x557104f88ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.251+0000 7f352ffff700 1 -- 172.21.15.44:0/3467163066 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7f880 msgr2=0x557104f88600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:17.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.251+0000 7f352ffff700 1 --2- 172.21.15.44:0/3467163066 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7f880 0x557104f88600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.251+0000 7f352ffff700 1 -- 172.21.15.44:0/3467163066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f83cc0 msgr2=0x557104f88ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.251+0000 7f352ffff700 1 --2- 172.21.15.44:0/3467163066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f83cc0 0x557104f88ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.251+0000 7f352ffff700 1 -- 172.21.15.44:0/3467163066 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3528011440 con 0x557104f7ef70 2023-11-16T16:50:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.251+0000 7f35377fe700 1 --2- 172.21.15.44:0/3467163066 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7f880 0x557104f88600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.251+0000 7f352ffff700 1 --2- 172.21.15.44:0/3467163066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7ef70 0x557104f8b380 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f35200098e0 tx=0x7f3520009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3038d3e9074399 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.251+0000 7f3516ffd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:50:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.251+0000 7f3516ffd700 1 -- 172.21.15.44:0/3467163066 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3520003d10 con 0x557104f7ef70 2023-11-16T16:50:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.252+0000 7f3516ffd700 1 -- 172.21.15.44:0/3467163066 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f352000b790 con 0x557104f7ef70 2023-11-16T16:50:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.252+0000 7f3551570b80 1 -- 172.21.15.44:0/3467163066 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557104f6db20 con 0x557104f7ef70 2023-11-16T16:50:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.252+0000 7f3516ffd700 1 -- 172.21.15.44:0/3467163066 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3520003e70 con 0x557104f7ef70 2023-11-16T16:50:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.252+0000 7f3551570b80 10 client.15567 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.252+0000 7f3551570b80 1 -- 172.21.15.44:0/3467163066 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x557104e92190 con 0x557104f7ef70 2023-11-16T16:50:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.252+0000 7f3516ffd700 1 -- 172.21.15.44:0/3467163066 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557104f6db20 con 0x557104f7ef70 2023-11-16T16:50:17.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.252+0000 7f3516ffd700 1 -- 172.21.15.44:0/3467163066 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f352000f8a0 con 0x557104f7ef70 2023-11-16T16:50:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.252+0000 7f3551570b80 10 client.15567 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.252+0000 7f3551570b80 10 client.15567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.252+0000 7f3551570b80 20 client.15567 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.252+0000 7f3551570b80 1 --2- 172.21.15.44:0/3467163066 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557104f98c50 0x557104f9b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.252+0000 7f3551570b80 4 client.15567 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.252+0000 7f3551570b80 1 -- 172.21.15.44:0/3467163066 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557104e8e0a0 con 0x557104f98c50 2023-11-16T16:50:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.253+0000 7f35377fe700 1 --2- 172.21.15.44:0/3467163066 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557104f98c50 0x557104f9b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.253+0000 7f3516ffd700 10 client.15567 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.253+0000 7f35377fe700 1 --2- 172.21.15.44:0/3467163066 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557104f98c50 0x557104f9b060 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c19abc5c25257b6 server_cookie=5ad6b14d9ecb8e53 in_seq=0 out_seq=0 2023-11-16T16:50:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.254+0000 7f3516ffd700 1 -- 172.21.15.44:0/3467163066 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557104e8e0a0 con 0x557104f98c50 2023-11-16T16:50:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.254+0000 7f3516ffd700 10 client.15567 handle_command_reply: tid=0 2023-11-16T16:50:17.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.254+0000 7f3551570b80 1 client.15567 shutdown 2023-11-16T16:50:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.254+0000 7f3551570b80 1 -- 172.21.15.44:0/3467163066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7ef70 msgr2=0x557104f8b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.254+0000 7f3551570b80 1 --2- 172.21.15.44:0/3467163066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7ef70 0x557104f8b380 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f35200098e0 tx=0x7f3520009310 comp rx=0 tx=0).stop 2023-11-16T16:50:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.254+0000 7f3551570b80 1 -- 172.21.15.44:0/3467163066 shutdown_connections 2023-11-16T16:50:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.254+0000 7f3551570b80 1 --2- 172.21.15.44:0/3467163066 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557104f98c50 0x557104f9b060 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.254+0000 7f3551570b80 1 --2- 172.21.15.44:0/3467163066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f83cc0 0x557104f88ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.254+0000 7f3516ffd700 0 client.15567 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.254+0000 7f3551570b80 1 --2- 172.21.15.44:0/3467163066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7ef70 0x557104f8b380 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.254+0000 7f3551570b80 1 --2- 172.21.15.44:0/3467163066 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7f880 0x557104f88600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.255+0000 7f3551570b80 1 -- 172.21.15.44:0/3467163066 >> 172.21.15.44:0/3467163066 conn(0x557104f70360 msgr2=0x557104f85170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.255+0000 7f3551570b80 1 -- 172.21.15.44:0/3467163066 shutdown_connections 2023-11-16T16:50:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.255+0000 7f3551570b80 1 -- 172.21.15.44:0/3467163066 wait complete. 2023-11-16T16:50:17.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.255+0000 7f3551570b80 20 client.15567 trim_cache size 0 max 16384 2023-11-16T16:50:17.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.268+0000 7f3551570b80 1 Processor -- start 2023-11-16T16:50:17.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.268+0000 7f3551570b80 1 -- start start 2023-11-16T16:50:17.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.269+0000 7f3551570b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7ef70 0x557104f8af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.269+0000 7f3551570b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f7f880 0x557104f81cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.269+0000 7f3551570b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f83cc0 0x557104f75620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.269+0000 7f3551570b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x557104f8bba0 con 0x557104f83cc0 2023-11-16T16:50:17.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.269+0000 7f3551570b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x557104f7ffd0 con 0x557104f7f880 2023-11-16T16:50:17.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.269+0000 7f3551570b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x557104f6fee0 con 0x557104f7ef70 2023-11-16T16:50:17.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.269+0000 7f352ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f83cc0 0x557104f75620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.269+0000 7f352ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f83cc0 0x557104f75620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47784/0 (socket says 172.21.15.44:47784) 2023-11-16T16:50:17.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.269+0000 7f352ffff700 1 -- 172.21.15.44:0/2335048717 learned_addr learned my addr 172.21.15.44:0/2335048717 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:17.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.269+0000 7f352ffff700 1 --2- 172.21.15.44:0/2335048717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7ef70 0x557104f8af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.269+0000 7f35377fe700 1 --2- 172.21.15.44:0/2335048717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f7f880 0x557104f81cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.269+0000 7f352ffff700 1 -- 172.21.15.44:0/2335048717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7ef70 msgr2=0x557104f8af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:17.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.269+0000 7f352ffff700 1 --2- 172.21.15.44:0/2335048717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7ef70 0x557104f8af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.269+0000 7f352ffff700 1 -- 172.21.15.44:0/2335048717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f7f880 msgr2=0x557104f81cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:17.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.269+0000 7f352ffff700 1 --2- 172.21.15.44:0/2335048717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f7f880 0x557104f81cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.269+0000 7f352ffff700 1 -- 172.21.15.44:0/2335048717 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557104f84360 con 0x557104f83cc0 2023-11-16T16:50:17.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.270+0000 7f352ffff700 1 --2- 172.21.15.44:0/2335048717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f83cc0 0x557104f75620 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3520005850 tx=0x7f3520003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c029c8026d702c2d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:17.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.270+0000 7f352e7fc700 1 -- 172.21.15.44:0/2335048717 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3520003d10 con 0x557104f83cc0 2023-11-16T16:50:17.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.270+0000 7f352e7fc700 1 -- 172.21.15.44:0/2335048717 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f352001e040 con 0x557104f83cc0 2023-11-16T16:50:17.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.270+0000 7f352e7fc700 1 -- 172.21.15.44:0/2335048717 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352000bcd0 con 0x557104f83cc0 2023-11-16T16:50:17.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.270+0000 7f3551570b80 1 -- 172.21.15.44:0/2335048717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f83cc0 msgr2=0x557104f75620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:17.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.270+0000 7f3551570b80 1 --2- 172.21.15.44:0/2335048717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f83cc0 0x557104f75620 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3520005850 tx=0x7f3520003680 comp rx=0 tx=0).stop 2023-11-16T16:50:17.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.270+0000 7f3551570b80 1 -- 172.21.15.44:0/2335048717 shutdown_connections 2023-11-16T16:50:17.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.270+0000 7f3551570b80 1 --2- 172.21.15.44:0/2335048717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f83cc0 0x557104f75620 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.270+0000 7f3551570b80 1 --2- 172.21.15.44:0/2335048717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7ef70 0x557104f8af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.270+0000 7f3551570b80 1 --2- 172.21.15.44:0/2335048717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f7f880 0x557104f81cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.271+0000 7f3551570b80 1 -- 172.21.15.44:0/2335048717 >> 172.21.15.44:0/2335048717 conn(0x557104f70360 msgr2=0x557104f94f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:17.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.271+0000 7f3551570b80 1 -- 172.21.15.44:0/2335048717 shutdown_connections 2023-11-16T16:50:17.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.271+0000 7f3551570b80 1 -- 172.21.15.44:0/2335048717 wait complete. 2023-11-16T16:50:17.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.271+0000 7f3551570b80 1 -- start start 2023-11-16T16:50:17.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.271+0000 7f3551570b80 1 Processor -- start 2023-11-16T16:50:17.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f3551570b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7ef70 0x557104f8d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f3551570b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7f880 0x557104f8da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f3551570b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f83cc0 0x557104f8dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f3551570b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x557104febad0 con 0x557104f7ef70 2023-11-16T16:50:17.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f3551570b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x557104febc10 con 0x557104f83cc0 2023-11-16T16:50:17.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f3551570b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x557104febdd0 con 0x557104f7f880 2023-11-16T16:50:17.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f3551570b80 10 client.0 _ll_register_callbacks cb 0x557104e8b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:17.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f352ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7ef70 0x557104f8d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f352ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7ef70 0x557104f8d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47800/0 (socket says 172.21.15.44:47800) 2023-11-16T16:50:17.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f352ffff700 1 -- 172.21.15.44:0/1800075381 learned_addr learned my addr 172.21.15.44:0/1800075381 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:17.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f352ffff700 1 --2- 172.21.15.44:0/1800075381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f83cc0 0x557104f8dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f35377fe700 1 --2- 172.21.15.44:0/1800075381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7f880 0x557104f8da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f352ffff700 1 -- 172.21.15.44:0/1800075381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7f880 msgr2=0x557104f8da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:17.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f352ffff700 1 --2- 172.21.15.44:0/1800075381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7f880 0x557104f8da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f352ffff700 1 -- 172.21.15.44:0/1800075381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f83cc0 msgr2=0x557104f8dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:17.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f352ffff700 1 --2- 172.21.15.44:0/1800075381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f83cc0 0x557104f8dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f352ffff700 1 -- 172.21.15.44:0/1800075381 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557104e92190 con 0x557104f7ef70 2023-11-16T16:50:17.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.272+0000 7f352ffff700 1 --2- 172.21.15.44:0/1800075381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7ef70 0x557104f8d550 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f35200058e0 tx=0x7f3520009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec33a210b2df0eb0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:17.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.273+0000 7f3516ffd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:50:17.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.273+0000 7f3516ffd700 1 -- 172.21.15.44:0/1800075381 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3520042070 con 0x557104f7ef70 2023-11-16T16:50:17.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.273+0000 7f3516ffd700 1 -- 172.21.15.44:0/1800075381 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f3520000e40 con 0x557104f7ef70 2023-11-16T16:50:17.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.273+0000 7f3551570b80 1 -- 172.21.15.44:0/1800075381 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f352800faa0 con 0x557104f7ef70 2023-11-16T16:50:17.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.273+0000 7f3516ffd700 1 -- 172.21.15.44:0/1800075381 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f352000bd70 con 0x557104f7ef70 2023-11-16T16:50:17.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.273+0000 7f3551570b80 10 client.15573 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:17.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.273+0000 7f3551570b80 1 -- 172.21.15.44:0/1800075381 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x557104cf9de0 con 0x557104f7ef70 2023-11-16T16:50:17.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.273+0000 7f3516ffd700 1 -- 172.21.15.44:0/1800075381 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f352800faa0 con 0x557104f7ef70 2023-11-16T16:50:17.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.273+0000 7f3516ffd700 1 -- 172.21.15.44:0/1800075381 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3520045020 con 0x557104f7ef70 2023-11-16T16:50:17.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.274+0000 7f3551570b80 10 client.15573 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:17.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.274+0000 7f3551570b80 10 client.15573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.274+0000 7f3551570b80 20 client.15573 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.274+0000 7f3551570b80 1 --2- 172.21.15.44:0/1800075381 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557104fed020 0x557104fef400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.274+0000 7f3551570b80 4 client.15573 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T16:50:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.274+0000 7f3551570b80 1 -- 172.21.15.44:0/1800075381 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557104e9a660 con 0x557104fed020 2023-11-16T16:50:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.274+0000 7f35377fe700 1 --2- 172.21.15.44:0/1800075381 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557104fed020 0x557104fef400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.274+0000 7f35377fe700 1 --2- 172.21.15.44:0/1800075381 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557104fed020 0x557104fef400 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13235ae29fba8241 server_cookie=c3f16b622dbf89ce in_seq=0 out_seq=0 2023-11-16T16:50:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.274+0000 7f3516ffd700 10 client.15573 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.275+0000 7f3516ffd700 1 -- 172.21.15.44:0/1800075381 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557104e9a660 con 0x557104fed020 2023-11-16T16:50:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.275+0000 7f3516ffd700 10 client.15573 handle_command_reply: tid=0 2023-11-16T16:50:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.275+0000 7f3551570b80 1 client.15573 shutdown 2023-11-16T16:50:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.275+0000 7f3551570b80 1 -- 172.21.15.44:0/1800075381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7ef70 msgr2=0x557104f8d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.275+0000 7f3551570b80 1 --2- 172.21.15.44:0/1800075381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7ef70 0x557104f8d550 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f35200058e0 tx=0x7f3520009cd0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.275+0000 7f3551570b80 1 -- 172.21.15.44:0/1800075381 shutdown_connections 2023-11-16T16:50:17.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.275+0000 7f3551570b80 1 --2- 172.21.15.44:0/1800075381 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557104fed020 0x557104fef400 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.275+0000 7f3551570b80 1 --2- 172.21.15.44:0/1800075381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557104f83cc0 0x557104f8dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.275+0000 7f3551570b80 1 --2- 172.21.15.44:0/1800075381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557104f7ef70 0x557104f8d550 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.275+0000 7f3551570b80 1 --2- 172.21.15.44:0/1800075381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557104f7f880 0x557104f8da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.275+0000 7f3516ffd700 0 client.15573 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:17.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.275+0000 7f3551570b80 1 -- 172.21.15.44:0/1800075381 >> 172.21.15.44:0/1800075381 conn(0x557104f70360 msgr2=0x557104f81cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:17.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.275+0000 7f3551570b80 1 -- 172.21.15.44:0/1800075381 shutdown_connections 2023-11-16T16:50:17.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.275+0000 7f3551570b80 1 -- 172.21.15.44:0/1800075381 wait complete. 2023-11-16T16:50:17.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.275+0000 7f3551570b80 20 client.15573 trim_cache size 0 max 16384 2023-11-16T16:50:17.283 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T16:50:17.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.276+0000 7f353dbff700 1 -- 172.21.15.44:0/3206573437 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f35240a13e0 msgr2=0x7f35240a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:17.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.276+0000 7f353dbff700 1 --2- 172.21.15.44:0/3206573437 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f35240a13e0 0x7f35240a3890 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f35300c9de0 tx=0x7f3530009f90 comp rx=0 tx=0).stop 2023-11-16T16:50:17.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.276+0000 7f353dbff700 1 -- 172.21.15.44:0/3206573437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3538105620 msgr2=0x7f353819ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:17.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.276+0000 7f353dbff700 1 --2- 172.21.15.44:0/3206573437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3538105620 0x7f353819ae50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3530007f10 tx=0x7f353000d3b0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.276+0000 7f353dbff700 1 -- 172.21.15.44:0/3206573437 shutdown_connections 2023-11-16T16:50:17.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.276+0000 7f353dbff700 1 --2- 172.21.15.44:0/3206573437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3538105620 0x7f353819ae50 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.276+0000 7f353dbff700 1 --2- 172.21.15.44:0/3206573437 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f35240a13e0 0x7f35240a3890 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.276+0000 7f353dbff700 1 --2- 172.21.15.44:0/3206573437 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3538100cc0 0x7f35381964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.276+0000 7f353dbff700 1 --2- 172.21.15.44:0/3206573437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3538101660 0x7f35381969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:17.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.276+0000 7f353dbff700 1 -- 172.21.15.44:0/3206573437 >> 172.21.15.44:0/3206573437 conn(0x7f3538077560 msgr2=0x7f35380ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:17.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.276+0000 7f353dbff700 1 -- 172.21.15.44:0/3206573437 shutdown_connections 2023-11-16T16:50:17.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:17.276+0000 7f353dbff700 1 -- 172.21.15.44:0/3206573437 wait complete. 2023-11-16T16:50:17.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:17 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/608643892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:50:18.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:17 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/608643892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:50:18.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:17 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/608643892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:50:18.296 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:50:18.296 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:50:18.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.515+0000 7fda04bff700 1 -- 172.21.15.44:0/3079849571 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda00101660 msgr2=0x7fda00104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.515+0000 7fda04bff700 1 --2- 172.21.15.44:0/3079849571 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda00101660 0x7fda00104ef0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4009920 tx=0x7fd9f4009c30 comp rx=0 tx=0).stop 2023-11-16T16:50:18.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.515+0000 7fda04bff700 1 -- 172.21.15.44:0/3079849571 shutdown_connections 2023-11-16T16:50:18.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.515+0000 7fda04bff700 1 --2- 172.21.15.44:0/3079849571 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda00105620 0x7fda0010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.515+0000 7fda04bff700 1 --2- 172.21.15.44:0/3079849571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00100cc0 0x7fda00101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.515+0000 7fda04bff700 1 --2- 172.21.15.44:0/3079849571 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda00101660 0x7fda00104ef0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.515+0000 7fda04bff700 1 -- 172.21.15.44:0/3079849571 >> 172.21.15.44:0/3079849571 conn(0x7fda00077560 msgr2=0x7fda00077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:18.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.515+0000 7fda04bff700 1 -- 172.21.15.44:0/3079849571 shutdown_connections 2023-11-16T16:50:18.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.515+0000 7fda04bff700 1 -- 172.21.15.44:0/3079849571 wait complete. 2023-11-16T16:50:18.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.516+0000 7fda04bff700 1 Processor -- start 2023-11-16T16:50:18.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.517+0000 7fda04bff700 1 -- start start 2023-11-16T16:50:18.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.517+0000 7fda04bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda00100cc0 0x7fda00072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.517+0000 7fda04bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00101660 0x7fda00073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.517+0000 7fda04bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda00105620 0x7fda001a4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.517+0000 7fda04bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda00073b60 con 0x7fda00100cc0 2023-11-16T16:50:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.517+0000 7fda04bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda001a4c90 con 0x7fda00101660 2023-11-16T16:50:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.517+0000 7fda04bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda001a4dd0 con 0x7fda00105620 2023-11-16T16:50:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.517+0000 7fd9fdd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00101660 0x7fda00073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.517+0000 7fd9fe59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda00100cc0 0x7fda00072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.517+0000 7fd9fdd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00101660 0x7fda00073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:36420/0 (socket says 172.21.15.44:36420) 2023-11-16T16:50:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.517+0000 7fd9fdd9b700 1 -- 172.21.15.44:0/2450008048 learned_addr learned my addr 172.21.15.44:0/2450008048 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.517+0000 7fd9fed9d700 1 --2- 172.21.15.44:0/2450008048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda00105620 0x7fda001a4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.518+0000 7fd9fe59c700 1 -- 172.21.15.44:0/2450008048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda00105620 msgr2=0x7fda001a4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.518+0000 7fd9fe59c700 1 --2- 172.21.15.44:0/2450008048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda00105620 0x7fda001a4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.518+0000 7fd9fe59c700 1 -- 172.21.15.44:0/2450008048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00101660 msgr2=0x7fda00073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.518+0000 7fd9fe59c700 1 --2- 172.21.15.44:0/2450008048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00101660 0x7fda00073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.518+0000 7fd9fe59c700 1 -- 172.21.15.44:0/2450008048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f00095f0 con 0x7fda00100cc0 2023-11-16T16:50:18.522 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.518+0000 7fd9fdd9b700 1 --2- 172.21.15.44:0/2450008048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00101660 0x7fda00073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:18.522 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.518+0000 7fd9fe59c700 1 --2- 172.21.15.44:0/2450008048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda00100cc0 0x7fda00072df0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000eb90 tx=0x7fd9f000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:18.522 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.518+0000 7fd9ef7fe700 1 -- 172.21.15.44:0/2450008048 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0009970 con 0x7fda00100cc0 2023-11-16T16:50:18.522 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.518+0000 7fd9ef7fe700 1 -- 172.21.15.44:0/2450008048 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd9f00044e0 con 0x7fda00100cc0 2023-11-16T16:50:18.522 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.518+0000 7fda04bff700 1 -- 172.21.15.44:0/2450008048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f4009520 con 0x7fda00100cc0 2023-11-16T16:50:18.522 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.518+0000 7fd9ef7fe700 1 -- 172.21.15.44:0/2450008048 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0003e50 con 0x7fda00100cc0 2023-11-16T16:50:18.522 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.518+0000 7fda04bff700 1 -- 172.21.15.44:0/2450008048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda00110370 con 0x7fda00100cc0 2023-11-16T16:50:18.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.521+0000 7fd9ef7fe700 1 -- 172.21.15.44:0/2450008048 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7fd9f0004000 con 0x7fda00100cc0 2023-11-16T16:50:18.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.523+0000 7fda185c7b80 1 Processor -- start 2023-11-16T16:50:18.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.523+0000 7fd9ef7fe700 1 --2- 172.21.15.44:0/2450008048 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd9e80a13e0 0x7fd9e80a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.523+0000 7fda185c7b80 1 -- start start 2023-11-16T16:50:18.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.523+0000 7fd9ef7fe700 1 -- 172.21.15.44:0/2450008048 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fd9f0014070 con 0x7fda00100cc0 2023-11-16T16:50:18.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.523+0000 7fd9fed9d700 1 --2- 172.21.15.44:0/2450008048 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd9e80a13e0 0x7fd9e80a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.523+0000 7fda185c7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.523+0000 7fda185c7b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 0x55d1086d23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.523+0000 7fda185c7b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086d2930 0x55d1086d2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.523+0000 7fda185c7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1085d30e0 con 0x55d1086cdbe0 2023-11-16T16:50:18.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.523+0000 7fda185c7b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d108625920 con 0x55d1086ce4f0 2023-11-16T16:50:18.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.523+0000 7fda185c7b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10861edb0 con 0x55d1086d2930 2023-11-16T16:50:18.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.524+0000 7fd9fe59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.524+0000 7fd9fe59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47830/0 (socket says 172.21.15.44:47830) 2023-11-16T16:50:18.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.524+0000 7fd9fe59c700 1 -- 172.21.15.44:0/2505707123 learned_addr learned my addr 172.21.15.44:0/2505707123 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:18.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.524+0000 7fd9fdd9b700 1 --2- 172.21.15.44:0/2505707123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 0x55d1086d23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.524+0000 7fd9fed9d700 1 --2- 172.21.15.44:0/2505707123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086d2930 0x55d1086d2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.524+0000 7fd9fe59c700 1 -- 172.21.15.44:0/2505707123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086d2930 msgr2=0x55d1086d2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:18.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.524+0000 7fd9fe59c700 1 --2- 172.21.15.44:0/2505707123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086d2930 0x55d1086d2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.524+0000 7fd9fe59c700 1 -- 172.21.15.44:0/2505707123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 msgr2=0x55d1086d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:18.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.524+0000 7fd9fe59c700 1 --2- 172.21.15.44:0/2505707123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 0x55d1086d23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.524+0000 7fd9fe59c700 1 -- 172.21.15.44:0/2505707123 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1084a06b0 con 0x55d1086cdbe0 2023-11-16T16:50:18.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.524+0000 7fd9fdd9b700 1 --2- 172.21.15.44:0/2505707123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 0x55d1086d23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:18.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.524+0000 7fd9fed9d700 1 --2- 172.21.15.44:0/2450008048 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd9e80a13e0 0x7fd9e80a3890 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fda00074f00 tx=0x7fd9f8009920 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:18.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.524+0000 7fd9fe59c700 1 --2- 172.21.15.44:0/2505707123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086cdfb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000c5b0 tx=0x7fd9f009a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4226df3983783d93 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.524+0000 7fd9ecff9700 1 -- 172.21.15.44:0/2505707123 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f009e170 con 0x55d1086cdbe0 2023-11-16T16:50:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.524+0000 7fd9ecff9700 1 -- 172.21.15.44:0/2505707123 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd9f000ce20 con 0x55d1086cdbe0 2023-11-16T16:50:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.525+0000 7fd9ecff9700 1 -- 172.21.15.44:0/2505707123 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f00cd570 con 0x55d1086cdbe0 2023-11-16T16:50:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.525+0000 7fda185c7b80 1 -- 172.21.15.44:0/2505707123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 msgr2=0x55d1086cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.525+0000 7fda185c7b80 1 --2- 172.21.15.44:0/2505707123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086cdfb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000c5b0 tx=0x7fd9f009a150 comp rx=0 tx=0).stop 2023-11-16T16:50:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.525+0000 7fda185c7b80 1 -- 172.21.15.44:0/2505707123 shutdown_connections 2023-11-16T16:50:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.525+0000 7fda185c7b80 1 --2- 172.21.15.44:0/2505707123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086d2930 0x55d1086d2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.525+0000 7fda185c7b80 1 --2- 172.21.15.44:0/2505707123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086cdfb0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.525+0000 7fda185c7b80 1 --2- 172.21.15.44:0/2505707123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 0x55d1086d23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.525+0000 7fda185c7b80 1 -- 172.21.15.44:0/2505707123 >> 172.21.15.44:0/2505707123 conn(0x55d1086c33e0 msgr2=0x55d1086c8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.525+0000 7fda185c7b80 1 -- 172.21.15.44:0/2505707123 shutdown_connections 2023-11-16T16:50:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.525+0000 7fda185c7b80 1 -- 172.21.15.44:0/2505707123 wait complete. 2023-11-16T16:50:18.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.526+0000 7fda185c7b80 1 -- start start 2023-11-16T16:50:18.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.526+0000 7fda185c7b80 1 Processor -- start 2023-11-16T16:50:18.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.526+0000 7fda185c7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086da0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.526+0000 7fda185c7b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 0x55d1086d73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.526+0000 7fda185c7b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086d2930 0x55d1086d7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.527+0000 7fda185c7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1085c0960 con 0x55d1086cdbe0 2023-11-16T16:50:18.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.527+0000 7fda185c7b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1085c0600 con 0x55d1086ce4f0 2023-11-16T16:50:18.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.527+0000 7fda185c7b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1084a29a0 con 0x55d1086d2930 2023-11-16T16:50:18.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.527+0000 7fda185c7b80 10 client.0 _ll_register_callbacks cb 0x55d1085de800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:18.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.527+0000 7fd9fe59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086da0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.527+0000 7fd9fe59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086da0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47836/0 (socket says 172.21.15.44:47836) 2023-11-16T16:50:18.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.527+0000 7fd9fe59c700 1 -- 172.21.15.44:0/1003246004 learned_addr learned my addr 172.21.15.44:0/1003246004 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:18.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.527+0000 7fd9fdd9b700 1 --2- 172.21.15.44:0/1003246004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 0x55d1086d73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.527+0000 7fd9fed9d700 1 --2- 172.21.15.44:0/1003246004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086d2930 0x55d1086d7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.527+0000 7fd9fe59c700 1 -- 172.21.15.44:0/1003246004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086d2930 msgr2=0x55d1086d7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:18.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.527+0000 7fd9fe59c700 1 --2- 172.21.15.44:0/1003246004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086d2930 0x55d1086d7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.527+0000 7fd9fe59c700 1 -- 172.21.15.44:0/1003246004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 msgr2=0x55d1086d73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:18.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.527+0000 7fd9fe59c700 1 --2- 172.21.15.44:0/1003246004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 0x55d1086d73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.527+0000 7fd9fe59c700 1 -- 172.21.15.44:0/1003246004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda00110370 con 0x55d1086cdbe0 2023-11-16T16:50:18.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.527+0000 7fd9fdd9b700 1 --2- 172.21.15.44:0/1003246004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 0x55d1086d73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:18.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.528+0000 7fd9fe59c700 1 --2- 172.21.15.44:0/1003246004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086da0e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000cfb0 tx=0x7fd9f009a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54581d32786aa536 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:18.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.528+0000 7fd9e5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:50:18.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.528+0000 7fd9e5ffb700 1 -- 172.21.15.44:0/1003246004 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f009a760 con 0x55d1086cdbe0 2023-11-16T16:50:18.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.528+0000 7fd9e5ffb700 1 -- 172.21.15.44:0/1003246004 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fd9f00b8030 con 0x55d1086cdbe0 2023-11-16T16:50:18.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.528+0000 7fda185c7b80 1 -- 172.21.15.44:0/1003246004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d1086c0ba0 con 0x55d1086cdbe0 2023-11-16T16:50:18.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.528+0000 7fd9e5ffb700 1 -- 172.21.15.44:0/1003246004 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9f00a2750 con 0x55d1086cdbe0 2023-11-16T16:50:18.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.528+0000 7fda185c7b80 10 client.15585 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:18.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.528+0000 7fda185c7b80 1 -- 172.21.15.44:0/1003246004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55d1085e5210 con 0x55d1086cdbe0 2023-11-16T16:50:18.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.528+0000 7fd9e5ffb700 1 -- 172.21.15.44:0/1003246004 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d1086c0ba0 con 0x55d1086cdbe0 2023-11-16T16:50:18.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.528+0000 7fd9e5ffb700 1 -- 172.21.15.44:0/1003246004 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd9f00a22c0 con 0x55d1086cdbe0 2023-11-16T16:50:18.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.529+0000 7fda185c7b80 10 client.15585 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:18.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.529+0000 7fda185c7b80 10 client.15585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:18.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.529+0000 7fda185c7b80 20 client.15585 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:18.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.529+0000 7fda185c7b80 1 --2- 172.21.15.44:0/1003246004 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d1086e75f0 0x55d1086e9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.529+0000 7fda185c7b80 4 client.15585 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:18.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.529+0000 7fda185c7b80 1 -- 172.21.15.44:0/1003246004 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d1085e1120 con 0x55d1086e75f0 2023-11-16T16:50:18.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.529+0000 7fd9fdd9b700 1 --2- 172.21.15.44:0/1003246004 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d1086e75f0 0x55d1086e9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.529+0000 7fd9fdd9b700 1 --2- 172.21.15.44:0/1003246004 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d1086e75f0 0x55d1086e9a00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62e390d308e583ac server_cookie=677a87af9cdbb99e in_seq=0 out_seq=0 2023-11-16T16:50:18.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.529+0000 7fd9e5ffb700 10 client.15585 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:18.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.530+0000 7fd9e5ffb700 1 -- 172.21.15.44:0/1003246004 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d1085e1120 con 0x55d1086e75f0 2023-11-16T16:50:18.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.530+0000 7fd9e5ffb700 10 client.15585 handle_command_reply: tid=0 2023-11-16T16:50:18.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.530+0000 7fda185c7b80 1 client.15585 shutdown 2023-11-16T16:50:18.536 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.530+0000 7fda185c7b80 1 -- 172.21.15.44:0/1003246004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 msgr2=0x55d1086da0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.530+0000 7fda185c7b80 1 --2- 172.21.15.44:0/1003246004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086da0e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000cfb0 tx=0x7fd9f009a300 comp rx=0 tx=0).stop 2023-11-16T16:50:18.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.530+0000 7fda185c7b80 1 -- 172.21.15.44:0/1003246004 shutdown_connections 2023-11-16T16:50:18.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.530+0000 7fda185c7b80 1 --2- 172.21.15.44:0/1003246004 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d1086e75f0 0x55d1086e9a00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.530+0000 7fda185c7b80 1 --2- 172.21.15.44:0/1003246004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086d2930 0x55d1086d7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.530+0000 7fda185c7b80 1 --2- 172.21.15.44:0/1003246004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086da0e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.530+0000 7fda185c7b80 1 --2- 172.21.15.44:0/1003246004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 0x55d1086d73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.530+0000 7fd9e5ffb700 0 client.15585 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:18.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.530+0000 7fda185c7b80 1 -- 172.21.15.44:0/1003246004 >> 172.21.15.44:0/1003246004 conn(0x55d1086c33e0 msgr2=0x55d1086d3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:18.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.531+0000 7fda185c7b80 1 -- 172.21.15.44:0/1003246004 shutdown_connections 2023-11-16T16:50:18.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.531+0000 7fda185c7b80 1 -- 172.21.15.44:0/1003246004 wait complete. 2023-11-16T16:50:18.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.531+0000 7fda185c7b80 20 client.15585 trim_cache size 0 max 16384 2023-11-16T16:50:18.550 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.548+0000 7fda185c7b80 1 Processor -- start 2023-11-16T16:50:18.550 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.548+0000 7fda185c7b80 1 -- start start 2023-11-16T16:50:18.551 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.549+0000 7fda185c7b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086cdbe0 0x55d1086d0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.551 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.549+0000 7fda185c7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086ce4f0 0x55d1086d20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.551 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.549+0000 7fda185c7b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086d2930 0x55d1086d9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.551 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.549+0000 7fda185c7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1086d5930 con 0x55d1086ce4f0 2023-11-16T16:50:18.551 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.549+0000 7fda185c7b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1086d15d0 con 0x55d1086d2930 2023-11-16T16:50:18.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.549+0000 7fda185c7b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1086d56d0 con 0x55d1086cdbe0 2023-11-16T16:50:18.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.549+0000 7fd9fe59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086cdbe0 0x55d1086d0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.549+0000 7fd9fdd9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086ce4f0 0x55d1086d20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.549+0000 7fd9fe59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086cdbe0 0x55d1086d0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:59090/0 (socket says 172.21.15.44:59090) 2023-11-16T16:50:18.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.549+0000 7fd9fe59c700 1 -- 172.21.15.44:0/1105578449 learned_addr learned my addr 172.21.15.44:0/1105578449 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:18.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.549+0000 7fd9fed9d700 1 --2- 172.21.15.44:0/1105578449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086d2930 0x55d1086d9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.549+0000 7fd9fdd9b700 1 -- 172.21.15.44:0/1105578449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086cdbe0 msgr2=0x55d1086d0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:18.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.549+0000 7fd9fdd9b700 1 --2- 172.21.15.44:0/1105578449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086cdbe0 0x55d1086d0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.549+0000 7fd9fdd9b700 1 -- 172.21.15.44:0/1105578449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086d2930 msgr2=0x55d1086d9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:18.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.549+0000 7fd9fdd9b700 1 --2- 172.21.15.44:0/1105578449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086d2930 0x55d1086d9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.549+0000 7fd9fdd9b700 1 -- 172.21.15.44:0/1105578449 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1085e4f60 con 0x55d1086ce4f0 2023-11-16T16:50:18.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.550+0000 7fd9fe59c700 1 --2- 172.21.15.44:0/1105578449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086cdbe0 0x55d1086d0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:18.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.550+0000 7fd9fed9d700 1 --2- 172.21.15.44:0/1105578449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086d2930 0x55d1086d9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:18.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.550+0000 7fd9fdd9b700 1 --2- 172.21.15.44:0/1105578449 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086ce4f0 0x55d1086d20f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fda000745e0 tx=0x7fd9f400b2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc7092a893a09ae server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:18.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.550+0000 7fd9ed7fa700 1 -- 172.21.15.44:0/1105578449 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f4029070 con 0x55d1086ce4f0 2023-11-16T16:50:18.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.550+0000 7fd9ed7fa700 1 -- 172.21.15.44:0/1105578449 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd9f400f890 con 0x55d1086ce4f0 2023-11-16T16:50:18.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.550+0000 7fd9ed7fa700 1 -- 172.21.15.44:0/1105578449 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f401c040 con 0x55d1086ce4f0 2023-11-16T16:50:18.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.551+0000 7fda185c7b80 1 -- 172.21.15.44:0/1105578449 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086ce4f0 msgr2=0x55d1086d20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.551+0000 7fda185c7b80 1 --2- 172.21.15.44:0/1105578449 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086ce4f0 0x55d1086d20f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fda000745e0 tx=0x7fd9f400b2e0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.551+0000 7fda185c7b80 1 -- 172.21.15.44:0/1105578449 shutdown_connections 2023-11-16T16:50:18.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.551+0000 7fda185c7b80 1 --2- 172.21.15.44:0/1105578449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086d2930 0x55d1086d9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.551+0000 7fda185c7b80 1 --2- 172.21.15.44:0/1105578449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086cdbe0 0x55d1086d0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.551+0000 7fda185c7b80 1 --2- 172.21.15.44:0/1105578449 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086ce4f0 0x55d1086d20f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.551+0000 7fda185c7b80 1 -- 172.21.15.44:0/1105578449 >> 172.21.15.44:0/1105578449 conn(0x55d1086c33e0 msgr2=0x55d1086d19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:18.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.551+0000 7fda185c7b80 1 -- 172.21.15.44:0/1105578449 shutdown_connections 2023-11-16T16:50:18.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.551+0000 7fda185c7b80 1 -- 172.21.15.44:0/1105578449 wait complete. 2023-11-16T16:50:18.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.551+0000 7fda185c7b80 1 -- start start 2023-11-16T16:50:18.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.551+0000 7fda185c7b80 1 Processor -- start 2023-11-16T16:50:18.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.552+0000 7fda185c7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086dec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.552+0000 7fda185c7b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 0x55d1086dd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.557 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.552+0000 7fda185c7b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086d2930 0x55d1086dd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.552+0000 7fda185c7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1086dde60 con 0x55d1086cdbe0 2023-11-16T16:50:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.552+0000 7fda185c7b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10873b480 con 0x55d1086ce4f0 2023-11-16T16:50:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.552+0000 7fda185c7b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10873b5c0 con 0x55d1086d2930 2023-11-16T16:50:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.552+0000 7fda185c7b80 10 client.0 _ll_register_callbacks cb 0x7fd9f00966d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.552+0000 7fd9fe59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086dec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.552+0000 7fd9fe59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086dec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47866/0 (socket says 172.21.15.44:47866) 2023-11-16T16:50:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.552+0000 7fd9fe59c700 1 -- 172.21.15.44:0/3559066460 learned_addr learned my addr 172.21.15.44:0/3559066460 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.552+0000 7fd9fdd9b700 1 --2- 172.21.15.44:0/3559066460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 0x55d1086dd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.552+0000 7fd9fed9d700 1 --2- 172.21.15.44:0/3559066460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086d2930 0x55d1086dd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.553+0000 7fd9fe59c700 1 -- 172.21.15.44:0/3559066460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086d2930 msgr2=0x55d1086dd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.553+0000 7fd9fe59c700 1 --2- 172.21.15.44:0/3559066460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086d2930 0x55d1086dd790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.553+0000 7fd9fe59c700 1 -- 172.21.15.44:0/3559066460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 msgr2=0x55d1086dd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:18.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.553+0000 7fd9fe59c700 1 --2- 172.21.15.44:0/3559066460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 0x55d1086dd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.553+0000 7fd9fe59c700 1 -- 172.21.15.44:0/3559066460 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d1085e5210 con 0x55d1086cdbe0 2023-11-16T16:50:18.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.553+0000 7fd9fe59c700 1 --2- 172.21.15.44:0/3559066460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086dec70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0096800 tx=0x7fd9f0095fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=158d3a487fa64a37 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:18.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.553+0000 7fd9e5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:50:18.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.553+0000 7fd9e5ffb700 1 -- 172.21.15.44:0/3559066460 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f00cafc0 con 0x55d1086cdbe0 2023-11-16T16:50:18.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.553+0000 7fd9e5ffb700 1 -- 172.21.15.44:0/3559066460 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fd9f00b8040 con 0x55d1086cdbe0 2023-11-16T16:50:18.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.553+0000 7fda185c7b80 1 -- 172.21.15.44:0/3559066460 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d1085ed6e0 con 0x55d1086cdbe0 2023-11-16T16:50:18.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.553+0000 7fd9e5ffb700 1 -- 172.21.15.44:0/3559066460 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9f009a370 con 0x55d1086cdbe0 2023-11-16T16:50:18.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.553+0000 7fda185c7b80 10 client.15591 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:18.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.553+0000 7fda185c7b80 1 -- 172.21.15.44:0/3559066460 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x7fda0010d9b0 con 0x55d1086cdbe0 2023-11-16T16:50:18.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.553+0000 7fd9e5ffb700 1 -- 172.21.15.44:0/3559066460 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d1085ed6e0 con 0x55d1086cdbe0 2023-11-16T16:50:18.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.554+0000 7fd9e5ffb700 1 -- 172.21.15.44:0/3559066460 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd9f0083b00 con 0x55d1086cdbe0 2023-11-16T16:50:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.554+0000 7fda185c7b80 10 client.15591 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.554+0000 7fda185c7b80 10 client.15591 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.554+0000 7fda185c7b80 20 client.15591 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.554+0000 7fda185c7b80 1 --2- 172.21.15.44:0/3559066460 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d10873c8f0 0x55d10873ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.554+0000 7fda185c7b80 4 client.15591 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:50:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.554+0000 7fda185c7b80 1 -- 172.21.15.44:0/3559066460 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d10846dff0 con 0x55d10873c8f0 2023-11-16T16:50:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.554+0000 7fd9fdd9b700 1 --2- 172.21.15.44:0/3559066460 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d10873c8f0 0x55d10873ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.555+0000 7fd9e5ffb700 10 client.15591 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.555+0000 7fd9fdd9b700 1 --2- 172.21.15.44:0/3559066460 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d10873c8f0 0x55d10873ed00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=114872a2119a5c35 server_cookie=1877009177b0f94e in_seq=0 out_seq=0 2023-11-16T16:50:18.561 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.556+0000 7fd9e5ffb700 1 -- 172.21.15.44:0/3559066460 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d10846dff0 con 0x55d10873c8f0 2023-11-16T16:50:18.561 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.556+0000 7fd9e5ffb700 10 client.15591 handle_command_reply: tid=0 2023-11-16T16:50:18.561 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.556+0000 7fda185c7b80 1 client.15591 shutdown 2023-11-16T16:50:18.561 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.556+0000 7fda185c7b80 1 -- 172.21.15.44:0/3559066460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 msgr2=0x55d1086dec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.561 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.556+0000 7fda185c7b80 1 --2- 172.21.15.44:0/3559066460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086dec70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0096800 tx=0x7fd9f0095fb0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.556+0000 7fda185c7b80 1 -- 172.21.15.44:0/3559066460 shutdown_connections 2023-11-16T16:50:18.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.556+0000 7fda185c7b80 1 --2- 172.21.15.44:0/3559066460 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d10873c8f0 0x55d10873ed00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.556+0000 7fda185c7b80 1 --2- 172.21.15.44:0/3559066460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d1086d2930 0x55d1086dd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.556+0000 7fda185c7b80 1 --2- 172.21.15.44:0/3559066460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d1086cdbe0 0x55d1086dec70 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.556+0000 7fda185c7b80 1 --2- 172.21.15.44:0/3559066460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1086ce4f0 0x55d1086dd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.556+0000 7fd9e5ffb700 0 client.15591 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:18.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.556+0000 7fda185c7b80 1 -- 172.21.15.44:0/3559066460 >> 172.21.15.44:0/3559066460 conn(0x55d1086c33e0 msgr2=0x55d1086cc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:18.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.556+0000 7fda185c7b80 1 -- 172.21.15.44:0/3559066460 shutdown_connections 2023-11-16T16:50:18.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.556+0000 7fda185c7b80 1 -- 172.21.15.44:0/3559066460 wait complete. 2023-11-16T16:50:18.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.556+0000 7fda185c7b80 20 client.15591 trim_cache size 0 max 16384 2023-11-16T16:50:18.563 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:50:18.563 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:50:18.563 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "0ebccee1-949f-48c0-8803-e70984593374", 2023-11-16T16:50:18.563 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:50:18.563 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:50:18.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.557+0000 7fda04bff700 1 -- 172.21.15.44:0/2450008048 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd9e80a13e0 msgr2=0x7fd9e80a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.557+0000 7fda04bff700 1 --2- 172.21.15.44:0/2450008048 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd9e80a13e0 0x7fd9e80a3890 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fda00074f00 tx=0x7fd9f8009920 comp rx=0 tx=0).stop 2023-11-16T16:50:18.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.557+0000 7fda04bff700 1 -- 172.21.15.44:0/2450008048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda00100cc0 msgr2=0x7fda00072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.557+0000 7fda04bff700 1 --2- 172.21.15.44:0/2450008048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda00100cc0 0x7fda00072df0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000eb90 tx=0x7fd9f000eea0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.557+0000 7fda04bff700 1 -- 172.21.15.44:0/2450008048 shutdown_connections 2023-11-16T16:50:18.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.557+0000 7fda04bff700 1 --2- 172.21.15.44:0/2450008048 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd9e80a13e0 0x7fd9e80a3890 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.557+0000 7fda04bff700 1 --2- 172.21.15.44:0/2450008048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda00105620 0x7fda001a4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.557+0000 7fda04bff700 1 --2- 172.21.15.44:0/2450008048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda00100cc0 0x7fda00072df0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.557+0000 7fda04bff700 1 --2- 172.21.15.44:0/2450008048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00101660 0x7fda00073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.557+0000 7fda04bff700 1 -- 172.21.15.44:0/2450008048 >> 172.21.15.44:0/2450008048 conn(0x7fda00077560 msgr2=0x7fda000ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:18.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.557+0000 7fda04bff700 1 -- 172.21.15.44:0/2450008048 shutdown_connections 2023-11-16T16:50:18.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.557+0000 7fda04bff700 1 -- 172.21.15.44:0/2450008048 wait complete. 2023-11-16T16:50:18.577 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:50:18.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.786+0000 7f2eddfff700 1 -- 172.21.15.44:0/1119543546 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ed8101390 msgr2=0x7f2ed810d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.786+0000 7f2eddfff700 1 --2- 172.21.15.44:0/1119543546 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ed8101390 0x7f2ed810d020 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000b0d0 tx=0x7f2ed000b3e0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.786+0000 7f2eddfff700 1 -- 172.21.15.44:0/1119543546 shutdown_connections 2023-11-16T16:50:18.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.786+0000 7f2eddfff700 1 --2- 172.21.15.44:0/1119543546 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ed8101390 0x7f2ed810d020 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.786+0000 7f2eddfff700 1 --2- 172.21.15.44:0/1119543546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed81051b0 0x7f2ed8105580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.786+0000 7f2eddfff700 1 --2- 172.21.15.44:0/1119543546 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ed81009e0 0x7f2ed8100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.786+0000 7f2eddfff700 1 -- 172.21.15.44:0/1119543546 >> 172.21.15.44:0/1119543546 conn(0x7f2ed8077530 msgr2=0x7f2ed8077930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:18.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.786+0000 7f2eddfff700 1 -- 172.21.15.44:0/1119543546 shutdown_connections 2023-11-16T16:50:18.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.786+0000 7f2eddfff700 1 -- 172.21.15.44:0/1119543546 wait complete. 2023-11-16T16:50:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.787+0000 7f2eddfff700 1 Processor -- start 2023-11-16T16:50:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.788+0000 7f2eddfff700 1 -- start start 2023-11-16T16:50:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.788+0000 7f2eddfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ed81009e0 0x7f2ed81986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.788+0000 7f2eddfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed8101390 0x7f2ed8198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.788+0000 7f2eddfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ed81051b0 0x7f2ed819bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.788+0000 7f2eddfff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed819c590 con 0x7f2ed81051b0 2023-11-16T16:50:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.788+0000 7f2eddfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed819c700 con 0x7f2ed8101390 2023-11-16T16:50:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.788+0000 7f2eddfff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed819c8c0 con 0x7f2ed81009e0 2023-11-16T16:50:18.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.788+0000 7f2ed7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ed81051b0 0x7f2ed819bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.788+0000 7f2ed77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ed81009e0 0x7f2ed81986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.788+0000 7f2ed77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ed81009e0 0x7f2ed81986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:59116/0 (socket says 172.21.15.44:59116) 2023-11-16T16:50:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.788+0000 7f2ed77fe700 1 -- 172.21.15.44:0/1613416604 learned_addr learned my addr 172.21.15.44:0/1613416604 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.788+0000 7f2ecffff700 1 --2- 172.21.15.44:0/1613416604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed8101390 0x7f2ed8198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.789+0000 7f2ed7fff700 1 -- 172.21.15.44:0/1613416604 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ed81009e0 msgr2=0x7f2ed81986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.789+0000 7f2ed7fff700 1 --2- 172.21.15.44:0/1613416604 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ed81009e0 0x7f2ed81986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.789+0000 7f2ed7fff700 1 -- 172.21.15.44:0/1613416604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed8101390 msgr2=0x7f2ed8198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.789+0000 7f2ed7fff700 1 --2- 172.21.15.44:0/1613416604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed8101390 0x7f2ed8198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.789+0000 7f2ed7fff700 1 -- 172.21.15.44:0/1613416604 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec8009520 con 0x7f2ed81051b0 2023-11-16T16:50:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.789+0000 7f2ed7fff700 1 --2- 172.21.15.44:0/1613416604 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ed81051b0 0x7f2ed819bf90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009fd0 tx=0x7f2ed00077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.789+0000 7f2ed57fa700 1 -- 172.21.15.44:0/1613416604 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed000e040 con 0x7f2ed81051b0 2023-11-16T16:50:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.789+0000 7f2ed57fa700 1 -- 172.21.15.44:0/1613416604 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ed000bcb0 con 0x7f2ed81051b0 2023-11-16T16:50:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.789+0000 7f2eddfff700 1 -- 172.21.15.44:0/1613416604 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed0009d00 con 0x7f2ed81051b0 2023-11-16T16:50:18.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.789+0000 7f2ed57fa700 1 -- 172.21.15.44:0/1613416604 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0005840 con 0x7f2ed81051b0 2023-11-16T16:50:18.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.789+0000 7f2eddfff700 1 -- 172.21.15.44:0/1613416604 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed81103b0 con 0x7f2ed81051b0 2023-11-16T16:50:18.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.792+0000 7f2ed57fa700 1 -- 172.21.15.44:0/1613416604 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f2ed0004a50 con 0x7f2ed81051b0 2023-11-16T16:50:18.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.794+0000 7f2ef1931b80 1 Processor -- start 2023-11-16T16:50:18.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.794+0000 7f2ef1931b80 1 -- start start 2023-11-16T16:50:18.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.794+0000 7f2ed57fa700 1 --2- 172.21.15.44:0/1613416604 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2ec40a13e0 0x7f2ec40a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.794+0000 7f2ef1931b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a8878b60 0x5573a8878f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.794+0000 7f2ed57fa700 1 -- 172.21.15.44:0/1613416604 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f2ed00ce450 con 0x7f2ed81051b0 2023-11-16T16:50:18.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.794+0000 7f2ef1931b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8879470 0x5573a887d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.794+0000 7f2ef1931b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a887d8b0 0x5573a887dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.794+0000 7f2ef1931b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573a877e060 con 0x5573a8879470 2023-11-16T16:50:18.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.794+0000 7f2ef1931b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573a87d08a0 con 0x5573a887d8b0 2023-11-16T16:50:18.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.794+0000 7f2ef1931b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573a87c9d30 con 0x5573a8878b60 2023-11-16T16:50:18.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.794+0000 7f2ecffff700 1 --2- 172.21.15.44:0/1613416604 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2ec40a13e0 0x7f2ec40a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.794+0000 7f2ed77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a8878b60 0x5573a8878f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.795+0000 7f2ed77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a8878b60 0x5573a8878f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:59122/0 (socket says 172.21.15.44:59122) 2023-11-16T16:50:18.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.795+0000 7f2ed77fe700 1 -- 172.21.15.44:0/4200742648 learned_addr learned my addr 172.21.15.44:0/4200742648 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:18.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.795+0000 7f2ecffff700 1 --2- 172.21.15.44:0/4200742648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8879470 0x5573a887d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.795+0000 7f2ed7fff700 1 --2- 172.21.15.44:0/4200742648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a887d8b0 0x5573a887dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.795+0000 7f2ed77fe700 1 -- 172.21.15.44:0/4200742648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a887d8b0 msgr2=0x5573a887dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.795+0000 7f2ed77fe700 1 --2- 172.21.15.44:0/4200742648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a887d8b0 0x5573a887dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.795+0000 7f2ed77fe700 1 -- 172.21.15.44:0/4200742648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8879470 msgr2=0x5573a887d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.795+0000 7f2ed77fe700 1 --2- 172.21.15.44:0/4200742648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8879470 0x5573a887d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.795+0000 7f2ed77fe700 1 -- 172.21.15.44:0/4200742648 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573a864b630 con 0x5573a8878b60 2023-11-16T16:50:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.795+0000 7f2ed7fff700 1 --2- 172.21.15.44:0/4200742648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a887d8b0 0x5573a887dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:50:18.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.795+0000 7f2ecffff700 1 --2- 172.21.15.44:0/1613416604 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2ec40a13e0 0x7f2ec40a3890 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8199cb0 tx=0x7f2ec0009bf0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.795+0000 7f2ed77fe700 1 --2- 172.21.15.44:0/4200742648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a8878b60 0x5573a8878f30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80114c0 tx=0x7f2ec80117d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca781612fa057aaa server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.795+0000 7f2ecdffb700 1 -- 172.21.15.44:0/4200742648 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec8011230 con 0x5573a8878b60 2023-11-16T16:50:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.796+0000 7f2ecdffb700 1 -- 172.21.15.44:0/4200742648 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ec80108f0 con 0x5573a8878b60 2023-11-16T16:50:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.796+0000 7f2ecdffb700 1 -- 172.21.15.44:0/4200742648 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec80198a0 con 0x5573a8878b60 2023-11-16T16:50:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.796+0000 7f2ef1931b80 1 -- 172.21.15.44:0/4200742648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a8878b60 msgr2=0x5573a8878f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.796+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/4200742648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a8878b60 0x5573a8878f30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80114c0 tx=0x7f2ec80117d0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.796+0000 7f2ef1931b80 1 -- 172.21.15.44:0/4200742648 shutdown_connections 2023-11-16T16:50:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.796+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/4200742648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a887d8b0 0x5573a887dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.796+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/4200742648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a8878b60 0x5573a8878f30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.796+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/4200742648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8879470 0x5573a887d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.796+0000 7f2ef1931b80 1 -- 172.21.15.44:0/4200742648 >> 172.21.15.44:0/4200742648 conn(0x5573a886e360 msgr2=0x5573a8873b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:18.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.796+0000 7f2ef1931b80 1 -- 172.21.15.44:0/4200742648 shutdown_connections 2023-11-16T16:50:18.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.797+0000 7f2ef1931b80 1 -- 172.21.15.44:0/4200742648 wait complete. 2023-11-16T16:50:18.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.797+0000 7f2ef1931b80 1 -- start start 2023-11-16T16:50:18.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.797+0000 7f2ef1931b80 1 Processor -- start 2023-11-16T16:50:18.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.798+0000 7f2ef1931b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8878b60 0x5573a8885070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.798+0000 7f2ef1931b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a8879470 0x5573a88822f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.798+0000 7f2ef1931b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a887d8b0 0x5573a8882920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.798+0000 7f2ef1931b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573a876b8e0 con 0x5573a8878b60 2023-11-16T16:50:18.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.798+0000 7f2ef1931b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573a876b580 con 0x5573a887d8b0 2023-11-16T16:50:18.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.798+0000 7f2ef1931b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573a864d920 con 0x5573a8879470 2023-11-16T16:50:18.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.798+0000 7f2ef1931b80 10 client.0 _ll_register_callbacks cb 0x5573a8789780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:18.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.799+0000 7f2ed77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8878b60 0x5573a8885070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.799+0000 7f2ed77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8878b60 0x5573a8885070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47890/0 (socket says 172.21.15.44:47890) 2023-11-16T16:50:18.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.799+0000 7f2ed77fe700 1 -- 172.21.15.44:0/1955867768 learned_addr learned my addr 172.21.15.44:0/1955867768 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:18.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.799+0000 7f2ecffff700 1 --2- 172.21.15.44:0/1955867768 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a8879470 0x5573a88822f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.799+0000 7f2ed7fff700 1 --2- 172.21.15.44:0/1955867768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a887d8b0 0x5573a8882920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.799+0000 7f2ed77fe700 1 -- 172.21.15.44:0/1955867768 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a8879470 msgr2=0x5573a88822f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:18.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.799+0000 7f2ed77fe700 1 --2- 172.21.15.44:0/1955867768 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a8879470 0x5573a88822f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.799+0000 7f2ed77fe700 1 -- 172.21.15.44:0/1955867768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a887d8b0 msgr2=0x5573a8882920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:18.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.799+0000 7f2ed77fe700 1 --2- 172.21.15.44:0/1955867768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a887d8b0 0x5573a8882920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.799+0000 7f2ed77fe700 1 -- 172.21.15.44:0/1955867768 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec8009c90 con 0x5573a8878b60 2023-11-16T16:50:18.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.799+0000 7f2ed77fe700 1 --2- 172.21.15.44:0/1955867768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8878b60 0x5573a8885070 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80097a0 tx=0x7f2ec80104b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c186cf416d0982c9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:18.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.799+0000 7f2eb6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:50:18.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.800+0000 7f2eb6ffd700 1 -- 172.21.15.44:0/1955867768 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec8010750 con 0x5573a8878b60 2023-11-16T16:50:18.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.800+0000 7f2ef1931b80 1 -- 172.21.15.44:0/1955867768 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5573a886bb20 con 0x5573a8878b60 2023-11-16T16:50:18.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.800+0000 7f2eb6ffd700 1 -- 172.21.15.44:0/1955867768 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f2ec8011140 con 0x5573a8878b60 2023-11-16T16:50:18.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.800+0000 7f2eb6ffd700 1 -- 172.21.15.44:0/1955867768 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ec8011300 con 0x5573a8878b60 2023-11-16T16:50:18.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.800+0000 7f2ef1931b80 10 client.15603 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:18.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.800+0000 7f2ef1931b80 1 -- 172.21.15.44:0/1955867768 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5573a8790190 con 0x5573a8878b60 2023-11-16T16:50:18.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.800+0000 7f2eb6ffd700 1 -- 172.21.15.44:0/1955867768 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5573a886bb20 con 0x5573a8878b60 2023-11-16T16:50:18.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.800+0000 7f2eb6ffd700 1 -- 172.21.15.44:0/1955867768 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2ec8019480 con 0x5573a8878b60 2023-11-16T16:50:18.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.800+0000 7f2ef1931b80 10 client.15603 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:18.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.800+0000 7f2ef1931b80 10 client.15603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:18.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.800+0000 7f2ef1931b80 20 client.15603 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:18.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.801+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/1955867768 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5573a88926a0 0x5573a8894ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.801+0000 7f2ef1931b80 4 client.15603 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:18.806 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.801+0000 7f2ef1931b80 1 -- 172.21.15.44:0/1955867768 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5573a878c0a0 con 0x5573a88926a0 2023-11-16T16:50:18.806 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.801+0000 7f2ecffff700 1 --2- 172.21.15.44:0/1955867768 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5573a88926a0 0x5573a8894ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.806 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.801+0000 7f2ecffff700 1 --2- 172.21.15.44:0/1955867768 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5573a88926a0 0x5573a8894ab0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ca5e625ecbbc116 server_cookie=ac1bef568b8f303c in_seq=0 out_seq=0 2023-11-16T16:50:18.806 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.801+0000 7f2eb6ffd700 10 client.15603 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:18.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.802+0000 7f2eb6ffd700 1 -- 172.21.15.44:0/1955867768 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5573a878c0a0 con 0x5573a88926a0 2023-11-16T16:50:18.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.802+0000 7f2eb6ffd700 10 client.15603 handle_command_reply: tid=0 2023-11-16T16:50:18.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.802+0000 7f2ef1931b80 1 client.15603 shutdown 2023-11-16T16:50:18.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.802+0000 7f2ef1931b80 1 -- 172.21.15.44:0/1955867768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8878b60 msgr2=0x5573a8885070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.802+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/1955867768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8878b60 0x5573a8885070 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80097a0 tx=0x7f2ec80104b0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.802+0000 7f2ef1931b80 1 -- 172.21.15.44:0/1955867768 shutdown_connections 2023-11-16T16:50:18.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.802+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/1955867768 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5573a88926a0 0x5573a8894ab0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.802+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/1955867768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a887d8b0 0x5573a8882920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.802+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/1955867768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8878b60 0x5573a8885070 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.802+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/1955867768 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a8879470 0x5573a88822f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.802+0000 7f2eb6ffd700 0 client.15603 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:18.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.802+0000 7f2ef1931b80 1 -- 172.21.15.44:0/1955867768 >> 172.21.15.44:0/1955867768 conn(0x5573a886e360 msgr2=0x5573a887ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:18.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.803+0000 7f2ef1931b80 1 -- 172.21.15.44:0/1955867768 shutdown_connections 2023-11-16T16:50:18.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.803+0000 7f2ef1931b80 1 -- 172.21.15.44:0/1955867768 wait complete. 2023-11-16T16:50:18.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.803+0000 7f2ef1931b80 20 client.15603 trim_cache size 0 max 16384 2023-11-16T16:50:18.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.820+0000 7f2ef1931b80 1 Processor -- start 2023-11-16T16:50:18.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.820+0000 7f2ef1931b80 1 -- start start 2023-11-16T16:50:18.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.821+0000 7f2ef1931b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a8878b60 0x5573a888ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.821+0000 7f2ef1931b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8879470 0x5573a8884c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.821+0000 7f2ef1931b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a887d8b0 0x5573a8881be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.821+0000 7f2ef1931b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573a88806a0 con 0x5573a8879470 2023-11-16T16:50:18.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.821+0000 7f2ef1931b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573a886ba10 con 0x5573a8878b60 2023-11-16T16:50:18.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.821+0000 7f2ef1931b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573a8879bf0 con 0x5573a887d8b0 2023-11-16T16:50:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.821+0000 7f2ecffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8879470 0x5573a8884c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.821+0000 7f2ed77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a8878b60 0x5573a888ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.821+0000 7f2ecffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8879470 0x5573a8884c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47894/0 (socket says 172.21.15.44:47894) 2023-11-16T16:50:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.821+0000 7f2ed77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a8878b60 0x5573a888ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:36478/0 (socket says 172.21.15.44:36478) 2023-11-16T16:50:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.821+0000 7f2ecffff700 1 -- 172.21.15.44:0/1452645282 learned_addr learned my addr 172.21.15.44:0/1452645282 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.821+0000 7f2ed7fff700 1 --2- 172.21.15.44:0/1452645282 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a887d8b0 0x5573a8881be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.821+0000 7f2ecffff700 1 -- 172.21.15.44:0/1452645282 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a887d8b0 msgr2=0x5573a8881be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.821+0000 7f2ecffff700 1 --2- 172.21.15.44:0/1452645282 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a887d8b0 0x5573a8881be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.821+0000 7f2ecffff700 1 -- 172.21.15.44:0/1452645282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a8878b60 msgr2=0x5573a888ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.821+0000 7f2ecffff700 1 --2- 172.21.15.44:0/1452645282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a8878b60 0x5573a888ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.822+0000 7f2ecffff700 1 -- 172.21.15.44:0/1452645282 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573a878fee0 con 0x5573a8879470 2023-11-16T16:50:18.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.822+0000 7f2ed77fe700 1 --2- 172.21.15.44:0/1452645282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a8878b60 0x5573a888ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:18.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.822+0000 7f2ecffff700 1 --2- 172.21.15.44:0/1452645282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8879470 0x5573a8884c10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x5573a88838a0 tx=0x7f2ec0021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38ac9d52b69c00c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:18.827 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.822+0000 7f2ece7fc700 1 -- 172.21.15.44:0/1452645282 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0016040 con 0x5573a8879470 2023-11-16T16:50:18.827 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.822+0000 7f2ece7fc700 1 -- 172.21.15.44:0/1452645282 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ec002bdd0 con 0x5573a8879470 2023-11-16T16:50:18.827 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.822+0000 7f2ece7fc700 1 -- 172.21.15.44:0/1452645282 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec000f980 con 0x5573a8879470 2023-11-16T16:50:18.827 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.823+0000 7f2ef1931b80 1 -- 172.21.15.44:0/1452645282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8879470 msgr2=0x5573a8884c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.827 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.823+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/1452645282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8879470 0x5573a8884c10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x5573a88838a0 tx=0x7f2ec0021000 comp rx=0 tx=0).stop 2023-11-16T16:50:18.827 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.823+0000 7f2ef1931b80 1 -- 172.21.15.44:0/1452645282 shutdown_connections 2023-11-16T16:50:18.827 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.823+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/1452645282 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a887d8b0 0x5573a8881be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.827 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.823+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/1452645282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a8878b60 0x5573a888ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.827 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.823+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/1452645282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8879470 0x5573a8884c10 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.823+0000 7f2ef1931b80 1 -- 172.21.15.44:0/1452645282 >> 172.21.15.44:0/1452645282 conn(0x5573a886e360 msgr2=0x5573a8872550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:18.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.823+0000 7f2ef1931b80 1 -- 172.21.15.44:0/1452645282 shutdown_connections 2023-11-16T16:50:18.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.823+0000 7f2ef1931b80 1 -- 172.21.15.44:0/1452645282 wait complete. 2023-11-16T16:50:18.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.823+0000 7f2ef1931b80 1 -- start start 2023-11-16T16:50:18.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.823+0000 7f2ef1931b80 1 Processor -- start 2023-11-16T16:50:18.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.824+0000 7f2ef1931b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8878b60 0x5573a8889710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.824+0000 7f2ef1931b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a8879470 0x5573a8887a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.824+0000 7f2ef1931b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a887d8b0 0x5573a8887f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.824+0000 7f2ef1931b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573a8889fe0 con 0x5573a8878b60 2023-11-16T16:50:18.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.824+0000 7f2ef1931b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573a88884b0 con 0x5573a8879470 2023-11-16T16:50:18.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.824+0000 7f2ef1931b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573a8888620 con 0x5573a887d8b0 2023-11-16T16:50:18.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.824+0000 7f2ef1931b80 10 client.0 _ll_register_callbacks cb 0x7f2ec0011500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:18.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.824+0000 7f2ed77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8878b60 0x5573a8889710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.824+0000 7f2ed77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8878b60 0x5573a8889710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47898/0 (socket says 172.21.15.44:47898) 2023-11-16T16:50:18.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.824+0000 7f2ed77fe700 1 -- 172.21.15.44:0/2946820407 learned_addr learned my addr 172.21.15.44:0/2946820407 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:18.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.824+0000 7f2ecffff700 1 --2- 172.21.15.44:0/2946820407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a8879470 0x5573a8887a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.824+0000 7f2ed7fff700 1 --2- 172.21.15.44:0/2946820407 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a887d8b0 0x5573a8887f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.825+0000 7f2ed77fe700 1 -- 172.21.15.44:0/2946820407 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a887d8b0 msgr2=0x5573a8887f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:18.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.825+0000 7f2ed77fe700 1 --2- 172.21.15.44:0/2946820407 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a887d8b0 0x5573a8887f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.825+0000 7f2ed77fe700 1 -- 172.21.15.44:0/2946820407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a8879470 msgr2=0x5573a8887a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:18.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.825+0000 7f2ed77fe700 1 --2- 172.21.15.44:0/2946820407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a8879470 0x5573a8887a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.825+0000 7f2ed77fe700 1 -- 172.21.15.44:0/2946820407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5573a8790190 con 0x5573a8878b60 2023-11-16T16:50:18.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.825+0000 7f2ed77fe700 1 --2- 172.21.15.44:0/2946820407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8878b60 0x5573a8889710 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80198e0 tx=0x7f2ec80117d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=735866c794278aba server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:18.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.825+0000 7f2eb6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:50:18.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.825+0000 7f2eb6ffd700 1 -- 172.21.15.44:0/2946820407 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec8019b50 con 0x5573a8878b60 2023-11-16T16:50:18.830 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.825+0000 7f2eb6ffd700 1 -- 172.21.15.44:0/2946820407 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f2ec80105d0 con 0x5573a8878b60 2023-11-16T16:50:18.830 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.825+0000 7f2ef1931b80 1 -- 172.21.15.44:0/2946820407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2ec80108f0 con 0x5573a8878b60 2023-11-16T16:50:18.830 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.825+0000 7f2eb6ffd700 1 -- 172.21.15.44:0/2946820407 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ec8010790 con 0x5573a8878b60 2023-11-16T16:50:18.830 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.825+0000 7f2ef1931b80 10 client.15609 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:18.830 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.825+0000 7f2ef1931b80 1 -- 172.21.15.44:0/2946820407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x7f2ec0025040 con 0x5573a8878b60 2023-11-16T16:50:18.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.826+0000 7f2eb6ffd700 1 -- 172.21.15.44:0/2946820407 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ec80108f0 con 0x5573a8878b60 2023-11-16T16:50:18.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.826+0000 7f2eb6ffd700 1 -- 172.21.15.44:0/2946820407 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2ec80111d0 con 0x5573a8878b60 2023-11-16T16:50:18.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.826+0000 7f2ef1931b80 10 client.15609 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:18.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.826+0000 7f2ef1931b80 10 client.15609 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:18.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.826+0000 7f2ef1931b80 20 client.15609 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:18.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.826+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/2946820407 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5573a88e7700 0x5573a88e9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:18.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.826+0000 7f2ef1931b80 4 client.15609 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:50:18.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.826+0000 7f2ef1931b80 1 -- 172.21.15.44:0/2946820407 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5573a8798660 con 0x5573a88e7700 2023-11-16T16:50:18.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.826+0000 7f2ecffff700 1 --2- 172.21.15.44:0/2946820407 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5573a88e7700 0x5573a88e9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:18.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.827+0000 7f2ecffff700 1 --2- 172.21.15.44:0/2946820407 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5573a88e7700 0x5573a88e9b10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5bf0686c991bfa4 server_cookie=f43c1d8d5c83c010 in_seq=0 out_seq=0 2023-11-16T16:50:18.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.827+0000 7f2eb6ffd700 10 client.15609 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:18.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.827+0000 7f2eb6ffd700 1 -- 172.21.15.44:0/2946820407 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5573a8798660 con 0x5573a88e7700 2023-11-16T16:50:18.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.827+0000 7f2eb6ffd700 10 client.15609 handle_command_reply: tid=0 2023-11-16T16:50:18.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.827+0000 7f2ef1931b80 1 client.15609 shutdown 2023-11-16T16:50:18.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.827+0000 7f2ef1931b80 1 -- 172.21.15.44:0/2946820407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8878b60 msgr2=0x5573a8889710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.827+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/2946820407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8878b60 0x5573a8889710 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80198e0 tx=0x7f2ec80117d0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.827+0000 7f2ef1931b80 1 -- 172.21.15.44:0/2946820407 shutdown_connections 2023-11-16T16:50:18.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.827+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/2946820407 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5573a88e7700 0x5573a88e9b10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.827+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/2946820407 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5573a887d8b0 0x5573a8887f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.827+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/2946820407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5573a8878b60 0x5573a8889710 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.827+0000 7f2eb6ffd700 0 client.15609 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:18.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.827+0000 7f2ef1931b80 1 --2- 172.21.15.44:0/2946820407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5573a8879470 0x5573a8887a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.827+0000 7f2ef1931b80 1 -- 172.21.15.44:0/2946820407 >> 172.21.15.44:0/2946820407 conn(0x5573a886e360 msgr2=0x5573a887b930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:18.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.827+0000 7f2ef1931b80 1 -- 172.21.15.44:0/2946820407 shutdown_connections 2023-11-16T16:50:18.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.827+0000 7f2ef1931b80 1 -- 172.21.15.44:0/2946820407 wait complete. 2023-11-16T16:50:18.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.828+0000 7f2ef1931b80 20 client.15609 trim_cache size 0 max 16384 2023-11-16T16:50:18.834 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:50:18.834 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2023-11-16T16:50:18.834 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2023-11-16T16:50:18.835 INFO:teuthology.orchestra.run.smithi044.stdout: "0ebccee1-949f-48c0-8803-e70984593374": { 2023-11-16T16:50:18.835 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2023-11-16T16:50:18.835 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "0ebccee1-949f-48c0-8803-e70984593374", 2023-11-16T16:50:18.835 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2023-11-16T16:50:18.835 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:50:18.835 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:50:18.835 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:50:18.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.828+0000 7f2eddfff700 1 -- 172.21.15.44:0/1613416604 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2ec40a13e0 msgr2=0x7f2ec40a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.828+0000 7f2eddfff700 1 --2- 172.21.15.44:0/1613416604 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2ec40a13e0 0x7f2ec40a3890 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8199cb0 tx=0x7f2ec0009bf0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.828+0000 7f2eddfff700 1 -- 172.21.15.44:0/1613416604 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ed81051b0 msgr2=0x7f2ed819bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:18.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.828+0000 7f2eddfff700 1 --2- 172.21.15.44:0/1613416604 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ed81051b0 0x7f2ed819bf90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009fd0 tx=0x7f2ed00077f0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.828+0000 7f2eddfff700 1 -- 172.21.15.44:0/1613416604 shutdown_connections 2023-11-16T16:50:18.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.828+0000 7f2eddfff700 1 --2- 172.21.15.44:0/1613416604 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ed81051b0 0x7f2ed819bf90 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.828+0000 7f2eddfff700 1 --2- 172.21.15.44:0/1613416604 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2ec40a13e0 0x7f2ec40a3890 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.828+0000 7f2eddfff700 1 --2- 172.21.15.44:0/1613416604 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ed81009e0 0x7f2ed81986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.828+0000 7f2eddfff700 1 --2- 172.21.15.44:0/1613416604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed8101390 0x7f2ed8198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:18.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.828+0000 7f2eddfff700 1 -- 172.21.15.44:0/1613416604 >> 172.21.15.44:0/1613416604 conn(0x7f2ed8077530 msgr2=0x7f2ed80fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:18.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.828+0000 7f2eddfff700 1 -- 172.21.15.44:0/1613416604 shutdown_connections 2023-11-16T16:50:18.837 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:18.828+0000 7f2eddfff700 1 -- 172.21.15.44:0/1613416604 wait complete. 2023-11-16T16:50:18.848 INFO:tasks.cephfs.filesystem:scrub status for tag:0ebccee1-949f-48c0-8803-e70984593374 - {'path': '/', 'tag': '0ebccee1-949f-48c0-8803-e70984593374', 'options': 'recursive,force'} 2023-11-16T16:50:18.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:18 smithi077 ceph-mon[131204]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 2.6 KiB/s wr, 7 op/s 2023-11-16T16:50:19.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:18 smithi044 ceph-mon[128750]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 2.6 KiB/s wr, 7 op/s 2023-11-16T16:50:19.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:18 smithi176 ceph-mon[131630]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 2.6 KiB/s wr, 7 op/s 2023-11-16T16:50:19.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:19 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:50:19.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:19 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:19 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:50:19.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:19 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:20.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:19 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:50:20.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:19 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:20.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:19 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:50:20.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:19 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:20.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:19 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:50:20.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:19 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:20.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:19 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:50:20.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:19 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:20.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:20 smithi077 ceph-mon[131204]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.8 KiB/s wr, 6 op/s 2023-11-16T16:50:21.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:20 smithi044 ceph-mon[128750]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.8 KiB/s wr, 6 op/s 2023-11-16T16:50:21.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:20 smithi176 ceph-mon[131630]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.8 KiB/s wr, 6 op/s 2023-11-16T16:50:22.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:22 smithi077 ceph-mon[131204]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-11-16T16:50:22.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:22 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:50:23.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:22 smithi044 ceph-mon[128750]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-11-16T16:50:23.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:22 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:50:23.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:22 smithi176 ceph-mon[131630]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-11-16T16:50:23.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:22 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:50:24.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:24 smithi077 ceph-mon[131204]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2023-11-16T16:50:24.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:24 smithi044 ceph-mon[128750]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2023-11-16T16:50:24.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:24 smithi176 ceph-mon[131630]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2023-11-16T16:50:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.908+0000 7f200e7ff700 1 -- 172.21.15.44:0/1020388384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20081055a0 msgr2=0x7f200810d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.908+0000 7f200e7ff700 1 --2- 172.21.15.44:0/1020388384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20081055a0 0x7f200810d170 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f200000b0d0 tx=0x7f200000b3e0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.908+0000 7f200e7ff700 1 -- 172.21.15.44:0/1020388384 shutdown_connections 2023-11-16T16:50:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.908+0000 7f200e7ff700 1 --2- 172.21.15.44:0/1020388384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20081055a0 0x7f200810d170 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.908+0000 7f200e7ff700 1 --2- 172.21.15.44:0/1020388384 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2008100560 0x7f2008100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.908+0000 7f200e7ff700 1 --2- 172.21.15.44:0/1020388384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2008100f00 0x7f2008104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.908+0000 7f200e7ff700 1 -- 172.21.15.44:0/1020388384 >> 172.21.15.44:0/1020388384 conn(0x7f20080781b0 msgr2=0x7f20080785b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.909+0000 7f200e7ff700 1 -- 172.21.15.44:0/1020388384 shutdown_connections 2023-11-16T16:50:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.909+0000 7f200e7ff700 1 -- 172.21.15.44:0/1020388384 wait complete. 2023-11-16T16:50:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.910+0000 7f200e7ff700 1 Processor -- start 2023-11-16T16:50:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.910+0000 7f200e7ff700 1 -- start start 2023-11-16T16:50:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.910+0000 7f200e7ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2008100560 0x7f2008198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.910+0000 7f200e7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2008100f00 0x7f2008198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.910+0000 7f200e7ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20081055a0 0x7f200819c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.910+0000 7f200e7ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200819c610 con 0x7f20081055a0 2023-11-16T16:50:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.910+0000 7f200e7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200819c780 con 0x7f2008100f00 2023-11-16T16:50:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.910+0000 7f200e7ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200819c940 con 0x7f2008100560 2023-11-16T16:50:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.911+0000 7f200cd9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20081055a0 0x7f200819c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.911+0000 7f200cd9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20081055a0 0x7f200819c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50702/0 (socket says 172.21.15.44:50702) 2023-11-16T16:50:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.911+0000 7f200cd9c700 1 -- 172.21.15.44:0/1838120147 learned_addr learned my addr 172.21.15.44:0/1838120147 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.911+0000 7f20077fe700 1 --2- 172.21.15.44:0/1838120147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2008100f00 0x7f2008198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.911+0000 7f2007fff700 1 --2- 172.21.15.44:0/1838120147 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2008100560 0x7f2008198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.911+0000 7f200cd9c700 1 -- 172.21.15.44:0/1838120147 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2008100560 msgr2=0x7f2008198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.911+0000 7f200cd9c700 1 --2- 172.21.15.44:0/1838120147 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2008100560 0x7f2008198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.911+0000 7f200cd9c700 1 -- 172.21.15.44:0/1838120147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2008100f00 msgr2=0x7f2008198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.911+0000 7f200cd9c700 1 --2- 172.21.15.44:0/1838120147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2008100f00 0x7f2008198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.911+0000 7f200cd9c700 1 -- 172.21.15.44:0/1838120147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2000009d00 con 0x7f20081055a0 2023-11-16T16:50:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.911+0000 7f200cd9c700 1 --2- 172.21.15.44:0/1838120147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20081055a0 0x7f200819c010 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f2000007820 tx=0x7f2000007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.912+0000 7f20057fa700 1 -- 172.21.15.44:0/1838120147 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200000e040 con 0x7f20081055a0 2023-11-16T16:50:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.912+0000 7f20057fa700 1 -- 172.21.15.44:0/1838120147 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f200000bcb0 con 0x7f20081055a0 2023-11-16T16:50:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.912+0000 7f20057fa700 1 -- 172.21.15.44:0/1838120147 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20000058a0 con 0x7f20081055a0 2023-11-16T16:50:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.912+0000 7f200e7ff700 1 -- 172.21.15.44:0/1838120147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20081103c0 con 0x7f20081055a0 2023-11-16T16:50:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.912+0000 7f200e7ff700 1 -- 172.21.15.44:0/1838120147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2008110230 con 0x7f20081055a0 2023-11-16T16:50:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.915+0000 7f20057fa700 1 -- 172.21.15.44:0/1838120147 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f2000004a50 con 0x7f20081055a0 2023-11-16T16:50:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.916+0000 7f202205fb80 1 Processor -- start 2023-11-16T16:50:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.916+0000 7f202205fb80 1 -- start start 2023-11-16T16:50:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f20057fa700 1 --2- 172.21.15.44:0/1838120147 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1ff00a1460 0x7f1ff00a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f202205fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 0x5624f2d8c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f202205fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8c720 0x5624f2d86560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f202205fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8cb90 0x5624f2d86b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f202205fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f2c8c000 con 0x5624f2d8cb90 2023-11-16T16:50:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f202205fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f2cde840 con 0x5624f2be26b0 2023-11-16T16:50:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f202205fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f2cd7cd0 con 0x5624f2d8c720 2023-11-16T16:50:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f20057fa700 1 -- 172.21.15.44:0/1838120147 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f20000ce480 con 0x7f20081055a0 2023-11-16T16:50:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f20077fe700 1 --2- 172.21.15.44:0/1838120147 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1ff00a1460 0x7f1ff00a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f200cd9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8cb90 0x5624f2d86b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f2007fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 0x5624f2d8c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f200cd9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8cb90 0x5624f2d86b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50714/0 (socket says 172.21.15.44:50714) 2023-11-16T16:50:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f200cd9c700 1 -- 172.21.15.44:0/1587331765 learned_addr learned my addr 172.21.15.44:0/1587331765 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f20077fe700 1 --2- 172.21.15.44:0/1587331765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8c720 0x5624f2d86560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f2007fff700 1 -- 172.21.15.44:0/1587331765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8c720 msgr2=0x5624f2d86560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f2007fff700 1 --2- 172.21.15.44:0/1587331765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8c720 0x5624f2d86560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f2007fff700 1 -- 172.21.15.44:0/1587331765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8cb90 msgr2=0x5624f2d86b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f2007fff700 1 --2- 172.21.15.44:0/1587331765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8cb90 0x5624f2d86b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.917+0000 7f2007fff700 1 -- 172.21.15.44:0/1587331765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624f2b595d0 con 0x5624f2be26b0 2023-11-16T16:50:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.918+0000 7f2007fff700 1 --2- 172.21.15.44:0/1587331765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 0x5624f2d8c1e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8009cb0 tx=0x7f1ff8011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5db3d7b21d5e17d0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.918+0000 7f20077fe700 1 --2- 172.21.15.44:0/1838120147 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1ff00a1460 0x7f1ff00a3910 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2008199d30 tx=0x7f1ffc009ac0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.918+0000 7f1fee7fc700 1 -- 172.21.15.44:0/1587331765 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8011d20 con 0x5624f2be26b0 2023-11-16T16:50:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.918+0000 7f1fee7fc700 1 -- 172.21.15.44:0/1587331765 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ff8011e80 con 0x5624f2be26b0 2023-11-16T16:50:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.918+0000 7f1fee7fc700 1 -- 172.21.15.44:0/1587331765 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8005280 con 0x5624f2be26b0 2023-11-16T16:50:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.919+0000 7f202205fb80 1 -- 172.21.15.44:0/1587331765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 msgr2=0x5624f2d8c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.919+0000 7f202205fb80 1 --2- 172.21.15.44:0/1587331765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 0x5624f2d8c1e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8009cb0 tx=0x7f1ff8011440 comp rx=0 tx=0).stop 2023-11-16T16:50:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.919+0000 7f202205fb80 1 -- 172.21.15.44:0/1587331765 shutdown_connections 2023-11-16T16:50:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.919+0000 7f202205fb80 1 --2- 172.21.15.44:0/1587331765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8cb90 0x5624f2d86b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.919+0000 7f202205fb80 1 --2- 172.21.15.44:0/1587331765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 0x5624f2d8c1e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.919+0000 7f202205fb80 1 --2- 172.21.15.44:0/1587331765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8c720 0x5624f2d86560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.919+0000 7f202205fb80 1 -- 172.21.15.44:0/1587331765 >> 172.21.15.44:0/1587331765 conn(0x5624f2d7bf50 msgr2=0x5624f2d81270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.919+0000 7f202205fb80 1 -- 172.21.15.44:0/1587331765 shutdown_connections 2023-11-16T16:50:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.919+0000 7f202205fb80 1 -- 172.21.15.44:0/1587331765 wait complete. 2023-11-16T16:50:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.920+0000 7f202205fb80 1 -- start start 2023-11-16T16:50:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.920+0000 7f202205fb80 1 Processor -- start 2023-11-16T16:50:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.920+0000 7f202205fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 0x5624f2d92a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.920+0000 7f202205fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8c720 0x5624f2d8fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.920+0000 7f202205fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8cb90 0x5624f2d90340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.920+0000 7f202205fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f2c79880 con 0x5624f2d8cb90 2023-11-16T16:50:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.920+0000 7f202205fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f2c79520 con 0x5624f2be26b0 2023-11-16T16:50:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.920+0000 7f202205fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f2b5b8c0 con 0x5624f2d8c720 2023-11-16T16:50:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.920+0000 7f202205fb80 10 client.0 _ll_register_callbacks cb 0x5624f2c97720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.921+0000 7f2007fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 0x5624f2d92a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.921+0000 7f2007fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 0x5624f2d92a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:48658/0 (socket says 172.21.15.44:48658) 2023-11-16T16:50:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.921+0000 7f2007fff700 1 -- 172.21.15.44:0/2458356088 learned_addr learned my addr 172.21.15.44:0/2458356088 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.921+0000 7f20077fe700 1 --2- 172.21.15.44:0/2458356088 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8c720 0x5624f2d8fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.921+0000 7f200cd9c700 1 --2- 172.21.15.44:0/2458356088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8cb90 0x5624f2d90340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.921+0000 7f2007fff700 1 -- 172.21.15.44:0/2458356088 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8c720 msgr2=0x5624f2d8fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.921+0000 7f2007fff700 1 --2- 172.21.15.44:0/2458356088 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8c720 0x5624f2d8fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.921+0000 7f2007fff700 1 -- 172.21.15.44:0/2458356088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8cb90 msgr2=0x5624f2d90340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.921+0000 7f2007fff700 1 --2- 172.21.15.44:0/2458356088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8cb90 0x5624f2d90340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.921+0000 7f2007fff700 1 -- 172.21.15.44:0/2458356088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff80098e0 con 0x5624f2be26b0 2023-11-16T16:50:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.921+0000 7f2007fff700 1 --2- 172.21.15.44:0/2458356088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 0x5624f2d92a20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8000c00 tx=0x7f1ff8011bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7aef95c6a2e7e6df server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.921+0000 7f1fe77fe700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:50:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.921+0000 7f1fe77fe700 1 -- 172.21.15.44:0/2458356088 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8011e70 con 0x5624f2be26b0 2023-11-16T16:50:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.922+0000 7f202205fb80 1 -- 172.21.15.44:0/2458356088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5624f2c9a040 con 0x5624f2be26b0 2023-11-16T16:50:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.922+0000 7f1fe77fe700 1 -- 172.21.15.44:0/2458356088 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f1ff8004260 con 0x5624f2be26b0 2023-11-16T16:50:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.922+0000 7f1fe77fe700 1 -- 172.21.15.44:0/2458356088 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ff8004420 con 0x5624f2be26b0 2023-11-16T16:50:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.922+0000 7f202205fb80 10 client.25637 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.922+0000 7f202205fb80 1 -- 172.21.15.44:0/2458356088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5624f2c9e130 con 0x5624f2be26b0 2023-11-16T16:50:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.922+0000 7f1fe77fe700 1 -- 172.21.15.44:0/2458356088 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5624f2c9a040 con 0x5624f2be26b0 2023-11-16T16:50:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.922+0000 7f1fe77fe700 1 -- 172.21.15.44:0/2458356088 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1ff8005130 con 0x5624f2be26b0 2023-11-16T16:50:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.922+0000 7f202205fb80 10 client.25637 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.922+0000 7f202205fb80 10 client.25637 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.922+0000 7f202205fb80 20 client.25637 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.922+0000 7f202205fb80 1 --2- 172.21.15.44:0/2458356088 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5624f2d9fe70 0x5624f2da2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.922+0000 7f202205fb80 4 client.25637 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.922+0000 7f202205fb80 1 -- 172.21.15.44:0/2458356088 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624f2ca6600 con 0x5624f2d9fe70 2023-11-16T16:50:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.923+0000 7f20077fe700 1 --2- 172.21.15.44:0/2458356088 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5624f2d9fe70 0x5624f2da2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.923+0000 7f20077fe700 1 --2- 172.21.15.44:0/2458356088 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5624f2d9fe70 0x5624f2da2280 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5e372dc57887ef1 server_cookie=6774638876955152 in_seq=0 out_seq=0 2023-11-16T16:50:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.923+0000 7f1fe77fe700 10 client.25637 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.924+0000 7f1fe77fe700 1 -- 172.21.15.44:0/2458356088 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5624f2ca6600 con 0x5624f2d9fe70 2023-11-16T16:50:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.924+0000 7f1fe77fe700 10 client.25637 handle_command_reply: tid=0 2023-11-16T16:50:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.924+0000 7f202205fb80 1 client.25637 shutdown 2023-11-16T16:50:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.924+0000 7f202205fb80 1 -- 172.21.15.44:0/2458356088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 msgr2=0x5624f2d92a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.924+0000 7f202205fb80 1 --2- 172.21.15.44:0/2458356088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 0x5624f2d92a20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8000c00 tx=0x7f1ff8011bd0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.924+0000 7f202205fb80 1 -- 172.21.15.44:0/2458356088 shutdown_connections 2023-11-16T16:50:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.924+0000 7f202205fb80 1 --2- 172.21.15.44:0/2458356088 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5624f2d9fe70 0x5624f2da2280 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.924+0000 7f202205fb80 1 --2- 172.21.15.44:0/2458356088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8cb90 0x5624f2d90340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.924+0000 7f202205fb80 1 --2- 172.21.15.44:0/2458356088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 0x5624f2d92a20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.924+0000 7f202205fb80 1 --2- 172.21.15.44:0/2458356088 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8c720 0x5624f2d8fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.924+0000 7f1fe77fe700 0 client.25637 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.924+0000 7f202205fb80 1 -- 172.21.15.44:0/2458356088 >> 172.21.15.44:0/2458356088 conn(0x5624f2d7bf50 msgr2=0x5624f2d807e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.924+0000 7f202205fb80 1 -- 172.21.15.44:0/2458356088 shutdown_connections 2023-11-16T16:50:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.924+0000 7f202205fb80 1 -- 172.21.15.44:0/2458356088 wait complete. 2023-11-16T16:50:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.925+0000 7f202205fb80 20 client.25637 trim_cache size 0 max 16384 2023-11-16T16:50:24.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.940+0000 7f202205fb80 1 Processor -- start 2023-11-16T16:50:24.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.940+0000 7f202205fb80 1 -- start start 2023-11-16T16:50:24.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.940+0000 7f202205fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 0x5624f2d97020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.940+0000 7f202205fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8c720 0x5624f2d925c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.940+0000 7f202205fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8cb90 0x5624f2d80f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.940+0000 7f202205fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f2d7bb70 con 0x5624f2d8c720 2023-11-16T16:50:24.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.940+0000 7f202205fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f2d8a9f0 con 0x5624f2be26b0 2023-11-16T16:50:24.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.940+0000 7f202205fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f2d8bdd0 con 0x5624f2d8cb90 2023-11-16T16:50:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.941+0000 7f20077fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8c720 0x5624f2d925c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.941+0000 7f20077fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8c720 0x5624f2d925c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50736/0 (socket says 172.21.15.44:50736) 2023-11-16T16:50:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.941+0000 7f20077fe700 1 -- 172.21.15.44:0/3504055325 learned_addr learned my addr 172.21.15.44:0/3504055325 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.941+0000 7f2007fff700 1 --2- 172.21.15.44:0/3504055325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 0x5624f2d97020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.941+0000 7f200cd9c700 1 --2- 172.21.15.44:0/3504055325 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8cb90 0x5624f2d80f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.941+0000 7f20077fe700 1 -- 172.21.15.44:0/3504055325 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8cb90 msgr2=0x5624f2d80f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.941+0000 7f20077fe700 1 --2- 172.21.15.44:0/3504055325 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8cb90 0x5624f2d80f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.941+0000 7f20077fe700 1 -- 172.21.15.44:0/3504055325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 msgr2=0x5624f2d97020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.941+0000 7f20077fe700 1 --2- 172.21.15.44:0/3504055325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 0x5624f2d97020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.941+0000 7f20077fe700 1 -- 172.21.15.44:0/3504055325 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624f2d7d6f0 con 0x5624f2d8c720 2023-11-16T16:50:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.941+0000 7f20077fe700 1 --2- 172.21.15.44:0/3504055325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8c720 0x5624f2d925c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009d80 tx=0x7f1ffc021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df17224e53418b84 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.941+0000 7f1feeffd700 1 -- 172.21.15.44:0/3504055325 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc028070 con 0x5624f2d8c720 2023-11-16T16:50:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.942+0000 7f1feeffd700 1 -- 172.21.15.44:0/3504055325 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ffc013260 con 0x5624f2d8c720 2023-11-16T16:50:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.942+0000 7f1feeffd700 1 -- 172.21.15.44:0/3504055325 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc011cd0 con 0x5624f2d8c720 2023-11-16T16:50:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.942+0000 7f202205fb80 1 -- 172.21.15.44:0/3504055325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8c720 msgr2=0x5624f2d925c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.942+0000 7f202205fb80 1 --2- 172.21.15.44:0/3504055325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8c720 0x5624f2d925c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009d80 tx=0x7f1ffc021000 comp rx=0 tx=0).stop 2023-11-16T16:50:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.942+0000 7f202205fb80 1 -- 172.21.15.44:0/3504055325 shutdown_connections 2023-11-16T16:50:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.942+0000 7f202205fb80 1 --2- 172.21.15.44:0/3504055325 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8cb90 0x5624f2d80f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.942+0000 7f202205fb80 1 --2- 172.21.15.44:0/3504055325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2be26b0 0x5624f2d97020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.942+0000 7f202205fb80 1 --2- 172.21.15.44:0/3504055325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2d8c720 0x5624f2d925c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.942+0000 7f202205fb80 1 -- 172.21.15.44:0/3504055325 >> 172.21.15.44:0/3504055325 conn(0x5624f2d7bf50 msgr2=0x5624f2d9c290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.942+0000 7f202205fb80 1 -- 172.21.15.44:0/3504055325 shutdown_connections 2023-11-16T16:50:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.942+0000 7f202205fb80 1 -- 172.21.15.44:0/3504055325 wait complete. 2023-11-16T16:50:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.943+0000 7f202205fb80 1 -- start start 2023-11-16T16:50:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.943+0000 7f202205fb80 1 Processor -- start 2023-11-16T16:50:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.943+0000 7f202205fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2be26b0 0x5624f2d88e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.943+0000 7f202205fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8c720 0x5624f2d89380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.943+0000 7f202205fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2d8cb90 0x5624f2d87c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.943+0000 7f202205fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f2d97590 con 0x5624f2be26b0 2023-11-16T16:50:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.943+0000 7f202205fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f2d88140 con 0x5624f2d8cb90 2023-11-16T16:50:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.943+0000 7f202205fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f2d88300 con 0x5624f2d8c720 2023-11-16T16:50:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.943+0000 7f202205fb80 10 client.0 _ll_register_callbacks cb 0x7f20000074e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.943+0000 7f2007fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2be26b0 0x5624f2d88e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.944+0000 7f2007fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2be26b0 0x5624f2d88e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50748/0 (socket says 172.21.15.44:50748) 2023-11-16T16:50:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.944+0000 7f2007fff700 1 -- 172.21.15.44:0/697150992 learned_addr learned my addr 172.21.15.44:0/697150992 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.944+0000 7f200cd9c700 1 --2- 172.21.15.44:0/697150992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2d8cb90 0x5624f2d87c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.944+0000 7f20077fe700 1 --2- 172.21.15.44:0/697150992 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8c720 0x5624f2d89380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.944+0000 7f2007fff700 1 -- 172.21.15.44:0/697150992 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8c720 msgr2=0x5624f2d89380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.944+0000 7f2007fff700 1 --2- 172.21.15.44:0/697150992 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8c720 0x5624f2d89380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.944+0000 7f2007fff700 1 -- 172.21.15.44:0/697150992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2d8cb90 msgr2=0x5624f2d87c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.944+0000 7f2007fff700 1 --2- 172.21.15.44:0/697150992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2d8cb90 0x5624f2d87c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.944+0000 7f2007fff700 1 -- 172.21.15.44:0/697150992 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ffc01b040 con 0x5624f2be26b0 2023-11-16T16:50:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.944+0000 7f2007fff700 1 --2- 172.21.15.44:0/697150992 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2be26b0 0x5624f2d88e40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8011790 tx=0x7f1ff8004260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbfb3b1425de0390 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.944+0000 7f1fe77fe700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:50:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.944+0000 7f1fe77fe700 1 -- 172.21.15.44:0/697150992 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8003680 con 0x5624f2be26b0 2023-11-16T16:50:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.944+0000 7f1fe77fe700 1 -- 172.21.15.44:0/697150992 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f1ff8000e40 con 0x5624f2be26b0 2023-11-16T16:50:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.944+0000 7f202205fb80 1 -- 172.21.15.44:0/697150992 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5624f2d79710 con 0x5624f2be26b0 2023-11-16T16:50:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.945+0000 7f1fe77fe700 1 -- 172.21.15.44:0/697150992 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ff80037e0 con 0x5624f2be26b0 2023-11-16T16:50:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.945+0000 7f202205fb80 10 client.15627 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.945+0000 7f202205fb80 1 -- 172.21.15.44:0/697150992 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5624f2b05d80 con 0x5624f2be26b0 2023-11-16T16:50:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.945+0000 7f1fe77fe700 1 -- 172.21.15.44:0/697150992 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5624f2d79710 con 0x5624f2be26b0 2023-11-16T16:50:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.945+0000 7f1fe77fe700 1 -- 172.21.15.44:0/697150992 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1ff8004f90 con 0x5624f2be26b0 2023-11-16T16:50:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.945+0000 7f202205fb80 10 client.15627 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.945+0000 7f202205fb80 10 client.15627 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.945+0000 7f202205fb80 20 client.15627 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.945+0000 7f202205fb80 1 --2- 172.21.15.44:0/697150992 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5624f2df4740 0x5624f2df6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.945+0000 7f202205fb80 4 client.15627 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:50:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.945+0000 7f202205fb80 1 -- 172.21.15.44:0/697150992 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5624f2b26f10 con 0x5624f2df4740 2023-11-16T16:50:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.946+0000 7f20077fe700 1 --2- 172.21.15.44:0/697150992 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5624f2df4740 0x5624f2df6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.946+0000 7f20077fe700 1 --2- 172.21.15.44:0/697150992 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5624f2df4740 0x5624f2df6b50 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27ee2cda6bcd72eb server_cookie=520009ef3919c871 in_seq=0 out_seq=0 2023-11-16T16:50:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.946+0000 7f1fe77fe700 10 client.15627 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:24.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.950+0000 7f1fe77fe700 1 -- 172.21.15.44:0/697150992 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5624f2b26f10 con 0x5624f2df4740 2023-11-16T16:50:24.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.950+0000 7f1fe77fe700 10 client.15627 handle_command_reply: tid=0 2023-11-16T16:50:24.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.950+0000 7f202205fb80 1 client.15627 shutdown 2023-11-16T16:50:24.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.950+0000 7f202205fb80 1 -- 172.21.15.44:0/697150992 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2be26b0 msgr2=0x5624f2d88e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:24.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.950+0000 7f202205fb80 1 --2- 172.21.15.44:0/697150992 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2be26b0 0x5624f2d88e40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8011790 tx=0x7f1ff8004260 comp rx=0 tx=0).stop 2023-11-16T16:50:24.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.950+0000 7f202205fb80 1 -- 172.21.15.44:0/697150992 shutdown_connections 2023-11-16T16:50:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.950+0000 7f202205fb80 1 --2- 172.21.15.44:0/697150992 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5624f2df4740 0x5624f2df6b50 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.950+0000 7f202205fb80 1 --2- 172.21.15.44:0/697150992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624f2d8cb90 0x5624f2d87c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.950+0000 7f202205fb80 1 --2- 172.21.15.44:0/697150992 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624f2be26b0 0x5624f2d88e40 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.950+0000 7f1fe77fe700 0 client.15627 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.950+0000 7f202205fb80 1 --2- 172.21.15.44:0/697150992 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5624f2d8c720 0x5624f2d89380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.950+0000 7f202205fb80 1 -- 172.21.15.44:0/697150992 >> 172.21.15.44:0/697150992 conn(0x5624f2d7bf50 msgr2=0x5624f2d8ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.950+0000 7f202205fb80 1 -- 172.21.15.44:0/697150992 shutdown_connections 2023-11-16T16:50:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.950+0000 7f202205fb80 1 -- 172.21.15.44:0/697150992 wait complete. 2023-11-16T16:50:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.950+0000 7f202205fb80 20 client.15627 trim_cache size 0 max 16384 2023-11-16T16:50:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:50:24.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:50:24.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:50:24.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:50:24.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.951+0000 7f200e7ff700 1 -- 172.21.15.44:0/1838120147 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1ff00a1460 msgr2=0x7f1ff00a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:24.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.951+0000 7f200e7ff700 1 --2- 172.21.15.44:0/1838120147 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1ff00a1460 0x7f1ff00a3910 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2008199d30 tx=0x7f1ffc009ac0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.951+0000 7f200e7ff700 1 -- 172.21.15.44:0/1838120147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20081055a0 msgr2=0x7f200819c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.951+0000 7f200e7ff700 1 --2- 172.21.15.44:0/1838120147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20081055a0 0x7f200819c010 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f2000007820 tx=0x7f2000007850 comp rx=0 tx=0).stop 2023-11-16T16:50:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.951+0000 7f200e7ff700 1 -- 172.21.15.44:0/1838120147 shutdown_connections 2023-11-16T16:50:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.951+0000 7f200e7ff700 1 --2- 172.21.15.44:0/1838120147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20081055a0 0x7f200819c010 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.951+0000 7f200e7ff700 1 --2- 172.21.15.44:0/1838120147 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1ff00a1460 0x7f1ff00a3910 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.951+0000 7f200e7ff700 1 --2- 172.21.15.44:0/1838120147 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2008100560 0x7f2008198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.951+0000 7f200e7ff700 1 --2- 172.21.15.44:0/1838120147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2008100f00 0x7f2008198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.951+0000 7f200e7ff700 1 -- 172.21.15.44:0/1838120147 >> 172.21.15.44:0/1838120147 conn(0x7f20080781b0 msgr2=0x7f20080fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.951+0000 7f200e7ff700 1 -- 172.21.15.44:0/1838120147 shutdown_connections 2023-11-16T16:50:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:24.951+0000 7f200e7ff700 1 -- 172.21.15.44:0/1838120147 wait complete. 2023-11-16T16:50:26.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:26 smithi077 ceph-mon[131204]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-11-16T16:50:27.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:26 smithi044 ceph-mon[128750]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-11-16T16:50:27.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:26 smithi176 ceph-mon[131630]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-11-16T16:50:28.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:28 smithi077 ceph-mon[131204]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2023-11-16T16:50:29.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:28 smithi044 ceph-mon[128750]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2023-11-16T16:50:29.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:28 smithi176 ceph-mon[131630]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2023-11-16T16:50:30.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:30 smithi077 ceph-mon[131204]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2023-11-16T16:50:31.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:30 smithi044 ceph-mon[128750]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2023-11-16T16:50:31.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:30 smithi176 ceph-mon[131630]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2023-11-16T16:50:32.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:32 smithi077 ceph-mon[131204]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2023-11-16T16:50:33.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:32 smithi044 ceph-mon[128750]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2023-11-16T16:50:33.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:32 smithi176 ceph-mon[131630]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2023-11-16T16:50:34.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:34 smithi077 ceph-mon[131204]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2023-11-16T16:50:34.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:34 smithi044 ceph-mon[128750]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2023-11-16T16:50:34.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:34 smithi176 ceph-mon[131630]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2023-11-16T16:50:36.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:36 smithi077 ceph-mon[131204]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2023-11-16T16:50:37.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:36 smithi044 ceph-mon[128750]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2023-11-16T16:50:37.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:36 smithi176 ceph-mon[131630]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2023-11-16T16:50:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.107+0000 7f24ad7ff700 1 -- 172.21.15.44:0/632099184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24a8074b00 msgr2=0x7f24a8073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.107+0000 7f24ad7ff700 1 --2- 172.21.15.44:0/632099184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24a8074b00 0x7f24a8073260 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f249c009990 tx=0x7f249c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.107+0000 7f24ad7ff700 1 -- 172.21.15.44:0/632099184 shutdown_connections 2023-11-16T16:50:38.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.107+0000 7f24ad7ff700 1 --2- 172.21.15.44:0/632099184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a810ce50 0x7f24a810f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.107+0000 7f24ad7ff700 1 --2- 172.21.15.44:0/632099184 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24a80746a0 0x7f24a8072d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.107+0000 7f24ad7ff700 1 --2- 172.21.15.44:0/632099184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24a8074b00 0x7f24a8073260 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.107+0000 7f24ad7ff700 1 -- 172.21.15.44:0/632099184 >> 172.21.15.44:0/632099184 conn(0x7f24a8077550 msgr2=0x7f24a8077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:38.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.107+0000 7f24ad7ff700 1 -- 172.21.15.44:0/632099184 shutdown_connections 2023-11-16T16:50:38.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.107+0000 7f24ad7ff700 1 -- 172.21.15.44:0/632099184 wait complete. 2023-11-16T16:50:38.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.109+0000 7f24ad7ff700 1 Processor -- start 2023-11-16T16:50:38.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.109+0000 7f24ad7ff700 1 -- start start 2023-11-16T16:50:38.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.109+0000 7f24ad7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a80746a0 0x7f24a81985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.109+0000 7f24ad7ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24a8074b00 0x7f24a8198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.109+0000 7f24ad7ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24a810ce50 0x7f24a819cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.109+0000 7f24ad7ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a819d5c0 con 0x7f24a8074b00 2023-11-16T16:50:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.109+0000 7f24ad7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a819d730 con 0x7f24a80746a0 2023-11-16T16:50:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.109+0000 7f24ad7ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a819d8f0 con 0x7f24a810ce50 2023-11-16T16:50:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.110+0000 7f24a67fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24a8074b00 0x7f24a8198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.110+0000 7f24a67fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24a8074b00 0x7f24a8198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60090/0 (socket says 172.21.15.44:60090) 2023-11-16T16:50:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.110+0000 7f24a67fc700 1 -- 172.21.15.44:0/3215640308 learned_addr learned my addr 172.21.15.44:0/3215640308 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.110+0000 7f24a6ffd700 1 --2- 172.21.15.44:0/3215640308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a80746a0 0x7f24a81985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.110+0000 7f24a77fe700 1 --2- 172.21.15.44:0/3215640308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24a810ce50 0x7f24a819cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.110+0000 7f24a67fc700 1 -- 172.21.15.44:0/3215640308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24a810ce50 msgr2=0x7f24a819cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.110+0000 7f24a67fc700 1 --2- 172.21.15.44:0/3215640308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24a810ce50 0x7f24a819cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.110+0000 7f24a67fc700 1 -- 172.21.15.44:0/3215640308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a80746a0 msgr2=0x7f24a81985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.110+0000 7f24a67fc700 1 --2- 172.21.15.44:0/3215640308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a80746a0 0x7f24a81985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.110+0000 7f24a67fc700 1 -- 172.21.15.44:0/3215640308 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249c0095f0 con 0x7f24a8074b00 2023-11-16T16:50:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.110+0000 7f24a6ffd700 1 --2- 172.21.15.44:0/3215640308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a80746a0 0x7f24a81985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.110+0000 7f24a67fc700 1 --2- 172.21.15.44:0/3215640308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24a8074b00 0x7f24a8198b20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f249c005910 tx=0x7f249c004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.111+0000 7f248ffff700 1 -- 172.21.15.44:0/3215640308 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c01d070 con 0x7f24a8074b00 2023-11-16T16:50:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.111+0000 7f24ad7ff700 1 -- 172.21.15.44:0/3215640308 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a8112570 con 0x7f24a8074b00 2023-11-16T16:50:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.111+0000 7f248ffff700 1 -- 172.21.15.44:0/3215640308 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f249c022470 con 0x7f24a8074b00 2023-11-16T16:50:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.111+0000 7f248ffff700 1 -- 172.21.15.44:0/3215640308 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c017650 con 0x7f24a8074b00 2023-11-16T16:50:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.111+0000 7f24ad7ff700 1 -- 172.21.15.44:0/3215640308 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a81123e0 con 0x7f24a8074b00 2023-11-16T16:50:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.114+0000 7f248ffff700 1 -- 172.21.15.44:0/3215640308 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f249c0225e0 con 0x7f24a8074b00 2023-11-16T16:50:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.116+0000 7f24c109bb80 1 Processor -- start 2023-11-16T16:50:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.116+0000 7f24c109bb80 1 -- start start 2023-11-16T16:50:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.116+0000 7f248ffff700 1 --2- 172.21.15.44:0/3215640308 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f24900a13e0 0x7f24900a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.116+0000 7f248ffff700 1 -- 172.21.15.44:0/3215640308 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f249c0ce530 con 0x7f24a8074b00 2023-11-16T16:50:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.116+0000 7f24c109bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5631620656b0 0x56316220f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.116+0000 7f24c109bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220f720 0x563162209560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.116+0000 7f24c109bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220fb90 0x563162209b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.116+0000 7f24c109bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56316210f000 con 0x56316220f720 2023-11-16T16:50:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.116+0000 7f24c109bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x563162161840 con 0x56316220fb90 2023-11-16T16:50:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.116+0000 7f24c109bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56316215acd0 con 0x5631620656b0 2023-11-16T16:50:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.117+0000 7f24a77fe700 1 --2- 172.21.15.44:0/3215640308 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f24900a13e0 0x7f24900a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.117+0000 7f24a6ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5631620656b0 0x56316220f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.117+0000 7f24a67fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220f720 0x563162209560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.117+0000 7f24a6ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5631620656b0 0x56316220f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:52836/0 (socket says 172.21.15.44:52836) 2023-11-16T16:50:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.117+0000 7f24a6ffd700 1 -- 172.21.15.44:0/2584278907 learned_addr learned my addr 172.21.15.44:0/2584278907 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.117+0000 7f24a77fe700 1 --2- 172.21.15.44:0/2584278907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220fb90 0x563162209b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.117+0000 7f24a67fc700 1 -- 172.21.15.44:0/2584278907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5631620656b0 msgr2=0x56316220f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.117+0000 7f24a67fc700 1 --2- 172.21.15.44:0/2584278907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5631620656b0 0x56316220f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.117+0000 7f24a67fc700 1 -- 172.21.15.44:0/2584278907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220fb90 msgr2=0x563162209b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.117+0000 7f24a67fc700 1 --2- 172.21.15.44:0/2584278907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220fb90 0x563162209b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.117+0000 7f24a67fc700 1 -- 172.21.15.44:0/2584278907 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563161fdc5d0 con 0x56316220f720 2023-11-16T16:50:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.117+0000 7f24a77fe700 1 --2- 172.21.15.44:0/2584278907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220fb90 0x563162209b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.117+0000 7f24a77fe700 1 --2- 172.21.15.44:0/3215640308 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f24900a13e0 0x7f24900a3890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009ec0 tx=0x7f24a0013250 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.118+0000 7f24a67fc700 1 --2- 172.21.15.44:0/2584278907 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220f720 0x563162209560 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f249c0cb780 tx=0x7f249c09ae90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3f4148b4b13c68b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.118+0000 7f248d7fa700 1 -- 172.21.15.44:0/2584278907 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c09aa20 con 0x56316220f720 2023-11-16T16:50:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.118+0000 7f248d7fa700 1 -- 172.21.15.44:0/2584278907 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f249c09b0c0 con 0x56316220f720 2023-11-16T16:50:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.118+0000 7f248d7fa700 1 -- 172.21.15.44:0/2584278907 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c0ce390 con 0x56316220f720 2023-11-16T16:50:38.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.119+0000 7f24c109bb80 1 -- 172.21.15.44:0/2584278907 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220f720 msgr2=0x563162209560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:38.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.119+0000 7f24c109bb80 1 --2- 172.21.15.44:0/2584278907 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220f720 0x563162209560 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f249c0cb780 tx=0x7f249c09ae90 comp rx=0 tx=0).stop 2023-11-16T16:50:38.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.119+0000 7f24c109bb80 1 -- 172.21.15.44:0/2584278907 shutdown_connections 2023-11-16T16:50:38.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.119+0000 7f24c109bb80 1 --2- 172.21.15.44:0/2584278907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220fb90 0x563162209b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.119+0000 7f24c109bb80 1 --2- 172.21.15.44:0/2584278907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5631620656b0 0x56316220f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.119+0000 7f24c109bb80 1 --2- 172.21.15.44:0/2584278907 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220f720 0x563162209560 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.119+0000 7f24c109bb80 1 -- 172.21.15.44:0/2584278907 >> 172.21.15.44:0/2584278907 conn(0x5631621fef50 msgr2=0x563162204270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.119+0000 7f24c109bb80 1 -- 172.21.15.44:0/2584278907 shutdown_connections 2023-11-16T16:50:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.119+0000 7f24c109bb80 1 -- 172.21.15.44:0/2584278907 wait complete. 2023-11-16T16:50:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.119+0000 7f24c109bb80 1 -- start start 2023-11-16T16:50:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.120+0000 7f24c109bb80 1 Processor -- start 2023-11-16T16:50:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.120+0000 7f24c109bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631620656b0 0x563162218780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.120+0000 7f24c109bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56316220f720 0x563162218cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.120+0000 7f24c109bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220fb90 0x5631622129d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.120+0000 7f24c109bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631620fc880 con 0x56316220fb90 2023-11-16T16:50:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.120+0000 7f24c109bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631620fc520 con 0x5631620656b0 2023-11-16T16:50:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.120+0000 7f24c109bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x563161fde8c0 con 0x56316220f720 2023-11-16T16:50:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.120+0000 7f24c109bb80 10 client.0 _ll_register_callbacks cb 0x56316211a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.121+0000 7f24a77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220fb90 0x5631622129d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.121+0000 7f24a67fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56316220f720 0x563162218cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.121+0000 7f24a77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220fb90 0x5631622129d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60110/0 (socket says 172.21.15.44:60110) 2023-11-16T16:50:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.121+0000 7f24a77fe700 1 -- 172.21.15.44:0/1308912952 learned_addr learned my addr 172.21.15.44:0/1308912952 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.121+0000 7f24a6ffd700 1 --2- 172.21.15.44:0/1308912952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631620656b0 0x563162218780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.121+0000 7f24a6ffd700 1 -- 172.21.15.44:0/1308912952 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56316220f720 msgr2=0x563162218cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.121+0000 7f24a6ffd700 1 --2- 172.21.15.44:0/1308912952 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56316220f720 0x563162218cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.121+0000 7f24a6ffd700 1 -- 172.21.15.44:0/1308912952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220fb90 msgr2=0x5631622129d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.121+0000 7f24a6ffd700 1 --2- 172.21.15.44:0/1308912952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220fb90 0x5631622129d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.121+0000 7f24a6ffd700 1 -- 172.21.15.44:0/1308912952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a81123e0 con 0x5631620656b0 2023-11-16T16:50:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.121+0000 7f24a77fe700 1 --2- 172.21.15.44:0/1308912952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220fb90 0x5631622129d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.121+0000 7f24a6ffd700 1 --2- 172.21.15.44:0/1308912952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631620656b0 0x563162218780 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2498012d20 tx=0x7f249800f6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de0a620a7f2f2ac8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.121+0000 7f248aff5700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:50:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.122+0000 7f248aff5700 1 -- 172.21.15.44:0/1308912952 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498017070 con 0x5631620656b0 2023-11-16T16:50:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.122+0000 7f24c109bb80 1 -- 172.21.15.44:0/1308912952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5631621fc710 con 0x5631620656b0 2023-11-16T16:50:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.122+0000 7f248aff5700 1 -- 172.21.15.44:0/1308912952 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f249800fb80 con 0x5631620656b0 2023-11-16T16:50:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.122+0000 7f248aff5700 1 -- 172.21.15.44:0/1308912952 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f249800fd40 con 0x5631620656b0 2023-11-16T16:50:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.122+0000 7f24c109bb80 10 client.25655 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.122+0000 7f24c109bb80 1 -- 172.21.15.44:0/1308912952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x563162121130 con 0x5631620656b0 2023-11-16T16:50:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.122+0000 7f248aff5700 1 -- 172.21.15.44:0/1308912952 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5631621fc710 con 0x5631620656b0 2023-11-16T16:50:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.123+0000 7f248aff5700 1 -- 172.21.15.44:0/1308912952 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2498003bb0 con 0x5631620656b0 2023-11-16T16:50:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.123+0000 7f24c109bb80 10 client.25655 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.123+0000 7f24c109bb80 10 client.25655 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.123+0000 7f24c109bb80 20 client.25655 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.123+0000 7f24c109bb80 1 --2- 172.21.15.44:0/1308912952 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x563162222c40 0x563162225050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.123+0000 7f24c109bb80 4 client.25655 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.123+0000 7f24c109bb80 1 -- 172.21.15.44:0/1308912952 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56316211d040 con 0x563162222c40 2023-11-16T16:50:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.123+0000 7f24a67fc700 1 --2- 172.21.15.44:0/1308912952 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x563162222c40 0x563162225050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.124+0000 7f24a67fc700 1 --2- 172.21.15.44:0/1308912952 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x563162222c40 0x563162225050 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e1dca2b7311139e server_cookie=f24ee1ece86d6031 in_seq=0 out_seq=0 2023-11-16T16:50:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.124+0000 7f248aff5700 10 client.25655 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.124+0000 7f248aff5700 1 -- 172.21.15.44:0/1308912952 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56316211d040 con 0x563162222c40 2023-11-16T16:50:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.124+0000 7f248aff5700 10 client.25655 handle_command_reply: tid=0 2023-11-16T16:50:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.124+0000 7f24c109bb80 1 client.25655 shutdown 2023-11-16T16:50:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.125+0000 7f24c109bb80 1 -- 172.21.15.44:0/1308912952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631620656b0 msgr2=0x563162218780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.125+0000 7f24c109bb80 1 --2- 172.21.15.44:0/1308912952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631620656b0 0x563162218780 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2498012d20 tx=0x7f249800f6a0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.125+0000 7f24c109bb80 1 -- 172.21.15.44:0/1308912952 shutdown_connections 2023-11-16T16:50:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.125+0000 7f24c109bb80 1 --2- 172.21.15.44:0/1308912952 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x563162222c40 0x563162225050 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.125+0000 7f24c109bb80 1 --2- 172.21.15.44:0/1308912952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220fb90 0x5631622129d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.125+0000 7f24c109bb80 1 --2- 172.21.15.44:0/1308912952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631620656b0 0x563162218780 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.125+0000 7f24c109bb80 1 --2- 172.21.15.44:0/1308912952 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56316220f720 0x563162218cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.125+0000 7f248aff5700 0 client.25655 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.125+0000 7f24c109bb80 1 -- 172.21.15.44:0/1308912952 >> 172.21.15.44:0/1308912952 conn(0x5631621fef50 msgr2=0x5631622037e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.125+0000 7f24c109bb80 1 -- 172.21.15.44:0/1308912952 shutdown_connections 2023-11-16T16:50:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.125+0000 7f24c109bb80 1 -- 172.21.15.44:0/1308912952 wait complete. 2023-11-16T16:50:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.125+0000 7f24c109bb80 20 client.25655 trim_cache size 0 max 16384 2023-11-16T16:50:38.142 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.141+0000 7f24c109bb80 1 Processor -- start 2023-11-16T16:50:38.143 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.141+0000 7f24c109bb80 1 -- start start 2023-11-16T16:50:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.141+0000 7f24c109bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631620656b0 0x56316220ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.141+0000 7f24c109bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220f720 0x563162203f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.141+0000 7f24c109bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56316220fb90 0x56316220e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.141+0000 7f24c109bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x563162201000 con 0x5631620656b0 2023-11-16T16:50:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.142+0000 7f24c109bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56316220d6f0 con 0x56316220f720 2023-11-16T16:50:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.142+0000 7f24c109bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56316220f1e0 con 0x56316220fb90 2023-11-16T16:50:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.142+0000 7f24a6ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631620656b0 0x56316220ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.142+0000 7f24a6ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631620656b0 0x56316220ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60124/0 (socket says 172.21.15.44:60124) 2023-11-16T16:50:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.142+0000 7f24a6ffd700 1 -- 172.21.15.44:0/3289890847 learned_addr learned my addr 172.21.15.44:0/3289890847 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.142+0000 7f24a67fc700 1 --2- 172.21.15.44:0/3289890847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220f720 0x563162203f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.142+0000 7f24a77fe700 1 --2- 172.21.15.44:0/3289890847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56316220fb90 0x56316220e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.142+0000 7f24a6ffd700 1 -- 172.21.15.44:0/3289890847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56316220fb90 msgr2=0x56316220e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.142+0000 7f24a6ffd700 1 --2- 172.21.15.44:0/3289890847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56316220fb90 0x56316220e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.142+0000 7f24a6ffd700 1 -- 172.21.15.44:0/3289890847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220f720 msgr2=0x563162203f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.142+0000 7f24a6ffd700 1 --2- 172.21.15.44:0/3289890847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220f720 0x563162203f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.142+0000 7f24a6ffd700 1 -- 172.21.15.44:0/3289890847 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563162120e80 con 0x5631620656b0 2023-11-16T16:50:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.142+0000 7f24a67fc700 1 --2- 172.21.15.44:0/3289890847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220f720 0x563162203f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.142+0000 7f24a6ffd700 1 --2- 172.21.15.44:0/3289890847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631620656b0 0x56316220ce60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2498009340 tx=0x7f2498005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e98f88f5f6a0b7d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.143+0000 7f248dffb700 1 -- 172.21.15.44:0/3289890847 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498017040 con 0x5631620656b0 2023-11-16T16:50:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.143+0000 7f248dffb700 1 -- 172.21.15.44:0/3289890847 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2498010550 con 0x5631620656b0 2023-11-16T16:50:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.143+0000 7f248dffb700 1 -- 172.21.15.44:0/3289890847 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24980053d0 con 0x5631620656b0 2023-11-16T16:50:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.143+0000 7f24c109bb80 1 -- 172.21.15.44:0/3289890847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631620656b0 msgr2=0x56316220ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.143+0000 7f24c109bb80 1 --2- 172.21.15.44:0/3289890847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631620656b0 0x56316220ce60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2498009340 tx=0x7f2498005e40 comp rx=0 tx=0).stop 2023-11-16T16:50:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.144+0000 7f24c109bb80 1 -- 172.21.15.44:0/3289890847 shutdown_connections 2023-11-16T16:50:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.144+0000 7f24c109bb80 1 --2- 172.21.15.44:0/3289890847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56316220fb90 0x56316220e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.144+0000 7f24c109bb80 1 --2- 172.21.15.44:0/3289890847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631620656b0 0x56316220ce60 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.144+0000 7f24c109bb80 1 --2- 172.21.15.44:0/3289890847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220f720 0x563162203f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.144+0000 7f24c109bb80 1 -- 172.21.15.44:0/3289890847 >> 172.21.15.44:0/3289890847 conn(0x5631621fef50 msgr2=0x563162213300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.144+0000 7f24c109bb80 1 -- 172.21.15.44:0/3289890847 shutdown_connections 2023-11-16T16:50:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.144+0000 7f24c109bb80 1 -- 172.21.15.44:0/3289890847 wait complete. 2023-11-16T16:50:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.144+0000 7f24c109bb80 1 -- start start 2023-11-16T16:50:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.144+0000 7f24c109bb80 1 Processor -- start 2023-11-16T16:50:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.145+0000 7f24c109bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5631620656b0 0x563162216430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.145+0000 7f24c109bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220f720 0x563162216970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.145+0000 7f24c109bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220fb90 0x563162275ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.145+0000 7f24c109bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x563162217120 con 0x56316220fb90 2023-11-16T16:50:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.145+0000 7f24c109bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x563162217290 con 0x56316220f720 2023-11-16T16:50:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.145+0000 7f24c109bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x563162276030 con 0x5631620656b0 2023-11-16T16:50:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.145+0000 7f24c109bb80 10 client.0 _ll_register_callbacks cb 0x7f24980127f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.145+0000 7f24a77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220fb90 0x563162275ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.145+0000 7f24a67fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220f720 0x563162216970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.145+0000 7f24a6ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5631620656b0 0x563162216430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.145+0000 7f24a77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220fb90 0x563162275ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60128/0 (socket says 172.21.15.44:60128) 2023-11-16T16:50:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.145+0000 7f24a67fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220f720 0x563162216970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:33394/0 (socket says 172.21.15.44:33394) 2023-11-16T16:50:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.145+0000 7f24a77fe700 1 -- 172.21.15.44:0/883699142 learned_addr learned my addr 172.21.15.44:0/883699142 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.145+0000 7f24a6ffd700 1 --2- 172.21.15.44:0/883699142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5631620656b0 0x563162216430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:52870/0 (socket says 172.21.15.44:52870) 2023-11-16T16:50:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.146+0000 7f24a67fc700 1 -- 172.21.15.44:0/883699142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5631620656b0 msgr2=0x563162216430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.146+0000 7f24a67fc700 1 --2- 172.21.15.44:0/883699142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5631620656b0 0x563162216430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.146+0000 7f24a67fc700 1 -- 172.21.15.44:0/883699142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220fb90 msgr2=0x563162275ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.146+0000 7f24a67fc700 1 --2- 172.21.15.44:0/883699142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220fb90 0x563162275ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.146+0000 7f24a67fc700 1 -- 172.21.15.44:0/883699142 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563161fdc5d0 con 0x56316220f720 2023-11-16T16:50:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.146+0000 7f24a67fc700 1 --2- 172.21.15.44:0/883699142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220f720 0x563162216970 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x56316220a1a0 tx=0x7f249c000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e2339013c5cacb0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.146+0000 7f248aff5700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:50:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.146+0000 7f248aff5700 1 -- 172.21.15.44:0/883699142 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c09f0f0 con 0x56316220f720 2023-11-16T16:50:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.146+0000 7f248aff5700 1 -- 172.21.15.44:0/883699142 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f249c0b7070 con 0x56316220f720 2023-11-16T16:50:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.146+0000 7f24c109bb80 1 -- 172.21.15.44:0/883699142 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f249c09b0c0 con 0x56316220f720 2023-11-16T16:50:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.147+0000 7f248aff5700 1 -- 172.21.15.44:0/883699142 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f249c0a3480 con 0x56316220f720 2023-11-16T16:50:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.147+0000 7f24c109bb80 10 client.25661 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.147+0000 7f24c109bb80 1 -- 172.21.15.44:0/883699142 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x563162121130 con 0x56316220f720 2023-11-16T16:50:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.147+0000 7f248aff5700 1 -- 172.21.15.44:0/883699142 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f249c07d020 con 0x56316220f720 2023-11-16T16:50:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.147+0000 7f248aff5700 1 -- 172.21.15.44:0/883699142 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f249c0ce200 con 0x56316220f720 2023-11-16T16:50:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.147+0000 7f24c109bb80 10 client.25661 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.147+0000 7f24c109bb80 10 client.25661 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.147+0000 7f24c109bb80 20 client.25661 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.147+0000 7f24c109bb80 1 --2- 172.21.15.44:0/883699142 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5631622773c0 0x5631622797d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.147+0000 7f24c109bb80 4 client.25661 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:50:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.147+0000 7f24c109bb80 1 -- 172.21.15.44:0/883699142 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563162129600 con 0x5631622773c0 2023-11-16T16:50:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.147+0000 7f24a6ffd700 1 --2- 172.21.15.44:0/883699142 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5631622773c0 0x5631622797d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.148+0000 7f24a6ffd700 1 --2- 172.21.15.44:0/883699142 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5631622773c0 0x5631622797d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb44fd698fa19dfd server_cookie=901cd1fce7cb8980 in_seq=0 out_seq=0 2023-11-16T16:50:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.148+0000 7f248aff5700 10 client.25661 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.155+0000 7f248aff5700 1 -- 172.21.15.44:0/883699142 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563162129600 con 0x5631622773c0 2023-11-16T16:50:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.155+0000 7f248aff5700 10 client.25661 handle_command_reply: tid=0 2023-11-16T16:50:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.156+0000 7f24c109bb80 1 client.25661 shutdown 2023-11-16T16:50:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.156+0000 7f24c109bb80 1 -- 172.21.15.44:0/883699142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220f720 msgr2=0x563162216970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.156+0000 7f24c109bb80 1 --2- 172.21.15.44:0/883699142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220f720 0x563162216970 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x56316220a1a0 tx=0x7f249c000bc0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.156+0000 7f24c109bb80 1 -- 172.21.15.44:0/883699142 shutdown_connections 2023-11-16T16:50:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.156+0000 7f24c109bb80 1 --2- 172.21.15.44:0/883699142 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5631622773c0 0x5631622797d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.156+0000 7f24c109bb80 1 --2- 172.21.15.44:0/883699142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56316220fb90 0x563162275ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.156+0000 7f24c109bb80 1 --2- 172.21.15.44:0/883699142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5631620656b0 0x563162216430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.156+0000 7f24c109bb80 1 --2- 172.21.15.44:0/883699142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56316220f720 0x563162216970 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.156+0000 7f248aff5700 0 client.25661 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.156+0000 7f24c109bb80 1 -- 172.21.15.44:0/883699142 >> 172.21.15.44:0/883699142 conn(0x5631621fef50 msgr2=0x56316220e390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.156+0000 7f24c109bb80 1 -- 172.21.15.44:0/883699142 shutdown_connections 2023-11-16T16:50:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.156+0000 7f24c109bb80 1 -- 172.21.15.44:0/883699142 wait complete. 2023-11-16T16:50:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.156+0000 7f24c109bb80 20 client.25661 trim_cache size 0 max 16384 2023-11-16T16:50:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:50:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:50:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:50:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:50:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.157+0000 7f24ad7ff700 1 -- 172.21.15.44:0/3215640308 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f24900a13e0 msgr2=0x7f24900a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.157+0000 7f24ad7ff700 1 --2- 172.21.15.44:0/3215640308 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f24900a13e0 0x7f24900a3890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009ec0 tx=0x7f24a0013250 comp rx=0 tx=0).stop 2023-11-16T16:50:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.157+0000 7f24ad7ff700 1 -- 172.21.15.44:0/3215640308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24a8074b00 msgr2=0x7f24a8198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.157+0000 7f24ad7ff700 1 --2- 172.21.15.44:0/3215640308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24a8074b00 0x7f24a8198b20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f249c005910 tx=0x7f249c004710 comp rx=0 tx=0).stop 2023-11-16T16:50:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.157+0000 7f24ad7ff700 1 -- 172.21.15.44:0/3215640308 shutdown_connections 2023-11-16T16:50:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.157+0000 7f24ad7ff700 1 --2- 172.21.15.44:0/3215640308 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f24900a13e0 0x7f24900a3890 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.157+0000 7f24ad7ff700 1 --2- 172.21.15.44:0/3215640308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24a810ce50 0x7f24a819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.157+0000 7f24ad7ff700 1 --2- 172.21.15.44:0/3215640308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a80746a0 0x7f24a81985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.157+0000 7f24ad7ff700 1 --2- 172.21.15.44:0/3215640308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24a8074b00 0x7f24a8198b20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.157+0000 7f24ad7ff700 1 -- 172.21.15.44:0/3215640308 >> 172.21.15.44:0/3215640308 conn(0x7f24a8077550 msgr2=0x7f24a8104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.157+0000 7f24ad7ff700 1 -- 172.21.15.44:0/3215640308 shutdown_connections 2023-11-16T16:50:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:38.157+0000 7f24ad7ff700 1 -- 172.21.15.44:0/3215640308 wait complete. 2023-11-16T16:50:38.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:38 smithi077 ceph-mon[131204]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2023-11-16T16:50:39.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:38 smithi044 ceph-mon[128750]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2023-11-16T16:50:39.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:38 smithi176 ceph-mon[131630]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2023-11-16T16:50:40.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:40 smithi077 ceph-mon[131204]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2023-11-16T16:50:41.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:40 smithi044 ceph-mon[128750]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2023-11-16T16:50:41.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:40 smithi176 ceph-mon[131630]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2023-11-16T16:50:42.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:42 smithi077 ceph-mon[131204]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.3 KiB/s wr, 4 op/s 2023-11-16T16:50:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:42 smithi044 ceph-mon[128750]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.3 KiB/s wr, 4 op/s 2023-11-16T16:50:43.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:42 smithi176 ceph-mon[131630]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.3 KiB/s wr, 4 op/s 2023-11-16T16:50:44.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:44 smithi077 ceph-mon[131204]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2023-11-16T16:50:44.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:44 smithi044 ceph-mon[128750]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2023-11-16T16:50:44.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:44 smithi176 ceph-mon[131630]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2023-11-16T16:50:46.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:46 smithi077 ceph-mon[131204]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2023-11-16T16:50:47.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:46 smithi044 ceph-mon[128750]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2023-11-16T16:50:47.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:46 smithi176 ceph-mon[131630]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2023-11-16T16:50:48.849 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:50:48.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:48 smithi077 ceph-mon[131204]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2023-11-16T16:50:49.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:48 smithi044 ceph-mon[128750]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2023-11-16T16:50:49.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.120+0000 7f56bd79d700 1 -- 172.21.15.44:0/882984817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b8068040 msgr2=0x7f56b8068410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:49.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.120+0000 7f56bd79d700 1 --2- 172.21.15.44:0/882984817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b8068040 0x7f56b8068410 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f56a80099c0 tx=0x7f56a8009cd0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.120+0000 7f56bd79d700 1 -- 172.21.15.44:0/882984817 shutdown_connections 2023-11-16T16:50:49.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.120+0000 7f56bd79d700 1 --2- 172.21.15.44:0/882984817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b8106500 0x7f56b810cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.120+0000 7f56bd79d700 1 --2- 172.21.15.44:0/882984817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b8068040 0x7f56b8068410 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.120+0000 7f56bd79d700 1 --2- 172.21.15.44:0/882984817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56b80689e0 0x7f56b8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.120+0000 7f56bd79d700 1 -- 172.21.15.44:0/882984817 >> 172.21.15.44:0/882984817 conn(0x7f56b80750c0 msgr2=0x7f56b80754c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:49.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.120+0000 7f56bd79d700 1 -- 172.21.15.44:0/882984817 shutdown_connections 2023-11-16T16:50:49.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.120+0000 7f56bd79d700 1 -- 172.21.15.44:0/882984817 wait complete. 2023-11-16T16:50:49.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.122+0000 7f56bd79d700 1 Processor -- start 2023-11-16T16:50:49.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.122+0000 7f56bd79d700 1 -- start start 2023-11-16T16:50:49.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56bd79d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b8068040 0x7f56b8196230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56bd79d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b80689e0 0x7f56b8196770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56bd79d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56b8106500 0x7f56b819abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56bd79d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b819b210 con 0x7f56b8106500 2023-11-16T16:50:49.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56bd79d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b819b380 con 0x7f56b8068040 2023-11-16T16:50:49.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56bd79d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b819b540 con 0x7f56b80689e0 2023-11-16T16:50:49.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56b77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b80689e0 0x7f56b8196770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56b7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b8068040 0x7f56b8196230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56b77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b80689e0 0x7f56b8196770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48222/0 (socket says 172.21.15.44:48222) 2023-11-16T16:50:49.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56b77fe700 1 -- 172.21.15.44:0/831080756 learned_addr learned my addr 172.21.15.44:0/831080756 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:49.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56bcf9c700 1 --2- 172.21.15.44:0/831080756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56b8106500 0x7f56b819abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56b77fe700 1 -- 172.21.15.44:0/831080756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b8068040 msgr2=0x7f56b8196230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:49.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56b77fe700 1 --2- 172.21.15.44:0/831080756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b8068040 0x7f56b8196230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56b77fe700 1 -- 172.21.15.44:0/831080756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56b8106500 msgr2=0x7f56b819abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:49.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56b77fe700 1 --2- 172.21.15.44:0/831080756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56b8106500 0x7f56b819abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56b77fe700 1 -- 172.21.15.44:0/831080756 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a80095f0 con 0x7f56b80689e0 2023-11-16T16:50:49.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56b7fff700 1 --2- 172.21.15.44:0/831080756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b8068040 0x7f56b8196230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:49.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.123+0000 7f56bcf9c700 1 --2- 172.21.15.44:0/831080756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56b8106500 0x7f56b819abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:49.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.124+0000 7f56b77fe700 1 --2- 172.21.15.44:0/831080756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b80689e0 0x7f56b8196770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00e940 tx=0x7f56ac00ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:49.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.124+0000 7f56b57fa700 1 -- 172.21.15.44:0/831080756 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac009730 con 0x7f56b80689e0 2023-11-16T16:50:49.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.124+0000 7f56b57fa700 1 -- 172.21.15.44:0/831080756 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56ac00ee90 con 0x7f56b80689e0 2023-11-16T16:50:49.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.124+0000 7f56b57fa700 1 -- 172.21.15.44:0/831080756 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac010ea0 con 0x7f56b80689e0 2023-11-16T16:50:49.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.124+0000 7f56bd79d700 1 -- 172.21.15.44:0/831080756 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b8110120 con 0x7f56b80689e0 2023-11-16T16:50:49.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.124+0000 7f56bd79d700 1 -- 172.21.15.44:0/831080756 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b810ff90 con 0x7f56b80689e0 2023-11-16T16:50:49.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.129+0000 7f56d2422b80 1 Processor -- start 2023-11-16T16:50:49.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.129+0000 7f56d2422b80 1 -- start start 2023-11-16T16:50:49.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.130+0000 7f56d2422b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e08d0 0x55ac011e0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.130+0000 7f56d2422b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e11e0 0x55ac011dabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.130+0000 7f56d2422b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011db100 0x55ac011db530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.130+0000 7f56b57fa700 1 -- 172.21.15.44:0/831080756 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f56ac00cbd0 con 0x7f56b80689e0 2023-11-16T16:50:49.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.130+0000 7f56d2422b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac010e0060 con 0x55ac011e11e0 2023-11-16T16:50:49.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.130+0000 7f56d2422b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac011328a0 con 0x55ac011db100 2023-11-16T16:50:49.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.130+0000 7f56d2422b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac0112bd30 con 0x55ac011e08d0 2023-11-16T16:50:49.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.130+0000 7f56bcf9c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e08d0 0x55ac011e0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.130+0000 7f56bcf9c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e08d0 0x55ac011e0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48230/0 (socket says 172.21.15.44:48230) 2023-11-16T16:50:49.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.130+0000 7f56bcf9c700 1 -- 172.21.15.44:0/427334929 learned_addr learned my addr 172.21.15.44:0/427334929 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:49.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.130+0000 7f56b7fff700 1 --2- 172.21.15.44:0/427334929 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e11e0 0x55ac011dabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.131+0000 7f56bcf9c700 1 -- 172.21.15.44:0/427334929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011db100 msgr2=0x55ac011db530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:50:49.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.131+0000 7f56bcf9c700 1 --2- 172.21.15.44:0/427334929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011db100 0x55ac011db530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.131+0000 7f56bcf9c700 1 -- 172.21.15.44:0/427334929 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e11e0 msgr2=0x55ac011dabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:49.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.131+0000 7f56bcf9c700 1 --2- 172.21.15.44:0/427334929 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e11e0 0x55ac011dabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.131+0000 7f56bcf9c700 1 -- 172.21.15.44:0/427334929 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac00fad630 con 0x55ac011e08d0 2023-11-16T16:50:49.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.131+0000 7f56b7fff700 1 --2- 172.21.15.44:0/427334929 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e11e0 0x55ac011dabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:49.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.131+0000 7f56bcf9c700 1 --2- 172.21.15.44:0/427334929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e08d0 0x55ac011e0ca0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009440 tx=0x7f56b0009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22df43f8d4527666 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:49.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.131+0000 7f569e7fc700 1 -- 172.21.15.44:0/427334929 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b000e070 con 0x55ac011e08d0 2023-11-16T16:50:49.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.131+0000 7f569e7fc700 1 -- 172.21.15.44:0/427334929 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56b000f040 con 0x55ac011e08d0 2023-11-16T16:50:49.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.131+0000 7f569e7fc700 1 -- 172.21.15.44:0/427334929 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0014a20 con 0x55ac011e08d0 2023-11-16T16:50:49.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.132+0000 7f56b57fa700 1 --2- 172.21.15.44:0/831080756 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f56a00a14e0 0x7f56a00a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.132+0000 7f56d2422b80 1 -- 172.21.15.44:0/427334929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e08d0 msgr2=0x55ac011e0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:49.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.132+0000 7f56d2422b80 1 --2- 172.21.15.44:0/427334929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e08d0 0x55ac011e0ca0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009440 tx=0x7f56b0009920 comp rx=0 tx=0).stop 2023-11-16T16:50:49.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.132+0000 7f56b57fa700 1 -- 172.21.15.44:0/831080756 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f56ac014070 con 0x7f56b80689e0 2023-11-16T16:50:49.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.132+0000 7f56d2422b80 1 -- 172.21.15.44:0/427334929 shutdown_connections 2023-11-16T16:50:49.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.132+0000 7f56b7fff700 1 --2- 172.21.15.44:0/831080756 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f56a00a14e0 0x7f56a00a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.132+0000 7f56d2422b80 1 --2- 172.21.15.44:0/427334929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011db100 0x55ac011db530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.132+0000 7f56d2422b80 1 --2- 172.21.15.44:0/427334929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e08d0 0x55ac011e0ca0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.132+0000 7f56d2422b80 1 --2- 172.21.15.44:0/427334929 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e11e0 0x55ac011dabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.132+0000 7f56d2422b80 1 -- 172.21.15.44:0/427334929 >> 172.21.15.44:0/427334929 conn(0x55ac011d05b0 msgr2=0x55ac011d5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:49.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.133+0000 7f56d2422b80 1 -- 172.21.15.44:0/427334929 shutdown_connections 2023-11-16T16:50:49.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.133+0000 7f56d2422b80 1 -- 172.21.15.44:0/427334929 wait complete. 2023-11-16T16:50:49.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.133+0000 7f56b7fff700 1 --2- 172.21.15.44:0/831080756 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f56a00a14e0 0x7f56a00a3990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009990 tx=0x7f56a8005fb0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:49.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.133+0000 7f56d2422b80 1 -- start start 2023-11-16T16:50:49.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.134+0000 7f56d2422b80 1 Processor -- start 2023-11-16T16:50:49.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.134+0000 7f56d2422b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011db100 0x55ac011e71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.134+0000 7f56d2422b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011e08d0 0x55ac011e4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.134+0000 7f56d2422b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e11e0 0x55ac011e4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.134+0000 7f56d2422b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac010cd8e0 con 0x55ac011db100 2023-11-16T16:50:49.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.134+0000 7f56d2422b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac010cd580 con 0x55ac011e08d0 2023-11-16T16:50:49.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.134+0000 7f56d2422b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac00faf920 con 0x55ac011e11e0 2023-11-16T16:50:49.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.135+0000 7f56d2422b80 10 client.0 _ll_register_callbacks cb 0x55ac010eb780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:49.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.135+0000 7f56b7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011db100 0x55ac011e71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.135+0000 7f56b7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011db100 0x55ac011e71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57638/0 (socket says 172.21.15.44:57638) 2023-11-16T16:50:49.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.135+0000 7f56b7fff700 1 -- 172.21.15.44:0/245487790 learned_addr learned my addr 172.21.15.44:0/245487790 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:49.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.135+0000 7f56b77fe700 1 --2- 172.21.15.44:0/245487790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011e08d0 0x55ac011e4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.135+0000 7f56bcf9c700 1 --2- 172.21.15.44:0/245487790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e11e0 0x55ac011e4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.135+0000 7f56b7fff700 1 -- 172.21.15.44:0/245487790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e11e0 msgr2=0x55ac011e4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:49.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.135+0000 7f56b7fff700 1 --2- 172.21.15.44:0/245487790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e11e0 0x55ac011e4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.135+0000 7f56b7fff700 1 -- 172.21.15.44:0/245487790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011e08d0 msgr2=0x55ac011e4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:49.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.135+0000 7f56b7fff700 1 --2- 172.21.15.44:0/245487790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011e08d0 0x55ac011e4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.135+0000 7f56b7fff700 1 -- 172.21.15.44:0/245487790 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b00090f0 con 0x55ac011db100 2023-11-16T16:50:49.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.135+0000 7f56b77fe700 1 --2- 172.21.15.44:0/245487790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011e08d0 0x55ac011e4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:49.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.136+0000 7f56b7fff700 1 --2- 172.21.15.44:0/245487790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011db100 0x55ac011e71b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f56a8000c00 tx=0x7f56a8009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb4ab31fdd46437 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:49.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.136+0000 7f56977fe700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:50:49.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.136+0000 7f56977fe700 1 -- 172.21.15.44:0/245487790 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a802b070 con 0x55ac011db100 2023-11-16T16:50:49.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.136+0000 7f56977fe700 1 -- 172.21.15.44:0/245487790 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f56a80055b0 con 0x55ac011db100 2023-11-16T16:50:49.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.136+0000 7f56d2422b80 1 -- 172.21.15.44:0/245487790 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ac011cdd70 con 0x55ac011db100 2023-11-16T16:50:49.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.137+0000 7f56977fe700 1 -- 172.21.15.44:0/245487790 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56a800f460 con 0x55ac011db100 2023-11-16T16:50:49.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.137+0000 7f56d2422b80 10 client.15657 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:49.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.137+0000 7f56d2422b80 1 -- 172.21.15.44:0/245487790 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55ac010f2190 con 0x55ac011db100 2023-11-16T16:50:49.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.137+0000 7f56977fe700 1 -- 172.21.15.44:0/245487790 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ac011cdd70 con 0x55ac011db100 2023-11-16T16:50:49.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.137+0000 7f56977fe700 1 -- 172.21.15.44:0/245487790 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f56a8037070 con 0x55ac011db100 2023-11-16T16:50:49.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.137+0000 7f56d2422b80 10 client.15657 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:49.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.137+0000 7f56d2422b80 10 client.15657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:49.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.137+0000 7f56d2422b80 20 client.15657 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:49.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.137+0000 7f56d2422b80 1 --2- 172.21.15.44:0/245487790 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ac011fd360 0x55ac011ff770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.137+0000 7f56d2422b80 4 client.15657 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:49.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.137+0000 7f56d2422b80 1 -- 172.21.15.44:0/245487790 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ac010ee0a0 con 0x55ac011fd360 2023-11-16T16:50:49.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.138+0000 7f56b77fe700 1 --2- 172.21.15.44:0/245487790 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ac011fd360 0x55ac011ff770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.138+0000 7f56b77fe700 1 --2- 172.21.15.44:0/245487790 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ac011fd360 0x55ac011ff770 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f9f45584487093b server_cookie=80c641b8acf2b877 in_seq=0 out_seq=0 2023-11-16T16:50:49.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.138+0000 7f56977fe700 10 client.15657 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:49.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.139+0000 7f56977fe700 1 -- 172.21.15.44:0/245487790 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ac010ee0a0 con 0x55ac011fd360 2023-11-16T16:50:49.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.139+0000 7f56977fe700 10 client.15657 handle_command_reply: tid=0 2023-11-16T16:50:49.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.139+0000 7f56d2422b80 1 client.15657 shutdown 2023-11-16T16:50:49.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.140+0000 7f56d2422b80 1 -- 172.21.15.44:0/245487790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011db100 msgr2=0x55ac011e71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:49.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.140+0000 7f56d2422b80 1 --2- 172.21.15.44:0/245487790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011db100 0x55ac011e71b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f56a8000c00 tx=0x7f56a8009cd0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.140+0000 7f56d2422b80 1 -- 172.21.15.44:0/245487790 shutdown_connections 2023-11-16T16:50:49.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.140+0000 7f56d2422b80 1 --2- 172.21.15.44:0/245487790 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ac011fd360 0x55ac011ff770 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.140+0000 7f56d2422b80 1 --2- 172.21.15.44:0/245487790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e11e0 0x55ac011e4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.140+0000 7f56977fe700 0 client.15657 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:49.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.140+0000 7f56d2422b80 1 --2- 172.21.15.44:0/245487790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011db100 0x55ac011e71b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.140+0000 7f56d2422b80 1 --2- 172.21.15.44:0/245487790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011e08d0 0x55ac011e4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.140+0000 7f56d2422b80 1 -- 172.21.15.44:0/245487790 >> 172.21.15.44:0/245487790 conn(0x55ac011d05b0 msgr2=0x55ac011d5030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:49.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.140+0000 7f56d2422b80 1 -- 172.21.15.44:0/245487790 shutdown_connections 2023-11-16T16:50:49.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.140+0000 7f56d2422b80 1 -- 172.21.15.44:0/245487790 wait complete. 2023-11-16T16:50:49.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.140+0000 7f56d2422b80 20 client.15657 trim_cache size 0 max 16384 2023-11-16T16:50:49.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:48 smithi176 ceph-mon[131630]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2023-11-16T16:50:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.174+0000 7f56d2422b80 1 Processor -- start 2023-11-16T16:50:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.174+0000 7f56d2422b80 1 -- start start 2023-11-16T16:50:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.174+0000 7f56d2422b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011db100 0x55ac011e6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.174+0000 7f56d2422b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e08d0 0x55ac011d57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.174+0000 7f56d2422b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011e11e0 0x55ac011e3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.174+0000 7f56d2422b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac011cd8b0 con 0x55ac011e08d0 2023-11-16T16:50:49.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.174+0000 7f56d2422b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac011e29b0 con 0x55ac011e11e0 2023-11-16T16:50:49.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.174+0000 7f56d2422b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac011df140 con 0x55ac011db100 2023-11-16T16:50:49.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.175+0000 7f56b77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e08d0 0x55ac011d57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.175+0000 7f56b77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e08d0 0x55ac011d57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57646/0 (socket says 172.21.15.44:57646) 2023-11-16T16:50:49.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.175+0000 7f56bcf9c700 1 --2- 172.21.15.44:0/2318041416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011e11e0 0x55ac011e3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.175+0000 7f56b77fe700 1 -- 172.21.15.44:0/2318041416 learned_addr learned my addr 172.21.15.44:0/2318041416 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:49.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.175+0000 7f56b7fff700 1 --2- 172.21.15.44:0/2318041416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011db100 0x55ac011e6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.175+0000 7f56bcf9c700 1 -- 172.21.15.44:0/2318041416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011db100 msgr2=0x55ac011e6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:49.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.175+0000 7f56bcf9c700 1 --2- 172.21.15.44:0/2318041416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011db100 0x55ac011e6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.175+0000 7f56bcf9c700 1 -- 172.21.15.44:0/2318041416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e08d0 msgr2=0x55ac011d57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:49.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.175+0000 7f56bcf9c700 1 --2- 172.21.15.44:0/2318041416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e08d0 0x55ac011d57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.175+0000 7f56bcf9c700 1 -- 172.21.15.44:0/2318041416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac010f1ee0 con 0x55ac011e11e0 2023-11-16T16:50:49.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.176+0000 7f56bcf9c700 1 --2- 172.21.15.44:0/2318041416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011e11e0 0x55ac011e3c10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f56b0016ff0 tx=0x7f56b0004c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc6e5b2d1091c431 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:49.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.176+0000 7f569effd700 1 -- 172.21.15.44:0/2318041416 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b000e040 con 0x55ac011e11e0 2023-11-16T16:50:49.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.176+0000 7f569effd700 1 -- 172.21.15.44:0/2318041416 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56b0004020 con 0x55ac011e11e0 2023-11-16T16:50:49.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.177+0000 7f56d2422b80 1 -- 172.21.15.44:0/2318041416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011e11e0 msgr2=0x55ac011e3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:49.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.177+0000 7f56d2422b80 1 --2- 172.21.15.44:0/2318041416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011e11e0 0x55ac011e3c10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f56b0016ff0 tx=0x7f56b0004c60 comp rx=0 tx=0).stop 2023-11-16T16:50:49.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.177+0000 7f56d2422b80 1 -- 172.21.15.44:0/2318041416 shutdown_connections 2023-11-16T16:50:49.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.177+0000 7f56d2422b80 1 --2- 172.21.15.44:0/2318041416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011e11e0 0x55ac011e3c10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.177+0000 7f56d2422b80 1 --2- 172.21.15.44:0/2318041416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011db100 0x55ac011e6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.177+0000 7f56d2422b80 1 --2- 172.21.15.44:0/2318041416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e08d0 0x55ac011d57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.177+0000 7f56d2422b80 1 -- 172.21.15.44:0/2318041416 >> 172.21.15.44:0/2318041416 conn(0x55ac011d05b0 msgr2=0x55ac011f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:49.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.177+0000 7f56d2422b80 1 -- 172.21.15.44:0/2318041416 shutdown_connections 2023-11-16T16:50:49.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.177+0000 7f56d2422b80 1 -- 172.21.15.44:0/2318041416 wait complete. 2023-11-16T16:50:49.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.178+0000 7f56d2422b80 1 -- start start 2023-11-16T16:50:49.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.178+0000 7f56d2422b80 1 Processor -- start 2023-11-16T16:50:49.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.179+0000 7f56d2422b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011db100 0x55ac011f41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.179+0000 7f56d2422b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e08d0 0x55ac011f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.179+0000 7f56d2422b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e11e0 0x55ac011f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.179+0000 7f56d2422b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac011f2f70 con 0x55ac011e11e0 2023-11-16T16:50:49.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.179+0000 7f56d2422b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac011f30e0 con 0x55ac011db100 2023-11-16T16:50:49.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.179+0000 7f56d2422b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac011f32a0 con 0x55ac011e08d0 2023-11-16T16:50:49.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.179+0000 7f56d2422b80 10 client.0 _ll_register_callbacks cb 0x7f56a80037d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:49.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.179+0000 7f56b7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011db100 0x55ac011f41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.179+0000 7f56b7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011db100 0x55ac011f41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:53746/0 (socket says 172.21.15.44:53746) 2023-11-16T16:50:49.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.179+0000 7f56b7fff700 1 -- 172.21.15.44:0/1178696454 learned_addr learned my addr 172.21.15.44:0/1178696454 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:49.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.179+0000 7f56bcf9c700 1 --2- 172.21.15.44:0/1178696454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e11e0 0x55ac011f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.180+0000 7f56b77fe700 1 --2- 172.21.15.44:0/1178696454 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e08d0 0x55ac011f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.180+0000 7f56b7fff700 1 -- 172.21.15.44:0/1178696454 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e08d0 msgr2=0x55ac011f24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:49.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.180+0000 7f56b7fff700 1 --2- 172.21.15.44:0/1178696454 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e08d0 0x55ac011f24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.180+0000 7f56b7fff700 1 -- 172.21.15.44:0/1178696454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e11e0 msgr2=0x55ac011f2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:49.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.180+0000 7f56b7fff700 1 --2- 172.21.15.44:0/1178696454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e11e0 0x55ac011f2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.180+0000 7f56b7fff700 1 -- 172.21.15.44:0/1178696454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ac00fad630 con 0x55ac011db100 2023-11-16T16:50:49.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.180+0000 7f56b77fe700 1 --2- 172.21.15.44:0/1178696454 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e08d0 0x55ac011f24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:49.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.180+0000 7f56bcf9c700 1 --2- 172.21.15.44:0/1178696454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e11e0 0x55ac011f2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:49.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.180+0000 7f56b7fff700 1 --2- 172.21.15.44:0/1178696454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011db100 0x55ac011f41c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009fd0 tx=0x7f56a8030bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=323cb5ed96a5d5fb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:49.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.180+0000 7f56977fe700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:50:49.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.181+0000 7f56977fe700 1 -- 172.21.15.44:0/1178696454 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a802b040 con 0x55ac011db100 2023-11-16T16:50:49.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.181+0000 7f56977fe700 1 -- 172.21.15.44:0/1178696454 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f56a800bd20 con 0x55ac011db100 2023-11-16T16:50:49.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.181+0000 7f56d2422b80 1 -- 172.21.15.44:0/1178696454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f56b000f040 con 0x55ac011db100 2023-11-16T16:50:49.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.181+0000 7f56977fe700 1 -- 172.21.15.44:0/1178696454 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56a8005c60 con 0x55ac011db100 2023-11-16T16:50:49.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.181+0000 7f56d2422b80 10 client.25676 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:49.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.181+0000 7f56d2422b80 1 -- 172.21.15.44:0/1178696454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55ac00f42c60 con 0x55ac011db100 2023-11-16T16:50:49.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.182+0000 7f56977fe700 1 -- 172.21.15.44:0/1178696454 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56b000f040 con 0x55ac011db100 2023-11-16T16:50:49.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.183+0000 7f56977fe700 1 -- 172.21.15.44:0/1178696454 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f56a8037070 con 0x55ac011db100 2023-11-16T16:50:49.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.183+0000 7f56d2422b80 10 client.25676 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:49.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.183+0000 7f56d2422b80 10 client.25676 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:49.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.183+0000 7f56d2422b80 20 client.25676 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:49.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.183+0000 7f56d2422b80 1 --2- 172.21.15.44:0/1178696454 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ac01252170 0x55ac01254550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.183+0000 7f56d2422b80 4 client.25676 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:50:49.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.183+0000 7f56d2422b80 1 -- 172.21.15.44:0/1178696454 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ac010fa660 con 0x55ac01252170 2023-11-16T16:50:49.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.183+0000 7f56b77fe700 1 --2- 172.21.15.44:0/1178696454 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ac01252170 0x55ac01254550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.184+0000 7f56b77fe700 1 --2- 172.21.15.44:0/1178696454 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ac01252170 0x55ac01254550 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca02c8ee51a33e7f server_cookie=fae982f905891a7c in_seq=0 out_seq=0 2023-11-16T16:50:49.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.184+0000 7f56977fe700 10 client.25676 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:49.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.184+0000 7f56977fe700 1 -- 172.21.15.44:0/1178696454 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ac010fa660 con 0x55ac01252170 2023-11-16T16:50:49.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.184+0000 7f56977fe700 10 client.25676 handle_command_reply: tid=0 2023-11-16T16:50:49.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.184+0000 7f56d2422b80 1 client.25676 shutdown 2023-11-16T16:50:49.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.184+0000 7f56d2422b80 1 -- 172.21.15.44:0/1178696454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011db100 msgr2=0x55ac011f41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:49.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.184+0000 7f56d2422b80 1 --2- 172.21.15.44:0/1178696454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011db100 0x55ac011f41c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009fd0 tx=0x7f56a8030bb0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.184+0000 7f56d2422b80 1 -- 172.21.15.44:0/1178696454 shutdown_connections 2023-11-16T16:50:49.189 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.184+0000 7f56d2422b80 1 --2- 172.21.15.44:0/1178696454 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ac01252170 0x55ac01254550 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.189 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.184+0000 7f56d2422b80 1 --2- 172.21.15.44:0/1178696454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ac011e11e0 0x55ac011f2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.189 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.184+0000 7f56d2422b80 1 --2- 172.21.15.44:0/1178696454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ac011db100 0x55ac011f41c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.189 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.184+0000 7f56d2422b80 1 --2- 172.21.15.44:0/1178696454 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ac011e08d0 0x55ac011f24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.189 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.184+0000 7f56d2422b80 1 -- 172.21.15.44:0/1178696454 >> 172.21.15.44:0/1178696454 conn(0x55ac011d05b0 msgr2=0x55ac011df550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:49.189 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.184+0000 7f56977fe700 0 client.25676 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:49.189 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.185+0000 7f56d2422b80 1 -- 172.21.15.44:0/1178696454 shutdown_connections 2023-11-16T16:50:49.189 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.185+0000 7f56d2422b80 1 -- 172.21.15.44:0/1178696454 wait complete. 2023-11-16T16:50:49.189 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.185+0000 7f56d2422b80 20 client.25676 trim_cache size 0 max 16384 2023-11-16T16:50:49.190 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:50:49.190 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T16:50:49.190 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T16:50:49.190 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:50:49.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.186+0000 7f56bd79d700 1 -- 172.21.15.44:0/831080756 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f56a00a14e0 msgr2=0x7f56a00a3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:49.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.186+0000 7f56bd79d700 1 --2- 172.21.15.44:0/831080756 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f56a00a14e0 0x7f56a00a3990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009990 tx=0x7f56a8005fb0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.186+0000 7f56bd79d700 1 -- 172.21.15.44:0/831080756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b80689e0 msgr2=0x7f56b8196770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:49.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.186+0000 7f56bd79d700 1 --2- 172.21.15.44:0/831080756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b80689e0 0x7f56b8196770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00e940 tx=0x7f56ac00ec50 comp rx=0 tx=0).stop 2023-11-16T16:50:49.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.186+0000 7f56bd79d700 1 -- 172.21.15.44:0/831080756 shutdown_connections 2023-11-16T16:50:49.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.186+0000 7f56bd79d700 1 --2- 172.21.15.44:0/831080756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56b8106500 0x7f56b819abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.186+0000 7f56bd79d700 1 --2- 172.21.15.44:0/831080756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b8068040 0x7f56b8196230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.186+0000 7f56bd79d700 1 --2- 172.21.15.44:0/831080756 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f56a00a14e0 0x7f56a00a3990 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.186+0000 7f56bd79d700 1 --2- 172.21.15.44:0/831080756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b80689e0 0x7f56b8196770 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.186+0000 7f56bd79d700 1 -- 172.21.15.44:0/831080756 >> 172.21.15.44:0/831080756 conn(0x7f56b80750c0 msgr2=0x7f56b8103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:49.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.186+0000 7f56bd79d700 1 -- 172.21.15.44:0/831080756 shutdown_connections 2023-11-16T16:50:49.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.186+0000 7f56bd79d700 1 -- 172.21.15.44:0/831080756 wait complete. 2023-11-16T16:50:49.216 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T16:50:49.217 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:50:49.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.558+0000 7ff03cbff700 1 -- 172.21.15.44:0/141633632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff038100550 msgr2=0x7ff038100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:49.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.558+0000 7ff03cbff700 1 --2- 172.21.15.44:0/141633632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff038100550 0x7ff038100920 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff0280099c0 tx=0x7ff028009cd0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.558+0000 7ff03cbff700 1 -- 172.21.15.44:0/141633632 shutdown_connections 2023-11-16T16:50:49.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.558+0000 7ff03cbff700 1 --2- 172.21.15.44:0/141633632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff038105540 0x7ff03810d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.558+0000 7ff03cbff700 1 --2- 172.21.15.44:0/141633632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff038100550 0x7ff038100920 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.558+0000 7ff03cbff700 1 --2- 172.21.15.44:0/141633632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff038100ef0 0x7ff038104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.558+0000 7ff03cbff700 1 -- 172.21.15.44:0/141633632 >> 172.21.15.44:0/141633632 conn(0x7ff038077020 msgr2=0x7ff038077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:49.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.558+0000 7ff03cbff700 1 -- 172.21.15.44:0/141633632 shutdown_connections 2023-11-16T16:50:49.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.558+0000 7ff03cbff700 1 -- 172.21.15.44:0/141633632 wait complete. 2023-11-16T16:50:49.561 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.560+0000 7ff03cbff700 1 Processor -- start 2023-11-16T16:50:49.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.560+0000 7ff03cbff700 1 -- start start 2023-11-16T16:50:49.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.561+0000 7ff03cbff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff038100550 0x7ff03810bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.561+0000 7ff03cbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff038100ef0 0x7ff03810c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.561+0000 7ff03cbff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff038105540 0x7ff0381b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.561+0000 7ff03cbff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0381b15b0 con 0x7ff038100550 2023-11-16T16:50:49.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.561+0000 7ff03cbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0381b16f0 con 0x7ff038100ef0 2023-11-16T16:50:49.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.561+0000 7ff03cbff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0381b1830 con 0x7ff038105540 2023-11-16T16:50:49.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.561+0000 7ff03659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff038100550 0x7ff03810bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.561+0000 7ff036d9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff038105540 0x7ff0381b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.561+0000 7ff03659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff038100550 0x7ff03810bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57690/0 (socket says 172.21.15.44:57690) 2023-11-16T16:50:49.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.561+0000 7ff036d9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff038105540 0x7ff0381b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48272/0 (socket says 172.21.15.44:48272) 2023-11-16T16:50:49.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.561+0000 7ff03659c700 1 -- 172.21.15.44:0/1208041720 learned_addr learned my addr 172.21.15.44:0/1208041720 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:49.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.561+0000 7ff035d9b700 1 --2- 172.21.15.44:0/1208041720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff038100ef0 0x7ff03810c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.562+0000 7ff036d9d700 1 -- 172.21.15.44:0/1208041720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff038100ef0 msgr2=0x7ff03810c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:49.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.562+0000 7ff036d9d700 1 --2- 172.21.15.44:0/1208041720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff038100ef0 0x7ff03810c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.562+0000 7ff036d9d700 1 -- 172.21.15.44:0/1208041720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff038100550 msgr2=0x7ff03810bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:49.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.562+0000 7ff036d9d700 1 --2- 172.21.15.44:0/1208041720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff038100550 0x7ff03810bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:49.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.562+0000 7ff036d9d700 1 -- 172.21.15.44:0/1208041720 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0280095f0 con 0x7ff038105540 2023-11-16T16:50:49.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.562+0000 7ff036d9d700 1 --2- 172.21.15.44:0/1208041720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff038105540 0x7ff0381b1070 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff030009f60 tx=0x7ff030009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:49.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.562+0000 7ff0277fe700 1 -- 172.21.15.44:0/1208041720 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff030018070 con 0x7ff038105540 2023-11-16T16:50:49.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.563+0000 7ff0277fe700 1 -- 172.21.15.44:0/1208041720 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff03000f040 con 0x7ff038105540 2023-11-16T16:50:49.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.563+0000 7ff0277fe700 1 -- 172.21.15.44:0/1208041720 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03000ccc0 con 0x7ff038105540 2023-11-16T16:50:49.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.563+0000 7ff03cbff700 1 -- 172.21.15.44:0/1208041720 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff038110350 con 0x7ff038105540 2023-11-16T16:50:49.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.564+0000 7ff03cbff700 1 -- 172.21.15.44:0/1208041720 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0381101c0 con 0x7ff038105540 2023-11-16T16:50:49.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.565+0000 7ff03cbff700 1 -- 172.21.15.44:0/1208041720 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0381104e0 con 0x7ff038105540 2023-11-16T16:50:49.570 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.569+0000 7ff0277fe700 1 -- 172.21.15.44:0/1208041720 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7ff030003a00 con 0x7ff038105540 2023-11-16T16:50:49.572 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.571+0000 7ff0277fe700 1 --2- 172.21.15.44:0/1208041720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff0200a1460 0x7ff0200a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:49.573 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.571+0000 7ff0277fe700 1 -- 172.21.15.44:0/1208041720 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7ff0300cc5e0 con 0x7ff038105540 2023-11-16T16:50:49.573 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.572+0000 7ff03659c700 1 --2- 172.21.15.44:0/1208041720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff0200a1460 0x7ff0200a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:49.574 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.573+0000 7ff03659c700 1 --2- 172.21.15.44:0/1208041720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff0200a1460 0x7ff0200a3910 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff0280036c0 tx=0x7ff028005fb0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:49.576 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:49.574+0000 7ff0277fe700 1 -- 172.21.15.44:0/1208041720 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff0300953c0 con 0x7ff038105540 2023-11-16T16:50:50.082 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.081+0000 7ff03cbff700 1 -- 172.21.15.44:0/1208041720 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff038104cf0 con 0x7ff038105540 2023-11-16T16:50:50.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.082+0000 7ff0277fe700 1 -- 172.21.15.44:0/1208041720 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7ff0300989e0 con 0x7ff038105540 2023-11-16T16:50:50.084 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:50:50.084 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:50:50.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.085+0000 7ff03cbff700 1 -- 172.21.15.44:0/1208041720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff0200a1460 msgr2=0x7ff0200a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.085+0000 7ff03cbff700 1 --2- 172.21.15.44:0/1208041720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff0200a1460 0x7ff0200a3910 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff0280036c0 tx=0x7ff028005fb0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.085+0000 7ff03cbff700 1 -- 172.21.15.44:0/1208041720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff038105540 msgr2=0x7ff0381b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.085+0000 7ff03cbff700 1 --2- 172.21.15.44:0/1208041720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff038105540 0x7ff0381b1070 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff030009f60 tx=0x7ff030009f90 comp rx=0 tx=0).stop 2023-11-16T16:50:50.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.085+0000 7ff03cbff700 1 -- 172.21.15.44:0/1208041720 shutdown_connections 2023-11-16T16:50:50.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.085+0000 7ff03cbff700 1 --2- 172.21.15.44:0/1208041720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff0200a1460 0x7ff0200a3910 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.085+0000 7ff03cbff700 1 --2- 172.21.15.44:0/1208041720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff038105540 0x7ff0381b1070 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.085+0000 7ff03cbff700 1 --2- 172.21.15.44:0/1208041720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff038100550 0x7ff03810bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.085+0000 7ff03cbff700 1 --2- 172.21.15.44:0/1208041720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff038100ef0 0x7ff03810c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.085+0000 7ff03cbff700 1 -- 172.21.15.44:0/1208041720 >> 172.21.15.44:0/1208041720 conn(0x7ff038077020 msgr2=0x7ff038101e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:50.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.085+0000 7ff03cbff700 1 -- 172.21.15.44:0/1208041720 shutdown_connections 2023-11-16T16:50:50.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.086+0000 7ff03cbff700 1 -- 172.21.15.44:0/1208041720 wait complete. 2023-11-16T16:50:50.088 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:50:50.118 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T16:50:50.465 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.463+0000 7f0f153ff700 1 -- 172.21.15.44:0/2461872899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f10100cc0 msgr2=0x7f0f10101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.465 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.463+0000 7f0f153ff700 1 --2- 172.21.15.44:0/2461872899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f10100cc0 0x7f0f10101090 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0f000099c0 tx=0x7f0f00009cd0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.465 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.463+0000 7f0f153ff700 1 -- 172.21.15.44:0/2461872899 shutdown_connections 2023-11-16T16:50:50.465 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.463+0000 7f0f153ff700 1 --2- 172.21.15.44:0/2461872899 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f10105620 0x7f0f1010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.465 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.463+0000 7f0f153ff700 1 --2- 172.21.15.44:0/2461872899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f10100cc0 0x7f0f10101090 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.465 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.463+0000 7f0f153ff700 1 --2- 172.21.15.44:0/2461872899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f10101660 0x7f0f10104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.465 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.463+0000 7f0f153ff700 1 -- 172.21.15.44:0/2461872899 >> 172.21.15.44:0/2461872899 conn(0x7f0f10077560 msgr2=0x7f0f10077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.463+0000 7f0f153ff700 1 -- 172.21.15.44:0/2461872899 shutdown_connections 2023-11-16T16:50:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.463+0000 7f0f153ff700 1 -- 172.21.15.44:0/2461872899 wait complete. 2023-11-16T16:50:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.465+0000 7f0f153ff700 1 Processor -- start 2023-11-16T16:50:50.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.465+0000 7f0f153ff700 1 -- start start 2023-11-16T16:50:50.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.466+0000 7f0f153ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f10100cc0 0x7f0f10198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.466+0000 7f0f153ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f10101660 0x7f0f10198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.466+0000 7f0f153ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f10105620 0x7f0f1019d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.466+0000 7f0f153ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1019d640 con 0x7f0f10101660 2023-11-16T16:50:50.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.466+0000 7f0f153ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1019d7b0 con 0x7f0f10100cc0 2023-11-16T16:50:50.468 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.466+0000 7f0f153ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1019d970 con 0x7f0f10105620 2023-11-16T16:50:50.469 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.466+0000 7f0f0f7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f10105620 0x7f0f1019d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.469 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.467+0000 7f0f0f7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f10105620 0x7f0f1019d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48292/0 (socket says 172.21.15.44:48292) 2023-11-16T16:50:50.469 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.467+0000 7f0f0f7fe700 1 -- 172.21.15.44:0/3164619921 learned_addr learned my addr 172.21.15.44:0/3164619921 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:50.469 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.467+0000 7f0f0effd700 1 --2- 172.21.15.44:0/3164619921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f10100cc0 0x7f0f10198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.469 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.467+0000 7f0f0e7fc700 1 --2- 172.21.15.44:0/3164619921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f10101660 0x7f0f10198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.469 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.467+0000 7f0f0effd700 1 -- 172.21.15.44:0/3164619921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f10105620 msgr2=0x7f0f1019d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.469 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.467+0000 7f0f0effd700 1 --2- 172.21.15.44:0/3164619921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f10105620 0x7f0f1019d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.469 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.467+0000 7f0f0effd700 1 -- 172.21.15.44:0/3164619921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f10101660 msgr2=0x7f0f10198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.469 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.467+0000 7f0f0effd700 1 --2- 172.21.15.44:0/3164619921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f10101660 0x7f0f10198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.469 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.467+0000 7f0f0effd700 1 -- 172.21.15.44:0/3164619921 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f000095f0 con 0x7f0f10100cc0 2023-11-16T16:50:50.469 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.467+0000 7f0f0e7fc700 1 --2- 172.21.15.44:0/3164619921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f10101660 0x7f0f10198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:50.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.467+0000 7f0f0effd700 1 --2- 172.21.15.44:0/3164619921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f10100cc0 0x7f0f10198660 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009f50 tx=0x7f0f00005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:50.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.468+0000 7f0ef7fff700 1 -- 172.21.15.44:0/3164619921 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0001d070 con 0x7f0f10100cc0 2023-11-16T16:50:50.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.468+0000 7f0ef7fff700 1 -- 172.21.15.44:0/3164619921 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f00003f90 con 0x7f0f10100cc0 2023-11-16T16:50:50.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.468+0000 7f0ef7fff700 1 -- 172.21.15.44:0/3164619921 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0000fc40 con 0x7f0f10100cc0 2023-11-16T16:50:50.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.469+0000 7f0f153ff700 1 -- 172.21.15.44:0/3164619921 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f101103d0 con 0x7f0f10100cc0 2023-11-16T16:50:50.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.469+0000 7f0f153ff700 1 -- 172.21.15.44:0/3164619921 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f10110240 con 0x7f0f10100cc0 2023-11-16T16:50:50.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.474+0000 7f0ef7fff700 1 -- 172.21.15.44:0/3164619921 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f0f0000f460 con 0x7f0f10100cc0 2023-11-16T16:50:50.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.475+0000 7f0f28dacb80 1 Processor -- start 2023-11-16T16:50:50.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.476+0000 7f0f28dacb80 1 -- start start 2023-11-16T16:50:50.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.476+0000 7f0f28dacb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3b5b60 0x563dcd3b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.476+0000 7f0f28dacb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b6470 0x563dcd3ba370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.476+0000 7f0f28dacb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3ba8b0 0x563dcd3bad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.476+0000 7f0f28dacb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcd2bb060 con 0x563dcd3ba8b0 2023-11-16T16:50:50.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.476+0000 7f0f28dacb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcd30d8a0 con 0x563dcd3b6470 2023-11-16T16:50:50.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.476+0000 7f0f28dacb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcd306d30 con 0x563dcd3b5b60 2023-11-16T16:50:50.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.477+0000 7f0ef7fff700 1 --2- 172.21.15.44:0/3164619921 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0ef80a14e0 0x7f0ef80a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.477+0000 7f0f0f7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3b5b60 0x563dcd3b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.477+0000 7f0f0f7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3b5b60 0x563dcd3b5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48306/0 (socket says 172.21.15.44:48306) 2023-11-16T16:50:50.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.477+0000 7f0f0f7fe700 1 -- 172.21.15.44:0/2272199888 learned_addr learned my addr 172.21.15.44:0/2272199888 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:50.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.477+0000 7f0ef7fff700 1 -- 172.21.15.44:0/3164619921 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f0f000cdd90 con 0x7f0f10100cc0 2023-11-16T16:50:50.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.477+0000 7f0f0f7fe700 1 --2- 172.21.15.44:0/3164619921 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0ef80a14e0 0x7f0ef80a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.477+0000 7f0f0e7fc700 1 --2- 172.21.15.44:0/2272199888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3ba8b0 0x563dcd3bad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.478+0000 7f0f0f7fe700 1 -- 172.21.15.44:0/2272199888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b6470 msgr2=0x563dcd3ba370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:50:50.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.478+0000 7f0f0f7fe700 1 --2- 172.21.15.44:0/2272199888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b6470 0x563dcd3ba370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.478+0000 7f0f0f7fe700 1 -- 172.21.15.44:0/2272199888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3ba8b0 msgr2=0x563dcd3bad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:50.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.478+0000 7f0f0f7fe700 1 --2- 172.21.15.44:0/2272199888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3ba8b0 0x563dcd3bad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.478+0000 7f0f0f7fe700 1 -- 172.21.15.44:0/2272199888 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563dcd188630 con 0x563dcd3b5b60 2023-11-16T16:50:50.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.478+0000 7f0f0f7fe700 1 --2- 172.21.15.44:0/2272199888 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3b5b60 0x563dcd3b5f30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0f08018430 tx=0x7f0f08018740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75a92d1094f83934 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:50.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.478+0000 7f0f0f7fe700 1 --2- 172.21.15.44:0/3164619921 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0ef80a14e0 0x7f0ef80a3990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0f08007c50 tx=0x7f0f08009000 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:50.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.479+0000 7f0ef57fa700 1 -- 172.21.15.44:0/2272199888 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f080112d0 con 0x563dcd3b5b60 2023-11-16T16:50:50.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.479+0000 7f0ef57fa700 1 -- 172.21.15.44:0/2272199888 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f0801d070 con 0x563dcd3b5b60 2023-11-16T16:50:50.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.479+0000 7f0ef57fa700 1 -- 172.21.15.44:0/2272199888 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f08021ce0 con 0x563dcd3b5b60 2023-11-16T16:50:50.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.480+0000 7f0f28dacb80 1 -- 172.21.15.44:0/2272199888 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3b5b60 msgr2=0x563dcd3b5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.480+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/2272199888 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3b5b60 0x563dcd3b5f30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0f08018430 tx=0x7f0f08018740 comp rx=0 tx=0).stop 2023-11-16T16:50:50.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.480+0000 7f0f28dacb80 1 -- 172.21.15.44:0/2272199888 shutdown_connections 2023-11-16T16:50:50.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.480+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/2272199888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3ba8b0 0x563dcd3bad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.480+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/2272199888 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3b5b60 0x563dcd3b5f30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.480+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/2272199888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b6470 0x563dcd3ba370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.480+0000 7f0f28dacb80 1 -- 172.21.15.44:0/2272199888 >> 172.21.15.44:0/2272199888 conn(0x563dcd3ab360 msgr2=0x563dcd3b0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:50.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.480+0000 7f0f28dacb80 1 -- 172.21.15.44:0/2272199888 shutdown_connections 2023-11-16T16:50:50.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.480+0000 7f0f28dacb80 1 -- 172.21.15.44:0/2272199888 wait complete. 2023-11-16T16:50:50.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.481+0000 7f0f28dacb80 1 -- start start 2023-11-16T16:50:50.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.481+0000 7f0f28dacb80 1 Processor -- start 2023-11-16T16:50:50.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.482+0000 7f0f28dacb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3b5b60 0x563dcd3bf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.482+0000 7f0f28dacb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b6470 0x563dcd3bf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.482+0000 7f0f28dacb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3ba8b0 0x563dcd3bfe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.482+0000 7f0f28dacb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcd2a88e0 con 0x563dcd3b6470 2023-11-16T16:50:50.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.482+0000 7f0f28dacb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcd2a8580 con 0x563dcd3ba8b0 2023-11-16T16:50:50.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.482+0000 7f0f28dacb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcd18a920 con 0x563dcd3b5b60 2023-11-16T16:50:50.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.482+0000 7f0f28dacb80 10 client.0 _ll_register_callbacks cb 0x563dcd2c6780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:50.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.483+0000 7f0f0e7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b6470 0x563dcd3bf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.483+0000 7f0f0effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3b5b60 0x563dcd3bf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.483+0000 7f0f0effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3b5b60 0x563dcd3bf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48318/0 (socket says 172.21.15.44:48318) 2023-11-16T16:50:50.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.483+0000 7f0f0effd700 1 -- 172.21.15.44:0/4193064419 learned_addr learned my addr 172.21.15.44:0/4193064419 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:50.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.483+0000 7f0f0effd700 1 -- 172.21.15.44:0/4193064419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3ba8b0 msgr2=0x563dcd3bfe40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T16:50:50.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.483+0000 7f0f0effd700 1 --2- 172.21.15.44:0/4193064419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3ba8b0 0x563dcd3bfe40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.483+0000 7f0f0effd700 1 -- 172.21.15.44:0/4193064419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b6470 msgr2=0x563dcd3bf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:50.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.483+0000 7f0f0effd700 1 --2- 172.21.15.44:0/4193064419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b6470 0x563dcd3bf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.483+0000 7f0f0effd700 1 -- 172.21.15.44:0/4193064419 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f08011c60 con 0x563dcd3b5b60 2023-11-16T16:50:50.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.483+0000 7f0f0e7fc700 1 --2- 172.21.15.44:0/4193064419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b6470 0x563dcd3bf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:50.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.484+0000 7f0f0effd700 1 --2- 172.21.15.44:0/4193064419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3b5b60 0x563dcd3bf3c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0f0009a850 tx=0x7f0f0009a350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68d2c3c2271c5b7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:50.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.484+0000 7f0ef2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:50:50.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.484+0000 7f0ef2ff5700 1 -- 172.21.15.44:0/4193064419 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0009a940 con 0x563dcd3b5b60 2023-11-16T16:50:50.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.484+0000 7f0f28dacb80 1 -- 172.21.15.44:0/4193064419 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563dcd3a8b20 con 0x563dcd3b5b60 2023-11-16T16:50:50.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.484+0000 7f0ef2ff5700 1 -- 172.21.15.44:0/4193064419 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f0f0009e8f0 con 0x563dcd3b5b60 2023-11-16T16:50:50.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.484+0000 7f0ef2ff5700 1 -- 172.21.15.44:0/4193064419 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f000a2c80 con 0x563dcd3b5b60 2023-11-16T16:50:50.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.484+0000 7f0f28dacb80 10 client.25687 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:50.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.484+0000 7f0f28dacb80 1 -- 172.21.15.44:0/4193064419 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x563dcd2cd190 con 0x563dcd3b5b60 2023-11-16T16:50:50.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.485+0000 7f0ef2ff5700 1 -- 172.21.15.44:0/4193064419 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f000ba0c0 con 0x563dcd3b5b60 2023-11-16T16:50:50.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.485+0000 7f0ef2ff5700 1 -- 172.21.15.44:0/4193064419 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0f0009bd20 con 0x563dcd3b5b60 2023-11-16T16:50:50.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.485+0000 7f0f28dacb80 10 client.25687 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:50.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.485+0000 7f0f28dacb80 10 client.25687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:50.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.485+0000 7f0f28dacb80 20 client.25687 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:50.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.485+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/4193064419 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x563dcd3cf2e0 0x563dcd3d16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.485+0000 7f0f28dacb80 4 client.25687 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:50.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.485+0000 7f0f28dacb80 1 -- 172.21.15.44:0/4193064419 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563dcd3a8b20 con 0x563dcd3cf2e0 2023-11-16T16:50:50.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.486+0000 7f0f0e7fc700 1 --2- 172.21.15.44:0/4193064419 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x563dcd3cf2e0 0x563dcd3d16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.486+0000 7f0f0e7fc700 1 --2- 172.21.15.44:0/4193064419 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x563dcd3cf2e0 0x563dcd3d16c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32eb12ec02a2f3bf server_cookie=e577b6a0171267c3 in_seq=0 out_seq=0 2023-11-16T16:50:50.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.486+0000 7f0ef2ff5700 10 client.25687 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:50.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.487+0000 7f0ef2ff5700 1 -- 172.21.15.44:0/4193064419 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563dcd3a8b20 con 0x563dcd3cf2e0 2023-11-16T16:50:50.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.487+0000 7f0ef2ff5700 10 client.25687 handle_command_reply: tid=0 2023-11-16T16:50:50.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.487+0000 7f0f28dacb80 1 client.25687 shutdown 2023-11-16T16:50:50.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.488+0000 7f0f28dacb80 1 -- 172.21.15.44:0/4193064419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3b5b60 msgr2=0x563dcd3bf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.488+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/4193064419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3b5b60 0x563dcd3bf3c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0f0009a850 tx=0x7f0f0009a350 comp rx=0 tx=0).stop 2023-11-16T16:50:50.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.488+0000 7f0f28dacb80 1 -- 172.21.15.44:0/4193064419 shutdown_connections 2023-11-16T16:50:50.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.488+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/4193064419 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x563dcd3cf2e0 0x563dcd3d16c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.488+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/4193064419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3ba8b0 0x563dcd3bfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.488+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/4193064419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3b5b60 0x563dcd3bf3c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.488+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/4193064419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b6470 0x563dcd3bf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.488+0000 7f0f28dacb80 1 -- 172.21.15.44:0/4193064419 >> 172.21.15.44:0/4193064419 conn(0x563dcd3ab360 msgr2=0x563dcd3bbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:50.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.488+0000 7f0ef2ff5700 0 client.25687 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:50.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.488+0000 7f0f28dacb80 1 -- 172.21.15.44:0/4193064419 shutdown_connections 2023-11-16T16:50:50.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.488+0000 7f0f28dacb80 1 -- 172.21.15.44:0/4193064419 wait complete. 2023-11-16T16:50:50.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.488+0000 7f0f28dacb80 20 client.25687 trim_cache size 0 max 16384 2023-11-16T16:50:50.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.515+0000 7f0f28dacb80 1 Processor -- start 2023-11-16T16:50:50.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.515+0000 7f0f28dacb80 1 -- start start 2023-11-16T16:50:50.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.516+0000 7f0f28dacb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b5b60 0x563dcd3b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.516+0000 7f0f28dacb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b6470 0x563dcd3c6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.516+0000 7f0f28dacb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3ba8b0 0x563dcd3bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.516+0000 7f0f28dacb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcd3b18b0 con 0x563dcd3b6470 2023-11-16T16:50:50.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.516+0000 7f0f28dacb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcd3c6e80 con 0x563dcd3b5b60 2023-11-16T16:50:50.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.516+0000 7f0f28dacb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcd3cb770 con 0x563dcd3ba8b0 2023-11-16T16:50:50.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.517+0000 7f0f0effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b5b60 0x563dcd3b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.517+0000 7f0f0e7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b6470 0x563dcd3c6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.517+0000 7f0f0effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b5b60 0x563dcd3b81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:53802/0 (socket says 172.21.15.44:53802) 2023-11-16T16:50:50.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.517+0000 7f0f0effd700 1 -- 172.21.15.44:0/458143415 learned_addr learned my addr 172.21.15.44:0/458143415 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:50.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.517+0000 7f0f0effd700 1 -- 172.21.15.44:0/458143415 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3ba8b0 msgr2=0x563dcd3bbc90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T16:50:50.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.517+0000 7f0f0effd700 1 --2- 172.21.15.44:0/458143415 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3ba8b0 0x563dcd3bbc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.517+0000 7f0f0effd700 1 -- 172.21.15.44:0/458143415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b6470 msgr2=0x563dcd3c6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:50.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.517+0000 7f0f0effd700 1 --2- 172.21.15.44:0/458143415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b6470 0x563dcd3c6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.517+0000 7f0f0effd700 1 -- 172.21.15.44:0/458143415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563dcd2ccee0 con 0x563dcd3b5b60 2023-11-16T16:50:50.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.517+0000 7f0f0effd700 1 --2- 172.21.15.44:0/458143415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b5b60 0x563dcd3b81a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0f0009d040 tx=0x7f0f00098e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ed04fc0410752e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:50.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.518+0000 7f0ef5ffb700 1 -- 172.21.15.44:0/458143415 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0000b540 con 0x563dcd3b5b60 2023-11-16T16:50:50.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.518+0000 7f0ef5ffb700 1 -- 172.21.15.44:0/458143415 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f0009e8f0 con 0x563dcd3b5b60 2023-11-16T16:50:50.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.518+0000 7f0ef5ffb700 1 -- 172.21.15.44:0/458143415 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f000ba0b0 con 0x563dcd3b5b60 2023-11-16T16:50:50.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.519+0000 7f0f28dacb80 1 -- 172.21.15.44:0/458143415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b5b60 msgr2=0x563dcd3b81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.519+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/458143415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b5b60 0x563dcd3b81a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0f0009d040 tx=0x7f0f00098e70 comp rx=0 tx=0).stop 2023-11-16T16:50:50.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.519+0000 7f0f28dacb80 1 -- 172.21.15.44:0/458143415 shutdown_connections 2023-11-16T16:50:50.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.519+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/458143415 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3ba8b0 0x563dcd3bbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.519+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/458143415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b5b60 0x563dcd3b81a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.519+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/458143415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b6470 0x563dcd3c6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.519+0000 7f0f28dacb80 1 -- 172.21.15.44:0/458143415 >> 172.21.15.44:0/458143415 conn(0x563dcd3ab360 msgr2=0x563dcd3b8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:50.522 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.519+0000 7f0f28dacb80 1 -- 172.21.15.44:0/458143415 shutdown_connections 2023-11-16T16:50:50.522 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.519+0000 7f0f28dacb80 1 -- 172.21.15.44:0/458143415 wait complete. 2023-11-16T16:50:50.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.520+0000 7f0f28dacb80 1 -- start start 2023-11-16T16:50:50.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.520+0000 7f0f28dacb80 1 Processor -- start 2023-11-16T16:50:50.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.521+0000 7f0f28dacb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b5b60 0x563dcd3c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.521+0000 7f0f28dacb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b6470 0x563dcd3c4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.521+0000 7f0f28dacb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3ba8b0 0x563dcd3c4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.521+0000 7f0f28dacb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcd3c5510 con 0x563dcd3b5b60 2023-11-16T16:50:50.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.521+0000 7f0f28dacb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcd3c5680 con 0x563dcd3b6470 2023-11-16T16:50:50.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.521+0000 7f0f28dacb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcd422170 con 0x563dcd3ba8b0 2023-11-16T16:50:50.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.521+0000 7f0f28dacb80 10 client.0 _ll_register_callbacks cb 0x7f0f0009a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:50.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.522+0000 7f0f0effd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b5b60 0x563dcd3c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.522+0000 7f0f0effd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b5b60 0x563dcd3c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57756/0 (socket says 172.21.15.44:57756) 2023-11-16T16:50:50.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.522+0000 7f0f0effd700 1 -- 172.21.15.44:0/4084857328 learned_addr learned my addr 172.21.15.44:0/4084857328 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:50.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.522+0000 7f0f0e7fc700 1 --2- 172.21.15.44:0/4084857328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b6470 0x563dcd3c4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.522+0000 7f0f0effd700 1 -- 172.21.15.44:0/4084857328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3ba8b0 msgr2=0x563dcd3c4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:50.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.522+0000 7f0f0f7fe700 1 --2- 172.21.15.44:0/4084857328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3ba8b0 0x563dcd3c4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.522+0000 7f0f0effd700 1 --2- 172.21.15.44:0/4084857328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3ba8b0 0x563dcd3c4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.522+0000 7f0f0effd700 1 -- 172.21.15.44:0/4084857328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b6470 msgr2=0x563dcd3c4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:50.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.522+0000 7f0f0effd700 1 --2- 172.21.15.44:0/4084857328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b6470 0x563dcd3c4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.522+0000 7f0f0effd700 1 -- 172.21.15.44:0/4084857328 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563dcd2cd190 con 0x563dcd3b5b60 2023-11-16T16:50:50.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.522+0000 7f0f0e7fc700 1 --2- 172.21.15.44:0/4084857328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b6470 0x563dcd3c4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:50.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.523+0000 7f0f0f7fe700 1 --2- 172.21.15.44:0/4084857328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3ba8b0 0x563dcd3c4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:50:50.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.523+0000 7f0f0effd700 1 --2- 172.21.15.44:0/4084857328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b5b60 0x563dcd3c6950 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0f00097330 tx=0x7f0f0009aeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf304ba99ad2988 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:50.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.523+0000 7f0ef2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:50:50.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.523+0000 7f0ef2ff5700 1 -- 172.21.15.44:0/4084857328 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0009ed60 con 0x563dcd3b5b60 2023-11-16T16:50:50.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.523+0000 7f0ef2ff5700 1 -- 172.21.15.44:0/4084857328 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f0f0009a8c0 con 0x563dcd3b5b60 2023-11-16T16:50:50.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.523+0000 7f0f28dacb80 1 -- 172.21.15.44:0/4084857328 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0f0801d070 con 0x563dcd3b5b60 2023-11-16T16:50:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.524+0000 7f0ef2ff5700 1 -- 172.21.15.44:0/4084857328 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f000a27f0 con 0x563dcd3b5b60 2023-11-16T16:50:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.524+0000 7f0f28dacb80 10 client.15687 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.524+0000 7f0f28dacb80 1 -- 172.21.15.44:0/4084857328 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x563dcd11dc60 con 0x563dcd3b5b60 2023-11-16T16:50:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.524+0000 7f0ef2ff5700 1 -- 172.21.15.44:0/4084857328 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f0801d070 con 0x563dcd3b5b60 2023-11-16T16:50:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.524+0000 7f0ef2ff5700 1 -- 172.21.15.44:0/4084857328 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0f0009ab20 con 0x563dcd3b5b60 2023-11-16T16:50:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.524+0000 7f0f28dacb80 10 client.15687 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.524+0000 7f0f28dacb80 10 client.15687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.524+0000 7f0f28dacb80 20 client.15687 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.525+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/4084857328 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x563dcd423240 0x563dcd425620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.525+0000 7f0f28dacb80 4 client.15687 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T16:50:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.525+0000 7f0f28dacb80 1 -- 172.21.15.44:0/4084857328 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563dcd2c90a0 con 0x563dcd423240 2023-11-16T16:50:50.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.525+0000 7f0f0e7fc700 1 --2- 172.21.15.44:0/4084857328 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x563dcd423240 0x563dcd425620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.526+0000 7f0f0e7fc700 1 --2- 172.21.15.44:0/4084857328 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x563dcd423240 0x563dcd425620 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e78c15599d84c5d server_cookie=3ee2944a1f61c0d7 in_seq=0 out_seq=0 2023-11-16T16:50:50.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.526+0000 7f0ef2ff5700 10 client.15687 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:50.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.526+0000 7f0ef2ff5700 1 -- 172.21.15.44:0/4084857328 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563dcd2c90a0 con 0x563dcd423240 2023-11-16T16:50:50.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.526+0000 7f0ef2ff5700 10 client.15687 handle_command_reply: tid=0 2023-11-16T16:50:50.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.526+0000 7f0f28dacb80 1 client.15687 shutdown 2023-11-16T16:50:50.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.526+0000 7f0f28dacb80 1 -- 172.21.15.44:0/4084857328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b5b60 msgr2=0x563dcd3c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.526+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/4084857328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b5b60 0x563dcd3c6950 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0f00097330 tx=0x7f0f0009aeb0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.527+0000 7f0f28dacb80 1 -- 172.21.15.44:0/4084857328 shutdown_connections 2023-11-16T16:50:50.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.527+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/4084857328 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x563dcd423240 0x563dcd425620 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.527+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/4084857328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x563dcd3ba8b0 0x563dcd3c4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.527+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/4084857328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563dcd3b5b60 0x563dcd3c6950 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.527+0000 7f0f28dacb80 1 --2- 172.21.15.44:0/4084857328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563dcd3b6470 0x563dcd3c4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.527+0000 7f0f28dacb80 1 -- 172.21.15.44:0/4084857328 >> 172.21.15.44:0/4084857328 conn(0x563dcd3ab360 msgr2=0x563dcd3bf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:50.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.527+0000 7f0ef2ff5700 0 client.15687 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:50.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.527+0000 7f0f28dacb80 1 -- 172.21.15.44:0/4084857328 shutdown_connections 2023-11-16T16:50:50.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.527+0000 7f0f28dacb80 1 -- 172.21.15.44:0/4084857328 wait complete. 2023-11-16T16:50:50.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.527+0000 7f0f28dacb80 20 client.15687 trim_cache size 0 max 16384 2023-11-16T16:50:50.530 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T16:50:50.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.528+0000 7f0f153ff700 1 -- 172.21.15.44:0/3164619921 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0ef80a14e0 msgr2=0x7f0ef80a3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.528+0000 7f0f153ff700 1 --2- 172.21.15.44:0/3164619921 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0ef80a14e0 0x7f0ef80a3990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0f08007c50 tx=0x7f0f08009000 comp rx=0 tx=0).stop 2023-11-16T16:50:50.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.528+0000 7f0f153ff700 1 -- 172.21.15.44:0/3164619921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f10100cc0 msgr2=0x7f0f10198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.528+0000 7f0f153ff700 1 --2- 172.21.15.44:0/3164619921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f10100cc0 0x7f0f10198660 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009f50 tx=0x7f0f00005740 comp rx=0 tx=0).stop 2023-11-16T16:50:50.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.528+0000 7f0f153ff700 1 -- 172.21.15.44:0/3164619921 shutdown_connections 2023-11-16T16:50:50.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.528+0000 7f0f153ff700 1 --2- 172.21.15.44:0/3164619921 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0ef80a14e0 0x7f0ef80a3990 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.528+0000 7f0f153ff700 1 --2- 172.21.15.44:0/3164619921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f10105620 0x7f0f1019d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.528+0000 7f0f153ff700 1 --2- 172.21.15.44:0/3164619921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f10100cc0 0x7f0f10198660 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.528+0000 7f0f153ff700 1 --2- 172.21.15.44:0/3164619921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f10101660 0x7f0f10198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.528+0000 7f0f153ff700 1 -- 172.21.15.44:0/3164619921 >> 172.21.15.44:0/3164619921 conn(0x7f0f10077560 msgr2=0x7f0f100feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:50.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.529+0000 7f0f153ff700 1 -- 172.21.15.44:0/3164619921 shutdown_connections 2023-11-16T16:50:50.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:50.529+0000 7f0f153ff700 1 -- 172.21.15.44:0/3164619921 wait complete. 2023-11-16T16:50:50.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:50 smithi044 ceph-mon[128750]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2023-11-16T16:50:50.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:50 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1208041720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:50:50.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:50 smithi077 ceph-mon[131204]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2023-11-16T16:50:50.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:50 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1208041720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:50:51.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:50 smithi176 ceph-mon[131630]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2023-11-16T16:50:51.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:50 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1208041720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:50:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.395+0000 7f50ecd9d700 1 -- 172.21.15.44:0/2862077348 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50e8100c80 msgr2=0x7f50e8104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.395+0000 7f50ecd9d700 1 --2- 172.21.15.44:0/2862077348 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50e8100c80 0x7f50e8104c40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0099c0 tx=0x7f50dc009cd0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.396+0000 7f50ecd9d700 1 -- 172.21.15.44:0/2862077348 shutdown_connections 2023-11-16T16:50:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.396+0000 7f50ecd9d700 1 --2- 172.21.15.44:0/2862077348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50e8105320 0x7f50e810cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.396+0000 7f50ecd9d700 1 --2- 172.21.15.44:0/2862077348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50e81002e0 0x7f50e81006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.396+0000 7f50ecd9d700 1 --2- 172.21.15.44:0/2862077348 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50e8100c80 0x7f50e8104c40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.396+0000 7f50ecd9d700 1 -- 172.21.15.44:0/2862077348 >> 172.21.15.44:0/2862077348 conn(0x7f50e80fbb70 msgr2=0x7f50e80fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.396+0000 7f50ecd9d700 1 -- 172.21.15.44:0/2862077348 shutdown_connections 2023-11-16T16:50:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.396+0000 7f50ecd9d700 1 -- 172.21.15.44:0/2862077348 wait complete. 2023-11-16T16:50:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.398+0000 7f50ecd9d700 1 Processor -- start 2023-11-16T16:50:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.398+0000 7f50ecd9d700 1 -- start start 2023-11-16T16:50:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.399+0000 7f50ecd9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50e81002e0 0x7f50e81984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.399+0000 7f50ecd9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50e8100c80 0x7f50e8198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.399+0000 7f50ecd9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50e8105320 0x7f50e819ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.399+0000 7f50ecd9d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e819d4b0 con 0x7f50e81002e0 2023-11-16T16:50:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.399+0000 7f50ecd9d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e819d620 con 0x7f50e8100c80 2023-11-16T16:50:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.399+0000 7f50ecd9d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e819d7e0 con 0x7f50e8105320 2023-11-16T16:50:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.399+0000 7f50e6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50e8100c80 0x7f50e8198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.399+0000 7f50e6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50e8100c80 0x7f50e8198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:53830/0 (socket says 172.21.15.44:53830) 2023-11-16T16:50:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.399+0000 7f50e6ffd700 1 -- 172.21.15.44:0/2371190729 learned_addr learned my addr 172.21.15.44:0/2371190729 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.399+0000 7f50e7fff700 1 --2- 172.21.15.44:0/2371190729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50e8105320 0x7f50e819ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.399+0000 7f50e77fe700 1 --2- 172.21.15.44:0/2371190729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50e81002e0 0x7f50e81984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.400+0000 7f50e6ffd700 1 -- 172.21.15.44:0/2371190729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50e8105320 msgr2=0x7f50e819ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.400+0000 7f50e6ffd700 1 --2- 172.21.15.44:0/2371190729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50e8105320 0x7f50e819ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.400+0000 7f50e6ffd700 1 -- 172.21.15.44:0/2371190729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50e81002e0 msgr2=0x7f50e81984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.400+0000 7f50e6ffd700 1 --2- 172.21.15.44:0/2371190729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50e81002e0 0x7f50e81984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.400+0000 7f50e6ffd700 1 -- 172.21.15.44:0/2371190729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50dc0095f0 con 0x7f50e8100c80 2023-11-16T16:50:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.400+0000 7f50e77fe700 1 --2- 172.21.15.44:0/2371190729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50e81002e0 0x7f50e81984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.400+0000 7f50e6ffd700 1 --2- 172.21.15.44:0/2371190729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50e8100c80 0x7f50e8198a10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f50dc009990 tx=0x7f50dc00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.400+0000 7f50e4ff9700 1 -- 172.21.15.44:0/2371190729 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc01d070 con 0x7f50e8100c80 2023-11-16T16:50:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.400+0000 7f50ecd9d700 1 -- 172.21.15.44:0/2371190729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e8112370 con 0x7f50e8100c80 2023-11-16T16:50:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.400+0000 7f50e4ff9700 1 -- 172.21.15.44:0/2371190729 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50dc0047d0 con 0x7f50e8100c80 2023-11-16T16:50:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.401+0000 7f50e4ff9700 1 -- 172.21.15.44:0/2371190729 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc005380 con 0x7f50e8100c80 2023-11-16T16:50:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.401+0000 7f50ecd9d700 1 -- 172.21.15.44:0/2371190729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e81121e0 con 0x7f50e8100c80 2023-11-16T16:50:51.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.406+0000 7f50e4ff9700 1 -- 172.21.15.44:0/2371190729 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f50dc004940 con 0x7f50e8100c80 2023-11-16T16:50:51.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.407+0000 7f51018bab80 1 Processor -- start 2023-11-16T16:50:51.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.407+0000 7f51018bab80 1 -- start start 2023-11-16T16:50:51.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.408+0000 7f51018bab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fa806b0 0x56446fc2a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.408+0000 7f51018bab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fc2a990 0x56446fc247b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.408+0000 7f51018bab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 0x56446fc25120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.408+0000 7f51018bab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56446fb2a000 con 0x56446fa806b0 2023-11-16T16:50:51.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.408+0000 7f51018bab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56446fb7c840 con 0x56446fc24cf0 2023-11-16T16:50:51.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.408+0000 7f51018bab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56446fb75cd0 con 0x56446fc2a990 2023-11-16T16:50:51.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.409+0000 7f50e6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 0x56446fc25120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.409+0000 7f50e4ff9700 1 --2- 172.21.15.44:0/2371190729 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f50d00a14b0 0x7f50d00a3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.409+0000 7f50e6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 0x56446fc25120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:53844/0 (socket says 172.21.15.44:53844) 2023-11-16T16:50:51.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.409+0000 7f50e6ffd700 1 -- 172.21.15.44:0/1222902699 learned_addr learned my addr 172.21.15.44:0/1222902699 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:51.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.409+0000 7f50e7fff700 1 --2- 172.21.15.44:0/1222902699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fa806b0 0x56446fc2a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.409+0000 7f50e77fe700 1 --2- 172.21.15.44:0/1222902699 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fc2a990 0x56446fc247b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.409+0000 7f50e4ff9700 1 -- 172.21.15.44:0/2371190729 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f50dc0cd9d0 con 0x7f50e8100c80 2023-11-16T16:50:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.409+0000 7f50e77fe700 1 -- 172.21.15.44:0/1222902699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 msgr2=0x56446fc25120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.409+0000 7f50e77fe700 1 --2- 172.21.15.44:0/1222902699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 0x56446fc25120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.409+0000 7f50e77fe700 1 -- 172.21.15.44:0/1222902699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fa806b0 msgr2=0x56446fc2a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.410+0000 7f50e77fe700 1 --2- 172.21.15.44:0/1222902699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fa806b0 0x56446fc2a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.410+0000 7f50e77fe700 1 -- 172.21.15.44:0/1222902699 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56446f9f75d0 con 0x56446fc2a990 2023-11-16T16:50:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.410+0000 7f50e6ffd700 1 --2- 172.21.15.44:0/1222902699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 0x56446fc25120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.410+0000 7f50e7fff700 1 --2- 172.21.15.44:0/1222902699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fa806b0 0x56446fc2a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.410+0000 7f50e77fe700 1 --2- 172.21.15.44:0/1222902699 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fc2a990 0x56446fc247b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f50d800bcd0 tx=0x7f50d800f360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e71c1f0732054c38 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.410+0000 7f50e7fff700 1 --2- 172.21.15.44:0/2371190729 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f50d00a14b0 0x7f50d00a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.410+0000 7f50cdffb700 1 -- 172.21.15.44:0/1222902699 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d800fbe0 con 0x56446fc2a990 2023-11-16T16:50:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.410+0000 7f50cdffb700 1 -- 172.21.15.44:0/1222902699 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50d800fd40 con 0x56446fc2a990 2023-11-16T16:50:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.410+0000 7f50cdffb700 1 -- 172.21.15.44:0/1222902699 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d801b720 con 0x56446fc2a990 2023-11-16T16:50:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.411+0000 7f51018bab80 1 -- 172.21.15.44:0/1222902699 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fc2a990 msgr2=0x56446fc247b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.411+0000 7f51018bab80 1 --2- 172.21.15.44:0/1222902699 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fc2a990 0x56446fc247b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f50d800bcd0 tx=0x7f50d800f360 comp rx=0 tx=0).stop 2023-11-16T16:50:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.411+0000 7f50e7fff700 1 --2- 172.21.15.44:0/2371190729 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f50d00a14b0 0x7f50d00a3960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f50e00076a0 tx=0x7f50e0009040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.412+0000 7f51018bab80 1 -- 172.21.15.44:0/1222902699 shutdown_connections 2023-11-16T16:50:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.412+0000 7f51018bab80 1 --2- 172.21.15.44:0/1222902699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 0x56446fc25120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.412+0000 7f51018bab80 1 --2- 172.21.15.44:0/1222902699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fa806b0 0x56446fc2a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.412+0000 7f51018bab80 1 --2- 172.21.15.44:0/1222902699 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fc2a990 0x56446fc247b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.412+0000 7f51018bab80 1 -- 172.21.15.44:0/1222902699 >> 172.21.15.44:0/1222902699 conn(0x56446fc1a1a0 msgr2=0x56446fc1f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.412+0000 7f51018bab80 1 -- 172.21.15.44:0/1222902699 shutdown_connections 2023-11-16T16:50:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.412+0000 7f51018bab80 1 -- 172.21.15.44:0/1222902699 wait complete. 2023-11-16T16:50:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.413+0000 7f51018bab80 1 -- start start 2023-11-16T16:50:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.413+0000 7f51018bab80 1 Processor -- start 2023-11-16T16:50:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.414+0000 7f51018bab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fa806b0 0x56446fc30c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.414+0000 7f51018bab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 0x56446fc2df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.414+0000 7f51018bab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fc2a990 0x56446fc2e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.414+0000 7f51018bab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56446fb17880 con 0x56446fc2a990 2023-11-16T16:50:51.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.414+0000 7f51018bab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56446fb17520 con 0x56446fc24cf0 2023-11-16T16:50:51.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.414+0000 7f51018bab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56446f9f98c0 con 0x56446fa806b0 2023-11-16T16:50:51.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.414+0000 7f51018bab80 10 client.0 _ll_register_callbacks cb 0x56446fb35720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:51.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.414+0000 7f50e77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fa806b0 0x56446fc30c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.414+0000 7f50e77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fa806b0 0x56446fc30c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48358/0 (socket says 172.21.15.44:48358) 2023-11-16T16:50:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.414+0000 7f50e77fe700 1 -- 172.21.15.44:0/719444643 learned_addr learned my addr 172.21.15.44:0/719444643 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.414+0000 7f50e7fff700 1 --2- 172.21.15.44:0/719444643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fc2a990 0x56446fc2e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.415+0000 7f50e6ffd700 1 --2- 172.21.15.44:0/719444643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 0x56446fc2df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.415+0000 7f50e77fe700 1 -- 172.21.15.44:0/719444643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 msgr2=0x56446fc2df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.415+0000 7f50e77fe700 1 --2- 172.21.15.44:0/719444643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 0x56446fc2df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.415+0000 7f50e77fe700 1 -- 172.21.15.44:0/719444643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fc2a990 msgr2=0x56446fc2e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.415+0000 7f50e77fe700 1 --2- 172.21.15.44:0/719444643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fc2a990 0x56446fc2e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.415+0000 7f50e77fe700 1 -- 172.21.15.44:0/719444643 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d800b900 con 0x56446fa806b0 2023-11-16T16:50:51.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.415+0000 7f50e7fff700 1 --2- 172.21.15.44:0/719444643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fc2a990 0x56446fc2e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:51.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.416+0000 7f50e77fe700 1 --2- 172.21.15.44:0/719444643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fa806b0 0x56446fc30c20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f50d800fc40 tx=0x7f50d800fc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21112e1dd991123e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.416+0000 7f50c6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:50:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.416+0000 7f50c6ffd700 1 -- 172.21.15.44:0/719444643 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d801f040 con 0x56446fa806b0 2023-11-16T16:50:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.416+0000 7f50c6ffd700 1 -- 172.21.15.44:0/719444643 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f50d8004450 con 0x56446fa806b0 2023-11-16T16:50:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.416+0000 7f51018bab80 1 -- 172.21.15.44:0/719444643 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56446fc17960 con 0x56446fa806b0 2023-11-16T16:50:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.417+0000 7f50c6ffd700 1 -- 172.21.15.44:0/719444643 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50d800feb0 con 0x56446fa806b0 2023-11-16T16:50:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.417+0000 7f51018bab80 10 client.25699 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.417+0000 7f51018bab80 1 -- 172.21.15.44:0/719444643 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x56446fb3c130 con 0x56446fa806b0 2023-11-16T16:50:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.417+0000 7f50c6ffd700 1 -- 172.21.15.44:0/719444643 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56446fc17960 con 0x56446fa806b0 2023-11-16T16:50:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.417+0000 7f50c6ffd700 1 -- 172.21.15.44:0/719444643 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f50d8003a90 con 0x56446fa806b0 2023-11-16T16:50:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.418+0000 7f51018bab80 10 client.25699 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.418+0000 7f51018bab80 10 client.25699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.418+0000 7f51018bab80 20 client.25699 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.418+0000 7f51018bab80 1 --2- 172.21.15.44:0/719444643 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56446fc3e010 0x56446fc40420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.418+0000 7f51018bab80 4 client.25699 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.418+0000 7f51018bab80 1 -- 172.21.15.44:0/719444643 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56446fb38040 con 0x56446fc3e010 2023-11-16T16:50:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.418+0000 7f50e6ffd700 1 --2- 172.21.15.44:0/719444643 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56446fc3e010 0x56446fc40420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.419+0000 7f50e6ffd700 1 --2- 172.21.15.44:0/719444643 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56446fc3e010 0x56446fc40420 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd9e3229436992c8 server_cookie=b2f1a12d5f58f9e in_seq=0 out_seq=0 2023-11-16T16:50:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.419+0000 7f50c6ffd700 10 client.25699 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.420+0000 7f50c6ffd700 1 -- 172.21.15.44:0/719444643 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56446fb38040 con 0x56446fc3e010 2023-11-16T16:50:51.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.420+0000 7f50c6ffd700 10 client.25699 handle_command_reply: tid=0 2023-11-16T16:50:51.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.420+0000 7f51018bab80 1 client.25699 shutdown 2023-11-16T16:50:51.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.420+0000 7f51018bab80 1 -- 172.21.15.44:0/719444643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fa806b0 msgr2=0x56446fc30c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.420+0000 7f51018bab80 1 --2- 172.21.15.44:0/719444643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fa806b0 0x56446fc30c20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f50d800fc40 tx=0x7f50d800fc70 comp rx=0 tx=0).stop 2023-11-16T16:50:51.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.420+0000 7f51018bab80 1 -- 172.21.15.44:0/719444643 shutdown_connections 2023-11-16T16:50:51.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.420+0000 7f51018bab80 1 --2- 172.21.15.44:0/719444643 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56446fc3e010 0x56446fc40420 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.420+0000 7f51018bab80 1 --2- 172.21.15.44:0/719444643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fc2a990 0x56446fc2e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.420+0000 7f51018bab80 1 --2- 172.21.15.44:0/719444643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fa806b0 0x56446fc30c20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.420+0000 7f51018bab80 1 --2- 172.21.15.44:0/719444643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 0x56446fc2df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.420+0000 7f51018bab80 1 -- 172.21.15.44:0/719444643 >> 172.21.15.44:0/719444643 conn(0x56446fc1a1a0 msgr2=0x56446fc1ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:51.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.420+0000 7f50c6ffd700 0 client.25699 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:51.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.421+0000 7f51018bab80 1 -- 172.21.15.44:0/719444643 shutdown_connections 2023-11-16T16:50:51.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.421+0000 7f51018bab80 1 -- 172.21.15.44:0/719444643 wait complete. 2023-11-16T16:50:51.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.421+0000 7f51018bab80 20 client.25699 trim_cache size 0 max 16384 2023-11-16T16:50:51.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.451+0000 7f51018bab80 1 Processor -- start 2023-11-16T16:50:51.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.451+0000 7f51018bab80 1 -- start start 2023-11-16T16:50:51.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.452+0000 7f51018bab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fa806b0 0x56446fc35220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.452+0000 7f51018bab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 0x56446fc1f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.452+0000 7f51018bab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fc2a990 0x56446fc307c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.452+0000 7f51018bab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56446fc2a4e0 con 0x56446fa806b0 2023-11-16T16:50:51.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.452+0000 7f51018bab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56446fc2eb90 con 0x56446fc24cf0 2023-11-16T16:50:51.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.452+0000 7f51018bab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56446fc35760 con 0x56446fc2a990 2023-11-16T16:50:51.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.452+0000 7f50e6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 0x56446fc1f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.452+0000 7f50e7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fc2a990 0x56446fc307c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.452+0000 7f50e6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 0x56446fc1f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:53862/0 (socket says 172.21.15.44:53862) 2023-11-16T16:50:51.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.452+0000 7f50e6ffd700 1 -- 172.21.15.44:0/1593161696 learned_addr learned my addr 172.21.15.44:0/1593161696 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:51.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.452+0000 7f50e7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fc2a990 0x56446fc307c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48366/0 (socket says 172.21.15.44:48366) 2023-11-16T16:50:51.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.452+0000 7f50e77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fa806b0 0x56446fc35220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.453+0000 7f50e6ffd700 1 -- 172.21.15.44:0/1593161696 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fc2a990 msgr2=0x56446fc307c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.453+0000 7f50e6ffd700 1 --2- 172.21.15.44:0/1593161696 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fc2a990 0x56446fc307c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.453+0000 7f50e6ffd700 1 -- 172.21.15.44:0/1593161696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fa806b0 msgr2=0x56446fc35220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.453+0000 7f50e6ffd700 1 --2- 172.21.15.44:0/1593161696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fa806b0 0x56446fc35220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.453+0000 7f50e6ffd700 1 -- 172.21.15.44:0/1593161696 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56446fb3be80 con 0x56446fc24cf0 2023-11-16T16:50:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.453+0000 7f50e6ffd700 1 --2- 172.21.15.44:0/1593161696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 0x56446fc1f180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f50dc09a490 tx=0x7f50dc085040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cf08c2855439952 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.454+0000 7f50ce7fc700 1 -- 172.21.15.44:0/1593161696 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc0a28c0 con 0x56446fc24cf0 2023-11-16T16:50:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.454+0000 7f50ce7fc700 1 -- 172.21.15.44:0/1593161696 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56446fb38040 con 0x56446fc24cf0 2023-11-16T16:50:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.454+0000 7f50ce7fc700 1 -- 172.21.15.44:0/1593161696 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc004c20 con 0x56446fc24cf0 2023-11-16T16:50:51.457 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.454+0000 7f51018bab80 1 -- 172.21.15.44:0/1593161696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 msgr2=0x56446fc1f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.457 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.454+0000 7f51018bab80 1 --2- 172.21.15.44:0/1593161696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 0x56446fc1f180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f50dc09a490 tx=0x7f50dc085040 comp rx=0 tx=0).stop 2023-11-16T16:50:51.457 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.455+0000 7f51018bab80 1 -- 172.21.15.44:0/1593161696 shutdown_connections 2023-11-16T16:50:51.457 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.455+0000 7f51018bab80 1 --2- 172.21.15.44:0/1593161696 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fc2a990 0x56446fc307c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.457 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.455+0000 7f51018bab80 1 --2- 172.21.15.44:0/1593161696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fa806b0 0x56446fc35220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.457 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.455+0000 7f51018bab80 1 --2- 172.21.15.44:0/1593161696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc24cf0 0x56446fc1f180 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.457 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.455+0000 7f51018bab80 1 -- 172.21.15.44:0/1593161696 >> 172.21.15.44:0/1593161696 conn(0x56446fc1a1a0 msgr2=0x56446fc28080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:51.457 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.455+0000 7f51018bab80 1 -- 172.21.15.44:0/1593161696 shutdown_connections 2023-11-16T16:50:51.457 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.455+0000 7f51018bab80 1 -- 172.21.15.44:0/1593161696 wait complete. 2023-11-16T16:50:51.457 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.455+0000 7f51018bab80 1 -- start start 2023-11-16T16:50:51.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.456+0000 7f51018bab80 1 Processor -- start 2023-11-16T16:50:51.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.456+0000 7f51018bab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fa806b0 0x56446fc27310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.456+0000 7f51018bab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fc24cf0 0x56446fc25de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.456+0000 7f51018bab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc2a990 0x56446fc26320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.456+0000 7f51018bab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56446fc27b60 con 0x56446fc24cf0 2023-11-16T16:50:51.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.456+0000 7f51018bab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56446fc26860 con 0x56446fc2a990 2023-11-16T16:50:51.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.456+0000 7f51018bab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56446fc26a20 con 0x56446fa806b0 2023-11-16T16:50:51.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.457+0000 7f51018bab80 10 client.0 _ll_register_callbacks cb 0x7f50e0009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:51.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.457+0000 7f50e6ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fc24cf0 0x56446fc25de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.457+0000 7f50e7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc2a990 0x56446fc26320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.457+0000 7f50e6ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fc24cf0 0x56446fc25de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57814/0 (socket says 172.21.15.44:57814) 2023-11-16T16:50:51.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.457+0000 7f50e6ffd700 1 -- 172.21.15.44:0/699281510 learned_addr learned my addr 172.21.15.44:0/699281510 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:51.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.457+0000 7f50e77fe700 1 --2- 172.21.15.44:0/699281510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fa806b0 0x56446fc27310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.458+0000 7f50e7fff700 1 -- 172.21.15.44:0/699281510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fa806b0 msgr2=0x56446fc27310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.458+0000 7f50e7fff700 1 --2- 172.21.15.44:0/699281510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fa806b0 0x56446fc27310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.458+0000 7f50e7fff700 1 -- 172.21.15.44:0/699281510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fc24cf0 msgr2=0x56446fc25de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.458+0000 7f50e7fff700 1 --2- 172.21.15.44:0/699281510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fc24cf0 0x56446fc25de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.458+0000 7f50e7fff700 1 -- 172.21.15.44:0/699281510 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e8112370 con 0x56446fc2a990 2023-11-16T16:50:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.458+0000 7f50e6ffd700 1 --2- 172.21.15.44:0/699281510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fc24cf0 0x56446fc25de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.458+0000 7f50e7fff700 1 --2- 172.21.15.44:0/699281510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc2a990 0x56446fc26320 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f50e0011390 tx=0x7f50e0004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5719445c9b4a782d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.458+0000 7f50c6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:50:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.459+0000 7f50c6ffd700 1 -- 172.21.15.44:0/699281510 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e000e040 con 0x56446fc2a990 2023-11-16T16:50:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.459+0000 7f51018bab80 1 -- 172.21.15.44:0/699281510 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f50d800fd40 con 0x56446fc2a990 2023-11-16T16:50:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.459+0000 7f50c6ffd700 1 -- 172.21.15.44:0/699281510 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f50e002d620 con 0x56446fc2a990 2023-11-16T16:50:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.459+0000 7f50c6ffd700 1 -- 172.21.15.44:0/699281510 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50e0011e80 con 0x56446fc2a990 2023-11-16T16:50:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.459+0000 7f51018bab80 10 client.25712 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.459+0000 7f51018bab80 1 -- 172.21.15.44:0/699281510 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x56446f98cc00 con 0x56446fc2a990 2023-11-16T16:50:51.462 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.459+0000 7f50c6ffd700 1 -- 172.21.15.44:0/699281510 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50d800fd40 con 0x56446fc2a990 2023-11-16T16:50:51.462 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.460+0000 7f50c6ffd700 1 -- 172.21.15.44:0/699281510 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f50e0024990 con 0x56446fc2a990 2023-11-16T16:50:51.462 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.460+0000 7f51018bab80 10 client.25712 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:51.462 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.460+0000 7f51018bab80 10 client.25712 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:51.462 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.460+0000 7f51018bab80 20 client.25712 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.460+0000 7f51018bab80 1 --2- 172.21.15.44:0/699281510 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56446fc9ba60 0x56446fc9de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.460+0000 7f51018bab80 4 client.25712 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:50:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.460+0000 7f51018bab80 1 -- 172.21.15.44:0/699281510 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56446fb44600 con 0x56446fc9ba60 2023-11-16T16:50:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.460+0000 7f50e77fe700 1 --2- 172.21.15.44:0/699281510 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56446fc9ba60 0x56446fc9de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.461+0000 7f50e77fe700 1 --2- 172.21.15.44:0/699281510 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56446fc9ba60 0x56446fc9de70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40660b0ed528e658 server_cookie=cb400c4b86d1914c in_seq=0 out_seq=0 2023-11-16T16:50:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.461+0000 7f50c6ffd700 10 client.25712 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:51.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.465+0000 7f50c6ffd700 1 -- 172.21.15.44:0/699281510 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56446fb44600 con 0x56446fc9ba60 2023-11-16T16:50:51.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.465+0000 7f50c6ffd700 10 client.25712 handle_command_reply: tid=0 2023-11-16T16:50:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.465+0000 7f51018bab80 1 client.25712 shutdown 2023-11-16T16:50:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.465+0000 7f51018bab80 1 -- 172.21.15.44:0/699281510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc2a990 msgr2=0x56446fc26320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.465+0000 7f51018bab80 1 --2- 172.21.15.44:0/699281510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc2a990 0x56446fc26320 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f50e0011390 tx=0x7f50e0004ad0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.466+0000 7f51018bab80 1 -- 172.21.15.44:0/699281510 shutdown_connections 2023-11-16T16:50:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.466+0000 7f51018bab80 1 --2- 172.21.15.44:0/699281510 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56446fc9ba60 0x56446fc9de70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.466+0000 7f51018bab80 1 --2- 172.21.15.44:0/699281510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446fc2a990 0x56446fc26320 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.466+0000 7f51018bab80 1 --2- 172.21.15.44:0/699281510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56446fa806b0 0x56446fc27310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.466+0000 7f51018bab80 1 --2- 172.21.15.44:0/699281510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56446fc24cf0 0x56446fc25de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.466+0000 7f51018bab80 1 -- 172.21.15.44:0/699281510 >> 172.21.15.44:0/699281510 conn(0x56446fc1a1a0 msgr2=0x56446fc28fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.466+0000 7f50c6ffd700 0 client.25712 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.466+0000 7f51018bab80 1 -- 172.21.15.44:0/699281510 shutdown_connections 2023-11-16T16:50:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.466+0000 7f51018bab80 1 -- 172.21.15.44:0/699281510 wait complete. 2023-11-16T16:50:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.466+0000 7f51018bab80 20 client.25712 trim_cache size 0 max 16384 2023-11-16T16:50:51.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:50:51.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:50:51.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:50:51.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:50:51.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.467+0000 7f50ecd9d700 1 -- 172.21.15.44:0/2371190729 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f50d00a14b0 msgr2=0x7f50d00a3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.467+0000 7f50ecd9d700 1 --2- 172.21.15.44:0/2371190729 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f50d00a14b0 0x7f50d00a3960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f50e00076a0 tx=0x7f50e0009040 comp rx=0 tx=0).stop 2023-11-16T16:50:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.467+0000 7f50ecd9d700 1 -- 172.21.15.44:0/2371190729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50e8100c80 msgr2=0x7f50e8198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.467+0000 7f50ecd9d700 1 --2- 172.21.15.44:0/2371190729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50e8100c80 0x7f50e8198a10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f50dc009990 tx=0x7f50dc00f690 comp rx=0 tx=0).stop 2023-11-16T16:50:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.467+0000 7f50ecd9d700 1 -- 172.21.15.44:0/2371190729 shutdown_connections 2023-11-16T16:50:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.467+0000 7f50ecd9d700 1 --2- 172.21.15.44:0/2371190729 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f50d00a14b0 0x7f50d00a3960 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.467+0000 7f50ecd9d700 1 --2- 172.21.15.44:0/2371190729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50e8105320 0x7f50e819ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.467+0000 7f50ecd9d700 1 --2- 172.21.15.44:0/2371190729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50e81002e0 0x7f50e81984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.467+0000 7f50ecd9d700 1 --2- 172.21.15.44:0/2371190729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50e8100c80 0x7f50e8198a10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.467+0000 7f50ecd9d700 1 -- 172.21.15.44:0/2371190729 >> 172.21.15.44:0/2371190729 conn(0x7f50e80fbb70 msgr2=0x7f50e80fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.467+0000 7f50ecd9d700 1 -- 172.21.15.44:0/2371190729 shutdown_connections 2023-11-16T16:50:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:50:51.467+0000 7f50ecd9d700 1 -- 172.21.15.44:0/2371190729 wait complete. 2023-11-16T16:50:51.561 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:50:51.561 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:50:51.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.913+0000 7fb3affff700 1 -- 172.21.15.44:0/3586450303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3a80746f0 msgr2=0x7fb3a8072d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.913+0000 7fb3affff700 1 --2- 172.21.15.44:0/3586450303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3a80746f0 0x7fb3a8072d20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4009990 tx=0x7fb3a4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.913+0000 7fb3affff700 1 -- 172.21.15.44:0/3586450303 shutdown_connections 2023-11-16T16:50:51.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.913+0000 7fb3affff700 1 --2- 172.21.15.44:0/3586450303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3a810cec0 0x7fb3a810f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.913+0000 7fb3affff700 1 --2- 172.21.15.44:0/3586450303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3a80746f0 0x7fb3a8072d20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.913+0000 7fb3affff700 1 --2- 172.21.15.44:0/3586450303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a8074ac0 0x7fb3a8073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.913+0000 7fb3affff700 1 -- 172.21.15.44:0/3586450303 >> 172.21.15.44:0/3586450303 conn(0x7fb3a8077560 msgr2=0x7fb3a8077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:51.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.914+0000 7fb3affff700 1 -- 172.21.15.44:0/3586450303 shutdown_connections 2023-11-16T16:50:51.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.914+0000 7fb3affff700 1 -- 172.21.15.44:0/3586450303 wait complete. 2023-11-16T16:50:51.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.915+0000 7fb3affff700 1 Processor -- start 2023-11-16T16:50:51.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.915+0000 7fb3affff700 1 -- start start 2023-11-16T16:50:51.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.916+0000 7fb3affff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3a80746f0 0x7fb3a8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.916+0000 7fb3affff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a8074ac0 0x7fb3a8198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.916+0000 7fb3affff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3a810cec0 0x7fb3a819c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.916+0000 7fb3affff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a819c600 con 0x7fb3a810cec0 2023-11-16T16:50:51.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.916+0000 7fb3affff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a819c770 con 0x7fb3a8074ac0 2023-11-16T16:50:51.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.916+0000 7fb3affff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a819c930 con 0x7fb3a80746f0 2023-11-16T16:50:51.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.917+0000 7fb3add9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3a80746f0 0x7fb3a8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.917+0000 7fb3add9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3a80746f0 0x7fb3a8198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48378/0 (socket says 172.21.15.44:48378) 2023-11-16T16:50:51.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.917+0000 7fb3add9b700 1 -- 172.21.15.44:0/1433500480 learned_addr learned my addr 172.21.15.44:0/1433500480 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:51.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.917+0000 7fb3ad59a700 1 --2- 172.21.15.44:0/1433500480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a8074ac0 0x7fb3a8198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.917+0000 7fb3ae59c700 1 --2- 172.21.15.44:0/1433500480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3a810cec0 0x7fb3a819c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.917+0000 7fb3ad59a700 1 -- 172.21.15.44:0/1433500480 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3a80746f0 msgr2=0x7fb3a8198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.917+0000 7fb3ad59a700 1 --2- 172.21.15.44:0/1433500480 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3a80746f0 0x7fb3a8198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.917+0000 7fb3ad59a700 1 -- 172.21.15.44:0/1433500480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3a810cec0 msgr2=0x7fb3a819c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.917+0000 7fb3ad59a700 1 --2- 172.21.15.44:0/1433500480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3a810cec0 0x7fb3a819c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.917+0000 7fb3ad59a700 1 -- 172.21.15.44:0/1433500480 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a40095f0 con 0x7fb3a8074ac0 2023-11-16T16:50:51.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.917+0000 7fb3ae59c700 1 --2- 172.21.15.44:0/1433500480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3a810cec0 0x7fb3a819c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.918+0000 7fb3ad59a700 1 --2- 172.21.15.44:0/1433500480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a8074ac0 0x7fb3a8198c50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb39800e9c0 tx=0x7fb39800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.918+0000 7fb39effd700 1 -- 172.21.15.44:0/1433500480 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39800cbd0 con 0x7fb3a8074ac0 2023-11-16T16:50:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.918+0000 7fb39effd700 1 -- 172.21.15.44:0/1433500480 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb39800eeb0 con 0x7fb3a8074ac0 2023-11-16T16:50:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.918+0000 7fb39effd700 1 -- 172.21.15.44:0/1433500480 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398018680 con 0x7fb3a8074ac0 2023-11-16T16:50:51.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.918+0000 7fb3affff700 1 -- 172.21.15.44:0/1433500480 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a81125a0 con 0x7fb3a8074ac0 2023-11-16T16:50:51.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.918+0000 7fb3affff700 1 -- 172.21.15.44:0/1433500480 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a8112410 con 0x7fb3a8074ac0 2023-11-16T16:50:51.925 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.924+0000 7fb39effd700 1 -- 172.21.15.44:0/1433500480 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7fb398010430 con 0x7fb3a8074ac0 2023-11-16T16:50:51.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.925+0000 7fb3c399fb80 1 Processor -- start 2023-11-16T16:50:51.927 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.925+0000 7fb3c399fb80 1 -- start start 2023-11-16T16:50:51.928 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.926+0000 7fb39effd700 1 --2- 172.21.15.44:0/1433500480 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fb3940a1460 0x7fb3940a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.928 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.927+0000 7fb3c399fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f4be0 0x5581814f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.927+0000 7fb3c399fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f54f0 0x5581814f93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.927+0000 7fb3c399fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f9930 0x5581814f9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.927+0000 7fb3c399fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581813fa0e0 con 0x5581814f4be0 2023-11-16T16:50:51.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.927+0000 7fb3c399fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55818144c920 con 0x5581814f54f0 2023-11-16T16:50:51.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.927+0000 7fb3c399fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x558181445db0 con 0x5581814f9930 2023-11-16T16:50:51.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.927+0000 7fb39effd700 1 -- 172.21.15.44:0/1433500480 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fb398014070 con 0x7fb3a8074ac0 2023-11-16T16:50:51.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.927+0000 7fb3ad59a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f54f0 0x5581814f93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.927+0000 7fb3add9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f4be0 0x5581814f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.928+0000 7fb3add9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f4be0 0x5581814f4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57846/0 (socket says 172.21.15.44:57846) 2023-11-16T16:50:51.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.928+0000 7fb3add9b700 1 -- 172.21.15.44:0/719204903 learned_addr learned my addr 172.21.15.44:0/719204903 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:51.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.928+0000 7fb3ae59c700 1 --2- 172.21.15.44:0/1433500480 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fb3940a1460 0x7fb3940a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.928+0000 7fb3ae59c700 1 --2- 172.21.15.44:0/719204903 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f9930 0x5581814f9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.928+0000 7fb3ad59a700 1 -- 172.21.15.44:0/719204903 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f9930 msgr2=0x5581814f9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:51.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.928+0000 7fb3ad59a700 1 --2- 172.21.15.44:0/719204903 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f9930 0x5581814f9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.928+0000 7fb3ad59a700 1 -- 172.21.15.44:0/719204903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f4be0 msgr2=0x5581814f4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:51.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.928+0000 7fb3ad59a700 1 --2- 172.21.15.44:0/719204903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f4be0 0x5581814f4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.928+0000 7fb3ad59a700 1 -- 172.21.15.44:0/719204903 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581812c76b0 con 0x5581814f54f0 2023-11-16T16:50:51.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.929+0000 7fb3ad59a700 1 --2- 172.21.15.44:0/719204903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f54f0 0x5581814f93f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb3980ca6c0 tx=0x7fb398096360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68158e4b9e0d7755 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:51.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.929+0000 7fb3ae59c700 1 --2- 172.21.15.44:0/1433500480 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fb3940a1460 0x7fb3940a3910 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb3a806a350 tx=0x7fb3a0009920 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:51.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.929+0000 7fb393fff700 1 -- 172.21.15.44:0/719204903 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398021c00 con 0x5581814f54f0 2023-11-16T16:50:51.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.929+0000 7fb393fff700 1 -- 172.21.15.44:0/719204903 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb398021a20 con 0x5581814f54f0 2023-11-16T16:50:51.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.929+0000 7fb393fff700 1 -- 172.21.15.44:0/719204903 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398099fe0 con 0x5581814f54f0 2023-11-16T16:50:51.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.930+0000 7fb3c399fb80 1 -- 172.21.15.44:0/719204903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f54f0 msgr2=0x5581814f93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.930+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/719204903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f54f0 0x5581814f93f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb3980ca6c0 tx=0x7fb398096360 comp rx=0 tx=0).stop 2023-11-16T16:50:51.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.930+0000 7fb3c399fb80 1 -- 172.21.15.44:0/719204903 shutdown_connections 2023-11-16T16:50:51.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.930+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/719204903 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f9930 0x5581814f9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.930+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/719204903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f4be0 0x5581814f4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.930+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/719204903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f54f0 0x5581814f93f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.930+0000 7fb3c399fb80 1 -- 172.21.15.44:0/719204903 >> 172.21.15.44:0/719204903 conn(0x5581814ea3e0 msgr2=0x5581814efbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:51.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.931+0000 7fb3c399fb80 1 -- 172.21.15.44:0/719204903 shutdown_connections 2023-11-16T16:50:51.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.931+0000 7fb3c399fb80 1 -- 172.21.15.44:0/719204903 wait complete. 2023-11-16T16:50:51.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.931+0000 7fb3c399fb80 1 -- start start 2023-11-16T16:50:51.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.932+0000 7fb3c399fb80 1 Processor -- start 2023-11-16T16:50:51.934 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.932+0000 7fb3c399fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f4be0 0x558181501040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.932+0000 7fb3c399fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f54f0 0x5581814fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.932+0000 7fb3c399fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f9930 0x5581814fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.932+0000 7fb3c399fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581813e7960 con 0x5581814f9930 2023-11-16T16:50:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.932+0000 7fb3c399fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581813e7600 con 0x5581814f54f0 2023-11-16T16:50:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.932+0000 7fb3c399fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581812c99a0 con 0x5581814f4be0 2023-11-16T16:50:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.933+0000 7fb3c399fb80 10 client.0 _ll_register_callbacks cb 0x558181405800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.933+0000 7fb3add9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f4be0 0x558181501040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.933+0000 7fb3ae59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f9930 0x5581814fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.933+0000 7fb3ae59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f9930 0x5581814fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57858/0 (socket says 172.21.15.44:57858) 2023-11-16T16:50:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.933+0000 7fb3ae59c700 1 -- 172.21.15.44:0/1358698811 learned_addr learned my addr 172.21.15.44:0/1358698811 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:51.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.934+0000 7fb3ae59c700 1 -- 172.21.15.44:0/1358698811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f4be0 msgr2=0x558181501040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:51.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.934+0000 7fb3ae59c700 1 --2- 172.21.15.44:0/1358698811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f4be0 0x558181501040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.934+0000 7fb3ae59c700 1 -- 172.21.15.44:0/1358698811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f54f0 msgr2=0x5581814fe2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:51.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.934+0000 7fb3ad59a700 1 --2- 172.21.15.44:0/1358698811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f54f0 0x5581814fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.934+0000 7fb3ae59c700 1 --2- 172.21.15.44:0/1358698811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f54f0 0x5581814fe2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.934+0000 7fb3ae59c700 1 -- 172.21.15.44:0/1358698811 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a8112410 con 0x5581814f9930 2023-11-16T16:50:51.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.934+0000 7fb3ad59a700 1 --2- 172.21.15.44:0/1358698811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f54f0 0x5581814fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:50:51.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.934+0000 7fb3ae59c700 1 --2- 172.21.15.44:0/1358698811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f9930 0x5581814fe8f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0017ce0 tx=0x7fb3a0004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5256269762a80959 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:51.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.934+0000 7fb3917fa700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:50:51.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.935+0000 7fb3917fa700 1 -- 172.21.15.44:0/1358698811 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a000e040 con 0x5581814f9930 2023-11-16T16:50:51.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.935+0000 7fb3917fa700 1 -- 172.21.15.44:0/1358698811 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fb3a001e040 con 0x5581814f9930 2023-11-16T16:50:51.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.935+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1358698811 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5581814e7ba0 con 0x5581814f9930 2023-11-16T16:50:51.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.935+0000 7fb3917fa700 1 -- 172.21.15.44:0/1358698811 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3a0017e70 con 0x5581814f9930 2023-11-16T16:50:51.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.935+0000 7fb3c399fb80 10 client.15717 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:51.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.935+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1358698811 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55818140c210 con 0x5581814f9930 2023-11-16T16:50:51.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.936+0000 7fb3917fa700 1 -- 172.21.15.44:0/1358698811 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5581814e7ba0 con 0x5581814f9930 2023-11-16T16:50:51.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.936+0000 7fb3917fa700 1 -- 172.21.15.44:0/1358698811 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb3a0022410 con 0x5581814f9930 2023-11-16T16:50:51.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.936+0000 7fb3c399fb80 10 client.15717 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:51.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.936+0000 7fb3c399fb80 10 client.15717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:51.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.936+0000 7fb3c399fb80 20 client.15717 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:51.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.936+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/1358698811 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558181517080 0x558181519490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.936+0000 7fb3c399fb80 4 client.15717 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:51.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.936+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1358698811 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558181408120 con 0x558181517080 2023-11-16T16:50:51.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.937+0000 7fb3add9b700 1 --2- 172.21.15.44:0/1358698811 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558181517080 0x558181519490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.937+0000 7fb3add9b700 1 --2- 172.21.15.44:0/1358698811 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558181517080 0x558181519490 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18b860dd7eddb389 server_cookie=28afb7812db2bb52 in_seq=0 out_seq=0 2023-11-16T16:50:51.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.937+0000 7fb3917fa700 10 client.15717 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:51.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.938+0000 7fb3917fa700 1 -- 172.21.15.44:0/1358698811 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x558181408120 con 0x558181517080 2023-11-16T16:50:51.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.938+0000 7fb3917fa700 10 client.15717 handle_command_reply: tid=0 2023-11-16T16:50:51.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.938+0000 7fb3c399fb80 1 client.15717 shutdown 2023-11-16T16:50:51.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.939+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1358698811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f9930 msgr2=0x5581814fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.939+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/1358698811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f9930 0x5581814fe8f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0017ce0 tx=0x7fb3a0004b40 comp rx=0 tx=0).stop 2023-11-16T16:50:51.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.939+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1358698811 shutdown_connections 2023-11-16T16:50:51.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.939+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/1358698811 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558181517080 0x558181519490 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.939+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/1358698811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f9930 0x5581814fe8f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.939+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/1358698811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f4be0 0x558181501040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.939+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/1358698811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f54f0 0x5581814fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.939+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1358698811 >> 172.21.15.44:0/1358698811 conn(0x5581814ea3e0 msgr2=0x5581814fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:51.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.939+0000 7fb3917fa700 0 client.15717 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:51.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.939+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1358698811 shutdown_connections 2023-11-16T16:50:51.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.939+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1358698811 wait complete. 2023-11-16T16:50:51.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.940+0000 7fb3c399fb80 20 client.15717 trim_cache size 0 max 16384 2023-11-16T16:50:51.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.974+0000 7fb3c399fb80 1 Processor -- start 2023-11-16T16:50:51.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.974+0000 7fb3c399fb80 1 -- start start 2023-11-16T16:50:51.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.975+0000 7fb3c399fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f4be0 0x55818150e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.977 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.975+0000 7fb3c399fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f54f0 0x5581814fb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.977 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.975+0000 7fb3c399fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f9930 0x5581814fba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.977 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.975+0000 7fb3c399fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x558181513a30 con 0x5581814f54f0 2023-11-16T16:50:51.977 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.975+0000 7fb3c399fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581814e76e0 con 0x5581814f4be0 2023-11-16T16:50:51.977 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.975+0000 7fb3c399fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581814f13f0 con 0x5581814f9930 2023-11-16T16:50:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.975+0000 7fb3add9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f4be0 0x55818150e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.975+0000 7fb3ae59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f9930 0x5581814fba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.975+0000 7fb3add9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f4be0 0x55818150e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:53920/0 (socket says 172.21.15.44:53920) 2023-11-16T16:50:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.975+0000 7fb3add9b700 1 -- 172.21.15.44:0/1111502583 learned_addr learned my addr 172.21.15.44:0/1111502583 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.975+0000 7fb3ad59a700 1 --2- 172.21.15.44:0/1111502583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f54f0 0x5581814fb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.976+0000 7fb3add9b700 1 -- 172.21.15.44:0/1111502583 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f9930 msgr2=0x5581814fba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.976+0000 7fb3add9b700 1 --2- 172.21.15.44:0/1111502583 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f9930 0x5581814fba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.976+0000 7fb3add9b700 1 -- 172.21.15.44:0/1111502583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f54f0 msgr2=0x5581814fb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.976+0000 7fb3add9b700 1 --2- 172.21.15.44:0/1111502583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f54f0 0x5581814fb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.976+0000 7fb3add9b700 1 -- 172.21.15.44:0/1111502583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55818140bf60 con 0x5581814f4be0 2023-11-16T16:50:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.976+0000 7fb3ad59a700 1 --2- 172.21.15.44:0/1111502583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f54f0 0x5581814fb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:50:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.976+0000 7fb3add9b700 1 --2- 172.21.15.44:0/1111502583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f4be0 0x55818150e170 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4006010 tx=0x7fb3a40058e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e526617ee0ffa740 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:51.979 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.976+0000 7fb39cff9700 1 -- 172.21.15.44:0/1111502583 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a401d070 con 0x5581814f4be0 2023-11-16T16:50:51.979 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.977+0000 7fb39cff9700 1 -- 172.21.15.44:0/1111502583 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb3a401e040 con 0x5581814f4be0 2023-11-16T16:50:51.979 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.977+0000 7fb39cff9700 1 -- 172.21.15.44:0/1111502583 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a400bbf0 con 0x5581814f4be0 2023-11-16T16:50:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.978+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1111502583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f4be0 msgr2=0x55818150e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.978+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/1111502583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f4be0 0x55818150e170 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4006010 tx=0x7fb3a40058e0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.978+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1111502583 shutdown_connections 2023-11-16T16:50:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.978+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/1111502583 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f9930 0x5581814fba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.978+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/1111502583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f4be0 0x55818150e170 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.978+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/1111502583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f54f0 0x5581814fb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.978+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1111502583 >> 172.21.15.44:0/1111502583 conn(0x5581814ea3e0 msgr2=0x5581814f79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.978+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1111502583 shutdown_connections 2023-11-16T16:50:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.978+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1111502583 wait complete. 2023-11-16T16:50:51.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.978+0000 7fb3c399fb80 1 -- start start 2023-11-16T16:50:51.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.979+0000 7fb3c399fb80 1 Processor -- start 2023-11-16T16:50:51.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.979+0000 7fb3c399fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f4be0 0x55818150cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.979+0000 7fb3c399fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f54f0 0x55818150d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.980+0000 7fb3c399fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f9930 0x55818156af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.980+0000 7fb3c399fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55818150d840 con 0x5581814f54f0 2023-11-16T16:50:51.982 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.980+0000 7fb3c399fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55818150d9b0 con 0x5581814f4be0 2023-11-16T16:50:51.982 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.980+0000 7fb3c399fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55818156b4e0 con 0x5581814f9930 2023-11-16T16:50:51.982 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.980+0000 7fb3c399fb80 10 client.0 _ll_register_callbacks cb 0x7fb3a00176b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:51.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.980+0000 7fb3add9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f4be0 0x55818150cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.980+0000 7fb3ad59a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f54f0 0x55818150d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.980+0000 7fb3add9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f4be0 0x55818150cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:53928/0 (socket says 172.21.15.44:53928) 2023-11-16T16:50:51.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.980+0000 7fb3add9b700 1 -- 172.21.15.44:0/1797692960 learned_addr learned my addr 172.21.15.44:0/1797692960 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:51.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.980+0000 7fb3ae59c700 1 --2- 172.21.15.44:0/1797692960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f9930 0x55818156af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.981+0000 7fb3add9b700 1 -- 172.21.15.44:0/1797692960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f9930 msgr2=0x55818156af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:51.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.981+0000 7fb3add9b700 1 --2- 172.21.15.44:0/1797692960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f9930 0x55818156af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.981+0000 7fb3add9b700 1 -- 172.21.15.44:0/1797692960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f54f0 msgr2=0x55818150d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:51.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.981+0000 7fb3add9b700 1 --2- 172.21.15.44:0/1797692960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f54f0 0x55818150d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.981+0000 7fb3add9b700 1 -- 172.21.15.44:0/1797692960 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a810fbd0 con 0x5581814f4be0 2023-11-16T16:50:51.983 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.981+0000 7fb3ad59a700 1 --2- 172.21.15.44:0/1797692960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f54f0 0x55818150d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.981+0000 7fb3add9b700 1 --2- 172.21.15.44:0/1797692960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f4be0 0x55818150cb10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40098e0 tx=0x7fb3a4005da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=204b41ec587d76a4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.981+0000 7fb3917fa700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:50:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.982+0000 7fb3917fa700 1 -- 172.21.15.44:0/1797692960 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a401d070 con 0x5581814f4be0 2023-11-16T16:50:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.982+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1797692960 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb398021a20 con 0x5581814f4be0 2023-11-16T16:50:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.982+0000 7fb3917fa700 1 -- 172.21.15.44:0/1797692960 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fb3a4028350 con 0x5581814f4be0 2023-11-16T16:50:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.982+0000 7fb3917fa700 1 -- 172.21.15.44:0/1797692960 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3a4028550 con 0x5581814f4be0 2023-11-16T16:50:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.982+0000 7fb3c399fb80 10 client.25727 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.982+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1797692960 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55818125cce0 con 0x5581814f4be0 2023-11-16T16:50:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.982+0000 7fb3917fa700 1 -- 172.21.15.44:0/1797692960 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb398021a20 con 0x5581814f4be0 2023-11-16T16:50:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.983+0000 7fb3917fa700 1 -- 172.21.15.44:0/1797692960 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb3a4004890 con 0x5581814f4be0 2023-11-16T16:50:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.983+0000 7fb3c399fb80 10 client.25727 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.983+0000 7fb3c399fb80 10 client.25727 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:51.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.983+0000 7fb3c399fb80 20 client.25727 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:51.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.983+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/1797692960 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55818156c7b0 0x55818156ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:51.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.983+0000 7fb3c399fb80 4 client.25727 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:50:51.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.983+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1797692960 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5581814146e0 con 0x55818156c7b0 2023-11-16T16:50:51.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.983+0000 7fb3ad59a700 1 --2- 172.21.15.44:0/1797692960 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55818156c7b0 0x55818156ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:51.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.984+0000 7fb3ad59a700 1 --2- 172.21.15.44:0/1797692960 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55818156c7b0 0x55818156ebc0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0a3e2f44c862f17 server_cookie=d8ea10f67a5a79be in_seq=0 out_seq=0 2023-11-16T16:50:51.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.984+0000 7fb3917fa700 10 client.25727 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:51.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.985+0000 7fb3917fa700 1 -- 172.21.15.44:0/1797692960 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5581814146e0 con 0x55818156c7b0 2023-11-16T16:50:51.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.985+0000 7fb3917fa700 10 client.25727 handle_command_reply: tid=0 2023-11-16T16:50:51.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.985+0000 7fb3c399fb80 1 client.25727 shutdown 2023-11-16T16:50:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.985+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1797692960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f4be0 msgr2=0x55818150cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.985+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/1797692960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f4be0 0x55818150cb10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40098e0 tx=0x7fb3a4005da0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.986+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1797692960 shutdown_connections 2023-11-16T16:50:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.986+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/1797692960 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55818156c7b0 0x55818156ebc0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.986+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/1797692960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5581814f9930 0x55818156af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.986+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/1797692960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5581814f4be0 0x55818150cb10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.986+0000 7fb3c399fb80 1 --2- 172.21.15.44:0/1797692960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581814f54f0 0x55818150d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.986+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1797692960 >> 172.21.15.44:0/1797692960 conn(0x5581814ea3e0 msgr2=0x5581814f8860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.986+0000 7fb3917fa700 0 client.25727 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.986+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1797692960 shutdown_connections 2023-11-16T16:50:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.986+0000 7fb3c399fb80 1 -- 172.21.15.44:0/1797692960 wait complete. 2023-11-16T16:50:51.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.986+0000 7fb3c399fb80 20 client.25727 trim_cache size 0 max 16384 2023-11-16T16:50:51.989 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:50:51.989 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:50:51.989 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "0679d52a-4c81-4c45-8fcf-c1ef608e7900", 2023-11-16T16:50:51.989 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:50:51.989 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:50:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.987+0000 7fb3affff700 1 -- 172.21.15.44:0/1433500480 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fb3940a1460 msgr2=0x7fb3940a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.987+0000 7fb3affff700 1 --2- 172.21.15.44:0/1433500480 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fb3940a1460 0x7fb3940a3910 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb3a806a350 tx=0x7fb3a0009920 comp rx=0 tx=0).stop 2023-11-16T16:50:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.987+0000 7fb3affff700 1 -- 172.21.15.44:0/1433500480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a8074ac0 msgr2=0x7fb3a8198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.987+0000 7fb3affff700 1 --2- 172.21.15.44:0/1433500480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a8074ac0 0x7fb3a8198c50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb39800e9c0 tx=0x7fb39800ecd0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.987+0000 7fb3affff700 1 -- 172.21.15.44:0/1433500480 shutdown_connections 2023-11-16T16:50:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.987+0000 7fb3affff700 1 --2- 172.21.15.44:0/1433500480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3a810cec0 0x7fb3a819c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.987+0000 7fb3affff700 1 --2- 172.21.15.44:0/1433500480 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fb3940a1460 0x7fb3940a3910 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.987+0000 7fb3affff700 1 --2- 172.21.15.44:0/1433500480 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3a80746f0 0x7fb3a8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.987+0000 7fb3affff700 1 --2- 172.21.15.44:0/1433500480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a8074ac0 0x7fb3a8198c50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.987+0000 7fb3affff700 1 -- 172.21.15.44:0/1433500480 >> 172.21.15.44:0/1433500480 conn(0x7fb3a8077560 msgr2=0x7fb3a8104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.987+0000 7fb3affff700 1 -- 172.21.15.44:0/1433500480 shutdown_connections 2023-11-16T16:50:51.991 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:51.988+0000 7fb3affff700 1 -- 172.21.15.44:0/1433500480 wait complete. 2023-11-16T16:50:52.019 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:50:52.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.367+0000 7f91449ff700 1 -- 172.21.15.44:0/2960455054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9140100cc0 msgr2=0x7f9140101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:52.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.367+0000 7f91449ff700 1 --2- 172.21.15.44:0/2960455054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9140100cc0 0x7f9140101090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f91300099c0 tx=0x7f9130009cd0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.367+0000 7f91449ff700 1 -- 172.21.15.44:0/2960455054 shutdown_connections 2023-11-16T16:50:52.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.367+0000 7f91449ff700 1 --2- 172.21.15.44:0/2960455054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9140105620 0x7f914010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.367+0000 7f91449ff700 1 --2- 172.21.15.44:0/2960455054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9140100cc0 0x7f9140101090 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.367+0000 7f91449ff700 1 --2- 172.21.15.44:0/2960455054 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9140101660 0x7f9140104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.367+0000 7f91449ff700 1 -- 172.21.15.44:0/2960455054 >> 172.21.15.44:0/2960455054 conn(0x7f9140077560 msgr2=0x7f9140077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:52.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.367+0000 7f91449ff700 1 -- 172.21.15.44:0/2960455054 shutdown_connections 2023-11-16T16:50:52.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.368+0000 7f91449ff700 1 -- 172.21.15.44:0/2960455054 wait complete. 2023-11-16T16:50:52.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.369+0000 7f91449ff700 1 Processor -- start 2023-11-16T16:50:52.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.370+0000 7f91449ff700 1 -- start start 2023-11-16T16:50:52.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.370+0000 7f91449ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9140100cc0 0x7f914010c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.370+0000 7f91449ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9140101660 0x7f914010c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.370+0000 7f91449ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9140105620 0x7f91401b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.370+0000 7f91449ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914010cd10 con 0x7f9140101660 2023-11-16T16:50:52.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.370+0000 7f91449ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91401b1730 con 0x7f9140105620 2023-11-16T16:50:52.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.370+0000 7f91449ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91401b18f0 con 0x7f9140100cc0 2023-11-16T16:50:52.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.371+0000 7f913e59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9140100cc0 0x7f914010c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.371+0000 7f913ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9140105620 0x7f91401b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.371+0000 7f913e59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9140100cc0 0x7f914010c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48428/0 (socket says 172.21.15.44:48428) 2023-11-16T16:50:52.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.371+0000 7f913e59c700 1 -- 172.21.15.44:0/1357955354 learned_addr learned my addr 172.21.15.44:0/1357955354 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:52.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.371+0000 7f913dd9b700 1 --2- 172.21.15.44:0/1357955354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9140101660 0x7f914010c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.371+0000 7f913e59c700 1 -- 172.21.15.44:0/1357955354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9140105620 msgr2=0x7f91401b11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:52.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.371+0000 7f913e59c700 1 --2- 172.21.15.44:0/1357955354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9140105620 0x7f91401b11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.371+0000 7f913e59c700 1 -- 172.21.15.44:0/1357955354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9140101660 msgr2=0x7f914010c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:52.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.371+0000 7f913e59c700 1 --2- 172.21.15.44:0/1357955354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9140101660 0x7f914010c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.371+0000 7f913e59c700 1 -- 172.21.15.44:0/1357955354 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91300095f0 con 0x7f9140100cc0 2023-11-16T16:50:52.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.371+0000 7f913dd9b700 1 --2- 172.21.15.44:0/1357955354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9140101660 0x7f914010c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:52.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.372+0000 7f913e59c700 1 --2- 172.21.15.44:0/1357955354 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9140100cc0 0x7f914010c030 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f91300098e0 tx=0x7f913000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:52.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.372+0000 7f912f7fe700 1 -- 172.21.15.44:0/1357955354 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913001d070 con 0x7f9140100cc0 2023-11-16T16:50:52.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.372+0000 7f91449ff700 1 -- 172.21.15.44:0/1357955354 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91401103d0 con 0x7f9140100cc0 2023-11-16T16:50:52.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.372+0000 7f912f7fe700 1 -- 172.21.15.44:0/1357955354 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9130022470 con 0x7f9140100cc0 2023-11-16T16:50:52.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.372+0000 7f912f7fe700 1 -- 172.21.15.44:0/1357955354 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9130017650 con 0x7f9140100cc0 2023-11-16T16:50:52.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.372+0000 7f91449ff700 1 -- 172.21.15.44:0/1357955354 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9140110240 con 0x7f9140100cc0 2023-11-16T16:50:52.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.378+0000 7f912f7fe700 1 -- 172.21.15.44:0/1357955354 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f9130022850 con 0x7f9140100cc0 2023-11-16T16:50:52.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.380+0000 7f91583b0b80 1 Processor -- start 2023-11-16T16:50:52.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.380+0000 7f91583b0b80 1 -- start start 2023-11-16T16:50:52.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.380+0000 7f912f7fe700 1 --2- 172.21.15.44:0/1357955354 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f91280a1460 0x7f91280a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.381+0000 7f912f7fe700 1 -- 172.21.15.44:0/1357955354 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f91300cdcd0 con 0x7f9140100cc0 2023-11-16T16:50:52.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.381+0000 7f91583b0b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e7b60 0x559cf79e7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.381+0000 7f91583b0b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79e8470 0x559cf79ec370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.381+0000 7f91583b0b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79ec8b0 0x559cf79ecd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.381+0000 7f91583b0b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf78ed060 con 0x559cf79e8470 2023-11-16T16:50:52.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.381+0000 7f91583b0b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf793f8a0 con 0x559cf79ec8b0 2023-11-16T16:50:52.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.381+0000 7f91583b0b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf7938d30 con 0x559cf79e7b60 2023-11-16T16:50:52.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.381+0000 7f913ed9d700 1 --2- 172.21.15.44:0/1357955354 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f91280a1460 0x7f91280a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.381+0000 7f913e59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e7b60 0x559cf79e7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.381+0000 7f913e59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e7b60 0x559cf79e7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48436/0 (socket says 172.21.15.44:48436) 2023-11-16T16:50:52.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.381+0000 7f913e59c700 1 -- 172.21.15.44:0/3656979770 learned_addr learned my addr 172.21.15.44:0/3656979770 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:52.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.382+0000 7f913dd9b700 1 --2- 172.21.15.44:0/3656979770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79e8470 0x559cf79ec370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.382+0000 7f913e59c700 1 -- 172.21.15.44:0/3656979770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79ec8b0 msgr2=0x559cf79ecd00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T16:50:52.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.382+0000 7f913e59c700 1 --2- 172.21.15.44:0/3656979770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79ec8b0 0x559cf79ecd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.382+0000 7f913e59c700 1 -- 172.21.15.44:0/3656979770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79e8470 msgr2=0x559cf79ec370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:52.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.382+0000 7f913e59c700 1 --2- 172.21.15.44:0/3656979770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79e8470 0x559cf79ec370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.382+0000 7f913e59c700 1 -- 172.21.15.44:0/3656979770 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559cf77ba630 con 0x559cf79e7b60 2023-11-16T16:50:52.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.382+0000 7f913dd9b700 1 --2- 172.21.15.44:0/3656979770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79e8470 0x559cf79ec370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:52.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.382+0000 7f913e59c700 1 --2- 172.21.15.44:0/3656979770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e7b60 0x559cf79e7f30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f913009a900 tx=0x7f913009a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99fa3caa0f62f0a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:52.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.382+0000 7f912cff9700 1 -- 172.21.15.44:0/3656979770 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91300a2d30 con 0x559cf79e7b60 2023-11-16T16:50:52.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.383+0000 7f913ed9d700 1 --2- 172.21.15.44:0/1357955354 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f91280a1460 0x7f91280a3910 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9138009ec0 tx=0x7f913800b040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:52.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.383+0000 7f912cff9700 1 -- 172.21.15.44:0/3656979770 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9130098d10 con 0x559cf79e7b60 2023-11-16T16:50:52.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.383+0000 7f912cff9700 1 -- 172.21.15.44:0/3656979770 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91300a28a0 con 0x559cf79e7b60 2023-11-16T16:50:52.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.384+0000 7f91583b0b80 1 -- 172.21.15.44:0/3656979770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e7b60 msgr2=0x559cf79e7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:52.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.384+0000 7f91583b0b80 1 --2- 172.21.15.44:0/3656979770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e7b60 0x559cf79e7f30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f913009a900 tx=0x7f913009a5d0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.384+0000 7f91583b0b80 1 -- 172.21.15.44:0/3656979770 shutdown_connections 2023-11-16T16:50:52.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.384+0000 7f91583b0b80 1 --2- 172.21.15.44:0/3656979770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79ec8b0 0x559cf79ecd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.384+0000 7f91583b0b80 1 --2- 172.21.15.44:0/3656979770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e7b60 0x559cf79e7f30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.384+0000 7f91583b0b80 1 --2- 172.21.15.44:0/3656979770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79e8470 0x559cf79ec370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.384+0000 7f91583b0b80 1 -- 172.21.15.44:0/3656979770 >> 172.21.15.44:0/3656979770 conn(0x559cf79dd360 msgr2=0x559cf79e2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:52.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.384+0000 7f91583b0b80 1 -- 172.21.15.44:0/3656979770 shutdown_connections 2023-11-16T16:50:52.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.384+0000 7f91583b0b80 1 -- 172.21.15.44:0/3656979770 wait complete. 2023-11-16T16:50:52.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.385+0000 7f91583b0b80 1 -- start start 2023-11-16T16:50:52.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.385+0000 7f91583b0b80 1 Processor -- start 2023-11-16T16:50:52.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.386+0000 7f91583b0b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79e7b60 0x559cf79f4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.386+0000 7f91583b0b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e8470 0x559cf79f12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.386+0000 7f91583b0b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79ec8b0 0x559cf79f18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.386+0000 7f91583b0b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf78da8e0 con 0x559cf79ec8b0 2023-11-16T16:50:52.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.386+0000 7f91583b0b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf78da580 con 0x559cf79e7b60 2023-11-16T16:50:52.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.386+0000 7f91583b0b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf77bc920 con 0x559cf79e8470 2023-11-16T16:50:52.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.386+0000 7f91583b0b80 10 client.0 _ll_register_callbacks cb 0x559cf78f8780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:52.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.386+0000 7f913e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79e7b60 0x559cf79f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.386+0000 7f913e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79e7b60 0x559cf79f4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:53966/0 (socket says 172.21.15.44:53966) 2023-11-16T16:50:52.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.386+0000 7f913e59c700 1 -- 172.21.15.44:0/1300772366 learned_addr learned my addr 172.21.15.44:0/1300772366 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:52.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.386+0000 7f913dd9b700 1 --2- 172.21.15.44:0/1300772366 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e8470 0x559cf79f12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.386+0000 7f913ed9d700 1 --2- 172.21.15.44:0/1300772366 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79ec8b0 0x559cf79f18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.387+0000 7f913e59c700 1 -- 172.21.15.44:0/1300772366 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e8470 msgr2=0x559cf79f12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:52.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.387+0000 7f913e59c700 1 --2- 172.21.15.44:0/1300772366 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e8470 0x559cf79f12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.387+0000 7f913e59c700 1 -- 172.21.15.44:0/1300772366 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79ec8b0 msgr2=0x559cf79f18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:52.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.387+0000 7f913e59c700 1 --2- 172.21.15.44:0/1300772366 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79ec8b0 0x559cf79f18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.387+0000 7f913e59c700 1 -- 172.21.15.44:0/1300772366 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9140110240 con 0x559cf79e7b60 2023-11-16T16:50:52.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.387+0000 7f913ed9d700 1 --2- 172.21.15.44:0/1300772366 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79ec8b0 0x559cf79f18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:50:52.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.387+0000 7f913e59c700 1 --2- 172.21.15.44:0/1300772366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79e7b60 0x559cf79f4000 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f91300cb140 tx=0x7f9130096de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6af322b8550e28c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:52.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.387+0000 7f9125ffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:50:52.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.388+0000 7f9125ffb700 1 -- 172.21.15.44:0/1300772366 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91300a2d30 con 0x559cf79e7b60 2023-11-16T16:50:52.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.388+0000 7f9125ffb700 1 -- 172.21.15.44:0/1300772366 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f91300a28a0 con 0x559cf79e7b60 2023-11-16T16:50:52.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.388+0000 7f91583b0b80 1 -- 172.21.15.44:0/1300772366 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559cf79dab20 con 0x559cf79e7b60 2023-11-16T16:50:52.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.388+0000 7f9125ffb700 1 -- 172.21.15.44:0/1300772366 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91300a2aa0 con 0x559cf79e7b60 2023-11-16T16:50:52.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.388+0000 7f91583b0b80 10 client.25736 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:52.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.388+0000 7f91583b0b80 1 -- 172.21.15.44:0/1300772366 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x559cf78ff190 con 0x559cf79e7b60 2023-11-16T16:50:52.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.388+0000 7f9125ffb700 1 -- 172.21.15.44:0/1300772366 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559cf79dab20 con 0x559cf79e7b60 2023-11-16T16:50:52.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.389+0000 7f9125ffb700 1 -- 172.21.15.44:0/1300772366 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f91300998d0 con 0x559cf79e7b60 2023-11-16T16:50:52.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.389+0000 7f91583b0b80 10 client.25736 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:52.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.389+0000 7f91583b0b80 10 client.25736 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:52.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.389+0000 7f91583b0b80 20 client.25736 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:52.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.389+0000 7f91583b0b80 1 --2- 172.21.15.44:0/1300772366 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559cf7a01150 0x559cf7a03560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.389+0000 7f91583b0b80 4 client.25736 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:52.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.389+0000 7f91583b0b80 1 -- 172.21.15.44:0/1300772366 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559cf78fb0a0 con 0x559cf7a01150 2023-11-16T16:50:52.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.390+0000 7f913dd9b700 1 --2- 172.21.15.44:0/1300772366 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559cf7a01150 0x559cf7a03560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.390+0000 7f913dd9b700 1 --2- 172.21.15.44:0/1300772366 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559cf7a01150 0x559cf7a03560 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dc9a8dcedb59fbe server_cookie=3980af7a78ff2731 in_seq=0 out_seq=0 2023-11-16T16:50:52.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.390+0000 7f9125ffb700 10 client.25736 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:52.394 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.391+0000 7f9125ffb700 1 -- 172.21.15.44:0/1300772366 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559cf78fb0a0 con 0x559cf7a01150 2023-11-16T16:50:52.394 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.391+0000 7f9125ffb700 10 client.25736 handle_command_reply: tid=0 2023-11-16T16:50:52.394 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.391+0000 7f91583b0b80 1 client.25736 shutdown 2023-11-16T16:50:52.394 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.392+0000 7f91583b0b80 1 -- 172.21.15.44:0/1300772366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79e7b60 msgr2=0x559cf79f4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:52.394 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.392+0000 7f91583b0b80 1 --2- 172.21.15.44:0/1300772366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79e7b60 0x559cf79f4000 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f91300cb140 tx=0x7f9130096de0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.394 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.392+0000 7f91583b0b80 1 -- 172.21.15.44:0/1300772366 shutdown_connections 2023-11-16T16:50:52.394 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.392+0000 7f91583b0b80 1 --2- 172.21.15.44:0/1300772366 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559cf7a01150 0x559cf7a03560 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.392+0000 7f91583b0b80 1 --2- 172.21.15.44:0/1300772366 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79ec8b0 0x559cf79f18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.392+0000 7f91583b0b80 1 --2- 172.21.15.44:0/1300772366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79e7b60 0x559cf79f4000 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.392+0000 7f91583b0b80 1 --2- 172.21.15.44:0/1300772366 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e8470 0x559cf79f12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.392+0000 7f91583b0b80 1 -- 172.21.15.44:0/1300772366 >> 172.21.15.44:0/1300772366 conn(0x559cf79dd360 msgr2=0x559cf79e1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:52.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.392+0000 7f9125ffb700 0 client.25736 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:52.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.392+0000 7f91583b0b80 1 -- 172.21.15.44:0/1300772366 shutdown_connections 2023-11-16T16:50:52.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.392+0000 7f91583b0b80 1 -- 172.21.15.44:0/1300772366 wait complete. 2023-11-16T16:50:52.395 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.392+0000 7f91583b0b80 20 client.25736 trim_cache size 0 max 16384 2023-11-16T16:50:52.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.427+0000 7f91583b0b80 1 Processor -- start 2023-11-16T16:50:52.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.427+0000 7f91583b0b80 1 -- start start 2023-11-16T16:50:52.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.427+0000 7f91583b0b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79e8470 0x559cf79e2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.427+0000 7f91583b0b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79ec8b0 0x559cf79f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.427+0000 7f91583b0b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e2aa0 0x559cf79f4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.427+0000 7f91583b0b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf79dcf70 con 0x559cf79ec8b0 2023-11-16T16:50:52.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.427+0000 7f91583b0b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf79ed330 con 0x559cf79e8470 2023-11-16T16:50:52.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.427+0000 7f91583b0b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf79e8870 con 0x559cf79e2aa0 2023-11-16T16:50:52.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.428+0000 7f913e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79e8470 0x559cf79e2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.428+0000 7f913e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79e8470 0x559cf79e2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:53978/0 (socket says 172.21.15.44:53978) 2023-11-16T16:50:52.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.428+0000 7f913e59c700 1 -- 172.21.15.44:0/3011131020 learned_addr learned my addr 172.21.15.44:0/3011131020 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:52.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.428+0000 7f913ed9d700 1 --2- 172.21.15.44:0/3011131020 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e2aa0 0x559cf79f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.428+0000 7f913dd9b700 1 --2- 172.21.15.44:0/3011131020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79ec8b0 0x559cf79f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.429+0000 7f913e59c700 1 -- 172.21.15.44:0/3011131020 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e2aa0 msgr2=0x559cf79f4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:52.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.429+0000 7f913e59c700 1 --2- 172.21.15.44:0/3011131020 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e2aa0 0x559cf79f4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.429+0000 7f913e59c700 1 -- 172.21.15.44:0/3011131020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79ec8b0 msgr2=0x559cf79f3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:52.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.429+0000 7f913e59c700 1 --2- 172.21.15.44:0/3011131020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79ec8b0 0x559cf79f3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.429+0000 7f913e59c700 1 -- 172.21.15.44:0/3011131020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559cf79deb00 con 0x559cf79e8470 2023-11-16T16:50:52.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.429+0000 7f913dd9b700 1 --2- 172.21.15.44:0/3011131020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79ec8b0 0x559cf79f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:52.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.429+0000 7f913e59c700 1 --2- 172.21.15.44:0/3011131020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79e8470 0x559cf79e2560 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9130000c00 tx=0x7f9130098f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f55349dc88e2fd04 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:52.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.429+0000 7f912d7fa700 1 -- 172.21.15.44:0/3011131020 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91300ba3f0 con 0x559cf79e8470 2023-11-16T16:50:52.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.429+0000 7f912d7fa700 1 -- 172.21.15.44:0/3011131020 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91300a2d30 con 0x559cf79e8470 2023-11-16T16:50:52.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.429+0000 7f912d7fa700 1 -- 172.21.15.44:0/3011131020 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91300a28a0 con 0x559cf79e8470 2023-11-16T16:50:52.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.430+0000 7f91583b0b80 1 -- 172.21.15.44:0/3011131020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79e8470 msgr2=0x559cf79e2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:52.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.430+0000 7f91583b0b80 1 --2- 172.21.15.44:0/3011131020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79e8470 0x559cf79e2560 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9130000c00 tx=0x7f9130098f20 comp rx=0 tx=0).stop 2023-11-16T16:50:52.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.431+0000 7f91583b0b80 1 -- 172.21.15.44:0/3011131020 shutdown_connections 2023-11-16T16:50:52.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.431+0000 7f91583b0b80 1 --2- 172.21.15.44:0/3011131020 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e2aa0 0x559cf79f4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.431+0000 7f91583b0b80 1 --2- 172.21.15.44:0/3011131020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79e8470 0x559cf79e2560 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.431+0000 7f91583b0b80 1 --2- 172.21.15.44:0/3011131020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79ec8b0 0x559cf79f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.431+0000 7f91583b0b80 1 -- 172.21.15.44:0/3011131020 >> 172.21.15.44:0/3011131020 conn(0x559cf79dd360 msgr2=0x559cf79fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:52.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.431+0000 7f91583b0b80 1 -- 172.21.15.44:0/3011131020 shutdown_connections 2023-11-16T16:50:52.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.431+0000 7f91583b0b80 1 -- 172.21.15.44:0/3011131020 wait complete. 2023-11-16T16:50:52.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.431+0000 7f91583b0b80 1 -- start start 2023-11-16T16:50:52.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.432+0000 7f91583b0b80 1 Processor -- start 2023-11-16T16:50:52.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.432+0000 7f91583b0b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79e2aa0 0x559cf79f65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.432+0000 7f91583b0b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e8470 0x559cf79f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.432+0000 7f91583b0b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79ec8b0 0x559cf79f7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.432+0000 7f91583b0b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf7a540b0 con 0x559cf79e2aa0 2023-11-16T16:50:52.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.432+0000 7f91583b0b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf7a541f0 con 0x559cf79ec8b0 2023-11-16T16:50:52.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.432+0000 7f91583b0b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf7a543b0 con 0x559cf79e8470 2023-11-16T16:50:52.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.433+0000 7f91583b0b80 10 client.0 _ll_register_callbacks cb 0x7f913009a4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:52.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.433+0000 7f913ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79ec8b0 0x559cf79f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.433+0000 7f913ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79ec8b0 0x559cf79f7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:53992/0 (socket says 172.21.15.44:53992) 2023-11-16T16:50:52.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.433+0000 7f913ed9d700 1 -- 172.21.15.44:0/3123988037 learned_addr learned my addr 172.21.15.44:0/3123988037 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:52.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.433+0000 7f913dd9b700 1 --2- 172.21.15.44:0/3123988037 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e8470 0x559cf79f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.433+0000 7f913e59c700 1 --2- 172.21.15.44:0/3123988037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79e2aa0 0x559cf79f65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.434+0000 7f913dd9b700 1 -- 172.21.15.44:0/3123988037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79ec8b0 msgr2=0x559cf79f7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:52.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.434+0000 7f913dd9b700 1 --2- 172.21.15.44:0/3123988037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79ec8b0 0x559cf79f7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.434+0000 7f913dd9b700 1 -- 172.21.15.44:0/3123988037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79e2aa0 msgr2=0x559cf79f65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:52.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.434+0000 7f913dd9b700 1 --2- 172.21.15.44:0/3123988037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79e2aa0 0x559cf79f65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.434+0000 7f913dd9b700 1 -- 172.21.15.44:0/3123988037 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913401e330 con 0x559cf79e8470 2023-11-16T16:50:52.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.434+0000 7f913dd9b700 1 --2- 172.21.15.44:0/3123988037 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e8470 0x559cf79f6b30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x559cf79f2960 tx=0x7f9134012400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0d08cddb461c944 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:52.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.434+0000 7f9125ffb700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:50:52.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.435+0000 7f9125ffb700 1 -- 172.21.15.44:0/3123988037 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9134017040 con 0x559cf79e8470 2023-11-16T16:50:52.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.435+0000 7f91583b0b80 1 -- 172.21.15.44:0/3123988037 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9130098d10 con 0x559cf79e8470 2023-11-16T16:50:52.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.435+0000 7f9125ffb700 1 -- 172.21.15.44:0/3123988037 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f9134004510 con 0x559cf79e8470 2023-11-16T16:50:52.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.435+0000 7f9125ffb700 1 -- 172.21.15.44:0/3123988037 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9134004830 con 0x559cf79e8470 2023-11-16T16:50:52.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.435+0000 7f91583b0b80 10 client.25741 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:52.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.435+0000 7f91583b0b80 1 -- 172.21.15.44:0/3123988037 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x559cf78ff190 con 0x559cf79e8470 2023-11-16T16:50:52.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.436+0000 7f9125ffb700 1 -- 172.21.15.44:0/3123988037 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9130098d10 con 0x559cf79e8470 2023-11-16T16:50:52.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.437+0000 7f9125ffb700 1 -- 172.21.15.44:0/3123988037 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f913401d030 con 0x559cf79e8470 2023-11-16T16:50:52.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.437+0000 7f91583b0b80 10 client.25741 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:52.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.437+0000 7f91583b0b80 10 client.25741 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:52.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.437+0000 7f91583b0b80 20 client.25741 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:52.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.437+0000 7f91583b0b80 1 --2- 172.21.15.44:0/3123988037 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559cf7a55b20 0x559cf7a57f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.437+0000 7f91583b0b80 4 client.25741 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:50:52.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.437+0000 7f91583b0b80 1 -- 172.21.15.44:0/3123988037 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559cf7907660 con 0x559cf7a55b20 2023-11-16T16:50:52.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.437+0000 7f913e59c700 1 --2- 172.21.15.44:0/3123988037 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559cf7a55b20 0x559cf7a57f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.438+0000 7f913e59c700 1 --2- 172.21.15.44:0/3123988037 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559cf7a55b20 0x559cf7a57f30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7329f854c43705f2 server_cookie=b410981f0aa69bda in_seq=0 out_seq=0 2023-11-16T16:50:52.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.438+0000 7f9125ffb700 10 client.25741 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:52.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.438+0000 7f9125ffb700 1 -- 172.21.15.44:0/3123988037 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x559cf7907660 con 0x559cf7a55b20 2023-11-16T16:50:52.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.438+0000 7f9125ffb700 10 client.25741 handle_command_reply: tid=0 2023-11-16T16:50:52.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.438+0000 7f91583b0b80 1 client.25741 shutdown 2023-11-16T16:50:52.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.439+0000 7f91583b0b80 1 -- 172.21.15.44:0/3123988037 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e8470 msgr2=0x559cf79f6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:52.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.439+0000 7f91583b0b80 1 --2- 172.21.15.44:0/3123988037 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e8470 0x559cf79f6b30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x559cf79f2960 tx=0x7f9134012400 comp rx=0 tx=0).stop 2023-11-16T16:50:52.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.439+0000 7f91583b0b80 1 -- 172.21.15.44:0/3123988037 shutdown_connections 2023-11-16T16:50:52.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.439+0000 7f91583b0b80 1 --2- 172.21.15.44:0/3123988037 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559cf7a55b20 0x559cf7a57f30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.439+0000 7f91583b0b80 1 --2- 172.21.15.44:0/3123988037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559cf79ec8b0 0x559cf79f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.439+0000 7f91583b0b80 1 --2- 172.21.15.44:0/3123988037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559cf79e2aa0 0x559cf79f65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.439+0000 7f9125ffb700 0 client.25741 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:52.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.439+0000 7f91583b0b80 1 --2- 172.21.15.44:0/3123988037 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559cf79e8470 0x559cf79f6b30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.439+0000 7f91583b0b80 1 -- 172.21.15.44:0/3123988037 >> 172.21.15.44:0/3123988037 conn(0x559cf79dd360 msgr2=0x559cf79eba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:52.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.439+0000 7f91583b0b80 1 -- 172.21.15.44:0/3123988037 shutdown_connections 2023-11-16T16:50:52.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.439+0000 7f91583b0b80 1 -- 172.21.15.44:0/3123988037 wait complete. 2023-11-16T16:50:52.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.440+0000 7f91583b0b80 20 client.25741 trim_cache size 0 max 16384 2023-11-16T16:50:52.442 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:50:52.442 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T16:50:52.442 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T16:50:52.442 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:50:52.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.440+0000 7f91449ff700 1 -- 172.21.15.44:0/1357955354 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f91280a1460 msgr2=0x7f91280a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:52.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.440+0000 7f91449ff700 1 --2- 172.21.15.44:0/1357955354 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f91280a1460 0x7f91280a3910 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9138009ec0 tx=0x7f913800b040 comp rx=0 tx=0).stop 2023-11-16T16:50:52.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.440+0000 7f91449ff700 1 -- 172.21.15.44:0/1357955354 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9140100cc0 msgr2=0x7f914010c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:52.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.440+0000 7f91449ff700 1 --2- 172.21.15.44:0/1357955354 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9140100cc0 0x7f914010c030 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f91300098e0 tx=0x7f913000f690 comp rx=0 tx=0).stop 2023-11-16T16:50:52.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.440+0000 7f91449ff700 1 -- 172.21.15.44:0/1357955354 shutdown_connections 2023-11-16T16:50:52.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.440+0000 7f91449ff700 1 --2- 172.21.15.44:0/1357955354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9140105620 0x7f91401b11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.440+0000 7f91449ff700 1 --2- 172.21.15.44:0/1357955354 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f91280a1460 0x7f91280a3910 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.440+0000 7f91449ff700 1 --2- 172.21.15.44:0/1357955354 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9140100cc0 0x7f914010c030 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.440+0000 7f91449ff700 1 --2- 172.21.15.44:0/1357955354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9140101660 0x7f914010c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.440+0000 7f91449ff700 1 -- 172.21.15.44:0/1357955354 >> 172.21.15.44:0/1357955354 conn(0x7f9140077560 msgr2=0x7f91400fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:52.444 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.441+0000 7f91449ff700 1 -- 172.21.15.44:0/1357955354 shutdown_connections 2023-11-16T16:50:52.444 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.441+0000 7f91449ff700 1 -- 172.21.15.44:0/1357955354 wait complete. 2023-11-16T16:50:52.471 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T16:50:52.471 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:50:52.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.812+0000 7f38739ff700 1 -- 172.21.15.44:0/498815354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f386c100510 msgr2=0x7f386c1008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:52.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.812+0000 7f38739ff700 1 --2- 172.21.15.44:0/498815354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f386c100510 0x7f386c1008e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f38680099c0 tx=0x7f3868009cd0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.812+0000 7f38739ff700 1 -- 172.21.15.44:0/498815354 shutdown_connections 2023-11-16T16:50:52.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.812+0000 7f38739ff700 1 --2- 172.21.15.44:0/498815354 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f386c105550 0x7f386c10d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.812+0000 7f38739ff700 1 --2- 172.21.15.44:0/498815354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f386c100510 0x7f386c1008e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.812+0000 7f38739ff700 1 --2- 172.21.15.44:0/498815354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f386c100eb0 0x7f386c104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.812+0000 7f38739ff700 1 -- 172.21.15.44:0/498815354 >> 172.21.15.44:0/498815354 conn(0x7f386c077050 msgr2=0x7f386c077450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:52.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.812+0000 7f38739ff700 1 -- 172.21.15.44:0/498815354 shutdown_connections 2023-11-16T16:50:52.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.812+0000 7f38739ff700 1 -- 172.21.15.44:0/498815354 wait complete. 2023-11-16T16:50:52.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.814+0000 7f38739ff700 1 Processor -- start 2023-11-16T16:50:52.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.814+0000 7f38739ff700 1 -- start start 2023-11-16T16:50:52.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.815+0000 7f38739ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f386c100510 0x7f386c1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.815+0000 7f38739ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f386c100eb0 0x7f386c198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.815+0000 7f38739ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f386c105550 0x7f386c19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.815+0000 7f38739ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386c19d590 con 0x7f386c100510 2023-11-16T16:50:52.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.815+0000 7f38739ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386c19d700 con 0x7f386c105550 2023-11-16T16:50:52.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.815+0000 7f38739ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386c19d8c0 con 0x7f386c100eb0 2023-11-16T16:50:52.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.815+0000 7f387179b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f386c100510 0x7f386c1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.815+0000 7f3871f9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f386c105550 0x7f386c19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.815+0000 7f387179b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f386c100510 0x7f386c1985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57932/0 (socket says 172.21.15.44:57932) 2023-11-16T16:50:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.815+0000 7f387179b700 1 -- 172.21.15.44:0/17574130 learned_addr learned my addr 172.21.15.44:0/17574130 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.815+0000 7f3870f9a700 1 --2- 172.21.15.44:0/17574130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f386c100eb0 0x7f386c198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.816+0000 7f3871f9c700 1 -- 172.21.15.44:0/17574130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f386c100eb0 msgr2=0x7f386c198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.816+0000 7f3871f9c700 1 --2- 172.21.15.44:0/17574130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f386c100eb0 0x7f386c198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.816+0000 7f3871f9c700 1 -- 172.21.15.44:0/17574130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f386c100510 msgr2=0x7f386c1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.816+0000 7f3871f9c700 1 --2- 172.21.15.44:0/17574130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f386c100510 0x7f386c1985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.816+0000 7f3871f9c700 1 -- 172.21.15.44:0/17574130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38680095f0 con 0x7f386c105550 2023-11-16T16:50:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.816+0000 7f3871f9c700 1 --2- 172.21.15.44:0/17574130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f386c105550 0x7f386c19cf50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f386400ab40 tx=0x7f386400ae50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.817+0000 7f38627fc700 1 -- 172.21.15.44:0/17574130 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386400f070 con 0x7f386c105550 2023-11-16T16:50:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.817+0000 7f38627fc700 1 -- 172.21.15.44:0/17574130 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3864007a70 con 0x7f386c105550 2023-11-16T16:50:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.817+0000 7f38739ff700 1 -- 172.21.15.44:0/17574130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f386c110360 con 0x7f386c105550 2023-11-16T16:50:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.817+0000 7f38627fc700 1 -- 172.21.15.44:0/17574130 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3864005280 con 0x7f386c105550 2023-11-16T16:50:52.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.817+0000 7f38739ff700 1 -- 172.21.15.44:0/17574130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f386c1101d0 con 0x7f386c105550 2023-11-16T16:50:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.818+0000 7f38739ff700 1 -- 172.21.15.44:0/17574130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f386c1104f0 con 0x7f386c105550 2023-11-16T16:50:52.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.822+0000 7f38627fc700 1 -- 172.21.15.44:0/17574130 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f3864004ad0 con 0x7f386c105550 2023-11-16T16:50:52.826 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.824+0000 7f38627fc700 1 --2- 172.21.15.44:0/17574130 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f38580a1460 0x7f38580a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:52.826 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.825+0000 7f38627fc700 1 -- 172.21.15.44:0/17574130 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f38640cc740 con 0x7f386c105550 2023-11-16T16:50:52.826 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.825+0000 7f387179b700 1 --2- 172.21.15.44:0/17574130 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f38580a1460 0x7f38580a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:52.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.826+0000 7f387179b700 1 --2- 172.21.15.44:0/17574130 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f38580a1460 0x7f38580a3910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f38680036c0 tx=0x7f3868005fb0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:52.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:52.827+0000 7f38627fc700 1 -- 172.21.15.44:0/17574130 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f38640955b0 con 0x7f386c105550 2023-11-16T16:50:52.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:52 smithi044 ceph-mon[128750]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2023-11-16T16:50:52.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:52 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:50:52.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:52 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:52.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:52 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:50:52.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:52 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:52.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:52 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:50:52.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:52 smithi077 ceph-mon[131204]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2023-11-16T16:50:52.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:52 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:50:52.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:52 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:52.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:52 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:50:52.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:52 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:52.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:52 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:50:53.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:52 smithi176 ceph-mon[131630]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2023-11-16T16:50:53.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:52 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:50:53.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:52 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:53.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:52 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:50:53.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:52 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:53.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:52 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:50:53.341 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.339+0000 7f38739ff700 1 -- 172.21.15.44:0/17574130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f386c19db10 con 0x7f386c105550 2023-11-16T16:50:53.341 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.340+0000 7f38627fc700 1 -- 172.21.15.44:0/17574130 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f3864004dd0 con 0x7f386c105550 2023-11-16T16:50:53.342 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:50:53.342 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:50:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.343+0000 7f38739ff700 1 -- 172.21.15.44:0/17574130 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f38580a1460 msgr2=0x7f38580a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.343+0000 7f38739ff700 1 --2- 172.21.15.44:0/17574130 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f38580a1460 0x7f38580a3910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f38680036c0 tx=0x7f3868005fb0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.343+0000 7f38739ff700 1 -- 172.21.15.44:0/17574130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f386c105550 msgr2=0x7f386c19cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.343+0000 7f38739ff700 1 --2- 172.21.15.44:0/17574130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f386c105550 0x7f386c19cf50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f386400ab40 tx=0x7f386400ae50 comp rx=0 tx=0).stop 2023-11-16T16:50:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.343+0000 7f38739ff700 1 -- 172.21.15.44:0/17574130 shutdown_connections 2023-11-16T16:50:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.343+0000 7f38739ff700 1 --2- 172.21.15.44:0/17574130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f386c105550 0x7f386c19cf50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.343+0000 7f38739ff700 1 --2- 172.21.15.44:0/17574130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f386c100510 0x7f386c1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.343+0000 7f38739ff700 1 --2- 172.21.15.44:0/17574130 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f38580a1460 0x7f38580a3910 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.343+0000 7f38739ff700 1 --2- 172.21.15.44:0/17574130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f386c100eb0 0x7f386c198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.343+0000 7f38739ff700 1 -- 172.21.15.44:0/17574130 >> 172.21.15.44:0/17574130 conn(0x7f386c077050 msgr2=0x7f386c0fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.344+0000 7f38739ff700 1 -- 172.21.15.44:0/17574130 shutdown_connections 2023-11-16T16:50:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.344+0000 7f38739ff700 1 -- 172.21.15.44:0/17574130 wait complete. 2023-11-16T16:50:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:50:53.376 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T16:50:53.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.724+0000 7f32477ff700 1 -- 172.21.15.44:0/4175221038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3240108550 msgr2=0x7f3240108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:53.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.724+0000 7f32477ff700 1 --2- 172.21.15.44:0/4175221038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3240108550 0x7f3240108920 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f323c009990 tx=0x7f323c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.724+0000 7f32477ff700 1 -- 172.21.15.44:0/4175221038 shutdown_connections 2023-11-16T16:50:53.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.724+0000 7f32477ff700 1 --2- 172.21.15.44:0/4175221038 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3240102ed0 0x7f324010f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.724+0000 7f32477ff700 1 --2- 172.21.15.44:0/4175221038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3240108550 0x7f3240108920 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.724+0000 7f32477ff700 1 --2- 172.21.15.44:0/4175221038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3240102520 0x7f3240102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.724+0000 7f32477ff700 1 -- 172.21.15.44:0/4175221038 >> 172.21.15.44:0/4175221038 conn(0x7f32400fdfc0 msgr2=0x7f32401003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:53.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.724+0000 7f32477ff700 1 -- 172.21.15.44:0/4175221038 shutdown_connections 2023-11-16T16:50:53.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.724+0000 7f32477ff700 1 -- 172.21.15.44:0/4175221038 wait complete. 2023-11-16T16:50:53.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.726+0000 7f32477ff700 1 Processor -- start 2023-11-16T16:50:53.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.726+0000 7f32477ff700 1 -- start start 2023-11-16T16:50:53.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.727+0000 7f32477ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3240102520 0x7f32401985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.727+0000 7f32477ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3240102ed0 0x7f3240198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.727+0000 7f32477ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3240108550 0x7f324019cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.727+0000 7f32477ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324019d580 con 0x7f3240102ed0 2023-11-16T16:50:53.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.727+0000 7f32477ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324019d6f0 con 0x7f3240108550 2023-11-16T16:50:53.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.727+0000 7f32477ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324019d8b0 con 0x7f3240102520 2023-11-16T16:50:53.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.727+0000 7f3244d9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3240102ed0 0x7f3240198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.727+0000 7f3245d9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3240108550 0x7f324019cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.727+0000 7f324559b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3240102520 0x7f32401985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.727+0000 7f3245d9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3240108550 0x7f324019cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:54038/0 (socket says 172.21.15.44:54038) 2023-11-16T16:50:53.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.727+0000 7f3245d9c700 1 -- 172.21.15.44:0/1726203592 learned_addr learned my addr 172.21.15.44:0/1726203592 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:53.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.728+0000 7f3245d9c700 1 -- 172.21.15.44:0/1726203592 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3240102520 msgr2=0x7f32401985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:53.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.728+0000 7f3245d9c700 1 --2- 172.21.15.44:0/1726203592 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3240102520 0x7f32401985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.728+0000 7f3245d9c700 1 -- 172.21.15.44:0/1726203592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3240102ed0 msgr2=0x7f3240198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:53.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.728+0000 7f3245d9c700 1 --2- 172.21.15.44:0/1726203592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3240102ed0 0x7f3240198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.728+0000 7f3245d9c700 1 -- 172.21.15.44:0/1726203592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c0095f0 con 0x7f3240108550 2023-11-16T16:50:53.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.728+0000 7f3244d9a700 1 --2- 172.21.15.44:0/1726203592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3240102ed0 0x7f3240198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:53.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.728+0000 7f3245d9c700 1 --2- 172.21.15.44:0/1726203592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3240108550 0x7f324019cf40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3238007ee0 tx=0x7f323800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:53.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.728+0000 7f32367fc700 1 -- 172.21.15.44:0/1726203592 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323800dd20 con 0x7f3240108550 2023-11-16T16:50:53.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.729+0000 7f32367fc700 1 -- 172.21.15.44:0/1726203592 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f323800f040 con 0x7f3240108550 2023-11-16T16:50:53.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.729+0000 7f32367fc700 1 -- 172.21.15.44:0/1726203592 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32380048a0 con 0x7f3240108550 2023-11-16T16:50:53.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.729+0000 7f32477ff700 1 -- 172.21.15.44:0/1726203592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3240112570 con 0x7f3240108550 2023-11-16T16:50:53.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.729+0000 7f32477ff700 1 -- 172.21.15.44:0/1726203592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32401123e0 con 0x7f3240108550 2023-11-16T16:50:53.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.734+0000 7f32367fc700 1 -- 172.21.15.44:0/1726203592 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f3238003a00 con 0x7f3240108550 2023-11-16T16:50:53.737 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.735+0000 7f325b152b80 1 Processor -- start 2023-11-16T16:50:53.737 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.736+0000 7f325b152b80 1 -- start start 2023-11-16T16:50:53.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.736+0000 7f325b152b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731db60 0x559b2731df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.736+0000 7f325b152b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b2731e470 0x559b27322370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.736+0000 7f325b152b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b273228b0 0x559b27322d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.736+0000 7f325b152b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b27223060 con 0x559b2731e470 2023-11-16T16:50:53.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.736+0000 7f325b152b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b272758a0 con 0x559b273228b0 2023-11-16T16:50:53.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.736+0000 7f325b152b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b2726ed30 con 0x559b2731db60 2023-11-16T16:50:53.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.737+0000 7f32367fc700 1 --2- 172.21.15.44:0/1726203592 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f322c0a14e0 0x7f322c0a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.737+0000 7f3244d9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b273228b0 0x559b27322d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.737+0000 7f3244d9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b273228b0 0x559b27322d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:54046/0 (socket says 172.21.15.44:54046) 2023-11-16T16:50:53.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.737+0000 7f3244d9a700 1 -- 172.21.15.44:0/2426823270 learned_addr learned my addr 172.21.15.44:0/2426823270 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:53.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.737+0000 7f3244d9a700 1 --2- 172.21.15.44:0/2426823270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731db60 0x559b2731df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.737+0000 7f32367fc700 1 -- 172.21.15.44:0/1726203592 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f32380ccb80 con 0x7f3240108550 2023-11-16T16:50:53.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.738+0000 7f3244d9a700 1 -- 172.21.15.44:0/2426823270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731db60 msgr2=0x559b2731df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:53.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.738+0000 7f3244d9a700 1 --2- 172.21.15.44:0/2426823270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731db60 0x559b2731df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.738+0000 7f3244d9a700 1 -- 172.21.15.44:0/2426823270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b2731e470 msgr2=0x559b27322370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:53.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.737+0000 7f324559b700 1 --2- 172.21.15.44:0/2426823270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b2731e470 0x559b27322370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.738+0000 7f3244d9a700 1 --2- 172.21.15.44:0/2426823270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b2731e470 0x559b27322370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.738+0000 7f3244d9a700 1 -- 172.21.15.44:0/2426823270 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b270f0630 con 0x559b273228b0 2023-11-16T16:50:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.738+0000 7f324559b700 1 --2- 172.21.15.44:0/2426823270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b2731e470 0x559b27322370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:50:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.738+0000 7f3244d9a700 1 --2- 172.21.15.44:0/2426823270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b273228b0 0x559b27322d00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3230032510 tx=0x7f3230015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be096fa242dd3c98 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.738+0000 7f3245d9c700 1 --2- 172.21.15.44:0/1726203592 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f322c0a14e0 0x7f322c0a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.739+0000 7f322b7fe700 1 -- 172.21.15.44:0/2426823270 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3230038070 con 0x559b273228b0 2023-11-16T16:50:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.739+0000 7f322b7fe700 1 -- 172.21.15.44:0/2426823270 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3230004020 con 0x559b273228b0 2023-11-16T16:50:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.739+0000 7f322b7fe700 1 -- 172.21.15.44:0/2426823270 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3230004a10 con 0x559b273228b0 2023-11-16T16:50:53.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.740+0000 7f325b152b80 1 -- 172.21.15.44:0/2426823270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b273228b0 msgr2=0x559b27322d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:53.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.740+0000 7f3245d9c700 1 --2- 172.21.15.44:0/1726203592 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f322c0a14e0 0x7f322c0a3990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f32380c9dd0 tx=0x7f3238004ad0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:53.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.740+0000 7f325b152b80 1 --2- 172.21.15.44:0/2426823270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b273228b0 0x559b27322d00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3230032510 tx=0x7f3230015040 comp rx=0 tx=0).stop 2023-11-16T16:50:53.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.740+0000 7f325b152b80 1 -- 172.21.15.44:0/2426823270 shutdown_connections 2023-11-16T16:50:53.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.740+0000 7f325b152b80 1 --2- 172.21.15.44:0/2426823270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b273228b0 0x559b27322d00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.740+0000 7f325b152b80 1 --2- 172.21.15.44:0/2426823270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731db60 0x559b2731df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.740+0000 7f325b152b80 1 --2- 172.21.15.44:0/2426823270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b2731e470 0x559b27322370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.740+0000 7f325b152b80 1 -- 172.21.15.44:0/2426823270 >> 172.21.15.44:0/2426823270 conn(0x559b27313360 msgr2=0x559b27318b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:53.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.740+0000 7f325b152b80 1 -- 172.21.15.44:0/2426823270 shutdown_connections 2023-11-16T16:50:53.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.740+0000 7f325b152b80 1 -- 172.21.15.44:0/2426823270 wait complete. 2023-11-16T16:50:53.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.741+0000 7f325b152b80 1 -- start start 2023-11-16T16:50:53.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.741+0000 7f325b152b80 1 Processor -- start 2023-11-16T16:50:53.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.742+0000 7f325b152b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731db60 0x559b27327350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.742+0000 7f325b152b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731e470 0x559b27327890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.742+0000 7f325b152b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b273228b0 0x559b27327dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.742+0000 7f325b152b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b272108e0 con 0x559b273228b0 2023-11-16T16:50:53.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.742+0000 7f325b152b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b27210580 con 0x559b2731db60 2023-11-16T16:50:53.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.742+0000 7f325b152b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b270f2920 con 0x559b2731e470 2023-11-16T16:50:53.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.742+0000 7f325b152b80 10 client.0 _ll_register_callbacks cb 0x559b2722e780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:53.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.743+0000 7f3244d9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731db60 0x559b27327350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.743+0000 7f3244d9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731db60 0x559b27327350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:54052/0 (socket says 172.21.15.44:54052) 2023-11-16T16:50:53.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.743+0000 7f3244d9a700 1 -- 172.21.15.44:0/3836963848 learned_addr learned my addr 172.21.15.44:0/3836963848 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:53.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.743+0000 7f3244d9a700 1 --2- 172.21.15.44:0/3836963848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b273228b0 0x559b27327dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.743+0000 7f324559b700 1 --2- 172.21.15.44:0/3836963848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731e470 0x559b27327890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.743+0000 7f3244d9a700 1 -- 172.21.15.44:0/3836963848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731e470 msgr2=0x559b27327890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:53.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.743+0000 7f3244d9a700 1 --2- 172.21.15.44:0/3836963848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731e470 0x559b27327890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.743+0000 7f3244d9a700 1 -- 172.21.15.44:0/3836963848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b273228b0 msgr2=0x559b27327dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:53.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.743+0000 7f3244d9a700 1 --2- 172.21.15.44:0/3836963848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b273228b0 0x559b27327dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.743+0000 7f3244d9a700 1 -- 172.21.15.44:0/3836963848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323001b040 con 0x559b2731db60 2023-11-16T16:50:53.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.743+0000 7f3244d9a700 1 --2- 172.21.15.44:0/3836963848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731db60 0x559b27327350 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f32300044a0 tx=0x7f3230009b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f23c8467fd4348ac server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:53.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.743+0000 7f3228ff9700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:50:53.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.744+0000 7f324559b700 1 --2- 172.21.15.44:0/3836963848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731e470 0x559b27327890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:50:53.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.744+0000 7f3228ff9700 1 -- 172.21.15.44:0/3836963848 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3230038070 con 0x559b2731db60 2023-11-16T16:50:53.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.744+0000 7f3228ff9700 1 -- 172.21.15.44:0/3836963848 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f3230000e40 con 0x559b2731db60 2023-11-16T16:50:53.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.744+0000 7f325b152b80 1 -- 172.21.15.44:0/3836963848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559b27310b20 con 0x559b2731db60 2023-11-16T16:50:53.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.744+0000 7f3228ff9700 1 -- 172.21.15.44:0/3836963848 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3230012c70 con 0x559b2731db60 2023-11-16T16:50:53.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.744+0000 7f325b152b80 10 client.25760 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:53.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.744+0000 7f325b152b80 1 -- 172.21.15.44:0/3836963848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x559b27235190 con 0x559b2731db60 2023-11-16T16:50:53.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.744+0000 7f3228ff9700 1 -- 172.21.15.44:0/3836963848 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559b27310b20 con 0x559b2731db60 2023-11-16T16:50:53.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.745+0000 7f3228ff9700 1 -- 172.21.15.44:0/3836963848 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3230004890 con 0x559b2731db60 2023-11-16T16:50:53.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.745+0000 7f325b152b80 10 client.25760 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:53.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.745+0000 7f325b152b80 10 client.25760 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:53.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.745+0000 7f325b152b80 20 client.25760 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:53.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.745+0000 7f325b152b80 1 --2- 172.21.15.44:0/3836963848 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b27337800 0x559b27339c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.745+0000 7f325b152b80 4 client.25760 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:53.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.745+0000 7f325b152b80 1 -- 172.21.15.44:0/3836963848 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559b272310a0 con 0x559b27337800 2023-11-16T16:50:53.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.745+0000 7f324559b700 1 --2- 172.21.15.44:0/3836963848 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b27337800 0x559b27339c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.746+0000 7f3228ff9700 10 client.25760 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.746+0000 7f324559b700 1 --2- 172.21.15.44:0/3836963848 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b27337800 0x559b27339c10 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8ff722491426697 server_cookie=97d008155647bdfb in_seq=0 out_seq=0 2023-11-16T16:50:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.746+0000 7f3228ff9700 1 -- 172.21.15.44:0/3836963848 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559b272310a0 con 0x559b27337800 2023-11-16T16:50:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.746+0000 7f3228ff9700 10 client.25760 handle_command_reply: tid=0 2023-11-16T16:50:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.747+0000 7f325b152b80 1 client.25760 shutdown 2023-11-16T16:50:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.747+0000 7f325b152b80 1 -- 172.21.15.44:0/3836963848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731db60 msgr2=0x559b27327350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.747+0000 7f325b152b80 1 --2- 172.21.15.44:0/3836963848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731db60 0x559b27327350 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f32300044a0 tx=0x7f3230009b70 comp rx=0 tx=0).stop 2023-11-16T16:50:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.747+0000 7f325b152b80 1 -- 172.21.15.44:0/3836963848 shutdown_connections 2023-11-16T16:50:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.747+0000 7f325b152b80 1 --2- 172.21.15.44:0/3836963848 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b27337800 0x559b27339c10 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.747+0000 7f325b152b80 1 --2- 172.21.15.44:0/3836963848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b273228b0 0x559b27327dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.747+0000 7f325b152b80 1 --2- 172.21.15.44:0/3836963848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731db60 0x559b27327350 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.747+0000 7f325b152b80 1 --2- 172.21.15.44:0/3836963848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731e470 0x559b27327890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.747+0000 7f325b152b80 1 -- 172.21.15.44:0/3836963848 >> 172.21.15.44:0/3836963848 conn(0x559b27313360 msgr2=0x559b27323d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:53.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.747+0000 7f3228ff9700 0 client.25760 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:53.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.747+0000 7f325b152b80 1 -- 172.21.15.44:0/3836963848 shutdown_connections 2023-11-16T16:50:53.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.747+0000 7f325b152b80 1 -- 172.21.15.44:0/3836963848 wait complete. 2023-11-16T16:50:53.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.748+0000 7f325b152b80 20 client.25760 trim_cache size 0 max 16384 2023-11-16T16:50:53.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.774+0000 7f325b152b80 1 Processor -- start 2023-11-16T16:50:53.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.774+0000 7f325b152b80 1 -- start start 2023-11-16T16:50:53.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.774+0000 7f325b152b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731db60 0x559b27318be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.774+0000 7f325b152b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731e470 0x559b2732e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.774+0000 7f325b152b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b273228b0 0x559b27327c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.774+0000 7f325b152b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b273214a0 con 0x559b273228b0 2023-11-16T16:50:53.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.774+0000 7f325b152b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b273287e0 con 0x559b2731db60 2023-11-16T16:50:53.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.774+0000 7f325b152b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b2732eeb0 con 0x559b2731e470 2023-11-16T16:50:53.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.775+0000 7f3244d9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731db60 0x559b27318be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.775+0000 7f3244d9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731db60 0x559b27318be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:54062/0 (socket says 172.21.15.44:54062) 2023-11-16T16:50:53.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.775+0000 7f3244d9a700 1 -- 172.21.15.44:0/604862288 learned_addr learned my addr 172.21.15.44:0/604862288 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:53.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.775+0000 7f3244d9a700 1 --2- 172.21.15.44:0/604862288 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b273228b0 0x559b27327c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.775+0000 7f324559b700 1 --2- 172.21.15.44:0/604862288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731e470 0x559b2732e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.775+0000 7f3244d9a700 1 -- 172.21.15.44:0/604862288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731e470 msgr2=0x559b2732e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:53.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.775+0000 7f3244d9a700 1 --2- 172.21.15.44:0/604862288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731e470 0x559b2732e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.775+0000 7f3244d9a700 1 -- 172.21.15.44:0/604862288 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b273228b0 msgr2=0x559b27327c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:53.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.775+0000 7f3244d9a700 1 --2- 172.21.15.44:0/604862288 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b273228b0 0x559b27327c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.775+0000 7f3244d9a700 1 -- 172.21.15.44:0/604862288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b27314b00 con 0x559b2731db60 2023-11-16T16:50:53.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.775+0000 7f324559b700 1 --2- 172.21.15.44:0/604862288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731e470 0x559b2732e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:50:53.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.775+0000 7f3244d9a700 1 --2- 172.21.15.44:0/604862288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731db60 0x559b27318be0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3230003720 tx=0x7f3230009b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc18da46d6e4dfc7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:53.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.776+0000 7f322bfff700 1 -- 172.21.15.44:0/604862288 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3230038040 con 0x559b2731db60 2023-11-16T16:50:53.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.776+0000 7f322bfff700 1 -- 172.21.15.44:0/604862288 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3230009660 con 0x559b2731db60 2023-11-16T16:50:53.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.776+0000 7f322bfff700 1 -- 172.21.15.44:0/604862288 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32300038a0 con 0x559b2731db60 2023-11-16T16:50:53.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.777+0000 7f325b152b80 1 -- 172.21.15.44:0/604862288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731db60 msgr2=0x559b27318be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:53.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.777+0000 7f325b152b80 1 --2- 172.21.15.44:0/604862288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731db60 0x559b27318be0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3230003720 tx=0x7f3230009b70 comp rx=0 tx=0).stop 2023-11-16T16:50:53.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.777+0000 7f325b152b80 1 -- 172.21.15.44:0/604862288 shutdown_connections 2023-11-16T16:50:53.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.777+0000 7f325b152b80 1 --2- 172.21.15.44:0/604862288 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b273228b0 0x559b27327c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.777+0000 7f325b152b80 1 --2- 172.21.15.44:0/604862288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731db60 0x559b27318be0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.777+0000 7f325b152b80 1 --2- 172.21.15.44:0/604862288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731e470 0x559b2732e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.777+0000 7f325b152b80 1 -- 172.21.15.44:0/604862288 >> 172.21.15.44:0/604862288 conn(0x559b27313360 msgr2=0x559b27320930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:53.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.777+0000 7f325b152b80 1 -- 172.21.15.44:0/604862288 shutdown_connections 2023-11-16T16:50:53.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.777+0000 7f325b152b80 1 -- 172.21.15.44:0/604862288 wait complete. 2023-11-16T16:50:53.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.778+0000 7f325b152b80 1 -- start start 2023-11-16T16:50:53.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.778+0000 7f325b152b80 1 Processor -- start 2023-11-16T16:50:53.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.779+0000 7f325b152b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731db60 0x559b2732e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.779+0000 7f325b152b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731e470 0x559b2732b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.779+0000 7f325b152b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b273228b0 0x559b2732be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.779+0000 7f325b152b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b2732c360 con 0x559b273228b0 2023-11-16T16:50:53.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.779+0000 7f325b152b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b2732c4a0 con 0x559b2731e470 2023-11-16T16:50:53.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.779+0000 7f325b152b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b2732c660 con 0x559b2731db60 2023-11-16T16:50:53.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.779+0000 7f325b152b80 10 client.0 _ll_register_callbacks cb 0x7f3230009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:53.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.779+0000 7f3244d9a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731db60 0x559b2732e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.779+0000 7f3244d9a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731db60 0x559b2732e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48550/0 (socket says 172.21.15.44:48550) 2023-11-16T16:50:53.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.779+0000 7f3244d9a700 1 -- 172.21.15.44:0/493574149 learned_addr learned my addr 172.21.15.44:0/493574149 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:53.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.779+0000 7f3244d9a700 1 --2- 172.21.15.44:0/493574149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b273228b0 0x559b2732be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.780+0000 7f324559b700 1 --2- 172.21.15.44:0/493574149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731e470 0x559b2732b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.780+0000 7f3244d9a700 1 -- 172.21.15.44:0/493574149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731e470 msgr2=0x559b2732b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:53.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.780+0000 7f3244d9a700 1 --2- 172.21.15.44:0/493574149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731e470 0x559b2732b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.780+0000 7f3244d9a700 1 -- 172.21.15.44:0/493574149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b273228b0 msgr2=0x559b2732be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:53.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.780+0000 7f3244d9a700 1 --2- 172.21.15.44:0/493574149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b273228b0 0x559b2732be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.780+0000 7f3244d9a700 1 -- 172.21.15.44:0/493574149 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b27235190 con 0x559b2731db60 2023-11-16T16:50:53.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.780+0000 7f3244d9a700 1 --2- 172.21.15.44:0/493574149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731db60 0x559b2732e5c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f32300046a0 tx=0x7f3230013100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=534f1b578fdd09d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:53.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.780+0000 7f3228ff9700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:50:53.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.781+0000 7f3228ff9700 1 -- 172.21.15.44:0/493574149 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3230038070 con 0x559b2731db60 2023-11-16T16:50:53.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.781+0000 7f3228ff9700 1 -- 172.21.15.44:0/493574149 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f3230000e40 con 0x559b2731db60 2023-11-16T16:50:53.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.781+0000 7f325b152b80 1 -- 172.21.15.44:0/493574149 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3230004020 con 0x559b2731db60 2023-11-16T16:50:53.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.781+0000 7f3228ff9700 1 -- 172.21.15.44:0/493574149 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3230004900 con 0x559b2731db60 2023-11-16T16:50:53.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.781+0000 7f325b152b80 10 client.25756 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:53.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.781+0000 7f325b152b80 1 -- 172.21.15.44:0/493574149 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x559b2709cde0 con 0x559b2731db60 2023-11-16T16:50:53.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.782+0000 7f3228ff9700 1 -- 172.21.15.44:0/493574149 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f323006a080 con 0x559b2731db60 2023-11-16T16:50:53.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.782+0000 7f3228ff9700 1 -- 172.21.15.44:0/493574149 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3230032820 con 0x559b2731db60 2023-11-16T16:50:53.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.782+0000 7f325b152b80 10 client.25756 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:53.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.782+0000 7f325b152b80 10 client.25756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:53.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.782+0000 7f325b152b80 20 client.25756 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:53.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.782+0000 7f325b152b80 1 --2- 172.21.15.44:0/493574149 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b2738b660 0x559b2738da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:53.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.782+0000 7f325b152b80 4 client.25756 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T16:50:53.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.782+0000 7f325b152b80 1 -- 172.21.15.44:0/493574149 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559b2723d660 con 0x559b2738b660 2023-11-16T16:50:53.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.783+0000 7f324559b700 1 --2- 172.21.15.44:0/493574149 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b2738b660 0x559b2738da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:53.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.783+0000 7f324559b700 1 --2- 172.21.15.44:0/493574149 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b2738b660 0x559b2738da70 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ebb9a6d4242c875 server_cookie=4f187745e21ca3f7 in_seq=0 out_seq=0 2023-11-16T16:50:53.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.783+0000 7f3228ff9700 10 client.25756 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:53.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.783+0000 7f3228ff9700 1 -- 172.21.15.44:0/493574149 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559b2723d660 con 0x559b2738b660 2023-11-16T16:50:53.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.783+0000 7f3228ff9700 10 client.25756 handle_command_reply: tid=0 2023-11-16T16:50:53.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.784+0000 7f325b152b80 1 client.25756 shutdown 2023-11-16T16:50:53.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.784+0000 7f325b152b80 1 -- 172.21.15.44:0/493574149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731db60 msgr2=0x559b2732e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:53.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.784+0000 7f325b152b80 1 --2- 172.21.15.44:0/493574149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731db60 0x559b2732e5c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f32300046a0 tx=0x7f3230013100 comp rx=0 tx=0).stop 2023-11-16T16:50:53.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.784+0000 7f325b152b80 1 -- 172.21.15.44:0/493574149 shutdown_connections 2023-11-16T16:50:53.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.784+0000 7f325b152b80 1 --2- 172.21.15.44:0/493574149 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b2738b660 0x559b2738da70 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.784+0000 7f325b152b80 1 --2- 172.21.15.44:0/493574149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b273228b0 0x559b2732be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.784+0000 7f325b152b80 1 --2- 172.21.15.44:0/493574149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b2731db60 0x559b2732e5c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.784+0000 7f3228ff9700 0 client.25756 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:53.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.784+0000 7f325b152b80 1 --2- 172.21.15.44:0/493574149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b2731e470 0x559b2732b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.784+0000 7f325b152b80 1 -- 172.21.15.44:0/493574149 >> 172.21.15.44:0/493574149 conn(0x559b27313360 msgr2=0x559b27327b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:53.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.784+0000 7f325b152b80 1 -- 172.21.15.44:0/493574149 shutdown_connections 2023-11-16T16:50:53.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.784+0000 7f325b152b80 1 -- 172.21.15.44:0/493574149 wait complete. 2023-11-16T16:50:53.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.785+0000 7f325b152b80 20 client.25756 trim_cache size 0 max 16384 2023-11-16T16:50:53.788 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T16:50:53.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.785+0000 7f32477ff700 1 -- 172.21.15.44:0/1726203592 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f322c0a14e0 msgr2=0x7f322c0a3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:53.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.785+0000 7f32477ff700 1 --2- 172.21.15.44:0/1726203592 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f322c0a14e0 0x7f322c0a3990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f32380c9dd0 tx=0x7f3238004ad0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.785+0000 7f32477ff700 1 -- 172.21.15.44:0/1726203592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3240108550 msgr2=0x7f324019cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:53.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.785+0000 7f32477ff700 1 --2- 172.21.15.44:0/1726203592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3240108550 0x7f324019cf40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3238007ee0 tx=0x7f323800d3b0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.786+0000 7f32477ff700 1 -- 172.21.15.44:0/1726203592 shutdown_connections 2023-11-16T16:50:53.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.786+0000 7f32477ff700 1 --2- 172.21.15.44:0/1726203592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3240108550 0x7f324019cf40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.786+0000 7f32477ff700 1 --2- 172.21.15.44:0/1726203592 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f322c0a14e0 0x7f322c0a3990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.786+0000 7f32477ff700 1 --2- 172.21.15.44:0/1726203592 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3240102520 0x7f32401985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.786+0000 7f32477ff700 1 --2- 172.21.15.44:0/1726203592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3240102ed0 0x7f3240198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:53.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.786+0000 7f32477ff700 1 -- 172.21.15.44:0/1726203592 >> 172.21.15.44:0/1726203592 conn(0x7f32400fdfc0 msgr2=0x7f32400ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:53.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.786+0000 7f32477ff700 1 -- 172.21.15.44:0/1726203592 shutdown_connections 2023-11-16T16:50:53.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:53.786+0000 7f32477ff700 1 -- 172.21.15.44:0/1726203592 wait complete. 2023-11-16T16:50:54.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:54 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/17574130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:50:54.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:54 smithi077 ceph-mon[131204]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 426 B/s wr, 2 op/s 2023-11-16T16:50:54.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:54 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/17574130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:50:54.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:54 smithi044 ceph-mon[128750]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 426 B/s wr, 2 op/s 2023-11-16T16:50:54.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:54 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/17574130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:50:54.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:54 smithi176 ceph-mon[131630]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 426 B/s wr, 2 op/s 2023-11-16T16:50:54.817 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:50:54.818 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:50:55.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.176+0000 7fe5ea3ff700 1 -- 172.21.15.44:0/1209147363 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5e4101390 msgr2=0x7fe5e4101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.176+0000 7fe5ea3ff700 1 --2- 172.21.15.44:0/1209147363 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5e4101390 0x7fe5e4101760 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4009990 tx=0x7fe5d4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.176+0000 7fe5ea3ff700 1 -- 172.21.15.44:0/1209147363 shutdown_connections 2023-11-16T16:50:55.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.176+0000 7fe5ea3ff700 1 --2- 172.21.15.44:0/1209147363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5e410a8b0 0x7fe5e410ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.176+0000 7fe5ea3ff700 1 --2- 172.21.15.44:0/1209147363 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5e4101390 0x7fe5e4101760 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.176+0000 7fe5ea3ff700 1 --2- 172.21.15.44:0/1209147363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e4101ca0 0x7fe5e410a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.176+0000 7fe5ea3ff700 1 -- 172.21.15.44:0/1209147363 >> 172.21.15.44:0/1209147363 conn(0x7fe5e40770b0 msgr2=0x7fe5e40774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:55.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.176+0000 7fe5ea3ff700 1 -- 172.21.15.44:0/1209147363 shutdown_connections 2023-11-16T16:50:55.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.176+0000 7fe5ea3ff700 1 -- 172.21.15.44:0/1209147363 wait complete. 2023-11-16T16:50:55.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.178+0000 7fe5ea3ff700 1 Processor -- start 2023-11-16T16:50:55.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.178+0000 7fe5ea3ff700 1 -- start start 2023-11-16T16:50:55.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.178+0000 7fe5ea3ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5e4101390 0x7fe5e4109270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.178+0000 7fe5ea3ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5e4101ca0 0x7fe5e41097b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.178+0000 7fe5ea3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e410a8b0 0x7fe5e41b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.179+0000 7fe5ea3ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e406a510 con 0x7fe5e4101ca0 2023-11-16T16:50:55.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.179+0000 7fe5ea3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e406a680 con 0x7fe5e410a8b0 2023-11-16T16:50:55.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.179+0000 7fe5ea3ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e406a840 con 0x7fe5e4101390 2023-11-16T16:50:55.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.179+0000 7fe5e37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5e4101ca0 0x7fe5e41097b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.179+0000 7fe5e37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5e4101ca0 0x7fe5e41097b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39796/0 (socket says 172.21.15.44:39796) 2023-11-16T16:50:55.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.179+0000 7fe5e37fe700 1 -- 172.21.15.44:0/3198099773 learned_addr learned my addr 172.21.15.44:0/3198099773 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:55.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.179+0000 7fe5e899c700 1 --2- 172.21.15.44:0/3198099773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e410a8b0 0x7fe5e41b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.179+0000 7fe5e3fff700 1 --2- 172.21.15.44:0/3198099773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5e4101390 0x7fe5e4109270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.180+0000 7fe5e899c700 1 -- 172.21.15.44:0/3198099773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5e4101390 msgr2=0x7fe5e4109270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.180+0000 7fe5e899c700 1 --2- 172.21.15.44:0/3198099773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5e4101390 0x7fe5e4109270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.180+0000 7fe5e899c700 1 -- 172.21.15.44:0/3198099773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5e4101ca0 msgr2=0x7fe5e41097b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.180+0000 7fe5e899c700 1 --2- 172.21.15.44:0/3198099773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5e4101ca0 0x7fe5e41097b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.180+0000 7fe5e899c700 1 -- 172.21.15.44:0/3198099773 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5dc009c40 con 0x7fe5e410a8b0 2023-11-16T16:50:55.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.180+0000 7fe5e899c700 1 --2- 172.21.15.44:0/3198099773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e410a8b0 0x7fe5e41b1170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe5e406a4a0 tx=0x7fe5dc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:55.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.180+0000 7fe5e17fa700 1 -- 172.21.15.44:0/3198099773 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc003a00 con 0x7fe5e410a8b0 2023-11-16T16:50:55.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.180+0000 7fe5e17fa700 1 -- 172.21.15.44:0/3198099773 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5dc00f040 con 0x7fe5e410a8b0 2023-11-16T16:50:55.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.180+0000 7fe5e17fa700 1 -- 172.21.15.44:0/3198099773 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc004570 con 0x7fe5e410a8b0 2023-11-16T16:50:55.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.180+0000 7fe5ea3ff700 1 -- 172.21.15.44:0/3198099773 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d40095f0 con 0x7fe5e410a8b0 2023-11-16T16:50:55.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.181+0000 7fe5ea3ff700 1 -- 172.21.15.44:0/3198099773 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e4110380 con 0x7fe5e410a8b0 2023-11-16T16:50:55.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.186+0000 7fe5e17fa700 1 -- 172.21.15.44:0/3198099773 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7fe5dc003b60 con 0x7fe5e410a8b0 2023-11-16T16:50:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.187+0000 7fe5fdc5bb80 1 Processor -- start 2023-11-16T16:50:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.188+0000 7fe5fdc5bb80 1 -- start start 2023-11-16T16:50:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.188+0000 7fe5fdc5bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd63fbe0 0x561ccd63ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.188+0000 7fe5fdc5bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd6404f0 0x561ccd6443f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.188+0000 7fe5fdc5bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd644930 0x561ccd644d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.188+0000 7fe5fdc5bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ccd5450e0 con 0x561ccd6404f0 2023-11-16T16:50:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.188+0000 7fe5fdc5bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ccd597920 con 0x561ccd644930 2023-11-16T16:50:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.188+0000 7fe5fdc5bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ccd590db0 con 0x561ccd63fbe0 2023-11-16T16:50:55.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.189+0000 7fe5e37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd644930 0x561ccd644d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.189+0000 7fe5e3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd6404f0 0x561ccd6443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.189+0000 7fe5e37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd644930 0x561ccd644d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:50374/0 (socket says 172.21.15.44:50374) 2023-11-16T16:50:55.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.189+0000 7fe5e37fe700 1 -- 172.21.15.44:0/3490156002 learned_addr learned my addr 172.21.15.44:0/3490156002 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:55.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.189+0000 7fe5e37fe700 1 --2- 172.21.15.44:0/3490156002 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd63fbe0 0x561ccd63ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.189+0000 7fe5e37fe700 1 -- 172.21.15.44:0/3490156002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd644930 msgr2=0x561ccd644d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:55.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.189+0000 7fe5e37fe700 1 --2- 172.21.15.44:0/3490156002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd644930 0x561ccd644d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.189+0000 7fe5e37fe700 1 -- 172.21.15.44:0/3490156002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd6404f0 msgr2=0x561ccd6443f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:55.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.189+0000 7fe5e37fe700 1 --2- 172.21.15.44:0/3490156002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd6404f0 0x561ccd6443f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.189+0000 7fe5e37fe700 1 -- 172.21.15.44:0/3490156002 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ccd4126b0 con 0x561ccd63fbe0 2023-11-16T16:50:55.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.189+0000 7fe5e3fff700 1 --2- 172.21.15.44:0/3490156002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd6404f0 0x561ccd6443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:55.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.190+0000 7fe5e37fe700 1 --2- 172.21.15.44:0/3490156002 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd63fbe0 0x561ccd63ffb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8012820 tx=0x7fe5d8007480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be9abe5348228f57 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:55.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.190+0000 7fe5e17fa700 1 --2- 172.21.15.44:0/3198099773 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe5cc0a1460 0x7fe5cc0a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.190+0000 7fe5ca7fc700 1 -- 172.21.15.44:0/3490156002 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8004030 con 0x561ccd63fbe0 2023-11-16T16:50:55.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.190+0000 7fe5ca7fc700 1 -- 172.21.15.44:0/3490156002 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5d8004830 con 0x561ccd63fbe0 2023-11-16T16:50:55.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.190+0000 7fe5ca7fc700 1 -- 172.21.15.44:0/3490156002 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8005280 con 0x561ccd63fbe0 2023-11-16T16:50:55.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.190+0000 7fe5e17fa700 1 -- 172.21.15.44:0/3198099773 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fe5dc0ccd40 con 0x7fe5e410a8b0 2023-11-16T16:50:55.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.190+0000 7fe5e3fff700 1 --2- 172.21.15.44:0/3198099773 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe5cc0a1460 0x7fe5cc0a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.191+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/3490156002 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd63fbe0 msgr2=0x561ccd63ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.191+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/3490156002 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd63fbe0 0x561ccd63ffb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8012820 tx=0x7fe5d8007480 comp rx=0 tx=0).stop 2023-11-16T16:50:55.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.191+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/3490156002 shutdown_connections 2023-11-16T16:50:55.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.191+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/3490156002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd644930 0x561ccd644d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.191+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/3490156002 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd63fbe0 0x561ccd63ffb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.191+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/3490156002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd6404f0 0x561ccd6443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.191+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/3490156002 >> 172.21.15.44:0/3490156002 conn(0x561ccd6353e0 msgr2=0x561ccd63abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:55.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.192+0000 7fe5e3fff700 1 --2- 172.21.15.44:0/3198099773 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe5cc0a1460 0x7fe5cc0a3910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe5e410a2d0 tx=0x7fe5d4005fd0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:55.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.192+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/3490156002 shutdown_connections 2023-11-16T16:50:55.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.192+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/3490156002 wait complete. 2023-11-16T16:50:55.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.192+0000 7fe5fdc5bb80 1 -- start start 2023-11-16T16:50:55.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.193+0000 7fe5fdc5bb80 1 Processor -- start 2023-11-16T16:50:55.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.194+0000 7fe5fdc5bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd63fbe0 0x561ccd64c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.194+0000 7fe5fdc5bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd6404f0 0x561ccd649370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.194+0000 7fe5fdc5bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd644930 0x561ccd649a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.194+0000 7fe5fdc5bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ccd532960 con 0x561ccd6404f0 2023-11-16T16:50:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.194+0000 7fe5fdc5bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ccd532600 con 0x561ccd644930 2023-11-16T16:50:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.194+0000 7fe5fdc5bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ccd4149a0 con 0x561ccd63fbe0 2023-11-16T16:50:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.194+0000 7fe5fdc5bb80 10 client.0 _ll_register_callbacks cb 0x561ccd550800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.194+0000 7fe5e3fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd63fbe0 0x561ccd64c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.194+0000 7fe5e3fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd63fbe0 0x561ccd64c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:43764/0 (socket says 172.21.15.44:43764) 2023-11-16T16:50:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.194+0000 7fe5e3fff700 1 -- 172.21.15.44:0/2278272832 learned_addr learned my addr 172.21.15.44:0/2278272832 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.194+0000 7fe5e37fe700 1 --2- 172.21.15.44:0/2278272832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd6404f0 0x561ccd649370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.194+0000 7fe5e899c700 1 --2- 172.21.15.44:0/2278272832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd644930 0x561ccd649a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.195+0000 7fe5e3fff700 1 -- 172.21.15.44:0/2278272832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd644930 msgr2=0x561ccd649a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:55.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.195+0000 7fe5e3fff700 1 --2- 172.21.15.44:0/2278272832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd644930 0x561ccd649a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.195+0000 7fe5e3fff700 1 -- 172.21.15.44:0/2278272832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd6404f0 msgr2=0x561ccd649370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:55.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.195+0000 7fe5e3fff700 1 --2- 172.21.15.44:0/2278272832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd6404f0 0x561ccd649370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.195+0000 7fe5e3fff700 1 -- 172.21.15.44:0/2278272832 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d801a040 con 0x561ccd63fbe0 2023-11-16T16:50:55.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.195+0000 7fe5e3fff700 1 --2- 172.21.15.44:0/2278272832 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd63fbe0 0x561ccd64c0f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4015000 tx=0x7fe5d4009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b25ef522a6e647a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:55.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.195+0000 7fe5c37fe700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:50:55.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.195+0000 7fe5c37fe700 1 -- 172.21.15.44:0/2278272832 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4004ea0 con 0x561ccd63fbe0 2023-11-16T16:50:55.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.196+0000 7fe5c37fe700 1 -- 172.21.15.44:0/2278272832 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fe5d4026460 con 0x561ccd63fbe0 2023-11-16T16:50:55.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.196+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/2278272832 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561ccd632ba0 con 0x561ccd63fbe0 2023-11-16T16:50:55.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.196+0000 7fe5c37fe700 1 -- 172.21.15.44:0/2278272832 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5d400bed0 con 0x561ccd63fbe0 2023-11-16T16:50:55.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.197+0000 7fe5fdc5bb80 10 client.25768 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:55.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.197+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/2278272832 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x561ccd557210 con 0x561ccd63fbe0 2023-11-16T16:50:55.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.197+0000 7fe5c37fe700 1 -- 172.21.15.44:0/2278272832 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561ccd632ba0 con 0x561ccd63fbe0 2023-11-16T16:50:55.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.198+0000 7fe5c37fe700 1 -- 172.21.15.44:0/2278272832 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe5d4038070 con 0x561ccd63fbe0 2023-11-16T16:50:55.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.198+0000 7fe5fdc5bb80 10 client.25768 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:55.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.198+0000 7fe5fdc5bb80 10 client.25768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:55.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.198+0000 7fe5fdc5bb80 20 client.25768 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:55.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.198+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/2278272832 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561ccd6598e0 0x561ccd65bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.198+0000 7fe5fdc5bb80 4 client.25768 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:55.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.198+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/2278272832 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561ccd553120 con 0x561ccd6598e0 2023-11-16T16:50:55.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.198+0000 7fe5e37fe700 1 --2- 172.21.15.44:0/2278272832 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561ccd6598e0 0x561ccd65bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.199+0000 7fe5e37fe700 1 --2- 172.21.15.44:0/2278272832 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561ccd6598e0 0x561ccd65bcf0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9a03385a2b048ed server_cookie=3c1d5546631e7ad7 in_seq=0 out_seq=0 2023-11-16T16:50:55.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.199+0000 7fe5c37fe700 10 client.25768 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:55.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.200+0000 7fe5c37fe700 1 -- 172.21.15.44:0/2278272832 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x561ccd553120 con 0x561ccd6598e0 2023-11-16T16:50:55.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.200+0000 7fe5c37fe700 10 client.25768 handle_command_reply: tid=0 2023-11-16T16:50:55.202 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.200+0000 7fe5fdc5bb80 1 client.25768 shutdown 2023-11-16T16:50:55.202 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.200+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/2278272832 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd63fbe0 msgr2=0x561ccd64c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.202 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.200+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/2278272832 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd63fbe0 0x561ccd64c0f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4015000 tx=0x7fe5d4009310 comp rx=0 tx=0).stop 2023-11-16T16:50:55.202 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.201+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/2278272832 shutdown_connections 2023-11-16T16:50:55.202 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.201+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/2278272832 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561ccd6598e0 0x561ccd65bcf0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.202 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.201+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/2278272832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd644930 0x561ccd649a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.201+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/2278272832 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd63fbe0 0x561ccd64c0f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.201+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/2278272832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd6404f0 0x561ccd649370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.201+0000 7fe5c37fe700 0 client.25768 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:55.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.201+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/2278272832 >> 172.21.15.44:0/2278272832 conn(0x561ccd6353e0 msgr2=0x561ccd639e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:55.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.201+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/2278272832 shutdown_connections 2023-11-16T16:50:55.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.201+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/2278272832 wait complete. 2023-11-16T16:50:55.203 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.201+0000 7fe5fdc5bb80 20 client.25768 trim_cache size 0 max 16384 2023-11-16T16:50:55.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.236+0000 7fe5fdc5bb80 1 Processor -- start 2023-11-16T16:50:55.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.236+0000 7fe5fdc5bb80 1 -- start start 2023-11-16T16:50:55.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.236+0000 7fe5fdc5bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd63fbe0 0x561ccd6509f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.236+0000 7fe5fdc5bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd6404f0 0x561ccd64bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.236+0000 7fe5fdc5bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd644930 0x561ccd642220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.237+0000 7fe5fdc5bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ccd63b9d0 con 0x561ccd63fbe0 2023-11-16T16:50:55.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.237+0000 7fe5fdc5bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ccd6478d0 con 0x561ccd6404f0 2023-11-16T16:50:55.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.237+0000 7fe5fdc5bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ccd6434f0 con 0x561ccd644930 2023-11-16T16:50:55.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.237+0000 7fe5e37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd6404f0 0x561ccd64bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.237+0000 7fe5e37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd6404f0 0x561ccd64bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:50392/0 (socket says 172.21.15.44:50392) 2023-11-16T16:50:55.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.237+0000 7fe5e37fe700 1 -- 172.21.15.44:0/3635853662 learned_addr learned my addr 172.21.15.44:0/3635853662 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:55.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.237+0000 7fe5e3fff700 1 --2- 172.21.15.44:0/3635853662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd63fbe0 0x561ccd6509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.237+0000 7fe5e899c700 1 --2- 172.21.15.44:0/3635853662 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd644930 0x561ccd642220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.238+0000 7fe5e37fe700 1 -- 172.21.15.44:0/3635853662 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd644930 msgr2=0x561ccd642220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:55.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.238+0000 7fe5e37fe700 1 --2- 172.21.15.44:0/3635853662 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd644930 0x561ccd642220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.238+0000 7fe5e37fe700 1 -- 172.21.15.44:0/3635853662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd63fbe0 msgr2=0x561ccd6509f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:55.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.238+0000 7fe5e37fe700 1 --2- 172.21.15.44:0/3635853662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd63fbe0 0x561ccd6509f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.238+0000 7fe5e37fe700 1 -- 172.21.15.44:0/3635853662 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ccd636b80 con 0x561ccd6404f0 2023-11-16T16:50:55.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.238+0000 7fe5e37fe700 1 --2- 172.21.15.44:0/3635853662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd6404f0 0x561ccd64bc90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80127c0 tx=0x7fe5d8004f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd2e7c813929b54d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:55.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.239+0000 7fe5caffd700 1 -- 172.21.15.44:0/3635853662 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8026070 con 0x561ccd6404f0 2023-11-16T16:50:55.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.239+0000 7fe5caffd700 1 -- 172.21.15.44:0/3635853662 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5d8027040 con 0x561ccd6404f0 2023-11-16T16:50:55.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.239+0000 7fe5caffd700 1 -- 172.21.15.44:0/3635853662 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8028dc0 con 0x561ccd6404f0 2023-11-16T16:50:55.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.240+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/3635853662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd6404f0 msgr2=0x561ccd64bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.240+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/3635853662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd6404f0 0x561ccd64bc90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80127c0 tx=0x7fe5d8004f10 comp rx=0 tx=0).stop 2023-11-16T16:50:55.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.240+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/3635853662 shutdown_connections 2023-11-16T16:50:55.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.240+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/3635853662 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd644930 0x561ccd642220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.240+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/3635853662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd63fbe0 0x561ccd6509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.240+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/3635853662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd6404f0 0x561ccd64bc90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.240+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/3635853662 >> 172.21.15.44:0/3635853662 conn(0x561ccd6353e0 msgr2=0x561ccd643a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:55.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.240+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/3635853662 shutdown_connections 2023-11-16T16:50:55.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.240+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/3635853662 wait complete. 2023-11-16T16:50:55.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.241+0000 7fe5fdc5bb80 1 -- start start 2023-11-16T16:50:55.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.241+0000 7fe5fdc5bb80 1 Processor -- start 2023-11-16T16:50:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.242+0000 7fe5fdc5bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd63fbe0 0x561ccd645d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.242+0000 7fe5fdc5bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd6404f0 0x561ccd6462c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.242+0000 7fe5fdc5bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd644930 0x561ccd646800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.242+0000 7fe5fdc5bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ccd6ad7a0 con 0x561ccd644930 2023-11-16T16:50:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.242+0000 7fe5fdc5bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ccd6ad8e0 con 0x561ccd63fbe0 2023-11-16T16:50:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.242+0000 7fe5fdc5bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ccd6ada20 con 0x561ccd6404f0 2023-11-16T16:50:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.242+0000 7fe5fdc5bb80 10 client.0 _ll_register_callbacks cb 0x7fe5d4005350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.242+0000 7fe5e899c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd644930 0x561ccd646800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.242+0000 7fe5e37fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd6404f0 0x561ccd6462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.242+0000 7fe5e37fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd6404f0 0x561ccd6462c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:43790/0 (socket says 172.21.15.44:43790) 2023-11-16T16:50:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.242+0000 7fe5e37fe700 1 -- 172.21.15.44:0/993609492 learned_addr learned my addr 172.21.15.44:0/993609492 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.242+0000 7fe5e3fff700 1 --2- 172.21.15.44:0/993609492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd63fbe0 0x561ccd645d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.243+0000 7fe5e3fff700 1 -- 172.21.15.44:0/993609492 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd6404f0 msgr2=0x561ccd6462c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:55.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.243+0000 7fe5e3fff700 1 --2- 172.21.15.44:0/993609492 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd6404f0 0x561ccd6462c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.243+0000 7fe5e3fff700 1 -- 172.21.15.44:0/993609492 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd644930 msgr2=0x561ccd646800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.243+0000 7fe5e3fff700 1 --2- 172.21.15.44:0/993609492 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd644930 0x561ccd646800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.243+0000 7fe5e3fff700 1 -- 172.21.15.44:0/993609492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ccd4126b0 con 0x561ccd63fbe0 2023-11-16T16:50:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.243+0000 7fe5e899c700 1 --2- 172.21.15.44:0/993609492 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd644930 0x561ccd646800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.243+0000 7fe5e3fff700 1 --2- 172.21.15.44:0/993609492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd63fbe0 0x561ccd645d80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4028000 tx=0x7fe5d400b2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5f9be94a6b9bb8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.243+0000 7fe5c37fe700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:50:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.243+0000 7fe5c37fe700 1 -- 172.21.15.44:0/993609492 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4004d70 con 0x561ccd63fbe0 2023-11-16T16:50:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.243+0000 7fe5c37fe700 1 -- 172.21.15.44:0/993609492 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fe5d400fa80 con 0x561ccd63fbe0 2023-11-16T16:50:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.244+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/993609492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5d8004830 con 0x561ccd63fbe0 2023-11-16T16:50:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.244+0000 7fe5c37fe700 1 -- 172.21.15.44:0/993609492 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5d4004ed0 con 0x561ccd63fbe0 2023-11-16T16:50:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.244+0000 7fe5fdc5bb80 10 client.25784 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.244+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/993609492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x561ccd3bee60 con 0x561ccd63fbe0 2023-11-16T16:50:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.244+0000 7fe5c37fe700 1 -- 172.21.15.44:0/993609492 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5d8004830 con 0x561ccd63fbe0 2023-11-16T16:50:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.245+0000 7fe5c37fe700 1 -- 172.21.15.44:0/993609492 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe5d404a050 con 0x561ccd63fbe0 2023-11-16T16:50:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.245+0000 7fe5fdc5bb80 10 client.25784 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.245+0000 7fe5fdc5bb80 10 client.25784 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.245+0000 7fe5fdc5bb80 20 client.25784 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.245+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/993609492 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561ccd6aecf0 0x561ccd6b1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.245+0000 7fe5fdc5bb80 4 client.25784 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:50:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.245+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/993609492 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x561ccd55f6e0 con 0x561ccd6aecf0 2023-11-16T16:50:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.245+0000 7fe5e37fe700 1 --2- 172.21.15.44:0/993609492 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561ccd6aecf0 0x561ccd6b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.246+0000 7fe5e37fe700 1 --2- 172.21.15.44:0/993609492 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561ccd6aecf0 0x561ccd6b1100 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73f0d588c0e493c6 server_cookie=1db216adb715f869 in_seq=0 out_seq=0 2023-11-16T16:50:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.246+0000 7fe5c37fe700 10 client.25784 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.247+0000 7fe5c37fe700 1 -- 172.21.15.44:0/993609492 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561ccd55f6e0 con 0x561ccd6aecf0 2023-11-16T16:50:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.247+0000 7fe5c37fe700 10 client.25784 handle_command_reply: tid=0 2023-11-16T16:50:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.247+0000 7fe5fdc5bb80 1 client.25784 shutdown 2023-11-16T16:50:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.247+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/993609492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd63fbe0 msgr2=0x561ccd645d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.247+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/993609492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd63fbe0 0x561ccd645d80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4028000 tx=0x7fe5d400b2c0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.248+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/993609492 shutdown_connections 2023-11-16T16:50:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.248+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/993609492 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561ccd6aecf0 0x561ccd6b1100 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.248+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/993609492 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ccd644930 0x561ccd646800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.248+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/993609492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ccd63fbe0 0x561ccd645d80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.248+0000 7fe5fdc5bb80 1 --2- 172.21.15.44:0/993609492 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561ccd6404f0 0x561ccd6462c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.248+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/993609492 >> 172.21.15.44:0/993609492 conn(0x561ccd6353e0 msgr2=0x561ccd63e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:55.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.248+0000 7fe5c37fe700 0 client.25784 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:55.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.248+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/993609492 shutdown_connections 2023-11-16T16:50:55.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.248+0000 7fe5fdc5bb80 1 -- 172.21.15.44:0/993609492 wait complete. 2023-11-16T16:50:55.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.248+0000 7fe5fdc5bb80 20 client.25784 trim_cache size 0 max 16384 2023-11-16T16:50:55.251 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:50:55.251 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:50:55.251 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "452d3c6b-e145-4fef-8021-c3ec192fbe5b", 2023-11-16T16:50:55.251 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:50:55.252 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:50:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.249+0000 7fe5ea3ff700 1 -- 172.21.15.44:0/3198099773 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe5cc0a1460 msgr2=0x7fe5cc0a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.249+0000 7fe5ea3ff700 1 --2- 172.21.15.44:0/3198099773 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe5cc0a1460 0x7fe5cc0a3910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe5e410a2d0 tx=0x7fe5d4005fd0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.249+0000 7fe5ea3ff700 1 -- 172.21.15.44:0/3198099773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e410a8b0 msgr2=0x7fe5e41b1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.249+0000 7fe5ea3ff700 1 --2- 172.21.15.44:0/3198099773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e410a8b0 0x7fe5e41b1170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe5e406a4a0 tx=0x7fe5dc00c3a0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.249+0000 7fe5ea3ff700 1 -- 172.21.15.44:0/3198099773 shutdown_connections 2023-11-16T16:50:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.249+0000 7fe5ea3ff700 1 --2- 172.21.15.44:0/3198099773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e410a8b0 0x7fe5e41b1170 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.249+0000 7fe5ea3ff700 1 --2- 172.21.15.44:0/3198099773 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe5cc0a1460 0x7fe5cc0a3910 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.249+0000 7fe5ea3ff700 1 --2- 172.21.15.44:0/3198099773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5e4101390 0x7fe5e4109270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.249+0000 7fe5ea3ff700 1 --2- 172.21.15.44:0/3198099773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5e4101ca0 0x7fe5e41097b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.249+0000 7fe5ea3ff700 1 -- 172.21.15.44:0/3198099773 >> 172.21.15.44:0/3198099773 conn(0x7fe5e40770b0 msgr2=0x7fe5e40ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:55.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.250+0000 7fe5ea3ff700 1 -- 172.21.15.44:0/3198099773 shutdown_connections 2023-11-16T16:50:55.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.250+0000 7fe5ea3ff700 1 -- 172.21.15.44:0/3198099773 wait complete. 2023-11-16T16:50:55.282 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:50:55.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.632+0000 7f55075ff700 1 -- 172.21.15.44:0/2086598163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55000ffdf0 msgr2=0x7f5500103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.632+0000 7f55075ff700 1 --2- 172.21.15.44:0/2086598163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55000ffdf0 0x7f5500103db0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f54f00099c0 tx=0x7f54f0009cd0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.632+0000 7f55075ff700 1 -- 172.21.15.44:0/2086598163 shutdown_connections 2023-11-16T16:50:55.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.632+0000 7f55075ff700 1 --2- 172.21.15.44:0/2086598163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5500104490 0x7f550010c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.632+0000 7f55075ff700 1 --2- 172.21.15.44:0/2086598163 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55000ff450 0x7f55000ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.632+0000 7f55075ff700 1 --2- 172.21.15.44:0/2086598163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55000ffdf0 0x7f5500103db0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.632+0000 7f55075ff700 1 -- 172.21.15.44:0/2086598163 >> 172.21.15.44:0/2086598163 conn(0x7f55000770b0 msgr2=0x7f55000774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:55.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.633+0000 7f55075ff700 1 -- 172.21.15.44:0/2086598163 shutdown_connections 2023-11-16T16:50:55.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.633+0000 7f55075ff700 1 -- 172.21.15.44:0/2086598163 wait complete. 2023-11-16T16:50:55.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.634+0000 7f55075ff700 1 Processor -- start 2023-11-16T16:50:55.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.635+0000 7f55075ff700 1 -- start start 2023-11-16T16:50:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.635+0000 7f55075ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55000ff450 0x7f5500106100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.635+0000 7f55075ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55000ffdf0 0x7f5500106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.636+0000 7f55075ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5500104490 0x7f5500106b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.636+0000 7f55075ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55001a47e0 con 0x7f5500104490 2023-11-16T16:50:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.636+0000 7f55075ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55001a4950 con 0x7f55000ff450 2023-11-16T16:50:55.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.636+0000 7f55075ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55001a4b10 con 0x7f55000ffdf0 2023-11-16T16:50:55.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.636+0000 7f5504b9a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55000ffdf0 0x7f5500106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.636+0000 7f5504b9a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55000ffdf0 0x7f5500106640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:43810/0 (socket says 172.21.15.44:43810) 2023-11-16T16:50:55.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.636+0000 7f5504b9a700 1 -- 172.21.15.44:0/2101019613 learned_addr learned my addr 172.21.15.44:0/2101019613 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:55.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.636+0000 7f550539b700 1 --2- 172.21.15.44:0/2101019613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55000ff450 0x7f5500106100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.637+0000 7f5505b9c700 1 --2- 172.21.15.44:0/2101019613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5500104490 0x7f5500106b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.637+0000 7f5504b9a700 1 -- 172.21.15.44:0/2101019613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55000ff450 msgr2=0x7f5500106100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.637+0000 7f5504b9a700 1 --2- 172.21.15.44:0/2101019613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55000ff450 0x7f5500106100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.637+0000 7f5504b9a700 1 -- 172.21.15.44:0/2101019613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5500104490 msgr2=0x7f5500106b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.637+0000 7f5504b9a700 1 --2- 172.21.15.44:0/2101019613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5500104490 0x7f5500106b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.637+0000 7f5504b9a700 1 -- 172.21.15.44:0/2101019613 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc009520 con 0x7f55000ffdf0 2023-11-16T16:50:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.637+0000 7f5505b9c700 1 --2- 172.21.15.44:0/2101019613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5500104490 0x7f5500106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:50:55.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.637+0000 7f5504b9a700 1 --2- 172.21.15.44:0/2101019613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55000ffdf0 0x7f5500106640 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f54f00058e0 tx=0x7f54f000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:55.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.637+0000 7f54f67fc700 1 -- 172.21.15.44:0/2101019613 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f001d070 con 0x7f55000ffdf0 2023-11-16T16:50:55.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.638+0000 7f54f67fc700 1 -- 172.21.15.44:0/2101019613 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54f0005050 con 0x7f55000ffdf0 2023-11-16T16:50:55.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.638+0000 7f54f67fc700 1 -- 172.21.15.44:0/2101019613 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f0004ad0 con 0x7f55000ffdf0 2023-11-16T16:50:55.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.638+0000 7f55075ff700 1 -- 172.21.15.44:0/2101019613 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f00095f0 con 0x7f55000ffdf0 2023-11-16T16:50:55.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.638+0000 7f55075ff700 1 -- 172.21.15.44:0/2101019613 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f550010f260 con 0x7f55000ffdf0 2023-11-16T16:50:55.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.643+0000 7f54f67fc700 1 -- 172.21.15.44:0/2101019613 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f54f0005430 con 0x7f55000ffdf0 2023-11-16T16:50:55.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.645+0000 7f551ae4fb80 1 Processor -- start 2023-11-16T16:50:55.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.645+0000 7f551ae4fb80 1 -- start start 2023-11-16T16:50:55.648 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.646+0000 7f551ae4fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664e1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.648 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.646+0000 7f551ae4fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e2470 0x55c1664e6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.648 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.646+0000 7f551ae4fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e68b0 0x55c1664e6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.648 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.646+0000 7f551ae4fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1663e7060 con 0x55c1664e68b0 2023-11-16T16:50:55.648 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.646+0000 7f551ae4fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1664398a0 con 0x55c1664e1b60 2023-11-16T16:50:55.648 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.646+0000 7f551ae4fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c166432d30 con 0x55c1664e2470 2023-11-16T16:50:55.648 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.646+0000 7f54f67fc700 1 --2- 172.21.15.44:0/2101019613 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f54ec0a1460 0x7f54ec0a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.649 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.646+0000 7f54f67fc700 1 -- 172.21.15.44:0/2101019613 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f54f00ce170 con 0x7f55000ffdf0 2023-11-16T16:50:55.649 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.647+0000 7f5505b9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664e1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.649 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.647+0000 7f5504b9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e68b0 0x55c1664e6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.649 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.647+0000 7f5505b9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664e1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:50436/0 (socket says 172.21.15.44:50436) 2023-11-16T16:50:55.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.647+0000 7f5505b9c700 1 -- 172.21.15.44:0/749199358 learned_addr learned my addr 172.21.15.44:0/749199358 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:55.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.647+0000 7f550539b700 1 --2- 172.21.15.44:0/749199358 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e2470 0x55c1664e6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.647+0000 7f5504b9a700 1 -- 172.21.15.44:0/749199358 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e2470 msgr2=0x55c1664e6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:55.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.648+0000 7f5504b9a700 1 --2- 172.21.15.44:0/749199358 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e2470 0x55c1664e6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.648+0000 7f5504b9a700 1 -- 172.21.15.44:0/749199358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 msgr2=0x55c1664e1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:55.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.648+0000 7f5504b9a700 1 --2- 172.21.15.44:0/749199358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664e1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.648+0000 7f5504b9a700 1 -- 172.21.15.44:0/749199358 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c1662b4630 con 0x55c1664e68b0 2023-11-16T16:50:55.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.648+0000 7f5505b9c700 1 --2- 172.21.15.44:0/749199358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664e1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:55.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.648+0000 7f5505b9c700 1 --2- 172.21.15.44:0/2101019613 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f54ec0a1460 0x7f54ec0a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.649+0000 7f5505b9c700 1 --2- 172.21.15.44:0/2101019613 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f54ec0a1460 0x7f54ec0a3910 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f54f800f5d0 tx=0x7f54f8009040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:55.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.649+0000 7f5504b9a700 1 --2- 172.21.15.44:0/749199358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e68b0 0x55c1664e6d00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f54f00cb3a0 tx=0x7f54f009a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d0e4fedd913a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:55.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.650+0000 7f54eb7fe700 1 -- 172.21.15.44:0/749199358 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f00a30a0 con 0x55c1664e68b0 2023-11-16T16:50:55.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.650+0000 7f54eb7fe700 1 -- 172.21.15.44:0/749199358 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54f009d070 con 0x55c1664e68b0 2023-11-16T16:50:55.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.650+0000 7f54eb7fe700 1 -- 172.21.15.44:0/749199358 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f009b180 con 0x55c1664e68b0 2023-11-16T16:50:55.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.651+0000 7f551ae4fb80 1 -- 172.21.15.44:0/749199358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e68b0 msgr2=0x55c1664e6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.651+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/749199358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e68b0 0x55c1664e6d00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f54f00cb3a0 tx=0x7f54f009a950 comp rx=0 tx=0).stop 2023-11-16T16:50:55.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.651+0000 7f551ae4fb80 1 -- 172.21.15.44:0/749199358 shutdown_connections 2023-11-16T16:50:55.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.651+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/749199358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e68b0 0x55c1664e6d00 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.651+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/749199358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664e1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.651+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/749199358 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e2470 0x55c1664e6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.651+0000 7f551ae4fb80 1 -- 172.21.15.44:0/749199358 >> 172.21.15.44:0/749199358 conn(0x55c1664d7360 msgr2=0x55c1664dcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:55.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.652+0000 7f551ae4fb80 1 -- 172.21.15.44:0/749199358 shutdown_connections 2023-11-16T16:50:55.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.652+0000 7f551ae4fb80 1 -- 172.21.15.44:0/749199358 wait complete. 2023-11-16T16:50:55.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.653+0000 7f551ae4fb80 1 -- start start 2023-11-16T16:50:55.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.653+0000 7f551ae4fb80 1 Processor -- start 2023-11-16T16:50:55.656 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.654+0000 7f551ae4fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664ee000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.656 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.654+0000 7f551ae4fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e2470 0x55c1664eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.656 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.654+0000 7f551ae4fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e68b0 0x55c1664eb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.655+0000 7f551ae4fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1663d48e0 con 0x55c1664e2470 2023-11-16T16:50:55.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.655+0000 7f551ae4fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1663d4580 con 0x55c1664e1b60 2023-11-16T16:50:55.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.655+0000 7f551ae4fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1662b6920 con 0x55c1664e68b0 2023-11-16T16:50:55.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.655+0000 7f551ae4fb80 10 client.0 _ll_register_callbacks cb 0x55c1663f2780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:55.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.655+0000 7f550539b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.655+0000 7f550539b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664ee000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:50448/0 (socket says 172.21.15.44:50448) 2023-11-16T16:50:55.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.655+0000 7f550539b700 1 -- 172.21.15.44:0/2224851770 learned_addr learned my addr 172.21.15.44:0/2224851770 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:55.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.655+0000 7f5504b9a700 1 --2- 172.21.15.44:0/2224851770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e2470 0x55c1664eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.655+0000 7f5505b9c700 1 --2- 172.21.15.44:0/2224851770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e68b0 0x55c1664eb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.658 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.655+0000 7f550539b700 1 -- 172.21.15.44:0/2224851770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e68b0 msgr2=0x55c1664eb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:55.658 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.655+0000 7f550539b700 1 --2- 172.21.15.44:0/2224851770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e68b0 0x55c1664eb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.658 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.655+0000 7f550539b700 1 -- 172.21.15.44:0/2224851770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e2470 msgr2=0x55c1664eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:55.658 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.655+0000 7f550539b700 1 --2- 172.21.15.44:0/2224851770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e2470 0x55c1664eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.655+0000 7f550539b700 1 -- 172.21.15.44:0/2224851770 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f550010f260 con 0x55c1664e1b60 2023-11-16T16:50:55.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.656+0000 7f550539b700 1 --2- 172.21.15.44:0/2224851770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664ee000 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f54fc0129e0 tx=0x7f54fc012cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84835f5587c41f7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:55.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.656+0000 7f54e8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:50:55.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.656+0000 7f54e8ff9700 1 -- 172.21.15.44:0/2224851770 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54fc017070 con 0x55c1664e1b60 2023-11-16T16:50:55.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.656+0000 7f551ae4fb80 1 -- 172.21.15.44:0/2224851770 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c1664d4b20 con 0x55c1664e1b60 2023-11-16T16:50:55.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.656+0000 7f54e8ff9700 1 -- 172.21.15.44:0/2224851770 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f54fc00f830 con 0x55c1664e1b60 2023-11-16T16:50:55.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.657+0000 7f54e8ff9700 1 -- 172.21.15.44:0/2224851770 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54fc0122e0 con 0x55c1664e1b60 2023-11-16T16:50:55.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.657+0000 7f551ae4fb80 10 client.25793 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:55.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.657+0000 7f551ae4fb80 1 -- 172.21.15.44:0/2224851770 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55c1663f9190 con 0x55c1664e1b60 2023-11-16T16:50:55.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.657+0000 7f54e8ff9700 1 -- 172.21.15.44:0/2224851770 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c1664d4b20 con 0x55c1664e1b60 2023-11-16T16:50:55.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.657+0000 7f54e8ff9700 1 -- 172.21.15.44:0/2224851770 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f54fc010c40 con 0x55c1664e1b60 2023-11-16T16:50:55.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.658+0000 7f551ae4fb80 10 client.25793 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:55.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.658+0000 7f551ae4fb80 10 client.25793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:55.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.658+0000 7f551ae4fb80 20 client.25793 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:55.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.658+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/2224851770 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c1664fb620 0x55c1664fda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.658+0000 7f551ae4fb80 4 client.25793 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:55.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.658+0000 7f551ae4fb80 1 -- 172.21.15.44:0/2224851770 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c1663f50a0 con 0x55c1664fb620 2023-11-16T16:50:55.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.658+0000 7f5504b9a700 1 --2- 172.21.15.44:0/2224851770 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c1664fb620 0x55c1664fda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.659+0000 7f5504b9a700 1 --2- 172.21.15.44:0/2224851770 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c1664fb620 0x55c1664fda30 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c495b2f0edda66cd server_cookie=d060259cef0d9c9d in_seq=0 out_seq=0 2023-11-16T16:50:55.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.659+0000 7f54e8ff9700 10 client.25793 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:55.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.659+0000 7f54e8ff9700 1 -- 172.21.15.44:0/2224851770 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55c1663f50a0 con 0x55c1664fb620 2023-11-16T16:50:55.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.659+0000 7f54e8ff9700 10 client.25793 handle_command_reply: tid=0 2023-11-16T16:50:55.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.659+0000 7f551ae4fb80 1 client.25793 shutdown 2023-11-16T16:50:55.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.660+0000 7f551ae4fb80 1 -- 172.21.15.44:0/2224851770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 msgr2=0x55c1664ee000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.660+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/2224851770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664ee000 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f54fc0129e0 tx=0x7f54fc012cf0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.660+0000 7f551ae4fb80 1 -- 172.21.15.44:0/2224851770 shutdown_connections 2023-11-16T16:50:55.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.660+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/2224851770 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c1664fb620 0x55c1664fda30 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.660+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/2224851770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e68b0 0x55c1664eb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.660+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/2224851770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664ee000 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.660+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/2224851770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e2470 0x55c1664eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.660+0000 7f551ae4fb80 1 -- 172.21.15.44:0/2224851770 >> 172.21.15.44:0/2224851770 conn(0x55c1664d7360 msgr2=0x55c1664e7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:55.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.660+0000 7f54e8ff9700 0 client.25793 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:55.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.660+0000 7f551ae4fb80 1 -- 172.21.15.44:0/2224851770 shutdown_connections 2023-11-16T16:50:55.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.660+0000 7f551ae4fb80 1 -- 172.21.15.44:0/2224851770 wait complete. 2023-11-16T16:50:55.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.661+0000 7f551ae4fb80 20 client.25793 trim_cache size 0 max 16384 2023-11-16T16:50:55.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.695+0000 7f551ae4fb80 1 Processor -- start 2023-11-16T16:50:55.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.695+0000 7f551ae4fb80 1 -- start start 2023-11-16T16:50:55.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.695+0000 7f551ae4fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664eb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.695+0000 7f551ae4fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e2470 0x55c1664dcca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.695+0000 7f551ae4fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e68b0 0x55c1664f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.695+0000 7f551ae4fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1664e2a00 con 0x55c1664e2470 2023-11-16T16:50:55.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.695+0000 7f551ae4fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1664e2bf0 con 0x55c1664e1b60 2023-11-16T16:50:55.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.695+0000 7f551ae4fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1664ebe10 con 0x55c1664e68b0 2023-11-16T16:50:55.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.696+0000 7f550539b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664eb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.696+0000 7f550539b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664eb1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:50452/0 (socket says 172.21.15.44:50452) 2023-11-16T16:50:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.696+0000 7f550539b700 1 -- 172.21.15.44:0/4237861483 learned_addr learned my addr 172.21.15.44:0/4237861483 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.696+0000 7f5505b9c700 1 --2- 172.21.15.44:0/4237861483 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e68b0 0x55c1664f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.696+0000 7f5504b9a700 1 --2- 172.21.15.44:0/4237861483 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e2470 0x55c1664dcca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.696+0000 7f550539b700 1 -- 172.21.15.44:0/4237861483 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e68b0 msgr2=0x55c1664f27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.696+0000 7f550539b700 1 --2- 172.21.15.44:0/4237861483 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e68b0 0x55c1664f27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.696+0000 7f550539b700 1 -- 172.21.15.44:0/4237861483 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e2470 msgr2=0x55c1664dcca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.696+0000 7f550539b700 1 --2- 172.21.15.44:0/4237861483 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e2470 0x55c1664dcca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.696+0000 7f550539b700 1 -- 172.21.15.44:0/4237861483 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c1663f8ee0 con 0x55c1664e1b60 2023-11-16T16:50:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.697+0000 7f5504b9a700 1 --2- 172.21.15.44:0/4237861483 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e2470 0x55c1664dcca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:50:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.697+0000 7f550539b700 1 --2- 172.21.15.44:0/4237861483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664eb1c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f54fc001eb0 tx=0x7f54fc012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95ee741f5c5c14f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.697+0000 7f54ebfff700 1 -- 172.21.15.44:0/4237861483 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54fc017070 con 0x55c1664e1b60 2023-11-16T16:50:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.697+0000 7f54ebfff700 1 -- 172.21.15.44:0/4237861483 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54fc018040 con 0x55c1664e1b60 2023-11-16T16:50:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.697+0000 7f54ebfff700 1 -- 172.21.15.44:0/4237861483 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54fc0054e0 con 0x55c1664e1b60 2023-11-16T16:50:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.698+0000 7f551ae4fb80 1 -- 172.21.15.44:0/4237861483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 msgr2=0x55c1664eb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.698+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/4237861483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664eb1c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f54fc001eb0 tx=0x7f54fc012000 comp rx=0 tx=0).stop 2023-11-16T16:50:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.698+0000 7f551ae4fb80 1 -- 172.21.15.44:0/4237861483 shutdown_connections 2023-11-16T16:50:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.698+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/4237861483 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e68b0 0x55c1664f27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.698+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/4237861483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664eb1c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.698+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/4237861483 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e2470 0x55c1664dcca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.698+0000 7f551ae4fb80 1 -- 172.21.15.44:0/4237861483 >> 172.21.15.44:0/4237861483 conn(0x55c1664d7360 msgr2=0x55c1664eaab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:55.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.699+0000 7f551ae4fb80 1 -- 172.21.15.44:0/4237861483 shutdown_connections 2023-11-16T16:50:55.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.699+0000 7f551ae4fb80 1 -- 172.21.15.44:0/4237861483 wait complete. 2023-11-16T16:50:55.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.700+0000 7f551ae4fb80 1 -- start start 2023-11-16T16:50:55.702 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.700+0000 7f551ae4fb80 1 Processor -- start 2023-11-16T16:50:55.703 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.701+0000 7f551ae4fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664f0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.703 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.701+0000 7f551ae4fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e2470 0x55c1664f0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.703 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.701+0000 7f551ae4fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e68b0 0x55c1664f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.703 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.701+0000 7f551ae4fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c16654e5d0 con 0x55c1664e68b0 2023-11-16T16:50:55.703 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.701+0000 7f551ae4fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c16654e740 con 0x55c1664e1b60 2023-11-16T16:50:55.703 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.701+0000 7f551ae4fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c16654e900 con 0x55c1664e2470 2023-11-16T16:50:55.703 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.701+0000 7f551ae4fb80 10 client.0 _ll_register_callbacks cb 0x55c1663f2780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:55.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.702+0000 7f5505b9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e68b0 0x55c1664f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.702+0000 7f5505b9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e68b0 0x55c1664f11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39908/0 (socket says 172.21.15.44:39908) 2023-11-16T16:50:55.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.702+0000 7f5505b9c700 1 -- 172.21.15.44:0/3955321241 learned_addr learned my addr 172.21.15.44:0/3955321241 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:50:55.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.703+0000 7f5504b9a700 1 --2- 172.21.15.44:0/3955321241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e2470 0x55c1664f0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.703+0000 7f550539b700 1 --2- 172.21.15.44:0/3955321241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664f0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.703+0000 7f5505b9c700 1 -- 172.21.15.44:0/3955321241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e2470 msgr2=0x55c1664f0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:55.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.703+0000 7f5505b9c700 1 --2- 172.21.15.44:0/3955321241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e2470 0x55c1664f0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.704+0000 7f5505b9c700 1 -- 172.21.15.44:0/3955321241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 msgr2=0x55c1664f0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:55.706 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.704+0000 7f5505b9c700 1 --2- 172.21.15.44:0/3955321241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664f0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.706 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.704+0000 7f5505b9c700 1 -- 172.21.15.44:0/3955321241 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c1663f9190 con 0x55c1664e68b0 2023-11-16T16:50:55.706 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.704+0000 7f5505b9c700 1 --2- 172.21.15.44:0/3955321241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e68b0 0x55c1664f11c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f54f8015c20 tx=0x7f54f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c0af18774d8731 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:55.706 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.704+0000 7f54e8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:50:55.707 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.706+0000 7f54e8ff9700 1 -- 172.21.15.44:0/3955321241 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f80168a0 con 0x55c1664e68b0 2023-11-16T16:50:55.707 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.706+0000 7f551ae4fb80 1 -- 172.21.15.44:0/3955321241 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f54f009d070 con 0x55c1664e68b0 2023-11-16T16:50:55.708 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.706+0000 7f54e8ff9700 1 -- 172.21.15.44:0/3955321241 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f54f8016a00 con 0x55c1664e68b0 2023-11-16T16:50:55.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.706+0000 7f54e8ff9700 1 -- 172.21.15.44:0/3955321241 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54f8016bc0 con 0x55c1664e68b0 2023-11-16T16:50:55.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.706+0000 7f551ae4fb80 10 client.15786 fetch_fsmap learned FSMap version 21 2023-11-16T16:50:55.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.706+0000 7f551ae4fb80 1 -- 172.21.15.44:0/3955321241 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55c166249c60 con 0x55c1664e68b0 2023-11-16T16:50:55.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.707+0000 7f54e8ff9700 1 -- 172.21.15.44:0/3955321241 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54f009d070 con 0x55c1664e68b0 2023-11-16T16:50:55.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.707+0000 7f54e8ff9700 1 -- 172.21.15.44:0/3955321241 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f54f80143c0 con 0x55c1664e68b0 2023-11-16T16:50:55.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.707+0000 7f551ae4fb80 10 client.15786 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:50:55.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.707+0000 7f551ae4fb80 10 client.15786 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:55.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.707+0000 7f551ae4fb80 20 client.15786 populate_metadata read hostname 'smithi044' 2023-11-16T16:50:55.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.707+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/3955321241 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c166550090 0x55c1665524a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:55.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.707+0000 7f551ae4fb80 4 client.15786 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:50:55.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.707+0000 7f551ae4fb80 1 -- 172.21.15.44:0/3955321241 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c166401660 con 0x55c166550090 2023-11-16T16:50:55.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.708+0000 7f550539b700 1 --2- 172.21.15.44:0/3955321241 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c166550090 0x55c1665524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:55.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.708+0000 7f550539b700 1 --2- 172.21.15.44:0/3955321241 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c166550090 0x55c1665524a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2376f2259e4bd704 server_cookie=bf09dd5f4524ad41 in_seq=0 out_seq=0 2023-11-16T16:50:55.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.708+0000 7f54e8ff9700 10 client.15786 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:55.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.709+0000 7f54e8ff9700 1 -- 172.21.15.44:0/3955321241 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c166401660 con 0x55c166550090 2023-11-16T16:50:55.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.709+0000 7f54e8ff9700 10 client.15786 handle_command_reply: tid=0 2023-11-16T16:50:55.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.709+0000 7f551ae4fb80 1 client.15786 shutdown 2023-11-16T16:50:55.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.709+0000 7f551ae4fb80 1 -- 172.21.15.44:0/3955321241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e68b0 msgr2=0x55c1664f11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.709+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/3955321241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e68b0 0x55c1664f11c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f54f8015c20 tx=0x7f54f8004ad0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.709+0000 7f551ae4fb80 1 -- 172.21.15.44:0/3955321241 shutdown_connections 2023-11-16T16:50:55.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.709+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/3955321241 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c166550090 0x55c1665524a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.709+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/3955321241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1664e68b0 0x55c1664f11c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.709+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/3955321241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1664e1b60 0x55c1664f0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.709+0000 7f54e8ff9700 0 client.15786 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:50:55.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.709+0000 7f551ae4fb80 1 --2- 172.21.15.44:0/3955321241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1664e2470 0x55c1664f0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.709+0000 7f551ae4fb80 1 -- 172.21.15.44:0/3955321241 >> 172.21.15.44:0/3955321241 conn(0x55c1664d7360 msgr2=0x55c1664e99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:55.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.710+0000 7f551ae4fb80 1 -- 172.21.15.44:0/3955321241 shutdown_connections 2023-11-16T16:50:55.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.710+0000 7f551ae4fb80 1 -- 172.21.15.44:0/3955321241 wait complete. 2023-11-16T16:50:55.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.710+0000 7f551ae4fb80 20 client.15786 trim_cache size 0 max 16384 2023-11-16T16:50:55.713 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:50:55.713 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2023-11-16T16:50:55.713 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2023-11-16T16:50:55.713 INFO:teuthology.orchestra.run.smithi044.stdout: "452d3c6b-e145-4fef-8021-c3ec192fbe5b": { 2023-11-16T16:50:55.713 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2023-11-16T16:50:55.713 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "452d3c6b-e145-4fef-8021-c3ec192fbe5b", 2023-11-16T16:50:55.713 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2023-11-16T16:50:55.713 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:50:55.713 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:50:55.713 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:50:55.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.711+0000 7f55075ff700 1 -- 172.21.15.44:0/2101019613 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f54ec0a1460 msgr2=0x7f54ec0a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.711+0000 7f55075ff700 1 --2- 172.21.15.44:0/2101019613 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f54ec0a1460 0x7f54ec0a3910 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f54f800f5d0 tx=0x7f54f8009040 comp rx=0 tx=0).stop 2023-11-16T16:50:55.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.711+0000 7f55075ff700 1 -- 172.21.15.44:0/2101019613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55000ffdf0 msgr2=0x7f5500106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:55.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.711+0000 7f55075ff700 1 --2- 172.21.15.44:0/2101019613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55000ffdf0 0x7f5500106640 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f54f00058e0 tx=0x7f54f000f690 comp rx=0 tx=0).stop 2023-11-16T16:50:55.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.711+0000 7f55075ff700 1 -- 172.21.15.44:0/2101019613 shutdown_connections 2023-11-16T16:50:55.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.711+0000 7f55075ff700 1 --2- 172.21.15.44:0/2101019613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5500104490 0x7f5500106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.711+0000 7f55075ff700 1 --2- 172.21.15.44:0/2101019613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55000ff450 0x7f5500106100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.711+0000 7f55075ff700 1 --2- 172.21.15.44:0/2101019613 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f54ec0a1460 0x7f54ec0a3910 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.711+0000 7f55075ff700 1 --2- 172.21.15.44:0/2101019613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55000ffdf0 0x7f5500106640 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:55.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.711+0000 7f55075ff700 1 -- 172.21.15.44:0/2101019613 >> 172.21.15.44:0/2101019613 conn(0x7f55000770b0 msgr2=0x7f5500102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:55.714 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.712+0000 7f55075ff700 1 -- 172.21.15.44:0/2101019613 shutdown_connections 2023-11-16T16:50:55.715 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:50:55.712+0000 7f55075ff700 1 -- 172.21.15.44:0/2101019613 wait complete. 2023-11-16T16:50:55.742 INFO:tasks.cephfs.filesystem:scrub status for tag:452d3c6b-e145-4fef-8021-c3ec192fbe5b - {'path': '/', 'tag': '452d3c6b-e145-4fef-8021-c3ec192fbe5b', 'options': 'recursive,force'} 2023-11-16T16:50:56.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:56 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:50:56.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:56 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:56.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:56 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:50:56.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:56 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:56.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:56 smithi077 ceph-mon[131204]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:50:57.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:56 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:50:57.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:56 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:57.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:56 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:50:57.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:56 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:57.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:56 smithi044 ceph-mon[128750]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:50:57.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:56 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:50:57.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:56 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:57.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:56 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:50:57.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:56 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:50:57.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:56 smithi176 ceph-mon[131630]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2023-11-16T16:50:57.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:57 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:50:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:57 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:50:58.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:57 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:50:58.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:50:58 smithi077 ceph-mon[131204]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-11-16T16:50:59.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:50:58 smithi044 ceph-mon[128750]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-11-16T16:50:59.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:50:58 smithi176 ceph-mon[131630]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-11-16T16:51:00.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:00 smithi077 ceph-mon[131204]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 4 op/s 2023-11-16T16:51:01.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:00 smithi044 ceph-mon[128750]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 4 op/s 2023-11-16T16:51:01.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:00 smithi176 ceph-mon[131630]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 4 op/s 2023-11-16T16:51:02.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:02 smithi077 ceph-mon[131204]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2023-11-16T16:51:03.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:02 smithi044 ceph-mon[128750]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2023-11-16T16:51:03.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:02 smithi176 ceph-mon[131630]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2023-11-16T16:51:04.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:04 smithi077 ceph-mon[131204]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 16 op/s 2023-11-16T16:51:04.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:04 smithi044 ceph-mon[128750]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 16 op/s 2023-11-16T16:51:04.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:04 smithi176 ceph-mon[131630]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 16 op/s 2023-11-16T16:51:04.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.719+0000 7f7ed79ff700 1 -- 172.21.15.44:0/1160653489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ed0105560 msgr2=0x7f7ed010d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:04.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.719+0000 7f7ed79ff700 1 --2- 172.21.15.44:0/1160653489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ed0105560 0x7f7ed010d170 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009fc0 tx=0x7f7ec800b330 comp rx=0 tx=0).stop 2023-11-16T16:51:04.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.719+0000 7f7ed79ff700 1 -- 172.21.15.44:0/1160653489 shutdown_connections 2023-11-16T16:51:04.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.720+0000 7f7ed79ff700 1 --2- 172.21.15.44:0/1160653489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ed0105560 0x7f7ed010d170 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.720+0000 7f7ed79ff700 1 --2- 172.21.15.44:0/1160653489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed0100c50 0x7f7ed0101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.720+0000 7f7ed79ff700 1 --2- 172.21.15.44:0/1160653489 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ed01015f0 0x7f7ed0104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.720+0000 7f7ed79ff700 1 -- 172.21.15.44:0/1160653489 >> 172.21.15.44:0/1160653489 conn(0x7f7ed0077550 msgr2=0x7f7ed0077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:04.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.720+0000 7f7ed79ff700 1 -- 172.21.15.44:0/1160653489 shutdown_connections 2023-11-16T16:51:04.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.720+0000 7f7ed79ff700 1 -- 172.21.15.44:0/1160653489 wait complete. 2023-11-16T16:51:04.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.722+0000 7f7ed79ff700 1 Processor -- start 2023-11-16T16:51:04.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.723+0000 7f7ed79ff700 1 -- start start 2023-11-16T16:51:04.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.724+0000 7f7ed79ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ed0100c50 0x7f7ed01ab0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.724+0000 7f7ed79ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ed01015f0 0x7f7ed01ab620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.724+0000 7f7ed79ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed0105560 0x7f7ed010bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.725+0000 7f7ed579b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ed0100c50 0x7f7ed01ab0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:04.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.725+0000 7f7ed579b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ed0100c50 0x7f7ed01ab0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39916/0 (socket says 172.21.15.44:39916) 2023-11-16T16:51:04.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.725+0000 7f7ed579b700 1 -- 172.21.15.44:0/3628743281 learned_addr learned my addr 172.21.15.44:0/3628743281 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:04.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.725+0000 7f7ed79ff700 1 -- 172.21.15.44:0/3628743281 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed010c5d0 con 0x7f7ed0100c50 2023-11-16T16:51:04.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.725+0000 7f7ed79ff700 1 -- 172.21.15.44:0/3628743281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed010c740 con 0x7f7ed0105560 2023-11-16T16:51:04.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.725+0000 7f7ed79ff700 1 -- 172.21.15.44:0/3628743281 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed010c900 con 0x7f7ed01015f0 2023-11-16T16:51:04.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.725+0000 7f7ed5f9c700 1 --2- 172.21.15.44:0/3628743281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed0105560 0x7f7ed010bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:04.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.725+0000 7f7ed4f9a700 1 --2- 172.21.15.44:0/3628743281 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ed01015f0 0x7f7ed01ab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:04.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.726+0000 7f7ed5f9c700 1 -- 172.21.15.44:0/3628743281 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ed01015f0 msgr2=0x7f7ed01ab620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:04.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.726+0000 7f7ed5f9c700 1 --2- 172.21.15.44:0/3628743281 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ed01015f0 0x7f7ed01ab620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.726+0000 7f7ed5f9c700 1 -- 172.21.15.44:0/3628743281 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ed0100c50 msgr2=0x7f7ed01ab0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:04.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.726+0000 7f7ed5f9c700 1 --2- 172.21.15.44:0/3628743281 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ed0100c50 0x7f7ed01ab0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.726+0000 7f7ed5f9c700 1 -- 172.21.15.44:0/3628743281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec8009c40 con 0x7f7ed0105560 2023-11-16T16:51:04.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.726+0000 7f7ed579b700 1 --2- 172.21.15.44:0/3628743281 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ed0100c50 0x7f7ed01ab0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:51:04.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.726+0000 7f7ed5f9c700 1 --2- 172.21.15.44:0/3628743281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed0105560 0x7f7ed010bef0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8015000 tx=0x7f7ec80093b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.727+0000 7f7ec67fc700 1 -- 172.21.15.44:0/3628743281 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec800e040 con 0x7f7ed0105560 2023-11-16T16:51:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.727+0000 7f7ec67fc700 1 -- 172.21.15.44:0/3628743281 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ec801d470 con 0x7f7ed0105560 2023-11-16T16:51:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.727+0000 7f7ec67fc700 1 -- 172.21.15.44:0/3628743281 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec80052a0 con 0x7f7ed0105560 2023-11-16T16:51:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.727+0000 7f7ed79ff700 1 -- 172.21.15.44:0/3628743281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed01103d0 con 0x7f7ed0105560 2023-11-16T16:51:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.727+0000 7f7ed79ff700 1 -- 172.21.15.44:0/3628743281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed0110240 con 0x7f7ed0105560 2023-11-16T16:51:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.732+0000 7f7ec67fc700 1 -- 172.21.15.44:0/3628743281 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f7ec801d5e0 con 0x7f7ed0105560 2023-11-16T16:51:04.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.734+0000 7f7eeb2f1b80 1 Processor -- start 2023-11-16T16:51:04.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.734+0000 7f7eeb2f1b80 1 -- start start 2023-11-16T16:51:04.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.735+0000 7f7eeb2f1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c7736b0 0x55ca2c91d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.735+0000 7f7eeb2f1b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c91d720 0x55ca2c917560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.735+0000 7f7eeb2f1b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c91db90 0x55ca2c917b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.735+0000 7f7eeb2f1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2c81d000 con 0x55ca2c7736b0 2023-11-16T16:51:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.735+0000 7f7eeb2f1b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2c86f840 con 0x55ca2c91db90 2023-11-16T16:51:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.735+0000 7f7eeb2f1b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2c868cd0 con 0x55ca2c91d720 2023-11-16T16:51:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.736+0000 7f7ed4f9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c7736b0 0x55ca2c91d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.736+0000 7f7ed579b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c91d720 0x55ca2c917560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.736+0000 7f7ed579b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c91d720 0x55ca2c917560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:43880/0 (socket says 172.21.15.44:43880) 2023-11-16T16:51:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.736+0000 7f7ed579b700 1 -- 172.21.15.44:0/2696714646 learned_addr learned my addr 172.21.15.44:0/2696714646 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:04.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.736+0000 7f7ec67fc700 1 --2- 172.21.15.44:0/3628743281 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7ebc0a1460 0x7f7ebc0a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.736+0000 7f7ec67fc700 1 -- 172.21.15.44:0/3628743281 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f7ec80ce6a0 con 0x7f7ed0105560 2023-11-16T16:51:04.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.736+0000 7f7ed5f9c700 1 --2- 172.21.15.44:0/3628743281 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7ebc0a1460 0x7f7ebc0a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:04.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.737+0000 7f7ed4f9a700 1 --2- 172.21.15.44:0/2696714646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c91db90 0x55ca2c917b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:04.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.737+0000 7f7ed5f9c700 1 --2- 172.21.15.44:0/3628743281 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7ebc0a1460 0x7f7ebc0a3910 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80cb8f0 tx=0x7f7ec8009920 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:04.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.738+0000 7f7ed4f9a700 1 -- 172.21.15.44:0/2696714646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c91d720 msgr2=0x55ca2c917560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:04.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.738+0000 7f7ed4f9a700 1 --2- 172.21.15.44:0/2696714646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c91d720 0x55ca2c917560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.738+0000 7f7ed4f9a700 1 -- 172.21.15.44:0/2696714646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c91db90 msgr2=0x55ca2c917b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:04.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.738+0000 7f7ed4f9a700 1 --2- 172.21.15.44:0/2696714646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c91db90 0x55ca2c917b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.738+0000 7f7ed4f9a700 1 -- 172.21.15.44:0/2696714646 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca2c6ea5d0 con 0x55ca2c7736b0 2023-11-16T16:51:04.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.739+0000 7f7ed4f9a700 1 --2- 172.21.15.44:0/2696714646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c7736b0 0x55ca2c91d1e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7ed006a260 tx=0x7f7ec0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb840ee2c7cdc78 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:04.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.739+0000 7f7ebb7fe700 1 -- 172.21.15.44:0/2696714646 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec00132d0 con 0x55ca2c7736b0 2023-11-16T16:51:04.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.739+0000 7f7ebb7fe700 1 -- 172.21.15.44:0/2696714646 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ec0011900 con 0x55ca2c7736b0 2023-11-16T16:51:04.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.739+0000 7f7ebb7fe700 1 -- 172.21.15.44:0/2696714646 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec00108f0 con 0x55ca2c7736b0 2023-11-16T16:51:04.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.740+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/2696714646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c7736b0 msgr2=0x55ca2c91d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:04.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.740+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/2696714646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c7736b0 0x55ca2c91d1e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7ed006a260 tx=0x7f7ec0009260 comp rx=0 tx=0).stop 2023-11-16T16:51:04.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.741+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/2696714646 shutdown_connections 2023-11-16T16:51:04.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.741+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/2696714646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c91db90 0x55ca2c917b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.741+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/2696714646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c7736b0 0x55ca2c91d1e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.741+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/2696714646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c91d720 0x55ca2c917560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.741+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/2696714646 >> 172.21.15.44:0/2696714646 conn(0x55ca2c90cf50 msgr2=0x55ca2c912270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:04.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.741+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/2696714646 shutdown_connections 2023-11-16T16:51:04.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.741+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/2696714646 wait complete. 2023-11-16T16:51:04.743 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.742+0000 7f7eeb2f1b80 1 -- start start 2023-11-16T16:51:04.743 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.743+0000 7f7eeb2f1b80 1 Processor -- start 2023-11-16T16:51:04.744 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.744+0000 7f7eeb2f1b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c7736b0 0x55ca2c926aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.744 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.744+0000 7f7eeb2f1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c91d720 0x55ca2c926fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.744 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.744+0000 7f7eeb2f1b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c91db90 0x55ca2c920d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.744+0000 7f7ed4f9a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c7736b0 0x55ca2c926aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:04.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.744+0000 7f7ed4f9a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c7736b0 0x55ca2c926aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:43884/0 (socket says 172.21.15.44:43884) 2023-11-16T16:51:04.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.744+0000 7f7eeb2f1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2c80a880 con 0x55ca2c91d720 2023-11-16T16:51:04.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.744+0000 7f7ed579b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c91d720 0x55ca2c926fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:04.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.744+0000 7f7ed4f9a700 1 -- 172.21.15.44:0/3633711998 learned_addr learned my addr 172.21.15.44:0/3633711998 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:04.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.745+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/3633711998 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2c80a520 con 0x55ca2c91db90 2023-11-16T16:51:04.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.745+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/3633711998 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2c6ec8c0 con 0x55ca2c7736b0 2023-11-16T16:51:04.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.745+0000 7f7eeb2f1b80 10 client.0 _ll_register_callbacks cb 0x55ca2c828720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:04.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.745+0000 7f7ed4f9a700 1 --2- 172.21.15.44:0/3633711998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c91db90 0x55ca2c920d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:04.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.745+0000 7f7ed579b700 1 -- 172.21.15.44:0/3633711998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c7736b0 msgr2=0x55ca2c926aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:04.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.745+0000 7f7ed579b700 1 --2- 172.21.15.44:0/3633711998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c7736b0 0x55ca2c926aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.745+0000 7f7ed579b700 1 -- 172.21.15.44:0/3633711998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c91db90 msgr2=0x55ca2c920d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:04.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.745+0000 7f7ed579b700 1 --2- 172.21.15.44:0/3633711998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c91db90 0x55ca2c920d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.745+0000 7f7ed579b700 1 -- 172.21.15.44:0/3633711998 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec0009dd0 con 0x55ca2c91d720 2023-11-16T16:51:04.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.746+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/3633711998 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ca2c90a710 con 0x55ca2c91d720 2023-11-16T16:51:04.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.746+0000 7f7ed579b700 1 --2- 172.21.15.44:0/3633711998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c91d720 0x55ca2c926fe0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc013770 tx=0x7f7ecc013a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e3c3e0a1154828 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:04.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.746+0000 7f7eb8ff9700 10 client.15798 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:51:04.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.747+0000 7f7eb8ff9700 1 -- 172.21.15.44:0/3633711998 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc004460 con 0x55ca2c91d720 2023-11-16T16:51:04.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.747+0000 7f7eb8ff9700 1 -- 172.21.15.44:0/3633711998 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f7ecc003680 con 0x55ca2c91d720 2023-11-16T16:51:04.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.747+0000 7f7eb8ff9700 1 -- 172.21.15.44:0/3633711998 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ecc004460 con 0x55ca2c91d720 2023-11-16T16:51:04.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.747+0000 7f7eeb2f1b80 10 client.15798 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:04.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.747+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/3633711998 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=21}) v3 -- 0x55ca2c82f130 con 0x55ca2c91d720 2023-11-16T16:51:04.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.748+0000 7f7eb8ff9700 1 -- 172.21.15.44:0/3633711998 <== mon.0 v2:172.21.15.44:3300/0 4 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7ecc003840 con 0x55ca2c91d720 2023-11-16T16:51:04.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.748+0000 7f7eeb2f1b80 10 client.15798 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:04.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.748+0000 7f7eeb2f1b80 10 client.15798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:04.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.748+0000 7f7eeb2f1b80 20 client.15798 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:04.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.748+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/3633711998 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ca2c930d80 0x55ca2c933190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.748+0000 7f7eeb2f1b80 4 client.15798 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:51:04.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.748+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/3633711998 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ca2c82b040 con 0x55ca2c930d80 2023-11-16T16:51:04.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.748+0000 7f7ed4f9a700 1 --2- 172.21.15.44:0/3633711998 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ca2c930d80 0x55ca2c933190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:04.750 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.749+0000 7f7eb8ff9700 10 client.15798 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:04.750 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.749+0000 7f7ed4f9a700 1 --2- 172.21.15.44:0/3633711998 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ca2c930d80 0x55ca2c933190 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3eb5e2c3b39e9c48 server_cookie=9a0e3ee3d24f4379 in_seq=0 out_seq=0 2023-11-16T16:51:04.750 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.750+0000 7f7eb8ff9700 1 -- 172.21.15.44:0/3633711998 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ca2c82b040 con 0x55ca2c930d80 2023-11-16T16:51:04.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.750+0000 7f7eb8ff9700 10 client.15798 handle_command_reply: tid=0 2023-11-16T16:51:04.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.750+0000 7f7eeb2f1b80 1 client.15798 shutdown 2023-11-16T16:51:04.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.750+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/3633711998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c91d720 msgr2=0x55ca2c926fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:04.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.750+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/3633711998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c91d720 0x55ca2c926fe0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc013770 tx=0x7f7ecc013a80 comp rx=0 tx=0).stop 2023-11-16T16:51:04.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.750+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/3633711998 shutdown_connections 2023-11-16T16:51:04.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.750+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/3633711998 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ca2c930d80 0x55ca2c933190 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.750+0000 7f7eb8ff9700 0 client.15798 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:04.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.750+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/3633711998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c91db90 0x55ca2c920d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.750+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/3633711998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c7736b0 0x55ca2c926aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.750+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/3633711998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c91d720 0x55ca2c926fe0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.750+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/3633711998 >> 172.21.15.44:0/3633711998 conn(0x55ca2c90cf50 msgr2=0x55ca2c9117e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:04.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.751+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/3633711998 shutdown_connections 2023-11-16T16:51:04.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.751+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/3633711998 wait complete. 2023-11-16T16:51:04.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.751+0000 7f7eeb2f1b80 20 client.15798 trim_cache size 0 max 16384 2023-11-16T16:51:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.781+0000 7f7eeb2f1b80 1 Processor -- start 2023-11-16T16:51:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.782+0000 7f7eeb2f1b80 1 -- start start 2023-11-16T16:51:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.782+0000 7f7eeb2f1b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c7736b0 0x55ca2c920ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.782+0000 7f7eeb2f1b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c91d720 0x55ca2c911f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.782+0000 7f7eeb2f1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c91db90 0x55ca2c919c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.782+0000 7f7eeb2f1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2c90a320 con 0x55ca2c91db90 2023-11-16T16:51:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.782+0000 7f7eeb2f1b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2c90efd0 con 0x55ca2c7736b0 2023-11-16T16:51:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.782+0000 7f7eeb2f1b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2c913c50 con 0x55ca2c91d720 2023-11-16T16:51:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.783+0000 7f7ed4f9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c7736b0 0x55ca2c920ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.783+0000 7f7ed4f9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c7736b0 0x55ca2c920ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:50510/0 (socket says 172.21.15.44:50510) 2023-11-16T16:51:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.783+0000 7f7ed4f9a700 1 -- 172.21.15.44:0/3424903173 learned_addr learned my addr 172.21.15.44:0/3424903173 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.783+0000 7f7ed579b700 1 --2- 172.21.15.44:0/3424903173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c91d720 0x55ca2c911f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.783+0000 7f7ed4f9a700 1 --2- 172.21.15.44:0/3424903173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c91db90 0x55ca2c919c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.783+0000 7f7ed579b700 1 -- 172.21.15.44:0/3424903173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c7736b0 msgr2=0x55ca2c920ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.783+0000 7f7ed579b700 1 --2- 172.21.15.44:0/3424903173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c7736b0 0x55ca2c920ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.783+0000 7f7ed579b700 1 -- 172.21.15.44:0/3424903173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c91db90 msgr2=0x55ca2c919c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.783+0000 7f7ed579b700 1 --2- 172.21.15.44:0/3424903173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c91db90 0x55ca2c919c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.783+0000 7f7ed579b700 1 -- 172.21.15.44:0/3424903173 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca2c90e6f0 con 0x55ca2c91d720 2023-11-16T16:51:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.784+0000 7f7ed579b700 1 --2- 172.21.15.44:0/3424903173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c91d720 0x55ca2c911f60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc000c00 tx=0x7f7ecc016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33c06483690d10e5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.784+0000 7f7ebbfff700 1 -- 172.21.15.44:0/3424903173 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc003d50 con 0x55ca2c91d720 2023-11-16T16:51:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.784+0000 7f7ebbfff700 1 -- 172.21.15.44:0/3424903173 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ca2c90a710 con 0x55ca2c91d720 2023-11-16T16:51:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.785+0000 7f7ebbfff700 1 -- 172.21.15.44:0/3424903173 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc005290 con 0x55ca2c91d720 2023-11-16T16:51:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.785+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/3424903173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c91d720 msgr2=0x55ca2c911f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.785+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/3424903173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c91d720 0x55ca2c911f60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc000c00 tx=0x7f7ecc016000 comp rx=0 tx=0).stop 2023-11-16T16:51:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.786+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/3424903173 shutdown_connections 2023-11-16T16:51:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.786+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/3424903173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c91db90 0x55ca2c919c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.786+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/3424903173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c7736b0 0x55ca2c920ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.786+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/3424903173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c91d720 0x55ca2c911f60 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc000c00 tx=0x7f7ecc016000 comp rx=0 tx=0).stop 2023-11-16T16:51:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.786+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/3424903173 >> 172.21.15.44:0/3424903173 conn(0x55ca2c90cf50 msgr2=0x55ca2c9205d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.786+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/3424903173 shutdown_connections 2023-11-16T16:51:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.786+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/3424903173 wait complete. 2023-11-16T16:51:04.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.786+0000 7f7eeb2f1b80 1 -- start start 2023-11-16T16:51:04.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.787+0000 7f7eeb2f1b80 1 Processor -- start 2023-11-16T16:51:04.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.787+0000 7f7eeb2f1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c7736b0 0x55ca2c926450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.787+0000 7f7eeb2f1b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c91db90 0x55ca2c926990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.787+0000 7f7eeb2f1b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c930d80 0x55ca2c97dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.787+0000 7f7eeb2f1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2c97e150 con 0x55ca2c7736b0 2023-11-16T16:51:04.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.787+0000 7f7eeb2f1b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2c97e2c0 con 0x55ca2c91db90 2023-11-16T16:51:04.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.787+0000 7f7eeb2f1b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2c97e480 con 0x55ca2c930d80 2023-11-16T16:51:04.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.788+0000 7f7eeb2f1b80 10 client.0 _ll_register_callbacks cb 0x7f7ecc009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:04.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.788+0000 7f7ed4f9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c7736b0 0x55ca2c926450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:04.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.788+0000 7f7ed4f9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c7736b0 0x55ca2c926450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39938/0 (socket says 172.21.15.44:39938) 2023-11-16T16:51:04.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.788+0000 7f7ed4f9a700 1 -- 172.21.15.44:0/2121337373 learned_addr learned my addr 172.21.15.44:0/2121337373 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:04.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.789+0000 7f7ed4f9a700 1 -- 172.21.15.44:0/2121337373 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c930d80 msgr2=0x55ca2c97dc10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:51:04.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.789+0000 7f7ed4f9a700 1 --2- 172.21.15.44:0/2121337373 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c930d80 0x55ca2c97dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.789+0000 7f7ed4f9a700 1 -- 172.21.15.44:0/2121337373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c91db90 msgr2=0x55ca2c926990 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T16:51:04.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.789+0000 7f7ed4f9a700 1 --2- 172.21.15.44:0/2121337373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c91db90 0x55ca2c926990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.789+0000 7f7ed4f9a700 1 -- 172.21.15.44:0/2121337373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ca2c82f130 con 0x55ca2c7736b0 2023-11-16T16:51:04.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.789+0000 7f7ed4f9a700 1 --2- 172.21.15.44:0/2121337373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c7736b0 0x55ca2c926450 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0010470 tx=0x7f7ec00092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4022f6b66e5bbafe server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:04.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.789+0000 7f7eb8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:51:04.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.789+0000 7f7eb8ff9700 1 -- 172.21.15.44:0/2121337373 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0005370 con 0x55ca2c7736b0 2023-11-16T16:51:04.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.790+0000 7f7eb8ff9700 1 -- 172.21.15.44:0/2121337373 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f7ec00054d0 con 0x55ca2c7736b0 2023-11-16T16:51:04.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.790+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/2121337373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ec0011900 con 0x55ca2c7736b0 2023-11-16T16:51:04.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.790+0000 7f7eb8ff9700 1 -- 172.21.15.44:0/2121337373 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ec0005690 con 0x55ca2c7736b0 2023-11-16T16:51:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.790+0000 7f7eeb2f1b80 10 client.15804 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.790+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/2121337373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55ca2c696d80 con 0x55ca2c7736b0 2023-11-16T16:51:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.790+0000 7f7eb8ff9700 1 -- 172.21.15.44:0/2121337373 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ec0011900 con 0x55ca2c7736b0 2023-11-16T16:51:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.790+0000 7f7eb8ff9700 1 -- 172.21.15.44:0/2121337373 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7ec00105f0 con 0x55ca2c7736b0 2023-11-16T16:51:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.791+0000 7f7eeb2f1b80 10 client.15804 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.791+0000 7f7eeb2f1b80 10 client.15804 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.791+0000 7f7eeb2f1b80 20 client.15804 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.791+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/2121337373 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ca2c985ab0 0x55ca2c987ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.791+0000 7f7eeb2f1b80 4 client.15804 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:51:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.791+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/2121337373 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ca2c837600 con 0x55ca2c985ab0 2023-11-16T16:51:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.791+0000 7f7ed579b700 1 --2- 172.21.15.44:0/2121337373 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ca2c985ab0 0x55ca2c987ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.792+0000 7f7ed579b700 1 --2- 172.21.15.44:0/2121337373 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ca2c985ab0 0x55ca2c987ec0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=937048a193a205f6 server_cookie=a43b1bcb38519f54 in_seq=0 out_seq=0 2023-11-16T16:51:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.792+0000 7f7eb8ff9700 10 client.15804 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.798+0000 7f7eb8ff9700 1 -- 172.21.15.44:0/2121337373 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ca2c837600 con 0x55ca2c985ab0 2023-11-16T16:51:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.798+0000 7f7eb8ff9700 10 client.15804 handle_command_reply: tid=0 2023-11-16T16:51:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.798+0000 7f7eeb2f1b80 1 client.15804 shutdown 2023-11-16T16:51:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.799+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/2121337373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c7736b0 msgr2=0x55ca2c926450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.799+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/2121337373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c7736b0 0x55ca2c926450 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0010470 tx=0x7f7ec00092d0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.799+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/2121337373 shutdown_connections 2023-11-16T16:51:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.799+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/2121337373 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ca2c985ab0 0x55ca2c987ec0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.799+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/2121337373 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ca2c930d80 0x55ca2c97dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.799+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/2121337373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca2c7736b0 0x55ca2c926450 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.799+0000 7f7eeb2f1b80 1 --2- 172.21.15.44:0/2121337373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca2c91db90 0x55ca2c926990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.799+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/2121337373 >> 172.21.15.44:0/2121337373 conn(0x55ca2c90cf50 msgr2=0x55ca2c91f500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.799+0000 7f7eb8ff9700 0 client.15804 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.799+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/2121337373 shutdown_connections 2023-11-16T16:51:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.799+0000 7f7eeb2f1b80 1 -- 172.21.15.44:0/2121337373 wait complete. 2023-11-16T16:51:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.800+0000 7f7eeb2f1b80 20 client.15804 trim_cache size 0 max 16384 2023-11-16T16:51:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:51:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:51:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:51:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:51:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.800+0000 7f7ed79ff700 1 -- 172.21.15.44:0/3628743281 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7ebc0a1460 msgr2=0x7f7ebc0a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.800+0000 7f7ed79ff700 1 --2- 172.21.15.44:0/3628743281 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7ebc0a1460 0x7f7ebc0a3910 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80cb8f0 tx=0x7f7ec8009920 comp rx=0 tx=0).stop 2023-11-16T16:51:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.800+0000 7f7ed79ff700 1 -- 172.21.15.44:0/3628743281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed0105560 msgr2=0x7f7ed010bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.800+0000 7f7ed79ff700 1 --2- 172.21.15.44:0/3628743281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed0105560 0x7f7ed010bef0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8015000 tx=0x7f7ec80093b0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.801+0000 7f7ed79ff700 1 -- 172.21.15.44:0/3628743281 shutdown_connections 2023-11-16T16:51:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.801+0000 7f7ed79ff700 1 --2- 172.21.15.44:0/3628743281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed0105560 0x7f7ed010bef0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.801+0000 7f7ed79ff700 1 --2- 172.21.15.44:0/3628743281 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ed0100c50 0x7f7ed01ab0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.801+0000 7f7ed79ff700 1 --2- 172.21.15.44:0/3628743281 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7ebc0a1460 0x7f7ebc0a3910 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.801+0000 7f7ed79ff700 1 --2- 172.21.15.44:0/3628743281 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ed01015f0 0x7f7ed01ab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.801+0000 7f7ed79ff700 1 -- 172.21.15.44:0/3628743281 >> 172.21.15.44:0/3628743281 conn(0x7f7ed0077550 msgr2=0x7f7ed00fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.801+0000 7f7ed79ff700 1 -- 172.21.15.44:0/3628743281 shutdown_connections 2023-11-16T16:51:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:04.801+0000 7f7ed79ff700 1 -- 172.21.15.44:0/3628743281 wait complete. 2023-11-16T16:51:05.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:05 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:51:05.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:05 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:51:05.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:05 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:51:05.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:05 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:51:05.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:05 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:51:05.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:05 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:51:05.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:05 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:51:05.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:05 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:51:05.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:05 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:51:06.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:06 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:51:06.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:06 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:51:06.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:06 smithi044 ceph-mon[128750]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 16 op/s 2023-11-16T16:51:06.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:06 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:51:06.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:06 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:51:06.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:06 smithi176 ceph-mon[131630]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 16 op/s 2023-11-16T16:51:06.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:06 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:51:06.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:06 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:51:06.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:06 smithi077 ceph-mon[131204]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 16 op/s 2023-11-16T16:51:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:07 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:51:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:07 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:51:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:07 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:51:07.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:07 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:51:07.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:07 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:51:07.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:07 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:51:07.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:07 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:51:07.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:07 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:51:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:07 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:51:08.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:08 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:51:08.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:08 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T16:51:08.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:08 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:51:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:08 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:51:08.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:08 smithi044 ceph-mon[128750]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 17 op/s 2023-11-16T16:51:08.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:08 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:51:08.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:08 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T16:51:08.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:08 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:51:08.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:08 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:51:08.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:08 smithi176 ceph-mon[131630]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 17 op/s 2023-11-16T16:51:08.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:08 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:51:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:08 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T16:51:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:08 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:51:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:08 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:51:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:08 smithi077 ceph-mon[131204]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 17 op/s 2023-11-16T16:51:10.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:10 smithi077 ceph-mon[131204]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 23 KiB/s wr, 14 op/s 2023-11-16T16:51:11.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:10 smithi044 ceph-mon[128750]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 23 KiB/s wr, 14 op/s 2023-11-16T16:51:11.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:10 smithi176 ceph-mon[131630]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 23 KiB/s wr, 14 op/s 2023-11-16T16:51:12.028 INFO:tasks.workunit.client.0.smithi044.stderr:Updating files: 62% (7558/12074) Updating files: 63% (7607/12074) Updating files: 64% (7728/12074) Updating files: 65% (7849/12074) Updating files: 66% (7969/12074) Updating files: 67% (8090/12074) Updating files: 68% (8211/12074) Updating files: 69% (8332/12074) Updating files: 70% (8452/12074) Updating files: 71% (8573/12074) Updating files: 72% (8694/12074) Updating files: 73% (8815/12074) Updating files: 74% (8935/12074) Updating files: 75% (9056/12074) Updating files: 76% (9177/12074) Updating files: 77% (9297/12074) Updating files: 78% (9418/12074) Updating files: 79% (9539/12074) Updating files: 80% (9660/12074) Updating files: 81% (9780/12074) Updating files: 82% (9901/12074) Updating files: 83% (10022/12074) Updating files: 84% (10143/12074) Updating files: 85% (10263/12074) Updating files: 86% (10384/12074) Updating files: 87% (10505/12074) Updating files: 88% (10626/12074) Updating files: 89% (10746/12074) Updating files: 90% (10867/12074) Updating files: 91% (10988/12074) Updating files: 92% (11109/12074) Updating files: 93% (11229/12074) Updating files: 94% (11350/12074) Updating files: 95% (11471/12074) Updating files: 96% (11592/12074) Updating files: 97% (11712/12074) Updating files: 98% (11833/12074) Updating files: 99% (11954/12074) Updating files: 100% (12074/12074) Updating files: 100% (12074/12074), done. 2023-11-16T16:51:12.775 INFO:tasks.workunit.client.0.smithi044.stderr:Note: switching to '5e4a743ad7fd1422ebd6c4dc950a21a303279207'. 2023-11-16T16:51:12.775 INFO:tasks.workunit.client.0.smithi044.stderr: 2023-11-16T16:51:12.775 INFO:tasks.workunit.client.0.smithi044.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-11-16T16:51:12.775 INFO:tasks.workunit.client.0.smithi044.stderr:changes and commit them, and you can discard any commits you make in this 2023-11-16T16:51:12.775 INFO:tasks.workunit.client.0.smithi044.stderr:state without impacting any branches by switching back to a branch. 2023-11-16T16:51:12.775 INFO:tasks.workunit.client.0.smithi044.stderr: 2023-11-16T16:51:12.775 INFO:tasks.workunit.client.0.smithi044.stderr:If you want to create a new branch to retain commits you create, you may 2023-11-16T16:51:12.776 INFO:tasks.workunit.client.0.smithi044.stderr:do so (now or later) by using -c with the switch command. Example: 2023-11-16T16:51:12.776 INFO:tasks.workunit.client.0.smithi044.stderr: 2023-11-16T16:51:12.776 INFO:tasks.workunit.client.0.smithi044.stderr: git switch -c 2023-11-16T16:51:12.776 INFO:tasks.workunit.client.0.smithi044.stderr: 2023-11-16T16:51:12.776 INFO:tasks.workunit.client.0.smithi044.stderr:Or undo this operation with: 2023-11-16T16:51:12.776 INFO:tasks.workunit.client.0.smithi044.stderr: 2023-11-16T16:51:12.776 INFO:tasks.workunit.client.0.smithi044.stderr: git switch - 2023-11-16T16:51:12.776 INFO:tasks.workunit.client.0.smithi044.stderr: 2023-11-16T16:51:12.776 INFO:tasks.workunit.client.0.smithi044.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-11-16T16:51:12.776 INFO:tasks.workunit.client.0.smithi044.stderr: 2023-11-16T16:51:12.776 INFO:tasks.workunit.client.0.smithi044.stderr:HEAD is now at 5e4a743ad7f doc/cephfs: Update docs to match remove functionality and respective vxattrs 2023-11-16T16:51:12.794 DEBUG:teuthology.orchestra.run.smithi044:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-11-16T16:51:12.887 INFO:tasks.workunit.client.0.smithi044.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-11-16T16:51:12.892 INFO:tasks.workunit.client.0.smithi044.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-16T16:51:12.892 INFO:tasks.workunit.client.0.smithi044.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-11-16T16:51:12.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:12 smithi077 ceph-mon[131204]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 23 KiB/s wr, 15 op/s 2023-11-16T16:51:13.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:12 smithi044 ceph-mon[128750]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 23 KiB/s wr, 15 op/s 2023-11-16T16:51:13.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:12 smithi176 ceph-mon[131630]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 23 KiB/s wr, 15 op/s 2023-11-16T16:51:13.550 INFO:tasks.workunit.client.0.smithi044.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-11-16T16:51:13.646 INFO:tasks.workunit.client.0.smithi044.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-11-16T16:51:13.712 INFO:tasks.workunit.client.0.smithi044.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-16T16:51:13.717 INFO:tasks.workunit.client.0.smithi044.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-16T16:51:13.717 INFO:tasks.workunit.client.0.smithi044.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-11-16T16:51:13.781 INFO:tasks.workunit.client.0.smithi044.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-16T16:51:13.788 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T16:51:13.789 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-11-16T16:51:13.833 INFO:tasks.workunit:Running workunits matching fs/misc on client.0... 2023-11-16T16:51:13.834 INFO:tasks.workunit:Running workunit fs/misc/acl.sh... 2023-11-16T16:51:13.835 DEBUG:teuthology.orchestra.run.smithi044:workunit test fs/misc/acl.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh 2023-11-16T16:51:13.930 INFO:tasks.workunit.client.0.smithi044.stderr:+ set -e 2023-11-16T16:51:13.931 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir -p testdir 2023-11-16T16:51:13.935 INFO:tasks.workunit.client.0.smithi044.stderr:+ cd testdir 2023-11-16T16:51:13.935 INFO:tasks.workunit.client.0.smithi044.stderr:+ set +e 2023-11-16T16:51:13.935 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfacl -d -m u:nobody:rw . 2023-11-16T16:51:13.943 INFO:tasks.workunit.client.0.smithi044.stderr:+ test 0 '!=' 0 2023-11-16T16:51:13.944 INFO:tasks.workunit.client.0.smithi044.stderr:+ set -e 2023-11-16T16:51:13.944 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=0 2023-11-16T16:51:13.944 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 0 -lt 100 ']' 2023-11-16T16:51:13.944 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 0 + 1 2023-11-16T16:51:13.946 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=1 2023-11-16T16:51:13.946 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:13.951 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:13.952 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:13.959 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:13.960 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:13.960 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:14.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:14 smithi044 ceph-mon[128750]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 682 B/s wr, 2 op/s 2023-11-16T16:51:14.476 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:14 smithi077 ceph-mon[131204]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 682 B/s wr, 2 op/s 2023-11-16T16:51:14.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:14 smithi176 ceph-mon[131630]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 682 B/s wr, 2 op/s 2023-11-16T16:51:16.678 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:16.678 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:16.840 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:16.840 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:16.864 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:16.864 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:16.864 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 1 -lt 100 ']' 2023-11-16T16:51:16.864 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 1 + 1 2023-11-16T16:51:16.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:16 smithi044 ceph-mon[128750]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 682 B/s wr, 2 op/s 2023-11-16T16:51:16.902 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=2 2023-11-16T16:51:16.902 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:16.936 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:16.936 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:16.939 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:16.940 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:16.940 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:16.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:16 smithi077 ceph-mon[131204]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 682 B/s wr, 2 op/s 2023-11-16T16:51:17.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:16 smithi176 ceph-mon[131630]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 682 B/s wr, 2 op/s 2023-11-16T16:51:17.370 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:17.370 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:17.490 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:17.490 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:17.505 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:17.505 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:17.505 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -lt 100 ']' 2023-11-16T16:51:17.505 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 2 + 1 2023-11-16T16:51:17.546 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=3 2023-11-16T16:51:17.547 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:17.568 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:17.568 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:17.571 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:17.571 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:17.572 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:18.186 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:18.187 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:18.382 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:18.382 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:18.397 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:18.397 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:18.397 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 3 -lt 100 ']' 2023-11-16T16:51:18.398 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 3 + 1 2023-11-16T16:51:18.435 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=4 2023-11-16T16:51:18.435 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:18.494 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:18.494 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:18.497 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:18.497 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:18.498 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:18.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:18 smithi077 ceph-mon[131204]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2023-11-16T16:51:19.064 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:18 smithi044 ceph-mon[128750]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2023-11-16T16:51:19.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:18 smithi176 ceph-mon[131630]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2023-11-16T16:51:19.203 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:19.203 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:19.423 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:19.423 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:19.448 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:19.448 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:19.448 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 4 -lt 100 ']' 2023-11-16T16:51:19.448 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 4 + 1 2023-11-16T16:51:19.493 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=5 2023-11-16T16:51:19.493 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:19.520 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:19.520 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:19.523 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:19.523 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:19.524 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:19.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.906+0000 7f92bcfff700 1 -- 172.21.15.44:0/3121289168 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b8102460 msgr2=0x7f92b81028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:19.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.906+0000 7f92bcfff700 1 --2- 172.21.15.44:0/3121289168 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b8102460 0x7f92b81028d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f92a0009990 tx=0x7f92a0009ca0 comp rx=0 tx=0).stop 2023-11-16T16:51:19.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.907+0000 7f92bcfff700 1 -- 172.21.15.44:0/3121289168 shutdown_connections 2023-11-16T16:51:19.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.907+0000 7f92bcfff700 1 --2- 172.21.15.44:0/3121289168 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92b8102e10 0x7f92b810f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:19.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.907+0000 7f92bcfff700 1 --2- 172.21.15.44:0/3121289168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92b8108470 0x7f92b8108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:19.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.907+0000 7f92bcfff700 1 --2- 172.21.15.44:0/3121289168 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b8102460 0x7f92b81028d0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.907+0000 7f92bcfff700 1 -- 172.21.15.44:0/3121289168 >> 172.21.15.44:0/3121289168 conn(0x7f92b80fdf60 msgr2=0x7f92b8100370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.907+0000 7f92bcfff700 1 -- 172.21.15.44:0/3121289168 shutdown_connections 2023-11-16T16:51:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.907+0000 7f92bcfff700 1 -- 172.21.15.44:0/3121289168 wait complete. 2023-11-16T16:51:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.908+0000 7f92bcfff700 1 Processor -- start 2023-11-16T16:51:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.908+0000 7f92bcfff700 1 -- start start 2023-11-16T16:51:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.908+0000 7f92bcfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b8102460 0x7f92b81985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.908+0000 7f92bcfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92b8102e10 0x7f92b8198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.908+0000 7f92bcfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92b8108470 0x7f92b819cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.908+0000 7f92bcfff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b819d5c0 con 0x7f92b8102460 2023-11-16T16:51:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.908+0000 7f92bcfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b819d730 con 0x7f92b8102e10 2023-11-16T16:51:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.908+0000 7f92bcfff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b819d8f0 con 0x7f92b8108470 2023-11-16T16:51:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.909+0000 7f92b659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b8102460 0x7f92b81985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.909+0000 7f92b659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b8102460 0x7f92b81985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47274/0 (socket says 172.21.15.44:47274) 2023-11-16T16:51:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.909+0000 7f92b659c700 1 -- 172.21.15.44:0/394473913 learned_addr learned my addr 172.21.15.44:0/394473913 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.909+0000 7f92affff700 1 --2- 172.21.15.44:0/394473913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92b8102e10 0x7f92b8198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.909+0000 7f92b6d9d700 1 --2- 172.21.15.44:0/394473913 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92b8108470 0x7f92b819cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.909+0000 7f92b659c700 1 -- 172.21.15.44:0/394473913 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92b8108470 msgr2=0x7f92b819cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.909+0000 7f92b659c700 1 --2- 172.21.15.44:0/394473913 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92b8108470 0x7f92b819cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.909+0000 7f92b659c700 1 -- 172.21.15.44:0/394473913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92b8102e10 msgr2=0x7f92b8198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.909+0000 7f92b659c700 1 --2- 172.21.15.44:0/394473913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92b8102e10 0x7f92b8198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.909+0000 7f92b659c700 1 -- 172.21.15.44:0/394473913 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a00095f0 con 0x7f92b8102460 2023-11-16T16:51:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.909+0000 7f92b659c700 1 --2- 172.21.15.44:0/394473913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b8102460 0x7f92b81985e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f92a800d950 tx=0x7f92a800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.909+0000 7f92af7fe700 1 -- 172.21.15.44:0/394473913 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8009940 con 0x7f92b8102460 2023-11-16T16:51:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.910+0000 7f92bcfff700 1 -- 172.21.15.44:0/394473913 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b81124b0 con 0x7f92b8102460 2023-11-16T16:51:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.910+0000 7f92af7fe700 1 -- 172.21.15.44:0/394473913 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f92a8010460 con 0x7f92b8102460 2023-11-16T16:51:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.910+0000 7f92af7fe700 1 -- 172.21.15.44:0/394473913 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a800f620 con 0x7f92b8102460 2023-11-16T16:51:19.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.910+0000 7f92bcfff700 1 -- 172.21.15.44:0/394473913 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b8112320 con 0x7f92b8102460 2023-11-16T16:51:19.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.913+0000 7f92af7fe700 1 -- 172.21.15.44:0/394473913 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f92a80108d0 con 0x7f92b8102460 2023-11-16T16:51:19.916 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.914+0000 7f92af7fe700 1 --2- 172.21.15.44:0/394473913 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f92a40a13e0 0x7f92a40a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:19.916 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.914+0000 7f92af7fe700 1 -- 172.21.15.44:0/394473913 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f92a80cd1b0 con 0x7f92b8102460 2023-11-16T16:51:19.916 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.914+0000 7f92affff700 1 --2- 172.21.15.44:0/394473913 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f92a40a13e0 0x7f92a40a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:19.917 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:19.915+0000 7f92affff700 1 --2- 172.21.15.44:0/394473913 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f92a40a13e0 0x7f92a40a3890 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f92b8199bf0 tx=0x7f92a0005fb0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:20.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.000+0000 7f92d097db80 1 Processor -- start 2023-11-16T16:51:20.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.000+0000 7f92d097db80 1 -- start start 2023-11-16T16:51:20.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.001+0000 7f92d097db80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 0x55f88a1ab1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.001+0000 7f92d097db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1a5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.001+0000 7f92d097db80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a1abb90 0x55f88a1a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.001+0000 7f92d097db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f88a0ab000 con 0x55f88a1ab720 2023-11-16T16:51:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.001+0000 7f92d097db80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f88a0fd840 con 0x55f88a0016b0 2023-11-16T16:51:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.001+0000 7f92d097db80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f88a0f6cd0 con 0x55f88a1abb90 2023-11-16T16:51:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.001+0000 7f92affff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.001+0000 7f92affff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1a5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47284/0 (socket says 172.21.15.44:47284) 2023-11-16T16:51:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.001+0000 7f92affff700 1 -- 172.21.15.44:0/1797352308 learned_addr learned my addr 172.21.15.44:0/1797352308 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.001+0000 7f92b659c700 1 --2- 172.21.15.44:0/1797352308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 0x55f88a1ab1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.001+0000 7f92b6d9d700 1 --2- 172.21.15.44:0/1797352308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a1abb90 0x55f88a1a5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.002+0000 7f92affff700 1 -- 172.21.15.44:0/1797352308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a1abb90 msgr2=0x55f88a1a5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:20.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.002+0000 7f92affff700 1 --2- 172.21.15.44:0/1797352308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a1abb90 0x55f88a1a5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.002+0000 7f92affff700 1 -- 172.21.15.44:0/1797352308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 msgr2=0x55f88a1ab1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:20.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.002+0000 7f92affff700 1 --2- 172.21.15.44:0/1797352308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 0x55f88a1ab1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.002+0000 7f92affff700 1 -- 172.21.15.44:0/1797352308 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f889f785d0 con 0x55f88a1ab720 2023-11-16T16:51:20.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.002+0000 7f92b659c700 1 --2- 172.21.15.44:0/1797352308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 0x55f88a1ab1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:20.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.002+0000 7f92affff700 1 --2- 172.21.15.44:0/1797352308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1a5560 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f92a0009990 tx=0x7f92a0020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94f5223e7f999cd server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:20.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.002+0000 7f92acff9700 1 -- 172.21.15.44:0/1797352308 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a002b070 con 0x55f88a1ab720 2023-11-16T16:51:20.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.002+0000 7f92acff9700 1 -- 172.21.15.44:0/1797352308 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f92a001e040 con 0x55f88a1ab720 2023-11-16T16:51:20.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.002+0000 7f92acff9700 1 -- 172.21.15.44:0/1797352308 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a000f6a0 con 0x55f88a1ab720 2023-11-16T16:51:20.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.003+0000 7f92d097db80 1 -- 172.21.15.44:0/1797352308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 msgr2=0x55f88a1a5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.003+0000 7f92d097db80 1 --2- 172.21.15.44:0/1797352308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1a5560 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f92a0009990 tx=0x7f92a0020000 comp rx=0 tx=0).stop 2023-11-16T16:51:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.003+0000 7f92d097db80 1 -- 172.21.15.44:0/1797352308 shutdown_connections 2023-11-16T16:51:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.003+0000 7f92d097db80 1 --2- 172.21.15.44:0/1797352308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a1abb90 0x55f88a1a5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.003+0000 7f92d097db80 1 --2- 172.21.15.44:0/1797352308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 0x55f88a1ab1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.003+0000 7f92d097db80 1 --2- 172.21.15.44:0/1797352308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1a5560 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.003+0000 7f92d097db80 1 -- 172.21.15.44:0/1797352308 >> 172.21.15.44:0/1797352308 conn(0x55f88a19af50 msgr2=0x55f88a1a0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.003+0000 7f92d097db80 1 -- 172.21.15.44:0/1797352308 shutdown_connections 2023-11-16T16:51:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.003+0000 7f92d097db80 1 -- 172.21.15.44:0/1797352308 wait complete. 2023-11-16T16:51:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.004+0000 7f92d097db80 1 -- start start 2023-11-16T16:51:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.004+0000 7f92d097db80 1 Processor -- start 2023-11-16T16:51:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.004+0000 7f92d097db80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 0x55f88a1b4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.004+0000 7f92d097db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.004+0000 7f92d097db80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a1abb90 0x55f88a1aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.004+0000 7f92d097db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f88a098880 con 0x55f88a1ab720 2023-11-16T16:51:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.004+0000 7f92d097db80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f88a098520 con 0x55f88a0016b0 2023-11-16T16:51:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.004+0000 7f92d097db80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f889f7a8c0 con 0x55f88a1abb90 2023-11-16T16:51:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.004+0000 7f92d097db80 10 client.0 _ll_register_callbacks cb 0x55f88a0b6720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.005+0000 7f92affff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.005+0000 7f92affff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1b4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47286/0 (socket says 172.21.15.44:47286) 2023-11-16T16:51:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.005+0000 7f92affff700 1 -- 172.21.15.44:0/710398273 learned_addr learned my addr 172.21.15.44:0/710398273 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.005+0000 7f92b659c700 1 --2- 172.21.15.44:0/710398273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 0x55f88a1b4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.005+0000 7f92b6d9d700 1 --2- 172.21.15.44:0/710398273 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a1abb90 0x55f88a1aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.005+0000 7f92affff700 1 -- 172.21.15.44:0/710398273 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a1abb90 msgr2=0x55f88a1aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.005+0000 7f92affff700 1 --2- 172.21.15.44:0/710398273 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a1abb90 0x55f88a1aecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.005+0000 7f92affff700 1 -- 172.21.15.44:0/710398273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 msgr2=0x55f88a1b4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.005+0000 7f92affff700 1 --2- 172.21.15.44:0/710398273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 0x55f88a1b4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.005+0000 7f92affff700 1 -- 172.21.15.44:0/710398273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b810fba0 con 0x55f88a1ab720 2023-11-16T16:51:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.005+0000 7f92affff700 1 --2- 172.21.15.44:0/710398273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1b4fb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f92a0000c00 tx=0x7f92a000fb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa7a1722c1241387 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.005+0000 7f929dffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:51:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.006+0000 7f929dffb700 1 -- 172.21.15.44:0/710398273 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a002b040 con 0x55f88a1ab720 2023-11-16T16:51:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.006+0000 7f929dffb700 1 -- 172.21.15.44:0/710398273 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f92a000be80 con 0x55f88a1ab720 2023-11-16T16:51:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.006+0000 7f92d097db80 1 -- 172.21.15.44:0/710398273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f88a198710 con 0x55f88a1ab720 2023-11-16T16:51:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.006+0000 7f929dffb700 1 -- 172.21.15.44:0/710398273 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92a0004c20 con 0x55f88a1ab720 2023-11-16T16:51:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.006+0000 7f92d097db80 10 client.15816 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.006+0000 7f92d097db80 1 -- 172.21.15.44:0/710398273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55f88a0bd130 con 0x55f88a1ab720 2023-11-16T16:51:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.006+0000 7f929dffb700 1 -- 172.21.15.44:0/710398273 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f88a198710 con 0x55f88a1ab720 2023-11-16T16:51:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.006+0000 7f929dffb700 1 -- 172.21.15.44:0/710398273 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f92a0024d10 con 0x55f88a1ab720 2023-11-16T16:51:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.006+0000 7f92d097db80 10 client.15816 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.006+0000 7f92d097db80 10 client.15816 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.007+0000 7f92d097db80 20 client.15816 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.007+0000 7f92d097db80 1 --2- 172.21.15.44:0/710398273 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f88a1be9f0 0x55f88a1c0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.007+0000 7f92d097db80 4 client.15816 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:51:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.007+0000 7f92d097db80 1 -- 172.21.15.44:0/710398273 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f88a0b9040 con 0x55f88a1be9f0 2023-11-16T16:51:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.007+0000 7f92b659c700 1 --2- 172.21.15.44:0/710398273 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f88a1be9f0 0x55f88a1c0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.007+0000 7f92b659c700 1 --2- 172.21.15.44:0/710398273 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f88a1be9f0 0x55f88a1c0dd0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1f8ba5d99477174 server_cookie=29daa5997bf2191a in_seq=0 out_seq=0 2023-11-16T16:51:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.007+0000 7f929dffb700 10 client.15816 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.008+0000 7f929dffb700 1 -- 172.21.15.44:0/710398273 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f88a0b9040 con 0x55f88a1be9f0 2023-11-16T16:51:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.008+0000 7f929dffb700 10 client.15816 handle_command_reply: tid=0 2023-11-16T16:51:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.008+0000 7f92d097db80 1 client.15816 shutdown 2023-11-16T16:51:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.008+0000 7f92d097db80 1 -- 172.21.15.44:0/710398273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 msgr2=0x55f88a1b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.008+0000 7f92d097db80 1 --2- 172.21.15.44:0/710398273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1b4fb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f92a0000c00 tx=0x7f92a000fb90 comp rx=0 tx=0).stop 2023-11-16T16:51:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.008+0000 7f92d097db80 1 -- 172.21.15.44:0/710398273 shutdown_connections 2023-11-16T16:51:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.008+0000 7f92d097db80 1 --2- 172.21.15.44:0/710398273 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f88a1be9f0 0x55f88a1c0dd0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.008+0000 7f92d097db80 1 --2- 172.21.15.44:0/710398273 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a1abb90 0x55f88a1aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.008+0000 7f92d097db80 1 --2- 172.21.15.44:0/710398273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 0x55f88a1b4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.008+0000 7f92d097db80 1 --2- 172.21.15.44:0/710398273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1b4fb0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.008+0000 7f929dffb700 0 client.15816 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.008+0000 7f92d097db80 1 -- 172.21.15.44:0/710398273 >> 172.21.15.44:0/710398273 conn(0x55f88a19af50 msgr2=0x55f88a19f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.009+0000 7f92d097db80 1 -- 172.21.15.44:0/710398273 shutdown_connections 2023-11-16T16:51:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.009+0000 7f92d097db80 1 -- 172.21.15.44:0/710398273 wait complete. 2023-11-16T16:51:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.009+0000 7f92d097db80 20 client.15816 trim_cache size 0 max 16384 2023-11-16T16:51:20.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.024+0000 7f92d097db80 1 Processor -- start 2023-11-16T16:51:20.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.024+0000 7f92d097db80 1 -- start start 2023-11-16T16:51:20.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.024+0000 7f92d097db80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a0016b0 0x55f88a1a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.024+0000 7f92d097db80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a1ab720 0x55f88a19ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.024+0000 7f92d097db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1abb90 0x55f88a1abf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.024+0000 7f92d097db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f88a1a0fb0 con 0x55f88a1abb90 2023-11-16T16:51:20.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.024+0000 7f92d097db80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f88a19ab70 con 0x55f88a1ab720 2023-11-16T16:51:20.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.024+0000 7f92d097db80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f88a1a9970 con 0x55f88a0016b0 2023-11-16T16:51:20.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.025+0000 7f92b659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a0016b0 0x55f88a1a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.025+0000 7f92b6d9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1abb90 0x55f88a1abf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.025+0000 7f92b659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a0016b0 0x55f88a1a1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:49280/0 (socket says 172.21.15.44:49280) 2023-11-16T16:51:20.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.025+0000 7f92b659c700 1 -- 172.21.15.44:0/384256901 learned_addr learned my addr 172.21.15.44:0/384256901 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:20.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.025+0000 7f92affff700 1 --2- 172.21.15.44:0/384256901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a1ab720 0x55f88a19ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.025+0000 7f92b659c700 1 -- 172.21.15.44:0/384256901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a1ab720 msgr2=0x55f88a19ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:20.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.025+0000 7f92b659c700 1 --2- 172.21.15.44:0/384256901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a1ab720 0x55f88a19ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.025+0000 7f92b659c700 1 -- 172.21.15.44:0/384256901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1abb90 msgr2=0x55f88a1abf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:20.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.025+0000 7f92b659c700 1 --2- 172.21.15.44:0/384256901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1abb90 0x55f88a1abf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.025+0000 7f92b659c700 1 -- 172.21.15.44:0/384256901 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f88a0bce80 con 0x55f88a0016b0 2023-11-16T16:51:20.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.025+0000 7f92b659c700 1 --2- 172.21.15.44:0/384256901 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a0016b0 0x55f88a1a1a00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f92a8099be0 tx=0x7f92a80cc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=894cdf1286614b3a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:20.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.026+0000 7f92ad7fa700 1 -- 172.21.15.44:0/384256901 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a80a20a0 con 0x55f88a0016b0 2023-11-16T16:51:20.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.026+0000 7f92ad7fa700 1 -- 172.21.15.44:0/384256901 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f92a8097ff0 con 0x55f88a0016b0 2023-11-16T16:51:20.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.026+0000 7f92ad7fa700 1 -- 172.21.15.44:0/384256901 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8099140 con 0x55f88a0016b0 2023-11-16T16:51:20.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.026+0000 7f92d097db80 1 -- 172.21.15.44:0/384256901 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a0016b0 msgr2=0x55f88a1a1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:20.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.026+0000 7f92d097db80 1 --2- 172.21.15.44:0/384256901 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a0016b0 0x55f88a1a1a00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f92a8099be0 tx=0x7f92a80cc040 comp rx=0 tx=0).stop 2023-11-16T16:51:20.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.026+0000 7f92d097db80 1 -- 172.21.15.44:0/384256901 shutdown_connections 2023-11-16T16:51:20.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.026+0000 7f92d097db80 1 --2- 172.21.15.44:0/384256901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1abb90 0x55f88a1abf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.026+0000 7f92d097db80 1 --2- 172.21.15.44:0/384256901 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a0016b0 0x55f88a1a1a00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.026+0000 7f92d097db80 1 --2- 172.21.15.44:0/384256901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a1ab720 0x55f88a19ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.026+0000 7f92d097db80 1 -- 172.21.15.44:0/384256901 >> 172.21.15.44:0/384256901 conn(0x55f88a19af50 msgr2=0x55f88a1af650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:20.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.026+0000 7f92d097db80 1 -- 172.21.15.44:0/384256901 shutdown_connections 2023-11-16T16:51:20.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.026+0000 7f92d097db80 1 -- 172.21.15.44:0/384256901 wait complete. 2023-11-16T16:51:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.027+0000 7f92d097db80 1 -- start start 2023-11-16T16:51:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.027+0000 7f92d097db80 1 Processor -- start 2023-11-16T16:51:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.027+0000 7f92d097db80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 0x55f88a1b2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.027+0000 7f92d097db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.027+0000 7f92d097db80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a1abb90 0x55f88a211850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.027+0000 7f92d097db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f88a1b3650 con 0x55f88a1ab720 2023-11-16T16:51:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.027+0000 7f92d097db80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f88a211d90 con 0x55f88a0016b0 2023-11-16T16:51:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.027+0000 7f92d097db80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f88a211ed0 con 0x55f88a1abb90 2023-11-16T16:51:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.028+0000 7f92d097db80 10 client.0 _ll_register_callbacks cb 0x7f92a8095f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.028+0000 7f92affff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.028+0000 7f92affff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47306/0 (socket says 172.21.15.44:47306) 2023-11-16T16:51:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.028+0000 7f92affff700 1 -- 172.21.15.44:0/1568016685 learned_addr learned my addr 172.21.15.44:0/1568016685 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.028+0000 7f92b6d9d700 1 --2- 172.21.15.44:0/1568016685 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a1abb90 0x55f88a211850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.028+0000 7f92b659c700 1 --2- 172.21.15.44:0/1568016685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 0x55f88a1b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.028+0000 7f92affff700 1 -- 172.21.15.44:0/1568016685 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a1abb90 msgr2=0x55f88a211850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.028+0000 7f92affff700 1 --2- 172.21.15.44:0/1568016685 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a1abb90 0x55f88a211850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.028+0000 7f92affff700 1 -- 172.21.15.44:0/1568016685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 msgr2=0x55f88a1b2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.028+0000 7f92affff700 1 --2- 172.21.15.44:0/1568016685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 0x55f88a1b2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.028+0000 7f92affff700 1 -- 172.21.15.44:0/1568016685 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b81124b0 con 0x55f88a1ab720 2023-11-16T16:51:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.028+0000 7f92b659c700 1 --2- 172.21.15.44:0/1568016685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 0x55f88a1b2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.028+0000 7f92affff700 1 --2- 172.21.15.44:0/1568016685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1b2d40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f92a00186d0 tx=0x7f92a0031de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81ee5f7a89f7dce server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:20.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.028+0000 7f929dffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:51:20.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.029+0000 7f929dffb700 1 -- 172.21.15.44:0/1568016685 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a002b040 con 0x55f88a1ab720 2023-11-16T16:51:20.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.029+0000 7f92d097db80 1 -- 172.21.15.44:0/1568016685 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f92a001e040 con 0x55f88a1ab720 2023-11-16T16:51:20.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.029+0000 7f929dffb700 1 -- 172.21.15.44:0/1568016685 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f92a000fcb0 con 0x55f88a1ab720 2023-11-16T16:51:20.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.029+0000 7f929dffb700 1 -- 172.21.15.44:0/1568016685 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92a0004ed0 con 0x55f88a1ab720 2023-11-16T16:51:20.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.029+0000 7f92d097db80 10 client.15822 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:20.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.029+0000 7f92d097db80 1 -- 172.21.15.44:0/1568016685 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55f889f0dc00 con 0x55f88a1ab720 2023-11-16T16:51:20.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.029+0000 7f929dffb700 1 -- 172.21.15.44:0/1568016685 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f92a001e040 con 0x55f88a1ab720 2023-11-16T16:51:20.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.029+0000 7f929dffb700 1 -- 172.21.15.44:0/1568016685 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f92a00048a0 con 0x55f88a1ab720 2023-11-16T16:51:20.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.029+0000 7f92d097db80 10 client.15822 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:20.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.029+0000 7f92d097db80 10 client.15822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.029+0000 7f92d097db80 20 client.15822 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.029+0000 7f92d097db80 1 --2- 172.21.15.44:0/1568016685 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f88a213270 0x55f88a215680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.029+0000 7f92d097db80 4 client.15822 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:51:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.029+0000 7f92d097db80 1 -- 172.21.15.44:0/1568016685 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f88a0c5600 con 0x55f88a213270 2023-11-16T16:51:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.030+0000 7f92b659c700 1 --2- 172.21.15.44:0/1568016685 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f88a213270 0x55f88a215680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.030+0000 7f92b659c700 1 --2- 172.21.15.44:0/1568016685 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f88a213270 0x55f88a215680 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a9b30435a8f3d6d server_cookie=52abb945457379fd in_seq=0 out_seq=0 2023-11-16T16:51:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.030+0000 7f929dffb700 10 client.15822 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:20.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.050+0000 7f929dffb700 1 -- 172.21.15.44:0/1568016685 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f88a0c5600 con 0x55f88a213270 2023-11-16T16:51:20.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.050+0000 7f929dffb700 10 client.15822 handle_command_reply: tid=0 2023-11-16T16:51:20.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.050+0000 7f92d097db80 1 client.15822 shutdown 2023-11-16T16:51:20.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.050+0000 7f92d097db80 1 -- 172.21.15.44:0/1568016685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 msgr2=0x55f88a1b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.050+0000 7f92d097db80 1 --2- 172.21.15.44:0/1568016685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1b2d40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f92a00186d0 tx=0x7f92a0031de0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.050+0000 7f92d097db80 1 -- 172.21.15.44:0/1568016685 shutdown_connections 2023-11-16T16:51:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.050+0000 7f92d097db80 1 --2- 172.21.15.44:0/1568016685 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f88a213270 0x55f88a215680 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.050+0000 7f92d097db80 1 --2- 172.21.15.44:0/1568016685 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f88a1abb90 0x55f88a211850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.050+0000 7f92d097db80 1 --2- 172.21.15.44:0/1568016685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f88a0016b0 0x55f88a1b2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.050+0000 7f92d097db80 1 --2- 172.21.15.44:0/1568016685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f88a1ab720 0x55f88a1b2d40 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.050+0000 7f929dffb700 0 client.15822 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.050+0000 7f92d097db80 1 -- 172.21.15.44:0/1568016685 >> 172.21.15.44:0/1568016685 conn(0x55f88a19af50 msgr2=0x55f88a1ac820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.050+0000 7f92d097db80 1 -- 172.21.15.44:0/1568016685 shutdown_connections 2023-11-16T16:51:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.050+0000 7f92d097db80 1 -- 172.21.15.44:0/1568016685 wait complete. 2023-11-16T16:51:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.051+0000 7f92d097db80 20 client.15822 trim_cache size 0 max 16384 2023-11-16T16:51:20.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:51:20.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:51:20.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:51:20.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:51:20.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.051+0000 7f92bcfff700 1 -- 172.21.15.44:0/394473913 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f92a40a13e0 msgr2=0x7f92a40a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:20.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.051+0000 7f92bcfff700 1 --2- 172.21.15.44:0/394473913 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f92a40a13e0 0x7f92a40a3890 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f92b8199bf0 tx=0x7f92a0005fb0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.051+0000 7f92bcfff700 1 -- 172.21.15.44:0/394473913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b8102460 msgr2=0x7f92b81985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:20.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.051+0000 7f92bcfff700 1 --2- 172.21.15.44:0/394473913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b8102460 0x7f92b81985e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f92a800d950 tx=0x7f92a800dc60 comp rx=0 tx=0).stop 2023-11-16T16:51:20.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.051+0000 7f92bcfff700 1 -- 172.21.15.44:0/394473913 shutdown_connections 2023-11-16T16:51:20.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.051+0000 7f92bcfff700 1 --2- 172.21.15.44:0/394473913 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f92a40a13e0 0x7f92a40a3890 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.051+0000 7f92bcfff700 1 --2- 172.21.15.44:0/394473913 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92b8108470 0x7f92b819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.051+0000 7f92bcfff700 1 --2- 172.21.15.44:0/394473913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b8102460 0x7f92b81985e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.051+0000 7f92bcfff700 1 --2- 172.21.15.44:0/394473913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92b8102e10 0x7f92b8198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.051+0000 7f92bcfff700 1 -- 172.21.15.44:0/394473913 >> 172.21.15.44:0/394473913 conn(0x7f92b80fdf60 msgr2=0x7f92b8105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:20.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.051+0000 7f92bcfff700 1 -- 172.21.15.44:0/394473913 shutdown_connections 2023-11-16T16:51:20.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:20.051+0000 7f92bcfff700 1 -- 172.21.15.44:0/394473913 wait complete. 2023-11-16T16:51:20.183 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:20.184 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:20.315 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:20.315 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:20.329 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:20.329 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:20.329 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 5 -lt 100 ']' 2023-11-16T16:51:20.330 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 5 + 1 2023-11-16T16:51:20.363 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=6 2023-11-16T16:51:20.363 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:20.394 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:20.394 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:20.396 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:20.397 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:20.397 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:20.927 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:20.927 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:20.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:20 smithi077 ceph-mon[131204]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 1 op/s 2023-11-16T16:51:21.047 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:21.048 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:21.063 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:21.063 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:21.063 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 6 -lt 100 ']' 2023-11-16T16:51:21.064 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 6 + 1 2023-11-16T16:51:21.096 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=7 2023-11-16T16:51:21.096 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:21.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:20 smithi044 ceph-mon[128750]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 1 op/s 2023-11-16T16:51:21.127 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:21.128 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:21.130 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:21.131 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:21.131 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:21.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:20 smithi176 ceph-mon[131630]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 1 op/s 2023-11-16T16:51:21.468 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:21.468 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:21.491 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:21.491 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:21.496 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:21.496 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:21.496 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 7 -lt 100 ']' 2023-11-16T16:51:21.497 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 7 + 1 2023-11-16T16:51:21.504 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=8 2023-11-16T16:51:21.504 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:21.514 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:21.515 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:21.517 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:21.518 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:21.518 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:21.688 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:21.689 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:21.711 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:21.711 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:21.717 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:21.717 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:21.717 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 8 -lt 100 ']' 2023-11-16T16:51:21.718 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 8 + 1 2023-11-16T16:51:21.724 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=9 2023-11-16T16:51:21.724 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:21.740 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:21.741 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:21.743 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:21.744 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:21.744 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:21.905 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:21.906 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:21.929 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:21.929 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:21.934 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:21.934 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:21.935 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 9 -lt 100 ']' 2023-11-16T16:51:21.935 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 9 + 1 2023-11-16T16:51:21.942 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=10 2023-11-16T16:51:21.942 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:21.953 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:21.953 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:21.956 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:21.956 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:21.957 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:22.211 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:22.211 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:22.314 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:22.314 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:22.330 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:22.330 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:22.330 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 10 -lt 100 ']' 2023-11-16T16:51:22.330 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 10 + 1 2023-11-16T16:51:22.362 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=11 2023-11-16T16:51:22.362 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:22.402 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:22.402 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:22.405 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:22.405 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:22.406 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:22.844 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:22.844 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:22.956 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:22.956 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:22.970 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:22.970 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:22.970 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 11 -lt 100 ']' 2023-11-16T16:51:22.971 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 11 + 1 2023-11-16T16:51:22.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:22 smithi077 ceph-mon[131204]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-11-16T16:51:23.004 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=12 2023-11-16T16:51:23.004 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:23.035 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:23.035 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:23.038 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:23.038 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:23.039 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:23.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:22 smithi044 ceph-mon[128750]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-11-16T16:51:23.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:22 smithi176 ceph-mon[131630]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-11-16T16:51:23.444 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:23.444 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:23.556 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:23.556 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:23.579 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:23.579 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:23.580 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 12 -lt 100 ']' 2023-11-16T16:51:23.580 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 12 + 1 2023-11-16T16:51:23.612 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=13 2023-11-16T16:51:23.612 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:23.634 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:23.635 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:23.637 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:23.638 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:23.638 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:24.043 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:24.044 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:24.180 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:24.180 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:24.204 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:24.204 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:24.204 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 13 -lt 100 ']' 2023-11-16T16:51:24.205 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 13 + 1 2023-11-16T16:51:24.242 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=14 2023-11-16T16:51:24.242 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:24.304 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:24.304 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:24.307 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:24.307 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:24.308 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:24.535 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:24 smithi044 ceph-mon[128750]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-11-16T16:51:24.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:24 smithi176 ceph-mon[131630]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-11-16T16:51:24.702 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:24.702 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:24.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:24 smithi077 ceph-mon[131204]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-11-16T16:51:24.830 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:24.830 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:24.854 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:24.855 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:24.855 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 14 -lt 100 ']' 2023-11-16T16:51:24.855 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 14 + 1 2023-11-16T16:51:24.887 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=15 2023-11-16T16:51:24.887 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:24.918 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:24.919 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:24.922 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:24.922 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:24.923 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:25.343 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:25.344 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:25.480 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:25.480 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:25.503 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:25.504 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:25.504 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 15 -lt 100 ']' 2023-11-16T16:51:25.504 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 15 + 1 2023-11-16T16:51:25.537 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=16 2023-11-16T16:51:25.537 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:25.560 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:25.560 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:25.562 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:25.563 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:25.563 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:25.743 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:51:26.077 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:26.077 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:26.154 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:26.154 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:26.161 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:26.161 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:26.161 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 16 -lt 100 ']' 2023-11-16T16:51:26.161 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 16 + 1 2023-11-16T16:51:26.193 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=17 2023-11-16T16:51:26.193 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:26.206 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:26.207 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:26.209 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:26.209 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:26.210 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:26.643 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:26.644 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:26.747 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:26.747 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:26.771 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:26.771 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:26.771 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 17 -lt 100 ']' 2023-11-16T16:51:26.772 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 17 + 1 2023-11-16T16:51:26.809 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=18 2023-11-16T16:51:26.809 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:26.841 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:26.841 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:26.843 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:26.844 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:26.844 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:27.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:26 smithi044 ceph-mon[128750]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-11-16T16:51:27.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:26 smithi176 ceph-mon[131630]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-11-16T16:51:27.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:26 smithi077 ceph-mon[131204]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-11-16T16:51:27.422 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:27.422 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:27.580 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:27.580 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:27.603 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:27.603 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:27.604 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 18 -lt 100 ']' 2023-11-16T16:51:27.604 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 18 + 1 2023-11-16T16:51:27.633 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=19 2023-11-16T16:51:27.633 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:27.687 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:27.688 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:27.690 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:27.690 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:27.691 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:28.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:27 smithi044 ceph-mon[128750]: mgrmap e25: x(active, since 91s), standbys: z, y 2023-11-16T16:51:28.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:27 smithi176 ceph-mon[131630]: mgrmap e25: x(active, since 91s), standbys: z, y 2023-11-16T16:51:28.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:27 smithi077 ceph-mon[131204]: mgrmap e25: x(active, since 91s), standbys: z, y 2023-11-16T16:51:28.585 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:28.585 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:28.797 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:28.797 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:28.856 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:28.856 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:28.856 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 19 -lt 100 ']' 2023-11-16T16:51:28.856 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 19 + 1 2023-11-16T16:51:28.903 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=20 2023-11-16T16:51:28.903 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:28.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.914+0000 7f6da75ff700 1 -- 172.21.15.44:0/2355380400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6da0105620 msgr2=0x7f6da010d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:28.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.914+0000 7f6da75ff700 1 --2- 172.21.15.44:0/2355380400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6da0105620 0x7f6da010d1a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6d9800b0d0 tx=0x7f6d9800b3e0 comp rx=0 tx=0).stop 2023-11-16T16:51:28.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.914+0000 7f6da75ff700 1 -- 172.21.15.44:0/2355380400 shutdown_connections 2023-11-16T16:51:28.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.914+0000 7f6da75ff700 1 --2- 172.21.15.44:0/2355380400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6da0105620 0x7f6da010d1a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:28.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.914+0000 7f6da75ff700 1 --2- 172.21.15.44:0/2355380400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6da0100cc0 0x7f6da0101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:28.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.914+0000 7f6da75ff700 1 --2- 172.21.15.44:0/2355380400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6da0101660 0x7f6da0104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:28.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.914+0000 7f6da75ff700 1 -- 172.21.15.44:0/2355380400 >> 172.21.15.44:0/2355380400 conn(0x7f6da0077560 msgr2=0x7f6da0077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:28.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.914+0000 7f6da75ff700 1 -- 172.21.15.44:0/2355380400 shutdown_connections 2023-11-16T16:51:28.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.914+0000 7f6da75ff700 1 -- 172.21.15.44:0/2355380400 wait complete. 2023-11-16T16:51:28.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.915+0000 7f6da75ff700 1 Processor -- start 2023-11-16T16:51:28.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.915+0000 7f6da75ff700 1 -- start start 2023-11-16T16:51:28.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.915+0000 7f6da75ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6da0100cc0 0x7f6da019e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:28.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.915+0000 7f6da75ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6da0101660 0x7f6da019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:28.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.916+0000 7f6da75ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6da0105620 0x7f6da0198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:28.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.916+0000 7f6da75ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da0198c90 con 0x7f6da0100cc0 2023-11-16T16:51:28.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.916+0000 7f6da75ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da0198e00 con 0x7f6da0101660 2023-11-16T16:51:28.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.916+0000 7f6da75ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da0198fc0 con 0x7f6da0105620 2023-11-16T16:51:28.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.916+0000 7f6da539b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6da0100cc0 0x7f6da019e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:28.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.916+0000 7f6da539b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6da0100cc0 0x7f6da019e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56956/0 (socket says 172.21.15.44:56956) 2023-11-16T16:51:28.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.916+0000 7f6da539b700 1 -- 172.21.15.44:0/977946302 learned_addr learned my addr 172.21.15.44:0/977946302 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:28.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.916+0000 7f6da4b9a700 1 --2- 172.21.15.44:0/977946302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6da0101660 0x7f6da019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:28.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.916+0000 7f6da5b9c700 1 --2- 172.21.15.44:0/977946302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6da0105620 0x7f6da0198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:28.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.916+0000 7f6da539b700 1 -- 172.21.15.44:0/977946302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6da0105620 msgr2=0x7f6da0198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:28.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.916+0000 7f6da539b700 1 --2- 172.21.15.44:0/977946302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6da0105620 0x7f6da0198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:28.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.916+0000 7f6da539b700 1 -- 172.21.15.44:0/977946302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6da0101660 msgr2=0x7f6da019eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:28.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.916+0000 7f6da539b700 1 --2- 172.21.15.44:0/977946302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6da0101660 0x7f6da019eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:28.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.916+0000 7f6da539b700 1 -- 172.21.15.44:0/977946302 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d98009d00 con 0x7f6da0100cc0 2023-11-16T16:51:28.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.916+0000 7f6da4b9a700 1 --2- 172.21.15.44:0/977946302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6da0101660 0x7f6da019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:51:28.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.916+0000 7f6da539b700 1 --2- 172.21.15.44:0/977946302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6da0100cc0 0x7f6da019e5b0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c00d980 tx=0x7f6d9c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:28.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.917+0000 7f6d967fc700 1 -- 172.21.15.44:0/977946302 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c00bb60 con 0x7f6da0100cc0 2023-11-16T16:51:28.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.917+0000 7f6d967fc700 1 -- 172.21.15.44:0/977946302 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d9c007640 con 0x7f6da0100cc0 2023-11-16T16:51:28.923 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.917+0000 7f6da75ff700 1 -- 172.21.15.44:0/977946302 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da01103e0 con 0x7f6da0100cc0 2023-11-16T16:51:28.923 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.917+0000 7f6d967fc700 1 -- 172.21.15.44:0/977946302 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c004e90 con 0x7f6da0100cc0 2023-11-16T16:51:28.923 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.917+0000 7f6da75ff700 1 -- 172.21.15.44:0/977946302 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da0110250 con 0x7f6da0100cc0 2023-11-16T16:51:28.924 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.920+0000 7f6d967fc700 1 -- 172.21.15.44:0/977946302 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f6d9c00bcc0 con 0x7f6da0100cc0 2023-11-16T16:51:28.924 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.921+0000 7f6d967fc700 1 --2- 172.21.15.44:0/977946302 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6d8c0a1340 0x7f6d8c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:28.924 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.921+0000 7f6d967fc700 1 -- 172.21.15.44:0/977946302 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f6d9c0ccf10 con 0x7f6da0100cc0 2023-11-16T16:51:28.924 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.921+0000 7f6da4b9a700 1 --2- 172.21.15.44:0/977946302 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6d8c0a1340 0x7f6d8c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:28.924 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.922+0000 7f6da4b9a700 1 --2- 172.21.15.44:0/977946302 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6d8c0a1340 0x7f6d8c0a37f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6da0106a40 tx=0x7f6d90009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:28.944 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:28.944 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:28.947 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:28.948 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:28.948 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:28.992 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.990+0000 7f6dbae62b80 1 Processor -- start 2023-11-16T16:51:28.993 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.991+0000 7f6dbae62b80 1 -- start start 2023-11-16T16:51:28.993 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.991+0000 7f6dbae62b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666ab60 0x5648c666af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:28.993 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.991+0000 7f6dbae62b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666b470 0x5648c666f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:28.993 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.991+0000 7f6dbae62b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666f8b0 0x5648c666fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:28.993 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.991+0000 7f6dbae62b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c6570060 con 0x5648c666f8b0 2023-11-16T16:51:28.993 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.991+0000 7f6dbae62b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c65c28a0 con 0x5648c666b470 2023-11-16T16:51:28.993 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.991+0000 7f6dbae62b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c65bbd30 con 0x5648c666ab60 2023-11-16T16:51:28.999 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.991+0000 7f6da5b9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666f8b0 0x5648c666fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:28.999 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.991+0000 7f6da4b9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666b470 0x5648c666f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:28.999 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.991+0000 7f6da539b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666ab60 0x5648c666af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:28.999 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.991+0000 7f6da5b9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666f8b0 0x5648c666fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56960/0 (socket says 172.21.15.44:56960) 2023-11-16T16:51:28.999 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.991+0000 7f6da5b9c700 1 -- 172.21.15.44:0/2151019924 learned_addr learned my addr 172.21.15.44:0/2151019924 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:28.999 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.992+0000 7f6da539b700 1 -- 172.21.15.44:0/2151019924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666b470 msgr2=0x5648c666f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:28.999 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.992+0000 7f6da539b700 1 --2- 172.21.15.44:0/2151019924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666b470 0x5648c666f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.000 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.992+0000 7f6da539b700 1 -- 172.21.15.44:0/2151019924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666f8b0 msgr2=0x5648c666fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:29.000 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.992+0000 7f6da539b700 1 --2- 172.21.15.44:0/2151019924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666f8b0 0x5648c666fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.000 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.992+0000 7f6da539b700 1 -- 172.21.15.44:0/2151019924 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648c643d630 con 0x5648c666ab60 2023-11-16T16:51:29.000 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.992+0000 7f6da5b9c700 1 --2- 172.21.15.44:0/2151019924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666f8b0 0x5648c666fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:29.000 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.992+0000 7f6da4b9a700 1 --2- 172.21.15.44:0/2151019924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666b470 0x5648c666f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:29.000 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.992+0000 7f6da539b700 1 --2- 172.21.15.44:0/2151019924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666ab60 0x5648c666af30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c0ca280 tx=0x7f6d9c004820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98e6cb9bccea177a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:29.000 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.992+0000 7f6d8b7fe700 1 -- 172.21.15.44:0/2151019924 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c099c00 con 0x5648c666ab60 2023-11-16T16:51:29.000 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.992+0000 7f6d8b7fe700 1 -- 172.21.15.44:0/2151019924 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d9c09dc30 con 0x5648c666ab60 2023-11-16T16:51:29.001 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.992+0000 7f6d8b7fe700 1 -- 172.21.15.44:0/2151019924 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c0990f0 con 0x5648c666ab60 2023-11-16T16:51:29.001 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.993+0000 7f6dbae62b80 1 -- 172.21.15.44:0/2151019924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666ab60 msgr2=0x5648c666af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:29.001 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.993+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/2151019924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666ab60 0x5648c666af30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c0ca280 tx=0x7f6d9c004820 comp rx=0 tx=0).stop 2023-11-16T16:51:29.001 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.993+0000 7f6dbae62b80 1 -- 172.21.15.44:0/2151019924 shutdown_connections 2023-11-16T16:51:29.001 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.993+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/2151019924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666f8b0 0x5648c666fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.001 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.993+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/2151019924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666ab60 0x5648c666af30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.001 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.993+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/2151019924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666b470 0x5648c666f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.001 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.993+0000 7f6dbae62b80 1 -- 172.21.15.44:0/2151019924 >> 172.21.15.44:0/2151019924 conn(0x5648c6660360 msgr2=0x5648c6665b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:29.002 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.993+0000 7f6dbae62b80 1 -- 172.21.15.44:0/2151019924 shutdown_connections 2023-11-16T16:51:29.002 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.993+0000 7f6dbae62b80 1 -- 172.21.15.44:0/2151019924 wait complete. 2023-11-16T16:51:29.002 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.994+0000 7f6dbae62b80 1 -- start start 2023-11-16T16:51:29.002 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.994+0000 7f6dbae62b80 1 Processor -- start 2023-11-16T16:51:29.002 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.994+0000 7f6dbae62b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666ab60 0x5648c6674380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:29.002 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.994+0000 7f6dbae62b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c66748c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:29.002 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.994+0000 7f6dbae62b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666f8b0 0x5648c6674e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:29.002 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.994+0000 7f6dbae62b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c655d8e0 con 0x5648c666b470 2023-11-16T16:51:29.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.994+0000 7f6dbae62b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c655d580 con 0x5648c666f8b0 2023-11-16T16:51:29.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.994+0000 7f6dbae62b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c643f920 con 0x5648c666ab60 2023-11-16T16:51:29.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.994+0000 7f6dbae62b80 10 client.0 _ll_register_callbacks cb 0x5648c657b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:29.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.995+0000 7f6da4b9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c66748c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:29.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.995+0000 7f6da4b9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c66748c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56976/0 (socket says 172.21.15.44:56976) 2023-11-16T16:51:29.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.995+0000 7f6da4b9a700 1 -- 172.21.15.44:0/356273267 learned_addr learned my addr 172.21.15.44:0/356273267 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:29.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.995+0000 7f6da5b9c700 1 --2- 172.21.15.44:0/356273267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666f8b0 0x5648c6674e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:29.003 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.995+0000 7f6da539b700 1 --2- 172.21.15.44:0/356273267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666ab60 0x5648c6674380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:29.004 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.995+0000 7f6da4b9a700 1 -- 172.21.15.44:0/356273267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666ab60 msgr2=0x5648c6674380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:29.004 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.995+0000 7f6da4b9a700 1 --2- 172.21.15.44:0/356273267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666ab60 0x5648c6674380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.004 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.995+0000 7f6da4b9a700 1 -- 172.21.15.44:0/356273267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666f8b0 msgr2=0x5648c6674e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:29.004 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.995+0000 7f6da4b9a700 1 --2- 172.21.15.44:0/356273267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666f8b0 0x5648c6674e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.004 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.995+0000 7f6da4b9a700 1 -- 172.21.15.44:0/356273267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da0110250 con 0x5648c666b470 2023-11-16T16:51:29.004 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.995+0000 7f6da5b9c700 1 --2- 172.21.15.44:0/356273267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666f8b0 0x5648c6674e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:29.004 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.995+0000 7f6da4b9a700 1 --2- 172.21.15.44:0/356273267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c66748c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6d90011c00 tx=0x7f6d900047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7910e478a3cdba68 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:29.004 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.995+0000 7f6d88ff9700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:51:29.004 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.996+0000 7f6d88ff9700 1 -- 172.21.15.44:0/356273267 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9001f840 con 0x5648c666b470 2023-11-16T16:51:29.005 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.996+0000 7f6d88ff9700 1 -- 172.21.15.44:0/356273267 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f6d90017070 con 0x5648c666b470 2023-11-16T16:51:29.005 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.996+0000 7f6dbae62b80 1 -- 172.21.15.44:0/356273267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5648c665db20 con 0x5648c666b470 2023-11-16T16:51:29.005 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.996+0000 7f6d88ff9700 1 -- 172.21.15.44:0/356273267 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d9001f9a0 con 0x5648c666b470 2023-11-16T16:51:29.005 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.996+0000 7f6dbae62b80 10 client.15834 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:29.005 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.996+0000 7f6dbae62b80 1 -- 172.21.15.44:0/356273267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5648c6582190 con 0x5648c666b470 2023-11-16T16:51:29.005 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.996+0000 7f6d88ff9700 1 -- 172.21.15.44:0/356273267 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5648c665db20 con 0x5648c666b470 2023-11-16T16:51:29.005 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.996+0000 7f6d88ff9700 1 -- 172.21.15.44:0/356273267 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6d90028af0 con 0x5648c666b470 2023-11-16T16:51:29.005 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.997+0000 7f6dbae62b80 10 client.15834 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:29.005 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.997+0000 7f6dbae62b80 10 client.15834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:29.006 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.997+0000 7f6dbae62b80 20 client.15834 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:29.006 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.997+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/356273267 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5648c6684a60 0x5648c6686e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:29.006 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.997+0000 7f6dbae62b80 4 client.15834 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:51:29.006 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.997+0000 7f6dbae62b80 1 -- 172.21.15.44:0/356273267 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5648c657e0a0 con 0x5648c6684a60 2023-11-16T16:51:29.006 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.997+0000 7f6da539b700 1 --2- 172.21.15.44:0/356273267 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5648c6684a60 0x5648c6686e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:29.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.997+0000 7f6da539b700 1 --2- 172.21.15.44:0/356273267 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5648c6684a60 0x5648c6686e70 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34a0cd677f980fb3 server_cookie=b4a003b21827edeb in_seq=0 out_seq=0 2023-11-16T16:51:29.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.997+0000 7f6d88ff9700 10 client.15834 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:29.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.998+0000 7f6d88ff9700 1 -- 172.21.15.44:0/356273267 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5648c657e0a0 con 0x5648c6684a60 2023-11-16T16:51:29.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.998+0000 7f6d88ff9700 10 client.15834 handle_command_reply: tid=0 2023-11-16T16:51:29.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.998+0000 7f6dbae62b80 1 client.15834 shutdown 2023-11-16T16:51:29.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.999+0000 7f6dbae62b80 1 -- 172.21.15.44:0/356273267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 msgr2=0x5648c66748c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:29.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.999+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/356273267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c66748c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6d90011c00 tx=0x7f6d900047c0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.999+0000 7f6dbae62b80 1 -- 172.21.15.44:0/356273267 shutdown_connections 2023-11-16T16:51:29.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.999+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/356273267 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5648c6684a60 0x5648c6686e70 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.999+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/356273267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666f8b0 0x5648c6674e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.999+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/356273267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666ab60 0x5648c6674380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.999+0000 7f6d88ff9700 0 client.15834 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:29.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.999+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/356273267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c66748c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.999+0000 7f6dbae62b80 1 -- 172.21.15.44:0/356273267 >> 172.21.15.44:0/356273267 conn(0x5648c6660360 msgr2=0x5648c6670d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:29.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.999+0000 7f6dbae62b80 1 -- 172.21.15.44:0/356273267 shutdown_connections 2023-11-16T16:51:29.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.999+0000 7f6dbae62b80 1 -- 172.21.15.44:0/356273267 wait complete. 2023-11-16T16:51:29.010 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:28.999+0000 7f6dbae62b80 20 client.15834 trim_cache size 0 max 16384 2023-11-16T16:51:29.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.017+0000 7f6dbae62b80 1 Processor -- start 2023-11-16T16:51:29.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.017+0000 7f6dbae62b80 1 -- start start 2023-11-16T16:51:29.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.017+0000 7f6dbae62b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666ab60 0x5648c6665be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:29.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.017+0000 7f6dbae62b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c667b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:29.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.017+0000 7f6dbae62b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666f8b0 0x5648c6674c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:29.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.017+0000 7f6dbae62b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c666b840 con 0x5648c666b470 2023-11-16T16:51:29.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.017+0000 7f6dbae62b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c6677730 con 0x5648c666ab60 2023-11-16T16:51:29.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.017+0000 7f6dbae62b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c665fee0 con 0x5648c666f8b0 2023-11-16T16:51:29.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.018+0000 7f6da4b9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c667b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:29.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.018+0000 7f6da4b9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c667b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56992/0 (socket says 172.21.15.44:56992) 2023-11-16T16:51:29.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.018+0000 7f6da4b9a700 1 -- 172.21.15.44:0/2458780472 learned_addr learned my addr 172.21.15.44:0/2458780472 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:29.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.018+0000 7f6da5b9c700 1 --2- 172.21.15.44:0/2458780472 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666f8b0 0x5648c6674c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:29.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.018+0000 7f6da539b700 1 --2- 172.21.15.44:0/2458780472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666ab60 0x5648c6665be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:29.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.018+0000 7f6da4b9a700 1 -- 172.21.15.44:0/2458780472 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666f8b0 msgr2=0x5648c6674c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:29.022 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.018+0000 7f6da4b9a700 1 --2- 172.21.15.44:0/2458780472 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666f8b0 0x5648c6674c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.022 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.018+0000 7f6da4b9a700 1 -- 172.21.15.44:0/2458780472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666ab60 msgr2=0x5648c6665be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:29.022 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.018+0000 7f6da4b9a700 1 --2- 172.21.15.44:0/2458780472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666ab60 0x5648c6665be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.022 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.018+0000 7f6da4b9a700 1 -- 172.21.15.44:0/2458780472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648c66701a0 con 0x5648c666b470 2023-11-16T16:51:29.022 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.018+0000 7f6da4b9a700 1 --2- 172.21.15.44:0/2458780472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c667b9a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6d90028f00 tx=0x7f6d900046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f2ad92c8f8646a3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:29.022 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.018+0000 7f6d8bfff700 1 -- 172.21.15.44:0/2458780472 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9001e5a0 con 0x5648c666b470 2023-11-16T16:51:29.022 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.019+0000 7f6d8bfff700 1 -- 172.21.15.44:0/2458780472 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d90015070 con 0x5648c666b470 2023-11-16T16:51:29.022 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.019+0000 7f6d8bfff700 1 -- 172.21.15.44:0/2458780472 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9001f8a0 con 0x5648c666b470 2023-11-16T16:51:29.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.019+0000 7f6dbae62b80 1 -- 172.21.15.44:0/2458780472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 msgr2=0x5648c667b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:29.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.019+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/2458780472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c667b9a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6d90028f00 tx=0x7f6d900046d0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.019+0000 7f6dbae62b80 1 -- 172.21.15.44:0/2458780472 shutdown_connections 2023-11-16T16:51:29.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.019+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/2458780472 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666f8b0 0x5648c6674c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.019+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/2458780472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666ab60 0x5648c6665be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.019+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/2458780472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c667b9a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.019+0000 7f6dbae62b80 1 -- 172.21.15.44:0/2458780472 >> 172.21.15.44:0/2458780472 conn(0x5648c6660360 msgr2=0x5648c666d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:29.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.019+0000 7f6dbae62b80 1 -- 172.21.15.44:0/2458780472 shutdown_connections 2023-11-16T16:51:29.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.019+0000 7f6dbae62b80 1 -- 172.21.15.44:0/2458780472 wait complete. 2023-11-16T16:51:29.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.020+0000 7f6dbae62b80 1 -- start start 2023-11-16T16:51:29.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.020+0000 7f6dbae62b80 1 Processor -- start 2023-11-16T16:51:29.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6dbae62b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666ab60 0x5648c6679990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:29.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6dbae62b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c6679ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:29.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6dbae62b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666f8b0 0x5648c667a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:29.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6dbae62b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c66d79c0 con 0x5648c666b470 2023-11-16T16:51:29.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6dbae62b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c66d7b00 con 0x5648c666f8b0 2023-11-16T16:51:29.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6dbae62b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c66d7cc0 con 0x5648c666ab60 2023-11-16T16:51:29.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6dbae62b80 10 client.0 _ll_register_callbacks cb 0x7f6d900117e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:29.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6da4b9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c6679ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:29.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6da5b9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666f8b0 0x5648c667a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:29.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6da4b9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c6679ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57000/0 (socket says 172.21.15.44:57000) 2023-11-16T16:51:29.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6da4b9a700 1 -- 172.21.15.44:0/1591411726 learned_addr learned my addr 172.21.15.44:0/1591411726 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:29.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6da539b700 1 --2- 172.21.15.44:0/1591411726 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666ab60 0x5648c6679990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:29.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6da4b9a700 1 -- 172.21.15.44:0/1591411726 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666ab60 msgr2=0x5648c6679990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:29.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6da4b9a700 1 --2- 172.21.15.44:0/1591411726 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666ab60 0x5648c6679990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6da4b9a700 1 -- 172.21.15.44:0/1591411726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666f8b0 msgr2=0x5648c667a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:29.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6da4b9a700 1 --2- 172.21.15.44:0/1591411726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666f8b0 0x5648c667a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6da4b9a700 1 -- 172.21.15.44:0/1591411726 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5648c6582190 con 0x5648c666b470 2023-11-16T16:51:29.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.021+0000 7f6da5b9c700 1 --2- 172.21.15.44:0/1591411726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666f8b0 0x5648c667a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:29.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.022+0000 7f6da4b9a700 1 --2- 172.21.15.44:0/1591411726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c6679ed0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6d90009b50 tx=0x7f6d9001ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9dd063a34163631 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:29.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.022+0000 7f6d88ff9700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:51:29.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.022+0000 7f6d88ff9700 1 -- 172.21.15.44:0/1591411726 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d90010ce0 con 0x5648c666b470 2023-11-16T16:51:29.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.022+0000 7f6dbae62b80 1 -- 172.21.15.44:0/1591411726 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d9c09dc30 con 0x5648c666b470 2023-11-16T16:51:29.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.022+0000 7f6d88ff9700 1 -- 172.21.15.44:0/1591411726 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f6d90017040 con 0x5648c666b470 2023-11-16T16:51:29.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.022+0000 7f6d88ff9700 1 -- 172.21.15.44:0/1591411726 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d9001e9f0 con 0x5648c666b470 2023-11-16T16:51:29.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.022+0000 7f6dbae62b80 10 client.15840 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:29.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.022+0000 7f6dbae62b80 1 -- 172.21.15.44:0/1591411726 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5648c63e9de0 con 0x5648c666b470 2023-11-16T16:51:29.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.022+0000 7f6d88ff9700 1 -- 172.21.15.44:0/1591411726 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d9c09dc30 con 0x5648c666b470 2023-11-16T16:51:29.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.023+0000 7f6d88ff9700 1 -- 172.21.15.44:0/1591411726 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6d90004c30 con 0x5648c666b470 2023-11-16T16:51:29.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.023+0000 7f6dbae62b80 10 client.15840 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:29.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.023+0000 7f6dbae62b80 10 client.15840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:29.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.023+0000 7f6dbae62b80 20 client.15840 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:29.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.023+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/1591411726 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5648c66d8f60 0x5648c66db370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:29.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.023+0000 7f6dbae62b80 4 client.15840 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:51:29.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.023+0000 7f6dbae62b80 1 -- 172.21.15.44:0/1591411726 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5648c658a660 con 0x5648c66d8f60 2023-11-16T16:51:29.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.023+0000 7f6da539b700 1 --2- 172.21.15.44:0/1591411726 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5648c66d8f60 0x5648c66db370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:29.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.024+0000 7f6da539b700 1 --2- 172.21.15.44:0/1591411726 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5648c66d8f60 0x5648c66db370 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f1df13bf11b2991 server_cookie=8db6a0b96b0bbf07 in_seq=0 out_seq=0 2023-11-16T16:51:29.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.024+0000 7f6d88ff9700 10 client.15840 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:29.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.024+0000 7f6d88ff9700 1 -- 172.21.15.44:0/1591411726 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5648c658a660 con 0x5648c66d8f60 2023-11-16T16:51:29.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.024+0000 7f6d88ff9700 10 client.15840 handle_command_reply: tid=0 2023-11-16T16:51:29.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.024+0000 7f6dbae62b80 1 client.15840 shutdown 2023-11-16T16:51:29.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.024+0000 7f6dbae62b80 1 -- 172.21.15.44:0/1591411726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 msgr2=0x5648c6679ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:29.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.024+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/1591411726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c6679ed0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6d90009b50 tx=0x7f6d9001ee50 comp rx=0 tx=0).stop 2023-11-16T16:51:29.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.024+0000 7f6dbae62b80 1 -- 172.21.15.44:0/1591411726 shutdown_connections 2023-11-16T16:51:29.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.024+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/1591411726 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5648c66d8f60 0x5648c66db370 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.024+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/1591411726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648c666f8b0 0x5648c667a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.024+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/1591411726 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5648c666ab60 0x5648c6679990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.024+0000 7f6dbae62b80 1 --2- 172.21.15.44:0/1591411726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648c666b470 0x5648c6679ed0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.024+0000 7f6dbae62b80 1 -- 172.21.15.44:0/1591411726 >> 172.21.15.44:0/1591411726 conn(0x5648c6660360 msgr2=0x5648c666e910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:29.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.024+0000 7f6d88ff9700 0 client.15840 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:29.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.024+0000 7f6dbae62b80 1 -- 172.21.15.44:0/1591411726 shutdown_connections 2023-11-16T16:51:29.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.024+0000 7f6dbae62b80 1 -- 172.21.15.44:0/1591411726 wait complete. 2023-11-16T16:51:29.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.025+0000 7f6dbae62b80 20 client.15840 trim_cache size 0 max 16384 2023-11-16T16:51:29.034 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:51:29.035 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T16:51:29.035 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T16:51:29.035 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:51:29.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.025+0000 7f6da75ff700 1 -- 172.21.15.44:0/977946302 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6d8c0a1340 msgr2=0x7f6d8c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:29.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.025+0000 7f6da75ff700 1 --2- 172.21.15.44:0/977946302 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6d8c0a1340 0x7f6d8c0a37f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6da0106a40 tx=0x7f6d90009260 comp rx=0 tx=0).stop 2023-11-16T16:51:29.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.025+0000 7f6da75ff700 1 -- 172.21.15.44:0/977946302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6da0100cc0 msgr2=0x7f6da019e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:29.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.025+0000 7f6da75ff700 1 --2- 172.21.15.44:0/977946302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6da0100cc0 0x7f6da019e5b0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c00d980 tx=0x7f6d9c00dd40 comp rx=0 tx=0).stop 2023-11-16T16:51:29.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.025+0000 7f6da75ff700 1 -- 172.21.15.44:0/977946302 shutdown_connections 2023-11-16T16:51:29.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.025+0000 7f6da75ff700 1 --2- 172.21.15.44:0/977946302 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6d8c0a1340 0x7f6d8c0a37f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.025+0000 7f6da75ff700 1 --2- 172.21.15.44:0/977946302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6da0105620 0x7f6da0198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.025+0000 7f6da75ff700 1 --2- 172.21.15.44:0/977946302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6da0100cc0 0x7f6da019e5b0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.025+0000 7f6da75ff700 1 --2- 172.21.15.44:0/977946302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6da0101660 0x7f6da019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:29.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.025+0000 7f6da75ff700 1 -- 172.21.15.44:0/977946302 >> 172.21.15.44:0/977946302 conn(0x7f6da0077560 msgr2=0x7f6da00feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:29.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.025+0000 7f6da75ff700 1 -- 172.21.15.44:0/977946302 shutdown_connections 2023-11-16T16:51:29.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:29.025+0000 7f6da75ff700 1 -- 172.21.15.44:0/977946302 wait complete. 2023-11-16T16:51:29.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:28 smithi044 ceph-mon[128750]: pgmap v49: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-11-16T16:51:29.064 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T16:51:29.064 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:51:29.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:28 smithi176 ceph-mon[131630]: pgmap v49: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-11-16T16:51:29.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:28 smithi077 ceph-mon[131204]: pgmap v49: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-11-16T16:51:29.501 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:29.502 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:29.613 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:29.614 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:29.645 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:29.646 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:29.646 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 20 -lt 100 ']' 2023-11-16T16:51:29.646 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 20 + 1 2023-11-16T16:51:29.675 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=21 2023-11-16T16:51:29.675 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:29.704 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:29.705 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:29.707 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:29.708 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:29.708 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:30.314 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:30.314 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:30.497 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:30.497 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:30.521 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:30.521 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:30.522 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 21 -lt 100 ']' 2023-11-16T16:51:30.522 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 21 + 1 2023-11-16T16:51:30.558 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=22 2023-11-16T16:51:30.558 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:30.604 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:30.604 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:30.606 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:30.607 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:30.607 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:31.084 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:30 smithi044 ceph-mon[128750]: pgmap v50: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-11-16T16:51:31.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:30 smithi176 ceph-mon[131630]: pgmap v50: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-11-16T16:51:31.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:30 smithi077 ceph-mon[131204]: pgmap v50: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-11-16T16:51:31.360 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:31.360 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:31.590 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.588+0000 7f7f639ff700 1 -- 172.21.15.44:0/3215956373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f5c100550 msgr2=0x7f7f5c100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:31.591 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.588+0000 7f7f639ff700 1 --2- 172.21.15.44:0/3215956373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f5c100550 0x7f7f5c100920 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7f580099c0 tx=0x7f7f58009cd0 comp rx=0 tx=0).stop 2023-11-16T16:51:31.591 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.588+0000 7f7f639ff700 1 -- 172.21.15.44:0/3215956373 shutdown_connections 2023-11-16T16:51:31.591 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.588+0000 7f7f639ff700 1 --2- 172.21.15.44:0/3215956373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c105540 0x7f7f5c10d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:31.591 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.588+0000 7f7f639ff700 1 --2- 172.21.15.44:0/3215956373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f5c100550 0x7f7f5c100920 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:31.591 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.588+0000 7f7f639ff700 1 --2- 172.21.15.44:0/3215956373 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f5c100ef0 0x7f7f5c104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:31.591 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.588+0000 7f7f639ff700 1 -- 172.21.15.44:0/3215956373 >> 172.21.15.44:0/3215956373 conn(0x7f7f5c077020 msgr2=0x7f7f5c077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:31.591 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.589+0000 7f7f639ff700 1 -- 172.21.15.44:0/3215956373 shutdown_connections 2023-11-16T16:51:31.591 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.589+0000 7f7f639ff700 1 -- 172.21.15.44:0/3215956373 wait complete. 2023-11-16T16:51:31.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.590+0000 7f7f639ff700 1 Processor -- start 2023-11-16T16:51:31.593 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.590+0000 7f7f639ff700 1 -- start start 2023-11-16T16:51:31.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.590+0000 7f7f639ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f5c100550 0x7f7f5c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:31.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.590+0000 7f7f639ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c100ef0 0x7f7f5c198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:31.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.590+0000 7f7f639ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f5c105540 0x7f7f5c19cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:31.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.590+0000 7f7f639ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c19d580 con 0x7f7f5c100550 2023-11-16T16:51:31.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.590+0000 7f7f639ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c19d6f0 con 0x7f7f5c100ef0 2023-11-16T16:51:31.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.590+0000 7f7f639ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c19d8b0 con 0x7f7f5c105540 2023-11-16T16:51:31.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.590+0000 7f7f6179b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f5c100550 0x7f7f5c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:31.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.590+0000 7f7f6179b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f5c100550 0x7f7f5c1985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57014/0 (socket says 172.21.15.44:57014) 2023-11-16T16:51:31.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.590+0000 7f7f6179b700 1 -- 172.21.15.44:0/255592417 learned_addr learned my addr 172.21.15.44:0/255592417 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:31.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.590+0000 7f7f60f9a700 1 --2- 172.21.15.44:0/255592417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c100ef0 0x7f7f5c198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:31.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.591+0000 7f7f61f9c700 1 --2- 172.21.15.44:0/255592417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f5c105540 0x7f7f5c19cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:31.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.591+0000 7f7f6179b700 1 -- 172.21.15.44:0/255592417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f5c105540 msgr2=0x7f7f5c19cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:31.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.591+0000 7f7f6179b700 1 --2- 172.21.15.44:0/255592417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f5c105540 0x7f7f5c19cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:31.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.591+0000 7f7f6179b700 1 -- 172.21.15.44:0/255592417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c100ef0 msgr2=0x7f7f5c198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:31.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.591+0000 7f7f6179b700 1 --2- 172.21.15.44:0/255592417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c100ef0 0x7f7f5c198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:31.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.591+0000 7f7f6179b700 1 -- 172.21.15.44:0/255592417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f580095f0 con 0x7f7f5c100550 2023-11-16T16:51:31.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.591+0000 7f7f61f9c700 1 --2- 172.21.15.44:0/255592417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f5c105540 0x7f7f5c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:51:31.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.591+0000 7f7f60f9a700 1 --2- 172.21.15.44:0/255592417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c100ef0 0x7f7f5c198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:31.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.591+0000 7f7f6179b700 1 --2- 172.21.15.44:0/255592417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f5c100550 0x7f7f5c1985a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7f580098e0 tx=0x7f7f5800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:31.596 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.591+0000 7f7f527fc700 1 -- 172.21.15.44:0/255592417 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5801d070 con 0x7f7f5c100550 2023-11-16T16:51:31.596 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.591+0000 7f7f527fc700 1 -- 172.21.15.44:0/255592417 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f5800fb80 con 0x7f7f5c100550 2023-11-16T16:51:31.596 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.591+0000 7f7f639ff700 1 -- 172.21.15.44:0/255592417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5c110350 con 0x7f7f5c100550 2023-11-16T16:51:31.596 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.591+0000 7f7f527fc700 1 -- 172.21.15.44:0/255592417 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f580173f0 con 0x7f7f5c100550 2023-11-16T16:51:31.596 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.592+0000 7f7f639ff700 1 -- 172.21.15.44:0/255592417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f5c1101c0 con 0x7f7f5c100550 2023-11-16T16:51:31.596 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.593+0000 7f7f639ff700 1 -- 172.21.15.44:0/255592417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f5c1104e0 con 0x7f7f5c100550 2023-11-16T16:51:31.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.594+0000 7f7f527fc700 1 -- 172.21.15.44:0/255592417 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f7f5800fcf0 con 0x7f7f5c100550 2023-11-16T16:51:31.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.596+0000 7f7f527fc700 1 --2- 172.21.15.44:0/255592417 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7f480a12c0 0x7f7f480a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:31.598 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.596+0000 7f7f527fc700 1 -- 172.21.15.44:0/255592417 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f7f580cdd30 con 0x7f7f5c100550 2023-11-16T16:51:31.598 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.596+0000 7f7f60f9a700 1 --2- 172.21.15.44:0/255592417 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7f480a12c0 0x7f7f480a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:31.598 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.597+0000 7f7f60f9a700 1 --2- 172.21.15.44:0/255592417 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7f480a12c0 0x7f7f480a3770 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c199bb0 tx=0x7f7f4c009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:31.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.607+0000 7f7f527fc700 1 -- 172.21.15.44:0/255592417 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7f58096df0 con 0x7f7f5c100550 2023-11-16T16:51:31.655 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:31.655 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:31.688 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:31.688 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:31.688 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 22 -lt 100 ']' 2023-11-16T16:51:31.689 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 22 + 1 2023-11-16T16:51:31.719 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=23 2023-11-16T16:51:31.720 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:31.748 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:31.749 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:31.751 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:31.752 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:31.752 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:31.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.869+0000 7f7f639ff700 1 -- 172.21.15.44:0/255592417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f5c067a60 con 0x7f7f5c100550 2023-11-16T16:51:31.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.870+0000 7f7f527fc700 1 -- 172.21.15.44:0/255592417 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f7f5809a410 con 0x7f7f5c100550 2023-11-16T16:51:31.872 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:51:31.872 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:51:31.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.872+0000 7f7f639ff700 1 -- 172.21.15.44:0/255592417 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7f480a12c0 msgr2=0x7f7f480a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:31.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.872+0000 7f7f639ff700 1 --2- 172.21.15.44:0/255592417 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7f480a12c0 0x7f7f480a3770 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c199bb0 tx=0x7f7f4c009260 comp rx=0 tx=0).stop 2023-11-16T16:51:31.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.872+0000 7f7f639ff700 1 -- 172.21.15.44:0/255592417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f5c100550 msgr2=0x7f7f5c1985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:31.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.872+0000 7f7f639ff700 1 --2- 172.21.15.44:0/255592417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f5c100550 0x7f7f5c1985a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7f580098e0 tx=0x7f7f5800f690 comp rx=0 tx=0).stop 2023-11-16T16:51:31.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.873+0000 7f7f639ff700 1 -- 172.21.15.44:0/255592417 shutdown_connections 2023-11-16T16:51:31.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.873+0000 7f7f639ff700 1 --2- 172.21.15.44:0/255592417 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7f480a12c0 0x7f7f480a3770 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:31.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.873+0000 7f7f639ff700 1 --2- 172.21.15.44:0/255592417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f5c105540 0x7f7f5c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:31.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.873+0000 7f7f639ff700 1 --2- 172.21.15.44:0/255592417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7f5c100550 0x7f7f5c1985a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:31.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.873+0000 7f7f639ff700 1 --2- 172.21.15.44:0/255592417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c100ef0 0x7f7f5c198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:31.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.873+0000 7f7f639ff700 1 -- 172.21.15.44:0/255592417 >> 172.21.15.44:0/255592417 conn(0x7f7f5c077020 msgr2=0x7f7f5c0fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:31.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.873+0000 7f7f639ff700 1 -- 172.21.15.44:0/255592417 shutdown_connections 2023-11-16T16:51:31.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:31.873+0000 7f7f639ff700 1 -- 172.21.15.44:0/255592417 wait complete. 2023-11-16T16:51:31.876 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:51:31.905 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T16:51:32.318 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:32.318 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:32.367 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:32.367 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:32.377 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:32.377 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:32.377 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 23 -lt 100 ']' 2023-11-16T16:51:32.377 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 23 + 1 2023-11-16T16:51:32.416 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=24 2023-11-16T16:51:32.416 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:32.431 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:32.432 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:32.434 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:32.435 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:32.436 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:33.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:32 smithi044 ceph-mon[128750]: pgmap v51: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-11-16T16:51:33.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:32 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/255592417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:51:33.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:32 smithi176 ceph-mon[131630]: pgmap v51: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-11-16T16:51:33.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:32 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/255592417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:51:33.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:32 smithi077 ceph-mon[131204]: pgmap v51: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-11-16T16:51:33.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:32 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/255592417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:51:33.234 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:33.234 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:33.463 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:33.463 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:33.479 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:33.479 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:33.479 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 24 -lt 100 ']' 2023-11-16T16:51:33.479 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 24 + 1 2023-11-16T16:51:33.533 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=25 2023-11-16T16:51:33.533 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:33.567 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:33.568 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:33.570 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:33.571 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:33.571 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:34.309 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:34.309 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:34.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:34 smithi044 ceph-mon[128750]: pgmap v52: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 3 op/s 2023-11-16T16:51:34.513 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:34.513 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:34.547 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:34.547 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:34.547 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 25 -lt 100 ']' 2023-11-16T16:51:34.547 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 25 + 1 2023-11-16T16:51:34.591 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=26 2023-11-16T16:51:34.591 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:34.634 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:34.635 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:34.637 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:34.637 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:34.638 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:34.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:34 smithi176 ceph-mon[131630]: pgmap v52: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 3 op/s 2023-11-16T16:51:34.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:34 smithi077 ceph-mon[131204]: pgmap v52: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 3 op/s 2023-11-16T16:51:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f43137ff700 1 -- 172.21.15.44:0/2533095412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c101440 msgr2=0x7f430c101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f43137ff700 1 --2- 172.21.15.44:0/2533095412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c101440 0x7f430c101810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4308009990 tx=0x7f4308009ca0 comp rx=0 tx=0).stop 2023-11-16T16:51:34.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f954f9ff700 1 -- 172.21.15.44:0/10214361 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95480746a0 msgr2=0x7f9548072d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:34.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f954f9ff700 1 --2- 172.21.15.44:0/10214361 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95480746a0 0x7f9548072d20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9544009990 tx=0x7f9544009ca0 comp rx=0 tx=0).stop 2023-11-16T16:51:34.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f954f9ff700 1 -- 172.21.15.44:0/10214361 shutdown_connections 2023-11-16T16:51:34.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f954f9ff700 1 --2- 172.21.15.44:0/10214361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f954810cec0 0x7f954810f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:34.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f954f9ff700 1 --2- 172.21.15.44:0/10214361 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95480746a0 0x7f9548072d20 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:34.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f954f9ff700 1 --2- 172.21.15.44:0/10214361 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9548074b00 0x7f9548073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:34.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f954f9ff700 1 -- 172.21.15.44:0/10214361 >> 172.21.15.44:0/10214361 conn(0x7f9548077590 msgr2=0x7f9548077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:34.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f43137ff700 1 -- 172.21.15.44:0/2533095412 shutdown_connections 2023-11-16T16:51:34.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f43137ff700 1 --2- 172.21.15.44:0/2533095412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f430c10a810 0x7f430c10cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:34.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f43137ff700 1 --2- 172.21.15.44:0/2533095412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c101440 0x7f430c101810 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:34.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f43137ff700 1 --2- 172.21.15.44:0/2533095412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f430c101d50 0x7f430c10a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:34.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f43137ff700 1 -- 172.21.15.44:0/2533095412 >> 172.21.15.44:0/2533095412 conn(0x7f430c0763a0 msgr2=0x7f430c0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:34.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f954f9ff700 1 -- 172.21.15.44:0/10214361 shutdown_connections 2023-11-16T16:51:34.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f43137ff700 1 -- 172.21.15.44:0/2533095412 shutdown_connections 2023-11-16T16:51:34.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f954f9ff700 1 -- 172.21.15.44:0/10214361 wait complete. 2023-11-16T16:51:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.947+0000 7f43137ff700 1 -- 172.21.15.44:0/2533095412 wait complete. 2023-11-16T16:51:34.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.948+0000 7f954f9ff700 1 Processor -- start 2023-11-16T16:51:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.948+0000 7f43137ff700 1 Processor -- start 2023-11-16T16:51:34.953 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.948+0000 7f954f9ff700 1 -- start start 2023-11-16T16:51:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.948+0000 7f43137ff700 1 -- start start 2023-11-16T16:51:34.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f954f9ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95480746a0 0x7f9548198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:34.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f954f9ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9548074b00 0x7f9548198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:34.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f954f9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f954810cec0 0x7f954819d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:34.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f954f9ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954819d650 con 0x7f95480746a0 2023-11-16T16:51:34.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f954f9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954819d7c0 con 0x7f954810cec0 2023-11-16T16:51:34.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f954f9ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954819d980 con 0x7f9548074b00 2023-11-16T16:51:34.955 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f43137ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f430c101440 0x7f430c0ff090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:34.955 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f43137ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c101d50 0x7f430c0ff5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:34.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f954cf9a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9548074b00 0x7f9548198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:34.955 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f43137ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f430c10a810 0x7f430c0ffb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:34.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f954d79b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95480746a0 0x7f9548198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f43137ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c1a8d20 con 0x7f430c10a810 2023-11-16T16:51:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f43137ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c1a8e60 con 0x7f430c101d50 2023-11-16T16:51:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f43137ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c1a8fa0 con 0x7f430c101440 2023-11-16T16:51:34.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f954cf9a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9548074b00 0x7f9548198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:36672/0 (socket says 172.21.15.44:36672) 2023-11-16T16:51:34.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f954cf9a700 1 -- 172.21.15.44:0/2052880286 learned_addr learned my addr 172.21.15.44:0/2052880286 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:34.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f954df9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f954810cec0 0x7f954819d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f4310d9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c101d50 0x7f430c0ff5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f431159b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f430c101440 0x7f430c0ff090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f4310d9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c101d50 0x7f430c0ff5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:52094/0 (socket says 172.21.15.44:52094) 2023-11-16T16:51:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.949+0000 7f4310d9a700 1 -- 172.21.15.44:0/1386808828 learned_addr learned my addr 172.21.15.44:0/1386808828 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f4311d9c700 1 --2- 172.21.15.44:0/1386808828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f430c10a810 0x7f430c0ffb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f431159b700 1 -- 172.21.15.44:0/1386808828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c101d50 msgr2=0x7f430c0ff5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:34.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f954df9c700 1 -- 172.21.15.44:0/2052880286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9548074b00 msgr2=0x7f9548198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:34.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f954df9c700 1 --2- 172.21.15.44:0/2052880286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9548074b00 0x7f9548198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:34.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f954df9c700 1 -- 172.21.15.44:0/2052880286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95480746a0 msgr2=0x7f9548198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:34.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f954df9c700 1 --2- 172.21.15.44:0/2052880286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95480746a0 0x7f9548198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:34.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f954df9c700 1 -- 172.21.15.44:0/2052880286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95440095f0 con 0x7f954810cec0 2023-11-16T16:51:34.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f431159b700 1 --2- 172.21.15.44:0/1386808828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c101d50 0x7f430c0ff5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:34.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f431159b700 1 -- 172.21.15.44:0/1386808828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f430c10a810 msgr2=0x7f430c0ffb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:34.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f431159b700 1 --2- 172.21.15.44:0/1386808828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f430c10a810 0x7f430c0ffb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:34.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f431159b700 1 -- 172.21.15.44:0/1386808828 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43080095f0 con 0x7f430c101440 2023-11-16T16:51:34.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f4310d9a700 1 --2- 172.21.15.44:0/1386808828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c101d50 0x7f430c0ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:34.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f4311d9c700 1 --2- 172.21.15.44:0/1386808828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f430c10a810 0x7f430c0ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:51:34.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f954df9c700 1 --2- 172.21.15.44:0/2052880286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f954810cec0 0x7f954819d010 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f954000ef90 tx=0x7f954000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:34.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f431159b700 1 --2- 172.21.15.44:0/1386808828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f430c101440 0x7f430c0ff090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4308000c00 tx=0x7f4308005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:34.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f953e7fc700 1 -- 172.21.15.44:0/2052880286 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9540018070 con 0x7f954810cec0 2023-11-16T16:51:34.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f953e7fc700 1 -- 172.21.15.44:0/2052880286 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f954000f040 con 0x7f954810cec0 2023-11-16T16:51:34.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f953e7fc700 1 -- 172.21.15.44:0/2052880286 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954000cc30 con 0x7f954810cec0 2023-11-16T16:51:34.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f954f9ff700 1 -- 172.21.15.44:0/2052880286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95481125c0 con 0x7f954810cec0 2023-11-16T16:51:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.950+0000 7f43027fc700 1 -- 172.21.15.44:0/1386808828 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430801d070 con 0x7f430c101440 2023-11-16T16:51:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.951+0000 7f43027fc700 1 -- 172.21.15.44:0/1386808828 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4308021410 con 0x7f430c101440 2023-11-16T16:51:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.951+0000 7f43027fc700 1 -- 172.21.15.44:0/1386808828 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308017650 con 0x7f430c101440 2023-11-16T16:51:34.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.951+0000 7f954f9ff700 1 -- 172.21.15.44:0/2052880286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9548112430 con 0x7f954810cec0 2023-11-16T16:51:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.951+0000 7f43137ff700 1 -- 172.21.15.44:0/1386808828 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f430c1102c0 con 0x7f430c101440 2023-11-16T16:51:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.951+0000 7f43137ff700 1 -- 172.21.15.44:0/1386808828 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430c110130 con 0x7f430c101440 2023-11-16T16:51:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.954+0000 7f43027fc700 1 -- 172.21.15.44:0/1386808828 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f43080217f0 con 0x7f430c101440 2023-11-16T16:51:34.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.954+0000 7f953e7fc700 1 -- 172.21.15.44:0/2052880286 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f9540003a00 con 0x7f954810cec0 2023-11-16T16:51:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.956+0000 7f43027fc700 1 --2- 172.21.15.44:0/1386808828 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f42f80a12c0 0x7f42f80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.957+0000 7f43027fc700 1 -- 172.21.15.44:0/1386808828 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f43080ce2b0 con 0x7f430c101440 2023-11-16T16:51:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.957+0000 7f4310d9a700 1 --2- 172.21.15.44:0/1386808828 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f42f80a12c0 0x7f42f80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:34.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.957+0000 7f953e7fc700 1 --2- 172.21.15.44:0/2052880286 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f95340a1340 0x7f95340a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:34.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.957+0000 7f953e7fc700 1 -- 172.21.15.44:0/2052880286 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f95400ccb40 con 0x7f954810cec0 2023-11-16T16:51:34.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.957+0000 7f954d79b700 1 --2- 172.21.15.44:0/2052880286 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f95340a1340 0x7f95340a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:34.957+0000 7f4310d9a700 1 --2- 172.21.15.44:0/1386808828 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f42f80a12c0 0x7f42f80a3770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f430c100880 tx=0x7f42fc009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:34.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:34.957+0000 7f954d79b700 1 --2- 172.21.15.44:0/2052880286 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f95340a1340 0x7f95340a37f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f954400b2e0 tx=0x7f9544005fb0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.047+0000 7f43271e2b80 1 Processor -- start 2023-11-16T16:51:35.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.047+0000 7f95633c3b80 1 Processor -- start 2023-11-16T16:51:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.047+0000 7f43271e2b80 1 -- start start 2023-11-16T16:51:35.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.047+0000 7f95633c3b80 1 -- start start 2023-11-16T16:51:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.047+0000 7f43271e2b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baae366b0 0x560baafe01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.047+0000 7f43271e2b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0720 0x560baafda560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f43271e2b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baafe0b90 0x560baafdab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f43271e2b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x560baaee0000 con 0x560baafe0720 2023-11-16T16:51:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f43271e2b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560baaf32840 con 0x560baafe0b90 2023-11-16T16:51:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f43271e2b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x560baaf2bcd0 con 0x560baae366b0 2023-11-16T16:51:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f431159b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baae366b0 0x560baafe01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f431159b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baae366b0 0x560baafe01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:36698/0 (socket says 172.21.15.44:36698) 2023-11-16T16:51:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f431159b700 1 -- 172.21.15.44:0/123577275 learned_addr learned my addr 172.21.15.44:0/123577275 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f4311d9c700 1 --2- 172.21.15.44:0/123577275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baafe0b90 0x560baafdab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f95633c3b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce63597b60 0x55ce63597f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f95633c3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63598470 0x55ce6359c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f95633c3b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce6359c8b0 0x55ce6359cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f95633c3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce6349d060 con 0x55ce63598470 2023-11-16T16:51:35.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f95633c3b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce634ef8a0 con 0x55ce63597b60 2023-11-16T16:51:35.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f95633c3b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce634e8d30 con 0x55ce6359c8b0 2023-11-16T16:51:35.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f954df9c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce6359c8b0 0x55ce6359cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f954d79b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce63597b60 0x55ce63597f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f954df9c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce6359c8b0 0x55ce6359cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:36712/0 (socket says 172.21.15.44:36712) 2023-11-16T16:51:35.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f954df9c700 1 -- 172.21.15.44:0/2071462515 learned_addr learned my addr 172.21.15.44:0/2071462515 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:35.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f954cf9a700 1 --2- 172.21.15.44:0/2071462515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63598470 0x55ce6359c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f431159b700 1 -- 172.21.15.44:0/123577275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baafe0b90 msgr2=0x560baafdab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f431159b700 1 --2- 172.21.15.44:0/123577275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baafe0b90 0x560baafdab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f431159b700 1 -- 172.21.15.44:0/123577275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0720 msgr2=0x560baafda560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:51:35.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.048+0000 7f954df9c700 1 -- 172.21.15.44:0/2071462515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce63597b60 msgr2=0x55ce63597f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:35.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.049+0000 7f954df9c700 1 --2- 172.21.15.44:0/2071462515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce63597b60 0x55ce63597f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.049+0000 7f954df9c700 1 -- 172.21.15.44:0/2071462515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63598470 msgr2=0x55ce6359c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:35.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.049+0000 7f954df9c700 1 --2- 172.21.15.44:0/2071462515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63598470 0x55ce6359c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.049+0000 7f954df9c700 1 -- 172.21.15.44:0/2071462515 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce6336a630 con 0x55ce6359c8b0 2023-11-16T16:51:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.049+0000 7f4310d9a700 1 --2- 172.21.15.44:0/123577275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0720 0x560baafda560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.049+0000 7f431159b700 1 --2- 172.21.15.44:0/123577275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0720 0x560baafda560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.049+0000 7f431159b700 1 -- 172.21.15.44:0/123577275 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560baadad5d0 con 0x560baae366b0 2023-11-16T16:51:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.049+0000 7f4310d9a700 1 --2- 172.21.15.44:0/123577275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0720 0x560baafda560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:51:35.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.049+0000 7f954df9c700 1 --2- 172.21.15.44:0/2071462515 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce6359c8b0 0x55ce6359cd00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f95400995c0 tx=0x7f9540099530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90f3635672356e9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.049+0000 7f431159b700 1 --2- 172.21.15.44:0/123577275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baae366b0 0x560baafe01e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f43080a2ec0 tx=0x7f430809abf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18d67c3eef948257 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:35.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.049+0000 7f95337fe700 1 -- 172.21.15.44:0/2071462515 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954009d8f0 con 0x55ce6359c8b0 2023-11-16T16:51:35.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.049+0000 7f95337fe700 1 -- 172.21.15.44:0/2071462515 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95400a1c80 con 0x55ce6359c8b0 2023-11-16T16:51:35.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.049+0000 7f95337fe700 1 -- 172.21.15.44:0/2071462515 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95400b9330 con 0x55ce6359c8b0 2023-11-16T16:51:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.049+0000 7f42f77fe700 1 -- 172.21.15.44:0/123577275 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430809b030 con 0x560baae366b0 2023-11-16T16:51:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.049+0000 7f42f77fe700 1 -- 172.21.15.44:0/123577275 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f430809f060 con 0x560baae366b0 2023-11-16T16:51:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.049+0000 7f42f77fe700 1 -- 172.21.15.44:0/123577275 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430809a520 con 0x560baae366b0 2023-11-16T16:51:35.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f95633c3b80 1 -- 172.21.15.44:0/2071462515 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce6359c8b0 msgr2=0x55ce6359cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:35.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f95633c3b80 1 --2- 172.21.15.44:0/2071462515 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce6359c8b0 0x55ce6359cd00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f95400995c0 tx=0x7f9540099530 comp rx=0 tx=0).stop 2023-11-16T16:51:35.057 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f43271e2b80 1 -- 172.21.15.44:0/123577275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baae366b0 msgr2=0x560baafe01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:35.057 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f43271e2b80 1 --2- 172.21.15.44:0/123577275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baae366b0 0x560baafe01e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f43080a2ec0 tx=0x7f430809abf0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f95633c3b80 1 -- 172.21.15.44:0/2071462515 shutdown_connections 2023-11-16T16:51:35.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f95633c3b80 1 --2- 172.21.15.44:0/2071462515 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce6359c8b0 0x55ce6359cd00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f95633c3b80 1 --2- 172.21.15.44:0/2071462515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce63597b60 0x55ce63597f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f95633c3b80 1 --2- 172.21.15.44:0/2071462515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63598470 0x55ce6359c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f95633c3b80 1 -- 172.21.15.44:0/2071462515 >> 172.21.15.44:0/2071462515 conn(0x55ce6358d360 msgr2=0x55ce63592b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f43271e2b80 1 -- 172.21.15.44:0/123577275 shutdown_connections 2023-11-16T16:51:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f43271e2b80 1 --2- 172.21.15.44:0/123577275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baafe0b90 0x560baafdab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f43271e2b80 1 --2- 172.21.15.44:0/123577275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baae366b0 0x560baafe01e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f43271e2b80 1 --2- 172.21.15.44:0/123577275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0720 0x560baafda560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f43271e2b80 1 -- 172.21.15.44:0/123577275 >> 172.21.15.44:0/123577275 conn(0x560baafcff50 msgr2=0x560baafd5270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:35.059 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f95633c3b80 1 -- 172.21.15.44:0/2071462515 shutdown_connections 2023-11-16T16:51:35.059 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f95633c3b80 1 -- 172.21.15.44:0/2071462515 wait complete. 2023-11-16T16:51:35.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f43271e2b80 1 -- 172.21.15.44:0/123577275 shutdown_connections 2023-11-16T16:51:35.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.050+0000 7f43271e2b80 1 -- 172.21.15.44:0/123577275 wait complete. 2023-11-16T16:51:35.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.051+0000 7f43271e2b80 1 -- start start 2023-11-16T16:51:35.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.051+0000 7f95633c3b80 1 -- start start 2023-11-16T16:51:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.051+0000 7f43271e2b80 1 Processor -- start 2023-11-16T16:51:35.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.051+0000 7f95633c3b80 1 Processor -- start 2023-11-16T16:51:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f43271e2b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baae366b0 0x560baafe9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f43271e2b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0720 0x560baafe9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f43271e2b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baafe0b90 0x560baafe3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f43271e2b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x560baaecd880 con 0x560baafe0720 2023-11-16T16:51:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f43271e2b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560baaecd520 con 0x560baae366b0 2023-11-16T16:51:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f43271e2b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x560baadaf8c0 con 0x560baafe0b90 2023-11-16T16:51:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f43271e2b80 10 client.0 _ll_register_callbacks cb 0x560baaeeb720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f4310d9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0720 0x560baafe9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f431159b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baae366b0 0x560baafe9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f95633c3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63597b60 0x55ce635a4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f95633c3b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce63598470 0x55ce635a12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f95633c3b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce6359c8b0 0x55ce635a1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f95633c3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce6348a8e0 con 0x55ce63597b60 2023-11-16T16:51:35.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f95633c3b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce6348a580 con 0x55ce63598470 2023-11-16T16:51:35.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f95633c3b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce6336c920 con 0x55ce6359c8b0 2023-11-16T16:51:35.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f95633c3b80 10 client.0 _ll_register_callbacks cb 0x55ce634a8780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f431159b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baae366b0 0x560baafe9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:52130/0 (socket says 172.21.15.44:52130) 2023-11-16T16:51:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f431159b700 1 -- 172.21.15.44:0/4062848643 learned_addr learned my addr 172.21.15.44:0/4062848643 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f4311d9c700 1 --2- 172.21.15.44:0/4062848643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baafe0b90 0x560baafe3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f954cf9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce63598470 0x55ce635a12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f954df9c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce6359c8b0 0x55ce635a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f954cf9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce63598470 0x55ce635a12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:52142/0 (socket says 172.21.15.44:52142) 2023-11-16T16:51:35.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f954cf9a700 1 -- 172.21.15.44:0/3757774927 learned_addr learned my addr 172.21.15.44:0/3757774927 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:35.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.052+0000 7f954d79b700 1 --2- 172.21.15.44:0/3757774927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63597b60 0x55ce635a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f431159b700 1 -- 172.21.15.44:0/4062848643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baafe0b90 msgr2=0x560baafe3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f431159b700 1 --2- 172.21.15.44:0/4062848643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baafe0b90 0x560baafe3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f431159b700 1 -- 172.21.15.44:0/4062848643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0720 msgr2=0x560baafe9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f431159b700 1 --2- 172.21.15.44:0/4062848643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0720 0x560baafe9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f431159b700 1 -- 172.21.15.44:0/4062848643 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430c110130 con 0x560baae366b0 2023-11-16T16:51:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f4310d9a700 1 --2- 172.21.15.44:0/4062848643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0720 0x560baafe9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:51:35.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f954df9c700 1 -- 172.21.15.44:0/3757774927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce63598470 msgr2=0x55ce635a12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:35.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f954df9c700 1 --2- 172.21.15.44:0/3757774927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce63598470 0x55ce635a12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f954df9c700 1 -- 172.21.15.44:0/3757774927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63597b60 msgr2=0x55ce635a4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:35.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f954df9c700 1 --2- 172.21.15.44:0/3757774927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63597b60 0x55ce635a4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f954df9c700 1 -- 172.21.15.44:0/3757774927 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9548112430 con 0x55ce6359c8b0 2023-11-16T16:51:35.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f954d79b700 1 --2- 172.21.15.44:0/3757774927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63597b60 0x55ce635a4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:51:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f431159b700 1 --2- 172.21.15.44:0/4062848643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baae366b0 0x560baafe9740 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4308021bd0 tx=0x7f4308022f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae4c64467967cfdb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f42f4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:51:35.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f954df9c700 1 --2- 172.21.15.44:0/3757774927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce6359c8b0 0x55ce635a1950 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f95400ca430 tx=0x7f9540095fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee741de4d8e099ed server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:35.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f9530ff9700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:51:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f42f4ff9700 1 -- 172.21.15.44:0/4062848643 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430809b490 con 0x560baae366b0 2023-11-16T16:51:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f42f4ff9700 1 -- 172.21.15.44:0/4062848643 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f430809b030 con 0x560baae366b0 2023-11-16T16:51:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f43271e2b80 1 -- 172.21.15.44:0/4062848643 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560baafcd710 con 0x560baae366b0 2023-11-16T16:51:35.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f9530ff9700 1 -- 172.21.15.44:0/3757774927 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9540099d20 con 0x55ce6359c8b0 2023-11-16T16:51:35.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f9530ff9700 1 -- 172.21.15.44:0/3757774927 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f95400092e0 con 0x55ce6359c8b0 2023-11-16T16:51:35.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f95633c3b80 1 -- 172.21.15.44:0/3757774927 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ce6358ab20 con 0x55ce6359c8b0 2023-11-16T16:51:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f42f4ff9700 1 -- 172.21.15.44:0/4062848643 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43080b9250 con 0x560baae366b0 2023-11-16T16:51:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f43271e2b80 10 client.25877 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.053+0000 7f43271e2b80 1 -- 172.21.15.44:0/4062848643 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x560baaef2130 con 0x560baae366b0 2023-11-16T16:51:35.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f9530ff9700 1 -- 172.21.15.44:0/3757774927 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f954009d8f0 con 0x55ce6359c8b0 2023-11-16T16:51:35.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f95633c3b80 10 client.25870 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:35.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f95633c3b80 1 -- 172.21.15.44:0/3757774927 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55ce634af190 con 0x55ce6359c8b0 2023-11-16T16:51:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f42f4ff9700 1 -- 172.21.15.44:0/4062848643 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43080cb020 con 0x560baae366b0 2023-11-16T16:51:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f42f4ff9700 1 -- 172.21.15.44:0/4062848643 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f43080a3080 con 0x560baae366b0 2023-11-16T16:51:35.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f9530ff9700 1 -- 172.21.15.44:0/3757774927 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f954007e080 con 0x55ce6359c8b0 2023-11-16T16:51:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f43271e2b80 10 client.25877 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:35.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f9530ff9700 1 -- 172.21.15.44:0/3757774927 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f95400b4070 con 0x55ce6359c8b0 2023-11-16T16:51:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f43271e2b80 10 client.25877 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f43271e2b80 20 client.25877 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f43271e2b80 1 --2- 172.21.15.44:0/4062848643 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560baaff3680 0x560baaff5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f43271e2b80 4 client.25877 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:51:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f43271e2b80 1 -- 172.21.15.44:0/4062848643 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560baaeee040 con 0x560baaff3680 2023-11-16T16:51:35.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f95633c3b80 10 client.25870 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:35.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f95633c3b80 10 client.25870 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:35.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f95633c3b80 20 client.25870 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:35.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f95633c3b80 1 --2- 172.21.15.44:0/3757774927 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ce635b9bd0 0x55ce635bbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f95633c3b80 4 client.25870 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:51:35.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.054+0000 7f95633c3b80 1 -- 172.21.15.44:0/3757774927 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ce634ab0a0 con 0x55ce635b9bd0 2023-11-16T16:51:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.055+0000 7f4310d9a700 1 --2- 172.21.15.44:0/4062848643 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560baaff3680 0x560baaff5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.055+0000 7f954d79b700 1 --2- 172.21.15.44:0/3757774927 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ce635b9bd0 0x55ce635bbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.055+0000 7f4310d9a700 1 --2- 172.21.15.44:0/4062848643 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560baaff3680 0x560baaff5a60 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d442475b45afdd2a server_cookie=5fc8f3a784c9e34 in_seq=0 out_seq=0 2023-11-16T16:51:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.055+0000 7f42f4ff9700 10 client.25877 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:35.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.055+0000 7f9530ff9700 10 client.25870 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:35.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.055+0000 7f954d79b700 1 --2- 172.21.15.44:0/3757774927 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ce635b9bd0 0x55ce635bbfe0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=929f1cd3cdc13339 server_cookie=b040f9943281a2d in_seq=0 out_seq=0 2023-11-16T16:51:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.056+0000 7f42f4ff9700 1 -- 172.21.15.44:0/4062848643 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x560baaeee040 con 0x560baaff3680 2023-11-16T16:51:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.056+0000 7f42f4ff9700 10 client.25877 handle_command_reply: tid=0 2023-11-16T16:51:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.056+0000 7f43271e2b80 1 client.25877 shutdown 2023-11-16T16:51:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.056+0000 7f43271e2b80 1 -- 172.21.15.44:0/4062848643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baae366b0 msgr2=0x560baafe9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.056+0000 7f43271e2b80 1 --2- 172.21.15.44:0/4062848643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baae366b0 0x560baafe9740 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4308021bd0 tx=0x7f4308022f30 comp rx=0 tx=0).stop 2023-11-16T16:51:35.074 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.056+0000 7f9530ff9700 1 -- 172.21.15.44:0/3757774927 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ce634ab0a0 con 0x55ce635b9bd0 2023-11-16T16:51:35.074 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.056+0000 7f9530ff9700 10 client.25870 handle_command_reply: tid=0 2023-11-16T16:51:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.056+0000 7f43271e2b80 1 -- 172.21.15.44:0/4062848643 shutdown_connections 2023-11-16T16:51:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.056+0000 7f43271e2b80 1 --2- 172.21.15.44:0/4062848643 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560baaff3680 0x560baaff5a60 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.056+0000 7f43271e2b80 1 --2- 172.21.15.44:0/4062848643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baafe0b90 0x560baafe3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.056+0000 7f43271e2b80 1 --2- 172.21.15.44:0/4062848643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baae366b0 0x560baafe9740 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.056+0000 7f43271e2b80 1 --2- 172.21.15.44:0/4062848643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0720 0x560baafe9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.056+0000 7f42f4ff9700 0 client.25877 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.056+0000 7f43271e2b80 1 -- 172.21.15.44:0/4062848643 >> 172.21.15.44:0/4062848643 conn(0x560baafcff50 msgr2=0x560baafd47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:35.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.056+0000 7f95633c3b80 1 client.25870 shutdown 2023-11-16T16:51:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.056+0000 7f43271e2b80 1 -- 172.21.15.44:0/4062848643 shutdown_connections 2023-11-16T16:51:35.076 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.057+0000 7f95633c3b80 1 -- 172.21.15.44:0/3757774927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce6359c8b0 msgr2=0x55ce635a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:35.076 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.057+0000 7f95633c3b80 1 --2- 172.21.15.44:0/3757774927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce6359c8b0 0x55ce635a1950 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f95400ca430 tx=0x7f9540095fd0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.057+0000 7f43271e2b80 1 -- 172.21.15.44:0/4062848643 wait complete. 2023-11-16T16:51:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.057+0000 7f95633c3b80 1 -- 172.21.15.44:0/3757774927 shutdown_connections 2023-11-16T16:51:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.057+0000 7f95633c3b80 1 --2- 172.21.15.44:0/3757774927 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ce635b9bd0 0x55ce635bbfe0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.057+0000 7f95633c3b80 1 --2- 172.21.15.44:0/3757774927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce6359c8b0 0x55ce635a1950 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.057+0000 7f95633c3b80 1 --2- 172.21.15.44:0/3757774927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63597b60 0x55ce635a4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.057+0000 7f95633c3b80 1 --2- 172.21.15.44:0/3757774927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce63598470 0x55ce635a12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.057+0000 7f95633c3b80 1 -- 172.21.15.44:0/3757774927 >> 172.21.15.44:0/3757774927 conn(0x55ce6358d360 msgr2=0x55ce6359dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:35.077 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.057+0000 7f9530ff9700 0 client.25870 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:35.077 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.057+0000 7f43271e2b80 20 client.25877 trim_cache size 0 max 16384 2023-11-16T16:51:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.057+0000 7f95633c3b80 1 -- 172.21.15.44:0/3757774927 shutdown_connections 2023-11-16T16:51:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.057+0000 7f95633c3b80 1 -- 172.21.15.44:0/3757774927 wait complete. 2023-11-16T16:51:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.057+0000 7f95633c3b80 20 client.25870 trim_cache size 0 max 16384 2023-11-16T16:51:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.072+0000 7f95633c3b80 1 Processor -- start 2023-11-16T16:51:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.072+0000 7f95633c3b80 1 -- start start 2023-11-16T16:51:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.072+0000 7f95633c3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63597b60 0x55ce635b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.072+0000 7f95633c3b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce63598470 0x55ce635a3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.072+0000 7f95633c3b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce6359c8b0 0x55ce6359a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.072+0000 7f95633c3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce635b6b00 con 0x55ce63597b60 2023-11-16T16:51:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.072+0000 7f95633c3b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce63598bc0 con 0x55ce6359c8b0 2023-11-16T16:51:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.072+0000 7f95633c3b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce6358cf10 con 0x55ce63598470 2023-11-16T16:51:35.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.073+0000 7f954d79b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63597b60 0x55ce635b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.073+0000 7f954d79b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63597b60 0x55ce635b11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34750/0 (socket says 172.21.15.44:34750) 2023-11-16T16:51:35.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.073+0000 7f954d79b700 1 -- 172.21.15.44:0/2974240370 learned_addr learned my addr 172.21.15.44:0/2974240370 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:35.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.073+0000 7f954cf9a700 1 --2- 172.21.15.44:0/2974240370 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce63598470 0x55ce635a3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.073+0000 7f954df9c700 1 --2- 172.21.15.44:0/2974240370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce6359c8b0 0x55ce6359a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.073+0000 7f43271e2b80 1 Processor -- start 2023-11-16T16:51:35.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.073+0000 7f954d79b700 1 -- 172.21.15.44:0/2974240370 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce63598470 msgr2=0x55ce635a3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:35.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.073+0000 7f954d79b700 1 --2- 172.21.15.44:0/2974240370 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce63598470 0x55ce635a3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.073+0000 7f954d79b700 1 -- 172.21.15.44:0/2974240370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce6359c8b0 msgr2=0x55ce6359a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:35.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.073+0000 7f954d79b700 1 --2- 172.21.15.44:0/2974240370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce6359c8b0 0x55ce6359a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.073+0000 7f954d79b700 1 -- 172.21.15.44:0/2974240370 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce634aeee0 con 0x55ce63597b60 2023-11-16T16:51:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.073+0000 7f43271e2b80 1 -- start start 2023-11-16T16:51:35.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.073+0000 7f954d79b700 1 --2- 172.21.15.44:0/2974240370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63597b60 0x55ce635b11b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9544005f50 tx=0x7f9544020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1f7bc23d6fb2d1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f43271e2b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baae366b0 0x560baafd4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f43271e2b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baafe0720 0x560baafe0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f43271e2b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baafe0b90 0x560baafe14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f43271e2b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x560baafcfbe0 con 0x560baae366b0 2023-11-16T16:51:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f43271e2b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560baafd7060 con 0x560baafe0b90 2023-11-16T16:51:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f43271e2b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x560baafe2070 con 0x560baafe0720 2023-11-16T16:51:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f431159b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baae366b0 0x560baafd4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f431159b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baae366b0 0x560baafd4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34752/0 (socket says 172.21.15.44:34752) 2023-11-16T16:51:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f431159b700 1 -- 172.21.15.44:0/1491322034 learned_addr learned my addr 172.21.15.44:0/1491322034 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f4310d9a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baafe0720 0x560baafe0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f4311d9c700 1 --2- 172.21.15.44:0/1491322034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baafe0b90 0x560baafe14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f9533fff700 1 -- 172.21.15.44:0/2974240370 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954402d040 con 0x55ce63597b60 2023-11-16T16:51:35.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f9533fff700 1 -- 172.21.15.44:0/2974240370 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f954401e040 con 0x55ce63597b60 2023-11-16T16:51:35.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f9533fff700 1 -- 172.21.15.44:0/2974240370 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9544004c50 con 0x55ce63597b60 2023-11-16T16:51:35.084 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f431159b700 1 -- 172.21.15.44:0/1491322034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baafe0720 msgr2=0x560baafe0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:35.084 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f431159b700 1 --2- 172.21.15.44:0/1491322034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baafe0720 0x560baafe0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.084 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f431159b700 1 -- 172.21.15.44:0/1491322034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baafe0b90 msgr2=0x560baafe14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:35.084 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f431159b700 1 --2- 172.21.15.44:0/1491322034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baafe0b90 0x560baafe14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.074+0000 7f431159b700 1 -- 172.21.15.44:0/1491322034 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560baaef1e80 con 0x560baae366b0 2023-11-16T16:51:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.075+0000 7f4311d9c700 1 --2- 172.21.15.44:0/1491322034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baafe0b90 0x560baafe14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.075+0000 7f431159b700 1 --2- 172.21.15.44:0/1491322034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baae366b0 0x560baafd4f60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4308097aa0 tx=0x7f43080cbb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d349d60ddbe47344 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.075+0000 7f42f7fff700 1 -- 172.21.15.44:0/1491322034 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308060ea0 con 0x560baae366b0 2023-11-16T16:51:35.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.075+0000 7f95633c3b80 1 -- 172.21.15.44:0/2974240370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63597b60 msgr2=0x55ce635b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:35.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.075+0000 7f95633c3b80 1 --2- 172.21.15.44:0/2974240370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63597b60 0x55ce635b11b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9544005f50 tx=0x7f9544020000 comp rx=0 tx=0).stop 2023-11-16T16:51:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.075+0000 7f42f7fff700 1 -- 172.21.15.44:0/1491322034 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43080cb040 con 0x560baae366b0 2023-11-16T16:51:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.075+0000 7f42f7fff700 1 -- 172.21.15.44:0/1491322034 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430809be10 con 0x560baae366b0 2023-11-16T16:51:35.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.075+0000 7f95633c3b80 1 -- 172.21.15.44:0/2974240370 shutdown_connections 2023-11-16T16:51:35.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.075+0000 7f95633c3b80 1 --2- 172.21.15.44:0/2974240370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce6359c8b0 0x55ce6359a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.075+0000 7f95633c3b80 1 --2- 172.21.15.44:0/2974240370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce63597b60 0x55ce635b11b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.075+0000 7f95633c3b80 1 --2- 172.21.15.44:0/2974240370 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce63598470 0x55ce635a3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.075+0000 7f95633c3b80 1 -- 172.21.15.44:0/2974240370 >> 172.21.15.44:0/2974240370 conn(0x55ce6358d360 msgr2=0x55ce635b63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:35.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.075+0000 7f95633c3b80 1 -- 172.21.15.44:0/2974240370 shutdown_connections 2023-11-16T16:51:35.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.075+0000 7f95633c3b80 1 -- 172.21.15.44:0/2974240370 wait complete. 2023-11-16T16:51:35.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.076+0000 7f95633c3b80 1 -- start start 2023-11-16T16:51:35.087 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.076+0000 7f43271e2b80 1 -- 172.21.15.44:0/1491322034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baae366b0 msgr2=0x560baafd4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:35.087 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.076+0000 7f43271e2b80 1 --2- 172.21.15.44:0/1491322034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baae366b0 0x560baafd4f60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4308097aa0 tx=0x7f43080cbb80 comp rx=0 tx=0).stop 2023-11-16T16:51:35.087 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.076+0000 7f43271e2b80 1 -- 172.21.15.44:0/1491322034 shutdown_connections 2023-11-16T16:51:35.087 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.076+0000 7f43271e2b80 1 --2- 172.21.15.44:0/1491322034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baafe0b90 0x560baafe14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.087 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.076+0000 7f43271e2b80 1 --2- 172.21.15.44:0/1491322034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baae366b0 0x560baafd4f60 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.087 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.076+0000 7f43271e2b80 1 --2- 172.21.15.44:0/1491322034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baafe0720 0x560baafe0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.087 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.076+0000 7f43271e2b80 1 -- 172.21.15.44:0/1491322034 >> 172.21.15.44:0/1491322034 conn(0x560baafcff50 msgr2=0x560baafe4400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:35.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.076+0000 7f95633c3b80 1 Processor -- start 2023-11-16T16:51:35.088 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.076+0000 7f43271e2b80 1 -- 172.21.15.44:0/1491322034 shutdown_connections 2023-11-16T16:51:35.088 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.076+0000 7f43271e2b80 1 -- 172.21.15.44:0/1491322034 wait complete. 2023-11-16T16:51:35.088 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.077+0000 7f43271e2b80 1 -- start start 2023-11-16T16:51:35.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.077+0000 7f95633c3b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce63597b60 0x55ce635af1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.077+0000 7f95633c3b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce63598470 0x55ce635af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.077+0000 7f95633c3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce6359c8b0 0x55ce635afc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.077+0000 7f95633c3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce6360ca50 con 0x55ce6359c8b0 2023-11-16T16:51:35.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.077+0000 7f95633c3b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce6360cb90 con 0x55ce63597b60 2023-11-16T16:51:35.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.077+0000 7f95633c3b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce6360ccd0 con 0x55ce63598470 2023-11-16T16:51:35.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.077+0000 7f95633c3b80 10 client.0 _ll_register_callbacks cb 0x7f9544005e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.077+0000 7f43271e2b80 1 Processor -- start 2023-11-16T16:51:35.090 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.077+0000 7f954df9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce6359c8b0 0x55ce635afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.090 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.077+0000 7f954df9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce6359c8b0 0x55ce635afc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34754/0 (socket says 172.21.15.44:34754) 2023-11-16T16:51:35.090 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.077+0000 7f954df9c700 1 -- 172.21.15.44:0/1701899843 learned_addr learned my addr 172.21.15.44:0/1701899843 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:35.091 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.077+0000 7f954cf9a700 1 --2- 172.21.15.44:0/1701899843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce63598470 0x55ce635af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.091 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.077+0000 7f954d79b700 1 --2- 172.21.15.44:0/1701899843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce63597b60 0x55ce635af1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.091 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f954df9c700 1 -- 172.21.15.44:0/1701899843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce63598470 msgr2=0x55ce635af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:35.091 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f954df9c700 1 --2- 172.21.15.44:0/1701899843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce63598470 0x55ce635af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f43271e2b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baae366b0 0x560baafe7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.091 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f954df9c700 1 -- 172.21.15.44:0/1701899843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce63597b60 msgr2=0x55ce635af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:35.091 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f954df9c700 1 --2- 172.21.15.44:0/1701899843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce63597b60 0x55ce635af1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f43271e2b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baafe0720 0x560baafe7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f954df9c700 1 -- 172.21.15.44:0/1701899843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f954810fb90 con 0x55ce6359c8b0 2023-11-16T16:51:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f43271e2b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0b90 0x560baafe7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f43271e2b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bab046580 con 0x560baafe0b90 2023-11-16T16:51:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f43271e2b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bab0466f0 con 0x560baae366b0 2023-11-16T16:51:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f43271e2b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bab0468b0 con 0x560baafe0720 2023-11-16T16:51:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f43271e2b80 10 client.0 _ll_register_callbacks cb 0x7f4304009d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f431159b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baae366b0 0x560baafe7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f954df9c700 1 --2- 172.21.15.44:0/1701899843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce6359c8b0 0x55ce635afc20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f95400988c0 tx=0x7f9540099530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48746e74062e9a73 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:35.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f9530ff9700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:51:35.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f954cf9a700 1 --2- 172.21.15.44:0/1701899843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce63598470 0x55ce635af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f431159b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baae366b0 0x560baafe7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:52172/0 (socket says 172.21.15.44:52172) 2023-11-16T16:51:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f431159b700 1 -- 172.21.15.44:0/2666137416 learned_addr learned my addr 172.21.15.44:0/2666137416 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f4310d9a700 1 --2- 172.21.15.44:0/2666137416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baafe0720 0x560baafe7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.094 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f9530ff9700 1 -- 172.21.15.44:0/1701899843 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954009d8f0 con 0x55ce6359c8b0 2023-11-16T16:51:35.094 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f9530ff9700 1 -- 172.21.15.44:0/1701899843 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f9540099d30 con 0x55ce6359c8b0 2023-11-16T16:51:35.095 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f95633c3b80 1 -- 172.21.15.44:0/1701899843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95400a1c80 con 0x55ce6359c8b0 2023-11-16T16:51:35.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f431159b700 1 -- 172.21.15.44:0/2666137416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baafe0720 msgr2=0x560baafe7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:35.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f431159b700 1 --2- 172.21.15.44:0/2666137416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baafe0720 0x560baafe7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.078+0000 7f431159b700 1 -- 172.21.15.44:0/2666137416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0b90 msgr2=0x560baafe7c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:51:35.095 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f9530ff9700 1 -- 172.21.15.44:0/1701899843 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9540098db0 con 0x55ce6359c8b0 2023-11-16T16:51:35.095 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f95633c3b80 10 client.15876 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:35.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f95633c3b80 1 -- 172.21.15.44:0/1701899843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55ce632ffc60 con 0x55ce6359c8b0 2023-11-16T16:51:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f4311d9c700 1 --2- 172.21.15.44:0/2666137416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0b90 0x560baafe7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f431159b700 1 --2- 172.21.15.44:0/2666137416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0b90 0x560baafe7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f431159b700 1 -- 172.21.15.44:0/2666137416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4304009250 con 0x560baae366b0 2023-11-16T16:51:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f4311d9c700 1 --2- 172.21.15.44:0/2666137416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0b90 0x560baafe7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:51:35.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f9530ff9700 1 -- 172.21.15.44:0/1701899843 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95400a1c80 con 0x55ce6359c8b0 2023-11-16T16:51:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f431159b700 1 --2- 172.21.15.44:0/2666137416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baae366b0 0x560baafe7210 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4308021b40 tx=0x7f430809b030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38b0e444e7f20865 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f42f4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:51:35.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f9530ff9700 1 -- 172.21.15.44:0/1701899843 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f95400ca130 con 0x55ce6359c8b0 2023-11-16T16:51:35.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f95633c3b80 10 client.15876 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:35.098 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f95633c3b80 10 client.15876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:35.098 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f95633c3b80 20 client.15876 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:35.098 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f95633c3b80 1 --2- 172.21.15.44:0/1701899843 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ce6360dea0 0x55ce63610280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f42f4ff9700 1 -- 172.21.15.44:0/2666137416 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308060ea0 con 0x560baae366b0 2023-11-16T16:51:35.098 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f95633c3b80 4 client.15876 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T16:51:35.098 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.079+0000 7f95633c3b80 1 -- 172.21.15.44:0/1701899843 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ce634b7660 con 0x55ce6360dea0 2023-11-16T16:51:35.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.080+0000 7f42f4ff9700 1 -- 172.21.15.44:0/2666137416 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f430800fcc0 con 0x560baae366b0 2023-11-16T16:51:35.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.080+0000 7f43271e2b80 1 -- 172.21.15.44:0/2666137416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f430809f060 con 0x560baae366b0 2023-11-16T16:51:35.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.080+0000 7f42f4ff9700 1 -- 172.21.15.44:0/2666137416 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f430809ab70 con 0x560baae366b0 2023-11-16T16:51:35.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.080+0000 7f43271e2b80 10 client.25892 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:35.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.080+0000 7f43271e2b80 1 -- 172.21.15.44:0/2666137416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x560baaef2130 con 0x560baae366b0 2023-11-16T16:51:35.100 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.080+0000 7f954d79b700 1 --2- 172.21.15.44:0/1701899843 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ce6360dea0 0x55ce63610280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.080+0000 7f42f4ff9700 1 -- 172.21.15.44:0/2666137416 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f430809f060 con 0x560baae366b0 2023-11-16T16:51:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.080+0000 7f42f4ff9700 1 -- 172.21.15.44:0/2666137416 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f43080a3060 con 0x560baae366b0 2023-11-16T16:51:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.080+0000 7f43271e2b80 10 client.25892 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.080+0000 7f43271e2b80 10 client.25892 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.080+0000 7f43271e2b80 20 client.25892 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.080+0000 7f43271e2b80 1 --2- 172.21.15.44:0/2666137416 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560bab047af0 0x560bab049ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:35.101 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.080+0000 7f954d79b700 1 --2- 172.21.15.44:0/1701899843 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ce6360dea0 0x55ce63610280 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80c4d88273b36b34 server_cookie=cff9b578660e8427 in_seq=0 out_seq=0 2023-11-16T16:51:35.101 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.080+0000 7f9530ff9700 10 client.15876 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.080+0000 7f43271e2b80 4 client.25892 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:51:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.080+0000 7f43271e2b80 1 -- 172.21.15.44:0/2666137416 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560baaefa600 con 0x560bab047af0 2023-11-16T16:51:35.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f4310d9a700 1 --2- 172.21.15.44:0/2666137416 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560bab047af0 0x560bab049ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:35.102 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f9530ff9700 1 -- 172.21.15.44:0/1701899843 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ce634b7660 con 0x55ce6360dea0 2023-11-16T16:51:35.102 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f9530ff9700 10 client.15876 handle_command_reply: tid=0 2023-11-16T16:51:35.102 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f95633c3b80 1 client.15876 shutdown 2023-11-16T16:51:35.102 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f95633c3b80 1 -- 172.21.15.44:0/1701899843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce6359c8b0 msgr2=0x55ce635afc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:35.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f95633c3b80 1 --2- 172.21.15.44:0/1701899843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce6359c8b0 0x55ce635afc20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f95400988c0 tx=0x7f9540099530 comp rx=0 tx=0).stop 2023-11-16T16:51:35.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f95633c3b80 1 -- 172.21.15.44:0/1701899843 shutdown_connections 2023-11-16T16:51:35.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f95633c3b80 1 --2- 172.21.15.44:0/1701899843 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ce6360dea0 0x55ce63610280 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f95633c3b80 1 --2- 172.21.15.44:0/1701899843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ce6359c8b0 0x55ce635afc20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f95633c3b80 1 --2- 172.21.15.44:0/1701899843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce63597b60 0x55ce635af1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f95633c3b80 1 --2- 172.21.15.44:0/1701899843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ce63598470 0x55ce635af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f9530ff9700 0 client.15876 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:35.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f95633c3b80 1 -- 172.21.15.44:0/1701899843 >> 172.21.15.44:0/1701899843 conn(0x55ce6358d360 msgr2=0x55ce6359a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:35.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f4310d9a700 1 --2- 172.21.15.44:0/2666137416 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560bab047af0 0x560bab049ed0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed4ccde1e591eee0 server_cookie=3a776158fd536dff in_seq=0 out_seq=0 2023-11-16T16:51:35.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f42f4ff9700 10 client.25892 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:35.104 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f95633c3b80 1 -- 172.21.15.44:0/1701899843 shutdown_connections 2023-11-16T16:51:35.104 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.081+0000 7f95633c3b80 1 -- 172.21.15.44:0/1701899843 wait complete. 2023-11-16T16:51:35.104 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.082+0000 7f95633c3b80 20 client.15876 trim_cache size 0 max 16384 2023-11-16T16:51:35.104 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T16:51:35.105 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.082+0000 7f954f9ff700 1 -- 172.21.15.44:0/2052880286 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f95340a1340 msgr2=0x7f95340a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:35.105 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.082+0000 7f954f9ff700 1 --2- 172.21.15.44:0/2052880286 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f95340a1340 0x7f95340a37f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f954400b2e0 tx=0x7f9544005fb0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.105 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.082+0000 7f954f9ff700 1 -- 172.21.15.44:0/2052880286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f954810cec0 msgr2=0x7f954819d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:35.105 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.082+0000 7f954f9ff700 1 --2- 172.21.15.44:0/2052880286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f954810cec0 0x7f954819d010 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f954000ef90 tx=0x7f954000c3a0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.105 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.082+0000 7f954f9ff700 1 -- 172.21.15.44:0/2052880286 shutdown_connections 2023-11-16T16:51:35.105 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.082+0000 7f954f9ff700 1 --2- 172.21.15.44:0/2052880286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f954810cec0 0x7f954819d010 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.106 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.082+0000 7f954f9ff700 1 --2- 172.21.15.44:0/2052880286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95480746a0 0x7f9548198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.106 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.082+0000 7f954f9ff700 1 --2- 172.21.15.44:0/2052880286 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f95340a1340 0x7f95340a37f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.106 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.082+0000 7f954f9ff700 1 --2- 172.21.15.44:0/2052880286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9548074b00 0x7f9548198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.106 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.082+0000 7f954f9ff700 1 -- 172.21.15.44:0/2052880286 >> 172.21.15.44:0/2052880286 conn(0x7f9548077590 msgr2=0x7f9548104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:35.106 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.082+0000 7f954f9ff700 1 -- 172.21.15.44:0/2052880286 shutdown_connections 2023-11-16T16:51:35.106 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:35.082+0000 7f954f9ff700 1 -- 172.21.15.44:0/2052880286 wait complete. 2023-11-16T16:51:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.104+0000 7f42f4ff9700 1 -- 172.21.15.44:0/2666137416 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560baaefa600 con 0x560bab047af0 2023-11-16T16:51:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.104+0000 7f42f4ff9700 10 client.25892 handle_command_reply: tid=0 2023-11-16T16:51:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.104+0000 7f43271e2b80 1 client.25892 shutdown 2023-11-16T16:51:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.104+0000 7f43271e2b80 1 -- 172.21.15.44:0/2666137416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baae366b0 msgr2=0x560baafe7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.104+0000 7f43271e2b80 1 --2- 172.21.15.44:0/2666137416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baae366b0 0x560baafe7210 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4308021b40 tx=0x7f430809b030 comp rx=0 tx=0).stop 2023-11-16T16:51:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.104+0000 7f43271e2b80 1 -- 172.21.15.44:0/2666137416 shutdown_connections 2023-11-16T16:51:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.104+0000 7f43271e2b80 1 --2- 172.21.15.44:0/2666137416 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560bab047af0 0x560bab049ed0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.104+0000 7f43271e2b80 1 --2- 172.21.15.44:0/2666137416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560baafe0b90 0x560baafe7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.104+0000 7f43271e2b80 1 --2- 172.21.15.44:0/2666137416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560baae366b0 0x560baafe7210 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.104+0000 7f42f4ff9700 0 client.25892 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.104+0000 7f43271e2b80 1 --2- 172.21.15.44:0/2666137416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560baafe0720 0x560baafe7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.104+0000 7f43271e2b80 1 -- 172.21.15.44:0/2666137416 >> 172.21.15.44:0/2666137416 conn(0x560baafcff50 msgr2=0x560baafdebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.104+0000 7f43271e2b80 1 -- 172.21.15.44:0/2666137416 shutdown_connections 2023-11-16T16:51:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.104+0000 7f43271e2b80 1 -- 172.21.15.44:0/2666137416 wait complete. 2023-11-16T16:51:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.104+0000 7f43271e2b80 20 client.25892 trim_cache size 0 max 16384 2023-11-16T16:51:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:51:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:51:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:51:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:51:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.105+0000 7f43137ff700 1 -- 172.21.15.44:0/1386808828 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f42f80a12c0 msgr2=0x7f42f80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.105+0000 7f43137ff700 1 --2- 172.21.15.44:0/1386808828 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f42f80a12c0 0x7f42f80a3770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f430c100880 tx=0x7f42fc009260 comp rx=0 tx=0).stop 2023-11-16T16:51:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.105+0000 7f43137ff700 1 -- 172.21.15.44:0/1386808828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f430c101440 msgr2=0x7f430c0ff090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.105+0000 7f43137ff700 1 --2- 172.21.15.44:0/1386808828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f430c101440 0x7f430c0ff090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4308000c00 tx=0x7f4308005740 comp rx=0 tx=0).stop 2023-11-16T16:51:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.105+0000 7f43137ff700 1 -- 172.21.15.44:0/1386808828 shutdown_connections 2023-11-16T16:51:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.105+0000 7f43137ff700 1 --2- 172.21.15.44:0/1386808828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f430c10a810 0x7f430c0ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.105+0000 7f43137ff700 1 --2- 172.21.15.44:0/1386808828 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f42f80a12c0 0x7f42f80a3770 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.105+0000 7f43137ff700 1 --2- 172.21.15.44:0/1386808828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f430c101440 0x7f430c0ff090 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.105+0000 7f43137ff700 1 --2- 172.21.15.44:0/1386808828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c101d50 0x7f430c0ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.105+0000 7f43137ff700 1 -- 172.21.15.44:0/1386808828 >> 172.21.15.44:0/1386808828 conn(0x7f430c0763a0 msgr2=0x7f430c0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.105+0000 7f43137ff700 1 -- 172.21.15.44:0/1386808828 shutdown_connections 2023-11-16T16:51:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:35.105+0000 7f43137ff700 1 -- 172.21.15.44:0/1386808828 wait complete. 2023-11-16T16:51:35.240 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:35.240 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:35.438 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:35.438 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:35.460 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:35.460 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:35.460 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 26 -lt 100 ']' 2023-11-16T16:51:35.460 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 26 + 1 2023-11-16T16:51:35.494 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=27 2023-11-16T16:51:35.494 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:35.525 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:35.525 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:35.527 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:35.528 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:35.528 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:36.058 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:36.059 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:36.204 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:36.204 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:36.219 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:36.219 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:36.219 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 27 -lt 100 ']' 2023-11-16T16:51:36.220 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 27 + 1 2023-11-16T16:51:36.239 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:51:36.239 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:51:36.252 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=28 2023-11-16T16:51:36.252 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:36.286 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:36.286 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:36.288 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:36.289 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:36.289 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:36.834 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:36.834 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:36.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:36 smithi077 ceph-mon[131204]: pgmap v53: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 3 op/s 2023-11-16T16:51:36.978 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:36.978 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:36.984 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:36.984 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:36.984 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 28 -lt 100 ']' 2023-11-16T16:51:36.984 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 28 + 1 2023-11-16T16:51:37.032 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=29 2023-11-16T16:51:37.033 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:37.074 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:37.074 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:37.077 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:37.077 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:37.078 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:37.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:36 smithi044 ceph-mon[128750]: pgmap v53: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 3 op/s 2023-11-16T16:51:37.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:36 smithi176 ceph-mon[131630]: pgmap v53: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 3 op/s 2023-11-16T16:51:37.817 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:37.817 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:38.204 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:38.204 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:38.235 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:38.235 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:38.235 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 29 -lt 100 ']' 2023-11-16T16:51:38.236 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 29 + 1 2023-11-16T16:51:38.274 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=30 2023-11-16T16:51:38.274 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:38.309 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:38.310 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:38.313 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:38.313 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:38.314 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:38.906 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.904+0000 7ff0883ff700 1 -- 172.21.15.44:0/1138628837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080105620 msgr2=0x7ff08010d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:38.906 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.904+0000 7ff0883ff700 1 --2- 172.21.15.44:0/1138628837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080105620 0x7ff08010d1a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff078009fc0 tx=0x7ff07800b330 comp rx=0 tx=0).stop 2023-11-16T16:51:38.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.904+0000 7ff0883ff700 1 -- 172.21.15.44:0/1138628837 shutdown_connections 2023-11-16T16:51:38.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.904+0000 7ff0883ff700 1 --2- 172.21.15.44:0/1138628837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080105620 0x7ff08010d1a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:38.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.904+0000 7ff0883ff700 1 --2- 172.21.15.44:0/1138628837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff080100cc0 0x7ff080101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:38.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.904+0000 7ff0883ff700 1 --2- 172.21.15.44:0/1138628837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff080101660 0x7ff080104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:38.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.904+0000 7ff0883ff700 1 -- 172.21.15.44:0/1138628837 >> 172.21.15.44:0/1138628837 conn(0x7ff080077560 msgr2=0x7ff080077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:38.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.904+0000 7ff0883ff700 1 -- 172.21.15.44:0/1138628837 shutdown_connections 2023-11-16T16:51:38.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.904+0000 7ff0883ff700 1 -- 172.21.15.44:0/1138628837 wait complete. 2023-11-16T16:51:38.909 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.905+0000 7ff0883ff700 1 Processor -- start 2023-11-16T16:51:38.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.905+0000 7ff0883ff700 1 -- start start 2023-11-16T16:51:38.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.906+0000 7ff0883ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff080100cc0 0x7ff080072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:38.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.906+0000 7ff0883ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080101660 0x7ff080073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:38.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.906+0000 7ff0883ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff080105620 0x7ff0801a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:38.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.906+0000 7ff0883ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff080073b60 con 0x7ff080101660 2023-11-16T16:51:38.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.906+0000 7ff0883ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0801a4cf0 con 0x7ff080100cc0 2023-11-16T16:51:38.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.906+0000 7ff0883ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0801a4e30 con 0x7ff080105620 2023-11-16T16:51:38.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.906+0000 7ff08599a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080101660 0x7ff080073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:38.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.906+0000 7ff08599a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080101660 0x7ff080073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34794/0 (socket says 172.21.15.44:34794) 2023-11-16T16:51:38.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.906+0000 7ff08599a700 1 -- 172.21.15.44:0/2851899988 learned_addr learned my addr 172.21.15.44:0/2851899988 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:38.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.906+0000 7ff08619b700 1 --2- 172.21.15.44:0/2851899988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff080100cc0 0x7ff080072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:38.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.906+0000 7ff08699c700 1 --2- 172.21.15.44:0/2851899988 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff080105620 0x7ff0801a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:38.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.906+0000 7ff08599a700 1 -- 172.21.15.44:0/2851899988 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff080105620 msgr2=0x7ff0801a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:38.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.906+0000 7ff08599a700 1 --2- 172.21.15.44:0/2851899988 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff080105620 0x7ff0801a47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:38.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.907+0000 7ff08599a700 1 -- 172.21.15.44:0/2851899988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff080100cc0 msgr2=0x7ff080072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:38.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.907+0000 7ff08599a700 1 --2- 172.21.15.44:0/2851899988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff080100cc0 0x7ff080072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:38.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.907+0000 7ff08599a700 1 -- 172.21.15.44:0/2851899988 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff078009c40 con 0x7ff080101660 2023-11-16T16:51:38.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.907+0000 7ff08619b700 1 --2- 172.21.15.44:0/2851899988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff080100cc0 0x7ff080072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:38.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.907+0000 7ff08599a700 1 --2- 172.21.15.44:0/2851899988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080101660 0x7ff080073330 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff07000e910 tx=0x7ff07000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:38.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.907+0000 7ff0777fe700 1 -- 172.21.15.44:0/2851899988 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07000cbd0 con 0x7ff080101660 2023-11-16T16:51:38.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.907+0000 7ff0883ff700 1 -- 172.21.15.44:0/2851899988 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff080110390 con 0x7ff080101660 2023-11-16T16:51:38.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.907+0000 7ff0777fe700 1 -- 172.21.15.44:0/2851899988 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff07000ee50 con 0x7ff080101660 2023-11-16T16:51:38.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.907+0000 7ff0777fe700 1 -- 172.21.15.44:0/2851899988 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0700055c0 con 0x7ff080101660 2023-11-16T16:51:38.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.907+0000 7ff0883ff700 1 -- 172.21.15.44:0/2851899988 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff080110200 con 0x7ff080101660 2023-11-16T16:51:38.913 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.910+0000 7ff0777fe700 1 -- 172.21.15.44:0/2851899988 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7ff070004750 con 0x7ff080101660 2023-11-16T16:51:38.913 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.912+0000 7ff0777fe700 1 --2- 172.21.15.44:0/2851899988 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff06c0a12c0 0x7ff06c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:38.914 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.912+0000 7ff0777fe700 1 -- 172.21.15.44:0/2851899988 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7ff070014070 con 0x7ff080101660 2023-11-16T16:51:38.914 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.912+0000 7ff08619b700 1 --2- 172.21.15.44:0/2851899988 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff06c0a12c0 0x7ff06c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:38.914 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:38.912+0000 7ff08619b700 1 --2- 172.21.15.44:0/2851899988 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff06c0a12c0 0x7ff06c0a3770 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff07c005fd0 tx=0x7ff07c009310 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:38.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:38 smithi077 ceph-mon[131204]: pgmap v54: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 30 KiB/s wr, 11 op/s 2023-11-16T16:51:38.984 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:38.984 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:39.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.023+0000 7ff09bd6ab80 1 Processor -- start 2023-11-16T16:51:39.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.023+0000 7ff09bd6ab80 1 -- start start 2023-11-16T16:51:39.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.023+0000 7ff09bd6ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13eefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:39.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.023+0000 7ff09bd6ab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:39.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.023+0000 7ff09bd6ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13f3930 0x55e4b13f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:39.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.023+0000 7ff09bd6ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b12f40e0 con 0x55e4b13eebe0 2023-11-16T16:51:39.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.023+0000 7ff09bd6ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b1346920 con 0x55e4b13f3930 2023-11-16T16:51:39.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.023+0000 7ff09bd6ab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b133fdb0 con 0x55e4b13ef4f0 2023-11-16T16:51:39.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.023+0000 7ff08619b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13eefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:39.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.023+0000 7ff08619b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13eefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34808/0 (socket says 172.21.15.44:34808) 2023-11-16T16:51:39.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.023+0000 7ff08619b700 1 -- 172.21.15.44:0/1387773743 learned_addr learned my addr 172.21.15.44:0/1387773743 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:39.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.023+0000 7ff08699c700 1 --2- 172.21.15.44:0/1387773743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13f3930 0x55e4b13f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:39.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.023+0000 7ff08599a700 1 --2- 172.21.15.44:0/1387773743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:39.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.024+0000 7ff08619b700 1 -- 172.21.15.44:0/1387773743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13ef4f0 msgr2=0x55e4b13f33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:39.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.024+0000 7ff08619b700 1 --2- 172.21.15.44:0/1387773743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.024+0000 7ff08619b700 1 -- 172.21.15.44:0/1387773743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13f3930 msgr2=0x55e4b13f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:39.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.024+0000 7ff08619b700 1 --2- 172.21.15.44:0/1387773743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13f3930 0x55e4b13f3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.024+0000 7ff08619b700 1 -- 172.21.15.44:0/1387773743 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4b11c16b0 con 0x55e4b13eebe0 2023-11-16T16:51:39.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.024+0000 7ff08699c700 1 --2- 172.21.15.44:0/1387773743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13f3930 0x55e4b13f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:39.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.024+0000 7ff08619b700 1 --2- 172.21.15.44:0/1387773743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13eefb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff07c0043c0 tx=0x7ff07c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22fd47e3390097ce server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:39.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.024+0000 7ff074bfe700 1 -- 172.21.15.44:0/1387773743 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c003d10 con 0x55e4b13eebe0 2023-11-16T16:51:39.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.024+0000 7ff074bfe700 1 -- 172.21.15.44:0/1387773743 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff07c003e70 con 0x55e4b13eebe0 2023-11-16T16:51:39.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.024+0000 7ff074bfe700 1 -- 172.21.15.44:0/1387773743 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c01e780 con 0x55e4b13eebe0 2023-11-16T16:51:39.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.025+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/1387773743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 msgr2=0x55e4b13eefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:39.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.025+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/1387773743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13eefb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff07c0043c0 tx=0x7ff07c014000 comp rx=0 tx=0).stop 2023-11-16T16:51:39.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.025+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/1387773743 shutdown_connections 2023-11-16T16:51:39.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.025+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/1387773743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13f3930 0x55e4b13f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.025+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/1387773743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13eefb0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.025+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/1387773743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.025+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/1387773743 >> 172.21.15.44:0/1387773743 conn(0x55e4b13e43e0 msgr2=0x55e4b13e9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:39.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.025+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/1387773743 shutdown_connections 2023-11-16T16:51:39.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.025+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/1387773743 wait complete. 2023-11-16T16:51:39.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.026+0000 7ff09bd6ab80 1 -- start start 2023-11-16T16:51:39.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.026+0000 7ff09bd6ab80 1 Processor -- start 2023-11-16T16:51:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff09bd6ab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13eebe0 0x55e4b13fb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff09bd6ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff09bd6ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13f3930 0x55e4b13f8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff09bd6ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b12e1960 con 0x55e4b13f3930 2023-11-16T16:51:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff09bd6ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b12e1600 con 0x55e4b13ef4f0 2023-11-16T16:51:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff09bd6ab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b11c39a0 con 0x55e4b13eebe0 2023-11-16T16:51:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff09bd6ab80 10 client.0 _ll_register_callbacks cb 0x55e4b12ff800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff08699c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13f3930 0x55e4b13f8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff08699c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13f3930 0x55e4b13f8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34812/0 (socket says 172.21.15.44:34812) 2023-11-16T16:51:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff08699c700 1 -- 172.21.15.44:0/2985975199 learned_addr learned my addr 172.21.15.44:0/2985975199 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff08619b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13eebe0 0x55e4b13fb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff08599a700 1 --2- 172.21.15.44:0/2985975199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff08599a700 1 -- 172.21.15.44:0/2985975199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13eebe0 msgr2=0x55e4b13fb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff08599a700 1 --2- 172.21.15.44:0/2985975199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13eebe0 0x55e4b13fb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff08599a700 1 -- 172.21.15.44:0/2985975199 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13f3930 msgr2=0x55e4b13f8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff08599a700 1 --2- 172.21.15.44:0/2985975199 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13f3930 0x55e4b13f8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff08599a700 1 -- 172.21.15.44:0/2985975199 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07c017070 con 0x55e4b13ef4f0 2023-11-16T16:51:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.027+0000 7ff08619b700 1 --2- 172.21.15.44:0/2985975199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13eebe0 0x55e4b13fb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.028+0000 7ff08599a700 1 --2- 172.21.15.44:0/2985975199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f8310 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff0700ca7a0 tx=0x7ff070099b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1389e3dd35768c39 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.028+0000 7ff069ffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:51:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.028+0000 7ff069ffb700 1 -- 172.21.15.44:0/2985975199 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07009a580 con 0x55e4b13ef4f0 2023-11-16T16:51:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.028+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/2985975199 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e4b13e1ba0 con 0x55e4b13ef4f0 2023-11-16T16:51:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.028+0000 7ff069ffb700 1 -- 172.21.15.44:0/2985975199 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7ff07009e150 con 0x55e4b13ef4f0 2023-11-16T16:51:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.028+0000 7ff069ffb700 1 -- 172.21.15.44:0/2985975199 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0700a2570 con 0x55e4b13ef4f0 2023-11-16T16:51:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.028+0000 7ff09bd6ab80 10 client.25904 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.028+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/2985975199 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55e4b1306210 con 0x55e4b13ef4f0 2023-11-16T16:51:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.028+0000 7ff069ffb700 1 -- 172.21.15.44:0/2985975199 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e4b13e1ba0 con 0x55e4b13ef4f0 2023-11-16T16:51:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.029+0000 7ff069ffb700 1 -- 172.21.15.44:0/2985975199 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff0700b2b00 con 0x55e4b13ef4f0 2023-11-16T16:51:39.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.029+0000 7ff09bd6ab80 10 client.25904 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:39.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.029+0000 7ff09bd6ab80 10 client.25904 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:39.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.029+0000 7ff09bd6ab80 20 client.25904 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:39.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.029+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/2985975199 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e4b1408930 0x55e4b140ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:39.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.029+0000 7ff09bd6ab80 4 client.25904 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:51:39.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.029+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/2985975199 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e4b1302120 con 0x55e4b1408930 2023-11-16T16:51:39.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.029+0000 7ff08619b700 1 --2- 172.21.15.44:0/2985975199 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e4b1408930 0x55e4b140ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:39.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.029+0000 7ff08619b700 1 --2- 172.21.15.44:0/2985975199 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e4b1408930 0x55e4b140ad40 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d892062353623ec server_cookie=881d6be9e6bc46ce in_seq=0 out_seq=0 2023-11-16T16:51:39.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.029+0000 7ff069ffb700 10 client.25904 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:39.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.030+0000 7ff069ffb700 1 -- 172.21.15.44:0/2985975199 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e4b1302120 con 0x55e4b1408930 2023-11-16T16:51:39.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.030+0000 7ff069ffb700 10 client.25904 handle_command_reply: tid=0 2023-11-16T16:51:39.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.030+0000 7ff09bd6ab80 1 client.25904 shutdown 2023-11-16T16:51:39.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.030+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/2985975199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 msgr2=0x55e4b13f8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:39.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.030+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/2985975199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f8310 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff0700ca7a0 tx=0x7ff070099b70 comp rx=0 tx=0).stop 2023-11-16T16:51:39.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.031+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/2985975199 shutdown_connections 2023-11-16T16:51:39.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.031+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/2985975199 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e4b1408930 0x55e4b140ad40 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.031+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/2985975199 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13f3930 0x55e4b13f8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.031+0000 7ff069ffb700 0 client.25904 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:39.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.031+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/2985975199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13eebe0 0x55e4b13fb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.031+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/2985975199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f8310 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.031+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/2985975199 >> 172.21.15.44:0/2985975199 conn(0x55e4b13e43e0 msgr2=0x55e4b13f4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:39.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.031+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/2985975199 shutdown_connections 2023-11-16T16:51:39.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.031+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/2985975199 wait complete. 2023-11-16T16:51:39.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.031+0000 7ff09bd6ab80 20 client.25904 trim_cache size 0 max 16384 2023-11-16T16:51:39.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.049+0000 7ff09bd6ab80 1 Processor -- start 2023-11-16T16:51:39.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.049+0000 7ff09bd6ab80 1 -- start start 2023-11-16T16:51:39.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.049+0000 7ff09bd6ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13ffa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:39.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.049+0000 7ff09bd6ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:39.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.049+0000 7ff09bd6ab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13f3930 0x55e4b13f55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:39.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.049+0000 7ff09bd6ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b13e8cc0 con 0x55e4b13eebe0 2023-11-16T16:51:39.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.049+0000 7ff09bd6ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b13e1790 con 0x55e4b13ef4f0 2023-11-16T16:51:39.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.049+0000 7ff09bd6ab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b13eb4b0 con 0x55e4b13f3930 2023-11-16T16:51:39.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.049+0000 7ff08619b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13ffa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:39.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.049+0000 7ff08619b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13ffa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34814/0 (socket says 172.21.15.44:34814) 2023-11-16T16:51:39.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.049+0000 7ff08619b700 1 -- 172.21.15.44:0/3711493740 learned_addr learned my addr 172.21.15.44:0/3711493740 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:39.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.049+0000 7ff08599a700 1 --2- 172.21.15.44:0/3711493740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:39.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.049+0000 7ff08699c700 1 --2- 172.21.15.44:0/3711493740 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13f3930 0x55e4b13f55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:39.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.050+0000 7ff08619b700 1 -- 172.21.15.44:0/3711493740 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13f3930 msgr2=0x55e4b13f55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:39.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.050+0000 7ff08619b700 1 --2- 172.21.15.44:0/3711493740 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13f3930 0x55e4b13f55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.050+0000 7ff08619b700 1 -- 172.21.15.44:0/3711493740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 msgr2=0x55e4b13f4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:39.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.050+0000 7ff08619b700 1 --2- 172.21.15.44:0/3711493740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.050+0000 7ff08619b700 1 -- 172.21.15.44:0/3711493740 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4b13e5b80 con 0x55e4b13eebe0 2023-11-16T16:51:39.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.050+0000 7ff08599a700 1 --2- 172.21.15.44:0/3711493740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:39.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.050+0000 7ff08619b700 1 --2- 172.21.15.44:0/3711493740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13ffa20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff07c003e40 tx=0x7ff07c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=757f2f538bfd2967 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:39.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.050+0000 7ff0753ff700 1 -- 172.21.15.44:0/3711493740 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c02b050 con 0x55e4b13eebe0 2023-11-16T16:51:39.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.050+0000 7ff0753ff700 1 -- 172.21.15.44:0/3711493740 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff07c02c040 con 0x55e4b13eebe0 2023-11-16T16:51:39.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.050+0000 7ff0753ff700 1 -- 172.21.15.44:0/3711493740 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c004930 con 0x55e4b13eebe0 2023-11-16T16:51:39.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.051+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/3711493740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 msgr2=0x55e4b13ffa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:39.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.051+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/3711493740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13ffa20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff07c003e40 tx=0x7ff07c017000 comp rx=0 tx=0).stop 2023-11-16T16:51:39.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.051+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/3711493740 shutdown_connections 2023-11-16T16:51:39.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.051+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/3711493740 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13f3930 0x55e4b13f55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.051+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/3711493740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13ffa20 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.051+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/3711493740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.051+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/3711493740 >> 172.21.15.44:0/3711493740 conn(0x55e4b13e43e0 msgr2=0x55e4b13f19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:39.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.051+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/3711493740 shutdown_connections 2023-11-16T16:51:39.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.051+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/3711493740 wait complete. 2023-11-16T16:51:39.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.051+0000 7ff09bd6ab80 1 -- start start 2023-11-16T16:51:39.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.051+0000 7ff09bd6ab80 1 Processor -- start 2023-11-16T16:51:39.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.052+0000 7ff09bd6ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13ffa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:39.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.052+0000 7ff09bd6ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:39.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.052+0000 7ff09bd6ab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13f3930 0x55e4b13f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:39.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.052+0000 7ff09bd6ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b13f7760 con 0x55e4b13eebe0 2023-11-16T16:51:39.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.052+0000 7ff09bd6ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b13f78d0 con 0x55e4b13ef4f0 2023-11-16T16:51:39.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.052+0000 7ff09bd6ab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b145c7b0 con 0x55e4b13f3930 2023-11-16T16:51:39.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.052+0000 7ff09bd6ab80 10 client.0 _ll_register_callbacks cb 0x7ff070099c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:39.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.052+0000 7ff08619b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13ffa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:39.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.052+0000 7ff08619b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13ffa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34816/0 (socket says 172.21.15.44:34816) 2023-11-16T16:51:39.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.052+0000 7ff08619b700 1 -- 172.21.15.44:0/1940185010 learned_addr learned my addr 172.21.15.44:0/1940185010 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:39.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.052+0000 7ff08699c700 1 --2- 172.21.15.44:0/1940185010 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13f3930 0x55e4b13f70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:39.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.052+0000 7ff08599a700 1 --2- 172.21.15.44:0/1940185010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:39.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.053+0000 7ff08619b700 1 -- 172.21.15.44:0/1940185010 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13f3930 msgr2=0x55e4b13f70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:39.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.053+0000 7ff08619b700 1 --2- 172.21.15.44:0/1940185010 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13f3930 0x55e4b13f70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.053+0000 7ff08619b700 1 -- 172.21.15.44:0/1940185010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 msgr2=0x55e4b13f6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:39.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.053+0000 7ff08619b700 1 --2- 172.21.15.44:0/1940185010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.053+0000 7ff08619b700 1 -- 172.21.15.44:0/1940185010 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4b1306210 con 0x55e4b13eebe0 2023-11-16T16:51:39.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.053+0000 7ff08619b700 1 --2- 172.21.15.44:0/1940185010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13ffa30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff07c005dd0 tx=0x7ff07c005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988cafd496a6f8f2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:39.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.053+0000 7ff069ffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:51:39.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.053+0000 7ff069ffb700 1 -- 172.21.15.44:0/1940185010 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c033550 con 0x55e4b13eebe0 2023-11-16T16:51:39.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.053+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/1940185010 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff07c003e70 con 0x55e4b13eebe0 2023-11-16T16:51:39.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.053+0000 7ff069ffb700 1 -- 172.21.15.44:0/1940185010 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7ff07c033a40 con 0x55e4b13eebe0 2023-11-16T16:51:39.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.053+0000 7ff069ffb700 1 -- 172.21.15.44:0/1940185010 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff07c02b020 con 0x55e4b13eebe0 2023-11-16T16:51:39.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.053+0000 7ff09bd6ab80 10 client.15894 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:39.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.053+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/1940185010 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55e4b11c16b0 con 0x55e4b13eebe0 2023-11-16T16:51:39.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.054+0000 7ff069ffb700 1 -- 172.21.15.44:0/1940185010 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff07c003e70 con 0x55e4b13eebe0 2023-11-16T16:51:39.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.054+0000 7ff069ffb700 1 -- 172.21.15.44:0/1940185010 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff07c03b070 con 0x55e4b13eebe0 2023-11-16T16:51:39.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.054+0000 7ff09bd6ab80 10 client.15894 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:39.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.054+0000 7ff09bd6ab80 10 client.15894 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:39.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.054+0000 7ff09bd6ab80 20 client.15894 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:39.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.054+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/1940185010 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e4b145d910 0x55e4b145fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:39.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.054+0000 7ff09bd6ab80 4 client.15894 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:51:39.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.054+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/1940185010 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e4b130e6e0 con 0x55e4b145d910 2023-11-16T16:51:39.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.054+0000 7ff08599a700 1 --2- 172.21.15.44:0/1940185010 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e4b145d910 0x55e4b145fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:39.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.055+0000 7ff08599a700 1 --2- 172.21.15.44:0/1940185010 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e4b145d910 0x55e4b145fcf0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daec2929a157d531 server_cookie=c313f11c9828a054 in_seq=0 out_seq=0 2023-11-16T16:51:39.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.055+0000 7ff069ffb700 10 client.15894 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:39.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.056+0000 7ff069ffb700 1 -- 172.21.15.44:0/1940185010 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e4b130e6e0 con 0x55e4b145d910 2023-11-16T16:51:39.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.056+0000 7ff069ffb700 10 client.15894 handle_command_reply: tid=0 2023-11-16T16:51:39.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.056+0000 7ff09bd6ab80 1 client.15894 shutdown 2023-11-16T16:51:39.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.056+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/1940185010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 msgr2=0x55e4b13ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:39.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.056+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/1940185010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13ffa30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff07c005dd0 tx=0x7ff07c005e00 comp rx=0 tx=0).stop 2023-11-16T16:51:39.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.056+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/1940185010 shutdown_connections 2023-11-16T16:51:39.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.056+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/1940185010 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e4b145d910 0x55e4b145fcf0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.056+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/1940185010 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e4b13f3930 0x55e4b13f70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.056+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/1940185010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4b13eebe0 0x55e4b13ffa30 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.056+0000 7ff09bd6ab80 1 --2- 172.21.15.44:0/1940185010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b13ef4f0 0x55e4b13f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.056+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/1940185010 >> 172.21.15.44:0/1940185010 conn(0x55e4b13e43e0 msgr2=0x55e4b13f55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:39.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.056+0000 7ff069ffb700 0 client.15894 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:39.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.057+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/1940185010 shutdown_connections 2023-11-16T16:51:39.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.057+0000 7ff09bd6ab80 1 -- 172.21.15.44:0/1940185010 wait complete. 2023-11-16T16:51:39.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.057+0000 7ff09bd6ab80 20 client.15894 trim_cache size 0 max 16384 2023-11-16T16:51:39.066 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:51:39.066 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:51:39.066 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "72a0e759-1bbb-4056-af62-d033c8b97e6b", 2023-11-16T16:51:39.066 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:51:39.066 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:51:39.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.057+0000 7ff0883ff700 1 -- 172.21.15.44:0/2851899988 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff06c0a12c0 msgr2=0x7ff06c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:39.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.057+0000 7ff0883ff700 1 --2- 172.21.15.44:0/2851899988 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff06c0a12c0 0x7ff06c0a3770 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff07c005fd0 tx=0x7ff07c009310 comp rx=0 tx=0).stop 2023-11-16T16:51:39.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.057+0000 7ff0883ff700 1 -- 172.21.15.44:0/2851899988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080101660 msgr2=0x7ff080073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:39.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.057+0000 7ff0883ff700 1 --2- 172.21.15.44:0/2851899988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080101660 0x7ff080073330 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff07000e910 tx=0x7ff07000ecd0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.057+0000 7ff0883ff700 1 -- 172.21.15.44:0/2851899988 shutdown_connections 2023-11-16T16:51:39.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.057+0000 7ff0883ff700 1 --2- 172.21.15.44:0/2851899988 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff06c0a12c0 0x7ff06c0a3770 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.057+0000 7ff0883ff700 1 --2- 172.21.15.44:0/2851899988 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff080105620 0x7ff0801a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.057+0000 7ff0883ff700 1 --2- 172.21.15.44:0/2851899988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff080100cc0 0x7ff080072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.057+0000 7ff0883ff700 1 --2- 172.21.15.44:0/2851899988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080101660 0x7ff080073330 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:39.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.057+0000 7ff0883ff700 1 -- 172.21.15.44:0/2851899988 >> 172.21.15.44:0/2851899988 conn(0x7ff080077560 msgr2=0x7ff080103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:39.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.058+0000 7ff0883ff700 1 -- 172.21.15.44:0/2851899988 shutdown_connections 2023-11-16T16:51:39.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:39.058+0000 7ff0883ff700 1 -- 172.21.15.44:0/2851899988 wait complete. 2023-11-16T16:51:39.101 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:38 smithi044 ceph-mon[128750]: pgmap v54: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 30 KiB/s wr, 11 op/s 2023-11-16T16:51:39.104 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:51:39.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:38 smithi176 ceph-mon[131630]: pgmap v54: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 30 KiB/s wr, 11 op/s 2023-11-16T16:51:39.203 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:39.203 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:39.218 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:39.219 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:39.219 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 30 -lt 100 ']' 2023-11-16T16:51:39.219 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 30 + 1 2023-11-16T16:51:39.249 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=31 2023-11-16T16:51:39.249 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:39.272 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:39.272 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:39.275 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:39.276 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:39.276 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:39.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:39 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:51:39.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:39 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:51:39.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:39 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:51:39.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:39 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:51:39.875 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:39.875 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:39.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:39 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:51:39.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:39 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:51:39.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:39 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:51:39.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:39 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:51:40.004 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:40.004 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:40.019 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:40.019 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:40.019 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 31 -lt 100 ']' 2023-11-16T16:51:40.020 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 31 + 1 2023-11-16T16:51:40.049 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=32 2023-11-16T16:51:40.049 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:40.097 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:40.097 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:40.099 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:40.100 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:40.100 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:39 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:51:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:39 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:51:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:39 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:51:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:39 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:51:40.708 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:40.709 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:40.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:40 smithi077 ceph-mon[131204]: pgmap v55: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-11-16T16:51:40.979 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:40.979 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:41.003 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:41.003 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:41.003 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 32 -lt 100 ']' 2023-11-16T16:51:41.004 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 32 + 1 2023-11-16T16:51:41.085 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=33 2023-11-16T16:51:41.085 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:41.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:40 smithi044 ceph-mon[128750]: pgmap v55: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-11-16T16:51:41.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:40 smithi176 ceph-mon[131630]: pgmap v55: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-11-16T16:51:41.166 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:41.166 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:41.169 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:41.169 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:41.170 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:42.241 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:42.242 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:42.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.437+0000 7fddcb1ff700 1 -- 172.21.15.44:0/674596128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddc4100cc0 msgr2=0x7fddc4101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:42.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.437+0000 7fddcb1ff700 1 --2- 172.21.15.44:0/674596128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddc4100cc0 0x7fddc4101090 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fddb80099c0 tx=0x7fddb8009cd0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.438+0000 7fddcb1ff700 1 -- 172.21.15.44:0/674596128 shutdown_connections 2023-11-16T16:51:42.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.438+0000 7fddcb1ff700 1 --2- 172.21.15.44:0/674596128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddc4105620 0x7fddc410d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.438+0000 7fddcb1ff700 1 --2- 172.21.15.44:0/674596128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddc4100cc0 0x7fddc4101090 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.438+0000 7fddcb1ff700 1 --2- 172.21.15.44:0/674596128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fddc4101660 0x7fddc4104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.438+0000 7fddcb1ff700 1 -- 172.21.15.44:0/674596128 >> 172.21.15.44:0/674596128 conn(0x7fddc4077560 msgr2=0x7fddc4077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:42.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.438+0000 7fddcb1ff700 1 -- 172.21.15.44:0/674596128 shutdown_connections 2023-11-16T16:51:42.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.438+0000 7fddcb1ff700 1 -- 172.21.15.44:0/674596128 wait complete. 2023-11-16T16:51:42.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.439+0000 7fddcb1ff700 1 Processor -- start 2023-11-16T16:51:42.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.439+0000 7fddcb1ff700 1 -- start start 2023-11-16T16:51:42.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.439+0000 7fddcb1ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fddc4100cc0 0x7fddc4198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:42.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.439+0000 7fddcb1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddc4101660 0x7fddc4198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:42.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.439+0000 7fddcb1ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddc4105620 0x7fddc419d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:42.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.439+0000 7fddcb1ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc419d660 con 0x7fddc4105620 2023-11-16T16:51:42.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.439+0000 7fddcb1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc419d7d0 con 0x7fddc4101660 2023-11-16T16:51:42.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.439+0000 7fddcb1ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc419d990 con 0x7fddc4100cc0 2023-11-16T16:51:42.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.440+0000 7fddc979c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddc4105620 0x7fddc419d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:42.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.440+0000 7fddc979c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddc4105620 0x7fddc419d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34832/0 (socket says 172.21.15.44:34832) 2023-11-16T16:51:42.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.440+0000 7fddc979c700 1 -- 172.21.15.44:0/3045070880 learned_addr learned my addr 172.21.15.44:0/3045070880 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:42.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.440+0000 7fddc3fff700 1 --2- 172.21.15.44:0/3045070880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddc4101660 0x7fddc4198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:42.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.440+0000 7fddc8f9b700 1 --2- 172.21.15.44:0/3045070880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fddc4100cc0 0x7fddc4198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:42.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.440+0000 7fddc979c700 1 -- 172.21.15.44:0/3045070880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fddc4100cc0 msgr2=0x7fddc4198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:42.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.440+0000 7fddc979c700 1 --2- 172.21.15.44:0/3045070880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fddc4100cc0 0x7fddc4198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.440+0000 7fddc979c700 1 -- 172.21.15.44:0/3045070880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddc4101660 msgr2=0x7fddc4198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:42.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.440+0000 7fddc979c700 1 --2- 172.21.15.44:0/3045070880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddc4101660 0x7fddc4198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.440+0000 7fddc979c700 1 -- 172.21.15.44:0/3045070880 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb80095f0 con 0x7fddc4105620 2023-11-16T16:51:42.444 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.441+0000 7fddc979c700 1 --2- 172.21.15.44:0/3045070880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddc4105620 0x7fddc419d020 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fddbc00ef30 tx=0x7fddbc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:42.444 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.441+0000 7fddc1ffb700 1 -- 172.21.15.44:0/3045070880 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddbc018070 con 0x7fddc4105620 2023-11-16T16:51:42.444 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.441+0000 7fddc1ffb700 1 -- 172.21.15.44:0/3045070880 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fddbc00f040 con 0x7fddc4105620 2023-11-16T16:51:42.444 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.441+0000 7fddc1ffb700 1 -- 172.21.15.44:0/3045070880 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddbc00cd00 con 0x7fddc4105620 2023-11-16T16:51:42.444 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.441+0000 7fddcb1ff700 1 -- 172.21.15.44:0/3045070880 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc41103d0 con 0x7fddc4105620 2023-11-16T16:51:42.444 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.441+0000 7fddcb1ff700 1 -- 172.21.15.44:0/3045070880 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc4110240 con 0x7fddc4105620 2023-11-16T16:51:42.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.444+0000 7fddc1ffb700 1 -- 172.21.15.44:0/3045070880 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fddbc003a00 con 0x7fddc4105620 2023-11-16T16:51:42.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.446+0000 7fddc1ffb700 1 --2- 172.21.15.44:0/3045070880 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fddb40a1340 0x7fddb40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:42.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.446+0000 7fddc1ffb700 1 -- 172.21.15.44:0/3045070880 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fddbc0ccb10 con 0x7fddc4105620 2023-11-16T16:51:42.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.446+0000 7fddc8f9b700 1 --2- 172.21.15.44:0/3045070880 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fddb40a1340 0x7fddb40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:42.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.447+0000 7fddc8f9b700 1 --2- 172.21.15.44:0/3045070880 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fddb40a1340 0x7fddb40a37f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009990 tx=0x7fddb8005fb0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:42.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.606+0000 7fdddeb58b80 1 Processor -- start 2023-11-16T16:51:42.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.606+0000 7fdddeb58b80 1 -- start start 2023-11-16T16:51:42.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.606+0000 7fdddeb58b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 0x55900c0bdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:42.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.606+0000 7fdddeb58b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0be470 0x55900c0c2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:42.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.607+0000 7fdddeb58b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 0x55900c0c2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:42.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.607+0000 7fdddeb58b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900bfc3060 con 0x55900c0bdb60 2023-11-16T16:51:42.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.607+0000 7fdddeb58b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900c0158a0 con 0x55900c0c28b0 2023-11-16T16:51:42.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.607+0000 7fdddeb58b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900c00ed30 con 0x55900c0be470 2023-11-16T16:51:42.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.607+0000 7fddc8f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 0x55900c0bdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:42.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.607+0000 7fddc8f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 0x55900c0bdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34834/0 (socket says 172.21.15.44:34834) 2023-11-16T16:51:42.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.607+0000 7fddc8f9b700 1 -- 172.21.15.44:0/591908249 learned_addr learned my addr 172.21.15.44:0/591908249 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:42.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.607+0000 7fddc3fff700 1 --2- 172.21.15.44:0/591908249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0be470 0x55900c0c2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:42.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.607+0000 7fddc979c700 1 --2- 172.21.15.44:0/591908249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 0x55900c0c2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:42.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.607+0000 7fddc8f9b700 1 -- 172.21.15.44:0/591908249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0be470 msgr2=0x55900c0c2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:42.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.607+0000 7fddc8f9b700 1 --2- 172.21.15.44:0/591908249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0be470 0x55900c0c2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.607+0000 7fddc8f9b700 1 -- 172.21.15.44:0/591908249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 msgr2=0x55900c0c2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:42.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.607+0000 7fddc8f9b700 1 --2- 172.21.15.44:0/591908249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 0x55900c0c2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.607+0000 7fddc8f9b700 1 -- 172.21.15.44:0/591908249 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55900be90630 con 0x55900c0bdb60 2023-11-16T16:51:42.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.607+0000 7fddc979c700 1 --2- 172.21.15.44:0/591908249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 0x55900c0c2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:42.610 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.608+0000 7fddc8f9b700 1 --2- 172.21.15.44:0/591908249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 0x55900c0bdf30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fddb80095a0 tx=0x7fddb801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22b29fb16f85780 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:42.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.608+0000 7fddaeffd700 1 -- 172.21.15.44:0/591908249 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8003880 con 0x55900c0bdb60 2023-11-16T16:51:42.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.608+0000 7fddaeffd700 1 -- 172.21.15.44:0/591908249 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fddb801e040 con 0x55900c0bdb60 2023-11-16T16:51:42.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.608+0000 7fddaeffd700 1 -- 172.21.15.44:0/591908249 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb80306e0 con 0x55900c0bdb60 2023-11-16T16:51:42.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.609+0000 7fdddeb58b80 1 -- 172.21.15.44:0/591908249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 msgr2=0x55900c0bdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:42.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.609+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/591908249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 0x55900c0bdf30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fddb80095a0 tx=0x7fddb801f000 comp rx=0 tx=0).stop 2023-11-16T16:51:42.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.609+0000 7fdddeb58b80 1 -- 172.21.15.44:0/591908249 shutdown_connections 2023-11-16T16:51:42.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.609+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/591908249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 0x55900c0c2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.609+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/591908249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 0x55900c0bdf30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.609+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/591908249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0be470 0x55900c0c2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.609+0000 7fdddeb58b80 1 -- 172.21.15.44:0/591908249 >> 172.21.15.44:0/591908249 conn(0x55900c0b3360 msgr2=0x55900c0b8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:42.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.609+0000 7fdddeb58b80 1 -- 172.21.15.44:0/591908249 shutdown_connections 2023-11-16T16:51:42.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.609+0000 7fdddeb58b80 1 -- 172.21.15.44:0/591908249 wait complete. 2023-11-16T16:51:42.613 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.610+0000 7fdddeb58b80 1 -- start start 2023-11-16T16:51:42.613 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.610+0000 7fdddeb58b80 1 Processor -- start 2023-11-16T16:51:42.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.611+0000 7fdddeb58b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0bdb60 0x55900c0ca010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:42.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.611+0000 7fdddeb58b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0be470 0x55900c0c7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:42.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.611+0000 7fdddeb58b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 0x55900c0c7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:42.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.611+0000 7fdddeb58b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900bfb08e0 con 0x55900c0be470 2023-11-16T16:51:42.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.611+0000 7fdddeb58b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900bfb0580 con 0x55900c0c28b0 2023-11-16T16:51:42.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.611+0000 7fdddeb58b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900be92920 con 0x55900c0bdb60 2023-11-16T16:51:42.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.611+0000 7fdddeb58b80 10 client.0 _ll_register_callbacks cb 0x55900bfce780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:42.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.611+0000 7fddc979c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 0x55900c0c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:42.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.611+0000 7fddc3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0be470 0x55900c0c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:42.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.611+0000 7fddc8f9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0bdb60 0x55900c0ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:42.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.611+0000 7fddc979c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 0x55900c0c7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:52254/0 (socket says 172.21.15.44:52254) 2023-11-16T16:51:42.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.611+0000 7fddc979c700 1 -- 172.21.15.44:0/1325144405 learned_addr learned my addr 172.21.15.44:0/1325144405 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:42.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.611+0000 7fddc979c700 1 -- 172.21.15.44:0/1325144405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0bdb60 msgr2=0x55900c0ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:42.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.611+0000 7fddc979c700 1 --2- 172.21.15.44:0/1325144405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0bdb60 0x55900c0ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.611+0000 7fddc979c700 1 -- 172.21.15.44:0/1325144405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0be470 msgr2=0x55900c0c7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:42.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.612+0000 7fddc979c700 1 --2- 172.21.15.44:0/1325144405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0be470 0x55900c0c7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.612+0000 7fddc979c700 1 -- 172.21.15.44:0/1325144405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc410d9b0 con 0x55900c0c28b0 2023-11-16T16:51:42.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.612+0000 7fddc8f9b700 1 --2- 172.21.15.44:0/1325144405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0bdb60 0x55900c0ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:42.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.612+0000 7fddc979c700 1 --2- 172.21.15.44:0/1325144405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 0x55900c0c7910 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fddbc099590 tx=0x7fddbc099270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=873247494f054c52 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:42.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.612+0000 7fdda3fff700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:51:42.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.612+0000 7fdda3fff700 1 -- 172.21.15.44:0/1325144405 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddbc099cf0 con 0x55900c0c28b0 2023-11-16T16:51:42.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.612+0000 7fdddeb58b80 1 -- 172.21.15.44:0/1325144405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55900c0b0b20 con 0x55900c0c28b0 2023-11-16T16:51:42.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.612+0000 7fdda3fff700 1 -- 172.21.15.44:0/1325144405 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fddbc0b92d0 con 0x55900c0c28b0 2023-11-16T16:51:42.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.612+0000 7fdda3fff700 1 -- 172.21.15.44:0/1325144405 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddbc00ce60 con 0x55900c0c28b0 2023-11-16T16:51:42.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.612+0000 7fdddeb58b80 10 client.25922 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:42.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.612+0000 7fdddeb58b80 1 -- 172.21.15.44:0/1325144405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55900bfd5190 con 0x55900c0c28b0 2023-11-16T16:51:42.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.613+0000 7fdda3fff700 1 -- 172.21.15.44:0/1325144405 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55900c0b0b20 con 0x55900c0c28b0 2023-11-16T16:51:42.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.613+0000 7fdda3fff700 1 -- 172.21.15.44:0/1325144405 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fddbc0a17c0 con 0x55900c0c28b0 2023-11-16T16:51:42.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.613+0000 7fdddeb58b80 10 client.25922 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:42.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.613+0000 7fdddeb58b80 10 client.25922 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:42.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.613+0000 7fdddeb58b80 20 client.25922 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:42.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.613+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/1325144405 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55900c0d7740 0x55900c0d9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:42.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.613+0000 7fdddeb58b80 4 client.25922 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:51:42.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.613+0000 7fdddeb58b80 1 -- 172.21.15.44:0/1325144405 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55900bfd10a0 con 0x55900c0d7740 2023-11-16T16:51:42.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.613+0000 7fddc8f9b700 1 --2- 172.21.15.44:0/1325144405 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55900c0d7740 0x55900c0d9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:42.618 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.614+0000 7fddc8f9b700 1 --2- 172.21.15.44:0/1325144405 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55900c0d7740 0x55900c0d9b50 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf5c71561ebad7fc server_cookie=c4434a29aa6a064d in_seq=0 out_seq=0 2023-11-16T16:51:42.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.614+0000 7fdda3fff700 10 client.25922 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:42.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.615+0000 7fdda3fff700 1 -- 172.21.15.44:0/1325144405 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55900bfd10a0 con 0x55900c0d7740 2023-11-16T16:51:42.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.615+0000 7fdda3fff700 10 client.25922 handle_command_reply: tid=0 2023-11-16T16:51:42.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.615+0000 7fdddeb58b80 1 client.25922 shutdown 2023-11-16T16:51:42.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.615+0000 7fdddeb58b80 1 -- 172.21.15.44:0/1325144405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 msgr2=0x55900c0c7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:42.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.615+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/1325144405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 0x55900c0c7910 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fddbc099590 tx=0x7fddbc099270 comp rx=0 tx=0).stop 2023-11-16T16:51:42.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.615+0000 7fdddeb58b80 1 -- 172.21.15.44:0/1325144405 shutdown_connections 2023-11-16T16:51:42.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.615+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/1325144405 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55900c0d7740 0x55900c0d9b50 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.615+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/1325144405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 0x55900c0c7910 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.615+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/1325144405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0bdb60 0x55900c0ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.615+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/1325144405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0be470 0x55900c0c7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:42.620 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.615+0000 7fdddeb58b80 1 -- 172.21.15.44:0/1325144405 >> 172.21.15.44:0/1325144405 conn(0x55900c0b3360 msgr2=0x55900c0c3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:42.620 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.615+0000 7fdda3fff700 0 client.25922 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:42.620 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.615+0000 7fdddeb58b80 1 -- 172.21.15.44:0/1325144405 shutdown_connections 2023-11-16T16:51:42.620 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.615+0000 7fdddeb58b80 1 -- 172.21.15.44:0/1325144405 wait complete. 2023-11-16T16:51:42.620 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.616+0000 7fdddeb58b80 20 client.25922 trim_cache size 0 max 16384 2023-11-16T16:51:42.628 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:42.628 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:42.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.634+0000 7fdddeb58b80 1 Processor -- start 2023-11-16T16:51:42.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.634+0000 7fdddeb58b80 1 -- start start 2023-11-16T16:51:43.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:42 smithi077 ceph-mon[131204]: pgmap v56: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 12 op/s 2023-11-16T16:51:43.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:42 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:51:43.745 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:42 smithi176 ceph-mon[131630]: pgmap v56: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 12 op/s 2023-11-16T16:51:43.745 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:42 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:51:43.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.634+0000 7fdddeb58b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 0x55900c0c0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:43.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.634+0000 7fdddeb58b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0be470 0x55900c0bc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:43.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.634+0000 7fdddeb58b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 0x55900c0bcea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:43.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.634+0000 7fdddeb58b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900c0ba470 con 0x55900c0bdb60 2023-11-16T16:51:43.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.634+0000 7fdddeb58b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900c0b7c00 con 0x55900c0c28b0 2023-11-16T16:51:43.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.634+0000 7fdddeb58b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900c0c7cf0 con 0x55900c0be470 2023-11-16T16:51:43.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.635+0000 7fddc8f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 0x55900c0c0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:43.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.635+0000 7fddc8f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 0x55900c0c0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34852/0 (socket says 172.21.15.44:34852) 2023-11-16T16:51:43.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.635+0000 7fddc8f9b700 1 -- 172.21.15.44:0/1109166919 learned_addr learned my addr 172.21.15.44:0/1109166919 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:43.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.635+0000 7fddc979c700 1 --2- 172.21.15.44:0/1109166919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 0x55900c0bcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:43.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.635+0000 7fddc3fff700 1 --2- 172.21.15.44:0/1109166919 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0be470 0x55900c0bc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:43.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.635+0000 7fddc8f9b700 1 -- 172.21.15.44:0/1109166919 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0be470 msgr2=0x55900c0bc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:43.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.635+0000 7fddc8f9b700 1 --2- 172.21.15.44:0/1109166919 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0be470 0x55900c0bc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.635+0000 7fddc8f9b700 1 -- 172.21.15.44:0/1109166919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 msgr2=0x55900c0bcea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:43.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.635+0000 7fddc8f9b700 1 --2- 172.21.15.44:0/1109166919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 0x55900c0bcea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.635+0000 7fddc8f9b700 1 -- 172.21.15.44:0/1109166919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55900c0c2f50 con 0x55900c0bdb60 2023-11-16T16:51:43.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.635+0000 7fddc979c700 1 --2- 172.21.15.44:0/1109166919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 0x55900c0bcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:43.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.635+0000 7fddc8f9b700 1 --2- 172.21.15.44:0/1109166919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 0x55900c0c0970 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fddb8000c00 tx=0x7fddb8004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c517e474359810 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:43.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.636+0000 7fddaf7fe700 1 -- 172.21.15.44:0/1109166919 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8034040 con 0x55900c0bdb60 2023-11-16T16:51:43.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.636+0000 7fddaf7fe700 1 -- 172.21.15.44:0/1109166919 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fddb8035040 con 0x55900c0bdb60 2023-11-16T16:51:43.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.636+0000 7fddaf7fe700 1 -- 172.21.15.44:0/1109166919 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8030790 con 0x55900c0bdb60 2023-11-16T16:51:43.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.636+0000 7fdddeb58b80 1 -- 172.21.15.44:0/1109166919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 msgr2=0x55900c0c0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:43.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.636+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/1109166919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 0x55900c0c0970 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fddb8000c00 tx=0x7fddb8004610 comp rx=0 tx=0).stop 2023-11-16T16:51:43.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.636+0000 7fdddeb58b80 1 -- 172.21.15.44:0/1109166919 shutdown_connections 2023-11-16T16:51:43.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.636+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/1109166919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0c28b0 0x55900c0bcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.636+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/1109166919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 0x55900c0c0970 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.636+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/1109166919 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0be470 0x55900c0bc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.636+0000 7fdddeb58b80 1 -- 172.21.15.44:0/1109166919 >> 172.21.15.44:0/1109166919 conn(0x55900c0b3360 msgr2=0x55900c0d30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:43.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.637+0000 7fdddeb58b80 1 -- 172.21.15.44:0/1109166919 shutdown_connections 2023-11-16T16:51:43.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.637+0000 7fdddeb58b80 1 -- 172.21.15.44:0/1109166919 wait complete. 2023-11-16T16:51:43.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.637+0000 7fdddeb58b80 1 -- start start 2023-11-16T16:51:43.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.637+0000 7fdddeb58b80 1 Processor -- start 2023-11-16T16:51:43.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fdddeb58b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 0x55900c11f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:43.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fdddeb58b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0be470 0x55900c11fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:43.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fdddeb58b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0c28b0 0x55900c11ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:43.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fdddeb58b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900c12a6b0 con 0x55900c0bdb60 2023-11-16T16:51:43.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fdddeb58b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900c12a7f0 con 0x55900c0be470 2023-11-16T16:51:43.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fdddeb58b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900c12a9b0 con 0x55900c0c28b0 2023-11-16T16:51:43.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fdddeb58b80 10 client.0 _ll_register_callbacks cb 0x7fddbc099750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:43.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fddc8f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 0x55900c11f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:43.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fddc3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0be470 0x55900c11fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:43.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fddc979c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0c28b0 0x55900c11ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:43.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fddc3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0be470 0x55900c11fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:52286/0 (socket says 172.21.15.44:52286) 2023-11-16T16:51:43.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fddc3fff700 1 -- 172.21.15.44:0/3480196568 learned_addr learned my addr 172.21.15.44:0/3480196568 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:43.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fddc3fff700 1 -- 172.21.15.44:0/3480196568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0c28b0 msgr2=0x55900c11ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:43.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fddc3fff700 1 --2- 172.21.15.44:0/3480196568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0c28b0 0x55900c11ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fddc3fff700 1 -- 172.21.15.44:0/3480196568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 msgr2=0x55900c11f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:43.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fddc3fff700 1 --2- 172.21.15.44:0/3480196568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 0x55900c11f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fddc3fff700 1 -- 172.21.15.44:0/3480196568 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55900bfd5190 con 0x55900c0be470 2023-11-16T16:51:43.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.638+0000 7fddc979c700 1 --2- 172.21.15.44:0/3480196568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0c28b0 0x55900c11ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:43.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.639+0000 7fddc3fff700 1 --2- 172.21.15.44:0/3480196568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0be470 0x55900c11fa90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fddc4101a30 tx=0x7fddb0012c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be9abcfe4773adc5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:43.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.639+0000 7fdda3fff700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:51:43.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.639+0000 7fdda3fff700 1 -- 172.21.15.44:0/3480196568 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0022070 con 0x55900c0be470 2023-11-16T16:51:43.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.639+0000 7fdddeb58b80 1 -- 172.21.15.44:0/3480196568 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fddb801e040 con 0x55900c0be470 2023-11-16T16:51:43.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.639+0000 7fdda3fff700 1 -- 172.21.15.44:0/3480196568 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fddb0004830 con 0x55900c0be470 2023-11-16T16:51:43.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.639+0000 7fdda3fff700 1 -- 172.21.15.44:0/3480196568 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddb0017040 con 0x55900c0be470 2023-11-16T16:51:43.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.639+0000 7fdddeb58b80 10 client.25928 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:43.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.639+0000 7fdddeb58b80 1 -- 172.21.15.44:0/3480196568 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55900be90630 con 0x55900c0be470 2023-11-16T16:51:43.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.640+0000 7fdda3fff700 1 -- 172.21.15.44:0/3480196568 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fddb801e040 con 0x55900c0be470 2023-11-16T16:51:43.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.640+0000 7fdda3fff700 1 -- 172.21.15.44:0/3480196568 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fddb0025020 con 0x55900c0be470 2023-11-16T16:51:43.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.640+0000 7fdddeb58b80 10 client.25928 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:43.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.640+0000 7fdddeb58b80 10 client.25928 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:43.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.640+0000 7fdddeb58b80 20 client.25928 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:43.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.640+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/3480196568 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55900c12c1f0 0x55900c12e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:43.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.640+0000 7fdddeb58b80 4 client.25928 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:51:43.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.640+0000 7fdddeb58b80 1 -- 172.21.15.44:0/3480196568 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55900bfdd660 con 0x55900c12c1f0 2023-11-16T16:51:43.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.640+0000 7fddc8f9b700 1 --2- 172.21.15.44:0/3480196568 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55900c12c1f0 0x55900c12e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:43.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.641+0000 7fdda3fff700 10 client.25928 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:43.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.641+0000 7fddc8f9b700 1 --2- 172.21.15.44:0/3480196568 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55900c12c1f0 0x55900c12e600 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=947e3f3b86dc1d8f server_cookie=1e6ab848e72700f4 in_seq=0 out_seq=0 2023-11-16T16:51:43.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.641+0000 7fdda3fff700 1 -- 172.21.15.44:0/3480196568 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55900bfdd660 con 0x55900c12c1f0 2023-11-16T16:51:43.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.641+0000 7fdda3fff700 10 client.25928 handle_command_reply: tid=0 2023-11-16T16:51:43.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.641+0000 7fdddeb58b80 1 client.25928 shutdown 2023-11-16T16:51:43.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.641+0000 7fdddeb58b80 1 -- 172.21.15.44:0/3480196568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0be470 msgr2=0x55900c11fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:43.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.641+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/3480196568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0be470 0x55900c11fa90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fddc4101a30 tx=0x7fddb0012c50 comp rx=0 tx=0).stop 2023-11-16T16:51:43.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.641+0000 7fdddeb58b80 1 -- 172.21.15.44:0/3480196568 shutdown_connections 2023-11-16T16:51:43.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.641+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/3480196568 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55900c12c1f0 0x55900c12e600 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.641+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/3480196568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55900c0c28b0 0x55900c11ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.641+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/3480196568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55900c0bdb60 0x55900c11f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.641+0000 7fdddeb58b80 1 --2- 172.21.15.44:0/3480196568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55900c0be470 0x55900c11fa90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.641+0000 7fdda3fff700 0 client.25928 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:43.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.641+0000 7fdddeb58b80 1 -- 172.21.15.44:0/3480196568 >> 172.21.15.44:0/3480196568 conn(0x55900c0b3360 msgr2=0x55900c0bc960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:43.755 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.641+0000 7fdddeb58b80 1 -- 172.21.15.44:0/3480196568 shutdown_connections 2023-11-16T16:51:43.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.641+0000 7fdddeb58b80 1 -- 172.21.15.44:0/3480196568 wait complete. 2023-11-16T16:51:43.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.642+0000 7fdddeb58b80 20 client.25928 trim_cache size 0 max 16384 2023-11-16T16:51:43.756 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:51:43.756 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T16:51:43.756 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T16:51:43.756 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:51:43.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.642+0000 7fddcb1ff700 1 -- 172.21.15.44:0/3045070880 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fddb40a1340 msgr2=0x7fddb40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:43.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.642+0000 7fddcb1ff700 1 --2- 172.21.15.44:0/3045070880 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fddb40a1340 0x7fddb40a37f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009990 tx=0x7fddb8005fb0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.642+0000 7fddcb1ff700 1 -- 172.21.15.44:0/3045070880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddc4105620 msgr2=0x7fddc419d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:43.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.642+0000 7fddcb1ff700 1 --2- 172.21.15.44:0/3045070880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddc4105620 0x7fddc419d020 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fddbc00ef30 tx=0x7fddbc00c3a0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.642+0000 7fddcb1ff700 1 -- 172.21.15.44:0/3045070880 shutdown_connections 2023-11-16T16:51:43.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.642+0000 7fddcb1ff700 1 --2- 172.21.15.44:0/3045070880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddc4105620 0x7fddc419d020 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.642+0000 7fddcb1ff700 1 --2- 172.21.15.44:0/3045070880 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fddb40a1340 0x7fddb40a37f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.642+0000 7fddcb1ff700 1 --2- 172.21.15.44:0/3045070880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fddc4100cc0 0x7fddc4198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.642+0000 7fddcb1ff700 1 --2- 172.21.15.44:0/3045070880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddc4101660 0x7fddc4198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:43.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.642+0000 7fddcb1ff700 1 -- 172.21.15.44:0/3045070880 >> 172.21.15.44:0/3045070880 conn(0x7fddc4077560 msgr2=0x7fddc40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:43.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.642+0000 7fddcb1ff700 1 -- 172.21.15.44:0/3045070880 shutdown_connections 2023-11-16T16:51:43.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:42.642+0000 7fddcb1ff700 1 -- 172.21.15.44:0/3045070880 wait complete. 2023-11-16T16:51:43.758 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:43.758 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:43.758 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 33 -lt 100 ']' 2023-11-16T16:51:43.758 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 33 + 1 2023-11-16T16:51:43.759 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T16:51:43.759 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:51:43.760 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=34 2023-11-16T16:51:43.760 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:43.760 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:43.760 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:43.760 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:43.760 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:43.760 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:43.761 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:42 smithi044 ceph-mon[128750]: pgmap v56: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 12 op/s 2023-11-16T16:51:43.761 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:42 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:51:43.762 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:43.762 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:43.762 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:43.762 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:43.762 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:43.762 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:43.762 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 34 -lt 100 ']' 2023-11-16T16:51:43.762 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 34 + 1 2023-11-16T16:51:43.762 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=35 2023-11-16T16:51:43.762 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:43.762 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:43.763 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:43.763 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:43.763 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:43.763 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:44.474 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:44.474 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:44.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:44 smithi044 ceph-mon[128750]: pgmap v57: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 11 op/s 2023-11-16T16:51:44.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:44 smithi176 ceph-mon[131630]: pgmap v57: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 11 op/s 2023-11-16T16:51:44.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:44 smithi077 ceph-mon[131204]: pgmap v57: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 11 op/s 2023-11-16T16:51:45.161 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:45.161 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:45.177 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:45.177 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:45.177 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 35 -lt 100 ']' 2023-11-16T16:51:45.178 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 35 + 1 2023-11-16T16:51:45.232 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=36 2023-11-16T16:51:45.232 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:45.307 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:45.308 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:45.311 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:45.311 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:45.312 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:46.045 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:46.045 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:46.278 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:46.278 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:46.311 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:46.311 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:46.311 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 36 -lt 100 ']' 2023-11-16T16:51:46.311 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 36 + 1 2023-11-16T16:51:46.631 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=37 2023-11-16T16:51:46.632 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:46.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:46 smithi077 ceph-mon[131204]: pgmap v58: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 11 op/s 2023-11-16T16:51:46.991 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:46.992 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:46.994 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:46.995 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:46.995 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:47.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:46 smithi044 ceph-mon[128750]: pgmap v58: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 11 op/s 2023-11-16T16:51:47.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:46 smithi176 ceph-mon[131630]: pgmap v58: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 11 op/s 2023-11-16T16:51:48.799 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:48.799 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:49.052 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:49.052 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:49.094 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:49.094 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:49.094 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 37 -lt 100 ']' 2023-11-16T16:51:49.094 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 37 + 1 2023-11-16T16:51:49.181 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=38 2023-11-16T16:51:49.181 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:49.248 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:49.249 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:49.252 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:49.252 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:49.253 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.395+0000 7f278cbff700 1 -- 172.21.15.44:0/1063654639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2788100560 msgr2=0x7f2788100930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.395+0000 7f278cbff700 1 --2- 172.21.15.44:0/1063654639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2788100560 0x7f2788100930 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f27780099c0 tx=0x7f2778009cd0 comp rx=0 tx=0).stop 2023-11-16T16:51:49.397 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.395+0000 7f278cbff700 1 -- 172.21.15.44:0/1063654639 shutdown_connections 2023-11-16T16:51:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.395+0000 7f278cbff700 1 --2- 172.21.15.44:0/1063654639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27881055a0 0x7f278810d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.395+0000 7f278cbff700 1 --2- 172.21.15.44:0/1063654639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2788100560 0x7f2788100930 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.395+0000 7f278cbff700 1 --2- 172.21.15.44:0/1063654639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2788100f00 0x7f2788104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.395+0000 7f278cbff700 1 -- 172.21.15.44:0/1063654639 >> 172.21.15.44:0/1063654639 conn(0x7f2788077080 msgr2=0x7f2788077480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.396+0000 7f278cbff700 1 -- 172.21.15.44:0/1063654639 shutdown_connections 2023-11-16T16:51:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.396+0000 7f278cbff700 1 -- 172.21.15.44:0/1063654639 wait complete. 2023-11-16T16:51:49.398 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.397+0000 7f278cbff700 1 Processor -- start 2023-11-16T16:51:49.399 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.397+0000 7f278cbff700 1 -- start start 2023-11-16T16:51:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.397+0000 7f278cbff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2788100560 0x7f27881985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.397+0000 7f278cbff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2788100f00 0x7f2788198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.397+0000 7f278cbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27881055a0 0x7f278819cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.397+0000 7f278cbff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278819d5c0 con 0x7f2788100560 2023-11-16T16:51:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.397+0000 7f278cbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278819d730 con 0x7f27881055a0 2023-11-16T16:51:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.397+0000 7f278cbff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278819d8f0 con 0x7f2788100f00 2023-11-16T16:51:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.398+0000 7f278659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2788100560 0x7f27881985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:49.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.398+0000 7f2785d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2788100f00 0x7f2788198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.398+0000 7f278659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2788100560 0x7f27881985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51758/0 (socket says 172.21.15.44:51758) 2023-11-16T16:51:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.398+0000 7f278659c700 1 -- 172.21.15.44:0/3591032523 learned_addr learned my addr 172.21.15.44:0/3591032523 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.398+0000 7f2786d9d700 1 --2- 172.21.15.44:0/3591032523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27881055a0 0x7f278819cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.398+0000 7f2785d9b700 1 -- 172.21.15.44:0/3591032523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27881055a0 msgr2=0x7f278819cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.398+0000 7f2785d9b700 1 --2- 172.21.15.44:0/3591032523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27881055a0 0x7f278819cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.398+0000 7f2785d9b700 1 -- 172.21.15.44:0/3591032523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2788100560 msgr2=0x7f27881985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.398+0000 7f2785d9b700 1 --2- 172.21.15.44:0/3591032523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2788100560 0x7f27881985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:49.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.398+0000 7f2785d9b700 1 -- 172.21.15.44:0/3591032523 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27780095f0 con 0x7f2788100f00 2023-11-16T16:51:49.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.399+0000 7f2785d9b700 1 --2- 172.21.15.44:0/3591032523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2788100f00 0x7f2788198b20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f27881012d0 tx=0x7f277c00eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:49.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.399+0000 7f27777fe700 1 -- 172.21.15.44:0/3591032523 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c010900 con 0x7f2788100f00 2023-11-16T16:51:49.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.399+0000 7f27777fe700 1 -- 172.21.15.44:0/3591032523 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f277c00c430 con 0x7f2788100f00 2023-11-16T16:51:49.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.399+0000 7f27777fe700 1 -- 172.21.15.44:0/3591032523 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c018680 con 0x7f2788100f00 2023-11-16T16:51:49.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.399+0000 7f278cbff700 1 -- 172.21.15.44:0/3591032523 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27881103b0 con 0x7f2788100f00 2023-11-16T16:51:49.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.399+0000 7f278cbff700 1 -- 172.21.15.44:0/3591032523 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2788110220 con 0x7f2788100f00 2023-11-16T16:51:49.403 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.401+0000 7f278cbff700 1 -- 172.21.15.44:0/3591032523 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2788110540 con 0x7f2788100f00 2023-11-16T16:51:49.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.403+0000 7f27777fe700 1 -- 172.21.15.44:0/3591032523 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f277c00c5a0 con 0x7f2788100f00 2023-11-16T16:51:49.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.404+0000 7f27777fe700 1 --2- 172.21.15.44:0/3591032523 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f27700a1340 0x7f27700a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:49.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.404+0000 7f27777fe700 1 -- 172.21.15.44:0/3591032523 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f277c014070 con 0x7f2788100f00 2023-11-16T16:51:49.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.404+0000 7f278659c700 1 --2- 172.21.15.44:0/3591032523 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f27700a1340 0x7f27700a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:49.407 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.405+0000 7f278659c700 1 --2- 172.21.15.44:0/3591032523 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f27700a1340 0x7f27700a37f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f27780036c0 tx=0x7f2778005fb0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:49.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.421+0000 7f27777fe700 1 -- 172.21.15.44:0/3591032523 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f277c051730 con 0x7f2788100f00 2023-11-16T16:51:49.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:49 smithi044 ceph-mon[128750]: pgmap v59: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 KiB/s wr, 13 op/s 2023-11-16T16:51:49.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:49 smithi176 ceph-mon[131630]: pgmap v59: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 KiB/s wr, 13 op/s 2023-11-16T16:51:49.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.687+0000 7f278cbff700 1 -- 172.21.15.44:0/3591032523 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2788067a60 con 0x7f2788100f00 2023-11-16T16:51:49.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.688+0000 7f27777fe700 1 -- 172.21.15.44:0/3591032523 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f277c099be0 con 0x7f2788100f00 2023-11-16T16:51:49.689 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:51:49.690 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:51:49.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.690+0000 7f278cbff700 1 -- 172.21.15.44:0/3591032523 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f27700a1340 msgr2=0x7f27700a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:49.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.690+0000 7f278cbff700 1 --2- 172.21.15.44:0/3591032523 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f27700a1340 0x7f27700a37f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f27780036c0 tx=0x7f2778005fb0 comp rx=0 tx=0).stop 2023-11-16T16:51:49.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.690+0000 7f278cbff700 1 -- 172.21.15.44:0/3591032523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2788100f00 msgr2=0x7f2788198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:49.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.690+0000 7f278cbff700 1 --2- 172.21.15.44:0/3591032523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2788100f00 0x7f2788198b20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f27881012d0 tx=0x7f277c00eca0 comp rx=0 tx=0).stop 2023-11-16T16:51:49.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.690+0000 7f278cbff700 1 -- 172.21.15.44:0/3591032523 shutdown_connections 2023-11-16T16:51:49.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.690+0000 7f278cbff700 1 --2- 172.21.15.44:0/3591032523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27881055a0 0x7f278819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:49.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.690+0000 7f278cbff700 1 --2- 172.21.15.44:0/3591032523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2788100560 0x7f27881985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:49.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.690+0000 7f278cbff700 1 --2- 172.21.15.44:0/3591032523 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f27700a1340 0x7f27700a37f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:49.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.690+0000 7f278cbff700 1 --2- 172.21.15.44:0/3591032523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2788100f00 0x7f2788198b20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:49.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.690+0000 7f278cbff700 1 -- 172.21.15.44:0/3591032523 >> 172.21.15.44:0/3591032523 conn(0x7f2788077080 msgr2=0x7f278810c440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:49.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.690+0000 7f278cbff700 1 -- 172.21.15.44:0/3591032523 shutdown_connections 2023-11-16T16:51:49.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:49.690+0000 7f278cbff700 1 -- 172.21.15.44:0/3591032523 wait complete. 2023-11-16T16:51:49.693 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:51:49.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:49 smithi077 ceph-mon[131204]: pgmap v59: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 KiB/s wr, 13 op/s 2023-11-16T16:51:49.846 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T16:51:49.847 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:49.847 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:50.043 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:50.044 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:50.059 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:50.059 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:50.059 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 38 -lt 100 ']' 2023-11-16T16:51:50.059 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 38 + 1 2023-11-16T16:51:50.097 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=39 2023-11-16T16:51:50.097 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:50.129 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:50.129 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:50.131 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:50.132 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:50.132 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:50.539 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:50 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3591032523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:51:50.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:50 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3591032523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:51:50.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:50 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3591032523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:51:50.940 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:50.940 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.061+0000 7fcce05ff700 1 -- 172.21.15.44:0/2415244063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccd8105560 msgr2=0x7fccd810d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.061+0000 7fcce05ff700 1 --2- 172.21.15.44:0/2415244063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccd8105560 0x7fccd810d170 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fccd000b0d0 tx=0x7fccd000b3e0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.061+0000 7fcce05ff700 1 -- 172.21.15.44:0/2415244063 shutdown_connections 2023-11-16T16:51:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.061+0000 7fcce05ff700 1 --2- 172.21.15.44:0/2415244063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccd8105560 0x7fccd810d170 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.061+0000 7fcce05ff700 1 --2- 172.21.15.44:0/2415244063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccd8100c50 0x7fccd8101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.061+0000 7fcce05ff700 1 --2- 172.21.15.44:0/2415244063 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fccd81015f0 0x7fccd8104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.061+0000 7fcce05ff700 1 -- 172.21.15.44:0/2415244063 >> 172.21.15.44:0/2415244063 conn(0x7fccd8077550 msgr2=0x7fccd8077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.061+0000 7fcce05ff700 1 -- 172.21.15.44:0/2415244063 shutdown_connections 2023-11-16T16:51:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.061+0000 7fcce05ff700 1 -- 172.21.15.44:0/2415244063 wait complete. 2023-11-16T16:51:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.062+0000 7fcce05ff700 1 Processor -- start 2023-11-16T16:51:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.062+0000 7fcce05ff700 1 -- start start 2023-11-16T16:51:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fcce05ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccd8100c50 0x7fccd8072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fcce05ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fccd81015f0 0x7fccd8073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fcce05ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccd8105560 0x7fccd81a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fcce05ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd8073b60 con 0x7fccd8105560 2023-11-16T16:51:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fcce05ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd81a4bd0 con 0x7fccd8100c50 2023-11-16T16:51:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fcce05ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd81a4d10 con 0x7fccd81015f0 2023-11-16T16:51:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fccdeb9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccd8105560 0x7fccd81a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fccdeb9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccd8105560 0x7fccd81a4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51776/0 (socket says 172.21.15.44:51776) 2023-11-16T16:51:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fccdeb9c700 1 -- 172.21.15.44:0/840272850 learned_addr learned my addr 172.21.15.44:0/840272850 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fccde39b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccd8100c50 0x7fccd8072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fccddb9a700 1 --2- 172.21.15.44:0/840272850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fccd81015f0 0x7fccd8073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fccdeb9c700 1 -- 172.21.15.44:0/840272850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fccd81015f0 msgr2=0x7fccd8073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fccdeb9c700 1 --2- 172.21.15.44:0/840272850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fccd81015f0 0x7fccd8073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fccdeb9c700 1 -- 172.21.15.44:0/840272850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccd8100c50 msgr2=0x7fccd8072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fccdeb9c700 1 --2- 172.21.15.44:0/840272850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccd8100c50 0x7fccd8072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fccdeb9c700 1 -- 172.21.15.44:0/840272850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd0009d00 con 0x7fccd8105560 2023-11-16T16:51:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.063+0000 7fccde39b700 1 --2- 172.21.15.44:0/840272850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccd8100c50 0x7fccd8072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.064+0000 7fccdeb9c700 1 --2- 172.21.15.44:0/840272850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccd8105560 0x7fccd81a4690 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fccd0007820 tx=0x7fccd0007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.064+0000 7fcccf7fe700 1 -- 172.21.15.44:0/840272850 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd000e040 con 0x7fccd8105560 2023-11-16T16:51:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.064+0000 7fcccf7fe700 1 -- 172.21.15.44:0/840272850 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fccd000bcb0 con 0x7fccd8105560 2023-11-16T16:51:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.064+0000 7fcce05ff700 1 -- 172.21.15.44:0/840272850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd81103d0 con 0x7fccd8105560 2023-11-16T16:51:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.064+0000 7fcccf7fe700 1 -- 172.21.15.44:0/840272850 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd00058a0 con 0x7fccd8105560 2023-11-16T16:51:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.064+0000 7fcce05ff700 1 -- 172.21.15.44:0/840272850 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd8110240 con 0x7fccd8105560 2023-11-16T16:51:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.067+0000 7fcccf7fe700 1 -- 172.21.15.44:0/840272850 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fccd0004a50 con 0x7fccd8105560 2023-11-16T16:51:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.069+0000 7fcccf7fe700 1 --2- 172.21.15.44:0/840272850 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fccc40a12c0 0x7fccc40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.069+0000 7fcccf7fe700 1 -- 172.21.15.44:0/840272850 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fccd00ce250 con 0x7fccd8105560 2023-11-16T16:51:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.069+0000 7fccde39b700 1 --2- 172.21.15.44:0/840272850 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fccc40a12c0 0x7fccc40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.069+0000 7fccde39b700 1 --2- 172.21.15.44:0/840272850 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fccc40a12c0 0x7fccc40a3770 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fccd4011490 tx=0x7fccd4009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:51.135 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:51.135 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:51.169 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:51.170 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:51.170 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 39 -lt 100 ']' 2023-11-16T16:51:51.170 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 39 + 1 2023-11-16T16:51:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.204+0000 7fccf3f28b80 1 Processor -- start 2023-11-16T16:51:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.205+0000 7fccf3f28b80 1 -- start start 2023-11-16T16:51:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.205+0000 7fccf3f28b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7b646b0 0x55e1c7d0e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.205+0000 7fccf3f28b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7d0e720 0x55e1c7d08560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.205+0000 7fccf3f28b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d08b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.205+0000 7fccf3f28b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1c7c0e000 con 0x55e1c7b646b0 2023-11-16T16:51:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.205+0000 7fccf3f28b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1c7c60840 con 0x55e1c7d0e720 2023-11-16T16:51:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.205+0000 7fccf3f28b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1c7c59cd0 con 0x55e1c7d0eb90 2023-11-16T16:51:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.205+0000 7fccde39b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7b646b0 0x55e1c7d0e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.205+0000 7fccde39b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7b646b0 0x55e1c7d0e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51782/0 (socket says 172.21.15.44:51782) 2023-11-16T16:51:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.205+0000 7fccde39b700 1 -- 172.21.15.44:0/2896532799 learned_addr learned my addr 172.21.15.44:0/2896532799 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.205+0000 7fccddb9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7d0e720 0x55e1c7d08560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.205+0000 7fccdeb9c700 1 --2- 172.21.15.44:0/2896532799 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d08b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.206+0000 7fccde39b700 1 -- 172.21.15.44:0/2896532799 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7d0eb90 msgr2=0x55e1c7d08b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.206+0000 7fccde39b700 1 --2- 172.21.15.44:0/2896532799 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d08b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.206+0000 7fccde39b700 1 -- 172.21.15.44:0/2896532799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7d0e720 msgr2=0x55e1c7d08560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.206+0000 7fccde39b700 1 --2- 172.21.15.44:0/2896532799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7d0e720 0x55e1c7d08560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.206+0000 7fccde39b700 1 -- 172.21.15.44:0/2896532799 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e1c7adb5d0 con 0x55e1c7b646b0 2023-11-16T16:51:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.206+0000 7fccde39b700 1 --2- 172.21.15.44:0/2896532799 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7b646b0 0x55e1c7d0e1e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fccd40043c0 tx=0x7fccd4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7954dc1b7f2b7726 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.206+0000 7fccccbfe700 1 -- 172.21.15.44:0/2896532799 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4003d10 con 0x55e1c7b646b0 2023-11-16T16:51:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.206+0000 7fccccbfe700 1 -- 172.21.15.44:0/2896532799 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fccd4003e70 con 0x55e1c7b646b0 2023-11-16T16:51:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.206+0000 7fccccbfe700 1 -- 172.21.15.44:0/2896532799 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd400e6a0 con 0x55e1c7b646b0 2023-11-16T16:51:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.207+0000 7fccf3f28b80 1 -- 172.21.15.44:0/2896532799 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7b646b0 msgr2=0x55e1c7d0e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.207+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/2896532799 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7b646b0 0x55e1c7d0e1e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fccd40043c0 tx=0x7fccd4014000 comp rx=0 tx=0).stop 2023-11-16T16:51:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.207+0000 7fccf3f28b80 1 -- 172.21.15.44:0/2896532799 shutdown_connections 2023-11-16T16:51:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.207+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/2896532799 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d08b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.207+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/2896532799 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7b646b0 0x55e1c7d0e1e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.207+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/2896532799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7d0e720 0x55e1c7d08560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.207+0000 7fccf3f28b80 1 -- 172.21.15.44:0/2896532799 >> 172.21.15.44:0/2896532799 conn(0x55e1c7cfdf50 msgr2=0x55e1c7d03270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.207+0000 7fccf3f28b80 1 -- 172.21.15.44:0/2896532799 shutdown_connections 2023-11-16T16:51:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.207+0000 7fccf3f28b80 1 -- 172.21.15.44:0/2896532799 wait complete. 2023-11-16T16:51:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.208+0000 7fccf3f28b80 1 -- start start 2023-11-16T16:51:51.211 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.208+0000 7fccf3f28b80 1 Processor -- start 2023-11-16T16:51:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccf3f28b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7b646b0 0x55e1c7d17ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccf3f28b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7d0e720 0x55e1c7d18000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccf3f28b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d11d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccf3f28b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1c7bfb880 con 0x55e1c7d0eb90 2023-11-16T16:51:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccf3f28b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1c7bfb520 con 0x55e1c7b646b0 2023-11-16T16:51:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccf3f28b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1c7add8c0 con 0x55e1c7d0e720 2023-11-16T16:51:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccf3f28b80 10 client.0 _ll_register_callbacks cb 0x55e1c7c19720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccdeb9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d11d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccdeb9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d11d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51794/0 (socket says 172.21.15.44:51794) 2023-11-16T16:51:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccdeb9c700 1 -- 172.21.15.44:0/651212574 learned_addr learned my addr 172.21.15.44:0/651212574 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccde39b700 1 --2- 172.21.15.44:0/651212574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7b646b0 0x55e1c7d17ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccddb9a700 1 --2- 172.21.15.44:0/651212574 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7d0e720 0x55e1c7d18000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccdeb9c700 1 -- 172.21.15.44:0/651212574 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7d0e720 msgr2=0x55e1c7d18000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccdeb9c700 1 --2- 172.21.15.44:0/651212574 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7d0e720 0x55e1c7d18000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccdeb9c700 1 -- 172.21.15.44:0/651212574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7b646b0 msgr2=0x55e1c7d17ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccdeb9c700 1 --2- 172.21.15.44:0/651212574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7b646b0 0x55e1c7d17ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.209+0000 7fccdeb9c700 1 -- 172.21.15.44:0/651212574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd4017070 con 0x55e1c7d0eb90 2023-11-16T16:51:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.210+0000 7fccdeb9c700 1 --2- 172.21.15.44:0/651212574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d11d10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fccd0097a40 tx=0x7fccd009ab10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5106eb33f65cf2be server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.210+0000 7fccc1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:51:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.210+0000 7fccc1ffb700 1 -- 172.21.15.44:0/651212574 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd0095de0 con 0x55e1c7d0eb90 2023-11-16T16:51:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.210+0000 7fccc1ffb700 1 -- 172.21.15.44:0/651212574 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fccd000be20 con 0x55e1c7d0eb90 2023-11-16T16:51:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.210+0000 7fccf3f28b80 1 -- 172.21.15.44:0/651212574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e1c7cfb710 con 0x55e1c7d0eb90 2023-11-16T16:51:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.210+0000 7fccc1ffb700 1 -- 172.21.15.44:0/651212574 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccd009afd0 con 0x55e1c7d0eb90 2023-11-16T16:51:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.210+0000 7fccf3f28b80 10 client.15930 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.210+0000 7fccf3f28b80 1 -- 172.21.15.44:0/651212574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55e1c7c20130 con 0x55e1c7d0eb90 2023-11-16T16:51:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.210+0000 7fccc1ffb700 1 -- 172.21.15.44:0/651212574 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e1c7cfb710 con 0x55e1c7d0eb90 2023-11-16T16:51:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.211+0000 7fccc1ffb700 1 -- 172.21.15.44:0/651212574 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fccd009a4c0 con 0x55e1c7d0eb90 2023-11-16T16:51:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.211+0000 7fccf3f28b80 10 client.15930 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.211+0000 7fccf3f28b80 10 client.15930 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.211+0000 7fccf3f28b80 20 client.15930 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.211+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/651212574 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e1c7d21cc0 0x55e1c7d240a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.211+0000 7fccf3f28b80 4 client.15930 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:51:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.211+0000 7fccf3f28b80 1 -- 172.21.15.44:0/651212574 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e1c7c1c040 con 0x55e1c7d21cc0 2023-11-16T16:51:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.211+0000 7fccde39b700 1 --2- 172.21.15.44:0/651212574 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e1c7d21cc0 0x55e1c7d240a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.212+0000 7fccc1ffb700 10 client.15930 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.212+0000 7fccde39b700 1 --2- 172.21.15.44:0/651212574 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e1c7d21cc0 0x55e1c7d240a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=990292ee8bf0ecef server_cookie=302215c2dafdc80a in_seq=0 out_seq=0 2023-11-16T16:51:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.213+0000 7fccc1ffb700 1 -- 172.21.15.44:0/651212574 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e1c7c1c040 con 0x55e1c7d21cc0 2023-11-16T16:51:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.213+0000 7fccc1ffb700 10 client.15930 handle_command_reply: tid=0 2023-11-16T16:51:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.213+0000 7fccf3f28b80 1 client.15930 shutdown 2023-11-16T16:51:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.213+0000 7fccf3f28b80 1 -- 172.21.15.44:0/651212574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0eb90 msgr2=0x55e1c7d11d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.213+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/651212574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d11d10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fccd0097a40 tx=0x7fccd009ab10 comp rx=0 tx=0).stop 2023-11-16T16:51:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.213+0000 7fccf3f28b80 1 -- 172.21.15.44:0/651212574 shutdown_connections 2023-11-16T16:51:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.213+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/651212574 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e1c7d21cc0 0x55e1c7d240a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.213+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/651212574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d11d10 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.213+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/651212574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7b646b0 0x55e1c7d17ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.213+0000 7fccc1ffb700 0 client.15930 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.213+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/651212574 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7d0e720 0x55e1c7d18000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.213+0000 7fccf3f28b80 1 -- 172.21.15.44:0/651212574 >> 172.21.15.44:0/651212574 conn(0x55e1c7cfdf50 msgr2=0x55e1c7d027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.213+0000 7fccf3f28b80 1 -- 172.21.15.44:0/651212574 shutdown_connections 2023-11-16T16:51:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.213+0000 7fccf3f28b80 1 -- 172.21.15.44:0/651212574 wait complete. 2023-11-16T16:51:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.213+0000 7fccf3f28b80 20 client.15930 trim_cache size 0 max 16384 2023-11-16T16:51:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.229+0000 7fccf3f28b80 1 Processor -- start 2023-11-16T16:51:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.229+0000 7fccf3f28b80 1 -- start start 2023-11-16T16:51:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.229+0000 7fccf3f28b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7b646b0 0x55e1c7d02f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.229+0000 7fccf3f28b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0e720 0x55e1c7d0ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.229+0000 7fccf3f28b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d0d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.229+0000 7fccf3f28b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1c7cfffd0 con 0x55e1c7d0e720 2023-11-16T16:51:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.229+0000 7fccf3f28b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1c7cfdb70 con 0x55e1c7d0eb90 2023-11-16T16:51:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.229+0000 7fccf3f28b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1c7d0ca20 con 0x55e1c7b646b0 2023-11-16T16:51:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.229+0000 7fccddb9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0e720 0x55e1c7d0ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.229+0000 7fccddb9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0e720 0x55e1c7d0ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51808/0 (socket says 172.21.15.44:51808) 2023-11-16T16:51:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.229+0000 7fccddb9a700 1 -- 172.21.15.44:0/3048881549 learned_addr learned my addr 172.21.15.44:0/3048881549 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.229+0000 7fccdeb9c700 1 --2- 172.21.15.44:0/3048881549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d0d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.229+0000 7fccde39b700 1 --2- 172.21.15.44:0/3048881549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7b646b0 0x55e1c7d02f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.230+0000 7fccddb9a700 1 -- 172.21.15.44:0/3048881549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7b646b0 msgr2=0x55e1c7d02f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.230+0000 7fccddb9a700 1 --2- 172.21.15.44:0/3048881549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7b646b0 0x55e1c7d02f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.230+0000 7fccddb9a700 1 -- 172.21.15.44:0/3048881549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7d0eb90 msgr2=0x55e1c7d0d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.230+0000 7fccddb9a700 1 --2- 172.21.15.44:0/3048881549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d0d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.230+0000 7fccddb9a700 1 -- 172.21.15.44:0/3048881549 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e1c7c1fe80 con 0x55e1c7d0e720 2023-11-16T16:51:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.230+0000 7fccdeb9c700 1 --2- 172.21.15.44:0/3048881549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d0d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.230+0000 7fccddb9a700 1 --2- 172.21.15.44:0/3048881549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0e720 0x55e1c7d0ce30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fccc8009e70 tx=0x7fccc8004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d408e1a3d61654a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.230+0000 7fcccd3ff700 1 -- 172.21.15.44:0/3048881549 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8004a30 con 0x55e1c7d0e720 2023-11-16T16:51:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.230+0000 7fcccd3ff700 1 -- 172.21.15.44:0/3048881549 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fccc8018040 con 0x55e1c7d0e720 2023-11-16T16:51:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.230+0000 7fcccd3ff700 1 -- 172.21.15.44:0/3048881549 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8003ac0 con 0x55e1c7d0e720 2023-11-16T16:51:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.231+0000 7fccf3f28b80 1 -- 172.21.15.44:0/3048881549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0e720 msgr2=0x55e1c7d0ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.231+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/3048881549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0e720 0x55e1c7d0ce30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fccc8009e70 tx=0x7fccc8004320 comp rx=0 tx=0).stop 2023-11-16T16:51:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.231+0000 7fccf3f28b80 1 -- 172.21.15.44:0/3048881549 shutdown_connections 2023-11-16T16:51:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.231+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/3048881549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d0d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.231+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/3048881549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7b646b0 0x55e1c7d02f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.231+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/3048881549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0e720 0x55e1c7d0ce30 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.231+0000 7fccf3f28b80 1 -- 172.21.15.44:0/3048881549 >> 172.21.15.44:0/3048881549 conn(0x55e1c7cfdf50 msgr2=0x55e1c7d126a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.231+0000 7fccf3f28b80 1 -- 172.21.15.44:0/3048881549 shutdown_connections 2023-11-16T16:51:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.231+0000 7fccf3f28b80 1 -- 172.21.15.44:0/3048881549 wait complete. 2023-11-16T16:51:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.231+0000 7fccf3f28b80 1 -- start start 2023-11-16T16:51:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.232+0000 7fccf3f28b80 1 Processor -- start 2023-11-16T16:51:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.232+0000 7fccf3f28b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7b646b0 0x55e1c7d17750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.232+0000 7fccf3f28b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d17c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.232+0000 7fccf3f28b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7d15c70 0x55e1c7d160e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.232+0000 7fccf3f28b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1c7d16620 con 0x55e1c7d0eb90 2023-11-16T16:51:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.232+0000 7fccf3f28b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1c7d16760 con 0x55e1c7b646b0 2023-11-16T16:51:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.232+0000 7fccf3f28b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1c7d16920 con 0x55e1c7d15c70 2023-11-16T16:51:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.232+0000 7fccf3f28b80 10 client.0 _ll_register_callbacks cb 0x7fccd0009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.232+0000 7fccddb9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d17c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.232+0000 7fccddb9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d17c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51810/0 (socket says 172.21.15.44:51810) 2023-11-16T16:51:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.232+0000 7fccddb9a700 1 -- 172.21.15.44:0/1359368009 learned_addr learned my addr 172.21.15.44:0/1359368009 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.232+0000 7fccde39b700 1 --2- 172.21.15.44:0/1359368009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7b646b0 0x55e1c7d17750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.233+0000 7fccdeb9c700 1 --2- 172.21.15.44:0/1359368009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7d15c70 0x55e1c7d160e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.233+0000 7fccde39b700 1 -- 172.21.15.44:0/1359368009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7d15c70 msgr2=0x55e1c7d160e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.233+0000 7fccde39b700 1 --2- 172.21.15.44:0/1359368009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7d15c70 0x55e1c7d160e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.233+0000 7fccde39b700 1 -- 172.21.15.44:0/1359368009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0eb90 msgr2=0x55e1c7d17c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.233+0000 7fccde39b700 1 --2- 172.21.15.44:0/1359368009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d17c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.233+0000 7fccde39b700 1 -- 172.21.15.44:0/1359368009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e1c7c20130 con 0x55e1c7b646b0 2023-11-16T16:51:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.233+0000 7fccde39b700 1 --2- 172.21.15.44:0/1359368009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7b646b0 0x55e1c7d17750 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fccd4011be0 tx=0x7fccd4011d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=744d516faf0799eb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.233+0000 7fccc1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:51:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.233+0000 7fccc1ffb700 1 -- 172.21.15.44:0/1359368009 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd402c040 con 0x55e1c7b646b0 2023-11-16T16:51:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.233+0000 7fccf3f28b80 1 -- 172.21.15.44:0/1359368009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fccd4003e70 con 0x55e1c7b646b0 2023-11-16T16:51:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.233+0000 7fccc1ffb700 1 -- 172.21.15.44:0/1359368009 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fccd4033bd0 con 0x55e1c7b646b0 2023-11-16T16:51:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.234+0000 7fccc1ffb700 1 -- 172.21.15.44:0/1359368009 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccd4033d90 con 0x55e1c7b646b0 2023-11-16T16:51:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.234+0000 7fccf3f28b80 10 client.25952 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.234+0000 7fccf3f28b80 1 -- 172.21.15.44:0/1359368009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x7fccc8009b20 con 0x55e1c7b646b0 2023-11-16T16:51:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.234+0000 7fccc1ffb700 1 -- 172.21.15.44:0/1359368009 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fccd4003e70 con 0x55e1c7b646b0 2023-11-16T16:51:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.234+0000 7fccc1ffb700 1 -- 172.21.15.44:0/1359368009 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fccd4024020 con 0x55e1c7b646b0 2023-11-16T16:51:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.234+0000 7fccf3f28b80 10 client.25952 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.234+0000 7fccf3f28b80 10 client.25952 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.234+0000 7fccf3f28b80 20 client.25952 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.234+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/1359368009 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e1c7d77010 0x55e1c7d79420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.234+0000 7fccf3f28b80 4 client.25952 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:51:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.234+0000 7fccf3f28b80 1 -- 172.21.15.44:0/1359368009 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e1c7c28600 con 0x55e1c7d77010 2023-11-16T16:51:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.235+0000 7fccddb9a700 1 --2- 172.21.15.44:0/1359368009 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e1c7d77010 0x55e1c7d79420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.235+0000 7fccddb9a700 1 --2- 172.21.15.44:0/1359368009 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e1c7d77010 0x55e1c7d79420 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b780d744f969525d server_cookie=26fec530a48d7cd1 in_seq=0 out_seq=0 2023-11-16T16:51:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.235+0000 7fccc1ffb700 10 client.25952 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:51.247 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=40 2023-11-16T16:51:51.247 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.255+0000 7fccc1ffb700 1 -- 172.21.15.44:0/1359368009 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e1c7c28600 con 0x55e1c7d77010 2023-11-16T16:51:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.255+0000 7fccc1ffb700 10 client.25952 handle_command_reply: tid=0 2023-11-16T16:51:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.255+0000 7fccf3f28b80 1 client.25952 shutdown 2023-11-16T16:51:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.255+0000 7fccf3f28b80 1 -- 172.21.15.44:0/1359368009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7b646b0 msgr2=0x55e1c7d17750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.255+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/1359368009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7b646b0 0x55e1c7d17750 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fccd4011be0 tx=0x7fccd4011d00 comp rx=0 tx=0).stop 2023-11-16T16:51:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.256+0000 7fccf3f28b80 1 -- 172.21.15.44:0/1359368009 shutdown_connections 2023-11-16T16:51:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.256+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/1359368009 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e1c7d77010 0x55e1c7d79420 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.256+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/1359368009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e1c7d15c70 0x55e1c7d160e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.256+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/1359368009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1c7b646b0 0x55e1c7d17750 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.256+0000 7fccc1ffb700 0 client.25952 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.256+0000 7fccf3f28b80 1 --2- 172.21.15.44:0/1359368009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e1c7d0eb90 0x55e1c7d17c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.256+0000 7fccf3f28b80 1 -- 172.21.15.44:0/1359368009 >> 172.21.15.44:0/1359368009 conn(0x55e1c7cfdf50 msgr2=0x55e1c7d0ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.256+0000 7fccf3f28b80 1 -- 172.21.15.44:0/1359368009 shutdown_connections 2023-11-16T16:51:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.256+0000 7fccf3f28b80 1 -- 172.21.15.44:0/1359368009 wait complete. 2023-11-16T16:51:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.256+0000 7fccf3f28b80 20 client.25952 trim_cache size 0 max 16384 2023-11-16T16:51:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:51:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:51:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:51:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:51:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.257+0000 7fcce05ff700 1 -- 172.21.15.44:0/840272850 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fccc40a12c0 msgr2=0x7fccc40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.257+0000 7fcce05ff700 1 --2- 172.21.15.44:0/840272850 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fccc40a12c0 0x7fccc40a3770 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fccd4011490 tx=0x7fccd4009260 comp rx=0 tx=0).stop 2023-11-16T16:51:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.257+0000 7fcce05ff700 1 -- 172.21.15.44:0/840272850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccd8105560 msgr2=0x7fccd81a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.257+0000 7fcce05ff700 1 --2- 172.21.15.44:0/840272850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccd8105560 0x7fccd81a4690 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fccd0007820 tx=0x7fccd0007850 comp rx=0 tx=0).stop 2023-11-16T16:51:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.257+0000 7fcce05ff700 1 -- 172.21.15.44:0/840272850 shutdown_connections 2023-11-16T16:51:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.257+0000 7fcce05ff700 1 --2- 172.21.15.44:0/840272850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccd8105560 0x7fccd81a4690 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.257+0000 7fcce05ff700 1 --2- 172.21.15.44:0/840272850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccd8100c50 0x7fccd8072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.257+0000 7fcce05ff700 1 --2- 172.21.15.44:0/840272850 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fccc40a12c0 0x7fccc40a3770 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.257+0000 7fcce05ff700 1 --2- 172.21.15.44:0/840272850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fccd81015f0 0x7fccd8073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.257+0000 7fcce05ff700 1 -- 172.21.15.44:0/840272850 >> 172.21.15.44:0/840272850 conn(0x7fccd8077550 msgr2=0x7fccd80fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.257+0000 7fcce05ff700 1 -- 172.21.15.44:0/840272850 shutdown_connections 2023-11-16T16:51:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:51:51.257+0000 7fcce05ff700 1 -- 172.21.15.44:0/840272850 wait complete. 2023-11-16T16:51:51.298 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:51.298 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:51.300 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:51.301 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:51.301 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:51.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:51 smithi044 ceph-mon[128750]: pgmap v60: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 5 op/s 2023-11-16T16:51:51.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:51 smithi176 ceph-mon[131630]: pgmap v60: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 5 op/s 2023-11-16T16:51:51.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:51 smithi077 ceph-mon[131204]: pgmap v60: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 5 op/s 2023-11-16T16:51:52.015 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:52.015 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:52.260 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:52.260 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:52.300 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:52.301 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:52.301 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 40 -lt 100 ']' 2023-11-16T16:51:52.301 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 40 + 1 2023-11-16T16:51:52.339 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=41 2023-11-16T16:51:52.339 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:52.414 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:52.415 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:52.417 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:52.418 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:52.418 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:52.604 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.602+0000 7fbdd69ff700 1 -- 172.21.15.44:0/1323834267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbdd0100c50 msgr2=0x7fbdd0101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:52.604 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.602+0000 7fbdd69ff700 1 --2- 172.21.15.44:0/1323834267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbdd0100c50 0x7fbdd0101020 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0009990 tx=0x7fbdc0009ca0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.604 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.602+0000 7fbdd69ff700 1 -- 172.21.15.44:0/1323834267 shutdown_connections 2023-11-16T16:51:52.604 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.602+0000 7fbdd69ff700 1 --2- 172.21.15.44:0/1323834267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdd0105630 0x7fbdd010d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.604 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.602+0000 7fbdd69ff700 1 --2- 172.21.15.44:0/1323834267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbdd0100c50 0x7fbdd0101020 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.602+0000 7fbdd69ff700 1 --2- 172.21.15.44:0/1323834267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdd01015f0 0x7fbdd0104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.602+0000 7fbdd69ff700 1 -- 172.21.15.44:0/1323834267 >> 172.21.15.44:0/1323834267 conn(0x7fbdd00775c0 msgr2=0x7fbdd00779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:52.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.602+0000 7fbdd69ff700 1 -- 172.21.15.44:0/1323834267 shutdown_connections 2023-11-16T16:51:52.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.602+0000 7fbdd69ff700 1 -- 172.21.15.44:0/1323834267 wait complete. 2023-11-16T16:51:52.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.604+0000 7fbdd69ff700 1 Processor -- start 2023-11-16T16:51:52.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.604+0000 7fbdd69ff700 1 -- start start 2023-11-16T16:51:52.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.605+0000 7fbdd69ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdd0100c50 0x7fbdd01986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.605+0000 7fbdd69ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbdd01015f0 0x7fbdd0198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.605+0000 7fbdd69ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdd0105630 0x7fbdd019bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.605+0000 7fbdd69ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd019c5d0 con 0x7fbdd0105630 2023-11-16T16:51:52.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.605+0000 7fbdd69ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd019c740 con 0x7fbdd0100c50 2023-11-16T16:51:52.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.605+0000 7fbdd69ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd019c900 con 0x7fbdd01015f0 2023-11-16T16:51:52.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.605+0000 7fbdcffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdd0100c50 0x7fbdd01986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.605+0000 7fbdcffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdd0100c50 0x7fbdd01986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:54444/0 (socket says 172.21.15.44:54444) 2023-11-16T16:51:52.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.605+0000 7fbdcffff700 1 -- 172.21.15.44:0/105039326 learned_addr learned my addr 172.21.15.44:0/105039326 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:52.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.605+0000 7fbdcf7fe700 1 --2- 172.21.15.44:0/105039326 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbdd01015f0 0x7fbdd0198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.605+0000 7fbdd4f9c700 1 --2- 172.21.15.44:0/105039326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdd0105630 0x7fbdd019bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.606+0000 7fbdcffff700 1 -- 172.21.15.44:0/105039326 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbdd01015f0 msgr2=0x7fbdd0198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:52.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.606+0000 7fbdcffff700 1 --2- 172.21.15.44:0/105039326 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbdd01015f0 0x7fbdd0198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.606+0000 7fbdcffff700 1 -- 172.21.15.44:0/105039326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdd0105630 msgr2=0x7fbdd019bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:52.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.606+0000 7fbdcffff700 1 --2- 172.21.15.44:0/105039326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdd0105630 0x7fbdd019bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.606+0000 7fbdcffff700 1 -- 172.21.15.44:0/105039326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc00095f0 con 0x7fbdd0100c50 2023-11-16T16:51:52.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.606+0000 7fbdcffff700 1 --2- 172.21.15.44:0/105039326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdd0100c50 0x7fbdd01986e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0000c00 tx=0x7fbdc0005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:52.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.606+0000 7fbdcd7fa700 1 -- 172.21.15.44:0/105039326 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc001d070 con 0x7fbdd0100c50 2023-11-16T16:51:52.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.606+0000 7fbdd69ff700 1 -- 172.21.15.44:0/105039326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd01103f0 con 0x7fbdd0100c50 2023-11-16T16:51:52.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.606+0000 7fbdcd7fa700 1 -- 172.21.15.44:0/105039326 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbdc0021410 con 0x7fbdd0100c50 2023-11-16T16:51:52.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.606+0000 7fbdcd7fa700 1 -- 172.21.15.44:0/105039326 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0017650 con 0x7fbdd0100c50 2023-11-16T16:51:52.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.606+0000 7fbdd69ff700 1 -- 172.21.15.44:0/105039326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd0110260 con 0x7fbdd0100c50 2023-11-16T16:51:52.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.609+0000 7fbdcd7fa700 1 -- 172.21.15.44:0/105039326 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fbdc0021580 con 0x7fbdd0100c50 2023-11-16T16:51:52.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.611+0000 7fbdcd7fa700 1 --2- 172.21.15.44:0/105039326 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbdb80a12c0 0x7fbdb80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.611+0000 7fbdcd7fa700 1 -- 172.21.15.44:0/105039326 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fbdc00ce380 con 0x7fbdd0100c50 2023-11-16T16:51:52.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.611+0000 7fbdcf7fe700 1 --2- 172.21.15.44:0/105039326 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbdb80a12c0 0x7fbdb80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.613 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.612+0000 7fbdcf7fe700 1 --2- 172.21.15.44:0/105039326 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbdb80a12c0 0x7fbdb80a3770 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0199cf0 tx=0x7fbdc4009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:52.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.687+0000 7fbdea31bb80 1 Processor -- start 2023-11-16T16:51:52.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.688+0000 7fbdea31bb80 1 -- start start 2023-11-16T16:51:52.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.688+0000 7fbdea31bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af6b60 0x5584b7af6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.688+0000 7fbdea31bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7af7470 0x5584b7afb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.688+0000 7fbdea31bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7afb8b0 0x5584b7afbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.688+0000 7fbdea31bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b79fc060 con 0x5584b7af6b60 2023-11-16T16:51:52.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.688+0000 7fbdea31bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b7a4e8a0 con 0x5584b7af7470 2023-11-16T16:51:52.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.688+0000 7fbdea31bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b7a47d30 con 0x5584b7afb8b0 2023-11-16T16:51:52.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.688+0000 7fbdcffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af6b60 0x5584b7af6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.688+0000 7fbdcffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af6b60 0x5584b7af6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51826/0 (socket says 172.21.15.44:51826) 2023-11-16T16:51:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.688+0000 7fbdcffff700 1 -- 172.21.15.44:0/3446779382 learned_addr learned my addr 172.21.15.44:0/3446779382 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.688+0000 7fbdcf7fe700 1 --2- 172.21.15.44:0/3446779382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7af7470 0x5584b7afb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.688+0000 7fbdd4f9c700 1 --2- 172.21.15.44:0/3446779382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7afb8b0 0x5584b7afbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.689+0000 7fbdcffff700 1 -- 172.21.15.44:0/3446779382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7afb8b0 msgr2=0x5584b7afbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.689+0000 7fbdcffff700 1 --2- 172.21.15.44:0/3446779382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7afb8b0 0x5584b7afbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.689+0000 7fbdcffff700 1 -- 172.21.15.44:0/3446779382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7af7470 msgr2=0x5584b7afb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.689+0000 7fbdcffff700 1 --2- 172.21.15.44:0/3446779382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7af7470 0x5584b7afb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.689+0000 7fbdcffff700 1 -- 172.21.15.44:0/3446779382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584b78c9630 con 0x5584b7af6b60 2023-11-16T16:51:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.689+0000 7fbdd4f9c700 1 --2- 172.21.15.44:0/3446779382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7afb8b0 0x5584b7afbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:51:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.689+0000 7fbdcffff700 1 --2- 172.21.15.44:0/3446779382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af6b60 0x5584b7af6f30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbdc00a2f90 tx=0x7fbdc0097550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d2362c5e7f28b9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.689+0000 7fbdb67fc700 1 -- 172.21.15.44:0/3446779382 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0021980 con 0x5584b7af6b60 2023-11-16T16:51:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.689+0000 7fbdb67fc700 1 -- 172.21.15.44:0/3446779382 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbdc00ce170 con 0x5584b7af6b60 2023-11-16T16:51:52.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.689+0000 7fbdb67fc700 1 -- 172.21.15.44:0/3446779382 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc009a7f0 con 0x5584b7af6b60 2023-11-16T16:51:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.690+0000 7fbdea31bb80 1 -- 172.21.15.44:0/3446779382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af6b60 msgr2=0x5584b7af6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.690+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/3446779382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af6b60 0x5584b7af6f30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbdc00a2f90 tx=0x7fbdc0097550 comp rx=0 tx=0).stop 2023-11-16T16:51:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.690+0000 7fbdea31bb80 1 -- 172.21.15.44:0/3446779382 shutdown_connections 2023-11-16T16:51:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.690+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/3446779382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7afb8b0 0x5584b7afbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.690+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/3446779382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af6b60 0x5584b7af6f30 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.690+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/3446779382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7af7470 0x5584b7afb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.690+0000 7fbdea31bb80 1 -- 172.21.15.44:0/3446779382 >> 172.21.15.44:0/3446779382 conn(0x5584b7aec360 msgr2=0x5584b7af1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.690+0000 7fbdea31bb80 1 -- 172.21.15.44:0/3446779382 shutdown_connections 2023-11-16T16:51:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.690+0000 7fbdea31bb80 1 -- 172.21.15.44:0/3446779382 wait complete. 2023-11-16T16:51:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.691+0000 7fbdea31bb80 1 -- start start 2023-11-16T16:51:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.691+0000 7fbdea31bb80 1 Processor -- start 2023-11-16T16:51:52.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.691+0000 7fbdea31bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af6b60 0x5584b7b02f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.691+0000 7fbdea31bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7af7470 0x5584b7b001c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.691+0000 7fbdea31bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7afb8b0 0x5584b7b00740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.691+0000 7fbdea31bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b79e98e0 con 0x5584b7af6b60 2023-11-16T16:51:52.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.691+0000 7fbdea31bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b79e9580 con 0x5584b7af7470 2023-11-16T16:51:52.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.691+0000 7fbdea31bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b78cb920 con 0x5584b7afb8b0 2023-11-16T16:51:52.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.691+0000 7fbdea31bb80 10 client.0 _ll_register_callbacks cb 0x5584b7a07780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:52.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.692+0000 7fbdcf7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7af7470 0x5584b7b001c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.692+0000 7fbdcffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af6b60 0x5584b7b02f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.692+0000 7fbdcf7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7af7470 0x5584b7b001c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:54464/0 (socket says 172.21.15.44:54464) 2023-11-16T16:51:52.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.692+0000 7fbdcf7fe700 1 -- 172.21.15.44:0/2061792322 learned_addr learned my addr 172.21.15.44:0/2061792322 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:52.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.692+0000 7fbdd4f9c700 1 --2- 172.21.15.44:0/2061792322 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7afb8b0 0x5584b7b00740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.692+0000 7fbdcf7fe700 1 -- 172.21.15.44:0/2061792322 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7afb8b0 msgr2=0x5584b7b00740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.692+0000 7fbdcf7fe700 1 --2- 172.21.15.44:0/2061792322 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7afb8b0 0x5584b7b00740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.692+0000 7fbdcf7fe700 1 -- 172.21.15.44:0/2061792322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af6b60 msgr2=0x5584b7b02f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.692+0000 7fbdcf7fe700 1 --2- 172.21.15.44:0/2061792322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af6b60 0x5584b7b02f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.692+0000 7fbdcf7fe700 1 -- 172.21.15.44:0/2061792322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd0110260 con 0x5584b7af7470 2023-11-16T16:51:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.692+0000 7fbdcf7fe700 1 --2- 172.21.15.44:0/2061792322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7af7470 0x5584b7b001c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4004500 tx=0x7fbdc40047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46a1add126af8076 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.693+0000 7fbdaf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:51:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.693+0000 7fbdaf7fe700 1 -- 172.21.15.44:0/2061792322 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4022070 con 0x5584b7af7470 2023-11-16T16:51:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.693+0000 7fbdea31bb80 1 -- 172.21.15.44:0/2061792322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5584b7ae9b20 con 0x5584b7af7470 2023-11-16T16:51:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.693+0000 7fbdaf7fe700 1 -- 172.21.15.44:0/2061792322 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fbdc4004530 con 0x5584b7af7470 2023-11-16T16:51:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.693+0000 7fbdaf7fe700 1 -- 172.21.15.44:0/2061792322 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbdc4027420 con 0x5584b7af7470 2023-11-16T16:51:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.693+0000 7fbdea31bb80 10 client.25964 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.693+0000 7fbdea31bb80 1 -- 172.21.15.44:0/2061792322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5584b7a0e190 con 0x5584b7af7470 2023-11-16T16:51:52.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.693+0000 7fbdaf7fe700 1 -- 172.21.15.44:0/2061792322 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5584b7ae9b20 con 0x5584b7af7470 2023-11-16T16:51:52.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.694+0000 7fbdaf7fe700 1 -- 172.21.15.44:0/2061792322 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbdc401d6e0 con 0x5584b7af7470 2023-11-16T16:51:52.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.694+0000 7fbdea31bb80 10 client.25964 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:52.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.694+0000 7fbdea31bb80 10 client.25964 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:52.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.694+0000 7fbdea31bb80 20 client.25964 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:52.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.694+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/2061792322 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5584b7b105d0 0x5584b7b129e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.694+0000 7fbdea31bb80 4 client.25964 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:51:52.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.694+0000 7fbdea31bb80 1 -- 172.21.15.44:0/2061792322 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5584b7a0a0a0 con 0x5584b7b105d0 2023-11-16T16:51:52.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.694+0000 7fbdcffff700 1 --2- 172.21.15.44:0/2061792322 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5584b7b105d0 0x5584b7b129e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.695+0000 7fbdcffff700 1 --2- 172.21.15.44:0/2061792322 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5584b7b105d0 0x5584b7b129e0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b774a4a7b7ff6dc server_cookie=9e4bf1bc1ea9c3e8 in_seq=0 out_seq=0 2023-11-16T16:51:52.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.695+0000 7fbdaf7fe700 10 client.25964 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:52.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.695+0000 7fbdaf7fe700 1 -- 172.21.15.44:0/2061792322 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5584b7a0a0a0 con 0x5584b7b105d0 2023-11-16T16:51:52.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.695+0000 7fbdaf7fe700 10 client.25964 handle_command_reply: tid=0 2023-11-16T16:51:52.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.695+0000 7fbdea31bb80 1 client.25964 shutdown 2023-11-16T16:51:52.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.696+0000 7fbdea31bb80 1 -- 172.21.15.44:0/2061792322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7af7470 msgr2=0x5584b7b001c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:52.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.696+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/2061792322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7af7470 0x5584b7b001c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4004500 tx=0x7fbdc40047c0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.696+0000 7fbdea31bb80 1 -- 172.21.15.44:0/2061792322 shutdown_connections 2023-11-16T16:51:52.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.696+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/2061792322 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5584b7b105d0 0x5584b7b129e0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.696+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/2061792322 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7afb8b0 0x5584b7b00740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.696+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/2061792322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af6b60 0x5584b7b02f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.696+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/2061792322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7af7470 0x5584b7b001c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.696+0000 7fbdea31bb80 1 -- 172.21.15.44:0/2061792322 >> 172.21.15.44:0/2061792322 conn(0x5584b7aec360 msgr2=0x5584b7afcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:52.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.696+0000 7fbdaf7fe700 0 client.25964 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:52.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.696+0000 7fbdea31bb80 1 -- 172.21.15.44:0/2061792322 shutdown_connections 2023-11-16T16:51:52.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.696+0000 7fbdea31bb80 1 -- 172.21.15.44:0/2061792322 wait complete. 2023-11-16T16:51:52.702 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.696+0000 7fbdea31bb80 20 client.25964 trim_cache size 0 max 16384 2023-11-16T16:51:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.711+0000 7fbdea31bb80 1 Processor -- start 2023-11-16T16:51:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.711+0000 7fbdea31bb80 1 -- start start 2023-11-16T16:51:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.711+0000 7fbdea31bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7af6b60 0x5584b7b0c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.711+0000 7fbdea31bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af7470 0x5584b7b02ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.711+0000 7fbdea31bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7afb8b0 0x5584b7af98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.711+0000 7fbdea31bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b7afa3e0 con 0x5584b7af7470 2023-11-16T16:51:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.711+0000 7fbdea31bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b7b12c00 con 0x5584b7afb8b0 2023-11-16T16:51:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.711+0000 7fbdea31bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b7b12dc0 con 0x5584b7af6b60 2023-11-16T16:51:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.712+0000 7fbdcf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af7470 0x5584b7b02ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.712+0000 7fbdcf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af7470 0x5584b7b02ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51838/0 (socket says 172.21.15.44:51838) 2023-11-16T16:51:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.712+0000 7fbdcf7fe700 1 -- 172.21.15.44:0/1067814186 learned_addr learned my addr 172.21.15.44:0/1067814186 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.712+0000 7fbdd4f9c700 1 --2- 172.21.15.44:0/1067814186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7afb8b0 0x5584b7af98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.712+0000 7fbdcffff700 1 --2- 172.21.15.44:0/1067814186 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7af6b60 0x5584b7b0c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.712+0000 7fbdcf7fe700 1 -- 172.21.15.44:0/1067814186 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7af6b60 msgr2=0x5584b7b0c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.712+0000 7fbdcf7fe700 1 --2- 172.21.15.44:0/1067814186 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7af6b60 0x5584b7b0c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.712+0000 7fbdcf7fe700 1 -- 172.21.15.44:0/1067814186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7afb8b0 msgr2=0x5584b7af98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.712+0000 7fbdcf7fe700 1 --2- 172.21.15.44:0/1067814186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7afb8b0 0x5584b7af98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.712+0000 7fbdcf7fe700 1 -- 172.21.15.44:0/1067814186 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584b7a0dee0 con 0x5584b7af7470 2023-11-16T16:51:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.712+0000 7fbdcf7fe700 1 --2- 172.21.15.44:0/1067814186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af7470 0x5584b7b02ae0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40097c0 tx=0x7fbdc4009350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2bea12577942e00 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.713+0000 7fbdb6ffd700 1 -- 172.21.15.44:0/1067814186 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4039070 con 0x5584b7af7470 2023-11-16T16:51:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.713+0000 7fbdb6ffd700 1 -- 172.21.15.44:0/1067814186 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbdc4004530 con 0x5584b7af7470 2023-11-16T16:51:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.713+0000 7fbdb6ffd700 1 -- 172.21.15.44:0/1067814186 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4022070 con 0x5584b7af7470 2023-11-16T16:51:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.713+0000 7fbdea31bb80 1 -- 172.21.15.44:0/1067814186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af7470 msgr2=0x5584b7b02ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.713+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/1067814186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af7470 0x5584b7b02ae0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40097c0 tx=0x7fbdc4009350 comp rx=0 tx=0).stop 2023-11-16T16:51:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.713+0000 7fbdea31bb80 1 -- 172.21.15.44:0/1067814186 shutdown_connections 2023-11-16T16:51:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.713+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/1067814186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7afb8b0 0x5584b7af98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.713+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/1067814186 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7af6b60 0x5584b7b0c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.713+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/1067814186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af7470 0x5584b7b02ae0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.713+0000 7fbdea31bb80 1 -- 172.21.15.44:0/1067814186 >> 172.21.15.44:0/1067814186 conn(0x5584b7aec360 msgr2=0x5584b7af0550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.713+0000 7fbdea31bb80 1 -- 172.21.15.44:0/1067814186 shutdown_connections 2023-11-16T16:51:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.714+0000 7fbdea31bb80 1 -- 172.21.15.44:0/1067814186 wait complete. 2023-11-16T16:51:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.714+0000 7fbdea31bb80 1 -- start start 2023-11-16T16:51:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.714+0000 7fbdea31bb80 1 Processor -- start 2023-11-16T16:51:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdea31bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7af6b60 0x5584b7b05740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdea31bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af7470 0x5584b7b05c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdea31bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7afb8b0 0x5584b7b63450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdea31bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b7b06590 con 0x5584b7af7470 2023-11-16T16:51:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdea31bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b7b63990 con 0x5584b7afb8b0 2023-11-16T16:51:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdea31bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b7b63ad0 con 0x5584b7af6b60 2023-11-16T16:51:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdea31bb80 10 client.0 _ll_register_callbacks cb 0x5584b7a07780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdcf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af7470 0x5584b7b05c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdcf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af7470 0x5584b7b05c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51850/0 (socket says 172.21.15.44:51850) 2023-11-16T16:51:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdcf7fe700 1 -- 172.21.15.44:0/3319465334 learned_addr learned my addr 172.21.15.44:0/3319465334 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdd4f9c700 1 --2- 172.21.15.44:0/3319465334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7afb8b0 0x5584b7b63450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdcffff700 1 --2- 172.21.15.44:0/3319465334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7af6b60 0x5584b7b05740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdd4f9c700 1 -- 172.21.15.44:0/3319465334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7af6b60 msgr2=0x5584b7b05740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdd4f9c700 1 --2- 172.21.15.44:0/3319465334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7af6b60 0x5584b7b05740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdd4f9c700 1 -- 172.21.15.44:0/3319465334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af7470 msgr2=0x5584b7b05c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdd4f9c700 1 --2- 172.21.15.44:0/3319465334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af7470 0x5584b7b05c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdd4f9c700 1 -- 172.21.15.44:0/3319465334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5584b7a0e190 con 0x5584b7afb8b0 2023-11-16T16:51:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.715+0000 7fbdcf7fe700 1 --2- 172.21.15.44:0/3319465334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af7470 0x5584b7b05c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.716+0000 7fbdd4f9c700 1 --2- 172.21.15.44:0/3319465334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7afb8b0 0x5584b7b63450 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009350 tx=0x7fbdc8009380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=962a258e79692c44 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.716+0000 7fbdaf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:51:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.716+0000 7fbdaf7fe700 1 -- 172.21.15.44:0/3319465334 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc800e040 con 0x5584b7afb8b0 2023-11-16T16:51:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.716+0000 7fbdea31bb80 1 -- 172.21.15.44:0/3319465334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbdc00ce170 con 0x5584b7afb8b0 2023-11-16T16:51:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.716+0000 7fbdaf7fe700 1 -- 172.21.15.44:0/3319465334 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fbdc8004db0 con 0x5584b7afb8b0 2023-11-16T16:51:52.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.716+0000 7fbdaf7fe700 1 -- 172.21.15.44:0/3319465334 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbdc8018b60 con 0x5584b7afb8b0 2023-11-16T16:51:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.716+0000 7fbdea31bb80 10 client.25970 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.716+0000 7fbdea31bb80 1 -- 172.21.15.44:0/3319465334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5584b785ec60 con 0x5584b7afb8b0 2023-11-16T16:51:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.717+0000 7fbdaf7fe700 1 -- 172.21.15.44:0/3319465334 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbdc00ce170 con 0x5584b7afb8b0 2023-11-16T16:51:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.717+0000 7fbdaf7fe700 1 -- 172.21.15.44:0/3319465334 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbdc8005020 con 0x5584b7afb8b0 2023-11-16T16:51:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.717+0000 7fbdea31bb80 10 client.25970 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.717+0000 7fbdea31bb80 10 client.25970 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.717+0000 7fbdea31bb80 20 client.25970 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.717+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/3319465334 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5584b7b65260 0x5584b7b67670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.717+0000 7fbdea31bb80 4 client.25970 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T16:51:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.717+0000 7fbdea31bb80 1 -- 172.21.15.44:0/3319465334 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5584b7a16660 con 0x5584b7b65260 2023-11-16T16:51:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.717+0000 7fbdcffff700 1 --2- 172.21.15.44:0/3319465334 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5584b7b65260 0x5584b7b67670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdcffff700 1 --2- 172.21.15.44:0/3319465334 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5584b7b65260 0x5584b7b67670 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c23b795b944e3eac server_cookie=bca9076187a5d564 in_seq=0 out_seq=0 2023-11-16T16:51:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdaf7fe700 10 client.25970 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdaf7fe700 1 -- 172.21.15.44:0/3319465334 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5584b7a16660 con 0x5584b7b65260 2023-11-16T16:51:52.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdaf7fe700 10 client.25970 handle_command_reply: tid=0 2023-11-16T16:51:52.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdea31bb80 1 client.25970 shutdown 2023-11-16T16:51:52.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdea31bb80 1 -- 172.21.15.44:0/3319465334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7afb8b0 msgr2=0x5584b7b63450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:52.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/3319465334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7afb8b0 0x5584b7b63450 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009350 tx=0x7fbdc8009380 comp rx=0 tx=0).stop 2023-11-16T16:51:52.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdea31bb80 1 -- 172.21.15.44:0/3319465334 shutdown_connections 2023-11-16T16:51:52.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/3319465334 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5584b7b65260 0x5584b7b67670 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/3319465334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b7afb8b0 0x5584b7b63450 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/3319465334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584b7af6b60 0x5584b7b05740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdea31bb80 1 --2- 172.21.15.44:0/3319465334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b7af7470 0x5584b7b05c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdaf7fe700 0 client.25970 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:52.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdea31bb80 1 -- 172.21.15.44:0/3319465334 >> 172.21.15.44:0/3319465334 conn(0x5584b7aec360 msgr2=0x5584b7b0c850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:52.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdea31bb80 1 -- 172.21.15.44:0/3319465334 shutdown_connections 2023-11-16T16:51:52.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdea31bb80 1 -- 172.21.15.44:0/3319465334 wait complete. 2023-11-16T16:51:52.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.718+0000 7fbdea31bb80 20 client.25970 trim_cache size 0 max 16384 2023-11-16T16:51:52.725 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T16:51:52.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.719+0000 7fbdd69ff700 1 -- 172.21.15.44:0/105039326 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbdb80a12c0 msgr2=0x7fbdb80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:52.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.719+0000 7fbdd69ff700 1 --2- 172.21.15.44:0/105039326 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbdb80a12c0 0x7fbdb80a3770 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0199cf0 tx=0x7fbdc4009260 comp rx=0 tx=0).stop 2023-11-16T16:51:52.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.719+0000 7fbdd69ff700 1 -- 172.21.15.44:0/105039326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdd0100c50 msgr2=0x7fbdd01986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:52.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.719+0000 7fbdd69ff700 1 --2- 172.21.15.44:0/105039326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdd0100c50 0x7fbdd01986e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0000c00 tx=0x7fbdc0005740 comp rx=0 tx=0).stop 2023-11-16T16:51:52.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.719+0000 7fbdd69ff700 1 -- 172.21.15.44:0/105039326 shutdown_connections 2023-11-16T16:51:52.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.719+0000 7fbdd69ff700 1 --2- 172.21.15.44:0/105039326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdd0105630 0x7fbdd019bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.719+0000 7fbdd69ff700 1 --2- 172.21.15.44:0/105039326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdd0100c50 0x7fbdd01986e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.719+0000 7fbdd69ff700 1 --2- 172.21.15.44:0/105039326 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbdb80a12c0 0x7fbdb80a3770 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.719+0000 7fbdd69ff700 1 --2- 172.21.15.44:0/105039326 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbdd01015f0 0x7fbdd0198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.719+0000 7fbdd69ff700 1 -- 172.21.15.44:0/105039326 >> 172.21.15.44:0/105039326 conn(0x7fbdd00775c0 msgr2=0x7fbdd00fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:52.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.719+0000 7fbdd69ff700 1 -- 172.21.15.44:0/105039326 shutdown_connections 2023-11-16T16:51:52.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:52.719+0000 7fbdd69ff700 1 -- 172.21.15.44:0/105039326 wait complete. 2023-11-16T16:51:53.357 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:53.357 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:53.568 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:53.569 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:53.592 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:53.593 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:53.593 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 41 -lt 100 ']' 2023-11-16T16:51:53.593 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 41 + 1 2023-11-16T16:51:53.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:53 smithi044 ceph-mon[128750]: pgmap v61: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 26 KiB/s wr, 12 op/s 2023-11-16T16:51:53.625 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=42 2023-11-16T16:51:53.625 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:53.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:53 smithi176 ceph-mon[131630]: pgmap v61: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 26 KiB/s wr, 12 op/s 2023-11-16T16:51:53.656 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:53.656 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:53.658 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:53.659 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:53.660 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:53.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:53 smithi077 ceph-mon[131204]: pgmap v61: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 26 KiB/s wr, 12 op/s 2023-11-16T16:51:53.754 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:51:53.754 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:51:54.164 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:54.165 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:54.262 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:54.262 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:54.274 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:54.275 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:54.275 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 42 -lt 100 ']' 2023-11-16T16:51:54.275 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 42 + 1 2023-11-16T16:51:54.313 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=43 2023-11-16T16:51:54.313 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:54.332 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:54.332 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:54.334 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:54.335 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:54.335 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:55.089 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:55.090 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:55.249 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:55.249 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:55.267 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:55.267 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:55.267 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 43 -lt 100 ']' 2023-11-16T16:51:55.268 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 43 + 1 2023-11-16T16:51:55.276 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=44 2023-11-16T16:51:55.277 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:55.322 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:55.323 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:55.325 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:55.326 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:55.326 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:55.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:55 smithi044 ceph-mon[128750]: pgmap v62: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-11-16T16:51:55.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:55 smithi176 ceph-mon[131630]: pgmap v62: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-11-16T16:51:55.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:55 smithi077 ceph-mon[131204]: pgmap v62: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-11-16T16:51:56.056 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:56.056 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:56.285 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:56.285 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:56.366 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:56.366 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:56.366 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 44 -lt 100 ']' 2023-11-16T16:51:56.367 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 44 + 1 2023-11-16T16:51:56.408 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=45 2023-11-16T16:51:56.408 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:56.448 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:56.448 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:56.451 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:56.451 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:56.451 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:56.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.685+0000 7f03efdff700 1 -- 172.21.15.44:0/2644733468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03e8101660 msgr2=0x7f03e8104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:56.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.685+0000 7f03efdff700 1 --2- 172.21.15.44:0/2644733468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03e8101660 0x7f03e8104ef0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f03d80099c0 tx=0x7f03d8009cd0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.685+0000 7f03efdff700 1 -- 172.21.15.44:0/2644733468 shutdown_connections 2023-11-16T16:51:56.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.685+0000 7f03efdff700 1 --2- 172.21.15.44:0/2644733468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e8105620 0x7f03e810d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.685+0000 7f03efdff700 1 --2- 172.21.15.44:0/2644733468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03e8100cc0 0x7f03e8101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.685+0000 7f03efdff700 1 --2- 172.21.15.44:0/2644733468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03e8101660 0x7f03e8104ef0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.685+0000 7f03efdff700 1 -- 172.21.15.44:0/2644733468 >> 172.21.15.44:0/2644733468 conn(0x7f03e8077560 msgr2=0x7f03e8077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:56.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.686+0000 7f03efdff700 1 -- 172.21.15.44:0/2644733468 shutdown_connections 2023-11-16T16:51:56.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.686+0000 7f03efdff700 1 -- 172.21.15.44:0/2644733468 wait complete. 2023-11-16T16:51:56.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.687+0000 7f03efdff700 1 Processor -- start 2023-11-16T16:51:56.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.687+0000 7f03efdff700 1 -- start start 2023-11-16T16:51:56.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.687+0000 7f03efdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e8100cc0 0x7f03e81964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.687+0000 7f03efdff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03e8101660 0x7f03e81969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.687+0000 7f03efdff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03e8105620 0x7f03e819ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.687+0000 7f03efdff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e819b490 con 0x7f03e8105620 2023-11-16T16:51:56.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.687+0000 7f03efdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e819b600 con 0x7f03e8100cc0 2023-11-16T16:51:56.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.687+0000 7f03efdff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e819b7c0 con 0x7f03e8101660 2023-11-16T16:51:56.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.688+0000 7f03edb9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e8100cc0 0x7f03e81964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.688+0000 7f03ed39a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03e8101660 0x7f03e81969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.688+0000 7f03ee39c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03e8105620 0x7f03e819ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.688+0000 7f03edb9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e8100cc0 0x7f03e81964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59626/0 (socket says 172.21.15.44:59626) 2023-11-16T16:51:56.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.688+0000 7f03edb9b700 1 -- 172.21.15.44:0/1877003295 learned_addr learned my addr 172.21.15.44:0/1877003295 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:56.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.688+0000 7f03ed39a700 1 -- 172.21.15.44:0/1877003295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e8100cc0 msgr2=0x7f03e81964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:56.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.688+0000 7f03ed39a700 1 --2- 172.21.15.44:0/1877003295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e8100cc0 0x7f03e81964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.688+0000 7f03ed39a700 1 -- 172.21.15.44:0/1877003295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03e8105620 msgr2=0x7f03e819ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:56.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.688+0000 7f03ed39a700 1 --2- 172.21.15.44:0/1877003295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03e8105620 0x7f03e819ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.688+0000 7f03ed39a700 1 -- 172.21.15.44:0/1877003295 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d80095f0 con 0x7f03e8101660 2023-11-16T16:51:56.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.688+0000 7f03edb9b700 1 --2- 172.21.15.44:0/1877003295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e8100cc0 0x7f03e81964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:56.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.688+0000 7f03ed39a700 1 --2- 172.21.15.44:0/1877003295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03e8101660 0x7f03e81969f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f03e806a350 tx=0x7f03d8009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:56.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.689+0000 7f03deffd700 1 -- 172.21.15.44:0/1877003295 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d801d070 con 0x7f03e8101660 2023-11-16T16:51:56.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.689+0000 7f03efdff700 1 -- 172.21.15.44:0/1877003295 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e8110370 con 0x7f03e8101660 2023-11-16T16:51:56.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.689+0000 7f03efdff700 1 -- 172.21.15.44:0/1877003295 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e81101e0 con 0x7f03e8101660 2023-11-16T16:51:56.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.689+0000 7f03deffd700 1 -- 172.21.15.44:0/1877003295 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03d8005020 con 0x7f03e8101660 2023-11-16T16:51:56.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.689+0000 7f03deffd700 1 -- 172.21.15.44:0/1877003295 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d8004970 con 0x7f03e8101660 2023-11-16T16:51:56.694 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.693+0000 7f03deffd700 1 -- 172.21.15.44:0/1877003295 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f03d8005420 con 0x7f03e8101660 2023-11-16T16:51:56.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.694+0000 7f03deffd700 1 --2- 172.21.15.44:0/1877003295 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f03d40a12c0 0x7f03d40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.694+0000 7f03deffd700 1 -- 172.21.15.44:0/1877003295 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f03d80cdb90 con 0x7f03e8101660 2023-11-16T16:51:56.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.694+0000 7f03edb9b700 1 --2- 172.21.15.44:0/1877003295 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f03d40a12c0 0x7f03d40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.695+0000 7f03edb9b700 1 --2- 172.21.15.44:0/1877003295 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f03d40a12c0 0x7f03d40a3770 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f03e4011550 tx=0x7f03e4009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:56.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.762+0000 7f04036acb80 1 Processor -- start 2023-11-16T16:51:56.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.762+0000 7f04036acb80 1 -- start start 2023-11-16T16:51:56.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.762+0000 7f04036acb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefbff0 0x559b3fefc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.762+0000 7f04036acb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3fefc900 0x559b3ff00800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.762+0000 7f04036acb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3ff00d40 0x559b3ff01190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.762+0000 7f04036acb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b3fdfd0e0 con 0x559b3fefbff0 2023-11-16T16:51:56.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.762+0000 7f04036acb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b3fe4f920 con 0x559b3fefc900 2023-11-16T16:51:56.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.762+0000 7f04036acb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b3fe48db0 con 0x559b3ff00d40 2023-11-16T16:51:56.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.763+0000 7f03edb9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefbff0 0x559b3fefc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.763+0000 7f03ed39a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3fefc900 0x559b3ff00800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.763+0000 7f03ed39a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3fefc900 0x559b3ff00800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59632/0 (socket says 172.21.15.44:59632) 2023-11-16T16:51:56.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.763+0000 7f03ed39a700 1 -- 172.21.15.44:0/3077721592 learned_addr learned my addr 172.21.15.44:0/3077721592 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:56.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.763+0000 7f03edb9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefbff0 0x559b3fefc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60238/0 (socket says 172.21.15.44:60238) 2023-11-16T16:51:56.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.763+0000 7f03ee39c700 1 --2- 172.21.15.44:0/3077721592 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3ff00d40 0x559b3ff01190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.763+0000 7f03ed39a700 1 -- 172.21.15.44:0/3077721592 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3ff00d40 msgr2=0x559b3ff01190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:56.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.763+0000 7f03ed39a700 1 --2- 172.21.15.44:0/3077721592 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3ff00d40 0x559b3ff01190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.763+0000 7f03ed39a700 1 -- 172.21.15.44:0/3077721592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefbff0 msgr2=0x559b3fefc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:56.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.763+0000 7f03ed39a700 1 --2- 172.21.15.44:0/3077721592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefbff0 0x559b3fefc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.763+0000 7f03ed39a700 1 -- 172.21.15.44:0/3077721592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b3fcca6b0 con 0x559b3fefc900 2023-11-16T16:51:56.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.763+0000 7f03ed39a700 1 --2- 172.21.15.44:0/3077721592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3fefc900 0x559b3ff00800 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f03d8097380 tx=0x7f03d809a360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3aa34500ff8234bc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:56.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.764+0000 7f03d3fff700 1 -- 172.21.15.44:0/3077721592 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d80a2cd0 con 0x559b3fefc900 2023-11-16T16:51:56.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.764+0000 7f03d3fff700 1 -- 172.21.15.44:0/3077721592 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03d8098cb0 con 0x559b3fefc900 2023-11-16T16:51:56.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.764+0000 7f03d3fff700 1 -- 172.21.15.44:0/3077721592 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d8028bf0 con 0x559b3fefc900 2023-11-16T16:51:56.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.764+0000 7f04036acb80 1 -- 172.21.15.44:0/3077721592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3fefc900 msgr2=0x559b3ff00800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:56.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.764+0000 7f04036acb80 1 --2- 172.21.15.44:0/3077721592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3fefc900 0x559b3ff00800 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f03d8097380 tx=0x7f03d809a360 comp rx=0 tx=0).stop 2023-11-16T16:51:56.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.764+0000 7f04036acb80 1 -- 172.21.15.44:0/3077721592 shutdown_connections 2023-11-16T16:51:56.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.764+0000 7f04036acb80 1 --2- 172.21.15.44:0/3077721592 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3ff00d40 0x559b3ff01190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.764+0000 7f04036acb80 1 --2- 172.21.15.44:0/3077721592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefbff0 0x559b3fefc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.764+0000 7f04036acb80 1 --2- 172.21.15.44:0/3077721592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3fefc900 0x559b3ff00800 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.765+0000 7f04036acb80 1 -- 172.21.15.44:0/3077721592 >> 172.21.15.44:0/3077721592 conn(0x559b3feed3e0 msgr2=0x559b3fef2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:56.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.765+0000 7f04036acb80 1 -- 172.21.15.44:0/3077721592 shutdown_connections 2023-11-16T16:51:56.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.765+0000 7f04036acb80 1 -- 172.21.15.44:0/3077721592 wait complete. 2023-11-16T16:51:56.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.765+0000 7f04036acb80 1 -- start start 2023-11-16T16:51:56.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.765+0000 7f04036acb80 1 Processor -- start 2023-11-16T16:51:56.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.766+0000 7f04036acb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3fefbff0 0x559b3ff083f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.766+0000 7f04036acb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefc900 0x559b3ff05670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.766+0000 7f04036acb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3ff00d40 0x559b3ff05c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.766+0000 7f04036acb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b3fdea960 con 0x559b3fefc900 2023-11-16T16:51:56.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.766+0000 7f04036acb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b3fdea600 con 0x559b3ff00d40 2023-11-16T16:51:56.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.766+0000 7f04036acb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b3fccc9a0 con 0x559b3fefbff0 2023-11-16T16:51:56.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.766+0000 7f04036acb80 10 client.0 _ll_register_callbacks cb 0x559b3fe08800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:56.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.766+0000 7f03ed39a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefc900 0x559b3ff05670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.766+0000 7f03edb9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3fefbff0 0x559b3ff083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.766+0000 7f03ed39a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefc900 0x559b3ff05670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60246/0 (socket says 172.21.15.44:60246) 2023-11-16T16:51:56.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.766+0000 7f03ed39a700 1 -- 172.21.15.44:0/1847981293 learned_addr learned my addr 172.21.15.44:0/1847981293 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:56.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.766+0000 7f03edb9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3fefbff0 0x559b3ff083f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:39492/0 (socket says 172.21.15.44:39492) 2023-11-16T16:51:56.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.766+0000 7f03ee39c700 1 --2- 172.21.15.44:0/1847981293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3ff00d40 0x559b3ff05c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.767+0000 7f03ed39a700 1 -- 172.21.15.44:0/1847981293 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3fefbff0 msgr2=0x559b3ff083f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:56.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.767+0000 7f03ed39a700 1 --2- 172.21.15.44:0/1847981293 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3fefbff0 0x559b3ff083f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.767+0000 7f03ed39a700 1 -- 172.21.15.44:0/1847981293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3ff00d40 msgr2=0x559b3ff05c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:56.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.767+0000 7f03ed39a700 1 --2- 172.21.15.44:0/1847981293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3ff00d40 0x559b3ff05c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.767+0000 7f03ed39a700 1 -- 172.21.15.44:0/1847981293 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e81101e0 con 0x559b3fefc900 2023-11-16T16:51:56.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.767+0000 7f03edb9b700 1 --2- 172.21.15.44:0/1847981293 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3fefbff0 0x559b3ff083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:56.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.767+0000 7f03ed39a700 1 --2- 172.21.15.44:0/1847981293 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefc900 0x559b3ff05670 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f03d80a27a0 tx=0x7f03d809aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb347883f6d92f5f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:56.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.767+0000 7f03d17fa700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:51:56.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.767+0000 7f03d17fa700 1 -- 172.21.15.44:0/1847981293 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d809a000 con 0x559b3fefc900 2023-11-16T16:51:56.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.767+0000 7f03d17fa700 1 -- 172.21.15.44:0/1847981293 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f03d80a2cd0 con 0x559b3fefc900 2023-11-16T16:51:56.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.767+0000 7f04036acb80 1 -- 172.21.15.44:0/1847981293 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559b3feeaba0 con 0x559b3fefc900 2023-11-16T16:51:56.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.767+0000 7f03d17fa700 1 -- 172.21.15.44:0/1847981293 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03d80a2840 con 0x559b3fefc900 2023-11-16T16:51:56.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.767+0000 7f04036acb80 10 client.15966 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:56.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.767+0000 7f04036acb80 1 -- 172.21.15.44:0/1847981293 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x559b3fe0f210 con 0x559b3fefc900 2023-11-16T16:51:56.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.768+0000 7f03d17fa700 1 -- 172.21.15.44:0/1847981293 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559b3feeaba0 con 0x559b3fefc900 2023-11-16T16:51:56.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.768+0000 7f03d17fa700 1 -- 172.21.15.44:0/1847981293 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f03d800f9e0 con 0x559b3fefc900 2023-11-16T16:51:56.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.768+0000 7f04036acb80 10 client.15966 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:56.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.768+0000 7f04036acb80 10 client.15966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:56.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.768+0000 7f04036acb80 20 client.15966 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:56.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.768+0000 7f04036acb80 1 --2- 172.21.15.44:0/1847981293 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b3ff15570 0x559b3ff17950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.768+0000 7f04036acb80 4 client.15966 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:51:56.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.768+0000 7f04036acb80 1 -- 172.21.15.44:0/1847981293 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559b3fe0b120 con 0x559b3ff15570 2023-11-16T16:51:56.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.768+0000 7f03edb9b700 1 --2- 172.21.15.44:0/1847981293 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b3ff15570 0x559b3ff17950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.769+0000 7f03edb9b700 1 --2- 172.21.15.44:0/1847981293 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b3ff15570 0x559b3ff17950 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c8ee51fc08fb553 server_cookie=245b75014f4ffde1 in_seq=0 out_seq=0 2023-11-16T16:51:56.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.769+0000 7f03d17fa700 10 client.15966 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:56.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.769+0000 7f03d17fa700 1 -- 172.21.15.44:0/1847981293 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559b3fe0b120 con 0x559b3ff15570 2023-11-16T16:51:56.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.769+0000 7f03d17fa700 10 client.15966 handle_command_reply: tid=0 2023-11-16T16:51:56.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.770+0000 7f04036acb80 1 client.15966 shutdown 2023-11-16T16:51:56.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.770+0000 7f04036acb80 1 -- 172.21.15.44:0/1847981293 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefc900 msgr2=0x559b3ff05670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:56.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.770+0000 7f04036acb80 1 --2- 172.21.15.44:0/1847981293 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefc900 0x559b3ff05670 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f03d80a27a0 tx=0x7f03d809aa90 comp rx=0 tx=0).stop 2023-11-16T16:51:56.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.770+0000 7f04036acb80 1 -- 172.21.15.44:0/1847981293 shutdown_connections 2023-11-16T16:51:56.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.770+0000 7f04036acb80 1 --2- 172.21.15.44:0/1847981293 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b3ff15570 0x559b3ff17950 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.770+0000 7f04036acb80 1 --2- 172.21.15.44:0/1847981293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3ff00d40 0x559b3ff05c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.770+0000 7f04036acb80 1 --2- 172.21.15.44:0/1847981293 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3fefbff0 0x559b3ff083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.770+0000 7f03d17fa700 0 client.15966 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:56.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.770+0000 7f04036acb80 1 --2- 172.21.15.44:0/1847981293 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefc900 0x559b3ff05670 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.770+0000 7f04036acb80 1 -- 172.21.15.44:0/1847981293 >> 172.21.15.44:0/1847981293 conn(0x559b3feed3e0 msgr2=0x559b3ff021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:56.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.770+0000 7f04036acb80 1 -- 172.21.15.44:0/1847981293 shutdown_connections 2023-11-16T16:51:56.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.770+0000 7f04036acb80 1 -- 172.21.15.44:0/1847981293 wait complete. 2023-11-16T16:51:56.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.770+0000 7f04036acb80 20 client.15966 trim_cache size 0 max 16384 2023-11-16T16:51:56.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.788+0000 7f04036acb80 1 Processor -- start 2023-11-16T16:51:56.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.788+0000 7f04036acb80 1 -- start start 2023-11-16T16:51:56.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.788+0000 7f04036acb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefbff0 0x559b3fef2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.788+0000 7f04036acb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3fefc900 0x559b3ff11f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.788+0000 7f04036acb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3ff00d40 0x559b3ff07f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.788+0000 7f04036acb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b3fefce40 con 0x559b3fefbff0 2023-11-16T16:51:56.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.788+0000 7f04036acb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b3fefd050 con 0x559b3ff00d40 2023-11-16T16:51:56.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.788+0000 7f04036acb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b3feecf60 con 0x559b3fefc900 2023-11-16T16:51:56.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.789+0000 7f03edb9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefbff0 0x559b3fef2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.789+0000 7f03edb9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefbff0 0x559b3fef2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60262/0 (socket says 172.21.15.44:60262) 2023-11-16T16:51:56.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.789+0000 7f03edb9b700 1 -- 172.21.15.44:0/4286378567 learned_addr learned my addr 172.21.15.44:0/4286378567 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:56.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.789+0000 7f03ed39a700 1 --2- 172.21.15.44:0/4286378567 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3fefc900 0x559b3ff11f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.789+0000 7f03ee39c700 1 --2- 172.21.15.44:0/4286378567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3ff00d40 0x559b3ff07f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.789+0000 7f03edb9b700 1 -- 172.21.15.44:0/4286378567 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3fefc900 msgr2=0x559b3ff11f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:56.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.789+0000 7f03edb9b700 1 --2- 172.21.15.44:0/4286378567 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3fefc900 0x559b3ff11f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.789+0000 7f03edb9b700 1 -- 172.21.15.44:0/4286378567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3ff00d40 msgr2=0x559b3ff07f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:56.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.789+0000 7f03edb9b700 1 --2- 172.21.15.44:0/4286378567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3ff00d40 0x559b3ff07f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.789+0000 7f03edb9b700 1 -- 172.21.15.44:0/4286378567 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b3fef18a0 con 0x559b3fefbff0 2023-11-16T16:51:56.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.789+0000 7f03edb9b700 1 --2- 172.21.15.44:0/4286378567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefbff0 0x559b3fef2d30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f03e4009cf0 tx=0x7f03e4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d59018467816836e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:56.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.790+0000 7f03dcbff700 1 -- 172.21.15.44:0/4286378567 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e4022040 con 0x559b3fefbff0 2023-11-16T16:51:56.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.790+0000 7f03dcbff700 1 -- 172.21.15.44:0/4286378567 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03e4023040 con 0x559b3fefbff0 2023-11-16T16:51:56.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.790+0000 7f03dcbff700 1 -- 172.21.15.44:0/4286378567 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e401de80 con 0x559b3fefbff0 2023-11-16T16:51:56.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.790+0000 7f04036acb80 1 -- 172.21.15.44:0/4286378567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefbff0 msgr2=0x559b3fef2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:56.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.790+0000 7f04036acb80 1 --2- 172.21.15.44:0/4286378567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefbff0 0x559b3fef2d30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f03e4009cf0 tx=0x7f03e4014000 comp rx=0 tx=0).stop 2023-11-16T16:51:56.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.790+0000 7f04036acb80 1 -- 172.21.15.44:0/4286378567 shutdown_connections 2023-11-16T16:51:56.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.790+0000 7f04036acb80 1 --2- 172.21.15.44:0/4286378567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3ff00d40 0x559b3ff07f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.790+0000 7f04036acb80 1 --2- 172.21.15.44:0/4286378567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefbff0 0x559b3fef2d30 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.790+0000 7f04036acb80 1 --2- 172.21.15.44:0/4286378567 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3fefc900 0x559b3ff11f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.790+0000 7f04036acb80 1 -- 172.21.15.44:0/4286378567 >> 172.21.15.44:0/4286378567 conn(0x559b3feed3e0 msgr2=0x559b3fef2620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:56.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.790+0000 7f04036acb80 1 -- 172.21.15.44:0/4286378567 shutdown_connections 2023-11-16T16:51:56.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.790+0000 7f04036acb80 1 -- 172.21.15.44:0/4286378567 wait complete. 2023-11-16T16:51:56.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.791+0000 7f04036acb80 1 -- start start 2023-11-16T16:51:56.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.791+0000 7f04036acb80 1 Processor -- start 2023-11-16T16:51:56.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.792+0000 7f04036acb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3fefbff0 0x559b3ff0cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.792+0000 7f04036acb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefc900 0x559b3ff0b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.792+0000 7f04036acb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3ff00d40 0x559b3ff0b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.792+0000 7f04036acb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b3ff0c0b0 con 0x559b3fefc900 2023-11-16T16:51:56.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.792+0000 7f04036acb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b3ff693a0 con 0x559b3fefbff0 2023-11-16T16:51:56.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.792+0000 7f04036acb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b3ff694e0 con 0x559b3ff00d40 2023-11-16T16:51:56.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.792+0000 7f04036acb80 10 client.0 _ll_register_callbacks cb 0x7f03e4009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:56.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.792+0000 7f03edb9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3fefbff0 0x559b3ff0cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.792+0000 7f03ed39a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefc900 0x559b3ff0b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.792+0000 7f03edb9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3fefbff0 0x559b3ff0cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59648/0 (socket says 172.21.15.44:59648) 2023-11-16T16:51:56.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.792+0000 7f03edb9b700 1 -- 172.21.15.44:0/3986290743 learned_addr learned my addr 172.21.15.44:0/3986290743 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:51:56.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.792+0000 7f03ee39c700 1 --2- 172.21.15.44:0/3986290743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3ff00d40 0x559b3ff0b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.793+0000 7f03edb9b700 1 -- 172.21.15.44:0/3986290743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3ff00d40 msgr2=0x559b3ff0b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:56.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.793+0000 7f03edb9b700 1 --2- 172.21.15.44:0/3986290743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3ff00d40 0x559b3ff0b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.793+0000 7f03edb9b700 1 -- 172.21.15.44:0/3986290743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefc900 msgr2=0x559b3ff0b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:56.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.793+0000 7f03edb9b700 1 --2- 172.21.15.44:0/3986290743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefc900 0x559b3ff0b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.793+0000 7f03edb9b700 1 -- 172.21.15.44:0/3986290743 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e4017070 con 0x559b3fefbff0 2023-11-16T16:51:56.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.793+0000 7f03edb9b700 1 --2- 172.21.15.44:0/3986290743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3fefbff0 0x559b3ff0cde0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f03e401d410 tx=0x7f03e401d440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fcb3b108ffae34f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:56.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.793+0000 7f03d17fa700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:51:56.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.793+0000 7f03d17fa700 1 -- 172.21.15.44:0/3986290743 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e4022070 con 0x559b3fefbff0 2023-11-16T16:51:56.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.793+0000 7f04036acb80 1 -- 172.21.15.44:0/3986290743 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f03d8098cb0 con 0x559b3fefbff0 2023-11-16T16:51:56.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.793+0000 7f03d17fa700 1 -- 172.21.15.44:0/3986290743 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f03e401d7d0 con 0x559b3fefbff0 2023-11-16T16:51:56.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.794+0000 7f03d17fa700 1 -- 172.21.15.44:0/3986290743 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03e401de90 con 0x559b3fefbff0 2023-11-16T16:51:56.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.794+0000 7f04036acb80 10 client.25988 fetch_fsmap learned FSMap version 21 2023-11-16T16:51:56.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.794+0000 7f04036acb80 1 -- 172.21.15.44:0/3986290743 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x559b3fc76e60 con 0x559b3fefbff0 2023-11-16T16:51:56.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.794+0000 7f03d17fa700 1 -- 172.21.15.44:0/3986290743 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03d8098cb0 con 0x559b3fefbff0 2023-11-16T16:51:56.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.794+0000 7f03d17fa700 1 -- 172.21.15.44:0/3986290743 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f03e4004750 con 0x559b3fefbff0 2023-11-16T16:51:56.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.794+0000 7f04036acb80 10 client.25988 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:51:56.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.794+0000 7f04036acb80 10 client.25988 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:56.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.794+0000 7f04036acb80 20 client.25988 populate_metadata read hostname 'smithi044' 2023-11-16T16:51:56.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.794+0000 7f04036acb80 1 --2- 172.21.15.44:0/3986290743 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b3ff6ab00 0x559b3ff6cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:56.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.794+0000 7f04036acb80 4 client.25988 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:51:56.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.794+0000 7f04036acb80 1 -- 172.21.15.44:0/3986290743 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x559b3fe176e0 con 0x559b3ff6ab00 2023-11-16T16:51:56.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.794+0000 7f03ed39a700 1 --2- 172.21.15.44:0/3986290743 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b3ff6ab00 0x559b3ff6cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:56.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.795+0000 7f03ed39a700 1 --2- 172.21.15.44:0/3986290743 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b3ff6ab00 0x559b3ff6cf10 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=970a9a6496147506 server_cookie=c40a4a476eff2719 in_seq=0 out_seq=0 2023-11-16T16:51:56.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.795+0000 7f03d17fa700 10 client.25988 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:56.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.796+0000 7f03d17fa700 1 -- 172.21.15.44:0/3986290743 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x559b3fe176e0 con 0x559b3ff6ab00 2023-11-16T16:51:56.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.796+0000 7f03d17fa700 10 client.25988 handle_command_reply: tid=0 2023-11-16T16:51:56.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.796+0000 7f04036acb80 1 client.25988 shutdown 2023-11-16T16:51:56.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.796+0000 7f04036acb80 1 -- 172.21.15.44:0/3986290743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3fefbff0 msgr2=0x559b3ff0cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:56.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.796+0000 7f04036acb80 1 --2- 172.21.15.44:0/3986290743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3fefbff0 0x559b3ff0cde0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f03e401d410 tx=0x7f03e401d440 comp rx=0 tx=0).stop 2023-11-16T16:51:56.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.797+0000 7f04036acb80 1 -- 172.21.15.44:0/3986290743 shutdown_connections 2023-11-16T16:51:56.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.797+0000 7f04036acb80 1 --2- 172.21.15.44:0/3986290743 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b3ff6ab00 0x559b3ff6cf10 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.797+0000 7f04036acb80 1 --2- 172.21.15.44:0/3986290743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b3ff00d40 0x559b3ff0b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.797+0000 7f04036acb80 1 --2- 172.21.15.44:0/3986290743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b3fefbff0 0x559b3ff0cde0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.797+0000 7f04036acb80 1 --2- 172.21.15.44:0/3986290743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b3fefc900 0x559b3ff0b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.797+0000 7f03d17fa700 0 client.25988 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:51:56.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.797+0000 7f04036acb80 1 -- 172.21.15.44:0/3986290743 >> 172.21.15.44:0/3986290743 conn(0x559b3feed3e0 msgr2=0x559b3fefadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:56.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.797+0000 7f04036acb80 1 -- 172.21.15.44:0/3986290743 shutdown_connections 2023-11-16T16:51:56.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.797+0000 7f04036acb80 1 -- 172.21.15.44:0/3986290743 wait complete. 2023-11-16T16:51:56.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.797+0000 7f04036acb80 20 client.25988 trim_cache size 0 max 16384 2023-11-16T16:51:56.801 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:51:56.801 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:51:56.801 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "060ecb0d-1fb2-4a5c-8a35-28be1d3abb8c", 2023-11-16T16:51:56.801 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:51:56.801 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:51:56.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.797+0000 7f03efdff700 1 -- 172.21.15.44:0/1877003295 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f03d40a12c0 msgr2=0x7f03d40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:56.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.797+0000 7f03efdff700 1 --2- 172.21.15.44:0/1877003295 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f03d40a12c0 0x7f03d40a3770 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f03e4011550 tx=0x7f03e4009260 comp rx=0 tx=0).stop 2023-11-16T16:51:56.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.797+0000 7f03efdff700 1 -- 172.21.15.44:0/1877003295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03e8101660 msgr2=0x7f03e81969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:56.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.797+0000 7f03efdff700 1 --2- 172.21.15.44:0/1877003295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03e8101660 0x7f03e81969f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f03e806a350 tx=0x7f03d8009580 comp rx=0 tx=0).stop 2023-11-16T16:51:56.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.798+0000 7f03efdff700 1 -- 172.21.15.44:0/1877003295 shutdown_connections 2023-11-16T16:51:56.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.798+0000 7f03efdff700 1 --2- 172.21.15.44:0/1877003295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03e8105620 0x7f03e819ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.798+0000 7f03efdff700 1 --2- 172.21.15.44:0/1877003295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e8100cc0 0x7f03e81964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.798+0000 7f03efdff700 1 --2- 172.21.15.44:0/1877003295 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f03d40a12c0 0x7f03d40a3770 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.798+0000 7f03efdff700 1 --2- 172.21.15.44:0/1877003295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03e8101660 0x7f03e81969f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:56.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.798+0000 7f03efdff700 1 -- 172.21.15.44:0/1877003295 >> 172.21.15.44:0/1877003295 conn(0x7f03e8077560 msgr2=0x7f03e80ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:56.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.798+0000 7f03efdff700 1 -- 172.21.15.44:0/1877003295 shutdown_connections 2023-11-16T16:51:56.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:51:56.798+0000 7f03efdff700 1 -- 172.21.15.44:0/1877003295 wait complete. 2023-11-16T16:51:56.860 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:51:57.164 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:57.165 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:57.234 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:57.234 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:57.274 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:57.274 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:57.274 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 45 -lt 100 ']' 2023-11-16T16:51:57.274 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 45 + 1 2023-11-16T16:51:57.330 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=46 2023-11-16T16:51:57.330 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:57.337 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:57.338 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:57.340 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:57.341 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:57.341 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:57.605 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:57 smithi044 ceph-mon[128750]: pgmap v63: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-11-16T16:51:57.605 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:57 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:51:57.605 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:57 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:51:57.605 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:57 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:51:57.605 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:57 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:51:57.605 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:57 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:51:57.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:57 smithi176 ceph-mon[131630]: pgmap v63: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-11-16T16:51:57.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:57 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:51:57.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:57 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:51:57.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:57 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:51:57.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:57 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:51:57.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:57 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:51:57.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:57 smithi077 ceph-mon[131204]: pgmap v63: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-11-16T16:51:57.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:57 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:51:57.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:57 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:51:57.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:57 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:51:57.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:57 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:51:57.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:57 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:51:57.789 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:57.789 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:57.923 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:57.924 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:57.941 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:57.941 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:57.941 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 46 -lt 100 ']' 2023-11-16T16:51:57.941 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 46 + 1 2023-11-16T16:51:57.971 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=47 2023-11-16T16:51:57.971 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:58.034 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:58.035 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:58.037 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:58.038 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:58.038 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:58.631 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:58.631 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:58.834 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:51:58.834 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:51:58.890 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:58.890 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:58.890 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 47 -lt 100 ']' 2023-11-16T16:51:58.891 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 47 + 1 2023-11-16T16:51:58.921 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=48 2023-11-16T16:51:58.921 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:51:58.981 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:58.981 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:58.984 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:51:58.984 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:51:58.984 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:59.567 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:51:59 smithi044 ceph-mon[128750]: pgmap v64: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 10 op/s 2023-11-16T16:51:59.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:51:59 smithi176 ceph-mon[131630]: pgmap v64: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 10 op/s 2023-11-16T16:51:59.722 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:51:59.723 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:51:59.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:51:59 smithi077 ceph-mon[131204]: pgmap v64: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 10 op/s 2023-11-16T16:52:00.268 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:00.268 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:00.300 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:00.300 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:00.301 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 48 -lt 100 ']' 2023-11-16T16:52:00.301 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 48 + 1 2023-11-16T16:52:00.413 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=49 2023-11-16T16:52:00.413 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:00.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.543+0000 7f40c99ff700 1 -- 172.21.15.44:0/1945789944 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c41063f0 msgr2=0x7f40c41067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:00.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.543+0000 7f40c99ff700 1 --2- 172.21.15.44:0/1945789944 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c41063f0 0x7f40c41067c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009990 tx=0x7f40b4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.543+0000 7f40c99ff700 1 -- 172.21.15.44:0/1945789944 shutdown_connections 2023-11-16T16:52:00.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.543+0000 7f40c99ff700 1 --2- 172.21.15.44:0/1945789944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40c4101400 0x7f40c410d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.543+0000 7f40c99ff700 1 --2- 172.21.15.44:0/1945789944 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c41063f0 0x7f40c41067c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.543+0000 7f40c99ff700 1 --2- 172.21.15.44:0/1945789944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40c4100a50 0x7f40c4100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.543+0000 7f40c99ff700 1 -- 172.21.15.44:0/1945789944 >> 172.21.15.44:0/1945789944 conn(0x7f40c4077590 msgr2=0x7f40c4077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:00.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.543+0000 7f40c99ff700 1 -- 172.21.15.44:0/1945789944 shutdown_connections 2023-11-16T16:52:00.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.543+0000 7f40c99ff700 1 -- 172.21.15.44:0/1945789944 wait complete. 2023-11-16T16:52:00.547 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.544+0000 7f40c99ff700 1 Processor -- start 2023-11-16T16:52:00.547 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.544+0000 7f40c99ff700 1 -- start start 2023-11-16T16:52:00.547 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.545+0000 7f40c99ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c4100a50 0x7f40c4198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.547 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.545+0000 7f40c99ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40c4101400 0x7f40c4198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.547 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.545+0000 7f40c99ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40c41063f0 0x7f40c419cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.547 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.545+0000 7f40c99ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c419d630 con 0x7f40c4100a50 2023-11-16T16:52:00.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.545+0000 7f40c99ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c419d7a0 con 0x7f40c41063f0 2023-11-16T16:52:00.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.545+0000 7f40c99ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c419d960 con 0x7f40c4101400 2023-11-16T16:52:00.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.545+0000 7f40bbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40c4101400 0x7f40c4198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.545+0000 7f40c2ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c4100a50 0x7f40c4198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.545+0000 7f40bbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40c4101400 0x7f40c4198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:39534/0 (socket says 172.21.15.44:39534) 2023-11-16T16:52:00.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.545+0000 7f40bbfff700 1 -- 172.21.15.44:0/56572725 learned_addr learned my addr 172.21.15.44:0/56572725 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:00.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.545+0000 7f40c37fe700 1 --2- 172.21.15.44:0/56572725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40c41063f0 0x7f40c419cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.545+0000 7f40bbfff700 1 -- 172.21.15.44:0/56572725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40c41063f0 msgr2=0x7f40c419cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:00.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.545+0000 7f40bbfff700 1 --2- 172.21.15.44:0/56572725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40c41063f0 0x7f40c419cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.545+0000 7f40bbfff700 1 -- 172.21.15.44:0/56572725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c4100a50 msgr2=0x7f40c4198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:00.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.545+0000 7f40bbfff700 1 --2- 172.21.15.44:0/56572725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c4100a50 0x7f40c4198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.545+0000 7f40bbfff700 1 -- 172.21.15.44:0/56572725 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b40095f0 con 0x7f40c4101400 2023-11-16T16:52:00.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.546+0000 7f40bbfff700 1 --2- 172.21.15.44:0/56572725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40c4101400 0x7f40c4198b90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00e9a0 tx=0x7f40ac00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:00.549 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.546+0000 7f40c89fd700 1 -- 172.21.15.44:0/56572725 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac009790 con 0x7f40c4101400 2023-11-16T16:52:00.549 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.546+0000 7f40c89fd700 1 -- 172.21.15.44:0/56572725 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40ac010430 con 0x7f40c4101400 2023-11-16T16:52:00.549 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.546+0000 7f40c99ff700 1 -- 172.21.15.44:0/56572725 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c4110380 con 0x7f40c4101400 2023-11-16T16:52:00.550 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.546+0000 7f40c89fd700 1 -- 172.21.15.44:0/56572725 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac018680 con 0x7f40c4101400 2023-11-16T16:52:00.550 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.546+0000 7f40c99ff700 1 -- 172.21.15.44:0/56572725 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c41101f0 con 0x7f40c4101400 2023-11-16T16:52:00.551 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.549+0000 7f40c89fd700 1 -- 172.21.15.44:0/56572725 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f40ac0105a0 con 0x7f40c4101400 2023-11-16T16:52:00.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.551+0000 7f40c89fd700 1 --2- 172.21.15.44:0/56572725 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f40b00a12c0 0x7f40b00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.551+0000 7f40c89fd700 1 -- 172.21.15.44:0/56572725 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f40ac014070 con 0x7f40c4101400 2023-11-16T16:52:00.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.552+0000 7f40c2ffd700 1 --2- 172.21.15.44:0/56572725 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f40b00a12c0 0x7f40b00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.554 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.552+0000 7f40c2ffd700 1 --2- 172.21.15.44:0/56572725 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f40b00a12c0 0x7f40b00a3770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f40b400b2e0 tx=0x7f40b4005960 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:00.555 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:00.555 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:00.558 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:00.558 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:00.559 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.628+0000 7f40dd3dbb80 1 Processor -- start 2023-11-16T16:52:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.628+0000 7f40dd3dbb80 1 -- start start 2023-11-16T16:52:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.629+0000 7f40dd3dbb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2ab60 0x56286ab2af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.629+0000 7f40dd3dbb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2b470 0x56286ab2f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.629+0000 7f40dd3dbb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2f8b0 0x56286ab2fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.629+0000 7f40dd3dbb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56286aa30060 con 0x56286ab2ab60 2023-11-16T16:52:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.629+0000 7f40dd3dbb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56286aa828a0 con 0x56286ab2b470 2023-11-16T16:52:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.629+0000 7f40dd3dbb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56286aa7bd30 con 0x56286ab2f8b0 2023-11-16T16:52:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.629+0000 7f40c2ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2ab60 0x56286ab2af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.629+0000 7f40c2ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2ab60 0x56286ab2af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60292/0 (socket says 172.21.15.44:60292) 2023-11-16T16:52:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.629+0000 7f40c37fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2f8b0 0x56286ab2fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.629+0000 7f40bbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2b470 0x56286ab2f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.629+0000 7f40c2ffd700 1 -- 172.21.15.44:0/3844776517 learned_addr learned my addr 172.21.15.44:0/3844776517 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.629+0000 7f40c2ffd700 1 -- 172.21.15.44:0/3844776517 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2f8b0 msgr2=0x56286ab2fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.629+0000 7f40c2ffd700 1 --2- 172.21.15.44:0/3844776517 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2f8b0 0x56286ab2fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.629+0000 7f40c2ffd700 1 -- 172.21.15.44:0/3844776517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2b470 msgr2=0x56286ab2f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.629+0000 7f40c2ffd700 1 --2- 172.21.15.44:0/3844776517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2b470 0x56286ab2f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.630+0000 7f40c2ffd700 1 -- 172.21.15.44:0/3844776517 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56286a8fd630 con 0x56286ab2ab60 2023-11-16T16:52:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.630+0000 7f40bbfff700 1 --2- 172.21.15.44:0/3844776517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2b470 0x56286ab2f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.630+0000 7f40c2ffd700 1 --2- 172.21.15.44:0/3844776517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2ab60 0x56286ab2af30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f40b400b8c0 tx=0x7f40b4020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a180c58f9598a3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.630+0000 7f40b9ffb700 1 -- 172.21.15.44:0/3844776517 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b402b070 con 0x56286ab2ab60 2023-11-16T16:52:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.630+0000 7f40b9ffb700 1 -- 172.21.15.44:0/3844776517 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40b401e040 con 0x56286ab2ab60 2023-11-16T16:52:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.630+0000 7f40b9ffb700 1 -- 172.21.15.44:0/3844776517 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b400f660 con 0x56286ab2ab60 2023-11-16T16:52:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.631+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/3844776517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2ab60 msgr2=0x56286ab2af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.631+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/3844776517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2ab60 0x56286ab2af30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f40b400b8c0 tx=0x7f40b4020000 comp rx=0 tx=0).stop 2023-11-16T16:52:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.631+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/3844776517 shutdown_connections 2023-11-16T16:52:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.631+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/3844776517 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2f8b0 0x56286ab2fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.631+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/3844776517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2ab60 0x56286ab2af30 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.631+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/3844776517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2b470 0x56286ab2f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.631+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/3844776517 >> 172.21.15.44:0/3844776517 conn(0x56286ab20360 msgr2=0x56286ab25b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.631+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/3844776517 shutdown_connections 2023-11-16T16:52:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.631+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/3844776517 wait complete. 2023-11-16T16:52:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.632+0000 7f40dd3dbb80 1 -- start start 2023-11-16T16:52:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.632+0000 7f40dd3dbb80 1 Processor -- start 2023-11-16T16:52:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.632+0000 7f40dd3dbb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2ab60 0x56286ab36f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.632+0000 7f40dd3dbb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2b470 0x56286ab341f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.632+0000 7f40dd3dbb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2f8b0 0x56286ab34790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.632+0000 7f40dd3dbb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56286aa1d8e0 con 0x56286ab2f8b0 2023-11-16T16:52:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.632+0000 7f40dd3dbb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56286aa1d580 con 0x56286ab2ab60 2023-11-16T16:52:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.632+0000 7f40dd3dbb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56286a8ff920 con 0x56286ab2b470 2023-11-16T16:52:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.632+0000 7f40dd3dbb80 10 client.0 _ll_register_callbacks cb 0x56286aa3b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.633+0000 7f40bbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2b470 0x56286ab341f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.633+0000 7f40c37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2f8b0 0x56286ab34790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.633+0000 7f40bbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2b470 0x56286ab341f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:39550/0 (socket says 172.21.15.44:39550) 2023-11-16T16:52:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.633+0000 7f40bbfff700 1 -- 172.21.15.44:0/1302331568 learned_addr learned my addr 172.21.15.44:0/1302331568 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.633+0000 7f40c2ffd700 1 --2- 172.21.15.44:0/1302331568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2ab60 0x56286ab36f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.633+0000 7f40c2ffd700 1 -- 172.21.15.44:0/1302331568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2b470 msgr2=0x56286ab341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.633+0000 7f40c2ffd700 1 --2- 172.21.15.44:0/1302331568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2b470 0x56286ab341f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.633+0000 7f40c2ffd700 1 -- 172.21.15.44:0/1302331568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2f8b0 msgr2=0x56286ab34790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.633+0000 7f40c2ffd700 1 --2- 172.21.15.44:0/1302331568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2f8b0 0x56286ab34790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.633+0000 7f40c2ffd700 1 -- 172.21.15.44:0/1302331568 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c410da50 con 0x56286ab2ab60 2023-11-16T16:52:00.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.633+0000 7f40bbfff700 1 --2- 172.21.15.44:0/1302331568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2b470 0x56286ab341f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:00.638 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.633+0000 7f40c37fe700 1 --2- 172.21.15.44:0/1302331568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2f8b0 0x56286ab34790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.633+0000 7f40c2ffd700 1 --2- 172.21.15.44:0/1302331568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2ab60 0x56286ab36f70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f40b4005f50 tx=0x7f40b400bcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a92d2e6605d05635 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.633+0000 7f40a2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:52:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.634+0000 7f40a2ffd700 1 -- 172.21.15.44:0/1302331568 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b402b040 con 0x56286ab2ab60 2023-11-16T16:52:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.634+0000 7f40a2ffd700 1 -- 172.21.15.44:0/1302331568 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f40b400f920 con 0x56286ab2ab60 2023-11-16T16:52:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.634+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/1302331568 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56286ab1db20 con 0x56286ab2ab60 2023-11-16T16:52:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.634+0000 7f40a2ffd700 1 -- 172.21.15.44:0/1302331568 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40b4004910 con 0x56286ab2ab60 2023-11-16T16:52:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.634+0000 7f40dd3dbb80 10 client.26000 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.634+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/1302331568 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x56286aa42190 con 0x56286ab2ab60 2023-11-16T16:52:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.634+0000 7f40a2ffd700 1 -- 172.21.15.44:0/1302331568 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56286ab1db20 con 0x56286ab2ab60 2023-11-16T16:52:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.634+0000 7f40a2ffd700 1 -- 172.21.15.44:0/1302331568 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f40b402f400 con 0x56286ab2ab60 2023-11-16T16:52:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.635+0000 7f40dd3dbb80 10 client.26000 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.635+0000 7f40dd3dbb80 10 client.26000 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.635+0000 7f40dd3dbb80 20 client.26000 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.635+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/1302331568 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56286ab44320 0x56286ab46700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.635+0000 7f40dd3dbb80 4 client.26000 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.635+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/1302331568 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56286aa3e0a0 con 0x56286ab44320 2023-11-16T16:52:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.635+0000 7f40bbfff700 1 --2- 172.21.15.44:0/1302331568 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56286ab44320 0x56286ab46700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.635+0000 7f40bbfff700 1 --2- 172.21.15.44:0/1302331568 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56286ab44320 0x56286ab46700 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f162020ced68e14 server_cookie=47191194613f204e in_seq=0 out_seq=0 2023-11-16T16:52:00.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.635+0000 7f40a2ffd700 10 client.26000 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:00.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.636+0000 7f40a2ffd700 1 -- 172.21.15.44:0/1302331568 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56286aa3e0a0 con 0x56286ab44320 2023-11-16T16:52:00.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.636+0000 7f40a2ffd700 10 client.26000 handle_command_reply: tid=0 2023-11-16T16:52:00.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.636+0000 7f40dd3dbb80 1 client.26000 shutdown 2023-11-16T16:52:00.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.636+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/1302331568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2ab60 msgr2=0x56286ab36f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:00.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.636+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/1302331568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2ab60 0x56286ab36f70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f40b4005f50 tx=0x7f40b400bcc0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.636+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/1302331568 shutdown_connections 2023-11-16T16:52:00.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.636+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/1302331568 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56286ab44320 0x56286ab46700 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.636+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/1302331568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2f8b0 0x56286ab34790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.636+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/1302331568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2ab60 0x56286ab36f70 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.637+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/1302331568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2b470 0x56286ab341f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.637+0000 7f40a2ffd700 0 client.26000 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:00.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.637+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/1302331568 >> 172.21.15.44:0/1302331568 conn(0x56286ab20360 msgr2=0x56286ab24de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:00.643 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.637+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/1302331568 shutdown_connections 2023-11-16T16:52:00.643 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.637+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/1302331568 wait complete. 2023-11-16T16:52:00.643 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.637+0000 7f40dd3dbb80 20 client.26000 trim_cache size 0 max 16384 2023-11-16T16:52:00.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.655+0000 7f40dd3dbb80 1 Processor -- start 2023-11-16T16:52:00.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.655+0000 7f40dd3dbb80 1 -- start start 2023-11-16T16:52:00.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.655+0000 7f40dd3dbb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2ab60 0x56286ab3b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.655+0000 7f40dd3dbb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2b470 0x56286ab2d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.655+0000 7f40dd3dbb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2f8b0 0x56286ab2d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.655+0000 7f40dd3dbb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56286ab1d9d0 con 0x56286ab2f8b0 2023-11-16T16:52:00.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.655+0000 7f40dd3dbb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56286ab2b840 con 0x56286ab2b470 2023-11-16T16:52:00.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.655+0000 7f40dd3dbb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56286ab20170 con 0x56286ab2ab60 2023-11-16T16:52:00.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.656+0000 7f40c2ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2ab60 0x56286ab3b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.656+0000 7f40c2ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2ab60 0x56286ab3b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:39564/0 (socket says 172.21.15.44:39564) 2023-11-16T16:52:00.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.656+0000 7f40c2ffd700 1 -- 172.21.15.44:0/2669225377 learned_addr learned my addr 172.21.15.44:0/2669225377 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:00.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.656+0000 7f40c37fe700 1 --2- 172.21.15.44:0/2669225377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2f8b0 0x56286ab2d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.656+0000 7f40bbfff700 1 --2- 172.21.15.44:0/2669225377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2b470 0x56286ab2d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.656+0000 7f40c2ffd700 1 -- 172.21.15.44:0/2669225377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2b470 msgr2=0x56286ab2d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:00.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.656+0000 7f40c2ffd700 1 --2- 172.21.15.44:0/2669225377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2b470 0x56286ab2d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.656+0000 7f40c2ffd700 1 -- 172.21.15.44:0/2669225377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2f8b0 msgr2=0x56286ab2d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:00.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.656+0000 7f40c2ffd700 1 --2- 172.21.15.44:0/2669225377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2f8b0 0x56286ab2d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.656+0000 7f40c2ffd700 1 -- 172.21.15.44:0/2669225377 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56286aa41ee0 con 0x56286ab2ab60 2023-11-16T16:52:00.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.656+0000 7f40bbfff700 1 --2- 172.21.15.44:0/2669225377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2b470 0x56286ab2d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:52:00.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.656+0000 7f40c2ffd700 1 --2- 172.21.15.44:0/2669225377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2ab60 0x56286ab3b940 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009990 tx=0x7f40b4009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7859783b9264f764 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:00.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.657+0000 7f40ba7fc700 1 -- 172.21.15.44:0/2669225377 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b402b070 con 0x56286ab2ab60 2023-11-16T16:52:00.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.657+0000 7f40ba7fc700 1 -- 172.21.15.44:0/2669225377 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40b4004bd0 con 0x56286ab2ab60 2023-11-16T16:52:00.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.657+0000 7f40ba7fc700 1 -- 172.21.15.44:0/2669225377 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b400fbc0 con 0x56286ab2ab60 2023-11-16T16:52:00.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.657+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/2669225377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2ab60 msgr2=0x56286ab3b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:00.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.657+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/2669225377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2ab60 0x56286ab3b940 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009990 tx=0x7f40b4009310 comp rx=0 tx=0).stop 2023-11-16T16:52:00.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.657+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/2669225377 shutdown_connections 2023-11-16T16:52:00.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.657+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/2669225377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2f8b0 0x56286ab2d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.657+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/2669225377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2ab60 0x56286ab3b940 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.657+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/2669225377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2b470 0x56286ab2d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.657+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/2669225377 >> 172.21.15.44:0/2669225377 conn(0x56286ab20360 msgr2=0x56286ab2e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:00.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.657+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/2669225377 shutdown_connections 2023-11-16T16:52:00.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.658+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/2669225377 wait complete. 2023-11-16T16:52:00.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.658+0000 7f40dd3dbb80 1 -- start start 2023-11-16T16:52:00.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.658+0000 7f40dd3dbb80 1 Processor -- start 2023-11-16T16:52:00.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40dd3dbb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2ab60 0x56286ab37920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40dd3dbb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2b470 0x56286ab37e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40dd3dbb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2f8b0 0x56286ab383a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40dd3dbb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56286ab97290 con 0x56286ab2b470 2023-11-16T16:52:00.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40dd3dbb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56286ab97400 con 0x56286ab2ab60 2023-11-16T16:52:00.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40dd3dbb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56286ab975c0 con 0x56286ab2f8b0 2023-11-16T16:52:00.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40dd3dbb80 10 client.0 _ll_register_callbacks cb 0x7f40bc009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:00.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40bbfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2b470 0x56286ab37e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40bbfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2b470 0x56286ab37e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60314/0 (socket says 172.21.15.44:60314) 2023-11-16T16:52:00.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40bbfff700 1 -- 172.21.15.44:0/1048043723 learned_addr learned my addr 172.21.15.44:0/1048043723 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:00.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40c37fe700 1 --2- 172.21.15.44:0/1048043723 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2f8b0 0x56286ab383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40c2ffd700 1 --2- 172.21.15.44:0/1048043723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2ab60 0x56286ab37920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.665 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40bbfff700 1 -- 172.21.15.44:0/1048043723 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2f8b0 msgr2=0x56286ab383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:00.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40bbfff700 1 --2- 172.21.15.44:0/1048043723 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2f8b0 0x56286ab383a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40bbfff700 1 -- 172.21.15.44:0/1048043723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2ab60 msgr2=0x56286ab37920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:00.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40bbfff700 1 --2- 172.21.15.44:0/1048043723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2ab60 0x56286ab37920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.659+0000 7f40bbfff700 1 -- 172.21.15.44:0/1048043723 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56286aa42190 con 0x56286ab2b470 2023-11-16T16:52:00.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.660+0000 7f40bbfff700 1 --2- 172.21.15.44:0/1048043723 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2b470 0x56286ab37e60 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f40ac099ca0 tx=0x7f40ac0cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3270c4ec499c763 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:00.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.660+0000 7f40a2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:52:00.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.660+0000 7f40a2ffd700 1 -- 172.21.15.44:0/1048043723 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac0a23f0 con 0x56286ab2b470 2023-11-16T16:52:00.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.660+0000 7f40a2ffd700 1 -- 172.21.15.44:0/1048043723 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f40ac0b8260 con 0x56286ab2b470 2023-11-16T16:52:00.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.660+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/1048043723 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f40b401e040 con 0x56286ab2b470 2023-11-16T16:52:00.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.660+0000 7f40a2ffd700 1 -- 172.21.15.44:0/1048043723 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40ac099490 con 0x56286ab2b470 2023-11-16T16:52:00.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.660+0000 7f40dd3dbb80 10 client.15990 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:00.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.660+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/1048043723 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x56286a892c60 con 0x56286ab2b470 2023-11-16T16:52:00.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.660+0000 7f40a2ffd700 1 -- 172.21.15.44:0/1048043723 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40ac063020 con 0x56286ab2b470 2023-11-16T16:52:00.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.661+0000 7f40a2ffd700 1 -- 172.21.15.44:0/1048043723 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f40ac0ca120 con 0x56286ab2b470 2023-11-16T16:52:00.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.661+0000 7f40dd3dbb80 10 client.15990 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:00.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.661+0000 7f40dd3dbb80 10 client.15990 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:00.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.661+0000 7f40dd3dbb80 20 client.15990 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:00.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.661+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/1048043723 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56286ab98d70 0x56286ab9b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:00.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.661+0000 7f40dd3dbb80 4 client.15990 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:52:00.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.661+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/1048043723 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56286aa4a660 con 0x56286ab98d70 2023-11-16T16:52:00.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.661+0000 7f40c2ffd700 1 --2- 172.21.15.44:0/1048043723 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56286ab98d70 0x56286ab9b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:00.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.661+0000 7f40c2ffd700 1 --2- 172.21.15.44:0/1048043723 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56286ab98d70 0x56286ab9b180 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a18c0a0d5bb84f7 server_cookie=a1e4170583b92b57 in_seq=0 out_seq=0 2023-11-16T16:52:00.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.661+0000 7f40a2ffd700 10 client.15990 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:00.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.662+0000 7f40a2ffd700 1 -- 172.21.15.44:0/1048043723 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56286aa4a660 con 0x56286ab98d70 2023-11-16T16:52:00.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.662+0000 7f40a2ffd700 10 client.15990 handle_command_reply: tid=0 2023-11-16T16:52:00.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.662+0000 7f40dd3dbb80 1 client.15990 shutdown 2023-11-16T16:52:00.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.662+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/1048043723 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2b470 msgr2=0x56286ab37e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:00.668 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.662+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/1048043723 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2b470 0x56286ab37e60 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f40ac099ca0 tx=0x7f40ac0cc040 comp rx=0 tx=0).stop 2023-11-16T16:52:00.668 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.662+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/1048043723 shutdown_connections 2023-11-16T16:52:00.668 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.662+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/1048043723 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56286ab98d70 0x56286ab9b180 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.668 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.662+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/1048043723 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56286ab2f8b0 0x56286ab383a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.668 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.662+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/1048043723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56286ab2ab60 0x56286ab37920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.668 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.662+0000 7f40a2ffd700 0 client.15990 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:00.668 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.662+0000 7f40dd3dbb80 1 --2- 172.21.15.44:0/1048043723 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56286ab2b470 0x56286ab37e60 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.668 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.662+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/1048043723 >> 172.21.15.44:0/1048043723 conn(0x56286ab20360 msgr2=0x56286ab29960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:00.668 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.662+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/1048043723 shutdown_connections 2023-11-16T16:52:00.668 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.662+0000 7f40dd3dbb80 1 -- 172.21.15.44:0/1048043723 wait complete. 2023-11-16T16:52:00.668 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.662+0000 7f40dd3dbb80 20 client.15990 trim_cache size 0 max 16384 2023-11-16T16:52:00.669 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:52:00.669 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T16:52:00.669 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T16:52:00.669 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:52:00.669 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.663+0000 7f40c99ff700 1 -- 172.21.15.44:0/56572725 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f40b00a12c0 msgr2=0x7f40b00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:00.669 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.663+0000 7f40c99ff700 1 --2- 172.21.15.44:0/56572725 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f40b00a12c0 0x7f40b00a3770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f40b400b2e0 tx=0x7f40b4005960 comp rx=0 tx=0).stop 2023-11-16T16:52:00.669 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.663+0000 7f40c99ff700 1 -- 172.21.15.44:0/56572725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40c4101400 msgr2=0x7f40c4198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:00.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.663+0000 7f40c99ff700 1 --2- 172.21.15.44:0/56572725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40c4101400 0x7f40c4198b90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00e9a0 tx=0x7f40ac00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.663+0000 7f40c99ff700 1 -- 172.21.15.44:0/56572725 shutdown_connections 2023-11-16T16:52:00.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.663+0000 7f40c99ff700 1 --2- 172.21.15.44:0/56572725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40c41063f0 0x7f40c419cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.663+0000 7f40c99ff700 1 --2- 172.21.15.44:0/56572725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c4100a50 0x7f40c4198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.663+0000 7f40c99ff700 1 --2- 172.21.15.44:0/56572725 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f40b00a12c0 0x7f40b00a3770 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.663+0000 7f40c99ff700 1 --2- 172.21.15.44:0/56572725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40c4101400 0x7f40c4198b90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:00.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.663+0000 7f40c99ff700 1 -- 172.21.15.44:0/56572725 >> 172.21.15.44:0/56572725 conn(0x7f40c4077590 msgr2=0x7f40c410b730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:00.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.663+0000 7f40c99ff700 1 -- 172.21.15.44:0/56572725 shutdown_connections 2023-11-16T16:52:00.670 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:00.663+0000 7f40c99ff700 1 -- 172.21.15.44:0/56572725 wait complete. 2023-11-16T16:52:00.717 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T16:52:00.718 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:52:01.197 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:01.197 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:01.290 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:01.290 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:01.298 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:01.298 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:01.299 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 49 -lt 100 ']' 2023-11-16T16:52:01.299 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 49 + 1 2023-11-16T16:52:01.338 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=50 2023-11-16T16:52:01.338 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:01.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:01 smithi044 ceph-mon[128750]: pgmap v65: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-11-16T16:52:01.381 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:01.381 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:01.383 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:01.384 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:01.384 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:01.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:01 smithi176 ceph-mon[131630]: pgmap v65: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-11-16T16:52:01.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:01 smithi077 ceph-mon[131204]: pgmap v65: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-11-16T16:52:01.980 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:01.981 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:02.176 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:02.176 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:02.224 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:02.225 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:02.225 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 50 -lt 100 ']' 2023-11-16T16:52:02.225 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 50 + 1 2023-11-16T16:52:02.329 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=51 2023-11-16T16:52:02.329 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:02.417 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:02.417 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:02.420 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:02.420 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:02.421 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:03.164 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:03.164 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:03.359 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:03.359 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:03.390 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:03.391 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:03.391 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 51 -lt 100 ']' 2023-11-16T16:52:03.391 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 51 + 1 2023-11-16T16:52:03.446 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=52 2023-11-16T16:52:03.446 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:03.503 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:03.503 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:03.505 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:03.506 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:03.506 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:03.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:03 smithi044 ceph-mon[128750]: pgmap v66: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 51 KiB/s wr, 23 op/s 2023-11-16T16:52:03.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:03 smithi176 ceph-mon[131630]: pgmap v66: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 51 KiB/s wr, 23 op/s 2023-11-16T16:52:03.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:03 smithi077 ceph-mon[131204]: pgmap v66: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 51 KiB/s wr, 23 op/s 2023-11-16T16:52:03.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.793+0000 7ff986bff700 1 -- 172.21.15.44:0/4168100991 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980100ef0 msgr2=0x7ff980104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:03.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.793+0000 7ff986bff700 1 --2- 172.21.15.44:0/4168100991 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980100ef0 0x7ff980104e60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff96c0099c0 tx=0x7ff96c009cd0 comp rx=0 tx=0).stop 2023-11-16T16:52:03.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.793+0000 7ff986bff700 1 -- 172.21.15.44:0/4168100991 shutdown_connections 2023-11-16T16:52:03.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.793+0000 7ff986bff700 1 --2- 172.21.15.44:0/4168100991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff980105540 0x7ff98010d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:03.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.793+0000 7ff986bff700 1 --2- 172.21.15.44:0/4168100991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff980100550 0x7ff980100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:03.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.793+0000 7ff986bff700 1 --2- 172.21.15.44:0/4168100991 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980100ef0 0x7ff980104e60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:03.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.793+0000 7ff986bff700 1 -- 172.21.15.44:0/4168100991 >> 172.21.15.44:0/4168100991 conn(0x7ff980077020 msgr2=0x7ff980077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:03.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.793+0000 7ff986bff700 1 -- 172.21.15.44:0/4168100991 shutdown_connections 2023-11-16T16:52:03.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.794+0000 7ff986bff700 1 -- 172.21.15.44:0/4168100991 wait complete. 2023-11-16T16:52:03.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.795+0000 7ff986bff700 1 Processor -- start 2023-11-16T16:52:03.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.795+0000 7ff986bff700 1 -- start start 2023-11-16T16:52:03.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.795+0000 7ff986bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980100550 0x7ff980198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:03.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.795+0000 7ff986bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff980100ef0 0x7ff980198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:03.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.795+0000 7ff986bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff980105540 0x7ff98019cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:03.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.795+0000 7ff986bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98019d580 con 0x7ff980100550 2023-11-16T16:52:03.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.795+0000 7ff986bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98019d6f0 con 0x7ff980100ef0 2023-11-16T16:52:03.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.795+0000 7ff986bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98019d8b0 con 0x7ff980105540 2023-11-16T16:52:03.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.795+0000 7ff98499b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980100550 0x7ff980198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:03.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.795+0000 7ff98499b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980100550 0x7ff980198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60332/0 (socket says 172.21.15.44:60332) 2023-11-16T16:52:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.795+0000 7ff98499b700 1 -- 172.21.15.44:0/4153950101 learned_addr learned my addr 172.21.15.44:0/4153950101 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.796+0000 7ff97ffff700 1 --2- 172.21.15.44:0/4153950101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff980100ef0 0x7ff980198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.796+0000 7ff98519c700 1 --2- 172.21.15.44:0/4153950101 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff980105540 0x7ff98019cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.796+0000 7ff98499b700 1 -- 172.21.15.44:0/4153950101 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff980105540 msgr2=0x7ff98019cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.796+0000 7ff98499b700 1 --2- 172.21.15.44:0/4153950101 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff980105540 0x7ff98019cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.796+0000 7ff98499b700 1 -- 172.21.15.44:0/4153950101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff980100ef0 msgr2=0x7ff980198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.796+0000 7ff98499b700 1 --2- 172.21.15.44:0/4153950101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff980100ef0 0x7ff980198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.796+0000 7ff98499b700 1 -- 172.21.15.44:0/4153950101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff96c0095f0 con 0x7ff980100550 2023-11-16T16:52:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.796+0000 7ff98499b700 1 --2- 172.21.15.44:0/4153950101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980100550 0x7ff980198610 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff97400e9a0 tx=0x7ff97400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.796+0000 7ff97dffb700 1 -- 172.21.15.44:0/4153950101 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff974009790 con 0x7ff980100550 2023-11-16T16:52:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.796+0000 7ff97dffb700 1 -- 172.21.15.44:0/4153950101 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff97400cbd0 con 0x7ff980100550 2023-11-16T16:52:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.796+0000 7ff97dffb700 1 -- 172.21.15.44:0/4153950101 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff974018680 con 0x7ff980100550 2023-11-16T16:52:03.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.797+0000 7ff986bff700 1 -- 172.21.15.44:0/4153950101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff980110310 con 0x7ff980100550 2023-11-16T16:52:03.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.797+0000 7ff986bff700 1 -- 172.21.15.44:0/4153950101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff980110180 con 0x7ff980100550 2023-11-16T16:52:03.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.798+0000 7ff986bff700 1 -- 172.21.15.44:0/4153950101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9801104a0 con 0x7ff980100550 2023-11-16T16:52:03.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.800+0000 7ff97dffb700 1 -- 172.21.15.44:0/4153950101 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7ff97400cd40 con 0x7ff980100550 2023-11-16T16:52:03.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.802+0000 7ff97dffb700 1 --2- 172.21.15.44:0/4153950101 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff9700a1340 0x7ff9700a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:03.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.802+0000 7ff97dffb700 1 -- 172.21.15.44:0/4153950101 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7ff974014070 con 0x7ff980100550 2023-11-16T16:52:03.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.802+0000 7ff97ffff700 1 --2- 172.21.15.44:0/4153950101 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff9700a1340 0x7ff9700a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:03.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.803+0000 7ff97ffff700 1 --2- 172.21.15.44:0/4153950101 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff9700a1340 0x7ff9700a37f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff96c0098e0 tx=0x7ff96c003680 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:03.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:03.820+0000 7ff97dffb700 1 -- 172.21.15.44:0/4153950101 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff974051730 con 0x7ff980100550 2023-11-16T16:52:04.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:04.079+0000 7ff986bff700 1 -- 172.21.15.44:0/4153950101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff98019da70 con 0x7ff980100550 2023-11-16T16:52:04.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:04.080+0000 7ff97dffb700 1 -- 172.21.15.44:0/4153950101 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7ff974096300 con 0x7ff980100550 2023-11-16T16:52:04.081 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:52:04.081 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:52:04.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:04.082+0000 7ff986bff700 1 -- 172.21.15.44:0/4153950101 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff9700a1340 msgr2=0x7ff9700a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:04.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:04.082+0000 7ff986bff700 1 --2- 172.21.15.44:0/4153950101 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff9700a1340 0x7ff9700a37f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff96c0098e0 tx=0x7ff96c003680 comp rx=0 tx=0).stop 2023-11-16T16:52:04.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:04.082+0000 7ff986bff700 1 -- 172.21.15.44:0/4153950101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980100550 msgr2=0x7ff980198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:04.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:04.082+0000 7ff986bff700 1 --2- 172.21.15.44:0/4153950101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980100550 0x7ff980198610 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff97400e9a0 tx=0x7ff97400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:52:04.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:04.083+0000 7ff986bff700 1 -- 172.21.15.44:0/4153950101 shutdown_connections 2023-11-16T16:52:04.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:04.083+0000 7ff986bff700 1 --2- 172.21.15.44:0/4153950101 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff9700a1340 0x7ff9700a37f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:04.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:04.083+0000 7ff986bff700 1 --2- 172.21.15.44:0/4153950101 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff980105540 0x7ff98019cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:04.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:04.083+0000 7ff986bff700 1 --2- 172.21.15.44:0/4153950101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980100550 0x7ff980198610 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:04.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:04.083+0000 7ff986bff700 1 --2- 172.21.15.44:0/4153950101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff980100ef0 0x7ff980198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:04.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:04.083+0000 7ff986bff700 1 -- 172.21.15.44:0/4153950101 >> 172.21.15.44:0/4153950101 conn(0x7ff980077020 msgr2=0x7ff9800fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:04.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:04.083+0000 7ff986bff700 1 -- 172.21.15.44:0/4153950101 shutdown_connections 2023-11-16T16:52:04.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:04.083+0000 7ff986bff700 1 -- 172.21.15.44:0/4153950101 wait complete. 2023-11-16T16:52:04.085 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:52:04.255 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:04.255 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:04.262 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T16:52:04.491 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:04.492 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:04.524 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:04.525 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:04.525 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 52 -lt 100 ']' 2023-11-16T16:52:04.525 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 52 + 1 2023-11-16T16:52:04.571 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=53 2023-11-16T16:52:04.571 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:04.584 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:04.585 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:04.587 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:04.588 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:04.588 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:04.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:04 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/4153950101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:52:04.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:04 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/4153950101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:52:04.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:04 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/4153950101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:52:05.147 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:05.147 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:05.296 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:05.296 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:05.365 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:05.365 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:05.365 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 53 -lt 100 ']' 2023-11-16T16:52:05.365 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 53 + 1 2023-11-16T16:52:05.395 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=54 2023-11-16T16:52:05.395 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:05.488 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:05.488 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:05.491 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:05.491 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:05.492 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:05.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:05 smithi044 ceph-mon[128750]: pgmap v67: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 36 KiB/s wr, 17 op/s 2023-11-16T16:52:05.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:05 smithi176 ceph-mon[131630]: pgmap v67: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 36 KiB/s wr, 17 op/s 2023-11-16T16:52:05.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:05 smithi077 ceph-mon[131204]: pgmap v67: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 36 KiB/s wr, 17 op/s 2023-11-16T16:52:06.214 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:06.215 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:06.375 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:06.375 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:06.393 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:06.393 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:06.393 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 54 -lt 100 ']' 2023-11-16T16:52:06.394 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 54 + 1 2023-11-16T16:52:06.445 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=55 2023-11-16T16:52:06.445 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:06.571 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:06.572 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:06.574 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:06.575 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:06.575 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:06.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.710+0000 7fbd9d1ff700 1 -- 172.21.15.44:0/2078315788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd98101390 msgr2=0x7fbd98101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.710+0000 7fbd9d1ff700 1 --2- 172.21.15.44:0/2078315788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd98101390 0x7fbd98101760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbd880098f0 tx=0x7fbd88009c00 comp rx=0 tx=0).stop 2023-11-16T16:52:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.710+0000 7fbd9d1ff700 1 -- 172.21.15.44:0/2078315788 shutdown_connections 2023-11-16T16:52:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.710+0000 7fbd9d1ff700 1 --2- 172.21.15.44:0/2078315788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd9810a8b0 0x7fbd9810ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.710+0000 7fbd9d1ff700 1 --2- 172.21.15.44:0/2078315788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd98101390 0x7fbd98101760 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.710+0000 7fbd9d1ff700 1 --2- 172.21.15.44:0/2078315788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd98101ca0 0x7fbd9810a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.710+0000 7fbd9d1ff700 1 -- 172.21.15.44:0/2078315788 >> 172.21.15.44:0/2078315788 conn(0x7fbd980770b0 msgr2=0x7fbd980774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.710+0000 7fbd9d1ff700 1 -- 172.21.15.44:0/2078315788 shutdown_connections 2023-11-16T16:52:06.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.710+0000 7fbd9d1ff700 1 -- 172.21.15.44:0/2078315788 wait complete. 2023-11-16T16:52:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.710+0000 7f532c5ff700 1 -- 172.21.15.44:0/2232414612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53240ff230 msgr2=0x7f53240ff600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.710+0000 7f532c5ff700 1 --2- 172.21.15.44:0/2232414612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53240ff230 0x7f53240ff600 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5320009990 tx=0x7f5320009ca0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.711+0000 7f532c5ff700 1 -- 172.21.15.44:0/2232414612 shutdown_connections 2023-11-16T16:52:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.711+0000 7f532c5ff700 1 --2- 172.21.15.44:0/2232414612 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5324104630 0x7f5324104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.711+0000 7f532c5ff700 1 --2- 172.21.15.44:0/2232414612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53240ff230 0x7f53240ff600 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.711+0000 7f532c5ff700 1 --2- 172.21.15.44:0/2232414612 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53240ffb40 0x7f5324104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.711+0000 7f532c5ff700 1 -- 172.21.15.44:0/2232414612 >> 172.21.15.44:0/2232414612 conn(0x7f53240763a0 msgr2=0x7f53240767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.711+0000 7f532c5ff700 1 -- 172.21.15.44:0/2232414612 shutdown_connections 2023-11-16T16:52:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.711+0000 7f532c5ff700 1 -- 172.21.15.44:0/2232414612 wait complete. 2023-11-16T16:52:06.715 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.711+0000 7fbd9d1ff700 1 Processor -- start 2023-11-16T16:52:06.715 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.712+0000 7fbd9d1ff700 1 -- start start 2023-11-16T16:52:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.712+0000 7f532c5ff700 1 Processor -- start 2023-11-16T16:52:06.716 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.712+0000 7fbd9d1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd98101390 0x7fbd980ff1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.712+0000 7fbd9d1ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd98101ca0 0x7fbd980ff700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.712+0000 7fbd9d1ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd9810a8b0 0x7fbd981a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.712+0000 7fbd9d1ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd980fff30 con 0x7fbd9810a8b0 2023-11-16T16:52:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.712+0000 7fbd9d1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd981a4ba0 con 0x7fbd98101390 2023-11-16T16:52:06.717 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.712+0000 7fbd9d1ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd981a4ce0 con 0x7fbd98101ca0 2023-11-16T16:52:06.718 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.712+0000 7f532c5ff700 1 -- start start 2023-11-16T16:52:06.718 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.712+0000 7fbd9759e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd9810a8b0 0x7fbd981a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.712+0000 7fbd96d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd98101390 0x7fbd980ff1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.712+0000 7fbd9659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd98101ca0 0x7fbd980ff700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.712+0000 7fbd9759e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd9810a8b0 0x7fbd981a4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57068/0 (socket says 172.21.15.44:57068) 2023-11-16T16:52:06.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.712+0000 7fbd9759e700 1 -- 172.21.15.44:0/2369863585 learned_addr learned my addr 172.21.15.44:0/2369863585 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.713+0000 7f532c5ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53240ff230 0x7f532419c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.713+0000 7fbd96d9d700 1 -- 172.21.15.44:0/2369863585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd98101ca0 msgr2=0x7fbd980ff700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.713+0000 7fbd96d9d700 1 --2- 172.21.15.44:0/2369863585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd98101ca0 0x7fbd980ff700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.713+0000 7fbd96d9d700 1 -- 172.21.15.44:0/2369863585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd9810a8b0 msgr2=0x7fbd981a4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.713+0000 7fbd96d9d700 1 --2- 172.21.15.44:0/2369863585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd9810a8b0 0x7fbd981a4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.713+0000 7fbd96d9d700 1 -- 172.21.15.44:0/2369863585 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd88009520 con 0x7fbd98101390 2023-11-16T16:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.713+0000 7f532c5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53240ffb40 0x7f532419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.713+0000 7f532c5ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5324104630 0x7f53241a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.713+0000 7f532c5ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53241a1940 con 0x7f53240ff230 2023-11-16T16:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.713+0000 7f532c5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53241a1ab0 con 0x7f53240ffb40 2023-11-16T16:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.713+0000 7f532c5ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53241a1c70 con 0x7f5324104630 2023-11-16T16:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.713+0000 7f532a39b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53240ff230 0x7f532419c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.713+0000 7f5329b9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53240ffb40 0x7f532419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.713+0000 7f5329b9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53240ffb40 0x7f532419cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:33496/0 (socket says 172.21.15.44:33496) 2023-11-16T16:52:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.713+0000 7f532a39b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53240ff230 0x7f532419c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57080/0 (socket says 172.21.15.44:57080) 2023-11-16T16:52:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.713+0000 7f5329b9a700 1 -- 172.21.15.44:0/1898209835 learned_addr learned my addr 172.21.15.44:0/1898209835 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.713+0000 7f532ab9c700 1 --2- 172.21.15.44:0/1898209835 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5324104630 0x7f53241a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.713+0000 7fbd96d9d700 1 --2- 172.21.15.44:0/2369863585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd98101390 0x7fbd980ff1c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009810 tx=0x7fbd8800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.714+0000 7f5329b9a700 1 -- 172.21.15.44:0/1898209835 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5324104630 msgr2=0x7f53241a1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.714+0000 7f5329b9a700 1 --2- 172.21.15.44:0/1898209835 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5324104630 0x7f53241a1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.714+0000 7f5329b9a700 1 -- 172.21.15.44:0/1898209835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53240ff230 msgr2=0x7f532419c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.714+0000 7f5329b9a700 1 --2- 172.21.15.44:0/1898209835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53240ff230 0x7f532419c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.714+0000 7f5329b9a700 1 -- 172.21.15.44:0/1898209835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53200095f0 con 0x7f53240ffb40 2023-11-16T16:52:06.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.714+0000 7fbd7ffff700 1 -- 172.21.15.44:0/2369863585 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8801c070 con 0x7fbd98101390 2023-11-16T16:52:06.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.714+0000 7fbd7ffff700 1 -- 172.21.15.44:0/2369863585 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd88005040 con 0x7fbd98101390 2023-11-16T16:52:06.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.714+0000 7fbd7ffff700 1 -- 172.21.15.44:0/2369863585 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd880038e0 con 0x7fbd98101390 2023-11-16T16:52:06.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.714+0000 7fbd9d1ff700 1 -- 172.21.15.44:0/2369863585 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd98110380 con 0x7fbd98101390 2023-11-16T16:52:06.726 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.714+0000 7fbd9d1ff700 1 -- 172.21.15.44:0/2369863585 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd981101f0 con 0x7fbd98101390 2023-11-16T16:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.714+0000 7f5329b9a700 1 --2- 172.21.15.44:0/1898209835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53240ffb40 0x7f532419cee0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f531400e9a0 tx=0x7f531400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.715+0000 7f531b7fe700 1 -- 172.21.15.44:0/1898209835 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5314009790 con 0x7f53240ffb40 2023-11-16T16:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.715+0000 7f531b7fe700 1 -- 172.21.15.44:0/1898209835 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f531400cbd0 con 0x7f53240ffb40 2023-11-16T16:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.715+0000 7f532c5ff700 1 -- 172.21.15.44:0/1898209835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53241168c0 con 0x7f53240ffb40 2023-11-16T16:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.715+0000 7f531b7fe700 1 -- 172.21.15.44:0/1898209835 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5314018680 con 0x7f53240ffb40 2023-11-16T16:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.715+0000 7f532c5ff700 1 -- 172.21.15.44:0/1898209835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5324116730 con 0x7f53240ffb40 2023-11-16T16:52:06.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.717+0000 7fbd7ffff700 1 -- 172.21.15.44:0/2369863585 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fbd88020030 con 0x7fbd98101390 2023-11-16T16:52:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.719+0000 7f531b7fe700 1 -- 172.21.15.44:0/1898209835 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f531400cd40 con 0x7f53240ffb40 2023-11-16T16:52:06.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.719+0000 7fbd7ffff700 1 --2- 172.21.15.44:0/2369863585 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbd800a1340 0x7fbd800a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.720+0000 7fbd7ffff700 1 -- 172.21.15.44:0/2369863585 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fbd880cdef0 con 0x7fbd98101390 2023-11-16T16:52:06.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.720+0000 7fbd9659c700 1 --2- 172.21.15.44:0/2369863585 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbd800a1340 0x7fbd800a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.720+0000 7f531b7fe700 1 --2- 172.21.15.44:0/1898209835 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f53100a1340 0x7f53100a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.720+0000 7fbd9659c700 1 --2- 172.21.15.44:0/2369863585 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbd800a1340 0x7fbd800a37f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbd981009b0 tx=0x7fbd8c009310 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.720+0000 7f531b7fe700 1 -- 172.21.15.44:0/1898209835 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f5314014070 con 0x7f53240ffb40 2023-11-16T16:52:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.720+0000 7f532a39b700 1 --2- 172.21.15.44:0/1898209835 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f53100a1340 0x7f53100a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.721+0000 7f532a39b700 1 --2- 172.21.15.44:0/1898209835 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f53100a1340 0x7f53100a37f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f532000b2e0 tx=0x7f5320005fb0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:06.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.836+0000 7f533ff70b80 1 Processor -- start 2023-11-16T16:52:06.837 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.836+0000 7fbdb0aa9b80 1 Processor -- start 2023-11-16T16:52:06.837 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.836+0000 7f533ff70b80 1 -- start start 2023-11-16T16:52:06.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.836+0000 7fbdb0aa9b80 1 -- start start 2023-11-16T16:52:06.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.836+0000 7f533ff70b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdab576b0 0x55ccdad011e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.836+0000 7fbdb0aa9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d1b60 0x55f9542d1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.836+0000 7f533ff70b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdad01720 0x55ccdacfb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.836+0000 7fbdb0aa9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d2470 0x55f9542d6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.837+0000 7fbdb0aa9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d68b0 0x55f9542d6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.837+0000 7fbdb0aa9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9541d7060 con 0x55f9542d1b60 2023-11-16T16:52:06.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.837+0000 7fbdb0aa9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9542298a0 con 0x55f9542d68b0 2023-11-16T16:52:06.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.837+0000 7fbdb0aa9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f954222d30 con 0x55f9542d2470 2023-11-16T16:52:06.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.837+0000 7f533ff70b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01b90 0x55ccdacfbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.837+0000 7f533ff70b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccdac01000 con 0x55ccdad01720 2023-11-16T16:52:06.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.837+0000 7f533ff70b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccdac53840 con 0x55ccdad01b90 2023-11-16T16:52:06.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.837+0000 7f533ff70b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccdac4ccd0 con 0x55ccdab576b0 2023-11-16T16:52:06.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.837+0000 7f5329b9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdad01720 0x55ccdacfb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.842 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.837+0000 7f532a39b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdab576b0 0x55ccdad011e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.842 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.837+0000 7f532a39b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdab576b0 0x55ccdad011e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:42180/0 (socket says 172.21.15.44:42180) 2023-11-16T16:52:06.842 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.837+0000 7f532a39b700 1 -- 172.21.15.44:0/432303732 learned_addr learned my addr 172.21.15.44:0/432303732 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:06.842 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.837+0000 7f532ab9c700 1 --2- 172.21.15.44:0/432303732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01b90 0x55ccdacfbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.837+0000 7fbd96d9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d1b60 0x55f9542d1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.837+0000 7fbd96d9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d1b60 0x55f9542d1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57088/0 (socket says 172.21.15.44:57088) 2023-11-16T16:52:06.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.837+0000 7fbd96d9d700 1 -- 172.21.15.44:0/3457124339 learned_addr learned my addr 172.21.15.44:0/3457124339 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:06.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.837+0000 7fbd9659c700 1 --2- 172.21.15.44:0/3457124339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d2470 0x55f9542d6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.843 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.837+0000 7fbd9759e700 1 --2- 172.21.15.44:0/3457124339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d68b0 0x55f9542d6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.843 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.837+0000 7f532a39b700 1 -- 172.21.15.44:0/432303732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01b90 msgr2=0x55ccdacfbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:06.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.837+0000 7f532a39b700 1 --2- 172.21.15.44:0/432303732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01b90 0x55ccdacfbb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.837+0000 7f532a39b700 1 -- 172.21.15.44:0/432303732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdad01720 msgr2=0x55ccdacfb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:06.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.837+0000 7f532a39b700 1 --2- 172.21.15.44:0/432303732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdad01720 0x55ccdacfb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.837+0000 7f532a39b700 1 -- 172.21.15.44:0/432303732 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ccdaace5d0 con 0x55ccdab576b0 2023-11-16T16:52:06.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.837+0000 7fbd96d9d700 1 -- 172.21.15.44:0/3457124339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d2470 msgr2=0x55f9542d6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:06.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.837+0000 7fbd96d9d700 1 --2- 172.21.15.44:0/3457124339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d2470 0x55f9542d6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.837+0000 7fbd96d9d700 1 -- 172.21.15.44:0/3457124339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d68b0 msgr2=0x55f9542d6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:06.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.837+0000 7fbd96d9d700 1 --2- 172.21.15.44:0/3457124339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d68b0 0x55f9542d6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.838+0000 7fbd96d9d700 1 -- 172.21.15.44:0/3457124339 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9540a4630 con 0x55f9542d1b60 2023-11-16T16:52:06.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.838+0000 7fbd9759e700 1 --2- 172.21.15.44:0/3457124339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d68b0 0x55f9542d6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:52:06.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.838+0000 7f532a39b700 1 --2- 172.21.15.44:0/432303732 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdab576b0 0x55ccdad011e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5320015000 tx=0x7f5320020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11dd87d3868ad19d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:06.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.838+0000 7fbd96d9d700 1 --2- 172.21.15.44:0/3457124339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d1b60 0x55f9542d1f30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fbd88097170 tx=0x7fbd8809a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dccbccee1cce10e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:06.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.838+0000 7f5318bfe700 1 -- 172.21.15.44:0/432303732 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532002b070 con 0x55ccdab576b0 2023-11-16T16:52:06.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.838+0000 7f5318bfe700 1 -- 172.21.15.44:0/432303732 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f532001e040 con 0x55ccdab576b0 2023-11-16T16:52:06.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.838+0000 7f5318bfe700 1 -- 172.21.15.44:0/432303732 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532000f6c0 con 0x55ccdab576b0 2023-11-16T16:52:06.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.838+0000 7fbd7d7fa700 1 -- 172.21.15.44:0/3457124339 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd880a3030 con 0x55f9542d1b60 2023-11-16T16:52:06.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.838+0000 7fbd7d7fa700 1 -- 172.21.15.44:0/3457124339 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd88099010 con 0x55f9542d1b60 2023-11-16T16:52:06.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.838+0000 7fbd7d7fa700 1 -- 172.21.15.44:0/3457124339 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd88005420 con 0x55f9542d1b60 2023-11-16T16:52:06.848 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.839+0000 7f533ff70b80 1 -- 172.21.15.44:0/432303732 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdab576b0 msgr2=0x55ccdad011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.848 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.839+0000 7f533ff70b80 1 --2- 172.21.15.44:0/432303732 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdab576b0 0x55ccdad011e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5320015000 tx=0x7f5320020000 comp rx=0 tx=0).stop 2023-11-16T16:52:06.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.839+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/3457124339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d1b60 msgr2=0x55f9542d1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.839+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/3457124339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d1b60 0x55f9542d1f30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fbd88097170 tx=0x7fbd8809a5d0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.849 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.839+0000 7f533ff70b80 1 -- 172.21.15.44:0/432303732 shutdown_connections 2023-11-16T16:52:06.849 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.839+0000 7f533ff70b80 1 --2- 172.21.15.44:0/432303732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01b90 0x55ccdacfbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.849 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.839+0000 7f533ff70b80 1 --2- 172.21.15.44:0/432303732 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdab576b0 0x55ccdad011e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.849 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.839+0000 7f533ff70b80 1 --2- 172.21.15.44:0/432303732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdad01720 0x55ccdacfb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.850 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.839+0000 7f533ff70b80 1 -- 172.21.15.44:0/432303732 >> 172.21.15.44:0/432303732 conn(0x55ccdacf0f50 msgr2=0x55ccdacf6270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:06.850 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.839+0000 7f533ff70b80 1 -- 172.21.15.44:0/432303732 shutdown_connections 2023-11-16T16:52:06.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.839+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/3457124339 shutdown_connections 2023-11-16T16:52:06.850 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.839+0000 7f533ff70b80 1 -- 172.21.15.44:0/432303732 wait complete. 2023-11-16T16:52:06.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.839+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/3457124339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d68b0 0x55f9542d6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.839+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/3457124339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d1b60 0x55f9542d1f30 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.839+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/3457124339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d2470 0x55f9542d6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.839+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/3457124339 >> 172.21.15.44:0/3457124339 conn(0x55f9542c7360 msgr2=0x55f9542ccb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:06.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.839+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/3457124339 shutdown_connections 2023-11-16T16:52:06.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.839+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/3457124339 wait complete. 2023-11-16T16:52:06.852 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.840+0000 7f533ff70b80 1 -- start start 2023-11-16T16:52:06.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.840+0000 7fbdb0aa9b80 1 -- start start 2023-11-16T16:52:06.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.840+0000 7f533ff70b80 1 Processor -- start 2023-11-16T16:52:06.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.840+0000 7fbdb0aa9b80 1 Processor -- start 2023-11-16T16:52:06.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.841+0000 7f533ff70b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdab576b0 0x55ccdad0aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.841+0000 7f533ff70b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdad01720 0x55ccdad0af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.854 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbdb0aa9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d1b60 0x55f9542ddee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.854 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbdb0aa9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d2470 0x55f9542db200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.841+0000 7f533ff70b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdad01b90 0x55ccdad04c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.841+0000 7f533ff70b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccdabee880 con 0x55ccdad01720 2023-11-16T16:52:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.841+0000 7f533ff70b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccdabee520 con 0x55ccdab576b0 2023-11-16T16:52:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.841+0000 7f533ff70b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccdaad08c0 con 0x55ccdad01b90 2023-11-16T16:52:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.841+0000 7f533ff70b80 10 client.0 _ll_register_callbacks cb 0x55ccdac0c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:06.856 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbdb0aa9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d68b0 0x55f9542db740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.856 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbdb0aa9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9541c48e0 con 0x55f9542d2470 2023-11-16T16:52:06.856 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbdb0aa9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9541c4580 con 0x55f9542d1b60 2023-11-16T16:52:06.856 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbdb0aa9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9540a6920 con 0x55f9542d68b0 2023-11-16T16:52:06.856 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbdb0aa9b80 10 client.0 _ll_register_callbacks cb 0x55f9541e2780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:06.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.841+0000 7f532a39b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdab576b0 0x55ccdad0aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.841+0000 7f532a39b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdab576b0 0x55ccdad0aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:33518/0 (socket says 172.21.15.44:33518) 2023-11-16T16:52:06.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.841+0000 7f532a39b700 1 -- 172.21.15.44:0/2465143885 learned_addr learned my addr 172.21.15.44:0/2465143885 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:06.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.841+0000 7f532ab9c700 1 --2- 172.21.15.44:0/2465143885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdad01b90 0x55ccdad04c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbd96d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d1b60 0x55f9542ddee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbd9759e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d68b0 0x55f9542db740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbd96d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d1b60 0x55f9542ddee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:33526/0 (socket says 172.21.15.44:33526) 2023-11-16T16:52:06.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbd96d9d700 1 -- 172.21.15.44:0/3756268926 learned_addr learned my addr 172.21.15.44:0/3756268926 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:06.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbd9659c700 1 --2- 172.21.15.44:0/3756268926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d2470 0x55f9542db200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbd96d9d700 1 -- 172.21.15.44:0/3756268926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d68b0 msgr2=0x55f9542db740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:06.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbd96d9d700 1 --2- 172.21.15.44:0/3756268926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d68b0 0x55f9542db740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbd96d9d700 1 -- 172.21.15.44:0/3756268926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d2470 msgr2=0x55f9542db200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:06.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbd96d9d700 1 --2- 172.21.15.44:0/3756268926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d2470 0x55f9542db200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.841+0000 7fbd96d9d700 1 -- 172.21.15.44:0/3756268926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd981101f0 con 0x55f9542d1b60 2023-11-16T16:52:06.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.841+0000 7f532ab9c700 1 -- 172.21.15.44:0/2465143885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdab576b0 msgr2=0x55ccdad0aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:06.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.841+0000 7f532ab9c700 1 --2- 172.21.15.44:0/2465143885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdab576b0 0x55ccdad0aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.841+0000 7f532ab9c700 1 -- 172.21.15.44:0/2465143885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdad01720 msgr2=0x55ccdad0af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:06.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.841+0000 7f532ab9c700 1 --2- 172.21.15.44:0/2465143885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdad01720 0x55ccdad0af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.841+0000 7f532ab9c700 1 -- 172.21.15.44:0/2465143885 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5324113f60 con 0x55ccdad01b90 2023-11-16T16:52:06.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.842+0000 7fbd96d9d700 1 --2- 172.21.15.44:0/3756268926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d1b60 0x55f9542ddee0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbd880cb2c0 tx=0x7fbd88099180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=461f96eeef5f3c9d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:06.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.842+0000 7fbd7aff5700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:52:06.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.842+0000 7f532ab9c700 1 --2- 172.21.15.44:0/2465143885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdad01b90 0x55ccdad04c90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x55ccdad01f60 tx=0x7f531c007b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c00ad0a10bed353 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:06.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.842+0000 7f530dffb700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:52:06.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.842+0000 7fbd7aff5700 1 -- 172.21.15.44:0/3756268926 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd880b9030 con 0x55f9542d1b60 2023-11-16T16:52:06.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.842+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/3756268926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f9542c4b20 con 0x55f9542d1b60 2023-11-16T16:52:06.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.842+0000 7fbd7aff5700 1 -- 172.21.15.44:0/3756268926 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fbd880b3210 con 0x55f9542d1b60 2023-11-16T16:52:06.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.842+0000 7f530dffb700 1 -- 172.21.15.44:0/2465143885 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531c00e070 con 0x55ccdad01b90 2023-11-16T16:52:06.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.842+0000 7f530dffb700 1 -- 172.21.15.44:0/2465143885 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f531c003a00 con 0x55ccdad01b90 2023-11-16T16:52:06.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.842+0000 7f533ff70b80 1 -- 172.21.15.44:0/2465143885 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ccdacee710 con 0x55ccdad01b90 2023-11-16T16:52:06.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.842+0000 7fbd7aff5700 1 -- 172.21.15.44:0/3756268926 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd88005420 con 0x55f9542d1b60 2023-11-16T16:52:06.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.842+0000 7fbdb0aa9b80 10 client.26036 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:06.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.842+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/3756268926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55f9541e9190 con 0x55f9542d1b60 2023-11-16T16:52:06.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.843+0000 7f530dffb700 1 -- 172.21.15.44:0/2465143885 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f531c003bc0 con 0x55ccdad01b90 2023-11-16T16:52:06.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.843+0000 7f533ff70b80 10 client.26023 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:06.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.843+0000 7f533ff70b80 1 -- 172.21.15.44:0/2465143885 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55ccdac13130 con 0x55ccdad01b90 2023-11-16T16:52:06.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.843+0000 7fbd7aff5700 1 -- 172.21.15.44:0/3756268926 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f9542c4b20 con 0x55f9542d1b60 2023-11-16T16:52:06.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.843+0000 7fbd7aff5700 1 -- 172.21.15.44:0/3756268926 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbd8809a160 con 0x55f9542d1b60 2023-11-16T16:52:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.843+0000 7f530dffb700 1 -- 172.21.15.44:0/2465143885 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ccdacee710 con 0x55ccdad01b90 2023-11-16T16:52:06.865 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.843+0000 7fbdb0aa9b80 10 client.26036 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:06.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.843+0000 7fbdb0aa9b80 10 client.26036 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:06.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.843+0000 7fbdb0aa9b80 20 client.26036 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:06.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.843+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/3756268926 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f9542eb040 0x55f9542ed450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.843+0000 7fbdb0aa9b80 4 client.26036 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:06.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.843+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/3756268926 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f9541e50a0 con 0x55f9542eb040 2023-11-16T16:52:06.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.843+0000 7f530dffb700 1 -- 172.21.15.44:0/2465143885 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f531c0168c0 con 0x55ccdad01b90 2023-11-16T16:52:06.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.843+0000 7f533ff70b80 10 client.26023 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:06.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.843+0000 7f533ff70b80 10 client.26023 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:06.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.843+0000 7f533ff70b80 20 client.26023 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:06.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.843+0000 7f533ff70b80 1 --2- 172.21.15.44:0/2465143885 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ccdad14f70 0x55ccdad17380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.843+0000 7f533ff70b80 4 client.26023 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.843+0000 7f533ff70b80 1 -- 172.21.15.44:0/2465143885 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ccdac0f040 con 0x55ccdad14f70 2023-11-16T16:52:06.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.844+0000 7fbd9659c700 1 --2- 172.21.15.44:0/3756268926 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f9542eb040 0x55f9542ed450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.844+0000 7f532a39b700 1 --2- 172.21.15.44:0/2465143885 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ccdad14f70 0x55ccdad17380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.844+0000 7f530dffb700 10 client.26023 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:06.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.844+0000 7f532a39b700 1 --2- 172.21.15.44:0/2465143885 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ccdad14f70 0x55ccdad17380 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d9e1040320276a6 server_cookie=3befc90a0730fbab in_seq=0 out_seq=0 2023-11-16T16:52:06.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.844+0000 7fbd9659c700 1 --2- 172.21.15.44:0/3756268926 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f9542eb040 0x55f9542ed450 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47097b5e62e012a2 server_cookie=2d7ba6db6c213fe7 in_seq=0 out_seq=0 2023-11-16T16:52:06.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.844+0000 7fbd7aff5700 10 client.26036 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.845+0000 7f530dffb700 1 -- 172.21.15.44:0/2465143885 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ccdac0f040 con 0x55ccdad14f70 2023-11-16T16:52:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.845+0000 7f530dffb700 10 client.26023 handle_command_reply: tid=0 2023-11-16T16:52:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.845+0000 7f533ff70b80 1 client.26023 shutdown 2023-11-16T16:52:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.845+0000 7f533ff70b80 1 -- 172.21.15.44:0/2465143885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdad01b90 msgr2=0x55ccdad04c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.845+0000 7f533ff70b80 1 --2- 172.21.15.44:0/2465143885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdad01b90 0x55ccdad04c90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x55ccdad01f60 tx=0x7f531c007b50 comp rx=0 tx=0).stop 2023-11-16T16:52:06.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.845+0000 7fbd7aff5700 1 -- 172.21.15.44:0/3756268926 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f9541e50a0 con 0x55f9542eb040 2023-11-16T16:52:06.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.845+0000 7fbd7aff5700 10 client.26036 handle_command_reply: tid=0 2023-11-16T16:52:06.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.845+0000 7f533ff70b80 1 -- 172.21.15.44:0/2465143885 shutdown_connections 2023-11-16T16:52:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.845+0000 7f533ff70b80 1 --2- 172.21.15.44:0/2465143885 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ccdad14f70 0x55ccdad17380 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.845+0000 7f533ff70b80 1 --2- 172.21.15.44:0/2465143885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdad01b90 0x55ccdad04c90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.845+0000 7f533ff70b80 1 --2- 172.21.15.44:0/2465143885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdab576b0 0x55ccdad0aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.845+0000 7f533ff70b80 1 --2- 172.21.15.44:0/2465143885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdad01720 0x55ccdad0af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.845+0000 7f530dffb700 0 client.26023 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.845+0000 7f533ff70b80 1 -- 172.21.15.44:0/2465143885 >> 172.21.15.44:0/2465143885 conn(0x55ccdacf0f50 msgr2=0x55ccdacf57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:06.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.845+0000 7fbdb0aa9b80 1 client.26036 shutdown 2023-11-16T16:52:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.846+0000 7f533ff70b80 1 -- 172.21.15.44:0/2465143885 shutdown_connections 2023-11-16T16:52:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.846+0000 7f533ff70b80 1 -- 172.21.15.44:0/2465143885 wait complete. 2023-11-16T16:52:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.846+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/3756268926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d1b60 msgr2=0x55f9542ddee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.846+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/3756268926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d1b60 0x55f9542ddee0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbd880cb2c0 tx=0x7fbd88099180 comp rx=0 tx=0).stop 2023-11-16T16:52:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.846+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/3756268926 shutdown_connections 2023-11-16T16:52:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.846+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/3756268926 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f9542eb040 0x55f9542ed450 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.846+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/3756268926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d68b0 0x55f9542db740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.846+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/3756268926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d1b60 0x55f9542ddee0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.846+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/3756268926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d2470 0x55f9542db200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.846+0000 7fbd7aff5700 0 client.26036 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:06.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.846+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/3756268926 >> 172.21.15.44:0/3756268926 conn(0x55f9542c7360 msgr2=0x55f9542d7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:06.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.846+0000 7f533ff70b80 20 client.26023 trim_cache size 0 max 16384 2023-11-16T16:52:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.846+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/3756268926 shutdown_connections 2023-11-16T16:52:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.846+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/3756268926 wait complete. 2023-11-16T16:52:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.846+0000 7fbdb0aa9b80 20 client.26036 trim_cache size 0 max 16384 2023-11-16T16:52:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.861+0000 7fbdb0aa9b80 1 Processor -- start 2023-11-16T16:52:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.861+0000 7fbdb0aa9b80 1 -- start start 2023-11-16T16:52:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.861+0000 7fbdb0aa9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d1b60 0x55f9542ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.861+0000 7fbdb0aa9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d2470 0x55f9542e27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.861+0000 7fbdb0aa9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d68b0 0x55f9542d7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.861+0000 7fbdb0aa9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9542d9760 con 0x55f9542d1b60 2023-11-16T16:52:06.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.861+0000 7fbdb0aa9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9542d5140 con 0x55f9542d2470 2023-11-16T16:52:06.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.861+0000 7fbdb0aa9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9542d5390 con 0x55f9542d68b0 2023-11-16T16:52:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.861+0000 7f533ff70b80 1 Processor -- start 2023-11-16T16:52:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.861+0000 7f533ff70b80 1 -- start start 2023-11-16T16:52:06.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.862+0000 7fbd96d9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d1b60 0x55f9542ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.862+0000 7fbd96d9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d1b60 0x55f9542ccce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57130/0 (socket says 172.21.15.44:57130) 2023-11-16T16:52:06.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.862+0000 7fbd96d9d700 1 -- 172.21.15.44:0/1653053197 learned_addr learned my addr 172.21.15.44:0/1653053197 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:06.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.862+0000 7fbd9759e700 1 --2- 172.21.15.44:0/1653053197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d68b0 0x55f9542d7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.862+0000 7fbd9659c700 1 --2- 172.21.15.44:0/1653053197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d2470 0x55f9542e27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.862+0000 7f533ff70b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdab576b0 0x55ccdad05620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.862+0000 7f533ff70b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01720 0x55ccdacf5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.862+0000 7f533ff70b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdad01b90 0x55ccdad01f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.862+0000 7fbd9759e700 1 -- 172.21.15.44:0/1653053197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d2470 msgr2=0x55f9542e27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.862+0000 7fbd9759e700 1 --2- 172.21.15.44:0/1653053197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d2470 0x55f9542e27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.862+0000 7fbd9759e700 1 -- 172.21.15.44:0/1653053197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d1b60 msgr2=0x55f9542ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.862+0000 7fbd9759e700 1 --2- 172.21.15.44:0/1653053197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d1b60 0x55f9542ccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.862+0000 7fbd9759e700 1 -- 172.21.15.44:0/1653053197 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9541e8ee0 con 0x55f9542d68b0 2023-11-16T16:52:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.862+0000 7fbd96d9d700 1 --2- 172.21.15.44:0/1653053197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d1b60 0x55f9542ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.862+0000 7f533ff70b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccdacf7020 con 0x55ccdad01b90 2023-11-16T16:52:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.862+0000 7f533ff70b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccdacf0ba0 con 0x55ccdad01720 2023-11-16T16:52:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.862+0000 7f533ff70b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccdacfc250 con 0x55ccdab576b0 2023-11-16T16:52:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.862+0000 7f5329b9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01720 0x55ccdacf5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.862+0000 7f5329b9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01720 0x55ccdacf5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:33552/0 (socket says 172.21.15.44:33552) 2023-11-16T16:52:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.862+0000 7f5329b9a700 1 -- 172.21.15.44:0/2610749496 learned_addr learned my addr 172.21.15.44:0/2610749496 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.862+0000 7f532a39b700 1 --2- 172.21.15.44:0/2610749496 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdab576b0 0x55ccdad05620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.863+0000 7f532ab9c700 1 --2- 172.21.15.44:0/2610749496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdad01b90 0x55ccdad01f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.863+0000 7fbd9759e700 1 --2- 172.21.15.44:0/1653053197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d68b0 0x55f9542d7c90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbd981012d0 tx=0x7fbd90009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b52ad0aa472a57cf server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.863+0000 7f5329b9a700 1 -- 172.21.15.44:0/2610749496 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdab576b0 msgr2=0x55ccdad05620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.863+0000 7f5329b9a700 1 --2- 172.21.15.44:0/2610749496 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdab576b0 0x55ccdad05620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.863+0000 7f5329b9a700 1 -- 172.21.15.44:0/2610749496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdad01b90 msgr2=0x55ccdad01f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.863+0000 7f5329b9a700 1 --2- 172.21.15.44:0/2610749496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdad01b90 0x55ccdad01f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.863+0000 7f5329b9a700 1 -- 172.21.15.44:0/2610749496 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ccdacf26f0 con 0x55ccdad01720 2023-11-16T16:52:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.863+0000 7f532ab9c700 1 --2- 172.21.15.44:0/2610749496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdad01b90 0x55ccdad01f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:52:06.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.863+0000 7fbd7dffb700 1 -- 172.21.15.44:0/1653053197 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9000e040 con 0x55f9542d68b0 2023-11-16T16:52:06.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.863+0000 7fbd7dffb700 1 -- 172.21.15.44:0/1653053197 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd9001b070 con 0x55f9542d68b0 2023-11-16T16:52:06.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.863+0000 7fbd7dffb700 1 -- 172.21.15.44:0/1653053197 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd900044f0 con 0x55f9542d68b0 2023-11-16T16:52:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.863+0000 7f5329b9a700 1 --2- 172.21.15.44:0/2610749496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01720 0x55ccdacf5f60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f53140964c0 tx=0x7f5314099eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da16c431b0fdbca6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.863+0000 7f53193ff700 1 -- 172.21.15.44:0/2610749496 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5314099fc0 con 0x55ccdad01720 2023-11-16T16:52:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.863+0000 7f53193ff700 1 -- 172.21.15.44:0/2610749496 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f531409dff0 con 0x55ccdad01720 2023-11-16T16:52:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.863+0000 7f53193ff700 1 -- 172.21.15.44:0/2610749496 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5314099710 con 0x55ccdad01720 2023-11-16T16:52:06.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.864+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/1653053197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d68b0 msgr2=0x55f9542d7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.864+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/1653053197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d68b0 0x55f9542d7c90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbd981012d0 tx=0x7fbd90009920 comp rx=0 tx=0).stop 2023-11-16T16:52:06.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.864+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/1653053197 shutdown_connections 2023-11-16T16:52:06.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.864+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/1653053197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d68b0 0x55f9542d7c90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.864+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/1653053197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d1b60 0x55f9542ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.864+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/1653053197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d2470 0x55f9542e27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.864+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/1653053197 >> 172.21.15.44:0/1653053197 conn(0x55f9542c7360 msgr2=0x55f9542cc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:06.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.864+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/1653053197 shutdown_connections 2023-11-16T16:52:06.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.864+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/1653053197 wait complete. 2023-11-16T16:52:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.864+0000 7f533ff70b80 1 -- 172.21.15.44:0/2610749496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01720 msgr2=0x55ccdacf5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.864+0000 7f533ff70b80 1 --2- 172.21.15.44:0/2610749496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01720 0x55ccdacf5f60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f53140964c0 tx=0x7f5314099eb0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.864+0000 7f533ff70b80 1 -- 172.21.15.44:0/2610749496 shutdown_connections 2023-11-16T16:52:06.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.864+0000 7f533ff70b80 1 --2- 172.21.15.44:0/2610749496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdad01b90 0x55ccdad01f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.864+0000 7f533ff70b80 1 --2- 172.21.15.44:0/2610749496 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdab576b0 0x55ccdad05620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.864+0000 7f533ff70b80 1 --2- 172.21.15.44:0/2610749496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01720 0x55ccdacf5f60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.864+0000 7f533ff70b80 1 -- 172.21.15.44:0/2610749496 >> 172.21.15.44:0/2610749496 conn(0x55ccdacf0f50 msgr2=0x55ccdad1bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:06.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.864+0000 7fbdb0aa9b80 1 -- start start 2023-11-16T16:52:06.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.864+0000 7f533ff70b80 1 -- 172.21.15.44:0/2610749496 shutdown_connections 2023-11-16T16:52:06.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.864+0000 7f533ff70b80 1 -- 172.21.15.44:0/2610749496 wait complete. 2023-11-16T16:52:06.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.865+0000 7fbdb0aa9b80 1 Processor -- start 2023-11-16T16:52:06.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.865+0000 7f533ff70b80 1 -- start start 2023-11-16T16:52:06.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.865+0000 7fbdb0aa9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d1b60 0x55f9542e2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.865+0000 7fbdb0aa9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d2470 0x55f9542e0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.865+0000 7fbdb0aa9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d68b0 0x55f9542e0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.865+0000 7fbdb0aa9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9542e10e0 con 0x55f9542d68b0 2023-11-16T16:52:06.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.865+0000 7fbdb0aa9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9542e1250 con 0x55f9542d1b60 2023-11-16T16:52:06.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.865+0000 7fbdb0aa9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9542e1410 con 0x55f9542d2470 2023-11-16T16:52:06.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.865+0000 7f533ff70b80 1 Processor -- start 2023-11-16T16:52:06.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.865+0000 7fbdb0aa9b80 10 client.0 _ll_register_callbacks cb 0x7fbd8809a7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:06.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.865+0000 7fbd96d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d1b60 0x55f9542e2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.865+0000 7fbd9759e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d68b0 0x55f9542e0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.866+0000 7fbd96d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d1b60 0x55f9542e2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:33554/0 (socket says 172.21.15.44:33554) 2023-11-16T16:52:06.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.866+0000 7fbd96d9d700 1 -- 172.21.15.44:0/2701317046 learned_addr learned my addr 172.21.15.44:0/2701317046 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:06.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.866+0000 7fbd9659c700 1 --2- 172.21.15.44:0/2701317046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d2470 0x55f9542e0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.866+0000 7f533ff70b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdab576b0 0x55ccdad086d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.866+0000 7f533ff70b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01720 0x55ccdad08c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.866+0000 7f533ff70b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdad01b90 0x55ccdad67e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.866+0000 7f533ff70b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccdad093a0 con 0x55ccdab576b0 2023-11-16T16:52:06.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.866+0000 7f533ff70b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccdad09510 con 0x55ccdad01720 2023-11-16T16:52:06.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.866+0000 7f533ff70b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccdad68370 con 0x55ccdad01b90 2023-11-16T16:52:06.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.866+0000 7f533ff70b80 10 client.0 _ll_register_callbacks cb 0x7f531c007500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:06.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.866+0000 7fbd9759e700 1 -- 172.21.15.44:0/2701317046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d2470 msgr2=0x55f9542e0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:06.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.866+0000 7fbd9759e700 1 --2- 172.21.15.44:0/2701317046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d2470 0x55f9542e0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.866+0000 7fbd9759e700 1 -- 172.21.15.44:0/2701317046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d1b60 msgr2=0x55f9542e2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:06.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.866+0000 7fbd9759e700 1 --2- 172.21.15.44:0/2701317046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d1b60 0x55f9542e2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.866+0000 7fbd9759e700 1 -- 172.21.15.44:0/2701317046 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f9541e9190 con 0x55f9542d68b0 2023-11-16T16:52:06.893 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.866+0000 7fbd9659c700 1 --2- 172.21.15.44:0/2701317046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d2470 0x55f9542e0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:06.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.866+0000 7f5329b9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01720 0x55ccdad08c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.866+0000 7f532ab9c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdad01b90 0x55ccdad67e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.866+0000 7f5329b9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01720 0x55ccdad08c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:33568/0 (socket says 172.21.15.44:33568) 2023-11-16T16:52:06.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.866+0000 7f5329b9a700 1 -- 172.21.15.44:0/1254967904 learned_addr learned my addr 172.21.15.44:0/1254967904 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:06.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.866+0000 7f532ab9c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdad01b90 0x55ccdad67e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:42228/0 (socket says 172.21.15.44:42228) 2023-11-16T16:52:06.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.866+0000 7f532a39b700 1 --2- 172.21.15.44:0/1254967904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdab576b0 0x55ccdad086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.895 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.866+0000 7fbd9759e700 1 --2- 172.21.15.44:0/2701317046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d68b0 0x55f9542e0ba0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fbd981012d0 tx=0x7fbd90003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e9b2a319f7b12d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:06.895 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.866+0000 7fbd7aff5700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:52:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.867+0000 7f5329b9a700 1 -- 172.21.15.44:0/1254967904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdad01b90 msgr2=0x55ccdad67e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.867+0000 7f5329b9a700 1 --2- 172.21.15.44:0/1254967904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdad01b90 0x55ccdad67e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.867+0000 7f5329b9a700 1 -- 172.21.15.44:0/1254967904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdab576b0 msgr2=0x55ccdad086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.867+0000 7f5329b9a700 1 --2- 172.21.15.44:0/1254967904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdab576b0 0x55ccdad086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.867+0000 7f5329b9a700 1 -- 172.21.15.44:0/1254967904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5324116730 con 0x55ccdad01720 2023-11-16T16:52:06.896 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.867+0000 7fbd7aff5700 1 -- 172.21.15.44:0/2701317046 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9000e070 con 0x55f9542d68b0 2023-11-16T16:52:06.896 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.867+0000 7fbd7aff5700 1 -- 172.21.15.44:0/2701317046 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fbd90004130 con 0x55f9542d68b0 2023-11-16T16:52:06.896 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.867+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/2701317046 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd88099010 con 0x55f9542d68b0 2023-11-16T16:52:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.867+0000 7f5329b9a700 1 --2- 172.21.15.44:0/1254967904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01720 0x55ccdad08c10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5314099f50 tx=0x7f53140a2410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91eac9b453180ce5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.867+0000 7f530dffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:52:06.897 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.867+0000 7fbd7aff5700 1 -- 172.21.15.44:0/2701317046 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd90003e20 con 0x55f9542d68b0 2023-11-16T16:52:06.897 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.867+0000 7fbdb0aa9b80 10 client.16026 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:06.897 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.867+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/2701317046 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x7fbd90009230 con 0x55f9542d68b0 2023-11-16T16:52:06.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.867+0000 7f530dffb700 1 -- 172.21.15.44:0/1254967904 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53140b82c0 con 0x55ccdad01720 2023-11-16T16:52:06.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.867+0000 7f533ff70b80 1 -- 172.21.15.44:0/1254967904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f532001e040 con 0x55ccdad01720 2023-11-16T16:52:06.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.867+0000 7f530dffb700 1 -- 172.21.15.44:0/1254967904 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f5314094bd0 con 0x55ccdad01720 2023-11-16T16:52:06.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.867+0000 7f530dffb700 1 -- 172.21.15.44:0/1254967904 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5314094d90 con 0x55ccdad01720 2023-11-16T16:52:06.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.867+0000 7f533ff70b80 10 client.26048 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:06.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.867+0000 7f533ff70b80 1 -- 172.21.15.44:0/1254967904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55ccdaa7ad80 con 0x55ccdad01720 2023-11-16T16:52:06.899 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.868+0000 7fbd7aff5700 1 -- 172.21.15.44:0/2701317046 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd88099010 con 0x55f9542d68b0 2023-11-16T16:52:06.899 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.868+0000 7fbd7aff5700 1 -- 172.21.15.44:0/2701317046 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbd900056b0 con 0x55f9542d68b0 2023-11-16T16:52:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.868+0000 7f530dffb700 1 -- 172.21.15.44:0/1254967904 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5314078e90 con 0x55ccdad01720 2023-11-16T16:52:06.900 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.868+0000 7fbdb0aa9b80 10 client.16026 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:06.900 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.868+0000 7fbdb0aa9b80 10 client.16026 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:06.900 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.868+0000 7fbdb0aa9b80 20 client.16026 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:06.900 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.868+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/2701317046 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f954340190 0x55f9543425a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.900 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.868+0000 7fbdb0aa9b80 4 client.16026 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T16:52:06.900 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.868+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/2701317046 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f9541f1660 con 0x55f954340190 2023-11-16T16:52:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.868+0000 7f530dffb700 1 -- 172.21.15.44:0/1254967904 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5314098f20 con 0x55ccdad01720 2023-11-16T16:52:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.868+0000 7f533ff70b80 10 client.26048 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.868+0000 7f533ff70b80 10 client.26048 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.868+0000 7f533ff70b80 20 client.26048 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.868+0000 7f533ff70b80 1 --2- 172.21.15.44:0/1254967904 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ccdad698f0 0x55ccdad6bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.868+0000 7f533ff70b80 4 client.26048 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:52:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.868+0000 7f533ff70b80 1 -- 172.21.15.44:0/1254967904 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ccdac1b600 con 0x55ccdad698f0 2023-11-16T16:52:06.902 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.868+0000 7fbd96d9d700 1 --2- 172.21.15.44:0/2701317046 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f954340190 0x55f9543425a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.868+0000 7f532a39b700 1 --2- 172.21.15.44:0/1254967904 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ccdad698f0 0x55ccdad6bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:06.902 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.869+0000 7fbd7aff5700 10 client.16026 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:06.903 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.869+0000 7fbd96d9d700 1 --2- 172.21.15.44:0/2701317046 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f954340190 0x55f9543425a0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58fb2fe4421f276c server_cookie=bbea183c1c14c4ef in_seq=0 out_seq=0 2023-11-16T16:52:06.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.869+0000 7f532a39b700 1 --2- 172.21.15.44:0/1254967904 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ccdad698f0 0x55ccdad6bd00 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe620b5c342b754f server_cookie=65d29d4ab56047b9 in_seq=0 out_seq=0 2023-11-16T16:52:06.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.869+0000 7f530dffb700 10 client.26048 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:06.904 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.869+0000 7fbd7aff5700 1 -- 172.21.15.44:0/2701317046 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f9541f1660 con 0x55f954340190 2023-11-16T16:52:06.904 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.869+0000 7fbd7aff5700 10 client.16026 handle_command_reply: tid=0 2023-11-16T16:52:06.904 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.869+0000 7fbdb0aa9b80 1 client.16026 shutdown 2023-11-16T16:52:06.904 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.869+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/2701317046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d68b0 msgr2=0x55f9542e0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.904 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.869+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/2701317046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d68b0 0x55f9542e0ba0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fbd981012d0 tx=0x7fbd90003c30 comp rx=0 tx=0).stop 2023-11-16T16:52:06.905 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.869+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/2701317046 shutdown_connections 2023-11-16T16:52:06.905 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.869+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/2701317046 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f954340190 0x55f9543425a0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.905 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.869+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/2701317046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f9542d68b0 0x55f9542e0ba0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.905 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.869+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/2701317046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f9542d1b60 0x55f9542e2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.905 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.869+0000 7fbdb0aa9b80 1 --2- 172.21.15.44:0/2701317046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f9542d2470 0x55f9542e0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.905 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.869+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/2701317046 >> 172.21.15.44:0/2701317046 conn(0x55f9542c7360 msgr2=0x55f9542d8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:06.905 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.869+0000 7fbd7aff5700 0 client.16026 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:06.905 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.869+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/2701317046 shutdown_connections 2023-11-16T16:52:06.905 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.869+0000 7fbdb0aa9b80 1 -- 172.21.15.44:0/2701317046 wait complete. 2023-11-16T16:52:06.905 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.870+0000 7fbdb0aa9b80 20 client.16026 trim_cache size 0 max 16384 2023-11-16T16:52:06.906 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T16:52:06.906 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.870+0000 7fbd9d1ff700 1 -- 172.21.15.44:0/2369863585 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbd800a1340 msgr2=0x7fbd800a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.906 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.870+0000 7fbd9d1ff700 1 --2- 172.21.15.44:0/2369863585 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbd800a1340 0x7fbd800a37f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbd981009b0 tx=0x7fbd8c009310 comp rx=0 tx=0).stop 2023-11-16T16:52:06.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.870+0000 7fbd9d1ff700 1 -- 172.21.15.44:0/2369863585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd98101390 msgr2=0x7fbd980ff1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.870+0000 7fbd9d1ff700 1 --2- 172.21.15.44:0/2369863585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd98101390 0x7fbd980ff1c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009810 tx=0x7fbd8800fa00 comp rx=0 tx=0).stop 2023-11-16T16:52:06.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.870+0000 7fbd9d1ff700 1 -- 172.21.15.44:0/2369863585 shutdown_connections 2023-11-16T16:52:06.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.870+0000 7fbd9d1ff700 1 --2- 172.21.15.44:0/2369863585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd9810a8b0 0x7fbd981a4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.870+0000 7fbd9d1ff700 1 --2- 172.21.15.44:0/2369863585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd98101390 0x7fbd980ff1c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.870+0000 7fbd9d1ff700 1 --2- 172.21.15.44:0/2369863585 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbd800a1340 0x7fbd800a37f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.870+0000 7fbd9d1ff700 1 --2- 172.21.15.44:0/2369863585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd98101ca0 0x7fbd980ff700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.870+0000 7fbd9d1ff700 1 -- 172.21.15.44:0/2369863585 >> 172.21.15.44:0/2369863585 conn(0x7fbd980770b0 msgr2=0x7fbd98102290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:06.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.870+0000 7fbd9d1ff700 1 -- 172.21.15.44:0/2369863585 shutdown_connections 2023-11-16T16:52:06.907 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:06.870+0000 7fbd9d1ff700 1 -- 172.21.15.44:0/2369863585 wait complete. 2023-11-16T16:52:06.908 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.894+0000 7f530dffb700 1 -- 172.21.15.44:0/1254967904 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ccdac1b600 con 0x55ccdad698f0 2023-11-16T16:52:06.908 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.894+0000 7f530dffb700 10 client.26048 handle_command_reply: tid=0 2023-11-16T16:52:06.908 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.894+0000 7f533ff70b80 1 client.26048 shutdown 2023-11-16T16:52:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.894+0000 7f533ff70b80 1 -- 172.21.15.44:0/1254967904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01720 msgr2=0x55ccdad08c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.894+0000 7f533ff70b80 1 --2- 172.21.15.44:0/1254967904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01720 0x55ccdad08c10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5314099f50 tx=0x7f53140a2410 comp rx=0 tx=0).stop 2023-11-16T16:52:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.894+0000 7f533ff70b80 1 -- 172.21.15.44:0/1254967904 shutdown_connections 2023-11-16T16:52:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.894+0000 7f533ff70b80 1 --2- 172.21.15.44:0/1254967904 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ccdad698f0 0x55ccdad6bd00 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.894+0000 7f533ff70b80 1 --2- 172.21.15.44:0/1254967904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ccdad01b90 0x55ccdad67e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.894+0000 7f533ff70b80 1 --2- 172.21.15.44:0/1254967904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ccdab576b0 0x55ccdad086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.894+0000 7f533ff70b80 1 --2- 172.21.15.44:0/1254967904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ccdad01720 0x55ccdad08c10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.894+0000 7f530dffb700 0 client.26048 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.894+0000 7f533ff70b80 1 -- 172.21.15.44:0/1254967904 >> 172.21.15.44:0/1254967904 conn(0x55ccdacf0f50 msgr2=0x55ccdacfee00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.894+0000 7f533ff70b80 1 -- 172.21.15.44:0/1254967904 shutdown_connections 2023-11-16T16:52:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.894+0000 7f533ff70b80 1 -- 172.21.15.44:0/1254967904 wait complete. 2023-11-16T16:52:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.894+0000 7f533ff70b80 20 client.26048 trim_cache size 0 max 16384 2023-11-16T16:52:06.910 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:52:06.910 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:52:06.910 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:52:06.910 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:52:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.895+0000 7f532c5ff700 1 -- 172.21.15.44:0/1898209835 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f53100a1340 msgr2=0x7f53100a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.895+0000 7f532c5ff700 1 --2- 172.21.15.44:0/1898209835 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f53100a1340 0x7f53100a37f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f532000b2e0 tx=0x7f5320005fb0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.895+0000 7f532c5ff700 1 -- 172.21.15.44:0/1898209835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53240ffb40 msgr2=0x7f532419cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.895+0000 7f532c5ff700 1 --2- 172.21.15.44:0/1898209835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53240ffb40 0x7f532419cee0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f531400e9a0 tx=0x7f531400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.895+0000 7f532c5ff700 1 -- 172.21.15.44:0/1898209835 shutdown_connections 2023-11-16T16:52:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.895+0000 7f532c5ff700 1 --2- 172.21.15.44:0/1898209835 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f53100a1340 0x7f53100a37f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.895+0000 7f532c5ff700 1 --2- 172.21.15.44:0/1898209835 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5324104630 0x7f53241a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.912 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.895+0000 7f532c5ff700 1 --2- 172.21.15.44:0/1898209835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53240ff230 0x7f532419c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.912 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.895+0000 7f532c5ff700 1 --2- 172.21.15.44:0/1898209835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53240ffb40 0x7f532419cee0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:06.912 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.895+0000 7f532c5ff700 1 -- 172.21.15.44:0/1898209835 >> 172.21.15.44:0/1898209835 conn(0x7f53240763a0 msgr2=0x7f5324102050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:06.912 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.895+0000 7f532c5ff700 1 -- 172.21.15.44:0/1898209835 shutdown_connections 2023-11-16T16:52:06.912 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:06.895+0000 7f532c5ff700 1 -- 172.21.15.44:0/1898209835 wait complete. 2023-11-16T16:52:07.338 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:07.338 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:07.483 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:07.484 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:07.515 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:07.515 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:07.515 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 55 -lt 100 ']' 2023-11-16T16:52:07.515 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 55 + 1 2023-11-16T16:52:07.548 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=56 2023-11-16T16:52:07.548 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:07.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:07 smithi044 ceph-mon[128750]: pgmap v68: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 36 KiB/s wr, 17 op/s 2023-11-16T16:52:07.637 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:07.638 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:07.640 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:07.641 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:07.642 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:07.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:07 smithi176 ceph-mon[131630]: pgmap v68: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 36 KiB/s wr, 17 op/s 2023-11-16T16:52:07.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:07 smithi077 ceph-mon[131204]: pgmap v68: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 36 KiB/s wr, 17 op/s 2023-11-16T16:52:07.919 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:52:07.920 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:52:08.104 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:08.104 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:08.144 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:08.144 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:08.172 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:08.173 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:08.173 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 56 -lt 100 ']' 2023-11-16T16:52:08.173 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 56 + 1 2023-11-16T16:52:08.212 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=57 2023-11-16T16:52:08.212 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:08.236 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:08.237 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:08.239 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:08.240 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:08.240 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:08.545 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:08 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:52:08.546 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:08 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:52:08.546 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:08 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:52:08.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:08 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:52:08.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:08 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:52:08.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:08 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:52:08.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:08 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:52:08.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:08 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:52:08.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:08 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:52:08.771 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:08.771 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:08.897 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:08.897 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:08.948 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:08.949 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:08.949 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 57 -lt 100 ']' 2023-11-16T16:52:08.949 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 57 + 1 2023-11-16T16:52:09.011 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=58 2023-11-16T16:52:09.012 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:09.097 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:09.097 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:09.100 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:09.100 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:09.101 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:09.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:09 smithi044 ceph-mon[128750]: pgmap v69: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 39 KiB/s wr, 19 op/s 2023-11-16T16:52:09.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:09 smithi176 ceph-mon[131630]: pgmap v69: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 39 KiB/s wr, 19 op/s 2023-11-16T16:52:09.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:09 smithi077 ceph-mon[131204]: pgmap v69: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 39 KiB/s wr, 19 op/s 2023-11-16T16:52:16.235 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:11 smithi176 ceph-mon[131630]: pgmap v70: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 38 KiB/s wr, 17 op/s 2023-11-16T16:52:16.235 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:12 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:52:16.235 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:12 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:16.235 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:12 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:52:16.235 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:12 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:16.235 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:12 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:52:16.235 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:13 smithi176 ceph-mon[131630]: pgmap v71: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 59 KiB/s wr, 26 op/s 2023-11-16T16:52:16.235 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:14 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:52:16.235 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:14 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:14 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:15 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:15 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:15 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:15 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:15 smithi176 ceph-mon[131630]: pgmap v72: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:11 smithi077 ceph-mon[131204]: pgmap v70: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 38 KiB/s wr, 17 op/s 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:12 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:12 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:12 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:12 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:12 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:52:16.237 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:13 smithi077 ceph-mon[131204]: pgmap v71: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 59 KiB/s wr, 26 op/s 2023-11-16T16:52:16.237 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:14 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:52:16.237 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:14 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:52:16.237 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:14 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:52:16.237 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:15 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:52:16.237 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:15 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T16:52:16.237 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:15 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:52:16.237 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:15 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:52:16.237 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:15 smithi077 ceph-mon[131204]: pgmap v72: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-11-16T16:52:16.238 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:16.238 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.238 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:16.238 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:16.238 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.238 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.238 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 58 -lt 100 ']' 2023-11-16T16:52:16.238 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 58 + 1 2023-11-16T16:52:16.239 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=59 2023-11-16T16:52:16.239 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:16.239 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:16.239 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.239 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:16.239 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:16.239 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.018+0000 7ffb927ff700 1 -- 172.21.15.44:0/3203881463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb8c102e80 msgr2=0x7ffb8c10f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:16.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.018+0000 7ffb927ff700 1 --2- 172.21.15.44:0/3203881463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb8c102e80 0x7ffb8c10f310 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ffb8400b0a0 tx=0x7ffb8400b3b0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.018+0000 7ffb927ff700 1 -- 172.21.15.44:0/3203881463 shutdown_connections 2023-11-16T16:52:16.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.018+0000 7ffb927ff700 1 --2- 172.21.15.44:0/3203881463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb8c102e80 0x7ffb8c10f310 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.018+0000 7ffb927ff700 1 --2- 172.21.15.44:0/3203881463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb8c1084e0 0x7ffb8c1088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.018+0000 7ffb927ff700 1 --2- 172.21.15.44:0/3203881463 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb8c1024d0 0x7ffb8c102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.018+0000 7ffb927ff700 1 -- 172.21.15.44:0/3203881463 >> 172.21.15.44:0/3203881463 conn(0x7ffb8c0fdfb0 msgr2=0x7ffb8c1003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:16.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.018+0000 7ffb927ff700 1 -- 172.21.15.44:0/3203881463 shutdown_connections 2023-11-16T16:52:16.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.018+0000 7ffb927ff700 1 -- 172.21.15.44:0/3203881463 wait complete. 2023-11-16T16:52:16.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.019+0000 7ffb927ff700 1 Processor -- start 2023-11-16T16:52:16.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.020+0000 7ffb927ff700 1 -- start start 2023-11-16T16:52:16.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.020+0000 7ffb927ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb8c1024d0 0x7ffb8c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.020+0000 7ffb927ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb8c102e80 0x7ffb8c198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.020+0000 7ffb927ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb8c1084e0 0x7ffb8c19bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.020+0000 7ffb927ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c19c5d0 con 0x7ffb8c1024d0 2023-11-16T16:52:16.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.020+0000 7ffb927ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c19c740 con 0x7ffb8c1084e0 2023-11-16T16:52:16.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.020+0000 7ffb927ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c19c900 con 0x7ffb8c102e80 2023-11-16T16:52:16.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.020+0000 7ffb8bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb8c1024d0 0x7ffb8c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.021+0000 7ffb8bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb8c1024d0 0x7ffb8c1986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57178/0 (socket says 172.21.15.44:57178) 2023-11-16T16:52:16.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.021+0000 7ffb8bfff700 1 -- 172.21.15.44:0/3092950580 learned_addr learned my addr 172.21.15.44:0/3092950580 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:16.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.021+0000 7ffb8b7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb8c102e80 0x7ffb8c198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.021+0000 7ffb90d9c700 1 --2- 172.21.15.44:0/3092950580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb8c1084e0 0x7ffb8c19bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.021+0000 7ffb8bfff700 1 -- 172.21.15.44:0/3092950580 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb8c102e80 msgr2=0x7ffb8c198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:16.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.021+0000 7ffb8bfff700 1 --2- 172.21.15.44:0/3092950580 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb8c102e80 0x7ffb8c198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.021+0000 7ffb8bfff700 1 -- 172.21.15.44:0/3092950580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb8c1084e0 msgr2=0x7ffb8c19bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:16.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.021+0000 7ffb8bfff700 1 --2- 172.21.15.44:0/3092950580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb8c1084e0 0x7ffb8c19bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.021+0000 7ffb8bfff700 1 -- 172.21.15.44:0/3092950580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb84009d00 con 0x7ffb8c1024d0 2023-11-16T16:52:16.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.021+0000 7ffb90d9c700 1 --2- 172.21.15.44:0/3092950580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb8c1084e0 0x7ffb8c19bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:16.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.021+0000 7ffb8bfff700 1 --2- 172.21.15.44:0/3092950580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb8c1024d0 0x7ffb8c1986e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00e9a0 tx=0x7ffb7c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:16.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.022+0000 7ffb897fa700 1 -- 172.21.15.44:0/3092950580 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c009790 con 0x7ffb8c1024d0 2023-11-16T16:52:16.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.022+0000 7ffb897fa700 1 -- 172.21.15.44:0/3092950580 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb7c010430 con 0x7ffb8c1024d0 2023-11-16T16:52:16.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.022+0000 7ffb897fa700 1 -- 172.21.15.44:0/3092950580 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c018680 con 0x7ffb8c1024d0 2023-11-16T16:52:16.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.022+0000 7ffb927ff700 1 -- 172.21.15.44:0/3092950580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8c112520 con 0x7ffb8c1024d0 2023-11-16T16:52:16.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.022+0000 7ffb927ff700 1 -- 172.21.15.44:0/3092950580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8c112390 con 0x7ffb8c1024d0 2023-11-16T16:52:16.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.025+0000 7ffb897fa700 1 -- 172.21.15.44:0/3092950580 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7ffb7c0105a0 con 0x7ffb8c1024d0 2023-11-16T16:52:16.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.027+0000 7ffb897fa700 1 --2- 172.21.15.44:0/3092950580 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ffb740a1340 0x7ffb740a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.027+0000 7ffb897fa700 1 -- 172.21.15.44:0/3092950580 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7ffb7c014070 con 0x7ffb8c1024d0 2023-11-16T16:52:16.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.027+0000 7ffb8b7fe700 1 --2- 172.21.15.44:0/3092950580 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ffb740a1340 0x7ffb740a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.028+0000 7ffb8b7fe700 1 --2- 172.21.15.44:0/3092950580 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ffb740a1340 0x7ffb740a37f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c199cf0 tx=0x7ffb80009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:16.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.103+0000 7ffba61e5b80 1 Processor -- start 2023-11-16T16:52:16.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.103+0000 7ffba61e5b80 1 -- start start 2023-11-16T16:52:16.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.103+0000 7ffba61e5b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de68526be0 0x55de68526fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffba61e5b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de685274f0 0x55de6852b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffba61e5b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de6852b930 0x55de6852bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffba61e5b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de6842c0e0 con 0x55de6852b930 2023-11-16T16:52:16.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffba61e5b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de6847e920 con 0x55de685274f0 2023-11-16T16:52:16.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffba61e5b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de68477db0 con 0x55de68526be0 2023-11-16T16:52:16.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffb90d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de6852b930 0x55de6852bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffb8b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de685274f0 0x55de6852b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffb8bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de68526be0 0x55de68526fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffb8b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de685274f0 0x55de6852b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:33594/0 (socket says 172.21.15.44:33594) 2023-11-16T16:52:16.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffb8b7fe700 1 -- 172.21.15.44:0/1629361420 learned_addr learned my addr 172.21.15.44:0/1629361420 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:16.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffb8bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de68526be0 0x55de68526fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:42262/0 (socket says 172.21.15.44:42262) 2023-11-16T16:52:16.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffb8b7fe700 1 -- 172.21.15.44:0/1629361420 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de68526be0 msgr2=0x55de68526fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:16.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffb8b7fe700 1 --2- 172.21.15.44:0/1629361420 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de68526be0 0x55de68526fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffb8b7fe700 1 -- 172.21.15.44:0/1629361420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de6852b930 msgr2=0x55de6852bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:16.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffb8b7fe700 1 --2- 172.21.15.44:0/1629361420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de6852b930 0x55de6852bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffb8b7fe700 1 -- 172.21.15.44:0/1629361420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de682f96b0 con 0x55de685274f0 2023-11-16T16:52:16.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.104+0000 7ffb8bfff700 1 --2- 172.21.15.44:0/1629361420 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de68526be0 0x55de68526fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:16.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.105+0000 7ffb8b7fe700 1 --2- 172.21.15.44:0/1629361420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de685274f0 0x55de6852b3f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ffb80011d20 tx=0x7ffb80014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e8678e882bf43da server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:16.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.105+0000 7ffb727fc700 1 -- 172.21.15.44:0/1629361420 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8000eb20 con 0x55de685274f0 2023-11-16T16:52:16.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.105+0000 7ffb727fc700 1 -- 172.21.15.44:0/1629361420 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb80021070 con 0x55de685274f0 2023-11-16T16:52:16.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.105+0000 7ffb727fc700 1 -- 172.21.15.44:0/1629361420 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8001d670 con 0x55de685274f0 2023-11-16T16:52:16.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.106+0000 7ffba61e5b80 1 -- 172.21.15.44:0/1629361420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de685274f0 msgr2=0x55de6852b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:16.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.106+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/1629361420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de685274f0 0x55de6852b3f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ffb80011d20 tx=0x7ffb80014000 comp rx=0 tx=0).stop 2023-11-16T16:52:16.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.106+0000 7ffba61e5b80 1 -- 172.21.15.44:0/1629361420 shutdown_connections 2023-11-16T16:52:16.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.106+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/1629361420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de6852b930 0x55de6852bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.106+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/1629361420 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de68526be0 0x55de68526fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.106+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/1629361420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de685274f0 0x55de6852b3f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.106+0000 7ffba61e5b80 1 -- 172.21.15.44:0/1629361420 >> 172.21.15.44:0/1629361420 conn(0x55de6851c3e0 msgr2=0x55de68521bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:16.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.106+0000 7ffba61e5b80 1 -- 172.21.15.44:0/1629361420 shutdown_connections 2023-11-16T16:52:16.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.106+0000 7ffba61e5b80 1 -- 172.21.15.44:0/1629361420 wait complete. 2023-11-16T16:52:16.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.107+0000 7ffba61e5b80 1 -- start start 2023-11-16T16:52:16.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.107+0000 7ffba61e5b80 1 Processor -- start 2023-11-16T16:52:16.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.108+0000 7ffba61e5b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de68526be0 0x55de68533090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.108+0000 7ffba61e5b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de685274f0 0x55de68530310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.108+0000 7ffba61e5b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de6852b930 0x55de68530990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.108+0000 7ffba61e5b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de68419960 con 0x55de685274f0 2023-11-16T16:52:16.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.108+0000 7ffba61e5b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de68419600 con 0x55de6852b930 2023-11-16T16:52:16.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.108+0000 7ffba61e5b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de682fb9a0 con 0x55de68526be0 2023-11-16T16:52:16.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.108+0000 7ffba61e5b80 10 client.0 _ll_register_callbacks cb 0x55de68437800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:16.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.108+0000 7ffb8bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de68526be0 0x55de68533090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.108+0000 7ffb8b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de685274f0 0x55de68530310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.108+0000 7ffb8bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de68526be0 0x55de68533090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:42272/0 (socket says 172.21.15.44:42272) 2023-11-16T16:52:16.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.108+0000 7ffb8bfff700 1 -- 172.21.15.44:0/2032812166 learned_addr learned my addr 172.21.15.44:0/2032812166 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:16.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.108+0000 7ffb90d9c700 1 --2- 172.21.15.44:0/2032812166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de6852b930 0x55de68530990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.108+0000 7ffb8bfff700 1 -- 172.21.15.44:0/2032812166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de6852b930 msgr2=0x55de68530990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:16.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.108+0000 7ffb8bfff700 1 --2- 172.21.15.44:0/2032812166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de6852b930 0x55de68530990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.109+0000 7ffb8bfff700 1 -- 172.21.15.44:0/2032812166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de685274f0 msgr2=0x55de68530310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:16.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.109+0000 7ffb8bfff700 1 --2- 172.21.15.44:0/2032812166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de685274f0 0x55de68530310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.109+0000 7ffb8bfff700 1 -- 172.21.15.44:0/2032812166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb80017070 con 0x55de68526be0 2023-11-16T16:52:16.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.109+0000 7ffb90d9c700 1 --2- 172.21.15.44:0/2032812166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de6852b930 0x55de68530990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:16.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.109+0000 7ffb8bfff700 1 --2- 172.21.15.44:0/2032812166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de68526be0 0x55de68533090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c0ca5f0 tx=0x7ffb7c099f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b016588cd5c88192 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:16.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.109+0000 7ffb6b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:52:16.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.109+0000 7ffb6b7fe700 1 -- 172.21.15.44:0/2032812166 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c099600 con 0x55de68526be0 2023-11-16T16:52:16.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.109+0000 7ffb6b7fe700 1 -- 172.21.15.44:0/2032812166 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7ffb7c0a2360 con 0x55de68526be0 2023-11-16T16:52:16.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.109+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2032812166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55de68519ba0 con 0x55de68526be0 2023-11-16T16:52:16.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.109+0000 7ffb6b7fe700 1 -- 172.21.15.44:0/2032812166 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb7c0b82b0 con 0x55de68526be0 2023-11-16T16:52:16.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.109+0000 7ffba61e5b80 10 client.26050 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:16.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.109+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2032812166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55de6843e210 con 0x55de68526be0 2023-11-16T16:52:16.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.110+0000 7ffb6b7fe700 1 -- 172.21.15.44:0/2032812166 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb7c0ca080 con 0x55de68526be0 2023-11-16T16:52:16.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.110+0000 7ffb6b7fe700 1 -- 172.21.15.44:0/2032812166 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ffb7c098e70 con 0x55de68526be0 2023-11-16T16:52:16.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.110+0000 7ffba61e5b80 10 client.26050 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:16.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.110+0000 7ffba61e5b80 10 client.26050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:16.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.110+0000 7ffba61e5b80 20 client.26050 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:16.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.110+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/2032812166 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55de68540840 0x55de68542c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.110+0000 7ffba61e5b80 4 client.26050 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:16.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.110+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2032812166 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de6843a120 con 0x55de68540840 2023-11-16T16:52:16.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.110+0000 7ffb8b7fe700 1 --2- 172.21.15.44:0/2032812166 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55de68540840 0x55de68542c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.111+0000 7ffb8b7fe700 1 --2- 172.21.15.44:0/2032812166 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55de68540840 0x55de68542c50 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68b6c729ccc18226 server_cookie=f03224d4975b2b71 in_seq=0 out_seq=0 2023-11-16T16:52:16.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.111+0000 7ffb6b7fe700 10 client.26050 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:16.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.112+0000 7ffb6b7fe700 1 -- 172.21.15.44:0/2032812166 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55de6843a120 con 0x55de68540840 2023-11-16T16:52:16.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.112+0000 7ffb6b7fe700 10 client.26050 handle_command_reply: tid=0 2023-11-16T16:52:16.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.112+0000 7ffba61e5b80 1 client.26050 shutdown 2023-11-16T16:52:16.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.112+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2032812166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de68526be0 msgr2=0x55de68533090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:16.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.112+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/2032812166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de68526be0 0x55de68533090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c0ca5f0 tx=0x7ffb7c099f90 comp rx=0 tx=0).stop 2023-11-16T16:52:16.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.112+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2032812166 shutdown_connections 2023-11-16T16:52:16.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.112+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/2032812166 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55de68540840 0x55de68542c50 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.112+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/2032812166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de6852b930 0x55de68530990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.112+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/2032812166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de68526be0 0x55de68533090 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.112+0000 7ffb6b7fe700 0 client.26050 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:16.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.112+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/2032812166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de685274f0 0x55de68530310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.112+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2032812166 >> 172.21.15.44:0/2032812166 conn(0x55de6851c3e0 msgr2=0x55de685259e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:16.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.112+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2032812166 shutdown_connections 2023-11-16T16:52:16.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.112+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2032812166 wait complete. 2023-11-16T16:52:16.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.112+0000 7ffba61e5b80 20 client.26050 trim_cache size 0 max 16384 2023-11-16T16:52:16.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.131+0000 7ffba61e5b80 1 Processor -- start 2023-11-16T16:52:16.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.132+0000 7ffba61e5b80 1 -- start start 2023-11-16T16:52:16.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.132+0000 7ffba61e5b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de68526be0 0x55de68537a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.132+0000 7ffba61e5b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de685274f0 0x55de6852dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.132+0000 7ffba61e5b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de6852b930 0x55de6852af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.132+0000 7ffba61e5b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de6851bf60 con 0x55de685274f0 2023-11-16T16:52:16.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.132+0000 7ffba61e5b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de685229d0 con 0x55de68526be0 2023-11-16T16:52:16.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.132+0000 7ffba61e5b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de6852a440 con 0x55de6852b930 2023-11-16T16:52:16.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.132+0000 7ffb8bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de68526be0 0x55de68537a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.132+0000 7ffb8b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de685274f0 0x55de6852dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.132+0000 7ffb8bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de68526be0 0x55de68537a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:33604/0 (socket says 172.21.15.44:33604) 2023-11-16T16:52:16.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.132+0000 7ffb8bfff700 1 -- 172.21.15.44:0/2975547774 learned_addr learned my addr 172.21.15.44:0/2975547774 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:16.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.133+0000 7ffb90d9c700 1 --2- 172.21.15.44:0/2975547774 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de6852b930 0x55de6852af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.133+0000 7ffb8bfff700 1 -- 172.21.15.44:0/2975547774 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de6852b930 msgr2=0x55de6852af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:16.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.133+0000 7ffb8bfff700 1 --2- 172.21.15.44:0/2975547774 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de6852b930 0x55de6852af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.133+0000 7ffb8bfff700 1 -- 172.21.15.44:0/2975547774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de685274f0 msgr2=0x55de6852dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:16.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.133+0000 7ffb8bfff700 1 --2- 172.21.15.44:0/2975547774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de685274f0 0x55de6852dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.133+0000 7ffb8bfff700 1 -- 172.21.15.44:0/2975547774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de6843df60 con 0x55de68526be0 2023-11-16T16:52:16.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.133+0000 7ffb8bfff700 1 --2- 172.21.15.44:0/2975547774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de68526be0 0x55de68537a20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c000c00 tx=0x7ffb7c010a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6881214c4986f6f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:16.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.133+0000 7ffb72ffd700 1 -- 172.21.15.44:0/2975547774 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c0a2360 con 0x55de68526be0 2023-11-16T16:52:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.133+0000 7ffb72ffd700 1 -- 172.21.15.44:0/2975547774 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55de68519ba0 con 0x55de68526be0 2023-11-16T16:52:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.133+0000 7ffb72ffd700 1 -- 172.21.15.44:0/2975547774 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c099660 con 0x55de68526be0 2023-11-16T16:52:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.134+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2975547774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de68526be0 msgr2=0x55de68537a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.134+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/2975547774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de68526be0 0x55de68537a20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c000c00 tx=0x7ffb7c010a90 comp rx=0 tx=0).stop 2023-11-16T16:52:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.134+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2975547774 shutdown_connections 2023-11-16T16:52:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.134+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/2975547774 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de6852b930 0x55de6852af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.134+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/2975547774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de68526be0 0x55de68537a20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.134+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/2975547774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de685274f0 0x55de6852dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.134+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2975547774 >> 172.21.15.44:0/2975547774 conn(0x55de6851c3e0 msgr2=0x55de6852a850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.134+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2975547774 shutdown_connections 2023-11-16T16:52:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.134+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2975547774 wait complete. 2023-11-16T16:52:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.135+0000 7ffba61e5b80 1 -- start start 2023-11-16T16:52:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.135+0000 7ffba61e5b80 1 Processor -- start 2023-11-16T16:52:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.136+0000 7ffba61e5b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de68526be0 0x55de685210d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.136+0000 7ffba61e5b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de685274f0 0x55de68521610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.136+0000 7ffba61e5b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de6852b930 0x55de68594700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.136+0000 7ffba61e5b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de68594c40 con 0x55de68526be0 2023-11-16T16:52:16.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.136+0000 7ffba61e5b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de68594d80 con 0x55de685274f0 2023-11-16T16:52:16.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.136+0000 7ffba61e5b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de68594ec0 con 0x55de6852b930 2023-11-16T16:52:16.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.136+0000 7ffba61e5b80 10 client.0 _ll_register_callbacks cb 0x55de68437800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:16.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.136+0000 7ffb8bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de68526be0 0x55de685210d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.136+0000 7ffb8bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de68526be0 0x55de685210d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57208/0 (socket says 172.21.15.44:57208) 2023-11-16T16:52:16.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.136+0000 7ffb8bfff700 1 -- 172.21.15.44:0/2508959639 learned_addr learned my addr 172.21.15.44:0/2508959639 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:16.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.136+0000 7ffb8b7fe700 1 --2- 172.21.15.44:0/2508959639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de685274f0 0x55de68521610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.136+0000 7ffb90d9c700 1 --2- 172.21.15.44:0/2508959639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de6852b930 0x55de68594700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.137+0000 7ffb8bfff700 1 -- 172.21.15.44:0/2508959639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de6852b930 msgr2=0x55de68594700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:16.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.137+0000 7ffb8bfff700 1 --2- 172.21.15.44:0/2508959639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de6852b930 0x55de68594700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.137+0000 7ffb8bfff700 1 -- 172.21.15.44:0/2508959639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de685274f0 msgr2=0x55de68521610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:16.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.137+0000 7ffb8bfff700 1 --2- 172.21.15.44:0/2508959639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de685274f0 0x55de68521610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.137+0000 7ffb8bfff700 1 -- 172.21.15.44:0/2508959639 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de6843e210 con 0x55de68526be0 2023-11-16T16:52:16.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.137+0000 7ffb8b7fe700 1 --2- 172.21.15.44:0/2508959639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de685274f0 0x55de68521610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:16.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.137+0000 7ffb8bfff700 1 --2- 172.21.15.44:0/2508959639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de68526be0 0x55de685210d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00e970 tx=0x7ffb7c099920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c0af17014f32f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:16.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.137+0000 7ffb6b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:52:16.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.137+0000 7ffb6b7fe700 1 -- 172.21.15.44:0/2508959639 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c099600 con 0x55de68526be0 2023-11-16T16:52:16.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.137+0000 7ffb6b7fe700 1 -- 172.21.15.44:0/2508959639 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7ffb7c0a2360 con 0x55de68526be0 2023-11-16T16:52:16.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.137+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2508959639 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb80021070 con 0x55de68526be0 2023-11-16T16:52:16.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.137+0000 7ffb6b7fe700 1 -- 172.21.15.44:0/2508959639 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb7c099f90 con 0x55de68526be0 2023-11-16T16:52:16.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.137+0000 7ffba61e5b80 10 client.16047 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:16.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.137+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2508959639 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55de6828ece0 con 0x55de68526be0 2023-11-16T16:52:16.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.138+0000 7ffb6b7fe700 1 -- 172.21.15.44:0/2508959639 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb80021070 con 0x55de68526be0 2023-11-16T16:52:16.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.138+0000 7ffb6b7fe700 1 -- 172.21.15.44:0/2508959639 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ffb7c00cce0 con 0x55de68526be0 2023-11-16T16:52:16.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.138+0000 7ffba61e5b80 10 client.16047 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:16.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.138+0000 7ffba61e5b80 10 client.16047 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:16.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.138+0000 7ffba61e5b80 20 client.16047 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:16.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.138+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/2508959639 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55de68596130 0x55de68598510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:16.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.138+0000 7ffba61e5b80 4 client.16047 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:52:16.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.138+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2508959639 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55de684466e0 con 0x55de68596130 2023-11-16T16:52:16.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.138+0000 7ffb8b7fe700 1 --2- 172.21.15.44:0/2508959639 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55de68596130 0x55de68598510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:16.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.139+0000 7ffb6b7fe700 10 client.16047 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:16.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.139+0000 7ffb8b7fe700 1 --2- 172.21.15.44:0/2508959639 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55de68596130 0x55de68598510 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=144cf3fd3f199641 server_cookie=1897fc7962218280 in_seq=0 out_seq=0 2023-11-16T16:52:16.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.140+0000 7ffb6b7fe700 1 -- 172.21.15.44:0/2508959639 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55de684466e0 con 0x55de68596130 2023-11-16T16:52:16.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.140+0000 7ffb6b7fe700 10 client.16047 handle_command_reply: tid=0 2023-11-16T16:52:16.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.140+0000 7ffba61e5b80 1 client.16047 shutdown 2023-11-16T16:52:16.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.140+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2508959639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de68526be0 msgr2=0x55de685210d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:16.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.140+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/2508959639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de68526be0 0x55de685210d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00e970 tx=0x7ffb7c099920 comp rx=0 tx=0).stop 2023-11-16T16:52:16.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.140+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2508959639 shutdown_connections 2023-11-16T16:52:16.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.140+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/2508959639 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55de68596130 0x55de68598510 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.140+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/2508959639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55de6852b930 0x55de68594700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.140+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/2508959639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de68526be0 0x55de685210d0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.140+0000 7ffba61e5b80 1 --2- 172.21.15.44:0/2508959639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de685274f0 0x55de68521610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.140+0000 7ffb6b7fe700 0 client.16047 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:16.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.140+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2508959639 >> 172.21.15.44:0/2508959639 conn(0x55de6851c3e0 msgr2=0x55de68526270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:16.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.141+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2508959639 shutdown_connections 2023-11-16T16:52:16.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.141+0000 7ffba61e5b80 1 -- 172.21.15.44:0/2508959639 wait complete. 2023-11-16T16:52:16.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.141+0000 7ffba61e5b80 20 client.16047 trim_cache size 0 max 16384 2023-11-16T16:52:16.318 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:52:16.318 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:52:16.319 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "120674eb-c29e-439b-8850-ac8247dd6145", 2023-11-16T16:52:16.319 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:52:16.319 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:52:16.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.142+0000 7ffb927ff700 1 -- 172.21.15.44:0/3092950580 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ffb740a1340 msgr2=0x7ffb740a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:16.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.142+0000 7ffb927ff700 1 --2- 172.21.15.44:0/3092950580 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ffb740a1340 0x7ffb740a37f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c199cf0 tx=0x7ffb80009260 comp rx=0 tx=0).stop 2023-11-16T16:52:16.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.142+0000 7ffb927ff700 1 -- 172.21.15.44:0/3092950580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb8c1024d0 msgr2=0x7ffb8c1986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:16.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.142+0000 7ffb927ff700 1 --2- 172.21.15.44:0/3092950580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb8c1024d0 0x7ffb8c1986e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00e9a0 tx=0x7ffb7c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.142+0000 7ffb927ff700 1 -- 172.21.15.44:0/3092950580 shutdown_connections 2023-11-16T16:52:16.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.142+0000 7ffb927ff700 1 --2- 172.21.15.44:0/3092950580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb8c1084e0 0x7ffb8c19bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.142+0000 7ffb927ff700 1 --2- 172.21.15.44:0/3092950580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffb8c1024d0 0x7ffb8c1986e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.142+0000 7ffb927ff700 1 --2- 172.21.15.44:0/3092950580 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ffb740a1340 0x7ffb740a37f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.142+0000 7ffb927ff700 1 --2- 172.21.15.44:0/3092950580 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb8c102e80 0x7ffb8c198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:16.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.142+0000 7ffb927ff700 1 -- 172.21.15.44:0/3092950580 >> 172.21.15.44:0/3092950580 conn(0x7ffb8c0fdfb0 msgr2=0x7ffb8c0ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:16.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.142+0000 7ffb927ff700 1 -- 172.21.15.44:0/3092950580 shutdown_connections 2023-11-16T16:52:16.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:11.142+0000 7ffb927ff700 1 -- 172.21.15.44:0/3092950580 wait complete. 2023-11-16T16:52:16.321 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:52:16.321 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:11 smithi044 ceph-mon[128750]: pgmap v70: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 38 KiB/s wr, 17 op/s 2023-11-16T16:52:16.323 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:16.323 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.323 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:16.323 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:16.323 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.323 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.323 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 59 -lt 100 ']' 2023-11-16T16:52:16.323 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 59 + 1 2023-11-16T16:52:16.323 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=60 2023-11-16T16:52:16.324 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:16.324 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:16.324 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.324 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:16.324 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:16.324 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:12 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:52:16.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:12 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:16.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:12 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:52:16.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:12 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:16.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:12 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:52:16.325 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:16.326 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.326 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:16.326 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:16.326 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.326 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.326 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 60 -lt 100 ']' 2023-11-16T16:52:16.326 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 60 + 1 2023-11-16T16:52:16.326 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=61 2023-11-16T16:52:16.326 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:16.326 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:16.326 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.326 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:16.327 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:16.327 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:13 smithi044 ceph-mon[128750]: pgmap v71: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 59 KiB/s wr, 26 op/s 2023-11-16T16:52:16.329 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:16.329 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.329 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:16.329 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:16.329 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.329 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.330 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 61 -lt 100 ']' 2023-11-16T16:52:16.330 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 61 + 1 2023-11-16T16:52:16.330 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=62 2023-11-16T16:52:16.330 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:16.330 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:16.330 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.330 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:16.330 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:16.330 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.330 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:16.330 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.330 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:16.330 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:16.330 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.330 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.331 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 62 -lt 100 ']' 2023-11-16T16:52:16.331 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 62 + 1 2023-11-16T16:52:16.331 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=63 2023-11-16T16:52:16.331 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:16.331 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:16.331 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.331 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:16.331 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:16.331 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.331 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:14 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:52:16.331 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:14 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:52:16.331 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:14 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:52:16.333 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:16.333 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.333 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:16.333 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:16.333 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.333 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.333 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 63 -lt 100 ']' 2023-11-16T16:52:16.333 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 63 + 1 2023-11-16T16:52:16.333 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=64 2023-11-16T16:52:16.333 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:16.333 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:16.334 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.334 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:16.334 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:16.334 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.334 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:15 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:52:16.334 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:15 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T16:52:16.334 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:15 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:52:16.334 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:15 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:52:16.334 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:15 smithi044 ceph-mon[128750]: pgmap v72: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-11-16T16:52:16.336 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:16.336 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.336 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:16.336 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:16.336 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.336 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.336 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 64 -lt 100 ']' 2023-11-16T16:52:16.336 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 64 + 1 2023-11-16T16:52:16.336 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=65 2023-11-16T16:52:16.336 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:16.336 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:16.336 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.336 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:16.336 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:16.337 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.353 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:16.354 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.464 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:16.465 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:16.472 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.472 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.472 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 65 -lt 100 ']' 2023-11-16T16:52:16.472 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 65 + 1 2023-11-16T16:52:16.502 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=66 2023-11-16T16:52:16.502 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:16.535 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:16.535 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.538 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:16.538 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:16.539 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:17.070 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:17.070 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:17.163 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:17.163 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:17.169 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:17.169 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:17.169 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 66 -lt 100 ']' 2023-11-16T16:52:17.169 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 66 + 1 2023-11-16T16:52:17.210 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=67 2023-11-16T16:52:17.211 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:17.253 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:17.253 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:17.255 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:17.256 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:17.256 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:17.602 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:17 smithi044 ceph-mon[128750]: pgmap v73: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-11-16T16:52:17.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:17 smithi176 ceph-mon[131630]: pgmap v73: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-11-16T16:52:17.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:17 smithi077 ceph-mon[131204]: pgmap v73: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-11-16T16:52:17.837 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:17.837 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:18.015 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:18.015 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:18.048 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:18.048 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:18.048 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 67 -lt 100 ']' 2023-11-16T16:52:18.048 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 67 + 1 2023-11-16T16:52:18.085 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=68 2023-11-16T16:52:18.086 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:18.139 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:18.139 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:18.142 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:18.142 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:18.143 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:19.045 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:19.045 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:19.291 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:19.291 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:19.331 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:19.331 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:19.331 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 68 -lt 100 ']' 2023-11-16T16:52:19.331 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 68 + 1 2023-11-16T16:52:19.402 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=69 2023-11-16T16:52:19.402 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:19.477 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:19.478 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:19.480 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:19.481 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:19.481 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:19.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:19 smithi044 ceph-mon[128750]: pgmap v74: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 12 op/s 2023-11-16T16:52:19.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:19 smithi176 ceph-mon[131630]: pgmap v74: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 12 op/s 2023-11-16T16:52:19.684 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.682+0000 7fa3d59ff700 1 -- 172.21.15.44:0/3975169198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3d0105620 msgr2=0x7fa3d010d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:19.684 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.682+0000 7fa3d59ff700 1 --2- 172.21.15.44:0/3975169198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3d0105620 0x7fa3d010d1a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800b0d0 tx=0x7fa3c800b3e0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.684 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.682+0000 7fa3d59ff700 1 -- 172.21.15.44:0/3975169198 shutdown_connections 2023-11-16T16:52:19.684 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.682+0000 7fa3d59ff700 1 --2- 172.21.15.44:0/3975169198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3d0105620 0x7fa3d010d1a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.684 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.682+0000 7fa3d59ff700 1 --2- 172.21.15.44:0/3975169198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3d0100cc0 0x7fa3d0101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.684 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.682+0000 7fa3d59ff700 1 --2- 172.21.15.44:0/3975169198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d0101660 0x7fa3d0104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.684 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.682+0000 7fa3d59ff700 1 -- 172.21.15.44:0/3975169198 >> 172.21.15.44:0/3975169198 conn(0x7fa3d0077560 msgr2=0x7fa3d0077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:19.685 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.682+0000 7fa3d59ff700 1 -- 172.21.15.44:0/3975169198 shutdown_connections 2023-11-16T16:52:19.685 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.682+0000 7fa3d59ff700 1 -- 172.21.15.44:0/3975169198 wait complete. 2023-11-16T16:52:19.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.683+0000 7fa3d59ff700 1 Processor -- start 2023-11-16T16:52:19.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.684+0000 7fa3d59ff700 1 -- start start 2023-11-16T16:52:19.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.684+0000 7fa3d59ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d0100cc0 0x7fa3d019e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.684+0000 7fa3d59ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3d0101660 0x7fa3d019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.684+0000 7fa3d59ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3d0105620 0x7fa3d0198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.684+0000 7fa3d59ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d0198c70 con 0x7fa3d0101660 2023-11-16T16:52:19.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.684+0000 7fa3d59ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d0198de0 con 0x7fa3d0100cc0 2023-11-16T16:52:19.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.684+0000 7fa3d59ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d0198fa0 con 0x7fa3d0105620 2023-11-16T16:52:19.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.684+0000 7fa3c7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3d0101660 0x7fa3d019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.684+0000 7fa3c7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3d0101660 0x7fa3d019ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60376/0 (socket says 172.21.15.44:60376) 2023-11-16T16:52:19.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.684+0000 7fa3c7fff700 1 -- 172.21.15.44:0/3440849534 learned_addr learned my addr 172.21.15.44:0/3440849534 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:19.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.684+0000 7fa3ceffd700 1 --2- 172.21.15.44:0/3440849534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d0100cc0 0x7fa3d019e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.684+0000 7fa3cf7fe700 1 --2- 172.21.15.44:0/3440849534 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3d0105620 0x7fa3d0198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.685+0000 7fa3c7fff700 1 -- 172.21.15.44:0/3440849534 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3d0105620 msgr2=0x7fa3d0198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:19.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.685+0000 7fa3c7fff700 1 --2- 172.21.15.44:0/3440849534 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3d0105620 0x7fa3d0198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.685+0000 7fa3c7fff700 1 -- 172.21.15.44:0/3440849534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d0100cc0 msgr2=0x7fa3d019e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:19.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.685+0000 7fa3c7fff700 1 --2- 172.21.15.44:0/3440849534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d0100cc0 0x7fa3d019e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.685+0000 7fa3c7fff700 1 -- 172.21.15.44:0/3440849534 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c8009d00 con 0x7fa3d0101660 2023-11-16T16:52:19.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.685+0000 7fa3c7fff700 1 --2- 172.21.15.44:0/3440849534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3d0101660 0x7fa3d019ead0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa3b800e910 tx=0x7fa3b800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:19.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.685+0000 7fa3d49fd700 1 -- 172.21.15.44:0/3440849534 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8004a30 con 0x7fa3d0101660 2023-11-16T16:52:19.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.685+0000 7fa3d49fd700 1 -- 172.21.15.44:0/3440849534 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3b80054e0 con 0x7fa3d0101660 2023-11-16T16:52:19.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.685+0000 7fa3d59ff700 1 -- 172.21.15.44:0/3440849534 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d01103e0 con 0x7fa3d0101660 2023-11-16T16:52:19.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.685+0000 7fa3d49fd700 1 -- 172.21.15.44:0/3440849534 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b800cc40 con 0x7fa3d0101660 2023-11-16T16:52:19.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.686+0000 7fa3d59ff700 1 -- 172.21.15.44:0/3440849534 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d0110250 con 0x7fa3d0101660 2023-11-16T16:52:19.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.689+0000 7fa3d49fd700 1 -- 172.21.15.44:0/3440849534 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fa3b8005000 con 0x7fa3d0101660 2023-11-16T16:52:19.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.691+0000 7fa3d49fd700 1 --2- 172.21.15.44:0/3440849534 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa3bc0a12c0 0x7fa3bc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.691+0000 7fa3d49fd700 1 -- 172.21.15.44:0/3440849534 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fa3b8014070 con 0x7fa3d0101660 2023-11-16T16:52:19.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.691+0000 7fa3ceffd700 1 --2- 172.21.15.44:0/3440849534 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa3bc0a12c0 0x7fa3bc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.692+0000 7fa3ceffd700 1 --2- 172.21.15.44:0/3440849534 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa3bc0a12c0 0x7fa3bc0a3770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0009c60 tx=0x7fa3c0009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:19.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:19 smithi077 ceph-mon[131204]: pgmap v74: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 12 op/s 2023-11-16T16:52:19.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.776+0000 7fa3e9320b80 1 Processor -- start 2023-11-16T16:52:19.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.776+0000 7fa3e9320b80 1 -- start start 2023-11-16T16:52:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.776+0000 7fa3e9320b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 0x55d0998d0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.776+0000 7fa3e9320b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d1470 0x55d0998d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.776+0000 7fa3e9320b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d0998d5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.776+0000 7fa3e9320b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0997d6060 con 0x55d0998d58b0 2023-11-16T16:52:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.776+0000 7fa3e9320b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0998288a0 con 0x55d0998d0b60 2023-11-16T16:52:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.776+0000 7fa3e9320b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d099821d30 con 0x55d0998d1470 2023-11-16T16:52:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.777+0000 7fa3cf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d0998d5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.777+0000 7fa3cf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d0998d5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60380/0 (socket says 172.21.15.44:60380) 2023-11-16T16:52:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.777+0000 7fa3cf7fe700 1 -- 172.21.15.44:0/3368479491 learned_addr learned my addr 172.21.15.44:0/3368479491 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.777+0000 7fa3ceffd700 1 --2- 172.21.15.44:0/3368479491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 0x55d0998d0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.777+0000 7fa3c7fff700 1 --2- 172.21.15.44:0/3368479491 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d1470 0x55d0998d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.777+0000 7fa3cf7fe700 1 -- 172.21.15.44:0/3368479491 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d1470 msgr2=0x55d0998d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:19.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.777+0000 7fa3cf7fe700 1 --2- 172.21.15.44:0/3368479491 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d1470 0x55d0998d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.777+0000 7fa3cf7fe700 1 -- 172.21.15.44:0/3368479491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 msgr2=0x55d0998d0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:19.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.777+0000 7fa3cf7fe700 1 --2- 172.21.15.44:0/3368479491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 0x55d0998d0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.777+0000 7fa3cf7fe700 1 -- 172.21.15.44:0/3368479491 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0996a3630 con 0x55d0998d58b0 2023-11-16T16:52:19.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.777+0000 7fa3c7fff700 1 --2- 172.21.15.44:0/3368479491 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d1470 0x55d0998d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:19.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.777+0000 7fa3ceffd700 1 --2- 172.21.15.44:0/3368479491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 0x55d0998d0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:19.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.777+0000 7fa3cf7fe700 1 --2- 172.21.15.44:0/3368479491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d0998d5d00 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8005b20 tx=0x7fa3c8005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de572316dc285e34 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:19.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.778+0000 7fa3c5ffb700 1 -- 172.21.15.44:0/3368479491 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c800e040 con 0x55d0998d58b0 2023-11-16T16:52:19.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.778+0000 7fa3c5ffb700 1 -- 172.21.15.44:0/3368479491 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3c80074e0 con 0x55d0998d58b0 2023-11-16T16:52:19.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.778+0000 7fa3c5ffb700 1 -- 172.21.15.44:0/3368479491 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8003de0 con 0x55d0998d58b0 2023-11-16T16:52:19.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.778+0000 7fa3e9320b80 1 -- 172.21.15.44:0/3368479491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 msgr2=0x55d0998d5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:19.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.778+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/3368479491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d0998d5d00 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8005b20 tx=0x7fa3c8005b50 comp rx=0 tx=0).stop 2023-11-16T16:52:19.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.779+0000 7fa3e9320b80 1 -- 172.21.15.44:0/3368479491 shutdown_connections 2023-11-16T16:52:19.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.779+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/3368479491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d0998d5d00 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.779+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/3368479491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 0x55d0998d0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.779+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/3368479491 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d1470 0x55d0998d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.779+0000 7fa3e9320b80 1 -- 172.21.15.44:0/3368479491 >> 172.21.15.44:0/3368479491 conn(0x55d0998c6360 msgr2=0x55d0998cbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:19.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.779+0000 7fa3e9320b80 1 -- 172.21.15.44:0/3368479491 shutdown_connections 2023-11-16T16:52:19.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.779+0000 7fa3e9320b80 1 -- 172.21.15.44:0/3368479491 wait complete. 2023-11-16T16:52:19.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.779+0000 7fa3e9320b80 1 -- start start 2023-11-16T16:52:19.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.780+0000 7fa3e9320b80 1 Processor -- start 2023-11-16T16:52:19.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.780+0000 7fa3e9320b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 0x55d0998dcfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.780+0000 7fa3e9320b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d1470 0x55d0998da2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.781+0000 7fa3e9320b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d0998da8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.781+0000 7fa3e9320b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0997c38e0 con 0x55d0998d58b0 2023-11-16T16:52:19.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.781+0000 7fa3e9320b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0997c3580 con 0x55d0998d0b60 2023-11-16T16:52:19.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.781+0000 7fa3e9320b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0996a5920 con 0x55d0998d1470 2023-11-16T16:52:19.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.781+0000 7fa3e9320b80 10 client.0 _ll_register_callbacks cb 0x55d0997e1780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:19.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.781+0000 7fa3cf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d0998da8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.781+0000 7fa3cf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d0998da8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60394/0 (socket says 172.21.15.44:60394) 2023-11-16T16:52:19.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.781+0000 7fa3cf7fe700 1 -- 172.21.15.44:0/1220953954 learned_addr learned my addr 172.21.15.44:0/1220953954 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:19.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.781+0000 7fa3c7fff700 1 --2- 172.21.15.44:0/1220953954 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d1470 0x55d0998da2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.781+0000 7fa3ceffd700 1 --2- 172.21.15.44:0/1220953954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 0x55d0998dcfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.781+0000 7fa3cf7fe700 1 -- 172.21.15.44:0/1220953954 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d1470 msgr2=0x55d0998da2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:19.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.781+0000 7fa3cf7fe700 1 --2- 172.21.15.44:0/1220953954 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d1470 0x55d0998da2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.781+0000 7fa3cf7fe700 1 -- 172.21.15.44:0/1220953954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 msgr2=0x55d0998dcfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:19.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.781+0000 7fa3cf7fe700 1 --2- 172.21.15.44:0/1220953954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 0x55d0998dcfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.781+0000 7fa3cf7fe700 1 -- 172.21.15.44:0/1220953954 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d010d9b0 con 0x55d0998d58b0 2023-11-16T16:52:19.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.782+0000 7fa3cf7fe700 1 --2- 172.21.15.44:0/1220953954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d0998da8e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80095c0 tx=0x7fa3c800bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b503da834c0d2f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:19.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.782+0000 7fa3aeffd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:52:19.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.782+0000 7fa3aeffd700 1 -- 172.21.15.44:0/1220953954 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c800e030 con 0x55d0998d58b0 2023-11-16T16:52:19.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.782+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1220953954 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d0998c3b20 con 0x55d0998d58b0 2023-11-16T16:52:19.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.782+0000 7fa3aeffd700 1 -- 172.21.15.44:0/1220953954 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fa3c8003e10 con 0x55d0998d58b0 2023-11-16T16:52:19.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.782+0000 7fa3aeffd700 1 -- 172.21.15.44:0/1220953954 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3c8006160 con 0x55d0998d58b0 2023-11-16T16:52:19.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.782+0000 7fa3e9320b80 10 client.16059 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:19.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.782+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1220953954 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55d0997e8190 con 0x55d0998d58b0 2023-11-16T16:52:19.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.782+0000 7fa3aeffd700 1 -- 172.21.15.44:0/1220953954 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d0998c3b20 con 0x55d0998d58b0 2023-11-16T16:52:19.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.783+0000 7fa3aeffd700 1 -- 172.21.15.44:0/1220953954 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa3c800b100 con 0x55d0998d58b0 2023-11-16T16:52:19.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.783+0000 7fa3e9320b80 10 client.16059 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:19.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.783+0000 7fa3e9320b80 10 client.16059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:19.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.783+0000 7fa3e9320b80 20 client.16059 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:19.786 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.783+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/1220953954 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d0998ea460 0x55d0998ec840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.783+0000 7fa3e9320b80 4 client.16059 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:19.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.783+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1220953954 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d0997e40a0 con 0x55d0998ea460 2023-11-16T16:52:19.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.783+0000 7fa3ceffd700 1 --2- 172.21.15.44:0/1220953954 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d0998ea460 0x55d0998ec840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.784+0000 7fa3ceffd700 1 --2- 172.21.15.44:0/1220953954 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d0998ea460 0x55d0998ec840 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aab46f19ad04e629 server_cookie=442434623948a41 in_seq=0 out_seq=0 2023-11-16T16:52:19.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.784+0000 7fa3aeffd700 10 client.16059 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:19.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.785+0000 7fa3aeffd700 1 -- 172.21.15.44:0/1220953954 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d0997e40a0 con 0x55d0998ea460 2023-11-16T16:52:19.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.785+0000 7fa3aeffd700 10 client.16059 handle_command_reply: tid=0 2023-11-16T16:52:19.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.785+0000 7fa3e9320b80 1 client.16059 shutdown 2023-11-16T16:52:19.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.785+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1220953954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 msgr2=0x55d0998da8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:19.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.785+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/1220953954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d0998da8e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80095c0 tx=0x7fa3c800bcd0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.785+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1220953954 shutdown_connections 2023-11-16T16:52:19.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.785+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/1220953954 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d0998ea460 0x55d0998ec840 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.785+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/1220953954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d0998da8e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.785+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/1220953954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 0x55d0998dcfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.785+0000 7fa3aeffd700 0 client.16059 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:19.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.785+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/1220953954 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d1470 0x55d0998da2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.785+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1220953954 >> 172.21.15.44:0/1220953954 conn(0x55d0998c6360 msgr2=0x55d0998d6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:19.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.785+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1220953954 shutdown_connections 2023-11-16T16:52:19.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.785+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1220953954 wait complete. 2023-11-16T16:52:19.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.786+0000 7fa3e9320b80 20 client.16059 trim_cache size 0 max 16384 2023-11-16T16:52:19.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.803+0000 7fa3e9320b80 1 Processor -- start 2023-11-16T16:52:19.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.803+0000 7fa3e9320b80 1 -- start start 2023-11-16T16:52:19.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.803+0000 7fa3e9320b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 0x55d0998d38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.803+0000 7fa3e9320b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d1470 0x55d0998d3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.803+0000 7fa3e9320b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d58b0 0x55d0998cb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.803+0000 7fa3e9320b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0998d1bc0 con 0x55d0998d1470 2023-11-16T16:52:19.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.803+0000 7fa3e9320b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0998cd1c0 con 0x55d0998d0b60 2023-11-16T16:52:19.805 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.803+0000 7fa3e9320b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0998d8830 con 0x55d0998d58b0 2023-11-16T16:52:19.806 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.804+0000 7fa3c7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d1470 0x55d0998d3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.806 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.804+0000 7fa3c7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d1470 0x55d0998d3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60408/0 (socket says 172.21.15.44:60408) 2023-11-16T16:52:19.806 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.804+0000 7fa3c7fff700 1 -- 172.21.15.44:0/1334238931 learned_addr learned my addr 172.21.15.44:0/1334238931 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:19.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.804+0000 7fa3ceffd700 1 --2- 172.21.15.44:0/1334238931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 0x55d0998d38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.804+0000 7fa3cf7fe700 1 --2- 172.21.15.44:0/1334238931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d58b0 0x55d0998cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.804+0000 7fa3c7fff700 1 -- 172.21.15.44:0/1334238931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d58b0 msgr2=0x55d0998cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:19.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.804+0000 7fa3c7fff700 1 --2- 172.21.15.44:0/1334238931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d58b0 0x55d0998cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.804+0000 7fa3c7fff700 1 -- 172.21.15.44:0/1334238931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 msgr2=0x55d0998d38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:19.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.804+0000 7fa3c7fff700 1 --2- 172.21.15.44:0/1334238931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 0x55d0998d38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.804+0000 7fa3c7fff700 1 -- 172.21.15.44:0/1334238931 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0998c7b00 con 0x55d0998d1470 2023-11-16T16:52:19.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.804+0000 7fa3ceffd700 1 --2- 172.21.15.44:0/1334238931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 0x55d0998d38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:19.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.804+0000 7fa3c7fff700 1 --2- 172.21.15.44:0/1334238931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d1470 0x55d0998d3e30 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8096520 tx=0x7fa3b8096360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69193ea5cc5c8063 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:19.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.804+0000 7fa3c67fc700 1 -- 172.21.15.44:0/1334238931 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8005650 con 0x55d0998d1470 2023-11-16T16:52:19.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.805+0000 7fa3c67fc700 1 -- 172.21.15.44:0/1334238931 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3b809a020 con 0x55d0998d1470 2023-11-16T16:52:19.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.805+0000 7fa3c67fc700 1 -- 172.21.15.44:0/1334238931 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8004b90 con 0x55d0998d1470 2023-11-16T16:52:19.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.805+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1334238931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d1470 msgr2=0x55d0998d3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:19.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.805+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/1334238931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d1470 0x55d0998d3e30 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8096520 tx=0x7fa3b8096360 comp rx=0 tx=0).stop 2023-11-16T16:52:19.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.805+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1334238931 shutdown_connections 2023-11-16T16:52:19.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.805+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/1334238931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d58b0 0x55d0998cb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.805+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/1334238931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d0b60 0x55d0998d38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.805+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/1334238931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d1470 0x55d0998d3e30 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.805+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1334238931 >> 172.21.15.44:0/1334238931 conn(0x55d0998c6360 msgr2=0x55d0998e6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:19.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.805+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1334238931 shutdown_connections 2023-11-16T16:52:19.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.805+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1334238931 wait complete. 2023-11-16T16:52:19.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.806+0000 7fa3e9320b80 1 -- start start 2023-11-16T16:52:19.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.806+0000 7fa3e9320b80 1 Processor -- start 2023-11-16T16:52:19.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.807+0000 7fa3e9320b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d0b60 0x55d099937670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.807+0000 7fa3e9320b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d1470 0x55d099937bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.807+0000 7fa3e9320b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d099932250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.807+0000 7fa3e9320b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d099932830 con 0x55d0998d58b0 2023-11-16T16:52:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.807+0000 7fa3e9320b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d099932970 con 0x55d0998d1470 2023-11-16T16:52:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.807+0000 7fa3e9320b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d099932b30 con 0x55d0998d0b60 2023-11-16T16:52:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.807+0000 7fa3e9320b80 10 client.0 _ll_register_callbacks cb 0x7fa3c00096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.807+0000 7fa3cf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d099932250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.807+0000 7fa3c7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d1470 0x55d099937bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.807+0000 7fa3c7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d1470 0x55d099937bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:54608/0 (socket says 172.21.15.44:54608) 2023-11-16T16:52:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.807+0000 7fa3c7fff700 1 -- 172.21.15.44:0/1690839480 learned_addr learned my addr 172.21.15.44:0/1690839480 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.807+0000 7fa3cf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d099932250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60416/0 (socket says 172.21.15.44:60416) 2023-11-16T16:52:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.807+0000 7fa3ceffd700 1 --2- 172.21.15.44:0/1690839480 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d0b60 0x55d099937670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.808+0000 7fa3cf7fe700 1 -- 172.21.15.44:0/1690839480 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d0b60 msgr2=0x55d099937670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.808+0000 7fa3cf7fe700 1 --2- 172.21.15.44:0/1690839480 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d0b60 0x55d099937670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.808+0000 7fa3cf7fe700 1 -- 172.21.15.44:0/1690839480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d1470 msgr2=0x55d099937bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:19.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.808+0000 7fa3cf7fe700 1 --2- 172.21.15.44:0/1690839480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d1470 0x55d099937bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.808+0000 7fa3cf7fe700 1 -- 172.21.15.44:0/1690839480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c0009e60 con 0x55d0998d58b0 2023-11-16T16:52:19.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.808+0000 7fa3c7fff700 1 --2- 172.21.15.44:0/1690839480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d1470 0x55d099937bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:19.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.808+0000 7fa3cf7fe700 1 --2- 172.21.15.44:0/1690839480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d099932250 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009a20 tx=0x7fa3c8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=133ea1ef9cf8b31f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:19.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.808+0000 7fa3aeffd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:52:19.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.808+0000 7fa3aeffd700 1 -- 172.21.15.44:0/1690839480 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c800e040 con 0x55d0998d58b0 2023-11-16T16:52:19.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.808+0000 7fa3aeffd700 1 -- 172.21.15.44:0/1690839480 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fa3c8025ad0 con 0x55d0998d58b0 2023-11-16T16:52:19.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.808+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1690839480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa3c80074e0 con 0x55d0998d58b0 2023-11-16T16:52:19.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.808+0000 7fa3aeffd700 1 -- 172.21.15.44:0/1690839480 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3c8006160 con 0x55d0998d58b0 2023-11-16T16:52:19.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.809+0000 7fa3e9320b80 10 client.16065 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:19.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.809+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1690839480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x7fa3d01103e0 con 0x55d0998d58b0 2023-11-16T16:52:19.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.809+0000 7fa3aeffd700 1 -- 172.21.15.44:0/1690839480 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3c80074e0 con 0x55d0998d58b0 2023-11-16T16:52:19.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.809+0000 7fa3aeffd700 1 -- 172.21.15.44:0/1690839480 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa3c801c410 con 0x55d0998d58b0 2023-11-16T16:52:19.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.809+0000 7fa3e9320b80 10 client.16065 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:19.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.809+0000 7fa3e9320b80 10 client.16065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:19.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.809+0000 7fa3e9320b80 20 client.16065 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:19.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.809+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/1690839480 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d09993ef60 0x55d099941340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:19.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.809+0000 7fa3e9320b80 4 client.16065 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:52:19.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.809+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1690839480 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d0997f0660 con 0x55d09993ef60 2023-11-16T16:52:19.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.809+0000 7fa3ceffd700 1 --2- 172.21.15.44:0/1690839480 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d09993ef60 0x55d099941340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:19.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.810+0000 7fa3ceffd700 1 --2- 172.21.15.44:0/1690839480 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d09993ef60 0x55d099941340 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f09bb28ab9624479 server_cookie=ebfbb3ac83b6f98f in_seq=0 out_seq=0 2023-11-16T16:52:19.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.810+0000 7fa3aeffd700 10 client.16065 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:19.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.810+0000 7fa3aeffd700 1 -- 172.21.15.44:0/1690839480 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d0997f0660 con 0x55d09993ef60 2023-11-16T16:52:19.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.810+0000 7fa3aeffd700 10 client.16065 handle_command_reply: tid=0 2023-11-16T16:52:19.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.810+0000 7fa3e9320b80 1 client.16065 shutdown 2023-11-16T16:52:19.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.810+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1690839480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 msgr2=0x55d099932250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:19.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.810+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/1690839480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d099932250 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009a20 tx=0x7fa3c8009780 comp rx=0 tx=0).stop 2023-11-16T16:52:19.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.811+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1690839480 shutdown_connections 2023-11-16T16:52:19.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.811+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/1690839480 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d09993ef60 0x55d099941340 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.811+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/1690839480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0998d58b0 0x55d099932250 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.811+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/1690839480 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0998d0b60 0x55d099937670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.811+0000 7fa3aeffd700 0 client.16065 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:19.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.811+0000 7fa3e9320b80 1 --2- 172.21.15.44:0/1690839480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0998d1470 0x55d099937bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.811+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1690839480 >> 172.21.15.44:0/1690839480 conn(0x55d0998c6360 msgr2=0x55d0998d38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:19.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.811+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1690839480 shutdown_connections 2023-11-16T16:52:19.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.811+0000 7fa3e9320b80 1 -- 172.21.15.44:0/1690839480 wait complete. 2023-11-16T16:52:19.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.811+0000 7fa3e9320b80 20 client.16065 trim_cache size 0 max 16384 2023-11-16T16:52:19.816 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:52:19.816 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T16:52:19.816 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T16:52:19.816 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:52:19.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.811+0000 7fa3d59ff700 1 -- 172.21.15.44:0/3440849534 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa3bc0a12c0 msgr2=0x7fa3bc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:19.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.811+0000 7fa3d59ff700 1 --2- 172.21.15.44:0/3440849534 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa3bc0a12c0 0x7fa3bc0a3770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0009c60 tx=0x7fa3c0009260 comp rx=0 tx=0).stop 2023-11-16T16:52:19.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.811+0000 7fa3d59ff700 1 -- 172.21.15.44:0/3440849534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3d0101660 msgr2=0x7fa3d019ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:19.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.811+0000 7fa3d59ff700 1 --2- 172.21.15.44:0/3440849534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3d0101660 0x7fa3d019ead0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa3b800e910 tx=0x7fa3b800ec20 comp rx=0 tx=0).stop 2023-11-16T16:52:19.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.812+0000 7fa3d59ff700 1 -- 172.21.15.44:0/3440849534 shutdown_connections 2023-11-16T16:52:19.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.812+0000 7fa3d59ff700 1 --2- 172.21.15.44:0/3440849534 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa3bc0a12c0 0x7fa3bc0a3770 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.812+0000 7fa3d59ff700 1 --2- 172.21.15.44:0/3440849534 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3d0105620 0x7fa3d0198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.812+0000 7fa3d59ff700 1 --2- 172.21.15.44:0/3440849534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d0100cc0 0x7fa3d019e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.812+0000 7fa3d59ff700 1 --2- 172.21.15.44:0/3440849534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3d0101660 0x7fa3d019ead0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:19.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.812+0000 7fa3d59ff700 1 -- 172.21.15.44:0/3440849534 >> 172.21.15.44:0/3440849534 conn(0x7fa3d0077560 msgr2=0x7fa3d00feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:19.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.812+0000 7fa3d59ff700 1 -- 172.21.15.44:0/3440849534 shutdown_connections 2023-11-16T16:52:19.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:19.812+0000 7fa3d59ff700 1 -- 172.21.15.44:0/3440849534 wait complete. 2023-11-16T16:52:19.873 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T16:52:19.873 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:52:20.428 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:20.428 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:20.614 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:20.614 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:20.631 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:20.631 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:20.631 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 69 -lt 100 ']' 2023-11-16T16:52:20.632 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 69 + 1 2023-11-16T16:52:20.718 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=70 2023-11-16T16:52:20.719 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:20.782 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:20.782 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:20.785 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:20.786 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:20.786 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:21.478 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:21.478 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:21.579 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:21.579 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:21.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:21 smithi044 ceph-mon[128750]: pgmap v75: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-11-16T16:52:21.621 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:21.621 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:21.621 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 70 -lt 100 ']' 2023-11-16T16:52:21.621 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 70 + 1 2023-11-16T16:52:21.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:21 smithi176 ceph-mon[131630]: pgmap v75: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-11-16T16:52:21.693 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=71 2023-11-16T16:52:21.693 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:21.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:21 smithi077 ceph-mon[131204]: pgmap v75: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-11-16T16:52:21.754 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:21.754 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:21.757 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:21.757 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:21.758 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:23.336 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:23.336 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:23.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.357+0000 7f3190dff700 1 -- 172.21.15.44:0/3499214641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f318c102480 msgr2=0x7f318c1028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.357+0000 7f3190dff700 1 --2- 172.21.15.44:0/3499214641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f318c102480 0x7f318c1028f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3174009990 tx=0x7f3174009ca0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.357+0000 7f3190dff700 1 -- 172.21.15.44:0/3499214641 shutdown_connections 2023-11-16T16:52:23.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.357+0000 7f3190dff700 1 --2- 172.21.15.44:0/3499214641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f318c102e30 0x7f318c10f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.357+0000 7f3190dff700 1 --2- 172.21.15.44:0/3499214641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f318c108490 0x7f318c108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.357+0000 7f3190dff700 1 --2- 172.21.15.44:0/3499214641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f318c102480 0x7f318c1028f0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.357+0000 7f3190dff700 1 -- 172.21.15.44:0/3499214641 >> 172.21.15.44:0/3499214641 conn(0x7f318c0fdf60 msgr2=0x7f318c100370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.357+0000 7f3190dff700 1 -- 172.21.15.44:0/3499214641 shutdown_connections 2023-11-16T16:52:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.358+0000 7f3190dff700 1 -- 172.21.15.44:0/3499214641 wait complete. 2023-11-16T16:52:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.359+0000 7f3190dff700 1 Processor -- start 2023-11-16T16:52:23.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.359+0000 7f3190dff700 1 -- start start 2023-11-16T16:52:23.362 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.360+0000 7f3190dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f318c102480 0x7f318c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.362 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.360+0000 7f3190dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f318c102e30 0x7f318c198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.362 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.360+0000 7f3190dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f318c108490 0x7f318c19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.362 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.360+0000 7f3190dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318c19d610 con 0x7f318c102480 2023-11-16T16:52:23.362 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.360+0000 7f3190dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318c19d780 con 0x7f318c108490 2023-11-16T16:52:23.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.360+0000 7f3190dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318c19d940 con 0x7f318c102e30 2023-11-16T16:52:23.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.360+0000 7f318a59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f318c102480 0x7f318c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.360+0000 7f318a59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f318c102480 0x7f318c198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60432/0 (socket says 172.21.15.44:60432) 2023-11-16T16:52:23.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.360+0000 7f318a59c700 1 -- 172.21.15.44:0/150566775 learned_addr learned my addr 172.21.15.44:0/150566775 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:23.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.361+0000 7f3183fff700 1 --2- 172.21.15.44:0/150566775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f318c102e30 0x7f318c198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.361+0000 7f318ad9d700 1 --2- 172.21.15.44:0/150566775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f318c108490 0x7f318c19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.361+0000 7f318a59c700 1 -- 172.21.15.44:0/150566775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f318c102e30 msgr2=0x7f318c198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.361+0000 7f318a59c700 1 --2- 172.21.15.44:0/150566775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f318c102e30 0x7f318c198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.361+0000 7f318a59c700 1 -- 172.21.15.44:0/150566775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f318c108490 msgr2=0x7f318c19cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.361+0000 7f318a59c700 1 --2- 172.21.15.44:0/150566775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f318c108490 0x7f318c19cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.361+0000 7f318a59c700 1 -- 172.21.15.44:0/150566775 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31740095f0 con 0x7f318c102480 2023-11-16T16:52:23.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.361+0000 7f318a59c700 1 --2- 172.21.15.44:0/150566775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f318c102480 0x7f318c198630 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f317c00e9a0 tx=0x7f317c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:23.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.361+0000 7f31837fe700 1 -- 172.21.15.44:0/150566775 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317c009790 con 0x7f318c102480 2023-11-16T16:52:23.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.362+0000 7f31837fe700 1 -- 172.21.15.44:0/150566775 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f317c010430 con 0x7f318c102480 2023-11-16T16:52:23.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.362+0000 7f31837fe700 1 -- 172.21.15.44:0/150566775 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317c018680 con 0x7f318c102480 2023-11-16T16:52:23.365 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.362+0000 7f3190dff700 1 -- 172.21.15.44:0/150566775 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318c1124d0 con 0x7f318c102480 2023-11-16T16:52:23.365 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.362+0000 7f3190dff700 1 -- 172.21.15.44:0/150566775 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318c112340 con 0x7f318c102480 2023-11-16T16:52:23.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.365+0000 7f31837fe700 1 -- 172.21.15.44:0/150566775 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f317c0105a0 con 0x7f318c102480 2023-11-16T16:52:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.367+0000 7f31837fe700 1 --2- 172.21.15.44:0/150566775 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f31780a12c0 0x7f31780a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.367+0000 7f31837fe700 1 -- 172.21.15.44:0/150566775 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f317c014070 con 0x7f318c102480 2023-11-16T16:52:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.367+0000 7f3183fff700 1 --2- 172.21.15.44:0/150566775 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f31780a12c0 0x7f31780a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.368+0000 7f3183fff700 1 --2- 172.21.15.44:0/150566775 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f31780a12c0 0x7f31780a3770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f318c199c40 tx=0x7f3174005fb0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:23.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.442+0000 7f31a476ab80 1 Processor -- start 2023-11-16T16:52:23.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.442+0000 7f31a476ab80 1 -- start start 2023-11-16T16:52:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.443+0000 7f31a476ab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 0x55bff4df51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.443+0000 7f31a476ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4def560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.443+0000 7f31a476ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 0x55bff4defb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.443+0000 7f31a476ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bff4cf5000 con 0x55bff4df5b90 2023-11-16T16:52:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.443+0000 7f31a476ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bff4d47840 con 0x55bff4df5720 2023-11-16T16:52:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.443+0000 7f31a476ab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bff4d40cd0 con 0x55bff4c4b6b0 2023-11-16T16:52:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.443+0000 7f318a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 0x55bff4df51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.443+0000 7f318a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 0x55bff4df51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:59846/0 (socket says 172.21.15.44:59846) 2023-11-16T16:52:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.443+0000 7f318a59c700 1 -- 172.21.15.44:0/3917184682 learned_addr learned my addr 172.21.15.44:0/3917184682 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.443+0000 7f318ad9d700 1 --2- 172.21.15.44:0/3917184682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 0x55bff4defb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.443+0000 7f3183fff700 1 --2- 172.21.15.44:0/3917184682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4def560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.444+0000 7f318a59c700 1 -- 172.21.15.44:0/3917184682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 msgr2=0x55bff4def560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:23.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.444+0000 7f318a59c700 1 --2- 172.21.15.44:0/3917184682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4def560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.444+0000 7f318a59c700 1 -- 172.21.15.44:0/3917184682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 msgr2=0x55bff4defb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:23.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.444+0000 7f318a59c700 1 --2- 172.21.15.44:0/3917184682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 0x55bff4defb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.444+0000 7f318a59c700 1 -- 172.21.15.44:0/3917184682 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bff4bc25d0 con 0x55bff4c4b6b0 2023-11-16T16:52:23.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.444+0000 7f318a59c700 1 --2- 172.21.15.44:0/3917184682 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 0x55bff4df51e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f317c096990 tx=0x7f317c099970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=779c6ed79ff213ba server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:23.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.444+0000 7f3180ff9700 1 -- 172.21.15.44:0/3917184682 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317c0b1070 con 0x55bff4c4b6b0 2023-11-16T16:52:23.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.445+0000 7f3180ff9700 1 -- 172.21.15.44:0/3917184682 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f317c0a2370 con 0x55bff4c4b6b0 2023-11-16T16:52:23.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.445+0000 7f3180ff9700 1 -- 172.21.15.44:0/3917184682 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317c099670 con 0x55bff4c4b6b0 2023-11-16T16:52:23.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.445+0000 7f31a476ab80 1 -- 172.21.15.44:0/3917184682 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 msgr2=0x55bff4df51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.445+0000 7f31a476ab80 1 --2- 172.21.15.44:0/3917184682 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 0x55bff4df51e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f317c096990 tx=0x7f317c099970 comp rx=0 tx=0).stop 2023-11-16T16:52:23.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.445+0000 7f31a476ab80 1 -- 172.21.15.44:0/3917184682 shutdown_connections 2023-11-16T16:52:23.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.446+0000 7f31a476ab80 1 --2- 172.21.15.44:0/3917184682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 0x55bff4defb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.446+0000 7f31a476ab80 1 --2- 172.21.15.44:0/3917184682 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 0x55bff4df51e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.446+0000 7f31a476ab80 1 --2- 172.21.15.44:0/3917184682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4def560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.446+0000 7f31a476ab80 1 -- 172.21.15.44:0/3917184682 >> 172.21.15.44:0/3917184682 conn(0x55bff4de4f50 msgr2=0x55bff4dea270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:23.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.446+0000 7f31a476ab80 1 -- 172.21.15.44:0/3917184682 shutdown_connections 2023-11-16T16:52:23.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.446+0000 7f31a476ab80 1 -- 172.21.15.44:0/3917184682 wait complete. 2023-11-16T16:52:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.474+0000 7f31a476ab80 1 -- start start 2023-11-16T16:52:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.474+0000 7f31a476ab80 1 Processor -- start 2023-11-16T16:52:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.475+0000 7f31a476ab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 0x55bff4dfbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.475+0000 7f31a476ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4df8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.475+0000 7f31a476ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 0x55bff4df9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.475+0000 7f31a476ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bff4ce2880 con 0x55bff4df5b90 2023-11-16T16:52:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.475+0000 7f31a476ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bff4ce2520 con 0x55bff4df5720 2023-11-16T16:52:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.475+0000 7f31a476ab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bff4bc48c0 con 0x55bff4c4b6b0 2023-11-16T16:52:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.475+0000 7f31a476ab80 10 client.0 _ll_register_callbacks cb 0x55bff4d00720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.475+0000 7f318ad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 0x55bff4df9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.475+0000 7f318ad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 0x55bff4df9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60444/0 (socket says 172.21.15.44:60444) 2023-11-16T16:52:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.475+0000 7f318ad9d700 1 -- 172.21.15.44:0/1492965415 learned_addr learned my addr 172.21.15.44:0/1492965415 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.475+0000 7f318a59c700 1 --2- 172.21.15.44:0/1492965415 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 0x55bff4dfbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.475+0000 7f3183fff700 1 --2- 172.21.15.44:0/1492965415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4df8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.475+0000 7f318a59c700 1 -- 172.21.15.44:0/1492965415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 msgr2=0x55bff4df8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.475+0000 7f318a59c700 1 --2- 172.21.15.44:0/1492965415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4df8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.476+0000 7f318a59c700 1 -- 172.21.15.44:0/1492965415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 msgr2=0x55bff4df9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.476+0000 7f318a59c700 1 --2- 172.21.15.44:0/1492965415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 0x55bff4df9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.476+0000 7f318a59c700 1 -- 172.21.15.44:0/1492965415 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318c112340 con 0x55bff4c4b6b0 2023-11-16T16:52:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.476+0000 7f318ad9d700 1 --2- 172.21.15.44:0/1492965415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 0x55bff4df9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.476+0000 7f318a59c700 1 --2- 172.21.15.44:0/1492965415 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 0x55bff4dfbaf0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f317c096990 tx=0x7f317c0ca510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3e8de4f2b167c29 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.476+0000 7f3171ffb700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:52:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.476+0000 7f3171ffb700 1 -- 172.21.15.44:0/1492965415 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317c0b1020 con 0x55bff4c4b6b0 2023-11-16T16:52:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.476+0000 7f3171ffb700 1 -- 172.21.15.44:0/1492965415 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f317c010940 con 0x55bff4c4b6b0 2023-11-16T16:52:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.476+0000 7f31a476ab80 1 -- 172.21.15.44:0/1492965415 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bff4de2710 con 0x55bff4c4b6b0 2023-11-16T16:52:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.476+0000 7f3171ffb700 1 -- 172.21.15.44:0/1492965415 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f317c0a1ee0 con 0x55bff4c4b6b0 2023-11-16T16:52:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.477+0000 7f31a476ab80 10 client.26086 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.477+0000 7f31a476ab80 1 -- 172.21.15.44:0/1492965415 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55bff4d07130 con 0x55bff4c4b6b0 2023-11-16T16:52:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.477+0000 7f3171ffb700 1 -- 172.21.15.44:0/1492965415 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bff4de2710 con 0x55bff4c4b6b0 2023-11-16T16:52:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.477+0000 7f3171ffb700 1 -- 172.21.15.44:0/1492965415 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f317c0990f0 con 0x55bff4c4b6b0 2023-11-16T16:52:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.477+0000 7f31a476ab80 10 client.26086 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.477+0000 7f31a476ab80 10 client.26086 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.477+0000 7f31a476ab80 20 client.26086 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.477+0000 7f31a476ab80 1 --2- 172.21.15.44:0/1492965415 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55bff4e08be0 0x55bff4e0afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.477+0000 7f31a476ab80 4 client.26086 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.477+0000 7f31a476ab80 1 -- 172.21.15.44:0/1492965415 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bff4d03040 con 0x55bff4e08be0 2023-11-16T16:52:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.478+0000 7f3183fff700 1 --2- 172.21.15.44:0/1492965415 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55bff4e08be0 0x55bff4e0afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.478+0000 7f3183fff700 1 --2- 172.21.15.44:0/1492965415 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55bff4e08be0 0x55bff4e0afc0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2b42f62dc58201d server_cookie=8ba45129730a0d47 in_seq=0 out_seq=0 2023-11-16T16:52:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.478+0000 7f3171ffb700 10 client.26086 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.479+0000 7f3171ffb700 1 -- 172.21.15.44:0/1492965415 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55bff4d03040 con 0x55bff4e08be0 2023-11-16T16:52:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.479+0000 7f3171ffb700 10 client.26086 handle_command_reply: tid=0 2023-11-16T16:52:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.479+0000 7f31a476ab80 1 client.26086 shutdown 2023-11-16T16:52:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.479+0000 7f31a476ab80 1 -- 172.21.15.44:0/1492965415 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 msgr2=0x55bff4dfbaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.479+0000 7f31a476ab80 1 --2- 172.21.15.44:0/1492965415 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 0x55bff4dfbaf0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f317c096990 tx=0x7f317c0ca510 comp rx=0 tx=0).stop 2023-11-16T16:52:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.479+0000 7f31a476ab80 1 -- 172.21.15.44:0/1492965415 shutdown_connections 2023-11-16T16:52:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.479+0000 7f31a476ab80 1 --2- 172.21.15.44:0/1492965415 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55bff4e08be0 0x55bff4e0afc0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.479+0000 7f31a476ab80 1 --2- 172.21.15.44:0/1492965415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 0x55bff4df9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.479+0000 7f31a476ab80 1 --2- 172.21.15.44:0/1492965415 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 0x55bff4dfbaf0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.479+0000 7f31a476ab80 1 --2- 172.21.15.44:0/1492965415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4df8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.479+0000 7f31a476ab80 1 -- 172.21.15.44:0/1492965415 >> 172.21.15.44:0/1492965415 conn(0x55bff4de4f50 msgr2=0x55bff4de97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.479+0000 7f3171ffb700 0 client.26086 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.480+0000 7f31a476ab80 1 -- 172.21.15.44:0/1492965415 shutdown_connections 2023-11-16T16:52:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.480+0000 7f31a476ab80 1 -- 172.21.15.44:0/1492965415 wait complete. 2023-11-16T16:52:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.480+0000 7f31a476ab80 20 client.26086 trim_cache size 0 max 16384 2023-11-16T16:52:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.496+0000 7f31a476ab80 1 Processor -- start 2023-11-16T16:52:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.496+0000 7f31a476ab80 1 -- start start 2023-11-16T16:52:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.496+0000 7f31a476ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4c4b6b0 0x55bff4dfb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.496+0000 7f31a476ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4def560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.496+0000 7f31a476ab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4df5b90 0x55bff4defaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.496+0000 7f31a476ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bff4deb400 con 0x55bff4c4b6b0 2023-11-16T16:52:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.496+0000 7f31a476ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bff4deffe0 con 0x55bff4df5720 2023-11-16T16:52:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.496+0000 7f31a476ab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bff4debc50 con 0x55bff4df5b90 2023-11-16T16:52:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.497+0000 7f318a59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4c4b6b0 0x55bff4dfb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.497+0000 7f318a59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4c4b6b0 0x55bff4dfb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60450/0 (socket says 172.21.15.44:60450) 2023-11-16T16:52:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.497+0000 7f3183fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4def560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.497+0000 7f318a59c700 1 -- 172.21.15.44:0/1966892842 learned_addr learned my addr 172.21.15.44:0/1966892842 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.497+0000 7f318ad9d700 1 --2- 172.21.15.44:0/1966892842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4df5b90 0x55bff4defaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.497+0000 7f318a59c700 1 -- 172.21.15.44:0/1966892842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4df5b90 msgr2=0x55bff4defaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.497+0000 7f318a59c700 1 --2- 172.21.15.44:0/1966892842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4df5b90 0x55bff4defaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.497+0000 7f318a59c700 1 -- 172.21.15.44:0/1966892842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 msgr2=0x55bff4def560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.497+0000 7f318a59c700 1 --2- 172.21.15.44:0/1966892842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4def560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.497+0000 7f318a59c700 1 -- 172.21.15.44:0/1966892842 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bff4d06e80 con 0x55bff4c4b6b0 2023-11-16T16:52:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.497+0000 7f318a59c700 1 --2- 172.21.15.44:0/1966892842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4c4b6b0 0x55bff4dfb690 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f317c098270 tx=0x7f317c010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a4a774994a522f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.498+0000 7f31817fa700 1 -- 172.21.15.44:0/1966892842 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317c0b1030 con 0x55bff4c4b6b0 2023-11-16T16:52:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.498+0000 7f31817fa700 1 -- 172.21.15.44:0/1966892842 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f317c010940 con 0x55bff4c4b6b0 2023-11-16T16:52:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.498+0000 7f31817fa700 1 -- 172.21.15.44:0/1966892842 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317c0a2120 con 0x55bff4c4b6b0 2023-11-16T16:52:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.498+0000 7f31a476ab80 1 -- 172.21.15.44:0/1966892842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4c4b6b0 msgr2=0x55bff4dfb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.498+0000 7f31a476ab80 1 --2- 172.21.15.44:0/1966892842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4c4b6b0 0x55bff4dfb690 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f317c098270 tx=0x7f317c010c80 comp rx=0 tx=0).stop 2023-11-16T16:52:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.498+0000 7f31a476ab80 1 -- 172.21.15.44:0/1966892842 shutdown_connections 2023-11-16T16:52:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.498+0000 7f31a476ab80 1 --2- 172.21.15.44:0/1966892842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4df5b90 0x55bff4defaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.498+0000 7f31a476ab80 1 --2- 172.21.15.44:0/1966892842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4c4b6b0 0x55bff4dfb690 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.499+0000 7f31a476ab80 1 --2- 172.21.15.44:0/1966892842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4def560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.499+0000 7f31a476ab80 1 -- 172.21.15.44:0/1966892842 >> 172.21.15.44:0/1966892842 conn(0x55bff4de4f50 msgr2=0x55bff4df2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.499+0000 7f31a476ab80 1 -- 172.21.15.44:0/1966892842 shutdown_connections 2023-11-16T16:52:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.499+0000 7f31a476ab80 1 -- 172.21.15.44:0/1966892842 wait complete. 2023-11-16T16:52:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.499+0000 7f31a476ab80 1 -- start start 2023-11-16T16:52:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.499+0000 7f31a476ab80 1 Processor -- start 2023-11-16T16:52:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.500+0000 7f31a476ab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 0x55bff4dffe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.500+0000 7f31a476ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4dfd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.500+0000 7f31a476ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 0x55bff4dfd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.500+0000 7f31a476ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bff4dfdbd0 con 0x55bff4df5b90 2023-11-16T16:52:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.500+0000 7f31a476ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bff4dfdd40 con 0x55bff4df5720 2023-11-16T16:52:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.500+0000 7f31a476ab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bff4dfdf00 con 0x55bff4c4b6b0 2023-11-16T16:52:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.500+0000 7f31a476ab80 10 client.0 _ll_register_callbacks cb 0x7f317c099b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.500+0000 7f318ad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 0x55bff4dfd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.500+0000 7f3183fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4dfd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.500+0000 7f318a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 0x55bff4dffe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.500+0000 7f3183fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4dfd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:54662/0 (socket says 172.21.15.44:54662) 2023-11-16T16:52:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.500+0000 7f318ad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 0x55bff4dfd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60462/0 (socket says 172.21.15.44:60462) 2023-11-16T16:52:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.500+0000 7f318ad9d700 1 -- 172.21.15.44:0/2247258670 learned_addr learned my addr 172.21.15.44:0/2247258670 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.501+0000 7f3183fff700 1 -- 172.21.15.44:0/2247258670 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 msgr2=0x55bff4dffe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.501+0000 7f3183fff700 1 --2- 172.21.15.44:0/2247258670 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 0x55bff4dffe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.501+0000 7f3183fff700 1 -- 172.21.15.44:0/2247258670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 msgr2=0x55bff4dfd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.501+0000 7f3183fff700 1 --2- 172.21.15.44:0/2247258670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 0x55bff4dfd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.501+0000 7f3183fff700 1 -- 172.21.15.44:0/2247258670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bff4d07130 con 0x55bff4df5720 2023-11-16T16:52:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.501+0000 7f318ad9d700 1 --2- 172.21.15.44:0/2247258670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 0x55bff4dfd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.501+0000 7f318a59c700 1 --2- 172.21.15.44:0/2247258670 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 0x55bff4dffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.501+0000 7f3183fff700 1 --2- 172.21.15.44:0/2247258670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4dfd150 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f317400b2e0 tx=0x7f3174009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8c9525eaf5bdb52 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.501+0000 7f3171ffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:52:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.501+0000 7f3171ffb700 1 -- 172.21.15.44:0/2247258670 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3174035cf0 con 0x55bff4df5720 2023-11-16T16:52:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.501+0000 7f31a476ab80 1 -- 172.21.15.44:0/2247258670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f317c0a2370 con 0x55bff4df5720 2023-11-16T16:52:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.501+0000 7f3171ffb700 1 -- 172.21.15.44:0/2247258670 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f317400f840 con 0x55bff4df5720 2023-11-16T16:52:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.501+0000 7f3171ffb700 1 -- 172.21.15.44:0/2247258670 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3174035e50 con 0x55bff4df5720 2023-11-16T16:52:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.502+0000 7f31a476ab80 10 client.26102 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.502+0000 7f31a476ab80 1 -- 172.21.15.44:0/2247258670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55bff4b57c00 con 0x55bff4df5720 2023-11-16T16:52:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.502+0000 7f3171ffb700 1 -- 172.21.15.44:0/2247258670 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f317c0a2370 con 0x55bff4df5720 2023-11-16T16:52:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.502+0000 7f3171ffb700 1 -- 172.21.15.44:0/2247258670 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3174033950 con 0x55bff4df5720 2023-11-16T16:52:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.502+0000 7f31a476ab80 10 client.26102 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.502+0000 7f31a476ab80 10 client.26102 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.502+0000 7f31a476ab80 20 client.26102 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.502+0000 7f31a476ab80 1 --2- 172.21.15.44:0/2247258670 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55bff4e5cec0 0x55bff4e5f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.502+0000 7f31a476ab80 4 client.26102 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:52:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.502+0000 7f31a476ab80 1 -- 172.21.15.44:0/2247258670 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bff4d0f600 con 0x55bff4e5cec0 2023-11-16T16:52:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.503+0000 7f318a59c700 1 --2- 172.21.15.44:0/2247258670 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55bff4e5cec0 0x55bff4e5f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.503+0000 7f318a59c700 1 --2- 172.21.15.44:0/2247258670 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55bff4e5cec0 0x55bff4e5f2a0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fdb46830a7925d9 server_cookie=d581a97e913a258f in_seq=0 out_seq=0 2023-11-16T16:52:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.503+0000 7f3171ffb700 10 client.26102 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.522+0000 7f3171ffb700 1 -- 172.21.15.44:0/2247258670 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bff4d0f600 con 0x55bff4e5cec0 2023-11-16T16:52:23.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.522+0000 7f3171ffb700 10 client.26102 handle_command_reply: tid=0 2023-11-16T16:52:23.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.523+0000 7f31a476ab80 1 client.26102 shutdown 2023-11-16T16:52:23.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.523+0000 7f31a476ab80 1 -- 172.21.15.44:0/2247258670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 msgr2=0x55bff4dfd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.523+0000 7f31a476ab80 1 --2- 172.21.15.44:0/2247258670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4dfd150 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f317400b2e0 tx=0x7f3174009ca0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.523+0000 7f31a476ab80 1 -- 172.21.15.44:0/2247258670 shutdown_connections 2023-11-16T16:52:23.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.523+0000 7f31a476ab80 1 --2- 172.21.15.44:0/2247258670 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55bff4e5cec0 0x55bff4e5f2a0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.523+0000 7f31a476ab80 1 --2- 172.21.15.44:0/2247258670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bff4df5b90 0x55bff4dfd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.523+0000 7f31a476ab80 1 --2- 172.21.15.44:0/2247258670 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55bff4c4b6b0 0x55bff4dffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.523+0000 7f31a476ab80 1 --2- 172.21.15.44:0/2247258670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bff4df5720 0x55bff4dfd150 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.523+0000 7f3171ffb700 0 client.26102 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.523+0000 7f31a476ab80 1 -- 172.21.15.44:0/2247258670 >> 172.21.15.44:0/2247258670 conn(0x55bff4de4f50 msgr2=0x55bff4df3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.523+0000 7f31a476ab80 1 -- 172.21.15.44:0/2247258670 shutdown_connections 2023-11-16T16:52:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.523+0000 7f31a476ab80 1 -- 172.21.15.44:0/2247258670 wait complete. 2023-11-16T16:52:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.523+0000 7f31a476ab80 20 client.26102 trim_cache size 0 max 16384 2023-11-16T16:52:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:52:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:52:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:52:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:52:23.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.524+0000 7f3190dff700 1 -- 172.21.15.44:0/150566775 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f31780a12c0 msgr2=0x7f31780a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.524+0000 7f3190dff700 1 --2- 172.21.15.44:0/150566775 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f31780a12c0 0x7f31780a3770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f318c199c40 tx=0x7f3174005fb0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.524+0000 7f3190dff700 1 -- 172.21.15.44:0/150566775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f318c102480 msgr2=0x7f318c198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.524+0000 7f3190dff700 1 --2- 172.21.15.44:0/150566775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f318c102480 0x7f318c198630 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f317c00e9a0 tx=0x7f317c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.524+0000 7f3190dff700 1 -- 172.21.15.44:0/150566775 shutdown_connections 2023-11-16T16:52:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.524+0000 7f3190dff700 1 --2- 172.21.15.44:0/150566775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f318c108490 0x7f318c19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.524+0000 7f3190dff700 1 --2- 172.21.15.44:0/150566775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f318c102480 0x7f318c198630 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.524+0000 7f3190dff700 1 --2- 172.21.15.44:0/150566775 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f31780a12c0 0x7f31780a3770 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.524+0000 7f3190dff700 1 --2- 172.21.15.44:0/150566775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f318c102e30 0x7f318c198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.524+0000 7f3190dff700 1 -- 172.21.15.44:0/150566775 >> 172.21.15.44:0/150566775 conn(0x7f318c0fdf60 msgr2=0x7f318c0fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.524+0000 7f3190dff700 1 -- 172.21.15.44:0/150566775 shutdown_connections 2023-11-16T16:52:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:23.524+0000 7f3190dff700 1 -- 172.21.15.44:0/150566775 wait complete. 2023-11-16T16:52:23.564 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:23.564 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:23.580 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:23.580 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:23.580 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 71 -lt 100 ']' 2023-11-16T16:52:23.580 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 71 + 1 2023-11-16T16:52:23.612 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=72 2023-11-16T16:52:23.613 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:23.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:23 smithi044 ceph-mon[128750]: pgmap v76: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 35 KiB/s wr, 12 op/s 2023-11-16T16:52:23.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:23 smithi176 ceph-mon[131630]: pgmap v76: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 35 KiB/s wr, 12 op/s 2023-11-16T16:52:23.694 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:23.694 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:23.697 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:23.697 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:23.698 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:23.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:23 smithi077 ceph-mon[131204]: pgmap v76: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 35 KiB/s wr, 12 op/s 2023-11-16T16:52:24.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.139+0000 7f504a3ff700 1 -- 172.21.15.44:0/3922933522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5044100550 msgr2=0x7f5044100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:24.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.139+0000 7f504a3ff700 1 --2- 172.21.15.44:0/3922933522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5044100550 0x7f5044100920 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f50340099c0 tx=0x7f5034009cd0 comp rx=0 tx=0).stop 2023-11-16T16:52:24.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.139+0000 7f504a3ff700 1 -- 172.21.15.44:0/3922933522 shutdown_connections 2023-11-16T16:52:24.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.139+0000 7f504a3ff700 1 --2- 172.21.15.44:0/3922933522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5044105540 0x7f504410d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:24.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.139+0000 7f504a3ff700 1 --2- 172.21.15.44:0/3922933522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5044100550 0x7f5044100920 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:24.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.139+0000 7f504a3ff700 1 --2- 172.21.15.44:0/3922933522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5044100ef0 0x7f5044104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:24.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.139+0000 7f504a3ff700 1 -- 172.21.15.44:0/3922933522 >> 172.21.15.44:0/3922933522 conn(0x7f5044077020 msgr2=0x7f5044077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:24.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.139+0000 7f504a3ff700 1 -- 172.21.15.44:0/3922933522 shutdown_connections 2023-11-16T16:52:24.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.139+0000 7f504a3ff700 1 -- 172.21.15.44:0/3922933522 wait complete. 2023-11-16T16:52:24.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.140+0000 7f504a3ff700 1 Processor -- start 2023-11-16T16:52:24.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.140+0000 7f504a3ff700 1 -- start start 2023-11-16T16:52:24.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.141+0000 7f504a3ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5044100550 0x7f50441985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:24.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.141+0000 7f504a3ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5044100ef0 0x7f5044198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:24.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.141+0000 7f504a3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5044105540 0x7f504419cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:24.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.141+0000 7f504a3ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504419d580 con 0x7f5044100ef0 2023-11-16T16:52:24.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.141+0000 7f504a3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504419d6f0 con 0x7f5044105540 2023-11-16T16:52:24.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.141+0000 7f504a3ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504419d8b0 con 0x7f5044100550 2023-11-16T16:52:24.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.141+0000 7f50437fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5044100ef0 0x7f5044198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:24.143 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.141+0000 7f504899c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5044105540 0x7f504419cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:24.144 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.141+0000 7f50437fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5044100ef0 0x7f5044198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60488/0 (socket says 172.21.15.44:60488) 2023-11-16T16:52:24.144 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.141+0000 7f504899c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5044105540 0x7f504419cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:54666/0 (socket says 172.21.15.44:54666) 2023-11-16T16:52:24.144 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.141+0000 7f50437fe700 1 -- 172.21.15.44:0/3808009498 learned_addr learned my addr 172.21.15.44:0/3808009498 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:24.144 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.141+0000 7f5043fff700 1 --2- 172.21.15.44:0/3808009498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5044100550 0x7f50441985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:24.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.142+0000 7f504899c700 1 -- 172.21.15.44:0/3808009498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5044100550 msgr2=0x7f50441985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:24.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.142+0000 7f504899c700 1 --2- 172.21.15.44:0/3808009498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5044100550 0x7f50441985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:24.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.142+0000 7f504899c700 1 -- 172.21.15.44:0/3808009498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5044100ef0 msgr2=0x7f5044198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:24.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.142+0000 7f504899c700 1 --2- 172.21.15.44:0/3808009498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5044100ef0 0x7f5044198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:24.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.142+0000 7f504899c700 1 -- 172.21.15.44:0/3808009498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c009c40 con 0x7f5044105540 2023-11-16T16:52:24.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.142+0000 7f50437fe700 1 --2- 172.21.15.44:0/3808009498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5044100ef0 0x7f5044198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:24.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.142+0000 7f504899c700 1 --2- 172.21.15.44:0/3808009498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5044105540 0x7f504419cf40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f503c00efb0 tx=0x7f503c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:24.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.142+0000 7f50417fa700 1 -- 172.21.15.44:0/3808009498 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c003a00 con 0x7f5044105540 2023-11-16T16:52:24.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.142+0000 7f50417fa700 1 -- 172.21.15.44:0/3808009498 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f503c00f040 con 0x7f5044105540 2023-11-16T16:52:24.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.142+0000 7f504a3ff700 1 -- 172.21.15.44:0/3808009498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50340095f0 con 0x7f5044105540 2023-11-16T16:52:24.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.142+0000 7f50417fa700 1 -- 172.21.15.44:0/3808009498 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c004540 con 0x7f5044105540 2023-11-16T16:52:24.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.142+0000 7f504a3ff700 1 -- 172.21.15.44:0/3808009498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5044110350 con 0x7f5044105540 2023-11-16T16:52:24.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.144+0000 7f504a3ff700 1 -- 172.21.15.44:0/3808009498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50441101c0 con 0x7f5044105540 2023-11-16T16:52:24.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.146+0000 7f50417fa700 1 -- 172.21.15.44:0/3808009498 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f503c003bf0 con 0x7f5044105540 2023-11-16T16:52:24.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.147+0000 7f50417fa700 1 --2- 172.21.15.44:0/3808009498 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f502c0a1340 0x7f502c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:24.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.147+0000 7f50417fa700 1 -- 172.21.15.44:0/3808009498 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f503c0ccc20 con 0x7f5044105540 2023-11-16T16:52:24.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.147+0000 7f5043fff700 1 --2- 172.21.15.44:0/3808009498 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f502c0a1340 0x7f502c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:24.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.148+0000 7f5043fff700 1 --2- 172.21.15.44:0/3808009498 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f502c0a1340 0x7f502c0a37f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5034009990 tx=0x7f5034005fb0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:24.151 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.149+0000 7f50417fa700 1 -- 172.21.15.44:0/3808009498 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f503c0cce60 con 0x7f5044105540 2023-11-16T16:52:24.423 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.421+0000 7f504a3ff700 1 -- 172.21.15.44:0/3808009498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50441104e0 con 0x7f5044105540 2023-11-16T16:52:24.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.423+0000 7f50417fa700 1 -- 172.21.15.44:0/3808009498 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f503c095ce0 con 0x7f5044105540 2023-11-16T16:52:24.424 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:52:24.425 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:52:24.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.425+0000 7f504a3ff700 1 -- 172.21.15.44:0/3808009498 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f502c0a1340 msgr2=0x7f502c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:24.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.425+0000 7f504a3ff700 1 --2- 172.21.15.44:0/3808009498 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f502c0a1340 0x7f502c0a37f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5034009990 tx=0x7f5034005fb0 comp rx=0 tx=0).stop 2023-11-16T16:52:24.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.425+0000 7f504a3ff700 1 -- 172.21.15.44:0/3808009498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5044105540 msgr2=0x7f504419cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:24.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.425+0000 7f504a3ff700 1 --2- 172.21.15.44:0/3808009498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5044105540 0x7f504419cf40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f503c00efb0 tx=0x7f503c00c3a0 comp rx=0 tx=0).stop 2023-11-16T16:52:24.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.425+0000 7f504a3ff700 1 -- 172.21.15.44:0/3808009498 shutdown_connections 2023-11-16T16:52:24.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.425+0000 7f504a3ff700 1 --2- 172.21.15.44:0/3808009498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5044105540 0x7f504419cf40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:24.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.425+0000 7f504a3ff700 1 --2- 172.21.15.44:0/3808009498 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f502c0a1340 0x7f502c0a37f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:24.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.425+0000 7f504a3ff700 1 --2- 172.21.15.44:0/3808009498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5044100550 0x7f50441985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:24.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.425+0000 7f504a3ff700 1 --2- 172.21.15.44:0/3808009498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5044100ef0 0x7f5044198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:24.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.425+0000 7f504a3ff700 1 -- 172.21.15.44:0/3808009498 >> 172.21.15.44:0/3808009498 conn(0x7f5044077020 msgr2=0x7f50440fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:24.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.425+0000 7f504a3ff700 1 -- 172.21.15.44:0/3808009498 shutdown_connections 2023-11-16T16:52:24.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:24.425+0000 7f504a3ff700 1 -- 172.21.15.44:0/3808009498 wait complete. 2023-11-16T16:52:24.429 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:52:24.444 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:24.444 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:24.525 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T16:52:24.655 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:24.656 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:24.681 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:24.681 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:24.681 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 72 -lt 100 ']' 2023-11-16T16:52:24.682 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 72 + 1 2023-11-16T16:52:24.726 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=73 2023-11-16T16:52:24.726 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:24.734 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:24.734 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:24.737 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:24.737 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:24.738 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:25.277 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:25.277 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:25.431 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:25.431 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:25.454 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:25.454 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:25.454 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 73 -lt 100 ']' 2023-11-16T16:52:25.455 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 73 + 1 2023-11-16T16:52:25.493 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=74 2023-11-16T16:52:25.493 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:25.518 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:25.518 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:25.520 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:25.521 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:25.521 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:25.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:25 smithi044 ceph-mon[128750]: pgmap v77: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 13 KiB/s wr, 3 op/s 2023-11-16T16:52:25.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:25 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3808009498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:52:25.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:25 smithi176 ceph-mon[131630]: pgmap v77: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 13 KiB/s wr, 3 op/s 2023-11-16T16:52:25.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:25 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3808009498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:52:25.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:25 smithi077 ceph-mon[131204]: pgmap v77: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 13 KiB/s wr, 3 op/s 2023-11-16T16:52:25.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:25 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3808009498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:52:26.102 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:26.103 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:26.297 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:26.297 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:26.322 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:26.322 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:26.322 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 74 -lt 100 ']' 2023-11-16T16:52:26.323 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 74 + 1 2023-11-16T16:52:26.371 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=75 2023-11-16T16:52:26.371 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:26.427 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:26.427 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:26.429 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:26.430 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:26.430 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:27.302 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:27.302 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:27.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.489+0000 7fe040dff700 1 -- 172.21.15.44:0/1572759603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe03c105620 msgr2=0x7fe03c10d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:27.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.489+0000 7fe040dff700 1 --2- 172.21.15.44:0/1572759603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe03c105620 0x7fe03c10d1a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe03400b0d0 tx=0x7fe03400b3e0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.489+0000 7fe040dff700 1 -- 172.21.15.44:0/1572759603 shutdown_connections 2023-11-16T16:52:27.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.489+0000 7fe040dff700 1 --2- 172.21.15.44:0/1572759603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe03c105620 0x7fe03c10d1a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.489+0000 7fe040dff700 1 --2- 172.21.15.44:0/1572759603 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe03c100cc0 0x7fe03c101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.489+0000 7fe040dff700 1 --2- 172.21.15.44:0/1572759603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03c101660 0x7fe03c104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.489+0000 7fe040dff700 1 -- 172.21.15.44:0/1572759603 >> 172.21.15.44:0/1572759603 conn(0x7fe03c077560 msgr2=0x7fe03c077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:27.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.489+0000 7fe040dff700 1 -- 172.21.15.44:0/1572759603 shutdown_connections 2023-11-16T16:52:27.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.489+0000 7fe040dff700 1 -- 172.21.15.44:0/1572759603 wait complete. 2023-11-16T16:52:27.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.490+0000 7fe040dff700 1 Processor -- start 2023-11-16T16:52:27.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.491+0000 7fe040dff700 1 -- start start 2023-11-16T16:52:27.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.491+0000 7fe040dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe03c100cc0 0x7fe03c19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.491+0000 7fe040dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe03c101660 0x7fe03c19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.491+0000 7fe040dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03c105620 0x7fe03c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.491+0000 7fe040dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c198d20 con 0x7fe03c100cc0 2023-11-16T16:52:27.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.491+0000 7fe040dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c198e90 con 0x7fe03c105620 2023-11-16T16:52:27.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.491+0000 7fe040dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c199050 con 0x7fe03c101660 2023-11-16T16:52:27.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.491+0000 7fe03a59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe03c100cc0 0x7fe03c19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.491+0000 7fe039d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe03c101660 0x7fe03c19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.492+0000 7fe039d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe03c101660 0x7fe03c19eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:42400/0 (socket says 172.21.15.44:42400) 2023-11-16T16:52:27.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.492+0000 7fe039d9b700 1 -- 172.21.15.44:0/2156276150 learned_addr learned my addr 172.21.15.44:0/2156276150 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:27.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.492+0000 7fe03ad9d700 1 --2- 172.21.15.44:0/2156276150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03c105620 0x7fe03c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.492+0000 7fe039d9b700 1 -- 172.21.15.44:0/2156276150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03c105620 msgr2=0x7fe03c1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:27.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.492+0000 7fe039d9b700 1 --2- 172.21.15.44:0/2156276150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03c105620 0x7fe03c1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.492+0000 7fe039d9b700 1 -- 172.21.15.44:0/2156276150 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe03c100cc0 msgr2=0x7fe03c19e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:27.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.492+0000 7fe039d9b700 1 --2- 172.21.15.44:0/2156276150 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe03c100cc0 0x7fe03c19e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.492+0000 7fe039d9b700 1 -- 172.21.15.44:0/2156276150 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe034009d00 con 0x7fe03c101660 2023-11-16T16:52:27.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.492+0000 7fe039d9b700 1 --2- 172.21.15.44:0/2156276150 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe03c101660 0x7fe03c19eb80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe03000e910 tx=0x7fe03000ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:27.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.492+0000 7fe02b7fe700 1 -- 172.21.15.44:0/2156276150 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe030004a30 con 0x7fe03c101660 2023-11-16T16:52:27.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.492+0000 7fe040dff700 1 -- 172.21.15.44:0/2156276150 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03c1103e0 con 0x7fe03c101660 2023-11-16T16:52:27.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.492+0000 7fe02b7fe700 1 -- 172.21.15.44:0/2156276150 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe0300054e0 con 0x7fe03c101660 2023-11-16T16:52:27.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.493+0000 7fe02b7fe700 1 -- 172.21.15.44:0/2156276150 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03000cc40 con 0x7fe03c101660 2023-11-16T16:52:27.497 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.493+0000 7fe040dff700 1 -- 172.21.15.44:0/2156276150 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03c110250 con 0x7fe03c101660 2023-11-16T16:52:27.497 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.496+0000 7fe02b7fe700 1 -- 172.21.15.44:0/2156276150 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fe030005000 con 0x7fe03c101660 2023-11-16T16:52:27.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.497+0000 7fe02b7fe700 1 --2- 172.21.15.44:0/2156276150 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe0240a12c0 0x7fe0240a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.497+0000 7fe02b7fe700 1 -- 172.21.15.44:0/2156276150 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fe030014070 con 0x7fe03c101660 2023-11-16T16:52:27.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.497+0000 7fe03a59c700 1 --2- 172.21.15.44:0/2156276150 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe0240a12c0 0x7fe0240a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.498+0000 7fe03a59c700 1 --2- 172.21.15.44:0/2156276150 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe0240a12c0 0x7fe0240a3770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe02c009c60 tx=0x7fe02c009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:27.598 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.596+0000 7fe0547e9b80 1 Processor -- start 2023-11-16T16:52:27.598 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.597+0000 7fe0547e9b80 1 -- start start 2023-11-16T16:52:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.597+0000 7fe0547e9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e97b60 0x561253e97f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.597+0000 7fe0547e9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253e9c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.597+0000 7fe0547e9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e9c8b0 0x561253e9cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.597+0000 7fe0547e9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561253d9d060 con 0x561253e98470 2023-11-16T16:52:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.597+0000 7fe0547e9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561253def8a0 con 0x561253e9c8b0 2023-11-16T16:52:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.597+0000 7fe0547e9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561253de8d30 con 0x561253e97b60 2023-11-16T16:52:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.597+0000 7fe039d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253e9c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.598+0000 7fe039d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253e9c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33318/0 (socket says 172.21.15.44:33318) 2023-11-16T16:52:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.598+0000 7fe039d9b700 1 -- 172.21.15.44:0/2001373129 learned_addr learned my addr 172.21.15.44:0/2001373129 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.598+0000 7fe03a59c700 1 --2- 172.21.15.44:0/2001373129 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e97b60 0x561253e97f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.598+0000 7fe03ad9d700 1 --2- 172.21.15.44:0/2001373129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e9c8b0 0x561253e9cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.598+0000 7fe039d9b700 1 -- 172.21.15.44:0/2001373129 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e97b60 msgr2=0x561253e97f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.598+0000 7fe039d9b700 1 --2- 172.21.15.44:0/2001373129 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e97b60 0x561253e97f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.598+0000 7fe039d9b700 1 -- 172.21.15.44:0/2001373129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e9c8b0 msgr2=0x561253e9cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.598+0000 7fe039d9b700 1 --2- 172.21.15.44:0/2001373129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e9c8b0 0x561253e9cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.598+0000 7fe039d9b700 1 -- 172.21.15.44:0/2001373129 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561253c6a630 con 0x561253e98470 2023-11-16T16:52:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.598+0000 7fe03ad9d700 1 --2- 172.21.15.44:0/2001373129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e9c8b0 0x561253e9cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:52:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.598+0000 7fe039d9b700 1 --2- 172.21.15.44:0/2001373129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253e9c370 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe0300a1f40 tx=0x7fe030099a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6a91b7195f06ffc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.598+0000 7fe028ff9700 1 -- 172.21.15.44:0/2001373129 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0300cd170 con 0x561253e98470 2023-11-16T16:52:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.598+0000 7fe028ff9700 1 -- 172.21.15.44:0/2001373129 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe0300a2470 con 0x561253e98470 2023-11-16T16:52:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.598+0000 7fe028ff9700 1 -- 172.21.15.44:0/2001373129 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe030096730 con 0x561253e98470 2023-11-16T16:52:27.603 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.599+0000 7fe0547e9b80 1 -- 172.21.15.44:0/2001373129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 msgr2=0x561253e9c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:27.603 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.599+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/2001373129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253e9c370 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe0300a1f40 tx=0x7fe030099a00 comp rx=0 tx=0).stop 2023-11-16T16:52:27.603 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.599+0000 7fe0547e9b80 1 -- 172.21.15.44:0/2001373129 shutdown_connections 2023-11-16T16:52:27.603 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.599+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/2001373129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e9c8b0 0x561253e9cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.603 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.599+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/2001373129 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e97b60 0x561253e97f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.603 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.599+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/2001373129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253e9c370 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.603 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.599+0000 7fe0547e9b80 1 -- 172.21.15.44:0/2001373129 >> 172.21.15.44:0/2001373129 conn(0x561253e8d360 msgr2=0x561253e92b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:27.603 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.599+0000 7fe0547e9b80 1 -- 172.21.15.44:0/2001373129 shutdown_connections 2023-11-16T16:52:27.603 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.599+0000 7fe0547e9b80 1 -- 172.21.15.44:0/2001373129 wait complete. 2023-11-16T16:52:27.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.600+0000 7fe0547e9b80 1 -- start start 2023-11-16T16:52:27.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.600+0000 7fe0547e9b80 1 Processor -- start 2023-11-16T16:52:27.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.600+0000 7fe0547e9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e97b60 0x561253ea4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.600+0000 7fe0547e9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253ea1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.600+0000 7fe0547e9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e9c8b0 0x561253ea1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.600+0000 7fe0547e9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561253d8a8e0 con 0x561253e98470 2023-11-16T16:52:27.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.600+0000 7fe0547e9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561253d8a580 con 0x561253e9c8b0 2023-11-16T16:52:27.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.600+0000 7fe0547e9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561253c6c920 con 0x561253e97b60 2023-11-16T16:52:27.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.601+0000 7fe0547e9b80 10 client.0 _ll_register_callbacks cb 0x561253da8780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:27.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.601+0000 7fe039d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253ea1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.601+0000 7fe039d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253ea1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33332/0 (socket says 172.21.15.44:33332) 2023-11-16T16:52:27.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.601+0000 7fe039d9b700 1 -- 172.21.15.44:0/3133857579 learned_addr learned my addr 172.21.15.44:0/3133857579 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:27.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.601+0000 7fe03a59c700 1 --2- 172.21.15.44:0/3133857579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e97b60 0x561253ea4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.601+0000 7fe03ad9d700 1 --2- 172.21.15.44:0/3133857579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e9c8b0 0x561253ea1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.601+0000 7fe039d9b700 1 -- 172.21.15.44:0/3133857579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e97b60 msgr2=0x561253ea4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:27.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.601+0000 7fe039d9b700 1 --2- 172.21.15.44:0/3133857579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e97b60 0x561253ea4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.601+0000 7fe039d9b700 1 -- 172.21.15.44:0/3133857579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e9c8b0 msgr2=0x561253ea1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:27.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.601+0000 7fe039d9b700 1 --2- 172.21.15.44:0/3133857579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e9c8b0 0x561253ea1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.601+0000 7fe039d9b700 1 -- 172.21.15.44:0/3133857579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03c110250 con 0x561253e98470 2023-11-16T16:52:27.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.601+0000 7fe039d9b700 1 --2- 172.21.15.44:0/3133857579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253ea1320 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe030005c90 tx=0x7fe03009e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f47da670f89e339 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:27.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.601+0000 7fe021ffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:52:27.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.602+0000 7fe021ffb700 1 -- 172.21.15.44:0/3133857579 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe030098160 con 0x561253e98470 2023-11-16T16:52:27.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.602+0000 7fe021ffb700 1 -- 172.21.15.44:0/3133857579 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fe03000cda0 con 0x561253e98470 2023-11-16T16:52:27.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.602+0000 7fe0547e9b80 1 -- 172.21.15.44:0/3133857579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561253e8ab20 con 0x561253e98470 2023-11-16T16:52:27.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.602+0000 7fe021ffb700 1 -- 172.21.15.44:0/3133857579 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe0300ca280 con 0x561253e98470 2023-11-16T16:52:27.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.602+0000 7fe0547e9b80 10 client.16101 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:27.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.602+0000 7fe0547e9b80 1 -- 172.21.15.44:0/3133857579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x561253daf190 con 0x561253e98470 2023-11-16T16:52:27.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.602+0000 7fe021ffb700 1 -- 172.21.15.44:0/3133857579 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe03005fe90 con 0x561253e98470 2023-11-16T16:52:27.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.602+0000 7fe021ffb700 1 -- 172.21.15.44:0/3133857579 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe0300add20 con 0x561253e98470 2023-11-16T16:52:27.608 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.602+0000 7fe0547e9b80 10 client.16101 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:27.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.602+0000 7fe0547e9b80 10 client.16101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:27.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.602+0000 7fe0547e9b80 20 client.16101 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:27.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.602+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/3133857579 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561253eb1410 0x561253eb37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.602+0000 7fe0547e9b80 4 client.16101 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:27.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.602+0000 7fe0547e9b80 1 -- 172.21.15.44:0/3133857579 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561253dab0a0 con 0x561253eb1410 2023-11-16T16:52:27.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.603+0000 7fe03a59c700 1 --2- 172.21.15.44:0/3133857579 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561253eb1410 0x561253eb37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.603+0000 7fe03a59c700 1 --2- 172.21.15.44:0/3133857579 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561253eb1410 0x561253eb37f0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e45f2fb5b8f1675a server_cookie=f721294467b7caee in_seq=0 out_seq=0 2023-11-16T16:52:27.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.603+0000 7fe021ffb700 10 client.16101 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:27.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.604+0000 7fe021ffb700 1 -- 172.21.15.44:0/3133857579 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x561253dab0a0 con 0x561253eb1410 2023-11-16T16:52:27.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.604+0000 7fe021ffb700 10 client.16101 handle_command_reply: tid=0 2023-11-16T16:52:27.609 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.604+0000 7fe0547e9b80 1 client.16101 shutdown 2023-11-16T16:52:27.610 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:27.610 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:27.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.604+0000 7fe0547e9b80 1 -- 172.21.15.44:0/3133857579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 msgr2=0x561253ea1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:27.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.604+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/3133857579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253ea1320 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe030005c90 tx=0x7fe03009e150 comp rx=0 tx=0).stop 2023-11-16T16:52:27.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.605+0000 7fe0547e9b80 1 -- 172.21.15.44:0/3133857579 shutdown_connections 2023-11-16T16:52:27.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.605+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/3133857579 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561253eb1410 0x561253eb37f0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.605+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/3133857579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e9c8b0 0x561253ea1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.605+0000 7fe021ffb700 0 client.16101 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:27.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.605+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/3133857579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e97b60 0x561253ea4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.605+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/3133857579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253ea1320 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.605+0000 7fe0547e9b80 1 -- 172.21.15.44:0/3133857579 >> 172.21.15.44:0/3133857579 conn(0x561253e8d360 msgr2=0x561253e9dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:27.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.605+0000 7fe0547e9b80 1 -- 172.21.15.44:0/3133857579 shutdown_connections 2023-11-16T16:52:27.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.605+0000 7fe0547e9b80 1 -- 172.21.15.44:0/3133857579 wait complete. 2023-11-16T16:52:27.611 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.605+0000 7fe0547e9b80 20 client.16101 trim_cache size 0 max 16384 2023-11-16T16:52:27.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:27 smithi044 ceph-mon[128750]: pgmap v78: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 13 KiB/s wr, 3 op/s 2023-11-16T16:52:27.621 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.619+0000 7fe0547e9b80 1 Processor -- start 2023-11-16T16:52:27.621 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.619+0000 7fe0547e9b80 1 -- start start 2023-11-16T16:52:27.621 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.619+0000 7fe0547e9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e97b60 0x561253ea05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.621 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.619+0000 7fe0547e9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253e91550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.619+0000 7fe0547e9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e9c8b0 0x561253eadc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.619+0000 7fe0547e9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561253e98c00 con 0x561253e98470 2023-11-16T16:52:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.619+0000 7fe0547e9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561253eb3db0 con 0x561253e9c8b0 2023-11-16T16:52:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.619+0000 7fe0547e9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561253e9b440 con 0x561253e97b60 2023-11-16T16:52:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.619+0000 7fe039d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253e91550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.619+0000 7fe039d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253e91550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33334/0 (socket says 172.21.15.44:33334) 2023-11-16T16:52:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.619+0000 7fe039d9b700 1 -- 172.21.15.44:0/3064085149 learned_addr learned my addr 172.21.15.44:0/3064085149 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.619+0000 7fe03ad9d700 1 --2- 172.21.15.44:0/3064085149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e9c8b0 0x561253eadc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.619+0000 7fe03a59c700 1 --2- 172.21.15.44:0/3064085149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e97b60 0x561253ea05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.620+0000 7fe039d9b700 1 -- 172.21.15.44:0/3064085149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e97b60 msgr2=0x561253ea05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.620+0000 7fe039d9b700 1 --2- 172.21.15.44:0/3064085149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e97b60 0x561253ea05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.620+0000 7fe039d9b700 1 -- 172.21.15.44:0/3064085149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e9c8b0 msgr2=0x561253eadc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.620+0000 7fe039d9b700 1 --2- 172.21.15.44:0/3064085149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e9c8b0 0x561253eadc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.623 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.620+0000 7fe039d9b700 1 -- 172.21.15.44:0/3064085149 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561253daeee0 con 0x561253e98470 2023-11-16T16:52:27.623 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.620+0000 7fe03a59c700 1 --2- 172.21.15.44:0/3064085149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e97b60 0x561253ea05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:52:27.623 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.620+0000 7fe039d9b700 1 --2- 172.21.15.44:0/3064085149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253e91550 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe030099fb0 tx=0x7fe030099a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23327666ff2d962d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:27.624 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.620+0000 7fe0297fa700 1 -- 172.21.15.44:0/3064085149 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe030098160 con 0x561253e98470 2023-11-16T16:52:27.624 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.620+0000 7fe0297fa700 1 -- 172.21.15.44:0/3064085149 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe03005fe90 con 0x561253e98470 2023-11-16T16:52:27.624 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.620+0000 7fe0297fa700 1 -- 172.21.15.44:0/3064085149 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0300a2240 con 0x561253e98470 2023-11-16T16:52:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.621+0000 7fe0547e9b80 1 -- 172.21.15.44:0/3064085149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 msgr2=0x561253e91550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.621+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/3064085149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253e91550 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe030099fb0 tx=0x7fe030099a00 comp rx=0 tx=0).stop 2023-11-16T16:52:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.621+0000 7fe0547e9b80 1 -- 172.21.15.44:0/3064085149 shutdown_connections 2023-11-16T16:52:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.621+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/3064085149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e9c8b0 0x561253eadc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.621+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/3064085149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e97b60 0x561253ea05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.621+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/3064085149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e98470 0x561253e91550 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.621+0000 7fe0547e9b80 1 -- 172.21.15.44:0/3064085149 >> 172.21.15.44:0/3064085149 conn(0x561253e8d360 msgr2=0x561253eb8070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.621+0000 7fe0547e9b80 1 -- 172.21.15.44:0/3064085149 shutdown_connections 2023-11-16T16:52:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.621+0000 7fe0547e9b80 1 -- 172.21.15.44:0/3064085149 wait complete. 2023-11-16T16:52:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.621+0000 7fe0547e9b80 1 -- start start 2023-11-16T16:52:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.622+0000 7fe0547e9b80 1 Processor -- start 2023-11-16T16:52:27.626 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.622+0000 7fe0547e9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e97b60 0x561253ea6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.626 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.622+0000 7fe0547e9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e98470 0x561253ea70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.626 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.623+0000 7fe0547e9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e9c8b0 0x561253f041c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.626 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.623+0000 7fe0547e9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561253f04700 con 0x561253e9c8b0 2023-11-16T16:52:27.626 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.623+0000 7fe0547e9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561253f04840 con 0x561253e97b60 2023-11-16T16:52:27.626 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.623+0000 7fe0547e9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561253f049b0 con 0x561253e98470 2023-11-16T16:52:27.626 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.623+0000 7fe0547e9b80 10 client.0 _ll_register_callbacks cb 0x561253da8780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:27.626 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.623+0000 7fe03ad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e9c8b0 0x561253f041c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.626 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.623+0000 7fe039d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e98470 0x561253ea70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.626 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.623+0000 7fe03ad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e9c8b0 0x561253f041c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33336/0 (socket says 172.21.15.44:33336) 2023-11-16T16:52:27.626 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.623+0000 7fe039d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e98470 0x561253ea70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:42434/0 (socket says 172.21.15.44:42434) 2023-11-16T16:52:27.627 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.623+0000 7fe03ad9d700 1 -- 172.21.15.44:0/1952237791 learned_addr learned my addr 172.21.15.44:0/1952237791 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:27.627 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.623+0000 7fe03a59c700 1 --2- 172.21.15.44:0/1952237791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e97b60 0x561253ea6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.628 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.623+0000 7fe03ad9d700 1 -- 172.21.15.44:0/1952237791 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e98470 msgr2=0x561253ea70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:27.628 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.623+0000 7fe03ad9d700 1 --2- 172.21.15.44:0/1952237791 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e98470 0x561253ea70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.628 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.623+0000 7fe03ad9d700 1 -- 172.21.15.44:0/1952237791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e97b60 msgr2=0x561253ea6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:27.628 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.623+0000 7fe03ad9d700 1 --2- 172.21.15.44:0/1952237791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e97b60 0x561253ea6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.628 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.623+0000 7fe03ad9d700 1 -- 172.21.15.44:0/1952237791 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561253daf190 con 0x561253e9c8b0 2023-11-16T16:52:27.628 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.624+0000 7fe03ad9d700 1 --2- 172.21.15.44:0/1952237791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e9c8b0 0x561253f041c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe034009fd0 tx=0x7fe034009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad053ba9310d7884 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:27.628 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.624+0000 7fe021ffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:52:27.629 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.624+0000 7fe021ffb700 1 -- 172.21.15.44:0/1952237791 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe034022070 con 0x561253e9c8b0 2023-11-16T16:52:27.629 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.624+0000 7fe021ffb700 1 -- 172.21.15.44:0/1952237791 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fe03400bd30 con 0x561253e9c8b0 2023-11-16T16:52:27.629 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.624+0000 7fe0547e9b80 1 -- 172.21.15.44:0/1952237791 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe0300a2470 con 0x561253e9c8b0 2023-11-16T16:52:27.629 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.624+0000 7fe021ffb700 1 -- 172.21.15.44:0/1952237791 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe03400e040 con 0x561253e9c8b0 2023-11-16T16:52:27.629 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.624+0000 7fe0547e9b80 10 client.16107 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:27.629 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.624+0000 7fe0547e9b80 1 -- 172.21.15.44:0/1952237791 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x561253bffc60 con 0x561253e9c8b0 2023-11-16T16:52:27.629 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.624+0000 7fe021ffb700 1 -- 172.21.15.44:0/1952237791 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe034030020 con 0x561253e9c8b0 2023-11-16T16:52:27.629 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.625+0000 7fe021ffb700 1 -- 172.21.15.44:0/1952237791 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe03400b100 con 0x561253e9c8b0 2023-11-16T16:52:27.629 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.625+0000 7fe0547e9b80 10 client.16107 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:27.629 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.625+0000 7fe0547e9b80 10 client.16107 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:27.629 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.625+0000 7fe0547e9b80 20 client.16107 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:27.629 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.625+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/1952237791 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561253f06180 0x561253f08590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:27.630 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.625+0000 7fe0547e9b80 4 client.16107 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T16:52:27.630 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.625+0000 7fe0547e9b80 1 -- 172.21.15.44:0/1952237791 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561253db7660 con 0x561253f06180 2023-11-16T16:52:27.630 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.625+0000 7fe03a59c700 1 --2- 172.21.15.44:0/1952237791 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561253f06180 0x561253f08590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:27.630 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.626+0000 7fe03a59c700 1 --2- 172.21.15.44:0/1952237791 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561253f06180 0x561253f08590 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d659554bac86bd9 server_cookie=c17b35ccfa28cb36 in_seq=0 out_seq=0 2023-11-16T16:52:27.630 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.626+0000 7fe021ffb700 10 client.16107 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:27.630 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.628+0000 7fe021ffb700 1 -- 172.21.15.44:0/1952237791 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561253db7660 con 0x561253f06180 2023-11-16T16:52:27.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.628+0000 7fe021ffb700 10 client.16107 handle_command_reply: tid=0 2023-11-16T16:52:27.631 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.628+0000 7fe0547e9b80 1 client.16107 shutdown 2023-11-16T16:52:27.631 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:27.631 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:27.631 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 75 -lt 100 ']' 2023-11-16T16:52:27.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.628+0000 7fe0547e9b80 1 -- 172.21.15.44:0/1952237791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e9c8b0 msgr2=0x561253f041c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:27.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.628+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/1952237791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e9c8b0 0x561253f041c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe034009fd0 tx=0x7fe034009c80 comp rx=0 tx=0).stop 2023-11-16T16:52:27.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.628+0000 7fe0547e9b80 1 -- 172.21.15.44:0/1952237791 shutdown_connections 2023-11-16T16:52:27.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.628+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/1952237791 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561253f06180 0x561253f08590 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.628+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/1952237791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561253e9c8b0 0x561253f041c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.628+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/1952237791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561253e97b60 0x561253ea6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.628+0000 7fe021ffb700 0 client.16107 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:27.632 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.628+0000 7fe0547e9b80 1 --2- 172.21.15.44:0/1952237791 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561253e98470 0x561253ea70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.633 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.628+0000 7fe0547e9b80 1 -- 172.21.15.44:0/1952237791 >> 172.21.15.44:0/1952237791 conn(0x561253e8d360 msgr2=0x561253eadc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:27.633 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.629+0000 7fe0547e9b80 1 -- 172.21.15.44:0/1952237791 shutdown_connections 2023-11-16T16:52:27.633 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.629+0000 7fe0547e9b80 1 -- 172.21.15.44:0/1952237791 wait complete. 2023-11-16T16:52:27.633 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.629+0000 7fe0547e9b80 20 client.16107 trim_cache size 0 max 16384 2023-11-16T16:52:27.633 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T16:52:27.633 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 75 + 1 2023-11-16T16:52:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.629+0000 7fe040dff700 1 -- 172.21.15.44:0/2156276150 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe0240a12c0 msgr2=0x7fe0240a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.629+0000 7fe040dff700 1 --2- 172.21.15.44:0/2156276150 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe0240a12c0 0x7fe0240a3770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe02c009c60 tx=0x7fe02c009260 comp rx=0 tx=0).stop 2023-11-16T16:52:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.629+0000 7fe040dff700 1 -- 172.21.15.44:0/2156276150 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe03c101660 msgr2=0x7fe03c19eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.629+0000 7fe040dff700 1 --2- 172.21.15.44:0/2156276150 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe03c101660 0x7fe03c19eb80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe03000e910 tx=0x7fe03000ec20 comp rx=0 tx=0).stop 2023-11-16T16:52:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.629+0000 7fe040dff700 1 -- 172.21.15.44:0/2156276150 shutdown_connections 2023-11-16T16:52:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.629+0000 7fe040dff700 1 --2- 172.21.15.44:0/2156276150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03c105620 0x7fe03c1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.629+0000 7fe040dff700 1 --2- 172.21.15.44:0/2156276150 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe03c100cc0 0x7fe03c19e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.629+0000 7fe040dff700 1 --2- 172.21.15.44:0/2156276150 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe0240a12c0 0x7fe0240a3770 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.629+0000 7fe040dff700 1 --2- 172.21.15.44:0/2156276150 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe03c101660 0x7fe03c19eb80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:27.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.629+0000 7fe040dff700 1 -- 172.21.15.44:0/2156276150 >> 172.21.15.44:0/2156276150 conn(0x7fe03c077560 msgr2=0x7fe03c0fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:27.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.630+0000 7fe040dff700 1 -- 172.21.15.44:0/2156276150 shutdown_connections 2023-11-16T16:52:27.635 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:27.630+0000 7fe040dff700 1 -- 172.21.15.44:0/2156276150 wait complete. 2023-11-16T16:52:27.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:27 smithi176 ceph-mon[131630]: pgmap v78: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 13 KiB/s wr, 3 op/s 2023-11-16T16:52:27.659 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=76 2023-11-16T16:52:27.659 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:27.704 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:27.704 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:27.707 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:27.707 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:27.707 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:27.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:27 smithi077 ceph-mon[131204]: pgmap v78: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 13 KiB/s wr, 3 op/s 2023-11-16T16:52:28.268 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:28.269 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:28.405 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:28.406 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:28.420 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:28.420 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:28.420 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 76 -lt 100 ']' 2023-11-16T16:52:28.421 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 76 + 1 2023-11-16T16:52:28.453 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=77 2023-11-16T16:52:28.454 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:28.485 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:28.485 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:28.488 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:28.488 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:28.488 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:28.689 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:52:28.689 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:52:28.985 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:28.985 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:29.063 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:29.063 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:29.104 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:29.105 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:29.105 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 77 -lt 100 ']' 2023-11-16T16:52:29.105 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 77 + 1 2023-11-16T16:52:29.134 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=78 2023-11-16T16:52:29.134 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:29.142 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:29.142 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:29.145 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:29.145 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:29.146 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:29.595 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:29 smithi044 ceph-mon[128750]: pgmap v79: 129 pgs: 129 active+clean; 2.6 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 28 KiB/s wr, 11 op/s 2023-11-16T16:52:29.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:29 smithi176 ceph-mon[131630]: pgmap v79: 129 pgs: 129 active+clean; 2.6 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 28 KiB/s wr, 11 op/s 2023-11-16T16:52:29.685 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:29.685 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:29.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:29 smithi077 ceph-mon[131204]: pgmap v79: 129 pgs: 129 active+clean; 2.6 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 28 KiB/s wr, 11 op/s 2023-11-16T16:52:29.805 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:29.805 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:29.812 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:29.812 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:29.812 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 78 -lt 100 ']' 2023-11-16T16:52:29.812 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 78 + 1 2023-11-16T16:52:29.851 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=79 2023-11-16T16:52:29.851 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:29.918 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:29.918 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:29.921 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:29.922 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:29.922 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:30.418 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:30.419 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:30.655 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:30.655 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:30.663 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:30.663 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:30.663 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 79 -lt 100 ']' 2023-11-16T16:52:30.664 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 79 + 1 2023-11-16T16:52:30.700 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=80 2023-11-16T16:52:30.700 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:30.746 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:30.746 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:30.749 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:30.749 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:30.750 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:31.443 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:31.444 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:31.572 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:31.572 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:31.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:31 smithi044 ceph-mon[128750]: pgmap v80: 129 pgs: 129 active+clean; 2.6 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 26 KiB/s wr, 9 op/s 2023-11-16T16:52:31.628 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:31.628 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:31.628 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 80 -lt 100 ']' 2023-11-16T16:52:31.628 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 80 + 1 2023-11-16T16:52:31.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:31 smithi176 ceph-mon[131630]: pgmap v80: 129 pgs: 129 active+clean; 2.6 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 26 KiB/s wr, 9 op/s 2023-11-16T16:52:31.667 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=81 2023-11-16T16:52:31.667 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:31.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:31 smithi077 ceph-mon[131204]: pgmap v80: 129 pgs: 129 active+clean; 2.6 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 26 KiB/s wr, 9 op/s 2023-11-16T16:52:31.753 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:31.753 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:31.755 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:31.756 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:31.756 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:31.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.930+0000 7f20f47ff700 1 -- 172.21.15.44:0/746180506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec10a8b0 msgr2=0x7f20ec10ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:31.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.930+0000 7f20f47ff700 1 --2- 172.21.15.44:0/746180506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec10a8b0 0x7f20ec10ccd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f20e4009990 tx=0x7f20e4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:52:31.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.930+0000 7f20f47ff700 1 -- 172.21.15.44:0/746180506 shutdown_connections 2023-11-16T16:52:31.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.930+0000 7f20f47ff700 1 --2- 172.21.15.44:0/746180506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec10a8b0 0x7f20ec10ccd0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:31.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.930+0000 7f20f47ff700 1 --2- 172.21.15.44:0/746180506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20ec101390 0x7f20ec101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:31.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.930+0000 7f20f47ff700 1 --2- 172.21.15.44:0/746180506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20ec101ca0 0x7f20ec10a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:31.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.930+0000 7f20f47ff700 1 -- 172.21.15.44:0/746180506 >> 172.21.15.44:0/746180506 conn(0x7f20ec0770b0 msgr2=0x7f20ec0774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:31.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.930+0000 7f20f47ff700 1 -- 172.21.15.44:0/746180506 shutdown_connections 2023-11-16T16:52:31.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.931+0000 7f20f47ff700 1 -- 172.21.15.44:0/746180506 wait complete. 2023-11-16T16:52:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.931+0000 7f20f47ff700 1 Processor -- start 2023-11-16T16:52:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.932+0000 7f20f47ff700 1 -- start start 2023-11-16T16:52:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.932+0000 7f20f47ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec101390 0x7f20ec198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.932+0000 7f20f47ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20ec101ca0 0x7f20ec198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.932+0000 7f20f47ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20ec10a8b0 0x7f20ec19d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.932+0000 7f20f47ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec19d650 con 0x7f20ec101390 2023-11-16T16:52:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.932+0000 7f20f47ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec19d7c0 con 0x7f20ec101ca0 2023-11-16T16:52:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.932+0000 7f20f47ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec19d980 con 0x7f20ec10a8b0 2023-11-16T16:52:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.932+0000 7f20f259b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec101390 0x7f20ec198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.932+0000 7f20f259b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec101390 0x7f20ec198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33358/0 (socket says 172.21.15.44:33358) 2023-11-16T16:52:31.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.932+0000 7f20f259b700 1 -- 172.21.15.44:0/3927562346 learned_addr learned my addr 172.21.15.44:0/3927562346 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:31.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.932+0000 7f20f1d9a700 1 --2- 172.21.15.44:0/3927562346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20ec101ca0 0x7f20ec198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:31.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.932+0000 7f20f2d9c700 1 --2- 172.21.15.44:0/3927562346 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20ec10a8b0 0x7f20ec19d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:31.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.933+0000 7f20f259b700 1 -- 172.21.15.44:0/3927562346 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20ec10a8b0 msgr2=0x7f20ec19d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:31.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.933+0000 7f20f259b700 1 --2- 172.21.15.44:0/3927562346 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20ec10a8b0 0x7f20ec19d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:31.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.933+0000 7f20f259b700 1 -- 172.21.15.44:0/3927562346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20ec101ca0 msgr2=0x7f20ec198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:31.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.933+0000 7f20f259b700 1 --2- 172.21.15.44:0/3927562346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20ec101ca0 0x7f20ec198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:31.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.933+0000 7f20f259b700 1 -- 172.21.15.44:0/3927562346 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e40095f0 con 0x7f20ec101390 2023-11-16T16:52:31.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.933+0000 7f20f259b700 1 --2- 172.21.15.44:0/3927562346 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec101390 0x7f20ec198670 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f20e800ef10 tx=0x7f20e800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:31.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.933+0000 7f20e37fe700 1 -- 172.21.15.44:0/3927562346 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e800c9c0 con 0x7f20ec101390 2023-11-16T16:52:31.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.933+0000 7f20e37fe700 1 -- 172.21.15.44:0/3927562346 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20e800f040 con 0x7f20ec101390 2023-11-16T16:52:31.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.933+0000 7f20e37fe700 1 -- 172.21.15.44:0/3927562346 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e8014480 con 0x7f20ec101390 2023-11-16T16:52:31.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.933+0000 7f20f47ff700 1 -- 172.21.15.44:0/3927562346 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ec110350 con 0x7f20ec101390 2023-11-16T16:52:31.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.934+0000 7f20f47ff700 1 -- 172.21.15.44:0/3927562346 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ec1101c0 con 0x7f20ec101390 2023-11-16T16:52:31.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.937+0000 7f20e37fe700 1 -- 172.21.15.44:0/3927562346 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f20e8007690 con 0x7f20ec101390 2023-11-16T16:52:31.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.938+0000 7f20e37fe700 1 --2- 172.21.15.44:0/3927562346 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f20d80a12c0 0x7f20d80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:31.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.938+0000 7f20e37fe700 1 -- 172.21.15.44:0/3927562346 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f20e80cd3c0 con 0x7f20ec101390 2023-11-16T16:52:31.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.938+0000 7f20f1d9a700 1 --2- 172.21.15.44:0/3927562346 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f20d80a12c0 0x7f20d80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:31.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:31.939+0000 7f20f1d9a700 1 --2- 172.21.15.44:0/3927562346 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f20d80a12c0 0x7f20d80a3770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f20ec199c80 tx=0x7f20dc00b410 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:32.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.033+0000 7f21080d9b80 1 Processor -- start 2023-11-16T16:52:32.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.033+0000 7f21080d9b80 1 -- start start 2023-11-16T16:52:32.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.033+0000 7f21080d9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39bbe0 0x555d6a39bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:32.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.033+0000 7f21080d9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39c4f0 0x555d6a3a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:32.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.033+0000 7f21080d9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a3a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:32.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.033+0000 7f21080d9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6a2a10e0 con 0x555d6a3a0930 2023-11-16T16:52:32.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.033+0000 7f21080d9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6a2f3920 con 0x555d6a39c4f0 2023-11-16T16:52:32.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.033+0000 7f21080d9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6a2ecdb0 con 0x555d6a39bbe0 2023-11-16T16:52:32.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.034+0000 7f20f2d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a3a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:32.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.034+0000 7f20f2d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a3a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33370/0 (socket says 172.21.15.44:33370) 2023-11-16T16:52:32.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.034+0000 7f20f2d9c700 1 -- 172.21.15.44:0/3313411388 learned_addr learned my addr 172.21.15.44:0/3313411388 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:32.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.034+0000 7f20f1d9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39c4f0 0x555d6a3a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:32.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.034+0000 7f20f259b700 1 --2- 172.21.15.44:0/3313411388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39bbe0 0x555d6a39bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:32.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.034+0000 7f20f2d9c700 1 -- 172.21.15.44:0/3313411388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39bbe0 msgr2=0x555d6a39bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:32.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.034+0000 7f20f2d9c700 1 --2- 172.21.15.44:0/3313411388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39bbe0 0x555d6a39bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.034+0000 7f20f2d9c700 1 -- 172.21.15.44:0/3313411388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39c4f0 msgr2=0x555d6a3a03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:32.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.034+0000 7f20f2d9c700 1 --2- 172.21.15.44:0/3313411388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39c4f0 0x555d6a3a03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.034+0000 7f20f2d9c700 1 -- 172.21.15.44:0/3313411388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d6a16e6b0 con 0x555d6a3a0930 2023-11-16T16:52:32.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.034+0000 7f20f1d9a700 1 --2- 172.21.15.44:0/3313411388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39c4f0 0x555d6a3a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:32.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.034+0000 7f20f2d9c700 1 --2- 172.21.15.44:0/3313411388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a3a0d80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f20e400b610 tx=0x7f20e400b640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d42260fce994ea38 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:32.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.035+0000 7f20e0ff9700 1 -- 172.21.15.44:0/3313411388 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e401d070 con 0x555d6a3a0930 2023-11-16T16:52:32.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.035+0000 7f20e0ff9700 1 -- 172.21.15.44:0/3313411388 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20e401e040 con 0x555d6a3a0930 2023-11-16T16:52:32.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.035+0000 7f20e0ff9700 1 -- 172.21.15.44:0/3313411388 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e4004920 con 0x555d6a3a0930 2023-11-16T16:52:32.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.035+0000 7f21080d9b80 1 -- 172.21.15.44:0/3313411388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 msgr2=0x555d6a3a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:32.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.035+0000 7f21080d9b80 1 --2- 172.21.15.44:0/3313411388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a3a0d80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f20e400b610 tx=0x7f20e400b640 comp rx=0 tx=0).stop 2023-11-16T16:52:32.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.035+0000 7f21080d9b80 1 -- 172.21.15.44:0/3313411388 shutdown_connections 2023-11-16T16:52:32.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.035+0000 7f21080d9b80 1 --2- 172.21.15.44:0/3313411388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a3a0d80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.036+0000 7f21080d9b80 1 --2- 172.21.15.44:0/3313411388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39bbe0 0x555d6a39bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.036+0000 7f21080d9b80 1 --2- 172.21.15.44:0/3313411388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39c4f0 0x555d6a3a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.036+0000 7f21080d9b80 1 -- 172.21.15.44:0/3313411388 >> 172.21.15.44:0/3313411388 conn(0x555d6a3913e0 msgr2=0x555d6a396bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:32.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.036+0000 7f21080d9b80 1 -- 172.21.15.44:0/3313411388 shutdown_connections 2023-11-16T16:52:32.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.036+0000 7f21080d9b80 1 -- 172.21.15.44:0/3313411388 wait complete. 2023-11-16T16:52:32.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.036+0000 7f21080d9b80 1 -- start start 2023-11-16T16:52:32.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.036+0000 7f21080d9b80 1 Processor -- start 2023-11-16T16:52:32.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.037+0000 7f21080d9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39bbe0 0x555d6a3a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:32.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.037+0000 7f21080d9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39c4f0 0x555d6a3a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:32.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.037+0000 7f21080d9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a3a5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:32.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.037+0000 7f21080d9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6a28e960 con 0x555d6a3a0930 2023-11-16T16:52:32.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.037+0000 7f21080d9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6a28e600 con 0x555d6a39c4f0 2023-11-16T16:52:32.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.037+0000 7f21080d9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6a1709a0 con 0x555d6a39bbe0 2023-11-16T16:52:32.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.037+0000 7f21080d9b80 10 client.0 _ll_register_callbacks cb 0x555d6a2ac800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:32.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.037+0000 7f20f2d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a3a5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:32.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.037+0000 7f20f2d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a3a5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33378/0 (socket says 172.21.15.44:33378) 2023-11-16T16:52:32.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.037+0000 7f20f2d9c700 1 -- 172.21.15.44:0/2298892740 learned_addr learned my addr 172.21.15.44:0/2298892740 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:32.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.037+0000 7f20f1d9a700 1 --2- 172.21.15.44:0/2298892740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39c4f0 0x555d6a3a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:32.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.037+0000 7f20f259b700 1 --2- 172.21.15.44:0/2298892740 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39bbe0 0x555d6a3a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:32.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.038+0000 7f20f2d9c700 1 -- 172.21.15.44:0/2298892740 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39bbe0 msgr2=0x555d6a3a53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:32.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.038+0000 7f20f2d9c700 1 --2- 172.21.15.44:0/2298892740 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39bbe0 0x555d6a3a53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.042 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.038+0000 7f20f2d9c700 1 -- 172.21.15.44:0/2298892740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39c4f0 msgr2=0x555d6a3a58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:32.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.038+0000 7f20f2d9c700 1 --2- 172.21.15.44:0/2298892740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39c4f0 0x555d6a3a58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.038+0000 7f20f2d9c700 1 -- 172.21.15.44:0/2298892740 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ec10d990 con 0x555d6a3a0930 2023-11-16T16:52:32.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.038+0000 7f20f2d9c700 1 --2- 172.21.15.44:0/2298892740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a3a5e30 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f20e400b6f0 tx=0x7f20e400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df65ef7dbd214ba server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:32.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.038+0000 7f20d5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:52:32.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.038+0000 7f20d5ffb700 1 -- 172.21.15.44:0/2298892740 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e401d040 con 0x555d6a3a0930 2023-11-16T16:52:32.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.038+0000 7f20d5ffb700 1 -- 172.21.15.44:0/2298892740 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f20e400f880 con 0x555d6a3a0930 2023-11-16T16:52:32.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.038+0000 7f21080d9b80 1 -- 172.21.15.44:0/2298892740 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555d6a38eba0 con 0x555d6a3a0930 2023-11-16T16:52:32.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.039+0000 7f20d5ffb700 1 -- 172.21.15.44:0/2298892740 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20e4005440 con 0x555d6a3a0930 2023-11-16T16:52:32.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.039+0000 7f21080d9b80 10 client.16119 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:32.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.039+0000 7f21080d9b80 1 -- 172.21.15.44:0/2298892740 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x555d6a2b3210 con 0x555d6a3a0930 2023-11-16T16:52:32.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.039+0000 7f20d5ffb700 1 -- 172.21.15.44:0/2298892740 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555d6a38eba0 con 0x555d6a3a0930 2023-11-16T16:52:32.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.039+0000 7f20d5ffb700 1 -- 172.21.15.44:0/2298892740 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f20e4004ae0 con 0x555d6a3a0930 2023-11-16T16:52:32.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.039+0000 7f21080d9b80 10 client.16119 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:32.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.039+0000 7f21080d9b80 10 client.16119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:32.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.039+0000 7f21080d9b80 20 client.16119 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:32.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.039+0000 7f21080d9b80 1 --2- 172.21.15.44:0/2298892740 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x555d6a3b5510 0x555d6a3b7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:32.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.039+0000 7f21080d9b80 4 client.16119 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:32.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.039+0000 7f21080d9b80 1 -- 172.21.15.44:0/2298892740 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d6a2af120 con 0x555d6a3b5510 2023-11-16T16:52:32.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.040+0000 7f20f259b700 1 --2- 172.21.15.44:0/2298892740 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x555d6a3b5510 0x555d6a3b7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:32.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.040+0000 7f20f259b700 1 --2- 172.21.15.44:0/2298892740 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x555d6a3b5510 0x555d6a3b7920 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85c05bdf98a041d8 server_cookie=14e9c2ccaa414bb7 in_seq=0 out_seq=0 2023-11-16T16:52:32.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.040+0000 7f20d5ffb700 10 client.16119 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:32.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.041+0000 7f20d5ffb700 1 -- 172.21.15.44:0/2298892740 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555d6a2af120 con 0x555d6a3b5510 2023-11-16T16:52:32.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.041+0000 7f20d5ffb700 10 client.16119 handle_command_reply: tid=0 2023-11-16T16:52:32.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.041+0000 7f21080d9b80 1 client.16119 shutdown 2023-11-16T16:52:32.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.041+0000 7f21080d9b80 1 -- 172.21.15.44:0/2298892740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 msgr2=0x555d6a3a5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:32.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.041+0000 7f21080d9b80 1 --2- 172.21.15.44:0/2298892740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a3a5e30 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f20e400b6f0 tx=0x7f20e400f690 comp rx=0 tx=0).stop 2023-11-16T16:52:32.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.042+0000 7f21080d9b80 1 -- 172.21.15.44:0/2298892740 shutdown_connections 2023-11-16T16:52:32.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.042+0000 7f21080d9b80 1 --2- 172.21.15.44:0/2298892740 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x555d6a3b5510 0x555d6a3b7920 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.042+0000 7f21080d9b80 1 --2- 172.21.15.44:0/2298892740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a3a5e30 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.042+0000 7f21080d9b80 1 --2- 172.21.15.44:0/2298892740 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39bbe0 0x555d6a3a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.042+0000 7f20d5ffb700 0 client.16119 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:32.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.042+0000 7f21080d9b80 1 --2- 172.21.15.44:0/2298892740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39c4f0 0x555d6a3a58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.042+0000 7f21080d9b80 1 -- 172.21.15.44:0/2298892740 >> 172.21.15.44:0/2298892740 conn(0x555d6a3913e0 msgr2=0x555d6a3a1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:32.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.042+0000 7f21080d9b80 1 -- 172.21.15.44:0/2298892740 shutdown_connections 2023-11-16T16:52:32.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.042+0000 7f21080d9b80 1 -- 172.21.15.44:0/2298892740 wait complete. 2023-11-16T16:52:32.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.042+0000 7f21080d9b80 20 client.16119 trim_cache size 0 max 16384 2023-11-16T16:52:32.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.060+0000 7f21080d9b80 1 Processor -- start 2023-11-16T16:52:32.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.060+0000 7f21080d9b80 1 -- start start 2023-11-16T16:52:32.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.060+0000 7f21080d9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39bbe0 0x555d6a3a5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:32.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.060+0000 7f21080d9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39c4f0 0x555d6a396600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:32.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.060+0000 7f21080d9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a396b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:32.063 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.060+0000 7f21080d9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6a3a8440 con 0x555d6a3a0930 2023-11-16T16:52:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.060+0000 7f21080d9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6a39f520 con 0x555d6a39bbe0 2023-11-16T16:52:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.060+0000 7f21080d9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6a397080 con 0x555d6a39c4f0 2023-11-16T16:52:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.061+0000 7f20f2d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a396b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.061+0000 7f20f2d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a396b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33384/0 (socket says 172.21.15.44:33384) 2023-11-16T16:52:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.061+0000 7f20f2d9c700 1 -- 172.21.15.44:0/1977936047 learned_addr learned my addr 172.21.15.44:0/1977936047 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.061+0000 7f20f1d9a700 1 --2- 172.21.15.44:0/1977936047 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39c4f0 0x555d6a396600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.061+0000 7f20f259b700 1 --2- 172.21.15.44:0/1977936047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39bbe0 0x555d6a3a5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.061+0000 7f20f2d9c700 1 -- 172.21.15.44:0/1977936047 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39c4f0 msgr2=0x555d6a396600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.061+0000 7f20f2d9c700 1 --2- 172.21.15.44:0/1977936047 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39c4f0 0x555d6a396600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.061+0000 7f20f2d9c700 1 -- 172.21.15.44:0/1977936047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39bbe0 msgr2=0x555d6a3a5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.061+0000 7f20f2d9c700 1 --2- 172.21.15.44:0/1977936047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39bbe0 0x555d6a3a5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.061+0000 7f20f2d9c700 1 -- 172.21.15.44:0/1977936047 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d6a2b2f60 con 0x555d6a3a0930 2023-11-16T16:52:32.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.061+0000 7f20f259b700 1 --2- 172.21.15.44:0/1977936047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39bbe0 0x555d6a3a5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:52:32.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.061+0000 7f20f2d9c700 1 --2- 172.21.15.44:0/1977936047 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a396b40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f20e40098e0 tx=0x7f20e4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed4e4cf3aa069e70 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:32.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.062+0000 7f20e17fa700 1 -- 172.21.15.44:0/1977936047 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e402b5f0 con 0x555d6a3a0930 2023-11-16T16:52:32.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.062+0000 7f20e17fa700 1 -- 172.21.15.44:0/1977936047 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20e4005bc0 con 0x555d6a3a0930 2023-11-16T16:52:32.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.062+0000 7f20e17fa700 1 -- 172.21.15.44:0/1977936047 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e4022790 con 0x555d6a3a0930 2023-11-16T16:52:32.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.062+0000 7f21080d9b80 1 -- 172.21.15.44:0/1977936047 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 msgr2=0x555d6a396b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:32.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.062+0000 7f21080d9b80 1 --2- 172.21.15.44:0/1977936047 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a396b40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f20e40098e0 tx=0x7f20e4000bc0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.062+0000 7f21080d9b80 1 -- 172.21.15.44:0/1977936047 shutdown_connections 2023-11-16T16:52:32.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.062+0000 7f21080d9b80 1 --2- 172.21.15.44:0/1977936047 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a396b40 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.062+0000 7f21080d9b80 1 --2- 172.21.15.44:0/1977936047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39bbe0 0x555d6a3a5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.062+0000 7f21080d9b80 1 --2- 172.21.15.44:0/1977936047 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39c4f0 0x555d6a396600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.062+0000 7f21080d9b80 1 -- 172.21.15.44:0/1977936047 >> 172.21.15.44:0/1977936047 conn(0x555d6a3913e0 msgr2=0x555d6a3a2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:32.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.063+0000 7f21080d9b80 1 -- 172.21.15.44:0/1977936047 shutdown_connections 2023-11-16T16:52:32.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.063+0000 7f21080d9b80 1 -- 172.21.15.44:0/1977936047 wait complete. 2023-11-16T16:52:32.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.063+0000 7f21080d9b80 1 -- start start 2023-11-16T16:52:32.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.063+0000 7f21080d9b80 1 Processor -- start 2023-11-16T16:52:32.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f21080d9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39bbe0 0x555d6a3aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:32.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f21080d9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39c4f0 0x555d6a3aa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:32.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f21080d9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a4083d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:32.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f21080d9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6a3ab090 con 0x555d6a3a0930 2023-11-16T16:52:32.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f21080d9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6a3ab200 con 0x555d6a39bbe0 2023-11-16T16:52:32.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f21080d9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6a408940 con 0x555d6a39c4f0 2023-11-16T16:52:32.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f21080d9b80 10 client.0 _ll_register_callbacks cb 0x7f20dc00b8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:32.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f20f259b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39bbe0 0x555d6a3aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:32.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f20f2d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a4083d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:32.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f20f259b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39bbe0 0x555d6a3aa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:47682/0 (socket says 172.21.15.44:47682) 2023-11-16T16:52:32.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f20f259b700 1 -- 172.21.15.44:0/1590266523 learned_addr learned my addr 172.21.15.44:0/1590266523 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:32.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f20f1d9a700 1 --2- 172.21.15.44:0/1590266523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39c4f0 0x555d6a3aa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:32.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f20f259b700 1 -- 172.21.15.44:0/1590266523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39c4f0 msgr2=0x555d6a3aa8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:32.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f20f259b700 1 --2- 172.21.15.44:0/1590266523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39c4f0 0x555d6a3aa8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f20f259b700 1 -- 172.21.15.44:0/1590266523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 msgr2=0x555d6a4083d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:32.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f20f259b700 1 --2- 172.21.15.44:0/1590266523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a4083d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f20f259b700 1 -- 172.21.15.44:0/1590266523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d6a2b3210 con 0x555d6a39bbe0 2023-11-16T16:52:32.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f20f259b700 1 --2- 172.21.15.44:0/1590266523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39bbe0 0x555d6a3aa390 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f20e8009c10 tx=0x7f20e80cd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6522550fcd7534f6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:32.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.064+0000 7f20d5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:52:32.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.065+0000 7f20d5ffb700 1 -- 172.21.15.44:0/1590266523 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e809e0e0 con 0x555d6a39bbe0 2023-11-16T16:52:32.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.065+0000 7f21080d9b80 1 -- 172.21.15.44:0/1590266523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555d6a2bb6e0 con 0x555d6a39bbe0 2023-11-16T16:52:32.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.065+0000 7f20d5ffb700 1 -- 172.21.15.44:0/1590266523 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f20e80a2500 con 0x555d6a39bbe0 2023-11-16T16:52:32.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.065+0000 7f20d5ffb700 1 -- 172.21.15.44:0/1590266523 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20e80b8260 con 0x555d6a39bbe0 2023-11-16T16:52:32.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.065+0000 7f21080d9b80 10 client.26144 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:32.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.065+0000 7f21080d9b80 1 -- 172.21.15.44:0/1590266523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x555d6a103ce0 con 0x555d6a39bbe0 2023-11-16T16:52:32.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.065+0000 7f20d5ffb700 1 -- 172.21.15.44:0/1590266523 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555d6a2bb6e0 con 0x555d6a39bbe0 2023-11-16T16:52:32.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.065+0000 7f20d5ffb700 1 -- 172.21.15.44:0/1590266523 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f20e80995a0 con 0x555d6a39bbe0 2023-11-16T16:52:32.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.065+0000 7f21080d9b80 10 client.26144 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:32.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.065+0000 7f21080d9b80 10 client.26144 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:32.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.065+0000 7f21080d9b80 20 client.26144 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:32.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.065+0000 7f21080d9b80 1 --2- 172.21.15.44:0/1590266523 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x555d6a40a0d0 0x555d6a40c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:32.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.065+0000 7f21080d9b80 4 client.26144 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:52:32.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.066+0000 7f21080d9b80 1 -- 172.21.15.44:0/1590266523 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x555d6a13bff0 con 0x555d6a40a0d0 2023-11-16T16:52:32.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.066+0000 7f20f1d9a700 1 --2- 172.21.15.44:0/1590266523 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x555d6a40a0d0 0x555d6a40c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:32.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.066+0000 7f20f1d9a700 1 --2- 172.21.15.44:0/1590266523 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x555d6a40a0d0 0x555d6a40c4b0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d10983e24b5c619 server_cookie=7960304d4e8763a6 in_seq=0 out_seq=0 2023-11-16T16:52:32.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.066+0000 7f20d5ffb700 10 client.26144 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:32.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.067+0000 7f20d5ffb700 1 -- 172.21.15.44:0/1590266523 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x555d6a13bff0 con 0x555d6a40a0d0 2023-11-16T16:52:32.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.067+0000 7f20d5ffb700 10 client.26144 handle_command_reply: tid=0 2023-11-16T16:52:32.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.068+0000 7f21080d9b80 1 client.26144 shutdown 2023-11-16T16:52:32.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.068+0000 7f21080d9b80 1 -- 172.21.15.44:0/1590266523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39bbe0 msgr2=0x555d6a3aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:32.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.068+0000 7f21080d9b80 1 --2- 172.21.15.44:0/1590266523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39bbe0 0x555d6a3aa390 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f20e8009c10 tx=0x7f20e80cd040 comp rx=0 tx=0).stop 2023-11-16T16:52:32.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.068+0000 7f21080d9b80 1 -- 172.21.15.44:0/1590266523 shutdown_connections 2023-11-16T16:52:32.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.068+0000 7f21080d9b80 1 --2- 172.21.15.44:0/1590266523 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x555d6a40a0d0 0x555d6a40c4b0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.068+0000 7f21080d9b80 1 --2- 172.21.15.44:0/1590266523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555d6a3a0930 0x555d6a4083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.068+0000 7f21080d9b80 1 --2- 172.21.15.44:0/1590266523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d6a39bbe0 0x555d6a3aa390 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.068+0000 7f21080d9b80 1 --2- 172.21.15.44:0/1590266523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555d6a39c4f0 0x555d6a3aa8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.068+0000 7f20d5ffb700 0 client.26144 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:32.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.068+0000 7f21080d9b80 1 -- 172.21.15.44:0/1590266523 >> 172.21.15.44:0/1590266523 conn(0x555d6a3913e0 msgr2=0x555d6a3a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:32.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.068+0000 7f21080d9b80 1 -- 172.21.15.44:0/1590266523 shutdown_connections 2023-11-16T16:52:32.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.068+0000 7f21080d9b80 1 -- 172.21.15.44:0/1590266523 wait complete. 2023-11-16T16:52:32.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.068+0000 7f21080d9b80 20 client.26144 trim_cache size 0 max 16384 2023-11-16T16:52:32.073 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:52:32.073 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:52:32.074 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "c848b8ec-0aa2-4eb7-9410-09592e931369", 2023-11-16T16:52:32.074 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:52:32.074 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:52:32.074 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.069+0000 7f20f47ff700 1 -- 172.21.15.44:0/3927562346 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f20d80a12c0 msgr2=0x7f20d80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:32.074 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.069+0000 7f20f47ff700 1 --2- 172.21.15.44:0/3927562346 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f20d80a12c0 0x7f20d80a3770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f20ec199c80 tx=0x7f20dc00b410 comp rx=0 tx=0).stop 2023-11-16T16:52:32.074 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.069+0000 7f20f47ff700 1 -- 172.21.15.44:0/3927562346 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec101390 msgr2=0x7f20ec198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.069+0000 7f20f47ff700 1 --2- 172.21.15.44:0/3927562346 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec101390 0x7f20ec198670 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f20e800ef10 tx=0x7f20e800c3a0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.069+0000 7f20f47ff700 1 -- 172.21.15.44:0/3927562346 shutdown_connections 2023-11-16T16:52:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.069+0000 7f20f47ff700 1 --2- 172.21.15.44:0/3927562346 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f20d80a12c0 0x7f20d80a3770 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.069+0000 7f20f47ff700 1 --2- 172.21.15.44:0/3927562346 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20ec10a8b0 0x7f20ec19d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.069+0000 7f20f47ff700 1 --2- 172.21.15.44:0/3927562346 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec101390 0x7f20ec198670 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.069+0000 7f20f47ff700 1 --2- 172.21.15.44:0/3927562346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20ec101ca0 0x7f20ec198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.069+0000 7f20f47ff700 1 -- 172.21.15.44:0/3927562346 >> 172.21.15.44:0/3927562346 conn(0x7f20ec0770b0 msgr2=0x7f20ec0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.069+0000 7f20f47ff700 1 -- 172.21.15.44:0/3927562346 shutdown_connections 2023-11-16T16:52:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:32.069+0000 7f20f47ff700 1 -- 172.21.15.44:0/3927562346 wait complete. 2023-11-16T16:52:32.130 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:52:32.476 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:32.477 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:32.571 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:32.571 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:32.579 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:32.579 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:32.579 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 81 -lt 100 ']' 2023-11-16T16:52:32.580 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 81 + 1 2023-11-16T16:52:32.617 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=82 2023-11-16T16:52:32.617 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:32.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:32 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:52:32.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:32 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:32.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:32 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:52:32.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:32 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:32.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:32 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:52:32.624 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:32.624 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:32.627 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:32.628 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:32.628 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:32.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:32 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:52:32.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:32 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:32.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:32 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:52:32.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:32 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:32.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:32 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:52:32.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:32 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:52:32.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:32 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:32.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:32 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:52:32.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:32 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:32.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:32 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:52:33.026 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:33.026 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:33.153 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:33.153 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:33.187 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:33.187 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:33.187 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 82 -lt 100 ']' 2023-11-16T16:52:33.188 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 82 + 1 2023-11-16T16:52:33.250 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=83 2023-11-16T16:52:33.250 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:33.279 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:33.279 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:33.282 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:33.282 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:33.283 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:33.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:33 smithi044 ceph-mon[128750]: pgmap v81: 129 pgs: 129 active+clean; 2.7 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-11-16T16:52:33.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:33 smithi176 ceph-mon[131630]: pgmap v81: 129 pgs: 129 active+clean; 2.7 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-11-16T16:52:33.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:33 smithi077 ceph-mon[131204]: pgmap v81: 129 pgs: 129 active+clean; 2.7 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-11-16T16:52:33.926 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:33.926 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:34.221 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:34.221 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:34.245 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:34.246 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:34.246 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 83 -lt 100 ']' 2023-11-16T16:52:34.246 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 83 + 1 2023-11-16T16:52:34.341 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=84 2023-11-16T16:52:34.342 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:34.417 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:34.418 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:34.420 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:34.421 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:34.421 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:35.526 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:35.526 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:35.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:35 smithi044 ceph-mon[128750]: pgmap v82: 129 pgs: 129 active+clean; 2.7 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-11-16T16:52:35.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:35 smithi176 ceph-mon[131630]: pgmap v82: 129 pgs: 129 active+clean; 2.7 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-11-16T16:52:35.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:35 smithi077 ceph-mon[131204]: pgmap v82: 129 pgs: 129 active+clean; 2.7 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-11-16T16:52:35.871 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:35.871 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:35.908 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.905+0000 7f0a06dff700 1 -- 172.21.15.44:0/307856317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a00100cc0 msgr2=0x7f0a00101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:35.908 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.905+0000 7f0a06dff700 1 --2- 172.21.15.44:0/307856317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a00100cc0 0x7f0a00101090 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f09f40099c0 tx=0x7f09f4009cd0 comp rx=0 tx=0).stop 2023-11-16T16:52:35.908 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.906+0000 7f0a06dff700 1 -- 172.21.15.44:0/307856317 shutdown_connections 2023-11-16T16:52:35.908 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.906+0000 7f0a06dff700 1 --2- 172.21.15.44:0/307856317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a00105620 0x7f0a0010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:35.908 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.906+0000 7f0a06dff700 1 --2- 172.21.15.44:0/307856317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a00100cc0 0x7f0a00101090 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:35.908 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.906+0000 7f0a06dff700 1 --2- 172.21.15.44:0/307856317 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a00101660 0x7f0a00104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:35.908 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.906+0000 7f0a06dff700 1 -- 172.21.15.44:0/307856317 >> 172.21.15.44:0/307856317 conn(0x7f0a00077560 msgr2=0x7f0a00077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:35.908 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.906+0000 7f0a06dff700 1 -- 172.21.15.44:0/307856317 shutdown_connections 2023-11-16T16:52:35.908 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.906+0000 7f0a06dff700 1 -- 172.21.15.44:0/307856317 wait complete. 2023-11-16T16:52:35.909 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.907+0000 7f0a06dff700 1 Processor -- start 2023-11-16T16:52:35.909 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.908+0000 7f0a06dff700 1 -- start start 2023-11-16T16:52:35.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.908+0000 7f0a06dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a00100cc0 0x7f0a00196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:35.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.908+0000 7f0a06dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a00101660 0x7f0a00196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:35.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.908+0000 7f0a06dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a00105620 0x7f0a0019aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:35.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.908+0000 7f0a06dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0019b500 con 0x7f0a00100cc0 2023-11-16T16:52:35.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.908+0000 7f0a06dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0019b670 con 0x7f0a00105620 2023-11-16T16:52:35.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.908+0000 7f0a06dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0019b830 con 0x7f0a00101660 2023-11-16T16:52:35.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.909+0000 7f0a04b9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a00100cc0 0x7f0a00196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:35.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.909+0000 7f0a04b9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a00100cc0 0x7f0a00196520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46750/0 (socket says 172.21.15.44:46750) 2023-11-16T16:52:35.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.909+0000 7f0a04b9b700 1 -- 172.21.15.44:0/3808974775 learned_addr learned my addr 172.21.15.44:0/3808974775 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:35.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.909+0000 7f0a0539c700 1 --2- 172.21.15.44:0/3808974775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a00105620 0x7f0a0019aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:35.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.909+0000 7f09fffff700 1 --2- 172.21.15.44:0/3808974775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a00101660 0x7f0a00196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:35.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.909+0000 7f0a04b9b700 1 -- 172.21.15.44:0/3808974775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a00101660 msgr2=0x7f0a00196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:35.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.909+0000 7f0a04b9b700 1 --2- 172.21.15.44:0/3808974775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a00101660 0x7f0a00196a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:35.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.909+0000 7f0a04b9b700 1 -- 172.21.15.44:0/3808974775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a00105620 msgr2=0x7f0a0019aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:35.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.909+0000 7f0a04b9b700 1 --2- 172.21.15.44:0/3808974775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a00105620 0x7f0a0019aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:35.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.909+0000 7f0a04b9b700 1 -- 172.21.15.44:0/3808974775 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f40095f0 con 0x7f0a00100cc0 2023-11-16T16:52:35.913 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.909+0000 7f0a04b9b700 1 --2- 172.21.15.44:0/3808974775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a00100cc0 0x7f0a00196520 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f09f400b580 tx=0x7f09f400bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:35.913 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.910+0000 7f09fdffb700 1 -- 172.21.15.44:0/3808974775 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f401d070 con 0x7f0a00100cc0 2023-11-16T16:52:35.913 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.910+0000 7f09fdffb700 1 -- 172.21.15.44:0/3808974775 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09f4005050 con 0x7f0a00100cc0 2023-11-16T16:52:35.913 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.910+0000 7f0a06dff700 1 -- 172.21.15.44:0/3808974775 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a001103d0 con 0x7f0a00100cc0 2023-11-16T16:52:35.913 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.910+0000 7f09fdffb700 1 -- 172.21.15.44:0/3808974775 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4004a40 con 0x7f0a00100cc0 2023-11-16T16:52:35.913 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.910+0000 7f0a06dff700 1 -- 172.21.15.44:0/3808974775 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a00110240 con 0x7f0a00100cc0 2023-11-16T16:52:35.913 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:35.913 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:35.914 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 84 -lt 100 ']' 2023-11-16T16:52:35.914 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 84 + 1 2023-11-16T16:52:35.914 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.913+0000 7f09fdffb700 1 -- 172.21.15.44:0/3808974775 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f09f40051c0 con 0x7f0a00100cc0 2023-11-16T16:52:35.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.914+0000 7f09fdffb700 1 --2- 172.21.15.44:0/3808974775 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f09f00a12c0 0x7f09f00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:35.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.915+0000 7f09fdffb700 1 -- 172.21.15.44:0/3808974775 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f09f40cde60 con 0x7f0a00100cc0 2023-11-16T16:52:35.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.915+0000 7f09fffff700 1 --2- 172.21.15.44:0/3808974775 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f09f00a12c0 0x7f09f00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:35.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:35.916+0000 7f09fffff700 1 --2- 172.21.15.44:0/3808974775 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f09f00a12c0 0x7f09f00a3770 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0a00197b30 tx=0x7f09ec009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:35.983 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=85 2023-11-16T16:52:35.983 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:36.017 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.015+0000 7f0a1a7f5b80 1 Processor -- start 2023-11-16T16:52:36.017 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.015+0000 7f0a1a7f5b80 1 -- start start 2023-11-16T16:52:36.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.016+0000 7f0a1a7f5b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e31f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:36.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.016+0000 7f0a1a7f5b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e32470 0x559dd2e36370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:36.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.016+0000 7f0a1a7f5b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e368b0 0x559dd2e36d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.016+0000 7f0a1a7f5b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd2d37060 con 0x559dd2e31b60 2023-11-16T16:52:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.016+0000 7f0a1a7f5b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd2d898a0 con 0x559dd2e368b0 2023-11-16T16:52:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.016+0000 7f0a1a7f5b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd2d82d30 con 0x559dd2e32470 2023-11-16T16:52:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.016+0000 7f0a04b9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e31f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.016+0000 7f0a0539c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e368b0 0x559dd2e36d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.016+0000 7f0a0539c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e368b0 0x559dd2e36d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56226/0 (socket says 172.21.15.44:56226) 2023-11-16T16:52:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.016+0000 7f0a04b9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e31f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46760/0 (socket says 172.21.15.44:46760) 2023-11-16T16:52:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.016+0000 7f0a0539c700 1 -- 172.21.15.44:0/1613992972 learned_addr learned my addr 172.21.15.44:0/1613992972 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.016+0000 7f09fffff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e32470 0x559dd2e36370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.017+0000 7f0a0539c700 1 -- 172.21.15.44:0/1613992972 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e32470 msgr2=0x559dd2e36370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.017+0000 7f0a0539c700 1 --2- 172.21.15.44:0/1613992972 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e32470 0x559dd2e36370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.019 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.017+0000 7f0a0539c700 1 -- 172.21.15.44:0/1613992972 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 msgr2=0x559dd2e31f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:36.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.017+0000 7f0a0539c700 1 --2- 172.21.15.44:0/1613992972 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e31f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.017+0000 7f0a0539c700 1 -- 172.21.15.44:0/1613992972 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559dd2c04630 con 0x559dd2e368b0 2023-11-16T16:52:36.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.017+0000 7f0a04b9b700 1 --2- 172.21.15.44:0/1613992972 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e31f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:36.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.017+0000 7f0a0539c700 1 --2- 172.21.15.44:0/1613992972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e368b0 0x559dd2e36d00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f09f8011680 tx=0x7f09f8009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50e018f68d8eb44f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:36.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.017+0000 7f09eaffd700 1 -- 172.21.15.44:0/1613992972 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f800e040 con 0x559dd2e368b0 2023-11-16T16:52:36.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.017+0000 7f09eaffd700 1 -- 172.21.15.44:0/1613992972 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09f800f040 con 0x559dd2e368b0 2023-11-16T16:52:36.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.017+0000 7f09eaffd700 1 -- 172.21.15.44:0/1613992972 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f8004860 con 0x559dd2e368b0 2023-11-16T16:52:36.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.018+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/1613992972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e368b0 msgr2=0x559dd2e36d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:36.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.018+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/1613992972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e368b0 0x559dd2e36d00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f09f8011680 tx=0x7f09f8009920 comp rx=0 tx=0).stop 2023-11-16T16:52:36.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.018+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/1613992972 shutdown_connections 2023-11-16T16:52:36.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.018+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/1613992972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e368b0 0x559dd2e36d00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.018+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/1613992972 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e31f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.018+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/1613992972 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e32470 0x559dd2e36370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.023 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.018+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/1613992972 >> 172.21.15.44:0/1613992972 conn(0x559dd2e27360 msgr2=0x559dd2e2cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:36.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.018+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/1613992972 shutdown_connections 2023-11-16T16:52:36.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.018+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/1613992972 wait complete. 2023-11-16T16:52:36.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.019+0000 7f0a1a7f5b80 1 -- start start 2023-11-16T16:52:36.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.019+0000 7f0a1a7f5b80 1 Processor -- start 2023-11-16T16:52:36.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.019+0000 7f0a1a7f5b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e3e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:36.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.019+0000 7f0a1a7f5b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e32470 0x559dd2e3b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:36.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.020+0000 7f0a1a7f5b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e368b0 0x559dd2e3b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:36.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.020+0000 7f0a1a7f5b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd2d248e0 con 0x559dd2e31b60 2023-11-16T16:52:36.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.020+0000 7f0a1a7f5b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd2d24580 con 0x559dd2e32470 2023-11-16T16:52:36.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.020+0000 7f0a1a7f5b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd2c06920 con 0x559dd2e368b0 2023-11-16T16:52:36.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.020+0000 7f0a1a7f5b80 10 client.0 _ll_register_callbacks cb 0x559dd2d42780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:36.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.020+0000 7f0a04b9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e3e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:36.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.020+0000 7f0a04b9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e3e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46776/0 (socket says 172.21.15.44:46776) 2023-11-16T16:52:36.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.020+0000 7f0a04b9b700 1 -- 172.21.15.44:0/2503377381 learned_addr learned my addr 172.21.15.44:0/2503377381 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:36.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.020+0000 7f09fffff700 1 --2- 172.21.15.44:0/2503377381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e32470 0x559dd2e3b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:36.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.020+0000 7f0a0539c700 1 --2- 172.21.15.44:0/2503377381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e368b0 0x559dd2e3b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:36.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.020+0000 7f0a04b9b700 1 -- 172.21.15.44:0/2503377381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e368b0 msgr2=0x559dd2e3b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:36.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.020+0000 7f0a04b9b700 1 --2- 172.21.15.44:0/2503377381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e368b0 0x559dd2e3b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.020+0000 7f0a04b9b700 1 -- 172.21.15.44:0/2503377381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e32470 msgr2=0x559dd2e3b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:36.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.020+0000 7f0a04b9b700 1 --2- 172.21.15.44:0/2503377381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e32470 0x559dd2e3b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.020+0000 7f0a04b9b700 1 -- 172.21.15.44:0/2503377381 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f8011330 con 0x559dd2e31b60 2023-11-16T16:52:36.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.021+0000 7f0a04b9b700 1 --2- 172.21.15.44:0/2503377381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e3e050 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f09f40036c0 tx=0x7f09f409ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f30f22490115c88 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:36.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.021+0000 7f09dffff700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:52:36.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.021+0000 7f09dffff700 1 -- 172.21.15.44:0/2503377381 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f409a0d0 con 0x559dd2e31b60 2023-11-16T16:52:36.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.021+0000 7f09dffff700 1 -- 172.21.15.44:0/2503377381 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f09f409a230 con 0x559dd2e31b60 2023-11-16T16:52:36.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.021+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/2503377381 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559dd2e24b20 con 0x559dd2e31b60 2023-11-16T16:52:36.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.021+0000 7f09dffff700 1 -- 172.21.15.44:0/2503377381 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09f40b3260 con 0x559dd2e31b60 2023-11-16T16:52:36.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.021+0000 7f0a1a7f5b80 10 client.16137 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:36.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.021+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/2503377381 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x559dd2d49190 con 0x559dd2e31b60 2023-11-16T16:52:36.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.021+0000 7f09dffff700 1 -- 172.21.15.44:0/2503377381 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559dd2e24b20 con 0x559dd2e31b60 2023-11-16T16:52:36.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.022+0000 7f09dffff700 1 -- 172.21.15.44:0/2503377381 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f09f4004cc0 con 0x559dd2e31b60 2023-11-16T16:52:36.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.022+0000 7f0a1a7f5b80 10 client.16137 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:36.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.022+0000 7f0a1a7f5b80 10 client.16137 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:36.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.022+0000 7f0a1a7f5b80 20 client.16137 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:36.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.022+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/2503377381 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559dd2e4b4b0 0x559dd2e4d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:36.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.022+0000 7f0a1a7f5b80 4 client.16137 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:36.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.022+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/2503377381 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559dd2d450a0 con 0x559dd2e4b4b0 2023-11-16T16:52:36.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.022+0000 7f09fffff700 1 --2- 172.21.15.44:0/2503377381 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559dd2e4b4b0 0x559dd2e4d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:36.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.023+0000 7f09fffff700 1 --2- 172.21.15.44:0/2503377381 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559dd2e4b4b0 0x559dd2e4d8c0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c965fdf2e8722051 server_cookie=27e50377b8f9d343 in_seq=0 out_seq=0 2023-11-16T16:52:36.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.023+0000 7f09dffff700 10 client.16137 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:36.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.023+0000 7f09dffff700 1 -- 172.21.15.44:0/2503377381 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559dd2d450a0 con 0x559dd2e4b4b0 2023-11-16T16:52:36.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.023+0000 7f09dffff700 10 client.16137 handle_command_reply: tid=0 2023-11-16T16:52:36.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.023+0000 7f0a1a7f5b80 1 client.16137 shutdown 2023-11-16T16:52:36.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.024+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/2503377381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 msgr2=0x559dd2e3e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:36.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.024+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/2503377381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e3e050 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f09f40036c0 tx=0x7f09f409ec80 comp rx=0 tx=0).stop 2023-11-16T16:52:36.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.024+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/2503377381 shutdown_connections 2023-11-16T16:52:36.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.024+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/2503377381 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559dd2e4b4b0 0x559dd2e4d8c0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.024+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/2503377381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e368b0 0x559dd2e3b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.024+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/2503377381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e3e050 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.024+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/2503377381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e32470 0x559dd2e3b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.024+0000 7f09dffff700 0 client.16137 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.024+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/2503377381 >> 172.21.15.44:0/2503377381 conn(0x559dd2e27360 msgr2=0x559dd2e37d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.024+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/2503377381 shutdown_connections 2023-11-16T16:52:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.024+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/2503377381 wait complete. 2023-11-16T16:52:36.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.024+0000 7f0a1a7f5b80 20 client.16137 trim_cache size 0 max 16384 2023-11-16T16:52:36.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.042+0000 7f0a1a7f5b80 1 Processor -- start 2023-11-16T16:52:36.043 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.042+0000 7f0a1a7f5b80 1 -- start start 2023-11-16T16:52:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.042+0000 7f0a1a7f5b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e429b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.042+0000 7f0a1a7f5b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e32470 0x559dd2e2b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.042+0000 7f0a1a7f5b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e368b0 0x559dd2e3dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.042+0000 7f0a1a7f5b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd2e3c310 con 0x559dd2e31b60 2023-11-16T16:52:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.042+0000 7f0a1a7f5b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd2e4df20 con 0x559dd2e32470 2023-11-16T16:52:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.042+0000 7f0a1a7f5b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd2e42ef0 con 0x559dd2e368b0 2023-11-16T16:52:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.043+0000 7f0a04b9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e429b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.043+0000 7f0a04b9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e429b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46778/0 (socket says 172.21.15.44:46778) 2023-11-16T16:52:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.043+0000 7f0a04b9b700 1 -- 172.21.15.44:0/455660864 learned_addr learned my addr 172.21.15.44:0/455660864 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.043+0000 7f09fffff700 1 --2- 172.21.15.44:0/455660864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e32470 0x559dd2e2b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.043+0000 7f0a0539c700 1 --2- 172.21.15.44:0/455660864 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e368b0 0x559dd2e3dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.043+0000 7f0a04b9b700 1 -- 172.21.15.44:0/455660864 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e368b0 msgr2=0x559dd2e3dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.043+0000 7f0a04b9b700 1 --2- 172.21.15.44:0/455660864 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e368b0 0x559dd2e3dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.043+0000 7f0a04b9b700 1 -- 172.21.15.44:0/455660864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e32470 msgr2=0x559dd2e2b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.043+0000 7f0a04b9b700 1 --2- 172.21.15.44:0/455660864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e32470 0x559dd2e2b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.043+0000 7f0a04b9b700 1 -- 172.21.15.44:0/455660864 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559dd2d48ee0 con 0x559dd2e31b60 2023-11-16T16:52:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.043+0000 7f0a04b9b700 1 --2- 172.21.15.44:0/455660864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e429b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f09f4097650 tx=0x7f09f4009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d05fed5ad2339389 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.043+0000 7f09eb7fe700 1 -- 172.21.15.44:0/455660864 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4096d80 con 0x559dd2e31b60 2023-11-16T16:52:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.043+0000 7f09eb7fe700 1 -- 172.21.15.44:0/455660864 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09f4096b90 con 0x559dd2e31b60 2023-11-16T16:52:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.043+0000 7f09eb7fe700 1 -- 172.21.15.44:0/455660864 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f409a330 con 0x559dd2e31b60 2023-11-16T16:52:36.048 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.044+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/455660864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 msgr2=0x559dd2e429b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.044+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/455660864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e429b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f09f4097650 tx=0x7f09f4009310 comp rx=0 tx=0).stop 2023-11-16T16:52:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.044+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/455660864 shutdown_connections 2023-11-16T16:52:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.044+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/455660864 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e368b0 0x559dd2e3dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.044+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/455660864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e429b0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.044+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/455660864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e32470 0x559dd2e2b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.044+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/455660864 >> 172.21.15.44:0/455660864 conn(0x559dd2e27360 msgr2=0x559dd2e47c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.044+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/455660864 shutdown_connections 2023-11-16T16:52:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.044+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/455660864 wait complete. 2023-11-16T16:52:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.045+0000 7f0a1a7f5b80 1 -- start start 2023-11-16T16:52:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.045+0000 7f0a1a7f5b80 1 Processor -- start 2023-11-16T16:52:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f0a1a7f5b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e9bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f0a1a7f5b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e32470 0x559dd2e9bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:36.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f0a1a7f5b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e368b0 0x559dd2e976d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:36.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f0a1a7f5b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd2e97de0 con 0x559dd2e31b60 2023-11-16T16:52:36.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f0a1a7f5b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd2e97f50 con 0x559dd2e368b0 2023-11-16T16:52:36.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f0a1a7f5b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd2e98110 con 0x559dd2e32470 2023-11-16T16:52:36.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f0a1a7f5b80 10 client.0 _ll_register_callbacks cb 0x7f09f409abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:36.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f09fffff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e32470 0x559dd2e9bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:36.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f0a0539c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e368b0 0x559dd2e976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:36.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f09fffff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e32470 0x559dd2e9bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:35916/0 (socket says 172.21.15.44:35916) 2023-11-16T16:52:36.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f0a04b9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e9bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:36.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f09fffff700 1 -- 172.21.15.44:0/3831428809 learned_addr learned my addr 172.21.15.44:0/3831428809 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:36.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f09fffff700 1 -- 172.21.15.44:0/3831428809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e368b0 msgr2=0x559dd2e976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:36.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f09fffff700 1 --2- 172.21.15.44:0/3831428809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e368b0 0x559dd2e976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f09fffff700 1 -- 172.21.15.44:0/3831428809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 msgr2=0x559dd2e9bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:36.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f09fffff700 1 --2- 172.21.15.44:0/3831428809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e9bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.046+0000 7f09fffff700 1 -- 172.21.15.44:0/3831428809 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559dd2d49190 con 0x559dd2e32470 2023-11-16T16:52:36.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.047+0000 7f0a0539c700 1 --2- 172.21.15.44:0/3831428809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e368b0 0x559dd2e976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:36.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.047+0000 7f09fffff700 1 --2- 172.21.15.44:0/3831428809 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e32470 0x559dd2e9bff0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f09ec013010 tx=0x7f09ec0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da892881db63ac3b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:36.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.047+0000 7f09dffff700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:52:36.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.047+0000 7f09dffff700 1 -- 172.21.15.44:0/3831428809 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec022040 con 0x559dd2e32470 2023-11-16T16:52:36.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.047+0000 7f09dffff700 1 -- 172.21.15.44:0/3831428809 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f09ec02ac50 con 0x559dd2e32470 2023-11-16T16:52:36.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.047+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/3831428809 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f09f800f040 con 0x559dd2e32470 2023-11-16T16:52:36.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.047+0000 7f09dffff700 1 -- 172.21.15.44:0/3831428809 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09ec02ae10 con 0x559dd2e32470 2023-11-16T16:52:36.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.048+0000 7f0a1a7f5b80 10 client.26152 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:36.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.048+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/3831428809 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x559dd2b99c60 con 0x559dd2e32470 2023-11-16T16:52:36.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.048+0000 7f09dffff700 1 -- 172.21.15.44:0/3831428809 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09f800f040 con 0x559dd2e32470 2023-11-16T16:52:36.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.048+0000 7f09dffff700 1 -- 172.21.15.44:0/3831428809 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f09ec0336d0 con 0x559dd2e32470 2023-11-16T16:52:36.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.048+0000 7f0a1a7f5b80 10 client.26152 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:36.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.048+0000 7f0a1a7f5b80 10 client.26152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:36.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.048+0000 7f0a1a7f5b80 20 client.26152 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:36.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.048+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/3831428809 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559dd2ea4a50 0x559dd2ea6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:36.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.048+0000 7f0a1a7f5b80 4 client.26152 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:52:36.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.048+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/3831428809 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559dd2d51660 con 0x559dd2ea4a50 2023-11-16T16:52:36.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.048+0000 7f0a04b9b700 1 --2- 172.21.15.44:0/3831428809 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559dd2ea4a50 0x559dd2ea6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:36.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.049+0000 7f0a04b9b700 1 --2- 172.21.15.44:0/3831428809 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559dd2ea4a50 0x559dd2ea6e60 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38f5579402bcca81 server_cookie=b8bd6903d38031d8 in_seq=0 out_seq=0 2023-11-16T16:52:36.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.049+0000 7f09dffff700 10 client.26152 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:36.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.049+0000 7f09dffff700 1 -- 172.21.15.44:0/3831428809 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x559dd2d51660 con 0x559dd2ea4a50 2023-11-16T16:52:36.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.049+0000 7f09dffff700 10 client.26152 handle_command_reply: tid=0 2023-11-16T16:52:36.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.049+0000 7f0a1a7f5b80 1 client.26152 shutdown 2023-11-16T16:52:36.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.049+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/3831428809 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e32470 msgr2=0x559dd2e9bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:36.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.049+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/3831428809 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e32470 0x559dd2e9bff0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f09ec013010 tx=0x7f09ec0047c0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.049+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/3831428809 shutdown_connections 2023-11-16T16:52:36.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.049+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/3831428809 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559dd2ea4a50 0x559dd2ea6e60 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.049+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/3831428809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559dd2e368b0 0x559dd2e976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.049+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/3831428809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559dd2e31b60 0x559dd2e9bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.049+0000 7f0a1a7f5b80 1 --2- 172.21.15.44:0/3831428809 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559dd2e32470 0x559dd2e9bff0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.049+0000 7f09dffff700 0 client.26152 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:36.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.049+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/3831428809 >> 172.21.15.44:0/3831428809 conn(0x559dd2e27360 msgr2=0x559dd2e2b550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:36.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.050+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/3831428809 shutdown_connections 2023-11-16T16:52:36.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.050+0000 7f0a1a7f5b80 1 -- 172.21.15.44:0/3831428809 wait complete. 2023-11-16T16:52:36.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.050+0000 7f0a1a7f5b80 20 client.26152 trim_cache size 0 max 16384 2023-11-16T16:52:36.055 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:36.056 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:52:36.056 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T16:52:36.056 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T16:52:36.056 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:52:36.056 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:36.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.050+0000 7f0a06dff700 1 -- 172.21.15.44:0/3808974775 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f09f00a12c0 msgr2=0x7f09f00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:36.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.050+0000 7f0a06dff700 1 --2- 172.21.15.44:0/3808974775 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f09f00a12c0 0x7f09f00a3770 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0a00197b30 tx=0x7f09ec009260 comp rx=0 tx=0).stop 2023-11-16T16:52:36.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.050+0000 7f0a06dff700 1 -- 172.21.15.44:0/3808974775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a00100cc0 msgr2=0x7f0a00196520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:36.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.050+0000 7f0a06dff700 1 --2- 172.21.15.44:0/3808974775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a00100cc0 0x7f0a00196520 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f09f400b580 tx=0x7f09f400bfd0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.050+0000 7f0a06dff700 1 -- 172.21.15.44:0/3808974775 shutdown_connections 2023-11-16T16:52:36.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.050+0000 7f0a06dff700 1 --2- 172.21.15.44:0/3808974775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a00105620 0x7f0a0019aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.050+0000 7f0a06dff700 1 --2- 172.21.15.44:0/3808974775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a00100cc0 0x7f0a00196520 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.050+0000 7f0a06dff700 1 --2- 172.21.15.44:0/3808974775 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f09f00a12c0 0x7f09f00a3770 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.050+0000 7f0a06dff700 1 --2- 172.21.15.44:0/3808974775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a00101660 0x7f0a00196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:36.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.051+0000 7f0a06dff700 1 -- 172.21.15.44:0/3808974775 >> 172.21.15.44:0/3808974775 conn(0x7f0a00077560 msgr2=0x7f0a000febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:36.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.051+0000 7f0a06dff700 1 -- 172.21.15.44:0/3808974775 shutdown_connections 2023-11-16T16:52:36.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:36.051+0000 7f0a06dff700 1 -- 172.21.15.44:0/3808974775 wait complete. 2023-11-16T16:52:36.058 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:36.058 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:36.059 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:36.088 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T16:52:36.088 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:52:36.709 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:36.709 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:36.770 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:36.771 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:36.802 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:36.802 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:36.803 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 85 -lt 100 ']' 2023-11-16T16:52:36.803 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 85 + 1 2023-11-16T16:52:36.833 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=86 2023-11-16T16:52:36.833 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:36.867 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:36.867 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:36.870 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:36.870 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:36.871 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:37.376 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:37.376 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:37.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:37 smithi044 ceph-mon[128750]: pgmap v83: 129 pgs: 129 active+clean; 2.7 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-11-16T16:52:37.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:37 smithi176 ceph-mon[131630]: pgmap v83: 129 pgs: 129 active+clean; 2.7 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-11-16T16:52:37.679 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:37.679 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:37.699 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:37.700 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:37.700 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 86 -lt 100 ']' 2023-11-16T16:52:37.700 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 86 + 1 2023-11-16T16:52:37.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:37 smithi077 ceph-mon[131204]: pgmap v83: 129 pgs: 129 active+clean; 2.7 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-11-16T16:52:37.741 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=87 2023-11-16T16:52:37.741 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:37.864 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:37.864 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:37.870 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:37.870 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:37.870 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:38.692 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:38.692 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:38.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.863+0000 7f6d3d7ff700 1 -- 172.21.15.44:0/1994041054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d381015f0 msgr2=0x7f6d38104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:38.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.863+0000 7f6d3d7ff700 1 --2- 172.21.15.44:0/1994041054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d381015f0 0x7f6d38104e80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c009920 tx=0x7f6d2c009c30 comp rx=0 tx=0).stop 2023-11-16T16:52:38.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.863+0000 7f6d3d7ff700 1 -- 172.21.15.44:0/1994041054 shutdown_connections 2023-11-16T16:52:38.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.863+0000 7f6d3d7ff700 1 --2- 172.21.15.44:0/1994041054 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6d38105560 0x7f6d3810d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.863+0000 7f6d3d7ff700 1 --2- 172.21.15.44:0/1994041054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d38100c50 0x7f6d38101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.863+0000 7f6d3d7ff700 1 --2- 172.21.15.44:0/1994041054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d381015f0 0x7f6d38104e80 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.863+0000 7f6d3d7ff700 1 -- 172.21.15.44:0/1994041054 >> 172.21.15.44:0/1994041054 conn(0x7f6d38077550 msgr2=0x7f6d38077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:38.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.863+0000 7fc0253ff700 1 -- 172.21.15.44:0/4081442738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc020072e90 msgr2=0x7fc020073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:38.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.863+0000 7fc0253ff700 1 --2- 172.21.15.44:0/4081442738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc020072e90 0x7fc020073260 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc010009990 tx=0x7fc010009ca0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.864+0000 7fc0253ff700 1 -- 172.21.15.44:0/4081442738 shutdown_connections 2023-11-16T16:52:38.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.864+0000 7fc0253ff700 1 --2- 172.21.15.44:0/4081442738 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0201112f0 0x7fc0201136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.867 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.864+0000 7fc0253ff700 1 --2- 172.21.15.44:0/4081442738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc020072e90 0x7fc020073260 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.867 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.864+0000 7fc0253ff700 1 --2- 172.21.15.44:0/4081442738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0200737a0 0x7fc020110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.867 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.864+0000 7fc0253ff700 1 -- 172.21.15.44:0/4081442738 >> 172.21.15.44:0/4081442738 conn(0x7fc020077020 msgr2=0x7fc020077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:38.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.864+0000 7f6d3d7ff700 1 -- 172.21.15.44:0/1994041054 shutdown_connections 2023-11-16T16:52:38.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.864+0000 7f6d3d7ff700 1 -- 172.21.15.44:0/1994041054 wait complete. 2023-11-16T16:52:38.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.864+0000 7fc0253ff700 1 -- 172.21.15.44:0/4081442738 shutdown_connections 2023-11-16T16:52:38.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.864+0000 7fc0253ff700 1 -- 172.21.15.44:0/4081442738 wait complete. 2023-11-16T16:52:38.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.865+0000 7f6d3d7ff700 1 Processor -- start 2023-11-16T16:52:38.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.865+0000 7fc0253ff700 1 Processor -- start 2023-11-16T16:52:38.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.865+0000 7fc0253ff700 1 -- start start 2023-11-16T16:52:38.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.866+0000 7f6d3d7ff700 1 -- start start 2023-11-16T16:52:38.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.866+0000 7fc0253ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc020072e90 0x7fc0201a27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:38.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.866+0000 7fc0253ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0200737a0 0x7fc0201a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:38.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.866+0000 7fc0253ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0201112f0 0x7fc02019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:38.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.866+0000 7fc0253ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02019cea0 con 0x7fc0201112f0 2023-11-16T16:52:38.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.866+0000 7fc0253ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02019d010 con 0x7fc020072e90 2023-11-16T16:52:38.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.866+0000 7fc0253ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02019d1d0 con 0x7fc0200737a0 2023-11-16T16:52:38.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.866+0000 7f6d3d7ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d38100c50 0x7f6d38072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:38.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.866+0000 7fc01effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc020072e90 0x7fc0201a27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:38.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.866+0000 7fc01effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc020072e90 0x7fc0201a27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56292/0 (socket says 172.21.15.44:56292) 2023-11-16T16:52:38.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.866+0000 7fc01effd700 1 -- 172.21.15.44:0/1249659974 learned_addr learned my addr 172.21.15.44:0/1249659974 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:38.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.866+0000 7f6d3d7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d381015f0 0x7f6d38073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:38.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.866+0000 7f6d3d7ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6d38105560 0x7f6d381a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:38.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.866+0000 7f6d3d7ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d38073b60 con 0x7f6d38100c50 2023-11-16T16:52:38.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.866+0000 7f6d3d7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d381a4ba0 con 0x7f6d381015f0 2023-11-16T16:52:38.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.866+0000 7f6d3d7ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d381a4ce0 con 0x7f6d38105560 2023-11-16T16:52:38.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.866+0000 7fc01e7fc700 1 --2- 172.21.15.44:0/1249659974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0200737a0 0x7fc0201a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:38.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.866+0000 7fc01f7fe700 1 --2- 172.21.15.44:0/1249659974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0201112f0 0x7fc02019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:38.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.867+0000 7f6d377fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6d38105560 0x7f6d381a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:38.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.867+0000 7fc01effd700 1 -- 172.21.15.44:0/1249659974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0200737a0 msgr2=0x7fc0201a2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:38.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.867+0000 7fc01effd700 1 --2- 172.21.15.44:0/1249659974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0200737a0 0x7fc0201a2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.867+0000 7fc01effd700 1 -- 172.21.15.44:0/1249659974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0201112f0 msgr2=0x7fc02019c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:38.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.867+0000 7fc01effd700 1 --2- 172.21.15.44:0/1249659974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0201112f0 0x7fc02019c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.867+0000 7fc01effd700 1 -- 172.21.15.44:0/1249659974 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0100095f0 con 0x7fc020072e90 2023-11-16T16:52:38.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.867+0000 7fc01f7fe700 1 --2- 172.21.15.44:0/1249659974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0201112f0 0x7fc02019c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:52:38.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.867+0000 7fc01e7fc700 1 --2- 172.21.15.44:0/1249659974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0200737a0 0x7fc0201a2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:52:38.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.867+0000 7f6d377fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6d38105560 0x7f6d381a4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:35948/0 (socket says 172.21.15.44:35948) 2023-11-16T16:52:38.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.867+0000 7f6d377fe700 1 -- 172.21.15.44:0/3316062042 learned_addr learned my addr 172.21.15.44:0/3316062042 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:38.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.867+0000 7f6d367fc700 1 --2- 172.21.15.44:0/3316062042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d381015f0 0x7f6d38073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:38.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.867+0000 7f6d36ffd700 1 --2- 172.21.15.44:0/3316062042 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d38100c50 0x7f6d38072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:38.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.867+0000 7f6d377fe700 1 -- 172.21.15.44:0/3316062042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d381015f0 msgr2=0x7f6d38073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:38.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.867+0000 7f6d377fe700 1 --2- 172.21.15.44:0/3316062042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d381015f0 0x7f6d38073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.867+0000 7f6d377fe700 1 -- 172.21.15.44:0/3316062042 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d38100c50 msgr2=0x7f6d38072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.867+0000 7f6d377fe700 1 --2- 172.21.15.44:0/3316062042 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d38100c50 0x7f6d38072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.867+0000 7fc01effd700 1 --2- 172.21.15.44:0/1249659974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc020072e90 0x7fc0201a27c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc010005910 tx=0x7fc0100057d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.867+0000 7f6d377fe700 1 -- 172.21.15.44:0/3316062042 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d280095f0 con 0x7f6d38105560 2023-11-16T16:52:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.867+0000 7f6d36ffd700 1 --2- 172.21.15.44:0/3316062042 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d38100c50 0x7f6d38072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:52:38.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.867+0000 7fc007fff700 1 -- 172.21.15.44:0/1249659974 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01001d070 con 0x7fc020072e90 2023-11-16T16:52:38.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.867+0000 7fc0253ff700 1 -- 172.21.15.44:0/1249659974 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc020116920 con 0x7fc020072e90 2023-11-16T16:52:38.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.867+0000 7fc007fff700 1 -- 172.21.15.44:0/1249659974 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc010022470 con 0x7fc020072e90 2023-11-16T16:52:38.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.867+0000 7fc007fff700 1 -- 172.21.15.44:0/1249659974 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc010017650 con 0x7fc020072e90 2023-11-16T16:52:38.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.867+0000 7f6d377fe700 1 --2- 172.21.15.44:0/3316062042 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6d38105560 0x7f6d381a4660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6d3000ef30 tx=0x7f6d3000c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:38.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.868+0000 7fc0253ff700 1 -- 172.21.15.44:0/1249659974 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc020116790 con 0x7fc020072e90 2023-11-16T16:52:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.868+0000 7f6d1ffff700 1 -- 172.21.15.44:0/3316062042 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30018070 con 0x7f6d38105560 2023-11-16T16:52:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.868+0000 7f6d1ffff700 1 -- 172.21.15.44:0/3316062042 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d3000f040 con 0x7f6d38105560 2023-11-16T16:52:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.868+0000 7f6d3d7ff700 1 -- 172.21.15.44:0/3316062042 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2c009520 con 0x7f6d38105560 2023-11-16T16:52:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.868+0000 7f6d1ffff700 1 -- 172.21.15.44:0/3316062042 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30003dc0 con 0x7f6d38105560 2023-11-16T16:52:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.868+0000 7f6d3d7ff700 1 -- 172.21.15.44:0/3316062042 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d38110370 con 0x7f6d38105560 2023-11-16T16:52:38.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.869+0000 7fc0253ff700 1 -- 172.21.15.44:0/1249659974 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc020116ab0 con 0x7fc020072e90 2023-11-16T16:52:38.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.872+0000 7fc007fff700 1 -- 172.21.15.44:0/1249659974 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fc0100225e0 con 0x7fc020072e90 2023-11-16T16:52:38.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.873+0000 7f6d1ffff700 1 -- 172.21.15.44:0/3316062042 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f6d3000caa0 con 0x7f6d38105560 2023-11-16T16:52:38.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.875+0000 7fc007fff700 1 --2- 172.21.15.44:0/1249659974 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc0080a12c0 0x7fc0080a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:38.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.875+0000 7fc007fff700 1 -- 172.21.15.44:0/1249659974 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fc0100ce500 con 0x7fc020072e90 2023-11-16T16:52:38.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.875+0000 7fc01e7fc700 1 --2- 172.21.15.44:0/1249659974 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc0080a12c0 0x7fc0080a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:38.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.875+0000 7f6d1ffff700 1 --2- 172.21.15.44:0/3316062042 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6d200a1340 0x7f6d200a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:38.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.875+0000 7fc01e7fc700 1 --2- 172.21.15.44:0/1249659974 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc0080a12c0 0x7fc0080a3770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc020113ac0 tx=0x7fc014009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:38.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.876+0000 7f6d36ffd700 1 --2- 172.21.15.44:0/3316062042 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6d200a1340 0x7f6d200a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:38.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.876+0000 7f6d1ffff700 1 -- 172.21.15.44:0/3316062042 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f6d300ccc70 con 0x7f6d38105560 2023-11-16T16:52:38.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.876+0000 7f6d36ffd700 1 --2- 172.21.15.44:0/3316062042 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6d200a1340 0x7f6d200a37f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6d28005fd0 tx=0x7f6d28009310 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:38.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:38.878+0000 7fc007fff700 1 -- 172.21.15.44:0/1249659974 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc010097620 con 0x7fc020072e90 2023-11-16T16:52:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.973+0000 7f6d51211b80 1 Processor -- start 2023-11-16T16:52:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.973+0000 7f6d51211b80 1 -- start start 2023-11-16T16:52:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.973+0000 7f6d51211b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c1e36b0 0x55f25c38d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.973+0000 7f6d51211b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c38d720 0x55f25c387560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.973+0000 7f6d51211b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c38db90 0x55f25c387b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.974+0000 7f6d51211b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25c28d000 con 0x55f25c38db90 2023-11-16T16:52:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.974+0000 7f6d51211b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25c2df840 con 0x55f25c38d720 2023-11-16T16:52:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.974+0000 7f6d51211b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25c2d8cd0 con 0x55f25c1e36b0 2023-11-16T16:52:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.974+0000 7f6d36ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c1e36b0 0x55f25c38d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.974+0000 7f6d377fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c38db90 0x55f25c387b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.974+0000 7f6d36ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c1e36b0 0x55f25c38d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:35954/0 (socket says 172.21.15.44:35954) 2023-11-16T16:52:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.974+0000 7f6d36ffd700 1 -- 172.21.15.44:0/133388895 learned_addr learned my addr 172.21.15.44:0/133388895 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.974+0000 7f6d367fc700 1 --2- 172.21.15.44:0/133388895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c38d720 0x55f25c387560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.974+0000 7f6d36ffd700 1 -- 172.21.15.44:0/133388895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c38d720 msgr2=0x55f25c387560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.974+0000 7f6d36ffd700 1 --2- 172.21.15.44:0/133388895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c38d720 0x55f25c387560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.974+0000 7f6d36ffd700 1 -- 172.21.15.44:0/133388895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c38db90 msgr2=0x55f25c387b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.974+0000 7f6d36ffd700 1 --2- 172.21.15.44:0/133388895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c38db90 0x55f25c387b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.974+0000 7f6d36ffd700 1 -- 172.21.15.44:0/133388895 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f25c15a5d0 con 0x55f25c1e36b0 2023-11-16T16:52:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.974+0000 7f6d377fe700 1 --2- 172.21.15.44:0/133388895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c38db90 0x55f25c387b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.975+0000 7f6d36ffd700 1 --2- 172.21.15.44:0/133388895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c1e36b0 0x55f25c38d1e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6d28010f10 tx=0x7f6d28013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e2260ffcd9577c0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.975+0000 7f6d1d7fa700 1 -- 172.21.15.44:0/133388895 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d28012dc0 con 0x55f25c1e36b0 2023-11-16T16:52:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.975+0000 7f6d1d7fa700 1 -- 172.21.15.44:0/133388895 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d28009a30 con 0x55f25c1e36b0 2023-11-16T16:52:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.975+0000 7f6d1d7fa700 1 -- 172.21.15.44:0/133388895 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2801e730 con 0x55f25c1e36b0 2023-11-16T16:52:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.976+0000 7f6d51211b80 1 -- 172.21.15.44:0/133388895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c1e36b0 msgr2=0x55f25c38d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.976+0000 7f6d51211b80 1 --2- 172.21.15.44:0/133388895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c1e36b0 0x55f25c38d1e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6d28010f10 tx=0x7f6d28013000 comp rx=0 tx=0).stop 2023-11-16T16:52:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.976+0000 7f6d51211b80 1 -- 172.21.15.44:0/133388895 shutdown_connections 2023-11-16T16:52:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.976+0000 7f6d51211b80 1 --2- 172.21.15.44:0/133388895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c38db90 0x55f25c387b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.976+0000 7f6d51211b80 1 --2- 172.21.15.44:0/133388895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c1e36b0 0x55f25c38d1e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.976+0000 7f6d51211b80 1 --2- 172.21.15.44:0/133388895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c38d720 0x55f25c387560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.976+0000 7f6d51211b80 1 -- 172.21.15.44:0/133388895 >> 172.21.15.44:0/133388895 conn(0x55f25c37cf50 msgr2=0x55f25c382270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.976+0000 7f6d51211b80 1 -- 172.21.15.44:0/133388895 shutdown_connections 2023-11-16T16:52:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.976+0000 7f6d51211b80 1 -- 172.21.15.44:0/133388895 wait complete. 2023-11-16T16:52:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.977+0000 7f6d51211b80 1 -- start start 2023-11-16T16:52:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.977+0000 7f6d51211b80 1 Processor -- start 2023-11-16T16:52:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.977+0000 7f6d51211b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c1e36b0 0x55f25c393ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.978+0000 7f6d51211b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c38d720 0x55f25c390ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.978+0000 7f6d51211b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 0x55f25c391420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.978+0000 7f6d51211b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25c27a880 con 0x55f25c1e36b0 2023-11-16T16:52:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.978+0000 7f6d51211b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25c27a520 con 0x55f25c38d720 2023-11-16T16:52:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.978+0000 7f6d51211b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25c15c8c0 con 0x55f25c38db90 2023-11-16T16:52:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.978+0000 7f6d51211b80 10 client.0 _ll_register_callbacks cb 0x55f25c298720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.978+0000 7f6d36ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c1e36b0 0x55f25c393ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.978+0000 7f6d36ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c1e36b0 0x55f25c393ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46828/0 (socket says 172.21.15.44:46828) 2023-11-16T16:52:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.978+0000 7f6d36ffd700 1 -- 172.21.15.44:0/966544864 learned_addr learned my addr 172.21.15.44:0/966544864 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.978+0000 7f6d367fc700 1 --2- 172.21.15.44:0/966544864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c38d720 0x55f25c390ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.978+0000 7f6d377fe700 1 --2- 172.21.15.44:0/966544864 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 0x55f25c391420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.978+0000 7f6d36ffd700 1 -- 172.21.15.44:0/966544864 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 msgr2=0x55f25c391420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.978+0000 7f6d36ffd700 1 --2- 172.21.15.44:0/966544864 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 0x55f25c391420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.978+0000 7f6d36ffd700 1 -- 172.21.15.44:0/966544864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c38d720 msgr2=0x55f25c390ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.978+0000 7f6d36ffd700 1 --2- 172.21.15.44:0/966544864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c38d720 0x55f25c390ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.978+0000 7f6d36ffd700 1 -- 172.21.15.44:0/966544864 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d28010bf0 con 0x55f25c1e36b0 2023-11-16T16:52:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.979+0000 7f6d36ffd700 1 --2- 172.21.15.44:0/966544864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c1e36b0 0x55f25c393ad0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6d28012a10 tx=0x7f6d28012df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c00f0005316627e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.979+0000 7f6d1aff5700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:52:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.979+0000 7f6d1aff5700 1 -- 172.21.15.44:0/966544864 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2802e070 con 0x55f25c1e36b0 2023-11-16T16:52:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.979+0000 7f6d51211b80 1 -- 172.21.15.44:0/966544864 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f25c29b040 con 0x55f25c1e36b0 2023-11-16T16:52:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.979+0000 7f6d1aff5700 1 -- 172.21.15.44:0/966544864 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f6d2801f660 con 0x55f25c1e36b0 2023-11-16T16:52:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.979+0000 7f6d1aff5700 1 -- 172.21.15.44:0/966544864 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d28005da0 con 0x55f25c1e36b0 2023-11-16T16:52:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.979+0000 7f6d51211b80 10 client.16158 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.979+0000 7f6d51211b80 1 -- 172.21.15.44:0/966544864 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55f25c29f130 con 0x55f25c1e36b0 2023-11-16T16:52:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.980+0000 7f6d1aff5700 1 -- 172.21.15.44:0/966544864 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f25c29b040 con 0x55f25c1e36b0 2023-11-16T16:52:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.980+0000 7f6d1aff5700 1 -- 172.21.15.44:0/966544864 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6d28028c30 con 0x55f25c1e36b0 2023-11-16T16:52:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.980+0000 7f6d51211b80 10 client.16158 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.980+0000 7f6d51211b80 10 client.16158 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.980+0000 7f6d51211b80 20 client.16158 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.980+0000 7f6d51211b80 1 --2- 172.21.15.44:0/966544864 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f25c3a0e60 0x55f25c3a3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.980+0000 7f6d51211b80 4 client.16158 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.980+0000 7f6d51211b80 1 -- 172.21.15.44:0/966544864 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f25c2a7600 con 0x55f25c3a0e60 2023-11-16T16:52:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.980+0000 7f6d367fc700 1 --2- 172.21.15.44:0/966544864 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f25c3a0e60 0x55f25c3a3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.981+0000 7f6d367fc700 1 --2- 172.21.15.44:0/966544864 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f25c3a0e60 0x55f25c3a3270 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2a5424774b4f35 server_cookie=53386f36ef3aa6dc in_seq=0 out_seq=0 2023-11-16T16:52:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.981+0000 7f6d1aff5700 10 client.16158 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.982+0000 7f6d1aff5700 1 -- 172.21.15.44:0/966544864 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f25c2a7600 con 0x55f25c3a0e60 2023-11-16T16:52:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.982+0000 7f6d1aff5700 10 client.16158 handle_command_reply: tid=0 2023-11-16T16:52:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.982+0000 7f6d51211b80 1 client.16158 shutdown 2023-11-16T16:52:38.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.982+0000 7f6d51211b80 1 -- 172.21.15.44:0/966544864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c1e36b0 msgr2=0x55f25c393ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:38.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.982+0000 7f6d51211b80 1 --2- 172.21.15.44:0/966544864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c1e36b0 0x55f25c393ad0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6d28012a10 tx=0x7f6d28012df0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.982+0000 7f6d51211b80 1 -- 172.21.15.44:0/966544864 shutdown_connections 2023-11-16T16:52:38.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.982+0000 7f6d51211b80 1 --2- 172.21.15.44:0/966544864 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f25c3a0e60 0x55f25c3a3270 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.982+0000 7f6d51211b80 1 --2- 172.21.15.44:0/966544864 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 0x55f25c391420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.982+0000 7f6d51211b80 1 --2- 172.21.15.44:0/966544864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c1e36b0 0x55f25c393ad0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.982+0000 7f6d51211b80 1 --2- 172.21.15.44:0/966544864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c38d720 0x55f25c390ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:38.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.982+0000 7f6d51211b80 1 -- 172.21.15.44:0/966544864 >> 172.21.15.44:0/966544864 conn(0x55f25c37cf50 msgr2=0x55f25c3817e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:38.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.982+0000 7f6d1aff5700 0 client.16158 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:38.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.982+0000 7f6d51211b80 1 -- 172.21.15.44:0/966544864 shutdown_connections 2023-11-16T16:52:38.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.982+0000 7f6d51211b80 1 -- 172.21.15.44:0/966544864 wait complete. 2023-11-16T16:52:38.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.982+0000 7f6d51211b80 20 client.16158 trim_cache size 0 max 16384 2023-11-16T16:52:39.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.998+0000 7f6d51211b80 1 Processor -- start 2023-11-16T16:52:39.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.998+0000 7f6d51211b80 1 -- start start 2023-11-16T16:52:39.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.999+0000 7f6d51211b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c1e36b0 0x55f25c398070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:39.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.999+0000 7f6d51211b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c38d720 0x55f25c381f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:39.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.999+0000 7f6d51211b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 0x55f25c393600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:39.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.999+0000 7f6d51211b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25c38d1e0 con 0x55f25c38d720 2023-11-16T16:52:39.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.999+0000 7f6d51211b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25c39d950 con 0x55f25c1e36b0 2023-11-16T16:52:39.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.999+0000 7f6d51211b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25c3833e0 con 0x55f25c38db90 2023-11-16T16:52:39.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.999+0000 7f6d36ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c1e36b0 0x55f25c398070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:39.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.999+0000 7f6d367fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c38d720 0x55f25c381f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:39.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.999+0000 7f6d36ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c1e36b0 0x55f25c398070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56326/0 (socket says 172.21.15.44:56326) 2023-11-16T16:52:39.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.999+0000 7f6d377fe700 1 --2- 172.21.15.44:0/2929950726 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 0x55f25c393600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:39.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:38.999+0000 7f6d36ffd700 1 -- 172.21.15.44:0/2929950726 learned_addr learned my addr 172.21.15.44:0/2929950726 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:39.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.000+0000 7f6d377fe700 1 -- 172.21.15.44:0/2929950726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c1e36b0 msgr2=0x55f25c398070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:39.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.000+0000 7f6d377fe700 1 --2- 172.21.15.44:0/2929950726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c1e36b0 0x55f25c398070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.000+0000 7f6d377fe700 1 -- 172.21.15.44:0/2929950726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c38d720 msgr2=0x55f25c381f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:39.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.000+0000 7f6d377fe700 1 --2- 172.21.15.44:0/2929950726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c38d720 0x55f25c381f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.000+0000 7f6d377fe700 1 -- 172.21.15.44:0/2929950726 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f25c29ee80 con 0x55f25c38db90 2023-11-16T16:52:39.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.000+0000 7f6d367fc700 1 --2- 172.21.15.44:0/2929950726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c38d720 0x55f25c381f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:39.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.000+0000 7f6d377fe700 1 --2- 172.21.15.44:0/2929950726 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 0x55f25c393600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6d300c9fe0 tx=0x7f6d3009f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1141b5c79d03f585 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:39.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.000+0000 7f6d1dffb700 1 -- 172.21.15.44:0/2929950726 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d300999f0 con 0x55f25c38db90 2023-11-16T16:52:39.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.000+0000 7f6d1dffb700 1 -- 172.21.15.44:0/2929950726 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d3009da20 con 0x55f25c38db90 2023-11-16T16:52:39.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.000+0000 7f6d1dffb700 1 -- 172.21.15.44:0/2929950726 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30099eb0 con 0x55f25c38db90 2023-11-16T16:52:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.001+0000 7f6d51211b80 1 -- 172.21.15.44:0/2929950726 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 msgr2=0x55f25c393600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.001+0000 7f6d51211b80 1 --2- 172.21.15.44:0/2929950726 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 0x55f25c393600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6d300c9fe0 tx=0x7f6d3009f040 comp rx=0 tx=0).stop 2023-11-16T16:52:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.001+0000 7f6d51211b80 1 -- 172.21.15.44:0/2929950726 shutdown_connections 2023-11-16T16:52:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.001+0000 7f6d51211b80 1 --2- 172.21.15.44:0/2929950726 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 0x55f25c393600 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.001+0000 7f6d51211b80 1 --2- 172.21.15.44:0/2929950726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c1e36b0 0x55f25c398070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.001+0000 7f6d51211b80 1 --2- 172.21.15.44:0/2929950726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c38d720 0x55f25c381f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.001+0000 7f6d51211b80 1 -- 172.21.15.44:0/2929950726 >> 172.21.15.44:0/2929950726 conn(0x55f25c37cf50 msgr2=0x55f25c38ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.001+0000 7f6d51211b80 1 -- 172.21.15.44:0/2929950726 shutdown_connections 2023-11-16T16:52:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.001+0000 7f6d51211b80 1 -- 172.21.15.44:0/2929950726 wait complete. 2023-11-16T16:52:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.001+0000 7f6d51211b80 1 -- start start 2023-11-16T16:52:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.002+0000 7f6d51211b80 1 Processor -- start 2023-11-16T16:52:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.002+0000 7f6d51211b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c1e36b0 0x55f25c398240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.002+0000 7f6d51211b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c38d720 0x55f25c389aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.002+0000 7f6d51211b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 0x55f25c38a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.002+0000 7f6d51211b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25c38a920 con 0x55f25c1e36b0 2023-11-16T16:52:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.002+0000 7f6d51211b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25c3f3cd0 con 0x55f25c38d720 2023-11-16T16:52:39.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.002+0000 7f6d51211b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25c3f3e10 con 0x55f25c38db90 2023-11-16T16:52:39.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.002+0000 7f6d51211b80 10 client.0 _ll_register_callbacks cb 0x7f6d300093e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:39.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.002+0000 7f6d377fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 0x55f25c38a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:39.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.002+0000 7f6d367fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c38d720 0x55f25c389aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:39.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.002+0000 7f6d377fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 0x55f25c38a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:35980/0 (socket says 172.21.15.44:35980) 2023-11-16T16:52:39.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.002+0000 7f6d377fe700 1 -- 172.21.15.44:0/2245332157 learned_addr learned my addr 172.21.15.44:0/2245332157 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:39.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.003+0000 7f6d36ffd700 1 --2- 172.21.15.44:0/2245332157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c1e36b0 0x55f25c398240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:39.007 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:39.007 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.003+0000 7f6d377fe700 1 -- 172.21.15.44:0/2245332157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c38d720 msgr2=0x55f25c389aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.003+0000 7f6d377fe700 1 --2- 172.21.15.44:0/2245332157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c38d720 0x55f25c389aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.003+0000 7f6d377fe700 1 -- 172.21.15.44:0/2245332157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c1e36b0 msgr2=0x55f25c398240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.003+0000 7f6d377fe700 1 --2- 172.21.15.44:0/2245332157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c1e36b0 0x55f25c398240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.003+0000 7f6d377fe700 1 -- 172.21.15.44:0/2245332157 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d3810d9b0 con 0x55f25c38db90 2023-11-16T16:52:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.003+0000 7f6d377fe700 1 --2- 172.21.15.44:0/2245332157 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 0x55f25c38a1e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6d30099980 tx=0x7f6d30004d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d699a090a595c7e2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.003+0000 7f6d1aff5700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:52:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.003+0000 7f6d1aff5700 1 -- 172.21.15.44:0/2245332157 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3000cd50 con 0x55f25c38db90 2023-11-16T16:52:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.003+0000 7f6d51211b80 1 -- 172.21.15.44:0/2245332157 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f25c37a710 con 0x55f25c38db90 2023-11-16T16:52:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.003+0000 7f6d1aff5700 1 -- 172.21.15.44:0/2245332157 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f6d30099c50 con 0x55f25c38db90 2023-11-16T16:52:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.003+0000 7f6d1aff5700 1 -- 172.21.15.44:0/2245332157 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d30099e30 con 0x55f25c38db90 2023-11-16T16:52:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.003+0000 7f6d51211b80 10 client.26176 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.003+0000 7f6d51211b80 1 -- 172.21.15.44:0/2245332157 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x7f6d2c009520 con 0x55f25c38db90 2023-11-16T16:52:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.004+0000 7f6d1aff5700 1 -- 172.21.15.44:0/2245332157 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f25c37a710 con 0x55f25c38db90 2023-11-16T16:52:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.004+0000 7f6d1aff5700 1 -- 172.21.15.44:0/2245332157 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6d3001b9d0 con 0x55f25c38db90 2023-11-16T16:52:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.004+0000 7f6d51211b80 10 client.26176 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.004+0000 7f6d51211b80 10 client.26176 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.004+0000 7f6d51211b80 20 client.26176 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.004+0000 7f6d51211b80 1 --2- 172.21.15.44:0/2245332157 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f25c3f5170 0x55f25c3f7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.004+0000 7f6d51211b80 4 client.26176 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:52:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.004+0000 7f6d51211b80 1 -- 172.21.15.44:0/2245332157 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f25c127f10 con 0x55f25c3f5170 2023-11-16T16:52:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.004+0000 7f6d36ffd700 1 --2- 172.21.15.44:0/2245332157 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f25c3f5170 0x55f25c3f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:39.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.005+0000 7f6d36ffd700 1 --2- 172.21.15.44:0/2245332157 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f25c3f5170 0x55f25c3f7550 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac71ae958535b5ca server_cookie=d5d3997c96b87f4f in_seq=0 out_seq=0 2023-11-16T16:52:39.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.005+0000 7f6d1aff5700 10 client.26176 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:39.028 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:39.028 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:39.028 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 87 -lt 100 ']' 2023-11-16T16:52:39.029 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 87 + 1 2023-11-16T16:52:39.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.030+0000 7f6d1aff5700 1 -- 172.21.15.44:0/2245332157 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f25c127f10 con 0x55f25c3f5170 2023-11-16T16:52:39.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.031+0000 7f6d1aff5700 10 client.26176 handle_command_reply: tid=0 2023-11-16T16:52:39.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.031+0000 7f6d51211b80 1 client.26176 shutdown 2023-11-16T16:52:39.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.031+0000 7f6d51211b80 1 -- 172.21.15.44:0/2245332157 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 msgr2=0x55f25c38a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:39.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.031+0000 7f6d51211b80 1 --2- 172.21.15.44:0/2245332157 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 0x55f25c38a1e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6d30099980 tx=0x7f6d30004d30 comp rx=0 tx=0).stop 2023-11-16T16:52:39.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.031+0000 7f6d51211b80 1 -- 172.21.15.44:0/2245332157 shutdown_connections 2023-11-16T16:52:39.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.031+0000 7f6d51211b80 1 --2- 172.21.15.44:0/2245332157 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f25c3f5170 0x55f25c3f7550 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.031+0000 7f6d51211b80 1 --2- 172.21.15.44:0/2245332157 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f25c38db90 0x55f25c38a1e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.031+0000 7f6d51211b80 1 --2- 172.21.15.44:0/2245332157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f25c1e36b0 0x55f25c398240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.031+0000 7f6d1aff5700 0 client.26176 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:39.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.031+0000 7f6d51211b80 1 --2- 172.21.15.44:0/2245332157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f25c38d720 0x55f25c389aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.031+0000 7f6d51211b80 1 -- 172.21.15.44:0/2245332157 >> 172.21.15.44:0/2245332157 conn(0x55f25c37cf50 msgr2=0x55f25c390580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:39.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.031+0000 7f6d51211b80 1 -- 172.21.15.44:0/2245332157 shutdown_connections 2023-11-16T16:52:39.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.031+0000 7f6d51211b80 1 -- 172.21.15.44:0/2245332157 wait complete. 2023-11-16T16:52:39.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.031+0000 7f6d51211b80 20 client.26176 trim_cache size 0 max 16384 2023-11-16T16:52:39.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:52:39.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:52:39.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:52:39.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:52:39.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.032+0000 7f6d3d7ff700 1 -- 172.21.15.44:0/3316062042 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6d200a1340 msgr2=0x7f6d200a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:39.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.032+0000 7f6d3d7ff700 1 --2- 172.21.15.44:0/3316062042 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6d200a1340 0x7f6d200a37f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6d28005fd0 tx=0x7f6d28009310 comp rx=0 tx=0).stop 2023-11-16T16:52:39.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.032+0000 7f6d3d7ff700 1 -- 172.21.15.44:0/3316062042 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6d38105560 msgr2=0x7f6d381a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:39.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.032+0000 7f6d3d7ff700 1 --2- 172.21.15.44:0/3316062042 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6d38105560 0x7f6d381a4660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6d3000ef30 tx=0x7f6d3000c450 comp rx=0 tx=0).stop 2023-11-16T16:52:39.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.032+0000 7f6d3d7ff700 1 -- 172.21.15.44:0/3316062042 shutdown_connections 2023-11-16T16:52:39.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.032+0000 7f6d3d7ff700 1 --2- 172.21.15.44:0/3316062042 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6d200a1340 0x7f6d200a37f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.032+0000 7f6d3d7ff700 1 --2- 172.21.15.44:0/3316062042 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6d38105560 0x7f6d381a4660 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.032+0000 7f6d3d7ff700 1 --2- 172.21.15.44:0/3316062042 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d38100c50 0x7f6d38072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.032+0000 7f6d3d7ff700 1 --2- 172.21.15.44:0/3316062042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d381015f0 0x7f6d38073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.032+0000 7f6d3d7ff700 1 -- 172.21.15.44:0/3316062042 >> 172.21.15.44:0/3316062042 conn(0x7f6d38077550 msgr2=0x7f6d380feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:39.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.032+0000 7f6d3d7ff700 1 -- 172.21.15.44:0/3316062042 shutdown_connections 2023-11-16T16:52:39.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:39.032+0000 7f6d3d7ff700 1 -- 172.21.15.44:0/3316062042 wait complete. 2023-11-16T16:52:39.057 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=88 2023-11-16T16:52:39.058 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:39.122 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:39.122 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:39.125 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:39.125 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:39.126 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:39.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:39.147+0000 7fc0253ff700 1 -- 172.21.15.44:0/1249659974 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc020067a60 con 0x7fc020072e90 2023-11-16T16:52:39.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:39.149+0000 7fc007fff700 1 -- 172.21.15.44:0/1249659974 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7fc01009ac40 con 0x7fc020072e90 2023-11-16T16:52:39.150 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:52:39.150 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:52:39.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:39.151+0000 7fc0253ff700 1 -- 172.21.15.44:0/1249659974 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc0080a12c0 msgr2=0x7fc0080a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:39.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:39.151+0000 7fc0253ff700 1 --2- 172.21.15.44:0/1249659974 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc0080a12c0 0x7fc0080a3770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc020113ac0 tx=0x7fc014009260 comp rx=0 tx=0).stop 2023-11-16T16:52:39.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:39.151+0000 7fc0253ff700 1 -- 172.21.15.44:0/1249659974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc020072e90 msgr2=0x7fc0201a27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:39.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:39.151+0000 7fc0253ff700 1 --2- 172.21.15.44:0/1249659974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc020072e90 0x7fc0201a27c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc010005910 tx=0x7fc0100057d0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:39.151+0000 7fc0253ff700 1 -- 172.21.15.44:0/1249659974 shutdown_connections 2023-11-16T16:52:39.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:39.151+0000 7fc0253ff700 1 --2- 172.21.15.44:0/1249659974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0201112f0 0x7fc02019c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:39.151+0000 7fc0253ff700 1 --2- 172.21.15.44:0/1249659974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc020072e90 0x7fc0201a27c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:39.151+0000 7fc0253ff700 1 --2- 172.21.15.44:0/1249659974 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc0080a12c0 0x7fc0080a3770 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:39.151+0000 7fc0253ff700 1 --2- 172.21.15.44:0/1249659974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0200737a0 0x7fc0201a2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:39.151+0000 7fc0253ff700 1 -- 172.21.15.44:0/1249659974 >> 172.21.15.44:0/1249659974 conn(0x7fc020077020 msgr2=0x7fc0200ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:39.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:39.151+0000 7fc0253ff700 1 -- 172.21.15.44:0/1249659974 shutdown_connections 2023-11-16T16:52:39.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:39.151+0000 7fc0253ff700 1 -- 172.21.15.44:0/1249659974 wait complete. 2023-11-16T16:52:39.154 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:52:39.178 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T16:52:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:39 smithi044 ceph-mon[128750]: pgmap v84: 129 pgs: 129 active+clean; 2.8 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 33 KiB/s wr, 12 op/s 2023-11-16T16:52:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:39 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1249659974' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:52:39.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:39 smithi176 ceph-mon[131630]: pgmap v84: 129 pgs: 129 active+clean; 2.8 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 33 KiB/s wr, 12 op/s 2023-11-16T16:52:39.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:39 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1249659974' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:52:39.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:39 smithi077 ceph-mon[131204]: pgmap v84: 129 pgs: 129 active+clean; 2.8 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 33 KiB/s wr, 12 op/s 2023-11-16T16:52:39.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:39 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1249659974' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:52:39.750 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:39.751 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:39.904 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:39.904 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:39.935 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:39.935 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:39.935 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 88 -lt 100 ']' 2023-11-16T16:52:39.936 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 88 + 1 2023-11-16T16:52:39.974 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=89 2023-11-16T16:52:39.974 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:40.018 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:40.018 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:40.021 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:40.021 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:40.022 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:40.721 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:40.722 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:40.887 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:40.888 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:40.911 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:40.911 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:40.911 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 89 -lt 100 ']' 2023-11-16T16:52:40.912 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 89 + 1 2023-11-16T16:52:40.940 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=90 2023-11-16T16:52:40.941 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:41.017 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:41.017 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:41.020 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:41.020 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:41.021 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:41.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:41 smithi176 ceph-mon[131630]: pgmap v85: 129 pgs: 129 active+clean; 2.8 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 4 op/s 2023-11-16T16:52:41.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:41 smithi077 ceph-mon[131204]: pgmap v85: 129 pgs: 129 active+clean; 2.8 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 4 op/s 2023-11-16T16:52:41.733 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:41 smithi044 ceph-mon[128750]: pgmap v85: 129 pgs: 129 active+clean; 2.8 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 4 op/s 2023-11-16T16:52:41.950 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:41.951 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:42.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.205+0000 7fc10e7ff700 1 -- 172.21.15.44:0/701342487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc108101ca0 msgr2=0x7fc10810a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:42.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.205+0000 7fc10e7ff700 1 --2- 172.21.15.44:0/701342487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc108101ca0 0x7fc10810a370 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc009990 tx=0x7fc0fc009ca0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.205+0000 7fc10e7ff700 1 -- 172.21.15.44:0/701342487 shutdown_connections 2023-11-16T16:52:42.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.205+0000 7fc10e7ff700 1 --2- 172.21.15.44:0/701342487 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc10810a8b0 0x7fc10810ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.205+0000 7fc10e7ff700 1 --2- 172.21.15.44:0/701342487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc108101390 0x7fc108101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.205+0000 7fc10e7ff700 1 --2- 172.21.15.44:0/701342487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc108101ca0 0x7fc10810a370 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.205+0000 7fc10e7ff700 1 -- 172.21.15.44:0/701342487 >> 172.21.15.44:0/701342487 conn(0x7fc1080770b0 msgr2=0x7fc1080774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:42.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.205+0000 7fc10e7ff700 1 -- 172.21.15.44:0/701342487 shutdown_connections 2023-11-16T16:52:42.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.205+0000 7fc10e7ff700 1 -- 172.21.15.44:0/701342487 wait complete. 2023-11-16T16:52:42.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.206+0000 7fc10e7ff700 1 Processor -- start 2023-11-16T16:52:42.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.206+0000 7fc10e7ff700 1 -- start start 2023-11-16T16:52:42.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.207+0000 7fc10e7ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc108101390 0x7fc1080ff1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.207+0000 7fc10e7ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc108101ca0 0x7fc1080ff700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.207+0000 7fc10e7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10810a8b0 0x7fc1081a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.207+0000 7fc10e7ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1080fff30 con 0x7fc108101390 2023-11-16T16:52:42.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.207+0000 7fc10e7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1081a4b40 con 0x7fc10810a8b0 2023-11-16T16:52:42.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.207+0000 7fc10e7ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1081a4c80 con 0x7fc108101ca0 2023-11-16T16:52:42.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.207+0000 7fc107fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc108101390 0x7fc1080ff1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.207+0000 7fc107fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc108101390 0x7fc1080ff1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46874/0 (socket says 172.21.15.44:46874) 2023-11-16T16:52:42.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.207+0000 7fc107fff700 1 -- 172.21.15.44:0/1174764521 learned_addr learned my addr 172.21.15.44:0/1174764521 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:42.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.207+0000 7fc1077fe700 1 --2- 172.21.15.44:0/1174764521 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc108101ca0 0x7fc1080ff700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.207+0000 7fc10cd9c700 1 --2- 172.21.15.44:0/1174764521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10810a8b0 0x7fc1081a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.208+0000 7fc1077fe700 1 -- 172.21.15.44:0/1174764521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10810a8b0 msgr2=0x7fc1081a4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:42.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.208+0000 7fc1077fe700 1 --2- 172.21.15.44:0/1174764521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10810a8b0 0x7fc1081a4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.208+0000 7fc1077fe700 1 -- 172.21.15.44:0/1174764521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc108101390 msgr2=0x7fc1080ff1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:42.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.208+0000 7fc1077fe700 1 --2- 172.21.15.44:0/1174764521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc108101390 0x7fc1080ff1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.208+0000 7fc1077fe700 1 -- 172.21.15.44:0/1174764521 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0fc0095f0 con 0x7fc108101ca0 2023-11-16T16:52:42.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.208+0000 7fc107fff700 1 --2- 172.21.15.44:0/1174764521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc108101390 0x7fc1080ff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:42.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.208+0000 7fc1077fe700 1 --2- 172.21.15.44:0/1174764521 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc108101ca0 0x7fc1080ff700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc000c00 tx=0x7fc0fc005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:42.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.209+0000 7fc1057fa700 1 -- 172.21.15.44:0/1174764521 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc01d070 con 0x7fc108101ca0 2023-11-16T16:52:42.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.209+0000 7fc10e7ff700 1 -- 172.21.15.44:0/1174764521 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc108110390 con 0x7fc108101ca0 2023-11-16T16:52:42.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.209+0000 7fc1057fa700 1 -- 172.21.15.44:0/1174764521 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0fc021410 con 0x7fc108101ca0 2023-11-16T16:52:42.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.209+0000 7fc1057fa700 1 -- 172.21.15.44:0/1174764521 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc017650 con 0x7fc108101ca0 2023-11-16T16:52:42.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.209+0000 7fc10e7ff700 1 -- 172.21.15.44:0/1174764521 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc108110200 con 0x7fc108101ca0 2023-11-16T16:52:42.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.212+0000 7fc1057fa700 1 -- 172.21.15.44:0/1174764521 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fc0fc0217f0 con 0x7fc108101ca0 2023-11-16T16:52:42.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.214+0000 7fc1057fa700 1 --2- 172.21.15.44:0/1174764521 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc0f00a1340 0x7fc0f00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.214+0000 7fc1057fa700 1 -- 172.21.15.44:0/1174764521 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fc0fc0ce300 con 0x7fc108101ca0 2023-11-16T16:52:42.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.214+0000 7fc107fff700 1 --2- 172.21.15.44:0/1174764521 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc0f00a1340 0x7fc0f00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.215+0000 7fc107fff700 1 --2- 172.21.15.44:0/1174764521 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc0f00a1340 0x7fc0f00a37f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8011550 tx=0x7fc0f8009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:42.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.315+0000 7fc122062b80 1 Processor -- start 2023-11-16T16:52:42.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.315+0000 7fc122062b80 1 -- start start 2023-11-16T16:52:42.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.315+0000 7fc122062b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe763b60 0x556afe763f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.315+0000 7fc122062b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe764470 0x556afe768370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.315+0000 7fc122062b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe7688b0 0x556afe768d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.315+0000 7fc122062b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afe669060 con 0x556afe764470 2023-11-16T16:52:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.315+0000 7fc122062b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afe6bb8a0 con 0x556afe7688b0 2023-11-16T16:52:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.315+0000 7fc122062b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afe6b4d30 con 0x556afe763b60 2023-11-16T16:52:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.316+0000 7fc1077fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe764470 0x556afe768370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.316+0000 7fc1077fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe764470 0x556afe768370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46888/0 (socket says 172.21.15.44:46888) 2023-11-16T16:52:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.316+0000 7fc1077fe700 1 -- 172.21.15.44:0/289476937 learned_addr learned my addr 172.21.15.44:0/289476937 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.316+0000 7fc107fff700 1 --2- 172.21.15.44:0/289476937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe763b60 0x556afe763f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.316+0000 7fc10cd9c700 1 --2- 172.21.15.44:0/289476937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe7688b0 0x556afe768d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.316+0000 7fc1077fe700 1 -- 172.21.15.44:0/289476937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe763b60 msgr2=0x556afe763f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.316+0000 7fc1077fe700 1 --2- 172.21.15.44:0/289476937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe763b60 0x556afe763f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.316+0000 7fc1077fe700 1 -- 172.21.15.44:0/289476937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe7688b0 msgr2=0x556afe768d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.316+0000 7fc1077fe700 1 --2- 172.21.15.44:0/289476937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe7688b0 0x556afe768d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.316+0000 7fc1077fe700 1 -- 172.21.15.44:0/289476937 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556afe536630 con 0x556afe764470 2023-11-16T16:52:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.316+0000 7fc1077fe700 1 --2- 172.21.15.44:0/289476937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe764470 0x556afe768370 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc097af0 tx=0x7fc0fc09aed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d1d64ed55271172 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.317+0000 7fc0ee7fc700 1 -- 172.21.15.44:0/289476937 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc09b080 con 0x556afe764470 2023-11-16T16:52:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.317+0000 7fc0ee7fc700 1 -- 172.21.15.44:0/289476937 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0fc021e90 con 0x556afe764470 2023-11-16T16:52:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.317+0000 7fc0ee7fc700 1 -- 172.21.15.44:0/289476937 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc09a790 con 0x556afe764470 2023-11-16T16:52:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.318+0000 7fc122062b80 1 -- 172.21.15.44:0/289476937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe764470 msgr2=0x556afe768370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.318+0000 7fc122062b80 1 --2- 172.21.15.44:0/289476937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe764470 0x556afe768370 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc097af0 tx=0x7fc0fc09aed0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.318+0000 7fc122062b80 1 -- 172.21.15.44:0/289476937 shutdown_connections 2023-11-16T16:52:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.318+0000 7fc122062b80 1 --2- 172.21.15.44:0/289476937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe7688b0 0x556afe768d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.318+0000 7fc122062b80 1 --2- 172.21.15.44:0/289476937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe763b60 0x556afe763f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.318+0000 7fc122062b80 1 --2- 172.21.15.44:0/289476937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe764470 0x556afe768370 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.318+0000 7fc122062b80 1 -- 172.21.15.44:0/289476937 >> 172.21.15.44:0/289476937 conn(0x556afe759360 msgr2=0x556afe75eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.318+0000 7fc122062b80 1 -- 172.21.15.44:0/289476937 shutdown_connections 2023-11-16T16:52:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.318+0000 7fc122062b80 1 -- 172.21.15.44:0/289476937 wait complete. 2023-11-16T16:52:42.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.318+0000 7fc122062b80 1 -- start start 2023-11-16T16:52:42.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.319+0000 7fc122062b80 1 Processor -- start 2023-11-16T16:52:42.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.319+0000 7fc122062b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe763b60 0x556afe76ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.319+0000 7fc122062b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe764470 0x556afe76d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.319+0000 7fc122062b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe7688b0 0x556afe76d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.319+0000 7fc122062b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afe6568e0 con 0x556afe7688b0 2023-11-16T16:52:42.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.319+0000 7fc122062b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afe656580 con 0x556afe764470 2023-11-16T16:52:42.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.319+0000 7fc122062b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afe538920 con 0x556afe763b60 2023-11-16T16:52:42.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.319+0000 7fc122062b80 10 client.0 _ll_register_callbacks cb 0x556afe674780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:42.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.320+0000 7fc10cd9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe7688b0 0x556afe76d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.320+0000 7fc107fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe763b60 0x556afe76ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.320+0000 7fc10cd9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe7688b0 0x556afe76d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46902/0 (socket says 172.21.15.44:46902) 2023-11-16T16:52:42.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.320+0000 7fc10cd9c700 1 -- 172.21.15.44:0/2014893278 learned_addr learned my addr 172.21.15.44:0/2014893278 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:42.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.320+0000 7fc1077fe700 1 --2- 172.21.15.44:0/2014893278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe764470 0x556afe76d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.320+0000 7fc107fff700 1 -- 172.21.15.44:0/2014893278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe764470 msgr2=0x556afe76d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:42.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.320+0000 7fc107fff700 1 --2- 172.21.15.44:0/2014893278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe764470 0x556afe76d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.320+0000 7fc107fff700 1 -- 172.21.15.44:0/2014893278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe7688b0 msgr2=0x556afe76d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:42.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.320+0000 7fc107fff700 1 --2- 172.21.15.44:0/2014893278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe7688b0 0x556afe76d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.320+0000 7fc107fff700 1 -- 172.21.15.44:0/2014893278 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc108110200 con 0x556afe763b60 2023-11-16T16:52:42.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.321+0000 7fc107fff700 1 --2- 172.21.15.44:0/2014893278 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe763b60 0x556afe76ffa0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009cf0 tx=0x7fc0f80047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed011ade9315650f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:42.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.321+0000 7fc0e77fe700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:52:42.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.321+0000 7fc0e77fe700 1 -- 172.21.15.44:0/2014893278 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8022070 con 0x556afe763b60 2023-11-16T16:52:42.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.321+0000 7fc0e77fe700 1 -- 172.21.15.44:0/2014893278 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fc0f801dad0 con 0x556afe763b60 2023-11-16T16:52:42.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.321+0000 7fc122062b80 1 -- 172.21.15.44:0/2014893278 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556afe756b20 con 0x556afe763b60 2023-11-16T16:52:42.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.321+0000 7fc0e77fe700 1 -- 172.21.15.44:0/2014893278 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0f801dc90 con 0x556afe763b60 2023-11-16T16:52:42.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.321+0000 7fc122062b80 10 client.26188 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:42.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.321+0000 7fc122062b80 1 -- 172.21.15.44:0/2014893278 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x556afe67b190 con 0x556afe763b60 2023-11-16T16:52:42.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.322+0000 7fc0e77fe700 1 -- 172.21.15.44:0/2014893278 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556afe756b20 con 0x556afe763b60 2023-11-16T16:52:42.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.322+0000 7fc0e77fe700 1 -- 172.21.15.44:0/2014893278 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc0f801c490 con 0x556afe763b60 2023-11-16T16:52:42.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.322+0000 7fc122062b80 10 client.26188 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:42.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.322+0000 7fc122062b80 10 client.26188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:42.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.322+0000 7fc122062b80 20 client.26188 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:42.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.322+0000 7fc122062b80 1 --2- 172.21.15.44:0/2014893278 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x556afe77d570 0x556afe77f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.322+0000 7fc122062b80 4 client.26188 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:42.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.322+0000 7fc122062b80 1 -- 172.21.15.44:0/2014893278 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556afe6770a0 con 0x556afe77d570 2023-11-16T16:52:42.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.323+0000 7fc1077fe700 1 --2- 172.21.15.44:0/2014893278 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x556afe77d570 0x556afe77f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.323+0000 7fc1077fe700 1 --2- 172.21.15.44:0/2014893278 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x556afe77d570 0x556afe77f980 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f7812ed0a6669ae server_cookie=18a4292b20858401 in_seq=0 out_seq=0 2023-11-16T16:52:42.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.323+0000 7fc0e77fe700 10 client.26188 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:42.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.324+0000 7fc0e77fe700 1 -- 172.21.15.44:0/2014893278 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x556afe6770a0 con 0x556afe77d570 2023-11-16T16:52:42.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.324+0000 7fc0e77fe700 10 client.26188 handle_command_reply: tid=0 2023-11-16T16:52:42.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.324+0000 7fc122062b80 1 client.26188 shutdown 2023-11-16T16:52:42.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.324+0000 7fc122062b80 1 -- 172.21.15.44:0/2014893278 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe763b60 msgr2=0x556afe76ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:42.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.324+0000 7fc122062b80 1 --2- 172.21.15.44:0/2014893278 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe763b60 0x556afe76ffa0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009cf0 tx=0x7fc0f80047c0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.324+0000 7fc122062b80 1 -- 172.21.15.44:0/2014893278 shutdown_connections 2023-11-16T16:52:42.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.324+0000 7fc122062b80 1 --2- 172.21.15.44:0/2014893278 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x556afe77d570 0x556afe77f980 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.324+0000 7fc122062b80 1 --2- 172.21.15.44:0/2014893278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe7688b0 0x556afe76d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.324+0000 7fc0e77fe700 0 client.26188 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:42.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.324+0000 7fc122062b80 1 --2- 172.21.15.44:0/2014893278 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe763b60 0x556afe76ffa0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.324+0000 7fc122062b80 1 --2- 172.21.15.44:0/2014893278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe764470 0x556afe76d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.324+0000 7fc122062b80 1 -- 172.21.15.44:0/2014893278 >> 172.21.15.44:0/2014893278 conn(0x556afe759360 msgr2=0x556afe769d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:42.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.324+0000 7fc122062b80 1 -- 172.21.15.44:0/2014893278 shutdown_connections 2023-11-16T16:52:42.333 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.325+0000 7fc122062b80 1 -- 172.21.15.44:0/2014893278 wait complete. 2023-11-16T16:52:42.333 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.325+0000 7fc122062b80 20 client.26188 trim_cache size 0 max 16384 2023-11-16T16:52:42.341 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.339+0000 7fc122062b80 1 Processor -- start 2023-11-16T16:52:42.341 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.339+0000 7fc122062b80 1 -- start start 2023-11-16T16:52:42.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.339+0000 7fc122062b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe763b60 0x556afe75ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.339+0000 7fc122062b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe764470 0x556afe76fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.339+0000 7fc122062b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe7688b0 0x556afe767730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.339+0000 7fc122062b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afe6770a0 con 0x556afe763b60 2023-11-16T16:52:42.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.339+0000 7fc122062b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afe77ac70 con 0x556afe764470 2023-11-16T16:52:42.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.339+0000 7fc122062b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afe779fc0 con 0x556afe7688b0 2023-11-16T16:52:42.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.340+0000 7fc107fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe763b60 0x556afe75ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.340+0000 7fc107fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe763b60 0x556afe75ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46912/0 (socket says 172.21.15.44:46912) 2023-11-16T16:52:42.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.340+0000 7fc107fff700 1 -- 172.21.15.44:0/2529127617 learned_addr learned my addr 172.21.15.44:0/2529127617 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:42.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.340+0000 7fc1077fe700 1 --2- 172.21.15.44:0/2529127617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe764470 0x556afe76fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.340+0000 7fc10cd9c700 1 --2- 172.21.15.44:0/2529127617 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe7688b0 0x556afe767730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.340+0000 7fc107fff700 1 -- 172.21.15.44:0/2529127617 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe7688b0 msgr2=0x556afe767730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:42.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.340+0000 7fc107fff700 1 --2- 172.21.15.44:0/2529127617 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe7688b0 0x556afe767730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.340+0000 7fc107fff700 1 -- 172.21.15.44:0/2529127617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe764470 msgr2=0x556afe76fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:42.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.340+0000 7fc107fff700 1 --2- 172.21.15.44:0/2529127617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe764470 0x556afe76fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.340+0000 7fc107fff700 1 -- 172.21.15.44:0/2529127617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556afe768f50 con 0x556afe763b60 2023-11-16T16:52:42.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.340+0000 7fc10cd9c700 1 --2- 172.21.15.44:0/2529127617 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe7688b0 0x556afe767730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:52:42.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.340+0000 7fc107fff700 1 --2- 172.21.15.44:0/2529127617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe763b60 0x556afe75ec00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8016040 tx=0x7fc0f801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd35f852cfb016dc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:42.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.341+0000 7fc0eeffd700 1 -- 172.21.15.44:0/2529127617 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8022050 con 0x556afe763b60 2023-11-16T16:52:42.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.341+0000 7fc0eeffd700 1 -- 172.21.15.44:0/2529127617 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0f801c400 con 0x556afe763b60 2023-11-16T16:52:42.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.341+0000 7fc0eeffd700 1 -- 172.21.15.44:0/2529127617 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8004e10 con 0x556afe763b60 2023-11-16T16:52:42.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.342+0000 7fc122062b80 1 -- 172.21.15.44:0/2529127617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe763b60 msgr2=0x556afe75ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:42.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.342+0000 7fc122062b80 1 --2- 172.21.15.44:0/2529127617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe763b60 0x556afe75ec00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8016040 tx=0x7fc0f801c9f0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.342+0000 7fc122062b80 1 -- 172.21.15.44:0/2529127617 shutdown_connections 2023-11-16T16:52:42.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.342+0000 7fc122062b80 1 --2- 172.21.15.44:0/2529127617 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe7688b0 0x556afe767730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.342+0000 7fc122062b80 1 --2- 172.21.15.44:0/2529127617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe763b60 0x556afe75ec00 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.342+0000 7fc122062b80 1 --2- 172.21.15.44:0/2529127617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe764470 0x556afe76fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.342+0000 7fc122062b80 1 -- 172.21.15.44:0/2529127617 >> 172.21.15.44:0/2529127617 conn(0x556afe759360 msgr2=0x556afe7798b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:42.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.342+0000 7fc122062b80 1 -- 172.21.15.44:0/2529127617 shutdown_connections 2023-11-16T16:52:42.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.342+0000 7fc122062b80 1 -- 172.21.15.44:0/2529127617 wait complete. 2023-11-16T16:52:42.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.342+0000 7fc122062b80 1 -- start start 2023-11-16T16:52:42.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.342+0000 7fc122062b80 1 Processor -- start 2023-11-16T16:52:42.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.343+0000 7fc122062b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe763b60 0x556afe7ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.343+0000 7fc122062b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe764470 0x556afe7cac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.343+0000 7fc122062b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe7688b0 0x556afe7c9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.343+0000 7fc122062b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afe7c99b0 con 0x556afe7688b0 2023-11-16T16:52:42.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.343+0000 7fc122062b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afe7c9b20 con 0x556afe764470 2023-11-16T16:52:42.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.343+0000 7fc122062b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afe7c9ce0 con 0x556afe763b60 2023-11-16T16:52:42.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.343+0000 7fc122062b80 10 client.0 _ll_register_callbacks cb 0x7fc0f800e9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:42.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.343+0000 7fc10cd9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe7688b0 0x556afe7c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.343+0000 7fc10cd9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe7688b0 0x556afe7c9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46914/0 (socket says 172.21.15.44:46914) 2023-11-16T16:52:42.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.343+0000 7fc10cd9c700 1 -- 172.21.15.44:0/1087439161 learned_addr learned my addr 172.21.15.44:0/1087439161 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:42.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.343+0000 7fc1077fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe764470 0x556afe7cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.343+0000 7fc107fff700 1 --2- 172.21.15.44:0/1087439161 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe763b60 0x556afe7ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.344+0000 7fc1077fe700 1 -- 172.21.15.44:0/1087439161 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe763b60 msgr2=0x556afe7ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:42.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.344+0000 7fc1077fe700 1 --2- 172.21.15.44:0/1087439161 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe763b60 0x556afe7ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.344+0000 7fc1077fe700 1 -- 172.21.15.44:0/1087439161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe7688b0 msgr2=0x556afe7c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:42.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.344+0000 7fc1077fe700 1 --2- 172.21.15.44:0/1087439161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe7688b0 0x556afe7c9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.344+0000 7fc1077fe700 1 -- 172.21.15.44:0/1087439161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556afe67b190 con 0x556afe764470 2023-11-16T16:52:42.349 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.344+0000 7fc10cd9c700 1 --2- 172.21.15.44:0/1087439161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe7688b0 0x556afe7c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:42.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.344+0000 7fc1077fe700 1 --2- 172.21.15.44:0/1087439161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe764470 0x556afe7cac20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc0a3600 tx=0x7fc0fc0057b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77177b1870c8e0fc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:42.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.344+0000 7fc0e77fe700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:52:42.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.344+0000 7fc0e77fe700 1 -- 172.21.15.44:0/1087439161 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc0a3440 con 0x556afe764470 2023-11-16T16:52:42.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.344+0000 7fc122062b80 1 -- 172.21.15.44:0/1087439161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc0fc021e90 con 0x556afe764470 2023-11-16T16:52:42.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.344+0000 7fc0e77fe700 1 -- 172.21.15.44:0/1087439161 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fc0fc09b080 con 0x556afe764470 2023-11-16T16:52:42.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.345+0000 7fc0e77fe700 1 -- 172.21.15.44:0/1087439161 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0fc09f0b0 con 0x556afe764470 2023-11-16T16:52:42.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.345+0000 7fc122062b80 10 client.26204 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:42.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.345+0000 7fc122062b80 1 -- 172.21.15.44:0/1087439161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x556afe4e2de0 con 0x556afe764470 2023-11-16T16:52:42.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.345+0000 7fc0e77fe700 1 -- 172.21.15.44:0/1087439161 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0fc021e90 con 0x556afe764470 2023-11-16T16:52:42.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.345+0000 7fc0e77fe700 1 -- 172.21.15.44:0/1087439161 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc0fc09b370 con 0x556afe764470 2023-11-16T16:52:42.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.345+0000 7fc122062b80 10 client.26204 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:42.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.345+0000 7fc122062b80 10 client.26204 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:42.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.345+0000 7fc122062b80 20 client.26204 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:42.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.345+0000 7fc122062b80 1 --2- 172.21.15.44:0/1087439161 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x556afe7d2240 0x556afe7d4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:42.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.345+0000 7fc122062b80 4 client.26204 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T16:52:42.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.345+0000 7fc122062b80 1 -- 172.21.15.44:0/1087439161 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556afe683660 con 0x556afe7d2240 2023-11-16T16:52:42.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.345+0000 7fc107fff700 1 --2- 172.21.15.44:0/1087439161 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x556afe7d2240 0x556afe7d4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:42.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.346+0000 7fc0e77fe700 10 client.26204 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:42.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.346+0000 7fc107fff700 1 --2- 172.21.15.44:0/1087439161 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x556afe7d2240 0x556afe7d4650 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f575d7ae418b102 server_cookie=6fe2f122f9005821 in_seq=0 out_seq=0 2023-11-16T16:52:42.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.346+0000 7fc0e77fe700 1 -- 172.21.15.44:0/1087439161 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556afe683660 con 0x556afe7d2240 2023-11-16T16:52:42.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.346+0000 7fc0e77fe700 10 client.26204 handle_command_reply: tid=0 2023-11-16T16:52:42.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.346+0000 7fc122062b80 1 client.26204 shutdown 2023-11-16T16:52:42.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.346+0000 7fc122062b80 1 -- 172.21.15.44:0/1087439161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe764470 msgr2=0x556afe7cac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:42.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.346+0000 7fc122062b80 1 --2- 172.21.15.44:0/1087439161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe764470 0x556afe7cac20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc0a3600 tx=0x7fc0fc0057b0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.346+0000 7fc122062b80 1 -- 172.21.15.44:0/1087439161 shutdown_connections 2023-11-16T16:52:42.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.347+0000 7fc122062b80 1 --2- 172.21.15.44:0/1087439161 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x556afe7d2240 0x556afe7d4650 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.347+0000 7fc122062b80 1 --2- 172.21.15.44:0/1087439161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556afe7688b0 0x556afe7c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.347+0000 7fc122062b80 1 --2- 172.21.15.44:0/1087439161 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556afe763b60 0x556afe7ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.347+0000 7fc122062b80 1 --2- 172.21.15.44:0/1087439161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556afe764470 0x556afe7cac20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.355 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.347+0000 7fc0e77fe700 0 client.26204 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:42.355 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.347+0000 7fc122062b80 1 -- 172.21.15.44:0/1087439161 >> 172.21.15.44:0/1087439161 conn(0x556afe759360 msgr2=0x556afe762960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:42.355 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.347+0000 7fc122062b80 1 -- 172.21.15.44:0/1087439161 shutdown_connections 2023-11-16T16:52:42.355 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.347+0000 7fc122062b80 1 -- 172.21.15.44:0/1087439161 wait complete. 2023-11-16T16:52:42.355 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.347+0000 7fc122062b80 20 client.26204 trim_cache size 0 max 16384 2023-11-16T16:52:42.355 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T16:52:42.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.347+0000 7fc10e7ff700 1 -- 172.21.15.44:0/1174764521 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc0f00a1340 msgr2=0x7fc0f00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:42.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.347+0000 7fc10e7ff700 1 --2- 172.21.15.44:0/1174764521 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc0f00a1340 0x7fc0f00a37f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8011550 tx=0x7fc0f8009260 comp rx=0 tx=0).stop 2023-11-16T16:52:42.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.347+0000 7fc10e7ff700 1 -- 172.21.15.44:0/1174764521 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc108101ca0 msgr2=0x7fc1080ff700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:42.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.347+0000 7fc10e7ff700 1 --2- 172.21.15.44:0/1174764521 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc108101ca0 0x7fc1080ff700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc000c00 tx=0x7fc0fc005740 comp rx=0 tx=0).stop 2023-11-16T16:52:42.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.348+0000 7fc10e7ff700 1 -- 172.21.15.44:0/1174764521 shutdown_connections 2023-11-16T16:52:42.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.348+0000 7fc10e7ff700 1 --2- 172.21.15.44:0/1174764521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10810a8b0 0x7fc1081a4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.348+0000 7fc10e7ff700 1 --2- 172.21.15.44:0/1174764521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc108101390 0x7fc1080ff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.348+0000 7fc10e7ff700 1 --2- 172.21.15.44:0/1174764521 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc0f00a1340 0x7fc0f00a37f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.348+0000 7fc10e7ff700 1 --2- 172.21.15.44:0/1174764521 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc108101ca0 0x7fc1080ff700 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:42.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.348+0000 7fc10e7ff700 1 -- 172.21.15.44:0/1174764521 >> 172.21.15.44:0/1174764521 conn(0x7fc1080770b0 msgr2=0x7fc1080fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:42.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.348+0000 7fc10e7ff700 1 -- 172.21.15.44:0/1174764521 shutdown_connections 2023-11-16T16:52:42.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:42.348+0000 7fc10e7ff700 1 -- 172.21.15.44:0/1174764521 wait complete. 2023-11-16T16:52:42.404 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:42.404 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:42.511 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:42.512 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:42.512 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 90 -lt 100 ']' 2023-11-16T16:52:42.512 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 90 + 1 2023-11-16T16:52:42.544 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=91 2023-11-16T16:52:42.544 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:42.575 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:42.575 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:42.578 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:42.578 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:42.579 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:43.066 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:43.067 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:43.203 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:43.204 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:43.218 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:43.219 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:43.219 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 91 -lt 100 ']' 2023-11-16T16:52:43.219 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 91 + 1 2023-11-16T16:52:43.252 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=92 2023-11-16T16:52:43.252 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:43.283 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:43.283 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:43.286 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:43.286 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:43.287 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:43.406 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:52:43.406 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:52:43.630 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:43 smithi044 ceph-mon[128750]: pgmap v86: 129 pgs: 129 active+clean; 3.0 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 36 KiB/s wr, 11 op/s 2023-11-16T16:52:43.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:43 smithi176 ceph-mon[131630]: pgmap v86: 129 pgs: 129 active+clean; 3.0 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 36 KiB/s wr, 11 op/s 2023-11-16T16:52:43.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:43 smithi077 ceph-mon[131204]: pgmap v86: 129 pgs: 129 active+clean; 3.0 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 36 KiB/s wr, 11 op/s 2023-11-16T16:52:43.742 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:43.742 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:43.787 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:43.788 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:43.795 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:43.796 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:43.796 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 92 -lt 100 ']' 2023-11-16T16:52:43.796 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 92 + 1 2023-11-16T16:52:43.832 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=93 2023-11-16T16:52:43.832 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:43.876 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:43.877 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:43.879 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:43.880 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:43.880 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:44.450 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:44.450 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:44.612 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:44.612 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:44.627 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:44.627 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:44.627 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 93 -lt 100 ']' 2023-11-16T16:52:44.628 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 93 + 1 2023-11-16T16:52:44.690 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=94 2023-11-16T16:52:44.690 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:44.790 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:44.790 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:44.792 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:44.793 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:44.793 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:45.533 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:45.533 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:45.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:45 smithi044 ceph-mon[128750]: pgmap v87: 129 pgs: 129 active+clean; 3.0 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 30 KiB/s wr, 10 op/s 2023-11-16T16:52:45.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:45 smithi176 ceph-mon[131630]: pgmap v87: 129 pgs: 129 active+clean; 3.0 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 30 KiB/s wr, 10 op/s 2023-11-16T16:52:45.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:45 smithi077 ceph-mon[131204]: pgmap v87: 129 pgs: 129 active+clean; 3.0 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 30 KiB/s wr, 10 op/s 2023-11-16T16:52:45.745 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:45.745 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:45.804 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:45.804 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:45.804 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 94 -lt 100 ']' 2023-11-16T16:52:45.805 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 94 + 1 2023-11-16T16:52:45.840 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=95 2023-11-16T16:52:45.840 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:45.883 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:45.883 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:45.886 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:45.886 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:45.887 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:46.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.188+0000 7fcc09bff700 1 -- 172.21.15.44:0/1744119048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc04100a50 msgr2=0x7fcc04100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:46.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.188+0000 7fcc09bff700 1 --2- 172.21.15.44:0/1744119048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc04100a50 0x7fcc04100ec0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0099c0 tx=0x7fcbec009cd0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.188+0000 7fcc09bff700 1 -- 172.21.15.44:0/1744119048 shutdown_connections 2023-11-16T16:52:46.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.188+0000 7fcc09bff700 1 --2- 172.21.15.44:0/1744119048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc04101400 0x7fcc0410d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.188+0000 7fcc09bff700 1 --2- 172.21.15.44:0/1744119048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc041063f0 0x7fcc041067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.188+0000 7fcc09bff700 1 --2- 172.21.15.44:0/1744119048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc04100a50 0x7fcc04100ec0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.188+0000 7fcc09bff700 1 -- 172.21.15.44:0/1744119048 >> 172.21.15.44:0/1744119048 conn(0x7fcc04077590 msgr2=0x7fcc04077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:46.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.189+0000 7fcc09bff700 1 -- 172.21.15.44:0/1744119048 shutdown_connections 2023-11-16T16:52:46.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.189+0000 7fcc09bff700 1 -- 172.21.15.44:0/1744119048 wait complete. 2023-11-16T16:52:46.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.190+0000 7fcc09bff700 1 Processor -- start 2023-11-16T16:52:46.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.190+0000 7fcc09bff700 1 -- start start 2023-11-16T16:52:46.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.191+0000 7fcc09bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc04100a50 0x7fcc04072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.191+0000 7fcc09bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc04101400 0x7fcc04073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.191+0000 7fcc09bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc041063f0 0x7fcc041a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.191+0000 7fcc09bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc04073b60 con 0x7fcc04100a50 2023-11-16T16:52:46.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.191+0000 7fcc09bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc041a4b30 con 0x7fcc041063f0 2023-11-16T16:52:46.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.191+0000 7fcc09bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc041a4c70 con 0x7fcc04101400 2023-11-16T16:52:46.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.191+0000 7fcc037fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc04100a50 0x7fcc04072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.191+0000 7fcbfbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc04101400 0x7fcc04073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.191+0000 7fcc03fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc041063f0 0x7fcc041a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.191+0000 7fcbfbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc04101400 0x7fcc04073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:53722/0 (socket says 172.21.15.44:53722) 2023-11-16T16:52:46.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.191+0000 7fcbfbfff700 1 -- 172.21.15.44:0/3124436894 learned_addr learned my addr 172.21.15.44:0/3124436894 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:46.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.192+0000 7fcbfbfff700 1 -- 172.21.15.44:0/3124436894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc041063f0 msgr2=0x7fcc041a45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:46.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.192+0000 7fcbfbfff700 1 --2- 172.21.15.44:0/3124436894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc041063f0 0x7fcc041a45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.192+0000 7fcbfbfff700 1 -- 172.21.15.44:0/3124436894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc04100a50 msgr2=0x7fcc04072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.192+0000 7fcbfbfff700 1 --2- 172.21.15.44:0/3124436894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc04100a50 0x7fcc04072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.192+0000 7fcbfbfff700 1 -- 172.21.15.44:0/3124436894 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbfc009c40 con 0x7fcc04101400 2023-11-16T16:52:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.192+0000 7fcbfbfff700 1 --2- 172.21.15.44:0/3124436894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc04101400 0x7fcc04073330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0098e0 tx=0x7fcbec00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.193+0000 7fcc017fa700 1 -- 172.21.15.44:0/3124436894 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec01d070 con 0x7fcc04101400 2023-11-16T16:52:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.193+0000 7fcc017fa700 1 -- 172.21.15.44:0/3124436894 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcbec00fb80 con 0x7fcc04101400 2023-11-16T16:52:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.193+0000 7fcc017fa700 1 -- 172.21.15.44:0/3124436894 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec022df0 con 0x7fcc04101400 2023-11-16T16:52:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.193+0000 7fcc09bff700 1 -- 172.21.15.44:0/3124436894 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbec0095f0 con 0x7fcc04101400 2023-11-16T16:52:46.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.193+0000 7fcc09bff700 1 -- 172.21.15.44:0/3124436894 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc04110410 con 0x7fcc04101400 2023-11-16T16:52:46.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.196+0000 7fcc017fa700 1 -- 172.21.15.44:0/3124436894 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fcbec00fcf0 con 0x7fcc04101400 2023-11-16T16:52:46.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.198+0000 7fcc017fa700 1 --2- 172.21.15.44:0/3124436894 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fcbf00a1340 0x7fcbf00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.198+0000 7fcc037fe700 1 --2- 172.21.15.44:0/3124436894 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fcbf00a1340 0x7fcbf00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.198+0000 7fcc017fa700 1 -- 172.21.15.44:0/3124436894 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fcbec0ce0d0 con 0x7fcc04101400 2023-11-16T16:52:46.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.199+0000 7fcc037fe700 1 --2- 172.21.15.44:0/3124436894 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fcbf00a1340 0x7fcbf00a37f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009be0 tx=0x7fcbf4009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:46.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.289+0000 7fcc1d59bb80 1 Processor -- start 2023-11-16T16:52:46.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.289+0000 7fcc1d59bb80 1 -- start start 2023-11-16T16:52:46.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.290+0000 7fcc1d59bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cd95fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.290+0000 7fcc1d59bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd964f0 0x55962cd9a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.292 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.290+0000 7fcc1d59bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd9a930 0x55962cd9ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.290+0000 7fcc1d59bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962cc9b0e0 con 0x55962cd95be0 2023-11-16T16:52:46.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.290+0000 7fcc1d59bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962cced920 con 0x55962cd9a930 2023-11-16T16:52:46.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.290+0000 7fcc1d59bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962cce6db0 con 0x55962cd964f0 2023-11-16T16:52:46.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.290+0000 7fcc037fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cd95fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.290+0000 7fcc037fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cd95fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56994/0 (socket says 172.21.15.44:56994) 2023-11-16T16:52:46.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.290+0000 7fcbfbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd964f0 0x55962cd9a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.290+0000 7fcc037fe700 1 -- 172.21.15.44:0/4264912634 learned_addr learned my addr 172.21.15.44:0/4264912634 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:46.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.290+0000 7fcc03fff700 1 --2- 172.21.15.44:0/4264912634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd9a930 0x55962cd9ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.291+0000 7fcc037fe700 1 -- 172.21.15.44:0/4264912634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd964f0 msgr2=0x55962cd9a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:46.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.291+0000 7fcc037fe700 1 --2- 172.21.15.44:0/4264912634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd964f0 0x55962cd9a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.291+0000 7fcc037fe700 1 -- 172.21.15.44:0/4264912634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd9a930 msgr2=0x55962cd9ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:46.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.291+0000 7fcc037fe700 1 --2- 172.21.15.44:0/4264912634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd9a930 0x55962cd9ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.291+0000 7fcc037fe700 1 -- 172.21.15.44:0/4264912634 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55962cb686b0 con 0x55962cd95be0 2023-11-16T16:52:46.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.291+0000 7fcbfbfff700 1 --2- 172.21.15.44:0/4264912634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd964f0 0x55962cd9a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:46.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.291+0000 7fcc037fe700 1 --2- 172.21.15.44:0/4264912634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cd95fb0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4011d70 tx=0x7fcbf4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d77911133f1728 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:46.297 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.291+0000 7fcbf9ffb700 1 -- 172.21.15.44:0/4264912634 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4011da0 con 0x55962cd95be0 2023-11-16T16:52:46.297 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.291+0000 7fcbf9ffb700 1 -- 172.21.15.44:0/4264912634 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcbf4004020 con 0x55962cd95be0 2023-11-16T16:52:46.297 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.291+0000 7fcbf9ffb700 1 -- 172.21.15.44:0/4264912634 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4003cb0 con 0x55962cd95be0 2023-11-16T16:52:46.297 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.292+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/4264912634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 msgr2=0x55962cd95fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:46.297 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.292+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/4264912634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cd95fb0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4011d70 tx=0x7fcbf4014000 comp rx=0 tx=0).stop 2023-11-16T16:52:46.297 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.292+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/4264912634 shutdown_connections 2023-11-16T16:52:46.297 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.292+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/4264912634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd9a930 0x55962cd9ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.297 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.292+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/4264912634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cd95fb0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.297 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.292+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/4264912634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd964f0 0x55962cd9a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.297 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.292+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/4264912634 >> 172.21.15.44:0/4264912634 conn(0x55962cd8b3e0 msgr2=0x55962cd90bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:46.297 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.292+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/4264912634 shutdown_connections 2023-11-16T16:52:46.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.292+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/4264912634 wait complete. 2023-11-16T16:52:46.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.293+0000 7fcc1d59bb80 1 -- start start 2023-11-16T16:52:46.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.293+0000 7fcc1d59bb80 1 Processor -- start 2023-11-16T16:52:46.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.294+0000 7fcc1d59bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cda1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.294+0000 7fcc1d59bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd964f0 0x55962cd9f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.294+0000 7fcc1d59bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd9a930 0x55962cd9f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.294+0000 7fcc1d59bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962cc88960 con 0x55962cd95be0 2023-11-16T16:52:46.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.294+0000 7fcc1d59bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962cc88600 con 0x55962cd964f0 2023-11-16T16:52:46.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.294+0000 7fcc1d59bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962cb6a9a0 con 0x55962cd9a930 2023-11-16T16:52:46.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.294+0000 7fcc1d59bb80 10 client.0 _ll_register_callbacks cb 0x55962cca6800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:46.298 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.294+0000 7fcc037fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cda1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.294+0000 7fcc037fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cda1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57000/0 (socket says 172.21.15.44:57000) 2023-11-16T16:52:46.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.294+0000 7fcc037fe700 1 -- 172.21.15.44:0/3464230406 learned_addr learned my addr 172.21.15.44:0/3464230406 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:46.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.294+0000 7fcbfbfff700 1 --2- 172.21.15.44:0/3464230406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd964f0 0x55962cd9f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.294+0000 7fcc03fff700 1 --2- 172.21.15.44:0/3464230406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd9a930 0x55962cd9f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.295+0000 7fcbfbfff700 1 -- 172.21.15.44:0/3464230406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd9a930 msgr2=0x55962cd9f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:46.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.295+0000 7fcbfbfff700 1 --2- 172.21.15.44:0/3464230406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd9a930 0x55962cd9f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.295+0000 7fcbfbfff700 1 -- 172.21.15.44:0/3464230406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 msgr2=0x55962cda1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:46.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.295+0000 7fcbfbfff700 1 --2- 172.21.15.44:0/3464230406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cda1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.295+0000 7fcbfbfff700 1 -- 172.21.15.44:0/3464230406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf4017070 con 0x55962cd964f0 2023-11-16T16:52:46.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.295+0000 7fcc037fe700 1 --2- 172.21.15.44:0/3464230406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cda1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:46.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.295+0000 7fcbfbfff700 1 --2- 172.21.15.44:0/3464230406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd964f0 0x55962cd9f2e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0cb440 tx=0x7fcbec09af70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71a1c541a0b5463c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:46.299 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.295+0000 7fcbe2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:52:46.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.295+0000 7fcbe2ffd700 1 -- 172.21.15.44:0/3464230406 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0b2070 con 0x55962cd964f0 2023-11-16T16:52:46.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.295+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/3464230406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55962cd88ba0 con 0x55962cd964f0 2023-11-16T16:52:46.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.295+0000 7fcbe2ffd700 1 -- 172.21.15.44:0/3464230406 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fcbec0a3330 con 0x55962cd964f0 2023-11-16T16:52:46.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.295+0000 7fcbe2ffd700 1 -- 172.21.15.44:0/3464230406 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbec09a670 con 0x55962cd964f0 2023-11-16T16:52:46.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.295+0000 7fcc1d59bb80 10 client.26216 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:46.300 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.295+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/3464230406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55962ccad210 con 0x55962cd964f0 2023-11-16T16:52:46.301 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.296+0000 7fcbe2ffd700 1 -- 172.21.15.44:0/3464230406 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcbec0c9080 con 0x55962cd964f0 2023-11-16T16:52:46.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.296+0000 7fcbe2ffd700 1 -- 172.21.15.44:0/3464230406 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fcbec0caf80 con 0x55962cd964f0 2023-11-16T16:52:46.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.296+0000 7fcc1d59bb80 10 client.26216 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:46.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.296+0000 7fcc1d59bb80 10 client.26216 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:46.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.296+0000 7fcc1d59bb80 20 client.26216 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:46.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.296+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/3464230406 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55962cdaf500 0x55962cdb1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.296+0000 7fcc1d59bb80 4 client.26216 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:46.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.296+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/3464230406 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55962cca9120 con 0x55962cdaf500 2023-11-16T16:52:46.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.296+0000 7fcc037fe700 1 --2- 172.21.15.44:0/3464230406 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55962cdaf500 0x55962cdb1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.297+0000 7fcc037fe700 1 --2- 172.21.15.44:0/3464230406 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55962cdaf500 0x55962cdb1910 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48a19894afcb6986 server_cookie=e569c8e87266ca2a in_seq=0 out_seq=0 2023-11-16T16:52:46.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.297+0000 7fcbe2ffd700 10 client.26216 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:46.302 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.297+0000 7fcbe2ffd700 1 -- 172.21.15.44:0/3464230406 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55962cca9120 con 0x55962cdaf500 2023-11-16T16:52:46.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.297+0000 7fcbe2ffd700 10 client.26216 handle_command_reply: tid=0 2023-11-16T16:52:46.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.298+0000 7fcc1d59bb80 1 client.26216 shutdown 2023-11-16T16:52:46.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.298+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/3464230406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd964f0 msgr2=0x55962cd9f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:46.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.298+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/3464230406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd964f0 0x55962cd9f2e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0cb440 tx=0x7fcbec09af70 comp rx=0 tx=0).stop 2023-11-16T16:52:46.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.298+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/3464230406 shutdown_connections 2023-11-16T16:52:46.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.298+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/3464230406 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55962cdaf500 0x55962cdb1910 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.298+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/3464230406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd9a930 0x55962cd9f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.298+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/3464230406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cda1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.298+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/3464230406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd964f0 0x55962cd9f2e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.298+0000 7fcbe2ffd700 0 client.26216 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:46.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.298+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/3464230406 >> 172.21.15.44:0/3464230406 conn(0x55962cd8b3e0 msgr2=0x55962cd9bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:46.303 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.298+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/3464230406 shutdown_connections 2023-11-16T16:52:46.304 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.298+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/3464230406 wait complete. 2023-11-16T16:52:46.304 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.298+0000 7fcc1d59bb80 20 client.26216 trim_cache size 0 max 16384 2023-11-16T16:52:46.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.316+0000 7fcc1d59bb80 1 Processor -- start 2023-11-16T16:52:46.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.316+0000 7fcc1d59bb80 1 -- start start 2023-11-16T16:52:46.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.317+0000 7fcc1d59bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cd98220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.317+0000 7fcc1d59bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd964f0 0x55962cda6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.317+0000 7fcc1d59bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd9a930 0x55962cd9cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.317+0000 7fcc1d59bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962cda0850 con 0x55962cd95be0 2023-11-16T16:52:46.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.317+0000 7fcc1d59bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962cd88790 con 0x55962cd964f0 2023-11-16T16:52:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.317+0000 7fcc1d59bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962cda2810 con 0x55962cd9a930 2023-11-16T16:52:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.317+0000 7fcc037fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cd98220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.317+0000 7fcc037fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cd98220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57016/0 (socket says 172.21.15.44:57016) 2023-11-16T16:52:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.317+0000 7fcc037fe700 1 -- 172.21.15.44:0/1753731362 learned_addr learned my addr 172.21.15.44:0/1753731362 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.317+0000 7fcbfbfff700 1 --2- 172.21.15.44:0/1753731362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd964f0 0x55962cda6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.317+0000 7fcc03fff700 1 --2- 172.21.15.44:0/1753731362 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd9a930 0x55962cd9cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.317+0000 7fcc037fe700 1 -- 172.21.15.44:0/1753731362 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd9a930 msgr2=0x55962cd9cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.317+0000 7fcc037fe700 1 --2- 172.21.15.44:0/1753731362 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd9a930 0x55962cd9cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.317+0000 7fcc037fe700 1 -- 172.21.15.44:0/1753731362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd964f0 msgr2=0x55962cda6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.317+0000 7fcc037fe700 1 --2- 172.21.15.44:0/1753731362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd964f0 0x55962cda6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.317+0000 7fcc037fe700 1 -- 172.21.15.44:0/1753731362 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55962ccacf60 con 0x55962cd95be0 2023-11-16T16:52:46.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.318+0000 7fcc037fe700 1 --2- 172.21.15.44:0/1753731362 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cd98220 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40091e0 tx=0x7fcbf40036e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda1b917127a5d98 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:46.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.318+0000 7fcbfa7fc700 1 -- 172.21.15.44:0/1753731362 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf402c040 con 0x55962cd95be0 2023-11-16T16:52:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.318+0000 7fcbfa7fc700 1 -- 172.21.15.44:0/1753731362 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcbf400eba0 con 0x55962cd95be0 2023-11-16T16:52:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.318+0000 7fcbfa7fc700 1 -- 172.21.15.44:0/1753731362 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4003bf0 con 0x55962cd95be0 2023-11-16T16:52:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.319+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/1753731362 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 msgr2=0x55962cd98220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.319+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/1753731362 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cd98220 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40091e0 tx=0x7fcbf40036e0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.319+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/1753731362 shutdown_connections 2023-11-16T16:52:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.319+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/1753731362 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd9a930 0x55962cd9cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.319+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/1753731362 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd95be0 0x55962cd98220 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.319+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/1753731362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd964f0 0x55962cda6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.319+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/1753731362 >> 172.21.15.44:0/1753731362 conn(0x55962cd8b3e0 msgr2=0x55962cd9c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.319+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/1753731362 shutdown_connections 2023-11-16T16:52:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.319+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/1753731362 wait complete. 2023-11-16T16:52:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.319+0000 7fcc1d59bb80 1 -- start start 2023-11-16T16:52:46.323 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.320+0000 7fcc1d59bb80 1 Processor -- start 2023-11-16T16:52:46.325 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.320+0000 7fcc1d59bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd95be0 0x55962cd9e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.325 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.320+0000 7fcc1d59bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd964f0 0x55962cd9e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.325 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.320+0000 7fcc1d59bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd9a930 0x55962ce033c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.325 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcc1d59bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962cd9ecb0 con 0x55962cd964f0 2023-11-16T16:52:46.325 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcc1d59bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962ce03900 con 0x55962cd9a930 2023-11-16T16:52:46.325 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcc1d59bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962ce03a40 con 0x55962cd95be0 2023-11-16T16:52:46.325 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcc1d59bb80 10 client.0 _ll_register_callbacks cb 0x7fcbec09aba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:46.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcbfbfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd964f0 0x55962cd9e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcbfbfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd964f0 0x55962cd9e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57030/0 (socket says 172.21.15.44:57030) 2023-11-16T16:52:46.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcbfbfff700 1 -- 172.21.15.44:0/631172329 learned_addr learned my addr 172.21.15.44:0/631172329 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:46.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcc037fe700 1 --2- 172.21.15.44:0/631172329 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd95be0 0x55962cd9e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcc03fff700 1 --2- 172.21.15.44:0/631172329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd9a930 0x55962ce033c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcbfbfff700 1 -- 172.21.15.44:0/631172329 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd95be0 msgr2=0x55962cd9e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:46.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcbfbfff700 1 --2- 172.21.15.44:0/631172329 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd95be0 0x55962cd9e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcbfbfff700 1 -- 172.21.15.44:0/631172329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd9a930 msgr2=0x55962ce033c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:46.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcbfbfff700 1 --2- 172.21.15.44:0/631172329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd9a930 0x55962ce033c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcbfbfff700 1 -- 172.21.15.44:0/631172329 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55962cb686b0 con 0x55962cd964f0 2023-11-16T16:52:46.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcc03fff700 1 --2- 172.21.15.44:0/631172329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd9a930 0x55962ce033c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:52:46.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcbfbfff700 1 --2- 172.21.15.44:0/631172329 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd964f0 0x55962cd9e5f0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fcbec09ac70 tx=0x7fcbec009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=154ab066dfdc0a44 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:46.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.321+0000 7fcbe2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:52:46.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.322+0000 7fcbe2ffd700 1 -- 172.21.15.44:0/631172329 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0aede0 con 0x55962cd964f0 2023-11-16T16:52:46.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.322+0000 7fcbe2ffd700 1 -- 172.21.15.44:0/631172329 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fcbec0cb170 con 0x55962cd964f0 2023-11-16T16:52:46.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.322+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/631172329 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55962ccb56e0 con 0x55962cd964f0 2023-11-16T16:52:46.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.322+0000 7fcbe2ffd700 1 -- 172.21.15.44:0/631172329 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbec0b2070 con 0x55962cd964f0 2023-11-16T16:52:46.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.322+0000 7fcc1d59bb80 10 client.16200 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:46.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.322+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/631172329 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55962cafdce0 con 0x55962cd964f0 2023-11-16T16:52:46.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.322+0000 7fcbe2ffd700 1 -- 172.21.15.44:0/631172329 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55962ccb56e0 con 0x55962cd964f0 2023-11-16T16:52:46.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.322+0000 7fcbe2ffd700 1 -- 172.21.15.44:0/631172329 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fcbec095c10 con 0x55962cd964f0 2023-11-16T16:52:46.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.322+0000 7fcc1d59bb80 10 client.16200 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:46.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.322+0000 7fcc1d59bb80 10 client.16200 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:46.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.322+0000 7fcc1d59bb80 20 client.16200 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:46.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.322+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/631172329 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55962ce04e40 0x55962ce07250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:46.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.322+0000 7fcc1d59bb80 4 client.16200 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:52:46.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.322+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/631172329 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55962cb35ff0 con 0x55962ce04e40 2023-11-16T16:52:46.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.323+0000 7fcc037fe700 1 --2- 172.21.15.44:0/631172329 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55962ce04e40 0x55962ce07250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:46.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.323+0000 7fcc037fe700 1 --2- 172.21.15.44:0/631172329 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55962ce04e40 0x55962ce07250 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5832463c387aa40e server_cookie=9925c37597f20919 in_seq=0 out_seq=0 2023-11-16T16:52:46.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.323+0000 7fcbe2ffd700 10 client.16200 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:46.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.324+0000 7fcbe2ffd700 1 -- 172.21.15.44:0/631172329 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55962cb35ff0 con 0x55962ce04e40 2023-11-16T16:52:46.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.324+0000 7fcbe2ffd700 10 client.16200 handle_command_reply: tid=0 2023-11-16T16:52:46.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.324+0000 7fcc1d59bb80 1 client.16200 shutdown 2023-11-16T16:52:46.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.325+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/631172329 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd964f0 msgr2=0x55962cd9e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:46.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.325+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/631172329 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd964f0 0x55962cd9e5f0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fcbec09ac70 tx=0x7fcbec009cd0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.325+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/631172329 shutdown_connections 2023-11-16T16:52:46.329 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.325+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/631172329 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55962ce04e40 0x55962ce07250 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.325+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/631172329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55962cd9a930 0x55962ce033c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.325+0000 7fcbe2ffd700 0 client.16200 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:46.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.325+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/631172329 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55962cd95be0 0x55962cd9e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.325+0000 7fcc1d59bb80 1 --2- 172.21.15.44:0/631172329 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55962cd964f0 0x55962cd9e5f0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.325+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/631172329 >> 172.21.15.44:0/631172329 conn(0x55962cd8b3e0 msgr2=0x55962cd9ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:46.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.325+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/631172329 shutdown_connections 2023-11-16T16:52:46.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.325+0000 7fcc1d59bb80 1 -- 172.21.15.44:0/631172329 wait complete. 2023-11-16T16:52:46.330 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.325+0000 7fcc1d59bb80 20 client.16200 trim_cache size 0 max 16384 2023-11-16T16:52:46.330 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:52:46.331 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:52:46.331 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "66ebce24-dcab-48f5-83e7-48471bd78e6c", 2023-11-16T16:52:46.331 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:52:46.331 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:52:46.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.326+0000 7fcc09bff700 1 -- 172.21.15.44:0/3124436894 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fcbf00a1340 msgr2=0x7fcbf00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.326+0000 7fcc09bff700 1 --2- 172.21.15.44:0/3124436894 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fcbf00a1340 0x7fcbf00a37f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009be0 tx=0x7fcbf4009260 comp rx=0 tx=0).stop 2023-11-16T16:52:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.326+0000 7fcc09bff700 1 -- 172.21.15.44:0/3124436894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc04101400 msgr2=0x7fcc04073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.326+0000 7fcc09bff700 1 --2- 172.21.15.44:0/3124436894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc04101400 0x7fcc04073330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0098e0 tx=0x7fcbec00f690 comp rx=0 tx=0).stop 2023-11-16T16:52:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.326+0000 7fcc09bff700 1 -- 172.21.15.44:0/3124436894 shutdown_connections 2023-11-16T16:52:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.326+0000 7fcc09bff700 1 --2- 172.21.15.44:0/3124436894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc041063f0 0x7fcc041a45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.326+0000 7fcc09bff700 1 --2- 172.21.15.44:0/3124436894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc04100a50 0x7fcc04072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.326+0000 7fcc09bff700 1 --2- 172.21.15.44:0/3124436894 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fcbf00a1340 0x7fcbf00a37f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.326+0000 7fcc09bff700 1 --2- 172.21.15.44:0/3124436894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc04101400 0x7fcc04073330 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.326+0000 7fcc09bff700 1 -- 172.21.15.44:0/3124436894 >> 172.21.15.44:0/3124436894 conn(0x7fcc04077590 msgr2=0x7fcc040ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.326+0000 7fcc09bff700 1 -- 172.21.15.44:0/3124436894 shutdown_connections 2023-11-16T16:52:46.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:46.326+0000 7fcc09bff700 1 -- 172.21.15.44:0/3124436894 wait complete. 2023-11-16T16:52:46.378 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:52:46.683 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:46.683 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:46.761 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:46.761 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:46.784 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:46.785 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:46.785 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 95 -lt 100 ']' 2023-11-16T16:52:46.785 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 95 + 1 2023-11-16T16:52:46.823 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=96 2023-11-16T16:52:46.823 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:46.831 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:46.831 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:46.833 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:46.834 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:46.834 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:47.524 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:47.525 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:47.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:47 smithi176 ceph-mon[131630]: pgmap v88: 129 pgs: 129 active+clean; 3.0 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 30 KiB/s wr, 10 op/s 2023-11-16T16:52:47.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:47 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:52:47.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:47 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:47.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:47 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:52:47.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:47 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:47.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:47 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:52:47.686 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:47.687 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:47.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:47 smithi077 ceph-mon[131204]: pgmap v88: 129 pgs: 129 active+clean; 3.0 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 30 KiB/s wr, 10 op/s 2023-11-16T16:52:47.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:47 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:52:47.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:47 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:47.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:47 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:52:47.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:47 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:47.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:47 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:52:47.727 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:47.727 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:47.727 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 96 -lt 100 ']' 2023-11-16T16:52:47.727 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 96 + 1 2023-11-16T16:52:47.757 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=97 2023-11-16T16:52:47.757 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:47.839 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:47.839 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:47.842 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:47.842 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:47.843 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:47 smithi044 ceph-mon[128750]: pgmap v88: 129 pgs: 129 active+clean; 3.0 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 30 KiB/s wr, 10 op/s 2023-11-16T16:52:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:47 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:52:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:47 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:47 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:52:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:47 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:47 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:52:48.486 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:48.486 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:48.670 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:48.670 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:48.697 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:48.697 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:48.697 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 97 -lt 100 ']' 2023-11-16T16:52:48.698 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 97 + 1 2023-11-16T16:52:48.734 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=98 2023-11-16T16:52:48.734 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:48.807 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:48.808 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:48.810 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:48.811 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:48.811 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:49.424 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:49.425 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:49.569 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:49.570 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:49.602 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:49.602 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:49.602 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 98 -lt 100 ']' 2023-11-16T16:52:49.603 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 98 + 1 2023-11-16T16:52:49.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:49 smithi044 ceph-mon[128750]: pgmap v89: 129 pgs: 129 active+clean; 3.1 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 40 KiB/s wr, 13 op/s 2023-11-16T16:52:49.640 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=99 2023-11-16T16:52:49.640 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:49.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:49 smithi176 ceph-mon[131630]: pgmap v89: 129 pgs: 129 active+clean; 3.1 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 40 KiB/s wr, 13 op/s 2023-11-16T16:52:49.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:49 smithi077 ceph-mon[131204]: pgmap v89: 129 pgs: 129 active+clean; 3.1 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 40 KiB/s wr, 13 op/s 2023-11-16T16:52:49.749 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:49.750 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:49.753 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:49.753 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:49.753 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:50.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.121+0000 7f4cd81ff700 1 -- 172.21.15.44:0/2619058950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4cd0105620 msgr2=0x7f4cd010d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:50.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.121+0000 7f4cd81ff700 1 --2- 172.21.15.44:0/2619058950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4cd0105620 0x7f4cd010d1a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800b0a0 tx=0x7f4cc800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.121+0000 7f4cd81ff700 1 -- 172.21.15.44:0/2619058950 shutdown_connections 2023-11-16T16:52:50.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.121+0000 7f4cd81ff700 1 --2- 172.21.15.44:0/2619058950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4cd0105620 0x7f4cd010d1a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.121+0000 7f4cd81ff700 1 --2- 172.21.15.44:0/2619058950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd0100cc0 0x7f4cd0101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.121+0000 7f4cd81ff700 1 --2- 172.21.15.44:0/2619058950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4cd0101660 0x7f4cd0104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.121+0000 7f4cd81ff700 1 -- 172.21.15.44:0/2619058950 >> 172.21.15.44:0/2619058950 conn(0x7f4cd0077560 msgr2=0x7f4cd0077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:50.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.121+0000 7f4cd81ff700 1 -- 172.21.15.44:0/2619058950 shutdown_connections 2023-11-16T16:52:50.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.121+0000 7f4cd81ff700 1 -- 172.21.15.44:0/2619058950 wait complete. 2023-11-16T16:52:50.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.122+0000 7f4cd81ff700 1 Processor -- start 2023-11-16T16:52:50.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.122+0000 7f4cd81ff700 1 -- start start 2023-11-16T16:52:50.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd81ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd0100cc0 0x7f4cd0072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd81ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4cd0101660 0x7f4cd0073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd81ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4cd0105620 0x7f4cd01a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd81ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd0073b60 con 0x7f4cd0105620 2023-11-16T16:52:50.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd81ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd01a4cf0 con 0x7f4cd0100cc0 2023-11-16T16:52:50.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd81ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd01a4e30 con 0x7f4cd0101660 2023-11-16T16:52:50.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd579a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4cd0101660 0x7f4cd0073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd5f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd0100cc0 0x7f4cd0072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd579a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4cd0101660 0x7f4cd0073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:53758/0 (socket says 172.21.15.44:53758) 2023-11-16T16:52:50.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd579a700 1 -- 172.21.15.44:0/443417151 learned_addr learned my addr 172.21.15.44:0/443417151 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:50.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd679c700 1 --2- 172.21.15.44:0/443417151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4cd0105620 0x7f4cd01a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd5f9b700 1 -- 172.21.15.44:0/443417151 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4cd0101660 msgr2=0x7f4cd0073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:50.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd5f9b700 1 --2- 172.21.15.44:0/443417151 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4cd0101660 0x7f4cd0073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd5f9b700 1 -- 172.21.15.44:0/443417151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4cd0105620 msgr2=0x7f4cd01a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:50.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd5f9b700 1 --2- 172.21.15.44:0/443417151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4cd0105620 0x7f4cd01a47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd5f9b700 1 -- 172.21.15.44:0/443417151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc8009d00 con 0x7f4cd0100cc0 2023-11-16T16:52:50.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.123+0000 7f4cd579a700 1 --2- 172.21.15.44:0/443417151 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4cd0101660 0x7f4cd0073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:50.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.124+0000 7f4cd5f9b700 1 --2- 172.21.15.44:0/443417151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd0100cc0 0x7f4cd0072df0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc00e9f0 tx=0x7f4ccc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:50.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.124+0000 7f4cc6ffd700 1 -- 172.21.15.44:0/443417151 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc00c9c0 con 0x7f4cd0100cc0 2023-11-16T16:52:50.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.124+0000 7f4cd81ff700 1 -- 172.21.15.44:0/443417151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd01103e0 con 0x7f4cd0100cc0 2023-11-16T16:52:50.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.124+0000 7f4cc6ffd700 1 -- 172.21.15.44:0/443417151 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ccc010430 con 0x7f4cd0100cc0 2023-11-16T16:52:50.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.124+0000 7f4cc6ffd700 1 -- 172.21.15.44:0/443417151 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc018680 con 0x7f4cd0100cc0 2023-11-16T16:52:50.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.124+0000 7f4cd81ff700 1 -- 172.21.15.44:0/443417151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd0110250 con 0x7f4cd0100cc0 2023-11-16T16:52:50.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.128+0000 7f4cc6ffd700 1 -- 172.21.15.44:0/443417151 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f4ccc010860 con 0x7f4cd0100cc0 2023-11-16T16:52:50.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.129+0000 7f4cc6ffd700 1 --2- 172.21.15.44:0/443417151 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4cbc0a12c0 0x7f4cbc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.129+0000 7f4cc6ffd700 1 -- 172.21.15.44:0/443417151 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f4ccc014070 con 0x7f4cd0100cc0 2023-11-16T16:52:50.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.130+0000 7f4cd579a700 1 --2- 172.21.15.44:0/443417151 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4cbc0a12c0 0x7f4cbc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.130+0000 7f4cd579a700 1 --2- 172.21.15.44:0/443417151 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4cbc0a12c0 0x7f4cbc0a3770 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00745e0 tx=0x7f4cc0009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:50.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.222+0000 7f4cebbf4b80 1 Processor -- start 2023-11-16T16:52:50.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.222+0000 7f4cebbf4b80 1 -- start start 2023-11-16T16:52:50.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.223+0000 7f4cebbf4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c927b60 0x55af4c927f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.223+0000 7f4cebbf4b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c928470 0x55af4c92c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.223+0000 7f4cebbf4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c92cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.223+0000 7f4cebbf4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4c82d060 con 0x55af4c92c8b0 2023-11-16T16:52:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.223+0000 7f4cebbf4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4c87f8a0 con 0x55af4c927b60 2023-11-16T16:52:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.223+0000 7f4cebbf4b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4c878d30 con 0x55af4c928470 2023-11-16T16:52:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.223+0000 7f4cd679c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c92cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.223+0000 7f4cd579a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c928470 0x55af4c92c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.223+0000 7f4cd579a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c928470 0x55af4c92c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:53762/0 (socket says 172.21.15.44:53762) 2023-11-16T16:52:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.223+0000 7f4cd579a700 1 -- 172.21.15.44:0/417677560 learned_addr learned my addr 172.21.15.44:0/417677560 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.223+0000 7f4cd5f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c927b60 0x55af4c927f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.223+0000 7f4cd579a700 1 -- 172.21.15.44:0/417677560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c927b60 msgr2=0x55af4c927f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.224+0000 7f4cd579a700 1 --2- 172.21.15.44:0/417677560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c927b60 0x55af4c927f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.224+0000 7f4cd579a700 1 -- 172.21.15.44:0/417677560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 msgr2=0x55af4c92cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.224+0000 7f4cd579a700 1 --2- 172.21.15.44:0/417677560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c92cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.224+0000 7f4cd579a700 1 -- 172.21.15.44:0/417677560 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af4c6fa630 con 0x55af4c928470 2023-11-16T16:52:50.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.224+0000 7f4cd5f9b700 1 --2- 172.21.15.44:0/417677560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c927b60 0x55af4c927f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:50.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.224+0000 7f4cd579a700 1 --2- 172.21.15.44:0/417677560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c928470 0x55af4c92c370 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4cc000e700 tx=0x7f4cc0014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e3cf6bfefe3f42b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.224+0000 7f4cbbfff700 1 -- 172.21.15.44:0/417677560 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc000eea0 con 0x55af4c928470 2023-11-16T16:52:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.224+0000 7f4cbbfff700 1 -- 172.21.15.44:0/417677560 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4cc001e8b0 con 0x55af4c928470 2023-11-16T16:52:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.224+0000 7f4cbbfff700 1 -- 172.21.15.44:0/417677560 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc001d8f0 con 0x55af4c928470 2023-11-16T16:52:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.225+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/417677560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c928470 msgr2=0x55af4c92c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.225+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/417677560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c928470 0x55af4c92c370 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4cc000e700 tx=0x7f4cc0014000 comp rx=0 tx=0).stop 2023-11-16T16:52:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.225+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/417677560 shutdown_connections 2023-11-16T16:52:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.225+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/417677560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c92cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.225+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/417677560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c927b60 0x55af4c927f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.225+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/417677560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c928470 0x55af4c92c370 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.225+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/417677560 >> 172.21.15.44:0/417677560 conn(0x55af4c91d360 msgr2=0x55af4c922b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:50.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.225+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/417677560 shutdown_connections 2023-11-16T16:52:50.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.225+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/417677560 wait complete. 2023-11-16T16:52:50.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.226+0000 7f4cebbf4b80 1 -- start start 2023-11-16T16:52:50.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.226+0000 7f4cebbf4b80 1 Processor -- start 2023-11-16T16:52:50.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.227+0000 7f4cebbf4b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c934010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.227+0000 7f4cebbf4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 0x55af4c931290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.227+0000 7f4cebbf4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c931910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.227+0000 7f4cebbf4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4c81a8e0 con 0x55af4c92c8b0 2023-11-16T16:52:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.227+0000 7f4cebbf4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4c81a580 con 0x55af4c928470 2023-11-16T16:52:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.227+0000 7f4cebbf4b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4c6fc920 con 0x55af4c927b60 2023-11-16T16:52:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.227+0000 7f4cebbf4b80 10 client.0 _ll_register_callbacks cb 0x55af4c838780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.228+0000 7f4cd5f9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c934010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.228+0000 7f4cd5f9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c934010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:53772/0 (socket says 172.21.15.44:53772) 2023-11-16T16:52:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.228+0000 7f4cd5f9b700 1 -- 172.21.15.44:0/2816455379 learned_addr learned my addr 172.21.15.44:0/2816455379 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.228+0000 7f4cd579a700 1 --2- 172.21.15.44:0/2816455379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 0x55af4c931290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.228+0000 7f4cd679c700 1 --2- 172.21.15.44:0/2816455379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c931910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.228+0000 7f4cd5f9b700 1 -- 172.21.15.44:0/2816455379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 msgr2=0x55af4c931290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.228+0000 7f4cd5f9b700 1 --2- 172.21.15.44:0/2816455379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 0x55af4c931290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.228+0000 7f4cd5f9b700 1 -- 172.21.15.44:0/2816455379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 msgr2=0x55af4c931910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.228+0000 7f4cd5f9b700 1 --2- 172.21.15.44:0/2816455379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c931910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.228+0000 7f4cd5f9b700 1 -- 172.21.15.44:0/2816455379 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc0017070 con 0x55af4c927b60 2023-11-16T16:52:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.228+0000 7f4cd5f9b700 1 --2- 172.21.15.44:0/2816455379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c934010 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc096410 tx=0x7f4ccc09a050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c1f0815021d80c1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.228+0000 7f4cb97fa700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:52:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.229+0000 7f4cb97fa700 1 -- 172.21.15.44:0/2816455379 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc09a370 con 0x55af4c927b60 2023-11-16T16:52:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.229+0000 7f4cb97fa700 1 -- 172.21.15.44:0/2816455379 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f4ccc099400 con 0x55af4c927b60 2023-11-16T16:52:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.229+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/2816455379 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55af4c91ab20 con 0x55af4c927b60 2023-11-16T16:52:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.229+0000 7f4cb97fa700 1 -- 172.21.15.44:0/2816455379 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ccc0995c0 con 0x55af4c927b60 2023-11-16T16:52:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.229+0000 7f4cebbf4b80 10 client.26224 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.229+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/2816455379 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55af4c83f190 con 0x55af4c927b60 2023-11-16T16:52:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.229+0000 7f4cb97fa700 1 -- 172.21.15.44:0/2816455379 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55af4c91ab20 con 0x55af4c927b60 2023-11-16T16:52:50.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.229+0000 7f4cb97fa700 1 -- 172.21.15.44:0/2816455379 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4ccc00cb20 con 0x55af4c927b60 2023-11-16T16:52:50.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.230+0000 7f4cebbf4b80 10 client.26224 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:50.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.230+0000 7f4cebbf4b80 10 client.26224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:50.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.230+0000 7f4cebbf4b80 20 client.26224 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:50.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.230+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/2816455379 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55af4c9418b0 0x55af4c943c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.230+0000 7f4cebbf4b80 4 client.26224 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:50.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.230+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/2816455379 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55af4c83b0a0 con 0x55af4c9418b0 2023-11-16T16:52:50.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.230+0000 7f4cd579a700 1 --2- 172.21.15.44:0/2816455379 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55af4c9418b0 0x55af4c943c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.231+0000 7f4cd579a700 1 --2- 172.21.15.44:0/2816455379 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55af4c9418b0 0x55af4c943c90 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6e2a7e3b970aaa2 server_cookie=3f8f12f69610a4e5 in_seq=0 out_seq=0 2023-11-16T16:52:50.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.231+0000 7f4cb97fa700 10 client.26224 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.231+0000 7f4cb97fa700 1 -- 172.21.15.44:0/2816455379 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55af4c83b0a0 con 0x55af4c9418b0 2023-11-16T16:52:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.232+0000 7f4cb97fa700 10 client.26224 handle_command_reply: tid=0 2023-11-16T16:52:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.232+0000 7f4cebbf4b80 1 client.26224 shutdown 2023-11-16T16:52:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.232+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/2816455379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 msgr2=0x55af4c934010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.232+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/2816455379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c934010 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc096410 tx=0x7f4ccc09a050 comp rx=0 tx=0).stop 2023-11-16T16:52:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.232+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/2816455379 shutdown_connections 2023-11-16T16:52:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.232+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/2816455379 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55af4c9418b0 0x55af4c943c90 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.232+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/2816455379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c931910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.232+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/2816455379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c934010 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.232+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/2816455379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 0x55af4c931290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.232+0000 7f4cb97fa700 0 client.26224 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.232+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/2816455379 >> 172.21.15.44:0/2816455379 conn(0x55af4c91d360 msgr2=0x55af4c92dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:50.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.232+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/2816455379 shutdown_connections 2023-11-16T16:52:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.232+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/2816455379 wait complete. 2023-11-16T16:52:50.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.232+0000 7f4cebbf4b80 20 client.26224 trim_cache size 0 max 16384 2023-11-16T16:52:50.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.251+0000 7f4cebbf4b80 1 Processor -- start 2023-11-16T16:52:50.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.251+0000 7f4cebbf4b80 1 -- start start 2023-11-16T16:52:50.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.252+0000 7f4cebbf4b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c9389a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.252+0000 7f4cebbf4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 0x55af4c933bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.252+0000 7f4cebbf4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c922560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.252+0000 7f4cebbf4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4c91a730 con 0x55af4c92c8b0 2023-11-16T16:52:50.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.252+0000 7f4cebbf4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4c92f7f0 con 0x55af4c928470 2023-11-16T16:52:50.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.252+0000 7f4cebbf4b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4c92b3c0 con 0x55af4c927b60 2023-11-16T16:52:50.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.252+0000 7f4cd5f9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c9389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.252+0000 7f4cd679c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c922560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.252+0000 7f4cd5f9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c9389a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:53776/0 (socket says 172.21.15.44:53776) 2023-11-16T16:52:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.252+0000 7f4cd5f9b700 1 -- 172.21.15.44:0/2233808719 learned_addr learned my addr 172.21.15.44:0/2233808719 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.252+0000 7f4cd579a700 1 --2- 172.21.15.44:0/2233808719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 0x55af4c933bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.252+0000 7f4cd5f9b700 1 -- 172.21.15.44:0/2233808719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 msgr2=0x55af4c933bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.252+0000 7f4cd5f9b700 1 --2- 172.21.15.44:0/2233808719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 0x55af4c933bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.252+0000 7f4cd5f9b700 1 -- 172.21.15.44:0/2233808719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 msgr2=0x55af4c922560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.252+0000 7f4cd5f9b700 1 --2- 172.21.15.44:0/2233808719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c922560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.253+0000 7f4cd5f9b700 1 -- 172.21.15.44:0/2233808719 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af4c83eee0 con 0x55af4c927b60 2023-11-16T16:52:50.256 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.253+0000 7f4cd5f9b700 1 --2- 172.21.15.44:0/2233808719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c9389a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc099c10 tx=0x7f4ccc099ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65e55633cf68f626 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:50.256 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.253+0000 7f4cc4ff9700 1 -- 172.21.15.44:0/2233808719 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc00cd20 con 0x55af4c927b60 2023-11-16T16:52:50.256 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.253+0000 7f4cc4ff9700 1 -- 172.21.15.44:0/2233808719 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ccc0c8020 con 0x55af4c927b60 2023-11-16T16:52:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.253+0000 7f4cc4ff9700 1 -- 172.21.15.44:0/2233808719 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc09a170 con 0x55af4c927b60 2023-11-16T16:52:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.254+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/2233808719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 msgr2=0x55af4c9389a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.254+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/2233808719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c9389a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc099c10 tx=0x7f4ccc099ce0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.254+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/2233808719 shutdown_connections 2023-11-16T16:52:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.254+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/2233808719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c922560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.254+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/2233808719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c9389a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.254+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/2233808719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 0x55af4c933bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.254+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/2233808719 >> 172.21.15.44:0/2233808719 conn(0x55af4c91d360 msgr2=0x55af4c92a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.254+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/2233808719 shutdown_connections 2023-11-16T16:52:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.254+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/2233808719 wait complete. 2023-11-16T16:52:50.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.254+0000 7f4cebbf4b80 1 -- start start 2023-11-16T16:52:50.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.255+0000 7f4cebbf4b80 1 Processor -- start 2023-11-16T16:52:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.255+0000 7f4cebbf4b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c930e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.255+0000 7f4cebbf4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 0x55af4c931340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.255+0000 7f4cebbf4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c92fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.255+0000 7f4cebbf4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4c92ff80 con 0x55af4c92c8b0 2023-11-16T16:52:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.255+0000 7f4cebbf4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4c9300f0 con 0x55af4c928470 2023-11-16T16:52:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.255+0000 7f4cebbf4b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4c9302b0 con 0x55af4c927b60 2023-11-16T16:52:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.255+0000 7f4cebbf4b80 10 client.0 _ll_register_callbacks cb 0x7f4ccc099870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.256+0000 7f4cd679c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c92fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.256+0000 7f4cd579a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 0x55af4c931340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.256+0000 7f4cd679c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c92fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57082/0 (socket says 172.21.15.44:57082) 2023-11-16T16:52:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.256+0000 7f4cd579a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 0x55af4c931340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:42600/0 (socket says 172.21.15.44:42600) 2023-11-16T16:52:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.256+0000 7f4cd5f9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c930e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.256+0000 7f4cd679c700 1 -- 172.21.15.44:0/1235347436 learned_addr learned my addr 172.21.15.44:0/1235347436 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.256+0000 7f4cd679c700 1 -- 172.21.15.44:0/1235347436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 msgr2=0x55af4c930e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.256+0000 7f4cd679c700 1 --2- 172.21.15.44:0/1235347436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c930e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.256+0000 7f4cd679c700 1 -- 172.21.15.44:0/1235347436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 msgr2=0x55af4c931340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.256+0000 7f4cd679c700 1 --2- 172.21.15.44:0/1235347436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 0x55af4c931340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.256+0000 7f4cd679c700 1 -- 172.21.15.44:0/1235347436 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af4c83f190 con 0x55af4c92c8b0 2023-11-16T16:52:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.256+0000 7f4cd5f9b700 1 --2- 172.21.15.44:0/1235347436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c930e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:50.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.256+0000 7f4cd579a700 1 --2- 172.21.15.44:0/1235347436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 0x55af4c931340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.256+0000 7f4cd679c700 1 --2- 172.21.15.44:0/1235347436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c92fa40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800b0a0 tx=0x7f4cc80095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d2b5ea6c00f4f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.256+0000 7f4cb97fa700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:52:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.257+0000 7f4cb97fa700 1 -- 172.21.15.44:0/1235347436 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc800e040 con 0x55af4c92c8b0 2023-11-16T16:52:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.257+0000 7f4cb97fa700 1 -- 172.21.15.44:0/1235347436 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f4cc80074e0 con 0x55af4c92c8b0 2023-11-16T16:52:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.257+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/1235347436 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4cc001e8b0 con 0x55af4c92c8b0 2023-11-16T16:52:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.257+0000 7f4cb97fa700 1 -- 172.21.15.44:0/1235347436 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4cc80076c0 con 0x55af4c92c8b0 2023-11-16T16:52:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.257+0000 7f4cebbf4b80 10 client.16218 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.257+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/1235347436 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55af4c68fc60 con 0x55af4c92c8b0 2023-11-16T16:52:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.257+0000 7f4cb97fa700 1 -- 172.21.15.44:0/1235347436 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4cc8030020 con 0x55af4c92c8b0 2023-11-16T16:52:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.258+0000 7f4cb97fa700 1 -- 172.21.15.44:0/1235347436 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4cc800b0d0 con 0x55af4c92c8b0 2023-11-16T16:52:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.258+0000 7f4cebbf4b80 10 client.16218 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:50.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.258+0000 7f4cebbf4b80 10 client.16218 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:50.263 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.258+0000 7f4cebbf4b80 20 client.16218 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:50.263 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.258+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/1235347436 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55af4c996720 0x55af4c998b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:50.263 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.258+0000 7f4cebbf4b80 4 client.16218 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:52:50.263 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.258+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/1235347436 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55af4c847660 con 0x55af4c996720 2023-11-16T16:52:50.263 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.258+0000 7f4cd5f9b700 1 --2- 172.21.15.44:0/1235347436 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55af4c996720 0x55af4c998b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:50.264 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.259+0000 7f4cd5f9b700 1 --2- 172.21.15.44:0/1235347436 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55af4c996720 0x55af4c998b30 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f10dd9b9c46ee72a server_cookie=805436cf30661403 in_seq=0 out_seq=0 2023-11-16T16:52:50.264 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.259+0000 7f4cb97fa700 10 client.16218 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:50.264 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.259+0000 7f4cb97fa700 1 -- 172.21.15.44:0/1235347436 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55af4c847660 con 0x55af4c996720 2023-11-16T16:52:50.264 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.259+0000 7f4cb97fa700 10 client.16218 handle_command_reply: tid=0 2023-11-16T16:52:50.264 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.259+0000 7f4cebbf4b80 1 client.16218 shutdown 2023-11-16T16:52:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.259+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/1235347436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 msgr2=0x55af4c92fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.259+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/1235347436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c92fa40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800b0a0 tx=0x7f4cc80095d0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.259+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/1235347436 shutdown_connections 2023-11-16T16:52:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.259+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/1235347436 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55af4c996720 0x55af4c998b30 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.259+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/1235347436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55af4c92c8b0 0x55af4c92fa40 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.259+0000 7f4cb97fa700 0 client.16218 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.259+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/1235347436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55af4c927b60 0x55af4c930e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.259+0000 7f4cebbf4b80 1 --2- 172.21.15.44:0/1235347436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af4c928470 0x55af4c931340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.259+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/1235347436 >> 172.21.15.44:0/1235347436 conn(0x55af4c91d360 msgr2=0x55af4c92e570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.259+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/1235347436 shutdown_connections 2023-11-16T16:52:50.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.259+0000 7f4cebbf4b80 1 -- 172.21.15.44:0/1235347436 wait complete. 2023-11-16T16:52:50.266 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.260+0000 7f4cebbf4b80 20 client.16218 trim_cache size 0 max 16384 2023-11-16T16:52:50.266 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:52:50.266 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T16:52:50.266 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T16:52:50.266 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:52:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.260+0000 7f4cd81ff700 1 -- 172.21.15.44:0/443417151 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4cbc0a12c0 msgr2=0x7f4cbc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.260+0000 7f4cd81ff700 1 --2- 172.21.15.44:0/443417151 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4cbc0a12c0 0x7f4cbc0a3770 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00745e0 tx=0x7f4cc0009260 comp rx=0 tx=0).stop 2023-11-16T16:52:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.260+0000 7f4cd81ff700 1 -- 172.21.15.44:0/443417151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd0100cc0 msgr2=0x7f4cd0072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.260+0000 7f4cd81ff700 1 --2- 172.21.15.44:0/443417151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd0100cc0 0x7f4cd0072df0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc00e9f0 tx=0x7f4ccc00ed00 comp rx=0 tx=0).stop 2023-11-16T16:52:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.260+0000 7f4cd81ff700 1 -- 172.21.15.44:0/443417151 shutdown_connections 2023-11-16T16:52:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.260+0000 7f4cd81ff700 1 --2- 172.21.15.44:0/443417151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4cd0105620 0x7f4cd01a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.260+0000 7f4cd81ff700 1 --2- 172.21.15.44:0/443417151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd0100cc0 0x7f4cd0072df0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.260+0000 7f4cd81ff700 1 --2- 172.21.15.44:0/443417151 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4cbc0a12c0 0x7f4cbc0a3770 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.260+0000 7f4cd81ff700 1 --2- 172.21.15.44:0/443417151 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4cd0101660 0x7f4cd0073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:50.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.260+0000 7f4cd81ff700 1 -- 172.21.15.44:0/443417151 >> 172.21.15.44:0/443417151 conn(0x7f4cd0077560 msgr2=0x7f4cd00feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:50.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.260+0000 7f4cd81ff700 1 -- 172.21.15.44:0/443417151 shutdown_connections 2023-11-16T16:52:50.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:50.261+0000 7f4cd81ff700 1 -- 172.21.15.44:0/443417151 wait complete. 2023-11-16T16:52:50.328 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T16:52:50.328 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:52:50.532 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:50.533 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:50.621 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:50.621 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:50.652 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:50.652 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:50.652 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 99 -lt 100 ']' 2023-11-16T16:52:50.652 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 99 + 1 2023-11-16T16:52:50.689 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=100 2023-11-16T16:52:50.690 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:50.704 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:50.704 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:50.707 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2023-11-16T16:52:50.708 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2023-11-16T16:52:50.708 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:51.257 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2023-11-16T16:52:51.258 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2023-11-16T16:52:51.411 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2023-11-16T16:52:51.411 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:51.450 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:51.450 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:51.450 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 100 -lt 100 ']' 2023-11-16T16:52:51.458 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2023-11-16T16:52:51.494 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -n system.posix_acl_access -v 0x02000000 d1 2023-11-16T16:52:51.500 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -n system.posix_acl_default -v 0x02000000 . 2023-11-16T16:52:51.503 INFO:tasks.workunit.client.0.smithi044.stderr:+ expect_failure getfattr -n system.posix_acl_access d1 2023-11-16T16:52:51.503 INFO:tasks.workunit.client.0.smithi044.stderr:+ getfattr -n system.posix_acl_access d1 2023-11-16T16:52:51.538 INFO:tasks.workunit.client.0.smithi044.stderr:d1: system.posix_acl_access: No such attribute 2023-11-16T16:52:51.539 INFO:tasks.workunit.client.0.smithi044.stderr:+ return 0 2023-11-16T16:52:51.539 INFO:tasks.workunit.client.0.smithi044.stderr:+ expect_failure getfattr -n system.posix_acl_default . 2023-11-16T16:52:51.539 INFO:tasks.workunit.client.0.smithi044.stderr:+ getfattr -n system.posix_acl_default . 2023-11-16T16:52:51.541 INFO:tasks.workunit.client.0.smithi044.stderr:.: system.posix_acl_default: No such attribute 2023-11-16T16:52:51.541 INFO:tasks.workunit.client.0.smithi044.stderr:+ return 0 2023-11-16T16:52:51.541 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2023-11-16T16:52:51.546 INFO:tasks.workunit.client.0.smithi044.stderr:+ cd .. 2023-11-16T16:52:51.546 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir testdir 2023-11-16T16:52:51.560 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo OK 2023-11-16T16:52:51.561 INFO:tasks.workunit.client.0.smithi044.stdout:OK 2023-11-16T16:52:51.561 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T16:52:51.561 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T16:52:51.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:51 smithi176 ceph-mon[131630]: pgmap v90: 129 pgs: 129 active+clean; 3.1 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-11-16T16:52:51.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:51 smithi077 ceph-mon[131204]: pgmap v90: 129 pgs: 129 active+clean; 3.1 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-11-16T16:52:51.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:51 smithi044 ceph-mon[128750]: pgmap v90: 129 pgs: 129 active+clean; 3.1 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-11-16T16:52:52.750 INFO:tasks.workunit:Running workunit fs/misc/chmod.sh... 2023-11-16T16:52:52.750 DEBUG:teuthology.orchestra.run.smithi044:workunit test fs/misc/chmod.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh 2023-11-16T16:52:52.927 INFO:tasks.workunit.client.0.smithi044.stderr:+ set -e 2023-11-16T16:52:52.927 INFO:tasks.workunit.client.0.smithi044.stderr:+ file=test_chmod.186078 2023-11-16T16:52:52.927 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo foo 2023-11-16T16:52:52.933 INFO:tasks.workunit.client.0.smithi044.stderr:+ test 0 '!=' 0 2023-11-16T16:52:52.933 INFO:tasks.workunit.client.0.smithi044.stderr:+ chmod 400 test_chmod.186078 2023-11-16T16:52:52.946 INFO:tasks.workunit.client.0.smithi044.stderr:+ test 0 '!=' 0 2023-11-16T16:52:52.946 INFO:tasks.workunit.client.0.smithi044.stderr:+ check_perms test_chmod.186078 -r-------- 2023-11-16T16:52:52.946 INFO:tasks.workunit.client.0.smithi044.stderr:+ file=test_chmod.186078 2023-11-16T16:52:52.946 INFO:tasks.workunit.client.0.smithi044.stderr:++ ls -la test_chmod.186078 2023-11-16T16:52:52.958 INFO:tasks.workunit.client.0.smithi044.stderr:+ r='-r--------. 1 ubuntu ubuntu 4 Nov 16 16:52 test_chmod.186078' 2023-11-16T16:52:52.958 INFO:tasks.workunit.client.0.smithi044.stderr:+ test 0 '!=' 0 2023-11-16T16:52:52.959 INFO:tasks.workunit.client.0.smithi044.stderr:+ perms=-r-------- 2023-11-16T16:52:52.960 INFO:tasks.workunit.client.0.smithi044.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 Nov 16 16:52 test_chmod.186078 2023-11-16T16:52:52.960 INFO:tasks.workunit.client.0.smithi044.stderr:++ awk '{print $1}' 2023-11-16T16:52:53.034 INFO:tasks.workunit.client.0.smithi044.stderr:+ test -r-------- '!=' -r--------. 2023-11-16T16:52:53.036 INFO:tasks.workunit.client.0.smithi044.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 Nov 16 16:52 test_chmod.186078 2023-11-16T16:52:53.036 INFO:tasks.workunit.client.0.smithi044.stderr:++ awk '{print $1}' 2023-11-16T16:52:53.040 INFO:tasks.workunit.client.0.smithi044.stderr:+ test -r--------. '!=' -r--------. 2023-11-16T16:52:53.040 INFO:tasks.workunit.client.0.smithi044.stderr:+ set +e 2023-11-16T16:52:53.040 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo bar 2023-11-16T16:52:53.040 INFO:tasks.workunit.client.0.smithi044.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh: line 40: test_chmod.186078: Permission denied 2023-11-16T16:52:53.041 INFO:tasks.workunit.client.0.smithi044.stderr:+ test 1 = 0 2023-11-16T16:52:53.041 INFO:tasks.workunit.client.0.smithi044.stderr:+ set -e 2023-11-16T16:52:53.041 INFO:tasks.workunit.client.0.smithi044.stderr:+ chmod 600 test_chmod.186078 2023-11-16T16:52:53.042 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo bar 2023-11-16T16:52:53.043 INFO:tasks.workunit.client.0.smithi044.stderr:+ test 0 '!=' 0 2023-11-16T16:52:53.043 INFO:tasks.workunit.client.0.smithi044.stderr:+ check_perms test_chmod.186078 -rw------- 2023-11-16T16:52:53.043 INFO:tasks.workunit.client.0.smithi044.stderr:+ file=test_chmod.186078 2023-11-16T16:52:53.043 INFO:tasks.workunit.client.0.smithi044.stderr:++ ls -la test_chmod.186078 2023-11-16T16:52:53.046 INFO:tasks.workunit.client.0.smithi044.stderr:+ r='-rw-------. 1 ubuntu ubuntu 8 Nov 16 16:52 test_chmod.186078' 2023-11-16T16:52:53.046 INFO:tasks.workunit.client.0.smithi044.stderr:+ test 0 '!=' 0 2023-11-16T16:52:53.046 INFO:tasks.workunit.client.0.smithi044.stderr:+ perms=-rw------- 2023-11-16T16:52:53.047 INFO:tasks.workunit.client.0.smithi044.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 Nov 16 16:52 test_chmod.186078 2023-11-16T16:52:53.048 INFO:tasks.workunit.client.0.smithi044.stderr:++ awk '{print $1}' 2023-11-16T16:52:53.052 INFO:tasks.workunit.client.0.smithi044.stderr:+ test -rw------- '!=' -rw-------. 2023-11-16T16:52:53.053 INFO:tasks.workunit.client.0.smithi044.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 Nov 16 16:52 test_chmod.186078 2023-11-16T16:52:53.053 INFO:tasks.workunit.client.0.smithi044.stderr:++ awk '{print $1}' 2023-11-16T16:52:53.057 INFO:tasks.workunit.client.0.smithi044.stderr:+ test -rw-------. '!=' -rw-------. 2023-11-16T16:52:53.057 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo foo 2023-11-16T16:52:53.058 INFO:tasks.workunit.client.0.smithi044.stderr:+ test 0 '!=' 0 2023-11-16T16:52:53.058 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T16:52:53.058 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T16:52:53.144 INFO:tasks.workunit:Running workunit fs/misc/dac_override.sh... 2023-11-16T16:52:53.144 DEBUG:teuthology.orchestra.run.smithi044:workunit test fs/misc/dac_override.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dac_override.sh 2023-11-16T16:52:53.203 INFO:tasks.workunit.client.0.smithi044.stderr:+ set -e 2023-11-16T16:52:53.203 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir -p testdir 2023-11-16T16:52:53.206 INFO:tasks.workunit.client.0.smithi044.stderr:+ file=test_chmod.186137 2023-11-16T16:52:53.207 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo foo 2023-11-16T16:52:53.208 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo chmod 600 testdir 2023-11-16T16:52:53.242 INFO:tasks.workunit.client.0.smithi044.stderr:+ expect_failure cat testdir/test_chmod.186137 2023-11-16T16:52:53.242 INFO:tasks.workunit.client.0.smithi044.stderr:+ cat testdir/test_chmod.186137 2023-11-16T16:52:53.252 INFO:tasks.workunit.client.0.smithi044.stderr:cat: testdir/test_chmod.186137: Permission denied 2023-11-16T16:52:53.253 INFO:tasks.workunit.client.0.smithi044.stderr:+ return 0 2023-11-16T16:52:53.253 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo cat testdir/test_chmod.186137 2023-11-16T16:52:53.280 INFO:tasks.workunit.client.0.smithi044.stdout:foo 2023-11-16T16:52:53.283 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T16:52:53.283 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T16:52:53.371 INFO:tasks.workunit:Running workunit fs/misc/direct_io.py... 2023-11-16T16:52:53.372 DEBUG:teuthology.orchestra.run.smithi044:workunit test fs/misc/direct_io.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/direct_io.py 2023-11-16T16:52:53.397 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.395+0000 7f61735ff700 1 -- 172.21.15.44:0/1515560858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f616c10a800 msgr2=0x7f616c10cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.397 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.395+0000 7f61735ff700 1 --2- 172.21.15.44:0/1515560858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f616c10a800 0x7f616c10cc90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6164009f90 tx=0x7f616400b330 comp rx=0 tx=0).stop 2023-11-16T16:52:53.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:53 smithi044 ceph-mon[128750]: pgmap v91: 129 pgs: 129 active+clean; 3.3 MiB data, 90 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 42 KiB/s wr, 12 op/s 2023-11-16T16:52:53.397 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.396+0000 7f61735ff700 1 -- 172.21.15.44:0/1515560858 shutdown_connections 2023-11-16T16:52:53.398 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.396+0000 7f61735ff700 1 --2- 172.21.15.44:0/1515560858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f616c10a800 0x7f616c10cc90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.398 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.396+0000 7f61735ff700 1 --2- 172.21.15.44:0/1515560858 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f616c1014c0 0x7f616c101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.398 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.396+0000 7f61735ff700 1 --2- 172.21.15.44:0/1515560858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f616c101dd0 0x7f616c10a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.398 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.396+0000 7f61735ff700 1 -- 172.21.15.44:0/1515560858 >> 172.21.15.44:0/1515560858 conn(0x7f616c076310 msgr2=0x7f616c076710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.396+0000 7f5c0dfff700 1 -- 172.21.15.44:0/1572833754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c08105560 msgr2=0x7f5c0810d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.396+0000 7f5c0dfff700 1 --2- 172.21.15.44:0/1572833754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c08105560 0x7f5c0810d170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5c00009f90 tx=0x7f5c0000b330 comp rx=0 tx=0).stop 2023-11-16T16:52:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.396+0000 7f5c0dfff700 1 -- 172.21.15.44:0/1572833754 shutdown_connections 2023-11-16T16:52:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.396+0000 7f5c0dfff700 1 --2- 172.21.15.44:0/1572833754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c08105560 0x7f5c0810d170 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.396+0000 7f5c0dfff700 1 --2- 172.21.15.44:0/1572833754 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c08100c50 0x7f5c08101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.396+0000 7f5c0dfff700 1 --2- 172.21.15.44:0/1572833754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c081015f0 0x7f5c08104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.396+0000 7f5c0dfff700 1 -- 172.21.15.44:0/1572833754 >> 172.21.15.44:0/1572833754 conn(0x7f5c08077550 msgr2=0x7f5c08077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:53.399 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.396+0000 7f61735ff700 1 -- 172.21.15.44:0/1515560858 shutdown_connections 2023-11-16T16:52:53.399 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.396+0000 7f61735ff700 1 -- 172.21.15.44:0/1515560858 wait complete. 2023-11-16T16:52:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.396+0000 7f5c0dfff700 1 -- 172.21.15.44:0/1572833754 shutdown_connections 2023-11-16T16:52:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.396+0000 7f5c0dfff700 1 -- 172.21.15.44:0/1572833754 wait complete. 2023-11-16T16:52:53.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.397+0000 7f61735ff700 1 Processor -- start 2023-11-16T16:52:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.397+0000 7f5c0dfff700 1 Processor -- start 2023-11-16T16:52:53.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.398+0000 7f61735ff700 1 -- start start 2023-11-16T16:52:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.398+0000 7f5c0dfff700 1 -- start start 2023-11-16T16:52:53.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.398+0000 7f61735ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f616c1014c0 0x7f616c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.398+0000 7f5c0dfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c08100c50 0x7f5c0819e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.398+0000 7f61735ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f616c101dd0 0x7f616c198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.398+0000 7f61735ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f616c10a800 0x7f616c19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.398+0000 7f61735ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c19d610 con 0x7f616c1014c0 2023-11-16T16:52:53.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.398+0000 7f61735ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c19d780 con 0x7f616c101dd0 2023-11-16T16:52:53.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.398+0000 7f61735ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c19d940 con 0x7f616c10a800 2023-11-16T16:52:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.398+0000 7f5c0dfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c081015f0 0x7f5c0819ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.398+0000 7f5c0dfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c08105560 0x7f5c08198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.398+0000 7f5c0dfff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c08199280 con 0x7f5c08105560 2023-11-16T16:52:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.398+0000 7f5c0dfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c081993f0 con 0x7f5c08100c50 2023-11-16T16:52:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.398+0000 7f5c0dfff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c081995b0 con 0x7f5c081015f0 2023-11-16T16:52:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.398+0000 7f5c077fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c08100c50 0x7f5c0819e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f5c077fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c08100c50 0x7f5c0819e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:42620/0 (socket says 172.21.15.44:42620) 2023-11-16T16:52:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f5c077fe700 1 -- 172.21.15.44:0/76628241 learned_addr learned my addr 172.21.15.44:0/76628241 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f5c06ffd700 1 --2- 172.21.15.44:0/76628241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c081015f0 0x7f5c0819ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f617139b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f616c1014c0 0x7f616c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f6170b9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f616c101dd0 0x7f616c198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f6170b9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f616c101dd0 0x7f616c198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:42634/0 (socket says 172.21.15.44:42634) 2023-11-16T16:52:53.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f6170b9a700 1 -- 172.21.15.44:0/1941454846 learned_addr learned my addr 172.21.15.44:0/1941454846 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:53.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f6171b9c700 1 --2- 172.21.15.44:0/1941454846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f616c10a800 0x7f616c19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f5c077fe700 1 -- 172.21.15.44:0/76628241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c081015f0 msgr2=0x7f5c0819ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f5c077fe700 1 --2- 172.21.15.44:0/76628241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c081015f0 0x7f5c0819ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f5c077fe700 1 -- 172.21.15.44:0/76628241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c08105560 msgr2=0x7f5c08198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f5c077fe700 1 --2- 172.21.15.44:0/76628241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c08105560 0x7f5c08198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f5c077fe700 1 -- 172.21.15.44:0/76628241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc0095f0 con 0x7f5c08100c50 2023-11-16T16:52:53.405 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f6170b9a700 1 -- 172.21.15.44:0/1941454846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f616c10a800 msgr2=0x7f616c19cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f6170b9a700 1 --2- 172.21.15.44:0/1941454846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f616c10a800 0x7f616c19cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f6170b9a700 1 -- 172.21.15.44:0/1941454846 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f616c1014c0 msgr2=0x7f616c198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f6170b9a700 1 --2- 172.21.15.44:0/1941454846 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f616c1014c0 0x7f616c198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f6170b9a700 1 -- 172.21.15.44:0/1941454846 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6164009c40 con 0x7f616c101dd0 2023-11-16T16:52:53.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.399+0000 7f6170b9a700 1 --2- 172.21.15.44:0/1941454846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f616c101dd0 0x7f616c198b70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f615c00e9a0 tx=0x7f615c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.400+0000 7f5c077fe700 1 --2- 172.21.15.44:0/76628241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c08100c50 0x7f5c0819e520 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5bf800e9a0 tx=0x7f5bf800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:53.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.400+0000 7f61627fc700 1 -- 172.21.15.44:0/1941454846 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c009790 con 0x7f616c101dd0 2023-11-16T16:52:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.400+0000 7f5c04ff9700 1 -- 172.21.15.44:0/76628241 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8009790 con 0x7f5c08100c50 2023-11-16T16:52:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.400+0000 7f5c0dfff700 1 -- 172.21.15.44:0/76628241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c00009c40 con 0x7f5c08100c50 2023-11-16T16:52:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.400+0000 7f5c04ff9700 1 -- 172.21.15.44:0/76628241 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5bf800cbd0 con 0x7f5c08100c50 2023-11-16T16:52:53.408 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.400+0000 7f61627fc700 1 -- 172.21.15.44:0/1941454846 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f615c00ce80 con 0x7f616c101dd0 2023-11-16T16:52:53.408 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.400+0000 7f61627fc700 1 -- 172.21.15.44:0/1941454846 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c018680 con 0x7f616c101dd0 2023-11-16T16:52:53.408 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.400+0000 7f61735ff700 1 -- 172.21.15.44:0/1941454846 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616c1102c0 con 0x7f616c101dd0 2023-11-16T16:52:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.400+0000 7f5c0dfff700 1 -- 172.21.15.44:0/76628241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c080735f0 con 0x7f5c08100c50 2023-11-16T16:52:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.400+0000 7f5c04ff9700 1 -- 172.21.15.44:0/76628241 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8018680 con 0x7f5c08100c50 2023-11-16T16:52:53.409 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.400+0000 7f61735ff700 1 -- 172.21.15.44:0/1941454846 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c110130 con 0x7f616c101dd0 2023-11-16T16:52:53.409 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.402+0000 7f61735ff700 1 -- 172.21.15.44:0/1941454846 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616c110450 con 0x7f616c101dd0 2023-11-16T16:52:53.409 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.403+0000 7f61627fc700 1 -- 172.21.15.44:0/1941454846 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f615c010430 con 0x7f616c101dd0 2023-11-16T16:52:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.404+0000 7f5c04ff9700 1 -- 172.21.15.44:0/76628241 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f5bf800cd40 con 0x7f5c08100c50 2023-11-16T16:52:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.405+0000 7f5c04ff9700 1 --2- 172.21.15.44:0/76628241 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f5bf00a12c0 0x7f5bf00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.405+0000 7f5c04ff9700 1 -- 172.21.15.44:0/76628241 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f5bf8014070 con 0x7f5c08100c50 2023-11-16T16:52:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.406+0000 7f61627fc700 1 --2- 172.21.15.44:0/1941454846 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f61580a1340 0x7f61580a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.406+0000 7f61627fc700 1 -- 172.21.15.44:0/1941454846 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f615c014070 con 0x7f616c101dd0 2023-11-16T16:52:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.406+0000 7f5c06ffd700 1 --2- 172.21.15.44:0/76628241 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f5bf00a12c0 0x7f5bf00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.406+0000 7f617139b700 1 --2- 172.21.15.44:0/1941454846 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f61580a1340 0x7f61580a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.407+0000 7f5c06ffd700 1 --2- 172.21.15.44:0/76628241 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f5bf00a12c0 0x7f5bf00a3770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5c0810cfd0 tx=0x7f5bfc009310 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:53.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.407+0000 7f617139b700 1 --2- 172.21.15.44:0/1941454846 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f61580a1340 0x7f61580a37f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6168005fd0 tx=0x7f6168009310 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:53.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.409+0000 7f61627fc700 1 -- 172.21.15.44:0/1941454846 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f615c051730 con 0x7f616c101dd0 2023-11-16T16:52:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.446+0000 7f5c21829b80 1 Processor -- start 2023-11-16T16:52:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.446+0000 7f5c21829b80 1 -- start start 2023-11-16T16:52:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.447+0000 7f5c21829b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1142566b0 0x55c1144001e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.447+0000 7f5c21829b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c114400720 0x55c1143fa560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.447+0000 7f5c21829b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400b90 0x55c1143fab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.447+0000 7f5c21829b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c114300000 con 0x55c1142566b0 2023-11-16T16:52:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.447+0000 7f5c21829b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c114352840 con 0x55c114400b90 2023-11-16T16:52:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.447+0000 7f5c21829b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c11434bcd0 con 0x55c114400720 2023-11-16T16:52:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.447+0000 7f5c077fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1142566b0 0x55c1144001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.447+0000 7f5c077fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1142566b0 0x55c1144001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57142/0 (socket says 172.21.15.44:57142) 2023-11-16T16:52:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.447+0000 7f5c077fe700 1 -- 172.21.15.44:0/3999512023 learned_addr learned my addr 172.21.15.44:0/3999512023 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.447+0000 7f5c07fff700 1 --2- 172.21.15.44:0/3999512023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400b90 0x55c1143fab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.447+0000 7f5c06ffd700 1 --2- 172.21.15.44:0/3999512023 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c114400720 0x55c1143fa560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.448+0000 7f5c07fff700 1 -- 172.21.15.44:0/3999512023 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c114400720 msgr2=0x55c1143fa560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.448+0000 7f5c07fff700 1 --2- 172.21.15.44:0/3999512023 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c114400720 0x55c1143fa560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.448+0000 7f5c07fff700 1 -- 172.21.15.44:0/3999512023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1142566b0 msgr2=0x55c1144001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.448+0000 7f5c07fff700 1 --2- 172.21.15.44:0/3999512023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1142566b0 0x55c1144001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.448+0000 7f5c07fff700 1 -- 172.21.15.44:0/3999512023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c1141cd5d0 con 0x55c114400b90 2023-11-16T16:52:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.448+0000 7f5c077fe700 1 --2- 172.21.15.44:0/3999512023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1142566b0 0x55c1144001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.448+0000 7f5c07fff700 1 --2- 172.21.15.44:0/3999512023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400b90 0x55c1143fab00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5c00015000 tx=0x7f5c00009350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc264cb59c566edc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.448+0000 7f5bedffb700 1 -- 172.21.15.44:0/3999512023 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0000e040 con 0x55c114400b90 2023-11-16T16:52:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.448+0000 7f5bedffb700 1 -- 172.21.15.44:0/3999512023 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5c00007d00 con 0x55c114400b90 2023-11-16T16:52:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.448+0000 7f5bedffb700 1 -- 172.21.15.44:0/3999512023 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0000bc30 con 0x55c114400b90 2023-11-16T16:52:53.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.449+0000 7f5c21829b80 1 -- 172.21.15.44:0/3999512023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400b90 msgr2=0x55c1143fab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.449+0000 7f5c21829b80 1 --2- 172.21.15.44:0/3999512023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400b90 0x55c1143fab00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5c00015000 tx=0x7f5c00009350 comp rx=0 tx=0).stop 2023-11-16T16:52:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.449+0000 7f5c21829b80 1 -- 172.21.15.44:0/3999512023 shutdown_connections 2023-11-16T16:52:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.449+0000 7f5c21829b80 1 --2- 172.21.15.44:0/3999512023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400b90 0x55c1143fab00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.449+0000 7f5c21829b80 1 --2- 172.21.15.44:0/3999512023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1142566b0 0x55c1144001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.449+0000 7f5c21829b80 1 --2- 172.21.15.44:0/3999512023 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c114400720 0x55c1143fa560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.449+0000 7f5c21829b80 1 -- 172.21.15.44:0/3999512023 >> 172.21.15.44:0/3999512023 conn(0x55c1143eff50 msgr2=0x55c1143f5270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.449+0000 7f5c21829b80 1 -- 172.21.15.44:0/3999512023 shutdown_connections 2023-11-16T16:52:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.449+0000 7f5c21829b80 1 -- 172.21.15.44:0/3999512023 wait complete. 2023-11-16T16:52:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.450+0000 7f5c21829b80 1 -- start start 2023-11-16T16:52:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.450+0000 7f5c21829b80 1 Processor -- start 2023-11-16T16:52:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.451+0000 7f5c21829b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1142566b0 0x55c114406a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.451+0000 7f5c21829b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c114400720 0x55c114403c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.451+0000 7f5c21829b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c114400b90 0x55c114404250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.451+0000 7f5c21829b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1142ed880 con 0x55c114400b90 2023-11-16T16:52:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.451+0000 7f5c21829b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1142ed520 con 0x55c1142566b0 2023-11-16T16:52:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.451+0000 7f5c21829b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1141cf8c0 con 0x55c114400720 2023-11-16T16:52:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.451+0000 7f5c21829b80 10 client.0 _ll_register_callbacks cb 0x55c11430b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.451+0000 7f5c077fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1142566b0 0x55c114406a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.451+0000 7f5c06ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c114400720 0x55c114403c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.451+0000 7f5c077fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1142566b0 0x55c114406a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:42644/0 (socket says 172.21.15.44:42644) 2023-11-16T16:52:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.451+0000 7f5c077fe700 1 -- 172.21.15.44:0/918573921 learned_addr learned my addr 172.21.15.44:0/918573921 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.451+0000 7f5c07fff700 1 --2- 172.21.15.44:0/918573921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c114400b90 0x55c114404250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.452+0000 7f5c077fe700 1 -- 172.21.15.44:0/918573921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c114400720 msgr2=0x55c114403c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.452+0000 7f5c077fe700 1 --2- 172.21.15.44:0/918573921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c114400720 0x55c114403c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.452+0000 7f5c077fe700 1 -- 172.21.15.44:0/918573921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c114400b90 msgr2=0x55c114404250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.452+0000 7f5c077fe700 1 --2- 172.21.15.44:0/918573921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c114400b90 0x55c114404250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.452+0000 7f5c077fe700 1 -- 172.21.15.44:0/918573921 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c0810d9b0 con 0x55c1142566b0 2023-11-16T16:52:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.452+0000 7f5c077fe700 1 --2- 172.21.15.44:0/918573921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1142566b0 0x55c114406a50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8096450 tx=0x7f5bf809a0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abeb55671ebd224f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.452+0000 7f5be6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:52:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.452+0000 7f5be6ffd700 1 -- 172.21.15.44:0/918573921 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf80a23f0 con 0x55c1142566b0 2023-11-16T16:52:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.452+0000 7f5be6ffd700 1 -- 172.21.15.44:0/918573921 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f5bf80a1f60 con 0x55c1142566b0 2023-11-16T16:52:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.452+0000 7f5c21829b80 1 -- 172.21.15.44:0/918573921 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c1143ed710 con 0x55c1142566b0 2023-11-16T16:52:53.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.452+0000 7f5be6ffd700 1 -- 172.21.15.44:0/918573921 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5bf80a2120 con 0x55c1142566b0 2023-11-16T16:52:53.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.452+0000 7f5c21829b80 10 client.26258 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:53.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.452+0000 7f5c21829b80 1 -- 172.21.15.44:0/918573921 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55c114312130 con 0x55c1142566b0 2023-11-16T16:52:53.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.453+0000 7f5be6ffd700 1 -- 172.21.15.44:0/918573921 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c1143ed710 con 0x55c1142566b0 2023-11-16T16:52:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.453+0000 7f5be6ffd700 1 -- 172.21.15.44:0/918573921 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5bf8098f90 con 0x55c1142566b0 2023-11-16T16:52:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.453+0000 7f5c21829b80 10 client.26258 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.453+0000 7f5c21829b80 10 client.26258 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.453+0000 7f5c21829b80 20 client.26258 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.453+0000 7f5c21829b80 1 --2- 172.21.15.44:0/918573921 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c114413fa0 0x55c1144163b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.453+0000 7f5c21829b80 4 client.26258 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.453+0000 7f5c21829b80 1 -- 172.21.15.44:0/918573921 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c11430e040 con 0x55c114413fa0 2023-11-16T16:52:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.453+0000 7f5c06ffd700 1 --2- 172.21.15.44:0/918573921 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c114413fa0 0x55c1144163b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.454+0000 7f5c06ffd700 1 --2- 172.21.15.44:0/918573921 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c114413fa0 0x55c1144163b0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b006cbaa5c31e089 server_cookie=dda8b53767833c66 in_seq=0 out_seq=0 2023-11-16T16:52:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.454+0000 7f5be6ffd700 10 client.26258 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.454+0000 7f5be6ffd700 1 -- 172.21.15.44:0/918573921 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55c11430e040 con 0x55c114413fa0 2023-11-16T16:52:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.454+0000 7f5be6ffd700 10 client.26258 handle_command_reply: tid=0 2023-11-16T16:52:53.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.454+0000 7f5c21829b80 1 client.26258 shutdown 2023-11-16T16:52:53.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.455+0000 7f5c21829b80 1 -- 172.21.15.44:0/918573921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1142566b0 msgr2=0x55c114406a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.455+0000 7f5c21829b80 1 --2- 172.21.15.44:0/918573921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1142566b0 0x55c114406a50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8096450 tx=0x7f5bf809a0e0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.455+0000 7f5c21829b80 1 -- 172.21.15.44:0/918573921 shutdown_connections 2023-11-16T16:52:53.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.455+0000 7f5c21829b80 1 --2- 172.21.15.44:0/918573921 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c114413fa0 0x55c1144163b0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.455+0000 7f5c21829b80 1 --2- 172.21.15.44:0/918573921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c114400b90 0x55c114404250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.455+0000 7f5be6ffd700 0 client.26258 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:53.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.455+0000 7f5c21829b80 1 --2- 172.21.15.44:0/918573921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c1142566b0 0x55c114406a50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.455+0000 7f5c21829b80 1 --2- 172.21.15.44:0/918573921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c114400720 0x55c114403c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.455+0000 7f5c21829b80 1 -- 172.21.15.44:0/918573921 >> 172.21.15.44:0/918573921 conn(0x55c1143eff50 msgr2=0x55c1143f47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:53.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.455+0000 7f5c21829b80 1 -- 172.21.15.44:0/918573921 shutdown_connections 2023-11-16T16:52:53.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.455+0000 7f5c21829b80 1 -- 172.21.15.44:0/918573921 wait complete. 2023-11-16T16:52:53.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.455+0000 7f5c21829b80 20 client.26258 trim_cache size 0 max 16384 2023-11-16T16:52:53.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.471+0000 7f5c21829b80 1 Processor -- start 2023-11-16T16:52:53.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.471+0000 7f5c21829b80 1 -- start start 2023-11-16T16:52:53.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.471+0000 7f5c21829b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1142566b0 0x55c1144016a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.471+0000 7f5c21829b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c114400720 0x55c11440b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.472+0000 7f5c21829b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400b90 0x55c1143f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.472+0000 7f5c21829b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c114404830 con 0x55c114400720 2023-11-16T16:52:53.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.472+0000 7f5c21829b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1144021e0 con 0x55c114400b90 2023-11-16T16:52:53.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.472+0000 7f5c21829b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1143f6a00 con 0x55c1142566b0 2023-11-16T16:52:53.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.472+0000 7f5c06ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c114400720 0x55c11440b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.472+0000 7f5c07fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400b90 0x55c1143f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.472+0000 7f5c077fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1142566b0 0x55c1144016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.472+0000 7f5c07fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400b90 0x55c1143f4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:42652/0 (socket says 172.21.15.44:42652) 2023-11-16T16:52:53.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.472+0000 7f5c06ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c114400720 0x55c11440b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57158/0 (socket says 172.21.15.44:57158) 2023-11-16T16:52:53.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.472+0000 7f5c07fff700 1 -- 172.21.15.44:0/2106910765 learned_addr learned my addr 172.21.15.44:0/2106910765 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:53.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.472+0000 7f5c077fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1142566b0 0x55c1144016a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:53854/0 (socket says 172.21.15.44:53854) 2023-11-16T16:52:53.476 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.472+0000 7f5c07fff700 1 -- 172.21.15.44:0/2106910765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1142566b0 msgr2=0x55c1144016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:53.476 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.472+0000 7f5c07fff700 1 --2- 172.21.15.44:0/2106910765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1142566b0 0x55c1144016a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.476 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.472+0000 7f5c07fff700 1 -- 172.21.15.44:0/2106910765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c114400720 msgr2=0x55c11440b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:53.476 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.472+0000 7f5c07fff700 1 --2- 172.21.15.44:0/2106910765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c114400720 0x55c11440b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.476 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.472+0000 7f5c07fff700 1 -- 172.21.15.44:0/2106910765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c114311e80 con 0x55c114400b90 2023-11-16T16:52:53.476 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.473+0000 7f5c07fff700 1 --2- 172.21.15.44:0/2106910765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400b90 0x55c1143f4f60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5c00003de0 tx=0x7f5c00003e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d1dfe0ea3ae3164 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:53.476 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.473+0000 7f5bee7fc700 1 -- 172.21.15.44:0/2106910765 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0000e040 con 0x55c114400b90 2023-11-16T16:52:53.476 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.473+0000 7f5bee7fc700 1 -- 172.21.15.44:0/2106910765 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5c00003bc0 con 0x55c114400b90 2023-11-16T16:52:53.476 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.473+0000 7f5bee7fc700 1 -- 172.21.15.44:0/2106910765 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c000054e0 con 0x55c114400b90 2023-11-16T16:52:53.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.474+0000 7f5c21829b80 1 -- 172.21.15.44:0/2106910765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400b90 msgr2=0x55c1143f4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.474+0000 7f5c21829b80 1 --2- 172.21.15.44:0/2106910765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400b90 0x55c1143f4f60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5c00003de0 tx=0x7f5c00003e10 comp rx=0 tx=0).stop 2023-11-16T16:52:53.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.474+0000 7f5c21829b80 1 -- 172.21.15.44:0/2106910765 shutdown_connections 2023-11-16T16:52:53.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.474+0000 7f5c21829b80 1 --2- 172.21.15.44:0/2106910765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400b90 0x55c1143f4f60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.474+0000 7f5c21829b80 1 --2- 172.21.15.44:0/2106910765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c1142566b0 0x55c1144016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.474+0000 7f5c21829b80 1 --2- 172.21.15.44:0/2106910765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c114400720 0x55c11440b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.474+0000 7f5c21829b80 1 -- 172.21.15.44:0/2106910765 >> 172.21.15.44:0/2106910765 conn(0x55c1143eff50 msgr2=0x55c114400f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:53.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.474+0000 7f5c21829b80 1 -- 172.21.15.44:0/2106910765 shutdown_connections 2023-11-16T16:52:53.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.474+0000 7f5c21829b80 1 -- 172.21.15.44:0/2106910765 wait complete. 2023-11-16T16:52:53.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.474+0000 7f5c21829b80 1 -- start start 2023-11-16T16:52:53.477 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.474+0000 7f5c21829b80 1 Processor -- start 2023-11-16T16:52:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.475+0000 7f5c21829b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1142566b0 0x55c11440b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.475+0000 7f5c21829b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400720 0x55c1143fa330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.475+0000 7f5c21829b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c114400b90 0x55c1143fa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.475+0000 7f5c21829b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1143faf70 con 0x55c1142566b0 2023-11-16T16:52:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.475+0000 7f5c21829b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1143fbc60 con 0x55c114400720 2023-11-16T16:52:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.475+0000 7f5c21829b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1143fbe20 con 0x55c114400b90 2023-11-16T16:52:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.475+0000 7f5c21829b80 10 client.0 _ll_register_callbacks cb 0x7f5bf80ca5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.475+0000 7f5c077fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1142566b0 0x55c11440b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.475+0000 7f5c06ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400720 0x55c1143fa330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.475+0000 7f5c06ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400720 0x55c1143fa330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:42656/0 (socket says 172.21.15.44:42656) 2023-11-16T16:52:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.475+0000 7f5c06ffd700 1 -- 172.21.15.44:0/289419981 learned_addr learned my addr 172.21.15.44:0/289419981 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.475+0000 7f5c07fff700 1 --2- 172.21.15.44:0/289419981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c114400b90 0x55c1143fa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.476+0000 7f5c06ffd700 1 -- 172.21.15.44:0/289419981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c114400b90 msgr2=0x55c1143fa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.476+0000 7f5c06ffd700 1 --2- 172.21.15.44:0/289419981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c114400b90 0x55c1143fa870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.476+0000 7f5c06ffd700 1 -- 172.21.15.44:0/289419981 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1142566b0 msgr2=0x55c11440b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.476+0000 7f5c06ffd700 1 --2- 172.21.15.44:0/289419981 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1142566b0 0x55c11440b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.476+0000 7f5c06ffd700 1 -- 172.21.15.44:0/289419981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c1141cd5d0 con 0x55c114400720 2023-11-16T16:52:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.476+0000 7f5c06ffd700 1 --2- 172.21.15.44:0/289419981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400720 0x55c1143fa330 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc005930 tx=0x7f5bfc0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37f93897b600e0a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.476+0000 7f5be6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:52:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.476+0000 7f5be6ffd700 1 -- 172.21.15.44:0/289419981 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc022040 con 0x55c114400720 2023-11-16T16:52:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.476+0000 7f5be6ffd700 1 -- 172.21.15.44:0/289419981 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f5bfc003880 con 0x55c114400720 2023-11-16T16:52:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.476+0000 7f5c21829b80 1 -- 172.21.15.44:0/289419981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c00007d00 con 0x55c114400720 2023-11-16T16:52:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.477+0000 7f5be6ffd700 1 -- 172.21.15.44:0/289419981 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5bfc02be50 con 0x55c114400720 2023-11-16T16:52:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.477+0000 7f5c21829b80 10 client.26264 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.477+0000 7f5c21829b80 1 -- 172.21.15.44:0/289419981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55c114162c00 con 0x55c114400720 2023-11-16T16:52:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.477+0000 7f5be6ffd700 1 -- 172.21.15.44:0/289419981 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5c00007d00 con 0x55c114400720 2023-11-16T16:52:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.477+0000 7f5be6ffd700 1 -- 172.21.15.44:0/289419981 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5bfc0289b0 con 0x55c114400720 2023-11-16T16:52:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.477+0000 7f5c21829b80 10 client.26264 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.477+0000 7f5c21829b80 10 client.26264 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.477+0000 7f5c21829b80 20 client.26264 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.477+0000 7f5c21829b80 1 --2- 172.21.15.44:0/289419981 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c114468580 0x55c11446a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.477+0000 7f5c21829b80 4 client.26264 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:52:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.477+0000 7f5c21829b80 1 -- 172.21.15.44:0/289419981 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c11431a600 con 0x55c114468580 2023-11-16T16:52:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.478+0000 7f5c077fe700 1 --2- 172.21.15.44:0/289419981 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c114468580 0x55c11446a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.478+0000 7f5c077fe700 1 --2- 172.21.15.44:0/289419981 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c114468580 0x55c11446a990 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee9fbd9d3837ed49 server_cookie=238260bb7b30906c in_seq=0 out_seq=0 2023-11-16T16:52:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.478+0000 7f5be6ffd700 10 client.26264 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:53.500 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.498+0000 7f5be6ffd700 1 -- 172.21.15.44:0/289419981 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c11431a600 con 0x55c114468580 2023-11-16T16:52:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.498+0000 7f5be6ffd700 10 client.26264 handle_command_reply: tid=0 2023-11-16T16:52:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.498+0000 7f5c21829b80 1 client.26264 shutdown 2023-11-16T16:52:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.499+0000 7f5c21829b80 1 -- 172.21.15.44:0/289419981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400720 msgr2=0x55c1143fa330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.499+0000 7f5c21829b80 1 --2- 172.21.15.44:0/289419981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400720 0x55c1143fa330 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc005930 tx=0x7f5bfc0047c0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.499+0000 7f5c21829b80 1 -- 172.21.15.44:0/289419981 shutdown_connections 2023-11-16T16:52:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.499+0000 7f5c21829b80 1 --2- 172.21.15.44:0/289419981 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c114468580 0x55c11446a990 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.499+0000 7f5c21829b80 1 --2- 172.21.15.44:0/289419981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c114400b90 0x55c1143fa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.499+0000 7f5c21829b80 1 --2- 172.21.15.44:0/289419981 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1142566b0 0x55c11440b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.499+0000 7f5be6ffd700 0 client.26264 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.499+0000 7f5c21829b80 1 --2- 172.21.15.44:0/289419981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c114400720 0x55c1143fa330 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.502 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.499+0000 7f5c21829b80 1 -- 172.21.15.44:0/289419981 >> 172.21.15.44:0/289419981 conn(0x55c1143eff50 msgr2=0x55c1144017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:53.502 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.499+0000 7f5c21829b80 1 -- 172.21.15.44:0/289419981 shutdown_connections 2023-11-16T16:52:53.502 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.499+0000 7f5c21829b80 1 -- 172.21.15.44:0/289419981 wait complete. 2023-11-16T16:52:53.502 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.499+0000 7f5c21829b80 20 client.26264 trim_cache size 0 max 16384 2023-11-16T16:52:53.502 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:52:53.502 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:52:53.502 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:52:53.502 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:52:53.503 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.500+0000 7f5c0dfff700 1 -- 172.21.15.44:0/76628241 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f5bf00a12c0 msgr2=0x7f5bf00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.503 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.500+0000 7f5c0dfff700 1 --2- 172.21.15.44:0/76628241 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f5bf00a12c0 0x7f5bf00a3770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5c0810cfd0 tx=0x7f5bfc009310 comp rx=0 tx=0).stop 2023-11-16T16:52:53.503 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.500+0000 7f5c0dfff700 1 -- 172.21.15.44:0/76628241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c08100c50 msgr2=0x7f5c0819e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.503 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.500+0000 7f5c0dfff700 1 --2- 172.21.15.44:0/76628241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c08100c50 0x7f5c0819e520 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5bf800e9a0 tx=0x7f5bf800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.503 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.501+0000 7f5c0dfff700 1 -- 172.21.15.44:0/76628241 shutdown_connections 2023-11-16T16:52:53.503 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.501+0000 7f5c0dfff700 1 --2- 172.21.15.44:0/76628241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c08105560 0x7f5c08198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.503 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.501+0000 7f5c0dfff700 1 --2- 172.21.15.44:0/76628241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c08100c50 0x7f5c0819e520 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.501+0000 7f5c0dfff700 1 --2- 172.21.15.44:0/76628241 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f5bf00a12c0 0x7f5bf00a3770 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.501+0000 7f5c0dfff700 1 --2- 172.21.15.44:0/76628241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c081015f0 0x7f5c0819ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.501+0000 7f5c0dfff700 1 -- 172.21.15.44:0/76628241 >> 172.21.15.44:0/76628241 conn(0x7f5c08077550 msgr2=0x7f5c08103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:53.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.501+0000 7f5c0dfff700 1 -- 172.21.15.44:0/76628241 shutdown_connections 2023-11-16T16:52:53.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:52:53.501+0000 7f5c0dfff700 1 -- 172.21.15.44:0/76628241 wait complete. 2023-11-16T16:52:53.685 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.683+0000 7f61735ff700 1 -- 172.21.15.44:0/1941454846 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f616c067a60 con 0x7f616c101dd0 2023-11-16T16:52:53.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.684+0000 7f61627fc700 1 -- 172.21.15.44:0/1941454846 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f615c099c70 con 0x7f616c101dd0 2023-11-16T16:52:53.686 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:52:53.686 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:52:53.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.687+0000 7f61735ff700 1 -- 172.21.15.44:0/1941454846 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f61580a1340 msgr2=0x7f61580a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.687+0000 7f61735ff700 1 --2- 172.21.15.44:0/1941454846 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f61580a1340 0x7f61580a37f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6168005fd0 tx=0x7f6168009310 comp rx=0 tx=0).stop 2023-11-16T16:52:53.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.687+0000 7f61735ff700 1 -- 172.21.15.44:0/1941454846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f616c101dd0 msgr2=0x7f616c198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.687+0000 7f61735ff700 1 --2- 172.21.15.44:0/1941454846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f616c101dd0 0x7f616c198b70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f615c00e9a0 tx=0x7f615c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.687+0000 7f61735ff700 1 -- 172.21.15.44:0/1941454846 shutdown_connections 2023-11-16T16:52:53.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.687+0000 7f61735ff700 1 --2- 172.21.15.44:0/1941454846 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f61580a1340 0x7f61580a37f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.687+0000 7f61735ff700 1 --2- 172.21.15.44:0/1941454846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f616c10a800 0x7f616c19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.687+0000 7f61735ff700 1 --2- 172.21.15.44:0/1941454846 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f616c1014c0 0x7f616c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.687+0000 7f61735ff700 1 --2- 172.21.15.44:0/1941454846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f616c101dd0 0x7f616c198b70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.687+0000 7f61735ff700 1 -- 172.21.15.44:0/1941454846 >> 172.21.15.44:0/1941454846 conn(0x7f616c076310 msgr2=0x7f616c10abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:53.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.687+0000 7f61735ff700 1 -- 172.21.15.44:0/1941454846 shutdown_connections 2023-11-16T16:52:53.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.687+0000 7f61735ff700 1 -- 172.21.15.44:0/1941454846 wait complete. 2023-11-16T16:52:53.690 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:52:53.712 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T16:52:53.717 INFO:tasks.workunit.client.0.smithi044.stdout:ok 2023-11-16T16:52:53.725 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T16:52:53.725 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T16:52:53.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:53 smithi077 ceph-mon[131204]: pgmap v91: 129 pgs: 129 active+clean; 3.3 MiB data, 90 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 42 KiB/s wr, 12 op/s 2023-11-16T16:52:53.850 INFO:tasks.workunit:Running workunit fs/misc/dirfrag.sh... 2023-11-16T16:52:53.850 DEBUG:teuthology.orchestra.run.smithi044:workunit test fs/misc/dirfrag.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dirfrag.sh 2023-11-16T16:52:53.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:53 smithi176 ceph-mon[131630]: pgmap v91: 129 pgs: 129 active+clean; 3.3 MiB data, 90 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 42 KiB/s wr, 12 op/s 2023-11-16T16:52:53.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.930+0000 7fc5ee9ff700 1 -- 172.21.15.44:0/1514994791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5e8101660 msgr2=0x7fc5e8104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.930+0000 7fc5ee9ff700 1 --2- 172.21.15.44:0/1514994791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5e8101660 0x7fc5e8104ef0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0099c0 tx=0x7fc5dc009cd0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.930+0000 7fc5ee9ff700 1 -- 172.21.15.44:0/1514994791 shutdown_connections 2023-11-16T16:52:53.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.930+0000 7fc5ee9ff700 1 --2- 172.21.15.44:0/1514994791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5e8105620 0x7fc5e810d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.930+0000 7fc5ee9ff700 1 --2- 172.21.15.44:0/1514994791 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5e8100cc0 0x7fc5e8101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.930+0000 7fc5ee9ff700 1 --2- 172.21.15.44:0/1514994791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5e8101660 0x7fc5e8104ef0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.930+0000 7fc5ee9ff700 1 -- 172.21.15.44:0/1514994791 >> 172.21.15.44:0/1514994791 conn(0x7fc5e8077560 msgr2=0x7fc5e8077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:53.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.930+0000 7fc5ee9ff700 1 -- 172.21.15.44:0/1514994791 shutdown_connections 2023-11-16T16:52:53.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.930+0000 7fc5ee9ff700 1 -- 172.21.15.44:0/1514994791 wait complete. 2023-11-16T16:52:53.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.932+0000 7fc5ee9ff700 1 Processor -- start 2023-11-16T16:52:53.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.932+0000 7fc5ee9ff700 1 -- start start 2023-11-16T16:52:53.934 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.932+0000 7fc5ee9ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5e8100cc0 0x7fc5e8072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.932+0000 7fc5ee9ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5e8101660 0x7fc5e8073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.932+0000 7fc5ee9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5e8105620 0x7fc5e81a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.932+0000 7fc5ee9ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e8073b60 con 0x7fc5e8100cc0 2023-11-16T16:52:53.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.932+0000 7fc5ee9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e81a4b70 con 0x7fc5e8105620 2023-11-16T16:52:53.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.932+0000 7fc5ee9ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e81a4cb0 con 0x7fc5e8101660 2023-11-16T16:52:53.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.933+0000 7fc5e7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5e8100cc0 0x7fc5e8072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.933+0000 7fc5e7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5e8100cc0 0x7fc5e8072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57250/0 (socket says 172.21.15.44:57250) 2023-11-16T16:52:53.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.933+0000 7fc5e7fff700 1 -- 172.21.15.44:0/1810913249 learned_addr learned my addr 172.21.15.44:0/1810913249 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:53.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.933+0000 7fc5ecf9c700 1 --2- 172.21.15.44:0/1810913249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5e8105620 0x7fc5e81a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.933+0000 7fc5e77fe700 1 --2- 172.21.15.44:0/1810913249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5e8101660 0x7fc5e8073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.933+0000 7fc5e7fff700 1 -- 172.21.15.44:0/1810913249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5e8101660 msgr2=0x7fc5e8073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.933+0000 7fc5e7fff700 1 --2- 172.21.15.44:0/1810913249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5e8101660 0x7fc5e8073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.933+0000 7fc5e7fff700 1 -- 172.21.15.44:0/1810913249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5e8105620 msgr2=0x7fc5e81a4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.933+0000 7fc5e7fff700 1 --2- 172.21.15.44:0/1810913249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5e8105620 0x7fc5e81a4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.933+0000 7fc5e7fff700 1 -- 172.21.15.44:0/1810913249 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5dc0095f0 con 0x7fc5e8100cc0 2023-11-16T16:52:53.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.933+0000 7fc5e7fff700 1 --2- 172.21.15.44:0/1810913249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5e8100cc0 0x7fc5e8072df0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800e9a0 tx=0x7fc5d800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:53.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.934+0000 7fc5e57fa700 1 -- 172.21.15.44:0/1810913249 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8009790 con 0x7fc5e8100cc0 2023-11-16T16:52:53.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.934+0000 7fc5e57fa700 1 -- 172.21.15.44:0/1810913249 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5d8010430 con 0x7fc5e8100cc0 2023-11-16T16:52:53.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.934+0000 7fc5ee9ff700 1 -- 172.21.15.44:0/1810913249 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e8110320 con 0x7fc5e8100cc0 2023-11-16T16:52:53.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.934+0000 7fc5e57fa700 1 -- 172.21.15.44:0/1810913249 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8018680 con 0x7fc5e8100cc0 2023-11-16T16:52:53.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.934+0000 7fc5ee9ff700 1 -- 172.21.15.44:0/1810913249 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e8110190 con 0x7fc5e8100cc0 2023-11-16T16:52:53.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.937+0000 7fc5e57fa700 1 -- 172.21.15.44:0/1810913249 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fc5d80105a0 con 0x7fc5e8100cc0 2023-11-16T16:52:53.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.938+0000 7fc60231eb80 1 Processor -- start 2023-11-16T16:52:53.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.938+0000 7fc5e57fa700 1 --2- 172.21.15.44:0/1810913249 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc5d00a12c0 0x7fc5d00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.938+0000 7fc60231eb80 1 -- start start 2023-11-16T16:52:53.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.939+0000 7fc5e57fa700 1 -- 172.21.15.44:0/1810913249 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fc5d8014070 con 0x7fc5e8100cc0 2023-11-16T16:52:53.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.939+0000 7fc60231eb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15ab60 0x55e0aa15af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.939+0000 7fc5ecf9c700 1 --2- 172.21.15.44:0/1810913249 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc5d00a12c0 0x7fc5d00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.939+0000 7fc60231eb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15b470 0x55e0aa15f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.939+0000 7fc60231eb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15f8b0 0x55e0aa15fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.939+0000 7fc60231eb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0aa060060 con 0x55e0aa15b470 2023-11-16T16:52:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.939+0000 7fc60231eb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0aa0b28a0 con 0x55e0aa15ab60 2023-11-16T16:52:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.939+0000 7fc60231eb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0aa0abd30 con 0x55e0aa15f8b0 2023-11-16T16:52:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.939+0000 7fc5e7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15ab60 0x55e0aa15af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.939+0000 7fc5e7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15ab60 0x55e0aa15af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:42698/0 (socket says 172.21.15.44:42698) 2023-11-16T16:52:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.939+0000 7fc5e7fff700 1 -- 172.21.15.44:0/801624722 learned_addr learned my addr 172.21.15.44:0/801624722 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.939+0000 7fc5e77fe700 1 --2- 172.21.15.44:0/801624722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15b470 0x55e0aa15f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.940+0000 7fc5ecf9c700 1 --2- 172.21.15.44:0/801624722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15f8b0 0x55e0aa15fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.940+0000 7fc5e7fff700 1 -- 172.21.15.44:0/801624722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15f8b0 msgr2=0x55e0aa15fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.940+0000 7fc5e7fff700 1 --2- 172.21.15.44:0/801624722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15f8b0 0x55e0aa15fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.940+0000 7fc5e7fff700 1 -- 172.21.15.44:0/801624722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15b470 msgr2=0x55e0aa15f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.940+0000 7fc5e7fff700 1 --2- 172.21.15.44:0/801624722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15b470 0x55e0aa15f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.940+0000 7fc5e7fff700 1 -- 172.21.15.44:0/801624722 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e0a9f2d630 con 0x55e0aa15ab60 2023-11-16T16:52:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.940+0000 7fc5e7fff700 1 --2- 172.21.15.44:0/801624722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15ab60 0x55e0aa15af30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8096a20 tx=0x7fc5d8099c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feda9916078a9b8f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:53.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.940+0000 7fc5ecf9c700 1 --2- 172.21.15.44:0/1810913249 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc5d00a12c0 0x7fc5d00a3770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8074f00 tx=0x7fc5e000fc00 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:53.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.940+0000 7fc5ce7fc700 1 -- 172.21.15.44:0/801624722 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8010880 con 0x55e0aa15ab60 2023-11-16T16:52:53.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.940+0000 7fc5ce7fc700 1 -- 172.21.15.44:0/801624722 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5d80109e0 con 0x55e0aa15ab60 2023-11-16T16:52:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.940+0000 7fc5ce7fc700 1 -- 172.21.15.44:0/801624722 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d80a1f70 con 0x55e0aa15ab60 2023-11-16T16:52:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.941+0000 7fc60231eb80 1 -- 172.21.15.44:0/801624722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15ab60 msgr2=0x55e0aa15af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.941+0000 7fc60231eb80 1 --2- 172.21.15.44:0/801624722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15ab60 0x55e0aa15af30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8096a20 tx=0x7fc5d8099c20 comp rx=0 tx=0).stop 2023-11-16T16:52:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.941+0000 7fc60231eb80 1 -- 172.21.15.44:0/801624722 shutdown_connections 2023-11-16T16:52:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.941+0000 7fc60231eb80 1 --2- 172.21.15.44:0/801624722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15f8b0 0x55e0aa15fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.941+0000 7fc60231eb80 1 --2- 172.21.15.44:0/801624722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15ab60 0x55e0aa15af30 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.941+0000 7fc60231eb80 1 --2- 172.21.15.44:0/801624722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15b470 0x55e0aa15f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.941+0000 7fc60231eb80 1 -- 172.21.15.44:0/801624722 >> 172.21.15.44:0/801624722 conn(0x55e0aa150360 msgr2=0x55e0aa155b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.941+0000 7fc60231eb80 1 -- 172.21.15.44:0/801624722 shutdown_connections 2023-11-16T16:52:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.941+0000 7fc60231eb80 1 -- 172.21.15.44:0/801624722 wait complete. 2023-11-16T16:52:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.942+0000 7fc60231eb80 1 -- start start 2023-11-16T16:52:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.942+0000 7fc60231eb80 1 Processor -- start 2023-11-16T16:52:53.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.943+0000 7fc60231eb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15ab60 0x55e0aa167060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.943+0000 7fc60231eb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15b470 0x55e0aa164350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.943+0000 7fc60231eb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15f8b0 0x55e0aa164a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.943+0000 7fc60231eb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0aa04d8e0 con 0x55e0aa15ab60 2023-11-16T16:52:53.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.943+0000 7fc60231eb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0aa04d580 con 0x55e0aa15f8b0 2023-11-16T16:52:53.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.943+0000 7fc60231eb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0a9f2f920 con 0x55e0aa15b470 2023-11-16T16:52:53.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.943+0000 7fc60231eb80 10 client.0 _ll_register_callbacks cb 0x55e0aa06b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:53.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.943+0000 7fc5e7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15ab60 0x55e0aa167060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.943+0000 7fc5ecf9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15f8b0 0x55e0aa164a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.943+0000 7fc5e7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15ab60 0x55e0aa167060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57274/0 (socket says 172.21.15.44:57274) 2023-11-16T16:52:53.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.943+0000 7fc5e7fff700 1 -- 172.21.15.44:0/3940854330 learned_addr learned my addr 172.21.15.44:0/3940854330 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:53.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.943+0000 7fc5e77fe700 1 --2- 172.21.15.44:0/3940854330 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15b470 0x55e0aa164350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.944+0000 7fc5ecf9c700 1 -- 172.21.15.44:0/3940854330 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15b470 msgr2=0x55e0aa164350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:53.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.944+0000 7fc5ecf9c700 1 --2- 172.21.15.44:0/3940854330 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15b470 0x55e0aa164350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.944+0000 7fc5ecf9c700 1 -- 172.21.15.44:0/3940854330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15ab60 msgr2=0x55e0aa167060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:53.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.944+0000 7fc5ecf9c700 1 --2- 172.21.15.44:0/3940854330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15ab60 0x55e0aa167060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.944+0000 7fc5ecf9c700 1 -- 172.21.15.44:0/3940854330 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e8110190 con 0x55e0aa15f8b0 2023-11-16T16:52:53.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.944+0000 7fc5e7fff700 1 --2- 172.21.15.44:0/3940854330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15ab60 0x55e0aa167060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:53.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.944+0000 7fc5ecf9c700 1 --2- 172.21.15.44:0/3940854330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15f8b0 0x55e0aa164a90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0018b50 tx=0x7fc5e0004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba1b002e1ce8a508 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:53.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.944+0000 7fc5c77fe700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:52:53.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.944+0000 7fc5c77fe700 1 -- 172.21.15.44:0/3940854330 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e000e040 con 0x55e0aa15f8b0 2023-11-16T16:52:53.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.944+0000 7fc60231eb80 1 -- 172.21.15.44:0/3940854330 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e0aa14db20 con 0x55e0aa15f8b0 2023-11-16T16:52:53.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.944+0000 7fc5c77fe700 1 -- 172.21.15.44:0/3940854330 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fc5e00093f0 con 0x55e0aa15f8b0 2023-11-16T16:52:53.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.945+0000 7fc5c77fe700 1 -- 172.21.15.44:0/3940854330 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5e0008920 con 0x55e0aa15f8b0 2023-11-16T16:52:53.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.945+0000 7fc60231eb80 10 client.26294 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:53.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.945+0000 7fc60231eb80 1 -- 172.21.15.44:0/3940854330 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55e0aa072190 con 0x55e0aa15f8b0 2023-11-16T16:52:53.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.945+0000 7fc5c77fe700 1 -- 172.21.15.44:0/3940854330 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e0aa14db20 con 0x55e0aa15f8b0 2023-11-16T16:52:53.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.945+0000 7fc5c77fe700 1 -- 172.21.15.44:0/3940854330 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc5e0024b20 con 0x55e0aa15f8b0 2023-11-16T16:52:53.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.945+0000 7fc60231eb80 10 client.26294 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:53.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.945+0000 7fc60231eb80 10 client.26294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:53.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.945+0000 7fc60231eb80 20 client.26294 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:53.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.945+0000 7fc60231eb80 1 --2- 172.21.15.44:0/3940854330 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e0aa174920 0x55e0aa176d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.945+0000 7fc60231eb80 4 client.26294 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:53.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.945+0000 7fc60231eb80 1 -- 172.21.15.44:0/3940854330 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e0aa06e0a0 con 0x55e0aa174920 2023-11-16T16:52:53.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.946+0000 7fc5e7fff700 1 --2- 172.21.15.44:0/3940854330 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e0aa174920 0x55e0aa176d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.946+0000 7fc5e7fff700 1 --2- 172.21.15.44:0/3940854330 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e0aa174920 0x55e0aa176d30 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a4a48691c87479e server_cookie=a416e2aaa49d7a4a in_seq=0 out_seq=0 2023-11-16T16:52:53.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.946+0000 7fc5c77fe700 10 client.26294 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:53.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.947+0000 7fc5c77fe700 1 -- 172.21.15.44:0/3940854330 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e0aa06e0a0 con 0x55e0aa174920 2023-11-16T16:52:53.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.947+0000 7fc5c77fe700 10 client.26294 handle_command_reply: tid=0 2023-11-16T16:52:53.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.947+0000 7fc60231eb80 1 client.26294 shutdown 2023-11-16T16:52:53.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.947+0000 7fc60231eb80 1 -- 172.21.15.44:0/3940854330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15f8b0 msgr2=0x55e0aa164a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.947+0000 7fc60231eb80 1 --2- 172.21.15.44:0/3940854330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15f8b0 0x55e0aa164a90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0018b50 tx=0x7fc5e0004b40 comp rx=0 tx=0).stop 2023-11-16T16:52:53.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.947+0000 7fc60231eb80 1 -- 172.21.15.44:0/3940854330 shutdown_connections 2023-11-16T16:52:53.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.947+0000 7fc60231eb80 1 --2- 172.21.15.44:0/3940854330 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e0aa174920 0x55e0aa176d30 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.947+0000 7fc60231eb80 1 --2- 172.21.15.44:0/3940854330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15f8b0 0x55e0aa164a90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.947+0000 7fc60231eb80 1 --2- 172.21.15.44:0/3940854330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15ab60 0x55e0aa167060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.947+0000 7fc60231eb80 1 --2- 172.21.15.44:0/3940854330 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15b470 0x55e0aa164350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.947+0000 7fc5c77fe700 0 client.26294 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:53.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.947+0000 7fc60231eb80 1 -- 172.21.15.44:0/3940854330 >> 172.21.15.44:0/3940854330 conn(0x55e0aa150360 msgr2=0x55e0aa154de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:53.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.947+0000 7fc60231eb80 1 -- 172.21.15.44:0/3940854330 shutdown_connections 2023-11-16T16:52:53.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.947+0000 7fc60231eb80 1 -- 172.21.15.44:0/3940854330 wait complete. 2023-11-16T16:52:53.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.948+0000 7fc60231eb80 20 client.26294 trim_cache size 0 max 16384 2023-11-16T16:52:53.953 INFO:tasks.workunit.client.0.smithi044.stdout:creating folder hierarchy 2023-11-16T16:52:53.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.962+0000 7fc60231eb80 1 Processor -- start 2023-11-16T16:52:53.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.962+0000 7fc60231eb80 1 -- start start 2023-11-16T16:52:53.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.962+0000 7fc60231eb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa150360 0x55e0aa15e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.962+0000 7fc60231eb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15ab60 0x55e0aa155560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.962+0000 7fc60231eb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15b470 0x55e0aa155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.962+0000 7fc60231eb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0aa1656a0 con 0x55e0aa150360 2023-11-16T16:52:53.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.962+0000 7fc60231eb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0aa14fee0 con 0x55e0aa15b470 2023-11-16T16:52:53.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.962+0000 7fc60231eb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0aa15f360 con 0x55e0aa15ab60 2023-11-16T16:52:53.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.963+0000 7fc5e7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa150360 0x55e0aa15e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.963+0000 7fc5e7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa150360 0x55e0aa15e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57288/0 (socket says 172.21.15.44:57288) 2023-11-16T16:52:53.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.963+0000 7fc5e7fff700 1 -- 172.21.15.44:0/3074516381 learned_addr learned my addr 172.21.15.44:0/3074516381 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:53.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.963+0000 7fc5ecf9c700 1 --2- 172.21.15.44:0/3074516381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15b470 0x55e0aa155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.963+0000 7fc5e77fe700 1 --2- 172.21.15.44:0/3074516381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15ab60 0x55e0aa155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.963+0000 7fc5ecf9c700 1 -- 172.21.15.44:0/3074516381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15ab60 msgr2=0x55e0aa155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:53.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.963+0000 7fc5ecf9c700 1 --2- 172.21.15.44:0/3074516381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15ab60 0x55e0aa155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.963+0000 7fc5ecf9c700 1 -- 172.21.15.44:0/3074516381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa150360 msgr2=0x55e0aa15e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:53.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.963+0000 7fc5ecf9c700 1 --2- 172.21.15.44:0/3074516381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa150360 0x55e0aa15e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.963+0000 7fc5ecf9c700 1 -- 172.21.15.44:0/3074516381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e0aa071ee0 con 0x55e0aa15b470 2023-11-16T16:52:53.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.963+0000 7fc5ecf9c700 1 --2- 172.21.15.44:0/3074516381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15b470 0x55e0aa155aa0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0024fd0 tx=0x7fc5e001a440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ded6c36c502e441 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:53.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.963+0000 7fc5ceffd700 1 -- 172.21.15.44:0/3074516381 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e000e030 con 0x55e0aa15b470 2023-11-16T16:52:53.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.963+0000 7fc5ceffd700 1 -- 172.21.15.44:0/3074516381 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5e000f040 con 0x55e0aa15b470 2023-11-16T16:52:53.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.963+0000 7fc5ceffd700 1 -- 172.21.15.44:0/3074516381 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e001ac30 con 0x55e0aa15b470 2023-11-16T16:52:53.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.964+0000 7fc60231eb80 1 -- 172.21.15.44:0/3074516381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15b470 msgr2=0x55e0aa155aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.964+0000 7fc60231eb80 1 --2- 172.21.15.44:0/3074516381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15b470 0x55e0aa155aa0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0024fd0 tx=0x7fc5e001a440 comp rx=0 tx=0).stop 2023-11-16T16:52:53.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.964+0000 7fc60231eb80 1 -- 172.21.15.44:0/3074516381 shutdown_connections 2023-11-16T16:52:53.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.964+0000 7fc60231eb80 1 --2- 172.21.15.44:0/3074516381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa15b470 0x55e0aa155aa0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.964+0000 7fc60231eb80 1 --2- 172.21.15.44:0/3074516381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa150360 0x55e0aa15e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.964+0000 7fc60231eb80 1 --2- 172.21.15.44:0/3074516381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15ab60 0x55e0aa155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.964+0000 7fc60231eb80 1 -- 172.21.15.44:0/3074516381 >> 172.21.15.44:0/3074516381 conn(0x55e0aa164c00 msgr2=0x55e0aa170bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:53.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.964+0000 7fc60231eb80 1 -- 172.21.15.44:0/3074516381 shutdown_connections 2023-11-16T16:52:53.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.964+0000 7fc60231eb80 1 -- 172.21.15.44:0/3074516381 wait complete. 2023-11-16T16:52:53.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.965+0000 7fc60231eb80 1 -- start start 2023-11-16T16:52:53.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.965+0000 7fc60231eb80 1 Processor -- start 2023-11-16T16:52:53.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.965+0000 7fc60231eb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa150360 0x55e0aa167b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.965+0000 7fc60231eb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15ab60 0x55e0aa168050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.965+0000 7fc60231eb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15b470 0x55e0aa1c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.965+0000 7fc60231eb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0aa1c7cf0 con 0x55e0aa15b470 2023-11-16T16:52:53.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.965+0000 7fc60231eb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0aa1c7e30 con 0x55e0aa150360 2023-11-16T16:52:53.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.965+0000 7fc60231eb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0aa1c7f70 con 0x55e0aa15ab60 2023-11-16T16:52:53.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.965+0000 7fc60231eb80 10 client.0 _ll_register_callbacks cb 0x7fc5e0009340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:53.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.966+0000 7fc5ecf9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15b470 0x55e0aa1c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.966+0000 7fc5ecf9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15b470 0x55e0aa1c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57294/0 (socket says 172.21.15.44:57294) 2023-11-16T16:52:53.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.966+0000 7fc5ecf9c700 1 -- 172.21.15.44:0/2417595445 learned_addr learned my addr 172.21.15.44:0/2417595445 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:53.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.966+0000 7fc5e77fe700 1 --2- 172.21.15.44:0/2417595445 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15ab60 0x55e0aa168050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.966+0000 7fc5e7fff700 1 --2- 172.21.15.44:0/2417595445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa150360 0x55e0aa167b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.966+0000 7fc5ecf9c700 1 -- 172.21.15.44:0/2417595445 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15ab60 msgr2=0x55e0aa168050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:53.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.966+0000 7fc5ecf9c700 1 --2- 172.21.15.44:0/2417595445 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15ab60 0x55e0aa168050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.966+0000 7fc5ecf9c700 1 -- 172.21.15.44:0/2417595445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa150360 msgr2=0x55e0aa167b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:53.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.966+0000 7fc5ecf9c700 1 --2- 172.21.15.44:0/2417595445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa150360 0x55e0aa167b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.966+0000 7fc5ecf9c700 1 -- 172.21.15.44:0/2417595445 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e0aa072190 con 0x55e0aa15b470 2023-11-16T16:52:53.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.966+0000 7fc5ecf9c700 1 --2- 172.21.15.44:0/2417595445 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15b470 0x55e0aa1c77b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0005b90 tx=0x7fc5e00094f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0ce934a114055f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:53.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.966+0000 7fc5c77fe700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:52:53.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.966+0000 7fc5c77fe700 1 -- 172.21.15.44:0/2417595445 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e000e040 con 0x55e0aa15b470 2023-11-16T16:52:53.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.966+0000 7fc5c77fe700 1 -- 172.21.15.44:0/2417595445 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fc5e001a9a0 con 0x55e0aa15b470 2023-11-16T16:52:53.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.966+0000 7fc60231eb80 1 -- 172.21.15.44:0/2417595445 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc5d80109e0 con 0x55e0aa15b470 2023-11-16T16:52:53.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.967+0000 7fc5c77fe700 1 -- 172.21.15.44:0/2417595445 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5e001aba0 con 0x55e0aa15b470 2023-11-16T16:52:53.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.967+0000 7fc60231eb80 10 client.16275 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:53.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.967+0000 7fc60231eb80 1 -- 172.21.15.44:0/2417595445 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55e0a9ec2c60 con 0x55e0aa15b470 2023-11-16T16:52:53.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.967+0000 7fc5c77fe700 1 -- 172.21.15.44:0/2417595445 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5d80109e0 con 0x55e0aa15b470 2023-11-16T16:52:53.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.967+0000 7fc5c77fe700 1 -- 172.21.15.44:0/2417595445 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc5e0024610 con 0x55e0aa15b470 2023-11-16T16:52:53.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.967+0000 7fc60231eb80 10 client.16275 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:53.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.967+0000 7fc60231eb80 10 client.16275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:53.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.967+0000 7fc60231eb80 20 client.16275 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:53.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.967+0000 7fc60231eb80 1 --2- 172.21.15.44:0/2417595445 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e0aa15f8b0 0x55e0aa1cb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:53.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.967+0000 7fc60231eb80 4 client.16275 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T16:52:53.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.967+0000 7fc60231eb80 1 -- 172.21.15.44:0/2417595445 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e0aa07a660 con 0x55e0aa15f8b0 2023-11-16T16:52:53.973 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.968+0000 7fc5e7fff700 1 --2- 172.21.15.44:0/2417595445 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e0aa15f8b0 0x55e0aa1cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:53.974 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.968+0000 7fc5e7fff700 1 --2- 172.21.15.44:0/2417595445 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e0aa15f8b0 0x55e0aa1cb150 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef3645186b0f4a17 server_cookie=ae6d8435f65299b8 in_seq=0 out_seq=0 2023-11-16T16:52:53.974 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.968+0000 7fc5c77fe700 10 client.16275 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:53.974 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.968+0000 7fc5c77fe700 1 -- 172.21.15.44:0/2417595445 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e0aa07a660 con 0x55e0aa15f8b0 2023-11-16T16:52:53.974 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.968+0000 7fc5c77fe700 10 client.16275 handle_command_reply: tid=0 2023-11-16T16:52:53.974 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.968+0000 7fc60231eb80 1 client.16275 shutdown 2023-11-16T16:52:53.974 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.968+0000 7fc60231eb80 1 -- 172.21.15.44:0/2417595445 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15b470 msgr2=0x55e0aa1c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.974 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.968+0000 7fc60231eb80 1 --2- 172.21.15.44:0/2417595445 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15b470 0x55e0aa1c77b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0005b90 tx=0x7fc5e00094f0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.974 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.968+0000 7fc60231eb80 1 -- 172.21.15.44:0/2417595445 shutdown_connections 2023-11-16T16:52:53.974 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.968+0000 7fc60231eb80 1 --2- 172.21.15.44:0/2417595445 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e0aa15f8b0 0x55e0aa1cb150 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.974 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.968+0000 7fc60231eb80 1 --2- 172.21.15.44:0/2417595445 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e0aa15b470 0x55e0aa1c77b0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.974 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.968+0000 7fc60231eb80 1 --2- 172.21.15.44:0/2417595445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0aa150360 0x55e0aa167b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.974 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.968+0000 7fc60231eb80 1 --2- 172.21.15.44:0/2417595445 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e0aa15ab60 0x55e0aa168050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.969+0000 7fc60231eb80 1 -- 172.21.15.44:0/2417595445 >> 172.21.15.44:0/2417595445 conn(0x55e0aa164c00 msgr2=0x55e0aa159960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:53.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.969+0000 7fc5c77fe700 0 client.16275 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:53.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.969+0000 7fc60231eb80 1 -- 172.21.15.44:0/2417595445 shutdown_connections 2023-11-16T16:52:53.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.969+0000 7fc60231eb80 1 -- 172.21.15.44:0/2417595445 wait complete. 2023-11-16T16:52:53.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.969+0000 7fc60231eb80 20 client.16275 trim_cache size 0 max 16384 2023-11-16T16:52:53.975 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T16:52:53.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.970+0000 7fc5ee9ff700 1 -- 172.21.15.44:0/1810913249 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc5d00a12c0 msgr2=0x7fc5d00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.970+0000 7fc5ee9ff700 1 --2- 172.21.15.44:0/1810913249 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc5d00a12c0 0x7fc5d00a3770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8074f00 tx=0x7fc5e000fc00 comp rx=0 tx=0).stop 2023-11-16T16:52:53.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.970+0000 7fc5ee9ff700 1 -- 172.21.15.44:0/1810913249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5e8100cc0 msgr2=0x7fc5e8072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:53.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.970+0000 7fc5ee9ff700 1 --2- 172.21.15.44:0/1810913249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5e8100cc0 0x7fc5e8072df0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800e9a0 tx=0x7fc5d800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.970+0000 7fc5ee9ff700 1 -- 172.21.15.44:0/1810913249 shutdown_connections 2023-11-16T16:52:53.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.970+0000 7fc5ee9ff700 1 --2- 172.21.15.44:0/1810913249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5e8105620 0x7fc5e81a4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.970+0000 7fc5ee9ff700 1 --2- 172.21.15.44:0/1810913249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5e8100cc0 0x7fc5e8072df0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.970+0000 7fc5ee9ff700 1 --2- 172.21.15.44:0/1810913249 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc5d00a12c0 0x7fc5d00a3770 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.970+0000 7fc5ee9ff700 1 --2- 172.21.15.44:0/1810913249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5e8101660 0x7fc5e8073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:53.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.970+0000 7fc5ee9ff700 1 -- 172.21.15.44:0/1810913249 >> 172.21.15.44:0/1810913249 conn(0x7fc5e8077560 msgr2=0x7fc5e80ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:53.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.970+0000 7fc5ee9ff700 1 -- 172.21.15.44:0/1810913249 shutdown_connections 2023-11-16T16:52:53.977 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:53.970+0000 7fc5ee9ff700 1 -- 172.21.15.44:0/1810913249 wait complete. 2023-11-16T16:52:54.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:54 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1941454846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:52:54.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:54 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1941454846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:52:54.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:54 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1941454846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:52:54.993 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:52:54.994 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:52:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.206+0000 7f7539dff700 1 -- 172.21.15.44:0/2817274023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7534101660 msgr2=0x7f7534104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.206+0000 7f7539dff700 1 --2- 172.21.15.44:0/2817274023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7534101660 0x7f7534104ef0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f751c0099c0 tx=0x7f751c009cd0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.206+0000 7f7539dff700 1 -- 172.21.15.44:0/2817274023 shutdown_connections 2023-11-16T16:52:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.206+0000 7f7539dff700 1 --2- 172.21.15.44:0/2817274023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7534105620 0x7f753410d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.206+0000 7f7539dff700 1 --2- 172.21.15.44:0/2817274023 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7534100cc0 0x7f7534101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.206+0000 7f7539dff700 1 --2- 172.21.15.44:0/2817274023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7534101660 0x7f7534104ef0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.206+0000 7f7539dff700 1 -- 172.21.15.44:0/2817274023 >> 172.21.15.44:0/2817274023 conn(0x7f7534077560 msgr2=0x7f7534077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.206+0000 7f7539dff700 1 -- 172.21.15.44:0/2817274023 shutdown_connections 2023-11-16T16:52:55.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.206+0000 7f7539dff700 1 -- 172.21.15.44:0/2817274023 wait complete. 2023-11-16T16:52:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.207+0000 7f7539dff700 1 Processor -- start 2023-11-16T16:52:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.207+0000 7f7539dff700 1 -- start start 2023-11-16T16:52:55.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f7539dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7534100cc0 0x7f7534072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f7539dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7534101660 0x7f7534073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f7539dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7534105620 0x7f75341a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f7539dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7534073b60 con 0x7f7534100cc0 2023-11-16T16:52:55.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f7539dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75341a4cd0 con 0x7f7534101660 2023-11-16T16:52:55.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f7539dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75341a4e10 con 0x7f7534105620 2023-11-16T16:52:55.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f75337fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7534100cc0 0x7f7534072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f75337fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7534100cc0 0x7f7534072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33212/0 (socket says 172.21.15.44:33212) 2023-11-16T16:52:55.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f75337fe700 1 -- 172.21.15.44:0/1703703122 learned_addr learned my addr 172.21.15.44:0/1703703122 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:55.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f752bfff700 1 --2- 172.21.15.44:0/1703703122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7534101660 0x7f7534073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f7533fff700 1 --2- 172.21.15.44:0/1703703122 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7534105620 0x7f75341a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f752bfff700 1 -- 172.21.15.44:0/1703703122 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7534105620 msgr2=0x7f75341a4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f752bfff700 1 --2- 172.21.15.44:0/1703703122 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7534105620 0x7f75341a4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f752bfff700 1 -- 172.21.15.44:0/1703703122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7534100cc0 msgr2=0x7f7534072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f752bfff700 1 --2- 172.21.15.44:0/1703703122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7534100cc0 0x7f7534072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f752bfff700 1 -- 172.21.15.44:0/1703703122 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751c0095f0 con 0x7f7534101660 2023-11-16T16:52:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.208+0000 7f75337fe700 1 --2- 172.21.15.44:0/1703703122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7534100cc0 0x7f7534072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.209+0000 7f752bfff700 1 --2- 172.21.15.44:0/1703703122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7534101660 0x7f7534073330 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f753406a350 tx=0x7f751c00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.209+0000 7f75317fa700 1 -- 172.21.15.44:0/1703703122 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751c01d070 con 0x7f7534101660 2023-11-16T16:52:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.209+0000 7f7539dff700 1 -- 172.21.15.44:0/1703703122 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7534110370 con 0x7f7534101660 2023-11-16T16:52:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.209+0000 7f75317fa700 1 -- 172.21.15.44:0/1703703122 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f751c005050 con 0x7f7534101660 2023-11-16T16:52:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.209+0000 7f75317fa700 1 -- 172.21.15.44:0/1703703122 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751c004970 con 0x7f7534101660 2023-11-16T16:52:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.209+0000 7f7539dff700 1 -- 172.21.15.44:0/1703703122 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75341101e0 con 0x7f7534101660 2023-11-16T16:52:55.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.212+0000 7f75317fa700 1 -- 172.21.15.44:0/1703703122 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f751c0054c0 con 0x7f7534101660 2023-11-16T16:52:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.214+0000 7f754d7b9b80 1 Processor -- start 2023-11-16T16:52:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.214+0000 7f754d7b9b80 1 -- start start 2023-11-16T16:52:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.214+0000 7f754d7b9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf1667be0 0x55ecf1667fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f754d7b9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf16684f0 0x55ecf166c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f754d7b9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf166c930 0x55ecf166cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f754d7b9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf156d0e0 con 0x55ecf16684f0 2023-11-16T16:52:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f754d7b9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf15bf920 con 0x55ecf1667be0 2023-11-16T16:52:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f754d7b9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf15b8db0 con 0x55ecf166c930 2023-11-16T16:52:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f7533fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf1667be0 0x55ecf1667fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f75317fa700 1 --2- 172.21.15.44:0/1703703122 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f75200a12c0 0x7f75200a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f75337fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf16684f0 0x55ecf166c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f7533fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf1667be0 0x55ecf1667fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:40672/0 (socket says 172.21.15.44:40672) 2023-11-16T16:52:55.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f7533fff700 1 -- 172.21.15.44:0/1922407341 learned_addr learned my addr 172.21.15.44:0/1922407341 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:55.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f752bfff700 1 --2- 172.21.15.44:0/1922407341 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf166c930 0x55ecf166cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f75317fa700 1 -- 172.21.15.44:0/1703703122 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f751c0cdd80 con 0x7f7534101660 2023-11-16T16:52:55.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f75337fe700 1 -- 172.21.15.44:0/1922407341 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf166c930 msgr2=0x55ecf166cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:55.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f75337fe700 1 --2- 172.21.15.44:0/1922407341 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf166c930 0x55ecf166cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f75337fe700 1 -- 172.21.15.44:0/1922407341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf1667be0 msgr2=0x55ecf1667fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:55.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f75337fe700 1 --2- 172.21.15.44:0/1922407341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf1667be0 0x55ecf1667fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f7533fff700 1 --2- 172.21.15.44:0/1703703122 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f75200a12c0 0x7f75200a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.215+0000 7f75337fe700 1 -- 172.21.15.44:0/1922407341 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecf143a6b0 con 0x55ecf16684f0 2023-11-16T16:52:55.219 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.216+0000 7f75337fe700 1 --2- 172.21.15.44:0/1922407341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf16684f0 0x55ecf166c3f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7524011570 tx=0x7f7524011880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=735b12ab72e2e6a5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:55.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.216+0000 7f7529ffb700 1 -- 172.21.15.44:0/1922407341 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7524004230 con 0x55ecf16684f0 2023-11-16T16:52:55.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.216+0000 7f7529ffb700 1 -- 172.21.15.44:0/1922407341 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7524004390 con 0x55ecf16684f0 2023-11-16T16:52:55.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.216+0000 7f7529ffb700 1 -- 172.21.15.44:0/1922407341 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75240198e0 con 0x55ecf16684f0 2023-11-16T16:52:55.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.216+0000 7f7533fff700 1 --2- 172.21.15.44:0/1703703122 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f75200a12c0 0x7f75200a3770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f752c0060b0 tx=0x7f752c004b40 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:55.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.217+0000 7f754d7b9b80 1 -- 172.21.15.44:0/1922407341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf16684f0 msgr2=0x55ecf166c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.217+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/1922407341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf16684f0 0x55ecf166c3f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7524011570 tx=0x7f7524011880 comp rx=0 tx=0).stop 2023-11-16T16:52:55.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.217+0000 7f754d7b9b80 1 -- 172.21.15.44:0/1922407341 shutdown_connections 2023-11-16T16:52:55.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.217+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/1922407341 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf166c930 0x55ecf166cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.217+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/1922407341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf1667be0 0x55ecf1667fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.217+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/1922407341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf16684f0 0x55ecf166c3f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.217+0000 7f754d7b9b80 1 -- 172.21.15.44:0/1922407341 >> 172.21.15.44:0/1922407341 conn(0x55ecf165d3e0 msgr2=0x55ecf1662bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:55.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.217+0000 7f754d7b9b80 1 -- 172.21.15.44:0/1922407341 shutdown_connections 2023-11-16T16:52:55.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.217+0000 7f754d7b9b80 1 -- 172.21.15.44:0/1922407341 wait complete. 2023-11-16T16:52:55.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.217+0000 7f754d7b9b80 1 -- start start 2023-11-16T16:52:55.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.218+0000 7f754d7b9b80 1 Processor -- start 2023-11-16T16:52:55.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.218+0000 7f754d7b9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 0x55ecf1674190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.218+0000 7f754d7b9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf16684f0 0x55ecf16713a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.218+0000 7f754d7b9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf166c930 0x55ecf1671ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.218+0000 7f754d7b9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf155a960 con 0x55ecf1667be0 2023-11-16T16:52:55.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.218+0000 7f754d7b9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf155a600 con 0x55ecf166c930 2023-11-16T16:52:55.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.218+0000 7f754d7b9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf143c9a0 con 0x55ecf16684f0 2023-11-16T16:52:55.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.218+0000 7f754d7b9b80 10 client.0 _ll_register_callbacks cb 0x55ecf1578800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:55.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.218+0000 7f75337fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 0x55ecf1674190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.218+0000 7f7533fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf166c930 0x55ecf1671ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.219+0000 7f7533fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf166c930 0x55ecf1671ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:40674/0 (socket says 172.21.15.44:40674) 2023-11-16T16:52:55.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.219+0000 7f7533fff700 1 -- 172.21.15.44:0/3537211028 learned_addr learned my addr 172.21.15.44:0/3537211028 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:55.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.219+0000 7f752bfff700 1 --2- 172.21.15.44:0/3537211028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf16684f0 0x55ecf16713a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.219+0000 7f7533fff700 1 -- 172.21.15.44:0/3537211028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf16684f0 msgr2=0x55ecf16713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:55.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.219+0000 7f7533fff700 1 --2- 172.21.15.44:0/3537211028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf16684f0 0x55ecf16713a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.219+0000 7f7533fff700 1 -- 172.21.15.44:0/3537211028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 msgr2=0x55ecf1674190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:55.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.219+0000 7f7533fff700 1 --2- 172.21.15.44:0/3537211028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 0x55ecf1674190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.219+0000 7f7533fff700 1 -- 172.21.15.44:0/3537211028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7524009d60 con 0x55ecf166c930 2023-11-16T16:52:55.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.219+0000 7f7533fff700 1 --2- 172.21.15.44:0/3537211028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf166c930 0x55ecf1671ae0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f752c007d60 tx=0x7f752c004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3631b20b187be05 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:55.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.219+0000 7f7512ffd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:52:55.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.219+0000 7f7512ffd700 1 -- 172.21.15.44:0/3537211028 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752c00e040 con 0x55ecf166c930 2023-11-16T16:52:55.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.219+0000 7f7512ffd700 1 -- 172.21.15.44:0/3537211028 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f752c025400 con 0x55ecf166c930 2023-11-16T16:52:55.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.219+0000 7f754d7b9b80 1 -- 172.21.15.44:0/3537211028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ecf165aba0 con 0x55ecf166c930 2023-11-16T16:52:55.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.220+0000 7f7512ffd700 1 -- 172.21.15.44:0/3537211028 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f752c01ae90 con 0x55ecf166c930 2023-11-16T16:52:55.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.220+0000 7f754d7b9b80 10 client.26312 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:55.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.220+0000 7f754d7b9b80 1 -- 172.21.15.44:0/3537211028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55ecf157f210 con 0x55ecf166c930 2023-11-16T16:52:55.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.220+0000 7f7512ffd700 1 -- 172.21.15.44:0/3537211028 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ecf165aba0 con 0x55ecf166c930 2023-11-16T16:52:55.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.220+0000 7f7512ffd700 1 -- 172.21.15.44:0/3537211028 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f752c019700 con 0x55ecf166c930 2023-11-16T16:52:55.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.220+0000 7f754d7b9b80 10 client.26312 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:55.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.220+0000 7f754d7b9b80 10 client.26312 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:55.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.220+0000 7f754d7b9b80 20 client.26312 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:55.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.220+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/3537211028 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ecf1681970 0x55ecf1683d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.220+0000 7f754d7b9b80 4 client.26312 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:55.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.220+0000 7f754d7b9b80 1 -- 172.21.15.44:0/3537211028 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ecf157b120 con 0x55ecf1681970 2023-11-16T16:52:55.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.220+0000 7f75337fe700 1 --2- 172.21.15.44:0/3537211028 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ecf1681970 0x55ecf1683d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.221+0000 7f75337fe700 1 --2- 172.21.15.44:0/3537211028 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ecf1681970 0x55ecf1683d80 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ffa411f4dab5aa7 server_cookie=38ee7c72fe9f8ff0 in_seq=0 out_seq=0 2023-11-16T16:52:55.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.221+0000 7f7512ffd700 10 client.26312 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:55.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.222+0000 7f7512ffd700 1 -- 172.21.15.44:0/3537211028 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ecf157b120 con 0x55ecf1681970 2023-11-16T16:52:55.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.222+0000 7f7512ffd700 10 client.26312 handle_command_reply: tid=0 2023-11-16T16:52:55.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.222+0000 7f754d7b9b80 1 client.26312 shutdown 2023-11-16T16:52:55.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.222+0000 7f754d7b9b80 1 -- 172.21.15.44:0/3537211028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf166c930 msgr2=0x55ecf1671ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.222+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/3537211028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf166c930 0x55ecf1671ae0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f752c007d60 tx=0x7f752c004ad0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.222+0000 7f754d7b9b80 1 -- 172.21.15.44:0/3537211028 shutdown_connections 2023-11-16T16:52:55.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.222+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/3537211028 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ecf1681970 0x55ecf1683d80 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.222+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/3537211028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf166c930 0x55ecf1671ae0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.222+0000 7f7512ffd700 0 client.26312 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:55.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.222+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/3537211028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 0x55ecf1674190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.222+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/3537211028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf16684f0 0x55ecf16713a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.222+0000 7f754d7b9b80 1 -- 172.21.15.44:0/3537211028 >> 172.21.15.44:0/3537211028 conn(0x55ecf165d3e0 msgr2=0x55ecf166dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:55.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.222+0000 7f754d7b9b80 1 -- 172.21.15.44:0/3537211028 shutdown_connections 2023-11-16T16:52:55.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.222+0000 7f754d7b9b80 1 -- 172.21.15.44:0/3537211028 wait complete. 2023-11-16T16:52:55.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.222+0000 7f754d7b9b80 20 client.26312 trim_cache size 0 max 16384 2023-11-16T16:52:55.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.240+0000 7f754d7b9b80 1 Processor -- start 2023-11-16T16:52:55.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.240+0000 7f754d7b9b80 1 -- start start 2023-11-16T16:52:55.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.240+0000 7f754d7b9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 0x55ecf1671350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.241+0000 7f754d7b9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf16684f0 0x55ecf1678a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.241+0000 7f754d7b9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf166c930 0x55ecf1673cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.241+0000 7f754d7b9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf166b4c0 con 0x55ecf1667be0 2023-11-16T16:52:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.241+0000 7f754d7b9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf16726f0 con 0x55ecf16684f0 2023-11-16T16:52:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.241+0000 7f754d7b9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf1678f60 con 0x55ecf166c930 2023-11-16T16:52:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.241+0000 7f75337fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 0x55ecf1671350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.241+0000 7f75337fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 0x55ecf1671350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33232/0 (socket says 172.21.15.44:33232) 2023-11-16T16:52:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.241+0000 7f75337fe700 1 -- 172.21.15.44:0/465893867 learned_addr learned my addr 172.21.15.44:0/465893867 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.241+0000 7f752bfff700 1 --2- 172.21.15.44:0/465893867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf16684f0 0x55ecf1678a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.241+0000 7f7533fff700 1 --2- 172.21.15.44:0/465893867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf166c930 0x55ecf1673cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.241+0000 7f75337fe700 1 -- 172.21.15.44:0/465893867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf166c930 msgr2=0x55ecf1673cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.241+0000 7f75337fe700 1 --2- 172.21.15.44:0/465893867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf166c930 0x55ecf1673cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.241+0000 7f75337fe700 1 -- 172.21.15.44:0/465893867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf16684f0 msgr2=0x55ecf1678a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.241+0000 7f75337fe700 1 --2- 172.21.15.44:0/465893867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf16684f0 0x55ecf1678a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.241+0000 7f75337fe700 1 -- 172.21.15.44:0/465893867 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecf165eb80 con 0x55ecf1667be0 2023-11-16T16:52:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.241+0000 7f752bfff700 1 --2- 172.21.15.44:0/465893867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf16684f0 0x55ecf1678a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:52:55.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.242+0000 7f75337fe700 1 --2- 172.21.15.44:0/465893867 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 0x55ecf1671350 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f75240096b0 tx=0x7f7524011880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcc64b2bc9ccb1ab server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.242+0000 7f752a7fc700 1 -- 172.21.15.44:0/465893867 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7524029040 con 0x55ecf1667be0 2023-11-16T16:52:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.242+0000 7f752a7fc700 1 -- 172.21.15.44:0/465893867 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f752402a7f0 con 0x55ecf1667be0 2023-11-16T16:52:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.242+0000 7f752a7fc700 1 -- 172.21.15.44:0/465893867 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7524004740 con 0x55ecf1667be0 2023-11-16T16:52:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.242+0000 7f754d7b9b80 1 -- 172.21.15.44:0/465893867 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 msgr2=0x55ecf1671350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.242+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/465893867 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 0x55ecf1671350 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f75240096b0 tx=0x7f7524011880 comp rx=0 tx=0).stop 2023-11-16T16:52:55.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.242+0000 7f754d7b9b80 1 -- 172.21.15.44:0/465893867 shutdown_connections 2023-11-16T16:52:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.242+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/465893867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf166c930 0x55ecf1673cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.242+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/465893867 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 0x55ecf1671350 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.242+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/465893867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf16684f0 0x55ecf1678a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.242+0000 7f754d7b9b80 1 -- 172.21.15.44:0/465893867 >> 172.21.15.44:0/465893867 conn(0x55ecf165d3e0 msgr2=0x55ecf1670c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.243+0000 7f754d7b9b80 1 -- 172.21.15.44:0/465893867 shutdown_connections 2023-11-16T16:52:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.243+0000 7f754d7b9b80 1 -- 172.21.15.44:0/465893867 wait complete. 2023-11-16T16:52:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.243+0000 7f754d7b9b80 1 -- start start 2023-11-16T16:52:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.243+0000 7f754d7b9b80 1 Processor -- start 2023-11-16T16:52:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f754d7b9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 0x55ecf1677230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f754d7b9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf16684f0 0x55ecf1677770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f754d7b9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf166c930 0x55ecf16d5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f754d7b9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf1678040 con 0x55ecf1667be0 2023-11-16T16:52:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f754d7b9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf16d5d80 con 0x55ecf16684f0 2023-11-16T16:52:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f754d7b9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf16d5ec0 con 0x55ecf166c930 2023-11-16T16:52:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f754d7b9b80 10 client.0 _ll_register_callbacks cb 0x55ecf1578800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f75337fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 0x55ecf1677230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f7533fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf166c930 0x55ecf16d5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f752bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf16684f0 0x55ecf1677770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f7533fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf166c930 0x55ecf16d5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:33050/0 (socket says 172.21.15.44:33050) 2023-11-16T16:52:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f7533fff700 1 -- 172.21.15.44:0/2126269237 learned_addr learned my addr 172.21.15.44:0/2126269237 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f752bfff700 1 -- 172.21.15.44:0/2126269237 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf166c930 msgr2=0x55ecf16d5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f752bfff700 1 --2- 172.21.15.44:0/2126269237 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf166c930 0x55ecf16d5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f752bfff700 1 -- 172.21.15.44:0/2126269237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 msgr2=0x55ecf1677230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f752bfff700 1 --2- 172.21.15.44:0/2126269237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 0x55ecf1677230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f752bfff700 1 -- 172.21.15.44:0/2126269237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ecf143a6b0 con 0x55ecf16684f0 2023-11-16T16:52:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.244+0000 7f75337fe700 1 --2- 172.21.15.44:0/2126269237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 0x55ecf1677230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.245+0000 7f752bfff700 1 --2- 172.21.15.44:0/2126269237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf16684f0 0x55ecf1677770 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f751c0cb0f0 tx=0x7f751c009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5bac7039bd81fb4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.245+0000 7f7512ffd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:52:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.245+0000 7f7512ffd700 1 -- 172.21.15.44:0/2126269237 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751c0cf040 con 0x55ecf16684f0 2023-11-16T16:52:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.245+0000 7f754d7b9b80 1 -- 172.21.15.44:0/2126269237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ecf166f3d0 con 0x55ecf16684f0 2023-11-16T16:52:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.245+0000 7f7512ffd700 1 -- 172.21.15.44:0/2126269237 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f751c0051c0 con 0x55ecf16684f0 2023-11-16T16:52:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.245+0000 7f7512ffd700 1 -- 172.21.15.44:0/2126269237 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f751c09a460 con 0x55ecf16684f0 2023-11-16T16:52:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.245+0000 7f754d7b9b80 10 client.26318 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.245+0000 7f754d7b9b80 1 -- 172.21.15.44:0/2126269237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55ecf13e6e60 con 0x55ecf16684f0 2023-11-16T16:52:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.245+0000 7f7512ffd700 1 -- 172.21.15.44:0/2126269237 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ecf166f3d0 con 0x55ecf16684f0 2023-11-16T16:52:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.246+0000 7f7512ffd700 1 -- 172.21.15.44:0/2126269237 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f751c099ff0 con 0x55ecf16684f0 2023-11-16T16:52:55.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.246+0000 7f754d7b9b80 10 client.26318 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:55.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.246+0000 7f754d7b9b80 10 client.26318 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:55.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.246+0000 7f754d7b9b80 20 client.26318 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:55.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.246+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/2126269237 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ecf16d7220 0x55ecf16d9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.246+0000 7f754d7b9b80 4 client.26318 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:52:55.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.246+0000 7f754d7b9b80 1 -- 172.21.15.44:0/2126269237 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x7f7524004390 con 0x55ecf16d7220 2023-11-16T16:52:55.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.246+0000 7f75337fe700 1 --2- 172.21.15.44:0/2126269237 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ecf16d7220 0x55ecf16d9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.246+0000 7f75337fe700 1 --2- 172.21.15.44:0/2126269237 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ecf16d7220 0x55ecf16d9630 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4189a1be5fc4a24 server_cookie=dfb5fd7b4b54ee29 in_seq=0 out_seq=0 2023-11-16T16:52:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.246+0000 7f7512ffd700 10 client.26318 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.247+0000 7f7512ffd700 1 -- 172.21.15.44:0/2126269237 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x7f7524004390 con 0x55ecf16d7220 2023-11-16T16:52:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.247+0000 7f7512ffd700 10 client.26318 handle_command_reply: tid=0 2023-11-16T16:52:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.247+0000 7f754d7b9b80 1 client.26318 shutdown 2023-11-16T16:52:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.248+0000 7f754d7b9b80 1 -- 172.21.15.44:0/2126269237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf16684f0 msgr2=0x55ecf1677770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.248+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/2126269237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf16684f0 0x55ecf1677770 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f751c0cb0f0 tx=0x7f751c009310 comp rx=0 tx=0).stop 2023-11-16T16:52:55.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.248+0000 7f754d7b9b80 1 -- 172.21.15.44:0/2126269237 shutdown_connections 2023-11-16T16:52:55.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.248+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/2126269237 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ecf16d7220 0x55ecf16d9630 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.248+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/2126269237 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ecf166c930 0x55ecf16d5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.248+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/2126269237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ecf1667be0 0x55ecf1677230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.248+0000 7f754d7b9b80 1 --2- 172.21.15.44:0/2126269237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ecf16684f0 0x55ecf1677770 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.248+0000 7f7512ffd700 0 client.26318 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:55.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.248+0000 7f754d7b9b80 1 -- 172.21.15.44:0/2126269237 >> 172.21.15.44:0/2126269237 conn(0x55ecf165d3e0 msgr2=0x55ecf1667270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:55.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.248+0000 7f754d7b9b80 1 -- 172.21.15.44:0/2126269237 shutdown_connections 2023-11-16T16:52:55.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.248+0000 7f754d7b9b80 1 -- 172.21.15.44:0/2126269237 wait complete. 2023-11-16T16:52:55.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.248+0000 7f754d7b9b80 20 client.26318 trim_cache size 0 max 16384 2023-11-16T16:52:55.253 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:52:55.253 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:52:55.254 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "abf214e5-acc2-4e8d-a6d4-d53613037095", 2023-11-16T16:52:55.254 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:52:55.254 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:52:55.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.249+0000 7f7539dff700 1 -- 172.21.15.44:0/1703703122 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f75200a12c0 msgr2=0x7f75200a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.249+0000 7f7539dff700 1 --2- 172.21.15.44:0/1703703122 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f75200a12c0 0x7f75200a3770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f752c0060b0 tx=0x7f752c004b40 comp rx=0 tx=0).stop 2023-11-16T16:52:55.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.249+0000 7f7539dff700 1 -- 172.21.15.44:0/1703703122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7534101660 msgr2=0x7f7534073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.249+0000 7f7539dff700 1 --2- 172.21.15.44:0/1703703122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7534101660 0x7f7534073330 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f753406a350 tx=0x7f751c00f690 comp rx=0 tx=0).stop 2023-11-16T16:52:55.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.249+0000 7f7539dff700 1 -- 172.21.15.44:0/1703703122 shutdown_connections 2023-11-16T16:52:55.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.249+0000 7f7539dff700 1 --2- 172.21.15.44:0/1703703122 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f75200a12c0 0x7f75200a3770 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.249+0000 7f7539dff700 1 --2- 172.21.15.44:0/1703703122 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7534105620 0x7f75341a4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.249+0000 7f7539dff700 1 --2- 172.21.15.44:0/1703703122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7534100cc0 0x7f7534072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.249+0000 7f7539dff700 1 --2- 172.21.15.44:0/1703703122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7534101660 0x7f7534073330 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.249+0000 7f7539dff700 1 -- 172.21.15.44:0/1703703122 >> 172.21.15.44:0/1703703122 conn(0x7f7534077560 msgr2=0x7f75340ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:55.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.249+0000 7f7539dff700 1 -- 172.21.15.44:0/1703703122 shutdown_connections 2023-11-16T16:52:55.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.249+0000 7f7539dff700 1 -- 172.21.15.44:0/1703703122 wait complete. 2023-11-16T16:52:55.269 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:52:55.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.477+0000 7f1d44bff700 1 -- 172.21.15.44:0/3024962621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d40100cc0 msgr2=0x7f1d40101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.477+0000 7f1d44bff700 1 --2- 172.21.15.44:0/3024962621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d40100cc0 0x7f1d40101090 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1d300099c0 tx=0x7f1d30009cd0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.477+0000 7f1d44bff700 1 -- 172.21.15.44:0/3024962621 shutdown_connections 2023-11-16T16:52:55.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.477+0000 7f1d44bff700 1 --2- 172.21.15.44:0/3024962621 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d40105620 0x7f1d4010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.477+0000 7f1d44bff700 1 --2- 172.21.15.44:0/3024962621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d40100cc0 0x7f1d40101090 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.477+0000 7f1d44bff700 1 --2- 172.21.15.44:0/3024962621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d40101660 0x7f1d40104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.477+0000 7f1d44bff700 1 -- 172.21.15.44:0/3024962621 >> 172.21.15.44:0/3024962621 conn(0x7f1d40077560 msgr2=0x7f1d40077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:55.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.477+0000 7f1d44bff700 1 -- 172.21.15.44:0/3024962621 shutdown_connections 2023-11-16T16:52:55.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.477+0000 7f1d44bff700 1 -- 172.21.15.44:0/3024962621 wait complete. 2023-11-16T16:52:55.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.478+0000 7f1d44bff700 1 Processor -- start 2023-11-16T16:52:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.478+0000 7f1d44bff700 1 -- start start 2023-11-16T16:52:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.479+0000 7f1d44bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d40100cc0 0x7f1d40072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.479+0000 7f1d44bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d40101660 0x7f1d40073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.479+0000 7f1d44bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d40105620 0x7f1d401a4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.479+0000 7f1d44bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d40073bd0 con 0x7f1d40105620 2023-11-16T16:52:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.479+0000 7f1d44bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d401a4c40 con 0x7f1d40100cc0 2023-11-16T16:52:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.479+0000 7f1d44bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d401a4d80 con 0x7f1d40101660 2023-11-16T16:52:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.479+0000 7f1d3ed9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d40105620 0x7f1d401a4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.479+0000 7f1d3e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d40100cc0 0x7f1d40072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.479+0000 7f1d3ed9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d40105620 0x7f1d401a4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33256/0 (socket says 172.21.15.44:33256) 2023-11-16T16:52:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.479+0000 7f1d3ed9d700 1 -- 172.21.15.44:0/2770603249 learned_addr learned my addr 172.21.15.44:0/2770603249 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.479+0000 7f1d37fff700 1 --2- 172.21.15.44:0/2770603249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d40101660 0x7f1d40073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.479+0000 7f1d3e59c700 1 -- 172.21.15.44:0/2770603249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d40101660 msgr2=0x7f1d40073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.479+0000 7f1d3e59c700 1 --2- 172.21.15.44:0/2770603249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d40101660 0x7f1d40073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.479+0000 7f1d3e59c700 1 -- 172.21.15.44:0/2770603249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d40105620 msgr2=0x7f1d401a4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.479+0000 7f1d3e59c700 1 --2- 172.21.15.44:0/2770603249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d40105620 0x7f1d401a4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.480+0000 7f1d3e59c700 1 -- 172.21.15.44:0/2770603249 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d300095f0 con 0x7f1d40100cc0 2023-11-16T16:52:55.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.480+0000 7f1d3e59c700 1 --2- 172.21.15.44:0/2770603249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d40100cc0 0x7f1d40072df0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1d3000b580 tx=0x7f1d3000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:55.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.480+0000 7f1d377fe700 1 -- 172.21.15.44:0/2770603249 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3001d070 con 0x7f1d40100cc0 2023-11-16T16:52:55.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.480+0000 7f1d377fe700 1 -- 172.21.15.44:0/2770603249 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d30022470 con 0x7f1d40100cc0 2023-11-16T16:52:55.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.480+0000 7f1d44bff700 1 -- 172.21.15.44:0/2770603249 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d401103d0 con 0x7f1d40100cc0 2023-11-16T16:52:55.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.480+0000 7f1d377fe700 1 -- 172.21.15.44:0/2770603249 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d30017650 con 0x7f1d40100cc0 2023-11-16T16:52:55.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.480+0000 7f1d44bff700 1 -- 172.21.15.44:0/2770603249 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d40110240 con 0x7f1d40100cc0 2023-11-16T16:52:55.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.485+0000 7f1d377fe700 1 -- 172.21.15.44:0/2770603249 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f1d300225e0 con 0x7f1d40100cc0 2023-11-16T16:52:55.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.486+0000 7f1d5860fb80 1 Processor -- start 2023-11-16T16:52:55.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.486+0000 7f1d5860fb80 1 -- start start 2023-11-16T16:52:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.486+0000 7f1d5860fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255db60 0x559ca255df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.486+0000 7f1d5860fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca255e470 0x559ca2562370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.486+0000 7f1d377fe700 1 --2- 172.21.15.44:0/2770603249 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1d2c0a1340 0x7f1d2c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.486+0000 7f1d5860fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca25628b0 0x559ca2562d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.486+0000 7f1d5860fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ca2463060 con 0x559ca25628b0 2023-11-16T16:52:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.486+0000 7f1d5860fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ca24b58a0 con 0x559ca255e470 2023-11-16T16:52:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.486+0000 7f1d5860fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ca24aed30 con 0x559ca255db60 2023-11-16T16:52:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.487+0000 7f1d377fe700 1 -- 172.21.15.44:0/2770603249 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f1d300cde90 con 0x7f1d40100cc0 2023-11-16T16:52:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.487+0000 7f1d3ed9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca25628b0 0x559ca2562d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.487+0000 7f1d3ed9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255db60 0x559ca255df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.487+0000 7f1d3e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca255e470 0x559ca2562370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.487+0000 7f1d37fff700 1 --2- 172.21.15.44:0/2770603249 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1d2c0a1340 0x7f1d2c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.487+0000 7f1d3ed9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255db60 0x559ca255df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:33080/0 (socket says 172.21.15.44:33080) 2023-11-16T16:52:55.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.487+0000 7f1d3ed9d700 1 -- 172.21.15.44:0/278290294 learned_addr learned my addr 172.21.15.44:0/278290294 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:55.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.487+0000 7f1d3e59c700 1 --2- 172.21.15.44:0/278290294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca255e470 0x559ca2562370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:40708/0 (socket says 172.21.15.44:40708) 2023-11-16T16:52:55.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.487+0000 7f1d3ed9d700 1 -- 172.21.15.44:0/278290294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca255e470 msgr2=0x559ca2562370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:55.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.487+0000 7f1d3ed9d700 1 --2- 172.21.15.44:0/278290294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca255e470 0x559ca2562370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.487+0000 7f1d3ed9d700 1 -- 172.21.15.44:0/278290294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca25628b0 msgr2=0x559ca2562d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:55.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.487+0000 7f1d3ed9d700 1 --2- 172.21.15.44:0/278290294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca25628b0 0x559ca2562d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.487+0000 7f1d3ed9d700 1 -- 172.21.15.44:0/278290294 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ca2330630 con 0x559ca255db60 2023-11-16T16:52:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.488+0000 7f1d37fff700 1 --2- 172.21.15.44:0/2770603249 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1d2c0a1340 0x7f1d2c0a37f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1d400745e0 tx=0x7f1d28009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.488+0000 7f1d3ed9d700 1 --2- 172.21.15.44:0/278290294 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255db60 0x559ca255df30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1d38008d60 tx=0x7f1d3800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e86c8d5ebbd497a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.488+0000 7f1d34ff9700 1 -- 172.21.15.44:0/278290294 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3800e070 con 0x559ca255db60 2023-11-16T16:52:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.488+0000 7f1d34ff9700 1 -- 172.21.15.44:0/278290294 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d3800f040 con 0x559ca255db60 2023-11-16T16:52:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.488+0000 7f1d34ff9700 1 -- 172.21.15.44:0/278290294 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d38007760 con 0x559ca255db60 2023-11-16T16:52:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.488+0000 7f1d5860fb80 1 -- 172.21.15.44:0/278290294 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255db60 msgr2=0x559ca255df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.488+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/278290294 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255db60 0x559ca255df30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1d38008d60 tx=0x7f1d3800b040 comp rx=0 tx=0).stop 2023-11-16T16:52:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.489+0000 7f1d5860fb80 1 -- 172.21.15.44:0/278290294 shutdown_connections 2023-11-16T16:52:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.489+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/278290294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca25628b0 0x559ca2562d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.489+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/278290294 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255db60 0x559ca255df30 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.489+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/278290294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca255e470 0x559ca2562370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.489+0000 7f1d5860fb80 1 -- 172.21.15.44:0/278290294 >> 172.21.15.44:0/278290294 conn(0x559ca2553360 msgr2=0x559ca2558b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.489+0000 7f1d5860fb80 1 -- 172.21.15.44:0/278290294 shutdown_connections 2023-11-16T16:52:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.489+0000 7f1d5860fb80 1 -- 172.21.15.44:0/278290294 wait complete. 2023-11-16T16:52:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.489+0000 7f1d5860fb80 1 -- start start 2023-11-16T16:52:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.489+0000 7f1d5860fb80 1 Processor -- start 2023-11-16T16:52:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.490+0000 7f1d5860fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca255db60 0x559ca256a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.490+0000 7f1d5860fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255e470 0x559ca2567350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.490+0000 7f1d5860fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca25628b0 0x559ca25679f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.490+0000 7f1d5860fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ca24508e0 con 0x559ca25628b0 2023-11-16T16:52:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.490+0000 7f1d5860fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ca2450580 con 0x559ca255db60 2023-11-16T16:52:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.490+0000 7f1d5860fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ca2332920 con 0x559ca255e470 2023-11-16T16:52:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.490+0000 7f1d5860fb80 10 client.0 _ll_register_callbacks cb 0x559ca246e780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.490+0000 7f1d3ed9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca25628b0 0x559ca25679f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.490+0000 7f1d3ed9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca25628b0 0x559ca25679f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33270/0 (socket says 172.21.15.44:33270) 2023-11-16T16:52:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.490+0000 7f1d3ed9d700 1 -- 172.21.15.44:0/2040027816 learned_addr learned my addr 172.21.15.44:0/2040027816 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.490+0000 7f1d3e59c700 1 --2- 172.21.15.44:0/2040027816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca255db60 0x559ca256a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.490+0000 7f1d37fff700 1 --2- 172.21.15.44:0/2040027816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255e470 0x559ca2567350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.491+0000 7f1d3ed9d700 1 -- 172.21.15.44:0/2040027816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255e470 msgr2=0x559ca2567350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.491+0000 7f1d3ed9d700 1 --2- 172.21.15.44:0/2040027816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255e470 0x559ca2567350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.491+0000 7f1d3ed9d700 1 -- 172.21.15.44:0/2040027816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca255db60 msgr2=0x559ca256a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.491+0000 7f1d3ed9d700 1 --2- 172.21.15.44:0/2040027816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca255db60 0x559ca256a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.491+0000 7f1d3ed9d700 1 -- 172.21.15.44:0/2040027816 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d38008a40 con 0x559ca25628b0 2023-11-16T16:52:55.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.491+0000 7f1d3ed9d700 1 --2- 172.21.15.44:0/2040027816 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca25628b0 0x559ca25679f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1d38009920 tx=0x7f1d38009e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9bad4824afa236 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:55.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.491+0000 7f1d25ffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:52:55.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.491+0000 7f1d25ffb700 1 -- 172.21.15.44:0/2040027816 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3800e070 con 0x559ca25628b0 2023-11-16T16:52:55.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.491+0000 7f1d5860fb80 1 -- 172.21.15.44:0/2040027816 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559ca2550b20 con 0x559ca25628b0 2023-11-16T16:52:55.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.491+0000 7f1d25ffb700 1 -- 172.21.15.44:0/2040027816 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f1d38003c00 con 0x559ca25628b0 2023-11-16T16:52:55.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.492+0000 7f1d25ffb700 1 -- 172.21.15.44:0/2040027816 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d38008e50 con 0x559ca25628b0 2023-11-16T16:52:55.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.492+0000 7f1d5860fb80 10 client.16305 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:55.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.492+0000 7f1d5860fb80 1 -- 172.21.15.44:0/2040027816 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x559ca2475190 con 0x559ca25628b0 2023-11-16T16:52:55.498 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.492+0000 7f1d25ffb700 1 -- 172.21.15.44:0/2040027816 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559ca2550b20 con 0x559ca25628b0 2023-11-16T16:52:55.498 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.492+0000 7f1d25ffb700 1 -- 172.21.15.44:0/2040027816 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1d38023040 con 0x559ca25628b0 2023-11-16T16:52:55.498 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.492+0000 7f1d5860fb80 10 client.16305 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:55.498 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.492+0000 7f1d5860fb80 10 client.16305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:55.498 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.492+0000 7f1d5860fb80 20 client.16305 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:55.498 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.492+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/2040027816 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559ca2577890 0x559ca2579ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.498 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.492+0000 7f1d5860fb80 4 client.16305 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:55.498 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.492+0000 7f1d5860fb80 1 -- 172.21.15.44:0/2040027816 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559ca24710a0 con 0x559ca2577890 2023-11-16T16:52:55.498 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.493+0000 7f1d3e59c700 1 --2- 172.21.15.44:0/2040027816 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559ca2577890 0x559ca2579ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.498 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.493+0000 7f1d3e59c700 1 --2- 172.21.15.44:0/2040027816 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559ca2577890 0x559ca2579ca0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5c93f2da783e705 server_cookie=b77b3a51187917cf in_seq=0 out_seq=0 2023-11-16T16:52:55.498 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.493+0000 7f1d25ffb700 10 client.16305 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:55.498 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.494+0000 7f1d25ffb700 1 -- 172.21.15.44:0/2040027816 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559ca24710a0 con 0x559ca2577890 2023-11-16T16:52:55.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.494+0000 7f1d25ffb700 10 client.16305 handle_command_reply: tid=0 2023-11-16T16:52:55.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.494+0000 7f1d5860fb80 1 client.16305 shutdown 2023-11-16T16:52:55.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.494+0000 7f1d5860fb80 1 -- 172.21.15.44:0/2040027816 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca25628b0 msgr2=0x559ca25679f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.494+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/2040027816 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca25628b0 0x559ca25679f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1d38009920 tx=0x7f1d38009e10 comp rx=0 tx=0).stop 2023-11-16T16:52:55.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.494+0000 7f1d5860fb80 1 -- 172.21.15.44:0/2040027816 shutdown_connections 2023-11-16T16:52:55.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.494+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/2040027816 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559ca2577890 0x559ca2579ca0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.494+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/2040027816 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca25628b0 0x559ca25679f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.494+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/2040027816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca255db60 0x559ca256a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.494+0000 7f1d25ffb700 0 client.16305 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:55.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.494+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/2040027816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255e470 0x559ca2567350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.494+0000 7f1d5860fb80 1 -- 172.21.15.44:0/2040027816 >> 172.21.15.44:0/2040027816 conn(0x559ca2553360 msgr2=0x559ca2563d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:55.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.494+0000 7f1d5860fb80 1 -- 172.21.15.44:0/2040027816 shutdown_connections 2023-11-16T16:52:55.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.494+0000 7f1d5860fb80 1 -- 172.21.15.44:0/2040027816 wait complete. 2023-11-16T16:52:55.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.494+0000 7f1d5860fb80 20 client.16305 trim_cache size 0 max 16384 2023-11-16T16:52:55.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.512+0000 7f1d5860fb80 1 Processor -- start 2023-11-16T16:52:55.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.512+0000 7f1d5860fb80 1 -- start start 2023-11-16T16:52:55.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.512+0000 7f1d5860fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255db60 0x559ca25601a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.512+0000 7f1d5860fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca255e470 0x559ca2569c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.512+0000 7f1d5860fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca25628b0 0x559ca25618f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.512+0000 7f1d5860fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ca256a730 con 0x559ca255e470 2023-11-16T16:52:55.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.512+0000 7f1d5860fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ca25742c0 con 0x559ca25628b0 2023-11-16T16:52:55.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.512+0000 7f1d5860fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ca255e970 con 0x559ca255db60 2023-11-16T16:52:55.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.513+0000 7f1d3e59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255db60 0x559ca25601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.513+0000 7f1d3e59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255db60 0x559ca25601a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:33104/0 (socket says 172.21.15.44:33104) 2023-11-16T16:52:55.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.513+0000 7f1d3e59c700 1 -- 172.21.15.44:0/1994355175 learned_addr learned my addr 172.21.15.44:0/1994355175 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:55.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.513+0000 7f1d37fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca255e470 0x559ca2569c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.513+0000 7f1d3ed9d700 1 --2- 172.21.15.44:0/1994355175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca25628b0 0x559ca25618f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.513+0000 7f1d3e59c700 1 -- 172.21.15.44:0/1994355175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca25628b0 msgr2=0x559ca25618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:55.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.513+0000 7f1d3e59c700 1 --2- 172.21.15.44:0/1994355175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca25628b0 0x559ca25618f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.513+0000 7f1d3e59c700 1 -- 172.21.15.44:0/1994355175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca255e470 msgr2=0x559ca2569c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:55.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.513+0000 7f1d3e59c700 1 --2- 172.21.15.44:0/1994355175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca255e470 0x559ca2569c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.513+0000 7f1d3e59c700 1 -- 172.21.15.44:0/1994355175 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ca2562f50 con 0x559ca255db60 2023-11-16T16:52:55.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.513+0000 7f1d3e59c700 1 --2- 172.21.15.44:0/1994355175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255db60 0x559ca25601a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1d3009ac00 tx=0x7f1d3001a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d13d556a5e739767 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:55.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.513+0000 7f1d357fa700 1 -- 172.21.15.44:0/1994355175 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3009eca0 con 0x559ca255db60 2023-11-16T16:52:55.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.514+0000 7f1d357fa700 1 -- 172.21.15.44:0/1994355175 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d300b2070 con 0x559ca255db60 2023-11-16T16:52:55.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.514+0000 7f1d357fa700 1 -- 172.21.15.44:0/1994355175 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d300b33d0 con 0x559ca255db60 2023-11-16T16:52:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.514+0000 7f1d5860fb80 1 -- 172.21.15.44:0/1994355175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255db60 msgr2=0x559ca25601a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.514+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/1994355175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255db60 0x559ca25601a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1d3009ac00 tx=0x7f1d3001a040 comp rx=0 tx=0).stop 2023-11-16T16:52:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.514+0000 7f1d5860fb80 1 -- 172.21.15.44:0/1994355175 shutdown_connections 2023-11-16T16:52:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.514+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/1994355175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca25628b0 0x559ca25618f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.514+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/1994355175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255db60 0x559ca25601a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.514+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/1994355175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca255e470 0x559ca2569c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.514+0000 7f1d5860fb80 1 -- 172.21.15.44:0/1994355175 >> 172.21.15.44:0/1994355175 conn(0x559ca2553360 msgr2=0x559ca2560a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.514+0000 7f1d5860fb80 1 -- 172.21.15.44:0/1994355175 shutdown_connections 2023-11-16T16:52:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.514+0000 7f1d5860fb80 1 -- 172.21.15.44:0/1994355175 wait complete. 2023-11-16T16:52:55.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.515+0000 7f1d5860fb80 1 -- start start 2023-11-16T16:52:55.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.515+0000 7f1d5860fb80 1 Processor -- start 2023-11-16T16:52:55.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.516+0000 7f1d5860fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca255db60 0x559ca256e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.516+0000 7f1d5860fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255e470 0x559ca256e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.516+0000 7f1d5860fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca25628b0 0x559ca256c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.516+0000 7f1d5860fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ca256d0a0 con 0x559ca255db60 2023-11-16T16:52:55.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.516+0000 7f1d5860fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ca256d210 con 0x559ca25628b0 2023-11-16T16:52:55.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.516+0000 7f1d5860fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ca256d3d0 con 0x559ca255e470 2023-11-16T16:52:55.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.516+0000 7f1d5860fb80 10 client.0 _ll_register_callbacks cb 0x7f1d380086a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:55.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.516+0000 7f1d3e59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca255db60 0x559ca256e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.516+0000 7f1d3ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca25628b0 0x559ca256c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.516+0000 7f1d3e59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca255db60 0x559ca256e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33284/0 (socket says 172.21.15.44:33284) 2023-11-16T16:52:55.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.516+0000 7f1d3e59c700 1 -- 172.21.15.44:0/2192910775 learned_addr learned my addr 172.21.15.44:0/2192910775 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:52:55.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.516+0000 7f1d37fff700 1 --2- 172.21.15.44:0/2192910775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255e470 0x559ca256e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.517+0000 7f1d3e59c700 1 -- 172.21.15.44:0/2192910775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255e470 msgr2=0x559ca256e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:55.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.517+0000 7f1d3e59c700 1 --2- 172.21.15.44:0/2192910775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255e470 0x559ca256e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.517+0000 7f1d3e59c700 1 -- 172.21.15.44:0/2192910775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca25628b0 msgr2=0x559ca256c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:55.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.517+0000 7f1d3e59c700 1 --2- 172.21.15.44:0/2192910775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca25628b0 0x559ca256c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.517+0000 7f1d3e59c700 1 -- 172.21.15.44:0/2192910775 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d40110240 con 0x559ca255db60 2023-11-16T16:52:55.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.517+0000 7f1d3e59c700 1 --2- 172.21.15.44:0/2192910775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca255db60 0x559ca256e420 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1d300ca830 tx=0x7f1d300a3030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1319514f79ed84b9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:55.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.517+0000 7f1d25ffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:52:55.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.517+0000 7f1d25ffb700 1 -- 172.21.15.44:0/2192910775 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d30022ad0 con 0x559ca255db60 2023-11-16T16:52:55.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.517+0000 7f1d25ffb700 1 -- 172.21.15.44:0/2192910775 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f1d3009eca0 con 0x559ca255db60 2023-11-16T16:52:55.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.517+0000 7f1d5860fb80 1 -- 172.21.15.44:0/2192910775 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1d3800f040 con 0x559ca255db60 2023-11-16T16:52:55.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.517+0000 7f1d25ffb700 1 -- 172.21.15.44:0/2192910775 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d300b3170 con 0x559ca255db60 2023-11-16T16:52:55.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.517+0000 7f1d5860fb80 10 client.16311 fetch_fsmap learned FSMap version 21 2023-11-16T16:52:55.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.517+0000 7f1d5860fb80 1 -- 172.21.15.44:0/2192910775 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x559ca22dcde0 con 0x559ca255db60 2023-11-16T16:52:55.522 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.518+0000 7f1d25ffb700 1 -- 172.21.15.44:0/2192910775 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d3800f040 con 0x559ca255db60 2023-11-16T16:52:55.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.518+0000 7f1d25ffb700 1 -- 172.21.15.44:0/2192910775 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1d300a2ba0 con 0x559ca255db60 2023-11-16T16:52:55.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.518+0000 7f1d5860fb80 10 client.16311 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:52:55.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.518+0000 7f1d5860fb80 10 client.16311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:55.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.518+0000 7f1d5860fb80 20 client.16311 populate_metadata read hostname 'smithi044' 2023-11-16T16:52:55.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.518+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/2192910775 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559ca25cc500 0x559ca25ce910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:55.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.518+0000 7f1d5860fb80 4 client.16311 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:52:55.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.518+0000 7f1d5860fb80 1 -- 172.21.15.44:0/2192910775 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559ca247d660 con 0x559ca25cc500 2023-11-16T16:52:55.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.518+0000 7f1d37fff700 1 --2- 172.21.15.44:0/2192910775 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559ca25cc500 0x559ca25ce910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:55.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.519+0000 7f1d25ffb700 10 client.16311 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:55.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.519+0000 7f1d37fff700 1 --2- 172.21.15.44:0/2192910775 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559ca25cc500 0x559ca25ce910 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc742ea56d78016e server_cookie=700c43bcaa36c52a in_seq=0 out_seq=0 2023-11-16T16:52:55.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.519+0000 7f1d25ffb700 1 -- 172.21.15.44:0/2192910775 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x559ca247d660 con 0x559ca25cc500 2023-11-16T16:52:55.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.519+0000 7f1d25ffb700 10 client.16311 handle_command_reply: tid=0 2023-11-16T16:52:55.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.519+0000 7f1d5860fb80 1 client.16311 shutdown 2023-11-16T16:52:55.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.519+0000 7f1d5860fb80 1 -- 172.21.15.44:0/2192910775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca255db60 msgr2=0x559ca256e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.519+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/2192910775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca255db60 0x559ca256e420 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1d300ca830 tx=0x7f1d300a3030 comp rx=0 tx=0).stop 2023-11-16T16:52:55.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.519+0000 7f1d5860fb80 1 -- 172.21.15.44:0/2192910775 shutdown_connections 2023-11-16T16:52:55.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.519+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/2192910775 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559ca25cc500 0x559ca25ce910 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.519+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/2192910775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ca25628b0 0x559ca256c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.519+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/2192910775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ca255db60 0x559ca256e420 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.519+0000 7f1d25ffb700 0 client.16311 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:52:55.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.519+0000 7f1d5860fb80 1 --2- 172.21.15.44:0/2192910775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559ca255e470 0x559ca256e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.519+0000 7f1d5860fb80 1 -- 172.21.15.44:0/2192910775 >> 172.21.15.44:0/2192910775 conn(0x559ca2553360 msgr2=0x559ca25618f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:55.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.519+0000 7f1d5860fb80 1 -- 172.21.15.44:0/2192910775 shutdown_connections 2023-11-16T16:52:55.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.519+0000 7f1d5860fb80 1 -- 172.21.15.44:0/2192910775 wait complete. 2023-11-16T16:52:55.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.520+0000 7f1d5860fb80 20 client.16311 trim_cache size 0 max 16384 2023-11-16T16:52:55.525 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:52:55.525 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2023-11-16T16:52:55.525 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2023-11-16T16:52:55.525 INFO:teuthology.orchestra.run.smithi044.stdout: "abf214e5-acc2-4e8d-a6d4-d53613037095": { 2023-11-16T16:52:55.525 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2023-11-16T16:52:55.525 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "abf214e5-acc2-4e8d-a6d4-d53613037095", 2023-11-16T16:52:55.526 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2023-11-16T16:52:55.526 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:52:55.526 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:52:55.526 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:52:55.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.520+0000 7f1d44bff700 1 -- 172.21.15.44:0/2770603249 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1d2c0a1340 msgr2=0x7f1d2c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.520+0000 7f1d44bff700 1 --2- 172.21.15.44:0/2770603249 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1d2c0a1340 0x7f1d2c0a37f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1d400745e0 tx=0x7f1d28009260 comp rx=0 tx=0).stop 2023-11-16T16:52:55.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.520+0000 7f1d44bff700 1 -- 172.21.15.44:0/2770603249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d40100cc0 msgr2=0x7f1d40072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.520+0000 7f1d44bff700 1 --2- 172.21.15.44:0/2770603249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d40100cc0 0x7f1d40072df0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1d3000b580 tx=0x7f1d3000f690 comp rx=0 tx=0).stop 2023-11-16T16:52:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.520+0000 7f1d44bff700 1 -- 172.21.15.44:0/2770603249 shutdown_connections 2023-11-16T16:52:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.520+0000 7f1d44bff700 1 --2- 172.21.15.44:0/2770603249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d40105620 0x7f1d401a4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.520+0000 7f1d44bff700 1 --2- 172.21.15.44:0/2770603249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d40100cc0 0x7f1d40072df0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.520+0000 7f1d44bff700 1 --2- 172.21.15.44:0/2770603249 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1d2c0a1340 0x7f1d2c0a37f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.520+0000 7f1d44bff700 1 --2- 172.21.15.44:0/2770603249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d40101660 0x7f1d40073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.520+0000 7f1d44bff700 1 -- 172.21.15.44:0/2770603249 >> 172.21.15.44:0/2770603249 conn(0x7f1d40077560 msgr2=0x7f1d400feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.520+0000 7f1d44bff700 1 -- 172.21.15.44:0/2770603249 shutdown_connections 2023-11-16T16:52:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:52:55.520+0000 7f1d44bff700 1 -- 172.21.15.44:0/2770603249 wait complete. 2023-11-16T16:52:55.541 INFO:tasks.cephfs.filesystem:scrub status for tag:abf214e5-acc2-4e8d-a6d4-d53613037095 - {'path': '/', 'tag': 'abf214e5-acc2-4e8d-a6d4-d53613037095', 'options': 'recursive,force'} 2023-11-16T16:52:55.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:55 smithi044 ceph-mon[128750]: pgmap v92: 129 pgs: 129 active+clean; 3.3 MiB data, 90 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 KiB/s wr, 5 op/s 2023-11-16T16:52:55.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:55 smithi077 ceph-mon[131204]: pgmap v92: 129 pgs: 129 active+clean; 3.3 MiB data, 90 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 KiB/s wr, 5 op/s 2023-11-16T16:52:55.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:55 smithi176 ceph-mon[131630]: pgmap v92: 129 pgs: 129 active+clean; 3.3 MiB data, 90 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 KiB/s wr, 5 op/s 2023-11-16T16:52:56.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:56 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:52:56.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:56 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:56.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:56 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:52:56.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:56 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:56.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:56 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:52:56.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:56 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:56.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:56 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:52:56.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:56 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:56.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:56 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:52:56.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:56 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:56.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:56 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:52:56.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:56 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:52:57.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:57 smithi077 ceph-mon[131204]: pgmap v93: 129 pgs: 129 active+clean; 3.3 MiB data, 90 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 KiB/s wr, 5 op/s 2023-11-16T16:52:57.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:57 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:52:57.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:57 smithi044 ceph-mon[128750]: pgmap v93: 129 pgs: 129 active+clean; 3.3 MiB data, 90 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 KiB/s wr, 5 op/s 2023-11-16T16:52:57.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:57 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:52:57.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:57 smithi176 ceph-mon[131630]: pgmap v93: 129 pgs: 129 active+clean; 3.3 MiB data, 90 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 KiB/s wr, 5 op/s 2023-11-16T16:52:57.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:57 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:52:59.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:52:59 smithi077 ceph-mon[131204]: pgmap v94: 129 pgs: 129 active+clean; 3.4 MiB data, 91 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 39 KiB/s wr, 17 op/s 2023-11-16T16:52:59.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:52:59 smithi044 ceph-mon[128750]: pgmap v94: 129 pgs: 129 active+clean; 3.4 MiB data, 91 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 39 KiB/s wr, 17 op/s 2023-11-16T16:52:59.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:52:59 smithi176 ceph-mon[131630]: pgmap v94: 129 pgs: 129 active+clean; 3.4 MiB data, 91 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 39 KiB/s wr, 17 op/s 2023-11-16T16:53:01.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:01 smithi077 ceph-mon[131204]: pgmap v95: 129 pgs: 129 active+clean; 3.4 MiB data, 91 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 28 KiB/s wr, 14 op/s 2023-11-16T16:53:01.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:01 smithi044 ceph-mon[128750]: pgmap v95: 129 pgs: 129 active+clean; 3.4 MiB data, 91 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 28 KiB/s wr, 14 op/s 2023-11-16T16:53:01.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:01 smithi176 ceph-mon[131630]: pgmap v95: 129 pgs: 129 active+clean; 3.4 MiB data, 91 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 28 KiB/s wr, 14 op/s 2023-11-16T16:53:03.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:03 smithi077 ceph-mon[131204]: pgmap v96: 129 pgs: 129 active+clean; 17 MiB data, 100 MiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 1.2 MiB/s wr, 40 op/s 2023-11-16T16:53:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:03 smithi044 ceph-mon[128750]: pgmap v96: 129 pgs: 129 active+clean; 17 MiB data, 100 MiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 1.2 MiB/s wr, 40 op/s 2023-11-16T16:53:03.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:03 smithi176 ceph-mon[131630]: pgmap v96: 129 pgs: 129 active+clean; 17 MiB data, 100 MiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 1.2 MiB/s wr, 40 op/s 2023-11-16T16:53:05.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:05 smithi077 ceph-mon[131204]: pgmap v97: 129 pgs: 129 active+clean; 17 MiB data, 100 MiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 1.2 MiB/s wr, 37 op/s 2023-11-16T16:53:05.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:05 smithi044 ceph-mon[128750]: pgmap v97: 129 pgs: 129 active+clean; 17 MiB data, 100 MiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 1.2 MiB/s wr, 37 op/s 2023-11-16T16:53:05.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:05 smithi176 ceph-mon[131630]: pgmap v97: 129 pgs: 129 active+clean; 17 MiB data, 100 MiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 1.2 MiB/s wr, 37 op/s 2023-11-16T16:53:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.728+0000 7fc4fe5ff700 1 -- 172.21.15.44:0/3859797541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4f80ffb40 msgr2=0x7fc4f8104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.728+0000 7fc4fe5ff700 1 --2- 172.21.15.44:0/3859797541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4f80ffb40 0x7fc4f8104010 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec009990 tx=0x7fc4ec009ca0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.728+0000 7fc4fe5ff700 1 -- 172.21.15.44:0/3859797541 shutdown_connections 2023-11-16T16:53:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.728+0000 7fc4fe5ff700 1 --2- 172.21.15.44:0/3859797541 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4f8104630 0x7fc4f8104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.728+0000 7fc4fe5ff700 1 --2- 172.21.15.44:0/3859797541 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4f80ff230 0x7fc4f80ff600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.728+0000 7fc4fe5ff700 1 --2- 172.21.15.44:0/3859797541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4f80ffb40 0x7fc4f8104010 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.728+0000 7fc4fe5ff700 1 -- 172.21.15.44:0/3859797541 >> 172.21.15.44:0/3859797541 conn(0x7fc4f80763a0 msgr2=0x7fc4f80767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.729+0000 7fc4fe5ff700 1 -- 172.21.15.44:0/3859797541 shutdown_connections 2023-11-16T16:53:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.730+0000 7fc4fe5ff700 1 -- 172.21.15.44:0/3859797541 wait complete. 2023-11-16T16:53:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.731+0000 7fc4fe5ff700 1 Processor -- start 2023-11-16T16:53:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.732+0000 7fc4fe5ff700 1 -- start start 2023-11-16T16:53:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.733+0000 7fc4fe5ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4f80ff230 0x7fc4f81a28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.733+0000 7fc4fe5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4f80ffb40 0x7fc4f81a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.733+0000 7fc4fe5ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4f8104630 0x7fc4f819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.733+0000 7fc4fe5ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f819cfd0 con 0x7fc4f8104630 2023-11-16T16:53:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.733+0000 7fc4fe5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f819d140 con 0x7fc4f80ffb40 2023-11-16T16:53:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.733+0000 7fc4fe5ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f819d300 con 0x7fc4f80ff230 2023-11-16T16:53:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.733+0000 7fc4f77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4f80ffb40 0x7fc4f81a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.733+0000 7fc4f77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4f80ffb40 0x7fc4f81a2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:39852/0 (socket says 172.21.15.44:39852) 2023-11-16T16:53:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.733+0000 7fc4f77fe700 1 -- 172.21.15.44:0/2284259735 learned_addr learned my addr 172.21.15.44:0/2284259735 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.733+0000 7fc4f7fff700 1 --2- 172.21.15.44:0/2284259735 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4f80ff230 0x7fc4f81a28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.734+0000 7fc4fcb9c700 1 --2- 172.21.15.44:0/2284259735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4f8104630 0x7fc4f819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.734+0000 7fc4f77fe700 1 -- 172.21.15.44:0/2284259735 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4f80ff230 msgr2=0x7fc4f81a28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.734+0000 7fc4f77fe700 1 --2- 172.21.15.44:0/2284259735 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4f80ff230 0x7fc4f81a28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.734+0000 7fc4f77fe700 1 -- 172.21.15.44:0/2284259735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4f8104630 msgr2=0x7fc4f819c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.734+0000 7fc4f77fe700 1 --2- 172.21.15.44:0/2284259735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4f8104630 0x7fc4f819c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.734+0000 7fc4f77fe700 1 -- 172.21.15.44:0/2284259735 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec0095f0 con 0x7fc4f80ffb40 2023-11-16T16:53:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.734+0000 7fc4f77fe700 1 --2- 172.21.15.44:0/2284259735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4f80ffb40 0x7fc4f81a2e30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec003cd0 tx=0x7fc4ec003db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.735+0000 7fc4f57fa700 1 -- 172.21.15.44:0/2284259735 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec01d070 con 0x7fc4f80ffb40 2023-11-16T16:53:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.735+0000 7fc4f57fa700 1 -- 172.21.15.44:0/2284259735 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4ec00f460 con 0x7fc4f80ffb40 2023-11-16T16:53:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.735+0000 7fc4f57fa700 1 -- 172.21.15.44:0/2284259735 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec017650 con 0x7fc4f80ffb40 2023-11-16T16:53:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.735+0000 7fc4fe5ff700 1 -- 172.21.15.44:0/2284259735 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f8116990 con 0x7fc4f80ffb40 2023-11-16T16:53:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.735+0000 7fc4fe5ff700 1 -- 172.21.15.44:0/2284259735 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f8116800 con 0x7fc4f80ffb40 2023-11-16T16:53:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.741+0000 7fc4f57fa700 1 -- 172.21.15.44:0/2284259735 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fc4ec00f5d0 con 0x7fc4f80ffb40 2023-11-16T16:53:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.741+0000 7fc511e7fb80 1 Processor -- start 2023-11-16T16:53:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.742+0000 7fc511e7fb80 1 -- start start 2023-11-16T16:53:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.742+0000 7fc511e7fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f382016b0 0x555f383ab1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.742+0000 7fc511e7fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f383ab720 0x555f383a5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.742+0000 7fc511e7fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383abb90 0x555f383a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.742+0000 7fc511e7fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f382ab000 con 0x555f382016b0 2023-11-16T16:53:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.742+0000 7fc511e7fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f382fd840 con 0x555f383abb90 2023-11-16T16:53:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.742+0000 7fc511e7fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f382f6cd0 con 0x555f383ab720 2023-11-16T16:53:06.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.743+0000 7fc4fcb9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f382016b0 0x555f383ab1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.743+0000 7fc4fcb9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f382016b0 0x555f383ab1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42722/0 (socket says 172.21.15.44:42722) 2023-11-16T16:53:06.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.743+0000 7fc4fcb9c700 1 -- 172.21.15.44:0/3396623065 learned_addr learned my addr 172.21.15.44:0/3396623065 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:06.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.743+0000 7fc4f7fff700 1 --2- 172.21.15.44:0/3396623065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f383ab720 0x555f383a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.743+0000 7fc4fcb9c700 1 -- 172.21.15.44:0/3396623065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f383ab720 msgr2=0x555f383a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:06.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.743+0000 7fc4fcb9c700 1 --2- 172.21.15.44:0/3396623065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f383ab720 0x555f383a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.743+0000 7fc4fcb9c700 1 -- 172.21.15.44:0/3396623065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383abb90 msgr2=0x555f383a5b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:53:06.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.743+0000 7fc4fcb9c700 1 --2- 172.21.15.44:0/3396623065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383abb90 0x555f383a5b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.743+0000 7fc4fcb9c700 1 -- 172.21.15.44:0/3396623065 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f381785d0 con 0x555f382016b0 2023-11-16T16:53:06.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.743+0000 7fc4f57fa700 1 --2- 172.21.15.44:0/2284259735 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc4e00a1400 0x7fc4e00a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.744+0000 7fc4fcb9c700 1 --2- 172.21.15.44:0/3396623065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f382016b0 0x555f383ab1e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0007430 tx=0x7fc4f0007740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e11ac777c5eca9a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:06.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.744+0000 7fc4f57fa700 1 -- 172.21.15.44:0/2284259735 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fc4ec0ce340 con 0x7fc4f80ffb40 2023-11-16T16:53:06.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.744+0000 7fc4f7fff700 1 --2- 172.21.15.44:0/2284259735 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc4e00a1400 0x7fc4e00a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.744+0000 7fc4de7fc700 1 -- 172.21.15.44:0/3396623065 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0011710 con 0x555f382016b0 2023-11-16T16:53:06.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.744+0000 7fc4de7fc700 1 -- 172.21.15.44:0/3396623065 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4f0007e70 con 0x555f382016b0 2023-11-16T16:53:06.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.744+0000 7fc4de7fc700 1 -- 172.21.15.44:0/3396623065 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0013cf0 con 0x555f382016b0 2023-11-16T16:53:06.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.745+0000 7fc511e7fb80 1 -- 172.21.15.44:0/3396623065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f382016b0 msgr2=0x555f383ab1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:06.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.745+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/3396623065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f382016b0 0x555f383ab1e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0007430 tx=0x7fc4f0007740 comp rx=0 tx=0).stop 2023-11-16T16:53:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.745+0000 7fc511e7fb80 1 -- 172.21.15.44:0/3396623065 shutdown_connections 2023-11-16T16:53:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.745+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/3396623065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383abb90 0x555f383a5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.745+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/3396623065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f382016b0 0x555f383ab1e0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.745+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/3396623065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f383ab720 0x555f383a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.745+0000 7fc511e7fb80 1 -- 172.21.15.44:0/3396623065 >> 172.21.15.44:0/3396623065 conn(0x555f3839af50 msgr2=0x555f383a0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:06.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.746+0000 7fc511e7fb80 1 -- 172.21.15.44:0/3396623065 shutdown_connections 2023-11-16T16:53:06.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.747+0000 7fc511e7fb80 1 -- 172.21.15.44:0/3396623065 wait complete. 2023-11-16T16:53:06.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.747+0000 7fc4f7fff700 1 --2- 172.21.15.44:0/2284259735 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc4e00a1400 0x7fc4e00a38b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8006fd0 tx=0x7fc4e8008040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:06.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.747+0000 7fc511e7fb80 1 -- start start 2023-11-16T16:53:06.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.748+0000 7fc511e7fb80 1 Processor -- start 2023-11-16T16:53:06.750 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.749+0000 7fc511e7fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f382016b0 0x555f383aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.750 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.749+0000 7fc511e7fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383ab720 0x555f383af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.750 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.749+0000 7fc511e7fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f383abb90 0x555f383af920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.749+0000 7fc511e7fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f38298880 con 0x555f383abb90 2023-11-16T16:53:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.749+0000 7fc511e7fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f38298520 con 0x555f383ab720 2023-11-16T16:53:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.749+0000 7fc511e7fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f3817a8c0 con 0x555f382016b0 2023-11-16T16:53:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.749+0000 7fc511e7fb80 10 client.0 _ll_register_callbacks cb 0x555f382b6720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.750+0000 7fc4f7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f382016b0 0x555f383aeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.750+0000 7fc4f77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383ab720 0x555f383af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.750+0000 7fc4fcb9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f383abb90 0x555f383af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.750+0000 7fc4f7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f382016b0 0x555f383aeea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:39780/0 (socket says 172.21.15.44:39780) 2023-11-16T16:53:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.750+0000 7fc4f7fff700 1 -- 172.21.15.44:0/820306545 learned_addr learned my addr 172.21.15.44:0/820306545 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.750+0000 7fc4f7fff700 1 -- 172.21.15.44:0/820306545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383ab720 msgr2=0x555f383af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.750+0000 7fc4f7fff700 1 --2- 172.21.15.44:0/820306545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383ab720 0x555f383af3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.750+0000 7fc4f7fff700 1 -- 172.21.15.44:0/820306545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f383abb90 msgr2=0x555f383af920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.750+0000 7fc4f7fff700 1 --2- 172.21.15.44:0/820306545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f383abb90 0x555f383af920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.750+0000 7fc4f7fff700 1 -- 172.21.15.44:0/820306545 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f0008ce0 con 0x555f382016b0 2023-11-16T16:53:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.751+0000 7fc4f7fff700 1 --2- 172.21.15.44:0/820306545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f382016b0 0x555f383aeea0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc4e800ff30 tx=0x7fc4e8004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41a4194af261b8f6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.751+0000 7fc4d77fe700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:53:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.751+0000 7fc4d77fe700 1 -- 172.21.15.44:0/820306545 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8020440 con 0x555f382016b0 2023-11-16T16:53:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.751+0000 7fc4d77fe700 1 -- 172.21.15.44:0/820306545 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fc4e80205a0 con 0x555f382016b0 2023-11-16T16:53:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.751+0000 7fc511e7fb80 1 -- 172.21.15.44:0/820306545 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555f38398710 con 0x555f382016b0 2023-11-16T16:53:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.752+0000 7fc4d77fe700 1 -- 172.21.15.44:0/820306545 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4e80207a0 con 0x555f382016b0 2023-11-16T16:53:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.752+0000 7fc511e7fb80 10 client.26335 fetch_fsmap learned FSMap version 21 2023-11-16T16:53:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.752+0000 7fc511e7fb80 1 -- 172.21.15.44:0/820306545 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x555f382bd130 con 0x555f382016b0 2023-11-16T16:53:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.752+0000 7fc4d77fe700 1 -- 172.21.15.44:0/820306545 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555f38398710 con 0x555f382016b0 2023-11-16T16:53:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.753+0000 7fc4d77fe700 1 -- 172.21.15.44:0/820306545 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc4e80158e0 con 0x555f382016b0 2023-11-16T16:53:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.753+0000 7fc511e7fb80 10 client.26335 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:53:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.753+0000 7fc511e7fb80 10 client.26335 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.753+0000 7fc511e7fb80 20 client.26335 populate_metadata read hostname 'smithi044' 2023-11-16T16:53:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.753+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/820306545 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x555f383bf100 0x555f383c1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.753+0000 7fc511e7fb80 4 client.26335 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:53:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.753+0000 7fc511e7fb80 1 -- 172.21.15.44:0/820306545 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555f382b9040 con 0x555f383bf100 2023-11-16T16:53:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.753+0000 7fc4f77fe700 1 --2- 172.21.15.44:0/820306545 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x555f383bf100 0x555f383c1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.754+0000 7fc4f77fe700 1 --2- 172.21.15.44:0/820306545 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x555f383bf100 0x555f383c1510 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29dbda819a5e1c2b server_cookie=9a1cbc8940cb45df in_seq=0 out_seq=0 2023-11-16T16:53:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.754+0000 7fc4d77fe700 10 client.26335 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.757+0000 7fc4d77fe700 1 -- 172.21.15.44:0/820306545 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555f382b9040 con 0x555f383bf100 2023-11-16T16:53:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.757+0000 7fc4d77fe700 10 client.26335 handle_command_reply: tid=0 2023-11-16T16:53:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.757+0000 7fc511e7fb80 1 client.26335 shutdown 2023-11-16T16:53:06.759 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.758+0000 7fc511e7fb80 1 -- 172.21.15.44:0/820306545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f382016b0 msgr2=0x555f383aeea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:06.760 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.758+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/820306545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f382016b0 0x555f383aeea0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc4e800ff30 tx=0x7fc4e8004750 comp rx=0 tx=0).stop 2023-11-16T16:53:06.760 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.758+0000 7fc511e7fb80 1 -- 172.21.15.44:0/820306545 shutdown_connections 2023-11-16T16:53:06.760 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.758+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/820306545 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x555f383bf100 0x555f383c1510 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.760 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.758+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/820306545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f383abb90 0x555f383af920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.760 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.758+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/820306545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f382016b0 0x555f383aeea0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.760 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.758+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/820306545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383ab720 0x555f383af3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.760 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.758+0000 7fc511e7fb80 1 -- 172.21.15.44:0/820306545 >> 172.21.15.44:0/820306545 conn(0x555f3839af50 msgr2=0x555f3839f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:06.760 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.758+0000 7fc4d77fe700 0 client.26335 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.759+0000 7fc511e7fb80 1 -- 172.21.15.44:0/820306545 shutdown_connections 2023-11-16T16:53:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.759+0000 7fc511e7fb80 1 -- 172.21.15.44:0/820306545 wait complete. 2023-11-16T16:53:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.760+0000 7fc511e7fb80 20 client.26335 trim_cache size 0 max 16384 2023-11-16T16:53:06.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.788+0000 7fc511e7fb80 1 Processor -- start 2023-11-16T16:53:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.788+0000 7fc511e7fb80 1 -- start start 2023-11-16T16:53:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.789+0000 7fc511e7fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f382016b0 0x555f383a5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.789+0000 7fc511e7fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f383ab720 0x555f383b6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.789+0000 7fc511e7fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383abb90 0x555f383af750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.789+0000 7fc511e7fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f383a0fb0 con 0x555f382016b0 2023-11-16T16:53:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.789+0000 7fc511e7fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f383a12b0 con 0x555f383abb90 2023-11-16T16:53:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.789+0000 7fc511e7fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f383bc7e0 con 0x555f383ab720 2023-11-16T16:53:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.789+0000 7fc4f7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f382016b0 0x555f383a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.789+0000 7fc4f7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f382016b0 0x555f383a5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42736/0 (socket says 172.21.15.44:42736) 2023-11-16T16:53:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.789+0000 7fc4f7fff700 1 -- 172.21.15.44:0/3650266059 learned_addr learned my addr 172.21.15.44:0/3650266059 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.789+0000 7fc4fcb9c700 1 --2- 172.21.15.44:0/3650266059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383abb90 0x555f383af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.790+0000 7fc4f77fe700 1 --2- 172.21.15.44:0/3650266059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f383ab720 0x555f383b6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.790+0000 7fc4f7fff700 1 -- 172.21.15.44:0/3650266059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f383ab720 msgr2=0x555f383b6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.790+0000 7fc4f7fff700 1 --2- 172.21.15.44:0/3650266059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f383ab720 0x555f383b6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.790+0000 7fc4f7fff700 1 -- 172.21.15.44:0/3650266059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383abb90 msgr2=0x555f383af750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.790+0000 7fc4f7fff700 1 --2- 172.21.15.44:0/3650266059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383abb90 0x555f383af750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.790+0000 7fc4f7fff700 1 -- 172.21.15.44:0/3650266059 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f3839c6f0 con 0x555f382016b0 2023-11-16T16:53:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.790+0000 7fc4f7fff700 1 --2- 172.21.15.44:0/3650266059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f382016b0 0x555f383a5560 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8006bd0 tx=0x7fc4e800bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e95229c2805c18 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.791+0000 7fc4deffd700 1 -- 172.21.15.44:0/3650266059 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8037070 con 0x555f382016b0 2023-11-16T16:53:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.791+0000 7fc4deffd700 1 -- 172.21.15.44:0/3650266059 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4e8019070 con 0x555f382016b0 2023-11-16T16:53:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.791+0000 7fc4deffd700 1 -- 172.21.15.44:0/3650266059 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e80207a0 con 0x555f382016b0 2023-11-16T16:53:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.792+0000 7fc511e7fb80 1 -- 172.21.15.44:0/3650266059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f382016b0 msgr2=0x555f383a5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.792+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/3650266059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f382016b0 0x555f383a5560 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8006bd0 tx=0x7fc4e800bf50 comp rx=0 tx=0).stop 2023-11-16T16:53:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.793+0000 7fc511e7fb80 1 -- 172.21.15.44:0/3650266059 shutdown_connections 2023-11-16T16:53:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.793+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/3650266059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383abb90 0x555f383af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.793+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/3650266059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f382016b0 0x555f383a5560 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.793+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/3650266059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f383ab720 0x555f383b6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.793+0000 7fc511e7fb80 1 -- 172.21.15.44:0/3650266059 >> 172.21.15.44:0/3650266059 conn(0x555f3839af50 msgr2=0x555f383bb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.794+0000 7fc511e7fb80 1 -- 172.21.15.44:0/3650266059 shutdown_connections 2023-11-16T16:53:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.795+0000 7fc511e7fb80 1 -- 172.21.15.44:0/3650266059 wait complete. 2023-11-16T16:53:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.795+0000 7fc511e7fb80 1 -- start start 2023-11-16T16:53:06.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.795+0000 7fc511e7fb80 1 Processor -- start 2023-11-16T16:53:06.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.796+0000 7fc511e7fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f382016b0 0x555f3840c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.796+0000 7fc511e7fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383ab720 0x555f3840c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.796+0000 7fc511e7fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f383abb90 0x555f3840af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.796+0000 7fc511e7fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f383b6460 con 0x555f383abb90 2023-11-16T16:53:06.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.796+0000 7fc511e7fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f383b65a0 con 0x555f383ab720 2023-11-16T16:53:06.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.796+0000 7fc511e7fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f3840b470 con 0x555f382016b0 2023-11-16T16:53:06.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.796+0000 7fc511e7fb80 10 client.0 _ll_register_callbacks cb 0x7fc4e800fb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:06.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.797+0000 7fc4f7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f382016b0 0x555f3840c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.797+0000 7fc4f7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f382016b0 0x555f3840c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:39806/0 (socket says 172.21.15.44:39806) 2023-11-16T16:53:06.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.797+0000 7fc4f7fff700 1 -- 172.21.15.44:0/2832642425 learned_addr learned my addr 172.21.15.44:0/2832642425 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:06.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.797+0000 7fc4f77fe700 1 --2- 172.21.15.44:0/2832642425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383ab720 0x555f3840c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.797+0000 7fc4fcb9c700 1 --2- 172.21.15.44:0/2832642425 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f383abb90 0x555f3840af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.797+0000 7fc4f7fff700 1 -- 172.21.15.44:0/2832642425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383ab720 msgr2=0x555f3840c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:06.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.797+0000 7fc4f7fff700 1 --2- 172.21.15.44:0/2832642425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383ab720 0x555f3840c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.797+0000 7fc4f7fff700 1 -- 172.21.15.44:0/2832642425 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f383abb90 msgr2=0x555f3840af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:06.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.797+0000 7fc4f7fff700 1 --2- 172.21.15.44:0/2832642425 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f383abb90 0x555f3840af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:06.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.797+0000 7fc4f7fff700 1 -- 172.21.15.44:0/2832642425 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f382bd130 con 0x555f382016b0 2023-11-16T16:53:06.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.798+0000 7fc4f7fff700 1 --2- 172.21.15.44:0/2832642425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f382016b0 0x555f3840c1d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8015cf0 tx=0x7fc4e8006d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d90a75811740fa8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:06.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.798+0000 7fc4d77fe700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:53:06.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.798+0000 7fc4fcb9c700 1 --2- 172.21.15.44:0/2832642425 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f383abb90 0x555f3840af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:53:06.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.798+0000 7fc4d77fe700 1 -- 172.21.15.44:0/2832642425 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8037040 con 0x555f382016b0 2023-11-16T16:53:06.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.798+0000 7fc4d77fe700 1 -- 172.21.15.44:0/2832642425 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fc4e8015820 con 0x555f382016b0 2023-11-16T16:53:06.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.798+0000 7fc511e7fb80 1 -- 172.21.15.44:0/2832642425 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4f0007e70 con 0x555f382016b0 2023-11-16T16:53:06.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.799+0000 7fc4d77fe700 1 -- 172.21.15.44:0/2832642425 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4e800b7c0 con 0x555f382016b0 2023-11-16T16:53:06.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.799+0000 7fc511e7fb80 10 client.26341 fetch_fsmap learned FSMap version 21 2023-11-16T16:53:06.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.799+0000 7fc511e7fb80 1 -- 172.21.15.44:0/2832642425 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x555f38124d80 con 0x555f382016b0 2023-11-16T16:53:06.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.799+0000 7fc4d77fe700 1 -- 172.21.15.44:0/2832642425 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4f0007e70 con 0x555f382016b0 2023-11-16T16:53:06.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.800+0000 7fc4d77fe700 1 -- 172.21.15.44:0/2832642425 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc4e8020500 con 0x555f382016b0 2023-11-16T16:53:06.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.800+0000 7fc511e7fb80 10 client.26341 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:53:06.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.800+0000 7fc511e7fb80 10 client.26341 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:06.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.800+0000 7fc511e7fb80 20 client.26341 populate_metadata read hostname 'smithi044' 2023-11-16T16:53:06.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.800+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/2832642425 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x555f384247f0 0x555f38426bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:06.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.800+0000 7fc511e7fb80 4 client.26341 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:53:06.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.800+0000 7fc511e7fb80 1 -- 172.21.15.44:0/2832642425 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555f382c5600 con 0x555f384247f0 2023-11-16T16:53:06.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.801+0000 7fc4f77fe700 1 --2- 172.21.15.44:0/2832642425 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x555f384247f0 0x555f38426bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:06.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.801+0000 7fc4f77fe700 1 --2- 172.21.15.44:0/2832642425 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x555f384247f0 0x555f38426bd0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=760bc6cf2625262a server_cookie=8e48421c9aba0f3a in_seq=0 out_seq=0 2023-11-16T16:53:06.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:06.801+0000 7fc4d77fe700 10 client.26341 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:07.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:07 smithi077 ceph-mon[131204]: pgmap v98: 129 pgs: 129 active+clean; 17 MiB data, 100 MiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 1.2 MiB/s wr, 37 op/s 2023-11-16T16:53:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:07 smithi044 ceph-mon[128750]: pgmap v98: 129 pgs: 129 active+clean; 17 MiB data, 100 MiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 1.2 MiB/s wr, 37 op/s 2023-11-16T16:53:07.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:07 smithi176 ceph-mon[131630]: pgmap v98: 129 pgs: 129 active+clean; 17 MiB data, 100 MiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 1.2 MiB/s wr, 37 op/s 2023-11-16T16:53:09.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:09 smithi077 ceph-mon[131204]: pgmap v99: 129 pgs: 129 active+clean; 32 MiB data, 130 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 107 op/s 2023-11-16T16:53:09.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:09 smithi044 ceph-mon[128750]: pgmap v99: 129 pgs: 129 active+clean; 32 MiB data, 130 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 107 op/s 2023-11-16T16:53:09.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:09 smithi176 ceph-mon[131630]: pgmap v99: 129 pgs: 129 active+clean; 32 MiB data, 130 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 107 op/s 2023-11-16T16:53:11.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.185+0000 7fc4d77fe700 1 -- 172.21.15.44:0/2832642425 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555f382c5600 con 0x555f384247f0 2023-11-16T16:53:11.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.185+0000 7fc4d77fe700 10 client.26341 handle_command_reply: tid=0 2023-11-16T16:53:11.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.185+0000 7fc511e7fb80 1 client.26341 shutdown 2023-11-16T16:53:11.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.185+0000 7fc511e7fb80 1 -- 172.21.15.44:0/2832642425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f382016b0 msgr2=0x555f3840c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:11.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.185+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/2832642425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f382016b0 0x555f3840c1d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8015cf0 tx=0x7fc4e8006d90 comp rx=0 tx=0).stop 2023-11-16T16:53:11.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.186+0000 7fc511e7fb80 1 -- 172.21.15.44:0/2832642425 shutdown_connections 2023-11-16T16:53:11.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.186+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/2832642425 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x555f384247f0 0x555f38426bd0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:11.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.186+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/2832642425 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f383abb90 0x555f3840af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:11.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.186+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/2832642425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x555f382016b0 0x555f3840c1d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:11.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.186+0000 7fc511e7fb80 1 --2- 172.21.15.44:0/2832642425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555f383ab720 0x555f3840c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:11.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.186+0000 7fc511e7fb80 1 -- 172.21.15.44:0/2832642425 >> 172.21.15.44:0/2832642425 conn(0x555f3839af50 msgr2=0x555f383ad670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:11.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.186+0000 7fc4d77fe700 0 client.26341 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:11.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.186+0000 7fc511e7fb80 1 -- 172.21.15.44:0/2832642425 shutdown_connections 2023-11-16T16:53:11.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.186+0000 7fc511e7fb80 1 -- 172.21.15.44:0/2832642425 wait complete. 2023-11-16T16:53:11.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.187+0000 7fc511e7fb80 20 client.26341 trim_cache size 0 max 16384 2023-11-16T16:53:11.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:53:11.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:53:11.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:53:11.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:53:11.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.187+0000 7fc4fe5ff700 1 -- 172.21.15.44:0/2284259735 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc4e00a1400 msgr2=0x7fc4e00a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:11.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.187+0000 7fc4fe5ff700 1 --2- 172.21.15.44:0/2284259735 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc4e00a1400 0x7fc4e00a38b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8006fd0 tx=0x7fc4e8008040 comp rx=0 tx=0).stop 2023-11-16T16:53:11.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.187+0000 7fc4fe5ff700 1 -- 172.21.15.44:0/2284259735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4f80ffb40 msgr2=0x7fc4f81a2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:11.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.187+0000 7fc4fe5ff700 1 --2- 172.21.15.44:0/2284259735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4f80ffb40 0x7fc4f81a2e30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec003cd0 tx=0x7fc4ec003db0 comp rx=0 tx=0).stop 2023-11-16T16:53:11.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.188+0000 7fc4fe5ff700 1 -- 172.21.15.44:0/2284259735 shutdown_connections 2023-11-16T16:53:11.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.188+0000 7fc4fe5ff700 1 --2- 172.21.15.44:0/2284259735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4f8104630 0x7fc4f819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:11.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.188+0000 7fc4fe5ff700 1 --2- 172.21.15.44:0/2284259735 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc4e00a1400 0x7fc4e00a38b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:11.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.188+0000 7fc4fe5ff700 1 --2- 172.21.15.44:0/2284259735 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4f80ff230 0x7fc4f81a28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:11.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.188+0000 7fc4fe5ff700 1 --2- 172.21.15.44:0/2284259735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4f80ffb40 0x7fc4f81a2e30 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:11.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.188+0000 7fc4fe5ff700 1 -- 172.21.15.44:0/2284259735 >> 172.21.15.44:0/2284259735 conn(0x7fc4f80763a0 msgr2=0x7fc4f8101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:11.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.188+0000 7fc4fe5ff700 1 -- 172.21.15.44:0/2284259735 shutdown_connections 2023-11-16T16:53:11.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:11.188+0000 7fc4fe5ff700 1 -- 172.21.15.44:0/2284259735 wait complete. 2023-11-16T16:53:11.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:11 smithi077 ceph-mon[131204]: pgmap v100: 129 pgs: 129 active+clean; 32 MiB data, 130 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 95 op/s 2023-11-16T16:53:11.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:11 smithi044 ceph-mon[128750]: pgmap v100: 129 pgs: 129 active+clean; 32 MiB data, 130 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 95 op/s 2023-11-16T16:53:11.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:11 smithi176 ceph-mon[131630]: pgmap v100: 129 pgs: 129 active+clean; 32 MiB data, 130 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 95 op/s 2023-11-16T16:53:13.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:13 smithi077 ceph-mon[131204]: pgmap v101: 129 pgs: 129 active+clean; 6.4 MiB data, 288 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 2.31k op/s 2023-11-16T16:53:13.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:13 smithi044 ceph-mon[128750]: pgmap v101: 129 pgs: 129 active+clean; 6.4 MiB data, 288 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 2.31k op/s 2023-11-16T16:53:13.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:13 smithi176 ceph-mon[131630]: pgmap v101: 129 pgs: 129 active+clean; 6.4 MiB data, 288 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 2.31k op/s 2023-11-16T16:53:14.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:14 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:53:14.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:14 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:53:14.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:14 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:53:15.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:15 smithi077 ceph-mon[131204]: pgmap v102: 129 pgs: 129 active+clean; 6.4 MiB data, 288 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 2.29k op/s 2023-11-16T16:53:15.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:15 smithi044 ceph-mon[128750]: pgmap v102: 129 pgs: 129 active+clean; 6.4 MiB data, 288 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 2.29k op/s 2023-11-16T16:53:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:15 smithi176 ceph-mon[131630]: pgmap v102: 129 pgs: 129 active+clean; 6.4 MiB data, 288 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 2.29k op/s 2023-11-16T16:53:17.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:17 smithi077 ceph-mon[131204]: pgmap v103: 129 pgs: 129 active+clean; 6.4 MiB data, 288 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 2.29k op/s 2023-11-16T16:53:17.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:17 smithi044 ceph-mon[128750]: pgmap v103: 129 pgs: 129 active+clean; 6.4 MiB data, 288 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 2.29k op/s 2023-11-16T16:53:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:17 smithi176 ceph-mon[131630]: pgmap v103: 129 pgs: 129 active+clean; 6.4 MiB data, 288 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 2.29k op/s 2023-11-16T16:53:19.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:19 smithi077 ceph-mon[131204]: pgmap v104: 129 pgs: 129 active+clean; 21 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 2.29k op/s 2023-11-16T16:53:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:19 smithi044 ceph-mon[128750]: pgmap v104: 129 pgs: 129 active+clean; 21 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 2.29k op/s 2023-11-16T16:53:19.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:19 smithi176 ceph-mon[131630]: pgmap v104: 129 pgs: 129 active+clean; 21 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 2.29k op/s 2023-11-16T16:53:20.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:20 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:53:20.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:20 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:53:20.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:20 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:53:20.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:20 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:53:20.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:20 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:53:20.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:20 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:53:20.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:20 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:53:20.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:20 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:53:20.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:20 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:53:21.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:21 smithi044 ceph-mon[128750]: pgmap v105: 129 pgs: 129 active+clean; 21 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 2.22k op/s 2023-11-16T16:53:21.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:21 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:53:21.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:21 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T16:53:21.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:21 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:53:21.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:21 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:53:21.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:21 smithi176 ceph-mon[131630]: pgmap v105: 129 pgs: 129 active+clean; 21 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 2.22k op/s 2023-11-16T16:53:21.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:21 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:53:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:21 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T16:53:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:21 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:53:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:21 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:53:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:21 smithi077 ceph-mon[131204]: pgmap v105: 129 pgs: 129 active+clean; 21 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 2.22k op/s 2023-11-16T16:53:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:21 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:53:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:21 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T16:53:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:21 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:53:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:21 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:53:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:23 smithi044 ceph-mon[128750]: pgmap v106: 129 pgs: 129 active+clean; 34 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 2.23k op/s 2023-11-16T16:53:23.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:23 smithi176 ceph-mon[131630]: pgmap v106: 129 pgs: 129 active+clean; 34 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 2.23k op/s 2023-11-16T16:53:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:23 smithi077 ceph-mon[131204]: pgmap v106: 129 pgs: 129 active+clean; 34 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 2.23k op/s 2023-11-16T16:53:24.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.375+0000 7f453edff700 1 -- 172.21.15.44:0/1149376240 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4538101d50 msgr2=0x7f453810a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:24.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.375+0000 7f453edff700 1 --2- 172.21.15.44:0/1149376240 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4538101d50 0x7f453810a240 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4524009990 tx=0x7f4524009ca0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.375+0000 7f453edff700 1 -- 172.21.15.44:0/1149376240 shutdown_connections 2023-11-16T16:53:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.375+0000 7f453edff700 1 --2- 172.21.15.44:0/1149376240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453810a810 0x7f453810cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.375+0000 7f453edff700 1 --2- 172.21.15.44:0/1149376240 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4538101440 0x7f4538101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.375+0000 7f453edff700 1 --2- 172.21.15.44:0/1149376240 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4538101d50 0x7f453810a240 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.375+0000 7f453edff700 1 -- 172.21.15.44:0/1149376240 >> 172.21.15.44:0/1149376240 conn(0x7f45380763a0 msgr2=0x7f45380767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:24.379 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.375+0000 7f453edff700 1 -- 172.21.15.44:0/1149376240 shutdown_connections 2023-11-16T16:53:24.379 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.376+0000 7f453edff700 1 -- 172.21.15.44:0/1149376240 wait complete. 2023-11-16T16:53:24.379 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.377+0000 7f453edff700 1 Processor -- start 2023-11-16T16:53:24.379 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.377+0000 7f453edff700 1 -- start start 2023-11-16T16:53:24.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.378+0000 7f453edff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4538101440 0x7f4538194150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.378+0000 7f453edff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4538101d50 0x7f4538194690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.378+0000 7f453edff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453810a810 0x7f4538198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.378+0000 7f453edff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4538199130 con 0x7f4538101d50 2023-11-16T16:53:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.378+0000 7f453edff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45381992a0 con 0x7f453810a810 2023-11-16T16:53:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.378+0000 7f453edff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4538199460 con 0x7f4538101440 2023-11-16T16:53:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.378+0000 7f453cb9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4538101440 0x7f4538194150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.378+0000 7f4537fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4538101d50 0x7f4538194690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.378+0000 7f453d39c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453810a810 0x7f4538198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.378+0000 7f453cb9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4538101440 0x7f4538194150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:46834/0 (socket says 172.21.15.44:46834) 2023-11-16T16:53:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.378+0000 7f4537fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4538101d50 0x7f4538194690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44124/0 (socket says 172.21.15.44:44124) 2023-11-16T16:53:24.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.378+0000 7f453d39c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453810a810 0x7f4538198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:47184/0 (socket says 172.21.15.44:47184) 2023-11-16T16:53:24.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.378+0000 7f453cb9b700 1 -- 172.21.15.44:0/2290188056 learned_addr learned my addr 172.21.15.44:0/2290188056 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:24.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.379+0000 7f453d39c700 1 -- 172.21.15.44:0/2290188056 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4538101440 msgr2=0x7f4538194150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:24.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.379+0000 7f453d39c700 1 --2- 172.21.15.44:0/2290188056 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4538101440 0x7f4538194150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.379+0000 7f453d39c700 1 -- 172.21.15.44:0/2290188056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4538101d50 msgr2=0x7f4538194690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:24.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.379+0000 7f453d39c700 1 --2- 172.21.15.44:0/2290188056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4538101d50 0x7f4538194690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.379+0000 7f453d39c700 1 -- 172.21.15.44:0/2290188056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4530009c40 con 0x7f453810a810 2023-11-16T16:53:24.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.379+0000 7f4537fff700 1 --2- 172.21.15.44:0/2290188056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4538101d50 0x7f4538194690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:24.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.379+0000 7f453d39c700 1 --2- 172.21.15.44:0/2290188056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453810a810 0x7f4538198af0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f453000efb0 tx=0x7f453000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:24.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.379+0000 7f4535ffb700 1 -- 172.21.15.44:0/2290188056 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4530003a00 con 0x7f453810a810 2023-11-16T16:53:24.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.380+0000 7f4535ffb700 1 -- 172.21.15.44:0/2290188056 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f453000f040 con 0x7f453810a810 2023-11-16T16:53:24.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.380+0000 7f453edff700 1 -- 172.21.15.44:0/2290188056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45240095f0 con 0x7f453810a810 2023-11-16T16:53:24.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.380+0000 7f4535ffb700 1 -- 172.21.15.44:0/2290188056 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4530004540 con 0x7f453810a810 2023-11-16T16:53:24.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.380+0000 7f453edff700 1 -- 172.21.15.44:0/2290188056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f453810e170 con 0x7f453810a810 2023-11-16T16:53:24.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.384+0000 7f4535ffb700 1 -- 172.21.15.44:0/2290188056 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f4530003bf0 con 0x7f453810a810 2023-11-16T16:53:24.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.386+0000 7f4535ffb700 1 --2- 172.21.15.44:0/2290188056 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f45280a1340 0x7f45280a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.386+0000 7f4535ffb700 1 -- 172.21.15.44:0/2290188056 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f45300ccba0 con 0x7f453810a810 2023-11-16T16:53:24.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.386+0000 7f453cb9b700 1 --2- 172.21.15.44:0/2290188056 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f45280a1340 0x7f45280a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.387+0000 7f4552666b80 1 Processor -- start 2023-11-16T16:53:24.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.387+0000 7f4552666b80 1 -- start start 2023-11-16T16:53:24.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.387+0000 7f453cb9b700 1 --2- 172.21.15.44:0/2290188056 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f45280a1340 0x7f45280a37f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f452c00bfd0 tx=0x7f452c00b810 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:24.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.388+0000 7f4552666b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8b006b0 0x55b1b8caa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.388+0000 7f4552666b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8caa720 0x55b1b8ca4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.388+0000 7f4552666b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caab90 0x55b1b8ca4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.388+0000 7f4552666b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1b8baa000 con 0x55b1b8b006b0 2023-11-16T16:53:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.388+0000 7f4552666b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1b8bfc840 con 0x55b1b8caab90 2023-11-16T16:53:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.388+0000 7f4552666b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1b8bf5cd0 con 0x55b1b8caa720 2023-11-16T16:53:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.388+0000 7f453cb9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8b006b0 0x55b1b8caa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.388+0000 7f4537fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8caa720 0x55b1b8ca4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.388+0000 7f453cb9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8b006b0 0x55b1b8caa1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44140/0 (socket says 172.21.15.44:44140) 2023-11-16T16:53:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.388+0000 7f453cb9b700 1 -- 172.21.15.44:0/4236181671 learned_addr learned my addr 172.21.15.44:0/4236181671 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.389+0000 7f4537fff700 1 -- 172.21.15.44:0/4236181671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caab90 msgr2=0x55b1b8ca4b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:53:24.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.389+0000 7f4537fff700 1 --2- 172.21.15.44:0/4236181671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caab90 0x55b1b8ca4b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.389+0000 7f4537fff700 1 -- 172.21.15.44:0/4236181671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8b006b0 msgr2=0x55b1b8caa1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:24.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.389+0000 7f4537fff700 1 --2- 172.21.15.44:0/4236181671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8b006b0 0x55b1b8caa1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.389+0000 7f4537fff700 1 -- 172.21.15.44:0/4236181671 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b1b8a775d0 con 0x55b1b8caa720 2023-11-16T16:53:24.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.389+0000 7f453cb9b700 1 --2- 172.21.15.44:0/4236181671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8b006b0 0x55b1b8caa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.389+0000 7f4537fff700 1 --2- 172.21.15.44:0/4236181671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8caa720 0x55b1b8ca4560 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4524006010 tx=0x7f4524005e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4792cb4b4e7774b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.390+0000 7f4522ffd700 1 -- 172.21.15.44:0/4236181671 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452401d070 con 0x55b1b8caa720 2023-11-16T16:53:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.390+0000 7f4522ffd700 1 -- 172.21.15.44:0/4236181671 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f452401e040 con 0x55b1b8caa720 2023-11-16T16:53:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.390+0000 7f4522ffd700 1 -- 172.21.15.44:0/4236181671 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4524004d20 con 0x55b1b8caa720 2023-11-16T16:53:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.391+0000 7f4552666b80 1 -- 172.21.15.44:0/4236181671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8caa720 msgr2=0x55b1b8ca4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.391+0000 7f4552666b80 1 --2- 172.21.15.44:0/4236181671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8caa720 0x55b1b8ca4560 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4524006010 tx=0x7f4524005e90 comp rx=0 tx=0).stop 2023-11-16T16:53:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.391+0000 7f4552666b80 1 -- 172.21.15.44:0/4236181671 shutdown_connections 2023-11-16T16:53:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.391+0000 7f4552666b80 1 --2- 172.21.15.44:0/4236181671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caab90 0x55b1b8ca4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.391+0000 7f4552666b80 1 --2- 172.21.15.44:0/4236181671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8b006b0 0x55b1b8caa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.391+0000 7f4552666b80 1 --2- 172.21.15.44:0/4236181671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8caa720 0x55b1b8ca4560 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.391+0000 7f4552666b80 1 -- 172.21.15.44:0/4236181671 >> 172.21.15.44:0/4236181671 conn(0x55b1b8c99f50 msgr2=0x55b1b8c9f270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.392+0000 7f4552666b80 1 -- 172.21.15.44:0/4236181671 shutdown_connections 2023-11-16T16:53:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.392+0000 7f4552666b80 1 -- 172.21.15.44:0/4236181671 wait complete. 2023-11-16T16:53:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.392+0000 7f4552666b80 1 -- start start 2023-11-16T16:53:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.393+0000 7f4552666b80 1 Processor -- start 2023-11-16T16:53:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.393+0000 7f4552666b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 0x55b1b8cadd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.393+0000 7f4552666b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caa720 0x55b1b8cae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.393+0000 7f4552666b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8caab90 0x55b1b8cae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.393+0000 7f4552666b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1b8b97880 con 0x55b1b8caab90 2023-11-16T16:53:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.393+0000 7f4552666b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1b8b97520 con 0x55b1b8caa720 2023-11-16T16:53:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.393+0000 7f4552666b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1b8a798c0 con 0x55b1b8b006b0 2023-11-16T16:53:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.393+0000 7f4552666b80 10 client.0 _ll_register_callbacks cb 0x55b1b8bb5720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.393+0000 7f4537fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caa720 0x55b1b8cae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.393+0000 7f4537fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caa720 0x55b1b8cae2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:47196/0 (socket says 172.21.15.44:47196) 2023-11-16T16:53:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.393+0000 7f4537fff700 1 -- 172.21.15.44:0/650610433 learned_addr learned my addr 172.21.15.44:0/650610433 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.394+0000 7f453d39c700 1 --2- 172.21.15.44:0/650610433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8caab90 0x55b1b8cae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.394+0000 7f453cb9b700 1 --2- 172.21.15.44:0/650610433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 0x55b1b8cadd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.394+0000 7f4537fff700 1 -- 172.21.15.44:0/650610433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 msgr2=0x55b1b8cadd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.394+0000 7f4537fff700 1 --2- 172.21.15.44:0/650610433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 0x55b1b8cadd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.394+0000 7f4537fff700 1 -- 172.21.15.44:0/650610433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8caab90 msgr2=0x55b1b8cae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.394+0000 7f4537fff700 1 --2- 172.21.15.44:0/650610433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8caab90 0x55b1b8cae7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.394+0000 7f4537fff700 1 -- 172.21.15.44:0/650610433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f453810d8f0 con 0x55b1b8caa720 2023-11-16T16:53:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.394+0000 7f4537fff700 1 --2- 172.21.15.44:0/650610433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caa720 0x55b1b8cae2a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f45240048a0 tx=0x7f4524022630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b4857d707dd18d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.394+0000 7f4517fff700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:53:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.394+0000 7f4517fff700 1 -- 172.21.15.44:0/650610433 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452401d070 con 0x55b1b8caa720 2023-11-16T16:53:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.395+0000 7f4517fff700 1 -- 172.21.15.44:0/650610433 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f4524022ce0 con 0x55b1b8caa720 2023-11-16T16:53:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.395+0000 7f4552666b80 1 -- 172.21.15.44:0/650610433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b1b8c97710 con 0x55b1b8caa720 2023-11-16T16:53:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.395+0000 7f4517fff700 1 -- 172.21.15.44:0/650610433 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4524022ea0 con 0x55b1b8caa720 2023-11-16T16:53:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.395+0000 7f4552666b80 10 client.26360 fetch_fsmap learned FSMap version 21 2023-11-16T16:53:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.395+0000 7f4552666b80 1 -- 172.21.15.44:0/650610433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55b1b8bbc130 con 0x55b1b8caa720 2023-11-16T16:53:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.395+0000 7f4517fff700 1 -- 172.21.15.44:0/650610433 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b1b8c97710 con 0x55b1b8caa720 2023-11-16T16:53:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.396+0000 7f4517fff700 1 -- 172.21.15.44:0/650610433 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f452402c540 con 0x55b1b8caa720 2023-11-16T16:53:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.396+0000 7f4552666b80 10 client.26360 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:53:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.396+0000 7f4552666b80 10 client.26360 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.396+0000 7f4552666b80 20 client.26360 populate_metadata read hostname 'smithi044' 2023-11-16T16:53:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.396+0000 7f4552666b80 1 --2- 172.21.15.44:0/650610433 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b1b8cc6400 0x55b1b8cc8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.396+0000 7f4552666b80 4 client.26360 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:53:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.396+0000 7f4552666b80 1 -- 172.21.15.44:0/650610433 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b1b8bb8040 con 0x55b1b8cc6400 2023-11-16T16:53:24.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.396+0000 7f453cb9b700 1 --2- 172.21.15.44:0/650610433 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b1b8cc6400 0x55b1b8cc8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.397+0000 7f453cb9b700 1 --2- 172.21.15.44:0/650610433 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b1b8cc6400 0x55b1b8cc8810 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cecfe86c2c0bd7d8 server_cookie=d80fea23196d053b in_seq=0 out_seq=0 2023-11-16T16:53:24.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.397+0000 7f4517fff700 10 client.26360 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.398+0000 7f4517fff700 1 -- 172.21.15.44:0/650610433 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55b1b8bb8040 con 0x55b1b8cc6400 2023-11-16T16:53:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.398+0000 7f4517fff700 10 client.26360 handle_command_reply: tid=0 2023-11-16T16:53:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.398+0000 7f4552666b80 1 client.26360 shutdown 2023-11-16T16:53:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.398+0000 7f4552666b80 1 -- 172.21.15.44:0/650610433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caa720 msgr2=0x55b1b8cae2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.398+0000 7f4552666b80 1 --2- 172.21.15.44:0/650610433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caa720 0x55b1b8cae2a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f45240048a0 tx=0x7f4524022630 comp rx=0 tx=0).stop 2023-11-16T16:53:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.398+0000 7f4552666b80 1 -- 172.21.15.44:0/650610433 shutdown_connections 2023-11-16T16:53:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.398+0000 7f4552666b80 1 --2- 172.21.15.44:0/650610433 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b1b8cc6400 0x55b1b8cc8810 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.398+0000 7f4552666b80 1 --2- 172.21.15.44:0/650610433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8caab90 0x55b1b8cae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.398+0000 7f4552666b80 1 --2- 172.21.15.44:0/650610433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 0x55b1b8cadd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.398+0000 7f4552666b80 1 --2- 172.21.15.44:0/650610433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caa720 0x55b1b8cae2a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.398+0000 7f4552666b80 1 -- 172.21.15.44:0/650610433 >> 172.21.15.44:0/650610433 conn(0x55b1b8c99f50 msgr2=0x55b1b8c9e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.398+0000 7f4517fff700 0 client.26360 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.399+0000 7f4552666b80 1 -- 172.21.15.44:0/650610433 shutdown_connections 2023-11-16T16:53:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.399+0000 7f4552666b80 1 -- 172.21.15.44:0/650610433 wait complete. 2023-11-16T16:53:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.399+0000 7f4552666b80 20 client.26360 trim_cache size 0 max 16384 2023-11-16T16:53:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.418+0000 7f4552666b80 1 Processor -- start 2023-11-16T16:53:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.419+0000 7f4552666b80 1 -- start start 2023-11-16T16:53:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.419+0000 7f4552666b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 0x55b1b8c9ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.419+0000 7f4552666b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8caa720 0x55b1b8ca4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.419+0000 7f4552666b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caab90 0x55b1b8ca4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.419+0000 7f4552666b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1b8ca5660 con 0x55b1b8caa720 2023-11-16T16:53:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.419+0000 7f4552666b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1b8cb0e10 con 0x55b1b8caab90 2023-11-16T16:53:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.419+0000 7f4552666b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1b8c9fd60 con 0x55b1b8b006b0 2023-11-16T16:53:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.419+0000 7f453cb9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 0x55b1b8c9ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.419+0000 7f453d39c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caab90 0x55b1b8ca4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.419+0000 7f4537fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8caa720 0x55b1b8ca4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.419+0000 7f453d39c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caab90 0x55b1b8ca4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:47200/0 (socket says 172.21.15.44:47200) 2023-11-16T16:53:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.419+0000 7f453d39c700 1 -- 172.21.15.44:0/2827770287 learned_addr learned my addr 172.21.15.44:0/2827770287 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.420+0000 7f453d39c700 1 -- 172.21.15.44:0/2827770287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 msgr2=0x55b1b8c9ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.420+0000 7f453d39c700 1 --2- 172.21.15.44:0/2827770287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 0x55b1b8c9ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.420+0000 7f453d39c700 1 -- 172.21.15.44:0/2827770287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8caa720 msgr2=0x55b1b8ca4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.420+0000 7f453d39c700 1 --2- 172.21.15.44:0/2827770287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8caa720 0x55b1b8ca4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.420+0000 7f453d39c700 1 -- 172.21.15.44:0/2827770287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b1b8c9b6f0 con 0x55b1b8caab90 2023-11-16T16:53:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.420+0000 7f453d39c700 1 --2- 172.21.15.44:0/2827770287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caab90 0x55b1b8ca4aa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4530099680 tx=0x7f4530009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5807df27797918d0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.420+0000 7f45237fe700 1 -- 172.21.15.44:0/2827770287 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45300112c0 con 0x55b1b8caab90 2023-11-16T16:53:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.420+0000 7f45237fe700 1 -- 172.21.15.44:0/2827770287 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4530099980 con 0x55b1b8caab90 2023-11-16T16:53:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.420+0000 7f45237fe700 1 -- 172.21.15.44:0/2827770287 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4530004900 con 0x55b1b8caab90 2023-11-16T16:53:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.421+0000 7f4552666b80 1 -- 172.21.15.44:0/2827770287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caab90 msgr2=0x55b1b8ca4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.421+0000 7f4552666b80 1 --2- 172.21.15.44:0/2827770287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caab90 0x55b1b8ca4aa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4530099680 tx=0x7f4530009f30 comp rx=0 tx=0).stop 2023-11-16T16:53:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.422+0000 7f4552666b80 1 -- 172.21.15.44:0/2827770287 shutdown_connections 2023-11-16T16:53:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.422+0000 7f4552666b80 1 --2- 172.21.15.44:0/2827770287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caab90 0x55b1b8ca4aa0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.422+0000 7f4552666b80 1 --2- 172.21.15.44:0/2827770287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 0x55b1b8c9ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.422+0000 7f4552666b80 1 --2- 172.21.15.44:0/2827770287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8caa720 0x55b1b8ca4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.422+0000 7f4552666b80 1 -- 172.21.15.44:0/2827770287 >> 172.21.15.44:0/2827770287 conn(0x55b1b8c99f50 msgr2=0x55b1b8cc2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.422+0000 7f4552666b80 1 -- 172.21.15.44:0/2827770287 shutdown_connections 2023-11-16T16:53:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.422+0000 7f4552666b80 1 -- 172.21.15.44:0/2827770287 wait complete. 2023-11-16T16:53:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.422+0000 7f4552666b80 1 -- start start 2023-11-16T16:53:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.423+0000 7f4552666b80 1 Processor -- start 2023-11-16T16:53:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.424+0000 7f4552666b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 0x55b1b8cbb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.424+0000 7f4552666b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caa720 0x55b1b8cbbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.424+0000 7f4552666b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8caab90 0x55b1b8d19280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.424+0000 7f4552666b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1b8cbc5e0 con 0x55b1b8caab90 2023-11-16T16:53:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.424+0000 7f4552666b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1b8cbc750 con 0x55b1b8caa720 2023-11-16T16:53:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.424+0000 7f4552666b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1b8d19840 con 0x55b1b8b006b0 2023-11-16T16:53:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.424+0000 7f4552666b80 10 client.0 _ll_register_callbacks cb 0x7f452c00b2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.424+0000 7f453cb9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 0x55b1b8cbb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.424+0000 7f453cb9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 0x55b1b8cbb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:46852/0 (socket says 172.21.15.44:46852) 2023-11-16T16:53:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.424+0000 7f453cb9b700 1 -- 172.21.15.44:0/3677547340 learned_addr learned my addr 172.21.15.44:0/3677547340 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.424+0000 7f4537fff700 1 --2- 172.21.15.44:0/3677547340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caa720 0x55b1b8cbbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.425+0000 7f453d39c700 1 --2- 172.21.15.44:0/3677547340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8caab90 0x55b1b8d19280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.425+0000 7f453cb9b700 1 -- 172.21.15.44:0/3677547340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caa720 msgr2=0x55b1b8cbbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.425+0000 7f453cb9b700 1 --2- 172.21.15.44:0/3677547340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caa720 0x55b1b8cbbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.425+0000 7f453cb9b700 1 -- 172.21.15.44:0/3677547340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8caab90 msgr2=0x55b1b8d19280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.425+0000 7f453cb9b700 1 --2- 172.21.15.44:0/3677547340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8caab90 0x55b1b8d19280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.425+0000 7f453cb9b700 1 -- 172.21.15.44:0/3677547340 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f453810e170 con 0x55b1b8b006b0 2023-11-16T16:53:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.425+0000 7f453d39c700 1 --2- 172.21.15.44:0/3677547340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8caab90 0x55b1b8d19280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:53:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.425+0000 7f4537fff700 1 --2- 172.21.15.44:0/3677547340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caa720 0x55b1b8cbbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:53:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.425+0000 7f453cb9b700 1 --2- 172.21.15.44:0/3677547340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 0x55b1b8cbb890 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55b1b8cc90b0 tx=0x7f452c00d050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1d62fc8826133a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.425+0000 7f4517fff700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:53:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.425+0000 7f4517fff700 1 -- 172.21.15.44:0/3677547340 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452c026620 con 0x55b1b8b006b0 2023-11-16T16:53:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.425+0000 7f4517fff700 1 -- 172.21.15.44:0/3677547340 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f452c026780 con 0x55b1b8b006b0 2023-11-16T16:53:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.425+0000 7f4552666b80 1 -- 172.21.15.44:0/3677547340 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f452401e040 con 0x55b1b8b006b0 2023-11-16T16:53:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.426+0000 7f4517fff700 1 -- 172.21.15.44:0/3677547340 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f452c026940 con 0x55b1b8b006b0 2023-11-16T16:53:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.426+0000 7f4552666b80 10 client.26359 fetch_fsmap learned FSMap version 21 2023-11-16T16:53:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.426+0000 7f4552666b80 1 -- 172.21.15.44:0/3677547340 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55b1b8a23d80 con 0x55b1b8b006b0 2023-11-16T16:53:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.426+0000 7f4517fff700 1 -- 172.21.15.44:0/3677547340 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f452401e040 con 0x55b1b8b006b0 2023-11-16T16:53:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.426+0000 7f4517fff700 1 -- 172.21.15.44:0/3677547340 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f452c033b70 con 0x55b1b8b006b0 2023-11-16T16:53:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.426+0000 7f4552666b80 10 client.26359 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:53:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.426+0000 7f4552666b80 10 client.26359 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.426+0000 7f4552666b80 20 client.26359 populate_metadata read hostname 'smithi044' 2023-11-16T16:53:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.426+0000 7f4552666b80 1 --2- 172.21.15.44:0/3677547340 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b1b8d1afe0 0x55b1b8d1d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.426+0000 7f4552666b80 4 client.26359 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:53:24.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.426+0000 7f4552666b80 1 -- 172.21.15.44:0/3677547340 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b1b8bc4600 con 0x55b1b8d1afe0 2023-11-16T16:53:24.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.427+0000 7f4537fff700 1 --2- 172.21.15.44:0/3677547340 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b1b8d1afe0 0x55b1b8d1d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:24.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.427+0000 7f4537fff700 1 --2- 172.21.15.44:0/3677547340 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b1b8d1afe0 0x55b1b8d1d3f0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff2c9bac96925ee4 server_cookie=9caa5606f344df44 in_seq=0 out_seq=0 2023-11-16T16:53:24.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:24.427+0000 7f4517fff700 10 client.26359 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:25.542 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:53:25.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:25 smithi044 ceph-mon[128750]: pgmap v107: 129 pgs: 129 active+clean; 34 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 12 op/s 2023-11-16T16:53:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:25 smithi176 ceph-mon[131630]: pgmap v107: 129 pgs: 129 active+clean; 34 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 12 op/s 2023-11-16T16:53:25.980 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:25 smithi077 ceph-mon[131204]: pgmap v107: 129 pgs: 129 active+clean; 34 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 12 op/s 2023-11-16T16:53:26.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.085+0000 7f9d7bfff700 1 -- 172.21.15.44:0/184908787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d74109510 msgr2=0x7f9d74111c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:26.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.085+0000 7f9d7bfff700 1 --2- 172.21.15.44:0/184908787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d74109510 0x7f9d74111c70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009990 tx=0x7f9d64009ca0 comp rx=0 tx=0).stop 2023-11-16T16:53:26.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.085+0000 7f9d7bfff700 1 -- 172.21.15.44:0/184908787 shutdown_connections 2023-11-16T16:53:26.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.085+0000 7f9d7bfff700 1 --2- 172.21.15.44:0/184908787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d741121b0 0x7f9d74114590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:26.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.085+0000 7f9d7bfff700 1 --2- 172.21.15.44:0/184908787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d74108c00 0x7f9d74108fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:26.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.085+0000 7f9d7bfff700 1 --2- 172.21.15.44:0/184908787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d74109510 0x7f9d74111c70 secure :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009990 tx=0x7f9d64009ca0 comp rx=0 tx=0).stop 2023-11-16T16:53:26.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.086+0000 7f9d7bfff700 1 -- 172.21.15.44:0/184908787 >> 172.21.15.44:0/184908787 conn(0x7f9d7406c7b0 msgr2=0x7f9d74071770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:26.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.087+0000 7f9d7bfff700 1 -- 172.21.15.44:0/184908787 shutdown_connections 2023-11-16T16:53:26.089 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.087+0000 7f9d7bfff700 1 -- 172.21.15.44:0/184908787 wait complete. 2023-11-16T16:53:26.090 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.089+0000 7f9d7bfff700 1 Processor -- start 2023-11-16T16:53:26.091 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.089+0000 7f9d7bfff700 1 -- start start 2023-11-16T16:53:26.091 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.089+0000 7f9d7bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d74108c00 0x7f9d74111670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:26.091 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.089+0000 7f9d7bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d741121b0 0x7f9d74111bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:26.091 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.089+0000 7f9d7bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d7410b6f0 0x7f9d7410bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:26.091 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.089+0000 7f9d7bfff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7410c360 con 0x7f9d74108c00 2023-11-16T16:53:26.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.089+0000 7f9d7bfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7410c4d0 con 0x7f9d7410b6f0 2023-11-16T16:53:26.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.089+0000 7f9d7bfff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d741ba6a0 con 0x7f9d741121b0 2023-11-16T16:53:26.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.091+0000 7f9d7a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d7410b6f0 0x7f9d7410bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:26.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.091+0000 7f9d7a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d7410b6f0 0x7f9d7410bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56458/0 (socket says 172.21.15.44:56458) 2023-11-16T16:53:26.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.091+0000 7f9d7a59c700 1 -- 172.21.15.44:0/3768744868 learned_addr learned my addr 172.21.15.44:0/3768744868 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:26.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.091+0000 7f9d79d9b700 1 --2- 172.21.15.44:0/3768744868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d74108c00 0x7f9d74111670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:26.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.091+0000 7f9d7959a700 1 --2- 172.21.15.44:0/3768744868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d741121b0 0x7f9d74111bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:26.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.093+0000 7f9d79d9b700 1 -- 172.21.15.44:0/3768744868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d741121b0 msgr2=0x7f9d74111bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:26.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.093+0000 7f9d79d9b700 1 --2- 172.21.15.44:0/3768744868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d741121b0 0x7f9d74111bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:26.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.093+0000 7f9d79d9b700 1 -- 172.21.15.44:0/3768744868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d7410b6f0 msgr2=0x7f9d7410bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:26.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.093+0000 7f9d79d9b700 1 --2- 172.21.15.44:0/3768744868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d7410b6f0 0x7f9d7410bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:26.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.093+0000 7f9d79d9b700 1 -- 172.21.15.44:0/3768744868 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d640095f0 con 0x7f9d74108c00 2023-11-16T16:53:26.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.094+0000 7f9d79d9b700 1 --2- 172.21.15.44:0/3768744868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d74108c00 0x7f9d74111670 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9d7000c970 tx=0x7f9d7000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:26.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.095+0000 7f9d6affd700 1 -- 172.21.15.44:0/3768744868 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70007b00 con 0x7f9d74108c00 2023-11-16T16:53:26.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.095+0000 7f9d6affd700 1 -- 172.21.15.44:0/3768744868 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d7000ce60 con 0x7f9d74108c00 2023-11-16T16:53:26.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.095+0000 7f9d7bfff700 1 -- 172.21.15.44:0/3768744868 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d74115be0 con 0x7f9d74108c00 2023-11-16T16:53:26.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.095+0000 7f9d6affd700 1 -- 172.21.15.44:0/3768744868 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70018660 con 0x7f9d74108c00 2023-11-16T16:53:26.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.095+0000 7f9d7bfff700 1 -- 172.21.15.44:0/3768744868 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d74115a50 con 0x7f9d74108c00 2023-11-16T16:53:26.107 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.105+0000 7f9d6affd700 1 -- 172.21.15.44:0/3768744868 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f9d70004020 con 0x7f9d74108c00 2023-11-16T16:53:26.109 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.108+0000 7f9d8f9d0b80 1 Processor -- start 2023-11-16T16:53:26.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.109+0000 7f9d8f9d0b80 1 -- start start 2023-11-16T16:53:26.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.109+0000 7f9d8f9d0b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0905b60 0x55d7c0905f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:26.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.109+0000 7f9d8f9d0b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c0906470 0x55d7c090a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:26.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.109+0000 7f9d8f9d0b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c090a8b0 0x55d7c090ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:26.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.109+0000 7f9d8f9d0b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c080b060 con 0x55d7c0906470 2023-11-16T16:53:26.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.109+0000 7f9d8f9d0b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c085d8a0 con 0x55d7c090a8b0 2023-11-16T16:53:26.112 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.109+0000 7f9d8f9d0b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c0856d30 con 0x55d7c0905b60 2023-11-16T16:53:26.112 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.109+0000 7f9d6affd700 1 --2- 172.21.15.44:0/3768744868 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9d600a1450 0x7f9d600a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:26.112 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.110+0000 7f9d6affd700 1 -- 172.21.15.44:0/3768744868 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f9d700ccec0 con 0x7f9d74108c00 2023-11-16T16:53:26.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.110+0000 7f9d7959a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c090a8b0 0x55d7c090ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:26.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.111+0000 7f9d7959a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c090a8b0 0x55d7c090ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56462/0 (socket says 172.21.15.44:56462) 2023-11-16T16:53:26.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.111+0000 7f9d7959a700 1 -- 172.21.15.44:0/2019046249 learned_addr learned my addr 172.21.15.44:0/2019046249 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:26.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.111+0000 7f9d79d9b700 1 --2- 172.21.15.44:0/2019046249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c0906470 0x55d7c090a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:26.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.111+0000 7f9d7a59c700 1 --2- 172.21.15.44:0/3768744868 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9d600a1450 0x7f9d600a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:26.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.111+0000 7f9d7a59c700 1 --2- 172.21.15.44:0/2019046249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0905b60 0x55d7c0905f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:26.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.113+0000 7f9d7a59c700 1 --2- 172.21.15.44:0/3768744868 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9d600a1450 0x7f9d600a3900 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0094d0 tx=0x7f9d6c008350 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:26.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.113+0000 7f9d7959a700 1 -- 172.21.15.44:0/2019046249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0905b60 msgr2=0x55d7c0905f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:26.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.113+0000 7f9d7959a700 1 --2- 172.21.15.44:0/2019046249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0905b60 0x55d7c0905f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:26.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.113+0000 7f9d7959a700 1 -- 172.21.15.44:0/2019046249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c0906470 msgr2=0x55d7c090a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:26.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.113+0000 7f9d7959a700 1 --2- 172.21.15.44:0/2019046249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c0906470 0x55d7c090a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:26.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.114+0000 7f9d7959a700 1 -- 172.21.15.44:0/2019046249 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d7c06d8630 con 0x55d7c090a8b0 2023-11-16T16:53:26.117 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.115+0000 7f9d7959a700 1 --2- 172.21.15.44:0/2019046249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c090a8b0 0x55d7c090ad00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009990 tx=0x7f9d64009f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d29af26c2adabfbc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:26.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.125+0000 7f9d5ffff700 1 -- 172.21.15.44:0/2019046249 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6400b7d0 con 0x55d7c090a8b0 2023-11-16T16:53:26.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.125+0000 7f9d5ffff700 1 -- 172.21.15.44:0/2019046249 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d6401e040 con 0x55d7c090a8b0 2023-11-16T16:53:26.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.125+0000 7f9d5ffff700 1 -- 172.21.15.44:0/2019046249 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6400b7d0 con 0x55d7c090a8b0 2023-11-16T16:53:26.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.126+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2019046249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c090a8b0 msgr2=0x55d7c090ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:26.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.126+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/2019046249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c090a8b0 0x55d7c090ad00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009990 tx=0x7f9d64009f40 comp rx=0 tx=0).stop 2023-11-16T16:53:26.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.128+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2019046249 shutdown_connections 2023-11-16T16:53:26.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.128+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/2019046249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c090a8b0 0x55d7c090ad00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:26.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.128+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/2019046249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0905b60 0x55d7c0905f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:26.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.128+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/2019046249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c0906470 0x55d7c090a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:26.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.128+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2019046249 >> 172.21.15.44:0/2019046249 conn(0x55d7c08fb360 msgr2=0x55d7c0900b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:26.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.128+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2019046249 shutdown_connections 2023-11-16T16:53:26.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.128+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2019046249 wait complete. 2023-11-16T16:53:26.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.129+0000 7f9d8f9d0b80 1 -- start start 2023-11-16T16:53:26.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.129+0000 7f9d8f9d0b80 1 Processor -- start 2023-11-16T16:53:26.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.130+0000 7f9d8f9d0b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0905b60 0x55d7c09120c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:26.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.130+0000 7f9d8f9d0b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c0906470 0x55d7c090f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:26.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.130+0000 7f9d8f9d0b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c090a8b0 0x55d7c090f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:26.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.130+0000 7f9d8f9d0b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c07f88e0 con 0x55d7c0906470 2023-11-16T16:53:26.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.130+0000 7f9d8f9d0b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c07f8580 con 0x55d7c090a8b0 2023-11-16T16:53:26.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.130+0000 7f9d8f9d0b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c06da920 con 0x55d7c0905b60 2023-11-16T16:53:26.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.130+0000 7f9d8f9d0b80 10 client.0 _ll_register_callbacks cb 0x55d7c0816780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:26.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.131+0000 7f9d79d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0905b60 0x55d7c09120c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:26.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.131+0000 7f9d79d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0905b60 0x55d7c09120c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:46464/0 (socket says 172.21.15.44:46464) 2023-11-16T16:53:26.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.131+0000 7f9d79d9b700 1 -- 172.21.15.44:0/2128387463 learned_addr learned my addr 172.21.15.44:0/2128387463 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:26.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.131+0000 7f9d7959a700 1 --2- 172.21.15.44:0/2128387463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c0906470 0x55d7c090f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:26.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.132+0000 7f9d79d9b700 1 -- 172.21.15.44:0/2128387463 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c090a8b0 msgr2=0x55d7c090f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:26.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.132+0000 7f9d79d9b700 1 --2- 172.21.15.44:0/2128387463 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c090a8b0 0x55d7c090f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:26.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.132+0000 7f9d79d9b700 1 -- 172.21.15.44:0/2128387463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c0906470 msgr2=0x55d7c090f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:26.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.132+0000 7f9d79d9b700 1 --2- 172.21.15.44:0/2128387463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c0906470 0x55d7c090f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:26.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.132+0000 7f9d79d9b700 1 -- 172.21.15.44:0/2128387463 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d74115080 con 0x55d7c0905b60 2023-11-16T16:53:26.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.133+0000 7f9d79d9b700 1 --2- 172.21.15.44:0/2128387463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0905b60 0x55d7c09120c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9d700074a0 tx=0x7f9d7001f3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e2c22484e4f703f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:26.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.133+0000 7f9d5d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:53:26.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.134+0000 7f9d5d7fa700 1 -- 172.21.15.44:0/2128387463 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d700b1070 con 0x55d7c0905b60 2023-11-16T16:53:26.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.134+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2128387463 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d7c08f8b20 con 0x55d7c0905b60 2023-11-16T16:53:26.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.135+0000 7f9d5d7fa700 1 -- 172.21.15.44:0/2128387463 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f9d70099c10 con 0x55d7c0905b60 2023-11-16T16:53:26.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.135+0000 7f9d5d7fa700 1 -- 172.21.15.44:0/2128387463 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d7009dc40 con 0x55d7c0905b60 2023-11-16T16:53:26.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.135+0000 7f9d8f9d0b80 10 client.26368 fetch_fsmap learned FSMap version 21 2023-11-16T16:53:26.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.135+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2128387463 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55d7c081d190 con 0x55d7c0905b60 2023-11-16T16:53:26.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.136+0000 7f9d5d7fa700 1 -- 172.21.15.44:0/2128387463 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d7c08f8b20 con 0x55d7c0905b60 2023-11-16T16:53:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.136+0000 7f9d5d7fa700 1 -- 172.21.15.44:0/2128387463 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9d70099f80 con 0x55d7c0905b60 2023-11-16T16:53:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.136+0000 7f9d8f9d0b80 10 client.26368 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:53:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.136+0000 7f9d8f9d0b80 10 client.26368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.137+0000 7f9d8f9d0b80 20 client.26368 populate_metadata read hostname 'smithi044' 2023-11-16T16:53:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.137+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/2128387463 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d7c091f720 0x55d7c0921b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.137+0000 7f9d8f9d0b80 4 client.26368 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:53:26.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.137+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2128387463 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d7c08190a0 con 0x55d7c091f720 2023-11-16T16:53:26.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.153+0000 7f9d7959a700 1 --2- 172.21.15.44:0/2128387463 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d7c091f720 0x55d7c0921b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:26.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.157+0000 7f9d5d7fa700 10 client.26368 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:26.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:26.157+0000 7f9d7959a700 1 --2- 172.21.15.44:0/2128387463 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d7c091f720 0x55d7c0921b00 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37636d61e3b78071 server_cookie=e77b0a53dfdd7636 in_seq=0 out_seq=0 2023-11-16T16:53:27.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:27 smithi044 ceph-mon[128750]: pgmap v108: 129 pgs: 129 active+clean; 34 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 12 op/s 2023-11-16T16:53:27.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:27 smithi176 ceph-mon[131630]: pgmap v108: 129 pgs: 129 active+clean; 34 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 12 op/s 2023-11-16T16:53:27.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:27 smithi077 ceph-mon[131204]: pgmap v108: 129 pgs: 129 active+clean; 34 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 12 op/s 2023-11-16T16:53:29.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:29 smithi044 ceph-mon[128750]: pgmap v109: 129 pgs: 129 active+clean; 43 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.0 MiB/s wr, 1.04k op/s 2023-11-16T16:53:29.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:29 smithi176 ceph-mon[131630]: pgmap v109: 129 pgs: 129 active+clean; 43 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.0 MiB/s wr, 1.04k op/s 2023-11-16T16:53:29.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:29 smithi077 ceph-mon[131204]: pgmap v109: 129 pgs: 129 active+clean; 43 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.0 MiB/s wr, 1.04k op/s 2023-11-16T16:53:31.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:31 smithi044 ceph-mon[128750]: pgmap v110: 129 pgs: 129 active+clean; 43 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 1.03k op/s 2023-11-16T16:53:31.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:31 smithi176 ceph-mon[131630]: pgmap v110: 129 pgs: 129 active+clean; 43 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 1.03k op/s 2023-11-16T16:53:31.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:31 smithi077 ceph-mon[131204]: pgmap v110: 129 pgs: 129 active+clean; 43 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 1.03k op/s 2023-11-16T16:53:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.092+0000 7f4517fff700 1 -- 172.21.15.44:0/3677547340 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b1b8bc4600 con 0x55b1b8d1afe0 2023-11-16T16:53:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.092+0000 7f4517fff700 10 client.26359 handle_command_reply: tid=0 2023-11-16T16:53:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.092+0000 7f4552666b80 1 client.26359 shutdown 2023-11-16T16:53:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.093+0000 7f4552666b80 1 -- 172.21.15.44:0/3677547340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 msgr2=0x55b1b8cbb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.093+0000 7f4552666b80 1 --2- 172.21.15.44:0/3677547340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 0x55b1b8cbb890 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55b1b8cc90b0 tx=0x7f452c00d050 comp rx=0 tx=0).stop 2023-11-16T16:53:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.093+0000 7f4552666b80 1 -- 172.21.15.44:0/3677547340 shutdown_connections 2023-11-16T16:53:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.093+0000 7f4552666b80 1 --2- 172.21.15.44:0/3677547340 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b1b8d1afe0 0x55b1b8d1d3f0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.093+0000 7f4552666b80 1 --2- 172.21.15.44:0/3677547340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b1b8caab90 0x55b1b8d19280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.093+0000 7f4552666b80 1 --2- 172.21.15.44:0/3677547340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b1b8b006b0 0x55b1b8cbb890 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.093+0000 7f4552666b80 1 --2- 172.21.15.44:0/3677547340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1b8caa720 0x55b1b8cbbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.093+0000 7f4517fff700 0 client.26359 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.093+0000 7f4552666b80 1 -- 172.21.15.44:0/3677547340 >> 172.21.15.44:0/3677547340 conn(0x55b1b8c99f50 msgr2=0x55b1b8ca8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.093+0000 7f4552666b80 1 -- 172.21.15.44:0/3677547340 shutdown_connections 2023-11-16T16:53:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.093+0000 7f4552666b80 1 -- 172.21.15.44:0/3677547340 wait complete. 2023-11-16T16:53:32.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.093+0000 7f9d5d7fa700 1 -- 172.21.15.44:0/2128387463 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d7c08190a0 con 0x55d7c091f720 2023-11-16T16:53:32.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.093+0000 7f9d5d7fa700 10 client.26368 handle_command_reply: tid=0 2023-11-16T16:53:32.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.093+0000 7f9d8f9d0b80 1 client.26368 shutdown 2023-11-16T16:53:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.093+0000 7f4552666b80 20 client.26359 trim_cache size 0 max 16384 2023-11-16T16:53:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2128387463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0905b60 msgr2=0x55d7c09120c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/2128387463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0905b60 0x55d7c09120c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9d700074a0 tx=0x7f9d7001f3b0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:53:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:53:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:53:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:53:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2128387463 shutdown_connections 2023-11-16T16:53:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/2128387463 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d7c091f720 0x55d7c0921b00 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/2128387463 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c090a8b0 0x55d7c090f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/2128387463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0905b60 0x55d7c09120c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/2128387463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c0906470 0x55d7c090f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.099 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2128387463 >> 172.21.15.44:0/2128387463 conn(0x55d7c08fb360 msgr2=0x55d7c090bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:32.099 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f9d5d7fa700 0 client.26368 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:32.099 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2128387463 shutdown_connections 2023-11-16T16:53:32.099 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2128387463 wait complete. 2023-11-16T16:53:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f453edff700 1 -- 172.21.15.44:0/2290188056 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f45280a1340 msgr2=0x7f45280a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f453edff700 1 --2- 172.21.15.44:0/2290188056 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f45280a1340 0x7f45280a37f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f452c00bfd0 tx=0x7f452c00b810 comp rx=0 tx=0).stop 2023-11-16T16:53:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f453edff700 1 -- 172.21.15.44:0/2290188056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453810a810 msgr2=0x7f4538198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f453edff700 1 --2- 172.21.15.44:0/2290188056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453810a810 0x7f4538198af0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f453000efb0 tx=0x7f453000c3a0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f453edff700 1 -- 172.21.15.44:0/2290188056 shutdown_connections 2023-11-16T16:53:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f453edff700 1 --2- 172.21.15.44:0/2290188056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453810a810 0x7f4538198af0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f453edff700 1 --2- 172.21.15.44:0/2290188056 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f45280a1340 0x7f45280a37f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f453edff700 1 --2- 172.21.15.44:0/2290188056 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4538101440 0x7f4538194150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f453edff700 1 --2- 172.21.15.44:0/2290188056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4538101d50 0x7f4538194690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f453edff700 1 -- 172.21.15.44:0/2290188056 >> 172.21.15.44:0/2290188056 conn(0x7f45380763a0 msgr2=0x7f453810b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:32.100 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.094+0000 7f9d8f9d0b80 20 client.26368 trim_cache size 0 max 16384 2023-11-16T16:53:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.095+0000 7f453edff700 1 -- 172.21.15.44:0/2290188056 shutdown_connections 2023-11-16T16:53:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:32.095+0000 7f453edff700 1 -- 172.21.15.44:0/2290188056 wait complete. 2023-11-16T16:53:32.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.126+0000 7f9d8f9d0b80 1 Processor -- start 2023-11-16T16:53:32.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.126+0000 7f9d8f9d0b80 1 -- start start 2023-11-16T16:53:32.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.126+0000 7f9d8f9d0b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c0905b60 0x55d7c0916950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.126+0000 7f9d8f9d0b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0906470 0x55d7c08ff550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.126+0000 7f9d8f9d0b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c090a8b0 0x55d7c0911c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.126+0000 7f9d8f9d0b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c08f89f0 con 0x55d7c0905b60 2023-11-16T16:53:32.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.126+0000 7f9d8f9d0b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c0906930 con 0x55d7c090a8b0 2023-11-16T16:53:32.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.126+0000 7f9d8f9d0b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c09094e0 con 0x55d7c0906470 2023-11-16T16:53:32.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.127+0000 7f9d7959a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0906470 0x55d7c08ff550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.127+0000 7f9d7959a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0906470 0x55d7c08ff550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:46472/0 (socket says 172.21.15.44:46472) 2023-11-16T16:53:32.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.127+0000 7f9d7959a700 1 -- 172.21.15.44:0/3894884599 learned_addr learned my addr 172.21.15.44:0/3894884599 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:32.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.127+0000 7f9d7a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c090a8b0 0x55d7c0911c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.127+0000 7f9d79d9b700 1 --2- 172.21.15.44:0/3894884599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c0905b60 0x55d7c0916950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.127+0000 7f9d7959a700 1 -- 172.21.15.44:0/3894884599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c090a8b0 msgr2=0x55d7c0911c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:32.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.127+0000 7f9d7959a700 1 --2- 172.21.15.44:0/3894884599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c090a8b0 0x55d7c0911c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.127+0000 7f9d7959a700 1 -- 172.21.15.44:0/3894884599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c0905b60 msgr2=0x55d7c0916950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:32.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.127+0000 7f9d7959a700 1 --2- 172.21.15.44:0/3894884599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c0905b60 0x55d7c0916950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.127+0000 7f9d7959a700 1 -- 172.21.15.44:0/3894884599 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d7c0900100 con 0x55d7c0906470 2023-11-16T16:53:32.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.127+0000 7f9d7959a700 1 --2- 172.21.15.44:0/3894884599 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0906470 0x55d7c08ff550 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9d640098e0 tx=0x7f9d6400b580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61467a97af1ca251 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:32.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.128+0000 7f9d68ff9700 1 -- 172.21.15.44:0/3894884599 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6400b7d0 con 0x55d7c0906470 2023-11-16T16:53:32.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.128+0000 7f9d68ff9700 1 -- 172.21.15.44:0/3894884599 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d6400b930 con 0x55d7c0906470 2023-11-16T16:53:32.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.128+0000 7f9d68ff9700 1 -- 172.21.15.44:0/3894884599 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d64003bc0 con 0x55d7c0906470 2023-11-16T16:53:32.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.128+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/3894884599 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0906470 msgr2=0x55d7c08ff550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.128+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/3894884599 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0906470 0x55d7c08ff550 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9d640098e0 tx=0x7f9d6400b580 comp rx=0 tx=0).stop 2023-11-16T16:53:32.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.128+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/3894884599 shutdown_connections 2023-11-16T16:53:32.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.128+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/3894884599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c090a8b0 0x55d7c0911c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.128+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/3894884599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c0905b60 0x55d7c0916950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.128+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/3894884599 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0906470 0x55d7c08ff550 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.128+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/3894884599 >> 172.21.15.44:0/3894884599 conn(0x55d7c08fb360 msgr2=0x55d7c091bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:32.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.129+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/3894884599 shutdown_connections 2023-11-16T16:53:32.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.129+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/3894884599 wait complete. 2023-11-16T16:53:32.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.129+0000 7f9d8f9d0b80 1 -- start start 2023-11-16T16:53:32.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.129+0000 7f9d8f9d0b80 1 Processor -- start 2023-11-16T16:53:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d8f9d0b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c0905b60 0x55d7c0914930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d8f9d0b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0906470 0x55d7c0914e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d8f9d0b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c090a8b0 0x55d7c09153b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d8f9d0b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c09726d0 con 0x55d7c090a8b0 2023-11-16T16:53:32.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d8f9d0b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c0972840 con 0x55d7c0905b60 2023-11-16T16:53:32.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d8f9d0b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c0972a00 con 0x55d7c0906470 2023-11-16T16:53:32.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d8f9d0b80 10 client.0 _ll_register_callbacks cb 0x7f9d700999e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:32.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d79d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c0905b60 0x55d7c0914930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d79d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c0905b60 0x55d7c0914930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56496/0 (socket says 172.21.15.44:56496) 2023-11-16T16:53:32.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d79d9b700 1 -- 172.21.15.44:0/2732896131 learned_addr learned my addr 172.21.15.44:0/2732896131 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:32.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d7959a700 1 --2- 172.21.15.44:0/2732896131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0906470 0x55d7c0914e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d7a59c700 1 --2- 172.21.15.44:0/2732896131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c090a8b0 0x55d7c09153b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d79d9b700 1 -- 172.21.15.44:0/2732896131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0906470 msgr2=0x55d7c0914e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:32.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d79d9b700 1 --2- 172.21.15.44:0/2732896131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0906470 0x55d7c0914e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d79d9b700 1 -- 172.21.15.44:0/2732896131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c090a8b0 msgr2=0x55d7c09153b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d79d9b700 1 --2- 172.21.15.44:0/2732896131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c090a8b0 0x55d7c09153b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d79d9b700 1 -- 172.21.15.44:0/2732896131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d7c06d8630 con 0x55d7c0905b60 2023-11-16T16:53:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d7959a700 1 --2- 172.21.15.44:0/2732896131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0906470 0x55d7c0914e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:53:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.130+0000 7f9d7a59c700 1 --2- 172.21.15.44:0/2732896131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c090a8b0 0x55d7c09153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:53:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.131+0000 7f9d79d9b700 1 --2- 172.21.15.44:0/2732896131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c0905b60 0x55d7c0914930 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9d70099ba0 tx=0x7f9d70086000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99c24e62e049bae1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.131+0000 7f9d5d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:53:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.131+0000 7f9d5d7fa700 1 -- 172.21.15.44:0/2732896131 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d700b1040 con 0x55d7c0905b60 2023-11-16T16:53:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.131+0000 7f9d5d7fa700 1 -- 172.21.15.44:0/2732896131 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f9d7009dc40 con 0x55d7c0905b60 2023-11-16T16:53:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.131+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2732896131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d6401e040 con 0x55d7c0905b60 2023-11-16T16:53:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.131+0000 7f9d5d7fa700 1 -- 172.21.15.44:0/2732896131 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d700ca1c0 con 0x55d7c0905b60 2023-11-16T16:53:32.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.131+0000 7f9d8f9d0b80 10 client.26381 fetch_fsmap learned FSMap version 21 2023-11-16T16:53:32.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.131+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2732896131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55d7c081cee0 con 0x55d7c0905b60 2023-11-16T16:53:32.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.131+0000 7f9d5d7fa700 1 -- 172.21.15.44:0/2732896131 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d6401e040 con 0x55d7c0905b60 2023-11-16T16:53:32.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.131+0000 7f9d5d7fa700 1 -- 172.21.15.44:0/2732896131 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9d70084b90 con 0x55d7c0905b60 2023-11-16T16:53:32.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.131+0000 7f9d8f9d0b80 10 client.26381 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:53:32.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.131+0000 7f9d8f9d0b80 10 client.26381 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:32.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.131+0000 7f9d8f9d0b80 20 client.26381 populate_metadata read hostname 'smithi044' 2023-11-16T16:53:32.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.131+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/2732896131 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d7c0973ce0 0x55d7c09760f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.132+0000 7f9d8f9d0b80 4 client.26381 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:53:32.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.132+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2732896131 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d7c0825660 con 0x55d7c0973ce0 2023-11-16T16:53:32.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.132+0000 7f9d7959a700 1 --2- 172.21.15.44:0/2732896131 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d7c0973ce0 0x55d7c09760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.132+0000 7f9d7959a700 1 --2- 172.21.15.44:0/2732896131 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d7c0973ce0 0x55d7c09760f0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1abb91c5b5d0d24f server_cookie=96c703aeec6c03ca in_seq=0 out_seq=0 2023-11-16T16:53:32.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.132+0000 7f9d5d7fa700 10 client.26381 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:32.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.133+0000 7f9d5d7fa700 1 -- 172.21.15.44:0/2732896131 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d7c0825660 con 0x55d7c0973ce0 2023-11-16T16:53:32.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.133+0000 7f9d5d7fa700 10 client.26381 handle_command_reply: tid=0 2023-11-16T16:53:32.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.133+0000 7f9d8f9d0b80 1 client.26381 shutdown 2023-11-16T16:53:32.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.133+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2732896131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c0905b60 msgr2=0x55d7c0914930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.133+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/2732896131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c0905b60 0x55d7c0914930 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9d70099ba0 tx=0x7f9d70086000 comp rx=0 tx=0).stop 2023-11-16T16:53:32.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.133+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2732896131 shutdown_connections 2023-11-16T16:53:32.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.133+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/2732896131 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d7c0973ce0 0x55d7c09760f0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.133+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/2732896131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d7c090a8b0 0x55d7c09153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.133+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/2732896131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7c0905b60 0x55d7c0914930 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.133+0000 7f9d5d7fa700 0 client.26381 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:32.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.133+0000 7f9d8f9d0b80 1 --2- 172.21.15.44:0/2732896131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d7c0906470 0x55d7c0914e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.133+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2732896131 >> 172.21.15.44:0/2732896131 conn(0x55d7c08fb360 msgr2=0x55d7c09089d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:32.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.134+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2732896131 shutdown_connections 2023-11-16T16:53:32.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.134+0000 7f9d8f9d0b80 1 -- 172.21.15.44:0/2732896131 wait complete. 2023-11-16T16:53:32.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.134+0000 7f9d8f9d0b80 20 client.26381 trim_cache size 0 max 16384 2023-11-16T16:53:32.140 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:53:32.140 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T16:53:32.140 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T16:53:32.140 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:53:32.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.134+0000 7f9d7bfff700 1 -- 172.21.15.44:0/3768744868 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9d600a1450 msgr2=0x7f9d600a3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.134+0000 7f9d7bfff700 1 --2- 172.21.15.44:0/3768744868 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9d600a1450 0x7f9d600a3900 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0094d0 tx=0x7f9d6c008350 comp rx=0 tx=0).stop 2023-11-16T16:53:32.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.134+0000 7f9d7bfff700 1 -- 172.21.15.44:0/3768744868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d74108c00 msgr2=0x7f9d74111670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.134+0000 7f9d7bfff700 1 --2- 172.21.15.44:0/3768744868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d74108c00 0x7f9d74111670 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9d7000c970 tx=0x7f9d7000cc80 comp rx=0 tx=0).stop 2023-11-16T16:53:32.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.135+0000 7f9d7bfff700 1 -- 172.21.15.44:0/3768744868 shutdown_connections 2023-11-16T16:53:32.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.135+0000 7f9d7bfff700 1 --2- 172.21.15.44:0/3768744868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d7410b6f0 0x7f9d7410bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.135+0000 7f9d7bfff700 1 --2- 172.21.15.44:0/3768744868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d74108c00 0x7f9d74111670 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.135+0000 7f9d7bfff700 1 --2- 172.21.15.44:0/3768744868 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9d600a1450 0x7f9d600a3900 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.135+0000 7f9d7bfff700 1 --2- 172.21.15.44:0/3768744868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d741121b0 0x7f9d74111bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.135+0000 7f9d7bfff700 1 -- 172.21.15.44:0/3768744868 >> 172.21.15.44:0/3768744868 conn(0x7f9d7406c7b0 msgr2=0x7f9d74114360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:32.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.135+0000 7f9d7bfff700 1 -- 172.21.15.44:0/3768744868 shutdown_connections 2023-11-16T16:53:32.142 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.135+0000 7f9d7bfff700 1 -- 172.21.15.44:0/3768744868 wait complete. 2023-11-16T16:53:32.156 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T16:53:32.157 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:53:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.375+0000 7fcc58bff700 1 -- 172.21.15.44:0/4156133446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc54100480 msgr2=0x7fcc54100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.375+0000 7fcc58bff700 1 --2- 172.21.15.44:0/4156133446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc54100480 0x7fcc54100850 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcc44009990 tx=0x7fcc44009ca0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.375+0000 7fcc58bff700 1 -- 172.21.15.44:0/4156133446 shutdown_connections 2023-11-16T16:53:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.375+0000 7fcc58bff700 1 --2- 172.21.15.44:0/4156133446 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc541054c0 0x7fcc5410d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.375+0000 7fcc58bff700 1 --2- 172.21.15.44:0/4156133446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc54100480 0x7fcc54100850 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.375+0000 7fcc58bff700 1 --2- 172.21.15.44:0/4156133446 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc54100e20 0x7fcc54104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.375+0000 7fcc58bff700 1 -- 172.21.15.44:0/4156133446 >> 172.21.15.44:0/4156133446 conn(0x7fcc540fbcf0 msgr2=0x7fcc540fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.375+0000 7fcc58bff700 1 -- 172.21.15.44:0/4156133446 shutdown_connections 2023-11-16T16:53:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.375+0000 7fcc58bff700 1 -- 172.21.15.44:0/4156133446 wait complete. 2023-11-16T16:53:32.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.376+0000 7fcc58bff700 1 Processor -- start 2023-11-16T16:53:32.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.377+0000 7fcc58bff700 1 -- start start 2023-11-16T16:53:32.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.377+0000 7fcc58bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc54100480 0x7fcc54198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.377+0000 7fcc58bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc54100e20 0x7fcc54198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.377+0000 7fcc58bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc541054c0 0x7fcc5419cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.377+0000 7fcc58bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5419d500 con 0x7fcc54100e20 2023-11-16T16:53:32.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.377+0000 7fcc58bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5419d670 con 0x7fcc541054c0 2023-11-16T16:53:32.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.377+0000 7fcc58bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5419d830 con 0x7fcc54100480 2023-11-16T16:53:32.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.377+0000 7fcc5259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc54100480 0x7fcc54198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.377+0000 7fcc5259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc54100480 0x7fcc54198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:46516/0 (socket says 172.21.15.44:46516) 2023-11-16T16:53:32.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.377+0000 7fcc5259c700 1 -- 172.21.15.44:0/2708291443 learned_addr learned my addr 172.21.15.44:0/2708291443 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:32.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.377+0000 7fcc52d9d700 1 --2- 172.21.15.44:0/2708291443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc541054c0 0x7fcc5419cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.377+0000 7fcc51d9b700 1 --2- 172.21.15.44:0/2708291443 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc54100e20 0x7fcc54198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.378+0000 7fcc5259c700 1 -- 172.21.15.44:0/2708291443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc541054c0 msgr2=0x7fcc5419cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.378+0000 7fcc5259c700 1 --2- 172.21.15.44:0/2708291443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc541054c0 0x7fcc5419cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.378+0000 7fcc5259c700 1 -- 172.21.15.44:0/2708291443 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc54100e20 msgr2=0x7fcc54198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.378+0000 7fcc5259c700 1 --2- 172.21.15.44:0/2708291443 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc54100e20 0x7fcc54198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.378+0000 7fcc5259c700 1 -- 172.21.15.44:0/2708291443 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc440095f0 con 0x7fcc54100480 2023-11-16T16:53:32.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.378+0000 7fcc52d9d700 1 --2- 172.21.15.44:0/2708291443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc541054c0 0x7fcc5419cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:32.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.378+0000 7fcc5259c700 1 --2- 172.21.15.44:0/2708291443 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc54100480 0x7fcc54198520 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcc44005910 tx=0x7fcc44005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:32.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.378+0000 7fcc437fe700 1 -- 172.21.15.44:0/2708291443 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4401d070 con 0x7fcc54100480 2023-11-16T16:53:32.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.378+0000 7fcc437fe700 1 -- 172.21.15.44:0/2708291443 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc44022470 con 0x7fcc54100480 2023-11-16T16:53:32.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.378+0000 7fcc437fe700 1 -- 172.21.15.44:0/2708291443 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44017650 con 0x7fcc54100480 2023-11-16T16:53:32.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.379+0000 7fcc58bff700 1 -- 172.21.15.44:0/2708291443 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc541102d0 con 0x7fcc54100480 2023-11-16T16:53:32.382 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.379+0000 7fcc58bff700 1 -- 172.21.15.44:0/2708291443 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc54110140 con 0x7fcc54100480 2023-11-16T16:53:32.382 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.380+0000 7fcc58bff700 1 -- 172.21.15.44:0/2708291443 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc54110460 con 0x7fcc54100480 2023-11-16T16:53:32.384 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.382+0000 7fcc437fe700 1 -- 172.21.15.44:0/2708291443 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fcc4400fad0 con 0x7fcc54100480 2023-11-16T16:53:32.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.384+0000 7fcc437fe700 1 --2- 172.21.15.44:0/2708291443 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fcc3c0a12c0 0x7fcc3c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.384+0000 7fcc437fe700 1 -- 172.21.15.44:0/2708291443 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fcc440cecb0 con 0x7fcc54100480 2023-11-16T16:53:32.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.384+0000 7fcc51d9b700 1 --2- 172.21.15.44:0/2708291443 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fcc3c0a12c0 0x7fcc3c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.385+0000 7fcc437fe700 1 -- 172.21.15.44:0/2708291443 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcc44097d70 con 0x7fcc54100480 2023-11-16T16:53:32.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.385+0000 7fcc51d9b700 1 --2- 172.21.15.44:0/2708291443 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fcc3c0a12c0 0x7fcc3c0a3770 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcc54199b30 tx=0x7fcc480073d0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:32.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.699+0000 7fcc58bff700 1 -- 172.21.15.44:0/2708291443 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc540679f0 con 0x7fcc54100480 2023-11-16T16:53:32.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.700+0000 7fcc437fe700 1 -- 172.21.15.44:0/2708291443 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7fcc4409b390 con 0x7fcc54100480 2023-11-16T16:53:32.702 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:53:32.702 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:53:32.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.702+0000 7fcc58bff700 1 -- 172.21.15.44:0/2708291443 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fcc3c0a12c0 msgr2=0x7fcc3c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.702+0000 7fcc58bff700 1 --2- 172.21.15.44:0/2708291443 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fcc3c0a12c0 0x7fcc3c0a3770 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcc54199b30 tx=0x7fcc480073d0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.702+0000 7fcc58bff700 1 -- 172.21.15.44:0/2708291443 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc54100480 msgr2=0x7fcc54198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.702+0000 7fcc58bff700 1 --2- 172.21.15.44:0/2708291443 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc54100480 0x7fcc54198520 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcc44005910 tx=0x7fcc44005740 comp rx=0 tx=0).stop 2023-11-16T16:53:32.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.702+0000 7fcc58bff700 1 -- 172.21.15.44:0/2708291443 shutdown_connections 2023-11-16T16:53:32.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.702+0000 7fcc58bff700 1 --2- 172.21.15.44:0/2708291443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc541054c0 0x7fcc5419cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.702+0000 7fcc58bff700 1 --2- 172.21.15.44:0/2708291443 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fcc3c0a12c0 0x7fcc3c0a3770 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.702+0000 7fcc58bff700 1 --2- 172.21.15.44:0/2708291443 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc54100480 0x7fcc54198520 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.702+0000 7fcc58bff700 1 --2- 172.21.15.44:0/2708291443 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc54100e20 0x7fcc54198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.702+0000 7fcc58bff700 1 -- 172.21.15.44:0/2708291443 >> 172.21.15.44:0/2708291443 conn(0x7fcc540fbcf0 msgr2=0x7fcc5410c360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:32.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.702+0000 7fcc58bff700 1 -- 172.21.15.44:0/2708291443 shutdown_connections 2023-11-16T16:53:32.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.702+0000 7fcc58bff700 1 -- 172.21.15.44:0/2708291443 wait complete. 2023-11-16T16:53:32.705 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:53:32.723 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T16:53:32.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.944+0000 7f17ad9ff700 1 -- 172.21.15.44:0/3310477752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a80ffa00 msgr2=0x7f17a810c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.944+0000 7f17ad9ff700 1 --2- 172.21.15.44:0/3310477752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a80ffa00 0x7f17a810c760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1798009990 tx=0x7f1798009ca0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.944+0000 7f17ad9ff700 1 -- 172.21.15.44:0/3310477752 shutdown_connections 2023-11-16T16:53:32.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.944+0000 7f17ad9ff700 1 --2- 172.21.15.44:0/3310477752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17a810cca0 0x7f17a810f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.944+0000 7f17ad9ff700 1 --2- 172.21.15.44:0/3310477752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17a80ff0f0 0x7f17a80ff4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.944+0000 7f17ad9ff700 1 --2- 172.21.15.44:0/3310477752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a80ffa00 0x7f17a810c760 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.944+0000 7f17ad9ff700 1 -- 172.21.15.44:0/3310477752 >> 172.21.15.44:0/3310477752 conn(0x7f17a80770b0 msgr2=0x7f17a80774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:32.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.945+0000 7f17ad9ff700 1 -- 172.21.15.44:0/3310477752 shutdown_connections 2023-11-16T16:53:32.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.945+0000 7f17ad9ff700 1 -- 172.21.15.44:0/3310477752 wait complete. 2023-11-16T16:53:32.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.946+0000 7f17ad9ff700 1 Processor -- start 2023-11-16T16:53:32.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.946+0000 7f17ad9ff700 1 -- start start 2023-11-16T16:53:32.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.946+0000 7f17ad9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a80ff0f0 0x7f17a8198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.946+0000 7f17ad9ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17a80ffa00 0x7f17a8198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.946+0000 7f17ad9ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17a810cca0 0x7f17a819cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.946+0000 7f17ad9ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a819d610 con 0x7f17a80ffa00 2023-11-16T16:53:32.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.946+0000 7f17ad9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a819d780 con 0x7f17a80ff0f0 2023-11-16T16:53:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.946+0000 7f17ad9ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a819d940 con 0x7f17a810cca0 2023-11-16T16:53:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.946+0000 7f17a67fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17a80ffa00 0x7f17a8198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.947+0000 7f17a67fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17a80ffa00 0x7f17a8198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44530/0 (socket says 172.21.15.44:44530) 2023-11-16T16:53:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.947+0000 7f17a67fc700 1 -- 172.21.15.44:0/2378962547 learned_addr learned my addr 172.21.15.44:0/2378962547 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.947+0000 7f17a6ffd700 1 --2- 172.21.15.44:0/2378962547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a80ff0f0 0x7f17a8198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.947+0000 7f17a77fe700 1 --2- 172.21.15.44:0/2378962547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17a810cca0 0x7f17a819cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.947+0000 7f17a6ffd700 1 -- 172.21.15.44:0/2378962547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17a810cca0 msgr2=0x7f17a819cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.947+0000 7f17a6ffd700 1 --2- 172.21.15.44:0/2378962547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17a810cca0 0x7f17a819cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.947+0000 7f17a6ffd700 1 -- 172.21.15.44:0/2378962547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17a80ffa00 msgr2=0x7f17a8198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.947+0000 7f17a6ffd700 1 --2- 172.21.15.44:0/2378962547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17a80ffa00 0x7f17a8198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.947+0000 7f17a6ffd700 1 -- 172.21.15.44:0/2378962547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17980095f0 con 0x7f17a80ff0f0 2023-11-16T16:53:32.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.947+0000 7f17a67fc700 1 --2- 172.21.15.44:0/2378962547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17a80ffa00 0x7f17a8198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:32.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.947+0000 7f17a77fe700 1 --2- 172.21.15.44:0/2378962547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17a810cca0 0x7f17a819cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:32.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.947+0000 7f17a6ffd700 1 --2- 172.21.15.44:0/2378962547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a80ff0f0 0x7f17a8198630 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f179000e9f0 tx=0x7f179000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:32.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.947+0000 7f17ac9fd700 1 -- 172.21.15.44:0/2378962547 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179000c9c0 con 0x7f17a80ff0f0 2023-11-16T16:53:32.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.947+0000 7f17ac9fd700 1 -- 172.21.15.44:0/2378962547 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1790010430 con 0x7f17a80ff0f0 2023-11-16T16:53:32.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.948+0000 7f17ad9ff700 1 -- 172.21.15.44:0/2378962547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a81125a0 con 0x7f17a80ff0f0 2023-11-16T16:53:32.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.948+0000 7f17ac9fd700 1 -- 172.21.15.44:0/2378962547 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1790018680 con 0x7f17a80ff0f0 2023-11-16T16:53:32.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.948+0000 7f17ad9ff700 1 -- 172.21.15.44:0/2378962547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a8112410 con 0x7f17a80ff0f0 2023-11-16T16:53:32.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.951+0000 7f17ac9fd700 1 -- 172.21.15.44:0/2378962547 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f1790010860 con 0x7f17a80ff0f0 2023-11-16T16:53:32.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.952+0000 7f17c141fb80 1 Processor -- start 2023-11-16T16:53:32.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.952+0000 7f17c141fb80 1 -- start start 2023-11-16T16:53:32.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.953+0000 7f17c141fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bcb60 0x55f1580bcf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.953+0000 7f17c141fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580bd470 0x55f1580c1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.953+0000 7f17ac9fd700 1 --2- 172.21.15.44:0/2378962547 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f17940a1340 0x7f17940a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.953+0000 7f17c141fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580c18b0 0x55f1580c1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.953+0000 7f17c141fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f157fc2060 con 0x55f1580bd470 2023-11-16T16:53:32.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.953+0000 7f17c141fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1580148a0 con 0x55f1580bcb60 2023-11-16T16:53:32.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.953+0000 7f17c141fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f15800dd30 con 0x55f1580c18b0 2023-11-16T16:53:32.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.953+0000 7f17ac9fd700 1 -- 172.21.15.44:0/2378962547 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f1790014070 con 0x7f17a80ff0f0 2023-11-16T16:53:32.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.953+0000 7f17a77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bcb60 0x55f1580bcf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.953+0000 7f17a77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bcb60 0x55f1580bcf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56540/0 (socket says 172.21.15.44:56540) 2023-11-16T16:53:32.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.953+0000 7f17a77fe700 1 -- 172.21.15.44:0/1336804008 learned_addr learned my addr 172.21.15.44:0/1336804008 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:32.956 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.953+0000 7f17a67fc700 1 --2- 172.21.15.44:0/1336804008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580bd470 0x55f1580c1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.953+0000 7f17a77fe700 1 --2- 172.21.15.44:0/1336804008 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580c18b0 0x55f1580c1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.953+0000 7f17a6ffd700 1 --2- 172.21.15.44:0/2378962547 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f17940a1340 0x7f17940a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.954+0000 7f17a77fe700 1 -- 172.21.15.44:0/1336804008 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580c18b0 msgr2=0x55f1580c1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:32.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.954+0000 7f17a77fe700 1 --2- 172.21.15.44:0/1336804008 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580c18b0 0x55f1580c1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.954+0000 7f17a77fe700 1 -- 172.21.15.44:0/1336804008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580bd470 msgr2=0x55f1580c1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:32.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.954+0000 7f17a77fe700 1 --2- 172.21.15.44:0/1336804008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580bd470 0x55f1580c1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.954+0000 7f17a77fe700 1 -- 172.21.15.44:0/1336804008 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f157e8f630 con 0x55f1580bcb60 2023-11-16T16:53:32.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.954+0000 7f17a77fe700 1 --2- 172.21.15.44:0/1336804008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bcb60 0x55f1580bcf30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f17a0003f50 tx=0x7f17a0009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=530bb2a3fdcceb30 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:32.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.954+0000 7f179dffb700 1 -- 172.21.15.44:0/1336804008 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a0009310 con 0x55f1580bcb60 2023-11-16T16:53:32.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.954+0000 7f179dffb700 1 -- 172.21.15.44:0/1336804008 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17a000f040 con 0x55f1580bcb60 2023-11-16T16:53:32.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.954+0000 7f179dffb700 1 -- 172.21.15.44:0/1336804008 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a0007c00 con 0x55f1580bcb60 2023-11-16T16:53:32.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.955+0000 7f17c141fb80 1 -- 172.21.15.44:0/1336804008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bcb60 msgr2=0x55f1580bcf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.955+0000 7f17a6ffd700 1 --2- 172.21.15.44:0/2378962547 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f17940a1340 0x7f17940a37f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1790009800 tx=0x7f1790009790 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:32.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.955+0000 7f17c141fb80 1 --2- 172.21.15.44:0/1336804008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bcb60 0x55f1580bcf30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f17a0003f50 tx=0x7f17a0009920 comp rx=0 tx=0).stop 2023-11-16T16:53:32.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.955+0000 7f17c141fb80 1 -- 172.21.15.44:0/1336804008 shutdown_connections 2023-11-16T16:53:32.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.955+0000 7f17c141fb80 1 --2- 172.21.15.44:0/1336804008 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580c18b0 0x55f1580c1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.955+0000 7f17c141fb80 1 --2- 172.21.15.44:0/1336804008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bcb60 0x55f1580bcf30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.955+0000 7f17c141fb80 1 --2- 172.21.15.44:0/1336804008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580bd470 0x55f1580c1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.955+0000 7f17c141fb80 1 -- 172.21.15.44:0/1336804008 >> 172.21.15.44:0/1336804008 conn(0x55f1580b2360 msgr2=0x55f1580b7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:32.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.955+0000 7f17c141fb80 1 -- 172.21.15.44:0/1336804008 shutdown_connections 2023-11-16T16:53:32.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.955+0000 7f17c141fb80 1 -- 172.21.15.44:0/1336804008 wait complete. 2023-11-16T16:53:32.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.955+0000 7f17c141fb80 1 -- start start 2023-11-16T16:53:32.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.956+0000 7f17c141fb80 1 Processor -- start 2023-11-16T16:53:32.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.956+0000 7f17c141fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bcb60 0x55f1580c9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.956+0000 7f17c141fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580bd470 0x55f1580c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.956+0000 7f17c141fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580c18b0 0x55f1580c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.956+0000 7f17c141fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f157faf8e0 con 0x55f1580c18b0 2023-11-16T16:53:32.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.956+0000 7f17c141fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f157faf580 con 0x55f1580bcb60 2023-11-16T16:53:32.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.956+0000 7f17c141fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f157e91920 con 0x55f1580bd470 2023-11-16T16:53:32.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.956+0000 7f17c141fb80 10 client.0 _ll_register_callbacks cb 0x55f157fcd780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:32.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.956+0000 7f17a77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bcb60 0x55f1580c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.956+0000 7f17a77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bcb60 0x55f1580c9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56550/0 (socket says 172.21.15.44:56550) 2023-11-16T16:53:32.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.956+0000 7f17a77fe700 1 -- 172.21.15.44:0/229935245 learned_addr learned my addr 172.21.15.44:0/229935245 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:32.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.956+0000 7f17a67fc700 1 --2- 172.21.15.44:0/229935245 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580bd470 0x55f1580c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.956+0000 7f17a77fe700 1 --2- 172.21.15.44:0/229935245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580c18b0 0x55f1580c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.957+0000 7f17a77fe700 1 -- 172.21.15.44:0/229935245 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580bd470 msgr2=0x55f1580c62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:32.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.957+0000 7f17a77fe700 1 --2- 172.21.15.44:0/229935245 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580bd470 0x55f1580c62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.957+0000 7f17a77fe700 1 -- 172.21.15.44:0/229935245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580c18b0 msgr2=0x55f1580c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:32.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.957+0000 7f17a77fe700 1 --2- 172.21.15.44:0/229935245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580c18b0 0x55f1580c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.957+0000 7f17a77fe700 1 -- 172.21.15.44:0/229935245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a0004040 con 0x55f1580bcb60 2023-11-16T16:53:32.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.957+0000 7f17a77fe700 1 --2- 172.21.15.44:0/229935245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bcb60 0x55f1580c9050 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f17a000efd0 tx=0x7f17a000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6445713f0507a84f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:32.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.957+0000 7f1786ffd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:53:32.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.957+0000 7f1786ffd700 1 -- 172.21.15.44:0/229935245 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a00079e0 con 0x55f1580bcb60 2023-11-16T16:53:32.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.957+0000 7f1786ffd700 1 -- 172.21.15.44:0/229935245 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f17a000ed30 con 0x55f1580bcb60 2023-11-16T16:53:32.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.957+0000 7f17c141fb80 1 -- 172.21.15.44:0/229935245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f1580afb20 con 0x55f1580bcb60 2023-11-16T16:53:32.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.957+0000 7f1786ffd700 1 -- 172.21.15.44:0/229935245 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17a0007b40 con 0x55f1580bcb60 2023-11-16T16:53:32.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.957+0000 7f17c141fb80 10 client.26399 fetch_fsmap learned FSMap version 21 2023-11-16T16:53:32.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.957+0000 7f17c141fb80 1 -- 172.21.15.44:0/229935245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55f157fd4190 con 0x55f1580bcb60 2023-11-16T16:53:32.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.958+0000 7f1786ffd700 1 -- 172.21.15.44:0/229935245 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f1580afb20 con 0x55f1580bcb60 2023-11-16T16:53:32.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.958+0000 7f1786ffd700 1 -- 172.21.15.44:0/229935245 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f17a0018b90 con 0x55f1580bcb60 2023-11-16T16:53:32.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.958+0000 7f17c141fb80 10 client.26399 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:53:32.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.958+0000 7f17c141fb80 10 client.26399 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:32.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.958+0000 7f17c141fb80 20 client.26399 populate_metadata read hostname 'smithi044' 2023-11-16T16:53:32.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.958+0000 7f17c141fb80 1 --2- 172.21.15.44:0/229935245 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f1580d67b0 0x55f1580d8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.958+0000 7f17c141fb80 4 client.26399 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:53:32.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.958+0000 7f17c141fb80 1 -- 172.21.15.44:0/229935245 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f157fd00a0 con 0x55f1580d67b0 2023-11-16T16:53:32.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.958+0000 7f17a67fc700 1 --2- 172.21.15.44:0/229935245 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f1580d67b0 0x55f1580d8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.959+0000 7f17a67fc700 1 --2- 172.21.15.44:0/229935245 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f1580d67b0 0x55f1580d8bc0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=656d7f70d9df184d server_cookie=8bb52f3319891897 in_seq=0 out_seq=0 2023-11-16T16:53:32.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.959+0000 7f1786ffd700 10 client.26399 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:32.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.962+0000 7f1786ffd700 1 -- 172.21.15.44:0/229935245 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f157fd00a0 con 0x55f1580d67b0 2023-11-16T16:53:32.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.962+0000 7f1786ffd700 10 client.26399 handle_command_reply: tid=0 2023-11-16T16:53:32.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.962+0000 7f17c141fb80 1 client.26399 shutdown 2023-11-16T16:53:32.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.962+0000 7f17c141fb80 1 -- 172.21.15.44:0/229935245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bcb60 msgr2=0x55f1580c9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.962+0000 7f17c141fb80 1 --2- 172.21.15.44:0/229935245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bcb60 0x55f1580c9050 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f17a000efd0 tx=0x7f17a000b040 comp rx=0 tx=0).stop 2023-11-16T16:53:32.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.962+0000 7f17c141fb80 1 -- 172.21.15.44:0/229935245 shutdown_connections 2023-11-16T16:53:32.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.962+0000 7f17c141fb80 1 --2- 172.21.15.44:0/229935245 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f1580d67b0 0x55f1580d8bc0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.962+0000 7f17c141fb80 1 --2- 172.21.15.44:0/229935245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580c18b0 0x55f1580c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.962+0000 7f17c141fb80 1 --2- 172.21.15.44:0/229935245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bcb60 0x55f1580c9050 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.962+0000 7f17c141fb80 1 --2- 172.21.15.44:0/229935245 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580bd470 0x55f1580c62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.962+0000 7f1786ffd700 0 client.26399 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:32.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.962+0000 7f17c141fb80 1 -- 172.21.15.44:0/229935245 >> 172.21.15.44:0/229935245 conn(0x55f1580b2360 msgr2=0x55f1580b6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:32.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.962+0000 7f17c141fb80 1 -- 172.21.15.44:0/229935245 shutdown_connections 2023-11-16T16:53:32.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.962+0000 7f17c141fb80 1 -- 172.21.15.44:0/229935245 wait complete. 2023-11-16T16:53:32.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.962+0000 7f17c141fb80 20 client.26399 trim_cache size 0 max 16384 2023-11-16T16:53:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.976+0000 7f17c141fb80 1 Processor -- start 2023-11-16T16:53:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.976+0000 7f17c141fb80 1 -- start start 2023-11-16T16:53:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.976+0000 7f17c141fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580bcb60 0x55f1580c01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.976+0000 7f17c141fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580bd470 0x55f1580cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.976+0000 7f17c141fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580c18b0 0x55f1580b7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.976+0000 7f17c141fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f157fd00a0 con 0x55f1580bcb60 2023-11-16T16:53:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.976+0000 7f17c141fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1580af710 con 0x55f1580c18b0 2023-11-16T16:53:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.976+0000 7f17c141fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1580c7290 con 0x55f1580bd470 2023-11-16T16:53:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.977+0000 7f17a77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580bcb60 0x55f1580c01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.977+0000 7f17a77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580bcb60 0x55f1580c01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44560/0 (socket says 172.21.15.44:44560) 2023-11-16T16:53:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.977+0000 7f17a77fe700 1 -- 172.21.15.44:0/700435473 learned_addr learned my addr 172.21.15.44:0/700435473 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.977+0000 7f17a67fc700 1 --2- 172.21.15.44:0/700435473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580bd470 0x55f1580cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.977+0000 7f17a77fe700 1 --2- 172.21.15.44:0/700435473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580c18b0 0x55f1580b7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.977+0000 7f17a77fe700 1 -- 172.21.15.44:0/700435473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580bd470 msgr2=0x55f1580cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.977+0000 7f17a77fe700 1 --2- 172.21.15.44:0/700435473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580bd470 0x55f1580cd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.977+0000 7f17a77fe700 1 -- 172.21.15.44:0/700435473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580c18b0 msgr2=0x55f1580b7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.977+0000 7f17a77fe700 1 --2- 172.21.15.44:0/700435473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580c18b0 0x55f1580b7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.977+0000 7f17a77fe700 1 -- 172.21.15.44:0/700435473 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1580b3b00 con 0x55f1580bcb60 2023-11-16T16:53:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.977+0000 7f17a77fe700 1 --2- 172.21.15.44:0/700435473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580bcb60 0x55f1580c01a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f17a000e850 tx=0x7f17a0000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4415397a8a8f64f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.977+0000 7f179e7fc700 1 -- 172.21.15.44:0/700435473 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a0009a00 con 0x55f1580bcb60 2023-11-16T16:53:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.978+0000 7f179e7fc700 1 -- 172.21.15.44:0/700435473 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17a0003ab0 con 0x55f1580bcb60 2023-11-16T16:53:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.978+0000 7f179e7fc700 1 -- 172.21.15.44:0/700435473 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a0007d10 con 0x55f1580bcb60 2023-11-16T16:53:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.978+0000 7f17c141fb80 1 -- 172.21.15.44:0/700435473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580bcb60 msgr2=0x55f1580c01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.978+0000 7f17c141fb80 1 --2- 172.21.15.44:0/700435473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580bcb60 0x55f1580c01a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f17a000e850 tx=0x7f17a0000f40 comp rx=0 tx=0).stop 2023-11-16T16:53:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.978+0000 7f17c141fb80 1 -- 172.21.15.44:0/700435473 shutdown_connections 2023-11-16T16:53:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.978+0000 7f17c141fb80 1 --2- 172.21.15.44:0/700435473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580c18b0 0x55f1580b7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.978+0000 7f17c141fb80 1 --2- 172.21.15.44:0/700435473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580bcb60 0x55f1580c01a0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.978+0000 7f17c141fb80 1 --2- 172.21.15.44:0/700435473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580bd470 0x55f1580cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.978+0000 7f17c141fb80 1 -- 172.21.15.44:0/700435473 >> 172.21.15.44:0/700435473 conn(0x55f1580b2360 msgr2=0x55f1580c1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.978+0000 7f17c141fb80 1 -- 172.21.15.44:0/700435473 shutdown_connections 2023-11-16T16:53:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.978+0000 7f17c141fb80 1 -- 172.21.15.44:0/700435473 wait complete. 2023-11-16T16:53:32.984 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.979+0000 7f17c141fb80 1 -- start start 2023-11-16T16:53:32.984 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.979+0000 7f17c141fb80 1 Processor -- start 2023-11-16T16:53:32.984 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.979+0000 7f17c141fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580bcb60 0x55f1580cb350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.984 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.979+0000 7f17c141fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bd470 0x55f1580cb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.984 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.979+0000 7f17c141fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580c18b0 0x55f158129670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.984 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.979+0000 7f17c141fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1580cc030 con 0x55f1580c18b0 2023-11-16T16:53:32.984 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.979+0000 7f17c141fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1580cc1a0 con 0x55f1580bd470 2023-11-16T16:53:32.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.979+0000 7f17c141fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f158129bb0 con 0x55f1580bcb60 2023-11-16T16:53:32.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.979+0000 7f17c141fb80 10 client.0 _ll_register_callbacks cb 0x7f17a000e880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:32.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.980+0000 7f17a67fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bd470 0x55f1580cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.980+0000 7f17a77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580bcb60 0x55f1580cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.980+0000 7f17a67fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bd470 0x55f1580cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56560/0 (socket says 172.21.15.44:56560) 2023-11-16T16:53:32.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.980+0000 7f17a67fc700 1 -- 172.21.15.44:0/2069655065 learned_addr learned my addr 172.21.15.44:0/2069655065 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:32.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.980+0000 7f17a77fe700 1 --2- 172.21.15.44:0/2069655065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580c18b0 0x55f158129670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.980+0000 7f17a67fc700 1 -- 172.21.15.44:0/2069655065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580bcb60 msgr2=0x55f1580cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:32.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.980+0000 7f17a67fc700 1 --2- 172.21.15.44:0/2069655065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580bcb60 0x55f1580cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.980+0000 7f17a67fc700 1 -- 172.21.15.44:0/2069655065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580c18b0 msgr2=0x55f158129670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:32.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.980+0000 7f17a67fc700 1 --2- 172.21.15.44:0/2069655065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580c18b0 0x55f158129670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.980+0000 7f17a67fc700 1 -- 172.21.15.44:0/2069655065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f157fd4190 con 0x55f1580bd470 2023-11-16T16:53:32.985 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.980+0000 7f17a77fe700 1 --2- 172.21.15.44:0/2069655065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580c18b0 0x55f158129670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:53:32.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.980+0000 7f17a67fc700 1 --2- 172.21.15.44:0/2069655065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bd470 0x55f1580cb890 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1798006010 tx=0x7f179800bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=871dbd09c656d762 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:32.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.980+0000 7f1786ffd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:53:32.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.981+0000 7f1786ffd700 1 -- 172.21.15.44:0/2069655065 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179801d040 con 0x55f1580bd470 2023-11-16T16:53:32.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.981+0000 7f17c141fb80 1 -- 172.21.15.44:0/2069655065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f157fdc660 con 0x55f1580bd470 2023-11-16T16:53:32.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.981+0000 7f1786ffd700 1 -- 172.21.15.44:0/2069655065 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f1798005170 con 0x55f1580bd470 2023-11-16T16:53:32.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.981+0000 7f1786ffd700 1 -- 172.21.15.44:0/2069655065 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1798005330 con 0x55f1580bd470 2023-11-16T16:53:32.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.981+0000 7f17c141fb80 10 client.26405 fetch_fsmap learned FSMap version 21 2023-11-16T16:53:32.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.981+0000 7f17c141fb80 1 -- 172.21.15.44:0/2069655065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55f157e3bde0 con 0x55f1580bd470 2023-11-16T16:53:32.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.981+0000 7f1786ffd700 1 -- 172.21.15.44:0/2069655065 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f157fdc660 con 0x55f1580bd470 2023-11-16T16:53:32.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.981+0000 7f1786ffd700 1 -- 172.21.15.44:0/2069655065 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f179802e020 con 0x55f1580bd470 2023-11-16T16:53:32.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.981+0000 7f17c141fb80 10 client.26405 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:53:32.986 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.981+0000 7f17c141fb80 10 client.26405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:32.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.981+0000 7f17c141fb80 20 client.26405 populate_metadata read hostname 'smithi044' 2023-11-16T16:53:32.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.981+0000 7f17c141fb80 1 --2- 172.21.15.44:0/2069655065 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f15812b130 0x55f15812d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:32.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.981+0000 7f17c141fb80 4 client.26405 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T16:53:32.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.981+0000 7f17c141fb80 1 -- 172.21.15.44:0/2069655065 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f157e5cf70 con 0x55f15812b130 2023-11-16T16:53:32.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.981+0000 7f17a77fe700 1 --2- 172.21.15.44:0/2069655065 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f15812b130 0x55f15812d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:32.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.982+0000 7f17a77fe700 1 --2- 172.21.15.44:0/2069655065 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f15812b130 0x55f15812d540 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e9cca2d94c85c5a server_cookie=1d3804dc1cb026d in_seq=0 out_seq=0 2023-11-16T16:53:32.987 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.982+0000 7f1786ffd700 10 client.26405 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:32.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.985+0000 7f1786ffd700 1 -- 172.21.15.44:0/2069655065 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f157e5cf70 con 0x55f15812b130 2023-11-16T16:53:32.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.985+0000 7f1786ffd700 10 client.26405 handle_command_reply: tid=0 2023-11-16T16:53:32.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.986+0000 7f17c141fb80 1 client.26405 shutdown 2023-11-16T16:53:32.988 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.986+0000 7f17c141fb80 1 -- 172.21.15.44:0/2069655065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bd470 msgr2=0x55f1580cb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.986+0000 7f17c141fb80 1 --2- 172.21.15.44:0/2069655065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bd470 0x55f1580cb890 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1798006010 tx=0x7f179800bfd0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.986+0000 7f17c141fb80 1 -- 172.21.15.44:0/2069655065 shutdown_connections 2023-11-16T16:53:32.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.986+0000 7f17c141fb80 1 --2- 172.21.15.44:0/2069655065 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f15812b130 0x55f15812d540 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.986+0000 7f17c141fb80 1 --2- 172.21.15.44:0/2069655065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1580c18b0 0x55f158129670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.986+0000 7f17c141fb80 1 --2- 172.21.15.44:0/2069655065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f1580bcb60 0x55f1580cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.986+0000 7f17c141fb80 1 --2- 172.21.15.44:0/2069655065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f1580bd470 0x55f1580cb890 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.986+0000 7f1786ffd700 0 client.26405 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:32.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.986+0000 7f17c141fb80 1 -- 172.21.15.44:0/2069655065 >> 172.21.15.44:0/2069655065 conn(0x55f1580b2360 msgr2=0x55f1580bb960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:32.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.986+0000 7f17c141fb80 1 -- 172.21.15.44:0/2069655065 shutdown_connections 2023-11-16T16:53:32.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.986+0000 7f17c141fb80 1 -- 172.21.15.44:0/2069655065 wait complete. 2023-11-16T16:53:32.989 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.986+0000 7f17c141fb80 20 client.26405 trim_cache size 0 max 16384 2023-11-16T16:53:32.990 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T16:53:32.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.987+0000 7f17ad9ff700 1 -- 172.21.15.44:0/2378962547 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f17940a1340 msgr2=0x7f17940a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.987+0000 7f17ad9ff700 1 --2- 172.21.15.44:0/2378962547 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f17940a1340 0x7f17940a37f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1790009800 tx=0x7f1790009790 comp rx=0 tx=0).stop 2023-11-16T16:53:32.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.987+0000 7f17ad9ff700 1 -- 172.21.15.44:0/2378962547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a80ff0f0 msgr2=0x7f17a8198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:32.990 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.987+0000 7f17ad9ff700 1 --2- 172.21.15.44:0/2378962547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a80ff0f0 0x7f17a8198630 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f179000e9f0 tx=0x7f179000ed00 comp rx=0 tx=0).stop 2023-11-16T16:53:32.991 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.987+0000 7f17ad9ff700 1 -- 172.21.15.44:0/2378962547 shutdown_connections 2023-11-16T16:53:32.991 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.987+0000 7f17ad9ff700 1 --2- 172.21.15.44:0/2378962547 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f17940a1340 0x7f17940a37f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.991 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.987+0000 7f17ad9ff700 1 --2- 172.21.15.44:0/2378962547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17a810cca0 0x7f17a819cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.991 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.987+0000 7f17ad9ff700 1 --2- 172.21.15.44:0/2378962547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a80ff0f0 0x7f17a8198630 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.991 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.987+0000 7f17ad9ff700 1 --2- 172.21.15.44:0/2378962547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17a80ffa00 0x7f17a8198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:32.991 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.987+0000 7f17ad9ff700 1 -- 172.21.15.44:0/2378962547 >> 172.21.15.44:0/2378962547 conn(0x7f17a80770b0 msgr2=0x7f17a810dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:32.991 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.987+0000 7f17ad9ff700 1 -- 172.21.15.44:0/2378962547 shutdown_connections 2023-11-16T16:53:32.991 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:32.987+0000 7f17ad9ff700 1 -- 172.21.15.44:0/2378962547 wait complete. 2023-11-16T16:53:33.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:33 smithi044 ceph-mon[128750]: pgmap v111: 129 pgs: 129 active+clean; 43 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 1.87k op/s 2023-11-16T16:53:33.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:33 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2708291443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:53:33.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:33 smithi176 ceph-mon[131630]: pgmap v111: 129 pgs: 129 active+clean; 43 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 1.87k op/s 2023-11-16T16:53:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:33 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2708291443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:53:33.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:33 smithi077 ceph-mon[131204]: pgmap v111: 129 pgs: 129 active+clean; 43 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 1.87k op/s 2023-11-16T16:53:33.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:33 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2708291443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:53:34.009 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:53:34.010 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:53:34.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.266+0000 7f366cdff700 1 -- 172.21.15.44:0/2582252855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3668100cc0 msgr2=0x7f3668101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.266+0000 7f366cdff700 1 --2- 172.21.15.44:0/2582252855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3668100cc0 0x7f3668101090 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f36580099c0 tx=0x7f3658009cd0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.266+0000 7f366cdff700 1 -- 172.21.15.44:0/2582252855 shutdown_connections 2023-11-16T16:53:34.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.266+0000 7f366cdff700 1 --2- 172.21.15.44:0/2582252855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3668105620 0x7f366810d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.266+0000 7f366cdff700 1 --2- 172.21.15.44:0/2582252855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3668100cc0 0x7f3668101090 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.266+0000 7f366cdff700 1 --2- 172.21.15.44:0/2582252855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3668101660 0x7f3668104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.266+0000 7f366cdff700 1 -- 172.21.15.44:0/2582252855 >> 172.21.15.44:0/2582252855 conn(0x7f3668077560 msgr2=0x7f3668077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:34.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.266+0000 7f366cdff700 1 -- 172.21.15.44:0/2582252855 shutdown_connections 2023-11-16T16:53:34.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.266+0000 7f366cdff700 1 -- 172.21.15.44:0/2582252855 wait complete. 2023-11-16T16:53:34.269 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.267+0000 7f366cdff700 1 Processor -- start 2023-11-16T16:53:34.269 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.268+0000 7f366cdff700 1 -- start start 2023-11-16T16:53:34.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.268+0000 7f366cdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3668100cc0 0x7f36681986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.268+0000 7f366cdff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3668101660 0x7f3668198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.268+0000 7f366cdff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3668105620 0x7f366819bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.268+0000 7f366cdff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366819c5a0 con 0x7f3668101660 2023-11-16T16:53:34.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.268+0000 7f366cdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366819c710 con 0x7f3668100cc0 2023-11-16T16:53:34.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.268+0000 7f366cdff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366819c8d0 con 0x7f3668105620 2023-11-16T16:53:34.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.268+0000 7f366659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3668100cc0 0x7f36681986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.268+0000 7f366659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3668100cc0 0x7f36681986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56588/0 (socket says 172.21.15.44:56588) 2023-11-16T16:53:34.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.268+0000 7f3665d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3668101660 0x7f3668198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.268+0000 7f366659c700 1 -- 172.21.15.44:0/2056618222 learned_addr learned my addr 172.21.15.44:0/2056618222 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:34.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.269+0000 7f3666d9d700 1 --2- 172.21.15.44:0/2056618222 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3668105620 0x7f366819bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.269+0000 7f366659c700 1 -- 172.21.15.44:0/2056618222 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3668105620 msgr2=0x7f366819bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.269+0000 7f366659c700 1 --2- 172.21.15.44:0/2056618222 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3668105620 0x7f366819bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.269+0000 7f366659c700 1 -- 172.21.15.44:0/2056618222 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3668101660 msgr2=0x7f3668198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.269+0000 7f366659c700 1 --2- 172.21.15.44:0/2056618222 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3668101660 0x7f3668198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.269+0000 7f366659c700 1 -- 172.21.15.44:0/2056618222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36580095f0 con 0x7f3668100cc0 2023-11-16T16:53:34.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.269+0000 7f366659c700 1 --2- 172.21.15.44:0/2056618222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3668100cc0 0x7f36681986b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3658009990 tx=0x7f3658005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:34.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.270+0000 7f36577fe700 1 -- 172.21.15.44:0/2056618222 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365801d070 con 0x7f3668100cc0 2023-11-16T16:53:34.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.270+0000 7f36577fe700 1 -- 172.21.15.44:0/2056618222 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3658003f90 con 0x7f3668100cc0 2023-11-16T16:53:34.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.270+0000 7f366cdff700 1 -- 172.21.15.44:0/2056618222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36681103d0 con 0x7f3668100cc0 2023-11-16T16:53:34.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.270+0000 7f366cdff700 1 -- 172.21.15.44:0/2056618222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3668110240 con 0x7f3668100cc0 2023-11-16T16:53:34.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.270+0000 7f36577fe700 1 -- 172.21.15.44:0/2056618222 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3658022ba0 con 0x7f3668100cc0 2023-11-16T16:53:34.277 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.274+0000 7f36577fe700 1 -- 172.21.15.44:0/2056618222 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f3658003ad0 con 0x7f3668100cc0 2023-11-16T16:53:34.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.276+0000 7f36577fe700 1 --2- 172.21.15.44:0/2056618222 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f36500a1340 0x7f36500a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.276+0000 7f36577fe700 1 -- 172.21.15.44:0/2056618222 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f36580ce4d0 con 0x7f3668100cc0 2023-11-16T16:53:34.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.277+0000 7f36806d4b80 1 Processor -- start 2023-11-16T16:53:34.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.277+0000 7f3665d9b700 1 --2- 172.21.15.44:0/2056618222 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f36500a1340 0x7f36500a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.277+0000 7f36806d4b80 1 -- start start 2023-11-16T16:53:34.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.277+0000 7f36806d4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9ebe0 0x556f25e9efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.277+0000 7f36806d4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 0x556f25ea33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.277+0000 7f36806d4b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 0x556f25ea3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.277+0000 7f36806d4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f25da40e0 con 0x556f25e9f4f0 2023-11-16T16:53:34.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.277+0000 7f36806d4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f25df6920 con 0x556f25e9ebe0 2023-11-16T16:53:34.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.277+0000 7f36806d4b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f25defdb0 con 0x556f25ea3930 2023-11-16T16:53:34.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.278+0000 7f3665d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 0x556f25ea33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.278+0000 7f366659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9ebe0 0x556f25e9efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.278+0000 7f3666d9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 0x556f25ea3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.278+0000 7f3666d9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 0x556f25ea3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:46606/0 (socket says 172.21.15.44:46606) 2023-11-16T16:53:34.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.278+0000 7f3665d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 0x556f25ea33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44616/0 (socket says 172.21.15.44:44616) 2023-11-16T16:53:34.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.278+0000 7f366659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9ebe0 0x556f25e9efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56604/0 (socket says 172.21.15.44:56604) 2023-11-16T16:53:34.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.278+0000 7f3666d9d700 1 -- 172.21.15.44:0/1482142611 learned_addr learned my addr 172.21.15.44:0/1482142611 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:34.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.278+0000 7f3665d9b700 1 --2- 172.21.15.44:0/2056618222 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f36500a1340 0x7f36500a37f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3668199cc0 tx=0x7f365c00b410 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:34.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.278+0000 7f3666d9d700 1 -- 172.21.15.44:0/1482142611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9ebe0 msgr2=0x556f25e9efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:34.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.278+0000 7f3666d9d700 1 --2- 172.21.15.44:0/1482142611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9ebe0 0x556f25e9efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.278+0000 7f3666d9d700 1 -- 172.21.15.44:0/1482142611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 msgr2=0x556f25ea33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:34.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.278+0000 7f3666d9d700 1 --2- 172.21.15.44:0/1482142611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 0x556f25ea33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.278+0000 7f3666d9d700 1 -- 172.21.15.44:0/1482142611 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f25c716b0 con 0x556f25ea3930 2023-11-16T16:53:34.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.278+0000 7f3665d9b700 1 --2- 172.21.15.44:0/1482142611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 0x556f25ea33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:34.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.279+0000 7f3666d9d700 1 --2- 172.21.15.44:0/1482142611 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 0x556f25ea3d80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3660003fa0 tx=0x7f3660009ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85789ef54e37a645 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:34.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.279+0000 7f3654ff9700 1 -- 172.21.15.44:0/1482142611 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3660019070 con 0x556f25ea3930 2023-11-16T16:53:34.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.279+0000 7f3654ff9700 1 -- 172.21.15.44:0/1482142611 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f366000f040 con 0x556f25ea3930 2023-11-16T16:53:34.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.279+0000 7f3654ff9700 1 -- 172.21.15.44:0/1482142611 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3660007a60 con 0x556f25ea3930 2023-11-16T16:53:34.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.280+0000 7f36806d4b80 1 -- 172.21.15.44:0/1482142611 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 msgr2=0x556f25ea3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.280+0000 7f36806d4b80 1 --2- 172.21.15.44:0/1482142611 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 0x556f25ea3d80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3660003fa0 tx=0x7f3660009ba0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.280+0000 7f36806d4b80 1 -- 172.21.15.44:0/1482142611 shutdown_connections 2023-11-16T16:53:34.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.280+0000 7f36806d4b80 1 --2- 172.21.15.44:0/1482142611 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 0x556f25ea3d80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.280+0000 7f36806d4b80 1 --2- 172.21.15.44:0/1482142611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9ebe0 0x556f25e9efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.280+0000 7f36806d4b80 1 --2- 172.21.15.44:0/1482142611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 0x556f25ea33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.280+0000 7f36806d4b80 1 -- 172.21.15.44:0/1482142611 >> 172.21.15.44:0/1482142611 conn(0x556f25e943e0 msgr2=0x556f25e99bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:34.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.280+0000 7f36806d4b80 1 -- 172.21.15.44:0/1482142611 shutdown_connections 2023-11-16T16:53:34.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.280+0000 7f36806d4b80 1 -- 172.21.15.44:0/1482142611 wait complete. 2023-11-16T16:53:34.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.281+0000 7f36806d4b80 1 -- start start 2023-11-16T16:53:34.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.281+0000 7f36806d4b80 1 Processor -- start 2023-11-16T16:53:34.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.282+0000 7f36806d4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9ebe0 0x556f25eab140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.282+0000 7f36806d4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9f4f0 0x556f25ea8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.282+0000 7f36806d4b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 0x556f25ea8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.282+0000 7f36806d4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f25d91960 con 0x556f25e9ebe0 2023-11-16T16:53:34.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.282+0000 7f36806d4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f25d91600 con 0x556f25e9f4f0 2023-11-16T16:53:34.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.282+0000 7f36806d4b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f25c739a0 con 0x556f25ea3930 2023-11-16T16:53:34.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.282+0000 7f36806d4b80 10 client.0 _ll_register_callbacks cb 0x556f25daf800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:34.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.282+0000 7f3665d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9f4f0 0x556f25ea8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.282+0000 7f3665d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9f4f0 0x556f25ea8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56610/0 (socket says 172.21.15.44:56610) 2023-11-16T16:53:34.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.282+0000 7f3665d9b700 1 -- 172.21.15.44:0/742205120 learned_addr learned my addr 172.21.15.44:0/742205120 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:34.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.282+0000 7f3666d9d700 1 --2- 172.21.15.44:0/742205120 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 0x556f25ea8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.282+0000 7f366659c700 1 --2- 172.21.15.44:0/742205120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9ebe0 0x556f25eab140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.283+0000 7f3666d9d700 1 -- 172.21.15.44:0/742205120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9f4f0 msgr2=0x556f25ea8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:34.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.283+0000 7f3666d9d700 1 --2- 172.21.15.44:0/742205120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9f4f0 0x556f25ea8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.283+0000 7f3666d9d700 1 -- 172.21.15.44:0/742205120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9ebe0 msgr2=0x556f25eab140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:34.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.283+0000 7f3666d9d700 1 --2- 172.21.15.44:0/742205120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9ebe0 0x556f25eab140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.283+0000 7f3666d9d700 1 -- 172.21.15.44:0/742205120 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3660003bd0 con 0x556f25ea3930 2023-11-16T16:53:34.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.283+0000 7f3665d9b700 1 --2- 172.21.15.44:0/742205120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9f4f0 0x556f25ea8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:34.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.283+0000 7f366659c700 1 --2- 172.21.15.44:0/742205120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9ebe0 0x556f25eab140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:53:34.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.283+0000 7f3666d9d700 1 --2- 172.21.15.44:0/742205120 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 0x556f25ea8a40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f36600075e0 tx=0x7f3660003b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f36dd8f193de23f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:34.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.283+0000 7f364dffb700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:53:34.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.283+0000 7f364dffb700 1 -- 172.21.15.44:0/742205120 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3660019040 con 0x556f25ea3930 2023-11-16T16:53:34.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.284+0000 7f364dffb700 1 -- 172.21.15.44:0/742205120 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f3660007a50 con 0x556f25ea3930 2023-11-16T16:53:34.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.284+0000 7f36806d4b80 1 -- 172.21.15.44:0/742205120 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556f25e91ba0 con 0x556f25ea3930 2023-11-16T16:53:34.288 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.284+0000 7f364dffb700 1 -- 172.21.15.44:0/742205120 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3660007c10 con 0x556f25ea3930 2023-11-16T16:53:34.288 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.284+0000 7f36806d4b80 10 client.26410 fetch_fsmap learned FSMap version 21 2023-11-16T16:53:34.288 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.284+0000 7f36806d4b80 1 -- 172.21.15.44:0/742205120 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x556f25db6210 con 0x556f25ea3930 2023-11-16T16:53:34.288 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.284+0000 7f364dffb700 1 -- 172.21.15.44:0/742205120 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556f25e91ba0 con 0x556f25ea3930 2023-11-16T16:53:34.288 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.285+0000 7f364dffb700 1 -- 172.21.15.44:0/742205120 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3660015480 con 0x556f25ea3930 2023-11-16T16:53:34.288 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.285+0000 7f36806d4b80 10 client.26410 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:53:34.288 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.285+0000 7f36806d4b80 10 client.26410 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:34.288 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.285+0000 7f36806d4b80 20 client.26410 populate_metadata read hostname 'smithi044' 2023-11-16T16:53:34.288 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.285+0000 7f36806d4b80 1 --2- 172.21.15.44:0/742205120 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x556f25eb8810 0x556f25ebac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.285+0000 7f36806d4b80 4 client.26410 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:53:34.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.285+0000 7f36806d4b80 1 -- 172.21.15.44:0/742205120 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556f25db2120 con 0x556f25eb8810 2023-11-16T16:53:34.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.286+0000 7f366659c700 1 --2- 172.21.15.44:0/742205120 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x556f25eb8810 0x556f25ebac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.286+0000 7f366659c700 1 --2- 172.21.15.44:0/742205120 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x556f25eb8810 0x556f25ebac20 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=449b579e52be7aa4 server_cookie=1f9f693036e89b9c in_seq=0 out_seq=0 2023-11-16T16:53:34.289 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.286+0000 7f364dffb700 10 client.26410 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:34.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.287+0000 7f364dffb700 1 -- 172.21.15.44:0/742205120 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x556f25db2120 con 0x556f25eb8810 2023-11-16T16:53:34.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.287+0000 7f364dffb700 10 client.26410 handle_command_reply: tid=0 2023-11-16T16:53:34.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.287+0000 7f36806d4b80 1 client.26410 shutdown 2023-11-16T16:53:34.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.287+0000 7f36806d4b80 1 -- 172.21.15.44:0/742205120 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 msgr2=0x556f25ea8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.287+0000 7f36806d4b80 1 --2- 172.21.15.44:0/742205120 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 0x556f25ea8a40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f36600075e0 tx=0x7f3660003b50 comp rx=0 tx=0).stop 2023-11-16T16:53:34.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.287+0000 7f36806d4b80 1 -- 172.21.15.44:0/742205120 shutdown_connections 2023-11-16T16:53:34.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.287+0000 7f36806d4b80 1 --2- 172.21.15.44:0/742205120 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x556f25eb8810 0x556f25ebac20 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.287+0000 7f36806d4b80 1 --2- 172.21.15.44:0/742205120 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 0x556f25ea8a40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.287+0000 7f36806d4b80 1 --2- 172.21.15.44:0/742205120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9ebe0 0x556f25eab140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.287+0000 7f36806d4b80 1 --2- 172.21.15.44:0/742205120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9f4f0 0x556f25ea8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.287+0000 7f364dffb700 0 client.26410 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:34.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.287+0000 7f36806d4b80 1 -- 172.21.15.44:0/742205120 >> 172.21.15.44:0/742205120 conn(0x556f25e943e0 msgr2=0x556f25ea4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:34.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.288+0000 7f36806d4b80 1 -- 172.21.15.44:0/742205120 shutdown_connections 2023-11-16T16:53:34.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.288+0000 7f36806d4b80 1 -- 172.21.15.44:0/742205120 wait complete. 2023-11-16T16:53:34.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.288+0000 7f36806d4b80 20 client.26410 trim_cache size 0 max 16384 2023-11-16T16:53:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.311+0000 7f36806d4b80 1 Processor -- start 2023-11-16T16:53:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.311+0000 7f36806d4b80 1 -- start start 2023-11-16T16:53:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.311+0000 7f36806d4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9ebe0 0x556f25ea2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.311+0000 7f36806d4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 0x556f25eaac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.311+0000 7f36806d4b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 0x556f25ea68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.311+0000 7f36806d4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f25e9fc70 con 0x556f25e9f4f0 2023-11-16T16:53:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.311+0000 7f36806d4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f25e941f0 con 0x556f25e9ebe0 2023-11-16T16:53:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.311+0000 7f36806d4b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f25e9f9d0 con 0x556f25ea3930 2023-11-16T16:53:34.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.312+0000 7f366659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9ebe0 0x556f25ea2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.312+0000 7f366659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9ebe0 0x556f25ea2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56620/0 (socket says 172.21.15.44:56620) 2023-11-16T16:53:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.312+0000 7f366659c700 1 -- 172.21.15.44:0/68623709 learned_addr learned my addr 172.21.15.44:0/68623709 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.312+0000 7f3666d9d700 1 --2- 172.21.15.44:0/68623709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 0x556f25ea68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.312+0000 7f3665d9b700 1 --2- 172.21.15.44:0/68623709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 0x556f25eaac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.312+0000 7f366659c700 1 -- 172.21.15.44:0/68623709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 msgr2=0x556f25ea68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.312+0000 7f366659c700 1 --2- 172.21.15.44:0/68623709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 0x556f25ea68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.312+0000 7f366659c700 1 -- 172.21.15.44:0/68623709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 msgr2=0x556f25eaac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.312+0000 7f366659c700 1 --2- 172.21.15.44:0/68623709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 0x556f25eaac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.312+0000 7f366659c700 1 -- 172.21.15.44:0/68623709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f25ea7ec0 con 0x556f25e9ebe0 2023-11-16T16:53:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.312+0000 7f3666d9d700 1 --2- 172.21.15.44:0/68623709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 0x556f25ea68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.312+0000 7f366659c700 1 --2- 172.21.15.44:0/68623709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9ebe0 0x556f25ea2fa0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f365800c010 tx=0x7f3658086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ec05cb5130c70a2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.313+0000 7f36557fa700 1 -- 172.21.15.44:0/68623709 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36580a3610 con 0x556f25e9ebe0 2023-11-16T16:53:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.313+0000 7f36557fa700 1 -- 172.21.15.44:0/68623709 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f36580a10b0 con 0x556f25e9ebe0 2023-11-16T16:53:34.316 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.313+0000 7f36557fa700 1 -- 172.21.15.44:0/68623709 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3658022470 con 0x556f25e9ebe0 2023-11-16T16:53:34.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.313+0000 7f36806d4b80 1 -- 172.21.15.44:0/68623709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9ebe0 msgr2=0x556f25ea2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.313+0000 7f36806d4b80 1 --2- 172.21.15.44:0/68623709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9ebe0 0x556f25ea2fa0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f365800c010 tx=0x7f3658086040 comp rx=0 tx=0).stop 2023-11-16T16:53:34.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.313+0000 7f36806d4b80 1 -- 172.21.15.44:0/68623709 shutdown_connections 2023-11-16T16:53:34.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.313+0000 7f36806d4b80 1 --2- 172.21.15.44:0/68623709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25ea3930 0x556f25ea68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.313+0000 7f36806d4b80 1 --2- 172.21.15.44:0/68623709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25e9ebe0 0x556f25ea2fa0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.313+0000 7f36806d4b80 1 --2- 172.21.15.44:0/68623709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 0x556f25eaac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.313+0000 7f36806d4b80 1 -- 172.21.15.44:0/68623709 >> 172.21.15.44:0/68623709 conn(0x556f25e943e0 msgr2=0x556f25ea2890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:34.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.314+0000 7f36806d4b80 1 -- 172.21.15.44:0/68623709 shutdown_connections 2023-11-16T16:53:34.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.314+0000 7f36806d4b80 1 -- 172.21.15.44:0/68623709 wait complete. 2023-11-16T16:53:34.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.314+0000 7f36806d4b80 1 -- start start 2023-11-16T16:53:34.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.314+0000 7f36806d4b80 1 Processor -- start 2023-11-16T16:53:34.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.315+0000 7f36806d4b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25e9ebe0 0x556f25eae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.315+0000 7f36806d4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 0x556f25eaef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.315+0000 7f36806d4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25ea3930 0x556f25f06720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.315+0000 7f36806d4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f25f06c90 con 0x556f25e9f4f0 2023-11-16T16:53:34.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.315+0000 7f36806d4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f25f06e00 con 0x556f25ea3930 2023-11-16T16:53:34.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.315+0000 7f36806d4b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f25f06fc0 con 0x556f25e9ebe0 2023-11-16T16:53:34.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.315+0000 7f36806d4b80 10 client.0 _ll_register_callbacks cb 0x7f3658097590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:34.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.315+0000 7f366659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25e9ebe0 0x556f25eae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.315+0000 7f366659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25e9ebe0 0x556f25eae9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:46632/0 (socket says 172.21.15.44:46632) 2023-11-16T16:53:34.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.315+0000 7f366659c700 1 -- 172.21.15.44:0/2562184181 learned_addr learned my addr 172.21.15.44:0/2562184181 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:34.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.315+0000 7f3665d9b700 1 --2- 172.21.15.44:0/2562184181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 0x556f25eaef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.316+0000 7f366659c700 1 -- 172.21.15.44:0/2562184181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25ea3930 msgr2=0x556f25f06720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:34.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.316+0000 7f366659c700 1 --2- 172.21.15.44:0/2562184181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25ea3930 0x556f25f06720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.316+0000 7f366659c700 1 -- 172.21.15.44:0/2562184181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 msgr2=0x556f25eaef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:34.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.316+0000 7f366659c700 1 --2- 172.21.15.44:0/2562184181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 0x556f25eaef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.316+0000 7f366659c700 1 -- 172.21.15.44:0/2562184181 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f25db6210 con 0x556f25e9ebe0 2023-11-16T16:53:34.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.316+0000 7f366659c700 1 --2- 172.21.15.44:0/2562184181 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25e9ebe0 0x556f25eae9f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f36580a30e0 tx=0x7f365809ade0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e4b599b70a32c3c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:34.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.316+0000 7f364dffb700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:53:34.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.316+0000 7f364dffb700 1 -- 172.21.15.44:0/2562184181 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3658099390 con 0x556f25e9ebe0 2023-11-16T16:53:34.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.316+0000 7f364dffb700 1 -- 172.21.15.44:0/2562184181 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f36580a3610 con 0x556f25e9ebe0 2023-11-16T16:53:34.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.316+0000 7f36806d4b80 1 -- 172.21.15.44:0/2562184181 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f366000f040 con 0x556f25e9ebe0 2023-11-16T16:53:34.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.316+0000 7f364dffb700 1 -- 172.21.15.44:0/2562184181 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3658022470 con 0x556f25e9ebe0 2023-11-16T16:53:34.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.316+0000 7f36806d4b80 10 client.26416 fetch_fsmap learned FSMap version 21 2023-11-16T16:53:34.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.316+0000 7f36806d4b80 1 -- 172.21.15.44:0/2562184181 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x7f3668110240 con 0x556f25e9ebe0 2023-11-16T16:53:34.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.317+0000 7f364dffb700 1 -- 172.21.15.44:0/2562184181 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f366000f040 con 0x556f25e9ebe0 2023-11-16T16:53:34.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.317+0000 7f364dffb700 1 -- 172.21.15.44:0/2562184181 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f365809a740 con 0x556f25e9ebe0 2023-11-16T16:53:34.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.317+0000 7f36806d4b80 10 client.26416 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:53:34.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.317+0000 7f36806d4b80 10 client.26416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:34.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.317+0000 7f36806d4b80 20 client.26416 populate_metadata read hostname 'smithi044' 2023-11-16T16:53:34.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.317+0000 7f36806d4b80 1 --2- 172.21.15.44:0/2562184181 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x556f25f0e380 0x556f25f10790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.317+0000 7f36806d4b80 4 client.26416 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:53:34.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.317+0000 7f36806d4b80 1 -- 172.21.15.44:0/2562184181 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x556f25dbe6e0 con 0x556f25f0e380 2023-11-16T16:53:34.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.317+0000 7f3665d9b700 1 --2- 172.21.15.44:0/2562184181 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x556f25f0e380 0x556f25f10790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.318+0000 7f364dffb700 10 client.26416 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:34.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.318+0000 7f3665d9b700 1 --2- 172.21.15.44:0/2562184181 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x556f25f0e380 0x556f25f10790 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eca3f5aece215675 server_cookie=92b906059792f022 in_seq=0 out_seq=0 2023-11-16T16:53:34.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.479+0000 7f364dffb700 1 -- 172.21.15.44:0/2562184181 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x556f25dbe6e0 con 0x556f25f0e380 2023-11-16T16:53:34.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.479+0000 7f364dffb700 10 client.26416 handle_command_reply: tid=0 2023-11-16T16:53:34.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.479+0000 7f36806d4b80 1 client.26416 shutdown 2023-11-16T16:53:34.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.479+0000 7f36806d4b80 1 -- 172.21.15.44:0/2562184181 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25e9ebe0 msgr2=0x556f25eae9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.479+0000 7f36806d4b80 1 --2- 172.21.15.44:0/2562184181 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25e9ebe0 0x556f25eae9f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f36580a30e0 tx=0x7f365809ade0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.479+0000 7f36806d4b80 1 -- 172.21.15.44:0/2562184181 shutdown_connections 2023-11-16T16:53:34.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.479+0000 7f36806d4b80 1 --2- 172.21.15.44:0/2562184181 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x556f25f0e380 0x556f25f10790 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.479+0000 7f36806d4b80 1 --2- 172.21.15.44:0/2562184181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f25ea3930 0x556f25f06720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.479+0000 7f36806d4b80 1 --2- 172.21.15.44:0/2562184181 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x556f25e9ebe0 0x556f25eae9f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.479+0000 7f364dffb700 0 client.26416 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:34.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.479+0000 7f36806d4b80 1 --2- 172.21.15.44:0/2562184181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556f25e9f4f0 0x556f25eaef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.479+0000 7f36806d4b80 1 -- 172.21.15.44:0/2562184181 >> 172.21.15.44:0/2562184181 conn(0x556f25e943e0 msgr2=0x556f25e9e270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:34.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.479+0000 7f36806d4b80 1 -- 172.21.15.44:0/2562184181 shutdown_connections 2023-11-16T16:53:34.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.479+0000 7f36806d4b80 1 -- 172.21.15.44:0/2562184181 wait complete. 2023-11-16T16:53:34.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.480+0000 7f36806d4b80 20 client.26416 trim_cache size 0 max 16384 2023-11-16T16:53:34.483 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:53:34.483 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:53:34.483 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "b7bb16f1-c54b-4b04-97f3-2de994ff79eb", 2023-11-16T16:53:34.483 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:53:34.483 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:53:34.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.480+0000 7f366cdff700 1 -- 172.21.15.44:0/2056618222 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f36500a1340 msgr2=0x7f36500a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.480+0000 7f366cdff700 1 --2- 172.21.15.44:0/2056618222 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f36500a1340 0x7f36500a37f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3668199cc0 tx=0x7f365c00b410 comp rx=0 tx=0).stop 2023-11-16T16:53:34.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.480+0000 7f366cdff700 1 -- 172.21.15.44:0/2056618222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3668100cc0 msgr2=0x7f36681986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.480+0000 7f366cdff700 1 --2- 172.21.15.44:0/2056618222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3668100cc0 0x7f36681986b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3658009990 tx=0x7f3658005740 comp rx=0 tx=0).stop 2023-11-16T16:53:34.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.480+0000 7f366cdff700 1 -- 172.21.15.44:0/2056618222 shutdown_connections 2023-11-16T16:53:34.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.480+0000 7f366cdff700 1 --2- 172.21.15.44:0/2056618222 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f36500a1340 0x7f36500a37f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.480+0000 7f366cdff700 1 --2- 172.21.15.44:0/2056618222 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3668105620 0x7f366819bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.480+0000 7f366cdff700 1 --2- 172.21.15.44:0/2056618222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3668100cc0 0x7f36681986b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.480+0000 7f366cdff700 1 --2- 172.21.15.44:0/2056618222 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3668101660 0x7f3668198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.480+0000 7f366cdff700 1 -- 172.21.15.44:0/2056618222 >> 172.21.15.44:0/2056618222 conn(0x7f3668077560 msgr2=0x7f36680feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:34.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.481+0000 7f366cdff700 1 -- 172.21.15.44:0/2056618222 shutdown_connections 2023-11-16T16:53:34.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.481+0000 7f366cdff700 1 -- 172.21.15.44:0/2056618222 wait complete. 2023-11-16T16:53:34.502 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:53:34.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.741+0000 7f39ff5ff700 1 -- 172.21.15.44:0/2477643519 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f80ff180 msgr2=0x7f39f80ff550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.741+0000 7f39ff5ff700 1 --2- 172.21.15.44:0/2477643519 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f80ff180 0x7f39f80ff550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f39f4009990 tx=0x7f39f4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.741+0000 7f39ff5ff700 1 -- 172.21.15.44:0/2477643519 shutdown_connections 2023-11-16T16:53:34.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.741+0000 7f39ff5ff700 1 --2- 172.21.15.44:0/2477643519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39f8104680 0x7f39f8104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.741+0000 7f39ff5ff700 1 --2- 172.21.15.44:0/2477643519 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f80ff180 0x7f39f80ff550 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.741+0000 7f39ff5ff700 1 --2- 172.21.15.44:0/2477643519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39f80ffa90 0x7f39f8104140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.741+0000 7f39ff5ff700 1 -- 172.21.15.44:0/2477643519 >> 172.21.15.44:0/2477643519 conn(0x7f39f80770b0 msgr2=0x7f39f80774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:34.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.741+0000 7f39ff5ff700 1 -- 172.21.15.44:0/2477643519 shutdown_connections 2023-11-16T16:53:34.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.741+0000 7f39ff5ff700 1 -- 172.21.15.44:0/2477643519 wait complete. 2023-11-16T16:53:34.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.742+0000 7f39ff5ff700 1 Processor -- start 2023-11-16T16:53:34.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.742+0000 7f39ff5ff700 1 -- start start 2023-11-16T16:53:34.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.743+0000 7f39ff5ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39f80ff180 0x7f39f81a27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.743+0000 7f39ff5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39f80ffa90 0x7f39f81a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.743+0000 7f39ff5ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f8104680 0x7f39f819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.743+0000 7f39ff5ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f819cec0 con 0x7f39f80ff180 2023-11-16T16:53:34.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.743+0000 7f39ff5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f819d030 con 0x7f39f80ffa90 2023-11-16T16:53:34.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.743+0000 7f39ff5ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f819d1f0 con 0x7f39f8104680 2023-11-16T16:53:34.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.743+0000 7f39fcb9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39f80ffa90 0x7f39f81a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.743+0000 7f39fcb9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39f80ffa90 0x7f39f81a2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56640/0 (socket says 172.21.15.44:56640) 2023-11-16T16:53:34.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.743+0000 7f39fcb9a700 1 -- 172.21.15.44:0/2414525608 learned_addr learned my addr 172.21.15.44:0/2414525608 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:34.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.743+0000 7f39fdb9c700 1 --2- 172.21.15.44:0/2414525608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f8104680 0x7f39f819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.743+0000 7f39fd39b700 1 --2- 172.21.15.44:0/2414525608 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39f80ff180 0x7f39f81a27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.744+0000 7f39fdb9c700 1 -- 172.21.15.44:0/2414525608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39f80ffa90 msgr2=0x7f39f81a2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.744+0000 7f39fdb9c700 1 --2- 172.21.15.44:0/2414525608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39f80ffa90 0x7f39f81a2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.744+0000 7f39fdb9c700 1 -- 172.21.15.44:0/2414525608 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39f80ff180 msgr2=0x7f39f81a27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.744+0000 7f39fdb9c700 1 --2- 172.21.15.44:0/2414525608 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39f80ff180 0x7f39f81a27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.744+0000 7f39fdb9c700 1 -- 172.21.15.44:0/2414525608 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f40095f0 con 0x7f39f8104680 2023-11-16T16:53:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.744+0000 7f39fcb9a700 1 --2- 172.21.15.44:0/2414525608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39f80ffa90 0x7f39f81a2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.744+0000 7f39fdb9c700 1 --2- 172.21.15.44:0/2414525608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f8104680 0x7f39f819c850 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f39f000efc0 tx=0x7f39f000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.744+0000 7f39ee7fc700 1 -- 172.21.15.44:0/2414525608 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f0018070 con 0x7f39f8104680 2023-11-16T16:53:34.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.744+0000 7f39ff5ff700 1 -- 172.21.15.44:0/2414525608 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f8116940 con 0x7f39f8104680 2023-11-16T16:53:34.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.744+0000 7f39ee7fc700 1 -- 172.21.15.44:0/2414525608 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39f000f040 con 0x7f39f8104680 2023-11-16T16:53:34.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.744+0000 7f39ee7fc700 1 -- 172.21.15.44:0/2414525608 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f000cd40 con 0x7f39f8104680 2023-11-16T16:53:34.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.745+0000 7f39ff5ff700 1 -- 172.21.15.44:0/2414525608 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f81167b0 con 0x7f39f8104680 2023-11-16T16:53:34.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.748+0000 7f39ee7fc700 1 -- 172.21.15.44:0/2414525608 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f39f0003a00 con 0x7f39f8104680 2023-11-16T16:53:34.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.749+0000 7f3a12f47b80 1 Processor -- start 2023-11-16T16:53:34.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.750+0000 7f3a12f47b80 1 -- start start 2023-11-16T16:53:34.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.750+0000 7f3a12f47b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597620d2b60 0x5597620d2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.750+0000 7f3a12f47b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 0x5597620d7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.752 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.750+0000 7f3a12f47b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 0x5597620d7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.750+0000 7f3a12f47b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559761fd8060 con 0x5597620d2b60 2023-11-16T16:53:34.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.750+0000 7f3a12f47b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976202a8a0 con 0x5597620d3470 2023-11-16T16:53:34.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.750+0000 7f3a12f47b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559762023d30 con 0x5597620d78b0 2023-11-16T16:53:34.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.750+0000 7f39fd39b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 0x5597620d7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.750+0000 7f39fcb9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597620d2b60 0x5597620d2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.750+0000 7f39fd39b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 0x5597620d7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56642/0 (socket says 172.21.15.44:56642) 2023-11-16T16:53:34.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.750+0000 7f39fd39b700 1 -- 172.21.15.44:0/1490394270 learned_addr learned my addr 172.21.15.44:0/1490394270 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:34.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.751+0000 7f39fcb9a700 1 --2- 172.21.15.44:0/1490394270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 0x5597620d7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.751+0000 7f39ee7fc700 1 --2- 172.21.15.44:0/2414525608 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f39e40a12c0 0x7f39e40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.751+0000 7f39fd39b700 1 -- 172.21.15.44:0/1490394270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 msgr2=0x5597620d7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:34.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.751+0000 7f39fd39b700 1 --2- 172.21.15.44:0/1490394270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 0x5597620d7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.753 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.751+0000 7f39fd39b700 1 -- 172.21.15.44:0/1490394270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597620d2b60 msgr2=0x5597620d2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:34.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.751+0000 7f39fd39b700 1 --2- 172.21.15.44:0/1490394270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597620d2b60 0x5597620d2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.751+0000 7f39fd39b700 1 -- 172.21.15.44:0/1490394270 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559761ea5630 con 0x5597620d3470 2023-11-16T16:53:34.754 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.751+0000 7f39fcb9a700 1 --2- 172.21.15.44:0/1490394270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 0x5597620d7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:53:34.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.751+0000 7f39ee7fc700 1 -- 172.21.15.44:0/2414525608 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f39f00ccbf0 con 0x7f39f8104680 2023-11-16T16:53:34.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.751+0000 7f39fdb9c700 1 --2- 172.21.15.44:0/2414525608 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f39e40a12c0 0x7f39e40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.751+0000 7f39fd39b700 1 --2- 172.21.15.44:0/1490394270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 0x5597620d7370 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f39f4005ea0 tx=0x7f39f4005ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bf703e27c66727f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:34.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.752+0000 7f39e37fe700 1 -- 172.21.15.44:0/1490394270 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f401d070 con 0x5597620d3470 2023-11-16T16:53:34.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.752+0000 7f39e37fe700 1 -- 172.21.15.44:0/1490394270 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39f401e040 con 0x5597620d3470 2023-11-16T16:53:34.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.752+0000 7f39e37fe700 1 -- 172.21.15.44:0/1490394270 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f4004990 con 0x5597620d3470 2023-11-16T16:53:34.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.752+0000 7f3a12f47b80 1 -- 172.21.15.44:0/1490394270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 msgr2=0x5597620d7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.752+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/1490394270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 0x5597620d7370 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f39f4005ea0 tx=0x7f39f4005ed0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.752+0000 7f39fdb9c700 1 --2- 172.21.15.44:0/2414525608 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f39e40a12c0 0x7f39e40a3770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f39f000c6f0 tx=0x7f39f0004dc0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:34.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.753+0000 7f3a12f47b80 1 -- 172.21.15.44:0/1490394270 shutdown_connections 2023-11-16T16:53:34.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.753+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/1490394270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 0x5597620d7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.753+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/1490394270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597620d2b60 0x5597620d2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.753+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/1490394270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 0x5597620d7370 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.753+0000 7f3a12f47b80 1 -- 172.21.15.44:0/1490394270 >> 172.21.15.44:0/1490394270 conn(0x5597620c8360 msgr2=0x5597620cdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:34.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.753+0000 7f3a12f47b80 1 -- 172.21.15.44:0/1490394270 shutdown_connections 2023-11-16T16:53:34.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.753+0000 7f3a12f47b80 1 -- 172.21.15.44:0/1490394270 wait complete. 2023-11-16T16:53:34.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.753+0000 7f3a12f47b80 1 -- start start 2023-11-16T16:53:34.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.754+0000 7f3a12f47b80 1 Processor -- start 2023-11-16T16:53:34.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.754+0000 7f3a12f47b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d2b60 0x5597620defe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.754+0000 7f3a12f47b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 0x5597620dc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.754+0000 7f3a12f47b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597620d78b0 0x5597620dc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.754+0000 7f3a12f47b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559761fc58e0 con 0x5597620d78b0 2023-11-16T16:53:34.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.754+0000 7f3a12f47b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559761fc5580 con 0x5597620d3470 2023-11-16T16:53:34.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.754+0000 7f3a12f47b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559761ea7920 con 0x5597620d2b60 2023-11-16T16:53:34.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.754+0000 7f3a12f47b80 10 client.0 _ll_register_callbacks cb 0x559761fe3780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:34.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.755+0000 7f39fd39b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 0x5597620dc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.755+0000 7f39fd39b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 0x5597620dc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56650/0 (socket says 172.21.15.44:56650) 2023-11-16T16:53:34.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.755+0000 7f39fd39b700 1 -- 172.21.15.44:0/674665013 learned_addr learned my addr 172.21.15.44:0/674665013 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:34.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.755+0000 7f39fcb9a700 1 --2- 172.21.15.44:0/674665013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d2b60 0x5597620defe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.755+0000 7f39fcb9a700 1 --2- 172.21.15.44:0/674665013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597620d78b0 0x5597620dc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.755+0000 7f39fd39b700 1 -- 172.21.15.44:0/674665013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d2b60 msgr2=0x5597620defe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:34.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.755+0000 7f39fd39b700 1 --2- 172.21.15.44:0/674665013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d2b60 0x5597620defe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.760 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.755+0000 7f39fd39b700 1 -- 172.21.15.44:0/674665013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597620d78b0 msgr2=0x5597620dc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:34.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.755+0000 7f39fd39b700 1 --2- 172.21.15.44:0/674665013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597620d78b0 0x5597620dc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.755+0000 7f39fd39b700 1 -- 172.21.15.44:0/674665013 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f8114090 con 0x5597620d3470 2023-11-16T16:53:34.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.755+0000 7f39fd39b700 1 --2- 172.21.15.44:0/674665013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 0x5597620dc260 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f39f4022b10 tx=0x7f39f4022b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3d74aac67754526 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:34.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.755+0000 7f39e0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:53:34.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.756+0000 7f39e0ff9700 1 -- 172.21.15.44:0/674665013 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f401d070 con 0x5597620d3470 2023-11-16T16:53:34.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.756+0000 7f39e0ff9700 1 -- 172.21.15.44:0/674665013 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f39f400f460 con 0x5597620d3470 2023-11-16T16:53:34.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.756+0000 7f3a12f47b80 1 -- 172.21.15.44:0/674665013 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5597620c5b20 con 0x5597620d3470 2023-11-16T16:53:34.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.756+0000 7f39e0ff9700 1 -- 172.21.15.44:0/674665013 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39f40056b0 con 0x5597620d3470 2023-11-16T16:53:34.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.756+0000 7f3a12f47b80 10 client.26432 fetch_fsmap learned FSMap version 21 2023-11-16T16:53:34.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.756+0000 7f3a12f47b80 1 -- 172.21.15.44:0/674665013 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x559761fea190 con 0x5597620d3470 2023-11-16T16:53:34.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.756+0000 7f39e0ff9700 1 -- 172.21.15.44:0/674665013 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5597620c5b20 con 0x5597620d3470 2023-11-16T16:53:34.761 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.756+0000 7f39e0ff9700 1 -- 172.21.15.44:0/674665013 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f39f402c400 con 0x5597620d3470 2023-11-16T16:53:34.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.757+0000 7f3a12f47b80 10 client.26432 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:53:34.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.757+0000 7f3a12f47b80 10 client.26432 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:34.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.757+0000 7f3a12f47b80 20 client.26432 populate_metadata read hostname 'smithi044' 2023-11-16T16:53:34.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.757+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/674665013 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5597620ec390 0x5597620ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.757+0000 7f3a12f47b80 4 client.26432 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:53:34.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.757+0000 7f3a12f47b80 1 -- 172.21.15.44:0/674665013 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559761fe60a0 con 0x5597620ec390 2023-11-16T16:53:34.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.757+0000 7f39fcb9a700 1 --2- 172.21.15.44:0/674665013 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5597620ec390 0x5597620ee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.757+0000 7f39fcb9a700 1 --2- 172.21.15.44:0/674665013 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5597620ec390 0x5597620ee7a0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1e8257eb84e5c56 server_cookie=5f29c7f263664bcf in_seq=0 out_seq=0 2023-11-16T16:53:34.762 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.757+0000 7f39e0ff9700 10 client.26432 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:34.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.842+0000 7f39e0ff9700 1 -- 172.21.15.44:0/674665013 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559761fe60a0 con 0x5597620ec390 2023-11-16T16:53:34.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.842+0000 7f39e0ff9700 10 client.26432 handle_command_reply: tid=0 2023-11-16T16:53:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.843+0000 7f3a12f47b80 1 client.26432 shutdown 2023-11-16T16:53:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.843+0000 7f3a12f47b80 1 -- 172.21.15.44:0/674665013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 msgr2=0x5597620dc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.843+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/674665013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 0x5597620dc260 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f39f4022b10 tx=0x7f39f4022b40 comp rx=0 tx=0).stop 2023-11-16T16:53:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.843+0000 7f3a12f47b80 1 -- 172.21.15.44:0/674665013 shutdown_connections 2023-11-16T16:53:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.843+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/674665013 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5597620ec390 0x5597620ee7a0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.843+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/674665013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597620d78b0 0x5597620dc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.843+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/674665013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d2b60 0x5597620defe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.843+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/674665013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 0x5597620dc260 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.843+0000 7f3a12f47b80 1 -- 172.21.15.44:0/674665013 >> 172.21.15.44:0/674665013 conn(0x5597620c8360 msgr2=0x5597620d8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.843+0000 7f39e0ff9700 0 client.26432 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.843+0000 7f3a12f47b80 1 -- 172.21.15.44:0/674665013 shutdown_connections 2023-11-16T16:53:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.843+0000 7f3a12f47b80 1 -- 172.21.15.44:0/674665013 wait complete. 2023-11-16T16:53:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.844+0000 7f3a12f47b80 20 client.26432 trim_cache size 0 max 16384 2023-11-16T16:53:34.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.866+0000 7f3a12f47b80 1 Processor -- start 2023-11-16T16:53:34.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.867+0000 7f3a12f47b80 1 -- start start 2023-11-16T16:53:34.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.867+0000 7f3a12f47b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d2b60 0x5597620d5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.867+0000 7f3a12f47b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597620d3470 0x5597620cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.867+0000 7f3a12f47b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 0x5597620cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.867+0000 7f3a12f47b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597620cf230 con 0x5597620d3470 2023-11-16T16:53:34.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.867+0000 7f3a12f47b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597620c7ee0 con 0x5597620d2b60 2023-11-16T16:53:34.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.867+0000 7f3a12f47b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597620d3840 con 0x5597620d78b0 2023-11-16T16:53:34.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.867+0000 7f39fcb9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d2b60 0x5597620d5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.867+0000 7f39fcb9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d2b60 0x5597620d5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56490/0 (socket says 172.21.15.44:56490) 2023-11-16T16:53:34.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.867+0000 7f39fcb9a700 1 -- 172.21.15.44:0/22270781 learned_addr learned my addr 172.21.15.44:0/22270781 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:34.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.867+0000 7f39fd39b700 1 --2- 172.21.15.44:0/22270781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597620d3470 0x5597620cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.868+0000 7f39fcb9a700 1 --2- 172.21.15.44:0/22270781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 0x5597620cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.868+0000 7f39fcb9a700 1 -- 172.21.15.44:0/22270781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 msgr2=0x5597620cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:34.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.868+0000 7f39fcb9a700 1 --2- 172.21.15.44:0/22270781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 0x5597620cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.868+0000 7f39fcb9a700 1 -- 172.21.15.44:0/22270781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597620d3470 msgr2=0x5597620cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:34.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.868+0000 7f39fcb9a700 1 --2- 172.21.15.44:0/22270781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597620d3470 0x5597620cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.868+0000 7f39fcb9a700 1 -- 172.21.15.44:0/22270781 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597620c9b00 con 0x5597620d2b60 2023-11-16T16:53:34.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.868+0000 7f39fcb9a700 1 --2- 172.21.15.44:0/22270781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d2b60 0x5597620d5970 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f39f81044e0 tx=0x7f39e80095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d23a39001002e221 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:34.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.868+0000 7f39e3fff700 1 -- 172.21.15.44:0/22270781 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e8004230 con 0x5597620d2b60 2023-11-16T16:53:34.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.869+0000 7f39e3fff700 1 -- 172.21.15.44:0/22270781 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39e802c020 con 0x5597620d2b60 2023-11-16T16:53:34.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.869+0000 7f39e3fff700 1 -- 172.21.15.44:0/22270781 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e80052f0 con 0x5597620d2b60 2023-11-16T16:53:34.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.869+0000 7f3a12f47b80 1 -- 172.21.15.44:0/22270781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d2b60 msgr2=0x5597620d5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.869+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/22270781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d2b60 0x5597620d5970 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f39f81044e0 tx=0x7f39e80095e0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.870+0000 7f3a12f47b80 1 -- 172.21.15.44:0/22270781 shutdown_connections 2023-11-16T16:53:34.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.870+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/22270781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 0x5597620cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.872 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.870+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/22270781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d2b60 0x5597620d5970 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.870+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/22270781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597620d3470 0x5597620cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.870+0000 7f3a12f47b80 1 -- 172.21.15.44:0/22270781 >> 172.21.15.44:0/22270781 conn(0x5597620c8360 msgr2=0x5597620ee3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:34.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.870+0000 7f3a12f47b80 1 -- 172.21.15.44:0/22270781 shutdown_connections 2023-11-16T16:53:34.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.870+0000 7f3a12f47b80 1 -- 172.21.15.44:0/22270781 wait complete. 2023-11-16T16:53:34.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.870+0000 7f3a12f47b80 1 -- start start 2023-11-16T16:53:34.873 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.870+0000 7f3a12f47b80 1 Processor -- start 2023-11-16T16:53:34.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.871+0000 7f3a12f47b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 0x5597621343d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.875 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.871+0000 7f3a12f47b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 0x559762134910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.871+0000 7f3a12f47b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55976213d250 0x55976213f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.871+0000 7f3a12f47b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976213fb70 con 0x55976213d250 2023-11-16T16:53:34.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.871+0000 7f3a12f47b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976213fcb0 con 0x5597620d3470 2023-11-16T16:53:34.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.871+0000 7f3a12f47b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976213fdf0 con 0x5597620d78b0 2023-11-16T16:53:34.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.871+0000 7f3a12f47b80 10 client.0 _ll_register_callbacks cb 0x7f39e8013460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:34.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.871+0000 7f39fd39b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 0x559762134910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.871+0000 7f39fcb9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 0x5597621343d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.871+0000 7f39fd39b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 0x559762134910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:34098/0 (socket says 172.21.15.44:34098) 2023-11-16T16:53:34.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.871+0000 7f39fd39b700 1 -- 172.21.15.44:0/3978411303 learned_addr learned my addr 172.21.15.44:0/3978411303 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:34.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.871+0000 7f39fcb9a700 1 --2- 172.21.15.44:0/3978411303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55976213d250 0x55976213f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.872+0000 7f39fd39b700 1 -- 172.21.15.44:0/3978411303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 msgr2=0x5597621343d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:34.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.872+0000 7f39fd39b700 1 --2- 172.21.15.44:0/3978411303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 0x5597621343d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.872+0000 7f39fd39b700 1 -- 172.21.15.44:0/3978411303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55976213d250 msgr2=0x55976213f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:34.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.872+0000 7f39fd39b700 1 --2- 172.21.15.44:0/3978411303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55976213d250 0x55976213f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.872+0000 7f39fd39b700 1 -- 172.21.15.44:0/3978411303 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559761e51de0 con 0x5597620d78b0 2023-11-16T16:53:34.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.872+0000 7f39fd39b700 1 --2- 172.21.15.44:0/3978411303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 0x559762134910 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f39f4001eb0 tx=0x7f39f402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f72c214307ca747 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:34.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.872+0000 7f39e0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:53:34.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.872+0000 7f39e0ff9700 1 -- 172.21.15.44:0/3978411303 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f401d050 con 0x5597620d78b0 2023-11-16T16:53:34.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.872+0000 7f3a12f47b80 1 -- 172.21.15.44:0/3978411303 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39f401e040 con 0x5597620d78b0 2023-11-16T16:53:34.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.872+0000 7f39e0ff9700 1 -- 172.21.15.44:0/3978411303 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f39f400be60 con 0x5597620d78b0 2023-11-16T16:53:34.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.873+0000 7f39e0ff9700 1 -- 172.21.15.44:0/3978411303 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39f4022bb0 con 0x5597620d78b0 2023-11-16T16:53:34.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.873+0000 7f3a12f47b80 10 client.26434 fetch_fsmap learned FSMap version 21 2023-11-16T16:53:34.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.873+0000 7f3a12f47b80 1 -- 172.21.15.44:0/3978411303 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x559761fe9ee0 con 0x5597620d78b0 2023-11-16T16:53:34.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.873+0000 7f39e0ff9700 1 -- 172.21.15.44:0/3978411303 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39f401e040 con 0x5597620d78b0 2023-11-16T16:53:34.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.873+0000 7f39e0ff9700 1 -- 172.21.15.44:0/3978411303 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f39f40177a0 con 0x5597620d78b0 2023-11-16T16:53:34.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.873+0000 7f3a12f47b80 10 client.26434 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:53:34.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.873+0000 7f3a12f47b80 10 client.26434 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:34.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.873+0000 7f3a12f47b80 20 client.26434 populate_metadata read hostname 'smithi044' 2023-11-16T16:53:34.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.873+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/3978411303 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559762141020 0x559762143430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:34.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.873+0000 7f3a12f47b80 4 client.26434 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:53:34.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.873+0000 7f3a12f47b80 1 -- 172.21.15.44:0/3978411303 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559761ff2660 con 0x559762141020 2023-11-16T16:53:34.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.873+0000 7f39fcb9a700 1 --2- 172.21.15.44:0/3978411303 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559762141020 0x559762143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:34.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.874+0000 7f39e0ff9700 10 client.26434 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:34.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.874+0000 7f39fcb9a700 1 --2- 172.21.15.44:0/3978411303 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559762141020 0x559762143430 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d316071b4a048976 server_cookie=2472287473b3f7dc in_seq=0 out_seq=0 2023-11-16T16:53:34.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.877+0000 7f39e0ff9700 1 -- 172.21.15.44:0/3978411303 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x559761ff2660 con 0x559762141020 2023-11-16T16:53:34.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.877+0000 7f39e0ff9700 10 client.26434 handle_command_reply: tid=0 2023-11-16T16:53:34.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.877+0000 7f3a12f47b80 1 client.26434 shutdown 2023-11-16T16:53:34.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.877+0000 7f3a12f47b80 1 -- 172.21.15.44:0/3978411303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 msgr2=0x559762134910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.877+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/3978411303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 0x559762134910 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f39f4001eb0 tx=0x7f39f402c6c0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.877+0000 7f3a12f47b80 1 -- 172.21.15.44:0/3978411303 shutdown_connections 2023-11-16T16:53:34.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.877+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/3978411303 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559762141020 0x559762143430 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.877+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/3978411303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55976213d250 0x55976213f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.877+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/3978411303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5597620d3470 0x5597621343d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.877+0000 7f3a12f47b80 1 --2- 172.21.15.44:0/3978411303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5597620d78b0 0x559762134910 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.877+0000 7f39e0ff9700 0 client.26434 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:34.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.877+0000 7f3a12f47b80 1 -- 172.21.15.44:0/3978411303 >> 172.21.15.44:0/3978411303 conn(0x5597620c8360 msgr2=0x5597620d5970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:34.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.877+0000 7f3a12f47b80 1 -- 172.21.15.44:0/3978411303 shutdown_connections 2023-11-16T16:53:34.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.877+0000 7f3a12f47b80 1 -- 172.21.15.44:0/3978411303 wait complete. 2023-11-16T16:53:34.881 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.877+0000 7f3a12f47b80 20 client.26434 trim_cache size 0 max 16384 2023-11-16T16:53:34.881 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:53:34.881 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "scrub active (7856 inodes in the stack)", 2023-11-16T16:53:34.881 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2023-11-16T16:53:34.881 INFO:teuthology.orchestra.run.smithi044.stdout: "b7bb16f1-c54b-4b04-97f3-2de994ff79eb": { 2023-11-16T16:53:34.881 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2023-11-16T16:53:34.881 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "b7bb16f1-c54b-4b04-97f3-2de994ff79eb", 2023-11-16T16:53:34.881 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2023-11-16T16:53:34.881 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:53:34.882 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:53:34.882 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:53:34.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.878+0000 7f39ff5ff700 1 -- 172.21.15.44:0/2414525608 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f39e40a12c0 msgr2=0x7f39e40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.878+0000 7f39ff5ff700 1 --2- 172.21.15.44:0/2414525608 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f39e40a12c0 0x7f39e40a3770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f39f000c6f0 tx=0x7f39f0004dc0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.878+0000 7f39ff5ff700 1 -- 172.21.15.44:0/2414525608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f8104680 msgr2=0x7f39f819c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:34.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.878+0000 7f39ff5ff700 1 --2- 172.21.15.44:0/2414525608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f8104680 0x7f39f819c850 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f39f000efc0 tx=0x7f39f000c3a0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.878+0000 7f39ff5ff700 1 -- 172.21.15.44:0/2414525608 shutdown_connections 2023-11-16T16:53:34.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.878+0000 7f39ff5ff700 1 --2- 172.21.15.44:0/2414525608 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f39e40a12c0 0x7f39e40a3770 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.878+0000 7f39ff5ff700 1 --2- 172.21.15.44:0/2414525608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f8104680 0x7f39f819c850 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.878+0000 7f39ff5ff700 1 --2- 172.21.15.44:0/2414525608 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39f80ff180 0x7f39f81a27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.878+0000 7f39ff5ff700 1 --2- 172.21.15.44:0/2414525608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39f80ffa90 0x7f39f81a2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:34.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.878+0000 7f39ff5ff700 1 -- 172.21.15.44:0/2414525608 >> 172.21.15.44:0/2414525608 conn(0x7f39f80770b0 msgr2=0x7f39f81021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:34.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.878+0000 7f39ff5ff700 1 -- 172.21.15.44:0/2414525608 shutdown_connections 2023-11-16T16:53:34.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:53:34.879+0000 7f39ff5ff700 1 -- 172.21.15.44:0/2414525608 wait complete. 2023-11-16T16:53:34.903 INFO:tasks.cephfs.filesystem:scrub status for tag:b7bb16f1-c54b-4b04-97f3-2de994ff79eb - {'path': '/', 'tag': 'b7bb16f1-c54b-4b04-97f3-2de994ff79eb', 'options': 'recursive,force'} 2023-11-16T16:53:35.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:35 smithi044 ceph-mon[128750]: pgmap v112: 129 pgs: 129 active+clean; 46 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.8 MiB/s wr, 4.04k op/s 2023-11-16T16:53:35.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:35 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:53:35.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:35 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:53:35.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:35 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:53:35.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:35 smithi176 ceph-mon[131630]: pgmap v112: 129 pgs: 129 active+clean; 46 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.8 MiB/s wr, 4.04k op/s 2023-11-16T16:53:35.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:35 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:53:35.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:35 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:53:35.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:35 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:53:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:35 smithi077 ceph-mon[131204]: pgmap v112: 129 pgs: 129 active+clean; 46 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.8 MiB/s wr, 4.04k op/s 2023-11-16T16:53:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:35 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:53:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:35 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:53:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:35 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:53:37.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:37 smithi044 ceph-mon[128750]: pgmap v113: 129 pgs: 129 active+clean; 46 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.8 MiB/s wr, 4.04k op/s 2023-11-16T16:53:37.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:37 smithi176 ceph-mon[131630]: pgmap v113: 129 pgs: 129 active+clean; 46 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.8 MiB/s wr, 4.04k op/s 2023-11-16T16:53:37.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:37 smithi077 ceph-mon[131204]: pgmap v113: 129 pgs: 129 active+clean; 46 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.8 MiB/s wr, 4.04k op/s 2023-11-16T16:53:39.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:39 smithi044 ceph-mon[128750]: pgmap v114: 129 pgs: 129 active+clean; 20 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 4.17k op/s 2023-11-16T16:53:39.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:39 smithi176 ceph-mon[131630]: pgmap v114: 129 pgs: 129 active+clean; 20 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 4.17k op/s 2023-11-16T16:53:39.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:39 smithi077 ceph-mon[131204]: pgmap v114: 129 pgs: 129 active+clean; 20 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 4.17k op/s 2023-11-16T16:53:41.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:41 smithi044 ceph-mon[128750]: pgmap v115: 129 pgs: 129 active+clean; 20 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 3.15k op/s 2023-11-16T16:53:41.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:41 smithi176 ceph-mon[131630]: pgmap v115: 129 pgs: 129 active+clean; 20 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 3.15k op/s 2023-11-16T16:53:41.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:41 smithi077 ceph-mon[131204]: pgmap v115: 129 pgs: 129 active+clean; 20 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 3.15k op/s 2023-11-16T16:53:43.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:43 smithi044 ceph-mon[128750]: pgmap v116: 129 pgs: 129 active+clean; 25 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 3.22k op/s 2023-11-16T16:53:43.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:43 smithi176 ceph-mon[131630]: pgmap v116: 129 pgs: 129 active+clean; 25 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 3.22k op/s 2023-11-16T16:53:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:43 smithi077 ceph-mon[131204]: pgmap v116: 129 pgs: 129 active+clean; 25 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 3.22k op/s 2023-11-16T16:53:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.258+0000 7fc2ba1ff700 1 -- 172.21.15.44:0/1323772833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2b41015f0 msgr2=0x7fc2b4104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.258+0000 7fc2ba1ff700 1 --2- 172.21.15.44:0/1323772833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2b41015f0 0x7fc2b4104e80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80099c0 tx=0x7fc2a8009cd0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.258+0000 7fc2ba1ff700 1 -- 172.21.15.44:0/1323772833 shutdown_connections 2023-11-16T16:53:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.258+0000 7fc2ba1ff700 1 --2- 172.21.15.44:0/1323772833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2b4105560 0x7fc2b410d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.258+0000 7fc2ba1ff700 1 --2- 172.21.15.44:0/1323772833 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2b4100c50 0x7fc2b4101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.258+0000 7fc2ba1ff700 1 --2- 172.21.15.44:0/1323772833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2b41015f0 0x7fc2b4104e80 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.258+0000 7fc2ba1ff700 1 -- 172.21.15.44:0/1323772833 >> 172.21.15.44:0/1323772833 conn(0x7fc2b4077550 msgr2=0x7fc2b4077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.258+0000 7fc2ba1ff700 1 -- 172.21.15.44:0/1323772833 shutdown_connections 2023-11-16T16:53:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.258+0000 7fc2ba1ff700 1 -- 172.21.15.44:0/1323772833 wait complete. 2023-11-16T16:53:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.260+0000 7fc2ba1ff700 1 Processor -- start 2023-11-16T16:53:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.260+0000 7fc2ba1ff700 1 -- start start 2023-11-16T16:53:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2ba1ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2b4100c50 0x7fc2b41985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2ba1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2b41015f0 0x7fc2b4198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2ba1ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2b4105560 0x7fc2b419cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2ba1ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b419d5d0 con 0x7fc2b4100c50 2023-11-16T16:53:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2ba1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b419d740 con 0x7fc2b41015f0 2023-11-16T16:53:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2ba1ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b419d900 con 0x7fc2b4105560 2023-11-16T16:53:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2b37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2b4100c50 0x7fc2b41985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2b2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2b41015f0 0x7fc2b4198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2b2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2b41015f0 0x7fc2b4198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:40778/0 (socket says 172.21.15.44:40778) 2023-11-16T16:53:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2b2ffd700 1 -- 172.21.15.44:0/3131814961 learned_addr learned my addr 172.21.15.44:0/3131814961 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2b37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2b4100c50 0x7fc2b41985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46016/0 (socket says 172.21.15.44:46016) 2023-11-16T16:53:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2b3fff700 1 --2- 172.21.15.44:0/3131814961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2b4105560 0x7fc2b419cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2b2ffd700 1 -- 172.21.15.44:0/3131814961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2b4105560 msgr2=0x7fc2b419cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2b2ffd700 1 --2- 172.21.15.44:0/3131814961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2b4105560 0x7fc2b419cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2b2ffd700 1 -- 172.21.15.44:0/3131814961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2b4100c50 msgr2=0x7fc2b41985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2b2ffd700 1 --2- 172.21.15.44:0/3131814961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2b4100c50 0x7fc2b41985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2b2ffd700 1 -- 172.21.15.44:0/3131814961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a80095f0 con 0x7fc2b41015f0 2023-11-16T16:53:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2b3fff700 1 --2- 172.21.15.44:0/3131814961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2b4105560 0x7fc2b419cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:53:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.261+0000 7fc2b2ffd700 1 --2- 172.21.15.44:0/3131814961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2b41015f0 0x7fc2b4198b30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80098e0 tx=0x7fc2a800f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.262+0000 7fc2b0ff9700 1 -- 172.21.15.44:0/3131814961 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a801d070 con 0x7fc2b41015f0 2023-11-16T16:53:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.262+0000 7fc2b0ff9700 1 -- 172.21.15.44:0/3131814961 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc2a8005510 con 0x7fc2b41015f0 2023-11-16T16:53:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.262+0000 7fc2ba1ff700 1 -- 172.21.15.44:0/3131814961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b4110370 con 0x7fc2b41015f0 2023-11-16T16:53:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.262+0000 7fc2b0ff9700 1 -- 172.21.15.44:0/3131814961 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8004a00 con 0x7fc2b41015f0 2023-11-16T16:53:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.262+0000 7fc2ba1ff700 1 -- 172.21.15.44:0/3131814961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b41101e0 con 0x7fc2b41015f0 2023-11-16T16:53:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.265+0000 7fc2b0ff9700 1 -- 172.21.15.44:0/3131814961 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fc2a8004bc0 con 0x7fc2b41015f0 2023-11-16T16:53:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.266+0000 7fc2cda32b80 1 Processor -- start 2023-11-16T16:53:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.266+0000 7fc2b0ff9700 1 --2- 172.21.15.44:0/3131814961 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc29c0a12c0 0x7fc29c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.267+0000 7fc2cda32b80 1 -- start start 2023-11-16T16:53:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.267+0000 7fc2b0ff9700 1 -- 172.21.15.44:0/3131814961 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fc2a80cde20 con 0x7fc2b41015f0 2023-11-16T16:53:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.267+0000 7fc2b3fff700 1 --2- 172.21.15.44:0/3131814961 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc29c0a12c0 0x7fc29c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.267+0000 7fc2cda32b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 0x55e6244e51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.267+0000 7fc2cda32b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5720 0x55e6244df560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.267+0000 7fc2cda32b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5b90 0x55e6244dfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.267+0000 7fc2cda32b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6243e5000 con 0x55e62433b6b0 2023-11-16T16:53:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.267+0000 7fc2cda32b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e624437840 con 0x55e6244e5720 2023-11-16T16:53:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.267+0000 7fc2cda32b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e624430cd0 con 0x55e6244e5b90 2023-11-16T16:53:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.267+0000 7fc2b2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5720 0x55e6244df560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.267+0000 7fc2b37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 0x55e6244e51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.267+0000 7fc2b2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5720 0x55e6244df560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:40792/0 (socket says 172.21.15.44:40792) 2023-11-16T16:53:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.267+0000 7fc2b2ffd700 1 -- 172.21.15.44:0/2552321643 learned_addr learned my addr 172.21.15.44:0/2552321643 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.267+0000 7fc2b3fff700 1 --2- 172.21.15.44:0/3131814961 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc29c0a12c0 0x7fc29c0a3770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0093a0 tx=0x7fc2ac00b040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.268+0000 7fc2b3fff700 1 --2- 172.21.15.44:0/2552321643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5b90 0x55e6244dfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.268+0000 7fc2b2ffd700 1 -- 172.21.15.44:0/2552321643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5b90 msgr2=0x55e6244dfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.268+0000 7fc2b2ffd700 1 --2- 172.21.15.44:0/2552321643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5b90 0x55e6244dfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.268+0000 7fc2b2ffd700 1 -- 172.21.15.44:0/2552321643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 msgr2=0x55e6244e51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.268+0000 7fc2b2ffd700 1 --2- 172.21.15.44:0/2552321643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 0x55e6244e51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.268+0000 7fc2b2ffd700 1 -- 172.21.15.44:0/2552321643 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6242b25d0 con 0x55e6244e5720 2023-11-16T16:53:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.268+0000 7fc2b2ffd700 1 --2- 172.21.15.44:0/2552321643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5720 0x55e6244df560 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80970a0 tx=0x7fc2a809aa80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd6211130d190ba7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.268+0000 7fc299ffb700 1 -- 172.21.15.44:0/2552321643 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a80a2f60 con 0x55e6244e5720 2023-11-16T16:53:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.268+0000 7fc299ffb700 1 -- 172.21.15.44:0/2552321643 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc2a80ba070 con 0x55e6244e5720 2023-11-16T16:53:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.268+0000 7fc299ffb700 1 -- 172.21.15.44:0/2552321643 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a809b060 con 0x55e6244e5720 2023-11-16T16:53:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.269+0000 7fc2cda32b80 1 -- 172.21.15.44:0/2552321643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5720 msgr2=0x55e6244df560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.269+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/2552321643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5720 0x55e6244df560 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80970a0 tx=0x7fc2a809aa80 comp rx=0 tx=0).stop 2023-11-16T16:53:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.269+0000 7fc2cda32b80 1 -- 172.21.15.44:0/2552321643 shutdown_connections 2023-11-16T16:53:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.269+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/2552321643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5b90 0x55e6244dfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.269+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/2552321643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 0x55e6244e51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.269+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/2552321643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5720 0x55e6244df560 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.269+0000 7fc2cda32b80 1 -- 172.21.15.44:0/2552321643 >> 172.21.15.44:0/2552321643 conn(0x55e6244d4f50 msgr2=0x55e6244da270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.269+0000 7fc2cda32b80 1 -- 172.21.15.44:0/2552321643 shutdown_connections 2023-11-16T16:53:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.269+0000 7fc2cda32b80 1 -- 172.21.15.44:0/2552321643 wait complete. 2023-11-16T16:53:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.270+0000 7fc2cda32b80 1 -- start start 2023-11-16T16:53:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.270+0000 7fc2cda32b80 1 Processor -- start 2023-11-16T16:53:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.270+0000 7fc2cda32b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 0x55e6244ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.270+0000 7fc2cda32b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5720 0x55e6244eece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.270+0000 7fc2cda32b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5b90 0x55e6244e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.270+0000 7fc2cda32b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6243d2880 con 0x55e62433b6b0 2023-11-16T16:53:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.270+0000 7fc2cda32b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6243d2520 con 0x55e6244e5b90 2023-11-16T16:53:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.270+0000 7fc2cda32b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6242b48c0 con 0x55e6244e5720 2023-11-16T16:53:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.270+0000 7fc2cda32b80 10 client.0 _ll_register_callbacks cb 0x55e6243f0720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.270+0000 7fc2b37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 0x55e6244ee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.271+0000 7fc2b3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5b90 0x55e6244e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.271+0000 7fc2b37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 0x55e6244ee7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46048/0 (socket says 172.21.15.44:46048) 2023-11-16T16:53:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.271+0000 7fc2b37fe700 1 -- 172.21.15.44:0/2442882072 learned_addr learned my addr 172.21.15.44:0/2442882072 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.271+0000 7fc2b2ffd700 1 --2- 172.21.15.44:0/2442882072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5720 0x55e6244eece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.271+0000 7fc2b3fff700 1 -- 172.21.15.44:0/2442882072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5720 msgr2=0x55e6244eece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.271+0000 7fc2b3fff700 1 --2- 172.21.15.44:0/2442882072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5720 0x55e6244eece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.271+0000 7fc2b3fff700 1 -- 172.21.15.44:0/2442882072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 msgr2=0x55e6244ee7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.271+0000 7fc2b3fff700 1 --2- 172.21.15.44:0/2442882072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 0x55e6244ee7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.271+0000 7fc2b3fff700 1 -- 172.21.15.44:0/2442882072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b41101e0 con 0x55e6244e5b90 2023-11-16T16:53:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.271+0000 7fc2b37fe700 1 --2- 172.21.15.44:0/2442882072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 0x55e6244ee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.271+0000 7fc2b3fff700 1 --2- 172.21.15.44:0/2442882072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5b90 0x55e6244e8a30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x55e6244e5f60 tx=0x7fc2ac004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfac1695f7501b28 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.271+0000 7fc292ffd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:53:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.271+0000 7fc292ffd700 1 -- 172.21.15.44:0/2442882072 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac00e050 con 0x55e6244e5b90 2023-11-16T16:53:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.272+0000 7fc292ffd700 1 -- 172.21.15.44:0/2442882072 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fc2ac021440 con 0x55e6244e5b90 2023-11-16T16:53:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.272+0000 7fc2cda32b80 1 -- 172.21.15.44:0/2442882072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e6244d2710 con 0x55e6244e5b90 2023-11-16T16:53:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.272+0000 7fc292ffd700 1 -- 172.21.15.44:0/2442882072 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2ac021640 con 0x55e6244e5b90 2023-11-16T16:53:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.272+0000 7fc2cda32b80 10 client.26450 fetch_fsmap learned FSMap version 21 2023-11-16T16:53:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.272+0000 7fc2cda32b80 1 -- 172.21.15.44:0/2442882072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55e6243f7130 con 0x55e6244e5b90 2023-11-16T16:53:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.272+0000 7fc292ffd700 1 -- 172.21.15.44:0/2442882072 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e6244d2710 con 0x55e6244e5b90 2023-11-16T16:53:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.272+0000 7fc292ffd700 1 -- 172.21.15.44:0/2442882072 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc2ac02a680 con 0x55e6244e5b90 2023-11-16T16:53:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.272+0000 7fc2cda32b80 10 client.26450 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:53:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.272+0000 7fc2cda32b80 10 client.26450 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.272+0000 7fc2cda32b80 20 client.26450 populate_metadata read hostname 'smithi044' 2023-11-16T16:53:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.272+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/2442882072 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e6244f8cc0 0x55e6244fb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.272+0000 7fc2cda32b80 4 client.26450 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:53:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.272+0000 7fc2cda32b80 1 -- 172.21.15.44:0/2442882072 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e6243f3040 con 0x55e6244f8cc0 2023-11-16T16:53:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.273+0000 7fc2b37fe700 1 --2- 172.21.15.44:0/2442882072 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e6244f8cc0 0x55e6244fb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.273+0000 7fc2b37fe700 1 --2- 172.21.15.44:0/2442882072 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e6244f8cc0 0x55e6244fb0d0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d6b6efa1ad8b427 server_cookie=af7ba85980e73790 in_seq=0 out_seq=0 2023-11-16T16:53:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.273+0000 7fc292ffd700 10 client.26450 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.275+0000 7fc292ffd700 1 -- 172.21.15.44:0/2442882072 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e6243f3040 con 0x55e6244f8cc0 2023-11-16T16:53:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.275+0000 7fc292ffd700 10 client.26450 handle_command_reply: tid=0 2023-11-16T16:53:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.276+0000 7fc2cda32b80 1 client.26450 shutdown 2023-11-16T16:53:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.276+0000 7fc2cda32b80 1 -- 172.21.15.44:0/2442882072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5b90 msgr2=0x55e6244e8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.276+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/2442882072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5b90 0x55e6244e8a30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x55e6244e5f60 tx=0x7fc2ac004ad0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.276+0000 7fc2cda32b80 1 -- 172.21.15.44:0/2442882072 shutdown_connections 2023-11-16T16:53:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.276+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/2442882072 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e6244f8cc0 0x55e6244fb0d0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.276+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/2442882072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5b90 0x55e6244e8a30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.276+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/2442882072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 0x55e6244ee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.276+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/2442882072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5720 0x55e6244eece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.276+0000 7fc292ffd700 0 client.26450 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.276+0000 7fc2cda32b80 1 -- 172.21.15.44:0/2442882072 >> 172.21.15.44:0/2442882072 conn(0x55e6244d4f50 msgr2=0x55e6244d97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.276+0000 7fc2cda32b80 1 -- 172.21.15.44:0/2442882072 shutdown_connections 2023-11-16T16:53:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.276+0000 7fc2cda32b80 1 -- 172.21.15.44:0/2442882072 wait complete. 2023-11-16T16:53:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.276+0000 7fc2cda32b80 20 client.26450 trim_cache size 0 max 16384 2023-11-16T16:53:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.292+0000 7fc2cda32b80 1 Processor -- start 2023-11-16T16:53:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.292+0000 7fc2cda32b80 1 -- start start 2023-11-16T16:53:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.292+0000 7fc2cda32b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 0x55e6244e8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.292+0000 7fc2cda32b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5720 0x55e6244e2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.292+0000 7fc2cda32b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5b90 0x55e6244d9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.292+0000 7fc2cda32b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6244d6fd0 con 0x55e62433b6b0 2023-11-16T16:53:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.292+0000 7fc2cda32b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6244dbba0 con 0x55e6244e5720 2023-11-16T16:53:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.292+0000 7fc2cda32b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6244db3e0 con 0x55e6244e5b90 2023-11-16T16:53:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.293+0000 7fc2b37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 0x55e6244e8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.293+0000 7fc2b37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 0x55e6244e8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46062/0 (socket says 172.21.15.44:46062) 2023-11-16T16:53:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.293+0000 7fc2b2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5720 0x55e6244e2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.293+0000 7fc2b37fe700 1 -- 172.21.15.44:0/1163887051 learned_addr learned my addr 172.21.15.44:0/1163887051 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.293+0000 7fc2b3fff700 1 --2- 172.21.15.44:0/1163887051 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5b90 0x55e6244d9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.293+0000 7fc2b37fe700 1 -- 172.21.15.44:0/1163887051 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5b90 msgr2=0x55e6244d9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.293+0000 7fc2b37fe700 1 --2- 172.21.15.44:0/1163887051 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5b90 0x55e6244d9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.293+0000 7fc2b37fe700 1 -- 172.21.15.44:0/1163887051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5720 msgr2=0x55e6244e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.293+0000 7fc2b37fe700 1 --2- 172.21.15.44:0/1163887051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5720 0x55e6244e2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.293+0000 7fc2b37fe700 1 -- 172.21.15.44:0/1163887051 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6244d66f0 con 0x55e62433b6b0 2023-11-16T16:53:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.293+0000 7fc2b37fe700 1 --2- 172.21.15.44:0/1163887051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 0x55e6244e8b60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc2a40046d0 tx=0x7fc2a4009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90446105b16c3b9d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.294+0000 7fc29a7fc700 1 -- 172.21.15.44:0/1163887051 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4010c90 con 0x55e62433b6b0 2023-11-16T16:53:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.294+0000 7fc29a7fc700 1 -- 172.21.15.44:0/1163887051 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc2a4023070 con 0x55e62433b6b0 2023-11-16T16:53:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.294+0000 7fc29a7fc700 1 -- 172.21.15.44:0/1163887051 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4004a90 con 0x55e62433b6b0 2023-11-16T16:53:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.294+0000 7fc2cda32b80 1 -- 172.21.15.44:0/1163887051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 msgr2=0x55e6244e8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.294+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/1163887051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 0x55e6244e8b60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc2a40046d0 tx=0x7fc2a4009fd0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.294+0000 7fc2cda32b80 1 -- 172.21.15.44:0/1163887051 shutdown_connections 2023-11-16T16:53:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.294+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/1163887051 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5b90 0x55e6244d9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.294+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/1163887051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e62433b6b0 0x55e6244e8b60 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.294+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/1163887051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6244e5720 0x55e6244e2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.294+0000 7fc2cda32b80 1 -- 172.21.15.44:0/1163887051 >> 172.21.15.44:0/1163887051 conn(0x55e6244d4f50 msgr2=0x55e6244e93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.294+0000 7fc2cda32b80 1 -- 172.21.15.44:0/1163887051 shutdown_connections 2023-11-16T16:53:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.294+0000 7fc2cda32b80 1 -- 172.21.15.44:0/1163887051 wait complete. 2023-11-16T16:53:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.295+0000 7fc2cda32b80 1 -- start start 2023-11-16T16:53:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.295+0000 7fc2cda32b80 1 Processor -- start 2023-11-16T16:53:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.295+0000 7fc2cda32b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e62433b6b0 0x55e6244eeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.295+0000 7fc2cda32b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6244e5720 0x55e6244ec350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.295+0000 7fc2cda32b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5b90 0x55e6244ec910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.295+0000 7fc2cda32b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6244ecfe0 con 0x55e6244e5720 2023-11-16T16:53:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.295+0000 7fc2cda32b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6244ed150 con 0x55e62433b6b0 2023-11-16T16:53:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.295+0000 7fc2cda32b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e62454bb40 con 0x55e6244e5b90 2023-11-16T16:53:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.295+0000 7fc2cda32b80 10 client.0 _ll_register_callbacks cb 0x7fc2ac0158c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.296+0000 7fc2b37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e62433b6b0 0x55e6244eeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.296+0000 7fc2b37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e62433b6b0 0x55e6244eeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:40824/0 (socket says 172.21.15.44:40824) 2023-11-16T16:53:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.296+0000 7fc2b37fe700 1 -- 172.21.15.44:0/4236298936 learned_addr learned my addr 172.21.15.44:0/4236298936 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:53:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.296+0000 7fc2b2ffd700 1 --2- 172.21.15.44:0/4236298936 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6244e5720 0x55e6244ec350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.296+0000 7fc2b3fff700 1 --2- 172.21.15.44:0/4236298936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5b90 0x55e6244ec910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.296+0000 7fc2b37fe700 1 -- 172.21.15.44:0/4236298936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5b90 msgr2=0x55e6244ec910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.296+0000 7fc2b37fe700 1 --2- 172.21.15.44:0/4236298936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5b90 0x55e6244ec910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.296+0000 7fc2b37fe700 1 -- 172.21.15.44:0/4236298936 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6244e5720 msgr2=0x55e6244ec350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.296+0000 7fc2b37fe700 1 --2- 172.21.15.44:0/4236298936 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6244e5720 0x55e6244ec350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.296+0000 7fc2b37fe700 1 -- 172.21.15.44:0/4236298936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a4009e40 con 0x55e62433b6b0 2023-11-16T16:53:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.296+0000 7fc2b2ffd700 1 --2- 172.21.15.44:0/4236298936 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6244e5720 0x55e6244ec350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.296+0000 7fc2b3fff700 1 --2- 172.21.15.44:0/4236298936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5b90 0x55e6244ec910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.296+0000 7fc2b37fe700 1 --2- 172.21.15.44:0/4236298936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e62433b6b0 0x55e6244eeb80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4010cc0 tx=0x7fc2a4010cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1da2805cb8e0b7b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.296+0000 7fc292ffd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:53:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.297+0000 7fc292ffd700 1 -- 172.21.15.44:0/4236298936 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a401f250 con 0x55e62433b6b0 2023-11-16T16:53:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.297+0000 7fc292ffd700 1 -- 172.21.15.44:0/4236298936 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fc2a401f3b0 con 0x55e62433b6b0 2023-11-16T16:53:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.297+0000 7fc2cda32b80 1 -- 172.21.15.44:0/4236298936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc2a80ba070 con 0x55e62433b6b0 2023-11-16T16:53:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.297+0000 7fc292ffd700 1 -- 172.21.15.44:0/4236298936 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2a401f5b0 con 0x55e62433b6b0 2023-11-16T16:53:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.297+0000 7fc2cda32b80 10 client.26456 fetch_fsmap learned FSMap version 21 2023-11-16T16:53:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.297+0000 7fc2cda32b80 1 -- 172.21.15.44:0/4236298936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55e62425ed80 con 0x55e62433b6b0 2023-11-16T16:53:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.297+0000 7fc292ffd700 1 -- 172.21.15.44:0/4236298936 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc2a80ba070 con 0x55e62433b6b0 2023-11-16T16:53:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.297+0000 7fc292ffd700 1 -- 172.21.15.44:0/4236298936 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc2a4029690 con 0x55e62433b6b0 2023-11-16T16:53:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.297+0000 7fc2cda32b80 10 client.26456 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:53:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.297+0000 7fc2cda32b80 10 client.26456 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.297+0000 7fc2cda32b80 20 client.26456 populate_metadata read hostname 'smithi044' 2023-11-16T16:53:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.297+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/4236298936 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e62454d160 0x55e62454f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.297+0000 7fc2cda32b80 4 client.26456 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:53:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.297+0000 7fc2cda32b80 1 -- 172.21.15.44:0/4236298936 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e6243ff600 con 0x55e62454d160 2023-11-16T16:53:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.298+0000 7fc2b2ffd700 1 --2- 172.21.15.44:0/4236298936 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e62454d160 0x55e62454f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:45.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.298+0000 7fc2b2ffd700 1 --2- 172.21.15.44:0/4236298936 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e62454d160 0x55e62454f570 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb34327872eda142 server_cookie=845f9746ae0d2134 in_seq=0 out_seq=0 2023-11-16T16:53:45.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:45.298+0000 7fc292ffd700 10 client.26456 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:45.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:45 smithi044 ceph-mon[128750]: pgmap v117: 129 pgs: 129 active+clean; 33 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 2.48k op/s 2023-11-16T16:53:45.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:45 smithi176 ceph-mon[131630]: pgmap v117: 129 pgs: 129 active+clean; 33 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 2.48k op/s 2023-11-16T16:53:45.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:45 smithi077 ceph-mon[131204]: pgmap v117: 129 pgs: 129 active+clean; 33 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 2.48k op/s 2023-11-16T16:53:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:47 smithi044 ceph-mon[128750]: pgmap v118: 129 pgs: 129 active+clean; 33 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 312 op/s 2023-11-16T16:53:47.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:47 smithi176 ceph-mon[131630]: pgmap v118: 129 pgs: 129 active+clean; 33 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 312 op/s 2023-11-16T16:53:47.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:47 smithi077 ceph-mon[131204]: pgmap v118: 129 pgs: 129 active+clean; 33 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 312 op/s 2023-11-16T16:53:49.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:49 smithi044 ceph-mon[128750]: pgmap v119: 129 pgs: 129 active+clean; 53 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.7 MiB/s wr, 444 op/s 2023-11-16T16:53:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:49 smithi176 ceph-mon[131630]: pgmap v119: 129 pgs: 129 active+clean; 53 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.7 MiB/s wr, 444 op/s 2023-11-16T16:53:49.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:49 smithi077 ceph-mon[131204]: pgmap v119: 129 pgs: 129 active+clean; 53 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.7 MiB/s wr, 444 op/s 2023-11-16T16:53:51.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:51 smithi044 ceph-mon[128750]: pgmap v120: 129 pgs: 129 active+clean; 53 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 307 op/s 2023-11-16T16:53:51.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:51 smithi176 ceph-mon[131630]: pgmap v120: 129 pgs: 129 active+clean; 53 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 307 op/s 2023-11-16T16:53:51.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:51 smithi077 ceph-mon[131204]: pgmap v120: 129 pgs: 129 active+clean; 53 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 307 op/s 2023-11-16T16:53:53.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:53 smithi044 ceph-mon[128750]: pgmap v121: 129 pgs: 129 active+clean; 61 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 1.07k op/s 2023-11-16T16:53:53.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:53 smithi176 ceph-mon[131630]: pgmap v121: 129 pgs: 129 active+clean; 61 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 1.07k op/s 2023-11-16T16:53:53.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:53 smithi077 ceph-mon[131204]: pgmap v121: 129 pgs: 129 active+clean; 61 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 1.07k op/s 2023-11-16T16:53:54.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.105+0000 7fc292ffd700 1 -- 172.21.15.44:0/4236298936 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e6243ff600 con 0x55e62454d160 2023-11-16T16:53:54.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.105+0000 7fc292ffd700 10 client.26456 handle_command_reply: tid=0 2023-11-16T16:53:54.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.105+0000 7fc2cda32b80 1 client.26456 shutdown 2023-11-16T16:53:54.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.105+0000 7fc2cda32b80 1 -- 172.21.15.44:0/4236298936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e62433b6b0 msgr2=0x55e6244eeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:54.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.105+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/4236298936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e62433b6b0 0x55e6244eeb80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4010cc0 tx=0x7fc2a4010cf0 comp rx=0 tx=0).stop 2023-11-16T16:53:54.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.105+0000 7fc2cda32b80 1 -- 172.21.15.44:0/4236298936 shutdown_connections 2023-11-16T16:53:54.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.105+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/4236298936 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e62454d160 0x55e62454f570 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:54.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.105+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/4236298936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e6244e5b90 0x55e6244ec910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:54.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.105+0000 7fc292ffd700 0 client.26456 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:53:54.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.105+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/4236298936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e62433b6b0 0x55e6244eeb80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.105+0000 7fc2cda32b80 1 --2- 172.21.15.44:0/4236298936 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6244e5720 0x55e6244ec350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.105+0000 7fc2cda32b80 1 -- 172.21.15.44:0/4236298936 >> 172.21.15.44:0/4236298936 conn(0x55e6244d4f50 msgr2=0x55e6244e2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.105+0000 7fc2cda32b80 1 -- 172.21.15.44:0/4236298936 shutdown_connections 2023-11-16T16:53:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.105+0000 7fc2cda32b80 1 -- 172.21.15.44:0/4236298936 wait complete. 2023-11-16T16:53:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.106+0000 7fc2cda32b80 20 client.26456 trim_cache size 0 max 16384 2023-11-16T16:53:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:53:54.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:53:54.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:53:54.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:53:54.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.106+0000 7fc2ba1ff700 1 -- 172.21.15.44:0/3131814961 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc29c0a12c0 msgr2=0x7fc29c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:54.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.106+0000 7fc2ba1ff700 1 --2- 172.21.15.44:0/3131814961 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc29c0a12c0 0x7fc29c0a3770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0093a0 tx=0x7fc2ac00b040 comp rx=0 tx=0).stop 2023-11-16T16:53:54.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.106+0000 7fc2ba1ff700 1 -- 172.21.15.44:0/3131814961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2b41015f0 msgr2=0x7fc2b4198b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:54.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.106+0000 7fc2ba1ff700 1 --2- 172.21.15.44:0/3131814961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2b41015f0 0x7fc2b4198b30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80098e0 tx=0x7fc2a800f690 comp rx=0 tx=0).stop 2023-11-16T16:53:54.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.107+0000 7fc2ba1ff700 1 -- 172.21.15.44:0/3131814961 shutdown_connections 2023-11-16T16:53:54.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.107+0000 7fc2ba1ff700 1 --2- 172.21.15.44:0/3131814961 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc29c0a12c0 0x7fc29c0a3770 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:54.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.107+0000 7fc2ba1ff700 1 --2- 172.21.15.44:0/3131814961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2b4105560 0x7fc2b419cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:54.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.107+0000 7fc2ba1ff700 1 --2- 172.21.15.44:0/3131814961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2b4100c50 0x7fc2b41985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:54.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.107+0000 7fc2ba1ff700 1 --2- 172.21.15.44:0/3131814961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2b41015f0 0x7fc2b4198b30 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:54.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.107+0000 7fc2ba1ff700 1 -- 172.21.15.44:0/3131814961 >> 172.21.15.44:0/3131814961 conn(0x7fc2b4077550 msgr2=0x7fc2b40fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:54.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.107+0000 7fc2ba1ff700 1 -- 172.21.15.44:0/3131814961 shutdown_connections 2023-11-16T16:53:54.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:53:54.107+0000 7fc2ba1ff700 1 -- 172.21.15.44:0/3131814961 wait complete. 2023-11-16T16:53:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:55 smithi044 ceph-mon[128750]: pgmap v122: 129 pgs: 129 active+clean; 61 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 2.28k op/s 2023-11-16T16:53:55.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:55 smithi176 ceph-mon[131630]: pgmap v122: 129 pgs: 129 active+clean; 61 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 2.28k op/s 2023-11-16T16:53:55.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:55 smithi077 ceph-mon[131204]: pgmap v122: 129 pgs: 129 active+clean; 61 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 2.28k op/s 2023-11-16T16:53:57.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:57 smithi044 ceph-mon[128750]: pgmap v123: 129 pgs: 129 active+clean; 61 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 2.18k op/s 2023-11-16T16:53:57.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:57 smithi176 ceph-mon[131630]: pgmap v123: 129 pgs: 129 active+clean; 61 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 2.18k op/s 2023-11-16T16:53:57.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:57 smithi077 ceph-mon[131204]: pgmap v123: 129 pgs: 129 active+clean; 61 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 2.18k op/s 2023-11-16T16:53:59.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:53:59 smithi044 ceph-mon[128750]: pgmap v124: 129 pgs: 129 active+clean; 25 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 2.51k op/s 2023-11-16T16:53:59.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:53:59 smithi176 ceph-mon[131630]: pgmap v124: 129 pgs: 129 active+clean; 25 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 2.51k op/s 2023-11-16T16:53:59.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:53:59 smithi077 ceph-mon[131204]: pgmap v124: 129 pgs: 129 active+clean; 25 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 2.51k op/s 2023-11-16T16:54:01.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:01 smithi044 ceph-mon[128750]: pgmap v125: 129 pgs: 129 active+clean; 25 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 2.38k op/s 2023-11-16T16:54:01.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:01 smithi176 ceph-mon[131630]: pgmap v125: 129 pgs: 129 active+clean; 25 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 2.38k op/s 2023-11-16T16:54:01.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:01 smithi077 ceph-mon[131204]: pgmap v125: 129 pgs: 129 active+clean; 25 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 2.38k op/s 2023-11-16T16:54:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:03 smithi044 ceph-mon[128750]: pgmap v126: 129 pgs: 129 active+clean; 29 MiB data, 672 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 2.51k op/s 2023-11-16T16:54:03.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:03 smithi176 ceph-mon[131630]: pgmap v126: 129 pgs: 129 active+clean; 29 MiB data, 672 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 2.51k op/s 2023-11-16T16:54:03.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:03 smithi077 ceph-mon[131204]: pgmap v126: 129 pgs: 129 active+clean; 29 MiB data, 672 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 2.51k op/s 2023-11-16T16:54:04.904 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:54:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.119+0000 7f65e8dff700 1 -- 172.21.15.44:0/1064901561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e4100cc0 msgr2=0x7f65e4101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.119+0000 7f65e8dff700 1 --2- 172.21.15.44:0/1064901561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e4100cc0 0x7f65e4101090 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f65d40099c0 tx=0x7f65d4009cd0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.119+0000 7f65e8dff700 1 -- 172.21.15.44:0/1064901561 shutdown_connections 2023-11-16T16:54:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.119+0000 7f65e8dff700 1 --2- 172.21.15.44:0/1064901561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65e4105620 0x7f65e410d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.119+0000 7f65e8dff700 1 --2- 172.21.15.44:0/1064901561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e4100cc0 0x7f65e4101090 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.119+0000 7f65e8dff700 1 --2- 172.21.15.44:0/1064901561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65e4101660 0x7f65e4104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.119+0000 7f65e8dff700 1 -- 172.21.15.44:0/1064901561 >> 172.21.15.44:0/1064901561 conn(0x7f65e4077560 msgr2=0x7f65e4077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.119+0000 7f65e8dff700 1 -- 172.21.15.44:0/1064901561 shutdown_connections 2023-11-16T16:54:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.119+0000 7f65e8dff700 1 -- 172.21.15.44:0/1064901561 wait complete. 2023-11-16T16:54:05.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.120+0000 7f65e8dff700 1 Processor -- start 2023-11-16T16:54:05.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.120+0000 7f65e8dff700 1 -- start start 2023-11-16T16:54:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.120+0000 7f65e8dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65e4100cc0 0x7f65e41986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65e8dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65e4101660 0x7f65e4198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65e8dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e4105620 0x7f65e419bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65e8dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e419c5a0 con 0x7f65e4101660 2023-11-16T16:54:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65e8dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e419c710 con 0x7f65e4105620 2023-11-16T16:54:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65e8dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e419c8d0 con 0x7f65e4100cc0 2023-11-16T16:54:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65dbfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65e4101660 0x7f65e4198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65e2d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e4105620 0x7f65e419bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65dbfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65e4101660 0x7f65e4198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52704/0 (socket says 172.21.15.44:52704) 2023-11-16T16:54:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65dbfff700 1 -- 172.21.15.44:0/1231094778 learned_addr learned my addr 172.21.15.44:0/1231094778 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65e259c700 1 --2- 172.21.15.44:0/1231094778 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65e4100cc0 0x7f65e41986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65dbfff700 1 -- 172.21.15.44:0/1231094778 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65e4100cc0 msgr2=0x7f65e41986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:05.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65dbfff700 1 --2- 172.21.15.44:0/1231094778 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65e4100cc0 0x7f65e41986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65dbfff700 1 -- 172.21.15.44:0/1231094778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e4105620 msgr2=0x7f65e419bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:05.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65dbfff700 1 --2- 172.21.15.44:0/1231094778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e4105620 0x7f65e419bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65dbfff700 1 -- 172.21.15.44:0/1231094778 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d40095f0 con 0x7f65e4101660 2023-11-16T16:54:05.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65e2d9d700 1 --2- 172.21.15.44:0/1231094778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e4105620 0x7f65e419bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:05.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.121+0000 7f65dbfff700 1 --2- 172.21.15.44:0/1231094778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65e4101660 0x7f65e4198bf0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00e910 tx=0x7f65cc00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:05.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.122+0000 7f65db7fe700 1 -- 172.21.15.44:0/1231094778 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc004a30 con 0x7f65e4101660 2023-11-16T16:54:05.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.122+0000 7f65db7fe700 1 -- 172.21.15.44:0/1231094778 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65cc0054e0 con 0x7f65e4101660 2023-11-16T16:54:05.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.122+0000 7f65db7fe700 1 -- 172.21.15.44:0/1231094778 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc00cc40 con 0x7f65e4101660 2023-11-16T16:54:05.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.122+0000 7f65e8dff700 1 -- 172.21.15.44:0/1231094778 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e41103d0 con 0x7f65e4101660 2023-11-16T16:54:05.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.122+0000 7f65e8dff700 1 -- 172.21.15.44:0/1231094778 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e4110240 con 0x7f65e4101660 2023-11-16T16:54:05.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.125+0000 7f65db7fe700 1 -- 172.21.15.44:0/1231094778 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f65cc005000 con 0x7f65e4101660 2023-11-16T16:54:05.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.127+0000 7f65db7fe700 1 --2- 172.21.15.44:0/1231094778 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f65d00a1340 0x7f65d00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.127+0000 7f65fc726b80 1 Processor -- start 2023-11-16T16:54:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.127+0000 7f65db7fe700 1 -- 172.21.15.44:0/1231094778 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f65cc014070 con 0x7f65e4101660 2023-11-16T16:54:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.127+0000 7f65fc726b80 1 -- start start 2023-11-16T16:54:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.127+0000 7f65e259c700 1 --2- 172.21.15.44:0/1231094778 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f65d00a1340 0x7f65d00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65fc726b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943da9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65fc726b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943daa470 0x55d943dae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65fc726b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943dae8b0 0x55d943daed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65fc726b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d943caf060 con 0x55d943dae8b0 2023-11-16T16:54:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65fc726b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d943d018a0 con 0x55d943da9b60 2023-11-16T16:54:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65fc726b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d943cfad30 con 0x55d943daa470 2023-11-16T16:54:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65e259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943da9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65e259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943da9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59844/0 (socket says 172.21.15.44:59844) 2023-11-16T16:54:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65e259c700 1 -- 172.21.15.44:0/2842122260 learned_addr learned my addr 172.21.15.44:0/2842122260 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65dbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943daa470 0x55d943dae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65e2d9d700 1 --2- 172.21.15.44:0/2842122260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943dae8b0 0x55d943daed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65dbfff700 1 -- 172.21.15.44:0/2842122260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 msgr2=0x55d943da9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65dbfff700 1 --2- 172.21.15.44:0/2842122260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943da9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65dbfff700 1 -- 172.21.15.44:0/2842122260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943dae8b0 msgr2=0x55d943daed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65dbfff700 1 --2- 172.21.15.44:0/2842122260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943dae8b0 0x55d943daed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65dbfff700 1 -- 172.21.15.44:0/2842122260 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d943b7c630 con 0x55d943daa470 2023-11-16T16:54:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.128+0000 7f65e2d9d700 1 --2- 172.21.15.44:0/2842122260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943dae8b0 0x55d943daed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:54:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.129+0000 7f65e259c700 1 --2- 172.21.15.44:0/1231094778 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f65d00a1340 0x7f65d00a37f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f65d40036c0 tx=0x7f65d4004c10 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.129+0000 7f65dbfff700 1 --2- 172.21.15.44:0/2842122260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943daa470 0x55d943dae370 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f65cc0ca6a0 tx=0x7f65cc099f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7804bdd91ec0620 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.129+0000 7f65d8ff9700 1 -- 172.21.15.44:0/2842122260 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc0cd170 con 0x55d943daa470 2023-11-16T16:54:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.129+0000 7f65d8ff9700 1 -- 172.21.15.44:0/2842122260 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65cc09e050 con 0x55d943daa470 2023-11-16T16:54:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.129+0000 7f65d8ff9700 1 -- 172.21.15.44:0/2842122260 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc00cea0 con 0x55d943daa470 2023-11-16T16:54:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.130+0000 7f65fc726b80 1 -- 172.21.15.44:0/2842122260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943daa470 msgr2=0x55d943dae370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.130+0000 7f65fc726b80 1 --2- 172.21.15.44:0/2842122260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943daa470 0x55d943dae370 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f65cc0ca6a0 tx=0x7f65cc099f20 comp rx=0 tx=0).stop 2023-11-16T16:54:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.130+0000 7f65fc726b80 1 -- 172.21.15.44:0/2842122260 shutdown_connections 2023-11-16T16:54:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.130+0000 7f65fc726b80 1 --2- 172.21.15.44:0/2842122260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943dae8b0 0x55d943daed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.130+0000 7f65fc726b80 1 --2- 172.21.15.44:0/2842122260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943da9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.130+0000 7f65fc726b80 1 --2- 172.21.15.44:0/2842122260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943daa470 0x55d943dae370 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.130+0000 7f65fc726b80 1 -- 172.21.15.44:0/2842122260 >> 172.21.15.44:0/2842122260 conn(0x55d943d9f360 msgr2=0x55d943da4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.130+0000 7f65fc726b80 1 -- 172.21.15.44:0/2842122260 shutdown_connections 2023-11-16T16:54:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.130+0000 7f65fc726b80 1 -- 172.21.15.44:0/2842122260 wait complete. 2023-11-16T16:54:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.130+0000 7f65fc726b80 1 -- start start 2023-11-16T16:54:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.131+0000 7f65fc726b80 1 Processor -- start 2023-11-16T16:54:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.131+0000 7f65fc726b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943db3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.131+0000 7f65fc726b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943daa470 0x55d943db3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.131+0000 7f65fc726b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943dae8b0 0x55d943db3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.131+0000 7f65fc726b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d943c9c8e0 con 0x55d943daa470 2023-11-16T16:54:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.131+0000 7f65fc726b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d943c9c580 con 0x55d943da9b60 2023-11-16T16:54:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.131+0000 7f65fc726b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d943b7e920 con 0x55d943dae8b0 2023-11-16T16:54:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.131+0000 7f65fc726b80 10 client.0 _ll_register_callbacks cb 0x55d943cba780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.131+0000 7f65e259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943db3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.131+0000 7f65e259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943db3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59860/0 (socket says 172.21.15.44:59860) 2023-11-16T16:54:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.131+0000 7f65e259c700 1 -- 172.21.15.44:0/1397788881 learned_addr learned my addr 172.21.15.44:0/1397788881 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.131+0000 7f65dbfff700 1 --2- 172.21.15.44:0/1397788881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943daa470 0x55d943db3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.132+0000 7f65e2d9d700 1 --2- 172.21.15.44:0/1397788881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943dae8b0 0x55d943db3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.132+0000 7f65e259c700 1 -- 172.21.15.44:0/1397788881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943dae8b0 msgr2=0x55d943db3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.132+0000 7f65e259c700 1 --2- 172.21.15.44:0/1397788881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943dae8b0 0x55d943db3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.132+0000 7f65e259c700 1 -- 172.21.15.44:0/1397788881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943daa470 msgr2=0x55d943db3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.132+0000 7f65e259c700 1 --2- 172.21.15.44:0/1397788881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943daa470 0x55d943db3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.132+0000 7f65e259c700 1 -- 172.21.15.44:0/1397788881 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e4110240 con 0x55d943da9b60 2023-11-16T16:54:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.132+0000 7f65e259c700 1 --2- 172.21.15.44:0/1397788881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943db3320 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f65d40058e0 tx=0x7f65d4009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72e80d65f5f08e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.132+0000 7f65c9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:54:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.132+0000 7f65c9ffb700 1 -- 172.21.15.44:0/1397788881 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d400bc60 con 0x55d943da9b60 2023-11-16T16:54:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.132+0000 7f65fc726b80 1 -- 172.21.15.44:0/1397788881 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d943d9cb20 con 0x55d943da9b60 2023-11-16T16:54:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.132+0000 7f65c9ffb700 1 -- 172.21.15.44:0/1397788881 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f65d400bdc0 con 0x55d943da9b60 2023-11-16T16:54:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.133+0000 7f65c9ffb700 1 -- 172.21.15.44:0/1397788881 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65d4031460 con 0x55d943da9b60 2023-11-16T16:54:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.133+0000 7f65fc726b80 10 client.26468 fetch_fsmap learned FSMap version 21 2023-11-16T16:54:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.133+0000 7f65fc726b80 1 -- 172.21.15.44:0/1397788881 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55d943cc1190 con 0x55d943da9b60 2023-11-16T16:54:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.133+0000 7f65c9ffb700 1 -- 172.21.15.44:0/1397788881 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d943d9cb20 con 0x55d943da9b60 2023-11-16T16:54:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.133+0000 7f65c9ffb700 1 -- 172.21.15.44:0/1397788881 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f65d400f680 con 0x55d943da9b60 2023-11-16T16:54:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.133+0000 7f65fc726b80 10 client.26468 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:54:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.133+0000 7f65fc726b80 10 client.26468 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.133+0000 7f65fc726b80 20 client.26468 populate_metadata read hostname 'smithi044' 2023-11-16T16:54:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.133+0000 7f65fc726b80 1 --2- 172.21.15.44:0/1397788881 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d943dc3770 0x55d943dc5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.133+0000 7f65fc726b80 4 client.26468 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:54:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.133+0000 7f65fc726b80 1 -- 172.21.15.44:0/1397788881 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d943cbd0a0 con 0x55d943dc3770 2023-11-16T16:54:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.133+0000 7f65dbfff700 1 --2- 172.21.15.44:0/1397788881 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d943dc3770 0x55d943dc5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.134+0000 7f65dbfff700 1 --2- 172.21.15.44:0/1397788881 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d943dc3770 0x55d943dc5b50 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6ab026c0faf63e7 server_cookie=f8989a95f33b427e in_seq=0 out_seq=0 2023-11-16T16:54:05.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.134+0000 7f65c9ffb700 10 client.26468 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:05.139 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.136+0000 7f65c9ffb700 1 -- 172.21.15.44:0/1397788881 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d943cbd0a0 con 0x55d943dc3770 2023-11-16T16:54:05.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.136+0000 7f65c9ffb700 10 client.26468 handle_command_reply: tid=0 2023-11-16T16:54:05.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.136+0000 7f65fc726b80 1 client.26468 shutdown 2023-11-16T16:54:05.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.136+0000 7f65fc726b80 1 -- 172.21.15.44:0/1397788881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 msgr2=0x55d943db3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:05.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.136+0000 7f65fc726b80 1 --2- 172.21.15.44:0/1397788881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943db3320 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f65d40058e0 tx=0x7f65d4009cd0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.136+0000 7f65fc726b80 1 -- 172.21.15.44:0/1397788881 shutdown_connections 2023-11-16T16:54:05.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.136+0000 7f65fc726b80 1 --2- 172.21.15.44:0/1397788881 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d943dc3770 0x55d943dc5b50 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.136+0000 7f65fc726b80 1 --2- 172.21.15.44:0/1397788881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943dae8b0 0x55d943db3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.136+0000 7f65fc726b80 1 --2- 172.21.15.44:0/1397788881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943db3320 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.136+0000 7f65fc726b80 1 --2- 172.21.15.44:0/1397788881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943daa470 0x55d943db3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.136+0000 7f65c9ffb700 0 client.26468 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:05.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.136+0000 7f65fc726b80 1 -- 172.21.15.44:0/1397788881 >> 172.21.15.44:0/1397788881 conn(0x55d943d9f360 msgr2=0x55d943dafd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:05.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.136+0000 7f65fc726b80 1 -- 172.21.15.44:0/1397788881 shutdown_connections 2023-11-16T16:54:05.140 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.136+0000 7f65fc726b80 1 -- 172.21.15.44:0/1397788881 wait complete. 2023-11-16T16:54:05.141 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.136+0000 7f65fc726b80 20 client.26468 trim_cache size 0 max 16384 2023-11-16T16:54:05.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.154+0000 7f65fc726b80 1 Processor -- start 2023-11-16T16:54:05.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.154+0000 7f65fc726b80 1 -- start start 2023-11-16T16:54:05.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.154+0000 7f65fc726b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943da9b60 0x55d943dae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.154+0000 7f65fc726b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943daa470 0x55d943db3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.154+0000 7f65fc726b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943dae8b0 0x55d943da4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.154+0000 7f65fc726b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d943dc0330 con 0x55d943da9b60 2023-11-16T16:54:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.154+0000 7f65fc726b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d943db18a0 con 0x55d943daa470 2023-11-16T16:54:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.154+0000 7f65fc726b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d943dad4e0 con 0x55d943dae8b0 2023-11-16T16:54:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.155+0000 7f65e259c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943da9b60 0x55d943dae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.155+0000 7f65dbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943daa470 0x55d943db3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.155+0000 7f65e259c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943da9b60 0x55d943dae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52730/0 (socket says 172.21.15.44:52730) 2023-11-16T16:54:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.155+0000 7f65e259c700 1 -- 172.21.15.44:0/451326867 learned_addr learned my addr 172.21.15.44:0/451326867 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.155+0000 7f65e2d9d700 1 --2- 172.21.15.44:0/451326867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943dae8b0 0x55d943da4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.155+0000 7f65dbfff700 1 -- 172.21.15.44:0/451326867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943dae8b0 msgr2=0x55d943da4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.155+0000 7f65dbfff700 1 --2- 172.21.15.44:0/451326867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943dae8b0 0x55d943da4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.155+0000 7f65dbfff700 1 -- 172.21.15.44:0/451326867 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943da9b60 msgr2=0x55d943dae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.155+0000 7f65dbfff700 1 --2- 172.21.15.44:0/451326867 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943da9b60 0x55d943dae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.155+0000 7f65dbfff700 1 -- 172.21.15.44:0/451326867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d943daef50 con 0x55d943daa470 2023-11-16T16:54:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.155+0000 7f65e259c700 1 --2- 172.21.15.44:0/451326867 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943da9b60 0x55d943dae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:54:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.156+0000 7f65dbfff700 1 --2- 172.21.15.44:0/451326867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943daa470 0x55d943db3bd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f65cc009630 tx=0x7f65cc004f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef5509b8b2920076 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.156+0000 7f65d97fa700 1 -- 172.21.15.44:0/451326867 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc0a24f0 con 0x55d943daa470 2023-11-16T16:54:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.156+0000 7f65d97fa700 1 -- 172.21.15.44:0/451326867 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65cc004b90 con 0x55d943daa470 2023-11-16T16:54:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.156+0000 7f65d97fa700 1 -- 172.21.15.44:0/451326867 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc09a280 con 0x55d943daa470 2023-11-16T16:54:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.157+0000 7f65fc726b80 1 -- 172.21.15.44:0/451326867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943daa470 msgr2=0x55d943db3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.157+0000 7f65fc726b80 1 --2- 172.21.15.44:0/451326867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943daa470 0x55d943db3bd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f65cc009630 tx=0x7f65cc004f60 comp rx=0 tx=0).stop 2023-11-16T16:54:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.157+0000 7f65fc726b80 1 -- 172.21.15.44:0/451326867 shutdown_connections 2023-11-16T16:54:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.157+0000 7f65fc726b80 1 --2- 172.21.15.44:0/451326867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943dae8b0 0x55d943da4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.157+0000 7f65fc726b80 1 --2- 172.21.15.44:0/451326867 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943da9b60 0x55d943dae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.157+0000 7f65fc726b80 1 --2- 172.21.15.44:0/451326867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943daa470 0x55d943db3bd0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.157+0000 7f65fc726b80 1 -- 172.21.15.44:0/451326867 >> 172.21.15.44:0/451326867 conn(0x55d943d9f360 msgr2=0x55d943dad8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:05.160 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.157+0000 7f65fc726b80 1 -- 172.21.15.44:0/451326867 shutdown_connections 2023-11-16T16:54:05.160 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.157+0000 7f65fc726b80 1 -- 172.21.15.44:0/451326867 wait complete. 2023-11-16T16:54:05.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.157+0000 7f65fc726b80 1 -- start start 2023-11-16T16:54:05.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.157+0000 7f65fc726b80 1 Processor -- start 2023-11-16T16:54:05.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.158+0000 7f65fc726b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943db8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.158+0000 7f65fc726b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943daa470 0x55d943db8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.158+0000 7f65fc726b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943dae8b0 0x55d943e105d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.158+0000 7f65fc726b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d943e10b10 con 0x55d943dae8b0 2023-11-16T16:54:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.158+0000 7f65fc726b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d943e10c50 con 0x55d943da9b60 2023-11-16T16:54:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.158+0000 7f65fc726b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d943e10e10 con 0x55d943daa470 2023-11-16T16:54:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.158+0000 7f65fc726b80 10 client.0 _ll_register_callbacks cb 0x55d943cba780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.158+0000 7f65e259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943db8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.158+0000 7f65dbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943daa470 0x55d943db8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.158+0000 7f65e259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943db8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59872/0 (socket says 172.21.15.44:59872) 2023-11-16T16:54:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.158+0000 7f65e259c700 1 -- 172.21.15.44:0/4245109823 learned_addr learned my addr 172.21.15.44:0/4245109823 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.158+0000 7f65e2d9d700 1 --2- 172.21.15.44:0/4245109823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943dae8b0 0x55d943e105d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.159+0000 7f65e259c700 1 -- 172.21.15.44:0/4245109823 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943daa470 msgr2=0x55d943db8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.159+0000 7f65e259c700 1 --2- 172.21.15.44:0/4245109823 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943daa470 0x55d943db8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.159+0000 7f65e259c700 1 -- 172.21.15.44:0/4245109823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943dae8b0 msgr2=0x55d943e105d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.159+0000 7f65e259c700 1 --2- 172.21.15.44:0/4245109823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943dae8b0 0x55d943e105d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.159+0000 7f65e259c700 1 -- 172.21.15.44:0/4245109823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d943b7c630 con 0x55d943da9b60 2023-11-16T16:54:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.159+0000 7f65e2d9d700 1 --2- 172.21.15.44:0/4245109823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943dae8b0 0x55d943e105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:54:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.159+0000 7f65e259c700 1 --2- 172.21.15.44:0/4245109823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943db8840 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f65d4005760 tx=0x7f65d4005790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32a4df3601e1dedb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.159+0000 7f65c9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:54:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.159+0000 7f65c9ffb700 1 -- 172.21.15.44:0/4245109823 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d400fd10 con 0x55d943da9b60 2023-11-16T16:54:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.159+0000 7f65c9ffb700 1 -- 172.21.15.44:0/4245109823 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f65d400ba60 con 0x55d943da9b60 2023-11-16T16:54:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.159+0000 7f65fc726b80 1 -- 172.21.15.44:0/4245109823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f65cc09e050 con 0x55d943da9b60 2023-11-16T16:54:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.160+0000 7f65c9ffb700 1 -- 172.21.15.44:0/4245109823 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65d400bc20 con 0x55d943da9b60 2023-11-16T16:54:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.160+0000 7f65fc726b80 10 client.26474 fetch_fsmap learned FSMap version 21 2023-11-16T16:54:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.160+0000 7f65fc726b80 1 -- 172.21.15.44:0/4245109823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55d943b28de0 con 0x55d943da9b60 2023-11-16T16:54:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.160+0000 7f65c9ffb700 1 -- 172.21.15.44:0/4245109823 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65cc09e050 con 0x55d943da9b60 2023-11-16T16:54:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.160+0000 7f65c9ffb700 1 -- 172.21.15.44:0/4245109823 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f65d4031580 con 0x55d943da9b60 2023-11-16T16:54:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.160+0000 7f65fc726b80 10 client.26474 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:54:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.160+0000 7f65fc726b80 10 client.26474 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:05.165 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.160+0000 7f65fc726b80 20 client.26474 populate_metadata read hostname 'smithi044' 2023-11-16T16:54:05.165 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.160+0000 7f65fc726b80 1 --2- 172.21.15.44:0/4245109823 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d943e18180 0x55d943e1a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:05.165 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.160+0000 7f65fc726b80 4 client.26474 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:54:05.165 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.160+0000 7f65fc726b80 1 -- 172.21.15.44:0/4245109823 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d943cc9660 con 0x55d943e18180 2023-11-16T16:54:05.165 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.160+0000 7f65dbfff700 1 --2- 172.21.15.44:0/4245109823 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d943e18180 0x55d943e1a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:05.165 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.161+0000 7f65dbfff700 1 --2- 172.21.15.44:0/4245109823 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d943e18180 0x55d943e1a590 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2126bbe267579aa1 server_cookie=d23b359e91910373 in_seq=0 out_seq=0 2023-11-16T16:54:05.165 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.161+0000 7f65c9ffb700 10 client.26474 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:05.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.162+0000 7f65c9ffb700 1 -- 172.21.15.44:0/4245109823 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55d943cc9660 con 0x55d943e18180 2023-11-16T16:54:05.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.162+0000 7f65c9ffb700 10 client.26474 handle_command_reply: tid=0 2023-11-16T16:54:05.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.162+0000 7f65fc726b80 1 client.26474 shutdown 2023-11-16T16:54:05.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.163+0000 7f65fc726b80 1 -- 172.21.15.44:0/4245109823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 msgr2=0x55d943db8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:05.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.163+0000 7f65fc726b80 1 --2- 172.21.15.44:0/4245109823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943db8840 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f65d4005760 tx=0x7f65d4005790 comp rx=0 tx=0).stop 2023-11-16T16:54:05.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.163+0000 7f65fc726b80 1 -- 172.21.15.44:0/4245109823 shutdown_connections 2023-11-16T16:54:05.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.163+0000 7f65fc726b80 1 --2- 172.21.15.44:0/4245109823 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d943e18180 0x55d943e1a590 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.163+0000 7f65fc726b80 1 --2- 172.21.15.44:0/4245109823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d943dae8b0 0x55d943e105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.163+0000 7f65fc726b80 1 --2- 172.21.15.44:0/4245109823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d943da9b60 0x55d943db8840 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.163+0000 7f65fc726b80 1 --2- 172.21.15.44:0/4245109823 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d943daa470 0x55d943db8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.163+0000 7f65c9ffb700 0 client.26474 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:05.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.163+0000 7f65fc726b80 1 -- 172.21.15.44:0/4245109823 >> 172.21.15.44:0/4245109823 conn(0x55d943d9f360 msgr2=0x55d943dae180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:05.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.163+0000 7f65fc726b80 1 -- 172.21.15.44:0/4245109823 shutdown_connections 2023-11-16T16:54:05.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.163+0000 7f65fc726b80 1 -- 172.21.15.44:0/4245109823 wait complete. 2023-11-16T16:54:05.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.163+0000 7f65fc726b80 20 client.26474 trim_cache size 0 max 16384 2023-11-16T16:54:05.167 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:54:05.168 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "scrub active (3487 inodes in the stack)", 2023-11-16T16:54:05.168 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2023-11-16T16:54:05.168 INFO:teuthology.orchestra.run.smithi044.stdout: "b7bb16f1-c54b-4b04-97f3-2de994ff79eb": { 2023-11-16T16:54:05.168 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2023-11-16T16:54:05.168 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "b7bb16f1-c54b-4b04-97f3-2de994ff79eb", 2023-11-16T16:54:05.168 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2023-11-16T16:54:05.168 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:54:05.168 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:54:05.168 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:54:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.164+0000 7f65e8dff700 1 -- 172.21.15.44:0/1231094778 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f65d00a1340 msgr2=0x7f65d00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.164+0000 7f65e8dff700 1 --2- 172.21.15.44:0/1231094778 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f65d00a1340 0x7f65d00a37f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f65d40036c0 tx=0x7f65d4004c10 comp rx=0 tx=0).stop 2023-11-16T16:54:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.164+0000 7f65e8dff700 1 -- 172.21.15.44:0/1231094778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65e4101660 msgr2=0x7f65e4198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.164+0000 7f65e8dff700 1 --2- 172.21.15.44:0/1231094778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65e4101660 0x7f65e4198bf0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00e910 tx=0x7f65cc00ec20 comp rx=0 tx=0).stop 2023-11-16T16:54:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.164+0000 7f65e8dff700 1 -- 172.21.15.44:0/1231094778 shutdown_connections 2023-11-16T16:54:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.164+0000 7f65e8dff700 1 --2- 172.21.15.44:0/1231094778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e4105620 0x7f65e419bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.164+0000 7f65e8dff700 1 --2- 172.21.15.44:0/1231094778 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f65d00a1340 0x7f65d00a37f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.164+0000 7f65e8dff700 1 --2- 172.21.15.44:0/1231094778 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65e4100cc0 0x7f65e41986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.164+0000 7f65e8dff700 1 --2- 172.21.15.44:0/1231094778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65e4101660 0x7f65e4198bf0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.164+0000 7f65e8dff700 1 -- 172.21.15.44:0/1231094778 >> 172.21.15.44:0/1231094778 conn(0x7f65e4077560 msgr2=0x7f65e40feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:05.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.164+0000 7f65e8dff700 1 -- 172.21.15.44:0/1231094778 shutdown_connections 2023-11-16T16:54:05.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:05.164+0000 7f65e8dff700 1 -- 172.21.15.44:0/1231094778 wait complete. 2023-11-16T16:54:05.184 INFO:tasks.cephfs.filesystem:scrub status for tag:b7bb16f1-c54b-4b04-97f3-2de994ff79eb - {'path': '/', 'tag': 'b7bb16f1-c54b-4b04-97f3-2de994ff79eb', 'options': 'recursive,force'} 2023-11-16T16:54:05.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:05 smithi044 ceph-mon[128750]: pgmap v127: 129 pgs: 129 active+clean; 41 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 1.99k op/s 2023-11-16T16:54:05.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:05 smithi176 ceph-mon[131630]: pgmap v127: 129 pgs: 129 active+clean; 41 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 1.99k op/s 2023-11-16T16:54:05.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:05 smithi077 ceph-mon[131204]: pgmap v127: 129 pgs: 129 active+clean; 41 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 1.99k op/s 2023-11-16T16:54:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.261+0000 7fe7caf9d700 1 -- 172.21.15.44:0/590069788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c40737a0 msgr2=0x7fe7c4110b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.261+0000 7fe7caf9d700 1 --2- 172.21.15.44:0/590069788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c40737a0 0x7fe7c4110b40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4009990 tx=0x7fe7b4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.261+0000 7fe7caf9d700 1 -- 172.21.15.44:0/590069788 shutdown_connections 2023-11-16T16:54:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.261+0000 7fe7caf9d700 1 --2- 172.21.15.44:0/590069788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7c4111080 0x7fe7c4113460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.261+0000 7fe7caf9d700 1 --2- 172.21.15.44:0/590069788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7c4072e90 0x7fe7c4073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.261+0000 7fe7caf9d700 1 --2- 172.21.15.44:0/590069788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c40737a0 0x7fe7c4110b40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.261+0000 7fe7caf9d700 1 -- 172.21.15.44:0/590069788 >> 172.21.15.44:0/590069788 conn(0x7fe7c40fbb90 msgr2=0x7fe7c40fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.261+0000 7fe7caf9d700 1 -- 172.21.15.44:0/590069788 shutdown_connections 2023-11-16T16:54:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.261+0000 7fe7caf9d700 1 -- 172.21.15.44:0/590069788 wait complete. 2023-11-16T16:54:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.262+0000 7fe7caf9d700 1 Processor -- start 2023-11-16T16:54:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.262+0000 7fe7caf9d700 1 -- start start 2023-11-16T16:54:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7caf9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7c4072e90 0x7fe7c41a2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7caf9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c40737a0 0x7fe7c41a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7caf9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7c4111080 0x7fe7c419c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7caf9d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c419cd40 con 0x7fe7c4072e90 2023-11-16T16:54:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7caf9d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c419ceb0 con 0x7fe7c40737a0 2023-11-16T16:54:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7caf9d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c419d070 con 0x7fe7c4111080 2023-11-16T16:54:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7c9f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7c4072e90 0x7fe7c41a2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7c979a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c40737a0 0x7fe7c41a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7c979a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c40737a0 0x7fe7c41a2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59882/0 (socket says 172.21.15.44:59882) 2023-11-16T16:54:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7c979a700 1 -- 172.21.15.44:0/2126132718 learned_addr learned my addr 172.21.15.44:0/2126132718 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7c9f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7c4072e90 0x7fe7c41a2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52748/0 (socket says 172.21.15.44:52748) 2023-11-16T16:54:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7ca79c700 1 --2- 172.21.15.44:0/2126132718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7c4111080 0x7fe7c419c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7c979a700 1 -- 172.21.15.44:0/2126132718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7c4111080 msgr2=0x7fe7c419c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7c979a700 1 --2- 172.21.15.44:0/2126132718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7c4111080 0x7fe7c419c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7c979a700 1 -- 172.21.15.44:0/2126132718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7c4072e90 msgr2=0x7fe7c41a2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7c979a700 1 --2- 172.21.15.44:0/2126132718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7c4072e90 0x7fe7c41a2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7c979a700 1 -- 172.21.15.44:0/2126132718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b40095f0 con 0x7fe7c40737a0 2023-11-16T16:54:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.263+0000 7fe7c9f9b700 1 --2- 172.21.15.44:0/2126132718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7c4072e90 0x7fe7c41a2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.264+0000 7fe7c979a700 1 --2- 172.21.15.44:0/2126132718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c40737a0 0x7fe7c41a2ba0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4005910 tx=0x7fe7b4004710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.264+0000 7fe7baffd700 1 -- 172.21.15.44:0/2126132718 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b401d070 con 0x7fe7c40737a0 2023-11-16T16:54:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.264+0000 7fe7caf9d700 1 -- 172.21.15.44:0/2126132718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c4116720 con 0x7fe7c40737a0 2023-11-16T16:54:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.264+0000 7fe7baffd700 1 -- 172.21.15.44:0/2126132718 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7b4022470 con 0x7fe7c40737a0 2023-11-16T16:54:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.264+0000 7fe7baffd700 1 -- 172.21.15.44:0/2126132718 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4017650 con 0x7fe7c40737a0 2023-11-16T16:54:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.264+0000 7fe7caf9d700 1 -- 172.21.15.44:0/2126132718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c4116590 con 0x7fe7c40737a0 2023-11-16T16:54:07.269 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.268+0000 7fe7baffd700 1 -- 172.21.15.44:0/2126132718 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fe7b40225e0 con 0x7fe7c40737a0 2023-11-16T16:54:07.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.268+0000 7fe7dfbd8b80 1 Processor -- start 2023-11-16T16:54:07.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.269+0000 7fe7dfbd8b80 1 -- start start 2023-11-16T16:54:07.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.269+0000 7fe7dfbd8b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab2546b0 0x55d0ab3fe450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.269+0000 7fe7dfbd8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab3fe990 0x55d0ab3f87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.269+0000 7fe7dfbd8b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab3f9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.269+0000 7fe7dfbd8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0ab2fe000 con 0x55d0ab3fe990 2023-11-16T16:54:07.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.269+0000 7fe7baffd700 1 --2- 172.21.15.44:0/2126132718 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe7b00a12c0 0x7fe7b00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.269+0000 7fe7dfbd8b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0ab350840 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.269+0000 7fe7dfbd8b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0ab349cd0 con 0x55d0ab2546b0 2023-11-16T16:54:07.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.269+0000 7fe7c9f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab3fe990 0x55d0ab3f87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.269+0000 7fe7baffd700 1 -- 172.21.15.44:0/2126132718 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fe7b40ce310 con 0x7fe7c40737a0 2023-11-16T16:54:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.269+0000 7fe7ca79c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab2546b0 0x55d0ab3fe450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.269+0000 7fe7c9f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab3fe990 0x55d0ab3f87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52754/0 (socket says 172.21.15.44:52754) 2023-11-16T16:54:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.269+0000 7fe7c9f9b700 1 -- 172.21.15.44:0/1962675053 learned_addr learned my addr 172.21.15.44:0/1962675053 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.269+0000 7fe7c979a700 1 --2- 172.21.15.44:0/1962675053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab3f9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.270+0000 7fe7ca79c700 1 --2- 172.21.15.44:0/2126132718 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe7b00a12c0 0x7fe7b00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.270+0000 7fe7c979a700 1 -- 172.21.15.44:0/1962675053 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab2546b0 msgr2=0x55d0ab3fe450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.270+0000 7fe7c979a700 1 --2- 172.21.15.44:0/1962675053 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab2546b0 0x55d0ab3fe450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.270+0000 7fe7c979a700 1 -- 172.21.15.44:0/1962675053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab3fe990 msgr2=0x55d0ab3f87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.270+0000 7fe7c979a700 1 --2- 172.21.15.44:0/1962675053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab3fe990 0x55d0ab3f87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.270+0000 7fe7c979a700 1 -- 172.21.15.44:0/1962675053 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0ab1cb5d0 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.270+0000 7fe7c9f9b700 1 --2- 172.21.15.44:0/1962675053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab3fe990 0x55d0ab3f87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.270+0000 7fe7c979a700 1 --2- 172.21.15.44:0/1962675053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab3f9120 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe7b40cb680 tx=0x7fe7b409aee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23aa3ef56d6e39b3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.270+0000 7fe7ca79c700 1 --2- 172.21.15.44:0/2126132718 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe7b00a12c0 0x7fe7b00a3770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc009550 tx=0x7fe7bc00b040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.270+0000 7fe7affff700 1 -- 172.21.15.44:0/1962675053 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b409b090 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.270+0000 7fe7affff700 1 -- 172.21.15.44:0/1962675053 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7b409f0c0 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.270+0000 7fe7affff700 1 -- 172.21.15.44:0/1962675053 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b40c4d70 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.271+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/1962675053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 msgr2=0x55d0ab3f9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.271+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/1962675053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab3f9120 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe7b40cb680 tx=0x7fe7b409aee0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.271+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/1962675053 shutdown_connections 2023-11-16T16:54:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.271+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/1962675053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab3f9120 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.271+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/1962675053 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab2546b0 0x55d0ab3fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.271+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/1962675053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab3fe990 0x55d0ab3f87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.271+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/1962675053 >> 172.21.15.44:0/1962675053 conn(0x55d0ab3ee1a0 msgr2=0x55d0ab3f33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.271+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/1962675053 shutdown_connections 2023-11-16T16:54:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.271+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/1962675053 wait complete. 2023-11-16T16:54:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.272+0000 7fe7dfbd8b80 1 -- start start 2023-11-16T16:54:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.272+0000 7fe7dfbd8b80 1 Processor -- start 2023-11-16T16:54:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7dfbd8b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab2546b0 0x55d0ab4078f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7dfbd8b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab407e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7dfbd8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab3fe990 0x55d0ab401b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7dfbd8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0ab2eb880 con 0x55d0ab3fe990 2023-11-16T16:54:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7dfbd8b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0ab2eb520 con 0x55d0ab2546b0 2023-11-16T16:54:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7dfbd8b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0ab1cd8c0 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7dfbd8b80 10 client.0 _ll_register_callbacks cb 0x55d0ab309720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7c9f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab2546b0 0x55d0ab4078f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7c9f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab2546b0 0x55d0ab4078f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59908/0 (socket says 172.21.15.44:59908) 2023-11-16T16:54:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7c9f9b700 1 -- 172.21.15.44:0/2641760717 learned_addr learned my addr 172.21.15.44:0/2641760717 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7ca79c700 1 --2- 172.21.15.44:0/2641760717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab3fe990 0x55d0ab401b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7c979a700 1 --2- 172.21.15.44:0/2641760717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab407e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7c9f9b700 1 -- 172.21.15.44:0/2641760717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3f8cf0 msgr2=0x55d0ab407e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7c9f9b700 1 --2- 172.21.15.44:0/2641760717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab407e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7c9f9b700 1 -- 172.21.15.44:0/2641760717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab3fe990 msgr2=0x55d0ab401b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7c9f9b700 1 --2- 172.21.15.44:0/2641760717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab3fe990 0x55d0ab401b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7c9f9b700 1 -- 172.21.15.44:0/2641760717 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c4116590 con 0x55d0ab2546b0 2023-11-16T16:54:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.273+0000 7fe7c979a700 1 --2- 172.21.15.44:0/2641760717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab407e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:54:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.274+0000 7fe7c9f9b700 1 --2- 172.21.15.44:0/2641760717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab2546b0 0x55d0ab4078f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0011d10 tx=0x7fe7c000f6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57ac174ee7ad8a5d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.274+0000 7fe7ad7fa700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:54:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.274+0000 7fe7ad7fa700 1 -- 172.21.15.44:0/2641760717 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c0016070 con 0x55d0ab2546b0 2023-11-16T16:54:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.274+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/2641760717 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d0ab3eb960 con 0x55d0ab2546b0 2023-11-16T16:54:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.274+0000 7fe7ad7fa700 1 -- 172.21.15.44:0/2641760717 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fe7c000fe40 con 0x55d0ab2546b0 2023-11-16T16:54:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.274+0000 7fe7ad7fa700 1 -- 172.21.15.44:0/2641760717 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7c001a3f0 con 0x55d0ab2546b0 2023-11-16T16:54:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.274+0000 7fe7dfbd8b80 10 client.26486 fetch_fsmap learned FSMap version 21 2023-11-16T16:54:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.274+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/2641760717 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55d0ab310130 con 0x55d0ab2546b0 2023-11-16T16:54:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.274+0000 7fe7ad7fa700 1 -- 172.21.15.44:0/2641760717 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d0ab3eb960 con 0x55d0ab2546b0 2023-11-16T16:54:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.274+0000 7fe7ad7fa700 1 -- 172.21.15.44:0/2641760717 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe7c00195d0 con 0x55d0ab2546b0 2023-11-16T16:54:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.275+0000 7fe7dfbd8b80 10 client.26486 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:54:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.275+0000 7fe7dfbd8b80 10 client.26486 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.275+0000 7fe7dfbd8b80 20 client.26486 populate_metadata read hostname 'smithi044' 2023-11-16T16:54:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.275+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/2641760717 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d0ab411e60 0x55d0ab414270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.275+0000 7fe7dfbd8b80 4 client.26486 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:54:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.275+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/2641760717 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d0ab30c040 con 0x55d0ab411e60 2023-11-16T16:54:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.275+0000 7fe7c979a700 1 --2- 172.21.15.44:0/2641760717 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d0ab411e60 0x55d0ab414270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.275+0000 7fe7ad7fa700 10 client.26486 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.275+0000 7fe7c979a700 1 --2- 172.21.15.44:0/2641760717 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d0ab411e60 0x55d0ab414270 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa69834660b0d8e0 server_cookie=b7b5ba1218c1c8d5 in_seq=0 out_seq=0 2023-11-16T16:54:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.277+0000 7fe7ad7fa700 1 -- 172.21.15.44:0/2641760717 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d0ab30c040 con 0x55d0ab411e60 2023-11-16T16:54:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.277+0000 7fe7ad7fa700 10 client.26486 handle_command_reply: tid=0 2023-11-16T16:54:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.277+0000 7fe7dfbd8b80 1 client.26486 shutdown 2023-11-16T16:54:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.277+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/2641760717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab2546b0 msgr2=0x55d0ab4078f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.277+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/2641760717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab2546b0 0x55d0ab4078f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0011d10 tx=0x7fe7c000f6a0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.278+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/2641760717 shutdown_connections 2023-11-16T16:54:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.278+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/2641760717 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d0ab411e60 0x55d0ab414270 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.278+0000 7fe7ad7fa700 0 client.26486 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.278+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/2641760717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab3fe990 0x55d0ab401b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.278+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/2641760717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab2546b0 0x55d0ab4078f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.278+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/2641760717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab407e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.278+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/2641760717 >> 172.21.15.44:0/2641760717 conn(0x55d0ab3ee1a0 msgr2=0x55d0ab3f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:07.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.278+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/2641760717 shutdown_connections 2023-11-16T16:54:07.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.278+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/2641760717 wait complete. 2023-11-16T16:54:07.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.278+0000 7fe7dfbd8b80 20 client.26486 trim_cache size 0 max 16384 2023-11-16T16:54:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.294+0000 7fe7dfbd8b80 1 Processor -- start 2023-11-16T16:54:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.294+0000 7fe7dfbd8b80 1 -- start start 2023-11-16T16:54:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7dfbd8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab2546b0 0x55d0ab3f87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7dfbd8b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab3ff7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7dfbd8b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3fe990 0x55d0ab4024b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7dfbd8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0ab400120 con 0x55d0ab2546b0 2023-11-16T16:54:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7dfbd8b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0ab3fc870 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7dfbd8b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0ab3f40a0 con 0x55d0ab3fe990 2023-11-16T16:54:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7c979a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab3ff7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7c979a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab3ff7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59920/0 (socket says 172.21.15.44:59920) 2023-11-16T16:54:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7c979a700 1 -- 172.21.15.44:0/796095224 learned_addr learned my addr 172.21.15.44:0/796095224 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7c9f9b700 1 --2- 172.21.15.44:0/796095224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab2546b0 0x55d0ab3f87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7ca79c700 1 --2- 172.21.15.44:0/796095224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3fe990 0x55d0ab4024b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7c979a700 1 -- 172.21.15.44:0/796095224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3fe990 msgr2=0x55d0ab4024b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7c979a700 1 --2- 172.21.15.44:0/796095224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3fe990 0x55d0ab4024b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7c979a700 1 -- 172.21.15.44:0/796095224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab2546b0 msgr2=0x55d0ab3f87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7c979a700 1 --2- 172.21.15.44:0/796095224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab2546b0 0x55d0ab3f87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7c979a700 1 -- 172.21.15.44:0/796095224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0ab30fe80 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7c9f9b700 1 --2- 172.21.15.44:0/796095224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab2546b0 0x55d0ab3f87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:54:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7ca79c700 1 --2- 172.21.15.44:0/796095224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3fe990 0x55d0ab4024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:54:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.295+0000 7fe7c979a700 1 --2- 172.21.15.44:0/796095224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab3ff7f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4022c80 tx=0x7fe7b4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6d9c9078cd65f8a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.296+0000 7fe7b8ff9700 1 -- 172.21.15.44:0/796095224 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b40ce040 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.296+0000 7fe7b8ff9700 1 -- 172.21.15.44:0/796095224 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7b4022900 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.296+0000 7fe7b8ff9700 1 -- 172.21.15.44:0/796095224 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b409a7e0 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.296+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/796095224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 msgr2=0x55d0ab3ff7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.296+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/796095224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab3ff7f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4022c80 tx=0x7fe7b4000bc0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.296+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/796095224 shutdown_connections 2023-11-16T16:54:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.296+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/796095224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3fe990 0x55d0ab4024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.296+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/796095224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab2546b0 0x55d0ab3f87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.296+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/796095224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab3ff7f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.296+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/796095224 >> 172.21.15.44:0/796095224 conn(0x55d0ab3ee1a0 msgr2=0x55d0ab3f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.296+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/796095224 shutdown_connections 2023-11-16T16:54:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.296+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/796095224 wait complete. 2023-11-16T16:54:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.297+0000 7fe7dfbd8b80 1 -- start start 2023-11-16T16:54:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.297+0000 7fe7dfbd8b80 1 Processor -- start 2023-11-16T16:54:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.297+0000 7fe7dfbd8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab2546b0 0x55d0ab407e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.297+0000 7fe7dfbd8b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab4054a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.297+0000 7fe7dfbd8b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3fe990 0x55d0ab405be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.297+0000 7fe7dfbd8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0ab4062c0 con 0x55d0ab2546b0 2023-11-16T16:54:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.297+0000 7fe7dfbd8b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0ab464d20 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.297+0000 7fe7dfbd8b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0ab464e60 con 0x55d0ab3fe990 2023-11-16T16:54:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.297+0000 7fe7dfbd8b80 10 client.0 _ll_register_callbacks cb 0x7fe7c00117e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.298+0000 7fe7c979a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab4054a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.298+0000 7fe7c979a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab4054a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:59930/0 (socket says 172.21.15.44:59930) 2023-11-16T16:54:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.298+0000 7fe7c979a700 1 -- 172.21.15.44:0/919121393 learned_addr learned my addr 172.21.15.44:0/919121393 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.298+0000 7fe7c9f9b700 1 --2- 172.21.15.44:0/919121393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab2546b0 0x55d0ab407e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.298+0000 7fe7ca79c700 1 --2- 172.21.15.44:0/919121393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3fe990 0x55d0ab405be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.298+0000 7fe7c979a700 1 -- 172.21.15.44:0/919121393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3fe990 msgr2=0x55d0ab405be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.298+0000 7fe7c979a700 1 --2- 172.21.15.44:0/919121393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3fe990 0x55d0ab405be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.298+0000 7fe7c979a700 1 -- 172.21.15.44:0/919121393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab2546b0 msgr2=0x55d0ab407e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.298+0000 7fe7c979a700 1 --2- 172.21.15.44:0/919121393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab2546b0 0x55d0ab407e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.298+0000 7fe7c979a700 1 -- 172.21.15.44:0/919121393 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d0ab1cb5d0 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.298+0000 7fe7c979a700 1 --2- 172.21.15.44:0/919121393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab4054a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe7b40993e0 tx=0x7fe7b4009290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12afdc2e6481c5c3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.298+0000 7fe7ad7fa700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:54:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.298+0000 7fe7ad7fa700 1 -- 172.21.15.44:0/919121393 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b40b91b0 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.298+0000 7fe7ad7fa700 1 -- 172.21.15.44:0/919121393 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fe7b40b9310 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.298+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/919121393 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe7b409f0c0 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.299+0000 7fe7ad7fa700 1 -- 172.21.15.44:0/919121393 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7b409a580 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.299+0000 7fe7dfbd8b80 10 client.26492 fetch_fsmap learned FSMap version 21 2023-11-16T16:54:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.299+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/919121393 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55d0ab160c00 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.299+0000 7fe7ad7fa700 1 -- 172.21.15.44:0/919121393 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7b40cb020 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.299+0000 7fe7ad7fa700 1 -- 172.21.15.44:0/919121393 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe7b40a3050 con 0x55d0ab3f8cf0 2023-11-16T16:54:07.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.299+0000 7fe7dfbd8b80 10 client.26492 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:54:07.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.299+0000 7fe7dfbd8b80 10 client.26492 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:07.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.299+0000 7fe7dfbd8b80 20 client.26492 populate_metadata read hostname 'smithi044' 2023-11-16T16:54:07.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.299+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/919121393 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d0ab411e60 0x55d0ab467fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:07.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.299+0000 7fe7dfbd8b80 4 client.26492 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:54:07.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.299+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/919121393 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d0ab318600 con 0x55d0ab411e60 2023-11-16T16:54:07.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.299+0000 7fe7c9f9b700 1 --2- 172.21.15.44:0/919121393 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d0ab411e60 0x55d0ab467fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:07.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.300+0000 7fe7c9f9b700 1 --2- 172.21.15.44:0/919121393 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d0ab411e60 0x55d0ab467fd0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=568a7bbfe15c64de server_cookie=a29b6e160a45c858 in_seq=0 out_seq=0 2023-11-16T16:54:07.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:07.300+0000 7fe7ad7fa700 10 client.26492 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:07 smithi044 ceph-mon[128750]: pgmap v128: 129 pgs: 129 active+clean; 41 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 703 op/s 2023-11-16T16:54:07.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:07 smithi176 ceph-mon[131630]: pgmap v128: 129 pgs: 129 active+clean; 41 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 703 op/s 2023-11-16T16:54:07.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:07 smithi077 ceph-mon[131204]: pgmap v128: 129 pgs: 129 active+clean; 41 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 703 op/s 2023-11-16T16:54:09.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:09 smithi044 ceph-mon[128750]: pgmap v129: 129 pgs: 129 active+clean; 65 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.7 MiB/s wr, 840 op/s 2023-11-16T16:54:09.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:09 smithi176 ceph-mon[131630]: pgmap v129: 129 pgs: 129 active+clean; 65 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.7 MiB/s wr, 840 op/s 2023-11-16T16:54:09.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:09 smithi077 ceph-mon[131204]: pgmap v129: 129 pgs: 129 active+clean; 65 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.7 MiB/s wr, 840 op/s 2023-11-16T16:54:11.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:11 smithi044 ceph-mon[128750]: pgmap v130: 129 pgs: 129 active+clean; 65 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.4 MiB/s wr, 509 op/s 2023-11-16T16:54:11.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:11 smithi176 ceph-mon[131630]: pgmap v130: 129 pgs: 129 active+clean; 65 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.4 MiB/s wr, 509 op/s 2023-11-16T16:54:11.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:11 smithi077 ceph-mon[131204]: pgmap v130: 129 pgs: 129 active+clean; 65 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.4 MiB/s wr, 509 op/s 2023-11-16T16:54:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.717+0000 7fe7ad7fa700 1 -- 172.21.15.44:0/919121393 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d0ab318600 con 0x55d0ab411e60 2023-11-16T16:54:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.717+0000 7fe7ad7fa700 10 client.26492 handle_command_reply: tid=0 2023-11-16T16:54:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.717+0000 7fe7dfbd8b80 1 client.26492 shutdown 2023-11-16T16:54:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.718+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/919121393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 msgr2=0x55d0ab4054a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.718+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/919121393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab4054a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe7b40993e0 tx=0x7fe7b4009290 comp rx=0 tx=0).stop 2023-11-16T16:54:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.718+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/919121393 shutdown_connections 2023-11-16T16:54:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.718+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/919121393 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55d0ab411e60 0x55d0ab467fd0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.718+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/919121393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d0ab3fe990 0x55d0ab405be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.718+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/919121393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d0ab2546b0 0x55d0ab407e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.718+0000 7fe7dfbd8b80 1 --2- 172.21.15.44:0/919121393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0ab3f8cf0 0x55d0ab4054a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.718+0000 7fe7ad7fa700 0 client.26492 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.718+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/919121393 >> 172.21.15.44:0/919121393 conn(0x55d0ab3ee1a0 msgr2=0x55d0ab4024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.718+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/919121393 shutdown_connections 2023-11-16T16:54:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.718+0000 7fe7dfbd8b80 1 -- 172.21.15.44:0/919121393 wait complete. 2023-11-16T16:54:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.719+0000 7fe7dfbd8b80 20 client.26492 trim_cache size 0 max 16384 2023-11-16T16:54:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:54:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:54:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:54:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:54:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.719+0000 7fe7caf9d700 1 -- 172.21.15.44:0/2126132718 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe7b00a12c0 msgr2=0x7fe7b00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.720+0000 7fe7caf9d700 1 --2- 172.21.15.44:0/2126132718 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe7b00a12c0 0x7fe7b00a3770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc009550 tx=0x7fe7bc00b040 comp rx=0 tx=0).stop 2023-11-16T16:54:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.720+0000 7fe7caf9d700 1 -- 172.21.15.44:0/2126132718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c40737a0 msgr2=0x7fe7c41a2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.720+0000 7fe7caf9d700 1 --2- 172.21.15.44:0/2126132718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c40737a0 0x7fe7c41a2ba0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4005910 tx=0x7fe7b4004710 comp rx=0 tx=0).stop 2023-11-16T16:54:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.720+0000 7fe7caf9d700 1 -- 172.21.15.44:0/2126132718 shutdown_connections 2023-11-16T16:54:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.720+0000 7fe7caf9d700 1 --2- 172.21.15.44:0/2126132718 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe7b00a12c0 0x7fe7b00a3770 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.720+0000 7fe7caf9d700 1 --2- 172.21.15.44:0/2126132718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7c4111080 0x7fe7c419c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.720+0000 7fe7caf9d700 1 --2- 172.21.15.44:0/2126132718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7c4072e90 0x7fe7c41a2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.720+0000 7fe7caf9d700 1 --2- 172.21.15.44:0/2126132718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c40737a0 0x7fe7c41a2ba0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.720+0000 7fe7caf9d700 1 -- 172.21.15.44:0/2126132718 >> 172.21.15.44:0/2126132718 conn(0x7fe7c40fbb90 msgr2=0x7fe7c40fc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.720+0000 7fe7caf9d700 1 -- 172.21.15.44:0/2126132718 shutdown_connections 2023-11-16T16:54:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:13.720+0000 7fe7caf9d700 1 -- 172.21.15.44:0/2126132718 wait complete. 2023-11-16T16:54:13.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:13 smithi044 ceph-mon[128750]: pgmap v131: 129 pgs: 129 active+clean; 73 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.6 MiB/s wr, 948 op/s 2023-11-16T16:54:13.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:13 smithi176 ceph-mon[131630]: pgmap v131: 129 pgs: 129 active+clean; 73 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.6 MiB/s wr, 948 op/s 2023-11-16T16:54:13.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:13 smithi077 ceph-mon[131204]: pgmap v131: 129 pgs: 129 active+clean; 73 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.6 MiB/s wr, 948 op/s 2023-11-16T16:54:15.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:15 smithi044 ceph-mon[128750]: pgmap v132: 129 pgs: 129 active+clean; 81 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.5 MiB/s wr, 1.55k op/s 2023-11-16T16:54:15.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:15 smithi176 ceph-mon[131630]: pgmap v132: 129 pgs: 129 active+clean; 81 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.5 MiB/s wr, 1.55k op/s 2023-11-16T16:54:15.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:15 smithi077 ceph-mon[131204]: pgmap v132: 129 pgs: 129 active+clean; 81 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.5 MiB/s wr, 1.55k op/s 2023-11-16T16:54:17.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:17 smithi044 ceph-mon[128750]: pgmap v133: 129 pgs: 129 active+clean; 81 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 1.32k op/s 2023-11-16T16:54:17.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:17 smithi176 ceph-mon[131630]: pgmap v133: 129 pgs: 129 active+clean; 81 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 1.32k op/s 2023-11-16T16:54:17.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:17 smithi077 ceph-mon[131204]: pgmap v133: 129 pgs: 129 active+clean; 81 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 1.32k op/s 2023-11-16T16:54:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:19 smithi044 ceph-mon[128750]: pgmap v134: 129 pgs: 129 active+clean; 22 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.7 MiB/s wr, 2.20k op/s 2023-11-16T16:54:19.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:19 smithi176 ceph-mon[131630]: pgmap v134: 129 pgs: 129 active+clean; 22 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.7 MiB/s wr, 2.20k op/s 2023-11-16T16:54:19.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:19 smithi077 ceph-mon[131204]: pgmap v134: 129 pgs: 129 active+clean; 22 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.7 MiB/s wr, 2.20k op/s 2023-11-16T16:54:20.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:20 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:54:20.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:20 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:54:21.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:20 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:54:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:21 smithi176 ceph-mon[131630]: pgmap v135: 129 pgs: 129 active+clean; 22 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 2.06k op/s 2023-11-16T16:54:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:21 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:54:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:21 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:54:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:21 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:54:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:21 smithi077 ceph-mon[131204]: pgmap v135: 129 pgs: 129 active+clean; 22 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 2.06k op/s 2023-11-16T16:54:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:21 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:54:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:21 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:54:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:21 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:54:22.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:21 smithi044 ceph-mon[128750]: pgmap v135: 129 pgs: 129 active+clean; 22 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 2.06k op/s 2023-11-16T16:54:22.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:21 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:54:22.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:21 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:54:22.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:21 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:54:22.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:22 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:54:22.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:22 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T16:54:22.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:22 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:54:22.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:22 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:54:22.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:22 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:54:22.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:22 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T16:54:22.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:22 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:54:22.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:22 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:54:23.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:22 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:54:23.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:22 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T16:54:23.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:22 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:54:23.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:22 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:54:23.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:23 smithi176 ceph-mon[131630]: pgmap v136: 129 pgs: 129 active+clean; 29 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.6 MiB/s wr, 2.28k op/s 2023-11-16T16:54:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:23 smithi077 ceph-mon[131204]: pgmap v136: 129 pgs: 129 active+clean; 29 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.6 MiB/s wr, 2.28k op/s 2023-11-16T16:54:24.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:23 smithi044 ceph-mon[128750]: pgmap v136: 129 pgs: 129 active+clean; 29 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.6 MiB/s wr, 2.28k op/s 2023-11-16T16:54:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:25 smithi176 ceph-mon[131630]: pgmap v137: 129 pgs: 129 active+clean; 34 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.0 MiB/s wr, 2.16k op/s 2023-11-16T16:54:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:25 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:54:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:25 smithi077 ceph-mon[131204]: pgmap v137: 129 pgs: 129 active+clean; 34 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.0 MiB/s wr, 2.16k op/s 2023-11-16T16:54:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:25 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:54:26.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:25 smithi044 ceph-mon[128750]: pgmap v137: 129 pgs: 129 active+clean; 34 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.0 MiB/s wr, 2.16k op/s 2023-11-16T16:54:26.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:25 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:54:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.888+0000 7f82ed3ff700 1 -- 172.21.15.44:0/3782264381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82e81015f0 msgr2=0x7f82e8104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.888+0000 7f82ed3ff700 1 --2- 172.21.15.44:0/3782264381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82e81015f0 0x7f82e8104e80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009990 tx=0x7f82dc009ca0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.888+0000 7f82ed3ff700 1 -- 172.21.15.44:0/3782264381 shutdown_connections 2023-11-16T16:54:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.888+0000 7f82ed3ff700 1 --2- 172.21.15.44:0/3782264381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f82e8105560 0x7f82e810d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.888+0000 7f82ed3ff700 1 --2- 172.21.15.44:0/3782264381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82e8100c50 0x7f82e8101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.888+0000 7f82ed3ff700 1 --2- 172.21.15.44:0/3782264381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82e81015f0 0x7f82e8104e80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.888+0000 7f82ed3ff700 1 -- 172.21.15.44:0/3782264381 >> 172.21.15.44:0/3782264381 conn(0x7f82e8077550 msgr2=0x7f82e8077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.888+0000 7f82ed3ff700 1 -- 172.21.15.44:0/3782264381 shutdown_connections 2023-11-16T16:54:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.888+0000 7f82ed3ff700 1 -- 172.21.15.44:0/3782264381 wait complete. 2023-11-16T16:54:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.889+0000 7f82ed3ff700 1 Processor -- start 2023-11-16T16:54:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.889+0000 7f82ed3ff700 1 -- start start 2023-11-16T16:54:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.890+0000 7f82ed3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82e8100c50 0x7f82e81985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.890+0000 7f82ed3ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82e81015f0 0x7f82e8198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.890+0000 7f82ed3ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f82e8105560 0x7f82e819cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.890+0000 7f82ed3ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e819d5d0 con 0x7f82e8105560 2023-11-16T16:54:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.890+0000 7f82ed3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e819d740 con 0x7f82e8100c50 2023-11-16T16:54:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.890+0000 7f82ed3ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e819d900 con 0x7f82e81015f0 2023-11-16T16:54:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.890+0000 7f82e6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82e8100c50 0x7f82e81985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.890+0000 7f82e6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82e8100c50 0x7f82e81985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:35130/0 (socket says 172.21.15.44:35130) 2023-11-16T16:54:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.890+0000 7f82e6ffd700 1 -- 172.21.15.44:0/3599718801 learned_addr learned my addr 172.21.15.44:0/3599718801 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.890+0000 7f82e67fc700 1 --2- 172.21.15.44:0/3599718801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82e81015f0 0x7f82e8198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.890+0000 7f82e77fe700 1 --2- 172.21.15.44:0/3599718801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f82e8105560 0x7f82e819cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.891+0000 7f82e6ffd700 1 -- 172.21.15.44:0/3599718801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82e81015f0 msgr2=0x7f82e8198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.891+0000 7f82e6ffd700 1 --2- 172.21.15.44:0/3599718801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82e81015f0 0x7f82e8198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.891+0000 7f82e6ffd700 1 -- 172.21.15.44:0/3599718801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f82e8105560 msgr2=0x7f82e819cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.891+0000 7f82e6ffd700 1 --2- 172.21.15.44:0/3599718801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f82e8105560 0x7f82e819cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.891+0000 7f82e6ffd700 1 -- 172.21.15.44:0/3599718801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc0095f0 con 0x7f82e8100c50 2023-11-16T16:54:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.891+0000 7f82e6ffd700 1 --2- 172.21.15.44:0/3599718801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82e8100c50 0x7f82e81985f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f82d800e9a0 tx=0x7f82d800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.891+0000 7f82cffff700 1 -- 172.21.15.44:0/3599718801 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d8009790 con 0x7f82e8100c50 2023-11-16T16:54:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.891+0000 7f82ed3ff700 1 -- 172.21.15.44:0/3599718801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e8110370 con 0x7f82e8100c50 2023-11-16T16:54:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.891+0000 7f82cffff700 1 -- 172.21.15.44:0/3599718801 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82d800cbd0 con 0x7f82e8100c50 2023-11-16T16:54:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.891+0000 7f82cffff700 1 -- 172.21.15.44:0/3599718801 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d8018680 con 0x7f82e8100c50 2023-11-16T16:54:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.891+0000 7f82ed3ff700 1 -- 172.21.15.44:0/3599718801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e81101e0 con 0x7f82e8100c50 2023-11-16T16:54:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.895+0000 7f82cffff700 1 -- 172.21.15.44:0/3599718801 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f82d800cd40 con 0x7f82e8100c50 2023-11-16T16:54:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.896+0000 7f8300defb80 1 Processor -- start 2023-11-16T16:54:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.896+0000 7f8300defb80 1 -- start start 2023-11-16T16:54:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.897+0000 7f8300defb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1a716b0 0x558fe1c1b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.897+0000 7f8300defb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1b720 0x558fe1c15560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.897+0000 7f8300defb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1c1bb90 0x558fe1c15b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.897+0000 7f8300defb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe1b1b000 con 0x558fe1a716b0 2023-11-16T16:54:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.897+0000 7f8300defb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe1b6d840 con 0x558fe1c1b720 2023-11-16T16:54:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.897+0000 7f8300defb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe1b66cd0 con 0x558fe1c1bb90 2023-11-16T16:54:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.897+0000 7f82e6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1b720 0x558fe1c15560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.897+0000 7f82e77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1a716b0 0x558fe1c1b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.897+0000 7f82e6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1b720 0x558fe1c15560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:35132/0 (socket says 172.21.15.44:35132) 2023-11-16T16:54:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.897+0000 7f82e6ffd700 1 -- 172.21.15.44:0/2568820693 learned_addr learned my addr 172.21.15.44:0/2568820693 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.898+0000 7f82e67fc700 1 --2- 172.21.15.44:0/2568820693 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1c1bb90 0x558fe1c15b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.898+0000 7f82cffff700 1 --2- 172.21.15.44:0/3599718801 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f82d00a12c0 0x7f82d00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.898+0000 7f82e6ffd700 1 -- 172.21.15.44:0/2568820693 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1c1bb90 msgr2=0x558fe1c15b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.898+0000 7f82cffff700 1 -- 172.21.15.44:0/3599718801 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f82d8014070 con 0x7f82e8100c50 2023-11-16T16:54:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.898+0000 7f82e6ffd700 1 --2- 172.21.15.44:0/2568820693 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1c1bb90 0x558fe1c15b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.898+0000 7f82e6ffd700 1 -- 172.21.15.44:0/2568820693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1a716b0 msgr2=0x558fe1c1b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.898+0000 7f82e6ffd700 1 --2- 172.21.15.44:0/2568820693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1a716b0 0x558fe1c1b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.898+0000 7f82e6ffd700 1 -- 172.21.15.44:0/2568820693 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fe19e85d0 con 0x558fe1c1b720 2023-11-16T16:54:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.898+0000 7f82e67fc700 1 --2- 172.21.15.44:0/2568820693 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1c1bb90 0x558fe1c15b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:54:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.898+0000 7f82e77fe700 1 --2- 172.21.15.44:0/2568820693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1a716b0 0x558fe1c1b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.898+0000 7f82e6ffd700 1 --2- 172.21.15.44:0/2568820693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1b720 0x558fe1c15560 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f82d80ca600 tx=0x7f82d801f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cad2b4aa736a638 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.899+0000 7f82e77fe700 1 --2- 172.21.15.44:0/3599718801 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f82d00a12c0 0x7f82d00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.899+0000 7f82cd7fa700 1 -- 172.21.15.44:0/2568820693 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d80cd170 con 0x558fe1c1b720 2023-11-16T16:54:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.899+0000 7f82cd7fa700 1 -- 172.21.15.44:0/2568820693 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82d8099e10 con 0x558fe1c1b720 2023-11-16T16:54:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.899+0000 7f82cd7fa700 1 -- 172.21.15.44:0/2568820693 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d809a4c0 con 0x558fe1c1b720 2023-11-16T16:54:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.899+0000 7f82e77fe700 1 --2- 172.21.15.44:0/3599718801 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f82d00a12c0 0x7f82d00a3770 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009fd0 tx=0x7f82e0004ad0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.899+0000 7f8300defb80 1 -- 172.21.15.44:0/2568820693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1b720 msgr2=0x558fe1c15560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.899+0000 7f8300defb80 1 --2- 172.21.15.44:0/2568820693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1b720 0x558fe1c15560 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f82d80ca600 tx=0x7f82d801f000 comp rx=0 tx=0).stop 2023-11-16T16:54:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.900+0000 7f8300defb80 1 -- 172.21.15.44:0/2568820693 shutdown_connections 2023-11-16T16:54:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.900+0000 7f8300defb80 1 --2- 172.21.15.44:0/2568820693 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1c1bb90 0x558fe1c15b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.900+0000 7f8300defb80 1 --2- 172.21.15.44:0/2568820693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1a716b0 0x558fe1c1b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.900+0000 7f8300defb80 1 --2- 172.21.15.44:0/2568820693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1b720 0x558fe1c15560 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.900+0000 7f8300defb80 1 -- 172.21.15.44:0/2568820693 >> 172.21.15.44:0/2568820693 conn(0x558fe1c0af50 msgr2=0x558fe1c10270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.900+0000 7f8300defb80 1 -- 172.21.15.44:0/2568820693 shutdown_connections 2023-11-16T16:54:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.900+0000 7f8300defb80 1 -- 172.21.15.44:0/2568820693 wait complete. 2023-11-16T16:54:26.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.901+0000 7f8300defb80 1 -- start start 2023-11-16T16:54:26.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.902+0000 7f8300defb80 1 Processor -- start 2023-11-16T16:54:26.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.903+0000 7f8300defb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c24a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.903+0000 7f8300defb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1b720 0x558fe1c24f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.903+0000 7f8300defb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1c1bb90 0x558fe1c1ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.903+0000 7f8300defb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe1b08880 con 0x558fe1c1bb90 2023-11-16T16:54:26.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.903+0000 7f8300defb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe1b08520 con 0x558fe1c1b720 2023-11-16T16:54:26.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.903+0000 7f8300defb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe19ea8c0 con 0x558fe1a716b0 2023-11-16T16:54:26.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.903+0000 7f8300defb80 10 client.0 _ll_register_callbacks cb 0x558fe1b26720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:26.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.903+0000 7f82e6ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c24a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.903+0000 7f82e6ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c24a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:36418/0 (socket says 172.21.15.44:36418) 2023-11-16T16:54:26.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.903+0000 7f82e6ffd700 1 -- 172.21.15.44:0/1754620902 learned_addr learned my addr 172.21.15.44:0/1754620902 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:26.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.903+0000 7f82e67fc700 1 --2- 172.21.15.44:0/1754620902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1b720 0x558fe1c24f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.903+0000 7f82e77fe700 1 --2- 172.21.15.44:0/1754620902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1c1bb90 0x558fe1c1ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.904+0000 7f82e6ffd700 1 -- 172.21.15.44:0/1754620902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1b720 msgr2=0x558fe1c24f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.904+0000 7f82e6ffd700 1 --2- 172.21.15.44:0/1754620902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1b720 0x558fe1c24f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.904+0000 7f82e6ffd700 1 -- 172.21.15.44:0/1754620902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1c1bb90 msgr2=0x558fe1c1ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.904+0000 7f82e6ffd700 1 --2- 172.21.15.44:0/1754620902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1c1bb90 0x558fe1c1ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.904+0000 7f82e6ffd700 1 -- 172.21.15.44:0/1754620902 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e81101e0 con 0x558fe1a716b0 2023-11-16T16:54:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.904+0000 7f82e6ffd700 1 --2- 172.21.15.44:0/1754620902 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c24a50 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f82d8010bc0 tx=0x7f82d809a550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6838d0613339e23 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.904+0000 7f82caff5700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:54:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.904+0000 7f82caff5700 1 -- 172.21.15.44:0/1754620902 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d80cd170 con 0x558fe1a716b0 2023-11-16T16:54:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.904+0000 7f82caff5700 1 -- 172.21.15.44:0/1754620902 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f82d80a2490 con 0x558fe1a716b0 2023-11-16T16:54:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.904+0000 7f8300defb80 1 -- 172.21.15.44:0/1754620902 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558fe1c08710 con 0x558fe1a716b0 2023-11-16T16:54:26.908 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.905+0000 7f82caff5700 1 -- 172.21.15.44:0/1754620902 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82d80999a0 con 0x558fe1a716b0 2023-11-16T16:54:26.908 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.905+0000 7f8300defb80 10 client.26494 fetch_fsmap learned FSMap version 21 2023-11-16T16:54:26.908 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.905+0000 7f8300defb80 1 -- 172.21.15.44:0/1754620902 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x558fe1b2d130 con 0x558fe1a716b0 2023-11-16T16:54:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.905+0000 7f82caff5700 1 -- 172.21.15.44:0/1754620902 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558fe1c08710 con 0x558fe1a716b0 2023-11-16T16:54:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.906+0000 7f82caff5700 1 -- 172.21.15.44:0/1754620902 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f82d8099160 con 0x558fe1a716b0 2023-11-16T16:54:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.906+0000 7f8300defb80 10 client.26494 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:54:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.906+0000 7f8300defb80 10 client.26494 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.906+0000 7f8300defb80 20 client.26494 populate_metadata read hostname 'smithi044' 2023-11-16T16:54:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.906+0000 7f8300defb80 1 --2- 172.21.15.44:0/1754620902 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558fe1c2e980 0x558fe1c30d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.906+0000 7f8300defb80 4 client.26494 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:54:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.906+0000 7f8300defb80 1 -- 172.21.15.44:0/1754620902 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558fe1b29040 con 0x558fe1c2e980 2023-11-16T16:54:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.906+0000 7f82e67fc700 1 --2- 172.21.15.44:0/1754620902 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558fe1c2e980 0x558fe1c30d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.906+0000 7f82e67fc700 1 --2- 172.21.15.44:0/1754620902 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558fe1c2e980 0x558fe1c30d60 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=956c128b7ebd23a3 server_cookie=332db4692e889925 in_seq=0 out_seq=0 2023-11-16T16:54:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.906+0000 7f82caff5700 10 client.26494 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:26.912 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.910+0000 7f82caff5700 1 -- 172.21.15.44:0/1754620902 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x558fe1b29040 con 0x558fe1c2e980 2023-11-16T16:54:26.912 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.910+0000 7f82caff5700 10 client.26494 handle_command_reply: tid=0 2023-11-16T16:54:26.912 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.910+0000 7f8300defb80 1 client.26494 shutdown 2023-11-16T16:54:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.911+0000 7f8300defb80 1 -- 172.21.15.44:0/1754620902 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 msgr2=0x558fe1c24a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.911+0000 7f8300defb80 1 --2- 172.21.15.44:0/1754620902 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c24a50 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f82d8010bc0 tx=0x7f82d809a550 comp rx=0 tx=0).stop 2023-11-16T16:54:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.911+0000 7f8300defb80 1 -- 172.21.15.44:0/1754620902 shutdown_connections 2023-11-16T16:54:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.911+0000 7f8300defb80 1 --2- 172.21.15.44:0/1754620902 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558fe1c2e980 0x558fe1c30d60 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.911+0000 7f8300defb80 1 --2- 172.21.15.44:0/1754620902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1c1bb90 0x558fe1c1ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.911+0000 7f8300defb80 1 --2- 172.21.15.44:0/1754620902 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c24a50 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.911+0000 7f8300defb80 1 --2- 172.21.15.44:0/1754620902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1b720 0x558fe1c24f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.911+0000 7f8300defb80 1 -- 172.21.15.44:0/1754620902 >> 172.21.15.44:0/1754620902 conn(0x558fe1c0af50 msgr2=0x558fe1c0f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.911+0000 7f82caff5700 0 client.26494 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.911+0000 7f8300defb80 1 -- 172.21.15.44:0/1754620902 shutdown_connections 2023-11-16T16:54:26.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.911+0000 7f8300defb80 1 -- 172.21.15.44:0/1754620902 wait complete. 2023-11-16T16:54:26.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.911+0000 7f8300defb80 20 client.26494 trim_cache size 0 max 16384 2023-11-16T16:54:26.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.928+0000 7f8300defb80 1 Processor -- start 2023-11-16T16:54:26.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.928+0000 7f8300defb80 1 -- start start 2023-11-16T16:54:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.929+0000 7f8300defb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c1ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.929+0000 7f8300defb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1c1b720 0x558fe1c15560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.929+0000 7f8300defb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1bb90 0x558fe1c0ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.929+0000 7f8300defb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe1c15aa0 con 0x558fe1c1b720 2023-11-16T16:54:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.929+0000 7f8300defb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe1c1d260 con 0x558fe1c1bb90 2023-11-16T16:54:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.929+0000 7f8300defb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe1c199d0 con 0x558fe1a716b0 2023-11-16T16:54:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.929+0000 7f82e6ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c1ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.929+0000 7f82e6ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c1ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:36432/0 (socket says 172.21.15.44:36432) 2023-11-16T16:54:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.929+0000 7f82e6ffd700 1 -- 172.21.15.44:0/1495964104 learned_addr learned my addr 172.21.15.44:0/1495964104 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:26.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.929+0000 7f82e67fc700 1 --2- 172.21.15.44:0/1495964104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1c1b720 0x558fe1c15560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.930+0000 7f82e6ffd700 1 -- 172.21.15.44:0/1495964104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1bb90 msgr2=0x558fe1c0ff60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:54:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.930+0000 7f82e6ffd700 1 --2- 172.21.15.44:0/1495964104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1bb90 0x558fe1c0ff60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.930+0000 7f82e6ffd700 1 -- 172.21.15.44:0/1495964104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1c1b720 msgr2=0x558fe1c15560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.930+0000 7f82e6ffd700 1 --2- 172.21.15.44:0/1495964104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1c1b720 0x558fe1c15560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.930+0000 7f82e6ffd700 1 -- 172.21.15.44:0/1495964104 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fe1b2ce80 con 0x558fe1a716b0 2023-11-16T16:54:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.930+0000 7f82e67fc700 1 --2- 172.21.15.44:0/1495964104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1c1b720 0x558fe1c15560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:54:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.930+0000 7f82e6ffd700 1 --2- 172.21.15.44:0/1495964104 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c1ee10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f82d8000c00 tx=0x7f82d800ef70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ec5bfcc16518505 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.930+0000 7f82cdffb700 1 -- 172.21.15.44:0/1495964104 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d80cd170 con 0x558fe1a716b0 2023-11-16T16:54:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.930+0000 7f82cdffb700 1 -- 172.21.15.44:0/1495964104 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82d80999a0 con 0x558fe1a716b0 2023-11-16T16:54:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.930+0000 7f82cdffb700 1 -- 172.21.15.44:0/1495964104 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d80a22a0 con 0x558fe1a716b0 2023-11-16T16:54:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.931+0000 7f8300defb80 1 -- 172.21.15.44:0/1495964104 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 msgr2=0x558fe1c1ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.931+0000 7f8300defb80 1 --2- 172.21.15.44:0/1495964104 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c1ee10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f82d8000c00 tx=0x7f82d800ef70 comp rx=0 tx=0).stop 2023-11-16T16:54:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.931+0000 7f8300defb80 1 -- 172.21.15.44:0/1495964104 shutdown_connections 2023-11-16T16:54:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.931+0000 7f8300defb80 1 --2- 172.21.15.44:0/1495964104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1bb90 0x558fe1c0ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.931+0000 7f8300defb80 1 --2- 172.21.15.44:0/1495964104 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c1ee10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.931+0000 7f8300defb80 1 --2- 172.21.15.44:0/1495964104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1c1b720 0x558fe1c15560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.931+0000 7f8300defb80 1 -- 172.21.15.44:0/1495964104 >> 172.21.15.44:0/1495964104 conn(0x558fe1c0af50 msgr2=0x558fe1c1f610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.931+0000 7f8300defb80 1 -- 172.21.15.44:0/1495964104 shutdown_connections 2023-11-16T16:54:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.931+0000 7f8300defb80 1 -- 172.21.15.44:0/1495964104 wait complete. 2023-11-16T16:54:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.932+0000 7f8300defb80 1 -- start start 2023-11-16T16:54:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.932+0000 7f8300defb80 1 Processor -- start 2023-11-16T16:54:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.932+0000 7f8300defb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c22720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.932+0000 7f8300defb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1c1b720 0x558fe1c22c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.932+0000 7f8300defb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1bb90 0x558fe1c81810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.936 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.932+0000 7f8300defb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe1c23440 con 0x558fe1c1b720 2023-11-16T16:54:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.932+0000 7f8300defb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe1c235b0 con 0x558fe1c1bb90 2023-11-16T16:54:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.932+0000 7f8300defb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe1c81da0 con 0x558fe1a716b0 2023-11-16T16:54:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.933+0000 7f8300defb80 10 client.0 _ll_register_callbacks cb 0x7f82e0009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.933+0000 7f82e6ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c22720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.933+0000 7f82e6ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c22720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:36448/0 (socket says 172.21.15.44:36448) 2023-11-16T16:54:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.933+0000 7f82e6ffd700 1 -- 172.21.15.44:0/1038234580 learned_addr learned my addr 172.21.15.44:0/1038234580 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.933+0000 7f82e77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1bb90 0x558fe1c81810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.933+0000 7f82e67fc700 1 --2- 172.21.15.44:0/1038234580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1c1b720 0x558fe1c22c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.933+0000 7f82e6ffd700 1 -- 172.21.15.44:0/1038234580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1bb90 msgr2=0x558fe1c81810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.933+0000 7f82e6ffd700 1 --2- 172.21.15.44:0/1038234580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1bb90 0x558fe1c81810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.933+0000 7f82e6ffd700 1 -- 172.21.15.44:0/1038234580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1c1b720 msgr2=0x558fe1c22c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.933+0000 7f82e6ffd700 1 --2- 172.21.15.44:0/1038234580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1c1b720 0x558fe1c22c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.933+0000 7f82e6ffd700 1 -- 172.21.15.44:0/1038234580 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558fe1b2d130 con 0x558fe1a716b0 2023-11-16T16:54:26.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.933+0000 7f82e6ffd700 1 --2- 172.21.15.44:0/1038234580 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c22720 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f82d8096ab0 tx=0x7f82d809aee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1178bd1ad5abce61 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:26.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.933+0000 7f82caff5700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:54:26.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.934+0000 7f82caff5700 1 -- 172.21.15.44:0/1038234580 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d80a2490 con 0x558fe1a716b0 2023-11-16T16:54:26.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.934+0000 7f8300defb80 1 -- 172.21.15.44:0/1038234580 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f82d8099e10 con 0x558fe1a716b0 2023-11-16T16:54:26.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.934+0000 7f82caff5700 1 -- 172.21.15.44:0/1038234580 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f82d809e070 con 0x558fe1a716b0 2023-11-16T16:54:26.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.934+0000 7f82caff5700 1 -- 172.21.15.44:0/1038234580 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82d809a040 con 0x558fe1a716b0 2023-11-16T16:54:26.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.934+0000 7f8300defb80 10 client.26500 fetch_fsmap learned FSMap version 21 2023-11-16T16:54:26.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.934+0000 7f8300defb80 1 -- 172.21.15.44:0/1038234580 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x558fe197dc00 con 0x558fe1a716b0 2023-11-16T16:54:26.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.934+0000 7f82caff5700 1 -- 172.21.15.44:0/1038234580 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82d8099e10 con 0x558fe1a716b0 2023-11-16T16:54:26.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.934+0000 7f82caff5700 1 -- 172.21.15.44:0/1038234580 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f82d80b2a20 con 0x558fe1a716b0 2023-11-16T16:54:26.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.935+0000 7f8300defb80 10 client.26500 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:54:26.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.935+0000 7f8300defb80 10 client.26500 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:26.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.935+0000 7f8300defb80 20 client.26500 populate_metadata read hostname 'smithi044' 2023-11-16T16:54:26.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.935+0000 7f8300defb80 1 --2- 172.21.15.44:0/1038234580 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558fe1c83000 0x558fe1c853e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:26.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.935+0000 7f8300defb80 4 client.26500 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:54:26.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.935+0000 7f8300defb80 1 -- 172.21.15.44:0/1038234580 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558fe1b35600 con 0x558fe1c83000 2023-11-16T16:54:26.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.935+0000 7f82e67fc700 1 --2- 172.21.15.44:0/1038234580 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558fe1c83000 0x558fe1c853e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:26.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.935+0000 7f82caff5700 10 client.26500 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:26.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:26.935+0000 7f82e67fc700 1 --2- 172.21.15.44:0/1038234580 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558fe1c83000 0x558fe1c853e0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da34476553a90427 server_cookie=3566dd9c9323ded0 in_seq=0 out_seq=0 2023-11-16T16:54:27.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:27 smithi077 ceph-mon[131204]: pgmap v138: 129 pgs: 129 active+clean; 34 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 1.41k op/s 2023-11-16T16:54:28.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:27 smithi044 ceph-mon[128750]: pgmap v138: 129 pgs: 129 active+clean; 34 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 1.41k op/s 2023-11-16T16:54:28.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:27 smithi176 ceph-mon[131630]: pgmap v138: 129 pgs: 129 active+clean; 34 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 1.41k op/s 2023-11-16T16:54:28.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:28 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:54:29.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:28 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:54:29.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:28 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:54:29.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:29 smithi077 ceph-mon[131204]: pgmap v139: 129 pgs: 129 active+clean; 47 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 1.81k op/s 2023-11-16T16:54:30.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:29 smithi044 ceph-mon[128750]: pgmap v139: 129 pgs: 129 active+clean; 47 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 1.81k op/s 2023-11-16T16:54:30.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:29 smithi176 ceph-mon[131630]: pgmap v139: 129 pgs: 129 active+clean; 47 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 1.81k op/s 2023-11-16T16:54:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.823+0000 7f82caff5700 1 -- 172.21.15.44:0/1038234580 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558fe1b35600 con 0x558fe1c83000 2023-11-16T16:54:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.824+0000 7f82caff5700 10 client.26500 handle_command_reply: tid=0 2023-11-16T16:54:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.824+0000 7f8300defb80 1 client.26500 shutdown 2023-11-16T16:54:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.824+0000 7f8300defb80 1 -- 172.21.15.44:0/1038234580 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 msgr2=0x558fe1c22720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.824+0000 7f8300defb80 1 --2- 172.21.15.44:0/1038234580 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c22720 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f82d8096ab0 tx=0x7f82d809aee0 comp rx=0 tx=0).stop 2023-11-16T16:54:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.825+0000 7f8300defb80 1 -- 172.21.15.44:0/1038234580 shutdown_connections 2023-11-16T16:54:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.825+0000 7f8300defb80 1 --2- 172.21.15.44:0/1038234580 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x558fe1c83000 0x558fe1c853e0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.825+0000 7f8300defb80 1 --2- 172.21.15.44:0/1038234580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fe1c1bb90 0x558fe1c81810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.825+0000 7f8300defb80 1 --2- 172.21.15.44:0/1038234580 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x558fe1a716b0 0x558fe1c22720 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.825+0000 7f8300defb80 1 --2- 172.21.15.44:0/1038234580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558fe1c1b720 0x558fe1c22c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.825+0000 7f8300defb80 1 -- 172.21.15.44:0/1038234580 >> 172.21.15.44:0/1038234580 conn(0x558fe1c0af50 msgr2=0x558fe1c19de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.825+0000 7f82caff5700 0 client.26500 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.825+0000 7f8300defb80 1 -- 172.21.15.44:0/1038234580 shutdown_connections 2023-11-16T16:54:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.825+0000 7f8300defb80 1 -- 172.21.15.44:0/1038234580 wait complete. 2023-11-16T16:54:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.826+0000 7f8300defb80 20 client.26500 trim_cache size 0 max 16384 2023-11-16T16:54:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:54:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:54:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:54:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:54:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.827+0000 7f82ed3ff700 1 -- 172.21.15.44:0/3599718801 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f82d00a12c0 msgr2=0x7f82d00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.827+0000 7f82ed3ff700 1 --2- 172.21.15.44:0/3599718801 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f82d00a12c0 0x7f82d00a3770 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009fd0 tx=0x7f82e0004ad0 comp rx=0 tx=0).stop 2023-11-16T16:54:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.827+0000 7f82ed3ff700 1 -- 172.21.15.44:0/3599718801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82e8100c50 msgr2=0x7f82e81985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.827+0000 7f82ed3ff700 1 --2- 172.21.15.44:0/3599718801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82e8100c50 0x7f82e81985f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f82d800e9a0 tx=0x7f82d800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:54:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.827+0000 7f82ed3ff700 1 -- 172.21.15.44:0/3599718801 shutdown_connections 2023-11-16T16:54:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.827+0000 7f82ed3ff700 1 --2- 172.21.15.44:0/3599718801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f82e8105560 0x7f82e819cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.827+0000 7f82ed3ff700 1 --2- 172.21.15.44:0/3599718801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82e8100c50 0x7f82e81985f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.827+0000 7f82ed3ff700 1 --2- 172.21.15.44:0/3599718801 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f82d00a12c0 0x7f82d00a3770 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.827+0000 7f82ed3ff700 1 --2- 172.21.15.44:0/3599718801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82e81015f0 0x7f82e8198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.827+0000 7f82ed3ff700 1 -- 172.21.15.44:0/3599718801 >> 172.21.15.44:0/3599718801 conn(0x7f82e8077550 msgr2=0x7f82e80fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.828+0000 7f82ed3ff700 1 -- 172.21.15.44:0/3599718801 shutdown_connections 2023-11-16T16:54:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:30.829+0000 7f82ed3ff700 1 -- 172.21.15.44:0/3599718801 wait complete. 2023-11-16T16:54:31.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:31 smithi077 ceph-mon[131204]: pgmap v140: 129 pgs: 129 active+clean; 47 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 934 op/s 2023-11-16T16:54:32.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:31 smithi044 ceph-mon[128750]: pgmap v140: 129 pgs: 129 active+clean; 47 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 934 op/s 2023-11-16T16:54:32.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:31 smithi176 ceph-mon[131630]: pgmap v140: 129 pgs: 129 active+clean; 47 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 934 op/s 2023-11-16T16:54:33.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:33 smithi077 ceph-mon[131204]: pgmap v141: 129 pgs: 129 active+clean; 32 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 1.82k op/s 2023-11-16T16:54:34.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:33 smithi044 ceph-mon[128750]: pgmap v141: 129 pgs: 129 active+clean; 32 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 1.82k op/s 2023-11-16T16:54:34.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:33 smithi176 ceph-mon[131630]: pgmap v141: 129 pgs: 129 active+clean; 32 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 1.82k op/s 2023-11-16T16:54:35.184 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:54:35.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.429+0000 7f4b815ff700 1 -- 172.21.15.44:0/2480796665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b7c072e70 msgr2=0x7f4b7c073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:35.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.429+0000 7f4b815ff700 1 --2- 172.21.15.44:0/2480796665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b7c072e70 0x7f4b7c073240 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c009990 tx=0x7f4b6c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.429+0000 7f4b815ff700 1 -- 172.21.15.44:0/2480796665 shutdown_connections 2023-11-16T16:54:35.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.429+0000 7f4b815ff700 1 --2- 172.21.15.44:0/2480796665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b7c10cba0 0x7f4b7c10ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.429+0000 7f4b815ff700 1 --2- 172.21.15.44:0/2480796665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b7c072e70 0x7f4b7c073240 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.429+0000 7f4b815ff700 1 --2- 172.21.15.44:0/2480796665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b7c073780 0x7f4b7c10c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.429+0000 7f4b815ff700 1 -- 172.21.15.44:0/2480796665 >> 172.21.15.44:0/2480796665 conn(0x7f4b7c077140 msgr2=0x7f4b7c077540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:35.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.429+0000 7f4b815ff700 1 -- 172.21.15.44:0/2480796665 shutdown_connections 2023-11-16T16:54:35.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.429+0000 7f4b815ff700 1 -- 172.21.15.44:0/2480796665 wait complete. 2023-11-16T16:54:35.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.430+0000 7f4b815ff700 1 Processor -- start 2023-11-16T16:54:35.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.431+0000 7f4b815ff700 1 -- start start 2023-11-16T16:54:35.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.431+0000 7f4b815ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b7c072e70 0x7f4b7c19e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.431+0000 7f4b815ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b7c073780 0x7f4b7c19ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.431+0000 7f4b815ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b7c10cba0 0x7f4b7c198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.431+0000 7f4b815ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c198bf0 con 0x7f4b7c073780 2023-11-16T16:54:35.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.431+0000 7f4b815ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c198d60 con 0x7f4b7c072e70 2023-11-16T16:54:35.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.431+0000 7f4b815ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c198f20 con 0x7f4b7c10cba0 2023-11-16T16:54:35.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.431+0000 7f4b7affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b7c072e70 0x7f4b7c19e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.432+0000 7f4b7affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b7c072e70 0x7f4b7c19e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60270/0 (socket says 172.21.15.44:60270) 2023-11-16T16:54:35.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.432+0000 7f4b7a7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b7c073780 0x7f4b7c19ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.432+0000 7f4b7affd700 1 -- 172.21.15.44:0/3965281882 learned_addr learned my addr 172.21.15.44:0/3965281882 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:35.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.432+0000 7f4b7b7fe700 1 --2- 172.21.15.44:0/3965281882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b7c10cba0 0x7f4b7c198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.432+0000 7f4b7affd700 1 -- 172.21.15.44:0/3965281882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b7c10cba0 msgr2=0x7f4b7c198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:35.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.432+0000 7f4b7affd700 1 --2- 172.21.15.44:0/3965281882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b7c10cba0 0x7f4b7c198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.432+0000 7f4b7affd700 1 -- 172.21.15.44:0/3965281882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b7c073780 msgr2=0x7f4b7c19ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:35.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.432+0000 7f4b7affd700 1 --2- 172.21.15.44:0/3965281882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b7c073780 0x7f4b7c19ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.432+0000 7f4b7affd700 1 -- 172.21.15.44:0/3965281882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c0095f0 con 0x7f4b7c072e70 2023-11-16T16:54:35.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.432+0000 7f4b7affd700 1 --2- 172.21.15.44:0/3965281882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b7c072e70 0x7f4b7c19e510 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c0098e0 tx=0x7f4b6c00f670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:35.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.432+0000 7f4b63fff700 1 -- 172.21.15.44:0/3965281882 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c01d070 con 0x7f4b7c072e70 2023-11-16T16:54:35.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.433+0000 7f4b63fff700 1 -- 172.21.15.44:0/3965281882 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b6c005050 con 0x7f4b7c072e70 2023-11-16T16:54:35.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.433+0000 7f4b63fff700 1 -- 172.21.15.44:0/3965281882 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c004b30 con 0x7f4b7c072e70 2023-11-16T16:54:35.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.433+0000 7f4b815ff700 1 -- 172.21.15.44:0/3965281882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7c112580 con 0x7f4b7c072e70 2023-11-16T16:54:35.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.433+0000 7f4b815ff700 1 -- 172.21.15.44:0/3965281882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7c1123f0 con 0x7f4b7c072e70 2023-11-16T16:54:35.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.437+0000 7f4b63fff700 1 -- 172.21.15.44:0/3965281882 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f4b6c0051c0 con 0x7f4b7c072e70 2023-11-16T16:54:35.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.438+0000 7f4b94f67b80 1 Processor -- start 2023-11-16T16:54:35.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.438+0000 7f4b94f67b80 1 -- start start 2023-11-16T16:54:35.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.439+0000 7f4b94f67b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 0x56396ad2ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.439+0000 7f4b94f67b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad34370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.439+0000 7f4b94f67b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad348b0 0x56396ad34d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.439+0000 7f4b94f67b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396ac35060 con 0x56396ad348b0 2023-11-16T16:54:35.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.439+0000 7f4b94f67b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396ac878a0 con 0x56396ad30470 2023-11-16T16:54:35.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.439+0000 7f4b94f67b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396ac80d30 con 0x56396ad2fb60 2023-11-16T16:54:35.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.439+0000 7f4b7affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad34370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.439+0000 7f4b7affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad34370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60278/0 (socket says 172.21.15.44:60278) 2023-11-16T16:54:35.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.439+0000 7f4b7affd700 1 -- 172.21.15.44:0/3863285776 learned_addr learned my addr 172.21.15.44:0/3863285776 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:35.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.439+0000 7f4b7b7fe700 1 --2- 172.21.15.44:0/3863285776 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 0x56396ad2ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.439+0000 7f4b7a7fc700 1 --2- 172.21.15.44:0/3863285776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad348b0 0x56396ad34d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.439+0000 7f4b63fff700 1 --2- 172.21.15.44:0/3965281882 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4b640a12c0 0x7f4b640a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.440+0000 7f4b63fff700 1 -- 172.21.15.44:0/3965281882 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f4b6c0cde20 con 0x7f4b7c072e70 2023-11-16T16:54:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.440+0000 7f4b7b7fe700 1 -- 172.21.15.44:0/3863285776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 msgr2=0x56396ad34370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.440+0000 7f4b7b7fe700 1 --2- 172.21.15.44:0/3863285776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad34370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.440+0000 7f4b7b7fe700 1 -- 172.21.15.44:0/3863285776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad348b0 msgr2=0x56396ad34d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.440+0000 7f4b7b7fe700 1 --2- 172.21.15.44:0/3863285776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad348b0 0x56396ad34d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.440+0000 7f4b7b7fe700 1 -- 172.21.15.44:0/3863285776 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56396ab02630 con 0x56396ad2fb60 2023-11-16T16:54:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.440+0000 7f4b7b7fe700 1 --2- 172.21.15.44:0/3965281882 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4b640a12c0 0x7f4b640a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.441+0000 7f4b7b7fe700 1 --2- 172.21.15.44:0/3863285776 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 0x56396ad2ff30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4b74003f20 tx=0x7f4b74009c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d83eecc1355ae2bd server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.441+0000 7f4b617fa700 1 -- 172.21.15.44:0/3863285776 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b74007e70 con 0x56396ad2fb60 2023-11-16T16:54:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.441+0000 7f4b617fa700 1 -- 172.21.15.44:0/3863285776 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b74026070 con 0x56396ad2fb60 2023-11-16T16:54:35.444 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.441+0000 7f4b617fa700 1 -- 172.21.15.44:0/3863285776 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b74022650 con 0x56396ad2fb60 2023-11-16T16:54:35.444 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.441+0000 7f4b7b7fe700 1 --2- 172.21.15.44:0/3965281882 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4b640a12c0 0x7f4b640a3770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4b74021000 tx=0x7f4b7401e000 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:35.444 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.442+0000 7f4b94f67b80 1 -- 172.21.15.44:0/3863285776 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 msgr2=0x56396ad2ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:35.445 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.442+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/3863285776 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 0x56396ad2ff30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4b74003f20 tx=0x7f4b74009c20 comp rx=0 tx=0).stop 2023-11-16T16:54:35.445 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.442+0000 7f4b94f67b80 1 -- 172.21.15.44:0/3863285776 shutdown_connections 2023-11-16T16:54:35.445 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.442+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/3863285776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad348b0 0x56396ad34d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.445 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.442+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/3863285776 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 0x56396ad2ff30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.445 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.442+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/3863285776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad34370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.445 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.442+0000 7f4b94f67b80 1 -- 172.21.15.44:0/3863285776 >> 172.21.15.44:0/3863285776 conn(0x56396ad25360 msgr2=0x56396ad2ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:35.445 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.442+0000 7f4b94f67b80 1 -- 172.21.15.44:0/3863285776 shutdown_connections 2023-11-16T16:54:35.445 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.442+0000 7f4b94f67b80 1 -- 172.21.15.44:0/3863285776 wait complete. 2023-11-16T16:54:35.445 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.443+0000 7f4b94f67b80 1 -- start start 2023-11-16T16:54:35.445 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.443+0000 7f4b94f67b80 1 Processor -- start 2023-11-16T16:54:35.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.444+0000 7f4b94f67b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 0x56396ad3c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.444+0000 7f4b94f67b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad39280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.444+0000 7f4b94f67b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad348b0 0x56396ad398b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.444+0000 7f4b94f67b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396ac228e0 con 0x56396ad348b0 2023-11-16T16:54:35.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.444+0000 7f4b94f67b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396ac22580 con 0x56396ad30470 2023-11-16T16:54:35.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.444+0000 7f4b94f67b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396ab04920 con 0x56396ad2fb60 2023-11-16T16:54:35.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.444+0000 7f4b94f67b80 10 client.0 _ll_register_callbacks cb 0x56396ac40780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:35.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.444+0000 7f4b7b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad348b0 0x56396ad398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.444+0000 7f4b7a7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad39280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.444+0000 7f4b7a7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad39280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60280/0 (socket says 172.21.15.44:60280) 2023-11-16T16:54:35.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.444+0000 7f4b7b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad348b0 0x56396ad398b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59970/0 (socket says 172.21.15.44:59970) 2023-11-16T16:54:35.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.444+0000 7f4b7a7fc700 1 -- 172.21.15.44:0/465427803 learned_addr learned my addr 172.21.15.44:0/465427803 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:35.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.444+0000 7f4b7affd700 1 --2- 172.21.15.44:0/465427803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 0x56396ad3c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.445+0000 7f4b7a7fc700 1 -- 172.21.15.44:0/465427803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 msgr2=0x56396ad3c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:35.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.445+0000 7f4b7a7fc700 1 --2- 172.21.15.44:0/465427803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 0x56396ad3c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.445+0000 7f4b7a7fc700 1 -- 172.21.15.44:0/465427803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad348b0 msgr2=0x56396ad398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:35.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.445+0000 7f4b7a7fc700 1 --2- 172.21.15.44:0/465427803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad348b0 0x56396ad398b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.445+0000 7f4b7a7fc700 1 -- 172.21.15.44:0/465427803 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b74003a90 con 0x56396ad30470 2023-11-16T16:54:35.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.445+0000 7f4b7a7fc700 1 --2- 172.21.15.44:0/465427803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad39280 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4b7000bac0 tx=0x7f4b7000bdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1adec111c130626b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:35.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.445+0000 7f4b5eff5700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:54:35.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.445+0000 7f4b5eff5700 1 -- 172.21.15.44:0/465427803 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70004520 con 0x56396ad30470 2023-11-16T16:54:35.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.445+0000 7f4b94f67b80 1 -- 172.21.15.44:0/465427803 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56396ad22b20 con 0x56396ad30470 2023-11-16T16:54:35.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.446+0000 7f4b5eff5700 1 -- 172.21.15.44:0/465427803 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f4b70004830 con 0x56396ad30470 2023-11-16T16:54:35.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.446+0000 7f4b5eff5700 1 -- 172.21.15.44:0/465427803 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b700049f0 con 0x56396ad30470 2023-11-16T16:54:35.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.446+0000 7f4b94f67b80 10 client.26519 fetch_fsmap learned FSMap version 21 2023-11-16T16:54:35.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.446+0000 7f4b94f67b80 1 -- 172.21.15.44:0/465427803 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x56396ac47190 con 0x56396ad30470 2023-11-16T16:54:35.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.446+0000 7f4b5eff5700 1 -- 172.21.15.44:0/465427803 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56396ad22b20 con 0x56396ad30470 2023-11-16T16:54:35.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.446+0000 7f4b5eff5700 1 -- 172.21.15.44:0/465427803 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4b70011440 con 0x56396ad30470 2023-11-16T16:54:35.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.447+0000 7f4b94f67b80 10 client.26519 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:54:35.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.447+0000 7f4b94f67b80 10 client.26519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:35.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.447+0000 7f4b94f67b80 20 client.26519 populate_metadata read hostname 'smithi044' 2023-11-16T16:54:35.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.447+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/465427803 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56396ad498e0 0x56396ad4bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.447+0000 7f4b94f67b80 4 client.26519 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:54:35.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.447+0000 7f4b94f67b80 1 -- 172.21.15.44:0/465427803 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56396ac430a0 con 0x56396ad498e0 2023-11-16T16:54:35.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.447+0000 7f4b7affd700 1 --2- 172.21.15.44:0/465427803 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56396ad498e0 0x56396ad4bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.451 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.447+0000 7f4b7affd700 1 --2- 172.21.15.44:0/465427803 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56396ad498e0 0x56396ad4bcf0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cd569dbeadd8301 server_cookie=444104247feb2ca9 in_seq=0 out_seq=0 2023-11-16T16:54:35.451 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.447+0000 7f4b5eff5700 10 client.26519 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:35.451 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.448+0000 7f4b5eff5700 1 -- 172.21.15.44:0/465427803 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56396ac430a0 con 0x56396ad498e0 2023-11-16T16:54:35.451 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.448+0000 7f4b5eff5700 10 client.26519 handle_command_reply: tid=0 2023-11-16T16:54:35.451 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.448+0000 7f4b94f67b80 1 client.26519 shutdown 2023-11-16T16:54:35.451 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.449+0000 7f4b94f67b80 1 -- 172.21.15.44:0/465427803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 msgr2=0x56396ad39280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:35.451 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.449+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/465427803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad39280 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4b7000bac0 tx=0x7f4b7000bdd0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.451 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.449+0000 7f4b94f67b80 1 -- 172.21.15.44:0/465427803 shutdown_connections 2023-11-16T16:54:35.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.449+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/465427803 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56396ad498e0 0x56396ad4bcf0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.449+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/465427803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad348b0 0x56396ad398b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.449+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/465427803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 0x56396ad3c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.449+0000 7f4b5eff5700 0 client.26519 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:35.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.449+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/465427803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad39280 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.449+0000 7f4b94f67b80 1 -- 172.21.15.44:0/465427803 >> 172.21.15.44:0/465427803 conn(0x56396ad25360 msgr2=0x56396ad35d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:35.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.449+0000 7f4b94f67b80 1 -- 172.21.15.44:0/465427803 shutdown_connections 2023-11-16T16:54:35.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.449+0000 7f4b94f67b80 1 -- 172.21.15.44:0/465427803 wait complete. 2023-11-16T16:54:35.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.450+0000 7f4b94f67b80 20 client.26519 trim_cache size 0 max 16384 2023-11-16T16:54:35.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.469+0000 7f4b94f67b80 1 Processor -- start 2023-11-16T16:54:35.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.469+0000 7f4b94f67b80 1 -- start start 2023-11-16T16:54:35.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.470+0000 7f4b94f67b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 0x56396ad409a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.470+0000 7f4b94f67b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad328b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.470+0000 7f4b94f67b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad348b0 0x56396ad2a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.470+0000 7f4b94f67b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396ad33230 con 0x56396ad348b0 2023-11-16T16:54:35.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.470+0000 7f4b94f67b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396ad22710 con 0x56396ad30470 2023-11-16T16:54:35.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.470+0000 7f4b94f67b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396ad24ee0 con 0x56396ad2fb60 2023-11-16T16:54:35.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.470+0000 7f4b7a7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad328b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.470+0000 7f4b7a7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad328b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60296/0 (socket says 172.21.15.44:60296) 2023-11-16T16:54:35.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.470+0000 7f4b7a7fc700 1 -- 172.21.15.44:0/1144889707 learned_addr learned my addr 172.21.15.44:0/1144889707 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:35.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.470+0000 7f4b7affd700 1 --2- 172.21.15.44:0/1144889707 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 0x56396ad409a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.470+0000 7f4b7b7fe700 1 --2- 172.21.15.44:0/1144889707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad348b0 0x56396ad2a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.471+0000 7f4b7a7fc700 1 -- 172.21.15.44:0/1144889707 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 msgr2=0x56396ad409a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:35.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.471+0000 7f4b7a7fc700 1 --2- 172.21.15.44:0/1144889707 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 0x56396ad409a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.471+0000 7f4b7a7fc700 1 -- 172.21.15.44:0/1144889707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad348b0 msgr2=0x56396ad2a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:35.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.471+0000 7f4b7a7fc700 1 --2- 172.21.15.44:0/1144889707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad348b0 0x56396ad2a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.471+0000 7f4b7a7fc700 1 -- 172.21.15.44:0/1144889707 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56396ac46ee0 con 0x56396ad30470 2023-11-16T16:54:35.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.471+0000 7f4b7affd700 1 --2- 172.21.15.44:0/1144889707 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 0x56396ad409a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:35.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.471+0000 7f4b7a7fc700 1 --2- 172.21.15.44:0/1144889707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad328b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4b70000c00 tx=0x7f4b70004170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40c1c20c5e4f847d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:35.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.471+0000 7f4b61ffb700 1 -- 172.21.15.44:0/1144889707 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70003960 con 0x56396ad30470 2023-11-16T16:54:35.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.471+0000 7f4b61ffb700 1 -- 172.21.15.44:0/1144889707 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b70011ac0 con 0x56396ad30470 2023-11-16T16:54:35.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.471+0000 7f4b61ffb700 1 -- 172.21.15.44:0/1144889707 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70005070 con 0x56396ad30470 2023-11-16T16:54:35.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.472+0000 7f4b94f67b80 1 -- 172.21.15.44:0/1144889707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 msgr2=0x56396ad328b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:35.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.472+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/1144889707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad328b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4b70000c00 tx=0x7f4b70004170 comp rx=0 tx=0).stop 2023-11-16T16:54:35.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.472+0000 7f4b94f67b80 1 -- 172.21.15.44:0/1144889707 shutdown_connections 2023-11-16T16:54:35.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.472+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/1144889707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad348b0 0x56396ad2a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.472+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/1144889707 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad2fb60 0x56396ad409a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.472+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/1144889707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad328b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.472+0000 7f4b94f67b80 1 -- 172.21.15.44:0/1144889707 >> 172.21.15.44:0/1144889707 conn(0x56396ad25360 msgr2=0x56396ad45c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:35.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.473+0000 7f4b94f67b80 1 -- 172.21.15.44:0/1144889707 shutdown_connections 2023-11-16T16:54:35.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.473+0000 7f4b94f67b80 1 -- 172.21.15.44:0/1144889707 wait complete. 2023-11-16T16:54:35.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.473+0000 7f4b94f67b80 1 -- start start 2023-11-16T16:54:35.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.473+0000 7f4b94f67b80 1 Processor -- start 2023-11-16T16:54:35.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.474+0000 7f4b94f67b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad2fb60 0x56396ad409b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.474+0000 7f4b94f67b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad37a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.474+0000 7f4b94f67b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad348b0 0x56396ad37f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.474+0000 7f4b94f67b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396ad384b0 con 0x56396ad2fb60 2023-11-16T16:54:35.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.474+0000 7f4b94f67b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396ad38620 con 0x56396ad30470 2023-11-16T16:54:35.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.474+0000 7f4b94f67b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396ad387e0 con 0x56396ad348b0 2023-11-16T16:54:35.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.474+0000 7f4b94f67b80 10 client.0 _ll_register_callbacks cb 0x7f4b700042a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:35.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.474+0000 7f4b7affd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad2fb60 0x56396ad409b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.474+0000 7f4b7a7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad37a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.474+0000 7f4b7a7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad37a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60300/0 (socket says 172.21.15.44:60300) 2023-11-16T16:54:35.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.474+0000 7f4b7a7fc700 1 -- 172.21.15.44:0/1291566161 learned_addr learned my addr 172.21.15.44:0/1291566161 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:35.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.475+0000 7f4b7b7fe700 1 --2- 172.21.15.44:0/1291566161 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad348b0 0x56396ad37f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.475+0000 7f4b7a7fc700 1 -- 172.21.15.44:0/1291566161 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad348b0 msgr2=0x56396ad37f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:35.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.475+0000 7f4b7a7fc700 1 --2- 172.21.15.44:0/1291566161 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad348b0 0x56396ad37f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.475+0000 7f4b7a7fc700 1 -- 172.21.15.44:0/1291566161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad2fb60 msgr2=0x56396ad409b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:35.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.475+0000 7f4b7a7fc700 1 --2- 172.21.15.44:0/1291566161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad2fb60 0x56396ad409b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.475+0000 7f4b7a7fc700 1 -- 172.21.15.44:0/1291566161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7c1123f0 con 0x56396ad30470 2023-11-16T16:54:35.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.475+0000 7f4b7a7fc700 1 --2- 172.21.15.44:0/1291566161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad37a30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4b70014040 tx=0x7f4b700116f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50ceb30aa15f7a05 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:35.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.475+0000 7f4b5eff5700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:54:35.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.476+0000 7f4b5eff5700 1 -- 172.21.15.44:0/1291566161 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70003d80 con 0x56396ad30470 2023-11-16T16:54:35.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.476+0000 7f4b5eff5700 1 -- 172.21.15.44:0/1291566161 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f4b70000e40 con 0x56396ad30470 2023-11-16T16:54:35.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.476+0000 7f4b94f67b80 1 -- 172.21.15.44:0/1291566161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4b74026070 con 0x56396ad30470 2023-11-16T16:54:35.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.476+0000 7f4b5eff5700 1 -- 172.21.15.44:0/1291566161 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b70005260 con 0x56396ad30470 2023-11-16T16:54:35.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.476+0000 7f4b94f67b80 10 client.26525 fetch_fsmap learned FSMap version 21 2023-11-16T16:54:35.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.476+0000 7f4b94f67b80 1 -- 172.21.15.44:0/1291566161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x7f4b7001f040 con 0x56396ad30470 2023-11-16T16:54:35.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.477+0000 7f4b5eff5700 1 -- 172.21.15.44:0/1291566161 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b74026070 con 0x56396ad30470 2023-11-16T16:54:35.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.477+0000 7f4b5eff5700 1 -- 172.21.15.44:0/1291566161 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4b70010b80 con 0x56396ad30470 2023-11-16T16:54:35.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.477+0000 7f4b94f67b80 10 client.26525 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:54:35.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.477+0000 7f4b94f67b80 10 client.26525 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:35.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.477+0000 7f4b94f67b80 20 client.26525 populate_metadata read hostname 'smithi044' 2023-11-16T16:54:35.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.477+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/1291566161 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56396ad9e8f0 0x56396ada0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.477+0000 7f4b94f67b80 4 client.26525 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:54:35.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.477+0000 7f4b94f67b80 1 -- 172.21.15.44:0/1291566161 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56396ac4f660 con 0x56396ad9e8f0 2023-11-16T16:54:35.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.477+0000 7f4b7affd700 1 --2- 172.21.15.44:0/1291566161 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56396ad9e8f0 0x56396ada0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.478+0000 7f4b7affd700 1 --2- 172.21.15.44:0/1291566161 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56396ad9e8f0 0x56396ada0d00 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7a61b3df5f859ac server_cookie=a4fe96d978cb9dce in_seq=0 out_seq=0 2023-11-16T16:54:35.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.478+0000 7f4b5eff5700 10 client.26525 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:35.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.478+0000 7f4b5eff5700 1 -- 172.21.15.44:0/1291566161 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56396ac4f660 con 0x56396ad9e8f0 2023-11-16T16:54:35.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.478+0000 7f4b5eff5700 10 client.26525 handle_command_reply: tid=0 2023-11-16T16:54:35.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.478+0000 7f4b94f67b80 1 client.26525 shutdown 2023-11-16T16:54:35.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.478+0000 7f4b94f67b80 1 -- 172.21.15.44:0/1291566161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 msgr2=0x56396ad37a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:35.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.478+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/1291566161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad37a30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4b70014040 tx=0x7f4b700116f0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.478+0000 7f4b94f67b80 1 -- 172.21.15.44:0/1291566161 shutdown_connections 2023-11-16T16:54:35.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.478+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/1291566161 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56396ad9e8f0 0x56396ada0d00 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.478+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/1291566161 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56396ad348b0 0x56396ad37f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.478+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/1291566161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56396ad2fb60 0x56396ad409b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.478+0000 7f4b94f67b80 1 --2- 172.21.15.44:0/1291566161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56396ad30470 0x56396ad37a30 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.478+0000 7f4b94f67b80 1 -- 172.21.15.44:0/1291566161 >> 172.21.15.44:0/1291566161 conn(0x56396ad25360 msgr2=0x56396ad45150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:35.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.478+0000 7f4b5eff5700 0 client.26525 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:35.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.479+0000 7f4b94f67b80 1 -- 172.21.15.44:0/1291566161 shutdown_connections 2023-11-16T16:54:35.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.479+0000 7f4b94f67b80 1 -- 172.21.15.44:0/1291566161 wait complete. 2023-11-16T16:54:35.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.479+0000 7f4b94f67b80 20 client.26525 trim_cache size 0 max 16384 2023-11-16T16:54:35.483 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:54:35.483 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T16:54:35.483 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T16:54:35.483 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:54:35.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.479+0000 7f4b815ff700 1 -- 172.21.15.44:0/3965281882 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4b640a12c0 msgr2=0x7f4b640a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:35.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.479+0000 7f4b815ff700 1 --2- 172.21.15.44:0/3965281882 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4b640a12c0 0x7f4b640a3770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4b74021000 tx=0x7f4b7401e000 comp rx=0 tx=0).stop 2023-11-16T16:54:35.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.479+0000 7f4b815ff700 1 -- 172.21.15.44:0/3965281882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b7c072e70 msgr2=0x7f4b7c19e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:35.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.479+0000 7f4b815ff700 1 --2- 172.21.15.44:0/3965281882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b7c072e70 0x7f4b7c19e510 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c0098e0 tx=0x7f4b6c00f670 comp rx=0 tx=0).stop 2023-11-16T16:54:35.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.480+0000 7f4b815ff700 1 -- 172.21.15.44:0/3965281882 shutdown_connections 2023-11-16T16:54:35.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.480+0000 7f4b815ff700 1 --2- 172.21.15.44:0/3965281882 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4b640a12c0 0x7f4b640a3770 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.480+0000 7f4b815ff700 1 --2- 172.21.15.44:0/3965281882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b7c10cba0 0x7f4b7c198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.480+0000 7f4b815ff700 1 --2- 172.21.15.44:0/3965281882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b7c072e70 0x7f4b7c19e510 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.480+0000 7f4b815ff700 1 --2- 172.21.15.44:0/3965281882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b7c073780 0x7f4b7c19ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.480+0000 7f4b815ff700 1 -- 172.21.15.44:0/3965281882 >> 172.21.15.44:0/3965281882 conn(0x7f4b7c077140 msgr2=0x7f4b7c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:35.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.480+0000 7f4b815ff700 1 -- 172.21.15.44:0/3965281882 shutdown_connections 2023-11-16T16:54:35.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.480+0000 7f4b815ff700 1 -- 172.21.15.44:0/3965281882 wait complete. 2023-11-16T16:54:35.504 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T16:54:35.504 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:54:35.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.739+0000 7f0c7ddff700 1 -- 172.21.15.44:0/1381215931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c7810a800 msgr2=0x7f0c7810cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:35.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.739+0000 7f0c7ddff700 1 --2- 172.21.15.44:0/1381215931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c7810a800 0x7f0c7810cc90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0c7000b0a0 tx=0x7f0c7000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.739+0000 7f0c7ddff700 1 -- 172.21.15.44:0/1381215931 shutdown_connections 2023-11-16T16:54:35.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.739+0000 7f0c7ddff700 1 --2- 172.21.15.44:0/1381215931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c7810a800 0x7f0c7810cc90 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.739+0000 7f0c7ddff700 1 --2- 172.21.15.44:0/1381215931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c781014c0 0x7f0c78101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.739+0000 7f0c7ddff700 1 --2- 172.21.15.44:0/1381215931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c78101dd0 0x7f0c7810a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.739+0000 7f0c7ddff700 1 -- 172.21.15.44:0/1381215931 >> 172.21.15.44:0/1381215931 conn(0x7f0c78076310 msgr2=0x7f0c78076710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:35.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.739+0000 7f0c7ddff700 1 -- 172.21.15.44:0/1381215931 shutdown_connections 2023-11-16T16:54:35.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.739+0000 7f0c7ddff700 1 -- 172.21.15.44:0/1381215931 wait complete. 2023-11-16T16:54:35.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.740+0000 7f0c7ddff700 1 Processor -- start 2023-11-16T16:54:35.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.740+0000 7f0c7ddff700 1 -- start start 2023-11-16T16:54:35.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.741+0000 7f0c7ddff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c781014c0 0x7f0c78194280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.741+0000 7f0c7ddff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c78101dd0 0x7f0c781947c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.741+0000 7f0c7ddff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c7810a800 0x7f0c78198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.741+0000 7f0c7ddff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c78199260 con 0x7f0c7810a800 2023-11-16T16:54:35.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.741+0000 7f0c7ddff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c781993d0 con 0x7f0c78101dd0 2023-11-16T16:54:35.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.741+0000 7f0c7ddff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c78199590 con 0x7f0c781014c0 2023-11-16T16:54:35.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.741+0000 7f0c76ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c78101dd0 0x7f0c781947c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.741+0000 7f0c777fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c781014c0 0x7f0c78194280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.741+0000 7f0c76ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c78101dd0 0x7f0c781947c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60320/0 (socket says 172.21.15.44:60320) 2023-11-16T16:54:35.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.741+0000 7f0c76ffd700 1 -- 172.21.15.44:0/542259461 learned_addr learned my addr 172.21.15.44:0/542259461 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:35.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.741+0000 7f0c77fff700 1 --2- 172.21.15.44:0/542259461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c7810a800 0x7f0c78198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.742+0000 7f0c777fe700 1 -- 172.21.15.44:0/542259461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c78101dd0 msgr2=0x7f0c781947c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:35.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.742+0000 7f0c777fe700 1 --2- 172.21.15.44:0/542259461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c78101dd0 0x7f0c781947c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.742+0000 7f0c777fe700 1 -- 172.21.15.44:0/542259461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c7810a800 msgr2=0x7f0c78198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:35.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.742+0000 7f0c777fe700 1 --2- 172.21.15.44:0/542259461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c7810a800 0x7f0c78198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:35.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.742+0000 7f0c777fe700 1 -- 172.21.15.44:0/542259461 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c70009d00 con 0x7f0c781014c0 2023-11-16T16:54:35.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.742+0000 7f0c76ffd700 1 --2- 172.21.15.44:0/542259461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c78101dd0 0x7f0c781947c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:35.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.742+0000 7f0c77fff700 1 --2- 172.21.15.44:0/542259461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c7810a800 0x7f0c78198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:54:35.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.742+0000 7f0c777fe700 1 --2- 172.21.15.44:0/542259461 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c781014c0 0x7f0c78194280 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0c68012c10 tx=0x7f0c68012f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:35.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.742+0000 7f0c74ff9700 1 -- 172.21.15.44:0/542259461 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68010720 con 0x7f0c781014c0 2023-11-16T16:54:35.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.742+0000 7f0c7ddff700 1 -- 172.21.15.44:0/542259461 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7810e1d0 con 0x7f0c781014c0 2023-11-16T16:54:35.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.742+0000 7f0c74ff9700 1 -- 172.21.15.44:0/542259461 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c68013040 con 0x7f0c781014c0 2023-11-16T16:54:35.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.742+0000 7f0c74ff9700 1 -- 172.21.15.44:0/542259461 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6800bdf0 con 0x7f0c781014c0 2023-11-16T16:54:35.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.742+0000 7f0c7ddff700 1 -- 172.21.15.44:0/542259461 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c7810e040 con 0x7f0c781014c0 2023-11-16T16:54:35.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.743+0000 7f0c7ddff700 1 -- 172.21.15.44:0/542259461 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7810e360 con 0x7f0c781014c0 2023-11-16T16:54:35.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.745+0000 7f0c74ff9700 1 -- 172.21.15.44:0/542259461 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f0c68010880 con 0x7f0c781014c0 2023-11-16T16:54:35.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.747+0000 7f0c74ff9700 1 --2- 172.21.15.44:0/542259461 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c600a12c0 0x7f0c600a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:35.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.747+0000 7f0c74ff9700 1 -- 172.21.15.44:0/542259461 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f0c680d0ff0 con 0x7f0c781014c0 2023-11-16T16:54:35.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.747+0000 7f0c76ffd700 1 --2- 172.21.15.44:0/542259461 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c600a12c0 0x7f0c600a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:35.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.748+0000 7f0c76ffd700 1 --2- 172.21.15.44:0/542259461 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c600a12c0 0x7f0c600a3770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0c78195890 tx=0x7f0c6c009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:35.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:35.749+0000 7f0c74ff9700 1 -- 172.21.15.44:0/542259461 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0c6809a060 con 0x7f0c781014c0 2023-11-16T16:54:35.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:35 smithi044 ceph-mon[128750]: pgmap v142: 129 pgs: 129 active+clean; 7.7 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 3.00k op/s 2023-11-16T16:54:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:35 smithi077 ceph-mon[131204]: pgmap v142: 129 pgs: 129 active+clean; 7.7 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 3.00k op/s 2023-11-16T16:54:36.067 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.066+0000 7f0c7ddff700 1 -- 172.21.15.44:0/542259461 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c78105120 con 0x7f0c781014c0 2023-11-16T16:54:36.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.067+0000 7f0c74ff9700 1 -- 172.21.15.44:0/542259461 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f0c6809d680 con 0x7f0c781014c0 2023-11-16T16:54:36.069 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:54:36.069 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:54:36.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.070+0000 7f0c7ddff700 1 -- 172.21.15.44:0/542259461 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c600a12c0 msgr2=0x7f0c600a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:36.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.070+0000 7f0c7ddff700 1 --2- 172.21.15.44:0/542259461 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c600a12c0 0x7f0c600a3770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0c78195890 tx=0x7f0c6c009260 comp rx=0 tx=0).stop 2023-11-16T16:54:36.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.070+0000 7f0c7ddff700 1 -- 172.21.15.44:0/542259461 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c781014c0 msgr2=0x7f0c78194280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:36.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.070+0000 7f0c7ddff700 1 --2- 172.21.15.44:0/542259461 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c781014c0 0x7f0c78194280 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0c68012c10 tx=0x7f0c68012f20 comp rx=0 tx=0).stop 2023-11-16T16:54:36.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.070+0000 7f0c7ddff700 1 -- 172.21.15.44:0/542259461 shutdown_connections 2023-11-16T16:54:36.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.070+0000 7f0c7ddff700 1 --2- 172.21.15.44:0/542259461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c7810a800 0x7f0c78198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.070+0000 7f0c7ddff700 1 --2- 172.21.15.44:0/542259461 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c600a12c0 0x7f0c600a3770 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.070+0000 7f0c7ddff700 1 --2- 172.21.15.44:0/542259461 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c781014c0 0x7f0c78194280 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.070+0000 7f0c7ddff700 1 --2- 172.21.15.44:0/542259461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c78101dd0 0x7f0c781947c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.070+0000 7f0c7ddff700 1 -- 172.21.15.44:0/542259461 >> 172.21.15.44:0/542259461 conn(0x7f0c78076310 msgr2=0x7f0c78102260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:36.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.070+0000 7f0c7ddff700 1 -- 172.21.15.44:0/542259461 shutdown_connections 2023-11-16T16:54:36.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.070+0000 7f0c7ddff700 1 -- 172.21.15.44:0/542259461 wait complete. 2023-11-16T16:54:36.073 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:54:36.095 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T16:54:36.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:35 smithi176 ceph-mon[131630]: pgmap v142: 129 pgs: 129 active+clean; 7.7 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 3.00k op/s 2023-11-16T16:54:36.333 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.332+0000 7fd31759d700 1 -- 172.21.15.44:0/3228762515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd310102e90 msgr2=0x7fd310106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:36.334 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.332+0000 7fd31759d700 1 --2- 172.21.15.44:0/3228762515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd310102e90 0x7fd310106e50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd300009990 tx=0x7fd300009ca0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.334 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.332+0000 7fd31759d700 1 -- 172.21.15.44:0/3228762515 shutdown_connections 2023-11-16T16:54:36.334 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.332+0000 7fd31759d700 1 --2- 172.21.15.44:0/3228762515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd310107530 0x7fd31010f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.334 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.332+0000 7fd31759d700 1 --2- 172.21.15.44:0/3228762515 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3101024f0 0x7fd3101028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.334 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.332+0000 7fd31759d700 1 --2- 172.21.15.44:0/3228762515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd310102e90 0x7fd310106e50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.334 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.332+0000 7fd31759d700 1 -- 172.21.15.44:0/3228762515 >> 172.21.15.44:0/3228762515 conn(0x7fd3100fdd60 msgr2=0x7fd310100170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:36.334 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.332+0000 7fd31759d700 1 -- 172.21.15.44:0/3228762515 shutdown_connections 2023-11-16T16:54:36.334 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.332+0000 7fd31759d700 1 -- 172.21.15.44:0/3228762515 wait complete. 2023-11-16T16:54:36.335 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.333+0000 7fd31759d700 1 Processor -- start 2023-11-16T16:54:36.335 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.333+0000 7fd31759d700 1 -- start start 2023-11-16T16:54:36.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.334+0000 7fd31759d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3101024f0 0x7fd3101983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.334+0000 7fd31759d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd310102e90 0x7fd3101988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.334+0000 7fd31759d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd310107530 0x7fd31019cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.334+0000 7fd31759d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31019d390 con 0x7fd310107530 2023-11-16T16:54:36.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.334+0000 7fd31759d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31019d500 con 0x7fd310102e90 2023-11-16T16:54:36.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.334+0000 7fd31759d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31019d6c0 con 0x7fd3101024f0 2023-11-16T16:54:36.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.334+0000 7fd315d9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd310102e90 0x7fd3101988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:36.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.334+0000 7fd315d9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd310102e90 0x7fd3101988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60346/0 (socket says 172.21.15.44:60346) 2023-11-16T16:54:36.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.334+0000 7fd315d9a700 1 -- 172.21.15.44:0/270931394 learned_addr learned my addr 172.21.15.44:0/270931394 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:36.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.334+0000 7fd31659b700 1 --2- 172.21.15.44:0/270931394 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3101024f0 0x7fd3101983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:36.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.334+0000 7fd316d9c700 1 --2- 172.21.15.44:0/270931394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd310107530 0x7fd31019cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:36.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.335+0000 7fd315d9a700 1 -- 172.21.15.44:0/270931394 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3101024f0 msgr2=0x7fd3101983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:36.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.335+0000 7fd315d9a700 1 --2- 172.21.15.44:0/270931394 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3101024f0 0x7fd3101983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.335+0000 7fd315d9a700 1 -- 172.21.15.44:0/270931394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd310107530 msgr2=0x7fd31019cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:36.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.335+0000 7fd315d9a700 1 --2- 172.21.15.44:0/270931394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd310107530 0x7fd31019cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.335+0000 7fd315d9a700 1 -- 172.21.15.44:0/270931394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3000095f0 con 0x7fd310102e90 2023-11-16T16:54:36.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.335+0000 7fd315d9a700 1 --2- 172.21.15.44:0/270931394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd310102e90 0x7fd3101988f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd300004710 tx=0x7fd300009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:36.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.335+0000 7fd3077fe700 1 -- 172.21.15.44:0/270931394 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30001d070 con 0x7fd310102e90 2023-11-16T16:54:36.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.335+0000 7fd3077fe700 1 -- 172.21.15.44:0/270931394 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd300021410 con 0x7fd310102e90 2023-11-16T16:54:36.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.335+0000 7fd3077fe700 1 -- 172.21.15.44:0/270931394 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd300017650 con 0x7fd310102e90 2023-11-16T16:54:36.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.335+0000 7fd31759d700 1 -- 172.21.15.44:0/270931394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd310112300 con 0x7fd310102e90 2023-11-16T16:54:36.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.336+0000 7fd31759d700 1 -- 172.21.15.44:0/270931394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd310112170 con 0x7fd310102e90 2023-11-16T16:54:36.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.339+0000 7fd3077fe700 1 -- 172.21.15.44:0/270931394 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fd300021580 con 0x7fd310102e90 2023-11-16T16:54:36.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.340+0000 7fd3077fe700 1 --2- 172.21.15.44:0/270931394 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd2fc0a1340 0x7fd2fc0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.340+0000 7fd32c17ab80 1 Processor -- start 2023-11-16T16:54:36.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.340+0000 7fd3077fe700 1 -- 172.21.15.44:0/270931394 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fd3000ce210 con 0x7fd310102e90 2023-11-16T16:54:36.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.341+0000 7fd32c17ab80 1 -- start start 2023-11-16T16:54:36.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.341+0000 7fd31659b700 1 --2- 172.21.15.44:0/270931394 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd2fc0a1340 0x7fd2fc0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:36.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.341+0000 7fd32c17ab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d218d0 0x55e085d21ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.341+0000 7fd32c17ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d1bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.341+0000 7fd32c17ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d1c100 0x55e085d1c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.341+0000 7fd32c17ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e085c21060 con 0x55e085d1c100 2023-11-16T16:54:36.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.341+0000 7fd32c17ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e085c738a0 con 0x55e085d221e0 2023-11-16T16:54:36.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.341+0000 7fd32c17ab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e085c6cd30 con 0x55e085d218d0 2023-11-16T16:54:36.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.341+0000 7fd315d9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d1bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:36.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.341+0000 7fd315d9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d1bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60354/0 (socket says 172.21.15.44:60354) 2023-11-16T16:54:36.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.341+0000 7fd315d9a700 1 -- 172.21.15.44:0/2434121684 learned_addr learned my addr 172.21.15.44:0/2434121684 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:36.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.342+0000 7fd31659b700 1 --2- 172.21.15.44:0/270931394 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd2fc0a1340 0x7fd2fc0a37f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00f540 tx=0x7fd30c00b040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:36.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.342+0000 7fd315d9a700 1 -- 172.21.15.44:0/2434121684 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d218d0 msgr2=0x55e085d21ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:36.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.342+0000 7fd315d9a700 1 --2- 172.21.15.44:0/2434121684 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d218d0 0x55e085d21ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.342+0000 7fd315d9a700 1 -- 172.21.15.44:0/2434121684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d1c100 msgr2=0x55e085d1c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:36.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.342+0000 7fd315d9a700 1 --2- 172.21.15.44:0/2434121684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d1c100 0x55e085d1c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.342+0000 7fd315d9a700 1 -- 172.21.15.44:0/2434121684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e085aee630 con 0x55e085d221e0 2023-11-16T16:54:36.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.342+0000 7fd315d9a700 1 --2- 172.21.15.44:0/2434121684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d1bbc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd300097b10 tx=0x7fd300097510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28596d216cb52f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:36.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.342+0000 7fd304ff9700 1 -- 172.21.15.44:0/2434121684 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3000ce030 con 0x55e085d221e0 2023-11-16T16:54:36.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.342+0000 7fd304ff9700 1 -- 172.21.15.44:0/2434121684 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd30009f0d0 con 0x55e085d221e0 2023-11-16T16:54:36.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.342+0000 7fd304ff9700 1 -- 172.21.15.44:0/2434121684 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30009a7f0 con 0x55e085d221e0 2023-11-16T16:54:36.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.343+0000 7fd32c17ab80 1 -- 172.21.15.44:0/2434121684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 msgr2=0x55e085d1bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:36.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.343+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/2434121684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d1bbc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd300097b10 tx=0x7fd300097510 comp rx=0 tx=0).stop 2023-11-16T16:54:36.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.343+0000 7fd32c17ab80 1 -- 172.21.15.44:0/2434121684 shutdown_connections 2023-11-16T16:54:36.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.343+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/2434121684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d1c100 0x55e085d1c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.343+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/2434121684 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d218d0 0x55e085d21ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.343+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/2434121684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d1bbc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.343+0000 7fd32c17ab80 1 -- 172.21.15.44:0/2434121684 >> 172.21.15.44:0/2434121684 conn(0x55e085d115b0 msgr2=0x55e085d16ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:36.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.343+0000 7fd32c17ab80 1 -- 172.21.15.44:0/2434121684 shutdown_connections 2023-11-16T16:54:36.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.343+0000 7fd32c17ab80 1 -- 172.21.15.44:0/2434121684 wait complete. 2023-11-16T16:54:36.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.344+0000 7fd32c17ab80 1 -- start start 2023-11-16T16:54:36.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.344+0000 7fd32c17ab80 1 Processor -- start 2023-11-16T16:54:36.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.345+0000 7fd32c17ab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d1c100 0x55e085d28200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.347 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.345+0000 7fd32c17ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d218d0 0x55e085d25480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.345+0000 7fd32c17ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d25b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.345+0000 7fd32c17ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e085c0e8e0 con 0x55e085d218d0 2023-11-16T16:54:36.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.345+0000 7fd32c17ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e085c0e580 con 0x55e085d221e0 2023-11-16T16:54:36.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.345+0000 7fd32c17ab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e085af0920 con 0x55e085d1c100 2023-11-16T16:54:36.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.345+0000 7fd32c17ab80 10 client.0 _ll_register_callbacks cb 0x55e085c2c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:36.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.345+0000 7fd31659b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d1c100 0x55e085d28200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:36.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.345+0000 7fd315d9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d218d0 0x55e085d25480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:36.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.345+0000 7fd31659b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d1c100 0x55e085d28200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:49354/0 (socket says 172.21.15.44:49354) 2023-11-16T16:54:36.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.345+0000 7fd315d9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d218d0 0x55e085d25480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60056/0 (socket says 172.21.15.44:60056) 2023-11-16T16:54:36.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.345+0000 7fd31659b700 1 -- 172.21.15.44:0/2172178865 learned_addr learned my addr 172.21.15.44:0/2172178865 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:36.348 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.345+0000 7fd316d9c700 1 --2- 172.21.15.44:0/2172178865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d25b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:36.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.346+0000 7fd316d9c700 1 -- 172.21.15.44:0/2172178865 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d1c100 msgr2=0x55e085d28200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:36.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.346+0000 7fd316d9c700 1 --2- 172.21.15.44:0/2172178865 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d1c100 0x55e085d28200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.346+0000 7fd316d9c700 1 -- 172.21.15.44:0/2172178865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d218d0 msgr2=0x55e085d25480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:36.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.346+0000 7fd316d9c700 1 --2- 172.21.15.44:0/2172178865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d218d0 0x55e085d25480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.346+0000 7fd316d9c700 1 -- 172.21.15.44:0/2172178865 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd310112170 con 0x55e085d221e0 2023-11-16T16:54:36.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.346+0000 7fd315d9a700 1 --2- 172.21.15.44:0/2172178865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d218d0 0x55e085d25480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:36.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.346+0000 7fd316d9c700 1 --2- 172.21.15.44:0/2172178865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d25b70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd3080116c0 tx=0x7fd3080119d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6aaeabbab85fc915 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:36.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.346+0000 7fd2f9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:54:36.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.346+0000 7fd2f9ffb700 1 -- 172.21.15.44:0/2172178865 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308004a30 con 0x55e085d221e0 2023-11-16T16:54:36.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.346+0000 7fd2f9ffb700 1 -- 172.21.15.44:0/2172178865 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fd308004b90 con 0x55e085d221e0 2023-11-16T16:54:36.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.346+0000 7fd32c17ab80 1 -- 172.21.15.44:0/2172178865 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e085d0ed70 con 0x55e085d221e0 2023-11-16T16:54:36.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.346+0000 7fd2f9ffb700 1 -- 172.21.15.44:0/2172178865 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd308004d50 con 0x55e085d221e0 2023-11-16T16:54:36.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.346+0000 7fd32c17ab80 10 client.26543 fetch_fsmap learned FSMap version 21 2023-11-16T16:54:36.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.347+0000 7fd32c17ab80 1 -- 172.21.15.44:0/2172178865 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55e085c33190 con 0x55e085d221e0 2023-11-16T16:54:36.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.347+0000 7fd2f9ffb700 1 -- 172.21.15.44:0/2172178865 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e085d0ed70 con 0x55e085d221e0 2023-11-16T16:54:36.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.347+0000 7fd2f9ffb700 1 -- 172.21.15.44:0/2172178865 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd3080052f0 con 0x55e085d221e0 2023-11-16T16:54:36.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.347+0000 7fd32c17ab80 10 client.26543 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:54:36.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.347+0000 7fd32c17ab80 10 client.26543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:36.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.347+0000 7fd32c17ab80 20 client.26543 populate_metadata read hostname 'smithi044' 2023-11-16T16:54:36.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.347+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/2172178865 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e085d35890 0x55e085d37ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.347+0000 7fd32c17ab80 4 client.26543 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:54:36.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.347+0000 7fd32c17ab80 1 -- 172.21.15.44:0/2172178865 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e085c2f0a0 con 0x55e085d35890 2023-11-16T16:54:36.352 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.348+0000 7fd31659b700 1 --2- 172.21.15.44:0/2172178865 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e085d35890 0x55e085d37ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:36.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.348+0000 7fd31659b700 1 --2- 172.21.15.44:0/2172178865 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e085d35890 0x55e085d37ca0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c64c5b82652f9a server_cookie=e2ce7b1938b1e00 in_seq=0 out_seq=0 2023-11-16T16:54:36.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.348+0000 7fd2f9ffb700 10 client.26543 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:36.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.349+0000 7fd2f9ffb700 1 -- 172.21.15.44:0/2172178865 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e085c2f0a0 con 0x55e085d35890 2023-11-16T16:54:36.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.349+0000 7fd2f9ffb700 10 client.26543 handle_command_reply: tid=0 2023-11-16T16:54:36.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.349+0000 7fd32c17ab80 1 client.26543 shutdown 2023-11-16T16:54:36.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.349+0000 7fd32c17ab80 1 -- 172.21.15.44:0/2172178865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 msgr2=0x55e085d25b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:36.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.349+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/2172178865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d25b70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd3080116c0 tx=0x7fd3080119d0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.349+0000 7fd32c17ab80 1 -- 172.21.15.44:0/2172178865 shutdown_connections 2023-11-16T16:54:36.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.349+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/2172178865 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e085d35890 0x55e085d37ca0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.349+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/2172178865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d25b70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.349+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/2172178865 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d1c100 0x55e085d28200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.349+0000 7fd2f9ffb700 0 client.26543 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:36.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.349+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/2172178865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d218d0 0x55e085d25480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.349+0000 7fd32c17ab80 1 -- 172.21.15.44:0/2172178865 >> 172.21.15.44:0/2172178865 conn(0x55e085d115b0 msgr2=0x55e085d16030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:36.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.349+0000 7fd32c17ab80 1 -- 172.21.15.44:0/2172178865 shutdown_connections 2023-11-16T16:54:36.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.349+0000 7fd32c17ab80 1 -- 172.21.15.44:0/2172178865 wait complete. 2023-11-16T16:54:36.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.349+0000 7fd32c17ab80 20 client.26543 trim_cache size 0 max 16384 2023-11-16T16:54:36.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.367+0000 7fd32c17ab80 1 Processor -- start 2023-11-16T16:54:36.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.367+0000 7fd32c17ab80 1 -- start start 2023-11-16T16:54:36.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.367+0000 7fd32c17ab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d1c100 0x55e085d1bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.367+0000 7fd32c17ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d218d0 0x55e085d167b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.367+0000 7fd32c17ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d27d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.367+0000 7fd32c17ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e085d20200 con 0x55e085d218d0 2023-11-16T16:54:36.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.367+0000 7fd32c17ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e085d11370 con 0x55e085d221e0 2023-11-16T16:54:36.369 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.367+0000 7fd32c17ab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e085d267d0 con 0x55e085d1c100 2023-11-16T16:54:36.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.368+0000 7fd315d9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d218d0 0x55e085d167b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:36.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.368+0000 7fd316d9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d27d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:36.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.368+0000 7fd316d9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d27d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60384/0 (socket says 172.21.15.44:60384) 2023-11-16T16:54:36.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.368+0000 7fd316d9c700 1 -- 172.21.15.44:0/1759834411 learned_addr learned my addr 172.21.15.44:0/1759834411 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:36.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.368+0000 7fd31659b700 1 --2- 172.21.15.44:0/1759834411 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d1c100 0x55e085d1bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:36.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.368+0000 7fd316d9c700 1 -- 172.21.15.44:0/1759834411 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d1c100 msgr2=0x55e085d1bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:36.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.368+0000 7fd316d9c700 1 --2- 172.21.15.44:0/1759834411 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d1c100 0x55e085d1bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.368+0000 7fd316d9c700 1 -- 172.21.15.44:0/1759834411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d218d0 msgr2=0x55e085d167b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:36.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.368+0000 7fd316d9c700 1 --2- 172.21.15.44:0/1759834411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d218d0 0x55e085d167b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.368+0000 7fd316d9c700 1 -- 172.21.15.44:0/1759834411 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e085c32ee0 con 0x55e085d221e0 2023-11-16T16:54:36.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.368+0000 7fd316d9c700 1 --2- 172.21.15.44:0/1759834411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d27d30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd308000c00 tx=0x7fd308005960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=607f09bee6b6f7ca server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.369+0000 7fd3057fa700 1 -- 172.21.15.44:0/1759834411 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308019690 con 0x55e085d221e0 2023-11-16T16:54:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.369+0000 7fd3057fa700 1 -- 172.21.15.44:0/1759834411 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd308019cb0 con 0x55e085d221e0 2023-11-16T16:54:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.369+0000 7fd3057fa700 1 -- 172.21.15.44:0/1759834411 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308031470 con 0x55e085d221e0 2023-11-16T16:54:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.370+0000 7fd32c17ab80 1 -- 172.21.15.44:0/1759834411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 msgr2=0x55e085d27d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.370+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/1759834411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d27d30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd308000c00 tx=0x7fd308005960 comp rx=0 tx=0).stop 2023-11-16T16:54:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.370+0000 7fd32c17ab80 1 -- 172.21.15.44:0/1759834411 shutdown_connections 2023-11-16T16:54:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.370+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/1759834411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d27d30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.370+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/1759834411 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d1c100 0x55e085d1bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.370+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/1759834411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d218d0 0x55e085d167b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.370+0000 7fd32c17ab80 1 -- 172.21.15.44:0/1759834411 >> 172.21.15.44:0/1759834411 conn(0x55e085d115b0 msgr2=0x55e085d3c600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:36.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.370+0000 7fd32c17ab80 1 -- 172.21.15.44:0/1759834411 shutdown_connections 2023-11-16T16:54:36.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.370+0000 7fd32c17ab80 1 -- 172.21.15.44:0/1759834411 wait complete. 2023-11-16T16:54:36.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.370+0000 7fd32c17ab80 1 -- start start 2023-11-16T16:54:36.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.371+0000 7fd32c17ab80 1 Processor -- start 2023-11-16T16:54:36.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.371+0000 7fd32c17ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d1c100 0x55e085d2c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.371+0000 7fd32c17ab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d218d0 0x55e085d2c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.371+0000 7fd32c17ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d2a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.371+0000 7fd32c17ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e085d2b030 con 0x55e085d1c100 2023-11-16T16:54:36.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.371+0000 7fd32c17ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e085d2b1a0 con 0x55e085d221e0 2023-11-16T16:54:36.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.371+0000 7fd32c17ab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e085d2b360 con 0x55e085d218d0 2023-11-16T16:54:36.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.371+0000 7fd32c17ab80 10 client.0 _ll_register_callbacks cb 0x55e085c2c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:36.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.372+0000 7fd31659b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d1c100 0x55e085d2c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:36.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.372+0000 7fd31659b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d1c100 0x55e085d2c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60078/0 (socket says 172.21.15.44:60078) 2023-11-16T16:54:36.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.372+0000 7fd31659b700 1 -- 172.21.15.44:0/3339523939 learned_addr learned my addr 172.21.15.44:0/3339523939 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:36.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.372+0000 7fd316d9c700 1 --2- 172.21.15.44:0/3339523939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d2a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:36.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.372+0000 7fd315d9a700 1 --2- 172.21.15.44:0/3339523939 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d218d0 0x55e085d2c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:36.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.372+0000 7fd316d9c700 1 -- 172.21.15.44:0/3339523939 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d218d0 msgr2=0x55e085d2c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.372+0000 7fd316d9c700 1 --2- 172.21.15.44:0/3339523939 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d218d0 0x55e085d2c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.372+0000 7fd316d9c700 1 -- 172.21.15.44:0/3339523939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d1c100 msgr2=0x55e085d2c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.372+0000 7fd316d9c700 1 --2- 172.21.15.44:0/3339523939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d1c100 0x55e085d2c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.372+0000 7fd316d9c700 1 -- 172.21.15.44:0/3339523939 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e085c33190 con 0x55e085d221e0 2023-11-16T16:54:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.372+0000 7fd31659b700 1 --2- 172.21.15.44:0/3339523939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d1c100 0x55e085d2c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.372+0000 7fd316d9c700 1 --2- 172.21.15.44:0/3339523939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d2a960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd308005750 tx=0x7fd308019eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd34edc846fd24ba server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.372+0000 7fd2f9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:54:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.373+0000 7fd2f9ffb700 1 -- 172.21.15.44:0/3339523939 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30800edb0 con 0x55e085d221e0 2023-11-16T16:54:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.373+0000 7fd2f9ffb700 1 -- 172.21.15.44:0/3339523939 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fd308019690 con 0x55e085d221e0 2023-11-16T16:54:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.373+0000 7fd32c17ab80 1 -- 172.21.15.44:0/3339523939 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd30009f0d0 con 0x55e085d221e0 2023-11-16T16:54:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.373+0000 7fd2f9ffb700 1 -- 172.21.15.44:0/3339523939 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd308019850 con 0x55e085d221e0 2023-11-16T16:54:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.373+0000 7fd32c17ab80 10 client.26549 fetch_fsmap learned FSMap version 21 2023-11-16T16:54:36.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.373+0000 7fd32c17ab80 1 -- 172.21.15.44:0/3339523939 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55e085a83c60 con 0x55e085d221e0 2023-11-16T16:54:36.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.373+0000 7fd2f9ffb700 1 -- 172.21.15.44:0/3339523939 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd30009f0d0 con 0x55e085d221e0 2023-11-16T16:54:36.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.374+0000 7fd2f9ffb700 1 -- 172.21.15.44:0/3339523939 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd308031d00 con 0x55e085d221e0 2023-11-16T16:54:36.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.374+0000 7fd32c17ab80 10 client.26549 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:54:36.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.374+0000 7fd32c17ab80 10 client.26549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:36.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.374+0000 7fd32c17ab80 20 client.26549 populate_metadata read hostname 'smithi044' 2023-11-16T16:54:36.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.374+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/3339523939 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e085d8a440 0x55e085d8c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:36.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.374+0000 7fd32c17ab80 4 client.26549 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T16:54:36.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.374+0000 7fd32c17ab80 1 -- 172.21.15.44:0/3339523939 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e085c3b660 con 0x55e085d8a440 2023-11-16T16:54:36.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.374+0000 7fd31659b700 1 --2- 172.21.15.44:0/3339523939 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e085d8a440 0x55e085d8c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:36.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.374+0000 7fd31659b700 1 --2- 172.21.15.44:0/3339523939 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e085d8a440 0x55e085d8c820 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd34c70d7320e5cb server_cookie=5ab002fc4869761c in_seq=0 out_seq=0 2023-11-16T16:54:36.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.374+0000 7fd2f9ffb700 10 client.26549 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:36.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.375+0000 7fd2f9ffb700 1 -- 172.21.15.44:0/3339523939 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e085c3b660 con 0x55e085d8a440 2023-11-16T16:54:36.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.375+0000 7fd2f9ffb700 10 client.26549 handle_command_reply: tid=0 2023-11-16T16:54:36.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.375+0000 7fd32c17ab80 1 client.26549 shutdown 2023-11-16T16:54:36.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.375+0000 7fd32c17ab80 1 -- 172.21.15.44:0/3339523939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 msgr2=0x55e085d2a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:36.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.375+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/3339523939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d2a960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd308005750 tx=0x7fd308019eb0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.375+0000 7fd32c17ab80 1 -- 172.21.15.44:0/3339523939 shutdown_connections 2023-11-16T16:54:36.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.375+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/3339523939 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e085d8a440 0x55e085d8c820 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.375+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/3339523939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e085d221e0 0x55e085d2a960 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.375+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/3339523939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e085d1c100 0x55e085d2c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.375+0000 7fd2f9ffb700 0 client.26549 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:36.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.375+0000 7fd32c17ab80 1 --2- 172.21.15.44:0/3339523939 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e085d218d0 0x55e085d2c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.375+0000 7fd32c17ab80 1 -- 172.21.15.44:0/3339523939 >> 172.21.15.44:0/3339523939 conn(0x55e085d115b0 msgr2=0x55e085d1f650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:36.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.375+0000 7fd32c17ab80 1 -- 172.21.15.44:0/3339523939 shutdown_connections 2023-11-16T16:54:36.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.375+0000 7fd32c17ab80 1 -- 172.21.15.44:0/3339523939 wait complete. 2023-11-16T16:54:36.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.375+0000 7fd32c17ab80 20 client.26549 trim_cache size 0 max 16384 2023-11-16T16:54:36.380 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T16:54:36.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.376+0000 7fd31759d700 1 -- 172.21.15.44:0/270931394 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd2fc0a1340 msgr2=0x7fd2fc0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:36.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.376+0000 7fd31759d700 1 --2- 172.21.15.44:0/270931394 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd2fc0a1340 0x7fd2fc0a37f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00f540 tx=0x7fd30c00b040 comp rx=0 tx=0).stop 2023-11-16T16:54:36.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.376+0000 7fd31759d700 1 -- 172.21.15.44:0/270931394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd310102e90 msgr2=0x7fd3101988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:36.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.376+0000 7fd31759d700 1 --2- 172.21.15.44:0/270931394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd310102e90 0x7fd3101988f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd300004710 tx=0x7fd300009580 comp rx=0 tx=0).stop 2023-11-16T16:54:36.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.376+0000 7fd31759d700 1 -- 172.21.15.44:0/270931394 shutdown_connections 2023-11-16T16:54:36.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.376+0000 7fd31759d700 1 --2- 172.21.15.44:0/270931394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd310107530 0x7fd31019cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.376+0000 7fd31759d700 1 --2- 172.21.15.44:0/270931394 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd2fc0a1340 0x7fd2fc0a37f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.376+0000 7fd31759d700 1 --2- 172.21.15.44:0/270931394 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3101024f0 0x7fd3101983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.376+0000 7fd31759d700 1 --2- 172.21.15.44:0/270931394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd310102e90 0x7fd3101988f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:36.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.376+0000 7fd31759d700 1 -- 172.21.15.44:0/270931394 >> 172.21.15.44:0/270931394 conn(0x7fd3100fdd60 msgr2=0x7fd3100fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:36.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.376+0000 7fd31759d700 1 -- 172.21.15.44:0/270931394 shutdown_connections 2023-11-16T16:54:36.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:36.376+0000 7fd31759d700 1 -- 172.21.15.44:0/270931394 wait complete. 2023-11-16T16:54:36.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:36 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/542259461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:54:37.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:36 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/542259461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:54:37.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:36 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/542259461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:54:37.399 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:54:37.399 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:54:37.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.639+0000 7fd87e9ff700 1 -- 172.21.15.44:0/3769278545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd878101660 msgr2=0x7fd878104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:37.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.639+0000 7fd87e9ff700 1 --2- 172.21.15.44:0/3769278545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd878101660 0x7fd878104ef0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd86c0099c0 tx=0x7fd86c009cd0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.639+0000 7fd87e9ff700 1 -- 172.21.15.44:0/3769278545 shutdown_connections 2023-11-16T16:54:37.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.639+0000 7fd87e9ff700 1 --2- 172.21.15.44:0/3769278545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd878105620 0x7fd87810d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.639+0000 7fd87e9ff700 1 --2- 172.21.15.44:0/3769278545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd878100cc0 0x7fd878101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.639+0000 7fd87e9ff700 1 --2- 172.21.15.44:0/3769278545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd878101660 0x7fd878104ef0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.639+0000 7fd87e9ff700 1 -- 172.21.15.44:0/3769278545 >> 172.21.15.44:0/3769278545 conn(0x7fd878077560 msgr2=0x7fd878077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:37.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.639+0000 7fd87e9ff700 1 -- 172.21.15.44:0/3769278545 shutdown_connections 2023-11-16T16:54:37.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.639+0000 7fd87e9ff700 1 -- 172.21.15.44:0/3769278545 wait complete. 2023-11-16T16:54:37.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.640+0000 7fd87e9ff700 1 Processor -- start 2023-11-16T16:54:37.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.640+0000 7fd87e9ff700 1 -- start start 2023-11-16T16:54:37.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.641+0000 7fd87e9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd878100cc0 0x7fd8781986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.641+0000 7fd87e9ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd878101660 0x7fd878198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.641+0000 7fd87e9ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd878105620 0x7fd87819bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.641+0000 7fd87e9ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87819c590 con 0x7fd878101660 2023-11-16T16:54:37.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.641+0000 7fd87e9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87819c700 con 0x7fd878100cc0 2023-11-16T16:54:37.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.641+0000 7fd87e9ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87819c8c0 con 0x7fd878105620 2023-11-16T16:54:37.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.641+0000 7fd877fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd878100cc0 0x7fd8781986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.641+0000 7fd877fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd878100cc0 0x7fd8781986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60422/0 (socket says 172.21.15.44:60422) 2023-11-16T16:54:37.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.641+0000 7fd877fff700 1 -- 172.21.15.44:0/3038968025 learned_addr learned my addr 172.21.15.44:0/3038968025 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:37.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.641+0000 7fd8777fe700 1 --2- 172.21.15.44:0/3038968025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd878101660 0x7fd878198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.641+0000 7fd87cf9c700 1 --2- 172.21.15.44:0/3038968025 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd878105620 0x7fd87819bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.641+0000 7fd877fff700 1 -- 172.21.15.44:0/3038968025 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd878105620 msgr2=0x7fd87819bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:37.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.641+0000 7fd877fff700 1 --2- 172.21.15.44:0/3038968025 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd878105620 0x7fd87819bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.641+0000 7fd877fff700 1 -- 172.21.15.44:0/3038968025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd878101660 msgr2=0x7fd878198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:37.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.641+0000 7fd877fff700 1 --2- 172.21.15.44:0/3038968025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd878101660 0x7fd878198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.641+0000 7fd877fff700 1 -- 172.21.15.44:0/3038968025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c0095f0 con 0x7fd878100cc0 2023-11-16T16:54:37.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.642+0000 7fd877fff700 1 --2- 172.21.15.44:0/3038968025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd878100cc0 0x7fd8781986a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd86800e9a0 tx=0x7fd86800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:37.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.642+0000 7fd8757fa700 1 -- 172.21.15.44:0/3038968025 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868009790 con 0x7fd878100cc0 2023-11-16T16:54:37.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.642+0000 7fd8757fa700 1 -- 172.21.15.44:0/3038968025 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd868010430 con 0x7fd878100cc0 2023-11-16T16:54:37.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.642+0000 7fd87e9ff700 1 -- 172.21.15.44:0/3038968025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd878110370 con 0x7fd878100cc0 2023-11-16T16:54:37.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.642+0000 7fd8757fa700 1 -- 172.21.15.44:0/3038968025 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868018680 con 0x7fd878100cc0 2023-11-16T16:54:37.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.642+0000 7fd87e9ff700 1 -- 172.21.15.44:0/3038968025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8781101e0 con 0x7fd878100cc0 2023-11-16T16:54:37.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.645+0000 7fd8757fa700 1 -- 172.21.15.44:0/3038968025 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fd8680105a0 con 0x7fd878100cc0 2023-11-16T16:54:37.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.646+0000 7fd892346b80 1 Processor -- start 2023-11-16T16:54:37.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.646+0000 7fd8757fa700 1 --2- 172.21.15.44:0/3038968025 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd8600a12c0 0x7fd8600a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.647+0000 7fd8757fa700 1 -- 172.21.15.44:0/3038968025 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fd868014070 con 0x7fd878100cc0 2023-11-16T16:54:37.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.647+0000 7fd892346b80 1 -- start start 2023-11-16T16:54:37.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.647+0000 7fd87cf9c700 1 --2- 172.21.15.44:0/3038968025 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd8600a12c0 0x7fd8600a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.647+0000 7fd892346b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdabe0 0x55ee25fdafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.647+0000 7fd892346b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdb4f0 0x55ee25fdf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.647+0000 7fd892346b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdf930 0x55ee25fdfd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.647+0000 7fd892346b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee25ee00e0 con 0x55ee25fdb4f0 2023-11-16T16:54:37.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.647+0000 7fd892346b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee25f32920 con 0x55ee25fdf930 2023-11-16T16:54:37.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.647+0000 7fd892346b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee25f2bdb0 con 0x55ee25fdabe0 2023-11-16T16:54:37.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.647+0000 7fd87cf9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdf930 0x55ee25fdfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.647+0000 7fd8777fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdb4f0 0x55ee25fdf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.647+0000 7fd8777fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdb4f0 0x55ee25fdf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60106/0 (socket says 172.21.15.44:60106) 2023-11-16T16:54:37.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.647+0000 7fd87cf9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdf930 0x55ee25fdfd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60428/0 (socket says 172.21.15.44:60428) 2023-11-16T16:54:37.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.647+0000 7fd8777fe700 1 -- 172.21.15.44:0/1417685540 learned_addr learned my addr 172.21.15.44:0/1417685540 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:37.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.647+0000 7fd877fff700 1 --2- 172.21.15.44:0/1417685540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdabe0 0x55ee25fdafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.648+0000 7fd87cf9c700 1 --2- 172.21.15.44:0/3038968025 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd8600a12c0 0x7fd8600a3770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd870003ee0 tx=0x7fd870009100 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:37.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.648+0000 7fd877fff700 1 -- 172.21.15.44:0/1417685540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdf930 msgr2=0x55ee25fdfd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:37.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.648+0000 7fd877fff700 1 --2- 172.21.15.44:0/1417685540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdf930 0x55ee25fdfd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.648+0000 7fd877fff700 1 -- 172.21.15.44:0/1417685540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdb4f0 msgr2=0x55ee25fdf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:37.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.648+0000 7fd877fff700 1 --2- 172.21.15.44:0/1417685540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdb4f0 0x55ee25fdf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.648+0000 7fd877fff700 1 -- 172.21.15.44:0/1417685540 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee25dad6b0 con 0x55ee25fdabe0 2023-11-16T16:54:37.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.648+0000 7fd8777fe700 1 --2- 172.21.15.44:0/1417685540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdb4f0 0x55ee25fdf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:37.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.648+0000 7fd877fff700 1 --2- 172.21.15.44:0/1417685540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdabe0 0x55ee25fdafb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd8680ca630 tx=0x7fd868099c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b3e9eb86b54b59e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:37.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.648+0000 7fd85e7fc700 1 -- 172.21.15.44:0/1417685540 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8680a2400 con 0x55ee25fdabe0 2023-11-16T16:54:37.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.648+0000 7fd85e7fc700 1 -- 172.21.15.44:0/1417685540 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd868010880 con 0x55ee25fdabe0 2023-11-16T16:54:37.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.648+0000 7fd85e7fc700 1 -- 172.21.15.44:0/1417685540 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8680a1f70 con 0x55ee25fdabe0 2023-11-16T16:54:37.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.649+0000 7fd892346b80 1 -- 172.21.15.44:0/1417685540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdabe0 msgr2=0x55ee25fdafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:37.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.649+0000 7fd892346b80 1 --2- 172.21.15.44:0/1417685540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdabe0 0x55ee25fdafb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd8680ca630 tx=0x7fd868099c00 comp rx=0 tx=0).stop 2023-11-16T16:54:37.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.649+0000 7fd892346b80 1 -- 172.21.15.44:0/1417685540 shutdown_connections 2023-11-16T16:54:37.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.649+0000 7fd892346b80 1 --2- 172.21.15.44:0/1417685540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdf930 0x55ee25fdfd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.649+0000 7fd892346b80 1 --2- 172.21.15.44:0/1417685540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdabe0 0x55ee25fdafb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.649+0000 7fd892346b80 1 --2- 172.21.15.44:0/1417685540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdb4f0 0x55ee25fdf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.649+0000 7fd892346b80 1 -- 172.21.15.44:0/1417685540 >> 172.21.15.44:0/1417685540 conn(0x55ee25fd03e0 msgr2=0x55ee25fd5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:37.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.649+0000 7fd892346b80 1 -- 172.21.15.44:0/1417685540 shutdown_connections 2023-11-16T16:54:37.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.649+0000 7fd892346b80 1 -- 172.21.15.44:0/1417685540 wait complete. 2023-11-16T16:54:37.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.650+0000 7fd892346b80 1 -- start start 2023-11-16T16:54:37.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.650+0000 7fd892346b80 1 Processor -- start 2023-11-16T16:54:37.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.650+0000 7fd892346b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdabe0 0x55ee25fe4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.650+0000 7fd892346b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdb4f0 0x55ee25fe4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd892346b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdf930 0x55ee25fe4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd892346b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee25ecd960 con 0x55ee25fdabe0 2023-11-16T16:54:37.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd892346b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee25ecd600 con 0x55ee25fdf930 2023-11-16T16:54:37.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd892346b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee25daf9a0 con 0x55ee25fdb4f0 2023-11-16T16:54:37.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd892346b80 10 client.0 _ll_register_callbacks cb 0x55ee25eeb800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:37.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd877fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdabe0 0x55ee25fe4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd877fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdabe0 0x55ee25fe4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60112/0 (socket says 172.21.15.44:60112) 2023-11-16T16:54:37.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd877fff700 1 -- 172.21.15.44:0/1480277094 learned_addr learned my addr 172.21.15.44:0/1480277094 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:37.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd8777fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdb4f0 0x55ee25fe4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd87cf9c700 1 --2- 172.21.15.44:0/1480277094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdf930 0x55ee25fe4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.660 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd877fff700 1 -- 172.21.15.44:0/1480277094 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdb4f0 msgr2=0x55ee25fe4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd877fff700 1 --2- 172.21.15.44:0/1480277094 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdb4f0 0x55ee25fe4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd877fff700 1 -- 172.21.15.44:0/1480277094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdf930 msgr2=0x55ee25fe4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd877fff700 1 --2- 172.21.15.44:0/1480277094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdf930 0x55ee25fe4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd877fff700 1 -- 172.21.15.44:0/1480277094 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8781101e0 con 0x55ee25fdabe0 2023-11-16T16:54:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd877fff700 1 --2- 172.21.15.44:0/1480277094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdabe0 0x55ee25fe4410 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd86800e970 tx=0x7fd868099750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b45f7550fca4ac9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.651+0000 7fd8577fe700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:54:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.652+0000 7fd8577fe700 1 -- 172.21.15.44:0/1480277094 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8680a2400 con 0x55ee25fdabe0 2023-11-16T16:54:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.652+0000 7fd8577fe700 1 -- 172.21.15.44:0/1480277094 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fd8680a2180 con 0x55ee25fdabe0 2023-11-16T16:54:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.652+0000 7fd892346b80 1 -- 172.21.15.44:0/1480277094 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ee25fcdba0 con 0x55ee25fdabe0 2023-11-16T16:54:37.661 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.652+0000 7fd8577fe700 1 -- 172.21.15.44:0/1480277094 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd868094bc0 con 0x55ee25fdabe0 2023-11-16T16:54:37.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.652+0000 7fd892346b80 10 client.16527 fetch_fsmap learned FSMap version 21 2023-11-16T16:54:37.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.652+0000 7fd892346b80 1 -- 172.21.15.44:0/1480277094 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55ee25ef2210 con 0x55ee25fdabe0 2023-11-16T16:54:37.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.652+0000 7fd8577fe700 1 -- 172.21.15.44:0/1480277094 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8680b8030 con 0x55ee25fdabe0 2023-11-16T16:54:37.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.652+0000 7fd8577fe700 1 -- 172.21.15.44:0/1480277094 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd868098f10 con 0x55ee25fdabe0 2023-11-16T16:54:37.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.652+0000 7fd892346b80 10 client.16527 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:54:37.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.652+0000 7fd892346b80 10 client.16527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:37.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.652+0000 7fd892346b80 20 client.16527 populate_metadata read hostname 'smithi044' 2023-11-16T16:54:37.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.652+0000 7fd892346b80 1 --2- 172.21.15.44:0/1480277094 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ee25ff4640 0x55ee25ff6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.662 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.652+0000 7fd892346b80 4 client.16527 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:54:37.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.652+0000 7fd892346b80 1 -- 172.21.15.44:0/1480277094 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ee25eee120 con 0x55ee25ff4640 2023-11-16T16:54:37.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.653+0000 7fd8777fe700 1 --2- 172.21.15.44:0/1480277094 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ee25ff4640 0x55ee25ff6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.653+0000 7fd8777fe700 1 --2- 172.21.15.44:0/1480277094 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ee25ff4640 0x55ee25ff6a20 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d08634f2f3ec2ea server_cookie=7c8cadb5fd90209c in_seq=0 out_seq=0 2023-11-16T16:54:37.663 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.653+0000 7fd8577fe700 10 client.16527 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:37.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.677+0000 7fd8577fe700 1 -- 172.21.15.44:0/1480277094 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ee25eee120 con 0x55ee25ff4640 2023-11-16T16:54:37.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.677+0000 7fd8577fe700 10 client.16527 handle_command_reply: tid=0 2023-11-16T16:54:37.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.677+0000 7fd892346b80 1 client.16527 shutdown 2023-11-16T16:54:37.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.677+0000 7fd892346b80 1 -- 172.21.15.44:0/1480277094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdabe0 msgr2=0x55ee25fe4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:37.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.677+0000 7fd892346b80 1 --2- 172.21.15.44:0/1480277094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdabe0 0x55ee25fe4410 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd86800e970 tx=0x7fd868099750 comp rx=0 tx=0).stop 2023-11-16T16:54:37.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.678+0000 7fd892346b80 1 -- 172.21.15.44:0/1480277094 shutdown_connections 2023-11-16T16:54:37.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.678+0000 7fd892346b80 1 --2- 172.21.15.44:0/1480277094 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ee25ff4640 0x55ee25ff6a20 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.678+0000 7fd892346b80 1 --2- 172.21.15.44:0/1480277094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdf930 0x55ee25fe4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.678+0000 7fd892346b80 1 --2- 172.21.15.44:0/1480277094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdabe0 0x55ee25fe4410 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.681 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.678+0000 7fd892346b80 1 --2- 172.21.15.44:0/1480277094 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdb4f0 0x55ee25fe4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.681 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.678+0000 7fd8577fe700 0 client.16527 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:37.681 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.678+0000 7fd892346b80 1 -- 172.21.15.44:0/1480277094 >> 172.21.15.44:0/1480277094 conn(0x55ee25fd03e0 msgr2=0x55ee25fe0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:37.681 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.678+0000 7fd892346b80 1 -- 172.21.15.44:0/1480277094 shutdown_connections 2023-11-16T16:54:37.681 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.678+0000 7fd892346b80 1 -- 172.21.15.44:0/1480277094 wait complete. 2023-11-16T16:54:37.681 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.679+0000 7fd892346b80 20 client.16527 trim_cache size 0 max 16384 2023-11-16T16:54:37.702 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.700+0000 7fd892346b80 1 Processor -- start 2023-11-16T16:54:37.702 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.700+0000 7fd892346b80 1 -- start start 2023-11-16T16:54:37.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.701+0000 7fd892346b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdabe0 0x55ee25fdd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.701+0000 7fd892346b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdb4f0 0x55ee25fdf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.701+0000 7fd892346b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdf930 0x55ee25fe4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.701+0000 7fd892346b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee25fd7300 con 0x55ee25fdf930 2023-11-16T16:54:37.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.701+0000 7fd892346b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee25fdb980 con 0x55ee25fdb4f0 2023-11-16T16:54:37.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.701+0000 7fd892346b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee25fe2410 con 0x55ee25fdabe0 2023-11-16T16:54:37.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.701+0000 7fd8777fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdb4f0 0x55ee25fdf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.701+0000 7fd877fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdabe0 0x55ee25fdd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.701+0000 7fd87cf9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdf930 0x55ee25fe4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.701+0000 7fd8777fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdb4f0 0x55ee25fdf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60454/0 (socket says 172.21.15.44:60454) 2023-11-16T16:54:37.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.701+0000 7fd877fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdabe0 0x55ee25fdd220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:49420/0 (socket says 172.21.15.44:49420) 2023-11-16T16:54:37.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.701+0000 7fd8777fe700 1 -- 172.21.15.44:0/2134049561 learned_addr learned my addr 172.21.15.44:0/2134049561 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:37.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.702+0000 7fd8777fe700 1 -- 172.21.15.44:0/2134049561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdabe0 msgr2=0x55ee25fdd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:37.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.702+0000 7fd8777fe700 1 --2- 172.21.15.44:0/2134049561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdabe0 0x55ee25fdd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.702+0000 7fd8777fe700 1 -- 172.21.15.44:0/2134049561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdf930 msgr2=0x55ee25fe4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:37.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.702+0000 7fd8777fe700 1 --2- 172.21.15.44:0/2134049561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdf930 0x55ee25fe4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.702+0000 7fd8777fe700 1 -- 172.21.15.44:0/2134049561 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee25fe0220 con 0x55ee25fdb4f0 2023-11-16T16:54:37.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.702+0000 7fd877fff700 1 --2- 172.21.15.44:0/2134049561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdabe0 0x55ee25fdd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:37.705 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.702+0000 7fd87cf9c700 1 --2- 172.21.15.44:0/2134049561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdf930 0x55ee25fe4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:37.706 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.702+0000 7fd8777fe700 1 --2- 172.21.15.44:0/2134049561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdb4f0 0x55ee25fdf2a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00b580 tx=0x7fd86c009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d5b1a3abac33cd3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:37.706 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.702+0000 7fd85effd700 1 -- 172.21.15.44:0/2134049561 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c02c070 con 0x55ee25fdb4f0 2023-11-16T16:54:37.706 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.702+0000 7fd85effd700 1 -- 172.21.15.44:0/2134049561 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd86c01e040 con 0x55ee25fdb4f0 2023-11-16T16:54:37.706 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.702+0000 7fd85effd700 1 -- 172.21.15.44:0/2134049561 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c003ae0 con 0x55ee25fdb4f0 2023-11-16T16:54:37.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.703+0000 7fd892346b80 1 -- 172.21.15.44:0/2134049561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdb4f0 msgr2=0x55ee25fdf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:37.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.703+0000 7fd892346b80 1 --2- 172.21.15.44:0/2134049561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdb4f0 0x55ee25fdf2a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00b580 tx=0x7fd86c009510 comp rx=0 tx=0).stop 2023-11-16T16:54:37.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.703+0000 7fd892346b80 1 -- 172.21.15.44:0/2134049561 shutdown_connections 2023-11-16T16:54:37.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.703+0000 7fd892346b80 1 --2- 172.21.15.44:0/2134049561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdf930 0x55ee25fe4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.703+0000 7fd892346b80 1 --2- 172.21.15.44:0/2134049561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdabe0 0x55ee25fdd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.709 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.703+0000 7fd892346b80 1 --2- 172.21.15.44:0/2134049561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdb4f0 0x55ee25fdf2a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.703+0000 7fd892346b80 1 -- 172.21.15.44:0/2134049561 >> 172.21.15.44:0/2134049561 conn(0x55ee25fd03e0 msgr2=0x55ee25fd99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:37.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.703+0000 7fd892346b80 1 -- 172.21.15.44:0/2134049561 shutdown_connections 2023-11-16T16:54:37.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.703+0000 7fd892346b80 1 -- 172.21.15.44:0/2134049561 wait complete. 2023-11-16T16:54:37.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.703+0000 7fd892346b80 1 -- start start 2023-11-16T16:54:37.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.704+0000 7fd892346b80 1 Processor -- start 2023-11-16T16:54:37.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.704+0000 7fd892346b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdabe0 0x55ee25fea920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.704+0000 7fd892346b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdb4f0 0x55ee25feae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.704+0000 7fd892346b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdf930 0x55ee26042470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.704+0000 7fd892346b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee260429b0 con 0x55ee25fdabe0 2023-11-16T16:54:37.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.704+0000 7fd892346b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee26042af0 con 0x55ee25fdb4f0 2023-11-16T16:54:37.710 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.704+0000 7fd892346b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee26042cb0 con 0x55ee25fdf930 2023-11-16T16:54:37.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.704+0000 7fd892346b80 10 client.0 _ll_register_callbacks cb 0x55ee25eeb800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:37.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.704+0000 7fd877fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdabe0 0x55ee25fea920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.704+0000 7fd8777fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdb4f0 0x55ee25feae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.704+0000 7fd8777fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdb4f0 0x55ee25feae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60466/0 (socket says 172.21.15.44:60466) 2023-11-16T16:54:37.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.704+0000 7fd877fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdabe0 0x55ee25fea920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60124/0 (socket says 172.21.15.44:60124) 2023-11-16T16:54:37.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.704+0000 7fd8777fe700 1 -- 172.21.15.44:0/3978768054 learned_addr learned my addr 172.21.15.44:0/3978768054 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:37.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.704+0000 7fd87cf9c700 1 --2- 172.21.15.44:0/3978768054 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdf930 0x55ee26042470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.705+0000 7fd8777fe700 1 -- 172.21.15.44:0/3978768054 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdf930 msgr2=0x55ee26042470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:37.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.705+0000 7fd8777fe700 1 --2- 172.21.15.44:0/3978768054 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdf930 0x55ee26042470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.705+0000 7fd8777fe700 1 -- 172.21.15.44:0/3978768054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdabe0 msgr2=0x55ee25fea920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:37.711 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.705+0000 7fd8777fe700 1 --2- 172.21.15.44:0/3978768054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdabe0 0x55ee25fea920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.705+0000 7fd8777fe700 1 -- 172.21.15.44:0/3978768054 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd870004420 con 0x55ee25fdb4f0 2023-11-16T16:54:37.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.705+0000 7fd8777fe700 1 --2- 172.21.15.44:0/3978768054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdb4f0 0x55ee25feae60 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd86c000c00 tx=0x7fd86c028b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4e43a8f73f3124f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:37.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.705+0000 7fd8577fe700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:54:37.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.705+0000 7fd8577fe700 1 -- 172.21.15.44:0/3978768054 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c02c040 con 0x55ee25fdb4f0 2023-11-16T16:54:37.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.705+0000 7fd8577fe700 1 -- 172.21.15.44:0/3978768054 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fd86c028560 con 0x55ee25fdb4f0 2023-11-16T16:54:37.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.705+0000 7fd892346b80 1 -- 172.21.15.44:0/3978768054 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd868010880 con 0x55ee25fdb4f0 2023-11-16T16:54:37.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.706+0000 7fd8577fe700 1 -- 172.21.15.44:0/3978768054 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd86c0037d0 con 0x55ee25fdb4f0 2023-11-16T16:54:37.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.706+0000 7fd892346b80 10 client.26567 fetch_fsmap learned FSMap version 21 2023-11-16T16:54:37.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.706+0000 7fd892346b80 1 -- 172.21.15.44:0/3978768054 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x7fd86c005550 con 0x55ee25fdb4f0 2023-11-16T16:54:37.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.706+0000 7fd8577fe700 1 -- 172.21.15.44:0/3978768054 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd86c03a080 con 0x55ee25fdb4f0 2023-11-16T16:54:37.712 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.706+0000 7fd8577fe700 1 -- 172.21.15.44:0/3978768054 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd86c02f070 con 0x55ee25fdb4f0 2023-11-16T16:54:37.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.706+0000 7fd892346b80 10 client.26567 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:54:37.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.706+0000 7fd892346b80 10 client.26567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:37.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.706+0000 7fd892346b80 20 client.26567 populate_metadata read hostname 'smithi044' 2023-11-16T16:54:37.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.706+0000 7fd892346b80 1 --2- 172.21.15.44:0/3978768054 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ee2604a030 0x55ee2604c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:37.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.706+0000 7fd892346b80 4 client.26567 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:54:37.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.706+0000 7fd892346b80 1 -- 172.21.15.44:0/3978768054 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55ee25efa6e0 con 0x55ee2604a030 2023-11-16T16:54:37.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.706+0000 7fd877fff700 1 --2- 172.21.15.44:0/3978768054 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ee2604a030 0x55ee2604c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:37.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.707+0000 7fd877fff700 1 --2- 172.21.15.44:0/3978768054 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ee2604a030 0x55ee2604c410 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92895b06a54b053b server_cookie=d4f6f2f96dbb538f in_seq=0 out_seq=0 2023-11-16T16:54:37.713 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.707+0000 7fd8577fe700 10 client.26567 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:37.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.833+0000 7fd8577fe700 1 -- 172.21.15.44:0/3978768054 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55ee25efa6e0 con 0x55ee2604a030 2023-11-16T16:54:37.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.833+0000 7fd8577fe700 10 client.26567 handle_command_reply: tid=0 2023-11-16T16:54:37.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.833+0000 7fd892346b80 1 client.26567 shutdown 2023-11-16T16:54:37.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.833+0000 7fd892346b80 1 -- 172.21.15.44:0/3978768054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdb4f0 msgr2=0x55ee25feae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:37.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.833+0000 7fd892346b80 1 --2- 172.21.15.44:0/3978768054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdb4f0 0x55ee25feae60 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd86c000c00 tx=0x7fd86c028b30 comp rx=0 tx=0).stop 2023-11-16T16:54:37.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.833+0000 7fd892346b80 1 -- 172.21.15.44:0/3978768054 shutdown_connections 2023-11-16T16:54:37.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.833+0000 7fd892346b80 1 --2- 172.21.15.44:0/3978768054 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55ee2604a030 0x55ee2604c410 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.833+0000 7fd892346b80 1 --2- 172.21.15.44:0/3978768054 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ee25fdf930 0x55ee26042470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.833+0000 7fd892346b80 1 --2- 172.21.15.44:0/3978768054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ee25fdabe0 0x55ee25fea920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.833+0000 7fd8577fe700 0 client.26567 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:37.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.833+0000 7fd892346b80 1 --2- 172.21.15.44:0/3978768054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee25fdb4f0 0x55ee25feae60 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.833+0000 7fd892346b80 1 -- 172.21.15.44:0/3978768054 >> 172.21.15.44:0/3978768054 conn(0x55ee25fd03e0 msgr2=0x55ee25fdf110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:37.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.834+0000 7fd892346b80 1 -- 172.21.15.44:0/3978768054 shutdown_connections 2023-11-16T16:54:37.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.834+0000 7fd892346b80 1 -- 172.21.15.44:0/3978768054 wait complete. 2023-11-16T16:54:37.837 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.834+0000 7fd892346b80 20 client.26567 trim_cache size 0 max 16384 2023-11-16T16:54:37.837 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:54:37.837 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:54:37.837 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "c6d82195-2d53-46f4-9f2b-dc4452b4a1b0", 2023-11-16T16:54:37.837 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:54:37.837 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:54:37.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.835+0000 7fd87e9ff700 1 -- 172.21.15.44:0/3038968025 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd8600a12c0 msgr2=0x7fd8600a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:37.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.835+0000 7fd87e9ff700 1 --2- 172.21.15.44:0/3038968025 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd8600a12c0 0x7fd8600a3770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd870003ee0 tx=0x7fd870009100 comp rx=0 tx=0).stop 2023-11-16T16:54:37.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.835+0000 7fd87e9ff700 1 -- 172.21.15.44:0/3038968025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd878100cc0 msgr2=0x7fd8781986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:37.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.835+0000 7fd87e9ff700 1 --2- 172.21.15.44:0/3038968025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd878100cc0 0x7fd8781986a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd86800e9a0 tx=0x7fd86800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.835+0000 7fd87e9ff700 1 -- 172.21.15.44:0/3038968025 shutdown_connections 2023-11-16T16:54:37.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.835+0000 7fd87e9ff700 1 --2- 172.21.15.44:0/3038968025 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd8600a12c0 0x7fd8600a3770 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.835+0000 7fd87e9ff700 1 --2- 172.21.15.44:0/3038968025 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd878105620 0x7fd87819bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.835+0000 7fd87e9ff700 1 --2- 172.21.15.44:0/3038968025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd878100cc0 0x7fd8781986a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.835+0000 7fd87e9ff700 1 --2- 172.21.15.44:0/3038968025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd878101660 0x7fd878198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:37.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.835+0000 7fd87e9ff700 1 -- 172.21.15.44:0/3038968025 >> 172.21.15.44:0/3038968025 conn(0x7fd878077560 msgr2=0x7fd8780fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:37.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.835+0000 7fd87e9ff700 1 -- 172.21.15.44:0/3038968025 shutdown_connections 2023-11-16T16:54:37.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:37.835+0000 7fd87e9ff700 1 -- 172.21.15.44:0/3038968025 wait complete. 2023-11-16T16:54:37.854 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:37 smithi044 ceph-mon[128750]: pgmap v143: 129 pgs: 129 active+clean; 7.7 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 2.68k op/s 2023-11-16T16:54:37.857 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:54:37.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:37 smithi077 ceph-mon[131204]: pgmap v143: 129 pgs: 129 active+clean; 7.7 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 2.68k op/s 2023-11-16T16:54:38.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.066+0000 7f21573ff700 1 -- 172.21.15.44:0/3028181714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2150101660 msgr2=0x7f2150104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:38.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.066+0000 7f21573ff700 1 --2- 172.21.15.44:0/3028181714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2150101660 0x7f2150104ef0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f21400099c0 tx=0x7f2140009cd0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.067+0000 7f21573ff700 1 -- 172.21.15.44:0/3028181714 shutdown_connections 2023-11-16T16:54:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.067+0000 7f21573ff700 1 --2- 172.21.15.44:0/3028181714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2150105620 0x7f215010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.067+0000 7f21573ff700 1 --2- 172.21.15.44:0/3028181714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2150100cc0 0x7f2150101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.067+0000 7f21573ff700 1 --2- 172.21.15.44:0/3028181714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2150101660 0x7f2150104ef0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.067+0000 7f21573ff700 1 -- 172.21.15.44:0/3028181714 >> 172.21.15.44:0/3028181714 conn(0x7f2150077560 msgr2=0x7f2150077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.067+0000 7f21573ff700 1 -- 172.21.15.44:0/3028181714 shutdown_connections 2023-11-16T16:54:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.067+0000 7f21573ff700 1 -- 172.21.15.44:0/3028181714 wait complete. 2023-11-16T16:54:38.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.068+0000 7f21573ff700 1 Processor -- start 2023-11-16T16:54:38.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.068+0000 7f21573ff700 1 -- start start 2023-11-16T16:54:38.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.068+0000 7f21573ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2150100cc0 0x7f21501986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.068+0000 7f21573ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2150101660 0x7f2150198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.068+0000 7f21573ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2150105620 0x7f215019bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.068+0000 7f21573ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215019c5e0 con 0x7f2150100cc0 2023-11-16T16:54:38.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.068+0000 7f21573ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215019c750 con 0x7f2150105620 2023-11-16T16:54:38.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.068+0000 7f21573ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215019c910 con 0x7f2150101660 2023-11-16T16:54:38.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.069+0000 7f215519b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2150100cc0 0x7f21501986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.069+0000 7f215599c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2150105620 0x7f215019bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.069+0000 7f215599c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2150105620 0x7f215019bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60496/0 (socket says 172.21.15.44:60496) 2023-11-16T16:54:38.071 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.069+0000 7f215519b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2150100cc0 0x7f21501986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60142/0 (socket says 172.21.15.44:60142) 2023-11-16T16:54:38.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.069+0000 7f215599c700 1 -- 172.21.15.44:0/3017777157 learned_addr learned my addr 172.21.15.44:0/3017777157 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:38.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.069+0000 7f215499a700 1 --2- 172.21.15.44:0/3017777157 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2150101660 0x7f2150198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.069+0000 7f215599c700 1 -- 172.21.15.44:0/3017777157 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2150101660 msgr2=0x7f2150198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:38.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.069+0000 7f215599c700 1 --2- 172.21.15.44:0/3017777157 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2150101660 0x7f2150198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.069+0000 7f215599c700 1 -- 172.21.15.44:0/3017777157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2150100cc0 msgr2=0x7f21501986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:38.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.069+0000 7f215599c700 1 --2- 172.21.15.44:0/3017777157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2150100cc0 0x7f21501986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.069+0000 7f215599c700 1 -- 172.21.15.44:0/3017777157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2148009c40 con 0x7f2150105620 2023-11-16T16:54:38.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.069+0000 7f215519b700 1 --2- 172.21.15.44:0/3017777157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2150100cc0 0x7f21501986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:38.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.069+0000 7f215599c700 1 --2- 172.21.15.44:0/3017777157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2150105620 0x7f215019bfe0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f214800c370 tx=0x7f214800c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:38.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.069+0000 7f21467fc700 1 -- 172.21.15.44:0/3017777157 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2148017070 con 0x7f2150105620 2023-11-16T16:54:38.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.070+0000 7f21467fc700 1 -- 172.21.15.44:0/3017777157 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f214800f040 con 0x7f2150105620 2023-11-16T16:54:38.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.070+0000 7f21467fc700 1 -- 172.21.15.44:0/3017777157 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214800e050 con 0x7f2150105620 2023-11-16T16:54:38.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.070+0000 7f21573ff700 1 -- 172.21.15.44:0/3017777157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21400095f0 con 0x7f2150105620 2023-11-16T16:54:38.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.070+0000 7f21573ff700 1 -- 172.21.15.44:0/3017777157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2150110370 con 0x7f2150105620 2023-11-16T16:54:38.074 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.073+0000 7f21467fc700 1 -- 172.21.15.44:0/3017777157 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f214800cb60 con 0x7f2150105620 2023-11-16T16:54:38.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.074+0000 7f216ade3b80 1 Processor -- start 2023-11-16T16:54:38.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.074+0000 7f216ade3b80 1 -- start start 2023-11-16T16:54:38.076 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.074+0000 7f21467fc700 1 --2- 172.21.15.44:0/3017777157 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f213c0a1340 0x7f213c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.077 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.074+0000 7f21467fc700 1 -- 172.21.15.44:0/3017777157 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f21480cc930 con 0x7f2150105620 2023-11-16T16:54:38.077 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.074+0000 7f216ade3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d3b60 0x559b5a1d3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.074+0000 7f215499a700 1 --2- 172.21.15.44:0/3017777157 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f213c0a1340 0x7f213c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.074+0000 7f216ade3b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d4470 0x559b5a1d8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.075+0000 7f216ade3b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 0x559b5a1d8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.075+0000 7f216ade3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b5a0d9060 con 0x559b5a1d3b60 2023-11-16T16:54:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.075+0000 7f216ade3b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b5a12b8a0 con 0x559b5a1d88b0 2023-11-16T16:54:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.075+0000 7f216ade3b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b5a124d30 con 0x559b5a1d4470 2023-11-16T16:54:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.075+0000 7f215519b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d3b60 0x559b5a1d3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.075+0000 7f215519b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d3b60 0x559b5a1d3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60144/0 (socket says 172.21.15.44:60144) 2023-11-16T16:54:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.075+0000 7f215519b700 1 -- 172.21.15.44:0/2435846477 learned_addr learned my addr 172.21.15.44:0/2435846477 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.075+0000 7f215599c700 1 --2- 172.21.15.44:0/2435846477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 0x559b5a1d8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.075+0000 7f215499a700 1 --2- 172.21.15.44:0/2435846477 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d4470 0x559b5a1d8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.075+0000 7f215599c700 1 -- 172.21.15.44:0/2435846477 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d4470 msgr2=0x559b5a1d8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.075+0000 7f215599c700 1 --2- 172.21.15.44:0/2435846477 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d4470 0x559b5a1d8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.075+0000 7f215599c700 1 -- 172.21.15.44:0/2435846477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d3b60 msgr2=0x559b5a1d3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.075+0000 7f215599c700 1 --2- 172.21.15.44:0/2435846477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d3b60 0x559b5a1d3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.075+0000 7f215599c700 1 -- 172.21.15.44:0/2435846477 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b59fa6630 con 0x559b5a1d88b0 2023-11-16T16:54:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.075+0000 7f215519b700 1 --2- 172.21.15.44:0/2435846477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d3b60 0x559b5a1d3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.075+0000 7f215499a700 1 --2- 172.21.15.44:0/2435846477 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d4470 0x559b5a1d8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:54:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.076+0000 7f215599c700 1 --2- 172.21.15.44:0/2435846477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 0x559b5a1d8d00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f21480c9ca0 tx=0x7f2148099160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd11aaa5b171151d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.076+0000 7f215499a700 1 --2- 172.21.15.44:0/3017777157 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f213c0a1340 0x7f213c0a37f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f21400058e0 tx=0x7f2140000bc0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.076+0000 7f213b7fe700 1 -- 172.21.15.44:0/2435846477 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21480b90d0 con 0x559b5a1d88b0 2023-11-16T16:54:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.076+0000 7f213b7fe700 1 -- 172.21.15.44:0/2435846477 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2148099620 con 0x559b5a1d88b0 2023-11-16T16:54:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.076+0000 7f213b7fe700 1 -- 172.21.15.44:0/2435846477 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21480840c0 con 0x559b5a1d88b0 2023-11-16T16:54:38.082 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.076+0000 7f216ade3b80 1 -- 172.21.15.44:0/2435846477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 msgr2=0x559b5a1d8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:38.082 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.076+0000 7f216ade3b80 1 --2- 172.21.15.44:0/2435846477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 0x559b5a1d8d00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f21480c9ca0 tx=0x7f2148099160 comp rx=0 tx=0).stop 2023-11-16T16:54:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.076+0000 7f216ade3b80 1 -- 172.21.15.44:0/2435846477 shutdown_connections 2023-11-16T16:54:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.076+0000 7f216ade3b80 1 --2- 172.21.15.44:0/2435846477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 0x559b5a1d8d00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.076+0000 7f216ade3b80 1 --2- 172.21.15.44:0/2435846477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d3b60 0x559b5a1d3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.076+0000 7f216ade3b80 1 --2- 172.21.15.44:0/2435846477 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d4470 0x559b5a1d8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.076+0000 7f216ade3b80 1 -- 172.21.15.44:0/2435846477 >> 172.21.15.44:0/2435846477 conn(0x559b5a1c9360 msgr2=0x559b5a1ceb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.077+0000 7f216ade3b80 1 -- 172.21.15.44:0/2435846477 shutdown_connections 2023-11-16T16:54:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.077+0000 7f216ade3b80 1 -- 172.21.15.44:0/2435846477 wait complete. 2023-11-16T16:54:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.077+0000 7f216ade3b80 1 -- start start 2023-11-16T16:54:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.077+0000 7f216ade3b80 1 Processor -- start 2023-11-16T16:54:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.078+0000 7f216ade3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d3b60 0x559b5a1e0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.078+0000 7f216ade3b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d4470 0x559b5a1dd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.078+0000 7f216ade3b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 0x559b5a1dda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.078+0000 7f216ade3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b5a0c68e0 con 0x559b5a1d3b60 2023-11-16T16:54:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.078+0000 7f216ade3b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b5a0c6580 con 0x559b5a1d88b0 2023-11-16T16:54:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.078+0000 7f216ade3b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b59fa8920 con 0x559b5a1d4470 2023-11-16T16:54:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.078+0000 7f216ade3b80 10 client.0 _ll_register_callbacks cb 0x559b5a0e4780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.078+0000 7f215519b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d3b60 0x559b5a1e0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.078+0000 7f215599c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 0x559b5a1dda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.078+0000 7f215519b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d3b60 0x559b5a1e0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60154/0 (socket says 172.21.15.44:60154) 2023-11-16T16:54:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.078+0000 7f215519b700 1 -- 172.21.15.44:0/1582999882 learned_addr learned my addr 172.21.15.44:0/1582999882 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.078+0000 7f215499a700 1 --2- 172.21.15.44:0/1582999882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d4470 0x559b5a1dd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.079+0000 7f215599c700 1 -- 172.21.15.44:0/1582999882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d4470 msgr2=0x559b5a1dd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.079+0000 7f215599c700 1 --2- 172.21.15.44:0/1582999882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d4470 0x559b5a1dd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.079+0000 7f215599c700 1 -- 172.21.15.44:0/1582999882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d3b60 msgr2=0x559b5a1e0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.079+0000 7f215599c700 1 --2- 172.21.15.44:0/1582999882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d3b60 0x559b5a1e0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.079+0000 7f215599c700 1 -- 172.21.15.44:0/1582999882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2150110370 con 0x559b5a1d88b0 2023-11-16T16:54:38.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.079+0000 7f215599c700 1 --2- 172.21.15.44:0/1582999882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 0x559b5a1dda40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f21480c9cd0 tx=0x7f2148099c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8284161ed18251bc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:38.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.079+0000 7f2138ff9700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:54:38.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.079+0000 7f2138ff9700 1 -- 172.21.15.44:0/1582999882 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21480b7070 con 0x559b5a1d88b0 2023-11-16T16:54:38.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.079+0000 7f2138ff9700 1 -- 172.21.15.44:0/1582999882 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f214809d650 con 0x559b5a1d88b0 2023-11-16T16:54:38.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.079+0000 7f216ade3b80 1 -- 172.21.15.44:0/1582999882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559b5a1c6b20 con 0x559b5a1d88b0 2023-11-16T16:54:38.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.079+0000 7f2138ff9700 1 -- 172.21.15.44:0/1582999882 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21480a1a70 con 0x559b5a1d88b0 2023-11-16T16:54:38.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.079+0000 7f216ade3b80 10 client.26579 fetch_fsmap learned FSMap version 21 2023-11-16T16:54:38.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.079+0000 7f216ade3b80 1 -- 172.21.15.44:0/1582999882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x559b5a0eb190 con 0x559b5a1d88b0 2023-11-16T16:54:38.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.080+0000 7f2138ff9700 1 -- 172.21.15.44:0/1582999882 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559b5a1c6b20 con 0x559b5a1d88b0 2023-11-16T16:54:38.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.080+0000 7f2138ff9700 1 -- 172.21.15.44:0/1582999882 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2148004bd0 con 0x559b5a1d88b0 2023-11-16T16:54:38.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.080+0000 7f216ade3b80 10 client.26579 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:54:38.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.080+0000 7f216ade3b80 10 client.26579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:38.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.080+0000 7f216ade3b80 20 client.26579 populate_metadata read hostname 'smithi044' 2023-11-16T16:54:38.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.080+0000 7f216ade3b80 1 --2- 172.21.15.44:0/1582999882 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b5a1ed2d0 0x559b5a1ef6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.080+0000 7f216ade3b80 4 client.26579 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:54:38.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.080+0000 7f216ade3b80 1 -- 172.21.15.44:0/1582999882 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559b5a0e70a0 con 0x559b5a1ed2d0 2023-11-16T16:54:38.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.080+0000 7f215519b700 1 --2- 172.21.15.44:0/1582999882 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b5a1ed2d0 0x559b5a1ef6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.081+0000 7f215519b700 1 --2- 172.21.15.44:0/1582999882 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b5a1ed2d0 0x559b5a1ef6b0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84c07d915fe7d2a0 server_cookie=756f469b634dbc27 in_seq=0 out_seq=0 2023-11-16T16:54:38.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.081+0000 7f2138ff9700 10 client.26579 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:38.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.127+0000 7f2138ff9700 1 -- 172.21.15.44:0/1582999882 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559b5a0e70a0 con 0x559b5a1ed2d0 2023-11-16T16:54:38.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.127+0000 7f2138ff9700 10 client.26579 handle_command_reply: tid=0 2023-11-16T16:54:38.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.127+0000 7f216ade3b80 1 client.26579 shutdown 2023-11-16T16:54:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.127+0000 7f216ade3b80 1 -- 172.21.15.44:0/1582999882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 msgr2=0x559b5a1dda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.127+0000 7f216ade3b80 1 --2- 172.21.15.44:0/1582999882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 0x559b5a1dda40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f21480c9cd0 tx=0x7f2148099c10 comp rx=0 tx=0).stop 2023-11-16T16:54:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.128+0000 7f216ade3b80 1 -- 172.21.15.44:0/1582999882 shutdown_connections 2023-11-16T16:54:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.128+0000 7f216ade3b80 1 --2- 172.21.15.44:0/1582999882 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b5a1ed2d0 0x559b5a1ef6b0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.128+0000 7f216ade3b80 1 --2- 172.21.15.44:0/1582999882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 0x559b5a1dda40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.128+0000 7f216ade3b80 1 --2- 172.21.15.44:0/1582999882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d3b60 0x559b5a1e0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.128+0000 7f216ade3b80 1 --2- 172.21.15.44:0/1582999882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d4470 0x559b5a1dd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.128+0000 7f2138ff9700 0 client.26579 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.128+0000 7f216ade3b80 1 -- 172.21.15.44:0/1582999882 >> 172.21.15.44:0/1582999882 conn(0x559b5a1c9360 msgr2=0x559b5a1d9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.128+0000 7f216ade3b80 1 -- 172.21.15.44:0/1582999882 shutdown_connections 2023-11-16T16:54:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.128+0000 7f216ade3b80 1 -- 172.21.15.44:0/1582999882 wait complete. 2023-11-16T16:54:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.128+0000 7f216ade3b80 20 client.26579 trim_cache size 0 max 16384 2023-11-16T16:54:38.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:37 smithi176 ceph-mon[131630]: pgmap v143: 129 pgs: 129 active+clean; 7.7 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 2.68k op/s 2023-11-16T16:54:38.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.146+0000 7f216ade3b80 1 Processor -- start 2023-11-16T16:54:38.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.146+0000 7f216ade3b80 1 -- start start 2023-11-16T16:54:38.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.146+0000 7f216ade3b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d3b60 0x559b5a1cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.146+0000 7f216ade3b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d4470 0x559b5a1e48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.146+0000 7f216ade3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d88b0 0x559b5a1d6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.147+0000 7f216ade3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b5a1de270 con 0x559b5a1d88b0 2023-11-16T16:54:38.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.147+0000 7f216ade3b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b5a1c8ee0 con 0x559b5a1d4470 2023-11-16T16:54:38.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.147+0000 7f216ade3b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b5a1efce0 con 0x559b5a1d3b60 2023-11-16T16:54:38.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.147+0000 7f215499a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d4470 0x559b5a1e48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.147+0000 7f215499a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d4470 0x559b5a1e48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60516/0 (socket says 172.21.15.44:60516) 2023-11-16T16:54:38.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.147+0000 7f215499a700 1 -- 172.21.15.44:0/3780272245 learned_addr learned my addr 172.21.15.44:0/3780272245 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:38.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.147+0000 7f215599c700 1 --2- 172.21.15.44:0/3780272245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d88b0 0x559b5a1d6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.147+0000 7f215519b700 1 --2- 172.21.15.44:0/3780272245 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d3b60 0x559b5a1cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.147+0000 7f215499a700 1 -- 172.21.15.44:0/3780272245 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d3b60 msgr2=0x559b5a1cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:38.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.147+0000 7f215499a700 1 --2- 172.21.15.44:0/3780272245 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d3b60 0x559b5a1cebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.147+0000 7f215499a700 1 -- 172.21.15.44:0/3780272245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d88b0 msgr2=0x559b5a1d6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:38.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.147+0000 7f215499a700 1 --2- 172.21.15.44:0/3780272245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d88b0 0x559b5a1d6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.147+0000 7f215499a700 1 -- 172.21.15.44:0/3780272245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b5a1d8f50 con 0x559b5a1d4470 2023-11-16T16:54:38.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.147+0000 7f215519b700 1 --2- 172.21.15.44:0/3780272245 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d3b60 0x559b5a1cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:54:38.152 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.148+0000 7f215499a700 1 --2- 172.21.15.44:0/3780272245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d4470 0x559b5a1e48e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2140015000 tx=0x7f2140003680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f82fffc24dc9677 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:38.152 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.148+0000 7f213bfff700 1 -- 172.21.15.44:0/3780272245 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21400053d0 con 0x559b5a1d4470 2023-11-16T16:54:38.152 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.148+0000 7f213bfff700 1 -- 172.21.15.44:0/3780272245 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f214002c040 con 0x559b5a1d4470 2023-11-16T16:54:38.152 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.148+0000 7f213bfff700 1 -- 172.21.15.44:0/3780272245 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214000f7e0 con 0x559b5a1d4470 2023-11-16T16:54:38.152 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.149+0000 7f216ade3b80 1 -- 172.21.15.44:0/3780272245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d4470 msgr2=0x559b5a1e48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:38.152 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.149+0000 7f216ade3b80 1 --2- 172.21.15.44:0/3780272245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d4470 0x559b5a1e48e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2140015000 tx=0x7f2140003680 comp rx=0 tx=0).stop 2023-11-16T16:54:38.152 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.149+0000 7f216ade3b80 1 -- 172.21.15.44:0/3780272245 shutdown_connections 2023-11-16T16:54:38.152 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.149+0000 7f216ade3b80 1 --2- 172.21.15.44:0/3780272245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d88b0 0x559b5a1d6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.149+0000 7f216ade3b80 1 --2- 172.21.15.44:0/3780272245 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d3b60 0x559b5a1cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.149+0000 7f216ade3b80 1 --2- 172.21.15.44:0/3780272245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d4470 0x559b5a1e48e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.149+0000 7f216ade3b80 1 -- 172.21.15.44:0/3780272245 >> 172.21.15.44:0/3780272245 conn(0x559b5a1c9360 msgr2=0x559b5a1e9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.149+0000 7f216ade3b80 1 -- 172.21.15.44:0/3780272245 shutdown_connections 2023-11-16T16:54:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.149+0000 7f216ade3b80 1 -- 172.21.15.44:0/3780272245 wait complete. 2023-11-16T16:54:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.149+0000 7f216ade3b80 1 -- start start 2023-11-16T16:54:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.149+0000 7f216ade3b80 1 Processor -- start 2023-11-16T16:54:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f216ade3b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d3b60 0x559b5a1dcec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f216ade3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d4470 0x559b5a1db860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f216ade3b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 0x559b5a1dbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f216ade3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b5a1dd4e0 con 0x559b5a1d4470 2023-11-16T16:54:38.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f216ade3b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b5a1dd650 con 0x559b5a1d88b0 2023-11-16T16:54:38.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f216ade3b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b5a1dc2e0 con 0x559b5a1d3b60 2023-11-16T16:54:38.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f216ade3b80 10 client.0 _ll_register_callbacks cb 0x7f214c009be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:38.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f215599c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 0x559b5a1dbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f215499a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d4470 0x559b5a1db860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f215599c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 0x559b5a1dbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60526/0 (socket says 172.21.15.44:60526) 2023-11-16T16:54:38.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f215599c700 1 -- 172.21.15.44:0/2947230260 learned_addr learned my addr 172.21.15.44:0/2947230260 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:38.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f215519b700 1 --2- 172.21.15.44:0/2947230260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d3b60 0x559b5a1dcec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f215599c700 1 -- 172.21.15.44:0/2947230260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d3b60 msgr2=0x559b5a1dcec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:38.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f215599c700 1 --2- 172.21.15.44:0/2947230260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d3b60 0x559b5a1dcec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f215599c700 1 -- 172.21.15.44:0/2947230260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d4470 msgr2=0x559b5a1db860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:38.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f215599c700 1 --2- 172.21.15.44:0/2947230260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d4470 0x559b5a1db860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f215599c700 1 -- 172.21.15.44:0/2947230260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2140020040 con 0x559b5a1d88b0 2023-11-16T16:54:38.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.150+0000 7f215519b700 1 --2- 172.21.15.44:0/2947230260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d3b60 0x559b5a1dcec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:54:38.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.151+0000 7f215599c700 1 --2- 172.21.15.44:0/2947230260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 0x559b5a1dbda0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2148099320 tx=0x7f2148003c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68aeef47c70788a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:38.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.151+0000 7f2138ff9700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:54:38.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.151+0000 7f2138ff9700 1 -- 172.21.15.44:0/2947230260 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21480b7040 con 0x559b5a1d88b0 2023-11-16T16:54:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.151+0000 7f2138ff9700 1 -- 172.21.15.44:0/2947230260 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f214809d650 con 0x559b5a1d88b0 2023-11-16T16:54:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.151+0000 7f216ade3b80 1 -- 172.21.15.44:0/2947230260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2148099620 con 0x559b5a1d88b0 2023-11-16T16:54:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.151+0000 7f2138ff9700 1 -- 172.21.15.44:0/2947230260 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21480a1a70 con 0x559b5a1d88b0 2023-11-16T16:54:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.151+0000 7f216ade3b80 10 client.26585 fetch_fsmap learned FSMap version 21 2023-11-16T16:54:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.151+0000 7f216ade3b80 1 -- 172.21.15.44:0/2947230260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x559b59f52de0 con 0x559b5a1d88b0 2023-11-16T16:54:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.151+0000 7f2138ff9700 1 -- 172.21.15.44:0/2947230260 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2148099620 con 0x559b5a1d88b0 2023-11-16T16:54:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.151+0000 7f2138ff9700 1 -- 172.21.15.44:0/2947230260 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2148098c30 con 0x559b5a1d88b0 2023-11-16T16:54:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.151+0000 7f216ade3b80 10 client.26585 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:54:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.151+0000 7f216ade3b80 10 client.26585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.151+0000 7f216ade3b80 20 client.26585 populate_metadata read hostname 'smithi044' 2023-11-16T16:54:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.151+0000 7f216ade3b80 1 --2- 172.21.15.44:0/2947230260 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b5a241d40 0x559b5a244120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.151+0000 7f216ade3b80 4 client.26585 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:54:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.151+0000 7f216ade3b80 1 -- 172.21.15.44:0/2947230260 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559b5a0f3660 con 0x559b5a241d40 2023-11-16T16:54:38.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.152+0000 7f215519b700 1 --2- 172.21.15.44:0/2947230260 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b5a241d40 0x559b5a244120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:38.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.152+0000 7f215519b700 1 --2- 172.21.15.44:0/2947230260 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b5a241d40 0x559b5a244120 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f50095809eafc617 server_cookie=e8cb88ae71ca8ede in_seq=0 out_seq=0 2023-11-16T16:54:38.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.152+0000 7f2138ff9700 10 client.26585 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:38.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.153+0000 7f2138ff9700 1 -- 172.21.15.44:0/2947230260 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x559b5a0f3660 con 0x559b5a241d40 2023-11-16T16:54:38.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.153+0000 7f2138ff9700 10 client.26585 handle_command_reply: tid=0 2023-11-16T16:54:38.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.153+0000 7f216ade3b80 1 client.26585 shutdown 2023-11-16T16:54:38.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.154+0000 7f216ade3b80 1 -- 172.21.15.44:0/2947230260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 msgr2=0x559b5a1dbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:38.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.154+0000 7f216ade3b80 1 --2- 172.21.15.44:0/2947230260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 0x559b5a1dbda0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2148099320 tx=0x7f2148003c30 comp rx=0 tx=0).stop 2023-11-16T16:54:38.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.154+0000 7f216ade3b80 1 -- 172.21.15.44:0/2947230260 shutdown_connections 2023-11-16T16:54:38.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.154+0000 7f216ade3b80 1 --2- 172.21.15.44:0/2947230260 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559b5a241d40 0x559b5a244120 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.154+0000 7f216ade3b80 1 --2- 172.21.15.44:0/2947230260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b5a1d88b0 0x559b5a1dbda0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.154+0000 7f216ade3b80 1 --2- 172.21.15.44:0/2947230260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559b5a1d3b60 0x559b5a1dcec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.154+0000 7f216ade3b80 1 --2- 172.21.15.44:0/2947230260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559b5a1d4470 0x559b5a1db860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.154+0000 7f2138ff9700 0 client.26585 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.154+0000 7f216ade3b80 1 -- 172.21.15.44:0/2947230260 >> 172.21.15.44:0/2947230260 conn(0x559b5a1c9360 msgr2=0x559b5a1dfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.154+0000 7f216ade3b80 1 -- 172.21.15.44:0/2947230260 shutdown_connections 2023-11-16T16:54:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.154+0000 7f216ade3b80 1 -- 172.21.15.44:0/2947230260 wait complete. 2023-11-16T16:54:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.154+0000 7f216ade3b80 20 client.26585 trim_cache size 0 max 16384 2023-11-16T16:54:38.159 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:54:38.159 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "scrub active (9998 inodes in the stack)", 2023-11-16T16:54:38.159 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2023-11-16T16:54:38.159 INFO:teuthology.orchestra.run.smithi044.stdout: "c6d82195-2d53-46f4-9f2b-dc4452b4a1b0": { 2023-11-16T16:54:38.160 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2023-11-16T16:54:38.160 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "c6d82195-2d53-46f4-9f2b-dc4452b4a1b0", 2023-11-16T16:54:38.160 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2023-11-16T16:54:38.160 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:54:38.160 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:54:38.160 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:54:38.160 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.155+0000 7f21573ff700 1 -- 172.21.15.44:0/3017777157 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f213c0a1340 msgr2=0x7f213c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:38.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.155+0000 7f21573ff700 1 --2- 172.21.15.44:0/3017777157 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f213c0a1340 0x7f213c0a37f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f21400058e0 tx=0x7f2140000bc0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.155+0000 7f21573ff700 1 -- 172.21.15.44:0/3017777157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2150105620 msgr2=0x7f215019bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:38.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.155+0000 7f21573ff700 1 --2- 172.21.15.44:0/3017777157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2150105620 0x7f215019bfe0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f214800c370 tx=0x7f214800c680 comp rx=0 tx=0).stop 2023-11-16T16:54:38.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.155+0000 7f21573ff700 1 -- 172.21.15.44:0/3017777157 shutdown_connections 2023-11-16T16:54:38.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.155+0000 7f21573ff700 1 --2- 172.21.15.44:0/3017777157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2150105620 0x7f215019bfe0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.155+0000 7f21573ff700 1 --2- 172.21.15.44:0/3017777157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2150100cc0 0x7f21501986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.155+0000 7f21573ff700 1 --2- 172.21.15.44:0/3017777157 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f213c0a1340 0x7f213c0a37f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.155+0000 7f21573ff700 1 --2- 172.21.15.44:0/3017777157 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2150101660 0x7f2150198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:38.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.155+0000 7f21573ff700 1 -- 172.21.15.44:0/3017777157 >> 172.21.15.44:0/3017777157 conn(0x7f2150077560 msgr2=0x7f21500fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:38.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.155+0000 7f21573ff700 1 -- 172.21.15.44:0/3017777157 shutdown_connections 2023-11-16T16:54:38.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:54:38.155+0000 7f21573ff700 1 -- 172.21.15.44:0/3017777157 wait complete. 2023-11-16T16:54:38.176 INFO:tasks.cephfs.filesystem:scrub status for tag:c6d82195-2d53-46f4-9f2b-dc4452b4a1b0 - {'path': '/', 'tag': 'c6d82195-2d53-46f4-9f2b-dc4452b4a1b0', 'options': 'recursive,force'} 2023-11-16T16:54:38.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:38 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:54:38.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:38 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:54:38.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:38 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:54:39.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:38 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:54:39.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:38 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:54:39.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:38 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:54:39.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:38 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:54:39.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:38 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:54:39.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:38 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:54:39.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:39 smithi077 ceph-mon[131204]: pgmap v144: 129 pgs: 129 active+clean; 23 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.9 MiB/s wr, 2.69k op/s 2023-11-16T16:54:40.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:39 smithi044 ceph-mon[128750]: pgmap v144: 129 pgs: 129 active+clean; 23 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.9 MiB/s wr, 2.69k op/s 2023-11-16T16:54:40.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:39 smithi176 ceph-mon[131630]: pgmap v144: 129 pgs: 129 active+clean; 23 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.9 MiB/s wr, 2.69k op/s 2023-11-16T16:54:40.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:40 smithi077 ceph-mon[131204]: pgmap v145: 129 pgs: 129 active+clean; 23 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 2.29k op/s 2023-11-16T16:54:41.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:40 smithi044 ceph-mon[128750]: pgmap v145: 129 pgs: 129 active+clean; 23 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 2.29k op/s 2023-11-16T16:54:41.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:40 smithi176 ceph-mon[131630]: pgmap v145: 129 pgs: 129 active+clean; 23 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 2.29k op/s 2023-11-16T16:54:43.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:42 smithi044 ceph-mon[128750]: pgmap v146: 129 pgs: 129 active+clean; 36 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 2.58k op/s 2023-11-16T16:54:43.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:42 smithi176 ceph-mon[131630]: pgmap v146: 129 pgs: 129 active+clean; 36 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 2.58k op/s 2023-11-16T16:54:43.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:42 smithi077 ceph-mon[131204]: pgmap v146: 129 pgs: 129 active+clean; 36 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 2.58k op/s 2023-11-16T16:54:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.994+0000 7f8ee95ff700 1 -- 172.21.15.44:0/2789496276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ee4100c50 msgr2=0x7f8ee4101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.994+0000 7f8ee95ff700 1 --2- 172.21.15.44:0/2789496276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ee4100c50 0x7f8ee4101020 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8ed40099c0 tx=0x7f8ed4009cd0 comp rx=0 tx=0).stop 2023-11-16T16:54:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.995+0000 7f8ee95ff700 1 -- 172.21.15.44:0/2789496276 shutdown_connections 2023-11-16T16:54:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.995+0000 7f8ee95ff700 1 --2- 172.21.15.44:0/2789496276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee4105560 0x7f8ee410d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.995+0000 7f8ee95ff700 1 --2- 172.21.15.44:0/2789496276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ee4100c50 0x7f8ee4101020 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.995+0000 7f8ee95ff700 1 --2- 172.21.15.44:0/2789496276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ee41015f0 0x7f8ee4104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.995+0000 7f8ee95ff700 1 -- 172.21.15.44:0/2789496276 >> 172.21.15.44:0/2789496276 conn(0x7f8ee4077550 msgr2=0x7f8ee4077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.995+0000 7f8ee95ff700 1 -- 172.21.15.44:0/2789496276 shutdown_connections 2023-11-16T16:54:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.995+0000 7f8ee95ff700 1 -- 172.21.15.44:0/2789496276 wait complete. 2023-11-16T16:54:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.996+0000 7f8ee95ff700 1 Processor -- start 2023-11-16T16:54:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.997+0000 7f8ee95ff700 1 -- start start 2023-11-16T16:54:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.997+0000 7f8ee95ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ee4100c50 0x7f8ee4072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.997+0000 7f8ee95ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ee41015f0 0x7f8ee4073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.997+0000 7f8ee95ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee4105560 0x7f8ee41a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.997+0000 7f8ee95ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee4073b60 con 0x7f8ee41015f0 2023-11-16T16:54:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.997+0000 7f8ee95ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee41a4b30 con 0x7f8ee4105560 2023-11-16T16:54:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.997+0000 7f8ee95ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee41a4c70 con 0x7f8ee4100c50 2023-11-16T16:54:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.998+0000 7f8ee37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee4105560 0x7f8ee41a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.998+0000 7f8ee37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee4105560 0x7f8ee41a45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60536/0 (socket says 172.21.15.44:60536) 2023-11-16T16:54:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.998+0000 7f8ee37fe700 1 -- 172.21.15.44:0/2883742124 learned_addr learned my addr 172.21.15.44:0/2883742124 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.998+0000 7f8ee2ffd700 1 --2- 172.21.15.44:0/2883742124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ee4100c50 0x7f8ee4072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.998+0000 7f8ee27fc700 1 --2- 172.21.15.44:0/2883742124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ee41015f0 0x7f8ee4073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.998+0000 7f8ee37fe700 1 -- 172.21.15.44:0/2883742124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ee4100c50 msgr2=0x7f8ee4072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.998+0000 7f8ee37fe700 1 --2- 172.21.15.44:0/2883742124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ee4100c50 0x7f8ee4072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.998+0000 7f8ee37fe700 1 -- 172.21.15.44:0/2883742124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ee41015f0 msgr2=0x7f8ee4073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.998+0000 7f8ee37fe700 1 --2- 172.21.15.44:0/2883742124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ee41015f0 0x7f8ee4073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.998+0000 7f8ee37fe700 1 -- 172.21.15.44:0/2883742124 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed40095f0 con 0x7f8ee4105560 2023-11-16T16:54:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.998+0000 7f8ee27fc700 1 --2- 172.21.15.44:0/2883742124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ee41015f0 0x7f8ee4073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:54:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.999+0000 7f8ee37fe700 1 --2- 172.21.15.44:0/2883742124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee4105560 0x7f8ee41a45f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00ef30 tx=0x7f8edc00c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.999+0000 7f8ecbfff700 1 -- 172.21.15.44:0/2883742124 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc018070 con 0x7f8ee4105560 2023-11-16T16:54:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.999+0000 7f8ecbfff700 1 -- 172.21.15.44:0/2883742124 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8edc00f040 con 0x7f8ee4105560 2023-11-16T16:54:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.999+0000 7f8ecbfff700 1 -- 172.21.15.44:0/2883742124 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc003dc0 con 0x7f8ee4105560 2023-11-16T16:54:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.999+0000 7f8ee95ff700 1 -- 172.21.15.44:0/2883742124 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee41103b0 con 0x7f8ee4105560 2023-11-16T16:54:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:43.999+0000 7f8ee95ff700 1 -- 172.21.15.44:0/2883742124 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee4110220 con 0x7f8ee4105560 2023-11-16T16:54:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.003+0000 7f8ecbfff700 1 -- 172.21.15.44:0/2883742124 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f8edc00caa0 con 0x7f8ee4105560 2023-11-16T16:54:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.004+0000 7f8efcec9b80 1 Processor -- start 2023-11-16T16:54:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.005+0000 7f8efcec9b80 1 -- start start 2023-11-16T16:54:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.005+0000 7f8ecbfff700 1 --2- 172.21.15.44:0/2883742124 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f8ecc0a1340 0x7f8ecc0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.005+0000 7f8efcec9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56276ff186b0 0x5627700c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.005+0000 7f8efcec9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627700c2720 0x5627700bc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.005+0000 7f8efcec9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2b90 0x5627700bcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.005+0000 7f8efcec9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56276ffc2000 con 0x5627700c2720 2023-11-16T16:54:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.005+0000 7f8efcec9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562770014840 con 0x56276ff186b0 2023-11-16T16:54:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.005+0000 7f8efcec9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56277000dcd0 con 0x5627700c2b90 2023-11-16T16:54:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.005+0000 7f8ecbfff700 1 -- 172.21.15.44:0/2883742124 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f8edc0ccc00 con 0x7f8ee4105560 2023-11-16T16:54:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.005+0000 7f8ee37fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2b90 0x5627700bcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.005+0000 7f8ee27fc700 1 --2- 172.21.15.44:0/2883742124 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f8ecc0a1340 0x7f8ecc0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.006+0000 7f8ee2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56276ff186b0 0x5627700c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.006+0000 7f8ee2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56276ff186b0 0x5627700c21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60546/0 (socket says 172.21.15.44:60546) 2023-11-16T16:54:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.006+0000 7f8ee2ffd700 1 -- 172.21.15.44:0/561256785 learned_addr learned my addr 172.21.15.44:0/561256785 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.006+0000 7f8ee27fc700 1 --2- 172.21.15.44:0/561256785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627700c2720 0x5627700bc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.006+0000 7f8ee2ffd700 1 -- 172.21.15.44:0/561256785 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2b90 msgr2=0x5627700bcb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.006+0000 7f8ee2ffd700 1 --2- 172.21.15.44:0/561256785 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2b90 0x5627700bcb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.006+0000 7f8ee2ffd700 1 -- 172.21.15.44:0/561256785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627700c2720 msgr2=0x5627700bc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.006+0000 7f8ee2ffd700 1 --2- 172.21.15.44:0/561256785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627700c2720 0x5627700bc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.006+0000 7f8ee2ffd700 1 -- 172.21.15.44:0/561256785 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56276fe8f5d0 con 0x56276ff186b0 2023-11-16T16:54:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.006+0000 7f8ee37fe700 1 --2- 172.21.15.44:0/561256785 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2b90 0x5627700bcb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.006+0000 7f8ee27fc700 1 --2- 172.21.15.44:0/561256785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627700c2720 0x5627700bc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:54:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.007+0000 7f8ee2ffd700 1 --2- 172.21.15.44:0/561256785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56276ff186b0 0x5627700c21e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8ed40036c0 tx=0x7f8ed4005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e1bbff8c74404a4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.007+0000 7f8ee27fc700 1 --2- 172.21.15.44:0/2883742124 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f8ecc0a1340 0x7f8ecc0a37f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40745e0 tx=0x7f8ed800b360 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.007+0000 7f8ec97fa700 1 -- 172.21.15.44:0/561256785 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed401d040 con 0x56276ff186b0 2023-11-16T16:54:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.007+0000 7f8ec97fa700 1 -- 172.21.15.44:0/561256785 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ed40037d0 con 0x56276ff186b0 2023-11-16T16:54:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.007+0000 7f8ec97fa700 1 -- 172.21.15.44:0/561256785 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed400bb10 con 0x56276ff186b0 2023-11-16T16:54:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.008+0000 7f8efcec9b80 1 -- 172.21.15.44:0/561256785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56276ff186b0 msgr2=0x5627700c21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.008+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/561256785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56276ff186b0 0x5627700c21e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8ed40036c0 tx=0x7f8ed4005e80 comp rx=0 tx=0).stop 2023-11-16T16:54:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.008+0000 7f8efcec9b80 1 -- 172.21.15.44:0/561256785 shutdown_connections 2023-11-16T16:54:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.008+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/561256785 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2b90 0x5627700bcb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.008+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/561256785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56276ff186b0 0x5627700c21e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.008+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/561256785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627700c2720 0x5627700bc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.008+0000 7f8efcec9b80 1 -- 172.21.15.44:0/561256785 >> 172.21.15.44:0/561256785 conn(0x5627700b1f50 msgr2=0x5627700b7270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.008+0000 7f8efcec9b80 1 -- 172.21.15.44:0/561256785 shutdown_connections 2023-11-16T16:54:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.008+0000 7f8efcec9b80 1 -- 172.21.15.44:0/561256785 wait complete. 2023-11-16T16:54:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.009+0000 7f8efcec9b80 1 -- start start 2023-11-16T16:54:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.009+0000 7f8efcec9b80 1 Processor -- start 2023-11-16T16:54:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8efcec9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56276ff186b0 0x5627700cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8efcec9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2720 0x5627700cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8efcec9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627700c2b90 0x5627700c5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8efcec9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56276ffaf880 con 0x5627700c2b90 2023-11-16T16:54:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8efcec9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56276ffaf520 con 0x5627700c2720 2023-11-16T16:54:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8efcec9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56276fe918c0 con 0x56276ff186b0 2023-11-16T16:54:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8efcec9b80 10 client.0 _ll_register_callbacks cb 0x56276ffcd720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8ee2ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56276ff186b0 0x5627700cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8ee2ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56276ff186b0 0x5627700cb9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:49530/0 (socket says 172.21.15.44:49530) 2023-11-16T16:54:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8ee27fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2720 0x5627700cbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8ee2ffd700 1 -- 172.21.15.44:0/2808952152 learned_addr learned my addr 172.21.15.44:0/2808952152 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8ee37fe700 1 --2- 172.21.15.44:0/2808952152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627700c2b90 0x5627700c5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8ee2ffd700 1 -- 172.21.15.44:0/2808952152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2720 msgr2=0x5627700cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8ee2ffd700 1 --2- 172.21.15.44:0/2808952152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2720 0x5627700cbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8ee2ffd700 1 -- 172.21.15.44:0/2808952152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627700c2b90 msgr2=0x5627700c5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8ee2ffd700 1 --2- 172.21.15.44:0/2808952152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627700c2b90 0x5627700c5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8ee2ffd700 1 -- 172.21.15.44:0/2808952152 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee410d9b0 con 0x56276ff186b0 2023-11-16T16:54:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.010+0000 7f8ee27fc700 1 --2- 172.21.15.44:0/2808952152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2720 0x5627700cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.011+0000 7f8ee2ffd700 1 --2- 172.21.15.44:0/2808952152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56276ff186b0 0x5627700cb9f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4006010 tx=0x7f8ed4003be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=539c939acacc582b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.011+0000 7f8ec6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:54:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.011+0000 7f8ec6ff5700 1 -- 172.21.15.44:0/2808952152 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed401d040 con 0x56276ff186b0 2023-11-16T16:54:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.011+0000 7f8ec6ff5700 1 -- 172.21.15.44:0/2808952152 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f8ed40054d0 con 0x56276ff186b0 2023-11-16T16:54:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.011+0000 7f8efcec9b80 1 -- 172.21.15.44:0/2808952152 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5627700af710 con 0x56276ff186b0 2023-11-16T16:54:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.012+0000 7f8ec6ff5700 1 -- 172.21.15.44:0/2808952152 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ed400b790 con 0x56276ff186b0 2023-11-16T16:54:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.012+0000 7f8efcec9b80 10 client.26584 fetch_fsmap learned FSMap version 21 2023-11-16T16:54:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.012+0000 7f8efcec9b80 1 -- 172.21.15.44:0/2808952152 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x56276ffd4130 con 0x56276ff186b0 2023-11-16T16:54:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.012+0000 7f8ec6ff5700 1 -- 172.21.15.44:0/2808952152 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5627700af710 con 0x56276ff186b0 2023-11-16T16:54:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.013+0000 7f8ec6ff5700 1 -- 172.21.15.44:0/2808952152 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8ed4032050 con 0x56276ff186b0 2023-11-16T16:54:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.013+0000 7f8efcec9b80 10 client.26584 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:54:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.013+0000 7f8efcec9b80 10 client.26584 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.013+0000 7f8efcec9b80 20 client.26584 populate_metadata read hostname 'smithi044' 2023-11-16T16:54:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.013+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/2808952152 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5627700d59b0 0x5627700d7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.013+0000 7f8efcec9b80 4 client.26584 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:54:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.013+0000 7f8efcec9b80 1 -- 172.21.15.44:0/2808952152 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56276ffd0040 con 0x5627700d59b0 2023-11-16T16:54:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.013+0000 7f8ee27fc700 1 --2- 172.21.15.44:0/2808952152 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5627700d59b0 0x5627700d7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.014+0000 7f8ec6ff5700 10 client.26584 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.014+0000 7f8ee27fc700 1 --2- 172.21.15.44:0/2808952152 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5627700d59b0 0x5627700d7d90 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d20410f6c9e1c82 server_cookie=fcf2846e37e2a90a in_seq=0 out_seq=0 2023-11-16T16:54:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.015+0000 7f8ec6ff5700 1 -- 172.21.15.44:0/2808952152 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56276ffd0040 con 0x5627700d59b0 2023-11-16T16:54:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.015+0000 7f8ec6ff5700 10 client.26584 handle_command_reply: tid=0 2023-11-16T16:54:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.015+0000 7f8efcec9b80 1 client.26584 shutdown 2023-11-16T16:54:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.015+0000 7f8efcec9b80 1 -- 172.21.15.44:0/2808952152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56276ff186b0 msgr2=0x5627700cb9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.015+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/2808952152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56276ff186b0 0x5627700cb9f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4006010 tx=0x7f8ed4003be0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.015+0000 7f8efcec9b80 1 -- 172.21.15.44:0/2808952152 shutdown_connections 2023-11-16T16:54:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.015+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/2808952152 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5627700d59b0 0x5627700d7d90 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.015+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/2808952152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627700c2b90 0x5627700c5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.015+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/2808952152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56276ff186b0 0x5627700cb9f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.015+0000 7f8ec6ff5700 0 client.26584 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.015+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/2808952152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2720 0x5627700cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.015+0000 7f8efcec9b80 1 -- 172.21.15.44:0/2808952152 >> 172.21.15.44:0/2808952152 conn(0x5627700b1f50 msgr2=0x5627700b67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.015+0000 7f8efcec9b80 1 -- 172.21.15.44:0/2808952152 shutdown_connections 2023-11-16T16:54:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.015+0000 7f8efcec9b80 1 -- 172.21.15.44:0/2808952152 wait complete. 2023-11-16T16:54:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.016+0000 7f8efcec9b80 20 client.26584 trim_cache size 0 max 16384 2023-11-16T16:54:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.035+0000 7f8efcec9b80 1 Processor -- start 2023-11-16T16:54:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.035+0000 7f8efcec9b80 1 -- start start 2023-11-16T16:54:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.035+0000 7f8efcec9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56276ff186b0 0x5627700c5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.035+0000 7f8efcec9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2720 0x5627700b6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.036+0000 7f8efcec9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2b90 0x5627700bc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.036+0000 7f8efcec9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627700c4260 con 0x56276ff186b0 2023-11-16T16:54:44.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.036+0000 7f8efcec9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627700c0930 con 0x5627700c2b90 2023-11-16T16:54:44.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.036+0000 7f8efcec9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627700d7fe0 con 0x5627700c2720 2023-11-16T16:54:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.036+0000 7f8ee27fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2720 0x5627700b6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.036+0000 7f8ee37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2b90 0x5627700bc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.036+0000 7f8ee27fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2720 0x5627700b6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:49540/0 (socket says 172.21.15.44:49540) 2023-11-16T16:54:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.036+0000 7f8ee37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2b90 0x5627700bc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:60558/0 (socket says 172.21.15.44:60558) 2023-11-16T16:54:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.036+0000 7f8ee27fc700 1 -- 172.21.15.44:0/1664574881 learned_addr learned my addr 172.21.15.44:0/1664574881 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.036+0000 7f8ee2ffd700 1 --2- 172.21.15.44:0/1664574881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56276ff186b0 0x5627700c5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.036+0000 7f8ee27fc700 1 -- 172.21.15.44:0/1664574881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2b90 msgr2=0x5627700bc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.036+0000 7f8ee27fc700 1 --2- 172.21.15.44:0/1664574881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2b90 0x5627700bc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.036+0000 7f8ee27fc700 1 -- 172.21.15.44:0/1664574881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56276ff186b0 msgr2=0x5627700c5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.036+0000 7f8ee27fc700 1 --2- 172.21.15.44:0/1664574881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56276ff186b0 0x5627700c5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.036+0000 7f8ee27fc700 1 -- 172.21.15.44:0/1664574881 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56276ffd3e80 con 0x5627700c2720 2023-11-16T16:54:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.037+0000 7f8ee27fc700 1 --2- 172.21.15.44:0/1664574881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2720 0x5627700b6f60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8ee406a260 tx=0x7f8ed80058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=855b1202a68cd08e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.037+0000 7f8ec9ffb700 1 -- 172.21.15.44:0/1664574881 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8025070 con 0x5627700c2720 2023-11-16T16:54:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.037+0000 7f8ec9ffb700 1 -- 172.21.15.44:0/1664574881 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ed8027310 con 0x5627700c2720 2023-11-16T16:54:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.037+0000 7f8ec9ffb700 1 -- 172.21.15.44:0/1664574881 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8020480 con 0x5627700c2720 2023-11-16T16:54:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.038+0000 7f8efcec9b80 1 -- 172.21.15.44:0/1664574881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2720 msgr2=0x5627700b6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.038+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/1664574881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2720 0x5627700b6f60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8ee406a260 tx=0x7f8ed80058e0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.038+0000 7f8efcec9b80 1 -- 172.21.15.44:0/1664574881 shutdown_connections 2023-11-16T16:54:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.038+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/1664574881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2b90 0x5627700bc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.038+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/1664574881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56276ff186b0 0x5627700c5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.038+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/1664574881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2720 0x5627700b6f60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.038+0000 7f8efcec9b80 1 -- 172.21.15.44:0/1664574881 >> 172.21.15.44:0/1664574881 conn(0x5627700b1f50 msgr2=0x5627700c6610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.038+0000 7f8efcec9b80 1 -- 172.21.15.44:0/1664574881 shutdown_connections 2023-11-16T16:54:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.038+0000 7f8efcec9b80 1 -- 172.21.15.44:0/1664574881 wait complete. 2023-11-16T16:54:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.039+0000 7f8efcec9b80 1 -- start start 2023-11-16T16:54:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.039+0000 7f8efcec9b80 1 Processor -- start 2023-11-16T16:54:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.040+0000 7f8efcec9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56276ff186b0 0x5627700c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.040+0000 7f8efcec9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2720 0x5627700c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.040+0000 7f8efcec9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2b90 0x5627700ca060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.040+0000 7f8efcec9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562770128970 con 0x56276ff186b0 2023-11-16T16:54:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.040+0000 7f8efcec9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562770128ae0 con 0x5627700c2720 2023-11-16T16:54:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.040+0000 7f8efcec9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x562770128ca0 con 0x5627700c2b90 2023-11-16T16:54:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.040+0000 7f8efcec9b80 10 client.0 _ll_register_callbacks cb 0x7f8edc0093e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.040+0000 7f8ee2ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56276ff186b0 0x5627700c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.040+0000 7f8ee37fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2b90 0x5627700ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.040+0000 7f8ee37fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2b90 0x5627700ca060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:49542/0 (socket says 172.21.15.44:49542) 2023-11-16T16:54:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.040+0000 7f8ee37fe700 1 -- 172.21.15.44:0/19215969 learned_addr learned my addr 172.21.15.44:0/19215969 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:54:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.040+0000 7f8ee2ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56276ff186b0 0x5627700c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60238/0 (socket says 172.21.15.44:60238) 2023-11-16T16:54:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.040+0000 7f8ee27fc700 1 --2- 172.21.15.44:0/19215969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2720 0x5627700c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.040+0000 7f8ee37fe700 1 -- 172.21.15.44:0/19215969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2720 msgr2=0x5627700c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.040+0000 7f8ee37fe700 1 --2- 172.21.15.44:0/19215969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2720 0x5627700c9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.041+0000 7f8ee37fe700 1 -- 172.21.15.44:0/19215969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56276ff186b0 msgr2=0x5627700c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.041+0000 7f8ee37fe700 1 --2- 172.21.15.44:0/19215969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56276ff186b0 0x5627700c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.041+0000 7f8ee37fe700 1 -- 172.21.15.44:0/19215969 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed8019070 con 0x5627700c2b90 2023-11-16T16:54:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.041+0000 7f8ee2ffd700 1 --2- 172.21.15.44:0/19215969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56276ff186b0 0x5627700c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.041+0000 7f8ee37fe700 1 --2- 172.21.15.44:0/19215969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2b90 0x5627700ca060 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8edc095ee0 tx=0x7f8edc0999e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa16464230682963 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.041+0000 7f8ec6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:54:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.042+0000 7f8ec6ff5700 1 -- 172.21.15.44:0/19215969 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc0a1da0 con 0x5627700c2b90 2023-11-16T16:54:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.042+0000 7f8efcec9b80 1 -- 172.21.15.44:0/19215969 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ed40037d0 con 0x5627700c2b90 2023-11-16T16:54:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.042+0000 7f8ec6ff5700 1 -- 172.21.15.44:0/19215969 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f8edc09ae40 con 0x5627700c2b90 2023-11-16T16:54:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.042+0000 7f8ec6ff5700 1 -- 172.21.15.44:0/19215969 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8edc00cd50 con 0x5627700c2b90 2023-11-16T16:54:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.042+0000 7f8efcec9b80 10 client.26590 fetch_fsmap learned FSMap version 21 2023-11-16T16:54:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.042+0000 7f8efcec9b80 1 -- 172.21.15.44:0/19215969 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x56276fe24c00 con 0x5627700c2b90 2023-11-16T16:54:44.046 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.043+0000 7f8ec6ff5700 1 -- 172.21.15.44:0/19215969 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ed40037d0 con 0x5627700c2b90 2023-11-16T16:54:44.046 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.043+0000 7f8ec6ff5700 1 -- 172.21.15.44:0/19215969 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8edc0957d0 con 0x5627700c2b90 2023-11-16T16:54:44.046 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.043+0000 7f8efcec9b80 10 client.26590 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:54:44.046 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.043+0000 7f8efcec9b80 10 client.26590 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:44.046 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.043+0000 7f8efcec9b80 20 client.26590 populate_metadata read hostname 'smithi044' 2023-11-16T16:54:44.046 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.043+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/19215969 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56277012a320 0x56277012c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.043+0000 7f8efcec9b80 4 client.26590 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:54:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.043+0000 7f8efcec9b80 1 -- 172.21.15.44:0/19215969 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56276ffdc600 con 0x56277012a320 2023-11-16T16:54:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.043+0000 7f8ee2ffd700 1 --2- 172.21.15.44:0/19215969 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56277012a320 0x56277012c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.044+0000 7f8ee2ffd700 1 --2- 172.21.15.44:0/19215969 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56277012a320 0x56277012c700 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f92b904951712378 server_cookie=c89d11f1dd996439 in_seq=0 out_seq=0 2023-11-16T16:54:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:44.044+0000 7f8ec6ff5700 10 client.26590 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:45.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:45 smithi044 ceph-mon[128750]: pgmap v147: 129 pgs: 129 active+clean; 36 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 2.21k op/s 2023-11-16T16:54:45.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:45 smithi176 ceph-mon[131630]: pgmap v147: 129 pgs: 129 active+clean; 36 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 2.21k op/s 2023-11-16T16:54:45.729 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:45 smithi077 ceph-mon[131204]: pgmap v147: 129 pgs: 129 active+clean; 36 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 2.21k op/s 2023-11-16T16:54:47.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:47 smithi044 ceph-mon[128750]: pgmap v148: 129 pgs: 129 active+clean; 36 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 811 op/s 2023-11-16T16:54:47.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:47 smithi176 ceph-mon[131630]: pgmap v148: 129 pgs: 129 active+clean; 36 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 811 op/s 2023-11-16T16:54:47.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:47 smithi077 ceph-mon[131204]: pgmap v148: 129 pgs: 129 active+clean; 36 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 811 op/s 2023-11-16T16:54:48.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.990+0000 7f8ec6ff5700 1 -- 172.21.15.44:0/19215969 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56276ffdc600 con 0x56277012a320 2023-11-16T16:54:48.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.990+0000 7f8ec6ff5700 10 client.26590 handle_command_reply: tid=0 2023-11-16T16:54:48.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.990+0000 7f8efcec9b80 1 client.26590 shutdown 2023-11-16T16:54:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.990+0000 7f8efcec9b80 1 -- 172.21.15.44:0/19215969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2b90 msgr2=0x5627700ca060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.990+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/19215969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2b90 0x5627700ca060 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8edc095ee0 tx=0x7f8edc0999e0 comp rx=0 tx=0).stop 2023-11-16T16:54:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.990+0000 7f8efcec9b80 1 -- 172.21.15.44:0/19215969 shutdown_connections 2023-11-16T16:54:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.990+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/19215969 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56277012a320 0x56277012c700 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.990+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/19215969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5627700c2b90 0x5627700ca060 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.990+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/19215969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56276ff186b0 0x5627700c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.990+0000 7f8efcec9b80 1 --2- 172.21.15.44:0/19215969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627700c2720 0x5627700c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.990+0000 7f8ec6ff5700 0 client.26590 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:54:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.990+0000 7f8efcec9b80 1 -- 172.21.15.44:0/19215969 >> 172.21.15.44:0/19215969 conn(0x5627700b1f50 msgr2=0x5627700c2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.990+0000 7f8efcec9b80 1 -- 172.21.15.44:0/19215969 shutdown_connections 2023-11-16T16:54:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.990+0000 7f8efcec9b80 1 -- 172.21.15.44:0/19215969 wait complete. 2023-11-16T16:54:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.991+0000 7f8efcec9b80 20 client.26590 trim_cache size 0 max 16384 2023-11-16T16:54:48.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:54:48.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:54:48.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:54:48.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:54:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.991+0000 7f8ee95ff700 1 -- 172.21.15.44:0/2883742124 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f8ecc0a1340 msgr2=0x7f8ecc0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.991+0000 7f8ee95ff700 1 --2- 172.21.15.44:0/2883742124 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f8ecc0a1340 0x7f8ecc0a37f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40745e0 tx=0x7f8ed800b360 comp rx=0 tx=0).stop 2023-11-16T16:54:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.991+0000 7f8ee95ff700 1 -- 172.21.15.44:0/2883742124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee4105560 msgr2=0x7f8ee41a45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.991+0000 7f8ee95ff700 1 --2- 172.21.15.44:0/2883742124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee4105560 0x7f8ee41a45f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00ef30 tx=0x7f8edc00c450 comp rx=0 tx=0).stop 2023-11-16T16:54:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.991+0000 7f8ee95ff700 1 -- 172.21.15.44:0/2883742124 shutdown_connections 2023-11-16T16:54:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.991+0000 7f8ee95ff700 1 --2- 172.21.15.44:0/2883742124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee4105560 0x7f8ee41a45f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.991+0000 7f8ee95ff700 1 --2- 172.21.15.44:0/2883742124 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f8ecc0a1340 0x7f8ecc0a37f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.991+0000 7f8ee95ff700 1 --2- 172.21.15.44:0/2883742124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ee4100c50 0x7f8ee4072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.991+0000 7f8ee95ff700 1 --2- 172.21.15.44:0/2883742124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ee41015f0 0x7f8ee4073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.991+0000 7f8ee95ff700 1 -- 172.21.15.44:0/2883742124 >> 172.21.15.44:0/2883742124 conn(0x7f8ee4077550 msgr2=0x7f8ee40fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.991+0000 7f8ee95ff700 1 -- 172.21.15.44:0/2883742124 shutdown_connections 2023-11-16T16:54:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:54:48.992+0000 7f8ee95ff700 1 -- 172.21.15.44:0/2883742124 wait complete. 2023-11-16T16:54:49.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:49 smithi044 ceph-mon[128750]: pgmap v149: 129 pgs: 129 active+clean; 41 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.5 MiB/s wr, 3.40k op/s 2023-11-16T16:54:49.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:49 smithi176 ceph-mon[131630]: pgmap v149: 129 pgs: 129 active+clean; 41 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.5 MiB/s wr, 3.40k op/s 2023-11-16T16:54:49.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:49 smithi077 ceph-mon[131204]: pgmap v149: 129 pgs: 129 active+clean; 41 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.5 MiB/s wr, 3.40k op/s 2023-11-16T16:54:51.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:51 smithi044 ceph-mon[128750]: pgmap v150: 129 pgs: 129 active+clean; 41 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 3.39k op/s 2023-11-16T16:54:51.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:51 smithi176 ceph-mon[131630]: pgmap v150: 129 pgs: 129 active+clean; 41 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 3.39k op/s 2023-11-16T16:54:51.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:51 smithi077 ceph-mon[131204]: pgmap v150: 129 pgs: 129 active+clean; 41 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 3.39k op/s 2023-11-16T16:54:52.420 INFO:tasks.workunit.client.0.smithi044.stdout:created hierarchy, now cleaning up 2023-11-16T16:54:53.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:53 smithi044 ceph-mon[128750]: pgmap v151: 129 pgs: 129 active+clean; 21 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 3.61k op/s 2023-11-16T16:54:53.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:53 smithi176 ceph-mon[131630]: pgmap v151: 129 pgs: 129 active+clean; 21 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 3.61k op/s 2023-11-16T16:54:53.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:53 smithi077 ceph-mon[131204]: pgmap v151: 129 pgs: 129 active+clean; 21 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 3.61k op/s 2023-11-16T16:54:55.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:55 smithi044 ceph-mon[128750]: pgmap v152: 129 pgs: 129 active+clean; 9.2 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.5 MiB/s wr, 4.97k op/s 2023-11-16T16:54:55.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:55 smithi176 ceph-mon[131630]: pgmap v152: 129 pgs: 129 active+clean; 9.2 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.5 MiB/s wr, 4.97k op/s 2023-11-16T16:54:55.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:55 smithi077 ceph-mon[131204]: pgmap v152: 129 pgs: 129 active+clean; 9.2 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.5 MiB/s wr, 4.97k op/s 2023-11-16T16:54:57.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:57 smithi044 ceph-mon[128750]: pgmap v153: 129 pgs: 129 active+clean; 9.2 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 4.46k op/s 2023-11-16T16:54:57.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:57 smithi176 ceph-mon[131630]: pgmap v153: 129 pgs: 129 active+clean; 9.2 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 4.46k op/s 2023-11-16T16:54:57.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:57 smithi077 ceph-mon[131204]: pgmap v153: 129 pgs: 129 active+clean; 9.2 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 4.46k op/s 2023-11-16T16:54:59.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:54:59 smithi044 ceph-mon[128750]: pgmap v154: 129 pgs: 129 active+clean; 17 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 5.04k op/s 2023-11-16T16:55:05.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:01 smithi044 ceph-mon[128750]: pgmap v155: 129 pgs: 129 active+clean; 17 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 2.46k op/s 2023-11-16T16:55:05.401 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:54:59 smithi077 ceph-mon[131204]: pgmap v154: 129 pgs: 129 active+clean; 17 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 5.04k op/s 2023-11-16T16:55:05.401 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:01 smithi077 ceph-mon[131204]: pgmap v155: 129 pgs: 129 active+clean; 17 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 2.46k op/s 2023-11-16T16:55:05.401 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:03 smithi077 ceph-mon[131204]: pgmap v156: 129 pgs: 129 active+clean; 17 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 2.54k op/s 2023-11-16T16:55:05.402 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:54:59 smithi176 ceph-mon[131630]: pgmap v154: 129 pgs: 129 active+clean; 17 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 5.04k op/s 2023-11-16T16:55:05.402 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:01 smithi176 ceph-mon[131630]: pgmap v155: 129 pgs: 129 active+clean; 17 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 2.46k op/s 2023-11-16T16:55:05.402 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:03 smithi176 ceph-mon[131630]: pgmap v156: 129 pgs: 129 active+clean; 17 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 2.54k op/s 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.147+0000 7efc41dff700 1 -- 172.21.15.44:0/1831671665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc3c100c50 msgr2=0x7efc3c101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.147+0000 7efc41dff700 1 --2- 172.21.15.44:0/1831671665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc3c100c50 0x7efc3c101020 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efc2c009990 tx=0x7efc2c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.147+0000 7efc41dff700 1 -- 172.21.15.44:0/1831671665 shutdown_connections 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.147+0000 7efc41dff700 1 --2- 172.21.15.44:0/1831671665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc3c105560 0x7efc3c10d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.147+0000 7efc41dff700 1 --2- 172.21.15.44:0/1831671665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc3c100c50 0x7efc3c101020 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.147+0000 7efc41dff700 1 --2- 172.21.15.44:0/1831671665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc3c1015f0 0x7efc3c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.147+0000 7efc41dff700 1 -- 172.21.15.44:0/1831671665 >> 172.21.15.44:0/1831671665 conn(0x7efc3c077550 msgr2=0x7efc3c077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.147+0000 7efc41dff700 1 -- 172.21.15.44:0/1831671665 shutdown_connections 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.147+0000 7efc41dff700 1 -- 172.21.15.44:0/1831671665 wait complete. 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.148+0000 7efc41dff700 1 Processor -- start 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.149+0000 7efc41dff700 1 -- start start 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.149+0000 7efc41dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc3c100c50 0x7efc3c1a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.149+0000 7efc41dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc3c1015f0 0x7efc3c1a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.149+0000 7efc41dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc3c105560 0x7efc3c10be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.149+0000 7efc41dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c10c5b0 con 0x7efc3c1015f0 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.149+0000 7efc41dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c10c720 con 0x7efc3c100c50 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.149+0000 7efc41dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c10c8e0 con 0x7efc3c105560 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.149+0000 7efc3affd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc3c1015f0 0x7efc3c1a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.149+0000 7efc3bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc3c105560 0x7efc3c10be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.149+0000 7efc3affd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc3c1015f0 0x7efc3c1a1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57800/0 (socket says 172.21.15.44:57800) 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.149+0000 7efc3affd700 1 -- 172.21.15.44:0/2263890226 learned_addr learned my addr 172.21.15.44:0/2263890226 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.149+0000 7efc3b7fe700 1 --2- 172.21.15.44:0/2263890226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc3c100c50 0x7efc3c1a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.150+0000 7efc3bfff700 1 -- 172.21.15.44:0/2263890226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc3c100c50 msgr2=0x7efc3c1a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.150+0000 7efc3bfff700 1 --2- 172.21.15.44:0/2263890226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc3c100c50 0x7efc3c1a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.150+0000 7efc3bfff700 1 -- 172.21.15.44:0/2263890226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc3c1015f0 msgr2=0x7efc3c1a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.150+0000 7efc3bfff700 1 --2- 172.21.15.44:0/2263890226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc3c1015f0 0x7efc3c1a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.150+0000 7efc3bfff700 1 -- 172.21.15.44:0/2263890226 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2c0095f0 con 0x7efc3c105560 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.150+0000 7efc3affd700 1 --2- 172.21.15.44:0/2263890226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc3c1015f0 0x7efc3c1a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.150+0000 7efc3bfff700 1 --2- 172.21.15.44:0/2263890226 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc3c105560 0x7efc3c10be50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efc3400efc0 tx=0x7efc3400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.150+0000 7efc38ff9700 1 -- 172.21.15.44:0/2263890226 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc34018070 con 0x7efc3c105560 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.150+0000 7efc38ff9700 1 -- 172.21.15.44:0/2263890226 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc3400f040 con 0x7efc3c105560 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.151+0000 7efc41dff700 1 -- 172.21.15.44:0/2263890226 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc3c1103b0 con 0x7efc3c105560 2023-11-16T16:55:05.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.151+0000 7efc38ff9700 1 -- 172.21.15.44:0/2263890226 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3400cd40 con 0x7efc3c105560 2023-11-16T16:55:05.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.151+0000 7efc41dff700 1 -- 172.21.15.44:0/2263890226 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc3c110220 con 0x7efc3c105560 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.154+0000 7efc38ff9700 1 -- 172.21.15.44:0/2263890226 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7efc34003a00 con 0x7efc3c105560 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.155+0000 7efc556bab80 1 Processor -- start 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.155+0000 7efc556bab80 1 -- start start 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.155+0000 7efc38ff9700 1 --2- 172.21.15.44:0/2263890226 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7efc240a1340 0x7efc240a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.155+0000 7efc38ff9700 1 -- 172.21.15.44:0/2263890226 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7efc340ccb70 con 0x7efc3c105560 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc556bab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073fa91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc556bab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9720 0x562073fa3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc556bab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9b90 0x562073fa3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc556bab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562073ea9000 con 0x562073fa9b90 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc556bab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562073efb840 con 0x562073dff6b0 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc556bab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x562073ef4cd0 con 0x562073fa9720 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc3affd700 1 --2- 172.21.15.44:0/2263890226 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7efc240a1340 0x7efc240a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc3b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073fa91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc3affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9720 0x562073fa3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc3b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073fa91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:40586/0 (socket says 172.21.15.44:40586) 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc3b7fe700 1 -- 172.21.15.44:0/333143836 learned_addr learned my addr 172.21.15.44:0/333143836 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc3bfff700 1 --2- 172.21.15.44:0/333143836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9b90 0x562073fa3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc3b7fe700 1 -- 172.21.15.44:0/333143836 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9720 msgr2=0x562073fa3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc3b7fe700 1 --2- 172.21.15.44:0/333143836 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9720 0x562073fa3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc3b7fe700 1 -- 172.21.15.44:0/333143836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9b90 msgr2=0x562073fa3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc3b7fe700 1 --2- 172.21.15.44:0/333143836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9b90 0x562073fa3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc3b7fe700 1 -- 172.21.15.44:0/333143836 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562073d765d0 con 0x562073dff6b0 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc3bfff700 1 --2- 172.21.15.44:0/333143836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9b90 0x562073fa3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.156+0000 7efc3affd700 1 --2- 172.21.15.44:0/2263890226 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7efc240a1340 0x7efc240a37f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efc3c10cf90 tx=0x7efc30007c50 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.157+0000 7efc3b7fe700 1 --2- 172.21.15.44:0/333143836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073fa91e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efc2c005f50 tx=0x7efc2c00b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e1bbd6a65f2a0d2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.157+0000 7efc21ffb700 1 -- 172.21.15.44:0/333143836 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c01d070 con 0x562073dff6b0 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.157+0000 7efc21ffb700 1 -- 172.21.15.44:0/333143836 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc2c01e040 con 0x562073dff6b0 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.157+0000 7efc21ffb700 1 -- 172.21.15.44:0/333143836 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c0048c0 con 0x562073dff6b0 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.157+0000 7efc556bab80 1 -- 172.21.15.44:0/333143836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 msgr2=0x562073fa91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.157+0000 7efc556bab80 1 --2- 172.21.15.44:0/333143836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073fa91e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efc2c005f50 tx=0x7efc2c00b610 comp rx=0 tx=0).stop 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.158+0000 7efc556bab80 1 -- 172.21.15.44:0/333143836 shutdown_connections 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.158+0000 7efc556bab80 1 --2- 172.21.15.44:0/333143836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9b90 0x562073fa3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.158+0000 7efc556bab80 1 --2- 172.21.15.44:0/333143836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073fa91e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.158+0000 7efc556bab80 1 --2- 172.21.15.44:0/333143836 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9720 0x562073fa3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.158+0000 7efc556bab80 1 -- 172.21.15.44:0/333143836 >> 172.21.15.44:0/333143836 conn(0x562073f98f50 msgr2=0x562073f9e270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.158+0000 7efc556bab80 1 -- 172.21.15.44:0/333143836 shutdown_connections 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.158+0000 7efc556bab80 1 -- 172.21.15.44:0/333143836 wait complete. 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.158+0000 7efc556bab80 1 -- start start 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.158+0000 7efc556bab80 1 Processor -- start 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.159+0000 7efc556bab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073fb2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.159+0000 7efc556bab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9720 0x562073fb2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.159+0000 7efc556bab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9b90 0x562073facc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.159+0000 7efc556bab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562073e96880 con 0x562073fa9720 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.159+0000 7efc556bab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562073e96520 con 0x562073dff6b0 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.159+0000 7efc556bab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x562073d788c0 con 0x562073fa9b90 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.159+0000 7efc556bab80 10 client.0 _ll_register_callbacks cb 0x562073eb4720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.159+0000 7efc3b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073fb2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.159+0000 7efc3b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073fb2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:40596/0 (socket says 172.21.15.44:40596) 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.159+0000 7efc3b7fe700 1 -- 172.21.15.44:0/3949094434 learned_addr learned my addr 172.21.15.44:0/3949094434 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.159+0000 7efc3affd700 1 --2- 172.21.15.44:0/3949094434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9720 0x562073fb2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.159+0000 7efc3bfff700 1 --2- 172.21.15.44:0/3949094434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9b90 0x562073facc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.160+0000 7efc3b7fe700 1 -- 172.21.15.44:0/3949094434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9b90 msgr2=0x562073facc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.160+0000 7efc3b7fe700 1 --2- 172.21.15.44:0/3949094434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9b90 0x562073facc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.160+0000 7efc3b7fe700 1 -- 172.21.15.44:0/3949094434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9720 msgr2=0x562073fb2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.160+0000 7efc3b7fe700 1 --2- 172.21.15.44:0/3949094434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9720 0x562073fb2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.160+0000 7efc3b7fe700 1 -- 172.21.15.44:0/3949094434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc3c10d9b0 con 0x562073dff6b0 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.160+0000 7efc3bfff700 1 --2- 172.21.15.44:0/3949094434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9b90 0x562073facc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.160+0000 7efc3b7fe700 1 --2- 172.21.15.44:0/3949094434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073fb2a40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efc2c009990 tx=0x7efc2c00b680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9012c7c5c1aa7090 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.160+0000 7efc1affd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.160+0000 7efc1affd700 1 -- 172.21.15.44:0/3949094434 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c01d040 con 0x562073dff6b0 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.160+0000 7efc556bab80 1 -- 172.21.15.44:0/3949094434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562073f96710 con 0x562073dff6b0 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.160+0000 7efc1affd700 1 -- 172.21.15.44:0/3949094434 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7efc2c00faf0 con 0x562073dff6b0 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.160+0000 7efc1affd700 1 -- 172.21.15.44:0/3949094434 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc2c00fcb0 con 0x562073dff6b0 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.160+0000 7efc556bab80 10 client.26615 fetch_fsmap learned FSMap version 21 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.160+0000 7efc556bab80 1 -- 172.21.15.44:0/3949094434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x562073ebb130 con 0x562073dff6b0 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.161+0000 7efc1affd700 1 -- 172.21.15.44:0/3949094434 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562073f96710 con 0x562073dff6b0 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.161+0000 7efc1affd700 1 -- 172.21.15.44:0/3949094434 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7efc2c031020 con 0x562073dff6b0 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.161+0000 7efc556bab80 10 client.26615 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.161+0000 7efc556bab80 10 client.26615 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.161+0000 7efc556bab80 20 client.26615 populate_metadata read hostname 'smithi044' 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.161+0000 7efc556bab80 1 --2- 172.21.15.44:0/3949094434 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562073fbca90 0x562073fbeea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.161+0000 7efc556bab80 4 client.26615 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.161+0000 7efc556bab80 1 -- 172.21.15.44:0/3949094434 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562073eb7040 con 0x562073fbca90 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.161+0000 7efc3affd700 1 --2- 172.21.15.44:0/3949094434 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562073fbca90 0x562073fbeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.162+0000 7efc3affd700 1 --2- 172.21.15.44:0/3949094434 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562073fbca90 0x562073fbeea0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ec96ac977de5bfa server_cookie=245cdb4f07117a6f in_seq=0 out_seq=0 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.162+0000 7efc1affd700 10 client.26615 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.164+0000 7efc1affd700 1 -- 172.21.15.44:0/3949094434 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562073eb7040 con 0x562073fbca90 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.164+0000 7efc1affd700 10 client.26615 handle_command_reply: tid=0 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.164+0000 7efc556bab80 1 client.26615 shutdown 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.164+0000 7efc556bab80 1 -- 172.21.15.44:0/3949094434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 msgr2=0x562073fb2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.164+0000 7efc556bab80 1 --2- 172.21.15.44:0/3949094434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073fb2a40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efc2c009990 tx=0x7efc2c00b680 comp rx=0 tx=0).stop 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.164+0000 7efc556bab80 1 -- 172.21.15.44:0/3949094434 shutdown_connections 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.164+0000 7efc556bab80 1 --2- 172.21.15.44:0/3949094434 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562073fbca90 0x562073fbeea0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.164+0000 7efc556bab80 1 --2- 172.21.15.44:0/3949094434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9b90 0x562073facc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.164+0000 7efc556bab80 1 --2- 172.21.15.44:0/3949094434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073fb2a40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.164+0000 7efc556bab80 1 --2- 172.21.15.44:0/3949094434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9720 0x562073fb2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.164+0000 7efc1affd700 0 client.26615 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.164+0000 7efc556bab80 1 -- 172.21.15.44:0/3949094434 >> 172.21.15.44:0/3949094434 conn(0x562073f98f50 msgr2=0x562073f9d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.164+0000 7efc556bab80 1 -- 172.21.15.44:0/3949094434 shutdown_connections 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.164+0000 7efc556bab80 1 -- 172.21.15.44:0/3949094434 wait complete. 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.165+0000 7efc556bab80 20 client.26615 trim_cache size 0 max 16384 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.180+0000 7efc556bab80 1 Processor -- start 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.180+0000 7efc556bab80 1 -- start start 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.180+0000 7efc556bab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073f9df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc556bab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9720 0x562073fac500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc556bab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9b90 0x562073faca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc556bab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562073fbf500 con 0x562073fa9720 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc556bab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562073fab1f0 con 0x562073dff6b0 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc556bab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x562073fa76f0 con 0x562073fa9b90 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc3b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073f9df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc3affd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9720 0x562073fac500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc3b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073f9df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:40608/0 (socket says 172.21.15.44:40608) 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc3b7fe700 1 -- 172.21.15.44:0/2392121798 learned_addr learned my addr 172.21.15.44:0/2392121798 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc3bfff700 1 --2- 172.21.15.44:0/2392121798 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9b90 0x562073faca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc3b7fe700 1 -- 172.21.15.44:0/2392121798 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9b90 msgr2=0x562073faca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc3b7fe700 1 --2- 172.21.15.44:0/2392121798 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9b90 0x562073faca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc3b7fe700 1 -- 172.21.15.44:0/2392121798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9720 msgr2=0x562073fac500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc3b7fe700 1 --2- 172.21.15.44:0/2392121798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9720 0x562073fac500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc3b7fe700 1 -- 172.21.15.44:0/2392121798 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562073f9a6f0 con 0x562073dff6b0 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc3affd700 1 --2- 172.21.15.44:0/2392121798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9720 0x562073fac500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.181+0000 7efc3b7fe700 1 --2- 172.21.15.44:0/2392121798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073f9df60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efc2c033fd0 tx=0x7efc2c022400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5127f3bae77f6ce server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.182+0000 7efc227fc700 1 -- 172.21.15.44:0/2392121798 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c02bd90 con 0x562073dff6b0 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.182+0000 7efc227fc700 1 -- 172.21.15.44:0/2392121798 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc2c02b8b0 con 0x562073dff6b0 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.182+0000 7efc227fc700 1 -- 172.21.15.44:0/2392121798 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c022e00 con 0x562073dff6b0 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.182+0000 7efc556bab80 1 -- 172.21.15.44:0/2392121798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 msgr2=0x562073f9df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.182+0000 7efc556bab80 1 --2- 172.21.15.44:0/2392121798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073f9df60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efc2c033fd0 tx=0x7efc2c022400 comp rx=0 tx=0).stop 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.182+0000 7efc556bab80 1 -- 172.21.15.44:0/2392121798 shutdown_connections 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.182+0000 7efc556bab80 1 --2- 172.21.15.44:0/2392121798 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9b90 0x562073faca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.182+0000 7efc556bab80 1 --2- 172.21.15.44:0/2392121798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073dff6b0 0x562073f9df60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.182+0000 7efc556bab80 1 --2- 172.21.15.44:0/2392121798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073fa9720 0x562073fac500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.182+0000 7efc556bab80 1 -- 172.21.15.44:0/2392121798 >> 172.21.15.44:0/2392121798 conn(0x562073f98f50 msgr2=0x562073f9fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.182+0000 7efc556bab80 1 -- 172.21.15.44:0/2392121798 shutdown_connections 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.182+0000 7efc556bab80 1 -- 172.21.15.44:0/2392121798 wait complete. 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.183+0000 7efc556bab80 1 -- start start 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.183+0000 7efc556bab80 1 Processor -- start 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc556bab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073dff6b0 0x562073fb2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc556bab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9720 0x562073fb05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc556bab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073fa9b90 0x562073fb0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc556bab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562073fb1200 con 0x562073dff6b0 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc556bab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562073fb1370 con 0x562073fa9b90 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc556bab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207400f910 con 0x562073fa9720 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc556bab80 10 client.0 _ll_register_callbacks cb 0x7efc2c005ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc3b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073dff6b0 0x562073fb2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc3bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073fa9b90 0x562073fb0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc3b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073dff6b0 0x562073fb2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57838/0 (socket says 172.21.15.44:57838) 2023-11-16T16:55:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc3b7fe700 1 -- 172.21.15.44:0/802987640 learned_addr learned my addr 172.21.15.44:0/802987640 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc3affd700 1 --2- 172.21.15.44:0/802987640 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9720 0x562073fb05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc3b7fe700 1 -- 172.21.15.44:0/802987640 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9720 msgr2=0x562073fb05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc3b7fe700 1 --2- 172.21.15.44:0/802987640 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9720 0x562073fb05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc3b7fe700 1 -- 172.21.15.44:0/802987640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073fa9b90 msgr2=0x562073fb0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc3b7fe700 1 --2- 172.21.15.44:0/802987640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073fa9b90 0x562073fb0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.184+0000 7efc3b7fe700 1 -- 172.21.15.44:0/802987640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc30017040 con 0x562073dff6b0 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.185+0000 7efc3b7fe700 1 --2- 172.21.15.44:0/802987640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073dff6b0 0x562073fb2de0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7efc2c022840 tx=0x7efc2c0058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807f1501ac7e2648 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.185+0000 7efc1affd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.185+0000 7efc1affd700 1 -- 172.21.15.44:0/802987640 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c044030 con 0x562073dff6b0 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.185+0000 7efc556bab80 1 -- 172.21.15.44:0/802987640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efc2c01e040 con 0x562073dff6b0 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.185+0000 7efc1affd700 1 -- 172.21.15.44:0/802987640 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7efc2c045070 con 0x562073dff6b0 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.185+0000 7efc1affd700 1 -- 172.21.15.44:0/802987640 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc2c01d030 con 0x562073dff6b0 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.185+0000 7efc556bab80 10 client.16587 fetch_fsmap learned FSMap version 21 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.185+0000 7efc556bab80 1 -- 172.21.15.44:0/802987640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x562073ebb130 con 0x562073dff6b0 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.185+0000 7efc1affd700 1 -- 172.21.15.44:0/802987640 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc2c01e040 con 0x562073dff6b0 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.186+0000 7efc1affd700 1 -- 172.21.15.44:0/802987640 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7efc2c00b850 con 0x562073dff6b0 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.186+0000 7efc556bab80 10 client.16587 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:55:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.186+0000 7efc556bab80 10 client.16587 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.186+0000 7efc556bab80 20 client.16587 populate_metadata read hostname 'smithi044' 2023-11-16T16:55:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.186+0000 7efc556bab80 1 --2- 172.21.15.44:0/802987640 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562074010b90 0x562074012f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.186+0000 7efc556bab80 4 client.16587 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:55:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.186+0000 7efc556bab80 1 -- 172.21.15.44:0/802987640 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562073ec3600 con 0x562074010b90 2023-11-16T16:55:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.186+0000 7efc3affd700 1 --2- 172.21.15.44:0/802987640 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562074010b90 0x562074012f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.186+0000 7efc3affd700 1 --2- 172.21.15.44:0/802987640 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562074010b90 0x562074012f70 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57f9da59c8f68cdb server_cookie=97b97923449c95f0 in_seq=0 out_seq=0 2023-11-16T16:55:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:02.186+0000 7efc1affd700 10 client.16587 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:05.470 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:03 smithi044 ceph-mon[128750]: pgmap v156: 129 pgs: 129 active+clean; 17 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 2.54k op/s 2023-11-16T16:55:05.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:05 smithi044 ceph-mon[128750]: pgmap v157: 129 pgs: 129 active+clean; 27 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 2.72k op/s 2023-11-16T16:55:05.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:05 smithi176 ceph-mon[131630]: pgmap v157: 129 pgs: 129 active+clean; 27 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 2.72k op/s 2023-11-16T16:55:05.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:05 smithi077 ceph-mon[131204]: pgmap v157: 129 pgs: 129 active+clean; 27 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 2.72k op/s 2023-11-16T16:55:07.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:07 smithi176 ceph-mon[131630]: pgmap v158: 129 pgs: 129 active+clean; 27 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 1.05k op/s 2023-11-16T16:55:07.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:07 smithi077 ceph-mon[131204]: pgmap v158: 129 pgs: 129 active+clean; 27 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 1.05k op/s 2023-11-16T16:55:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:07 smithi044 ceph-mon[128750]: pgmap v158: 129 pgs: 129 active+clean; 27 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 1.05k op/s 2023-11-16T16:55:08.177 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:55:08.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.400+0000 7f3409bff700 1 -- 172.21.15.44:0/3127826333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3404074400 msgr2=0x7f34040747d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:08.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.400+0000 7f3409bff700 1 --2- 172.21.15.44:0/3127826333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3404074400 0x7f34040747d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f33f4009990 tx=0x7f33f4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:55:08.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.400+0000 7f3409bff700 1 -- 172.21.15.44:0/3127826333 shutdown_connections 2023-11-16T16:55:08.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.400+0000 7f3409bff700 1 --2- 172.21.15.44:0/3127826333 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f34040736d0 0x7f34041136a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:08.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.400+0000 7f3409bff700 1 --2- 172.21.15.44:0/3127826333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3404074400 0x7f34040747d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:08.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.400+0000 7f3409bff700 1 --2- 172.21.15.44:0/3127826333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3404072d20 0x7f3404073190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:08.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.400+0000 7f3409bff700 1 -- 172.21.15.44:0/3127826333 >> 172.21.15.44:0/3127826333 conn(0x7f3404077560 msgr2=0x7f3404077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:08.403 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.400+0000 7f3409bff700 1 -- 172.21.15.44:0/3127826333 shutdown_connections 2023-11-16T16:55:08.403 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.400+0000 7f3409bff700 1 -- 172.21.15.44:0/3127826333 wait complete. 2023-11-16T16:55:08.403 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.401+0000 7f3409bff700 1 Processor -- start 2023-11-16T16:55:08.403 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.402+0000 7f3409bff700 1 -- start start 2023-11-16T16:55:08.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.402+0000 7f3409bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3404072d20 0x7f34041a2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:08.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.402+0000 7f3409bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34040736d0 0x7f34041a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:08.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.402+0000 7f3409bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3404074400 0x7f340419c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:08.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.402+0000 7f3409bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340419cfe0 con 0x7f34040736d0 2023-11-16T16:55:08.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.402+0000 7f3409bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340419d150 con 0x7f3404074400 2023-11-16T16:55:08.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.402+0000 7f3409bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340419d310 con 0x7f3404072d20 2023-11-16T16:55:08.405 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.402+0000 7f3403fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3404074400 0x7f340419c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:08.405 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.402+0000 7f34037fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3404072d20 0x7f34041a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:08.405 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.402+0000 7f3402ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34040736d0 0x7f34041a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:08.405 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.402+0000 7f34037fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3404072d20 0x7f34041a2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:57000/0 (socket says 172.21.15.44:57000) 2023-11-16T16:55:08.405 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.402+0000 7f3403fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3404074400 0x7f340419c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:36110/0 (socket says 172.21.15.44:36110) 2023-11-16T16:55:08.405 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.402+0000 7f34037fe700 1 -- 172.21.15.44:0/3447492184 learned_addr learned my addr 172.21.15.44:0/3447492184 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:08.405 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.403+0000 7f3403fff700 1 -- 172.21.15.44:0/3447492184 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3404072d20 msgr2=0x7f34041a2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:08.405 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.403+0000 7f3403fff700 1 --2- 172.21.15.44:0/3447492184 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3404072d20 0x7f34041a2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:08.405 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.403+0000 7f3403fff700 1 -- 172.21.15.44:0/3447492184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34040736d0 msgr2=0x7f34041a2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:08.405 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.403+0000 7f3403fff700 1 --2- 172.21.15.44:0/3447492184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34040736d0 0x7f34041a2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:08.405 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.403+0000 7f3403fff700 1 -- 172.21.15.44:0/3447492184 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f8009520 con 0x7f3404074400 2023-11-16T16:55:08.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.403+0000 7f3403fff700 1 --2- 172.21.15.44:0/3447492184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3404074400 0x7f340419c970 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f33fc00efc0 tx=0x7f33fc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:08.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.403+0000 7f3400ff9700 1 -- 172.21.15.44:0/3447492184 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc018070 con 0x7f3404074400 2023-11-16T16:55:08.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.403+0000 7f3400ff9700 1 -- 172.21.15.44:0/3447492184 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33fc00f040 con 0x7f3404074400 2023-11-16T16:55:08.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.403+0000 7f3409bff700 1 -- 172.21.15.44:0/3447492184 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f40095f0 con 0x7f3404074400 2023-11-16T16:55:08.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.403+0000 7f3400ff9700 1 -- 172.21.15.44:0/3447492184 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc00cd40 con 0x7f3404074400 2023-11-16T16:55:08.407 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.403+0000 7f3409bff700 1 -- 172.21.15.44:0/3447492184 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3404116970 con 0x7f3404074400 2023-11-16T16:55:08.408 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.407+0000 7f3400ff9700 1 -- 172.21.15.44:0/3447492184 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f33fc003a00 con 0x7f3404074400 2023-11-16T16:55:08.409 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.408+0000 7f341d5dcb80 1 Processor -- start 2023-11-16T16:55:08.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.408+0000 7f341d5dcb80 1 -- start start 2023-11-16T16:55:08.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f341d5dcb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1501b60 0x55abd1501f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:08.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f341d5dcb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1502470 0x55abd1506370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:08.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f341d5dcb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 0x55abd1506d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:08.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f341d5dcb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abd1407060 con 0x55abd1502470 2023-11-16T16:55:08.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f341d5dcb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abd14598a0 con 0x55abd1501b60 2023-11-16T16:55:08.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f341d5dcb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abd1452d30 con 0x55abd15068b0 2023-11-16T16:55:08.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f3402ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1501b60 0x55abd1501f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:08.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f3400ff9700 1 --2- 172.21.15.44:0/3447492184 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f33ec0a12c0 0x7f33ec0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:08.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f3402ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1501b60 0x55abd1501f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:36122/0 (socket says 172.21.15.44:36122) 2023-11-16T16:55:08.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f3402ffd700 1 -- 172.21.15.44:0/3462409060 learned_addr learned my addr 172.21.15.44:0/3462409060 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:08.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f3400ff9700 1 -- 172.21.15.44:0/3447492184 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f33fc0ccbf0 con 0x7f3404074400 2023-11-16T16:55:08.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f34037fe700 1 --2- 172.21.15.44:0/3462409060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1502470 0x55abd1506370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:08.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f3402ffd700 1 --2- 172.21.15.44:0/3462409060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 0x55abd1506d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:08.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f3403fff700 1 --2- 172.21.15.44:0/3447492184 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f33ec0a12c0 0x7f33ec0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:08.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f3402ffd700 1 -- 172.21.15.44:0/3462409060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 msgr2=0x55abd1506d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:08.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f3402ffd700 1 --2- 172.21.15.44:0/3462409060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 0x55abd1506d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:08.413 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f3402ffd700 1 -- 172.21.15.44:0/3462409060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1502470 msgr2=0x55abd1506370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:08.413 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f3402ffd700 1 --2- 172.21.15.44:0/3462409060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1502470 0x55abd1506370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:08.413 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.409+0000 7f3402ffd700 1 -- 172.21.15.44:0/3462409060 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55abd12d4630 con 0x55abd1501b60 2023-11-16T16:55:08.413 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.410+0000 7f3402ffd700 1 --2- 172.21.15.44:0/3462409060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1501b60 0x55abd1501f30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f33f8013780 tx=0x7f33f8009a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f1d21b4e6adcc13 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:08.413 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.410+0000 7f33e9ffb700 1 -- 172.21.15.44:0/3462409060 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f8004050 con 0x55abd1501b60 2023-11-16T16:55:08.413 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.410+0000 7f33e9ffb700 1 -- 172.21.15.44:0/3462409060 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33f8003680 con 0x55abd1501b60 2023-11-16T16:55:08.413 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.410+0000 7f33e9ffb700 1 -- 172.21.15.44:0/3462409060 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f8010730 con 0x55abd1501b60 2023-11-16T16:55:08.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.410+0000 7f341d5dcb80 1 -- 172.21.15.44:0/3462409060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1501b60 msgr2=0x55abd1501f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:08.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.410+0000 7f3403fff700 1 --2- 172.21.15.44:0/3447492184 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f33ec0a12c0 0x7f33ec0a3770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f33fc00c6f0 tx=0x7f33fc004dc0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:08.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.410+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/3462409060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1501b60 0x55abd1501f30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f33f8013780 tx=0x7f33f8009a10 comp rx=0 tx=0).stop 2023-11-16T16:55:08.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.411+0000 7f341d5dcb80 1 -- 172.21.15.44:0/3462409060 shutdown_connections 2023-11-16T16:55:08.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.411+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/3462409060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 0x55abd1506d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:08.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.411+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/3462409060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1501b60 0x55abd1501f30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:08.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.411+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/3462409060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1502470 0x55abd1506370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:08.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.411+0000 7f341d5dcb80 1 -- 172.21.15.44:0/3462409060 >> 172.21.15.44:0/3462409060 conn(0x55abd14f7360 msgr2=0x55abd14fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:08.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.411+0000 7f341d5dcb80 1 -- 172.21.15.44:0/3462409060 shutdown_connections 2023-11-16T16:55:08.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.411+0000 7f341d5dcb80 1 -- 172.21.15.44:0/3462409060 wait complete. 2023-11-16T16:55:08.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.411+0000 7f341d5dcb80 1 -- start start 2023-11-16T16:55:08.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.411+0000 7f341d5dcb80 1 Processor -- start 2023-11-16T16:55:08.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.412+0000 7f341d5dcb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 0x55abd150e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:08.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.412+0000 7f341d5dcb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1502470 0x55abd150b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:08.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.412+0000 7f341d5dcb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 0x55abd150bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:08.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.412+0000 7f341d5dcb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abd13f48e0 con 0x55abd1501b60 2023-11-16T16:55:08.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.412+0000 7f341d5dcb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abd13f4580 con 0x55abd1502470 2023-11-16T16:55:08.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.412+0000 7f341d5dcb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abd12d6920 con 0x55abd15068b0 2023-11-16T16:55:08.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.412+0000 7f341d5dcb80 10 client.0 _ll_register_callbacks cb 0x55abd1412780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:08.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.412+0000 7f34037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1502470 0x55abd150b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:08.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.412+0000 7f3402ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 0x55abd150e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:08.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.412+0000 7f34037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1502470 0x55abd150b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:36134/0 (socket says 172.21.15.44:36134) 2023-11-16T16:55:08.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.412+0000 7f34037fe700 1 -- 172.21.15.44:0/3922649586 learned_addr learned my addr 172.21.15.44:0/3922649586 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:08.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.412+0000 7f3402ffd700 1 --2- 172.21.15.44:0/3922649586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 0x55abd150e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45604/0 (socket says 172.21.15.44:45604) 2023-11-16T16:55:08.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.413+0000 7f3402ffd700 1 --2- 172.21.15.44:0/3922649586 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 0x55abd150bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:08.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.413+0000 7f34037fe700 1 -- 172.21.15.44:0/3922649586 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 msgr2=0x55abd150bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:08.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.413+0000 7f34037fe700 1 --2- 172.21.15.44:0/3922649586 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 0x55abd150bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:08.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.413+0000 7f34037fe700 1 -- 172.21.15.44:0/3922649586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 msgr2=0x55abd150e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:08.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.413+0000 7f34037fe700 1 --2- 172.21.15.44:0/3922649586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 0x55abd150e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:08.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.413+0000 7f34037fe700 1 -- 172.21.15.44:0/3922649586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f8013460 con 0x55abd1502470 2023-11-16T16:55:08.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.413+0000 7f3402ffd700 1 --2- 172.21.15.44:0/3922649586 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 0x55abd150bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:55:08.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.413+0000 7f34037fe700 1 --2- 172.21.15.44:0/3922649586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1502470 0x55abd150b380 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f33f400bf90 tx=0x7f33f400bfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=976b040307ff18ba server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:08.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.413+0000 7f33e2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:55:08.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.413+0000 7f33e2ffd700 1 -- 172.21.15.44:0/3922649586 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f401d070 con 0x55abd1502470 2023-11-16T16:55:08.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.413+0000 7f341d5dcb80 1 -- 172.21.15.44:0/3922649586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55abd14f4b20 con 0x55abd1502470 2023-11-16T16:55:08.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.413+0000 7f33e2ffd700 1 -- 172.21.15.44:0/3922649586 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f33f4004500 con 0x55abd1502470 2023-11-16T16:55:08.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.413+0000 7f33e2ffd700 1 -- 172.21.15.44:0/3922649586 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33f40046c0 con 0x55abd1502470 2023-11-16T16:55:08.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.413+0000 7f341d5dcb80 10 client.26633 fetch_fsmap learned FSMap version 21 2023-11-16T16:55:08.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.413+0000 7f341d5dcb80 1 -- 172.21.15.44:0/3922649586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55abd1419190 con 0x55abd1502470 2023-11-16T16:55:08.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.414+0000 7f33e2ffd700 1 -- 172.21.15.44:0/3922649586 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55abd14f4b20 con 0x55abd1502470 2023-11-16T16:55:08.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.414+0000 7f33e2ffd700 1 -- 172.21.15.44:0/3922649586 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f33f4024850 con 0x55abd1502470 2023-11-16T16:55:08.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.414+0000 7f341d5dcb80 10 client.26633 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:55:08.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.414+0000 7f341d5dcb80 10 client.26633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:08.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.414+0000 7f341d5dcb80 20 client.26633 populate_metadata read hostname 'smithi044' 2023-11-16T16:55:08.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.414+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/3922649586 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55abd151b8b0 0x55abd151dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:08.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.414+0000 7f341d5dcb80 4 client.26633 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:55:08.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.414+0000 7f341d5dcb80 1 -- 172.21.15.44:0/3922649586 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55abd14150a0 con 0x55abd151b8b0 2023-11-16T16:55:08.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.414+0000 7f3402ffd700 1 --2- 172.21.15.44:0/3922649586 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55abd151b8b0 0x55abd151dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:08.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.415+0000 7f3402ffd700 1 --2- 172.21.15.44:0/3922649586 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55abd151b8b0 0x55abd151dcc0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8a359d0f631ba97 server_cookie=a723b8f597fbd019 in_seq=0 out_seq=0 2023-11-16T16:55:08.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:08.415+0000 7f33e2ffd700 10 client.26633 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:09.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:09 smithi176 ceph-mon[131630]: pgmap v159: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.7 MiB/s wr, 1.85k op/s 2023-11-16T16:55:09.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:09 smithi077 ceph-mon[131204]: pgmap v159: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.7 MiB/s wr, 1.85k op/s 2023-11-16T16:55:09.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:09 smithi044 ceph-mon[128750]: pgmap v159: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.7 MiB/s wr, 1.85k op/s 2023-11-16T16:55:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.492+0000 7efc1affd700 1 -- 172.21.15.44:0/802987640 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562073ec3600 con 0x562074010b90 2023-11-16T16:55:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.492+0000 7efc1affd700 10 client.16587 handle_command_reply: tid=0 2023-11-16T16:55:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.492+0000 7efc556bab80 1 client.16587 shutdown 2023-11-16T16:55:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.493+0000 7efc556bab80 1 -- 172.21.15.44:0/802987640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073dff6b0 msgr2=0x562073fb2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.493+0000 7efc556bab80 1 --2- 172.21.15.44:0/802987640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073dff6b0 0x562073fb2de0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7efc2c022840 tx=0x7efc2c0058e0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.493+0000 7efc556bab80 1 -- 172.21.15.44:0/802987640 shutdown_connections 2023-11-16T16:55:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.493+0000 7efc556bab80 1 --2- 172.21.15.44:0/802987640 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562074010b90 0x562074012f70 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.493+0000 7efc556bab80 1 --2- 172.21.15.44:0/802987640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562073fa9b90 0x562073fb0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.493+0000 7efc556bab80 1 --2- 172.21.15.44:0/802987640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562073dff6b0 0x562073fb2de0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.493+0000 7efc556bab80 1 --2- 172.21.15.44:0/802987640 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562073fa9720 0x562073fb05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.493+0000 7efc1affd700 0 client.16587 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.493+0000 7efc556bab80 1 -- 172.21.15.44:0/802987640 >> 172.21.15.44:0/802987640 conn(0x562073f98f50 msgr2=0x562073fac500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:10.497 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.493+0000 7f33e2ffd700 1 -- 172.21.15.44:0/3922649586 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55abd14150a0 con 0x55abd151b8b0 2023-11-16T16:55:10.497 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.493+0000 7f33e2ffd700 10 client.26633 handle_command_reply: tid=0 2023-11-16T16:55:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.493+0000 7efc556bab80 1 -- 172.21.15.44:0/802987640 shutdown_connections 2023-11-16T16:55:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.493+0000 7efc556bab80 1 -- 172.21.15.44:0/802987640 wait complete. 2023-11-16T16:55:10.498 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.493+0000 7f341d5dcb80 1 client.26633 shutdown 2023-11-16T16:55:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.493+0000 7efc556bab80 20 client.16587 trim_cache size 0 max 16384 2023-11-16T16:55:10.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.493+0000 7f341d5dcb80 1 -- 172.21.15.44:0/3922649586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1502470 msgr2=0x55abd150b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:10.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.493+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/3922649586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1502470 0x55abd150b380 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f33f400bf90 tx=0x7f33f400bfc0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.493+0000 7f341d5dcb80 1 -- 172.21.15.44:0/3922649586 shutdown_connections 2023-11-16T16:55:10.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.493+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/3922649586 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55abd151b8b0 0x55abd151dcc0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.493+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/3922649586 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 0x55abd150bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.493+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/3922649586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 0x55abd150e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.493+0000 7f33e2ffd700 0 client.26633 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:10.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.493+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/3922649586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1502470 0x55abd150b380 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.493+0000 7f341d5dcb80 1 -- 172.21.15.44:0/3922649586 >> 172.21.15.44:0/3922649586 conn(0x55abd14f7360 msgr2=0x55abd14fbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:55:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:55:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:55:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:55:10.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.494+0000 7f341d5dcb80 1 -- 172.21.15.44:0/3922649586 shutdown_connections 2023-11-16T16:55:10.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.494+0000 7f341d5dcb80 1 -- 172.21.15.44:0/3922649586 wait complete. 2023-11-16T16:55:10.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.494+0000 7f341d5dcb80 20 client.26633 trim_cache size 0 max 16384 2023-11-16T16:55:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.494+0000 7efc41dff700 1 -- 172.21.15.44:0/2263890226 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7efc240a1340 msgr2=0x7efc240a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.494+0000 7efc41dff700 1 --2- 172.21.15.44:0/2263890226 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7efc240a1340 0x7efc240a37f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efc3c10cf90 tx=0x7efc30007c50 comp rx=0 tx=0).stop 2023-11-16T16:55:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.494+0000 7efc41dff700 1 -- 172.21.15.44:0/2263890226 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc3c105560 msgr2=0x7efc3c10be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.494+0000 7efc41dff700 1 --2- 172.21.15.44:0/2263890226 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc3c105560 0x7efc3c10be50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efc3400efc0 tx=0x7efc3400c3a0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.494+0000 7efc41dff700 1 -- 172.21.15.44:0/2263890226 shutdown_connections 2023-11-16T16:55:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.494+0000 7efc41dff700 1 --2- 172.21.15.44:0/2263890226 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7efc240a1340 0x7efc240a37f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.494+0000 7efc41dff700 1 --2- 172.21.15.44:0/2263890226 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc3c105560 0x7efc3c10be50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.494+0000 7efc41dff700 1 --2- 172.21.15.44:0/2263890226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc3c100c50 0x7efc3c1a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.494+0000 7efc41dff700 1 --2- 172.21.15.44:0/2263890226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc3c1015f0 0x7efc3c1a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.494+0000 7efc41dff700 1 -- 172.21.15.44:0/2263890226 >> 172.21.15.44:0/2263890226 conn(0x7efc3c077550 msgr2=0x7efc3c0fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.494+0000 7efc41dff700 1 -- 172.21.15.44:0/2263890226 shutdown_connections 2023-11-16T16:55:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:10.494+0000 7efc41dff700 1 -- 172.21.15.44:0/2263890226 wait complete. 2023-11-16T16:55:10.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.512+0000 7f341d5dcb80 1 Processor -- start 2023-11-16T16:55:10.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.512+0000 7f341d5dcb80 1 -- start start 2023-11-16T16:55:10.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.512+0000 7f341d5dcb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 0x55abd15051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:10.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.512+0000 7f341d5dcb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1502470 0x55abd14fc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:10.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.512+0000 7f341d5dcb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 0x55abd14fcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:10.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.512+0000 7f341d5dcb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abd150c8f0 con 0x55abd1501b60 2023-11-16T16:55:10.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.512+0000 7f341d5dcb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abd14f6ee0 con 0x55abd1502470 2023-11-16T16:55:10.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.512+0000 7f341d5dcb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abd15098e0 con 0x55abd15068b0 2023-11-16T16:55:10.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.512+0000 7f34037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1502470 0x55abd14fc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:10.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.512+0000 7f34037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1502470 0x55abd14fc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:36148/0 (socket says 172.21.15.44:36148) 2023-11-16T16:55:10.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.512+0000 7f34037fe700 1 -- 172.21.15.44:0/1571293337 learned_addr learned my addr 172.21.15.44:0/1571293337 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:10.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.513+0000 7f3402ffd700 1 --2- 172.21.15.44:0/1571293337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 0x55abd15051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:10.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.513+0000 7f3402ffd700 1 --2- 172.21.15.44:0/1571293337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 0x55abd14fcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:10.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.513+0000 7f34037fe700 1 -- 172.21.15.44:0/1571293337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 msgr2=0x55abd14fcaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:10.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.513+0000 7f34037fe700 1 --2- 172.21.15.44:0/1571293337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 0x55abd14fcaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.513+0000 7f34037fe700 1 -- 172.21.15.44:0/1571293337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 msgr2=0x55abd15051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:10.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.513+0000 7f34037fe700 1 --2- 172.21.15.44:0/1571293337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 0x55abd15051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.513+0000 7f34037fe700 1 -- 172.21.15.44:0/1571293337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55abd1418ee0 con 0x55abd1502470 2023-11-16T16:55:10.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.513+0000 7f3402ffd700 1 --2- 172.21.15.44:0/1571293337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 0x55abd14fcaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:55:10.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.513+0000 7f34037fe700 1 --2- 172.21.15.44:0/1571293337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1502470 0x55abd14fc560 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f33f4024c60 tx=0x7f33f4009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aeed90ea21d66243 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:10.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.513+0000 7f33ea7fc700 1 -- 172.21.15.44:0/1571293337 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f401d070 con 0x55abd1502470 2023-11-16T16:55:10.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.513+0000 7f33ea7fc700 1 -- 172.21.15.44:0/1571293337 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33f401e040 con 0x55abd1502470 2023-11-16T16:55:10.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.513+0000 7f33ea7fc700 1 -- 172.21.15.44:0/1571293337 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f40178c0 con 0x55abd1502470 2023-11-16T16:55:10.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.514+0000 7f341d5dcb80 1 -- 172.21.15.44:0/1571293337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1502470 msgr2=0x55abd14fc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:10.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.514+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/1571293337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1502470 0x55abd14fc560 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f33f4024c60 tx=0x7f33f4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.514+0000 7f341d5dcb80 1 -- 172.21.15.44:0/1571293337 shutdown_connections 2023-11-16T16:55:10.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.514+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/1571293337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd15068b0 0x55abd14fcaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.514+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/1571293337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 0x55abd15051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.514+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/1571293337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd1502470 0x55abd14fc560 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.514+0000 7f341d5dcb80 1 -- 172.21.15.44:0/1571293337 >> 172.21.15.44:0/1571293337 conn(0x55abd14f7360 msgr2=0x55abd1506c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:10.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.514+0000 7f341d5dcb80 1 -- 172.21.15.44:0/1571293337 shutdown_connections 2023-11-16T16:55:10.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.514+0000 7f341d5dcb80 1 -- 172.21.15.44:0/1571293337 wait complete. 2023-11-16T16:55:10.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.515+0000 7f341d5dcb80 1 -- start start 2023-11-16T16:55:10.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.515+0000 7f341d5dcb80 1 Processor -- start 2023-11-16T16:55:10.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.515+0000 7f341d5dcb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 0x55abd150eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:10.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.515+0000 7f341d5dcb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd1502470 0x55abd150f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:10.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.515+0000 7f341d5dcb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd15068b0 0x55abd156e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:10.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.515+0000 7f341d5dcb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abd156ec60 con 0x55abd1501b60 2023-11-16T16:55:10.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.515+0000 7f341d5dcb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abd156eda0 con 0x55abd15068b0 2023-11-16T16:55:10.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.515+0000 7f341d5dcb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abd156eee0 con 0x55abd1502470 2023-11-16T16:55:10.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.515+0000 7f341d5dcb80 10 client.0 _ll_register_callbacks cb 0x55abd1412780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:10.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.516+0000 7f3402ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 0x55abd150eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:10.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.516+0000 7f34037fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd1502470 0x55abd150f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:10.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.516+0000 7f3402ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 0x55abd150eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45620/0 (socket says 172.21.15.44:45620) 2023-11-16T16:55:10.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.516+0000 7f3402ffd700 1 -- 172.21.15.44:0/883020805 learned_addr learned my addr 172.21.15.44:0/883020805 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:10.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.516+0000 7f3402ffd700 1 --2- 172.21.15.44:0/883020805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd15068b0 0x55abd156e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:10.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.516+0000 7f34037fe700 1 -- 172.21.15.44:0/883020805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd15068b0 msgr2=0x55abd156e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:10.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.516+0000 7f34037fe700 1 --2- 172.21.15.44:0/883020805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd15068b0 0x55abd156e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.516+0000 7f34037fe700 1 -- 172.21.15.44:0/883020805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 msgr2=0x55abd150eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:10.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.516+0000 7f34037fe700 1 --2- 172.21.15.44:0/883020805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 0x55abd150eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.516+0000 7f34037fe700 1 -- 172.21.15.44:0/883020805 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55abd1419190 con 0x55abd1502470 2023-11-16T16:55:10.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.516+0000 7f34037fe700 1 --2- 172.21.15.44:0/883020805 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd1502470 0x55abd150f0b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x55abd14fd850 tx=0x7f33f4005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7974ca2e5f3a01c5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:10.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.516+0000 7f33e2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:55:10.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.517+0000 7f33e2ffd700 1 -- 172.21.15.44:0/883020805 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f401d040 con 0x55abd1502470 2023-11-16T16:55:10.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.517+0000 7f33e2ffd700 1 -- 172.21.15.44:0/883020805 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f33f4000e40 con 0x55abd1502470 2023-11-16T16:55:10.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.517+0000 7f341d5dcb80 1 -- 172.21.15.44:0/883020805 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33f8003680 con 0x55abd1502470 2023-11-16T16:55:10.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.517+0000 7f33e2ffd700 1 -- 172.21.15.44:0/883020805 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33f4024890 con 0x55abd1502470 2023-11-16T16:55:10.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.517+0000 7f341d5dcb80 10 client.26620 fetch_fsmap learned FSMap version 21 2023-11-16T16:55:10.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.517+0000 7f341d5dcb80 1 -- 172.21.15.44:0/883020805 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55abd1269c60 con 0x55abd1502470 2023-11-16T16:55:10.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.517+0000 7f33e2ffd700 1 -- 172.21.15.44:0/883020805 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33f8003680 con 0x55abd1502470 2023-11-16T16:55:10.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.517+0000 7f33e2ffd700 1 -- 172.21.15.44:0/883020805 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f33f400f4b0 con 0x55abd1502470 2023-11-16T16:55:10.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.517+0000 7f341d5dcb80 10 client.26620 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:55:10.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.517+0000 7f341d5dcb80 10 client.26620 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:10.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.517+0000 7f341d5dcb80 20 client.26620 populate_metadata read hostname 'smithi044' 2023-11-16T16:55:10.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.517+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/883020805 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55abd15700b0 0x55abd1572490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:10.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.517+0000 7f341d5dcb80 4 client.26620 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:55:10.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.517+0000 7f341d5dcb80 1 -- 172.21.15.44:0/883020805 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55abd1421660 con 0x55abd15700b0 2023-11-16T16:55:10.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.518+0000 7f3402ffd700 1 --2- 172.21.15.44:0/883020805 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55abd15700b0 0x55abd1572490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:10.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.518+0000 7f3402ffd700 1 --2- 172.21.15.44:0/883020805 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55abd15700b0 0x55abd1572490 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e593a1fbe89442d2 server_cookie=147ae619cf46282d in_seq=0 out_seq=0 2023-11-16T16:55:10.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.518+0000 7f33e2ffd700 10 client.26620 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:10.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.520+0000 7f33e2ffd700 1 -- 172.21.15.44:0/883020805 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55abd1421660 con 0x55abd15700b0 2023-11-16T16:55:10.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.520+0000 7f33e2ffd700 10 client.26620 handle_command_reply: tid=0 2023-11-16T16:55:10.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.520+0000 7f341d5dcb80 1 client.26620 shutdown 2023-11-16T16:55:10.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.520+0000 7f341d5dcb80 1 -- 172.21.15.44:0/883020805 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd1502470 msgr2=0x55abd150f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:10.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.520+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/883020805 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd1502470 0x55abd150f0b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x55abd14fd850 tx=0x7f33f4005a20 comp rx=0 tx=0).stop 2023-11-16T16:55:10.529 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.520+0000 7f341d5dcb80 1 -- 172.21.15.44:0/883020805 shutdown_connections 2023-11-16T16:55:10.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.520+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/883020805 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55abd15700b0 0x55abd1572490 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.520+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/883020805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abd15068b0 0x55abd156e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.520+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/883020805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55abd1501b60 0x55abd150eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.520+0000 7f341d5dcb80 1 --2- 172.21.15.44:0/883020805 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55abd1502470 0x55abd150f0b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.520+0000 7f341d5dcb80 1 -- 172.21.15.44:0/883020805 >> 172.21.15.44:0/883020805 conn(0x55abd14f7360 msgr2=0x55abd1500960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:10.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.520+0000 7f33e2ffd700 0 client.26620 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:10.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.520+0000 7f341d5dcb80 1 -- 172.21.15.44:0/883020805 shutdown_connections 2023-11-16T16:55:10.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.521+0000 7f341d5dcb80 1 -- 172.21.15.44:0/883020805 wait complete. 2023-11-16T16:55:10.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.521+0000 7f341d5dcb80 20 client.26620 trim_cache size 0 max 16384 2023-11-16T16:55:10.531 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:55:10.531 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "scrub active (7004 inodes in the stack)", 2023-11-16T16:55:10.531 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2023-11-16T16:55:10.531 INFO:teuthology.orchestra.run.smithi044.stdout: "c6d82195-2d53-46f4-9f2b-dc4452b4a1b0": { 2023-11-16T16:55:10.531 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2023-11-16T16:55:10.531 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "c6d82195-2d53-46f4-9f2b-dc4452b4a1b0", 2023-11-16T16:55:10.532 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2023-11-16T16:55:10.532 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:55:10.532 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:55:10.532 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:55:10.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.521+0000 7f3409bff700 1 -- 172.21.15.44:0/3447492184 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f33ec0a12c0 msgr2=0x7f33ec0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:10.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.521+0000 7f3409bff700 1 --2- 172.21.15.44:0/3447492184 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f33ec0a12c0 0x7f33ec0a3770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f33fc00c6f0 tx=0x7f33fc004dc0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.521+0000 7f3409bff700 1 -- 172.21.15.44:0/3447492184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3404074400 msgr2=0x7f340419c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:10.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.521+0000 7f3409bff700 1 --2- 172.21.15.44:0/3447492184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3404074400 0x7f340419c970 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f33fc00efc0 tx=0x7f33fc00c3a0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.521+0000 7f3409bff700 1 -- 172.21.15.44:0/3447492184 shutdown_connections 2023-11-16T16:55:10.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.521+0000 7f3409bff700 1 --2- 172.21.15.44:0/3447492184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3404074400 0x7f340419c970 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.521+0000 7f3409bff700 1 --2- 172.21.15.44:0/3447492184 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f33ec0a12c0 0x7f33ec0a3770 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.521+0000 7f3409bff700 1 --2- 172.21.15.44:0/3447492184 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3404072d20 0x7f34041a2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.521+0000 7f3409bff700 1 --2- 172.21.15.44:0/3447492184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34040736d0 0x7f34041a2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:10.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.522+0000 7f3409bff700 1 -- 172.21.15.44:0/3447492184 >> 172.21.15.44:0/3447492184 conn(0x7f3404077560 msgr2=0x7f3404112890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:10.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.522+0000 7f3409bff700 1 -- 172.21.15.44:0/3447492184 shutdown_connections 2023-11-16T16:55:10.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:10.522+0000 7f3409bff700 1 -- 172.21.15.44:0/3447492184 wait complete. 2023-11-16T16:55:10.542 INFO:tasks.cephfs.filesystem:scrub status for tag:c6d82195-2d53-46f4-9f2b-dc4452b4a1b0 - {'path': '/', 'tag': 'c6d82195-2d53-46f4-9f2b-dc4452b4a1b0', 'options': 'recursive,force'} 2023-11-16T16:55:11.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:11 smithi176 ceph-mon[131630]: pgmap v160: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 1.27k op/s 2023-11-16T16:55:11.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:11 smithi077 ceph-mon[131204]: pgmap v160: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 1.27k op/s 2023-11-16T16:55:11.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:11 smithi044 ceph-mon[128750]: pgmap v160: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 1.27k op/s 2023-11-16T16:55:13.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:13 smithi176 ceph-mon[131630]: pgmap v161: 129 pgs: 129 active+clean; 57 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 1.27k op/s 2023-11-16T16:55:13.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:13 smithi077 ceph-mon[131204]: pgmap v161: 129 pgs: 129 active+clean; 57 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 1.27k op/s 2023-11-16T16:55:13.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:13 smithi044 ceph-mon[128750]: pgmap v161: 129 pgs: 129 active+clean; 57 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 1.27k op/s 2023-11-16T16:55:15.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:15 smithi176 ceph-mon[131630]: pgmap v162: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.1 MiB/s wr, 1.21k op/s 2023-11-16T16:55:15.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:15 smithi077 ceph-mon[131204]: pgmap v162: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.1 MiB/s wr, 1.21k op/s 2023-11-16T16:55:15.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:15 smithi044 ceph-mon[128750]: pgmap v162: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.1 MiB/s wr, 1.21k op/s 2023-11-16T16:55:17.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:17 smithi176 ceph-mon[131630]: pgmap v163: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.2 MiB/s wr, 810 op/s 2023-11-16T16:55:17.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:17 smithi077 ceph-mon[131204]: pgmap v163: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.2 MiB/s wr, 810 op/s 2023-11-16T16:55:17.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:17 smithi044 ceph-mon[128750]: pgmap v163: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.2 MiB/s wr, 810 op/s 2023-11-16T16:55:19.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:19 smithi176 ceph-mon[131630]: pgmap v164: 129 pgs: 129 active+clean; 61 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.9 MiB/s wr, 840 op/s 2023-11-16T16:55:19.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:19 smithi077 ceph-mon[131204]: pgmap v164: 129 pgs: 129 active+clean; 61 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.9 MiB/s wr, 840 op/s 2023-11-16T16:55:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:19 smithi044 ceph-mon[128750]: pgmap v164: 129 pgs: 129 active+clean; 61 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.9 MiB/s wr, 840 op/s 2023-11-16T16:55:21.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:21 smithi176 ceph-mon[131630]: pgmap v165: 129 pgs: 129 active+clean; 61 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.7 MiB/s wr, 39 op/s 2023-11-16T16:55:21.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:21 smithi077 ceph-mon[131204]: pgmap v165: 129 pgs: 129 active+clean; 61 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.7 MiB/s wr, 39 op/s 2023-11-16T16:55:21.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:21 smithi044 ceph-mon[128750]: pgmap v165: 129 pgs: 129 active+clean; 61 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.7 MiB/s wr, 39 op/s 2023-11-16T16:55:22.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:22 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:55:22.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:22 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:55:22.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:22 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:55:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.654+0000 7f5195bff700 1 -- 172.21.15.44:0/2195637482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5190100560 msgr2=0x7f5190100930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.654+0000 7f5195bff700 1 --2- 172.21.15.44:0/2195637482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5190100560 0x7f5190100930 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5180009990 tx=0x7f5180009ca0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.654+0000 7f5195bff700 1 -- 172.21.15.44:0/2195637482 shutdown_connections 2023-11-16T16:55:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.654+0000 7f5195bff700 1 --2- 172.21.15.44:0/2195637482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51901055a0 0x7f519010d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.654+0000 7f5195bff700 1 --2- 172.21.15.44:0/2195637482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5190100560 0x7f5190100930 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.654+0000 7f5195bff700 1 --2- 172.21.15.44:0/2195637482 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5190100f00 0x7f5190104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.654+0000 7f5195bff700 1 -- 172.21.15.44:0/2195637482 >> 172.21.15.44:0/2195637482 conn(0x7f51900781b0 msgr2=0x7f51900785b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.654+0000 7f5195bff700 1 -- 172.21.15.44:0/2195637482 shutdown_connections 2023-11-16T16:55:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.654+0000 7f5195bff700 1 -- 172.21.15.44:0/2195637482 wait complete. 2023-11-16T16:55:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.655+0000 7f5195bff700 1 Processor -- start 2023-11-16T16:55:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.655+0000 7f5195bff700 1 -- start start 2023-11-16T16:55:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.656+0000 7f5195bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5190100560 0x7f5190072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.656+0000 7f5195bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5190100f00 0x7f5190073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.656+0000 7f5195bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51901055a0 0x7f51901a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.656+0000 7f5195bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5190073b60 con 0x7f51901055a0 2023-11-16T16:55:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.656+0000 7f5195bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51901a4b80 con 0x7f5190100560 2023-11-16T16:55:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.656+0000 7f5195bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51901a4cc0 con 0x7f5190100f00 2023-11-16T16:55:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.656+0000 7f518f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5190100560 0x7f5190072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.656+0000 7f518f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5190100560 0x7f5190072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:38978/0 (socket says 172.21.15.44:38978) 2023-11-16T16:55:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.656+0000 7f518f7fe700 1 -- 172.21.15.44:0/3647378416 learned_addr learned my addr 172.21.15.44:0/3647378416 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.656+0000 7f518effd700 1 --2- 172.21.15.44:0/3647378416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5190100f00 0x7f5190073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.656+0000 7f518ffff700 1 --2- 172.21.15.44:0/3647378416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51901055a0 0x7f51901a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.656+0000 7f518f7fe700 1 -- 172.21.15.44:0/3647378416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5190100f00 msgr2=0x7f5190073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.657+0000 7f518f7fe700 1 --2- 172.21.15.44:0/3647378416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5190100f00 0x7f5190073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.657+0000 7f518f7fe700 1 -- 172.21.15.44:0/3647378416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51901055a0 msgr2=0x7f51901a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.657+0000 7f518f7fe700 1 --2- 172.21.15.44:0/3647378416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51901055a0 0x7f51901a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.657+0000 7f518f7fe700 1 -- 172.21.15.44:0/3647378416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5188009c40 con 0x7f5190100560 2023-11-16T16:55:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.657+0000 7f518f7fe700 1 --2- 172.21.15.44:0/3647378416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5190100560 0x7f5190072df0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5180005910 tx=0x7f5180004710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.657+0000 7f518cff9700 1 -- 172.21.15.44:0/3647378416 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518001d070 con 0x7f5190100560 2023-11-16T16:55:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.657+0000 7f518cff9700 1 -- 172.21.15.44:0/3647378416 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5180022470 con 0x7f5190100560 2023-11-16T16:55:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.657+0000 7f5195bff700 1 -- 172.21.15.44:0/3647378416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51800095f0 con 0x7f5190100560 2023-11-16T16:55:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.657+0000 7f518cff9700 1 -- 172.21.15.44:0/3647378416 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5180017650 con 0x7f5190100560 2023-11-16T16:55:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.657+0000 7f5195bff700 1 -- 172.21.15.44:0/3647378416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51901103b0 con 0x7f5190100560 2023-11-16T16:55:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.662+0000 7f518cff9700 1 -- 172.21.15.44:0/3647378416 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f51800225e0 con 0x7f5190100560 2023-11-16T16:55:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.663+0000 7f51a95a2b80 1 Processor -- start 2023-11-16T16:55:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.663+0000 7f51a95a2b80 1 -- start start 2023-11-16T16:55:23.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.663+0000 7f51a95a2b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278fd976b0 0x56278ff411e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.663+0000 7f51a95a2b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278ff41720 0x56278ff3b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.663+0000 7f51a95a2b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41b90 0x56278ff3bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.663+0000 7f51a95a2b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278fe41000 con 0x56278ff41720 2023-11-16T16:55:23.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.663+0000 7f51a95a2b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278fe93840 con 0x56278fd976b0 2023-11-16T16:55:23.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.663+0000 7f51a95a2b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278fe8ccd0 con 0x56278ff41b90 2023-11-16T16:55:23.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.664+0000 7f518ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278fd976b0 0x56278ff411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.664+0000 7f518ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278fd976b0 0x56278ff411e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:38980/0 (socket says 172.21.15.44:38980) 2023-11-16T16:55:23.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.664+0000 7f518ffff700 1 -- 172.21.15.44:0/2685733276 learned_addr learned my addr 172.21.15.44:0/2685733276 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:23.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.664+0000 7f518f7fe700 1 --2- 172.21.15.44:0/2685733276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278ff41720 0x56278ff3b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.664+0000 7f518effd700 1 --2- 172.21.15.44:0/2685733276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41b90 0x56278ff3bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.664+0000 7f518cff9700 1 --2- 172.21.15.44:0/3647378416 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f51780a1340 0x7f51780a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.664+0000 7f518ffff700 1 -- 172.21.15.44:0/2685733276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41b90 msgr2=0x56278ff3bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:23.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.664+0000 7f518ffff700 1 --2- 172.21.15.44:0/2685733276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41b90 0x56278ff3bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.664+0000 7f518ffff700 1 -- 172.21.15.44:0/2685733276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278ff41720 msgr2=0x56278ff3b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:23.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.664+0000 7f518ffff700 1 --2- 172.21.15.44:0/2685733276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278ff41720 0x56278ff3b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.664+0000 7f518ffff700 1 -- 172.21.15.44:0/2685733276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56278fd0e5d0 con 0x56278fd976b0 2023-11-16T16:55:23.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.664+0000 7f518f7fe700 1 --2- 172.21.15.44:0/2685733276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278ff41720 0x56278ff3b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:55:23.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.664+0000 7f518cff9700 1 -- 172.21.15.44:0/3647378416 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f51800ce210 con 0x7f5190100560 2023-11-16T16:55:23.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.664+0000 7f518ffff700 1 --2- 172.21.15.44:0/3647378416 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f51780a1340 0x7f51780a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.665+0000 7f518ffff700 1 --2- 172.21.15.44:0/2685733276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278fd976b0 0x56278ff411e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f518800e700 tx=0x7f5188009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6372060f98115d3b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:23.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.665+0000 7f5175ffb700 1 -- 172.21.15.44:0/2685733276 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518800e040 con 0x56278fd976b0 2023-11-16T16:55:23.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.665+0000 7f5175ffb700 1 -- 172.21.15.44:0/2685733276 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f518800f040 con 0x56278fd976b0 2023-11-16T16:55:23.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.665+0000 7f5175ffb700 1 -- 172.21.15.44:0/2685733276 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188015450 con 0x56278fd976b0 2023-11-16T16:55:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.666+0000 7f51a95a2b80 1 -- 172.21.15.44:0/2685733276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278fd976b0 msgr2=0x56278ff411e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.666+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/2685733276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278fd976b0 0x56278ff411e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f518800e700 tx=0x7f5188009920 comp rx=0 tx=0).stop 2023-11-16T16:55:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.666+0000 7f518ffff700 1 --2- 172.21.15.44:0/3647378416 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f51780a1340 0x7f51780a37f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5188020000 tx=0x7f5188023000 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.666+0000 7f51a95a2b80 1 -- 172.21.15.44:0/2685733276 shutdown_connections 2023-11-16T16:55:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.666+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/2685733276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41b90 0x56278ff3bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.666+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/2685733276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278fd976b0 0x56278ff411e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.666+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/2685733276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278ff41720 0x56278ff3b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.666+0000 7f51a95a2b80 1 -- 172.21.15.44:0/2685733276 >> 172.21.15.44:0/2685733276 conn(0x56278ff30f50 msgr2=0x56278ff36270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.666+0000 7f51a95a2b80 1 -- 172.21.15.44:0/2685733276 shutdown_connections 2023-11-16T16:55:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.666+0000 7f51a95a2b80 1 -- 172.21.15.44:0/2685733276 wait complete. 2023-11-16T16:55:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.667+0000 7f51a95a2b80 1 -- start start 2023-11-16T16:55:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.667+0000 7f51a95a2b80 1 Processor -- start 2023-11-16T16:55:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.667+0000 7f51a95a2b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278fd976b0 0x56278ff47a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.667+0000 7f51a95a2b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41720 0x56278ff44d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.667+0000 7f51a95a2b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41b90 0x56278ff453a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.667+0000 7f51a95a2b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278fe2e880 con 0x56278fd976b0 2023-11-16T16:55:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.667+0000 7f51a95a2b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278fe2e520 con 0x56278ff41720 2023-11-16T16:55:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.667+0000 7f51a95a2b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278fd108c0 con 0x56278ff41b90 2023-11-16T16:55:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.667+0000 7f51a95a2b80 10 client.0 _ll_register_callbacks cb 0x56278fe4c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.668+0000 7f518f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278fd976b0 0x56278ff47a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.668+0000 7f518effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41720 0x56278ff44d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.668+0000 7f518f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278fd976b0 0x56278ff47a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51458/0 (socket says 172.21.15.44:51458) 2023-11-16T16:55:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.668+0000 7f518f7fe700 1 -- 172.21.15.44:0/2164281844 learned_addr learned my addr 172.21.15.44:0/2164281844 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.668+0000 7f518ffff700 1 --2- 172.21.15.44:0/2164281844 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41b90 0x56278ff453a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.668+0000 7f518effd700 1 -- 172.21.15.44:0/2164281844 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41b90 msgr2=0x56278ff453a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.668+0000 7f518effd700 1 --2- 172.21.15.44:0/2164281844 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41b90 0x56278ff453a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.668+0000 7f518effd700 1 -- 172.21.15.44:0/2164281844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278fd976b0 msgr2=0x56278ff47a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.668+0000 7f518effd700 1 --2- 172.21.15.44:0/2164281844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278fd976b0 0x56278ff47a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.668+0000 7f518effd700 1 -- 172.21.15.44:0/2164281844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f518800e3b0 con 0x56278ff41720 2023-11-16T16:55:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.668+0000 7f518ffff700 1 --2- 172.21.15.44:0/2164281844 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41b90 0x56278ff453a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:55:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.668+0000 7f518f7fe700 1 --2- 172.21.15.44:0/2164281844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278fd976b0 0x56278ff47a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:55:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.668+0000 7f518effd700 1 --2- 172.21.15.44:0/2164281844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41720 0x56278ff44d40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f51900745e0 tx=0x7f5184011a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2a4fd50f1541eca server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.668+0000 7f516effd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:55:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.669+0000 7f516effd700 1 -- 172.21.15.44:0/2164281844 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5184016070 con 0x56278ff41720 2023-11-16T16:55:23.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.669+0000 7f51a95a2b80 1 -- 172.21.15.44:0/2164281844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56278ff2e710 con 0x56278ff41720 2023-11-16T16:55:23.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.669+0000 7f516effd700 1 -- 172.21.15.44:0/2164281844 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f5184004320 con 0x56278ff41720 2023-11-16T16:55:23.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.669+0000 7f516effd700 1 -- 172.21.15.44:0/2164281844 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51840044e0 con 0x56278ff41720 2023-11-16T16:55:23.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.669+0000 7f51a95a2b80 10 client.26651 fetch_fsmap learned FSMap version 21 2023-11-16T16:55:23.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.669+0000 7f51a95a2b80 1 -- 172.21.15.44:0/2164281844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x56278fe53130 con 0x56278ff41720 2023-11-16T16:55:23.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.669+0000 7f516effd700 1 -- 172.21.15.44:0/2164281844 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56278ff2e710 con 0x56278ff41720 2023-11-16T16:55:23.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.669+0000 7f516effd700 1 -- 172.21.15.44:0/2164281844 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f518400f790 con 0x56278ff41720 2023-11-16T16:55:23.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.669+0000 7f51a95a2b80 10 client.26651 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:55:23.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.669+0000 7f51a95a2b80 10 client.26651 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:23.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.669+0000 7f51a95a2b80 20 client.26651 populate_metadata read hostname 'smithi044' 2023-11-16T16:55:23.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.669+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/2164281844 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56278ff54f10 0x56278ff57320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.669+0000 7f51a95a2b80 4 client.26651 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:55:23.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.669+0000 7f51a95a2b80 1 -- 172.21.15.44:0/2164281844 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56278fe4f040 con 0x56278ff54f10 2023-11-16T16:55:23.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.670+0000 7f518f7fe700 1 --2- 172.21.15.44:0/2164281844 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56278ff54f10 0x56278ff57320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.670+0000 7f516effd700 10 client.26651 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:23.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.670+0000 7f518f7fe700 1 --2- 172.21.15.44:0/2164281844 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56278ff54f10 0x56278ff57320 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=262789f34305e3ad server_cookie=ab31ea620760a31f in_seq=0 out_seq=0 2023-11-16T16:55:23.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.671+0000 7f516effd700 1 -- 172.21.15.44:0/2164281844 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56278fe4f040 con 0x56278ff54f10 2023-11-16T16:55:23.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.671+0000 7f516effd700 10 client.26651 handle_command_reply: tid=0 2023-11-16T16:55:23.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.671+0000 7f51a95a2b80 1 client.26651 shutdown 2023-11-16T16:55:23.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.672+0000 7f51a95a2b80 1 -- 172.21.15.44:0/2164281844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41720 msgr2=0x56278ff44d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:23.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.672+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/2164281844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41720 0x56278ff44d40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f51900745e0 tx=0x7f5184011a50 comp rx=0 tx=0).stop 2023-11-16T16:55:23.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.672+0000 7f51a95a2b80 1 -- 172.21.15.44:0/2164281844 shutdown_connections 2023-11-16T16:55:23.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.672+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/2164281844 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56278ff54f10 0x56278ff57320 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.672+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/2164281844 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41b90 0x56278ff453a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.672+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/2164281844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278fd976b0 0x56278ff47a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.672+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/2164281844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41720 0x56278ff44d40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.672+0000 7f516effd700 0 client.26651 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:23.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.672+0000 7f51a95a2b80 1 -- 172.21.15.44:0/2164281844 >> 172.21.15.44:0/2164281844 conn(0x56278ff30f50 msgr2=0x56278ff357e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:23.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.672+0000 7f51a95a2b80 1 -- 172.21.15.44:0/2164281844 shutdown_connections 2023-11-16T16:55:23.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.672+0000 7f51a95a2b80 1 -- 172.21.15.44:0/2164281844 wait complete. 2023-11-16T16:55:23.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.672+0000 7f51a95a2b80 20 client.26651 trim_cache size 0 max 16384 2023-11-16T16:55:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.690+0000 7f51a95a2b80 1 Processor -- start 2023-11-16T16:55:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.690+0000 7f51a95a2b80 1 -- start start 2023-11-16T16:55:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.690+0000 7f51a95a2b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278fd976b0 0x56278ff4c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.690+0000 7f51a95a2b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41720 0x56278ff40510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.690+0000 7f51a95a2b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41b90 0x56278ff475f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.690+0000 7f51a95a2b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278ff52690 con 0x56278fd976b0 2023-11-16T16:55:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.690+0000 7f51a95a2b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278ff38010 con 0x56278ff41b90 2023-11-16T16:55:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.690+0000 7f51a95a2b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278ff432c0 con 0x56278ff41720 2023-11-16T16:55:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.690+0000 7f518f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278fd976b0 0x56278ff4c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.690+0000 7f518ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41b90 0x56278ff475f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.690+0000 7f518ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41b90 0x56278ff475f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:38994/0 (socket says 172.21.15.44:38994) 2023-11-16T16:55:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.690+0000 7f518f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278fd976b0 0x56278ff4c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51468/0 (socket says 172.21.15.44:51468) 2023-11-16T16:55:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.690+0000 7f518ffff700 1 -- 172.21.15.44:0/3270292698 learned_addr learned my addr 172.21.15.44:0/3270292698 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.690+0000 7f518effd700 1 --2- 172.21.15.44:0/3270292698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41720 0x56278ff40510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.691+0000 7f518ffff700 1 -- 172.21.15.44:0/3270292698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41720 msgr2=0x56278ff40510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.691+0000 7f518ffff700 1 --2- 172.21.15.44:0/3270292698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41720 0x56278ff40510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.691+0000 7f518ffff700 1 -- 172.21.15.44:0/3270292698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278fd976b0 msgr2=0x56278ff4c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.691+0000 7f518ffff700 1 --2- 172.21.15.44:0/3270292698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278fd976b0 0x56278ff4c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.691+0000 7f518ffff700 1 -- 172.21.15.44:0/3270292698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56278fe52e80 con 0x56278ff41b90 2023-11-16T16:55:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.691+0000 7f518effd700 1 --2- 172.21.15.44:0/3270292698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41720 0x56278ff40510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:55:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.691+0000 7f518f7fe700 1 --2- 172.21.15.44:0/3270292698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278fd976b0 0x56278ff4c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:55:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.691+0000 7f518ffff700 1 --2- 172.21.15.44:0/3270292698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41b90 0x56278ff475f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f518800e6d0 tx=0x7f5188025000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86a78c6ca94c7287 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.691+0000 7f51767fc700 1 -- 172.21.15.44:0/3270292698 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518800e040 con 0x56278ff41b90 2023-11-16T16:55:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.691+0000 7f51767fc700 1 -- 172.21.15.44:0/3270292698 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5188003c10 con 0x56278ff41b90 2023-11-16T16:55:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.691+0000 7f51767fc700 1 -- 172.21.15.44:0/3270292698 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188004e60 con 0x56278ff41b90 2023-11-16T16:55:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.692+0000 7f51a95a2b80 1 -- 172.21.15.44:0/3270292698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41b90 msgr2=0x56278ff475f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.692+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/3270292698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41b90 0x56278ff475f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f518800e6d0 tx=0x7f5188025000 comp rx=0 tx=0).stop 2023-11-16T16:55:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.692+0000 7f51a95a2b80 1 -- 172.21.15.44:0/3270292698 shutdown_connections 2023-11-16T16:55:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.692+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/3270292698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41b90 0x56278ff475f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.692+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/3270292698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278fd976b0 0x56278ff4c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.692+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/3270292698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278ff41720 0x56278ff40510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.692+0000 7f51a95a2b80 1 -- 172.21.15.44:0/3270292698 >> 172.21.15.44:0/3270292698 conn(0x56278ff30f50 msgr2=0x56278ff3fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.692+0000 7f51a95a2b80 1 -- 172.21.15.44:0/3270292698 shutdown_connections 2023-11-16T16:55:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.692+0000 7f51a95a2b80 1 -- 172.21.15.44:0/3270292698 wait complete. 2023-11-16T16:55:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.693+0000 7f51a95a2b80 1 -- start start 2023-11-16T16:55:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.693+0000 7f51a95a2b80 1 Processor -- start 2023-11-16T16:55:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.693+0000 7f51a95a2b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278fd976b0 0x56278ff3cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.693+0000 7f51a95a2b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41720 0x56278ff3d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.693+0000 7f51a95a2b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278ff41b90 0x56278ff3d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.693+0000 7f51a95a2b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278ff3dc20 con 0x56278ff41b90 2023-11-16T16:55:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.693+0000 7f51a95a2b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278ff3dd90 con 0x56278ff41720 2023-11-16T16:55:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.693+0000 7f51a95a2b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278ff3df50 con 0x56278fd976b0 2023-11-16T16:55:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.693+0000 7f51a95a2b80 10 client.0 _ll_register_callbacks cb 0x7f5184009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.694+0000 7f518effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41720 0x56278ff3d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.694+0000 7f518effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41720 0x56278ff3d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:38998/0 (socket says 172.21.15.44:38998) 2023-11-16T16:55:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.694+0000 7f518effd700 1 -- 172.21.15.44:0/500672619 learned_addr learned my addr 172.21.15.44:0/500672619 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.694+0000 7f518ffff700 1 --2- 172.21.15.44:0/500672619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278ff41b90 0x56278ff3d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.694+0000 7f518f7fe700 1 --2- 172.21.15.44:0/500672619 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278fd976b0 0x56278ff3cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.694+0000 7f518effd700 1 -- 172.21.15.44:0/500672619 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278fd976b0 msgr2=0x56278ff3cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.694+0000 7f518effd700 1 --2- 172.21.15.44:0/500672619 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278fd976b0 0x56278ff3cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.694+0000 7f518effd700 1 -- 172.21.15.44:0/500672619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278ff41b90 msgr2=0x56278ff3d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.694+0000 7f518effd700 1 --2- 172.21.15.44:0/500672619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278ff41b90 0x56278ff3d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.694+0000 7f518effd700 1 -- 172.21.15.44:0/500672619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51901103b0 con 0x56278ff41720 2023-11-16T16:55:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.694+0000 7f518ffff700 1 --2- 172.21.15.44:0/500672619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278ff41b90 0x56278ff3d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:55:23.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.694+0000 7f518effd700 1 --2- 172.21.15.44:0/500672619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41720 0x56278ff3d1a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x56278ff46280 tx=0x7f51840056f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4622c14c0f1f13b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:23.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.694+0000 7f516effd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:55:23.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.695+0000 7f516effd700 1 -- 172.21.15.44:0/500672619 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5184016040 con 0x56278ff41720 2023-11-16T16:55:23.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.695+0000 7f516effd700 1 -- 172.21.15.44:0/500672619 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f5184005ce0 con 0x56278ff41720 2023-11-16T16:55:23.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.695+0000 7f51a95a2b80 1 -- 172.21.15.44:0/500672619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f518800f040 con 0x56278ff41720 2023-11-16T16:55:23.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.695+0000 7f516effd700 1 -- 172.21.15.44:0/500672619 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f518400fd70 con 0x56278ff41720 2023-11-16T16:55:23.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.695+0000 7f51a95a2b80 10 client.26657 fetch_fsmap learned FSMap version 21 2023-11-16T16:55:23.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.695+0000 7f51a95a2b80 1 -- 172.21.15.44:0/500672619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x56278fe53130 con 0x56278ff41720 2023-11-16T16:55:23.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.695+0000 7f516effd700 1 -- 172.21.15.44:0/500672619 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f518800f040 con 0x56278ff41720 2023-11-16T16:55:23.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.696+0000 7f516effd700 1 -- 172.21.15.44:0/500672619 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5184031060 con 0x56278ff41720 2023-11-16T16:55:23.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.696+0000 7f51a95a2b80 10 client.26657 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:55:23.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.696+0000 7f51a95a2b80 10 client.26657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:23.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.696+0000 7f51a95a2b80 20 client.26657 populate_metadata read hostname 'smithi044' 2023-11-16T16:55:23.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.696+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/500672619 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56278ffa9850 0x56278ffabc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:23.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.696+0000 7f51a95a2b80 4 client.26657 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:55:23.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.696+0000 7f51a95a2b80 1 -- 172.21.15.44:0/500672619 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56278fe5b600 con 0x56278ffa9850 2023-11-16T16:55:23.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.696+0000 7f518f7fe700 1 --2- 172.21.15.44:0/500672619 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56278ffa9850 0x56278ffabc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:23.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.696+0000 7f518f7fe700 1 --2- 172.21.15.44:0/500672619 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56278ffa9850 0x56278ffabc60 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d9b5e3f10241cf server_cookie=b71150c8475abc6e in_seq=0 out_seq=0 2023-11-16T16:55:23.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:23.696+0000 7f516effd700 10 client.26657 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:23.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:23 smithi077 ceph-mon[131204]: pgmap v166: 129 pgs: 129 active+clean; 65 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.0 MiB/s wr, 76 op/s 2023-11-16T16:55:23.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:23 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:55:23.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:23 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:55:23.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:23 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:55:23.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:23 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:55:23.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:23 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T16:55:23.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:23 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:55:23.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:23 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:55:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:23 smithi044 ceph-mon[128750]: pgmap v166: 129 pgs: 129 active+clean; 65 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.0 MiB/s wr, 76 op/s 2023-11-16T16:55:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:23 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:55:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:23 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:55:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:23 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:55:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:23 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:55:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:23 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T16:55:23.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:23 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:55:23.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:23 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:55:23.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:23 smithi176 ceph-mon[131630]: pgmap v166: 129 pgs: 129 active+clean; 65 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.0 MiB/s wr, 76 op/s 2023-11-16T16:55:23.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:23 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:55:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:23 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:55:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:23 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:55:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:23 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:55:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:23 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T16:55:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:23 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:55:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:23 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:55:25.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:25 smithi077 ceph-mon[131204]: pgmap v167: 129 pgs: 129 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.7 MiB/s wr, 251 op/s 2023-11-16T16:55:25.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:25 smithi044 ceph-mon[128750]: pgmap v167: 129 pgs: 129 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.7 MiB/s wr, 251 op/s 2023-11-16T16:55:25.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:25 smithi176 ceph-mon[131630]: pgmap v167: 129 pgs: 129 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.7 MiB/s wr, 251 op/s 2023-11-16T16:55:27.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:27 smithi077 ceph-mon[131204]: pgmap v168: 129 pgs: 129 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.3 MiB/s wr, 243 op/s 2023-11-16T16:55:27.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:27 smithi044 ceph-mon[128750]: pgmap v168: 129 pgs: 129 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.3 MiB/s wr, 243 op/s 2023-11-16T16:55:27.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:27 smithi176 ceph-mon[131630]: pgmap v168: 129 pgs: 129 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.3 MiB/s wr, 243 op/s 2023-11-16T16:55:29.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:29 smithi077 ceph-mon[131204]: pgmap v169: 129 pgs: 129 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.2 MiB/s wr, 1.22k op/s 2023-11-16T16:55:29.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:29 smithi044 ceph-mon[128750]: pgmap v169: 129 pgs: 129 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.2 MiB/s wr, 1.22k op/s 2023-11-16T16:55:29.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:29 smithi176 ceph-mon[131630]: pgmap v169: 129 pgs: 129 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.2 MiB/s wr, 1.22k op/s 2023-11-16T16:55:31.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:31 smithi077 ceph-mon[131204]: pgmap v170: 129 pgs: 129 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 1.19k op/s 2023-11-16T16:55:31.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:31 smithi044 ceph-mon[128750]: pgmap v170: 129 pgs: 129 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 1.19k op/s 2023-11-16T16:55:31.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:31 smithi176 ceph-mon[131630]: pgmap v170: 129 pgs: 129 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 1.19k op/s 2023-11-16T16:55:33.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:33 smithi077 ceph-mon[131204]: pgmap v171: 129 pgs: 129 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 1.20k op/s 2023-11-16T16:55:33.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:33 smithi044 ceph-mon[128750]: pgmap v171: 129 pgs: 129 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 1.20k op/s 2023-11-16T16:55:33.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:33 smithi176 ceph-mon[131630]: pgmap v171: 129 pgs: 129 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.5 MiB/s wr, 1.20k op/s 2023-11-16T16:55:34.247 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.245+0000 7f516effd700 1 -- 172.21.15.44:0/500672619 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56278fe5b600 con 0x56278ffa9850 2023-11-16T16:55:34.247 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.245+0000 7f516effd700 10 client.26657 handle_command_reply: tid=0 2023-11-16T16:55:34.247 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.245+0000 7f51a95a2b80 1 client.26657 shutdown 2023-11-16T16:55:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.246+0000 7f51a95a2b80 1 -- 172.21.15.44:0/500672619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41720 msgr2=0x56278ff3d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.246+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/500672619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41720 0x56278ff3d1a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x56278ff46280 tx=0x7f51840056f0 comp rx=0 tx=0).stop 2023-11-16T16:55:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.249+0000 7f51a95a2b80 1 -- 172.21.15.44:0/500672619 shutdown_connections 2023-11-16T16:55:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.249+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/500672619 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56278ffa9850 0x56278ffabc60 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.249+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/500672619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56278ff41b90 0x56278ff3d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.249+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/500672619 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56278fd976b0 0x56278ff3cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.249+0000 7f51a95a2b80 1 --2- 172.21.15.44:0/500672619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56278ff41720 0x56278ff3d1a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.249+0000 7f51a95a2b80 1 -- 172.21.15.44:0/500672619 >> 172.21.15.44:0/500672619 conn(0x56278ff30f50 msgr2=0x56278ff40690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:34.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.249+0000 7f516effd700 0 client.26657 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:34.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.250+0000 7f51a95a2b80 1 -- 172.21.15.44:0/500672619 shutdown_connections 2023-11-16T16:55:34.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.250+0000 7f51a95a2b80 1 -- 172.21.15.44:0/500672619 wait complete. 2023-11-16T16:55:34.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.251+0000 7f51a95a2b80 20 client.26657 trim_cache size 0 max 16384 2023-11-16T16:55:34.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:55:34.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:55:34.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:55:34.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:55:34.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.253+0000 7f5195bff700 1 -- 172.21.15.44:0/3647378416 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f51780a1340 msgr2=0x7f51780a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:34.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.253+0000 7f5195bff700 1 --2- 172.21.15.44:0/3647378416 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f51780a1340 0x7f51780a37f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5188020000 tx=0x7f5188023000 comp rx=0 tx=0).stop 2023-11-16T16:55:34.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.253+0000 7f5195bff700 1 -- 172.21.15.44:0/3647378416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5190100560 msgr2=0x7f5190072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:34.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.253+0000 7f5195bff700 1 --2- 172.21.15.44:0/3647378416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5190100560 0x7f5190072df0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5180005910 tx=0x7f5180004710 comp rx=0 tx=0).stop 2023-11-16T16:55:34.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.253+0000 7f5195bff700 1 -- 172.21.15.44:0/3647378416 shutdown_connections 2023-11-16T16:55:34.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.253+0000 7f5195bff700 1 --2- 172.21.15.44:0/3647378416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51901055a0 0x7f51901a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:34.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.253+0000 7f5195bff700 1 --2- 172.21.15.44:0/3647378416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5190100560 0x7f5190072df0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:34.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.253+0000 7f5195bff700 1 --2- 172.21.15.44:0/3647378416 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f51780a1340 0x7f51780a37f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:34.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.253+0000 7f5195bff700 1 --2- 172.21.15.44:0/3647378416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5190100f00 0x7f5190073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:34.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.253+0000 7f5195bff700 1 -- 172.21.15.44:0/3647378416 >> 172.21.15.44:0/3647378416 conn(0x7f51900781b0 msgr2=0x7f5190103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:34.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.254+0000 7f5195bff700 1 -- 172.21.15.44:0/3647378416 shutdown_connections 2023-11-16T16:55:34.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:34.254+0000 7f5195bff700 1 -- 172.21.15.44:0/3647378416 wait complete. 2023-11-16T16:55:35.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:35 smithi077 ceph-mon[131204]: pgmap v172: 129 pgs: 129 active+clean; 102 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.8 MiB/s wr, 1.19k op/s 2023-11-16T16:55:35.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:35 smithi044 ceph-mon[128750]: pgmap v172: 129 pgs: 129 active+clean; 102 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.8 MiB/s wr, 1.19k op/s 2023-11-16T16:55:35.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:35 smithi176 ceph-mon[131630]: pgmap v172: 129 pgs: 129 active+clean; 102 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.8 MiB/s wr, 1.19k op/s 2023-11-16T16:55:37.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:37 smithi077 ceph-mon[131204]: pgmap v173: 129 pgs: 129 active+clean; 102 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 1.01k op/s 2023-11-16T16:55:37.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:37 smithi044 ceph-mon[128750]: pgmap v173: 129 pgs: 129 active+clean; 102 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 1.01k op/s 2023-11-16T16:55:37.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:37 smithi176 ceph-mon[131630]: pgmap v173: 129 pgs: 129 active+clean; 102 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 1.01k op/s 2023-11-16T16:55:39.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:39 smithi077 ceph-mon[131204]: pgmap v174: 129 pgs: 129 active+clean; 29 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.1 MiB/s wr, 1.57k op/s 2023-11-16T16:55:39.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:39 smithi044 ceph-mon[128750]: pgmap v174: 129 pgs: 129 active+clean; 29 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.1 MiB/s wr, 1.57k op/s 2023-11-16T16:55:39.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:39 smithi176 ceph-mon[131630]: pgmap v174: 129 pgs: 129 active+clean; 29 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.1 MiB/s wr, 1.57k op/s 2023-11-16T16:55:40.544 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:55:40.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.797+0000 7fe72e7ff700 1 -- 172.21.15.44:0/4251170476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7280736d0 msgr2=0x7fe7281136a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:40.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.797+0000 7fe72e7ff700 1 --2- 172.21.15.44:0/4251170476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7280736d0 0x7fe7281136a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe72000b0a0 tx=0x7fe72000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.797+0000 7fe72e7ff700 1 -- 172.21.15.44:0/4251170476 shutdown_connections 2023-11-16T16:55:40.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.797+0000 7fe72e7ff700 1 --2- 172.21.15.44:0/4251170476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7280736d0 0x7fe7281136a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.797+0000 7fe72e7ff700 1 --2- 172.21.15.44:0/4251170476 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe728074400 0x7fe7280747d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.797+0000 7fe72e7ff700 1 --2- 172.21.15.44:0/4251170476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe728072d20 0x7fe728073190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.797+0000 7fe72e7ff700 1 -- 172.21.15.44:0/4251170476 >> 172.21.15.44:0/4251170476 conn(0x7fe728077590 msgr2=0x7fe728077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:40.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.798+0000 7fe72e7ff700 1 -- 172.21.15.44:0/4251170476 shutdown_connections 2023-11-16T16:55:40.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.798+0000 7fe72e7ff700 1 -- 172.21.15.44:0/4251170476 wait complete. 2023-11-16T16:55:40.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.799+0000 7fe72e7ff700 1 Processor -- start 2023-11-16T16:55:40.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.799+0000 7fe72e7ff700 1 -- start start 2023-11-16T16:55:40.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.800+0000 7fe72e7ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe728072d20 0x7fe7281a2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.800+0000 7fe72e7ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7280736d0 0x7fe7281a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.800+0000 7fe72e7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe728074400 0x7fe72819c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.800+0000 7fe72e7ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72819d030 con 0x7fe728072d20 2023-11-16T16:55:40.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.800+0000 7fe72e7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72819d1a0 con 0x7fe728074400 2023-11-16T16:55:40.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.800+0000 7fe72e7ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72819d360 con 0x7fe7280736d0 2023-11-16T16:55:40.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.800+0000 7fe727fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe728072d20 0x7fe7281a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.800+0000 7fe72cd9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe728074400 0x7fe72819c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.800+0000 7fe727fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe728072d20 0x7fe7281a2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42700/0 (socket says 172.21.15.44:42700) 2023-11-16T16:55:40.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.800+0000 7fe727fff700 1 -- 172.21.15.44:0/288423419 learned_addr learned my addr 172.21.15.44:0/288423419 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:40.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.800+0000 7fe7277fe700 1 --2- 172.21.15.44:0/288423419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7280736d0 0x7fe7281a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.801+0000 7fe72cd9c700 1 -- 172.21.15.44:0/288423419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7280736d0 msgr2=0x7fe7281a2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:40.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.801+0000 7fe72cd9c700 1 --2- 172.21.15.44:0/288423419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7280736d0 0x7fe7281a2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.801+0000 7fe72cd9c700 1 -- 172.21.15.44:0/288423419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe728072d20 msgr2=0x7fe7281a2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:40.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.801+0000 7fe72cd9c700 1 --2- 172.21.15.44:0/288423419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe728072d20 0x7fe7281a2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.801+0000 7fe72cd9c700 1 -- 172.21.15.44:0/288423419 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe720009d00 con 0x7fe728074400 2023-11-16T16:55:40.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.801+0000 7fe72cd9c700 1 --2- 172.21.15.44:0/288423419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe728074400 0x7fe72819c9c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe720009fd0 tx=0x7fe720007930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:40.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.801+0000 7fe7257fa700 1 -- 172.21.15.44:0/288423419 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72000e040 con 0x7fe728074400 2023-11-16T16:55:40.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.801+0000 7fe7257fa700 1 -- 172.21.15.44:0/288423419 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe72001d470 con 0x7fe728074400 2023-11-16T16:55:40.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.801+0000 7fe7257fa700 1 -- 172.21.15.44:0/288423419 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7200052a0 con 0x7fe728074400 2023-11-16T16:55:40.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.801+0000 7fe72e7ff700 1 -- 172.21.15.44:0/288423419 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe728116a00 con 0x7fe728074400 2023-11-16T16:55:40.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.801+0000 7fe72e7ff700 1 -- 172.21.15.44:0/288423419 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe728116870 con 0x7fe728074400 2023-11-16T16:55:40.806 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.805+0000 7fe7257fa700 1 -- 172.21.15.44:0/288423419 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fe72001d5e0 con 0x7fe728074400 2023-11-16T16:55:40.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.806+0000 7fe7420e9b80 1 Processor -- start 2023-11-16T16:55:40.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.807+0000 7fe7257fa700 1 --2- 172.21.15.44:0/288423419 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe7100a1340 0x7fe7100a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.807+0000 7fe7420e9b80 1 -- start start 2023-11-16T16:55:40.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.807+0000 7fe7257fa700 1 -- 172.21.15.44:0/288423419 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fe7200ce490 con 0x7fe728074400 2023-11-16T16:55:40.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.807+0000 7fe7420e9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8ab60 0x561fe6d8af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.807+0000 7fe7277fe700 1 --2- 172.21.15.44:0/288423419 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe7100a1340 0x7fe7100a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.807+0000 7fe7420e9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 0x561fe6d8f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.807+0000 7fe7420e9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8f8b0 0x561fe6d8fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.807+0000 7fe7420e9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fe6c90060 con 0x561fe6d8f8b0 2023-11-16T16:55:40.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.807+0000 7fe7420e9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fe6ce28a0 con 0x561fe6d8ab60 2023-11-16T16:55:40.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.807+0000 7fe7420e9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fe6cdbd30 con 0x561fe6d8b470 2023-11-16T16:55:40.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.807+0000 7fe7277fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 0x561fe6d8f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.807+0000 7fe727fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8ab60 0x561fe6d8af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.807+0000 7fe7277fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 0x561fe6d8f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:51722/0 (socket says 172.21.15.44:51722) 2023-11-16T16:55:40.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.807+0000 7fe7277fe700 1 -- 172.21.15.44:0/1189155295 learned_addr learned my addr 172.21.15.44:0/1189155295 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:40.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.808+0000 7fe72cd9c700 1 --2- 172.21.15.44:0/1189155295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8f8b0 0x561fe6d8fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.808+0000 7fe7277fe700 1 -- 172.21.15.44:0/1189155295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8ab60 msgr2=0x561fe6d8af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:40.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.808+0000 7fe7277fe700 1 --2- 172.21.15.44:0/1189155295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8ab60 0x561fe6d8af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.808+0000 7fe7277fe700 1 -- 172.21.15.44:0/1189155295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8f8b0 msgr2=0x561fe6d8fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:40.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.808+0000 7fe7277fe700 1 --2- 172.21.15.44:0/1189155295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8f8b0 0x561fe6d8fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.808+0000 7fe7277fe700 1 -- 172.21.15.44:0/1189155295 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561fe6b5d630 con 0x561fe6d8b470 2023-11-16T16:55:40.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.808+0000 7fe7277fe700 1 --2- 172.21.15.44:0/1189155295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 0x561fe6d8f370 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe71c00f6a0 tx=0x7fe71c01d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=636f8ce55359976f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:40.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.809+0000 7fe7277fe700 1 --2- 172.21.15.44:0/288423419 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe7100a1340 0x7fe7100a37f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe71c013a70 tx=0x7fe71c009c00 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:40.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.809+0000 7fe70e7fc700 1 -- 172.21.15.44:0/1189155295 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c013170 con 0x561fe6d8b470 2023-11-16T16:55:40.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.809+0000 7fe70e7fc700 1 -- 172.21.15.44:0/1189155295 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe71c0099c0 con 0x561fe6d8b470 2023-11-16T16:55:40.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.809+0000 7fe70e7fc700 1 -- 172.21.15.44:0/1189155295 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c005480 con 0x561fe6d8b470 2023-11-16T16:55:40.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.810+0000 7fe7420e9b80 1 -- 172.21.15.44:0/1189155295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 msgr2=0x561fe6d8f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:40.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.810+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/1189155295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 0x561fe6d8f370 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe71c00f6a0 tx=0x7fe71c01d000 comp rx=0 tx=0).stop 2023-11-16T16:55:40.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.810+0000 7fe7420e9b80 1 -- 172.21.15.44:0/1189155295 shutdown_connections 2023-11-16T16:55:40.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.810+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/1189155295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8f8b0 0x561fe6d8fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.810+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/1189155295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8ab60 0x561fe6d8af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.810+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/1189155295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 0x561fe6d8f370 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.810+0000 7fe7420e9b80 1 -- 172.21.15.44:0/1189155295 >> 172.21.15.44:0/1189155295 conn(0x561fe6d80360 msgr2=0x561fe6d85b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:40.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.810+0000 7fe7420e9b80 1 -- 172.21.15.44:0/1189155295 shutdown_connections 2023-11-16T16:55:40.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.810+0000 7fe7420e9b80 1 -- 172.21.15.44:0/1189155295 wait complete. 2023-11-16T16:55:40.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.811+0000 7fe7420e9b80 1 -- start start 2023-11-16T16:55:40.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.811+0000 7fe7420e9b80 1 Processor -- start 2023-11-16T16:55:40.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.811+0000 7fe7420e9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8ab60 0x561fe6d970e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.811+0000 7fe7420e9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 0x561fe6d942f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.811+0000 7fe7420e9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8f8b0 0x561fe6d94a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.811+0000 7fe7420e9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fe6c7d8e0 con 0x561fe6d8f8b0 2023-11-16T16:55:40.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.811+0000 7fe7420e9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fe6c7d580 con 0x561fe6d8ab60 2023-11-16T16:55:40.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.811+0000 7fe7420e9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fe6b5f920 con 0x561fe6d8b470 2023-11-16T16:55:40.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.812+0000 7fe7420e9b80 10 client.0 _ll_register_callbacks cb 0x561fe6c9b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:40.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.812+0000 7fe727fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8ab60 0x561fe6d970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.812+0000 7fe727fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8ab60 0x561fe6d970e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:44944/0 (socket says 172.21.15.44:44944) 2023-11-16T16:55:40.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.812+0000 7fe727fff700 1 -- 172.21.15.44:0/659694399 learned_addr learned my addr 172.21.15.44:0/659694399 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:40.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.812+0000 7fe7277fe700 1 --2- 172.21.15.44:0/659694399 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 0x561fe6d942f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.812+0000 7fe72cd9c700 1 --2- 172.21.15.44:0/659694399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8f8b0 0x561fe6d94a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.812+0000 7fe727fff700 1 -- 172.21.15.44:0/659694399 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 msgr2=0x561fe6d942f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:40.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.812+0000 7fe727fff700 1 --2- 172.21.15.44:0/659694399 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 0x561fe6d942f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.812+0000 7fe727fff700 1 -- 172.21.15.44:0/659694399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8f8b0 msgr2=0x561fe6d94a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:40.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.812+0000 7fe727fff700 1 --2- 172.21.15.44:0/659694399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8f8b0 0x561fe6d94a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.812+0000 7fe727fff700 1 -- 172.21.15.44:0/659694399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71c01b040 con 0x561fe6d8ab60 2023-11-16T16:55:40.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.812+0000 7fe72cd9c700 1 --2- 172.21.15.44:0/659694399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8f8b0 0x561fe6d94a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:55:40.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.812+0000 7fe7277fe700 1 --2- 172.21.15.44:0/659694399 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 0x561fe6d942f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:55:40.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.812+0000 7fe727fff700 1 --2- 172.21.15.44:0/659694399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8ab60 0x561fe6d970e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe7180128a0 tx=0x7fe718012c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b852f743bc9ca10 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:40.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.812+0000 7fe7077fe700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:55:40.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.813+0000 7fe7077fe700 1 -- 172.21.15.44:0/659694399 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe718012ea0 con 0x561fe6d8ab60 2023-11-16T16:55:40.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.813+0000 7fe7077fe700 1 -- 172.21.15.44:0/659694399 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fe718004830 con 0x561fe6d8ab60 2023-11-16T16:55:40.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.813+0000 7fe7420e9b80 1 -- 172.21.15.44:0/659694399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561fe6d7db20 con 0x561fe6d8ab60 2023-11-16T16:55:40.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.814+0000 7fe7077fe700 1 -- 172.21.15.44:0/659694399 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe718017070 con 0x561fe6d8ab60 2023-11-16T16:55:40.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.814+0000 7fe7420e9b80 10 client.26669 fetch_fsmap learned FSMap version 21 2023-11-16T16:55:40.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.814+0000 7fe7420e9b80 1 -- 172.21.15.44:0/659694399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x561fe6ca2190 con 0x561fe6d8ab60 2023-11-16T16:55:40.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.814+0000 7fe7077fe700 1 -- 172.21.15.44:0/659694399 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561fe6d7db20 con 0x561fe6d8ab60 2023-11-16T16:55:40.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.814+0000 7fe7077fe700 1 -- 172.21.15.44:0/659694399 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe71801c990 con 0x561fe6d8ab60 2023-11-16T16:55:40.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.814+0000 7fe7420e9b80 10 client.26669 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:55:40.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.814+0000 7fe7420e9b80 10 client.26669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:40.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.814+0000 7fe7420e9b80 20 client.26669 populate_metadata read hostname 'smithi044' 2023-11-16T16:55:40.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.814+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/659694399 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561fe6da47d0 0x561fe6da6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.814+0000 7fe7420e9b80 4 client.26669 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:55:40.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.814+0000 7fe7420e9b80 1 -- 172.21.15.44:0/659694399 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561fe6c9e0a0 con 0x561fe6da47d0 2023-11-16T16:55:40.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.815+0000 7fe7277fe700 1 --2- 172.21.15.44:0/659694399 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561fe6da47d0 0x561fe6da6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.815+0000 7fe7277fe700 1 --2- 172.21.15.44:0/659694399 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561fe6da47d0 0x561fe6da6be0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0d8d3a4d2fe0369 server_cookie=cc7a11e39273f87e in_seq=0 out_seq=0 2023-11-16T16:55:40.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.815+0000 7fe7077fe700 10 client.26669 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:40.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.820+0000 7fe7077fe700 1 -- 172.21.15.44:0/659694399 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x561fe6c9e0a0 con 0x561fe6da47d0 2023-11-16T16:55:40.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.820+0000 7fe7077fe700 10 client.26669 handle_command_reply: tid=0 2023-11-16T16:55:40.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.820+0000 7fe7420e9b80 1 client.26669 shutdown 2023-11-16T16:55:40.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.821+0000 7fe7420e9b80 1 -- 172.21.15.44:0/659694399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8ab60 msgr2=0x561fe6d970e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:40.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.821+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/659694399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8ab60 0x561fe6d970e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe7180128a0 tx=0x7fe718012c60 comp rx=0 tx=0).stop 2023-11-16T16:55:40.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.821+0000 7fe7420e9b80 1 -- 172.21.15.44:0/659694399 shutdown_connections 2023-11-16T16:55:40.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.821+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/659694399 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561fe6da47d0 0x561fe6da6be0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.821+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/659694399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8f8b0 0x561fe6d94a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.821+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/659694399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8ab60 0x561fe6d970e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.821+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/659694399 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 0x561fe6d942f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.821+0000 7fe7420e9b80 1 -- 172.21.15.44:0/659694399 >> 172.21.15.44:0/659694399 conn(0x561fe6d80360 msgr2=0x561fe6d90d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:40.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.821+0000 7fe7077fe700 0 client.26669 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:40.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.821+0000 7fe7420e9b80 1 -- 172.21.15.44:0/659694399 shutdown_connections 2023-11-16T16:55:40.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.821+0000 7fe7420e9b80 1 -- 172.21.15.44:0/659694399 wait complete. 2023-11-16T16:55:40.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.821+0000 7fe7420e9b80 20 client.26669 trim_cache size 0 max 16384 2023-11-16T16:55:40.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.842+0000 7fe7420e9b80 1 Processor -- start 2023-11-16T16:55:40.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.842+0000 7fe7420e9b80 1 -- start start 2023-11-16T16:55:40.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.843+0000 7fe7420e9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8ab60 0x561fe6d85c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.843+0000 7fe7420e9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8b470 0x561fe6d96c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.843+0000 7fe7420e9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8f8b0 0x561fe6d91c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.843+0000 7fe7420e9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fe6d95780 con 0x561fe6d8f8b0 2023-11-16T16:55:40.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.843+0000 7fe7420e9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fe6d8b900 con 0x561fe6d8b470 2023-11-16T16:55:40.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.843+0000 7fe7420e9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fe6da6e00 con 0x561fe6d8ab60 2023-11-16T16:55:40.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.843+0000 7fe7277fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8b470 0x561fe6d96c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.843+0000 7fe7277fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8b470 0x561fe6d96c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:44950/0 (socket says 172.21.15.44:44950) 2023-11-16T16:55:40.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.843+0000 7fe7277fe700 1 -- 172.21.15.44:0/801058376 learned_addr learned my addr 172.21.15.44:0/801058376 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:40.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.843+0000 7fe727fff700 1 --2- 172.21.15.44:0/801058376 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8ab60 0x561fe6d85c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.843+0000 7fe72cd9c700 1 --2- 172.21.15.44:0/801058376 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8f8b0 0x561fe6d91c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.844+0000 7fe7277fe700 1 -- 172.21.15.44:0/801058376 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8ab60 msgr2=0x561fe6d85c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:40.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.844+0000 7fe7277fe700 1 --2- 172.21.15.44:0/801058376 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8ab60 0x561fe6d85c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.844+0000 7fe7277fe700 1 -- 172.21.15.44:0/801058376 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8f8b0 msgr2=0x561fe6d91c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:40.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.844+0000 7fe7277fe700 1 --2- 172.21.15.44:0/801058376 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8f8b0 0x561fe6d91c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.844+0000 7fe7277fe700 1 -- 172.21.15.44:0/801058376 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561fe6d81b00 con 0x561fe6d8b470 2023-11-16T16:55:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.844+0000 7fe7277fe700 1 --2- 172.21.15.44:0/801058376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8b470 0x561fe6d96c10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe71c013510 tx=0x7fe71c00ff00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c95abe38a39544dc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.844+0000 7fe70effd700 1 -- 172.21.15.44:0/801058376 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c0131f0 con 0x561fe6d8b470 2023-11-16T16:55:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.844+0000 7fe70effd700 1 -- 172.21.15.44:0/801058376 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe71c030250 con 0x561fe6d8b470 2023-11-16T16:55:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.844+0000 7fe70effd700 1 -- 172.21.15.44:0/801058376 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c005550 con 0x561fe6d8b470 2023-11-16T16:55:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.845+0000 7fe7420e9b80 1 -- 172.21.15.44:0/801058376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8b470 msgr2=0x561fe6d96c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.845+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/801058376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8b470 0x561fe6d96c10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe71c013510 tx=0x7fe71c00ff00 comp rx=0 tx=0).stop 2023-11-16T16:55:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.845+0000 7fe7420e9b80 1 -- 172.21.15.44:0/801058376 shutdown_connections 2023-11-16T16:55:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.845+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/801058376 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8f8b0 0x561fe6d91c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.845+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/801058376 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8ab60 0x561fe6d85c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.845+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/801058376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8b470 0x561fe6d96c10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.845+0000 7fe7420e9b80 1 -- 172.21.15.44:0/801058376 >> 172.21.15.44:0/801058376 conn(0x561fe6d80360 msgr2=0x561fe6d8d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.845+0000 7fe7420e9b80 1 -- 172.21.15.44:0/801058376 shutdown_connections 2023-11-16T16:55:40.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.845+0000 7fe7420e9b80 1 -- 172.21.15.44:0/801058376 wait complete. 2023-11-16T16:55:40.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.846+0000 7fe7420e9b80 1 -- start start 2023-11-16T16:55:40.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.846+0000 7fe7420e9b80 1 Processor -- start 2023-11-16T16:55:40.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.846+0000 7fe7420e9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8ab60 0x561fe6d998d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.846+0000 7fe7420e9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 0x561fe6d99e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.846+0000 7fe7420e9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8f8b0 0x561fe6d9a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.846+0000 7fe7420e9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fe6df7860 con 0x561fe6d8ab60 2023-11-16T16:55:40.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.846+0000 7fe7420e9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fe6df79d0 con 0x561fe6d8f8b0 2023-11-16T16:55:40.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.846+0000 7fe7420e9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fe6df7b90 con 0x561fe6d8b470 2023-11-16T16:55:40.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.846+0000 7fe7420e9b80 10 client.0 _ll_register_callbacks cb 0x7fe718009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:40.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.847+0000 7fe727fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8ab60 0x561fe6d998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.847+0000 7fe727fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8ab60 0x561fe6d998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42740/0 (socket says 172.21.15.44:42740) 2023-11-16T16:55:40.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.847+0000 7fe727fff700 1 -- 172.21.15.44:0/3256623444 learned_addr learned my addr 172.21.15.44:0/3256623444 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.847+0000 7fe72cd9c700 1 --2- 172.21.15.44:0/3256623444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8f8b0 0x561fe6d9a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.847+0000 7fe7277fe700 1 --2- 172.21.15.44:0/3256623444 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 0x561fe6d99e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.847+0000 7fe72cd9c700 1 -- 172.21.15.44:0/3256623444 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 msgr2=0x561fe6d99e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.847+0000 7fe72cd9c700 1 --2- 172.21.15.44:0/3256623444 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 0x561fe6d99e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.847+0000 7fe72cd9c700 1 -- 172.21.15.44:0/3256623444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8ab60 msgr2=0x561fe6d998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.847+0000 7fe72cd9c700 1 --2- 172.21.15.44:0/3256623444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8ab60 0x561fe6d998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.847+0000 7fe72cd9c700 1 -- 172.21.15.44:0/3256623444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561fe6b5d630 con 0x561fe6d8f8b0 2023-11-16T16:55:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.847+0000 7fe727fff700 1 --2- 172.21.15.44:0/3256623444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8ab60 0x561fe6d998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:55:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.847+0000 7fe72cd9c700 1 --2- 172.21.15.44:0/3256623444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8f8b0 0x561fe6d9a390 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe7200cb800 tx=0x7fe72009ae20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e42353e2064cd72 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.847+0000 7fe7077fe700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:55:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.848+0000 7fe7077fe700 1 -- 172.21.15.44:0/3256623444 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7200ce220 con 0x561fe6d8f8b0 2023-11-16T16:55:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.848+0000 7fe7077fe700 1 -- 172.21.15.44:0/3256623444 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fe72009b210 con 0x561fe6d8f8b0 2023-11-16T16:55:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.848+0000 7fe7420e9b80 1 -- 172.21.15.44:0/3256623444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe71c0099c0 con 0x561fe6d8f8b0 2023-11-16T16:55:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.848+0000 7fe7077fe700 1 -- 172.21.15.44:0/3256623444 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe72001daf0 con 0x561fe6d8f8b0 2023-11-16T16:55:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.848+0000 7fe7420e9b80 10 client.26675 fetch_fsmap learned FSMap version 21 2023-11-16T16:55:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.848+0000 7fe7420e9b80 1 -- 172.21.15.44:0/3256623444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x561fe6b09de0 con 0x561fe6d8f8b0 2023-11-16T16:55:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.848+0000 7fe7077fe700 1 -- 172.21.15.44:0/3256623444 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe71c0099c0 con 0x561fe6d8f8b0 2023-11-16T16:55:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.848+0000 7fe7077fe700 1 -- 172.21.15.44:0/3256623444 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe72009a700 con 0x561fe6d8f8b0 2023-11-16T16:55:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.848+0000 7fe7420e9b80 10 client.26675 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:55:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.848+0000 7fe7420e9b80 10 client.26675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.848+0000 7fe7420e9b80 20 client.26675 populate_metadata read hostname 'smithi044' 2023-11-16T16:55:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.848+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/3256623444 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561fe6df9110 0x561fe6dfb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.848+0000 7fe7420e9b80 4 client.26675 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:55:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.848+0000 7fe7420e9b80 1 -- 172.21.15.44:0/3256623444 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561fe6caa660 con 0x561fe6df9110 2023-11-16T16:55:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.849+0000 7fe727fff700 1 --2- 172.21.15.44:0/3256623444 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561fe6df9110 0x561fe6dfb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:40.854 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.849+0000 7fe727fff700 1 --2- 172.21.15.44:0/3256623444 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561fe6df9110 0x561fe6dfb4f0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd176feb270fb252 server_cookie=2f35dc27353a4bff in_seq=0 out_seq=0 2023-11-16T16:55:40.854 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.849+0000 7fe7077fe700 10 client.26675 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:40.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.854+0000 7fe7077fe700 1 -- 172.21.15.44:0/3256623444 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x561fe6caa660 con 0x561fe6df9110 2023-11-16T16:55:40.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.854+0000 7fe7077fe700 10 client.26675 handle_command_reply: tid=0 2023-11-16T16:55:40.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.854+0000 7fe7420e9b80 1 client.26675 shutdown 2023-11-16T16:55:40.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.855+0000 7fe7420e9b80 1 -- 172.21.15.44:0/3256623444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8f8b0 msgr2=0x561fe6d9a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:40.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.855+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/3256623444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8f8b0 0x561fe6d9a390 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe7200cb800 tx=0x7fe72009ae20 comp rx=0 tx=0).stop 2023-11-16T16:55:40.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.855+0000 7fe7420e9b80 1 -- 172.21.15.44:0/3256623444 shutdown_connections 2023-11-16T16:55:40.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.855+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/3256623444 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561fe6df9110 0x561fe6dfb4f0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.855+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/3256623444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fe6d8f8b0 0x561fe6d9a390 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.855+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/3256623444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561fe6d8ab60 0x561fe6d998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.855+0000 7fe7077fe700 0 client.26675 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:40.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.855+0000 7fe7420e9b80 1 --2- 172.21.15.44:0/3256623444 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561fe6d8b470 0x561fe6d99e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.855+0000 7fe7420e9b80 1 -- 172.21.15.44:0/3256623444 >> 172.21.15.44:0/3256623444 conn(0x561fe6d80360 msgr2=0x561fe6d91c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:40.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.855+0000 7fe7420e9b80 1 -- 172.21.15.44:0/3256623444 shutdown_connections 2023-11-16T16:55:40.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.855+0000 7fe7420e9b80 1 -- 172.21.15.44:0/3256623444 wait complete. 2023-11-16T16:55:40.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.855+0000 7fe7420e9b80 20 client.26675 trim_cache size 0 max 16384 2023-11-16T16:55:40.858 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:55:40.858 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "scrub active (5643 inodes in the stack)", 2023-11-16T16:55:40.858 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2023-11-16T16:55:40.858 INFO:teuthology.orchestra.run.smithi044.stdout: "c6d82195-2d53-46f4-9f2b-dc4452b4a1b0": { 2023-11-16T16:55:40.859 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2023-11-16T16:55:40.859 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "c6d82195-2d53-46f4-9f2b-dc4452b4a1b0", 2023-11-16T16:55:40.859 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2023-11-16T16:55:40.859 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:55:40.859 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:55:40.859 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:55:40.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.856+0000 7fe72e7ff700 1 -- 172.21.15.44:0/288423419 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe7100a1340 msgr2=0x7fe7100a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:40.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.856+0000 7fe72e7ff700 1 --2- 172.21.15.44:0/288423419 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe7100a1340 0x7fe7100a37f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe71c013a70 tx=0x7fe71c009c00 comp rx=0 tx=0).stop 2023-11-16T16:55:40.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.856+0000 7fe72e7ff700 1 -- 172.21.15.44:0/288423419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe728074400 msgr2=0x7fe72819c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:40.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.856+0000 7fe72e7ff700 1 --2- 172.21.15.44:0/288423419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe728074400 0x7fe72819c9c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe720009fd0 tx=0x7fe720007930 comp rx=0 tx=0).stop 2023-11-16T16:55:40.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.856+0000 7fe72e7ff700 1 -- 172.21.15.44:0/288423419 shutdown_connections 2023-11-16T16:55:40.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.856+0000 7fe72e7ff700 1 --2- 172.21.15.44:0/288423419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe728074400 0x7fe72819c9c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.856+0000 7fe72e7ff700 1 --2- 172.21.15.44:0/288423419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe728072d20 0x7fe7281a2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.856+0000 7fe72e7ff700 1 --2- 172.21.15.44:0/288423419 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe7100a1340 0x7fe7100a37f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.856+0000 7fe72e7ff700 1 --2- 172.21.15.44:0/288423419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7280736d0 0x7fe7281a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:40.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.856+0000 7fe72e7ff700 1 -- 172.21.15.44:0/288423419 >> 172.21.15.44:0/288423419 conn(0x7fe728077590 msgr2=0x7fe7281126e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:40.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.857+0000 7fe72e7ff700 1 -- 172.21.15.44:0/288423419 shutdown_connections 2023-11-16T16:55:40.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:55:40.857+0000 7fe72e7ff700 1 -- 172.21.15.44:0/288423419 wait complete. 2023-11-16T16:55:40.879 INFO:tasks.cephfs.filesystem:scrub status for tag:c6d82195-2d53-46f4-9f2b-dc4452b4a1b0 - {'path': '/', 'tag': 'c6d82195-2d53-46f4-9f2b-dc4452b4a1b0', 'options': 'recursive,force'} 2023-11-16T16:55:41.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:41 smithi077 ceph-mon[131204]: pgmap v175: 129 pgs: 129 active+clean; 29 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 585 op/s 2023-11-16T16:55:41.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:41 smithi044 ceph-mon[128750]: pgmap v175: 129 pgs: 129 active+clean; 29 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 585 op/s 2023-11-16T16:55:41.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:41 smithi176 ceph-mon[131630]: pgmap v175: 129 pgs: 129 active+clean; 29 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 585 op/s 2023-11-16T16:55:43.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:43 smithi077 ceph-mon[131204]: pgmap v176: 129 pgs: 129 active+clean; 33 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 632 op/s 2023-11-16T16:55:43.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:43 smithi044 ceph-mon[128750]: pgmap v176: 129 pgs: 129 active+clean; 33 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 632 op/s 2023-11-16T16:55:43.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:43 smithi176 ceph-mon[131630]: pgmap v176: 129 pgs: 129 active+clean; 33 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 632 op/s 2023-11-16T16:55:45.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:45 smithi077 ceph-mon[131204]: pgmap v177: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 858 op/s 2023-11-16T16:55:45.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:45 smithi044 ceph-mon[128750]: pgmap v177: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 858 op/s 2023-11-16T16:55:45.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:45 smithi176 ceph-mon[131630]: pgmap v177: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 858 op/s 2023-11-16T16:55:47.601 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.599+0000 7f9a327ff700 1 -- 172.21.15.44:0/314872071 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a2c104400 msgr2=0x7f9a2c10bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:47.601 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.599+0000 7f9a327ff700 1 --2- 172.21.15.44:0/314872071 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a2c104400 0x7f9a2c10bfd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9a20009a40 tx=0x7f9a20009d50 comp rx=0 tx=0).stop 2023-11-16T16:55:47.601 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.599+0000 7f9a327ff700 1 -- 172.21.15.44:0/314872071 shutdown_connections 2023-11-16T16:55:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.599+0000 7f9a327ff700 1 --2- 172.21.15.44:0/314872071 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a2c104400 0x7f9a2c10bfd0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.599+0000 7f9a327ff700 1 --2- 172.21.15.44:0/314872071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a2c0ff3c0 0x7f9a2c0ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.600+0000 7f9a327ff700 1 --2- 172.21.15.44:0/314872071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a2c0ffd60 0x7f9a2c103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.600+0000 7f9a327ff700 1 -- 172.21.15.44:0/314872071 >> 172.21.15.44:0/314872071 conn(0x7f9a2c0763a0 msgr2=0x7f9a2c0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:47.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.603+0000 7f9a327ff700 1 -- 172.21.15.44:0/314872071 shutdown_connections 2023-11-16T16:55:47.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.604+0000 7f9a327ff700 1 -- 172.21.15.44:0/314872071 wait complete. 2023-11-16T16:55:47.607 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.606+0000 7f9a327ff700 1 Processor -- start 2023-11-16T16:55:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.607+0000 7f9a327ff700 1 -- start start 2023-11-16T16:55:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.607+0000 7f9a327ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a2c0ff3c0 0x7f9a2c10be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.608+0000 7f9a327ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a2c0ffd60 0x7f9a2c119990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.608+0000 7f9a327ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a2c104400 0x7f9a2c109e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.608+0000 7f9a2bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a2c0ff3c0 0x7f9a2c10be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.608+0000 7f9a327ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c10a440 con 0x7f9a2c0ffd60 2023-11-16T16:55:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.608+0000 7f9a327ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c10a5b0 con 0x7f9a2c104400 2023-11-16T16:55:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.608+0000 7f9a327ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c10a770 con 0x7f9a2c0ff3c0 2023-11-16T16:55:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.608+0000 7f9a2bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a2c0ff3c0 0x7f9a2c10be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:50446/0 (socket says 172.21.15.44:50446) 2023-11-16T16:55:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.608+0000 7f9a2bfff700 1 -- 172.21.15.44:0/1890098656 learned_addr learned my addr 172.21.15.44:0/1890098656 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.609+0000 7f9a2bfff700 1 -- 172.21.15.44:0/1890098656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a2c104400 msgr2=0x7f9a2c109e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:55:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.609+0000 7f9a2bfff700 1 --2- 172.21.15.44:0/1890098656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a2c104400 0x7f9a2c109e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.609+0000 7f9a2bfff700 1 -- 172.21.15.44:0/1890098656 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a2c0ffd60 msgr2=0x7f9a2c119990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.609+0000 7f9a2bfff700 1 --2- 172.21.15.44:0/1890098656 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a2c0ffd60 0x7f9a2c119990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.609+0000 7f9a2bfff700 1 -- 172.21.15.44:0/1890098656 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a200095f0 con 0x7f9a2c0ff3c0 2023-11-16T16:55:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.610+0000 7f9a2bfff700 1 --2- 172.21.15.44:0/1890098656 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a2c0ff3c0 0x7f9a2c10be20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00d980 tx=0x7f9a1c00dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.610+0000 7f9a297fa700 1 -- 172.21.15.44:0/1890098656 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c00bb60 con 0x7f9a2c0ff3c0 2023-11-16T16:55:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.611+0000 7f9a327ff700 1 -- 172.21.15.44:0/1890098656 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2c10d710 con 0x7f9a2c0ff3c0 2023-11-16T16:55:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.611+0000 7f9a327ff700 1 -- 172.21.15.44:0/1890098656 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a2c10d580 con 0x7f9a2c0ff3c0 2023-11-16T16:55:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.612+0000 7f9a297fa700 1 -- 172.21.15.44:0/1890098656 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a1c007640 con 0x7f9a2c0ff3c0 2023-11-16T16:55:47.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.612+0000 7f9a297fa700 1 -- 172.21.15.44:0/1890098656 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c00bb60 con 0x7f9a2c0ff3c0 2023-11-16T16:55:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.623+0000 7f9a297fa700 1 -- 172.21.15.44:0/1890098656 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f9a1c003a40 con 0x7f9a2c0ff3c0 2023-11-16T16:55:47.628 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.626+0000 7f9a297fa700 1 --2- 172.21.15.44:0/1890098656 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9a140a1450 0x7f9a140a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.627+0000 7f9a297fa700 1 -- 172.21.15.44:0/1890098656 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f9a1c0cc7f0 con 0x7f9a2c0ff3c0 2023-11-16T16:55:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.627+0000 7f9a2b7fe700 1 --2- 172.21.15.44:0/1890098656 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9a140a1450 0x7f9a140a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.627+0000 7f9a46065b80 1 Processor -- start 2023-11-16T16:55:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.627+0000 7f9a46065b80 1 -- start start 2023-11-16T16:55:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.628+0000 7f9a46065b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e55686b0 0x5584e57121e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.628+0000 7f9a46065b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e5712720 0x5584e570c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.628+0000 7f9a46065b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712b90 0x5584e570cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.629+0000 7f9a46065b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e5612000 con 0x5584e5712720 2023-11-16T16:55:47.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.629+0000 7f9a46065b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e5664840 con 0x5584e5712b90 2023-11-16T16:55:47.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.629+0000 7f9a46065b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e565dcd0 con 0x5584e55686b0 2023-11-16T16:55:47.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.630+0000 7f9a2bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e55686b0 0x5584e57121e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:47.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.630+0000 7f9a2bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e55686b0 0x5584e57121e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:50452/0 (socket says 172.21.15.44:50452) 2023-11-16T16:55:47.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.630+0000 7f9a2bfff700 1 -- 172.21.15.44:0/1148983502 learned_addr learned my addr 172.21.15.44:0/1148983502 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.630+0000 7f9a2b7fe700 1 --2- 172.21.15.44:0/1148983502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e5712720 0x5584e570c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.630+0000 7f9a2bfff700 1 -- 172.21.15.44:0/1148983502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712b90 msgr2=0x5584e570cb00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T16:55:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.630+0000 7f9a2bfff700 1 --2- 172.21.15.44:0/1148983502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712b90 0x5584e570cb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.630+0000 7f9a2bfff700 1 -- 172.21.15.44:0/1148983502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e5712720 msgr2=0x5584e570c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.631+0000 7f9a2bfff700 1 --2- 172.21.15.44:0/1148983502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e5712720 0x5584e570c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.631+0000 7f9a2bfff700 1 -- 172.21.15.44:0/1148983502 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584e54df5d0 con 0x5584e55686b0 2023-11-16T16:55:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.631+0000 7f9a2b7fe700 1 --2- 172.21.15.44:0/1148983502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e5712720 0x5584e570c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:55:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.631+0000 7f9a2b7fe700 1 --2- 172.21.15.44:0/1890098656 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9a140a1450 0x7f9a140a3900 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9a24009710 tx=0x7f9a24009620 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.631+0000 7f9a2bfff700 1 --2- 172.21.15.44:0/1148983502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e55686b0 0x5584e57121e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0a1400 tx=0x7f9a1c099150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cae103b6c9f4d6d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.632+0000 7f9a127fc700 1 -- 172.21.15.44:0/1148983502 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c09d510 con 0x5584e55686b0 2023-11-16T16:55:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.633+0000 7f9a127fc700 1 -- 172.21.15.44:0/1148983502 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a1c0a1930 con 0x5584e55686b0 2023-11-16T16:55:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.633+0000 7f9a127fc700 1 -- 172.21.15.44:0/1148983502 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c0c9de0 con 0x5584e55686b0 2023-11-16T16:55:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.637+0000 7f9a46065b80 1 -- 172.21.15.44:0/1148983502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e55686b0 msgr2=0x5584e57121e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.637+0000 7f9a46065b80 1 --2- 172.21.15.44:0/1148983502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e55686b0 0x5584e57121e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0a1400 tx=0x7f9a1c099150 comp rx=0 tx=0).stop 2023-11-16T16:55:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.638+0000 7f9a46065b80 1 -- 172.21.15.44:0/1148983502 shutdown_connections 2023-11-16T16:55:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.638+0000 7f9a46065b80 1 --2- 172.21.15.44:0/1148983502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712b90 0x5584e570cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.638+0000 7f9a46065b80 1 --2- 172.21.15.44:0/1148983502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e55686b0 0x5584e57121e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.638+0000 7f9a46065b80 1 --2- 172.21.15.44:0/1148983502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e5712720 0x5584e570c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.638+0000 7f9a46065b80 1 -- 172.21.15.44:0/1148983502 >> 172.21.15.44:0/1148983502 conn(0x5584e5701f50 msgr2=0x5584e5707270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.639+0000 7f9a46065b80 1 -- 172.21.15.44:0/1148983502 shutdown_connections 2023-11-16T16:55:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.639+0000 7f9a46065b80 1 -- 172.21.15.44:0/1148983502 wait complete. 2023-11-16T16:55:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.640+0000 7f9a46065b80 1 -- start start 2023-11-16T16:55:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.640+0000 7f9a46065b80 1 Processor -- start 2023-11-16T16:55:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.644+0000 7f9a46065b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e55686b0 0x5584e5718a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.644+0000 7f9a46065b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e5712720 0x5584e5715ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.644+0000 7f9a46065b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712b90 0x5584e57162d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.644+0000 7f9a2bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e55686b0 0x5584e5718a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.644+0000 7f9a46065b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e55ff880 con 0x5584e55686b0 2023-11-16T16:55:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.644+0000 7f9a46065b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e55ff520 con 0x5584e5712b90 2023-11-16T16:55:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.644+0000 7f9a46065b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e54e18c0 con 0x5584e5712720 2023-11-16T16:55:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.644+0000 7f9a2bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e55686b0 0x5584e5718a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33266/0 (socket says 172.21.15.44:33266) 2023-11-16T16:55:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.644+0000 7f9a2bfff700 1 -- 172.21.15.44:0/3681381746 learned_addr learned my addr 172.21.15.44:0/3681381746 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.644+0000 7f9a46065b80 10 client.0 _ll_register_callbacks cb 0x5584e561d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.644+0000 7f9a30d9c700 1 --2- 172.21.15.44:0/3681381746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712b90 0x5584e57162d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.645+0000 7f9a2bfff700 1 -- 172.21.15.44:0/3681381746 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e5712720 msgr2=0x5584e5715ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T16:55:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.645+0000 7f9a2bfff700 1 --2- 172.21.15.44:0/3681381746 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e5712720 0x5584e5715ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.645+0000 7f9a2bfff700 1 -- 172.21.15.44:0/3681381746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712b90 msgr2=0x5584e57162d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.645+0000 7f9a2bfff700 1 --2- 172.21.15.44:0/3681381746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712b90 0x5584e57162d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.645+0000 7f9a2bfff700 1 -- 172.21.15.44:0/3681381746 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a2c10d580 con 0x5584e55686b0 2023-11-16T16:55:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.645+0000 7f9a2bfff700 1 --2- 172.21.15.44:0/3681381746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e55686b0 0x5584e5718a20 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00b540 tx=0x7f9a1c0c9bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61c9ab9af71d56b5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.645+0000 7f9a0b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:55:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.646+0000 7f9a0b7fe700 1 -- 172.21.15.44:0/3681381746 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c097880 con 0x5584e55686b0 2023-11-16T16:55:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.646+0000 7f9a0b7fe700 1 -- 172.21.15.44:0/3681381746 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f9a1c0652c0 con 0x5584e55686b0 2023-11-16T16:55:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.647+0000 7f9a46065b80 1 -- 172.21.15.44:0/3681381746 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5584e56ff710 con 0x5584e55686b0 2023-11-16T16:55:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.648+0000 7f9a0b7fe700 1 -- 172.21.15.44:0/3681381746 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a1c00bcc0 con 0x5584e55686b0 2023-11-16T16:55:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.648+0000 7f9a46065b80 10 client.16644 fetch_fsmap learned FSMap version 21 2023-11-16T16:55:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.648+0000 7f9a46065b80 1 -- 172.21.15.44:0/3681381746 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5584e5624130 con 0x5584e55686b0 2023-11-16T16:55:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.648+0000 7f9a0b7fe700 1 -- 172.21.15.44:0/3681381746 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5584e56ff710 con 0x5584e55686b0 2023-11-16T16:55:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.649+0000 7f9a0b7fe700 1 -- 172.21.15.44:0/3681381746 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9a1c0cb070 con 0x5584e55686b0 2023-11-16T16:55:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.649+0000 7f9a46065b80 10 client.16644 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:55:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.649+0000 7f9a46065b80 10 client.16644 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.649+0000 7f9a46065b80 20 client.16644 populate_metadata read hostname 'smithi044' 2023-11-16T16:55:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.649+0000 7f9a46065b80 1 --2- 172.21.15.44:0/3681381746 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5584e5725aa0 0x5584e5727eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.649+0000 7f9a46065b80 4 client.16644 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:55:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.649+0000 7f9a46065b80 1 -- 172.21.15.44:0/3681381746 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5584e5620040 con 0x5584e5725aa0 2023-11-16T16:55:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.649+0000 7f9a2b7fe700 1 --2- 172.21.15.44:0/3681381746 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5584e5725aa0 0x5584e5727eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.650+0000 7f9a0b7fe700 10 client.16644 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.650+0000 7f9a2b7fe700 1 --2- 172.21.15.44:0/3681381746 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5584e5725aa0 0x5584e5727eb0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=884bb6bf2468e78f server_cookie=5689578fcb835efd in_seq=0 out_seq=0 2023-11-16T16:55:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.665+0000 7f9a0b7fe700 1 -- 172.21.15.44:0/3681381746 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5584e5620040 con 0x5584e5725aa0 2023-11-16T16:55:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.665+0000 7f9a0b7fe700 10 client.16644 handle_command_reply: tid=0 2023-11-16T16:55:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.666+0000 7f9a46065b80 1 client.16644 shutdown 2023-11-16T16:55:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.666+0000 7f9a46065b80 1 -- 172.21.15.44:0/3681381746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e55686b0 msgr2=0x5584e5718a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.666+0000 7f9a46065b80 1 --2- 172.21.15.44:0/3681381746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e55686b0 0x5584e5718a20 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00b540 tx=0x7f9a1c0c9bb0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.667+0000 7f9a46065b80 1 -- 172.21.15.44:0/3681381746 shutdown_connections 2023-11-16T16:55:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.667+0000 7f9a46065b80 1 --2- 172.21.15.44:0/3681381746 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5584e5725aa0 0x5584e5727eb0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.667+0000 7f9a46065b80 1 --2- 172.21.15.44:0/3681381746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712b90 0x5584e57162d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.667+0000 7f9a46065b80 1 --2- 172.21.15.44:0/3681381746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e55686b0 0x5584e5718a20 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.667+0000 7f9a46065b80 1 --2- 172.21.15.44:0/3681381746 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e5712720 0x5584e5715ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.667+0000 7f9a46065b80 1 -- 172.21.15.44:0/3681381746 >> 172.21.15.44:0/3681381746 conn(0x5584e5701f50 msgr2=0x5584e57067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.667+0000 7f9a0b7fe700 0 client.16644 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.667+0000 7f9a46065b80 1 -- 172.21.15.44:0/3681381746 shutdown_connections 2023-11-16T16:55:47.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.667+0000 7f9a46065b80 1 -- 172.21.15.44:0/3681381746 wait complete. 2023-11-16T16:55:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.669+0000 7f9a46065b80 20 client.16644 trim_cache size 0 max 16384 2023-11-16T16:55:47.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.711+0000 7f9a46065b80 1 Processor -- start 2023-11-16T16:55:47.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.711+0000 7f9a46065b80 1 -- start start 2023-11-16T16:55:47.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.711+0000 7f9a46065b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e55686b0 0x5584e571d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.711+0000 7f9a46065b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e5712720 0x5584e57185c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.711+0000 7f9a46065b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712b90 0x5584e5715460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.712+0000 7f9a46065b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e57109a0 con 0x5584e5712720 2023-11-16T16:55:47.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.712+0000 7f9a46065b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e571d560 con 0x5584e5712b90 2023-11-16T16:55:47.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.712+0000 7f9a46065b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e5711d50 con 0x5584e55686b0 2023-11-16T16:55:47.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.712+0000 7f9a30d9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712b90 0x5584e5715460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.712+0000 7f9a30d9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712b90 0x5584e5715460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:53778/0 (socket says 172.21.15.44:53778) 2023-11-16T16:55:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.712+0000 7f9a30d9c700 1 -- 172.21.15.44:0/291930374 learned_addr learned my addr 172.21.15.44:0/291930374 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.712+0000 7f9a2b7fe700 1 --2- 172.21.15.44:0/291930374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e5712720 0x5584e57185c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.712+0000 7f9a2bfff700 1 --2- 172.21.15.44:0/291930374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e55686b0 0x5584e571d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.713+0000 7f9a2bfff700 1 -- 172.21.15.44:0/291930374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712b90 msgr2=0x5584e5715460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.713+0000 7f9a2bfff700 1 --2- 172.21.15.44:0/291930374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712b90 0x5584e5715460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.713+0000 7f9a2bfff700 1 -- 172.21.15.44:0/291930374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e5712720 msgr2=0x5584e57185c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.713+0000 7f9a2bfff700 1 --2- 172.21.15.44:0/291930374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e5712720 0x5584e57185c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.713+0000 7f9a2bfff700 1 -- 172.21.15.44:0/291930374 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584e5623e80 con 0x5584e55686b0 2023-11-16T16:55:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.713+0000 7f9a2bfff700 1 --2- 172.21.15.44:0/291930374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e55686b0 0x5584e571d020 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0a14f0 tx=0x7f9a1c0842d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68c91ec7b4e4afe5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.713+0000 7f9a12ffd700 1 -- 172.21.15.44:0/291930374 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c0c9ec0 con 0x5584e55686b0 2023-11-16T16:55:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.713+0000 7f9a12ffd700 1 -- 172.21.15.44:0/291930374 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a1c0652c0 con 0x5584e55686b0 2023-11-16T16:55:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.714+0000 7f9a12ffd700 1 -- 172.21.15.44:0/291930374 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c097980 con 0x5584e55686b0 2023-11-16T16:55:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.714+0000 7f9a46065b80 1 -- 172.21.15.44:0/291930374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e55686b0 msgr2=0x5584e571d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.714+0000 7f9a46065b80 1 --2- 172.21.15.44:0/291930374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e55686b0 0x5584e571d020 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0a14f0 tx=0x7f9a1c0842d0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.714+0000 7f9a46065b80 1 -- 172.21.15.44:0/291930374 shutdown_connections 2023-11-16T16:55:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.714+0000 7f9a46065b80 1 --2- 172.21.15.44:0/291930374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712b90 0x5584e5715460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.714+0000 7f9a46065b80 1 --2- 172.21.15.44:0/291930374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e55686b0 0x5584e571d020 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.714+0000 7f9a46065b80 1 --2- 172.21.15.44:0/291930374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e5712720 0x5584e57185c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.714+0000 7f9a46065b80 1 -- 172.21.15.44:0/291930374 >> 172.21.15.44:0/291930374 conn(0x5584e5701f50 msgr2=0x5584e570fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.714+0000 7f9a46065b80 1 -- 172.21.15.44:0/291930374 shutdown_connections 2023-11-16T16:55:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.715+0000 7f9a46065b80 1 -- 172.21.15.44:0/291930374 wait complete. 2023-11-16T16:55:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.715+0000 7f9a46065b80 1 -- start start 2023-11-16T16:55:47.718 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.715+0000 7f9a46065b80 1 Processor -- start 2023-11-16T16:55:47.719 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.716+0000 7f9a46065b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e55686b0 0x5584e570c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.719 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.716+0000 7f9a46065b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712720 0x5584e570cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.719 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.716+0000 7f9a46065b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e5712b90 0x5584e570b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.719 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.716+0000 7f9a46065b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e570d230 con 0x5584e55686b0 2023-11-16T16:55:47.719 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.716+0000 7f9a46065b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e570ba90 con 0x5584e5712720 2023-11-16T16:55:47.719 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.716+0000 7f9a46065b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e570bc50 con 0x5584e5712b90 2023-11-16T16:55:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.716+0000 7f9a46065b80 10 client.0 _ll_register_callbacks cb 0x5584e561d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.716+0000 7f9a2b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712720 0x5584e570cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.716+0000 7f9a2bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e55686b0 0x5584e570c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.716+0000 7f9a2b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712720 0x5584e570cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:53788/0 (socket says 172.21.15.44:53788) 2023-11-16T16:55:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.716+0000 7f9a2b7fe700 1 -- 172.21.15.44:0/1179357723 learned_addr learned my addr 172.21.15.44:0/1179357723 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:55:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.716+0000 7f9a30d9c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e5712b90 0x5584e570b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.716+0000 7f9a2b7fe700 1 -- 172.21.15.44:0/1179357723 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e5712b90 msgr2=0x5584e570b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.717+0000 7f9a2b7fe700 1 --2- 172.21.15.44:0/1179357723 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e5712b90 0x5584e570b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.717+0000 7f9a2b7fe700 1 -- 172.21.15.44:0/1179357723 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e55686b0 msgr2=0x5584e570c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.717+0000 7f9a2b7fe700 1 --2- 172.21.15.44:0/1179357723 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e55686b0 0x5584e570c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.717+0000 7f9a2b7fe700 1 -- 172.21.15.44:0/1179357723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5584e5624130 con 0x5584e5712720 2023-11-16T16:55:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.717+0000 7f9a30d9c700 1 --2- 172.21.15.44:0/1179357723 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e5712b90 0x5584e570b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:55:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.717+0000 7f9a2b7fe700 1 --2- 172.21.15.44:0/1179357723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712720 0x5584e570cb80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c06a3c0 tx=0x7f9a24004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4f40aebd0250516 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.717+0000 7f9a0b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:55:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.717+0000 7f9a0b7fe700 1 -- 172.21.15.44:0/1179357723 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a24023240 con 0x5584e5712720 2023-11-16T16:55:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.717+0000 7f9a0b7fe700 1 -- 172.21.15.44:0/1179357723 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f9a240233a0 con 0x5584e5712720 2023-11-16T16:55:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.717+0000 7f9a46065b80 1 -- 172.21.15.44:0/1179357723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9a1c0a1930 con 0x5584e5712720 2023-11-16T16:55:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.717+0000 7f9a0b7fe700 1 -- 172.21.15.44:0/1179357723 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a24023560 con 0x5584e5712720 2023-11-16T16:55:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.717+0000 7f9a46065b80 10 client.26687 fetch_fsmap learned FSMap version 21 2023-11-16T16:55:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.717+0000 7f9a46065b80 1 -- 172.21.15.44:0/1179357723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5584e5474c00 con 0x5584e5712720 2023-11-16T16:55:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.718+0000 7f9a0b7fe700 1 -- 172.21.15.44:0/1179357723 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a1c0a1930 con 0x5584e5712720 2023-11-16T16:55:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.718+0000 7f9a0b7fe700 1 -- 172.21.15.44:0/1179357723 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9a2402c470 con 0x5584e5712720 2023-11-16T16:55:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.718+0000 7f9a46065b80 10 client.26687 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:55:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.718+0000 7f9a46065b80 10 client.26687 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.719+0000 7f9a46065b80 20 client.26687 populate_metadata read hostname 'smithi044' 2023-11-16T16:55:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.719+0000 7f9a46065b80 1 --2- 172.21.15.44:0/1179357723 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5584e577a610 0x5584e577ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.719+0000 7f9a46065b80 4 client.26687 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:55:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.719+0000 7f9a46065b80 1 -- 172.21.15.44:0/1179357723 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5584e562c600 con 0x5584e577a610 2023-11-16T16:55:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.719+0000 7f9a2bfff700 1 --2- 172.21.15.44:0/1179357723 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5584e577a610 0x5584e577ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.719+0000 7f9a2bfff700 1 --2- 172.21.15.44:0/1179357723 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5584e577a610 0x5584e577ca20 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=777647180e9397b8 server_cookie=1864694ceee931ee in_seq=0 out_seq=0 2023-11-16T16:55:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:47.719+0000 7f9a0b7fe700 10 client.26687 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:47 smithi044 ceph-mon[128750]: pgmap v178: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 827 op/s 2023-11-16T16:55:47.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:47 smithi176 ceph-mon[131630]: pgmap v178: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 827 op/s 2023-11-16T16:55:47.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:47 smithi077 ceph-mon[131204]: pgmap v178: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 827 op/s 2023-11-16T16:55:49.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:49 smithi044 ceph-mon[128750]: pgmap v179: 129 pgs: 129 active+clean; 55 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 1.20k op/s 2023-11-16T16:55:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:49 smithi176 ceph-mon[131630]: pgmap v179: 129 pgs: 129 active+clean; 55 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 1.20k op/s 2023-11-16T16:55:49.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:49 smithi077 ceph-mon[131204]: pgmap v179: 129 pgs: 129 active+clean; 55 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 1.20k op/s 2023-11-16T16:55:51.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:51 smithi044 ceph-mon[128750]: pgmap v180: 129 pgs: 129 active+clean; 55 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 647 op/s 2023-11-16T16:55:51.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:51 smithi176 ceph-mon[131630]: pgmap v180: 129 pgs: 129 active+clean; 55 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 647 op/s 2023-11-16T16:55:51.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:51 smithi077 ceph-mon[131204]: pgmap v180: 129 pgs: 129 active+clean; 55 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 647 op/s 2023-11-16T16:55:53.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:53 smithi044 ceph-mon[128750]: pgmap v181: 129 pgs: 129 active+clean; 59 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 690 op/s 2023-11-16T16:55:53.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:53 smithi176 ceph-mon[131630]: pgmap v181: 129 pgs: 129 active+clean; 59 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 690 op/s 2023-11-16T16:55:53.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:53 smithi077 ceph-mon[131204]: pgmap v181: 129 pgs: 129 active+clean; 59 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 690 op/s 2023-11-16T16:55:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:55 smithi044 ceph-mon[128750]: pgmap v182: 129 pgs: 129 active+clean; 67 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 869 op/s 2023-11-16T16:55:55.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:55 smithi176 ceph-mon[131630]: pgmap v182: 129 pgs: 129 active+clean; 67 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 869 op/s 2023-11-16T16:55:55.989 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:55 smithi077 ceph-mon[131204]: pgmap v182: 129 pgs: 129 active+clean; 67 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 869 op/s 2023-11-16T16:55:57.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:57 smithi044 ceph-mon[128750]: pgmap v183: 129 pgs: 129 active+clean; 67 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 641 op/s 2023-11-16T16:55:57.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:57 smithi176 ceph-mon[131630]: pgmap v183: 129 pgs: 129 active+clean; 67 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 641 op/s 2023-11-16T16:55:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.935+0000 7f9a0b7fe700 1 -- 172.21.15.44:0/1179357723 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5584e562c600 con 0x5584e577a610 2023-11-16T16:55:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.935+0000 7f9a0b7fe700 10 client.26687 handle_command_reply: tid=0 2023-11-16T16:55:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.935+0000 7f9a46065b80 1 client.26687 shutdown 2023-11-16T16:55:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.935+0000 7f9a46065b80 1 -- 172.21.15.44:0/1179357723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712720 msgr2=0x5584e570cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.935+0000 7f9a46065b80 1 --2- 172.21.15.44:0/1179357723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712720 0x5584e570cb80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c06a3c0 tx=0x7f9a24004ad0 comp rx=0 tx=0).stop 2023-11-16T16:55:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.935+0000 7f9a46065b80 1 -- 172.21.15.44:0/1179357723 shutdown_connections 2023-11-16T16:55:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.935+0000 7f9a46065b80 1 --2- 172.21.15.44:0/1179357723 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5584e577a610 0x5584e577ca20 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.935+0000 7f9a46065b80 1 --2- 172.21.15.44:0/1179357723 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5584e5712b90 0x5584e570b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.935+0000 7f9a46065b80 1 --2- 172.21.15.44:0/1179357723 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584e55686b0 0x5584e570c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.935+0000 7f9a0b7fe700 0 client.26687 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:55:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.935+0000 7f9a46065b80 1 --2- 172.21.15.44:0/1179357723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584e5712720 0x5584e570cb80 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.935+0000 7f9a46065b80 1 -- 172.21.15.44:0/1179357723 >> 172.21.15.44:0/1179357723 conn(0x5584e5701f50 msgr2=0x5584e5712f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.935+0000 7f9a46065b80 1 -- 172.21.15.44:0/1179357723 shutdown_connections 2023-11-16T16:55:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.935+0000 7f9a46065b80 1 -- 172.21.15.44:0/1179357723 wait complete. 2023-11-16T16:55:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.936+0000 7f9a46065b80 20 client.26687 trim_cache size 0 max 16384 2023-11-16T16:55:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:55:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:55:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:55:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:55:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.936+0000 7f9a327ff700 1 -- 172.21.15.44:0/1890098656 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9a140a1450 msgr2=0x7f9a140a3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.936+0000 7f9a327ff700 1 --2- 172.21.15.44:0/1890098656 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9a140a1450 0x7f9a140a3900 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9a24009710 tx=0x7f9a24009620 comp rx=0 tx=0).stop 2023-11-16T16:55:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.936+0000 7f9a327ff700 1 -- 172.21.15.44:0/1890098656 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a2c0ff3c0 msgr2=0x7f9a2c10be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.936+0000 7f9a327ff700 1 --2- 172.21.15.44:0/1890098656 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a2c0ff3c0 0x7f9a2c10be20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00d980 tx=0x7f9a1c00dd40 comp rx=0 tx=0).stop 2023-11-16T16:55:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.936+0000 7f9a327ff700 1 -- 172.21.15.44:0/1890098656 shutdown_connections 2023-11-16T16:55:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.936+0000 7f9a327ff700 1 --2- 172.21.15.44:0/1890098656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a2c104400 0x7f9a2c109e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.936+0000 7f9a327ff700 1 --2- 172.21.15.44:0/1890098656 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f9a140a1450 0x7f9a140a3900 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:57.941 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.936+0000 7f9a327ff700 1 --2- 172.21.15.44:0/1890098656 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a2c0ff3c0 0x7f9a2c10be20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:57.941 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.936+0000 7f9a327ff700 1 --2- 172.21.15.44:0/1890098656 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a2c0ffd60 0x7f9a2c119990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:57.941 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.936+0000 7f9a327ff700 1 -- 172.21.15.44:0/1890098656 >> 172.21.15.44:0/1890098656 conn(0x7f9a2c0763a0 msgr2=0x7f9a2c10ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:57.941 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.937+0000 7f9a327ff700 1 -- 172.21.15.44:0/1890098656 shutdown_connections 2023-11-16T16:55:57.941 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:55:57.937+0000 7f9a327ff700 1 -- 172.21.15.44:0/1890098656 wait complete. 2023-11-16T16:55:57.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:57 smithi077 ceph-mon[131204]: pgmap v183: 129 pgs: 129 active+clean; 67 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 641 op/s 2023-11-16T16:55:59.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:55:59 smithi044 ceph-mon[128750]: pgmap v184: 129 pgs: 129 active+clean; 77 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 1.61k op/s 2023-11-16T16:55:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:55:59 smithi176 ceph-mon[131630]: pgmap v184: 129 pgs: 129 active+clean; 77 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 1.61k op/s 2023-11-16T16:55:59.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:55:59 smithi077 ceph-mon[131204]: pgmap v184: 129 pgs: 129 active+clean; 77 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 1.61k op/s 2023-11-16T16:56:01.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:01 smithi044 ceph-mon[128750]: pgmap v185: 129 pgs: 129 active+clean; 77 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 1.24k op/s 2023-11-16T16:56:01.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:01 smithi176 ceph-mon[131630]: pgmap v185: 129 pgs: 129 active+clean; 77 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 1.24k op/s 2023-11-16T16:56:01.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:01 smithi077 ceph-mon[131204]: pgmap v185: 129 pgs: 129 active+clean; 77 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 1.24k op/s 2023-11-16T16:56:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:03 smithi044 ceph-mon[128750]: pgmap v186: 129 pgs: 129 active+clean; 65 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 1.31k op/s 2023-11-16T16:56:03.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:03 smithi176 ceph-mon[131630]: pgmap v186: 129 pgs: 129 active+clean; 65 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 1.31k op/s 2023-11-16T16:56:03.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:03 smithi077 ceph-mon[131204]: pgmap v186: 129 pgs: 129 active+clean; 65 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 1.31k op/s 2023-11-16T16:56:05.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:05 smithi044 ceph-mon[128750]: pgmap v187: 129 pgs: 129 active+clean; 24 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 1.65k op/s 2023-11-16T16:56:05.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:05 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:56:05.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:05 smithi176 ceph-mon[131630]: pgmap v187: 129 pgs: 129 active+clean; 24 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 1.65k op/s 2023-11-16T16:56:05.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:05 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:56:05.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:05 smithi077 ceph-mon[131204]: pgmap v187: 129 pgs: 129 active+clean; 24 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 1.65k op/s 2023-11-16T16:56:05.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:05 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:56:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:07 smithi044 ceph-mon[128750]: pgmap v188: 129 pgs: 129 active+clean; 24 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 1.43k op/s 2023-11-16T16:56:07.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:07 smithi176 ceph-mon[131630]: pgmap v188: 129 pgs: 129 active+clean; 24 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 1.43k op/s 2023-11-16T16:56:07.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:07 smithi077 ceph-mon[131204]: pgmap v188: 129 pgs: 129 active+clean; 24 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 1.43k op/s 2023-11-16T16:56:08.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:08 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:56:08.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:08 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:56:08.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:08 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:56:09.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:09 smithi044 ceph-mon[128750]: pgmap v189: 129 pgs: 129 active+clean; 28 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 1.94k op/s 2023-11-16T16:56:09.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:09 smithi176 ceph-mon[131630]: pgmap v189: 129 pgs: 129 active+clean; 28 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 1.94k op/s 2023-11-16T16:56:09.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:09 smithi077 ceph-mon[131204]: pgmap v189: 129 pgs: 129 active+clean; 28 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 1.94k op/s 2023-11-16T16:56:10.880 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:56:11.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.261+0000 7fa10ebff700 1 -- 172.21.15.44:0/3911561869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1081015f0 msgr2=0x7fa108104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:11.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.261+0000 7fa10ebff700 1 --2- 172.21.15.44:0/3911561869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1081015f0 0x7fa108104e80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa100009f90 tx=0x7fa10000b330 comp rx=0 tx=0).stop 2023-11-16T16:56:11.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.261+0000 7fa10ebff700 1 -- 172.21.15.44:0/3911561869 shutdown_connections 2023-11-16T16:56:11.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.261+0000 7fa10ebff700 1 --2- 172.21.15.44:0/3911561869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa108105560 0x7fa10810d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.261+0000 7fa10ebff700 1 --2- 172.21.15.44:0/3911561869 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa108100c50 0x7fa108101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.261+0000 7fa10ebff700 1 --2- 172.21.15.44:0/3911561869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1081015f0 0x7fa108104e80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.261+0000 7fa10ebff700 1 -- 172.21.15.44:0/3911561869 >> 172.21.15.44:0/3911561869 conn(0x7fa108077550 msgr2=0x7fa108077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:11.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.262+0000 7fa10ebff700 1 -- 172.21.15.44:0/3911561869 shutdown_connections 2023-11-16T16:56:11.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.262+0000 7fa10ebff700 1 -- 172.21.15.44:0/3911561869 wait complete. 2023-11-16T16:56:11.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.263+0000 7fa10ebff700 1 Processor -- start 2023-11-16T16:56:11.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.264+0000 7fa10ebff700 1 -- start start 2023-11-16T16:56:11.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.265+0000 7fa10ebff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa108100c50 0x7fa10810bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.265+0000 7fa10ebff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1081015f0 0x7fa10810c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.265+0000 7fa10ebff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa108105560 0x7fa1081b1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.265+0000 7fa10ebff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1081b15d0 con 0x7fa108105560 2023-11-16T16:56:11.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.265+0000 7fa10ebff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1081b1710 con 0x7fa1081015f0 2023-11-16T16:56:11.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.265+0000 7fa10ebff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1081b1850 con 0x7fa108100c50 2023-11-16T16:56:11.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.266+0000 7fa107fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1081015f0 0x7fa10810c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.266+0000 7fa107fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1081015f0 0x7fa10810c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:55546/0 (socket says 172.21.15.44:55546) 2023-11-16T16:56:11.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.266+0000 7fa107fff700 1 -- 172.21.15.44:0/476454041 learned_addr learned my addr 172.21.15.44:0/476454041 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:11.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.267+0000 7fa10d19c700 1 --2- 172.21.15.44:0/476454041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa108105560 0x7fa1081b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.269 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.267+0000 7fa107fff700 1 -- 172.21.15.44:0/476454041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa108100c50 msgr2=0x7fa10810bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:11.269 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.268+0000 7fa107fff700 1 --2- 172.21.15.44:0/476454041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa108100c50 0x7fa10810bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.269 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.268+0000 7fa107fff700 1 -- 172.21.15.44:0/476454041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa108105560 msgr2=0x7fa1081b1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:11.269 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.268+0000 7fa107fff700 1 --2- 172.21.15.44:0/476454041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa108105560 0x7fa1081b1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.268+0000 7fa107fff700 1 -- 172.21.15.44:0/476454041 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa100009c40 con 0x7fa1081015f0 2023-11-16T16:56:11.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.268+0000 7fa10d19c700 1 --2- 172.21.15.44:0/476454041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa108105560 0x7fa1081b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:56:11.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.268+0000 7fa107fff700 1 --2- 172.21.15.44:0/476454041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1081015f0 0x7fa10810c4e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa10000cfa0 tx=0x7fa10000cfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:11.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.269+0000 7fa105ffb700 1 -- 172.21.15.44:0/476454041 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10000e050 con 0x7fa1081015f0 2023-11-16T16:56:11.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.269+0000 7fa105ffb700 1 -- 172.21.15.44:0/476454041 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa100005570 con 0x7fa1081015f0 2023-11-16T16:56:11.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.269+0000 7fa105ffb700 1 -- 172.21.15.44:0/476454041 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10001bab0 con 0x7fa1081015f0 2023-11-16T16:56:11.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.270+0000 7fa10ebff700 1 -- 172.21.15.44:0/476454041 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa108110370 con 0x7fa1081015f0 2023-11-16T16:56:11.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.270+0000 7fa10ebff700 1 -- 172.21.15.44:0/476454041 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1081101e0 con 0x7fa1081015f0 2023-11-16T16:56:11.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.276+0000 7fa122604b80 1 Processor -- start 2023-11-16T16:56:11.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.277+0000 7fa122604b80 1 -- start start 2023-11-16T16:56:11.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.277+0000 7fa105ffb700 1 -- 172.21.15.44:0/476454041 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fa1000056e0 con 0x7fa1081015f0 2023-11-16T16:56:11.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.277+0000 7fa122604b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b59956b0 0x55f0b5b3f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.277+0000 7fa122604b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 0x55f0b5b39560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.277+0000 7fa122604b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b39b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.277+0000 7fa122604b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0b5a3f000 con 0x55f0b5b3f720 2023-11-16T16:56:11.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.277+0000 7fa122604b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0b5a91840 con 0x55f0b5b3fb90 2023-11-16T16:56:11.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.277+0000 7fa122604b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0b5a8acd0 con 0x55f0b59956b0 2023-11-16T16:56:11.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.278+0000 7fa107fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b39b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.278+0000 7fa10d19c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b59956b0 0x55f0b5b3f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.278+0000 7fa107fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b39b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:55558/0 (socket says 172.21.15.44:55558) 2023-11-16T16:56:11.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.278+0000 7fa107fff700 1 -- 172.21.15.44:0/1463509996 learned_addr learned my addr 172.21.15.44:0/1463509996 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:11.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.278+0000 7fa107fff700 1 -- 172.21.15.44:0/1463509996 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b59956b0 msgr2=0x55f0b5b3f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:11.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.278+0000 7fa107fff700 1 --2- 172.21.15.44:0/1463509996 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b59956b0 0x55f0b5b3f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.278+0000 7fa10c99b700 1 --2- 172.21.15.44:0/1463509996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 0x55f0b5b39560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.278+0000 7fa107fff700 1 -- 172.21.15.44:0/1463509996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 msgr2=0x55f0b5b39560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:11.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.279+0000 7fa107fff700 1 --2- 172.21.15.44:0/1463509996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 0x55f0b5b39560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.279+0000 7fa107fff700 1 -- 172.21.15.44:0/1463509996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0b590c5d0 con 0x55f0b5b3fb90 2023-11-16T16:56:11.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.279+0000 7fa10c99b700 1 --2- 172.21.15.44:0/1463509996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 0x55f0b5b39560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:56:11.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.280+0000 7fa107fff700 1 --2- 172.21.15.44:0/1463509996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b39b00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa1000cbdf0 tx=0x7fa1000d2ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb5d2785f42a5fd3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:11.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.280+0000 7fa0eeffd700 1 -- 172.21.15.44:0/1463509996 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1000d7600 con 0x55f0b5b3fb90 2023-11-16T16:56:11.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.280+0000 7fa0eeffd700 1 -- 172.21.15.44:0/1463509996 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa1000d2750 con 0x55f0b5b3fb90 2023-11-16T16:56:11.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.280+0000 7fa105ffb700 1 --2- 172.21.15.44:0/476454041 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa0f00a1340 0x7fa0f00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.280+0000 7fa0eeffd700 1 -- 172.21.15.44:0/1463509996 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1000dbc20 con 0x55f0b5b3fb90 2023-11-16T16:56:11.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.281+0000 7fa10d19c700 1 --2- 172.21.15.44:0/476454041 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa0f00a1340 0x7fa0f00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.282+0000 7fa122604b80 1 -- 172.21.15.44:0/1463509996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b5b3fb90 msgr2=0x55f0b5b39b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:11.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.282+0000 7fa122604b80 1 --2- 172.21.15.44:0/1463509996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b39b00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa1000cbdf0 tx=0x7fa1000d2ae0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.282+0000 7fa105ffb700 1 -- 172.21.15.44:0/476454041 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fa1000ceb40 con 0x7fa1081015f0 2023-11-16T16:56:11.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.282+0000 7fa10d19c700 1 --2- 172.21.15.44:0/476454041 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa0f00a1340 0x7fa0f00a37f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc00ddd0 tx=0x7fa0fc00d540 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:11.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.282+0000 7fa122604b80 1 -- 172.21.15.44:0/1463509996 shutdown_connections 2023-11-16T16:56:11.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.282+0000 7fa122604b80 1 --2- 172.21.15.44:0/1463509996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b39b00 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.282+0000 7fa122604b80 1 --2- 172.21.15.44:0/1463509996 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b59956b0 0x55f0b5b3f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.282+0000 7fa122604b80 1 --2- 172.21.15.44:0/1463509996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 0x55f0b5b39560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.282+0000 7fa122604b80 1 -- 172.21.15.44:0/1463509996 >> 172.21.15.44:0/1463509996 conn(0x55f0b5b2ef50 msgr2=0x55f0b5b34270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:11.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.283+0000 7fa122604b80 1 -- 172.21.15.44:0/1463509996 shutdown_connections 2023-11-16T16:56:11.285 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.283+0000 7fa122604b80 1 -- 172.21.15.44:0/1463509996 wait complete. 2023-11-16T16:56:11.285 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.284+0000 7fa122604b80 1 -- start start 2023-11-16T16:56:11.285 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.284+0000 7fa122604b80 1 Processor -- start 2023-11-16T16:56:11.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.285+0000 7fa122604b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b59956b0 0x55f0b5b488f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.285+0000 7fa122604b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 0x55f0b5b48e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.285+0000 7fa122604b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b42b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.285+0000 7fa122604b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0b5a2c880 con 0x55f0b5b3f720 2023-11-16T16:56:11.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.285+0000 7fa122604b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0b5a2c520 con 0x55f0b59956b0 2023-11-16T16:56:11.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.285+0000 7fa122604b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0b590e8c0 con 0x55f0b5b3fb90 2023-11-16T16:56:11.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.285+0000 7fa122604b80 10 client.0 _ll_register_callbacks cb 0x55f0b5a4a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:11.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.285+0000 7fa10c99b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b59956b0 0x55f0b5b488f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.285+0000 7fa10c99b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b59956b0 0x55f0b5b488f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:55560/0 (socket says 172.21.15.44:55560) 2023-11-16T16:56:11.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.285+0000 7fa10c99b700 1 -- 172.21.15.44:0/1185456392 learned_addr learned my addr 172.21.15.44:0/1185456392 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:11.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.285+0000 7fa10d19c700 1 --2- 172.21.15.44:0/1185456392 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b42b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.286+0000 7fa107fff700 1 --2- 172.21.15.44:0/1185456392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 0x55f0b5b48e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.286+0000 7fa10c99b700 1 -- 172.21.15.44:0/1185456392 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b5b3fb90 msgr2=0x55f0b5b42b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:11.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.286+0000 7fa10c99b700 1 --2- 172.21.15.44:0/1185456392 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b42b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.286+0000 7fa10c99b700 1 -- 172.21.15.44:0/1185456392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 msgr2=0x55f0b5b48e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:11.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.286+0000 7fa10c99b700 1 --2- 172.21.15.44:0/1185456392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 0x55f0b5b48e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.289 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.286+0000 7fa10c99b700 1 -- 172.21.15.44:0/1185456392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1081101e0 con 0x55f0b59956b0 2023-11-16T16:56:11.289 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.286+0000 7fa107fff700 1 --2- 172.21.15.44:0/1185456392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 0x55f0b5b48e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:56:11.289 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.286+0000 7fa10c99b700 1 --2- 172.21.15.44:0/1185456392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b59956b0 0x55f0b5b488f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80105f0 tx=0x7fa0f8010900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba516c1585de608 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:11.289 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.286+0000 7fa0e7fff700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:56:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.286+0000 7fa0e7fff700 1 -- 172.21.15.44:0/1185456392 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8004a30 con 0x55f0b59956b0 2023-11-16T16:56:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.286+0000 7fa0e7fff700 1 -- 172.21.15.44:0/1185456392 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fa0f8016070 con 0x55f0b59956b0 2023-11-16T16:56:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.287+0000 7fa122604b80 1 -- 172.21.15.44:0/1185456392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f0b5a4d040 con 0x55f0b59956b0 2023-11-16T16:56:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.287+0000 7fa0e7fff700 1 -- 172.21.15.44:0/1185456392 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0f8004b90 con 0x55f0b59956b0 2023-11-16T16:56:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.287+0000 7fa122604b80 10 client.26699 fetch_fsmap learned FSMap version 21 2023-11-16T16:56:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.287+0000 7fa122604b80 1 -- 172.21.15.44:0/1185456392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55f0b5a51130 con 0x55f0b59956b0 2023-11-16T16:56:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.287+0000 7fa0e7fff700 1 -- 172.21.15.44:0/1185456392 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f0b5a4d040 con 0x55f0b59956b0 2023-11-16T16:56:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.287+0000 7fa0e7fff700 1 -- 172.21.15.44:0/1185456392 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa0f8005080 con 0x55f0b59956b0 2023-11-16T16:56:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.287+0000 7fa122604b80 10 client.26699 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:56:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.287+0000 7fa122604b80 10 client.26699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.287+0000 7fa122604b80 20 client.26699 populate_metadata read hostname 'smithi044' 2023-11-16T16:56:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.287+0000 7fa122604b80 1 --2- 172.21.15.44:0/1185456392 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f0b5b52ca0 0x55f0b5b550b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.288+0000 7fa122604b80 4 client.26699 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:56:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.288+0000 7fa122604b80 1 -- 172.21.15.44:0/1185456392 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f0b5a59600 con 0x55f0b5b52ca0 2023-11-16T16:56:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.288+0000 7fa107fff700 1 --2- 172.21.15.44:0/1185456392 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f0b5b52ca0 0x55f0b5b550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.288+0000 7fa107fff700 1 --2- 172.21.15.44:0/1185456392 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f0b5b52ca0 0x55f0b5b550b0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f68e98413d25567a server_cookie=4cb22427d40040d5 in_seq=0 out_seq=0 2023-11-16T16:56:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.288+0000 7fa0e7fff700 10 client.26699 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.292+0000 7fa0e7fff700 1 -- 172.21.15.44:0/1185456392 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f0b5a59600 con 0x55f0b5b52ca0 2023-11-16T16:56:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.292+0000 7fa0e7fff700 10 client.26699 handle_command_reply: tid=0 2023-11-16T16:56:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.292+0000 7fa122604b80 1 client.26699 shutdown 2023-11-16T16:56:11.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.292+0000 7fa122604b80 1 -- 172.21.15.44:0/1185456392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b59956b0 msgr2=0x55f0b5b488f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:11.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.292+0000 7fa122604b80 1 --2- 172.21.15.44:0/1185456392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b59956b0 0x55f0b5b488f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80105f0 tx=0x7fa0f8010900 comp rx=0 tx=0).stop 2023-11-16T16:56:11.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.292+0000 7fa122604b80 1 -- 172.21.15.44:0/1185456392 shutdown_connections 2023-11-16T16:56:11.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.292+0000 7fa122604b80 1 --2- 172.21.15.44:0/1185456392 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f0b5b52ca0 0x55f0b5b550b0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.292+0000 7fa122604b80 1 --2- 172.21.15.44:0/1185456392 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b42b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.292+0000 7fa122604b80 1 --2- 172.21.15.44:0/1185456392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b59956b0 0x55f0b5b488f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.292+0000 7fa122604b80 1 --2- 172.21.15.44:0/1185456392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 0x55f0b5b48e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.292+0000 7fa122604b80 1 -- 172.21.15.44:0/1185456392 >> 172.21.15.44:0/1185456392 conn(0x55f0b5b2ef50 msgr2=0x55f0b5b337e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:11.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.292+0000 7fa0e7fff700 0 client.26699 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:11.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.293+0000 7fa122604b80 1 -- 172.21.15.44:0/1185456392 shutdown_connections 2023-11-16T16:56:11.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.293+0000 7fa122604b80 1 -- 172.21.15.44:0/1185456392 wait complete. 2023-11-16T16:56:11.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.293+0000 7fa122604b80 20 client.26699 trim_cache size 0 max 16384 2023-11-16T16:56:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.311+0000 7fa122604b80 1 Processor -- start 2023-11-16T16:56:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.311+0000 7fa122604b80 1 -- start start 2023-11-16T16:56:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.312+0000 7fa122604b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b59956b0 0x55f0b5b406a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.312+0000 7fa122604b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b5b3f720 0x55f0b5b33f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.312+0000 7fa122604b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b39560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.312+0000 7fa122604b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0b5a59600 con 0x55f0b5b3fb90 2023-11-16T16:56:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.312+0000 7fa122604b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0b5a4d040 con 0x55f0b59956b0 2023-11-16T16:56:11.314 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.312+0000 7fa122604b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0b5b3a0d0 con 0x55f0b5b3f720 2023-11-16T16:56:11.314 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.312+0000 7fa10c99b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b59956b0 0x55f0b5b406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.312+0000 7fa10c99b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b59956b0 0x55f0b5b406a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:55574/0 (socket says 172.21.15.44:55574) 2023-11-16T16:56:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.312+0000 7fa10c99b700 1 -- 172.21.15.44:0/1007746204 learned_addr learned my addr 172.21.15.44:0/1007746204 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.312+0000 7fa107fff700 1 --2- 172.21.15.44:0/1007746204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b5b3f720 0x55f0b5b33f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.312+0000 7fa10c99b700 1 -- 172.21.15.44:0/1007746204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b5b3f720 msgr2=0x55f0b5b33f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.312+0000 7fa10c99b700 1 --2- 172.21.15.44:0/1007746204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b5b3f720 0x55f0b5b33f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.312+0000 7fa10c99b700 1 -- 172.21.15.44:0/1007746204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3fb90 msgr2=0x55f0b5b39560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.312+0000 7fa10c99b700 1 --2- 172.21.15.44:0/1007746204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b39560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.312+0000 7fa10c99b700 1 -- 172.21.15.44:0/1007746204 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0b5a50e80 con 0x55f0b59956b0 2023-11-16T16:56:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.313+0000 7fa10c99b700 1 --2- 172.21.15.44:0/1007746204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b59956b0 0x55f0b5b406a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8001eb0 tx=0x7fa0f8009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7173b82b82b33bb1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.313+0000 7fa0ef7fe700 1 -- 172.21.15.44:0/1007746204 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f80057d0 con 0x55f0b59956b0 2023-11-16T16:56:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.313+0000 7fa0ef7fe700 1 -- 172.21.15.44:0/1007746204 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa0f8012b20 con 0x55f0b59956b0 2023-11-16T16:56:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.313+0000 7fa0ef7fe700 1 -- 172.21.15.44:0/1007746204 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f801acb0 con 0x55f0b59956b0 2023-11-16T16:56:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.314+0000 7fa122604b80 1 -- 172.21.15.44:0/1007746204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b59956b0 msgr2=0x55f0b5b406a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.314+0000 7fa122604b80 1 --2- 172.21.15.44:0/1007746204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b59956b0 0x55f0b5b406a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8001eb0 tx=0x7fa0f8009820 comp rx=0 tx=0).stop 2023-11-16T16:56:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.314+0000 7fa122604b80 1 -- 172.21.15.44:0/1007746204 shutdown_connections 2023-11-16T16:56:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.314+0000 7fa122604b80 1 --2- 172.21.15.44:0/1007746204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b39560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.314+0000 7fa122604b80 1 --2- 172.21.15.44:0/1007746204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b59956b0 0x55f0b5b406a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.314+0000 7fa122604b80 1 --2- 172.21.15.44:0/1007746204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b5b3f720 0x55f0b5b33f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.314+0000 7fa122604b80 1 -- 172.21.15.44:0/1007746204 >> 172.21.15.44:0/1007746204 conn(0x55f0b5b2ef50 msgr2=0x55f0b5b3ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.315+0000 7fa122604b80 1 -- 172.21.15.44:0/1007746204 shutdown_connections 2023-11-16T16:56:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.315+0000 7fa122604b80 1 -- 172.21.15.44:0/1007746204 wait complete. 2023-11-16T16:56:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.315+0000 7fa122604b80 1 -- start start 2023-11-16T16:56:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.316+0000 7fa122604b80 1 Processor -- start 2023-11-16T16:56:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.316+0000 7fa122604b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b59956b0 0x55f0b5b48af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.316+0000 7fa122604b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 0x55f0b5b47630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.316+0000 7fa122604b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b47b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.316+0000 7fa122604b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0b5b480f0 con 0x55f0b5b3f720 2023-11-16T16:56:11.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.316+0000 7fa122604b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0b5b48230 con 0x55f0b5b3fb90 2023-11-16T16:56:11.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.316+0000 7fa122604b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0b5b483f0 con 0x55f0b59956b0 2023-11-16T16:56:11.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.317+0000 7fa122604b80 10 client.0 _ll_register_callbacks cb 0x7fa0f8009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.317+0000 7fa10d19c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b47b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.317+0000 7fa10d19c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b47b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:55582/0 (socket says 172.21.15.44:55582) 2023-11-16T16:56:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.317+0000 7fa10d19c700 1 -- 172.21.15.44:0/3930242658 learned_addr learned my addr 172.21.15.44:0/3930242658 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.317+0000 7fa107fff700 1 --2- 172.21.15.44:0/3930242658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 0x55f0b5b47630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.317+0000 7fa10c99b700 1 --2- 172.21.15.44:0/3930242658 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b59956b0 0x55f0b5b48af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.317+0000 7fa10d19c700 1 -- 172.21.15.44:0/3930242658 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b59956b0 msgr2=0x55f0b5b48af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.317+0000 7fa10d19c700 1 --2- 172.21.15.44:0/3930242658 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b59956b0 0x55f0b5b48af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.317+0000 7fa10d19c700 1 -- 172.21.15.44:0/3930242658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 msgr2=0x55f0b5b47630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.317+0000 7fa10d19c700 1 --2- 172.21.15.44:0/3930242658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 0x55f0b5b47630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.317+0000 7fa10d19c700 1 -- 172.21.15.44:0/3930242658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f801f040 con 0x55f0b5b3fb90 2023-11-16T16:56:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.318+0000 7fa10d19c700 1 --2- 172.21.15.44:0/3930242658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b47b70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc012ac0 tx=0x7fa0fc004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cf7d04aef395884 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.318+0000 7fa0e7fff700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:56:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.318+0000 7fa0e7fff700 1 -- 172.21.15.44:0/3930242658 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc022040 con 0x55f0b5b3fb90 2023-11-16T16:56:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.318+0000 7fa122604b80 1 -- 172.21.15.44:0/3930242658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f0b5b2c710 con 0x55f0b5b3fb90 2023-11-16T16:56:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.319+0000 7fa0e7fff700 1 -- 172.21.15.44:0/3930242658 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7fa0fc01de20 con 0x55f0b5b3fb90 2023-11-16T16:56:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.319+0000 7fa0e7fff700 1 -- 172.21.15.44:0/3930242658 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0fc01bc00 con 0x55f0b5b3fb90 2023-11-16T16:56:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.319+0000 7fa122604b80 10 client.26705 fetch_fsmap learned FSMap version 21 2023-11-16T16:56:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.319+0000 7fa122604b80 1 -- 172.21.15.44:0/3930242658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55f0b58a1c00 con 0x55f0b5b3fb90 2023-11-16T16:56:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.319+0000 7fa0e7fff700 1 -- 172.21.15.44:0/3930242658 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f0b5b2c710 con 0x55f0b5b3fb90 2023-11-16T16:56:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.320+0000 7fa0e7fff700 1 -- 172.21.15.44:0/3930242658 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa0fc0283f0 con 0x55f0b5b3fb90 2023-11-16T16:56:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.320+0000 7fa122604b80 10 client.26705 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:56:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.320+0000 7fa122604b80 10 client.26705 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.320+0000 7fa122604b80 20 client.26705 populate_metadata read hostname 'smithi044' 2023-11-16T16:56:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.321+0000 7fa122604b80 1 --2- 172.21.15.44:0/3930242658 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f0b5ba7100 0x55f0b5ba9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.321+0000 7fa122604b80 4 client.26705 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:56:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.321+0000 7fa122604b80 1 -- 172.21.15.44:0/3930242658 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f0b58d9f10 con 0x55f0b5ba7100 2023-11-16T16:56:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.323+0000 7fa10c99b700 1 --2- 172.21.15.44:0/3930242658 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f0b5ba7100 0x55f0b5ba9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.323+0000 7fa10c99b700 1 --2- 172.21.15.44:0/3930242658 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f0b5ba7100 0x55f0b5ba9510 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f2a73903b68bae4 server_cookie=c736d0aaa2a2ed9e in_seq=0 out_seq=0 2023-11-16T16:56:11.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:11.323+0000 7fa0e7fff700 10 client.26705 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:11.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.338+0000 7f2b42bff700 1 -- 172.21.15.44:0/3970976837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b3c101390 msgr2=0x7f2b3c101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:11.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.338+0000 7f2b42bff700 1 --2- 172.21.15.44:0/3970976837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b3c101390 0x7f2b3c101760 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2b30009990 tx=0x7f2b30009ca0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.338+0000 7f2b42bff700 1 -- 172.21.15.44:0/3970976837 shutdown_connections 2023-11-16T16:56:11.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.338+0000 7f2b42bff700 1 --2- 172.21.15.44:0/3970976837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b3c10a8b0 0x7f2b3c10ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.338+0000 7f2b42bff700 1 --2- 172.21.15.44:0/3970976837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b3c101390 0x7f2b3c101760 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.338+0000 7f2b42bff700 1 --2- 172.21.15.44:0/3970976837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b3c101ca0 0x7f2b3c10a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.338+0000 7f2b42bff700 1 -- 172.21.15.44:0/3970976837 >> 172.21.15.44:0/3970976837 conn(0x7f2b3c0770b0 msgr2=0x7f2b3c0774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:11.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.338+0000 7f2b42bff700 1 -- 172.21.15.44:0/3970976837 shutdown_connections 2023-11-16T16:56:11.340 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.338+0000 7f2b42bff700 1 -- 172.21.15.44:0/3970976837 wait complete. 2023-11-16T16:56:11.341 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.339+0000 7f2b42bff700 1 Processor -- start 2023-11-16T16:56:11.341 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.340+0000 7f2b42bff700 1 -- start start 2023-11-16T16:56:11.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.340+0000 7f2b42bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b3c101390 0x7f2b3c19e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.340+0000 7f2b42bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b3c101ca0 0x7f2b3c19ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.340+0000 7f2b42bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b3c10a8b0 0x7f2b3c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.340+0000 7f2b42bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c198be0 con 0x7f2b3c10a8b0 2023-11-16T16:56:11.342 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.340+0000 7f2b42bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c198d50 con 0x7f2b3c101390 2023-11-16T16:56:11.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.340+0000 7f2b42bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c198f10 con 0x7f2b3c101ca0 2023-11-16T16:56:11.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.341+0000 7f2b3bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b3c101ca0 0x7f2b3c19ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.341+0000 7f2b3bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b3c101ca0 0x7f2b3c19ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:47222/0 (socket says 172.21.15.44:47222) 2023-11-16T16:56:11.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.341+0000 7f2b3bfff700 1 -- 172.21.15.44:0/2230678019 learned_addr learned my addr 172.21.15.44:0/2230678019 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:11.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.341+0000 7f2b4099b700 1 --2- 172.21.15.44:0/2230678019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b3c101390 0x7f2b3c19e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.343 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.341+0000 7f2b4119c700 1 --2- 172.21.15.44:0/2230678019 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b3c10a8b0 0x7f2b3c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.342+0000 7f2b3bfff700 1 -- 172.21.15.44:0/2230678019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b3c101390 msgr2=0x7f2b3c19e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:11.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.342+0000 7f2b3bfff700 1 --2- 172.21.15.44:0/2230678019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b3c101390 0x7f2b3c19e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.342+0000 7f2b3bfff700 1 -- 172.21.15.44:0/2230678019 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b3c10a8b0 msgr2=0x7f2b3c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:11.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.342+0000 7f2b3bfff700 1 --2- 172.21.15.44:0/2230678019 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b3c10a8b0 0x7f2b3c198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.344 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.342+0000 7f2b3bfff700 1 -- 172.21.15.44:0/2230678019 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b300095f0 con 0x7f2b3c101ca0 2023-11-16T16:56:11.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.342+0000 7f2b3bfff700 1 --2- 172.21.15.44:0/2230678019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b3c101ca0 0x7f2b3c19ea40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800c920 tx=0x7f2b2800cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:11.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.342+0000 7f2b39ffb700 1 -- 172.21.15.44:0/2230678019 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b28007b00 con 0x7f2b3c101ca0 2023-11-16T16:56:11.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.343+0000 7f2b42bff700 1 -- 172.21.15.44:0/2230678019 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3c110380 con 0x7f2b3c101ca0 2023-11-16T16:56:11.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.343+0000 7f2b42bff700 1 -- 172.21.15.44:0/2230678019 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3c1101f0 con 0x7f2b3c101ca0 2023-11-16T16:56:11.345 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.343+0000 7f2b39ffb700 1 -- 172.21.15.44:0/2230678019 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b2800ce70 con 0x7f2b3c101ca0 2023-11-16T16:56:11.346 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.343+0000 7f2b39ffb700 1 -- 172.21.15.44:0/2230678019 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b28005460 con 0x7f2b3c101ca0 2023-11-16T16:56:11.351 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.349+0000 7f2b39ffb700 1 -- 172.21.15.44:0/2230678019 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f2b28004830 con 0x7f2b3c101ca0 2023-11-16T16:56:11.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.351+0000 7f2b5644cb80 1 Processor -- start 2023-11-16T16:56:11.353 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.351+0000 7f2b39ffb700 1 --2- 172.21.15.44:0/2230678019 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2b2c0a13c0 0x7f2b2c0a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.351+0000 7f2b39ffb700 1 -- 172.21.15.44:0/2230678019 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f2b280ccb40 con 0x7f2b3c101ca0 2023-11-16T16:56:11.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.352+0000 7f2b5644cb80 1 -- start start 2023-11-16T16:56:11.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.352+0000 7f2b5644cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb802b60 0x5620cb802f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.352+0000 7f2b4119c700 1 --2- 172.21.15.44:0/2230678019 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2b2c0a13c0 0x7f2b2c0a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.352+0000 7f2b5644cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb803470 0x5620cb807370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.352+0000 7f2b5644cb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb8078b0 0x5620cb807d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.354 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.352+0000 7f2b5644cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620cb708060 con 0x5620cb803470 2023-11-16T16:56:11.355 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.352+0000 7f2b5644cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620cb75a8a0 con 0x5620cb802b60 2023-11-16T16:56:11.355 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.352+0000 7f2b5644cb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620cb753d30 con 0x5620cb8078b0 2023-11-16T16:56:11.355 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.353+0000 7f2b4099b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb802b60 0x5620cb802f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.355 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.353+0000 7f2b4099b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb802b60 0x5620cb802f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:55604/0 (socket says 172.21.15.44:55604) 2023-11-16T16:56:11.355 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.353+0000 7f2b4099b700 1 -- 172.21.15.44:0/4196993815 learned_addr learned my addr 172.21.15.44:0/4196993815 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:11.355 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.353+0000 7f2b3bfff700 1 --2- 172.21.15.44:0/4196993815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb803470 0x5620cb807370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.355 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.353+0000 7f2b4099b700 1 -- 172.21.15.44:0/4196993815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb8078b0 msgr2=0x5620cb807d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:11.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.353+0000 7f2b4099b700 1 --2- 172.21.15.44:0/4196993815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb8078b0 0x5620cb807d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.353+0000 7f2b4099b700 1 -- 172.21.15.44:0/4196993815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb803470 msgr2=0x5620cb807370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:11.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.353+0000 7f2b4099b700 1 --2- 172.21.15.44:0/4196993815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb803470 0x5620cb807370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.353+0000 7f2b4099b700 1 -- 172.21.15.44:0/4196993815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5620cb5d5630 con 0x5620cb802b60 2023-11-16T16:56:11.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.353+0000 7f2b3bfff700 1 --2- 172.21.15.44:0/4196993815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb803470 0x5620cb807370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:56:11.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.353+0000 7f2b4119c700 1 --2- 172.21.15.44:0/2230678019 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2b2c0a13c0 0x7f2b2c0a3870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c19f200 tx=0x7f2b34009310 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:11.356 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.354+0000 7f2b4099b700 1 --2- 172.21.15.44:0/4196993815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb802b60 0x5620cb802f30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2b30015000 tx=0x7f2b30009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=304a1e3062e0405a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:11.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.355+0000 7f2b26ffd700 1 -- 172.21.15.44:0/4196993815 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3001d050 con 0x5620cb802b60 2023-11-16T16:56:11.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.355+0000 7f2b26ffd700 1 -- 172.21.15.44:0/4196993815 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b3001e040 con 0x5620cb802b60 2023-11-16T16:56:11.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.355+0000 7f2b26ffd700 1 -- 172.21.15.44:0/4196993815 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b30005660 con 0x5620cb802b60 2023-11-16T16:56:11.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.355+0000 7f2b5644cb80 1 -- 172.21.15.44:0/4196993815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb802b60 msgr2=0x5620cb802f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:11.357 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.355+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/4196993815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb802b60 0x5620cb802f30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2b30015000 tx=0x7f2b30009f90 comp rx=0 tx=0).stop 2023-11-16T16:56:11.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.356+0000 7f2b5644cb80 1 -- 172.21.15.44:0/4196993815 shutdown_connections 2023-11-16T16:56:11.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.356+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/4196993815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb8078b0 0x5620cb807d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.356+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/4196993815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb802b60 0x5620cb802f30 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.356+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/4196993815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb803470 0x5620cb807370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.356+0000 7f2b5644cb80 1 -- 172.21.15.44:0/4196993815 >> 172.21.15.44:0/4196993815 conn(0x5620cb7f8360 msgr2=0x5620cb7fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:11.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.357+0000 7f2b5644cb80 1 -- 172.21.15.44:0/4196993815 shutdown_connections 2023-11-16T16:56:11.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.357+0000 7f2b5644cb80 1 -- 172.21.15.44:0/4196993815 wait complete. 2023-11-16T16:56:11.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.357+0000 7f2b5644cb80 1 -- start start 2023-11-16T16:56:11.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.358+0000 7f2b5644cb80 1 Processor -- start 2023-11-16T16:56:11.360 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.358+0000 7f2b5644cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb802b60 0x5620cb80eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b5644cb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb803470 0x5620cb80c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b5644cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb8078b0 0x5620cb80c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b5644cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620cb6f58e0 con 0x5620cb802b60 2023-11-16T16:56:11.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b5644cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620cb6f5580 con 0x5620cb8078b0 2023-11-16T16:56:11.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b5644cb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620cb5d7920 con 0x5620cb803470 2023-11-16T16:56:11.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b4099b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb802b60 0x5620cb80eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b5644cb80 10 client.0 _ll_register_callbacks cb 0x5620cb713780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:11.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b4099b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb802b60 0x5620cb80eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36118/0 (socket says 172.21.15.44:36118) 2023-11-16T16:56:11.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b4099b700 1 -- 172.21.15.44:0/4276440795 learned_addr learned my addr 172.21.15.44:0/4276440795 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:11.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b3bfff700 1 --2- 172.21.15.44:0/4276440795 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb803470 0x5620cb80c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.361 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b4119c700 1 --2- 172.21.15.44:0/4276440795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb8078b0 0x5620cb80c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b4099b700 1 -- 172.21.15.44:0/4276440795 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb803470 msgr2=0x5620cb80c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:11.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b4099b700 1 --2- 172.21.15.44:0/4276440795 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb803470 0x5620cb80c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b4099b700 1 -- 172.21.15.44:0/4276440795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb8078b0 msgr2=0x5620cb80c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:11.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b4099b700 1 --2- 172.21.15.44:0/4276440795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb8078b0 0x5620cb80c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:11.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b4099b700 1 -- 172.21.15.44:0/4276440795 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3c10d990 con 0x5620cb802b60 2023-11-16T16:56:11.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.359+0000 7f2b4119c700 1 --2- 172.21.15.44:0/4276440795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb8078b0 0x5620cb80c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:56:11.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.360+0000 7f2b4099b700 1 --2- 172.21.15.44:0/4276440795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb802b60 0x5620cb80eff0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f2b30009990 tx=0x7f2b30005450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1598a49b5a5d8c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:11.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.360+0000 7f2b1bfff700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:56:11.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.360+0000 7f2b1bfff700 1 -- 172.21.15.44:0/4276440795 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b30003a90 con 0x5620cb802b60 2023-11-16T16:56:11.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.360+0000 7f2b1bfff700 1 -- 172.21.15.44:0/4276440795 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9423+0+0 (secure 0 0 0) 0x7f2b30003bf0 con 0x5620cb802b60 2023-11-16T16:56:11.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.360+0000 7f2b5644cb80 1 -- 172.21.15.44:0/4276440795 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5620cb7f5b20 con 0x5620cb802b60 2023-11-16T16:56:11.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.361+0000 7f2b1bfff700 1 -- 172.21.15.44:0/4276440795 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b3001d020 con 0x5620cb802b60 2023-11-16T16:56:11.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.361+0000 7f2b5644cb80 10 client.16671 fetch_fsmap learned FSMap version 21 2023-11-16T16:56:11.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.361+0000 7f2b5644cb80 1 -- 172.21.15.44:0/4276440795 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5620cb71a190 con 0x5620cb802b60 2023-11-16T16:56:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.362+0000 7f2b1bfff700 1 -- 172.21.15.44:0/4276440795 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5620cb7f5b20 con 0x5620cb802b60 2023-11-16T16:56:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.362+0000 7f2b1bfff700 1 -- 172.21.15.44:0/4276440795 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2b30023460 con 0x5620cb802b60 2023-11-16T16:56:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.362+0000 7f2b5644cb80 10 client.16671 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:56:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.362+0000 7f2b5644cb80 10 client.16671 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.362+0000 7f2b5644cb80 20 client.16671 populate_metadata read hostname 'smithi044' 2023-11-16T16:56:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.362+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/4276440795 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5620cb81c3e0 0x5620cb81e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.362+0000 7f2b5644cb80 4 client.16671 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:56:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.362+0000 7f2b5644cb80 1 -- 172.21.15.44:0/4276440795 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5620cb7160a0 con 0x5620cb81c3e0 2023-11-16T16:56:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.363+0000 7f2b3bfff700 1 --2- 172.21.15.44:0/4276440795 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5620cb81c3e0 0x5620cb81e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:11.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.363+0000 7f2b3bfff700 1 --2- 172.21.15.44:0/4276440795 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5620cb81c3e0 0x5620cb81e7c0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce707e97bd4f8289 server_cookie=d2bd6d0cd992956a in_seq=0 out_seq=0 2023-11-16T16:56:11.366 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:11.363+0000 7f2b1bfff700 10 client.16671 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:11.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:11 smithi044 ceph-mon[128750]: pgmap v190: 129 pgs: 129 active+clean; 28 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 966 op/s 2023-11-16T16:56:11.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:11 smithi176 ceph-mon[131630]: pgmap v190: 129 pgs: 129 active+clean; 28 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 966 op/s 2023-11-16T16:56:11.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:11 smithi077 ceph-mon[131204]: pgmap v190: 129 pgs: 129 active+clean; 28 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 966 op/s 2023-11-16T16:56:13.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:13 smithi044 ceph-mon[128750]: pgmap v191: 129 pgs: 129 active+clean; 28 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 983 op/s 2023-11-16T16:56:13.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:13 smithi044 ceph-mon[128750]: osdmap e93: 12 total, 12 up, 12 in 2023-11-16T16:56:13.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:13 smithi176 ceph-mon[131630]: pgmap v191: 129 pgs: 129 active+clean; 28 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 983 op/s 2023-11-16T16:56:13.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:13 smithi176 ceph-mon[131630]: osdmap e93: 12 total, 12 up, 12 in 2023-11-16T16:56:13.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:13 smithi077 ceph-mon[131204]: pgmap v191: 129 pgs: 129 active+clean; 28 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 983 op/s 2023-11-16T16:56:13.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:13 smithi077 ceph-mon[131204]: osdmap e93: 12 total, 12 up, 12 in 2023-11-16T16:56:15.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:15 smithi044 ceph-mon[128750]: pgmap v193: 129 pgs: 129 active+clean; 44 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 782 op/s 2023-11-16T16:56:15.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:15 smithi176 ceph-mon[131630]: pgmap v193: 129 pgs: 129 active+clean; 44 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 782 op/s 2023-11-16T16:56:15.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:15 smithi077 ceph-mon[131204]: pgmap v193: 129 pgs: 129 active+clean; 44 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 782 op/s 2023-11-16T16:56:17.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:17 smithi044 ceph-mon[128750]: pgmap v194: 129 pgs: 129 active+clean; 44 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 782 op/s 2023-11-16T16:56:17.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:17 smithi176 ceph-mon[131630]: pgmap v194: 129 pgs: 129 active+clean; 44 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 782 op/s 2023-11-16T16:56:17.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:17 smithi077 ceph-mon[131204]: pgmap v194: 129 pgs: 129 active+clean; 44 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 782 op/s 2023-11-16T16:56:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:19 smithi044 ceph-mon[128750]: pgmap v195: 129 pgs: 129 active+clean; 58 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 568 op/s 2023-11-16T16:56:19.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:19 smithi176 ceph-mon[131630]: pgmap v195: 129 pgs: 129 active+clean; 58 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 568 op/s 2023-11-16T16:56:19.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:19 smithi077 ceph-mon[131204]: pgmap v195: 129 pgs: 129 active+clean; 58 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 568 op/s 2023-11-16T16:56:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.250+0000 7fa0e7fff700 1 -- 172.21.15.44:0/3930242658 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f0b58d9f10 con 0x55f0b5ba7100 2023-11-16T16:56:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.250+0000 7fa0e7fff700 10 client.26705 handle_command_reply: tid=0 2023-11-16T16:56:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.250+0000 7fa122604b80 1 client.26705 shutdown 2023-11-16T16:56:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.251+0000 7fa122604b80 1 -- 172.21.15.44:0/3930242658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b5b3fb90 msgr2=0x55f0b5b47b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.251+0000 7fa122604b80 1 --2- 172.21.15.44:0/3930242658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b47b70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc012ac0 tx=0x7fa0fc004750 comp rx=0 tx=0).stop 2023-11-16T16:56:21.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.251+0000 7f2b1bfff700 1 -- 172.21.15.44:0/4276440795 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5620cb7160a0 con 0x5620cb81c3e0 2023-11-16T16:56:21.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.251+0000 7f2b1bfff700 10 client.16671 handle_command_reply: tid=0 2023-11-16T16:56:21.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.251+0000 7f2b5644cb80 1 client.16671 shutdown 2023-11-16T16:56:21.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.251+0000 7f2b5644cb80 1 -- 172.21.15.44:0/4276440795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb802b60 msgr2=0x5620cb80eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:21.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.251+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/4276440795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb802b60 0x5620cb80eff0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f2b30009990 tx=0x7f2b30005450 comp rx=0 tx=0).stop 2023-11-16T16:56:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.252+0000 7fa122604b80 1 -- 172.21.15.44:0/3930242658 shutdown_connections 2023-11-16T16:56:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.252+0000 7fa122604b80 1 --2- 172.21.15.44:0/3930242658 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55f0b5ba7100 0x55f0b5ba9510 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.252+0000 7fa122604b80 1 --2- 172.21.15.44:0/3930242658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f0b5b3fb90 0x55f0b5b47b70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.252+0000 7fa122604b80 1 --2- 172.21.15.44:0/3930242658 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f0b59956b0 0x55f0b5b48af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.252+0000 7fa122604b80 1 --2- 172.21.15.44:0/3930242658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0b5b3f720 0x55f0b5b47630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.252+0000 7fa122604b80 1 -- 172.21.15.44:0/3930242658 >> 172.21.15.44:0/3930242658 conn(0x55f0b5b2ef50 msgr2=0x55f0b5b423b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:21.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.251+0000 7f2b5644cb80 1 -- 172.21.15.44:0/4276440795 shutdown_connections 2023-11-16T16:56:21.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.251+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/4276440795 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5620cb81c3e0 0x5620cb81e7c0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.251+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/4276440795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb8078b0 0x5620cb80c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.251+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/4276440795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb802b60 0x5620cb80eff0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.251+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/4276440795 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb803470 0x5620cb80c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.252+0000 7f2b5644cb80 1 -- 172.21.15.44:0/4276440795 >> 172.21.15.44:0/4276440795 conn(0x5620cb7f8360 msgr2=0x5620cb7fcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:21.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.252+0000 7f2b1bfff700 0 client.16671 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:21.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.252+0000 7fa0e7fff700 0 client.26705 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.252+0000 7fa122604b80 1 -- 172.21.15.44:0/3930242658 shutdown_connections 2023-11-16T16:56:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.252+0000 7fa122604b80 1 -- 172.21.15.44:0/3930242658 wait complete. 2023-11-16T16:56:21.256 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.252+0000 7f2b5644cb80 1 -- 172.21.15.44:0/4276440795 shutdown_connections 2023-11-16T16:56:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.253+0000 7fa122604b80 20 client.26705 trim_cache size 0 max 16384 2023-11-16T16:56:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:56:21.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:56:21.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:56:21.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:56:21.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.253+0000 7f2b5644cb80 1 -- 172.21.15.44:0/4276440795 wait complete. 2023-11-16T16:56:21.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.254+0000 7f2b5644cb80 20 client.16671 trim_cache size 0 max 16384 2023-11-16T16:56:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.254+0000 7fa10ebff700 1 -- 172.21.15.44:0/476454041 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa0f00a1340 msgr2=0x7fa0f00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.254+0000 7fa10ebff700 1 --2- 172.21.15.44:0/476454041 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa0f00a1340 0x7fa0f00a37f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc00ddd0 tx=0x7fa0fc00d540 comp rx=0 tx=0).stop 2023-11-16T16:56:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.254+0000 7fa10ebff700 1 -- 172.21.15.44:0/476454041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1081015f0 msgr2=0x7fa10810c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.254+0000 7fa10ebff700 1 --2- 172.21.15.44:0/476454041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1081015f0 0x7fa10810c4e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa10000cfa0 tx=0x7fa10000cfd0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.254+0000 7fa10ebff700 1 -- 172.21.15.44:0/476454041 shutdown_connections 2023-11-16T16:56:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.254+0000 7fa10ebff700 1 --2- 172.21.15.44:0/476454041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa108105560 0x7fa1081b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.254+0000 7fa10ebff700 1 --2- 172.21.15.44:0/476454041 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa0f00a1340 0x7fa0f00a37f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.254+0000 7fa10ebff700 1 --2- 172.21.15.44:0/476454041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa108100c50 0x7fa10810bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.254+0000 7fa10ebff700 1 --2- 172.21.15.44:0/476454041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1081015f0 0x7fa10810c4e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.254+0000 7fa10ebff700 1 -- 172.21.15.44:0/476454041 >> 172.21.15.44:0/476454041 conn(0x7fa108077550 msgr2=0x7fa1080fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:21.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.254+0000 7fa10ebff700 1 -- 172.21.15.44:0/476454041 shutdown_connections 2023-11-16T16:56:21.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:21.255+0000 7fa10ebff700 1 -- 172.21.15.44:0/476454041 wait complete. 2023-11-16T16:56:21.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.275+0000 7f2b5644cb80 1 Processor -- start 2023-11-16T16:56:21.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.275+0000 7f2b5644cb80 1 -- start start 2023-11-16T16:56:21.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b5644cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb803470 0x5620cb8139e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:21.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b5644cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb8078b0 0x5620cb80eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:21.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b5644cb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb7fde40 0x5620cb80bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:21.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b5644cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620cb7fe990 con 0x5620cb8078b0 2023-11-16T16:56:21.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b5644cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620cb807260 con 0x5620cb803470 2023-11-16T16:56:21.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b5644cb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620cb8074a0 con 0x5620cb7fde40 2023-11-16T16:56:21.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b4099b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb803470 0x5620cb8139e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:21.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b3bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb8078b0 0x5620cb80eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:21.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b4099b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb803470 0x5620cb8139e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:51556/0 (socket says 172.21.15.44:51556) 2023-11-16T16:56:21.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b4099b700 1 -- 172.21.15.44:0/3193615440 learned_addr learned my addr 172.21.15.44:0/3193615440 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:21.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b4119c700 1 --2- 172.21.15.44:0/3193615440 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb7fde40 0x5620cb80bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:21.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b4099b700 1 -- 172.21.15.44:0/3193615440 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb7fde40 msgr2=0x5620cb80bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:21.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b4099b700 1 --2- 172.21.15.44:0/3193615440 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb7fde40 0x5620cb80bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b4099b700 1 -- 172.21.15.44:0/3193615440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb8078b0 msgr2=0x5620cb80eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:21.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b4099b700 1 --2- 172.21.15.44:0/3193615440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb8078b0 0x5620cb80eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b4099b700 1 -- 172.21.15.44:0/3193615440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5620cb818f40 con 0x5620cb803470 2023-11-16T16:56:21.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.276+0000 7f2b4119c700 1 --2- 172.21.15.44:0/3193615440 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb7fde40 0x5620cb80bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:56:21.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.277+0000 7f2b4099b700 1 --2- 172.21.15.44:0/3193615440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb803470 0x5620cb8139e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2b300238c0 tx=0x7f2b30005050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63beb68e6c8d1d9a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:21.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.277+0000 7f2b277fe700 1 -- 172.21.15.44:0/3193615440 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3001d040 con 0x5620cb803470 2023-11-16T16:56:21.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.277+0000 7f2b277fe700 1 -- 172.21.15.44:0/3193615440 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b300050c0 con 0x5620cb803470 2023-11-16T16:56:21.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.277+0000 7f2b277fe700 1 -- 172.21.15.44:0/3193615440 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b300406d0 con 0x5620cb803470 2023-11-16T16:56:21.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.278+0000 7f2b5644cb80 1 -- 172.21.15.44:0/3193615440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb803470 msgr2=0x5620cb8139e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:21.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.278+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/3193615440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb803470 0x5620cb8139e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2b300238c0 tx=0x7f2b30005050 comp rx=0 tx=0).stop 2023-11-16T16:56:21.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.278+0000 7f2b5644cb80 1 -- 172.21.15.44:0/3193615440 shutdown_connections 2023-11-16T16:56:21.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.278+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/3193615440 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb7fde40 0x5620cb80bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.278+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/3193615440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb803470 0x5620cb8139e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.278+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/3193615440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb8078b0 0x5620cb80eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.278+0000 7f2b5644cb80 1 -- 172.21.15.44:0/3193615440 >> 172.21.15.44:0/3193615440 conn(0x5620cb7f8360 msgr2=0x5620cb7fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:21.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.278+0000 7f2b5644cb80 1 -- 172.21.15.44:0/3193615440 shutdown_connections 2023-11-16T16:56:21.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.278+0000 7f2b5644cb80 1 -- 172.21.15.44:0/3193615440 wait complete. 2023-11-16T16:56:21.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.279+0000 7f2b5644cb80 1 -- start start 2023-11-16T16:56:21.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.279+0000 7f2b5644cb80 1 Processor -- start 2023-11-16T16:56:21.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.279+0000 7f2b5644cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb7fde40 0x5620cb813870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:21.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.280+0000 7f2b5644cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb803470 0x5620cb811ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:21.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.280+0000 7f2b5644cb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb8078b0 0x5620cb8120e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:21.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.280+0000 7f2b5644cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620cb812620 con 0x5620cb803470 2023-11-16T16:56:21.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.280+0000 7f2b5644cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620cb812790 con 0x5620cb7fde40 2023-11-16T16:56:21.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.280+0000 7f2b5644cb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620cb812950 con 0x5620cb8078b0 2023-11-16T16:56:21.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.280+0000 7f2b5644cb80 10 client.0 _ll_register_callbacks cb 0x7f2b30009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:21.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.280+0000 7f2b4099b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb7fde40 0x5620cb813870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:21.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.280+0000 7f2b3bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb803470 0x5620cb811ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:21.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.280+0000 7f2b4099b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb7fde40 0x5620cb813870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:51564/0 (socket says 172.21.15.44:51564) 2023-11-16T16:56:21.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.280+0000 7f2b4099b700 1 -- 172.21.15.44:0/422140017 learned_addr learned my addr 172.21.15.44:0/422140017 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:21.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.280+0000 7f2b4119c700 1 --2- 172.21.15.44:0/422140017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb8078b0 0x5620cb8120e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:21.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.280+0000 7f2b4099b700 1 -- 172.21.15.44:0/422140017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb8078b0 msgr2=0x5620cb8120e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:21.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.280+0000 7f2b4099b700 1 --2- 172.21.15.44:0/422140017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb8078b0 0x5620cb8120e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.280+0000 7f2b4099b700 1 -- 172.21.15.44:0/422140017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb803470 msgr2=0x5620cb811ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:21.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.280+0000 7f2b4099b700 1 --2- 172.21.15.44:0/422140017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb803470 0x5620cb811ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.281+0000 7f2b4099b700 1 -- 172.21.15.44:0/422140017 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5620cb71a190 con 0x5620cb7fde40 2023-11-16T16:56:21.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.281+0000 7f2b4119c700 1 --2- 172.21.15.44:0/422140017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb8078b0 0x5620cb8120e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:56:21.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.281+0000 7f2b3bfff700 1 --2- 172.21.15.44:0/422140017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb803470 0x5620cb811ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:56:21.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.281+0000 7f2b4099b700 1 --2- 172.21.15.44:0/422140017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb7fde40 0x5620cb813870 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2b30006010 tx=0x7f2b30005450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fafbd3f56a4f77b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:21.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.281+0000 7f2b1bfff700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:56:21.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.281+0000 7f2b1bfff700 1 -- 172.21.15.44:0/422140017 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3001d040 con 0x5620cb7fde40 2023-11-16T16:56:21.285 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.281+0000 7f2b5644cb80 1 -- 172.21.15.44:0/422140017 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5620cb722660 con 0x5620cb7fde40 2023-11-16T16:56:21.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.281+0000 7f2b1bfff700 1 -- 172.21.15.44:0/422140017 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f2b30003ca0 con 0x5620cb7fde40 2023-11-16T16:56:21.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.282+0000 7f2b1bfff700 1 -- 172.21.15.44:0/422140017 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b30040470 con 0x5620cb7fde40 2023-11-16T16:56:21.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.282+0000 7f2b5644cb80 10 client.26723 fetch_fsmap learned FSMap version 21 2023-11-16T16:56:21.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.282+0000 7f2b5644cb80 1 -- 172.21.15.44:0/422140017 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5620cb581de0 con 0x5620cb7fde40 2023-11-16T16:56:21.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.282+0000 7f2b1bfff700 1 -- 172.21.15.44:0/422140017 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5620cb722660 con 0x5620cb7fde40 2023-11-16T16:56:21.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.282+0000 7f2b1bfff700 1 -- 172.21.15.44:0/422140017 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2b3000f580 con 0x5620cb7fde40 2023-11-16T16:56:21.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.282+0000 7f2b5644cb80 10 client.26723 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:56:21.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.282+0000 7f2b5644cb80 10 client.26723 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:21.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.282+0000 7f2b5644cb80 20 client.26723 populate_metadata read hostname 'smithi044' 2023-11-16T16:56:21.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.282+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/422140017 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5620cb871230 0x5620cb873610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:21.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.282+0000 7f2b5644cb80 4 client.26723 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:56:21.286 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.282+0000 7f2b5644cb80 1 -- 172.21.15.44:0/422140017 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5620cb5a2f70 con 0x5620cb871230 2023-11-16T16:56:21.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.282+0000 7f2b3bfff700 1 --2- 172.21.15.44:0/422140017 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5620cb871230 0x5620cb873610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:21.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.283+0000 7f2b1bfff700 10 client.26723 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:21.287 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.283+0000 7f2b3bfff700 1 --2- 172.21.15.44:0/422140017 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5620cb871230 0x5620cb873610 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d2031626496dfb server_cookie=5e1bdbab7578f18 in_seq=0 out_seq=0 2023-11-16T16:56:21.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.289+0000 7f2b1bfff700 1 -- 172.21.15.44:0/422140017 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5620cb5a2f70 con 0x5620cb871230 2023-11-16T16:56:21.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.289+0000 7f2b1bfff700 10 client.26723 handle_command_reply: tid=0 2023-11-16T16:56:21.290 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.289+0000 7f2b5644cb80 1 client.26723 shutdown 2023-11-16T16:56:21.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.289+0000 7f2b5644cb80 1 -- 172.21.15.44:0/422140017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb7fde40 msgr2=0x5620cb813870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:21.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.289+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/422140017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb7fde40 0x5620cb813870 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2b30006010 tx=0x7f2b30005450 comp rx=0 tx=0).stop 2023-11-16T16:56:21.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.289+0000 7f2b5644cb80 1 -- 172.21.15.44:0/422140017 shutdown_connections 2023-11-16T16:56:21.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.289+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/422140017 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5620cb871230 0x5620cb873610 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.289+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/422140017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5620cb8078b0 0x5620cb8120e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.289+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/422140017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620cb7fde40 0x5620cb813870 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.289+0000 7f2b5644cb80 1 --2- 172.21.15.44:0/422140017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5620cb803470 0x5620cb811ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.289+0000 7f2b5644cb80 1 -- 172.21.15.44:0/422140017 >> 172.21.15.44:0/422140017 conn(0x5620cb7f8360 msgr2=0x5620cb8021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:21.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.289+0000 7f2b1bfff700 0 client.26723 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:21.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.289+0000 7f2b5644cb80 1 -- 172.21.15.44:0/422140017 shutdown_connections 2023-11-16T16:56:21.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.290+0000 7f2b5644cb80 1 -- 172.21.15.44:0/422140017 wait complete. 2023-11-16T16:56:21.291 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.290+0000 7f2b5644cb80 20 client.26723 trim_cache size 0 max 16384 2023-11-16T16:56:21.292 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:56:21.292 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T16:56:21.292 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T16:56:21.292 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:56:21.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.290+0000 7f2b42bff700 1 -- 172.21.15.44:0/2230678019 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2b2c0a13c0 msgr2=0x7f2b2c0a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:21.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.290+0000 7f2b42bff700 1 --2- 172.21.15.44:0/2230678019 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2b2c0a13c0 0x7f2b2c0a3870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c19f200 tx=0x7f2b34009310 comp rx=0 tx=0).stop 2023-11-16T16:56:21.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.290+0000 7f2b42bff700 1 -- 172.21.15.44:0/2230678019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b3c101ca0 msgr2=0x7f2b3c19ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:21.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.290+0000 7f2b42bff700 1 --2- 172.21.15.44:0/2230678019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b3c101ca0 0x7f2b3c19ea40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800c920 tx=0x7f2b2800cc30 comp rx=0 tx=0).stop 2023-11-16T16:56:21.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.290+0000 7f2b42bff700 1 -- 172.21.15.44:0/2230678019 shutdown_connections 2023-11-16T16:56:21.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.290+0000 7f2b42bff700 1 --2- 172.21.15.44:0/2230678019 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b3c10a8b0 0x7f2b3c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.290+0000 7f2b42bff700 1 --2- 172.21.15.44:0/2230678019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b3c101390 0x7f2b3c19e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.290+0000 7f2b42bff700 1 --2- 172.21.15.44:0/2230678019 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2b2c0a13c0 0x7f2b2c0a3870 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.290+0000 7f2b42bff700 1 --2- 172.21.15.44:0/2230678019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b3c101ca0 0x7f2b3c19ea40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.293 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.290+0000 7f2b42bff700 1 -- 172.21.15.44:0/2230678019 >> 172.21.15.44:0/2230678019 conn(0x7f2b3c0770b0 msgr2=0x7f2b3c0ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:21.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.291+0000 7f2b42bff700 1 -- 172.21.15.44:0/2230678019 shutdown_connections 2023-11-16T16:56:21.294 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.291+0000 7f2b42bff700 1 -- 172.21.15.44:0/2230678019 wait complete. 2023-11-16T16:56:21.313 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T16:56:21.313 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:56:21.530 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.529+0000 7f3eba3ff700 1 -- 172.21.15.44:0/2883731210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb4072e90 msgr2=0x7f3eb4073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:21.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.529+0000 7f3eba3ff700 1 --2- 172.21.15.44:0/2883731210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb4072e90 0x7f3eb4073260 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400b0a0 tx=0x7f3ea400b3b0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.529+0000 7f3eba3ff700 1 -- 172.21.15.44:0/2883731210 shutdown_connections 2023-11-16T16:56:21.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.529+0000 7f3eba3ff700 1 --2- 172.21.15.44:0/2883731210 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3eb410cae0 0x7f3eb410eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.529+0000 7f3eba3ff700 1 --2- 172.21.15.44:0/2883731210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb4072e90 0x7f3eb4073260 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.529+0000 7f3eba3ff700 1 --2- 172.21.15.44:0/2883731210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3eb40737a0 0x7f3eb410c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.529+0000 7f3eba3ff700 1 -- 172.21.15.44:0/2883731210 >> 172.21.15.44:0/2883731210 conn(0x7f3eb4077050 msgr2=0x7f3eb4077450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:21.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.530+0000 7f3eba3ff700 1 -- 172.21.15.44:0/2883731210 shutdown_connections 2023-11-16T16:56:21.531 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.530+0000 7f3eba3ff700 1 -- 172.21.15.44:0/2883731210 wait complete. 2023-11-16T16:56:21.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.531+0000 7f3eba3ff700 1 Processor -- start 2023-11-16T16:56:21.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.531+0000 7f3eba3ff700 1 -- start start 2023-11-16T16:56:21.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.531+0000 7f3eba3ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3eb4072e90 0x7f3eb41986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:21.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.531+0000 7f3eba3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb40737a0 0x7f3eb4198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:21.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.531+0000 7f3eba3ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3eb410cae0 0x7f3eb419bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:21.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.531+0000 7f3eba3ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb419c590 con 0x7f3eb4072e90 2023-11-16T16:56:21.532 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.531+0000 7f3eba3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb419c700 con 0x7f3eb40737a0 2023-11-16T16:56:21.533 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.531+0000 7f3eba3ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb419c8c0 con 0x7f3eb410cae0 2023-11-16T16:56:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.532+0000 7f3eb3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3eb4072e90 0x7f3eb41986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.532+0000 7f3eab7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb40737a0 0x7f3eb4198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.532+0000 7f3eb3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3eb4072e90 0x7f3eb41986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33752/0 (socket says 172.21.15.44:33752) 2023-11-16T16:56:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.532+0000 7f3eb3fff700 1 -- 172.21.15.44:0/1694594508 learned_addr learned my addr 172.21.15.44:0/1694594508 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.532+0000 7f3eb899c700 1 --2- 172.21.15.44:0/1694594508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3eb410cae0 0x7f3eb419bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.532+0000 7f3eab7fe700 1 -- 172.21.15.44:0/1694594508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3eb410cae0 msgr2=0x7f3eb419bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.532+0000 7f3eab7fe700 1 --2- 172.21.15.44:0/1694594508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3eb410cae0 0x7f3eb419bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.532+0000 7f3eab7fe700 1 -- 172.21.15.44:0/1694594508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3eb4072e90 msgr2=0x7f3eb41986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.532+0000 7f3eab7fe700 1 --2- 172.21.15.44:0/1694594508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3eb4072e90 0x7f3eb41986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.532+0000 7f3eab7fe700 1 -- 172.21.15.44:0/1694594508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea4009d00 con 0x7f3eb40737a0 2023-11-16T16:56:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.532+0000 7f3eb3fff700 1 --2- 172.21.15.44:0/1694594508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3eb4072e90 0x7f3eb41986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:56:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.533+0000 7f3eab7fe700 1 --2- 172.21.15.44:0/1694594508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb40737a0 0x7f3eb4198be0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c00e9f0 tx=0x7f3e9c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.533+0000 7f3eb1ffb700 1 -- 172.21.15.44:0/1694594508 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9c00c9c0 con 0x7f3eb40737a0 2023-11-16T16:56:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.533+0000 7f3eb1ffb700 1 -- 172.21.15.44:0/1694594508 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e9c010430 con 0x7f3eb40737a0 2023-11-16T16:56:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.533+0000 7f3eb1ffb700 1 -- 172.21.15.44:0/1694594508 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9c018680 con 0x7f3eb40737a0 2023-11-16T16:56:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.533+0000 7f3eba3ff700 1 -- 172.21.15.44:0/1694594508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb4112510 con 0x7f3eb40737a0 2023-11-16T16:56:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.533+0000 7f3eba3ff700 1 -- 172.21.15.44:0/1694594508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb4112380 con 0x7f3eb40737a0 2023-11-16T16:56:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.534+0000 7f3eba3ff700 1 -- 172.21.15.44:0/1694594508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb41126a0 con 0x7f3eb40737a0 2023-11-16T16:56:21.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.536+0000 7f3eb1ffb700 1 -- 172.21.15.44:0/1694594508 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f3e9c010860 con 0x7f3eb40737a0 2023-11-16T16:56:21.538 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.538+0000 7f3eb1ffb700 1 --2- 172.21.15.44:0/1694594508 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3ea00a1340 0x7f3ea00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:21.538 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.538+0000 7f3eb1ffb700 1 -- 172.21.15.44:0/1694594508 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f3e9c014070 con 0x7f3eb40737a0 2023-11-16T16:56:21.539 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.538+0000 7f3eb3fff700 1 --2- 172.21.15.44:0/1694594508 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3ea00a1340 0x7f3ea00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:21.539 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.539+0000 7f3eb3fff700 1 --2- 172.21.15.44:0/1694594508 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3ea00a1340 0x7f3ea00a37f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400b780 tx=0x7f3ea4009340 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:21.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.542+0000 7f3eb1ffb700 1 -- 172.21.15.44:0/1694594508 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3e9c051730 con 0x7f3eb40737a0 2023-11-16T16:56:21.837 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.836+0000 7f3eba3ff700 1 -- 172.21.15.44:0/1694594508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3eb4107400 con 0x7f3eb40737a0 2023-11-16T16:56:21.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.837+0000 7f3eb1ffb700 1 -- 172.21.15.44:0/1694594508 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f3e9c0961f0 con 0x7f3eb40737a0 2023-11-16T16:56:21.838 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:56:21.838 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:56:21.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.840+0000 7f3eba3ff700 1 -- 172.21.15.44:0/1694594508 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3ea00a1340 msgr2=0x7f3ea00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:21.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.840+0000 7f3eba3ff700 1 --2- 172.21.15.44:0/1694594508 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3ea00a1340 0x7f3ea00a37f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400b780 tx=0x7f3ea4009340 comp rx=0 tx=0).stop 2023-11-16T16:56:21.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.840+0000 7f3eba3ff700 1 -- 172.21.15.44:0/1694594508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb40737a0 msgr2=0x7f3eb4198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:21.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.840+0000 7f3eba3ff700 1 --2- 172.21.15.44:0/1694594508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb40737a0 0x7f3eb4198be0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c00e9f0 tx=0x7f3e9c00ed00 comp rx=0 tx=0).stop 2023-11-16T16:56:21.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.840+0000 7f3eba3ff700 1 -- 172.21.15.44:0/1694594508 shutdown_connections 2023-11-16T16:56:21.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.840+0000 7f3eba3ff700 1 --2- 172.21.15.44:0/1694594508 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3ea00a1340 0x7f3ea00a37f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.840+0000 7f3eba3ff700 1 --2- 172.21.15.44:0/1694594508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3eb410cae0 0x7f3eb419bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.840+0000 7f3eba3ff700 1 --2- 172.21.15.44:0/1694594508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3eb4072e90 0x7f3eb41986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.840+0000 7f3eba3ff700 1 --2- 172.21.15.44:0/1694594508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb40737a0 0x7f3eb4198be0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:21.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.840+0000 7f3eba3ff700 1 -- 172.21.15.44:0/1694594508 >> 172.21.15.44:0/1694594508 conn(0x7f3eb4077050 msgr2=0x7f3eb4104540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:21.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.840+0000 7f3eba3ff700 1 -- 172.21.15.44:0/1694594508 shutdown_connections 2023-11-16T16:56:21.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:21.840+0000 7f3eba3ff700 1 -- 172.21.15.44:0/1694594508 wait complete. 2023-11-16T16:56:21.842 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:56:21.862 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T16:56:21.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:21 smithi044 ceph-mon[128750]: pgmap v196: 129 pgs: 129 active+clean; 58 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 568 op/s 2023-11-16T16:56:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:21 smithi176 ceph-mon[131630]: pgmap v196: 129 pgs: 129 active+clean; 58 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 568 op/s 2023-11-16T16:56:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:21 smithi077 ceph-mon[131204]: pgmap v196: 129 pgs: 129 active+clean; 58 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 568 op/s 2023-11-16T16:56:22.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.122+0000 7f891cfff700 1 -- 172.21.15.44:0/614795553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8918102ed0 msgr2=0x7f891810f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:22.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.122+0000 7f891cfff700 1 --2- 172.21.15.44:0/614795553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8918102ed0 0x7f891810f310 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f891000b0a0 tx=0x7f891000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.124+0000 7f891cfff700 1 -- 172.21.15.44:0/614795553 shutdown_connections 2023-11-16T16:56:22.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.124+0000 7f891cfff700 1 --2- 172.21.15.44:0/614795553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8918102ed0 0x7f891810f310 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.124+0000 7f891cfff700 1 --2- 172.21.15.44:0/614795553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8918108550 0x7f8918108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.124+0000 7f891cfff700 1 --2- 172.21.15.44:0/614795553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8918102520 0x7f8918102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.124+0000 7f891cfff700 1 -- 172.21.15.44:0/614795553 >> 172.21.15.44:0/614795553 conn(0x7f89180fe020 msgr2=0x7f8918100430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:22.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.127+0000 7f891cfff700 1 -- 172.21.15.44:0/614795553 shutdown_connections 2023-11-16T16:56:22.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.127+0000 7f891cfff700 1 -- 172.21.15.44:0/614795553 wait complete. 2023-11-16T16:56:22.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.130+0000 7f891cfff700 1 Processor -- start 2023-11-16T16:56:22.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.131+0000 7f891cfff700 1 -- start start 2023-11-16T16:56:22.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.132+0000 7f891cfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8918102520 0x7f891810ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.132+0000 7f891cfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8918102ed0 0x7f891810be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.132+0000 7f891cfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8918108550 0x7f891810c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.132+0000 7f891cfff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891810c890 con 0x7f8918102520 2023-11-16T16:56:22.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.132+0000 7f891cfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891810c9d0 con 0x7f8918108550 2023-11-16T16:56:22.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.132+0000 7f891cfff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891810cb90 con 0x7f8918102ed0 2023-11-16T16:56:22.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.132+0000 7f8916d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8918108550 0x7f891810c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:22.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.132+0000 7f8916d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8918108550 0x7f891810c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:51610/0 (socket says 172.21.15.44:51610) 2023-11-16T16:56:22.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.132+0000 7f8916d9d700 1 -- 172.21.15.44:0/280487177 learned_addr learned my addr 172.21.15.44:0/280487177 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:22.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.133+0000 7f8916d9d700 1 -- 172.21.15.44:0/280487177 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8918102ed0 msgr2=0x7f891810be10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:56:22.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.133+0000 7f8916d9d700 1 --2- 172.21.15.44:0/280487177 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8918102ed0 0x7f891810be10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.133+0000 7f8916d9d700 1 -- 172.21.15.44:0/280487177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8918102520 msgr2=0x7f891810ee50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:56:22.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.133+0000 7f8916d9d700 1 --2- 172.21.15.44:0/280487177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8918102520 0x7f891810ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.133+0000 7f8916d9d700 1 -- 172.21.15.44:0/280487177 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8910009d00 con 0x7f8918108550 2023-11-16T16:56:22.135 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.134+0000 7f8916d9d700 1 --2- 172.21.15.44:0/280487177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8918108550 0x7f891810c350 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8910009fd0 tx=0x7f891000bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:22.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.137+0000 7f89077fe700 1 -- 172.21.15.44:0/280487177 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891000e040 con 0x7f8918108550 2023-11-16T16:56:22.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.137+0000 7f89077fe700 1 -- 172.21.15.44:0/280487177 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8910007930 con 0x7f8918108550 2023-11-16T16:56:22.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.137+0000 7f89077fe700 1 -- 172.21.15.44:0/280487177 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8910004680 con 0x7f8918108550 2023-11-16T16:56:22.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.137+0000 7f891cfff700 1 -- 172.21.15.44:0/280487177 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8918112570 con 0x7f8918108550 2023-11-16T16:56:22.138 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.137+0000 7f891cfff700 1 -- 172.21.15.44:0/280487177 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89181123e0 con 0x7f8918108550 2023-11-16T16:56:22.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.152+0000 7f89077fe700 1 -- 172.21.15.44:0/280487177 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f8910004c60 con 0x7f8918108550 2023-11-16T16:56:22.160 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.158+0000 7f89077fe700 1 --2- 172.21.15.44:0/280487177 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f89000a1340 0x7f89000a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.160 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.159+0000 7f89077fe700 1 -- 172.21.15.44:0/280487177 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f89100cde30 con 0x7f8918108550 2023-11-16T16:56:22.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.160+0000 7f8930831b80 1 Processor -- start 2023-11-16T16:56:22.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.160+0000 7f891659c700 1 --2- 172.21.15.44:0/280487177 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f89000a1340 0x7f89000a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:22.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.161+0000 7f8930831b80 1 -- start start 2023-11-16T16:56:22.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.161+0000 7f8930831b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d82b60 0x5626b9d82f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.161+0000 7f8930831b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d83470 0x5626b9d87370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.161+0000 7f8930831b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d878b0 0x5626b9d87d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.162+0000 7f8915d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d83470 0x5626b9d87370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:22.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.162+0000 7f8915d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d83470 0x5626b9d87370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:59338/0 (socket says 172.21.15.44:59338) 2023-11-16T16:56:22.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.162+0000 7f891659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d82b60 0x5626b9d82f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:22.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.162+0000 7f8930831b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b9c88060 con 0x5626b9d878b0 2023-11-16T16:56:22.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.162+0000 7f8930831b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b9cda8a0 con 0x5626b9d82b60 2023-11-16T16:56:22.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.162+0000 7f8930831b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b9cd3d30 con 0x5626b9d83470 2023-11-16T16:56:22.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.162+0000 7f891659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d82b60 0x5626b9d82f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:51624/0 (socket says 172.21.15.44:51624) 2023-11-16T16:56:22.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.162+0000 7f891659c700 1 -- 172.21.15.44:0/1868860825 learned_addr learned my addr 172.21.15.44:0/1868860825 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:22.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.162+0000 7f8916d9d700 1 --2- 172.21.15.44:0/1868860825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d878b0 0x5626b9d87d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:22.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.163+0000 7f8916d9d700 1 -- 172.21.15.44:0/1868860825 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d83470 msgr2=0x5626b9d87370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:22.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.163+0000 7f8916d9d700 1 --2- 172.21.15.44:0/1868860825 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d83470 0x5626b9d87370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.163+0000 7f8916d9d700 1 -- 172.21.15.44:0/1868860825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d82b60 msgr2=0x5626b9d82f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:22.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.163+0000 7f8916d9d700 1 --2- 172.21.15.44:0/1868860825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d82b60 0x5626b9d82f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.163+0000 7f8916d9d700 1 -- 172.21.15.44:0/1868860825 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626b9b55630 con 0x5626b9d878b0 2023-11-16T16:56:22.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.164+0000 7f8916d9d700 1 --2- 172.21.15.44:0/1868860825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d878b0 0x5626b9d87d00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f89100cb150 tx=0x7f891000b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b4868365043156d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:22.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.165+0000 7f8904ff9700 1 -- 172.21.15.44:0/1868860825 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89100a2f20 con 0x5626b9d878b0 2023-11-16T16:56:22.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.165+0000 7f8904ff9700 1 -- 172.21.15.44:0/1868860825 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f89100047e0 con 0x5626b9d878b0 2023-11-16T16:56:22.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.165+0000 7f8904ff9700 1 -- 172.21.15.44:0/1868860825 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891009b000 con 0x5626b9d878b0 2023-11-16T16:56:22.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.166+0000 7f8930831b80 1 -- 172.21.15.44:0/1868860825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d878b0 msgr2=0x5626b9d87d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:22.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.166+0000 7f8930831b80 1 --2- 172.21.15.44:0/1868860825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d878b0 0x5626b9d87d00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f89100cb150 tx=0x7f891000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.167+0000 7f8930831b80 1 -- 172.21.15.44:0/1868860825 shutdown_connections 2023-11-16T16:56:22.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.167+0000 7f8930831b80 1 --2- 172.21.15.44:0/1868860825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d878b0 0x5626b9d87d00 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.167+0000 7f8930831b80 1 --2- 172.21.15.44:0/1868860825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d82b60 0x5626b9d82f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.167+0000 7f8930831b80 1 --2- 172.21.15.44:0/1868860825 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d83470 0x5626b9d87370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.167+0000 7f8930831b80 1 -- 172.21.15.44:0/1868860825 >> 172.21.15.44:0/1868860825 conn(0x5626b9d78360 msgr2=0x5626b9d7db40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:22.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.167+0000 7f8930831b80 1 -- 172.21.15.44:0/1868860825 shutdown_connections 2023-11-16T16:56:22.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.167+0000 7f891659c700 1 --2- 172.21.15.44:0/280487177 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f89000a1340 0x7f89000a37f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8908009b60 tx=0x7f8908009620 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:22.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.168+0000 7f8930831b80 1 -- 172.21.15.44:0/1868860825 wait complete. 2023-11-16T16:56:22.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.169+0000 7f8930831b80 1 -- start start 2023-11-16T16:56:22.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.169+0000 7f8930831b80 1 Processor -- start 2023-11-16T16:56:22.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.170+0000 7f8930831b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d82b60 0x5626b9d8f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.170+0000 7f8930831b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d83470 0x5626b9d8c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.170+0000 7f8930831b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d878b0 0x5626b9d8c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.171+0000 7f8930831b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b9c758e0 con 0x5626b9d83470 2023-11-16T16:56:22.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.171+0000 7f8930831b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b9c75580 con 0x5626b9d82b60 2023-11-16T16:56:22.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.171+0000 7f8930831b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b9b57920 con 0x5626b9d878b0 2023-11-16T16:56:22.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.171+0000 7f8930831b80 10 client.0 _ll_register_callbacks cb 0x5626b9c93780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:22.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.171+0000 7f8916d9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d878b0 0x5626b9d8c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:22.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.171+0000 7f891659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d82b60 0x5626b9d8f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:22.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.171+0000 7f8916d9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d878b0 0x5626b9d8c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:59346/0 (socket says 172.21.15.44:59346) 2023-11-16T16:56:22.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.171+0000 7f8916d9d700 1 -- 172.21.15.44:0/2391667096 learned_addr learned my addr 172.21.15.44:0/2391667096 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:22.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.172+0000 7f8915d9b700 1 --2- 172.21.15.44:0/2391667096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d83470 0x5626b9d8c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:22.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.172+0000 7f891659c700 1 -- 172.21.15.44:0/2391667096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d878b0 msgr2=0x5626b9d8c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:22.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.172+0000 7f891659c700 1 --2- 172.21.15.44:0/2391667096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d878b0 0x5626b9d8c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.172+0000 7f891659c700 1 -- 172.21.15.44:0/2391667096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d83470 msgr2=0x5626b9d8c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:22.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.172+0000 7f891659c700 1 --2- 172.21.15.44:0/2391667096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d83470 0x5626b9d8c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.172+0000 7f891659c700 1 -- 172.21.15.44:0/2391667096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89181123e0 con 0x5626b9d82b60 2023-11-16T16:56:22.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.173+0000 7f891659c700 1 --2- 172.21.15.44:0/2391667096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d82b60 0x5626b9d8f020 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f890800ead0 tx=0x7f8908004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adc2771bbd2c3fdf server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:22.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.173+0000 7f88fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:56:22.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.175+0000 7f88fdffb700 1 -- 172.21.15.44:0/2391667096 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908018040 con 0x5626b9d82b60 2023-11-16T16:56:22.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.175+0000 7f88fdffb700 1 -- 172.21.15.44:0/2391667096 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f8908004830 con 0x5626b9d82b60 2023-11-16T16:56:22.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.175+0000 7f8930831b80 1 -- 172.21.15.44:0/2391667096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5626b9d75b20 con 0x5626b9d82b60 2023-11-16T16:56:22.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.176+0000 7f88fdffb700 1 -- 172.21.15.44:0/2391667096 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89080049f0 con 0x5626b9d82b60 2023-11-16T16:56:22.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.176+0000 7f8930831b80 10 client.26741 fetch_fsmap learned FSMap version 21 2023-11-16T16:56:22.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.176+0000 7f8930831b80 1 -- 172.21.15.44:0/2391667096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5626b9c9a190 con 0x5626b9d82b60 2023-11-16T16:56:22.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.178+0000 7f88fdffb700 1 -- 172.21.15.44:0/2391667096 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5626b9d75b20 con 0x5626b9d82b60 2023-11-16T16:56:22.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.178+0000 7f88fdffb700 1 -- 172.21.15.44:0/2391667096 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f890801f560 con 0x5626b9d82b60 2023-11-16T16:56:22.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.178+0000 7f8930831b80 10 client.26741 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:56:22.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.178+0000 7f8930831b80 10 client.26741 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:22.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.178+0000 7f8930831b80 20 client.26741 populate_metadata read hostname 'smithi044' 2023-11-16T16:56:22.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.178+0000 7f8930831b80 1 --2- 172.21.15.44:0/2391667096 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5626b9d9c630 0x5626b9d9ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.178+0000 7f8930831b80 4 client.26741 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:56:22.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.178+0000 7f8930831b80 1 -- 172.21.15.44:0/2391667096 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5626b9c960a0 con 0x5626b9d9c630 2023-11-16T16:56:22.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.183+0000 7f8915d9b700 1 --2- 172.21.15.44:0/2391667096 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5626b9d9c630 0x5626b9d9ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:22.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.186+0000 7f8915d9b700 1 --2- 172.21.15.44:0/2391667096 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5626b9d9c630 0x5626b9d9ea40 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf91291fc804a8a7 server_cookie=5b303b980a716612 in_seq=0 out_seq=0 2023-11-16T16:56:22.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.186+0000 7f88fdffb700 10 client.26741 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:22.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.193+0000 7f88fdffb700 1 -- 172.21.15.44:0/2391667096 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5626b9c960a0 con 0x5626b9d9c630 2023-11-16T16:56:22.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.193+0000 7f88fdffb700 10 client.26741 handle_command_reply: tid=0 2023-11-16T16:56:22.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.193+0000 7f8930831b80 1 client.26741 shutdown 2023-11-16T16:56:22.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.193+0000 7f8930831b80 1 -- 172.21.15.44:0/2391667096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d82b60 msgr2=0x5626b9d8f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:22.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.193+0000 7f8930831b80 1 --2- 172.21.15.44:0/2391667096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d82b60 0x5626b9d8f020 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f890800ead0 tx=0x7f8908004750 comp rx=0 tx=0).stop 2023-11-16T16:56:22.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.194+0000 7f8930831b80 1 -- 172.21.15.44:0/2391667096 shutdown_connections 2023-11-16T16:56:22.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.194+0000 7f8930831b80 1 --2- 172.21.15.44:0/2391667096 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5626b9d9c630 0x5626b9d9ea40 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.194+0000 7f8930831b80 1 --2- 172.21.15.44:0/2391667096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d878b0 0x5626b9d8c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.194+0000 7f8930831b80 1 --2- 172.21.15.44:0/2391667096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d82b60 0x5626b9d8f020 secure :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f890800ead0 tx=0x7f8908004750 comp rx=0 tx=0).stop 2023-11-16T16:56:22.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.194+0000 7f8930831b80 1 --2- 172.21.15.44:0/2391667096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d83470 0x5626b9d8c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.194+0000 7f8930831b80 1 -- 172.21.15.44:0/2391667096 >> 172.21.15.44:0/2391667096 conn(0x5626b9d78360 msgr2=0x5626b9d88d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:22.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.194+0000 7f88fdffb700 0 client.26741 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:22.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.195+0000 7f8930831b80 1 -- 172.21.15.44:0/2391667096 shutdown_connections 2023-11-16T16:56:22.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.196+0000 7f8930831b80 1 -- 172.21.15.44:0/2391667096 wait complete. 2023-11-16T16:56:22.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.197+0000 7f8930831b80 20 client.26741 trim_cache size 0 max 16384 2023-11-16T16:56:22.239 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.237+0000 7f8930831b80 1 Processor -- start 2023-11-16T16:56:22.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.239+0000 7f8930831b80 1 -- start start 2023-11-16T16:56:22.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.240+0000 7f8930831b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d83470 0x5626b9d937b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.240+0000 7f8930831b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d878b0 0x5626b9d8ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.240+0000 7f8930831b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d8f100 0x5626b9d8bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.240+0000 7f8930831b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b9d89390 con 0x5626b9d8f100 2023-11-16T16:56:22.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.240+0000 7f8930831b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b9d8d200 con 0x5626b9d878b0 2023-11-16T16:56:22.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.240+0000 7f8930831b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b9d8d880 con 0x5626b9d83470 2023-11-16T16:56:22.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.240+0000 7f891659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d83470 0x5626b9d937b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:22.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.240+0000 7f891659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d83470 0x5626b9d937b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:59352/0 (socket says 172.21.15.44:59352) 2023-11-16T16:56:22.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.240+0000 7f891659c700 1 -- 172.21.15.44:0/2333807910 learned_addr learned my addr 172.21.15.44:0/2333807910 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:22.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.241+0000 7f8915d9b700 1 --2- 172.21.15.44:0/2333807910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d878b0 0x5626b9d8ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:22.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.241+0000 7f8916d9d700 1 --2- 172.21.15.44:0/2333807910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d8f100 0x5626b9d8bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:22.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.243+0000 7f891659c700 1 -- 172.21.15.44:0/2333807910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d878b0 msgr2=0x5626b9d8ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:22.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.243+0000 7f891659c700 1 --2- 172.21.15.44:0/2333807910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d878b0 0x5626b9d8ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.243+0000 7f891659c700 1 -- 172.21.15.44:0/2333807910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d8f100 msgr2=0x5626b9d8bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:22.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.243+0000 7f891659c700 1 --2- 172.21.15.44:0/2333807910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d8f100 0x5626b9d8bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.243+0000 7f891659c700 1 -- 172.21.15.44:0/2333807910 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626b9c99ee0 con 0x5626b9d83470 2023-11-16T16:56:22.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.243+0000 7f8916d9d700 1 --2- 172.21.15.44:0/2333807910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d8f100 0x5626b9d8bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:56:22.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.243+0000 7f891659c700 1 --2- 172.21.15.44:0/2333807910 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d83470 0x5626b9d937b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f890801f8f0 tx=0x7f890801fa90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c26bbc5b5495a3c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:22.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.244+0000 7f89057fa700 1 -- 172.21.15.44:0/2333807910 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908005ce0 con 0x5626b9d83470 2023-11-16T16:56:22.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.244+0000 7f89057fa700 1 -- 172.21.15.44:0/2333807910 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f890801fbe0 con 0x5626b9d83470 2023-11-16T16:56:22.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.244+0000 7f89057fa700 1 -- 172.21.15.44:0/2333807910 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908004a90 con 0x5626b9d83470 2023-11-16T16:56:22.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.245+0000 7f8930831b80 1 -- 172.21.15.44:0/2333807910 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d83470 msgr2=0x5626b9d937b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:22.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.245+0000 7f8930831b80 1 --2- 172.21.15.44:0/2333807910 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d83470 0x5626b9d937b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f890801f8f0 tx=0x7f890801fa90 comp rx=0 tx=0).stop 2023-11-16T16:56:22.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.246+0000 7f8930831b80 1 -- 172.21.15.44:0/2333807910 shutdown_connections 2023-11-16T16:56:22.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.246+0000 7f8930831b80 1 --2- 172.21.15.44:0/2333807910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d8f100 0x5626b9d8bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.246+0000 7f8930831b80 1 --2- 172.21.15.44:0/2333807910 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d83470 0x5626b9d937b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.246+0000 7f8930831b80 1 --2- 172.21.15.44:0/2333807910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d878b0 0x5626b9d8ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.246+0000 7f8930831b80 1 -- 172.21.15.44:0/2333807910 >> 172.21.15.44:0/2333807910 conn(0x5626b9d78360 msgr2=0x5626b9d989b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:22.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.246+0000 7f8930831b80 1 -- 172.21.15.44:0/2333807910 shutdown_connections 2023-11-16T16:56:22.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.246+0000 7f8930831b80 1 -- 172.21.15.44:0/2333807910 wait complete. 2023-11-16T16:56:22.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.247+0000 7f8930831b80 1 -- start start 2023-11-16T16:56:22.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.247+0000 7f8930831b80 1 Processor -- start 2023-11-16T16:56:22.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.248+0000 7f8930831b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d83470 0x5626b9d937c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.248+0000 7f8930831b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d878b0 0x5626b9d8aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.248+0000 7f8930831b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d8f100 0x5626b9d8af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.248+0000 7f8930831b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b9d8b4d0 con 0x5626b9d878b0 2023-11-16T16:56:22.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.248+0000 7f8930831b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b9d8b640 con 0x5626b9d83470 2023-11-16T16:56:22.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.248+0000 7f8930831b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626b9d8b800 con 0x5626b9d8f100 2023-11-16T16:56:22.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.248+0000 7f8930831b80 10 client.0 _ll_register_callbacks cb 0x7f8908010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:22.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.249+0000 7f8916d9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d8f100 0x5626b9d8af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:22.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.249+0000 7f8916d9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d8f100 0x5626b9d8af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:59354/0 (socket says 172.21.15.44:59354) 2023-11-16T16:56:22.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.249+0000 7f8916d9d700 1 -- 172.21.15.44:0/999027790 learned_addr learned my addr 172.21.15.44:0/999027790 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:22.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.249+0000 7f8915d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d878b0 0x5626b9d8aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:22.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.250+0000 7f8916d9d700 1 -- 172.21.15.44:0/999027790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d83470 msgr2=0x5626b9d937c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T16:56:22.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.250+0000 7f8916d9d700 1 --2- 172.21.15.44:0/999027790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d83470 0x5626b9d937c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.250+0000 7f8916d9d700 1 -- 172.21.15.44:0/999027790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d878b0 msgr2=0x5626b9d8aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:22.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.250+0000 7f8916d9d700 1 --2- 172.21.15.44:0/999027790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d878b0 0x5626b9d8aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.250+0000 7f8916d9d700 1 -- 172.21.15.44:0/999027790 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626b9c9a190 con 0x5626b9d8f100 2023-11-16T16:56:22.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.251+0000 7f8916d9d700 1 --2- 172.21.15.44:0/999027790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d8f100 0x5626b9d8af90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8910097060 tx=0x7f8910006160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a8e2353292bb698 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:22.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.251+0000 7f88fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:56:22.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.251+0000 7f88fdffb700 1 -- 172.21.15.44:0/999027790 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89100b3270 con 0x5626b9d8f100 2023-11-16T16:56:22.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.251+0000 7f88fdffb700 1 -- 172.21.15.44:0/999027790 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f89100a2f20 con 0x5626b9d8f100 2023-11-16T16:56:22.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.252+0000 7f8930831b80 1 -- 172.21.15.44:0/999027790 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f89100047e0 con 0x5626b9d8f100 2023-11-16T16:56:22.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.252+0000 7f88fdffb700 1 -- 172.21.15.44:0/999027790 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f891009a050 con 0x5626b9d8f100 2023-11-16T16:56:22.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.252+0000 7f8930831b80 10 client.26719 fetch_fsmap learned FSMap version 21 2023-11-16T16:56:22.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.252+0000 7f8930831b80 1 -- 172.21.15.44:0/999027790 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5626b9aeac60 con 0x5626b9d8f100 2023-11-16T16:56:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.253+0000 7f88fdffb700 1 -- 172.21.15.44:0/999027790 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f89100047e0 con 0x5626b9d8f100 2023-11-16T16:56:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.254+0000 7f88fdffb700 1 -- 172.21.15.44:0/999027790 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f89100a2a90 con 0x5626b9d8f100 2023-11-16T16:56:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.257+0000 7f8930831b80 10 client.26719 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:56:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.257+0000 7f8930831b80 10 client.26719 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.257+0000 7f8930831b80 20 client.26719 populate_metadata read hostname 'smithi044' 2023-11-16T16:56:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.257+0000 7f8930831b80 1 --2- 172.21.15.44:0/999027790 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5626b9df14a0 0x5626b9df3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.257+0000 7f8930831b80 4 client.26719 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T16:56:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.257+0000 7f8930831b80 1 -- 172.21.15.44:0/999027790 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5626b9ca2660 con 0x5626b9df14a0 2023-11-16T16:56:22.259 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.259+0000 7f891659c700 1 --2- 172.21.15.44:0/999027790 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5626b9df14a0 0x5626b9df3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:22.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.260+0000 7f88fdffb700 10 client.26719 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:22.260 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.260+0000 7f891659c700 1 --2- 172.21.15.44:0/999027790 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5626b9df14a0 0x5626b9df3880 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bb33d1c9a1693fc server_cookie=13eb0456052798f1 in_seq=0 out_seq=0 2023-11-16T16:56:22.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.267+0000 7f88fdffb700 1 -- 172.21.15.44:0/999027790 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5626b9ca2660 con 0x5626b9df14a0 2023-11-16T16:56:22.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.267+0000 7f88fdffb700 10 client.26719 handle_command_reply: tid=0 2023-11-16T16:56:22.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.267+0000 7f8930831b80 1 client.26719 shutdown 2023-11-16T16:56:22.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.268+0000 7f8930831b80 1 -- 172.21.15.44:0/999027790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d8f100 msgr2=0x5626b9d8af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:22.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.268+0000 7f8930831b80 1 --2- 172.21.15.44:0/999027790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d8f100 0x5626b9d8af90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8910097060 tx=0x7f8910006160 comp rx=0 tx=0).stop 2023-11-16T16:56:22.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.269+0000 7f8930831b80 1 -- 172.21.15.44:0/999027790 shutdown_connections 2023-11-16T16:56:22.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.269+0000 7f8930831b80 1 --2- 172.21.15.44:0/999027790 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5626b9df14a0 0x5626b9df3880 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.269+0000 7f8930831b80 1 --2- 172.21.15.44:0/999027790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5626b9d8f100 0x5626b9d8af90 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.269+0000 7f8930831b80 1 --2- 172.21.15.44:0/999027790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626b9d83470 0x5626b9d937c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.269+0000 7f8930831b80 1 --2- 172.21.15.44:0/999027790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5626b9d878b0 0x5626b9d8aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.269+0000 7f8930831b80 1 -- 172.21.15.44:0/999027790 >> 172.21.15.44:0/999027790 conn(0x5626b9d78360 msgr2=0x5626b9d89570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:22.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.269+0000 7f88fdffb700 0 client.26719 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:22.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.270+0000 7f8930831b80 1 -- 172.21.15.44:0/999027790 shutdown_connections 2023-11-16T16:56:22.271 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.270+0000 7f8930831b80 1 -- 172.21.15.44:0/999027790 wait complete. 2023-11-16T16:56:22.272 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.270+0000 7f8930831b80 20 client.26719 trim_cache size 0 max 16384 2023-11-16T16:56:22.272 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T16:56:22.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.272+0000 7f891cfff700 1 -- 172.21.15.44:0/280487177 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f89000a1340 msgr2=0x7f89000a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:22.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.272+0000 7f891cfff700 1 --2- 172.21.15.44:0/280487177 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f89000a1340 0x7f89000a37f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8908009b60 tx=0x7f8908009620 comp rx=0 tx=0).stop 2023-11-16T16:56:22.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.273+0000 7f891cfff700 1 -- 172.21.15.44:0/280487177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8918108550 msgr2=0x7f891810c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:22.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.273+0000 7f891cfff700 1 --2- 172.21.15.44:0/280487177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8918108550 0x7f891810c350 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8910009fd0 tx=0x7f891000bd90 comp rx=0 tx=0).stop 2023-11-16T16:56:22.275 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.274+0000 7f891cfff700 1 -- 172.21.15.44:0/280487177 shutdown_connections 2023-11-16T16:56:22.275 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.274+0000 7f891cfff700 1 --2- 172.21.15.44:0/280487177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8918108550 0x7f891810c350 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.275 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.275+0000 7f891cfff700 1 --2- 172.21.15.44:0/280487177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8918102520 0x7f891810ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.275 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.275+0000 7f891cfff700 1 --2- 172.21.15.44:0/280487177 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f89000a1340 0x7f89000a37f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.275 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.275+0000 7f891cfff700 1 --2- 172.21.15.44:0/280487177 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8918102ed0 0x7f891810be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:22.275 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.275+0000 7f891cfff700 1 -- 172.21.15.44:0/280487177 >> 172.21.15.44:0/280487177 conn(0x7f89180fe020 msgr2=0x7f8918106120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:22.276 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.275+0000 7f891cfff700 1 -- 172.21.15.44:0/280487177 shutdown_connections 2023-11-16T16:56:22.276 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:22.275+0000 7f891cfff700 1 -- 172.21.15.44:0/280487177 wait complete. 2023-11-16T16:56:22.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:22 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1694594508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:56:22.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:22 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1694594508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:56:22.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:22 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1694594508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:56:23.325 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:56:23.325 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:56:23.675 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.674+0000 7f1bd73ff700 1 -- 172.21.15.44:0/3627825417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bd010cec0 msgr2=0x7f1bd010f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:23.675 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.674+0000 7f1bd73ff700 1 --2- 172.21.15.44:0/3627825417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bd010cec0 0x7f1bd010f2a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009f90 tx=0x7f1bc800b330 comp rx=0 tx=0).stop 2023-11-16T16:56:23.676 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.674+0000 7f1bd73ff700 1 -- 172.21.15.44:0/3627825417 shutdown_connections 2023-11-16T16:56:23.676 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.674+0000 7f1bd73ff700 1 --2- 172.21.15.44:0/3627825417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bd010cec0 0x7f1bd010f2a0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.676 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.675+0000 7f1bd73ff700 1 --2- 172.21.15.44:0/3627825417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bd00746f0 0x7f1bd0072d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.676 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.675+0000 7f1bd73ff700 1 --2- 172.21.15.44:0/3627825417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd0074ac0 0x7f1bd0073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.676 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.675+0000 7f1bd73ff700 1 -- 172.21.15.44:0/3627825417 >> 172.21.15.44:0/3627825417 conn(0x7f1bd0077560 msgr2=0x7f1bd0077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:23.676 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.675+0000 7f1bd73ff700 1 -- 172.21.15.44:0/3627825417 shutdown_connections 2023-11-16T16:56:23.676 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.675+0000 7f1bd73ff700 1 -- 172.21.15.44:0/3627825417 wait complete. 2023-11-16T16:56:23.676 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.676+0000 7f1bd73ff700 1 Processor -- start 2023-11-16T16:56:23.677 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.676+0000 7f1bd73ff700 1 -- start start 2023-11-16T16:56:23.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.677+0000 7f1bd73ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bd00746f0 0x7f1bd019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.677+0000 7f1bd73ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bd0074ac0 0x7f1bd019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.677+0000 7f1bd73ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd010cec0 0x7f1bd01986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.677+0000 7f1bd73ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd0198d60 con 0x7f1bd010cec0 2023-11-16T16:56:23.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.677+0000 7f1bd73ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd0198ed0 con 0x7f1bd00746f0 2023-11-16T16:56:23.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.677+0000 7f1bd73ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd0199090 con 0x7f1bd0074ac0 2023-11-16T16:56:23.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.677+0000 7f1bd519b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bd00746f0 0x7f1bd019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.677+0000 7f1bd519b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bd00746f0 0x7f1bd019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:51664/0 (socket says 172.21.15.44:51664) 2023-11-16T16:56:23.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.677+0000 7f1bd519b700 1 -- 172.21.15.44:0/3501341290 learned_addr learned my addr 172.21.15.44:0/3501341290 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:23.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.677+0000 7f1bd499a700 1 --2- 172.21.15.44:0/3501341290 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bd0074ac0 0x7f1bd019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.678+0000 7f1bd599c700 1 --2- 172.21.15.44:0/3501341290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd010cec0 0x7f1bd01986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.678+0000 7f1bd519b700 1 -- 172.21.15.44:0/3501341290 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bd0074ac0 msgr2=0x7f1bd019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:23.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.678+0000 7f1bd519b700 1 --2- 172.21.15.44:0/3501341290 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bd0074ac0 0x7f1bd019ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.678+0000 7f1bd519b700 1 -- 172.21.15.44:0/3501341290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd010cec0 msgr2=0x7f1bd01986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:23.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.678+0000 7f1bd519b700 1 --2- 172.21.15.44:0/3501341290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd010cec0 0x7f1bd01986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.678+0000 7f1bd519b700 1 -- 172.21.15.44:0/3501341290 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc8009c40 con 0x7f1bd00746f0 2023-11-16T16:56:23.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.678+0000 7f1bd519b700 1 --2- 172.21.15.44:0/3501341290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bd00746f0 0x7f1bd019e680 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00e9a0 tx=0x7f1bcc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:23.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.678+0000 7f1bc67fc700 1 -- 172.21.15.44:0/3501341290 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc009790 con 0x7f1bd00746f0 2023-11-16T16:56:23.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.678+0000 7f1bc67fc700 1 -- 172.21.15.44:0/3501341290 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1bcc00ce80 con 0x7f1bd00746f0 2023-11-16T16:56:23.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.678+0000 7f1bd73ff700 1 -- 172.21.15.44:0/3501341290 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd01125c0 con 0x7f1bd00746f0 2023-11-16T16:56:23.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.679+0000 7f1bc67fc700 1 -- 172.21.15.44:0/3501341290 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc018680 con 0x7f1bd00746f0 2023-11-16T16:56:23.681 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.679+0000 7f1bd73ff700 1 -- 172.21.15.44:0/3501341290 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd0112430 con 0x7f1bd00746f0 2023-11-16T16:56:23.684 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.683+0000 7f1bc67fc700 1 -- 172.21.15.44:0/3501341290 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f1bcc010430 con 0x7f1bd00746f0 2023-11-16T16:56:23.685 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.684+0000 7f1bc67fc700 1 --2- 172.21.15.44:0/3501341290 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1bbc0a12c0 0x7f1bbc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.685 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.685+0000 7f1bc67fc700 1 -- 172.21.15.44:0/3501341290 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f1bcc014070 con 0x7f1bd00746f0 2023-11-16T16:56:23.685 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.685+0000 7f1bd499a700 1 --2- 172.21.15.44:0/3501341290 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1bbc0a12c0 0x7f1bbc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.686+0000 7f1beadafb80 1 Processor -- start 2023-11-16T16:56:23.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.686+0000 7f1beadafb80 1 -- start start 2023-11-16T16:56:23.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.686+0000 7f1bd499a700 1 --2- 172.21.15.44:0/3501341290 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1bbc0a12c0 0x7f1bbc0a3770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0199b60 tx=0x7f1bc0009310 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:23.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.686+0000 7f1beadafb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a85be0 0x564244a85fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.686+0000 7f1beadafb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a864f0 0x564244a8a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.686+0000 7f1beadafb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a8a930 0x564244a8ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.686+0000 7f1beadafb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56424498b0e0 con 0x564244a864f0 2023-11-16T16:56:23.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.686+0000 7f1beadafb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642449dd920 con 0x564244a85be0 2023-11-16T16:56:23.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.686+0000 7f1beadafb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642449d6db0 con 0x564244a8a930 2023-11-16T16:56:23.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.686+0000 7f1bd519b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a85be0 0x564244a85fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.686+0000 7f1bd519b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a85be0 0x564244a85fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:51676/0 (socket says 172.21.15.44:51676) 2023-11-16T16:56:23.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.686+0000 7f1bd519b700 1 -- 172.21.15.44:0/1363059761 learned_addr learned my addr 172.21.15.44:0/1363059761 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:23.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.687+0000 7f1bd499a700 1 --2- 172.21.15.44:0/1363059761 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a864f0 0x564244a8a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.689 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.687+0000 7f1bd599c700 1 --2- 172.21.15.44:0/1363059761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a8a930 0x564244a8ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.687+0000 7f1bd519b700 1 -- 172.21.15.44:0/1363059761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a8a930 msgr2=0x564244a8ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:23.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.687+0000 7f1bd519b700 1 --2- 172.21.15.44:0/1363059761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a8a930 0x564244a8ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.687+0000 7f1bd519b700 1 -- 172.21.15.44:0/1363059761 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a864f0 msgr2=0x564244a8a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:23.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.687+0000 7f1bd519b700 1 --2- 172.21.15.44:0/1363059761 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a864f0 0x564244a8a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.687+0000 7f1bd519b700 1 -- 172.21.15.44:0/1363059761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5642448586b0 con 0x564244a85be0 2023-11-16T16:56:23.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.687+0000 7f1bd599c700 1 --2- 172.21.15.44:0/1363059761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a8a930 0x564244a8ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:56:23.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.687+0000 7f1bd499a700 1 --2- 172.21.15.44:0/1363059761 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a864f0 0x564244a8a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:56:23.690 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.687+0000 7f1bd519b700 1 --2- 172.21.15.44:0/1363059761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a85be0 0x564244a85fb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc0a1f50 tx=0x7f1bcc09a0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9f66774d82d92c3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:23.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.687+0000 7f1bbb7fe700 1 -- 172.21.15.44:0/1363059761 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc0cd0b0 con 0x564244a85be0 2023-11-16T16:56:23.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.687+0000 7f1bbb7fe700 1 -- 172.21.15.44:0/1363059761 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1bcc010710 con 0x564244a85be0 2023-11-16T16:56:23.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.687+0000 7f1bbb7fe700 1 -- 172.21.15.44:0/1363059761 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc0a1ff0 con 0x564244a85be0 2023-11-16T16:56:23.691 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.688+0000 7f1beadafb80 1 -- 172.21.15.44:0/1363059761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a85be0 msgr2=0x564244a85fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:23.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.688+0000 7f1beadafb80 1 --2- 172.21.15.44:0/1363059761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a85be0 0x564244a85fb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc0a1f50 tx=0x7f1bcc09a0b0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.689+0000 7f1beadafb80 1 -- 172.21.15.44:0/1363059761 shutdown_connections 2023-11-16T16:56:23.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.689+0000 7f1beadafb80 1 --2- 172.21.15.44:0/1363059761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a8a930 0x564244a8ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.689+0000 7f1beadafb80 1 --2- 172.21.15.44:0/1363059761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a85be0 0x564244a85fb0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.689+0000 7f1beadafb80 1 --2- 172.21.15.44:0/1363059761 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a864f0 0x564244a8a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.689+0000 7f1beadafb80 1 -- 172.21.15.44:0/1363059761 >> 172.21.15.44:0/1363059761 conn(0x564244a7b3e0 msgr2=0x564244a80bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:23.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.690+0000 7f1beadafb80 1 -- 172.21.15.44:0/1363059761 shutdown_connections 2023-11-16T16:56:23.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.690+0000 7f1beadafb80 1 -- 172.21.15.44:0/1363059761 wait complete. 2023-11-16T16:56:23.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.692+0000 7f1beadafb80 1 -- start start 2023-11-16T16:56:23.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.692+0000 7f1beadafb80 1 Processor -- start 2023-11-16T16:56:23.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.693+0000 7f1beadafb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a85be0 0x564244a8f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.696 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.693+0000 7f1beadafb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a864f0 0x564244a8f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.693+0000 7f1beadafb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a8a930 0x564244a8fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.693+0000 7f1beadafb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x564244978960 con 0x564244a8a930 2023-11-16T16:56:23.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.693+0000 7f1beadafb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x564244978600 con 0x564244a85be0 2023-11-16T16:56:23.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.693+0000 7f1beadafb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56424485a9a0 con 0x564244a864f0 2023-11-16T16:56:23.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.693+0000 7f1beadafb80 10 client.0 _ll_register_callbacks cb 0x564244996800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:23.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.693+0000 7f1bd519b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a85be0 0x564244a8f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.693+0000 7f1bd599c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a8a930 0x564244a8fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.693+0000 7f1bd519b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a85be0 0x564244a8f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:51680/0 (socket says 172.21.15.44:51680) 2023-11-16T16:56:23.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.693+0000 7f1bd519b700 1 -- 172.21.15.44:0/2321520213 learned_addr learned my addr 172.21.15.44:0/2321520213 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:23.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.694+0000 7f1bd499a700 1 --2- 172.21.15.44:0/2321520213 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a864f0 0x564244a8f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.694+0000 7f1bd519b700 1 -- 172.21.15.44:0/2321520213 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a864f0 msgr2=0x564244a8f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:23.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.694+0000 7f1bd519b700 1 --2- 172.21.15.44:0/2321520213 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a864f0 0x564244a8f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.694+0000 7f1bd519b700 1 -- 172.21.15.44:0/2321520213 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a8a930 msgr2=0x564244a8fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:23.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.694+0000 7f1bd519b700 1 --2- 172.21.15.44:0/2321520213 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a8a930 0x564244a8fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.694+0000 7f1bd519b700 1 -- 172.21.15.44:0/2321520213 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd0112430 con 0x564244a85be0 2023-11-16T16:56:23.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.694+0000 7f1bd519b700 1 --2- 172.21.15.44:0/2321520213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a85be0 0x564244a8f3d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc099fc0 tx=0x7f1bcc0a2600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee2bbc8aba2325f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:23.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.694+0000 7f1bb8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:56:23.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.694+0000 7f1bb8ff9700 1 -- 172.21.15.44:0/2321520213 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc099730 con 0x564244a85be0 2023-11-16T16:56:23.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.695+0000 7f1bb8ff9700 1 -- 172.21.15.44:0/2321520213 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f1bcc0b82a0 con 0x564244a85be0 2023-11-16T16:56:23.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.695+0000 7f1beadafb80 1 -- 172.21.15.44:0/2321520213 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564244a78ba0 con 0x564244a85be0 2023-11-16T16:56:23.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.695+0000 7f1bb8ff9700 1 -- 172.21.15.44:0/2321520213 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bcc0b8460 con 0x564244a85be0 2023-11-16T16:56:23.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.695+0000 7f1beadafb80 10 client.26756 fetch_fsmap learned FSMap version 21 2023-11-16T16:56:23.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.695+0000 7f1beadafb80 1 -- 172.21.15.44:0/2321520213 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x56424499d210 con 0x564244a85be0 2023-11-16T16:56:23.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.695+0000 7f1bb8ff9700 1 -- 172.21.15.44:0/2321520213 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564244a78ba0 con 0x564244a85be0 2023-11-16T16:56:23.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.695+0000 7f1bb8ff9700 1 -- 172.21.15.44:0/2321520213 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1bcc095e20 con 0x564244a85be0 2023-11-16T16:56:23.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.695+0000 7f1beadafb80 10 client.26756 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:56:23.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.695+0000 7f1beadafb80 10 client.26756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:23.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.695+0000 7f1beadafb80 20 client.26756 populate_metadata read hostname 'smithi044' 2023-11-16T16:56:23.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.695+0000 7f1beadafb80 1 --2- 172.21.15.44:0/2321520213 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564244a9f4e0 0x564244aa18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.696+0000 7f1beadafb80 4 client.26756 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:56:23.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.696+0000 7f1beadafb80 1 -- 172.21.15.44:0/2321520213 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564244999120 con 0x564244a9f4e0 2023-11-16T16:56:23.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.696+0000 7f1bd499a700 1 --2- 172.21.15.44:0/2321520213 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564244a9f4e0 0x564244aa18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.697+0000 7f1bd499a700 1 --2- 172.21.15.44:0/2321520213 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564244a9f4e0 0x564244aa18f0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfa7eca454a566d5 server_cookie=22032378700f4edf in_seq=0 out_seq=0 2023-11-16T16:56:23.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.697+0000 7f1bb8ff9700 10 client.26756 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:23.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.698+0000 7f1bb8ff9700 1 -- 172.21.15.44:0/2321520213 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x564244999120 con 0x564244a9f4e0 2023-11-16T16:56:23.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.698+0000 7f1bb8ff9700 10 client.26756 handle_command_reply: tid=0 2023-11-16T16:56:23.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.698+0000 7f1beadafb80 1 client.26756 shutdown 2023-11-16T16:56:23.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.698+0000 7f1beadafb80 1 -- 172.21.15.44:0/2321520213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a85be0 msgr2=0x564244a8f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:23.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.698+0000 7f1beadafb80 1 --2- 172.21.15.44:0/2321520213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a85be0 0x564244a8f3d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc099fc0 tx=0x7f1bcc0a2600 comp rx=0 tx=0).stop 2023-11-16T16:56:23.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.698+0000 7f1beadafb80 1 -- 172.21.15.44:0/2321520213 shutdown_connections 2023-11-16T16:56:23.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.698+0000 7f1beadafb80 1 --2- 172.21.15.44:0/2321520213 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564244a9f4e0 0x564244aa18f0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.698+0000 7f1beadafb80 1 --2- 172.21.15.44:0/2321520213 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a8a930 0x564244a8fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.698+0000 7f1beadafb80 1 --2- 172.21.15.44:0/2321520213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a85be0 0x564244a8f3d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.698+0000 7f1bb8ff9700 0 client.26756 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:23.702 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.698+0000 7f1beadafb80 1 --2- 172.21.15.44:0/2321520213 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a864f0 0x564244a8f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.702 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.698+0000 7f1beadafb80 1 -- 172.21.15.44:0/2321520213 >> 172.21.15.44:0/2321520213 conn(0x564244a7b3e0 msgr2=0x564244a8bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:23.702 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.698+0000 7f1beadafb80 1 -- 172.21.15.44:0/2321520213 shutdown_connections 2023-11-16T16:56:23.702 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.698+0000 7f1beadafb80 1 -- 172.21.15.44:0/2321520213 wait complete. 2023-11-16T16:56:23.702 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.699+0000 7f1beadafb80 20 client.26756 trim_cache size 0 max 16384 2023-11-16T16:56:23.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.718+0000 7f1beadafb80 1 Processor -- start 2023-11-16T16:56:23.719 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.719+0000 7f1beadafb80 1 -- start start 2023-11-16T16:56:23.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.719+0000 7f1beadafb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a85be0 0x564244a88220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.719+0000 7f1beadafb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a864f0 0x564244a969c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.719+0000 7f1beadafb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a8a930 0x564244a8a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.719+0000 7f1beadafb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x564244a90860 con 0x564244a85be0 2023-11-16T16:56:23.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.719+0000 7f1beadafb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x564244a96f00 con 0x564244a8a930 2023-11-16T16:56:23.720 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.719+0000 7f1beadafb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x564244a8a630 con 0x564244a864f0 2023-11-16T16:56:23.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.719+0000 7f1bd499a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a864f0 0x564244a969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.719+0000 7f1bd499a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a864f0 0x564244a969c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:59408/0 (socket says 172.21.15.44:59408) 2023-11-16T16:56:23.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.719+0000 7f1bd499a700 1 -- 172.21.15.44:0/2608094464 learned_addr learned my addr 172.21.15.44:0/2608094464 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:23.721 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.719+0000 7f1bd599c700 1 --2- 172.21.15.44:0/2608094464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a8a930 0x564244a8a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.720+0000 7f1bd499a700 1 -- 172.21.15.44:0/2608094464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a8a930 msgr2=0x564244a8a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.720+0000 7f1bd499a700 1 --2- 172.21.15.44:0/2608094464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a8a930 0x564244a8a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.720+0000 7f1bd499a700 1 -- 172.21.15.44:0/2608094464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a85be0 msgr2=0x564244a88220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.720+0000 7f1bd499a700 1 --2- 172.21.15.44:0/2608094464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a85be0 0x564244a88220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.720+0000 7f1bd499a700 1 -- 172.21.15.44:0/2608094464 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564244a8ef40 con 0x564244a864f0 2023-11-16T16:56:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.720+0000 7f1bd499a700 1 --2- 172.21.15.44:0/2608094464 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a864f0 0x564244a969c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1bd006a120 tx=0x7f1bc0014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ea25fdaf1fe2df9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.720+0000 7f1bbbfff700 1 -- 172.21.15.44:0/2608094464 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0024040 con 0x564244a864f0 2023-11-16T16:56:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.720+0000 7f1bbbfff700 1 -- 172.21.15.44:0/2608094464 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1bc0004500 con 0x564244a864f0 2023-11-16T16:56:23.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.721+0000 7f1bbbfff700 1 -- 172.21.15.44:0/2608094464 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc001f430 con 0x564244a864f0 2023-11-16T16:56:23.723 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.721+0000 7f1beadafb80 1 -- 172.21.15.44:0/2608094464 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a864f0 msgr2=0x564244a969c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:23.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.721+0000 7f1beadafb80 1 --2- 172.21.15.44:0/2608094464 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a864f0 0x564244a969c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1bd006a120 tx=0x7f1bc0014000 comp rx=0 tx=0).stop 2023-11-16T16:56:23.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.722+0000 7f1beadafb80 1 -- 172.21.15.44:0/2608094464 shutdown_connections 2023-11-16T16:56:23.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.722+0000 7f1beadafb80 1 --2- 172.21.15.44:0/2608094464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a8a930 0x564244a8a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.722+0000 7f1beadafb80 1 --2- 172.21.15.44:0/2608094464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a85be0 0x564244a88220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.722+0000 7f1beadafb80 1 --2- 172.21.15.44:0/2608094464 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a864f0 0x564244a969c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.722+0000 7f1beadafb80 1 -- 172.21.15.44:0/2608094464 >> 172.21.15.44:0/2608094464 conn(0x564244a7b3e0 msgr2=0x564244a899e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:23.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.722+0000 7f1beadafb80 1 -- 172.21.15.44:0/2608094464 shutdown_connections 2023-11-16T16:56:23.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.722+0000 7f1beadafb80 1 -- 172.21.15.44:0/2608094464 wait complete. 2023-11-16T16:56:23.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.722+0000 7f1beadafb80 1 -- start start 2023-11-16T16:56:23.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.723+0000 7f1beadafb80 1 Processor -- start 2023-11-16T16:56:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.723+0000 7f1beadafb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a85be0 0x564244aed7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.723+0000 7f1beadafb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a864f0 0x564244aedd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.724+0000 7f1beadafb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a8a930 0x564244a956f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.724+0000 7f1beadafb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x564244a95cf0 con 0x564244a85be0 2023-11-16T16:56:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.724+0000 7f1beadafb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x564244a95e60 con 0x564244a864f0 2023-11-16T16:56:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.724+0000 7f1beadafb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x564244a96020 con 0x564244a8a930 2023-11-16T16:56:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.724+0000 7f1beadafb80 10 client.0 _ll_register_callbacks cb 0x7f1bcc099a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.724+0000 7f1bd519b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a85be0 0x564244aed7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.724+0000 7f1bd519b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a85be0 0x564244aed7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33860/0 (socket says 172.21.15.44:33860) 2023-11-16T16:56:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.724+0000 7f1bd519b700 1 -- 172.21.15.44:0/3510733368 learned_addr learned my addr 172.21.15.44:0/3510733368 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.724+0000 7f1bd599c700 1 --2- 172.21.15.44:0/3510733368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a8a930 0x564244a956f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.724+0000 7f1bd499a700 1 --2- 172.21.15.44:0/3510733368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a864f0 0x564244aedd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.724+0000 7f1bd519b700 1 -- 172.21.15.44:0/3510733368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a8a930 msgr2=0x564244a956f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.724+0000 7f1bd519b700 1 --2- 172.21.15.44:0/3510733368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a8a930 0x564244a956f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.724+0000 7f1bd519b700 1 -- 172.21.15.44:0/3510733368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a864f0 msgr2=0x564244aedd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:23.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.724+0000 7f1bd519b700 1 --2- 172.21.15.44:0/3510733368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a864f0 0x564244aedd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:23.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.724+0000 7f1bd519b700 1 -- 172.21.15.44:0/3510733368 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc0017070 con 0x564244a85be0 2023-11-16T16:56:23.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.725+0000 7f1bd499a700 1 --2- 172.21.15.44:0/3510733368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a864f0 0x564244aedd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:56:23.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.725+0000 7f1bd519b700 1 --2- 172.21.15.44:0/3510733368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a85be0 0x564244aed7e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc096aa0 tx=0x7f1bcc010a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=304442e1e1f5b023 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:23.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.725+0000 7f1bb8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:56:23.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.725+0000 7f1bb8ff9700 1 -- 172.21.15.44:0/3510733368 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc05fea0 con 0x564244a85be0 2023-11-16T16:56:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.725+0000 7f1bb8ff9700 1 -- 172.21.15.44:0/3510733368 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f1bcc0b2a10 con 0x564244a85be0 2023-11-16T16:56:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.725+0000 7f1beadafb80 1 -- 172.21.15.44:0/3510733368 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1bcc010710 con 0x564244a85be0 2023-11-16T16:56:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.726+0000 7f1bb8ff9700 1 -- 172.21.15.44:0/3510733368 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bcc0b2c10 con 0x564244a85be0 2023-11-16T16:56:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.726+0000 7f1beadafb80 10 client.16713 fetch_fsmap learned FSMap version 21 2023-11-16T16:56:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.726+0000 7f1beadafb80 1 -- 172.21.15.44:0/3510733368 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x564244804e60 con 0x564244a85be0 2023-11-16T16:56:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.726+0000 7f1bb8ff9700 1 -- 172.21.15.44:0/3510733368 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1bcc010710 con 0x564244a85be0 2023-11-16T16:56:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.726+0000 7f1bb8ff9700 1 -- 172.21.15.44:0/3510733368 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1bcc0ca1a0 con 0x564244a85be0 2023-11-16T16:56:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.726+0000 7f1beadafb80 10 client.16713 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:56:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.726+0000 7f1beadafb80 10 client.16713 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.726+0000 7f1beadafb80 20 client.16713 populate_metadata read hostname 'smithi044' 2023-11-16T16:56:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.726+0000 7f1beadafb80 1 --2- 172.21.15.44:0/3510733368 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564244afdac0 0x564244affed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.726+0000 7f1beadafb80 4 client.16713 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:56:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.726+0000 7f1beadafb80 1 -- 172.21.15.44:0/3510733368 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5642449a56e0 con 0x564244afdac0 2023-11-16T16:56:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.726+0000 7f1bd499a700 1 --2- 172.21.15.44:0/3510733368 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564244afdac0 0x564244affed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.727+0000 7f1bd499a700 1 --2- 172.21.15.44:0/3510733368 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564244afdac0 0x564244affed0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be8852255573bff8 server_cookie=43b63409cf4f823 in_seq=0 out_seq=0 2023-11-16T16:56:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:23.727+0000 7f1bb8ff9700 10 client.16713 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:23 smithi044 ceph-mon[128750]: pgmap v197: 129 pgs: 129 active+clean; 66 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 702 op/s 2023-11-16T16:56:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:23 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:56:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:23 smithi044 ceph-mon[128750]: osdmap e94: 12 total, 12 up, 12 in 2023-11-16T16:56:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:23 smithi176 ceph-mon[131630]: pgmap v197: 129 pgs: 129 active+clean; 66 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 702 op/s 2023-11-16T16:56:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:23 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:56:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:23 smithi176 ceph-mon[131630]: osdmap e94: 12 total, 12 up, 12 in 2023-11-16T16:56:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:23 smithi077 ceph-mon[131204]: pgmap v197: 129 pgs: 129 active+clean; 66 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 702 op/s 2023-11-16T16:56:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:23 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:56:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:23 smithi077 ceph-mon[131204]: osdmap e94: 12 total, 12 up, 12 in 2023-11-16T16:56:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.737+0000 7f1bb8ff9700 1 -- 172.21.15.44:0/3510733368 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5642449a56e0 con 0x564244afdac0 2023-11-16T16:56:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.737+0000 7f1bb8ff9700 10 client.16713 handle_command_reply: tid=0 2023-11-16T16:56:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.737+0000 7f1beadafb80 1 client.16713 shutdown 2023-11-16T16:56:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.737+0000 7f1beadafb80 1 -- 172.21.15.44:0/3510733368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a85be0 msgr2=0x564244aed7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.737+0000 7f1beadafb80 1 --2- 172.21.15.44:0/3510733368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a85be0 0x564244aed7e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc096aa0 tx=0x7f1bcc010a30 comp rx=0 tx=0).stop 2023-11-16T16:56:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.738+0000 7f1beadafb80 1 -- 172.21.15.44:0/3510733368 shutdown_connections 2023-11-16T16:56:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.738+0000 7f1beadafb80 1 --2- 172.21.15.44:0/3510733368 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564244afdac0 0x564244affed0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.738+0000 7f1beadafb80 1 --2- 172.21.15.44:0/3510733368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564244a8a930 0x564244a956f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.738+0000 7f1bb8ff9700 0 client.16713 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.738+0000 7f1beadafb80 1 --2- 172.21.15.44:0/3510733368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564244a85be0 0x564244aed7e0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.738+0000 7f1beadafb80 1 --2- 172.21.15.44:0/3510733368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564244a864f0 0x564244aedd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.738+0000 7f1beadafb80 1 -- 172.21.15.44:0/3510733368 >> 172.21.15.44:0/3510733368 conn(0x564244a7b3e0 msgr2=0x564244a849e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.738+0000 7f1beadafb80 1 -- 172.21.15.44:0/3510733368 shutdown_connections 2023-11-16T16:56:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.738+0000 7f1beadafb80 1 -- 172.21.15.44:0/3510733368 wait complete. 2023-11-16T16:56:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.738+0000 7f1beadafb80 20 client.16713 trim_cache size 0 max 16384 2023-11-16T16:56:24.740 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:56:24.741 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:56:24.741 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "58fab0cd-3198-4023-82f3-75a84820f59b", 2023-11-16T16:56:24.741 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:56:24.741 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:56:24.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.739+0000 7f1bd73ff700 1 -- 172.21.15.44:0/3501341290 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1bbc0a12c0 msgr2=0x7f1bbc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:24.741 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.739+0000 7f1bd73ff700 1 --2- 172.21.15.44:0/3501341290 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1bbc0a12c0 0x7f1bbc0a3770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0199b60 tx=0x7f1bc0009310 comp rx=0 tx=0).stop 2023-11-16T16:56:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.739+0000 7f1bd73ff700 1 -- 172.21.15.44:0/3501341290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bd00746f0 msgr2=0x7f1bd019e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.739+0000 7f1bd73ff700 1 --2- 172.21.15.44:0/3501341290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bd00746f0 0x7f1bd019e680 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00e9a0 tx=0x7f1bcc00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:56:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.739+0000 7f1bd73ff700 1 -- 172.21.15.44:0/3501341290 shutdown_connections 2023-11-16T16:56:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.739+0000 7f1bd73ff700 1 --2- 172.21.15.44:0/3501341290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd010cec0 0x7f1bd01986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.739+0000 7f1bd73ff700 1 --2- 172.21.15.44:0/3501341290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bd00746f0 0x7f1bd019e680 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.739+0000 7f1bd73ff700 1 --2- 172.21.15.44:0/3501341290 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f1bbc0a12c0 0x7f1bbc0a3770 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.739+0000 7f1bd73ff700 1 --2- 172.21.15.44:0/3501341290 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bd0074ac0 0x7f1bd019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.739+0000 7f1bd73ff700 1 -- 172.21.15.44:0/3501341290 >> 172.21.15.44:0/3501341290 conn(0x7f1bd0077560 msgr2=0x7f1bd0104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.739+0000 7f1bd73ff700 1 -- 172.21.15.44:0/3501341290 shutdown_connections 2023-11-16T16:56:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:24.739+0000 7f1bd73ff700 1 -- 172.21.15.44:0/3501341290 wait complete. 2023-11-16T16:56:24.776 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:56:24.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:24 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:56:24.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:24 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:56:24.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:24 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:56:24.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:24 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:56:24.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:24 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:56:24.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:24 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:56:25.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.043+0000 7f4a1319d700 1 -- 172.21.15.44:0/1333908579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c108070 msgr2=0x7f4a0c1084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:25.044 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.043+0000 7f4a1319d700 1 --2- 172.21.15.44:0/1333908579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c108070 0x7f4a0c1084e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f49fc009990 tx=0x7f49fc009ca0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.044+0000 7f4a1319d700 1 -- 172.21.15.44:0/1333908579 shutdown_connections 2023-11-16T16:56:25.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.044+0000 7f4a1319d700 1 --2- 172.21.15.44:0/1333908579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a0c112e70 0x7f4a0c115250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.044+0000 7f4a1319d700 1 --2- 172.21.15.44:0/1333908579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a0c06cd80 0x7f4a0c107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.044+0000 7f4a1319d700 1 --2- 172.21.15.44:0/1333908579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c108070 0x7f4a0c1084e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.045 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.044+0000 7f4a1319d700 1 -- 172.21.15.44:0/1333908579 >> 172.21.15.44:0/1333908579 conn(0x7f4a0c06b400 msgr2=0x7f4a0c0714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:25.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.044+0000 7f4a1319d700 1 -- 172.21.15.44:0/1333908579 shutdown_connections 2023-11-16T16:56:25.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.045+0000 7f4a1319d700 1 -- 172.21.15.44:0/1333908579 wait complete. 2023-11-16T16:56:25.048 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.047+0000 7f4a1319d700 1 Processor -- start 2023-11-16T16:56:25.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.048+0000 7f4a1319d700 1 -- start start 2023-11-16T16:56:25.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.048+0000 7f4a1319d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a0c06cd80 0x7f4a0c1a9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.048+0000 7f4a1319d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c108070 0x7f4a0c1aa150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.049+0000 7f4a1319d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a0c112e70 0x7f4a0c111510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.049+0000 7f4a1319d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0c111ce0 con 0x7f4a0c06cd80 2023-11-16T16:56:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.049+0000 7f4a1319d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0c111e50 con 0x7f4a0c108070 2023-11-16T16:56:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.049+0000 7f4a1319d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0c112010 con 0x7f4a0c112e70 2023-11-16T16:56:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.049+0000 7f4a1199a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c108070 0x7f4a0c1aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.049+0000 7f4a1199a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c108070 0x7f4a0c1aa150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:45686/0 (socket says 172.21.15.44:45686) 2023-11-16T16:56:25.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.049+0000 7f4a1199a700 1 -- 172.21.15.44:0/2223803985 learned_addr learned my addr 172.21.15.44:0/2223803985 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:25.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.050+0000 7f4a1299c700 1 --2- 172.21.15.44:0/2223803985 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a0c112e70 0x7f4a0c111510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.051+0000 7f4a1219b700 1 --2- 172.21.15.44:0/2223803985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a0c06cd80 0x7f4a0c1a9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.051+0000 7f4a1299c700 1 -- 172.21.15.44:0/2223803985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c108070 msgr2=0x7f4a0c1aa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.052+0000 7f4a1299c700 1 --2- 172.21.15.44:0/2223803985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c108070 0x7f4a0c1aa150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.052+0000 7f4a1299c700 1 -- 172.21.15.44:0/2223803985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a0c06cd80 msgr2=0x7f4a0c1a9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.052+0000 7f4a1299c700 1 --2- 172.21.15.44:0/2223803985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a0c06cd80 0x7f4a0c1a9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.052+0000 7f4a1299c700 1 -- 172.21.15.44:0/2223803985 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49fc0095f0 con 0x7f4a0c112e70 2023-11-16T16:56:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.052+0000 7f4a1199a700 1 --2- 172.21.15.44:0/2223803985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c108070 0x7f4a0c1aa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:56:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.053+0000 7f4a1299c700 1 --2- 172.21.15.44:0/2223803985 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a0c112e70 0x7f4a0c111510 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400bdf0 tx=0x7f4a04009ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.054+0000 7f4a037fe700 1 -- 172.21.15.44:0/2223803985 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0400cc10 con 0x7f4a0c112e70 2023-11-16T16:56:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.054+0000 7f4a037fe700 1 -- 172.21.15.44:0/2223803985 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a0400cd70 con 0x7f4a0c112e70 2023-11-16T16:56:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.054+0000 7f4a037fe700 1 -- 172.21.15.44:0/2223803985 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04012650 con 0x7f4a0c112e70 2023-11-16T16:56:25.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.055+0000 7f4a1319d700 1 -- 172.21.15.44:0/2223803985 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a0c118900 con 0x7f4a0c112e70 2023-11-16T16:56:25.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.055+0000 7f4a1319d700 1 -- 172.21.15.44:0/2223803985 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a0c118770 con 0x7f4a0c112e70 2023-11-16T16:56:25.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.065+0000 7f4a037fe700 1 -- 172.21.15.44:0/2223803985 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f4a04014440 con 0x7f4a0c112e70 2023-11-16T16:56:25.068 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.067+0000 7f4a037fe700 1 --2- 172.21.15.44:0/2223803985 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f49f80a1340 0x7f49f80a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.067+0000 7f4a037fe700 1 -- 172.21.15.44:0/2223803985 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f4a04006e40 con 0x7f4a0c112e70 2023-11-16T16:56:25.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.068+0000 7f4a1219b700 1 --2- 172.21.15.44:0/2223803985 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f49f80a1340 0x7f49f80a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.070+0000 7f4a27d52b80 1 Processor -- start 2023-11-16T16:56:25.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.071+0000 7f4a1219b700 1 --2- 172.21.15.44:0/2223803985 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f49f80a1340 0x7f49f80a37f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4a08005ea0 tx=0x7f4a08005e30 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:25.074 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.072+0000 7f4a27d52b80 1 -- start start 2023-11-16T16:56:25.074 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.073+0000 7f4a27d52b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfa8d0 0x564fdcdfaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.074 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.073+0000 7f4a27d52b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.074 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.073+0000 7f4a27d52b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 0x564fdcdf5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.074+0000 7f4a1199a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.074+0000 7f4a1199a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:45690/0 (socket says 172.21.15.44:45690) 2023-11-16T16:56:25.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.074+0000 7f4a1199a700 1 -- 172.21.15.44:0/3649632931 learned_addr learned my addr 172.21.15.44:0/3649632931 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:25.076 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.075+0000 7f4a1219b700 1 --2- 172.21.15.44:0/3649632931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfa8d0 0x564fdcdfaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.076 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.075+0000 7f4a27d52b80 1 -- 172.21.15.44:0/3649632931 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fdccfa060 con 0x564fdcdfa8d0 2023-11-16T16:56:25.077 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.075+0000 7f4a27d52b80 1 -- 172.21.15.44:0/3649632931 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fdcd4c8a0 con 0x564fdcdfb1e0 2023-11-16T16:56:25.077 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.075+0000 7f4a27d52b80 1 -- 172.21.15.44:0/3649632931 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fdcd45d30 con 0x564fdcdf5100 2023-11-16T16:56:25.077 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.075+0000 7f4a1199a700 1 -- 172.21.15.44:0/3649632931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 msgr2=0x564fdcdf5530 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T16:56:25.077 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.076+0000 7f4a1299c700 1 --2- 172.21.15.44:0/3649632931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 0x564fdcdf5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.078 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.076+0000 7f4a1199a700 1 --2- 172.21.15.44:0/3649632931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 0x564fdcdf5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.076+0000 7f4a1199a700 1 -- 172.21.15.44:0/3649632931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfa8d0 msgr2=0x564fdcdfaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:25.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.076+0000 7f4a1199a700 1 --2- 172.21.15.44:0/3649632931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfa8d0 0x564fdcdfaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.076+0000 7f4a1199a700 1 -- 172.21.15.44:0/3649632931 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564fdcbc7630 con 0x564fdcdfb1e0 2023-11-16T16:56:25.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.077+0000 7f4a1199a700 1 --2- 172.21.15.44:0/3649632931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf4bc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00bb80 tx=0x7f49fc00bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3237cecb92f4a022 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:25.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.077+0000 7f4a00bfe700 1 -- 172.21.15.44:0/3649632931 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc01d050 con 0x564fdcdfb1e0 2023-11-16T16:56:25.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.078+0000 7f4a00bfe700 1 -- 172.21.15.44:0/3649632931 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49fc01e040 con 0x564fdcdfb1e0 2023-11-16T16:56:25.079 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.078+0000 7f4a00bfe700 1 -- 172.21.15.44:0/3649632931 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc023460 con 0x564fdcdfb1e0 2023-11-16T16:56:25.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.079+0000 7f4a27d52b80 1 -- 172.21.15.44:0/3649632931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfb1e0 msgr2=0x564fdcdf4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:25.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.079+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/3649632931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf4bc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00bb80 tx=0x7f49fc00bbb0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.079+0000 7f4a27d52b80 1 -- 172.21.15.44:0/3649632931 shutdown_connections 2023-11-16T16:56:25.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.079+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/3649632931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 0x564fdcdf5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.079+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/3649632931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfa8d0 0x564fdcdfaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.079+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/3649632931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf4bc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.079+0000 7f4a27d52b80 1 -- 172.21.15.44:0/3649632931 >> 172.21.15.44:0/3649632931 conn(0x564fdcdea5b0 msgr2=0x564fdcdefad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:25.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.079+0000 7f4a27d52b80 1 -- 172.21.15.44:0/3649632931 shutdown_connections 2023-11-16T16:56:25.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.079+0000 7f4a27d52b80 1 -- 172.21.15.44:0/3649632931 wait complete. 2023-11-16T16:56:25.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.080+0000 7f4a27d52b80 1 -- start start 2023-11-16T16:56:25.081 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.080+0000 7f4a27d52b80 1 Processor -- start 2023-11-16T16:56:25.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.081+0000 7f4a27d52b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 0x564fdcdfe470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.082+0000 7f4a27d52b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfa8d0 0x564fdcdfe9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.082+0000 7f4a27d52b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfb1e0 0x564fdcdfeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.082+0000 7f4a1219b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 0x564fdcdfe470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.082+0000 7f4a1219b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 0x564fdcdfe470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:56898/0 (socket says 172.21.15.44:56898) 2023-11-16T16:56:25.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.082+0000 7f4a1219b700 1 -- 172.21.15.44:0/1453880569 learned_addr learned my addr 172.21.15.44:0/1453880569 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:25.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.082+0000 7f4a27d52b80 1 -- 172.21.15.44:0/1453880569 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fdcce78e0 con 0x564fdcdfb1e0 2023-11-16T16:56:25.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.082+0000 7f4a27d52b80 1 -- 172.21.15.44:0/1453880569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fdcce7580 con 0x564fdcdfa8d0 2023-11-16T16:56:25.083 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.082+0000 7f4a27d52b80 1 -- 172.21.15.44:0/1453880569 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fdcbc9920 con 0x564fdcdf5100 2023-11-16T16:56:25.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.082+0000 7f4a27d52b80 10 client.0 _ll_register_callbacks cb 0x564fdcd05780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:25.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.082+0000 7f4a1199a700 1 --2- 172.21.15.44:0/1453880569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfa8d0 0x564fdcdfe9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.083+0000 7f4a1299c700 1 --2- 172.21.15.44:0/1453880569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfb1e0 0x564fdcdfeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.085 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.084+0000 7f4a1199a700 1 -- 172.21.15.44:0/1453880569 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 msgr2=0x564fdcdfe470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:25.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.084+0000 7f4a1199a700 1 --2- 172.21.15.44:0/1453880569 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 0x564fdcdfe470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.084+0000 7f4a1199a700 1 -- 172.21.15.44:0/1453880569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfb1e0 msgr2=0x564fdcdfeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:25.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.084+0000 7f4a1199a700 1 --2- 172.21.15.44:0/1453880569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfb1e0 0x564fdcdfeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.084+0000 7f4a1199a700 1 -- 172.21.15.44:0/1453880569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a0c115ec0 con 0x564fdcdfa8d0 2023-11-16T16:56:25.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.084+0000 7f4a1219b700 1 --2- 172.21.15.44:0/1453880569 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 0x564fdcdfe470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:56:25.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.085+0000 7f4a1199a700 1 --2- 172.21.15.44:0/1453880569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfa8d0 0x564fdcdfe9b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00bb80 tx=0x7f49fc004a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c37d9a6dc858052a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:25.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.085+0000 7f49f5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:56:25.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.085+0000 7f49f5ffb700 1 -- 172.21.15.44:0/1453880569 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc005910 con 0x564fdcdfa8d0 2023-11-16T16:56:25.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.085+0000 7f4a27d52b80 1 -- 172.21.15.44:0/1453880569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564fdcde7d70 con 0x564fdcdfa8d0 2023-11-16T16:56:25.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.086+0000 7f49f5ffb700 1 -- 172.21.15.44:0/1453880569 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f49fc00be80 con 0x564fdcdfa8d0 2023-11-16T16:56:25.088 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.086+0000 7f49f5ffb700 1 -- 172.21.15.44:0/1453880569 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49fc005910 con 0x564fdcdfa8d0 2023-11-16T16:56:25.091 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.088+0000 7f4a27d52b80 10 client.26774 fetch_fsmap learned FSMap version 21 2023-11-16T16:56:25.091 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.088+0000 7f4a27d52b80 1 -- 172.21.15.44:0/1453880569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x564fdcd0c190 con 0x564fdcdfa8d0 2023-11-16T16:56:25.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.088+0000 7f49f5ffb700 1 -- 172.21.15.44:0/1453880569 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564fdcde7d70 con 0x564fdcdfa8d0 2023-11-16T16:56:25.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.089+0000 7f49f5ffb700 1 -- 172.21.15.44:0/1453880569 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f49fc02d560 con 0x564fdcdfa8d0 2023-11-16T16:56:25.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.089+0000 7f4a27d52b80 10 client.26774 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:56:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.089+0000 7f4a27d52b80 10 client.26774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.089+0000 7f4a27d52b80 20 client.26774 populate_metadata read hostname 'smithi044' 2023-11-16T16:56:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.089+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/1453880569 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564fdce0e600 0x564fdce10a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.089+0000 7f4a27d52b80 4 client.26774 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:56:25.094 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.089+0000 7f4a27d52b80 1 -- 172.21.15.44:0/1453880569 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564fdcd080a0 con 0x564fdce0e600 2023-11-16T16:56:25.094 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.089+0000 7f4a1219b700 1 --2- 172.21.15.44:0/1453880569 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564fdce0e600 0x564fdce10a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.094 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.090+0000 7f4a1219b700 1 --2- 172.21.15.44:0/1453880569 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564fdce0e600 0x564fdce10a10 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89eb37a03b774d43 server_cookie=9fce0f84a383fae9 in_seq=0 out_seq=0 2023-11-16T16:56:25.094 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.090+0000 7f49f5ffb700 10 client.26774 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:25.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.093+0000 7f49f5ffb700 1 -- 172.21.15.44:0/1453880569 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x564fdcd080a0 con 0x564fdce0e600 2023-11-16T16:56:25.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.093+0000 7f49f5ffb700 10 client.26774 handle_command_reply: tid=0 2023-11-16T16:56:25.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.094+0000 7f4a27d52b80 1 client.26774 shutdown 2023-11-16T16:56:25.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.094+0000 7f4a27d52b80 1 -- 172.21.15.44:0/1453880569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfa8d0 msgr2=0x564fdcdfe9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:25.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.094+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/1453880569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfa8d0 0x564fdcdfe9b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00bb80 tx=0x7f49fc004a10 comp rx=0 tx=0).stop 2023-11-16T16:56:25.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.095+0000 7f4a27d52b80 1 -- 172.21.15.44:0/1453880569 shutdown_connections 2023-11-16T16:56:25.098 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.095+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/1453880569 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564fdce0e600 0x564fdce10a10 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.098 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.095+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/1453880569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfb1e0 0x564fdcdfeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.098 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.095+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/1453880569 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 0x564fdcdfe470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.098 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.095+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/1453880569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfa8d0 0x564fdcdfe9b0 secure :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00bb80 tx=0x7f49fc004a10 comp rx=0 tx=0).stop 2023-11-16T16:56:25.098 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.095+0000 7f4a27d52b80 1 -- 172.21.15.44:0/1453880569 >> 172.21.15.44:0/1453880569 conn(0x564fdcdea5b0 msgr2=0x564fdcdef030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:25.099 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.095+0000 7f49f5ffb700 0 client.26774 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:25.100 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.096+0000 7f4a27d52b80 1 -- 172.21.15.44:0/1453880569 shutdown_connections 2023-11-16T16:56:25.100 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.097+0000 7f4a27d52b80 1 -- 172.21.15.44:0/1453880569 wait complete. 2023-11-16T16:56:25.100 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.098+0000 7f4a27d52b80 20 client.26774 trim_cache size 0 max 16384 2023-11-16T16:56:25.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:24 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:56:25.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:24 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:56:25.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:24 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:56:25.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.144+0000 7f4a27d52b80 1 Processor -- start 2023-11-16T16:56:25.145 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.144+0000 7f4a27d52b80 1 -- start start 2023-11-16T16:56:25.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.145+0000 7f4a27d52b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 0x564fdce05b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.145+0000 7f4a27d52b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.145+0000 7f4a27d52b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfc5c0 0x564fdcdef7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.145+0000 7f4a27d52b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fdcdf0bc0 con 0x564fdcdfc5c0 2023-11-16T16:56:25.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.145+0000 7f4a27d52b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fdce018f0 con 0x564fdcdfb1e0 2023-11-16T16:56:25.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.145+0000 7f4a27d52b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fdcdf0730 con 0x564fdcdf5100 2023-11-16T16:56:25.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.145+0000 7f4a1199a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.145+0000 7f4a1219b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 0x564fdce05b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.145+0000 7f4a1199a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:45710/0 (socket says 172.21.15.44:45710) 2023-11-16T16:56:25.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.145+0000 7f4a1199a700 1 -- 172.21.15.44:0/464479284 learned_addr learned my addr 172.21.15.44:0/464479284 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:25.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.146+0000 7f4a1199a700 1 -- 172.21.15.44:0/464479284 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 msgr2=0x564fdce05b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:25.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.146+0000 7f4a1199a700 1 --2- 172.21.15.44:0/464479284 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 0x564fdce05b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.146+0000 7f4a1199a700 1 -- 172.21.15.44:0/464479284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfc5c0 msgr2=0x564fdcdef7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:25.148 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.146+0000 7f4a1199a700 1 --2- 172.21.15.44:0/464479284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfc5c0 0x564fdcdef7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.148 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.146+0000 7f4a1199a700 1 -- 172.21.15.44:0/464479284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564fdcdfdfe0 con 0x564fdcdfb1e0 2023-11-16T16:56:25.148 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.147+0000 7f4a1199a700 1 --2- 172.21.15.44:0/464479284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf9c40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f49fc000c00 tx=0x7f49fc00bbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff231c3c379cf6d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:25.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.149+0000 7f4a013ff700 1 -- 172.21.15.44:0/464479284 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc01d040 con 0x564fdcdfb1e0 2023-11-16T16:56:25.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.149+0000 7f4a013ff700 1 -- 172.21.15.44:0/464479284 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49fc005040 con 0x564fdcdfb1e0 2023-11-16T16:56:25.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.149+0000 7f4a013ff700 1 -- 172.21.15.44:0/464479284 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc02d6a0 con 0x564fdcdfb1e0 2023-11-16T16:56:25.151 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.150+0000 7f4a27d52b80 1 -- 172.21.15.44:0/464479284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfb1e0 msgr2=0x564fdcdf9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:25.151 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.150+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/464479284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf9c40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f49fc000c00 tx=0x7f49fc00bbf0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.151 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.150+0000 7f4a27d52b80 1 -- 172.21.15.44:0/464479284 shutdown_connections 2023-11-16T16:56:25.151 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.150+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/464479284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfc5c0 0x564fdcdef7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.151 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.150+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/464479284 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdf5100 0x564fdce05b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.151 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.150+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/464479284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf9c40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.151 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.150+0000 7f4a27d52b80 1 -- 172.21.15.44:0/464479284 >> 172.21.15.44:0/464479284 conn(0x564fdcdea5b0 msgr2=0x564fdcdf9530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:25.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.153+0000 7f4a27d52b80 1 -- 172.21.15.44:0/464479284 shutdown_connections 2023-11-16T16:56:25.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.154+0000 7f4a27d52b80 1 -- 172.21.15.44:0/464479284 wait complete. 2023-11-16T16:56:25.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.155+0000 7f4a27d52b80 1 -- start start 2023-11-16T16:56:25.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.155+0000 7f4a27d52b80 1 Processor -- start 2023-11-16T16:56:25.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.156+0000 7f4a27d52b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdf5100 0x564fdcdf7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.157+0000 7f4a27d52b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.157+0000 7f4a27d52b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfc5c0 0x564fdcdf6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.157+0000 7f4a1199a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.157+0000 7f4a1199a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:56918/0 (socket says 172.21.15.44:56918) 2023-11-16T16:56:25.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.158+0000 7f4a1199a700 1 -- 172.21.15.44:0/465722971 learned_addr learned my addr 172.21.15.44:0/465722971 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:25.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.158+0000 7f4a27d52b80 1 -- 172.21.15.44:0/465722971 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fdcdf7ea0 con 0x564fdcdfc5c0 2023-11-16T16:56:25.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.158+0000 7f4a27d52b80 1 -- 172.21.15.44:0/465722971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fdcdf8010 con 0x564fdcdf5100 2023-11-16T16:56:25.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.158+0000 7f4a27d52b80 1 -- 172.21.15.44:0/465722971 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fdcdf6d90 con 0x564fdcdfb1e0 2023-11-16T16:56:25.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.158+0000 7f4a27d52b80 10 client.0 _ll_register_callbacks cb 0x7f4a08014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:25.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.158+0000 7f4a1219b700 1 --2- 172.21.15.44:0/465722971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdf5100 0x564fdcdf7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.160 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.158+0000 7f4a1299c700 1 --2- 172.21.15.44:0/465722971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfc5c0 0x564fdcdf6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.160 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.159+0000 7f4a1299c700 1 -- 172.21.15.44:0/465722971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdfb1e0 msgr2=0x564fdcdf6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:25.160 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.160+0000 7f4a1299c700 1 --2- 172.21.15.44:0/465722971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.160+0000 7f4a1299c700 1 -- 172.21.15.44:0/465722971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdf5100 msgr2=0x564fdcdf7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:25.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.160+0000 7f4a1299c700 1 --2- 172.21.15.44:0/465722971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdf5100 0x564fdcdf7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.160+0000 7f4a1299c700 1 -- 172.21.15.44:0/465722971 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564fdcd0c190 con 0x564fdcdfc5c0 2023-11-16T16:56:25.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.160+0000 7f4a1199a700 1 --2- 172.21.15.44:0/465722971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:56:25.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.160+0000 7f4a1219b700 1 --2- 172.21.15.44:0/465722971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdf5100 0x564fdcdf7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:56:25.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.160+0000 7f4a1299c700 1 --2- 172.21.15.44:0/465722971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfc5c0 0x564fdcdf6850 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4a040c98e0 tx=0x7f4a040956b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d07a1a34becc9c9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:25.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.160+0000 7f49f5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:56:25.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.161+0000 7f49f5ffb700 1 -- 172.21.15.44:0/465722971 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a040996c0 con 0x564fdcdfc5c0 2023-11-16T16:56:25.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.161+0000 7f49f5ffb700 1 -- 172.21.15.44:0/465722971 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f4a0409d290 con 0x564fdcdfc5c0 2023-11-16T16:56:25.163 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.162+0000 7f4a27d52b80 1 -- 172.21.15.44:0/465722971 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f49fc01e040 con 0x564fdcdfc5c0 2023-11-16T16:56:25.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.162+0000 7f49f5ffb700 1 -- 172.21.15.44:0/465722971 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a040a16b0 con 0x564fdcdfc5c0 2023-11-16T16:56:25.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.163+0000 7f4a27d52b80 10 client.16725 fetch_fsmap learned FSMap version 21 2023-11-16T16:56:25.164 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.163+0000 7f4a27d52b80 1 -- 172.21.15.44:0/465722971 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x564fdcb73de0 con 0x564fdcdfc5c0 2023-11-16T16:56:25.165 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.163+0000 7f49f5ffb700 1 -- 172.21.15.44:0/465722971 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49fc01e040 con 0x564fdcdfc5c0 2023-11-16T16:56:25.165 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.163+0000 7f49f5ffb700 1 -- 172.21.15.44:0/465722971 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4a040981c0 con 0x564fdcdfc5c0 2023-11-16T16:56:25.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.163+0000 7f4a27d52b80 10 client.16725 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:56:25.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.163+0000 7f4a27d52b80 10 client.16725 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:25.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.163+0000 7f4a27d52b80 20 client.16725 populate_metadata read hostname 'smithi044' 2023-11-16T16:56:25.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.163+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/465722971 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564fdce634b0 0x564fdce658c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:25.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.163+0000 7f4a27d52b80 4 client.16725 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:56:25.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.163+0000 7f4a27d52b80 1 -- 172.21.15.44:0/465722971 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564fdcd14660 con 0x564fdce634b0 2023-11-16T16:56:25.166 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.164+0000 7f4a1219b700 1 --2- 172.21.15.44:0/465722971 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564fdce634b0 0x564fdce658c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:25.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.169+0000 7f4a1219b700 1 --2- 172.21.15.44:0/465722971 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564fdce634b0 0x564fdce658c0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2a9251e12e6cf4e server_cookie=4402810c70c3ff79 in_seq=0 out_seq=0 2023-11-16T16:56:25.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.169+0000 7f49f5ffb700 10 client.16725 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:25.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.172+0000 7f49f5ffb700 1 -- 172.21.15.44:0/465722971 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x564fdcd14660 con 0x564fdce634b0 2023-11-16T16:56:25.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.172+0000 7f49f5ffb700 10 client.16725 handle_command_reply: tid=0 2023-11-16T16:56:25.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.172+0000 7f4a27d52b80 1 client.16725 shutdown 2023-11-16T16:56:25.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.174+0000 7f4a27d52b80 1 -- 172.21.15.44:0/465722971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfc5c0 msgr2=0x564fdcdf6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:25.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.174+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/465722971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfc5c0 0x564fdcdf6850 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4a040c98e0 tx=0x7f4a040956b0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.175+0000 7f4a27d52b80 1 -- 172.21.15.44:0/465722971 shutdown_connections 2023-11-16T16:56:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.175+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/465722971 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564fdce634b0 0x564fdce658c0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.175+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/465722971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564fdcdfc5c0 0x564fdcdf6850 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.175+0000 7f49f5ffb700 0 client.16725 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.175+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/465722971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fdcdf5100 0x564fdcdf7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.175+0000 7f4a27d52b80 1 --2- 172.21.15.44:0/465722971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564fdcdfb1e0 0x564fdcdf6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.175+0000 7f4a27d52b80 1 -- 172.21.15.44:0/465722971 >> 172.21.15.44:0/465722971 conn(0x564fdcdea5b0 msgr2=0x564fdcdfbe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:25.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.175+0000 7f4a27d52b80 1 -- 172.21.15.44:0/465722971 shutdown_connections 2023-11-16T16:56:25.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.177+0000 7f4a27d52b80 1 -- 172.21.15.44:0/465722971 wait complete. 2023-11-16T16:56:25.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.177+0000 7f4a27d52b80 20 client.16725 trim_cache size 0 max 16384 2023-11-16T16:56:25.178 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:56:25.178 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "scrub active (3 inodes in the stack)", 2023-11-16T16:56:25.178 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2023-11-16T16:56:25.178 INFO:teuthology.orchestra.run.smithi044.stdout: "58fab0cd-3198-4023-82f3-75a84820f59b": { 2023-11-16T16:56:25.179 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2023-11-16T16:56:25.179 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "58fab0cd-3198-4023-82f3-75a84820f59b", 2023-11-16T16:56:25.179 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2023-11-16T16:56:25.179 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:56:25.179 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:56:25.179 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:56:25.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.178+0000 7f4a1319d700 1 -- 172.21.15.44:0/2223803985 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f49f80a1340 msgr2=0x7f49f80a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.178+0000 7f4a1319d700 1 --2- 172.21.15.44:0/2223803985 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f49f80a1340 0x7f49f80a37f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4a08005ea0 tx=0x7f4a08005e30 comp rx=0 tx=0).stop 2023-11-16T16:56:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.178+0000 7f4a1319d700 1 -- 172.21.15.44:0/2223803985 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a0c112e70 msgr2=0x7f4a0c111510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.178+0000 7f4a1319d700 1 --2- 172.21.15.44:0/2223803985 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a0c112e70 0x7f4a0c111510 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400bdf0 tx=0x7f4a04009ed0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.178+0000 7f4a1319d700 1 -- 172.21.15.44:0/2223803985 shutdown_connections 2023-11-16T16:56:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.178+0000 7f4a1319d700 1 --2- 172.21.15.44:0/2223803985 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f49f80a1340 0x7f49f80a37f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.178+0000 7f4a1319d700 1 --2- 172.21.15.44:0/2223803985 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a0c112e70 0x7f4a0c111510 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.178+0000 7f4a1319d700 1 --2- 172.21.15.44:0/2223803985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a0c06cd80 0x7f4a0c1a9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.178+0000 7f4a1319d700 1 --2- 172.21.15.44:0/2223803985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c108070 0x7f4a0c1aa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:25.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.178+0000 7f4a1319d700 1 -- 172.21.15.44:0/2223803985 >> 172.21.15.44:0/2223803985 conn(0x7f4a0c06b400 msgr2=0x7f4a0c108c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:25.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.179+0000 7f4a1319d700 1 -- 172.21.15.44:0/2223803985 shutdown_connections 2023-11-16T16:56:25.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:25.179+0000 7f4a1319d700 1 -- 172.21.15.44:0/2223803985 wait complete. 2023-11-16T16:56:25.216 INFO:tasks.cephfs.filesystem:scrub status for tag:58fab0cd-3198-4023-82f3-75a84820f59b - {'path': '/', 'tag': '58fab0cd-3198-4023-82f3-75a84820f59b', 'options': 'recursive,force'} 2023-11-16T16:56:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:25 smithi176 ceph-mon[131630]: pgmap v199: 129 pgs: 129 active+clean; 28 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 1.19k op/s 2023-11-16T16:56:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:25 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:56:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:25 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T16:56:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:25 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:56:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:25 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:56:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:25 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:56:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:25 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:56:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:25 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:56:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:25 smithi077 ceph-mon[131204]: pgmap v199: 129 pgs: 129 active+clean; 28 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 1.19k op/s 2023-11-16T16:56:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:25 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:56:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:25 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T16:56:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:25 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:56:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:25 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:56:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:25 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:56:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:25 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:56:25.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:25 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:56:26.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:25 smithi044 ceph-mon[128750]: pgmap v199: 129 pgs: 129 active+clean; 28 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 1.19k op/s 2023-11-16T16:56:26.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:25 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:56:26.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:25 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T16:56:26.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:25 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:56:26.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:25 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:56:26.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:25 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:56:26.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:25 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:56:26.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:25 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:56:27.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:27 smithi176 ceph-mon[131630]: pgmap v200: 129 pgs: 129 active+clean; 28 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 1.19k op/s 2023-11-16T16:56:27.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:27 smithi077 ceph-mon[131204]: pgmap v200: 129 pgs: 129 active+clean; 28 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 1.19k op/s 2023-11-16T16:56:28.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:27 smithi044 ceph-mon[128750]: pgmap v200: 129 pgs: 129 active+clean; 28 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 1.19k op/s 2023-11-16T16:56:29.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:29 smithi176 ceph-mon[131630]: pgmap v201: 129 pgs: 129 active+clean; 40 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 1.18k op/s 2023-11-16T16:56:29.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:29 smithi077 ceph-mon[131204]: pgmap v201: 129 pgs: 129 active+clean; 40 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 1.18k op/s 2023-11-16T16:56:30.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:29 smithi044 ceph-mon[128750]: pgmap v201: 129 pgs: 129 active+clean; 40 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 1.18k op/s 2023-11-16T16:56:31.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:31 smithi077 ceph-mon[131204]: pgmap v202: 129 pgs: 129 active+clean; 40 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 1.18k op/s 2023-11-16T16:56:32.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:31 smithi044 ceph-mon[128750]: pgmap v202: 129 pgs: 129 active+clean; 40 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 1.18k op/s 2023-11-16T16:56:32.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:31 smithi176 ceph-mon[131630]: pgmap v202: 129 pgs: 129 active+clean; 40 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 1.18k op/s 2023-11-16T16:56:33.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:33 smithi077 ceph-mon[131204]: pgmap v203: 129 pgs: 129 active+clean; 44 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 1.08k op/s 2023-11-16T16:56:34.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:33 smithi044 ceph-mon[128750]: pgmap v203: 129 pgs: 129 active+clean; 44 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 1.08k op/s 2023-11-16T16:56:34.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:33 smithi176 ceph-mon[131630]: pgmap v203: 129 pgs: 129 active+clean; 44 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 1.08k op/s 2023-11-16T16:56:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.525+0000 7f6fd599d700 1 -- 172.21.15.44:0/4061245518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fc8095e80 msgr2=0x7f6fc80a2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.525+0000 7f6fd599d700 1 --2- 172.21.15.44:0/4061245518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fc8095e80 0x7f6fc80a2330 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6fd004fde0 tx=0x7f6fd0067260 comp rx=0 tx=0).stop 2023-11-16T16:56:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.526+0000 7f6fd599d700 1 -- 172.21.15.44:0/4061245518 shutdown_connections 2023-11-16T16:56:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.526+0000 7f6fd599d700 1 --2- 172.21.15.44:0/4061245518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fc8095e80 0x7f6fc80a2330 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.526+0000 7f6fd599d700 1 --2- 172.21.15.44:0/4061245518 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fc80972e0 0x7f6fc80976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.526+0000 7f6fd599d700 1 --2- 172.21.15.44:0/4061245518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc80954d0 0x7f6fc8095940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.526+0000 7f6fd599d700 1 -- 172.21.15.44:0/4061245518 >> 172.21.15.44:0/4061245518 conn(0x7f6fc800b280 msgr2=0x7f6fc800b680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.527+0000 7f6fd599d700 1 -- 172.21.15.44:0/4061245518 shutdown_connections 2023-11-16T16:56:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.527+0000 7f6fd599d700 1 -- 172.21.15.44:0/4061245518 wait complete. 2023-11-16T16:56:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.529+0000 7f6fd599d700 1 Processor -- start 2023-11-16T16:56:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.529+0000 7f6fd599d700 1 -- start start 2023-11-16T16:56:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.530+0000 7f6fd599d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fc80954d0 0x7f6fc813e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.530+0000 7f6fd599d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc8095e80 0x7f6fc813ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.530+0000 7f6fd599d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fc80972e0 0x7f6fc8141fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.531+0000 7f6fd599d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc81425b0 con 0x7f6fc80972e0 2023-11-16T16:56:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.531+0000 7f6fd599d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc8142720 con 0x7f6fc8095e80 2023-11-16T16:56:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.531+0000 7f6fd599d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc81428e0 con 0x7f6fc80954d0 2023-11-16T16:56:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.530+0000 7f6fcffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc8095e80 0x7f6fc813ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.531+0000 7f6fcffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc8095e80 0x7f6fc813ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:45730/0 (socket says 172.21.15.44:45730) 2023-11-16T16:56:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.531+0000 7f6fcffff700 1 -- 172.21.15.44:0/3881458130 learned_addr learned my addr 172.21.15.44:0/3881458130 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.531+0000 7f6fcffff700 1 -- 172.21.15.44:0/3881458130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fc80954d0 msgr2=0x7f6fc813e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.531+0000 7f6fcffff700 1 --2- 172.21.15.44:0/3881458130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fc80954d0 0x7f6fc813e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.531+0000 7f6fcffff700 1 -- 172.21.15.44:0/3881458130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fc80972e0 msgr2=0x7f6fc8141fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.531+0000 7f6fcffff700 1 --2- 172.21.15.44:0/3881458130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fc80972e0 0x7f6fc8141fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.531+0000 7f6fcffff700 1 -- 172.21.15.44:0/3881458130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd00670d0 con 0x7f6fc8095e80 2023-11-16T16:56:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.532+0000 7f6fcffff700 1 --2- 172.21.15.44:0/3881458130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc8095e80 0x7f6fc813ec00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00e990 tx=0x7f6fbc00ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.532+0000 7f6fcdffb700 1 -- 172.21.15.44:0/3881458130 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc00cb70 con 0x7f6fc8095e80 2023-11-16T16:56:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.532+0000 7f6fcdffb700 1 -- 172.21.15.44:0/3881458130 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fbc00ee70 con 0x7f6fc8095e80 2023-11-16T16:56:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.532+0000 7f6fcdffb700 1 -- 172.21.15.44:0/3881458130 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc005020 con 0x7f6fc8095e80 2023-11-16T16:56:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.532+0000 7f6fd599d700 1 -- 172.21.15.44:0/3881458130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc80a5490 con 0x7f6fc8095e80 2023-11-16T16:56:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.532+0000 7f6fd599d700 1 -- 172.21.15.44:0/3881458130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc80a5300 con 0x7f6fc8095e80 2023-11-16T16:56:34.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.541+0000 7f6fcdffb700 1 -- 172.21.15.44:0/3881458130 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f6fbc004020 con 0x7f6fc8095e80 2023-11-16T16:56:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.545+0000 7f6fea54db80 1 Processor -- start 2023-11-16T16:56:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.546+0000 7f6fea54db80 1 -- start start 2023-11-16T16:56:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.546+0000 7f6fcdffb700 1 --2- 172.21.15.44:0/3881458130 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6fc00a13c0 0x7f6fc00a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.546+0000 7f6fcdffb700 1 -- 172.21.15.44:0/3881458130 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f6fbc0cd010 con 0x7f6fc8095e80 2023-11-16T16:56:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.549+0000 7f6fea54db80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc6077d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.549+0000 7f6fea54db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc6077d990 0x55cc607777b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.549+0000 7f6fea54db80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 0x55cc60778120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.549+0000 7f6fea54db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc6067d000 con 0x55cc6077d990 2023-11-16T16:56:34.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.549+0000 7f6fea54db80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc606cf840 con 0x55cc60777cf0 2023-11-16T16:56:34.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.549+0000 7f6fea54db80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc606c8cd0 con 0x55cc605d36b0 2023-11-16T16:56:34.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.549+0000 7f6fd499b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc6077d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:34.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.549+0000 7f6fd499b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc6077d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:56948/0 (socket says 172.21.15.44:56948) 2023-11-16T16:56:34.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.549+0000 7f6fd499b700 1 -- 172.21.15.44:0/1021082089 learned_addr learned my addr 172.21.15.44:0/1021082089 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:34.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.550+0000 7f6fcffff700 1 --2- 172.21.15.44:0/1021082089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc6077d990 0x55cc607777b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:34.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.550+0000 7f6fd499b700 1 -- 172.21.15.44:0/1021082089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 msgr2=0x55cc60778120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:34.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.550+0000 7f6fd499b700 1 --2- 172.21.15.44:0/1021082089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 0x55cc60778120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.550+0000 7f6fd519c700 1 --2- 172.21.15.44:0/3881458130 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6fc00a13c0 0x7f6fc00a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:34.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.550+0000 7f6fd499b700 1 -- 172.21.15.44:0/1021082089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc6077d990 msgr2=0x55cc607777b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:34.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.550+0000 7f6fd499b700 1 --2- 172.21.15.44:0/1021082089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc6077d990 0x55cc607777b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.550+0000 7f6fd499b700 1 -- 172.21.15.44:0/1021082089 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc6054a5d0 con 0x55cc605d36b0 2023-11-16T16:56:34.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.550+0000 7f6fd499b700 1 --2- 172.21.15.44:0/1021082089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc6077d450 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6fc400bb70 tx=0x7f6fc400be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e52a24044ea0325b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:34.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.551+0000 7f6fbaffd700 1 -- 172.21.15.44:0/1021082089 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc400d890 con 0x55cc605d36b0 2023-11-16T16:56:34.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.551+0000 7f6fbaffd700 1 -- 172.21.15.44:0/1021082089 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fc400deb0 con 0x55cc605d36b0 2023-11-16T16:56:34.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.551+0000 7f6fbaffd700 1 -- 172.21.15.44:0/1021082089 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc40183c0 con 0x55cc605d36b0 2023-11-16T16:56:34.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.552+0000 7f6fd519c700 1 --2- 172.21.15.44:0/3881458130 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6fc00a13c0 0x7f6fc00a3870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6fd007f000 tx=0x7f6fd0077040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.553+0000 7f6fea54db80 1 -- 172.21.15.44:0/1021082089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 msgr2=0x55cc6077d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.553+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1021082089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc6077d450 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6fc400bb70 tx=0x7f6fc400be80 comp rx=0 tx=0).stop 2023-11-16T16:56:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.553+0000 7f6fea54db80 1 -- 172.21.15.44:0/1021082089 shutdown_connections 2023-11-16T16:56:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.553+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1021082089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 0x55cc60778120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.553+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1021082089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc6077d450 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.553+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1021082089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc6077d990 0x55cc607777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.554+0000 7f6fea54db80 1 -- 172.21.15.44:0/1021082089 >> 172.21.15.44:0/1021082089 conn(0x55cc6076d1a0 msgr2=0x55cc607723e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.554+0000 7f6fea54db80 1 -- 172.21.15.44:0/1021082089 shutdown_connections 2023-11-16T16:56:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.554+0000 7f6fea54db80 1 -- 172.21.15.44:0/1021082089 wait complete. 2023-11-16T16:56:34.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.555+0000 7f6fea54db80 1 -- start start 2023-11-16T16:56:34.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.556+0000 7f6fea54db80 1 Processor -- start 2023-11-16T16:56:34.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.558+0000 7f6fea54db80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc60786c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.558+0000 7f6fea54db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc60777cf0 0x55cc60787150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.558+0000 7f6fd499b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc60786c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.559+0000 7f6fea54db80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc6077d990 0x55cc60780ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.559+0000 7f6fea54db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc6066a880 con 0x55cc60777cf0 2023-11-16T16:56:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.559+0000 7f6fd499b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc60786c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:56960/0 (socket says 172.21.15.44:56960) 2023-11-16T16:56:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.559+0000 7f6fea54db80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc6066a520 con 0x55cc6077d990 2023-11-16T16:56:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.559+0000 7f6fea54db80 1 -- 172.21.15.44:0/1196546416 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc6054c8c0 con 0x55cc605d36b0 2023-11-16T16:56:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.559+0000 7f6fd499b700 1 -- 172.21.15.44:0/1196546416 learned_addr learned my addr 172.21.15.44:0/1196546416 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.559+0000 7f6fea54db80 10 client.0 _ll_register_callbacks cb 0x55cc60688720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.559+0000 7f6fd519c700 1 --2- 172.21.15.44:0/1196546416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc6077d990 0x55cc60780ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:34.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.560+0000 7f6fd499b700 1 -- 172.21.15.44:0/1196546416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc6077d990 msgr2=0x55cc60780ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:34.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.560+0000 7f6fd499b700 1 --2- 172.21.15.44:0/1196546416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc6077d990 0x55cc60780ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.560+0000 7f6fd499b700 1 -- 172.21.15.44:0/1196546416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc60777cf0 msgr2=0x55cc60787150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:56:34.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.560+0000 7f6fd499b700 1 --2- 172.21.15.44:0/1196546416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc60777cf0 0x55cc60787150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.560+0000 7f6fd499b700 1 -- 172.21.15.44:0/1196546416 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc400b7a0 con 0x55cc605d36b0 2023-11-16T16:56:34.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.561+0000 7f6fd499b700 1 --2- 172.21.15.44:0/1196546416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc60786c10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4011fd0 tx=0x7f6fc4012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8002bbf5e6fb4450 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:34.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.561+0000 7f6faffff700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:56:34.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.561+0000 7f6faffff700 1 -- 172.21.15.44:0/1196546416 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc401e070 con 0x55cc605d36b0 2023-11-16T16:56:34.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.561+0000 7f6faffff700 1 -- 172.21.15.44:0/1196546416 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f6fc4010670 con 0x55cc605d36b0 2023-11-16T16:56:34.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.562+0000 7f6fea54db80 1 -- 172.21.15.44:0/1196546416 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cc6076a960 con 0x55cc605d36b0 2023-11-16T16:56:34.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.564+0000 7f6faffff700 1 -- 172.21.15.44:0/1196546416 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fc400d890 con 0x55cc605d36b0 2023-11-16T16:56:34.566 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.564+0000 7f6fea54db80 10 client.26755 fetch_fsmap learned FSMap version 21 2023-11-16T16:56:34.566 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.564+0000 7f6fea54db80 1 -- 172.21.15.44:0/1196546416 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55cc6068f130 con 0x55cc605d36b0 2023-11-16T16:56:34.566 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.564+0000 7f6faffff700 1 -- 172.21.15.44:0/1196546416 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cc6076a960 con 0x55cc605d36b0 2023-11-16T16:56:34.567 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.565+0000 7f6faffff700 1 -- 172.21.15.44:0/1196546416 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6fc4003840 con 0x55cc605d36b0 2023-11-16T16:56:34.567 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.565+0000 7f6fea54db80 10 client.26755 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:56:34.567 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.565+0000 7f6fea54db80 10 client.26755 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:34.567 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.565+0000 7f6fea54db80 20 client.26755 populate_metadata read hostname 'smithi044' 2023-11-16T16:56:34.567 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.565+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1196546416 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55cc60790fd0 0x55cc607933b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.567 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.565+0000 7f6fea54db80 4 client.26755 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:56:34.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.565+0000 7f6fea54db80 1 -- 172.21.15.44:0/1196546416 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc6068b040 con 0x55cc60790fd0 2023-11-16T16:56:34.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.566+0000 7f6fcffff700 1 --2- 172.21.15.44:0/1196546416 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55cc60790fd0 0x55cc607933b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:34.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.567+0000 7f6faffff700 10 client.26755 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:34.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.567+0000 7f6fcffff700 1 --2- 172.21.15.44:0/1196546416 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55cc60790fd0 0x55cc607933b0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e71529dca091c8d server_cookie=f9961b797576efc5 in_seq=0 out_seq=0 2023-11-16T16:56:34.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.569+0000 7f6faffff700 1 -- 172.21.15.44:0/1196546416 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55cc6068b040 con 0x55cc60790fd0 2023-11-16T16:56:34.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.569+0000 7f6faffff700 10 client.26755 handle_command_reply: tid=0 2023-11-16T16:56:34.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.569+0000 7f6fea54db80 1 client.26755 shutdown 2023-11-16T16:56:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.570+0000 7f6fea54db80 1 -- 172.21.15.44:0/1196546416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 msgr2=0x55cc60786c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.570+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1196546416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc60786c10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4011fd0 tx=0x7f6fc4012000 comp rx=0 tx=0).stop 2023-11-16T16:56:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.571+0000 7f6fea54db80 1 -- 172.21.15.44:0/1196546416 shutdown_connections 2023-11-16T16:56:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.571+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1196546416 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55cc60790fd0 0x55cc607933b0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.571+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1196546416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc6077d990 0x55cc60780ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.571+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1196546416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc60786c10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.571+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1196546416 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc60777cf0 0x55cc60787150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.571+0000 7f6fea54db80 1 -- 172.21.15.44:0/1196546416 >> 172.21.15.44:0/1196546416 conn(0x55cc6076d1a0 msgr2=0x55cc60771a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.571+0000 7f6faffff700 0 client.26755 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.571+0000 7f6fea54db80 1 -- 172.21.15.44:0/1196546416 shutdown_connections 2023-11-16T16:56:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.571+0000 7f6fea54db80 1 -- 172.21.15.44:0/1196546416 wait complete. 2023-11-16T16:56:34.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.573+0000 7f6fea54db80 20 client.26755 trim_cache size 0 max 16384 2023-11-16T16:56:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.603+0000 7f6fea54db80 1 Processor -- start 2023-11-16T16:56:34.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.603+0000 7f6fea54db80 1 -- start start 2023-11-16T16:56:34.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.604+0000 7f6fea54db80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc607777b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.604+0000 7f6fea54db80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 0x55cc60781000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.605+0000 7f6fea54db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc6077d990 0x55cc607817a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.607 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.605+0000 7f6fea54db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc6077bd00 con 0x55cc6077d990 2023-11-16T16:56:34.607 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.605+0000 7f6fea54db80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc6077d720 con 0x55cc60777cf0 2023-11-16T16:56:34.607 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.605+0000 7f6fea54db80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc60781ce0 con 0x55cc605d36b0 2023-11-16T16:56:34.607 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.605+0000 7f6fd499b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc607777b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:34.607 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.605+0000 7f6fcffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 0x55cc60781000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:34.607 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.605+0000 7f6fcffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 0x55cc60781000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:45748/0 (socket says 172.21.15.44:45748) 2023-11-16T16:56:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.605+0000 7f6fcffff700 1 -- 172.21.15.44:0/1823025559 learned_addr learned my addr 172.21.15.44:0/1823025559 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.606+0000 7f6fd519c700 1 --2- 172.21.15.44:0/1823025559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc6077d990 0x55cc607817a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.606+0000 7f6fd519c700 1 -- 172.21.15.44:0/1823025559 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 msgr2=0x55cc607777b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.606+0000 7f6fd519c700 1 --2- 172.21.15.44:0/1823025559 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc607777b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.606+0000 7f6fd519c700 1 -- 172.21.15.44:0/1823025559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 msgr2=0x55cc60781000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.607+0000 7f6fd519c700 1 --2- 172.21.15.44:0/1823025559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 0x55cc60781000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.607+0000 7f6fd519c700 1 -- 172.21.15.44:0/1823025559 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc6068ee80 con 0x55cc6077d990 2023-11-16T16:56:34.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.607+0000 7f6fcffff700 1 --2- 172.21.15.44:0/1823025559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 0x55cc60781000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:56:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.609+0000 7f6fd519c700 1 --2- 172.21.15.44:0/1823025559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc6077d990 0x55cc607817a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6fd004fb00 tx=0x7f6fd006e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc6f88738b1460f5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.609+0000 7f6fbb7fe700 1 -- 172.21.15.44:0/1823025559 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0073ea0 con 0x55cc6077d990 2023-11-16T16:56:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.609+0000 7f6fbb7fe700 1 -- 172.21.15.44:0/1823025559 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fd007b070 con 0x55cc6077d990 2023-11-16T16:56:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.609+0000 7f6fbb7fe700 1 -- 172.21.15.44:0/1823025559 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0073ea0 con 0x55cc6077d990 2023-11-16T16:56:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.610+0000 7f6fea54db80 1 -- 172.21.15.44:0/1823025559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc6077d990 msgr2=0x55cc607817a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.610+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1823025559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc6077d990 0x55cc607817a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6fd004fb00 tx=0x7f6fd006e000 comp rx=0 tx=0).stop 2023-11-16T16:56:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.610+0000 7f6fea54db80 1 -- 172.21.15.44:0/1823025559 shutdown_connections 2023-11-16T16:56:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.611+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1823025559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc6077d990 0x55cc607817a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.611+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1823025559 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc607777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.611+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1823025559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 0x55cc60781000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.611+0000 7f6fea54db80 1 -- 172.21.15.44:0/1823025559 >> 172.21.15.44:0/1823025559 conn(0x55cc6076d1a0 msgr2=0x55cc607780c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.611+0000 7f6fea54db80 1 -- 172.21.15.44:0/1823025559 shutdown_connections 2023-11-16T16:56:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.611+0000 7f6fea54db80 1 -- 172.21.15.44:0/1823025559 wait complete. 2023-11-16T16:56:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.612+0000 7f6fea54db80 1 -- start start 2023-11-16T16:56:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.612+0000 7f6fea54db80 1 Processor -- start 2023-11-16T16:56:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.613+0000 7f6fea54db80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc60784b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.613+0000 7f6fea54db80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 0x55cc60785090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.613+0000 7f6fea54db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc6077d990 0x55cc607e3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.614+0000 7f6fea54db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc60785770 con 0x55cc6077d990 2023-11-16T16:56:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.614+0000 7f6fea54db80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc607e43c0 con 0x55cc60777cf0 2023-11-16T16:56:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.614+0000 7f6fea54db80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc607e4500 con 0x55cc605d36b0 2023-11-16T16:56:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.614+0000 7f6fea54db80 10 client.0 _ll_register_callbacks cb 0x7f6fbc0047f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.614+0000 7f6fcffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 0x55cc60785090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.614+0000 7f6fcffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 0x55cc60785090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:45752/0 (socket says 172.21.15.44:45752) 2023-11-16T16:56:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.614+0000 7f6fcffff700 1 -- 172.21.15.44:0/1577694029 learned_addr learned my addr 172.21.15.44:0/1577694029 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.614+0000 7f6fd519c700 1 --2- 172.21.15.44:0/1577694029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc6077d990 0x55cc607e3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.614+0000 7f6fcffff700 1 -- 172.21.15.44:0/1577694029 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 msgr2=0x55cc60784b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.614+0000 7f6fcffff700 1 --2- 172.21.15.44:0/1577694029 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc60784b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.614+0000 7f6fcffff700 1 -- 172.21.15.44:0/1577694029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc6077d990 msgr2=0x55cc607e3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.614+0000 7f6fcffff700 1 --2- 172.21.15.44:0/1577694029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc6077d990 0x55cc607e3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.614+0000 7f6fcffff700 1 -- 172.21.15.44:0/1577694029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc80a5300 con 0x55cc60777cf0 2023-11-16T16:56:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.615+0000 7f6fcffff700 1 --2- 172.21.15.44:0/1577694029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 0x55cc60785090 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc096710 tx=0x7f6fbc00ccd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5f532d831e895fb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.615+0000 7f6faffff700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:56:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.615+0000 7f6faffff700 1 -- 172.21.15.44:0/1577694029 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc0044d0 con 0x55cc60777cf0 2023-11-16T16:56:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.615+0000 7f6fea54db80 1 -- 172.21.15.44:0/1577694029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6fc400deb0 con 0x55cc60777cf0 2023-11-16T16:56:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.616+0000 7f6faffff700 1 -- 172.21.15.44:0/1577694029 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f6fbc098100 con 0x55cc60777cf0 2023-11-16T16:56:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.616+0000 7f6faffff700 1 -- 172.21.15.44:0/1577694029 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fbc0ca5a0 con 0x55cc60777cf0 2023-11-16T16:56:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.617+0000 7f6fea54db80 10 client.26795 fetch_fsmap learned FSMap version 21 2023-11-16T16:56:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.617+0000 7f6fea54db80 1 -- 172.21.15.44:0/1577694029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x7f6fc80a2c30 con 0x55cc60777cf0 2023-11-16T16:56:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.617+0000 7f6faffff700 1 -- 172.21.15.44:0/1577694029 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fbc07c080 con 0x55cc60777cf0 2023-11-16T16:56:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.618+0000 7f6faffff700 1 -- 172.21.15.44:0/1577694029 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6fbc099190 con 0x55cc60777cf0 2023-11-16T16:56:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.618+0000 7f6fea54db80 10 client.26795 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:56:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.618+0000 7f6fea54db80 10 client.26795 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.618+0000 7f6fea54db80 20 client.26795 populate_metadata read hostname 'smithi044' 2023-11-16T16:56:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.618+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1577694029 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55cc607e5bd0 0x55cc607e7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.618+0000 7f6fea54db80 4 client.26795 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:56:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.618+0000 7f6fea54db80 1 -- 172.21.15.44:0/1577694029 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55cc60697600 con 0x55cc607e5bd0 2023-11-16T16:56:34.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.619+0000 7f6fd499b700 1 --2- 172.21.15.44:0/1577694029 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55cc607e5bd0 0x55cc607e7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:34.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.620+0000 7f6faffff700 10 client.26795 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:34.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:34.620+0000 7f6fd499b700 1 --2- 172.21.15.44:0/1577694029 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55cc607e5bd0 0x55cc607e7fe0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5612589144a16d72 server_cookie=2f165cea8e038ee2 in_seq=0 out_seq=0 2023-11-16T16:56:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:35 smithi077 ceph-mon[131204]: pgmap v204: 129 pgs: 129 active+clean; 52 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 669 op/s 2023-11-16T16:56:36.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:35 smithi044 ceph-mon[128750]: pgmap v204: 129 pgs: 129 active+clean; 52 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 669 op/s 2023-11-16T16:56:36.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:35 smithi176 ceph-mon[131630]: pgmap v204: 129 pgs: 129 active+clean; 52 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 669 op/s 2023-11-16T16:56:37.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:37 smithi077 ceph-mon[131204]: pgmap v205: 129 pgs: 129 active+clean; 52 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 580 op/s 2023-11-16T16:56:38.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:37 smithi044 ceph-mon[128750]: pgmap v205: 129 pgs: 129 active+clean; 52 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 580 op/s 2023-11-16T16:56:38.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:37 smithi176 ceph-mon[131630]: pgmap v205: 129 pgs: 129 active+clean; 52 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 580 op/s 2023-11-16T16:56:39.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:39 smithi077 ceph-mon[131204]: pgmap v206: 129 pgs: 129 active+clean; 69 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 864 op/s 2023-11-16T16:56:40.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:39 smithi044 ceph-mon[128750]: pgmap v206: 129 pgs: 129 active+clean; 69 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 864 op/s 2023-11-16T16:56:40.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:39 smithi176 ceph-mon[131630]: pgmap v206: 129 pgs: 129 active+clean; 69 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 864 op/s 2023-11-16T16:56:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.294+0000 7f6faffff700 1 -- 172.21.15.44:0/1577694029 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55cc60697600 con 0x55cc607e5bd0 2023-11-16T16:56:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.294+0000 7f6faffff700 10 client.26795 handle_command_reply: tid=0 2023-11-16T16:56:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.294+0000 7f6fea54db80 1 client.26795 shutdown 2023-11-16T16:56:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.294+0000 7f6fea54db80 1 -- 172.21.15.44:0/1577694029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 msgr2=0x55cc60785090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.294+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1577694029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 0x55cc60785090 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc096710 tx=0x7f6fbc00ccd0 comp rx=0 tx=0).stop 2023-11-16T16:56:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.294+0000 7f6fea54db80 1 -- 172.21.15.44:0/1577694029 shutdown_connections 2023-11-16T16:56:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.294+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1577694029 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55cc607e5bd0 0x55cc607e7fe0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.294+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1577694029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cc6077d990 0x55cc607e3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.294+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1577694029 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cc605d36b0 0x55cc60784b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.294+0000 7f6fea54db80 1 --2- 172.21.15.44:0/1577694029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc60777cf0 0x55cc60785090 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.294+0000 7f6fea54db80 1 -- 172.21.15.44:0/1577694029 >> 172.21.15.44:0/1577694029 conn(0x55cc6076d1a0 msgr2=0x55cc6077c030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.294+0000 7f6faffff700 0 client.26795 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.294+0000 7f6fea54db80 1 -- 172.21.15.44:0/1577694029 shutdown_connections 2023-11-16T16:56:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.294+0000 7f6fea54db80 1 -- 172.21.15.44:0/1577694029 wait complete. 2023-11-16T16:56:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.295+0000 7f6fea54db80 20 client.26795 trim_cache size 0 max 16384 2023-11-16T16:56:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:56:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:56:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:56:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:56:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.295+0000 7f6fd599d700 1 -- 172.21.15.44:0/3881458130 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6fc00a13c0 msgr2=0x7f6fc00a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.295+0000 7f6fd599d700 1 --2- 172.21.15.44:0/3881458130 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6fc00a13c0 0x7f6fc00a3870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6fd007f000 tx=0x7f6fd0077040 comp rx=0 tx=0).stop 2023-11-16T16:56:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.295+0000 7f6fd599d700 1 -- 172.21.15.44:0/3881458130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc8095e80 msgr2=0x7f6fc813ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.295+0000 7f6fd599d700 1 --2- 172.21.15.44:0/3881458130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc8095e80 0x7f6fc813ec00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00e990 tx=0x7f6fbc00ed50 comp rx=0 tx=0).stop 2023-11-16T16:56:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.295+0000 7f6fd599d700 1 -- 172.21.15.44:0/3881458130 shutdown_connections 2023-11-16T16:56:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.295+0000 7f6fd599d700 1 --2- 172.21.15.44:0/3881458130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fc80972e0 0x7f6fc8141fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.295+0000 7f6fd599d700 1 --2- 172.21.15.44:0/3881458130 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f6fc00a13c0 0x7f6fc00a3870 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.295+0000 7f6fd599d700 1 --2- 172.21.15.44:0/3881458130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fc80954d0 0x7f6fc813e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.295+0000 7f6fd599d700 1 --2- 172.21.15.44:0/3881458130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc8095e80 0x7f6fc813ec00 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.295+0000 7f6fd599d700 1 -- 172.21.15.44:0/3881458130 >> 172.21.15.44:0/3881458130 conn(0x7f6fc800b280 msgr2=0x7f6fc80980a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.296+0000 7f6fd599d700 1 -- 172.21.15.44:0/3881458130 shutdown_connections 2023-11-16T16:56:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:41.296+0000 7f6fd599d700 1 -- 172.21.15.44:0/3881458130 wait complete. 2023-11-16T16:56:41.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:41 smithi077 ceph-mon[131204]: pgmap v207: 129 pgs: 129 active+clean; 69 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 544 op/s 2023-11-16T16:56:42.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:41 smithi044 ceph-mon[128750]: pgmap v207: 129 pgs: 129 active+clean; 69 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 544 op/s 2023-11-16T16:56:42.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:41 smithi176 ceph-mon[131630]: pgmap v207: 129 pgs: 129 active+clean; 69 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 544 op/s 2023-11-16T16:56:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:43 smithi077 ceph-mon[131204]: pgmap v208: 129 pgs: 129 active+clean; 53 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 637 op/s 2023-11-16T16:56:44.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:43 smithi044 ceph-mon[128750]: pgmap v208: 129 pgs: 129 active+clean; 53 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 637 op/s 2023-11-16T16:56:44.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:43 smithi176 ceph-mon[131630]: pgmap v208: 129 pgs: 129 active+clean; 53 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 637 op/s 2023-11-16T16:56:45.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:45 smithi077 ceph-mon[131204]: pgmap v209: 129 pgs: 129 active+clean; 21 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 1.02k op/s 2023-11-16T16:56:46.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:45 smithi044 ceph-mon[128750]: pgmap v209: 129 pgs: 129 active+clean; 21 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 1.02k op/s 2023-11-16T16:56:46.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:45 smithi176 ceph-mon[131630]: pgmap v209: 129 pgs: 129 active+clean; 21 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 1.02k op/s 2023-11-16T16:56:48.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:47 smithi044 ceph-mon[128750]: pgmap v210: 129 pgs: 129 active+clean; 21 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 807 op/s 2023-11-16T16:56:48.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:47 smithi176 ceph-mon[131630]: pgmap v210: 129 pgs: 129 active+clean; 21 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 807 op/s 2023-11-16T16:56:48.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:47 smithi077 ceph-mon[131204]: pgmap v210: 129 pgs: 129 active+clean; 21 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 807 op/s 2023-11-16T16:56:49.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:49 smithi044 ceph-mon[128750]: pgmap v211: 129 pgs: 129 active+clean; 33 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 1.11k op/s 2023-11-16T16:56:49.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:49 smithi176 ceph-mon[131630]: pgmap v211: 129 pgs: 129 active+clean; 33 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 1.11k op/s 2023-11-16T16:56:49.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:49 smithi077 ceph-mon[131204]: pgmap v211: 129 pgs: 129 active+clean; 33 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 1.11k op/s 2023-11-16T16:56:51.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:51 smithi044 ceph-mon[128750]: pgmap v212: 129 pgs: 129 active+clean; 33 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 828 op/s 2023-11-16T16:56:51.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:51 smithi176 ceph-mon[131630]: pgmap v212: 129 pgs: 129 active+clean; 33 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 828 op/s 2023-11-16T16:56:51.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:51 smithi077 ceph-mon[131204]: pgmap v212: 129 pgs: 129 active+clean; 33 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 828 op/s 2023-11-16T16:56:53.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:53 smithi044 ceph-mon[128750]: pgmap v213: 129 pgs: 129 active+clean; 33 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 883 op/s 2023-11-16T16:56:53.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:53 smithi176 ceph-mon[131630]: pgmap v213: 129 pgs: 129 active+clean; 33 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 883 op/s 2023-11-16T16:56:53.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:53 smithi077 ceph-mon[131204]: pgmap v213: 129 pgs: 129 active+clean; 33 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 883 op/s 2023-11-16T16:56:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.506+0000 7fe6493ff700 1 -- 172.21.15.44:0/3644352225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6441111d0 msgr2=0x7fe644113660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.506+0000 7fe6493ff700 1 --2- 172.21.15.44:0/3644352225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6441111d0 0x7fe644113660 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe634008570 tx=0x7fe63400acb0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.507+0000 7fe6493ff700 1 -- 172.21.15.44:0/3644352225 shutdown_connections 2023-11-16T16:56:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.507+0000 7fe6493ff700 1 --2- 172.21.15.44:0/3644352225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6441111d0 0x7fe644113660 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.508+0000 7fe6493ff700 1 --2- 172.21.15.44:0/3644352225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe644107d60 0x7fe644108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.508+0000 7fe6493ff700 1 --2- 172.21.15.44:0/3644352225 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe644108670 0x7fe644110b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.508+0000 7fe6493ff700 1 -- 172.21.15.44:0/3644352225 >> 172.21.15.44:0/3644352225 conn(0x7fe64406c510 msgr2=0x7fe6440716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:54.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.511+0000 7fe6493ff700 1 -- 172.21.15.44:0/3644352225 shutdown_connections 2023-11-16T16:56:54.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.511+0000 7fe6493ff700 1 -- 172.21.15.44:0/3644352225 wait complete. 2023-11-16T16:56:54.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.514+0000 7fe6493ff700 1 Processor -- start 2023-11-16T16:56:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.514+0000 7fe6493ff700 1 -- start start 2023-11-16T16:56:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.515+0000 7fe6493ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe644107d60 0x7fe6441a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.515+0000 7fe6493ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe644108670 0x7fe6441a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.515+0000 7fe6493ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe6441111d0 0x7fe6441a9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.515+0000 7fe6493ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6441aa190 con 0x7fe6441111d0 2023-11-16T16:56:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.515+0000 7fe6493ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6441aa300 con 0x7fe644107d60 2023-11-16T16:56:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.515+0000 7fe6493ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6441aa4c0 con 0x7fe644108670 2023-11-16T16:56:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.516+0000 7fe642ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe644107d60 0x7fe6441a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.516+0000 7fe6427fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe644108670 0x7fe6441a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.516+0000 7fe642ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe644107d60 0x7fe6441a51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:42604/0 (socket says 172.21.15.44:42604) 2023-11-16T16:56:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.516+0000 7fe642ffd700 1 -- 172.21.15.44:0/4024543215 learned_addr learned my addr 172.21.15.44:0/4024543215 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.516+0000 7fe6437fe700 1 --2- 172.21.15.44:0/4024543215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe6441111d0 0x7fe6441a9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.517+0000 7fe642ffd700 1 -- 172.21.15.44:0/4024543215 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe644108670 msgr2=0x7fe6441a56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.517+0000 7fe642ffd700 1 --2- 172.21.15.44:0/4024543215 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe644108670 0x7fe6441a56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.517+0000 7fe642ffd700 1 -- 172.21.15.44:0/4024543215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe6441111d0 msgr2=0x7fe6441a9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.517+0000 7fe642ffd700 1 --2- 172.21.15.44:0/4024543215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe6441111d0 0x7fe6441a9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.517+0000 7fe642ffd700 1 -- 172.21.15.44:0/4024543215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6340081d0 con 0x7fe644107d60 2023-11-16T16:56:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.518+0000 7fe642ffd700 1 --2- 172.21.15.44:0/4024543215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe644107d60 0x7fe6441a51b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe6380097d0 tx=0x7fe638009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.518+0000 7fe62bfff700 1 -- 172.21.15.44:0/4024543215 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe638010040 con 0x7fe644107d60 2023-11-16T16:56:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.518+0000 7fe62bfff700 1 -- 172.21.15.44:0/4024543215 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe638013c50 con 0x7fe644107d60 2023-11-16T16:56:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.518+0000 7fe6493ff700 1 -- 172.21.15.44:0/4024543215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe644116910 con 0x7fe644107d60 2023-11-16T16:56:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.519+0000 7fe62bfff700 1 -- 172.21.15.44:0/4024543215 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe638014d60 con 0x7fe644107d60 2023-11-16T16:56:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.519+0000 7fe6493ff700 1 -- 172.21.15.44:0/4024543215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe644116780 con 0x7fe644107d60 2023-11-16T16:56:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.531+0000 7fe65cdaeb80 1 Processor -- start 2023-11-16T16:56:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.532+0000 7fe65cdaeb80 1 -- start start 2023-11-16T16:56:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.532+0000 7fe65cdaeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875bc26b0 0x564875d6c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.533+0000 7fe65cdaeb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d6c720 0x564875d66560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.533+0000 7fe65cdaeb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875d6cb90 0x564875d66b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.533+0000 7fe6437fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875bc26b0 0x564875d6c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.533+0000 7fe65cdaeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x564875c6c000 con 0x564875bc26b0 2023-11-16T16:56:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.533+0000 7fe65cdaeb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x564875cbe840 con 0x564875d6c720 2023-11-16T16:56:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.533+0000 7fe65cdaeb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x564875cb7cd0 con 0x564875d6cb90 2023-11-16T16:56:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.533+0000 7fe6427fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875d6cb90 0x564875d66b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.533+0000 7fe6427fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875d6cb90 0x564875d66b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:45958/0 (socket says 172.21.15.44:45958) 2023-11-16T16:56:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.533+0000 7fe6427fc700 1 -- 172.21.15.44:0/2572632594 learned_addr learned my addr 172.21.15.44:0/2572632594 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.534+0000 7fe6427fc700 1 -- 172.21.15.44:0/2572632594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d6c720 msgr2=0x564875d66560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:56:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.534+0000 7fe62bfff700 1 -- 172.21.15.44:0/4024543215 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fe63801e460 con 0x7fe644107d60 2023-11-16T16:56:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.536+0000 7fe642ffd700 1 --2- 172.21.15.44:0/2572632594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d6c720 0x564875d66560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.537+0000 7fe6427fc700 1 --2- 172.21.15.44:0/2572632594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d6c720 0x564875d66560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.537+0000 7fe6427fc700 1 -- 172.21.15.44:0/2572632594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875bc26b0 msgr2=0x564875d6c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.537+0000 7fe6427fc700 1 --2- 172.21.15.44:0/2572632594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875bc26b0 0x564875d6c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.537+0000 7fe6427fc700 1 -- 172.21.15.44:0/2572632594 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564875b395d0 con 0x564875d6cb90 2023-11-16T16:56:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.537+0000 7fe6437fe700 1 --2- 172.21.15.44:0/2572632594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875bc26b0 0x564875d6c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:56:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.537+0000 7fe642ffd700 1 --2- 172.21.15.44:0/2572632594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d6c720 0x564875d66560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:56:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.537+0000 7fe6427fc700 1 --2- 172.21.15.44:0/2572632594 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875d6cb90 0x564875d66b00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009f50 tx=0x7fe63c009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6d9ae903f44369d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.538+0000 7fe6297fa700 1 -- 172.21.15.44:0/2572632594 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c011070 con 0x564875d6cb90 2023-11-16T16:56:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.538+0000 7fe62bfff700 1 --2- 172.21.15.44:0/4024543215 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe62c0a1490 0x7fe62c0a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.538+0000 7fe6297fa700 1 -- 172.21.15.44:0/2572632594 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe63c009ab0 con 0x564875d6cb90 2023-11-16T16:56:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.539+0000 7fe62bfff700 1 -- 172.21.15.44:0/4024543215 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fe6380ccde0 con 0x7fe644107d60 2023-11-16T16:56:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.539+0000 7fe6297fa700 1 -- 172.21.15.44:0/2572632594 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c00a400 con 0x564875d6cb90 2023-11-16T16:56:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.540+0000 7fe6437fe700 1 --2- 172.21.15.44:0/4024543215 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe62c0a1490 0x7fe62c0a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.540+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/2572632594 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875d6cb90 msgr2=0x564875d66b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.540+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/2572632594 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875d6cb90 0x564875d66b00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009f50 tx=0x7fe63c009f80 comp rx=0 tx=0).stop 2023-11-16T16:56:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.541+0000 7fe6437fe700 1 --2- 172.21.15.44:0/4024543215 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe62c0a1490 0x7fe62c0a3940 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe634005cd0 tx=0x7fe634005c00 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.542+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/2572632594 shutdown_connections 2023-11-16T16:56:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.542+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/2572632594 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875d6cb90 0x564875d66b00 secure :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009f50 tx=0x7fe63c009f80 comp rx=0 tx=0).stop 2023-11-16T16:56:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.542+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/2572632594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875bc26b0 0x564875d6c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.542+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/2572632594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d6c720 0x564875d66560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.542+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/2572632594 >> 172.21.15.44:0/2572632594 conn(0x564875d5bf50 msgr2=0x564875d61270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.542+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/2572632594 shutdown_connections 2023-11-16T16:56:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.542+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/2572632594 wait complete. 2023-11-16T16:56:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.543+0000 7fe65cdaeb80 1 -- start start 2023-11-16T16:56:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.544+0000 7fe65cdaeb80 1 Processor -- start 2023-11-16T16:56:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.545+0000 7fe65cdaeb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875bc26b0 0x564875d6fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.545+0000 7fe65cdaeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875d6c720 0x564875d70360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.545+0000 7fe65cdaeb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875d708a0 0x564875d7c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.545+0000 7fe65cdaeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x564875c59880 con 0x564875d6c720 2023-11-16T16:56:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.545+0000 7fe65cdaeb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x564875c59520 con 0x564875bc26b0 2023-11-16T16:56:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.545+0000 7fe65cdaeb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x564875b3b8c0 con 0x564875d708a0 2023-11-16T16:56:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.546+0000 7fe642ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875bc26b0 0x564875d6fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.546+0000 7fe65cdaeb80 10 client.0 _ll_register_callbacks cb 0x564875c77720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.546+0000 7fe642ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875bc26b0 0x564875d6fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:42616/0 (socket says 172.21.15.44:42616) 2023-11-16T16:56:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.546+0000 7fe642ffd700 1 -- 172.21.15.44:0/3891236387 learned_addr learned my addr 172.21.15.44:0/3891236387 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.546+0000 7fe642ffd700 1 -- 172.21.15.44:0/3891236387 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875d708a0 msgr2=0x564875d7c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.546+0000 7fe642ffd700 1 --2- 172.21.15.44:0/3891236387 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875d708a0 0x564875d7c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.546+0000 7fe642ffd700 1 -- 172.21.15.44:0/3891236387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875d6c720 msgr2=0x564875d70360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.546+0000 7fe642ffd700 1 --2- 172.21.15.44:0/3891236387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875d6c720 0x564875d70360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.546+0000 7fe642ffd700 1 -- 172.21.15.44:0/3891236387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe63c009c30 con 0x564875bc26b0 2023-11-16T16:56:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.547+0000 7fe642ffd700 1 --2- 172.21.15.44:0/3891236387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875bc26b0 0x564875d6fe20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe638099770 tx=0x7fe638095ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34573140bf6938c7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:54.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.549+0000 7fe626ff5700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:56:54.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.549+0000 7fe626ff5700 1 -- 172.21.15.44:0/3891236387 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6380a1ec0 con 0x564875bc26b0 2023-11-16T16:56:54.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.549+0000 7fe626ff5700 1 -- 172.21.15.44:0/3891236387 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fe638099ee0 con 0x564875bc26b0 2023-11-16T16:56:54.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.549+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/3891236387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564875c7a040 con 0x564875bc26b0 2023-11-16T16:56:54.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.550+0000 7fe626ff5700 1 -- 172.21.15.44:0/3891236387 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6380b2250 con 0x564875bc26b0 2023-11-16T16:56:54.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.550+0000 7fe65cdaeb80 10 client.26804 fetch_fsmap learned FSMap version 21 2023-11-16T16:56:54.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.550+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/3891236387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x564875c7e130 con 0x564875bc26b0 2023-11-16T16:56:54.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.551+0000 7fe626ff5700 1 -- 172.21.15.44:0/3891236387 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564875c7a040 con 0x564875bc26b0 2023-11-16T16:56:54.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.551+0000 7fe626ff5700 1 -- 172.21.15.44:0/3891236387 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe638014420 con 0x564875bc26b0 2023-11-16T16:56:54.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.551+0000 7fe65cdaeb80 10 client.26804 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:56:54.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.551+0000 7fe65cdaeb80 10 client.26804 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:54.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.551+0000 7fe65cdaeb80 20 client.26804 populate_metadata read hostname 'smithi044' 2023-11-16T16:56:54.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.551+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/3891236387 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564875d80560 0x564875d82990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.551+0000 7fe65cdaeb80 4 client.26804 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:56:54.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.551+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/3891236387 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564875c86600 con 0x564875d80560 2023-11-16T16:56:54.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.560+0000 7fe6427fc700 1 --2- 172.21.15.44:0/3891236387 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564875d80560 0x564875d82990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.562+0000 7fe6427fc700 1 --2- 172.21.15.44:0/3891236387 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564875d80560 0x564875d82990 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=879140a20411eb6e server_cookie=504fd4dc9101b637 in_seq=0 out_seq=0 2023-11-16T16:56:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.562+0000 7fe626ff5700 10 client.26804 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:54.577 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.575+0000 7fe626ff5700 1 -- 172.21.15.44:0/3891236387 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x564875c86600 con 0x564875d80560 2023-11-16T16:56:54.577 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.575+0000 7fe626ff5700 10 client.26804 handle_command_reply: tid=0 2023-11-16T16:56:54.577 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.575+0000 7fe65cdaeb80 1 client.26804 shutdown 2023-11-16T16:56:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.576+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/3891236387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875bc26b0 msgr2=0x564875d6fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.576+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/3891236387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875bc26b0 0x564875d6fe20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe638099770 tx=0x7fe638095ee0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.577+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/3891236387 shutdown_connections 2023-11-16T16:56:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.577+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/3891236387 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564875d80560 0x564875d82990 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.577+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/3891236387 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875d708a0 0x564875d7c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.577+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/3891236387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875bc26b0 0x564875d6fe20 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.577+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/3891236387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875d6c720 0x564875d70360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.577+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/3891236387 >> 172.21.15.44:0/3891236387 conn(0x564875d5bf50 msgr2=0x564875d607e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.577+0000 7fe626ff5700 0 client.26804 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.577+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/3891236387 shutdown_connections 2023-11-16T16:56:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.577+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/3891236387 wait complete. 2023-11-16T16:56:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.578+0000 7fe65cdaeb80 20 client.26804 trim_cache size 0 max 16384 2023-11-16T16:56:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.623+0000 7fe65cdaeb80 1 Processor -- start 2023-11-16T16:56:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.623+0000 7fe65cdaeb80 1 -- start start 2023-11-16T16:56:54.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.624+0000 7fe65cdaeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875bc26b0 0x564875d7c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.624+0000 7fe65cdaeb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875d6c720 0x564875d77190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.624+0000 7fe65cdaeb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d708a0 0x564875d60f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.624+0000 7fe65cdaeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x564875d7dc60 con 0x564875bc26b0 2023-11-16T16:56:54.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.624+0000 7fe65cdaeb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x564875d70dd0 con 0x564875d708a0 2023-11-16T16:56:54.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.624+0000 7fe65cdaeb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x564875d62440 con 0x564875d6c720 2023-11-16T16:56:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.624+0000 7fe6427fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875d6c720 0x564875d77190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.624+0000 7fe6437fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d708a0 0x564875d60f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.625+0000 7fe6437fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d708a0 0x564875d60f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:42632/0 (socket says 172.21.15.44:42632) 2023-11-16T16:56:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.625+0000 7fe6437fe700 1 -- 172.21.15.44:0/1446426340 learned_addr learned my addr 172.21.15.44:0/1446426340 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.625+0000 7fe642ffd700 1 --2- 172.21.15.44:0/1446426340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875bc26b0 0x564875d7c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.627+0000 7fe6437fe700 1 -- 172.21.15.44:0/1446426340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875d6c720 msgr2=0x564875d77190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.627+0000 7fe6437fe700 1 --2- 172.21.15.44:0/1446426340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875d6c720 0x564875d77190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.627+0000 7fe6437fe700 1 -- 172.21.15.44:0/1446426340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875bc26b0 msgr2=0x564875d7c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.627+0000 7fe6437fe700 1 --2- 172.21.15.44:0/1446426340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875bc26b0 0x564875d7c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.627+0000 7fe6437fe700 1 -- 172.21.15.44:0/1446426340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564875d5d6f0 con 0x564875d708a0 2023-11-16T16:56:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.628+0000 7fe6437fe700 1 --2- 172.21.15.44:0/1446426340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d708a0 0x564875d60f60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe6340051d0 tx=0x7fe63400c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30f1effcf3be8ba3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.628+0000 7fe629ffb700 1 -- 172.21.15.44:0/1446426340 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe634005200 con 0x564875d708a0 2023-11-16T16:56:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.630+0000 7fe629ffb700 1 -- 172.21.15.44:0/1446426340 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6340055c0 con 0x564875d708a0 2023-11-16T16:56:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.630+0000 7fe629ffb700 1 -- 172.21.15.44:0/1446426340 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63402a870 con 0x564875d708a0 2023-11-16T16:56:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.632+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/1446426340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d708a0 msgr2=0x564875d60f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.632+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/1446426340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d708a0 0x564875d60f60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe6340051d0 tx=0x7fe63400c040 comp rx=0 tx=0).stop 2023-11-16T16:56:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.634+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/1446426340 shutdown_connections 2023-11-16T16:56:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.634+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/1446426340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d708a0 0x564875d60f60 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.634+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/1446426340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875bc26b0 0x564875d7c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.634+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/1446426340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875d6c720 0x564875d77190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.634+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/1446426340 >> 172.21.15.44:0/1446426340 conn(0x564875d5bf50 msgr2=0x564875d7b940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.635+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/1446426340 shutdown_connections 2023-11-16T16:56:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.635+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/1446426340 wait complete. 2023-11-16T16:56:54.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.636+0000 7fe65cdaeb80 1 -- start start 2023-11-16T16:56:54.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.637+0000 7fe65cdaeb80 1 Processor -- start 2023-11-16T16:56:54.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.638+0000 7fe65cdaeb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875bc26b0 0x564875d77390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.638+0000 7fe65cdaeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875d6c720 0x564875d75180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.638+0000 7fe65cdaeb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d708a0 0x564875d75870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.638+0000 7fe65cdaeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x564875d75f80 con 0x564875d6c720 2023-11-16T16:56:54.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.638+0000 7fe65cdaeb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x564875dd3480 con 0x564875d708a0 2023-11-16T16:56:54.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.638+0000 7fe65cdaeb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x564875de45d0 con 0x564875bc26b0 2023-11-16T16:56:54.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.638+0000 7fe65cdaeb80 10 client.0 _ll_register_callbacks cb 0x7fe6380998f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:54.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.639+0000 7fe6437fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d708a0 0x564875d75870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:54.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.639+0000 7fe6437fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d708a0 0x564875d75870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:42640/0 (socket says 172.21.15.44:42640) 2023-11-16T16:56:54.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.639+0000 7fe6437fe700 1 -- 172.21.15.44:0/859566224 learned_addr learned my addr 172.21.15.44:0/859566224 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:54.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.639+0000 7fe642ffd700 1 --2- 172.21.15.44:0/859566224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875bc26b0 0x564875d77390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:54.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.639+0000 7fe6437fe700 1 -- 172.21.15.44:0/859566224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875bc26b0 msgr2=0x564875d77390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.639+0000 7fe6437fe700 1 --2- 172.21.15.44:0/859566224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875bc26b0 0x564875d77390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.639+0000 7fe6437fe700 1 -- 172.21.15.44:0/859566224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875d6c720 msgr2=0x564875d75180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.640+0000 7fe6437fe700 1 --2- 172.21.15.44:0/859566224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875d6c720 0x564875d75180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.640+0000 7fe6437fe700 1 -- 172.21.15.44:0/859566224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe644114320 con 0x564875d708a0 2023-11-16T16:56:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.640+0000 7fe6437fe700 1 --2- 172.21.15.44:0/859566224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d708a0 0x564875d75870 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe634004cb0 tx=0x7fe63400a900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=760d55ebf2d45b8e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.640+0000 7fe626ff5700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:56:54.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.640+0000 7fe626ff5700 1 -- 172.21.15.44:0/859566224 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe634038070 con 0x564875d708a0 2023-11-16T16:56:54.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.641+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/859566224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564875d59710 con 0x564875d708a0 2023-11-16T16:56:54.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.642+0000 7fe626ff5700 1 -- 172.21.15.44:0/859566224 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fe63400a2e0 con 0x564875d708a0 2023-11-16T16:56:54.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.643+0000 7fe626ff5700 1 -- 172.21.15.44:0/859566224 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe634004ee0 con 0x564875d708a0 2023-11-16T16:56:54.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.643+0000 7fe65cdaeb80 10 client.26810 fetch_fsmap learned FSMap version 21 2023-11-16T16:56:54.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.643+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/859566224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x564875ae5d80 con 0x564875d708a0 2023-11-16T16:56:54.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.644+0000 7fe626ff5700 1 -- 172.21.15.44:0/859566224 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564875d59710 con 0x564875d708a0 2023-11-16T16:56:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.645+0000 7fe626ff5700 1 -- 172.21.15.44:0/859566224 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe634033450 con 0x564875d708a0 2023-11-16T16:56:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.645+0000 7fe65cdaeb80 10 client.26810 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:56:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.645+0000 7fe65cdaeb80 10 client.26810 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.645+0000 7fe65cdaeb80 20 client.26810 populate_metadata read hostname 'smithi044' 2023-11-16T16:56:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.645+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/859566224 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564875de58e0 0x564875de7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.645+0000 7fe65cdaeb80 4 client.26810 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:56:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.645+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/859566224 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564875b06f10 con 0x564875de58e0 2023-11-16T16:56:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.646+0000 7fe642ffd700 1 --2- 172.21.15.44:0/859566224 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564875de58e0 0x564875de7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.647+0000 7fe626ff5700 10 client.26810 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:56:54.647+0000 7fe642ffd700 1 --2- 172.21.15.44:0/859566224 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564875de58e0 0x564875de7cf0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ed06ea790e055d3 server_cookie=4e6d8cd53abd0b9d in_seq=0 out_seq=0 2023-11-16T16:56:55.218 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:56:55.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.444+0000 7ff5431ff700 1 -- 172.21.15.44:0/1215827585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff53c0746f0 msgr2=0x7ff53c072d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:55.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.444+0000 7ff5431ff700 1 --2- 172.21.15.44:0/1215827585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff53c0746f0 0x7ff53c072d20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff530009990 tx=0x7ff530009ca0 comp rx=0 tx=0).stop 2023-11-16T16:56:55.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.447+0000 7ff5431ff700 1 -- 172.21.15.44:0/1215827585 shutdown_connections 2023-11-16T16:56:55.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.447+0000 7ff5431ff700 1 --2- 172.21.15.44:0/1215827585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff53c10cec0 0x7ff53c10f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:55.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.447+0000 7ff5431ff700 1 --2- 172.21.15.44:0/1215827585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff53c0746f0 0x7ff53c072d20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:55.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.447+0000 7ff5431ff700 1 --2- 172.21.15.44:0/1215827585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff53c074ac0 0x7ff53c073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:55.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.447+0000 7ff5431ff700 1 -- 172.21.15.44:0/1215827585 >> 172.21.15.44:0/1215827585 conn(0x7ff53c077560 msgr2=0x7ff53c077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:55.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.448+0000 7ff5431ff700 1 -- 172.21.15.44:0/1215827585 shutdown_connections 2023-11-16T16:56:55.450 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.448+0000 7ff5431ff700 1 -- 172.21.15.44:0/1215827585 wait complete. 2023-11-16T16:56:55.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.450+0000 7ff5431ff700 1 Processor -- start 2023-11-16T16:56:55.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.451+0000 7ff5431ff700 1 -- start start 2023-11-16T16:56:55.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.452+0000 7ff5431ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff53c0746f0 0x7ff53c19e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:55.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.452+0000 7ff5431ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff53c074ac0 0x7ff53c19ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:55.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.452+0000 7ff5431ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff53c10cec0 0x7ff53c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:55.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.452+0000 7ff5431ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c198be0 con 0x7ff53c10cec0 2023-11-16T16:56:55.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.452+0000 7ff5431ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c198d50 con 0x7ff53c0746f0 2023-11-16T16:56:55.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.452+0000 7ff5431ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c198f10 con 0x7ff53c074ac0 2023-11-16T16:56:55.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.454+0000 7ff53bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff53c074ac0 0x7ff53c19ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:55.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.454+0000 7ff53bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff53c074ac0 0x7ff53c19ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:35218/0 (socket says 172.21.15.44:35218) 2023-11-16T16:56:55.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.454+0000 7ff53bfff700 1 -- 172.21.15.44:0/930624780 learned_addr learned my addr 172.21.15.44:0/930624780 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:55.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.454+0000 7ff540f9b700 1 --2- 172.21.15.44:0/930624780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff53c0746f0 0x7ff53c19e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:55.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.455+0000 7ff53bfff700 1 -- 172.21.15.44:0/930624780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff53c0746f0 msgr2=0x7ff53c19e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:55.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.455+0000 7ff53bfff700 1 --2- 172.21.15.44:0/930624780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff53c0746f0 0x7ff53c19e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:55.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.455+0000 7ff53bfff700 1 -- 172.21.15.44:0/930624780 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff53c10cec0 msgr2=0x7ff53c198570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:56:55.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.455+0000 7ff53bfff700 1 --2- 172.21.15.44:0/930624780 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff53c10cec0 0x7ff53c198570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:55.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.455+0000 7ff53bfff700 1 -- 172.21.15.44:0/930624780 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5300095f0 con 0x7ff53c074ac0 2023-11-16T16:56:55.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.455+0000 7ff53bfff700 1 --2- 172.21.15.44:0/930624780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff53c074ac0 0x7ff53c19ea40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff52800e970 tx=0x7ff52800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:55.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.456+0000 7ff539ffb700 1 -- 172.21.15.44:0/930624780 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528009790 con 0x7ff53c074ac0 2023-11-16T16:56:55.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.456+0000 7ff539ffb700 1 -- 172.21.15.44:0/930624780 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff52800eec0 con 0x7ff53c074ac0 2023-11-16T16:56:55.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.456+0000 7ff539ffb700 1 -- 172.21.15.44:0/930624780 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528005520 con 0x7ff53c074ac0 2023-11-16T16:56:55.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.456+0000 7ff5431ff700 1 -- 172.21.15.44:0/930624780 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53c112600 con 0x7ff53c074ac0 2023-11-16T16:56:55.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.456+0000 7ff5431ff700 1 -- 172.21.15.44:0/930624780 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53c112470 con 0x7ff53c074ac0 2023-11-16T16:56:55.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.469+0000 7ff539ffb700 1 -- 172.21.15.44:0/930624780 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7ff528004750 con 0x7ff53c074ac0 2023-11-16T16:56:55.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.473+0000 7ff539ffb700 1 --2- 172.21.15.44:0/930624780 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff52c0a13c0 0x7ff52c0a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:55.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.473+0000 7ff539ffb700 1 -- 172.21.15.44:0/930624780 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9511+0+0 (secure 0 0 0) 0x7ff528014070 con 0x7ff53c074ac0 2023-11-16T16:56:55.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.473+0000 7ff556a4eb80 1 Processor -- start 2023-11-16T16:56:55.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.475+0000 7ff540f9b700 1 --2- 172.21.15.44:0/930624780 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff52c0a13c0 0x7ff52c0a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:55.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.475+0000 7ff556a4eb80 1 -- start start 2023-11-16T16:56:55.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.475+0000 7ff556a4eb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8c370 0x5611aed8c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:55.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.476+0000 7ff556a4eb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed8cc80 0x5611aed90b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:55.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.476+0000 7ff556a4eb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed910c0 0x5611aed91510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:55.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.476+0000 7ff556a4eb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611aec89060 con 0x5611aed910c0 2023-11-16T16:56:55.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.476+0000 7ff556a4eb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611aecdb8a0 con 0x5611aed8c370 2023-11-16T16:56:55.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.476+0000 7ff556a4eb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611aecd4d30 con 0x5611aed8cc80 2023-11-16T16:56:55.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.476+0000 7ff53bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed8cc80 0x5611aed90b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:55.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.476+0000 7ff540f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8c370 0x5611aed8c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:55.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.477+0000 7ff540f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8c370 0x5611aed8c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:37720/0 (socket says 172.21.15.44:37720) 2023-11-16T16:56:55.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.477+0000 7ff540f9b700 1 -- 172.21.15.44:0/3536746931 learned_addr learned my addr 172.21.15.44:0/3536746931 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:55.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.477+0000 7ff53bfff700 1 -- 172.21.15.44:0/3536746931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8c370 msgr2=0x5611aed8c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:55.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.477+0000 7ff53bfff700 1 --2- 172.21.15.44:0/3536746931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8c370 0x5611aed8c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:55.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.477+0000 7ff53bfff700 1 -- 172.21.15.44:0/3536746931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed910c0 msgr2=0x5611aed91510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.477+0000 7ff53bfff700 1 --2- 172.21.15.44:0/3536746931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed910c0 0x5611aed91510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.478+0000 7ff540f9b700 1 --2- 172.21.15.44:0/930624780 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff52c0a13c0 0x7ff52c0a3870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff530015000 tx=0x7ff530005360 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.478+0000 7ff53bfff700 1 -- 172.21.15.44:0/3536746931 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611aeb56630 con 0x5611aed8cc80 2023-11-16T16:56:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.479+0000 7ff53bfff700 1 --2- 172.21.15.44:0/3536746931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed8cc80 0x5611aed90b80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff5280ca6b0 tx=0x7ff52809a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57393a91ff850445 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:55.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.479+0000 7ff526ffd700 1 -- 172.21.15.44:0/3536746931 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5280a2480 con 0x5611aed8cc80 2023-11-16T16:56:55.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.479+0000 7ff526ffd700 1 -- 172.21.15.44:0/3536746931 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff528098170 con 0x5611aed8cc80 2023-11-16T16:56:55.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.479+0000 7ff526ffd700 1 -- 172.21.15.44:0/3536746931 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52809a4f0 con 0x5611aed8cc80 2023-11-16T16:56:55.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.480+0000 7ff556a4eb80 1 -- 172.21.15.44:0/3536746931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed8cc80 msgr2=0x5611aed90b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:55.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.480+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/3536746931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed8cc80 0x5611aed90b80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff5280ca6b0 tx=0x7ff52809a180 comp rx=0 tx=0).stop 2023-11-16T16:56:55.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.482+0000 7ff556a4eb80 1 -- 172.21.15.44:0/3536746931 shutdown_connections 2023-11-16T16:56:55.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.482+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/3536746931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed910c0 0x5611aed91510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:55.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.482+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/3536746931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8c370 0x5611aed8c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:55.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.482+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/3536746931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed8cc80 0x5611aed90b80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:55.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.482+0000 7ff556a4eb80 1 -- 172.21.15.44:0/3536746931 >> 172.21.15.44:0/3536746931 conn(0x5611aed79360 msgr2=0x5611aed7eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:55.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.484+0000 7ff556a4eb80 1 -- 172.21.15.44:0/3536746931 shutdown_connections 2023-11-16T16:56:55.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.484+0000 7ff556a4eb80 1 -- 172.21.15.44:0/3536746931 wait complete. 2023-11-16T16:56:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.485+0000 7ff556a4eb80 1 -- start start 2023-11-16T16:56:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.486+0000 7ff556a4eb80 1 Processor -- start 2023-11-16T16:56:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.486+0000 7ff556a4eb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed8c370 0x5611aed95b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.486+0000 7ff556a4eb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8cc80 0x5611aed960d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.486+0000 7ff556a4eb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed910c0 0x5611aed96610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:55.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.486+0000 7ff556a4eb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611aec768e0 con 0x5611aed8c370 2023-11-16T16:56:55.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.486+0000 7ff556a4eb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611aec76580 con 0x5611aed8cc80 2023-11-16T16:56:55.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.486+0000 7ff556a4eb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611aeb58920 con 0x5611aed910c0 2023-11-16T16:56:55.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.486+0000 7ff556a4eb80 10 client.0 _ll_register_callbacks cb 0x5611aec94780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:55.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.487+0000 7ff54179c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed910c0 0x5611aed96610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:55.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.487+0000 7ff54179c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed910c0 0x5611aed96610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:35232/0 (socket says 172.21.15.44:35232) 2023-11-16T16:56:55.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.487+0000 7ff54179c700 1 -- 172.21.15.44:0/1554281928 learned_addr learned my addr 172.21.15.44:0/1554281928 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:56:55.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.489+0000 7ff53bfff700 1 --2- 172.21.15.44:0/1554281928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8cc80 0x5611aed960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:55.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.489+0000 7ff540f9b700 1 --2- 172.21.15.44:0/1554281928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed8c370 0x5611aed95b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:55.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.490+0000 7ff53bfff700 1 -- 172.21.15.44:0/1554281928 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed910c0 msgr2=0x5611aed96610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:55.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.490+0000 7ff53bfff700 1 --2- 172.21.15.44:0/1554281928 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed910c0 0x5611aed96610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:55.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.490+0000 7ff53bfff700 1 -- 172.21.15.44:0/1554281928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed8c370 msgr2=0x5611aed95b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:55.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.490+0000 7ff53bfff700 1 --2- 172.21.15.44:0/1554281928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed8c370 0x5611aed95b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:55.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.490+0000 7ff53bfff700 1 -- 172.21.15.44:0/1554281928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53c112470 con 0x5611aed8cc80 2023-11-16T16:56:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.490+0000 7ff540f9b700 1 --2- 172.21.15.44:0/1554281928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed8c370 0x5611aed95b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:56:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.491+0000 7ff53bfff700 1 --2- 172.21.15.44:0/1554281928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8cc80 0x5611aed960d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x5611aed839f0 tx=0x7ff5280cd220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c328fc2ee2e8099d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.491+0000 7ff51bfff700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:56:55.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.496+0000 7ff51bfff700 1 -- 172.21.15.44:0/1554281928 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528099730 con 0x5611aed8cc80 2023-11-16T16:56:55.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.498+0000 7ff51bfff700 1 -- 172.21.15.44:0/1554281928 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9511+0+0 (secure 0 0 0) 0x7ff52809a290 con 0x5611aed8cc80 2023-11-16T16:56:55.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.498+0000 7ff556a4eb80 1 -- 172.21.15.44:0/1554281928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5611aed76b20 con 0x5611aed8cc80 2023-11-16T16:56:55.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.499+0000 7ff51bfff700 1 -- 172.21.15.44:0/1554281928 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff52809a450 con 0x5611aed8cc80 2023-11-16T16:56:55.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.499+0000 7ff556a4eb80 10 client.26822 fetch_fsmap learned FSMap version 21 2023-11-16T16:56:55.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.499+0000 7ff556a4eb80 1 -- 172.21.15.44:0/1554281928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5611aec9b190 con 0x5611aed8cc80 2023-11-16T16:56:55.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.499+0000 7ff51bfff700 1 -- 172.21.15.44:0/1554281928 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5611aed76b20 con 0x5611aed8cc80 2023-11-16T16:56:55.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.500+0000 7ff51bfff700 1 -- 172.21.15.44:0/1554281928 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff5280ca1b0 con 0x5611aed8cc80 2023-11-16T16:56:55.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.501+0000 7ff556a4eb80 10 client.26822 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:56:55.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.501+0000 7ff556a4eb80 10 client.26822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:55.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.501+0000 7ff556a4eb80 20 client.26822 populate_metadata read hostname 'smithi044' 2023-11-16T16:56:55.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.501+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/1554281928 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5611aedbf6f0 0x5611aedc1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:55.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.501+0000 7ff556a4eb80 4 client.26822 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:56:55.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.501+0000 7ff556a4eb80 1 -- 172.21.15.44:0/1554281928 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5611aec970a0 con 0x5611aedbf6f0 2023-11-16T16:56:55.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.502+0000 7ff540f9b700 1 --2- 172.21.15.44:0/1554281928 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5611aedbf6f0 0x5611aedc1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:55.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.505+0000 7ff51bfff700 10 client.26822 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:56:55.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:56:55.505+0000 7ff540f9b700 1 --2- 172.21.15.44:0/1554281928 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5611aedbf6f0 0x5611aedc1ad0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98b9d4fbbdb6ed0c server_cookie=9e708a63635b5953 in_seq=0 out_seq=0 2023-11-16T16:56:55.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:55 smithi044 ceph-mon[128750]: pgmap v214: 129 pgs: 129 active+clean; 49 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 1.06k op/s 2023-11-16T16:56:55.644 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:55 smithi176 ceph-mon[131630]: pgmap v214: 129 pgs: 129 active+clean; 49 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 1.06k op/s 2023-11-16T16:56:55.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:55 smithi077 ceph-mon[131204]: pgmap v214: 129 pgs: 129 active+clean; 49 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 1.06k op/s 2023-11-16T16:56:57.646 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:57 smithi176 ceph-mon[131630]: pgmap v215: 129 pgs: 129 active+clean; 49 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 631 op/s 2023-11-16T16:56:57.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:57 smithi077 ceph-mon[131204]: pgmap v215: 129 pgs: 129 active+clean; 49 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 631 op/s 2023-11-16T16:56:57.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:57 smithi044 ceph-mon[128750]: pgmap v215: 129 pgs: 129 active+clean; 49 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 631 op/s 2023-11-16T16:56:59.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:56:59 smithi176 ceph-mon[131630]: pgmap v216: 129 pgs: 129 active+clean; 51 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 903 op/s 2023-11-16T16:56:59.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:56:59 smithi077 ceph-mon[131204]: pgmap v216: 129 pgs: 129 active+clean; 51 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 903 op/s 2023-11-16T16:56:59.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:56:59 smithi044 ceph-mon[128750]: pgmap v216: 129 pgs: 129 active+clean; 51 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 903 op/s 2023-11-16T16:57:01.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.094+0000 7fe626ff5700 1 -- 172.21.15.44:0/859566224 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564875b06f10 con 0x564875de58e0 2023-11-16T16:57:01.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.095+0000 7fe626ff5700 10 client.26810 handle_command_reply: tid=0 2023-11-16T16:57:01.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.095+0000 7fe65cdaeb80 1 client.26810 shutdown 2023-11-16T16:57:01.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.095+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/859566224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d708a0 msgr2=0x564875d75870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:01.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.095+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/859566224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d708a0 0x564875d75870 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe634004cb0 tx=0x7fe63400a900 comp rx=0 tx=0).stop 2023-11-16T16:57:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.095+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/859566224 shutdown_connections 2023-11-16T16:57:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.095+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/859566224 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x564875de58e0 0x564875de7cf0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.095+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/859566224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564875d708a0 0x564875d75870 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.095+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/859566224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x564875bc26b0 0x564875d77390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.095+0000 7fe65cdaeb80 1 --2- 172.21.15.44:0/859566224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564875d6c720 0x564875d75180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.095+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/859566224 >> 172.21.15.44:0/859566224 conn(0x564875d5bf50 msgr2=0x564875d6af00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.095+0000 7fe626ff5700 0 client.26810 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:01.100 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.095+0000 7ff51bfff700 1 -- 172.21.15.44:0/1554281928 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5611aec970a0 con 0x5611aedbf6f0 2023-11-16T16:57:01.100 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.095+0000 7ff51bfff700 10 client.26822 handle_command_reply: tid=0 2023-11-16T16:57:01.100 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.095+0000 7ff556a4eb80 1 client.26822 shutdown 2023-11-16T16:57:01.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.096+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/859566224 shutdown_connections 2023-11-16T16:57:01.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.096+0000 7fe65cdaeb80 1 -- 172.21.15.44:0/859566224 wait complete. 2023-11-16T16:57:01.101 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.096+0000 7ff556a4eb80 1 -- 172.21.15.44:0/1554281928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8cc80 msgr2=0x5611aed960d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:01.101 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.096+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/1554281928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8cc80 0x5611aed960d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x5611aed839f0 tx=0x7ff5280cd220 comp rx=0 tx=0).stop 2023-11-16T16:57:01.101 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.096+0000 7ff556a4eb80 1 -- 172.21.15.44:0/1554281928 shutdown_connections 2023-11-16T16:57:01.101 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.096+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/1554281928 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5611aedbf6f0 0x5611aedc1ad0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.102 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.096+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/1554281928 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed910c0 0x5611aed96610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.102 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.096+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/1554281928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed8c370 0x5611aed95b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.102 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.096+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/1554281928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8cc80 0x5611aed960d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.096+0000 7fe65cdaeb80 20 client.26810 trim_cache size 0 max 16384 2023-11-16T16:57:01.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.096+0000 7ff556a4eb80 1 -- 172.21.15.44:0/1554281928 >> 172.21.15.44:0/1554281928 conn(0x5611aed79360 msgr2=0x5611aed92570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:01.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.096+0000 7ff51bfff700 0 client.26822 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:01.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.096+0000 7ff556a4eb80 1 -- 172.21.15.44:0/1554281928 shutdown_connections 2023-11-16T16:57:01.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.096+0000 7ff556a4eb80 1 -- 172.21.15.44:0/1554281928 wait complete. 2023-11-16T16:57:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:57:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:57:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:57:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:57:01.104 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.097+0000 7ff556a4eb80 20 client.26822 trim_cache size 0 max 16384 2023-11-16T16:57:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.097+0000 7fe6493ff700 1 -- 172.21.15.44:0/4024543215 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe62c0a1490 msgr2=0x7fe62c0a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.097+0000 7fe6493ff700 1 --2- 172.21.15.44:0/4024543215 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe62c0a1490 0x7fe62c0a3940 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe634005cd0 tx=0x7fe634005c00 comp rx=0 tx=0).stop 2023-11-16T16:57:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.097+0000 7fe6493ff700 1 -- 172.21.15.44:0/4024543215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe644107d60 msgr2=0x7fe6441a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.097+0000 7fe6493ff700 1 --2- 172.21.15.44:0/4024543215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe644107d60 0x7fe6441a51b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe6380097d0 tx=0x7fe638009b90 comp rx=0 tx=0).stop 2023-11-16T16:57:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.097+0000 7fe6493ff700 1 -- 172.21.15.44:0/4024543215 shutdown_connections 2023-11-16T16:57:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.097+0000 7fe6493ff700 1 --2- 172.21.15.44:0/4024543215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe6441111d0 0x7fe6441a9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.097+0000 7fe6493ff700 1 --2- 172.21.15.44:0/4024543215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe644107d60 0x7fe6441a51b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.097+0000 7fe6493ff700 1 --2- 172.21.15.44:0/4024543215 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe62c0a1490 0x7fe62c0a3940 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.097+0000 7fe6493ff700 1 --2- 172.21.15.44:0/4024543215 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe644108670 0x7fe6441a56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.097+0000 7fe6493ff700 1 -- 172.21.15.44:0/4024543215 >> 172.21.15.44:0/4024543215 conn(0x7fe64406c510 msgr2=0x7fe644113330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.097+0000 7fe6493ff700 1 -- 172.21.15.44:0/4024543215 shutdown_connections 2023-11-16T16:57:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:01.097+0000 7fe6493ff700 1 -- 172.21.15.44:0/4024543215 wait complete. 2023-11-16T16:57:01.118 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.116+0000 7ff556a4eb80 1 Processor -- start 2023-11-16T16:57:01.118 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.116+0000 7ff556a4eb80 1 -- start start 2023-11-16T16:57:01.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.117+0000 7ff556a4eb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed8c370 0x5611aed87d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:01.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.117+0000 7ff556a4eb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8cc80 0x5611aed900d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:01.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.117+0000 7ff556a4eb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed910c0 0x5611aed90610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:01.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.117+0000 7ff556a4eb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611aed8fd50 con 0x5611aed910c0 2023-11-16T16:57:01.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.117+0000 7ff556a4eb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611aed8d110 con 0x5611aed8cc80 2023-11-16T16:57:01.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.117+0000 7ff556a4eb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611aed88260 con 0x5611aed8c370 2023-11-16T16:57:01.120 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.117+0000 7ff53bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8cc80 0x5611aed900d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:01.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.117+0000 7ff53bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8cc80 0x5611aed900d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:37750/0 (socket says 172.21.15.44:37750) 2023-11-16T16:57:01.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.117+0000 7ff53bfff700 1 -- 172.21.15.44:0/2647315009 learned_addr learned my addr 172.21.15.44:0/2647315009 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:01.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.117+0000 7ff54179c700 1 --2- 172.21.15.44:0/2647315009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed910c0 0x5611aed90610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:01.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.117+0000 7ff540f9b700 1 --2- 172.21.15.44:0/2647315009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed8c370 0x5611aed87d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:01.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.117+0000 7ff53bfff700 1 -- 172.21.15.44:0/2647315009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed8c370 msgr2=0x5611aed87d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:01.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.117+0000 7ff53bfff700 1 --2- 172.21.15.44:0/2647315009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed8c370 0x5611aed87d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.117+0000 7ff53bfff700 1 -- 172.21.15.44:0/2647315009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed910c0 msgr2=0x5611aed90610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:01.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.117+0000 7ff53bfff700 1 --2- 172.21.15.44:0/2647315009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed910c0 0x5611aed90610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.117+0000 7ff53bfff700 1 -- 172.21.15.44:0/2647315009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611aec9aee0 con 0x5611aed8cc80 2023-11-16T16:57:01.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.118+0000 7ff53bfff700 1 --2- 172.21.15.44:0/2647315009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8cc80 0x5611aed900d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff52800ce00 tx=0x7ff5280b3000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14b94de5506d4bc0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:01.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.118+0000 7ff5277fe700 1 -- 172.21.15.44:0/2647315009 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528099b70 con 0x5611aed8cc80 2023-11-16T16:57:01.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.118+0000 7ff5277fe700 1 -- 172.21.15.44:0/2647315009 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff528099730 con 0x5611aed8cc80 2023-11-16T16:57:01.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.118+0000 7ff5277fe700 1 -- 172.21.15.44:0/2647315009 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528016d20 con 0x5611aed8cc80 2023-11-16T16:57:01.126 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.119+0000 7ff556a4eb80 1 -- 172.21.15.44:0/2647315009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8cc80 msgr2=0x5611aed900d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:01.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.119+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/2647315009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8cc80 0x5611aed900d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff52800ce00 tx=0x7ff5280b3000 comp rx=0 tx=0).stop 2023-11-16T16:57:01.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.119+0000 7ff556a4eb80 1 -- 172.21.15.44:0/2647315009 shutdown_connections 2023-11-16T16:57:01.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.119+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/2647315009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed910c0 0x5611aed90610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.119+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/2647315009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed8c370 0x5611aed87d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.119+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/2647315009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8cc80 0x5611aed900d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.119+0000 7ff556a4eb80 1 -- 172.21.15.44:0/2647315009 >> 172.21.15.44:0/2647315009 conn(0x5611aed79360 msgr2=0x5611aed8f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:01.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.119+0000 7ff556a4eb80 1 -- 172.21.15.44:0/2647315009 shutdown_connections 2023-11-16T16:57:01.127 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.119+0000 7ff556a4eb80 1 -- 172.21.15.44:0/2647315009 wait complete. 2023-11-16T16:57:01.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.119+0000 7ff556a4eb80 1 -- start start 2023-11-16T16:57:01.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.119+0000 7ff556a4eb80 1 Processor -- start 2023-11-16T16:57:01.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.120+0000 7ff556a4eb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8c370 0x5611aed83d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:01.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.120+0000 7ff556a4eb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed8cc80 0x5611aed84240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:01.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.120+0000 7ff556a4eb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed910c0 0x5611aed84780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:01.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.120+0000 7ff556a4eb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611aee12660 con 0x5611aed8cc80 2023-11-16T16:57:01.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.120+0000 7ff556a4eb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611aee127d0 con 0x5611aed8c370 2023-11-16T16:57:01.128 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.120+0000 7ff556a4eb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611aee12990 con 0x5611aed910c0 2023-11-16T16:57:01.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.120+0000 7ff540f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8c370 0x5611aed83d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:01.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.120+0000 7ff556a4eb80 10 client.0 _ll_register_callbacks cb 0x7ff53000b7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:57:01.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.120+0000 7ff540f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8c370 0x5611aed83d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:37764/0 (socket says 172.21.15.44:37764) 2023-11-16T16:57:01.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.120+0000 7ff540f9b700 1 -- 172.21.15.44:0/657989391 learned_addr learned my addr 172.21.15.44:0/657989391 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:01.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.120+0000 7ff53bfff700 1 --2- 172.21.15.44:0/657989391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed8cc80 0x5611aed84240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:01.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.120+0000 7ff54179c700 1 --2- 172.21.15.44:0/657989391 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed910c0 0x5611aed84780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:01.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.120+0000 7ff540f9b700 1 -- 172.21.15.44:0/657989391 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed910c0 msgr2=0x5611aed84780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:01.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.121+0000 7ff540f9b700 1 --2- 172.21.15.44:0/657989391 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed910c0 0x5611aed84780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.121+0000 7ff540f9b700 1 -- 172.21.15.44:0/657989391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed8cc80 msgr2=0x5611aed84240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:01.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.121+0000 7ff540f9b700 1 --2- 172.21.15.44:0/657989391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed8cc80 0x5611aed84240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.121+0000 7ff540f9b700 1 -- 172.21.15.44:0/657989391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5611aec9b190 con 0x5611aed8c370 2023-11-16T16:57:01.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.121+0000 7ff54179c700 1 --2- 172.21.15.44:0/657989391 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed910c0 0x5611aed84780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:57:01.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.121+0000 7ff540f9b700 1 --2- 172.21.15.44:0/657989391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8c370 0x5611aed83d00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff5300098e0 tx=0x7ff530009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4d5453b68163c61 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:01.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.121+0000 7ff51bfff700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:57:01.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.121+0000 7ff51bfff700 1 -- 172.21.15.44:0/657989391 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53000bd00 con 0x5611aed8c370 2023-11-16T16:57:01.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.121+0000 7ff556a4eb80 1 -- 172.21.15.44:0/657989391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff528098170 con 0x5611aed8c370 2023-11-16T16:57:01.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.121+0000 7ff51bfff700 1 -- 172.21.15.44:0/657989391 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9511+0+0 (secure 0 0 0) 0x7ff53002e040 con 0x5611aed8c370 2023-11-16T16:57:01.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.121+0000 7ff51bfff700 1 -- 172.21.15.44:0/657989391 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff53000be60 con 0x5611aed8c370 2023-11-16T16:57:01.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.121+0000 7ff556a4eb80 10 client.26828 fetch_fsmap learned FSMap version 21 2023-11-16T16:57:01.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.121+0000 7ff556a4eb80 1 -- 172.21.15.44:0/657989391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5611aeaebc60 con 0x5611aed8c370 2023-11-16T16:57:01.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.122+0000 7ff51bfff700 1 -- 172.21.15.44:0/657989391 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff528098170 con 0x5611aed8c370 2023-11-16T16:57:01.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.122+0000 7ff51bfff700 1 -- 172.21.15.44:0/657989391 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff53000f4f0 con 0x5611aed8c370 2023-11-16T16:57:01.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.122+0000 7ff556a4eb80 10 client.26828 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:57:01.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.122+0000 7ff556a4eb80 10 client.26828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:57:01.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.122+0000 7ff556a4eb80 20 client.26828 populate_metadata read hostname 'smithi044' 2023-11-16T16:57:01.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.122+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/657989391 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5611aee14160 0x5611aee16570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:01.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.122+0000 7ff556a4eb80 4 client.26828 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:57:01.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.122+0000 7ff556a4eb80 1 -- 172.21.15.44:0/657989391 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5611aeca3660 con 0x5611aee14160 2023-11-16T16:57:01.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.122+0000 7ff53bfff700 1 --2- 172.21.15.44:0/657989391 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5611aee14160 0x5611aee16570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:01.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.122+0000 7ff53bfff700 1 --2- 172.21.15.44:0/657989391 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5611aee14160 0x5611aee16570 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48d976d1d6aade61 server_cookie=b1bb47f1b5f623ce in_seq=0 out_seq=0 2023-11-16T16:57:01.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.123+0000 7ff51bfff700 10 client.26828 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:01.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.124+0000 7ff51bfff700 1 -- 172.21.15.44:0/657989391 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x5611aeca3660 con 0x5611aee14160 2023-11-16T16:57:01.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.124+0000 7ff51bfff700 10 client.26828 handle_command_reply: tid=0 2023-11-16T16:57:01.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.124+0000 7ff556a4eb80 1 client.26828 shutdown 2023-11-16T16:57:01.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.124+0000 7ff556a4eb80 1 -- 172.21.15.44:0/657989391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8c370 msgr2=0x5611aed83d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:01.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.124+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/657989391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8c370 0x5611aed83d00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff5300098e0 tx=0x7ff530009ca0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.124+0000 7ff556a4eb80 1 -- 172.21.15.44:0/657989391 shutdown_connections 2023-11-16T16:57:01.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.124+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/657989391 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5611aee14160 0x5611aee16570 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.124+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/657989391 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5611aed910c0 0x5611aed84780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.124+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/657989391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611aed8c370 0x5611aed83d00 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.124+0000 7ff51bfff700 0 client.26828 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:01.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.124+0000 7ff556a4eb80 1 --2- 172.21.15.44:0/657989391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5611aed8cc80 0x5611aed84240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.124+0000 7ff556a4eb80 1 -- 172.21.15.44:0/657989391 >> 172.21.15.44:0/657989391 conn(0x5611aed79360 msgr2=0x5611aed934a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:01.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.124+0000 7ff556a4eb80 1 -- 172.21.15.44:0/657989391 shutdown_connections 2023-11-16T16:57:01.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.124+0000 7ff556a4eb80 1 -- 172.21.15.44:0/657989391 wait complete. 2023-11-16T16:57:01.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.125+0000 7ff556a4eb80 20 client.26828 trim_cache size 0 max 16384 2023-11-16T16:57:01.134 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:57:01.134 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "scrub active (8931 inodes in the stack)", 2023-11-16T16:57:01.134 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2023-11-16T16:57:01.134 INFO:teuthology.orchestra.run.smithi044.stdout: "58fab0cd-3198-4023-82f3-75a84820f59b": { 2023-11-16T16:57:01.134 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2023-11-16T16:57:01.135 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "58fab0cd-3198-4023-82f3-75a84820f59b", 2023-11-16T16:57:01.135 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2023-11-16T16:57:01.135 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:57:01.135 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:57:01.135 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:57:01.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.125+0000 7ff5431ff700 1 -- 172.21.15.44:0/930624780 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff52c0a13c0 msgr2=0x7ff52c0a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:01.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.125+0000 7ff5431ff700 1 --2- 172.21.15.44:0/930624780 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff52c0a13c0 0x7ff52c0a3870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff530015000 tx=0x7ff530005360 comp rx=0 tx=0).stop 2023-11-16T16:57:01.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.125+0000 7ff5431ff700 1 -- 172.21.15.44:0/930624780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff53c074ac0 msgr2=0x7ff53c19ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:01.136 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.125+0000 7ff5431ff700 1 --2- 172.21.15.44:0/930624780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff53c074ac0 0x7ff53c19ea40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff52800e970 tx=0x7ff52800ec80 comp rx=0 tx=0).stop 2023-11-16T16:57:01.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.125+0000 7ff5431ff700 1 -- 172.21.15.44:0/930624780 shutdown_connections 2023-11-16T16:57:01.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.125+0000 7ff5431ff700 1 --2- 172.21.15.44:0/930624780 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff53c10cec0 0x7ff53c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.125+0000 7ff5431ff700 1 --2- 172.21.15.44:0/930624780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff53c0746f0 0x7ff53c19e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.125+0000 7ff5431ff700 1 --2- 172.21.15.44:0/930624780 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff52c0a13c0 0x7ff52c0a3870 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.125+0000 7ff5431ff700 1 --2- 172.21.15.44:0/930624780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff53c074ac0 0x7ff53c19ea40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:01.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.125+0000 7ff5431ff700 1 -- 172.21.15.44:0/930624780 >> 172.21.15.44:0/930624780 conn(0x7ff53c077560 msgr2=0x7ff53c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:01.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.125+0000 7ff5431ff700 1 -- 172.21.15.44:0/930624780 shutdown_connections 2023-11-16T16:57:01.137 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:01.125+0000 7ff5431ff700 1 -- 172.21.15.44:0/930624780 wait complete. 2023-11-16T16:57:01.146 INFO:tasks.cephfs.filesystem:scrub status for tag:58fab0cd-3198-4023-82f3-75a84820f59b - {'path': '/', 'tag': '58fab0cd-3198-4023-82f3-75a84820f59b', 'options': 'recursive,force'} 2023-11-16T16:57:01.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:01 smithi044 ceph-mon[128750]: pgmap v217: 129 pgs: 129 active+clean; 59 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 632 op/s 2023-11-16T16:57:01.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:01 smithi176 ceph-mon[131630]: pgmap v217: 129 pgs: 129 active+clean; 59 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 632 op/s 2023-11-16T16:57:01.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:01 smithi077 ceph-mon[131204]: pgmap v217: 129 pgs: 129 active+clean; 59 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 632 op/s 2023-11-16T16:57:03.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:03 smithi077 ceph-mon[131204]: pgmap v218: 129 pgs: 129 active+clean; 51 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 790 op/s 2023-11-16T16:57:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:03 smithi044 ceph-mon[128750]: pgmap v218: 129 pgs: 129 active+clean; 51 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 790 op/s 2023-11-16T16:57:03.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:03 smithi176 ceph-mon[131630]: pgmap v218: 129 pgs: 129 active+clean; 51 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 790 op/s 2023-11-16T16:57:05.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:05 smithi077 ceph-mon[131204]: pgmap v219: 129 pgs: 129 active+clean; 16 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.8 MiB/s wr, 1.58k op/s 2023-11-16T16:57:05.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:05 smithi044 ceph-mon[128750]: pgmap v219: 129 pgs: 129 active+clean; 16 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.8 MiB/s wr, 1.58k op/s 2023-11-16T16:57:05.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:05 smithi176 ceph-mon[131630]: pgmap v219: 129 pgs: 129 active+clean; 16 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.8 MiB/s wr, 1.58k op/s 2023-11-16T16:57:07.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:07 smithi077 ceph-mon[131204]: pgmap v220: 129 pgs: 129 active+clean; 16 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 1.31k op/s 2023-11-16T16:57:07.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:07 smithi044 ceph-mon[128750]: pgmap v220: 129 pgs: 129 active+clean; 16 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 1.31k op/s 2023-11-16T16:57:07.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:07 smithi176 ceph-mon[131630]: pgmap v220: 129 pgs: 129 active+clean; 16 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 1.31k op/s 2023-11-16T16:57:09.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:09 smithi077 ceph-mon[131204]: pgmap v221: 129 pgs: 129 active+clean; 20 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 2.31k op/s 2023-11-16T16:57:09.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:09 smithi044 ceph-mon[128750]: pgmap v221: 129 pgs: 129 active+clean; 20 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 2.31k op/s 2023-11-16T16:57:09.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:09 smithi176 ceph-mon[131630]: pgmap v221: 129 pgs: 129 active+clean; 20 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 2.31k op/s 2023-11-16T16:57:11.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:11 smithi077 ceph-mon[131204]: pgmap v222: 129 pgs: 129 active+clean; 20 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 2.12k op/s 2023-11-16T16:57:11.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:11 smithi044 ceph-mon[128750]: pgmap v222: 129 pgs: 129 active+clean; 20 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 2.12k op/s 2023-11-16T16:57:11.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:11 smithi176 ceph-mon[131630]: pgmap v222: 129 pgs: 129 active+clean; 20 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 2.12k op/s 2023-11-16T16:57:13.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:13 smithi077 ceph-mon[131204]: pgmap v223: 129 pgs: 129 active+clean; 20 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 2.16k op/s 2023-11-16T16:57:13.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:13 smithi044 ceph-mon[128750]: pgmap v223: 129 pgs: 129 active+clean; 20 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 2.16k op/s 2023-11-16T16:57:13.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:13 smithi176 ceph-mon[131630]: pgmap v223: 129 pgs: 129 active+clean; 20 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 2.16k op/s 2023-11-16T16:57:14.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.356+0000 7f0febfff700 1 -- 172.21.15.44:0/693079261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fe409d5f0 msgr2=0x7f0fe409d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:14.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.356+0000 7f0febfff700 1 --2- 172.21.15.44:0/693079261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fe409d5f0 0x7f0fe409d9c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0fec065e20 tx=0x7f0fec067260 comp rx=0 tx=0).stop 2023-11-16T16:57:14.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.359+0000 7f0febfff700 1 -- 172.21.15.44:0/693079261 shutdown_connections 2023-11-16T16:57:14.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.359+0000 7f0febfff700 1 --2- 172.21.15.44:0/693079261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe4097f70 0x7f0fe40a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.359+0000 7f0febfff700 1 --2- 172.21.15.44:0/693079261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fe409d5f0 0x7f0fe409d9c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.359+0000 7f0febfff700 1 --2- 172.21.15.44:0/693079261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fe40975c0 0x7f0fe4097a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.359+0000 7f0febfff700 1 -- 172.21.15.44:0/693079261 >> 172.21.15.44:0/693079261 conn(0x7f0fe4093080 msgr2=0x7f0fe4095490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:14.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.359+0000 7f0febfff700 1 -- 172.21.15.44:0/693079261 shutdown_connections 2023-11-16T16:57:14.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.359+0000 7f0febfff700 1 -- 172.21.15.44:0/693079261 wait complete. 2023-11-16T16:57:14.365 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.363+0000 7f0febfff700 1 Processor -- start 2023-11-16T16:57:14.365 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.364+0000 7f0febfff700 1 -- start start 2023-11-16T16:57:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.364+0000 7f0febfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe40975c0 0x7f0fe412d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.364+0000 7f0febfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fe4097f70 0x7f0fe412dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.364+0000 7f0febfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fe409d5f0 0x7f0fe4132010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.364+0000 7f0febfff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe4132650 con 0x7f0fe4097f70 2023-11-16T16:57:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.364+0000 7f0febfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe41327c0 con 0x7f0fe40975c0 2023-11-16T16:57:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.364+0000 7f0febfff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe4132980 con 0x7f0fe409d5f0 2023-11-16T16:57:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.364+0000 7f0feaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe40975c0 0x7f0fe412d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.364+0000 7f0feaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe40975c0 0x7f0fe412d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:55584/0 (socket says 172.21.15.44:55584) 2023-11-16T16:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.364+0000 7f0feaffd700 1 -- 172.21.15.44:0/3824600941 learned_addr learned my addr 172.21.15.44:0/3824600941 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.365+0000 7f0feb7fe700 1 --2- 172.21.15.44:0/3824600941 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fe409d5f0 0x7f0fe4132010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.365+0000 7f0feaffd700 1 -- 172.21.15.44:0/3824600941 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fe409d5f0 msgr2=0x7f0fe4132010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.365+0000 7f0feaffd700 1 --2- 172.21.15.44:0/3824600941 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fe409d5f0 0x7f0fe4132010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.365+0000 7f0feaffd700 1 -- 172.21.15.44:0/3824600941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fe4097f70 msgr2=0x7f0fe412dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.365+0000 7f0feaffd700 1 --2- 172.21.15.44:0/3824600941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fe4097f70 0x7f0fe412dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.365+0000 7f0feaffd700 1 -- 172.21.15.44:0/3824600941 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec0670d0 con 0x7f0fe40975c0 2023-11-16T16:57:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.365+0000 7f0feaffd700 1 --2- 172.21.15.44:0/3824600941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe40975c0 0x7f0fe412d670 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0fec04fde0 tx=0x7f0fec07e9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.366+0000 7f0fd3fff700 1 -- 172.21.15.44:0/3824600941 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec073ea0 con 0x7f0fe40975c0 2023-11-16T16:57:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.366+0000 7f0fd3fff700 1 -- 172.21.15.44:0/3824600941 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fec07f8e0 con 0x7f0fe40975c0 2023-11-16T16:57:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.366+0000 7f0fd3fff700 1 -- 172.21.15.44:0/3824600941 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec07a040 con 0x7f0fe40975c0 2023-11-16T16:57:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.366+0000 7f0febfff700 1 -- 172.21.15.44:0/3824600941 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe40a7590 con 0x7f0fe40975c0 2023-11-16T16:57:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.367+0000 7f0febfff700 1 -- 172.21.15.44:0/3824600941 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe40a7400 con 0x7f0fe40975c0 2023-11-16T16:57:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.372+0000 7f0fd3fff700 1 -- 172.21.15.44:0/3824600941 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f0fec07f420 con 0x7f0fe40975c0 2023-11-16T16:57:14.379 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.377+0000 7f0fd3fff700 1 --2- 172.21.15.44:0/3824600941 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0fd40a1340 0x7f0fd40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.377+0000 7f0fd3fff700 1 -- 172.21.15.44:0/3824600941 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f0fec138bb0 con 0x7f0fe40975c0 2023-11-16T16:57:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.378+0000 7f0fea7fc700 1 --2- 172.21.15.44:0/3824600941 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0fd40a1340 0x7f0fd40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.379+0000 7f0fea7fc700 1 --2- 172.21.15.44:0/3824600941 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0fd40a1340 0x7f0fd40a37f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc005950 tx=0x7f0fdc0058e0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.383+0000 7f1004cbfb80 1 Processor -- start 2023-11-16T16:57:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.384+0000 7f1004cbfb80 1 -- start start 2023-11-16T16:57:14.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.384+0000 7f1004cbfb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e7ba6b0 0x55654e9671b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.384+0000 7f1004cbfb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 0x55654e96b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.384+0000 7f1004cbfb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e967b60 0x55654e96bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.385+0000 7f0fea7fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 0x55654e96b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.385+0000 7f0fea7fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 0x55654e96b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:60498/0 (socket says 172.21.15.44:60498) 2023-11-16T16:57:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.385+0000 7f0feaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e7ba6b0 0x55654e9671b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.385+0000 7f0feaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e7ba6b0 0x55654e9671b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:55592/0 (socket says 172.21.15.44:55592) 2023-11-16T16:57:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.385+0000 7f0feb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e967b60 0x55654e96bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.385+0000 7f0feb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e967b60 0x55654e96bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40926/0 (socket says 172.21.15.44:40926) 2023-11-16T16:57:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.385+0000 7f0fea7fc700 1 -- 172.21.15.44:0/305618754 learned_addr learned my addr 172.21.15.44:0/305618754 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.385+0000 7f1004cbfb80 1 -- 172.21.15.44:0/305618754 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654e864000 con 0x55654e967b60 2023-11-16T16:57:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.385+0000 7f1004cbfb80 1 -- 172.21.15.44:0/305618754 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654e8b6840 con 0x55654e7ba6b0 2023-11-16T16:57:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.385+0000 7f1004cbfb80 1 -- 172.21.15.44:0/305618754 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654e8afcd0 con 0x55654e9676f0 2023-11-16T16:57:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.386+0000 7f0feaffd700 1 -- 172.21.15.44:0/305618754 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 msgr2=0x55654e96b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.386+0000 7f0feaffd700 1 --2- 172.21.15.44:0/305618754 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 0x55654e96b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.386+0000 7f0feaffd700 1 -- 172.21.15.44:0/305618754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e967b60 msgr2=0x55654e96bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.386+0000 7f0feaffd700 1 --2- 172.21.15.44:0/305618754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e967b60 0x55654e96bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.386+0000 7f0feaffd700 1 -- 172.21.15.44:0/305618754 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55654e7315d0 con 0x55654e7ba6b0 2023-11-16T16:57:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.387+0000 7f0feaffd700 1 --2- 172.21.15.44:0/305618754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e7ba6b0 0x55654e9671b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0fec135ed0 tx=0x7f0fec1054f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13ec4098a7590bab server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.387+0000 7f0fd17fa700 1 -- 172.21.15.44:0/305618754 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec109910 con 0x55654e7ba6b0 2023-11-16T16:57:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.387+0000 7f0fd17fa700 1 -- 172.21.15.44:0/305618754 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fec10dca0 con 0x55654e7ba6b0 2023-11-16T16:57:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.387+0000 7f0fd17fa700 1 -- 172.21.15.44:0/305618754 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec105b40 con 0x55654e7ba6b0 2023-11-16T16:57:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.389+0000 7f1004cbfb80 1 -- 172.21.15.44:0/305618754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e7ba6b0 msgr2=0x55654e9671b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.389+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/305618754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e7ba6b0 0x55654e9671b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0fec135ed0 tx=0x7f0fec1054f0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.390+0000 7f1004cbfb80 1 -- 172.21.15.44:0/305618754 shutdown_connections 2023-11-16T16:57:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.390+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/305618754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e967b60 0x55654e96bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.390+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/305618754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e7ba6b0 0x55654e9671b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.390+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/305618754 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 0x55654e96b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.390+0000 7f1004cbfb80 1 -- 172.21.15.44:0/305618754 >> 172.21.15.44:0/305618754 conn(0x55654e9541a0 msgr2=0x55654e9593e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:14.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.390+0000 7f1004cbfb80 1 -- 172.21.15.44:0/305618754 shutdown_connections 2023-11-16T16:57:14.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.390+0000 7f1004cbfb80 1 -- 172.21.15.44:0/305618754 wait complete. 2023-11-16T16:57:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.392+0000 7f1004cbfb80 1 -- start start 2023-11-16T16:57:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.392+0000 7f1004cbfb80 1 Processor -- start 2023-11-16T16:57:14.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.393+0000 7f1004cbfb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e7ba6b0 0x55654e976130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.393+0000 7f1004cbfb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e9676f0 0x55654e976670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.393+0000 7f1004cbfb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e967b60 0x55654e970380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.393+0000 7f1004cbfb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654e851880 con 0x55654e7ba6b0 2023-11-16T16:57:14.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.393+0000 7f1004cbfb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654e851520 con 0x55654e9676f0 2023-11-16T16:57:14.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.393+0000 7f1004cbfb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654e7338c0 con 0x55654e967b60 2023-11-16T16:57:14.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.393+0000 7f1004cbfb80 10 client.0 _ll_register_callbacks cb 0x55654e86f720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:57:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.394+0000 7f0feb7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e967b60 0x55654e970380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.394+0000 7f0feb7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e967b60 0x55654e970380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:60508/0 (socket says 172.21.15.44:60508) 2023-11-16T16:57:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.394+0000 7f0feb7fe700 1 -- 172.21.15.44:0/2745387727 learned_addr learned my addr 172.21.15.44:0/2745387727 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.394+0000 7f0fea7fc700 1 --2- 172.21.15.44:0/2745387727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e9676f0 0x55654e976670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.394+0000 7f0feb7fe700 1 -- 172.21.15.44:0/2745387727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e9676f0 msgr2=0x55654e976670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.394+0000 7f0feb7fe700 1 --2- 172.21.15.44:0/2745387727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e9676f0 0x55654e976670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.394+0000 7f0feaffd700 1 --2- 172.21.15.44:0/2745387727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e7ba6b0 0x55654e976130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.395+0000 7f0feb7fe700 1 -- 172.21.15.44:0/2745387727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e7ba6b0 msgr2=0x55654e976130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.395+0000 7f0feb7fe700 1 --2- 172.21.15.44:0/2745387727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e7ba6b0 0x55654e976130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.395+0000 7f0feb7fe700 1 -- 172.21.15.44:0/2745387727 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe40a7400 con 0x55654e967b60 2023-11-16T16:57:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.395+0000 7f0fea7fc700 1 --2- 172.21.15.44:0/2745387727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e9676f0 0x55654e976670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:57:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.395+0000 7f0feaffd700 1 --2- 172.21.15.44:0/2745387727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e7ba6b0 0x55654e976130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:57:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.396+0000 7f0feb7fe700 1 --2- 172.21.15.44:0/2745387727 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e967b60 0x55654e970380 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0011a30 tx=0x7f0fe0011df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1828e49968ccfb4e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.396+0000 7f0fceff5700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:57:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.398+0000 7f0fceff5700 1 -- 172.21.15.44:0/2745387727 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe000c810 con 0x55654e967b60 2023-11-16T16:57:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.398+0000 7f0fceff5700 1 -- 172.21.15.44:0/2745387727 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f0fe000fcc0 con 0x55654e967b60 2023-11-16T16:57:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.399+0000 7f1004cbfb80 1 -- 172.21.15.44:0/2745387727 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55654e951960 con 0x55654e967b60 2023-11-16T16:57:14.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.400+0000 7f0fceff5700 1 -- 172.21.15.44:0/2745387727 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fe000fe80 con 0x55654e967b60 2023-11-16T16:57:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.400+0000 7f1004cbfb80 10 client.26797 fetch_fsmap learned FSMap version 21 2023-11-16T16:57:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.400+0000 7f1004cbfb80 1 -- 172.21.15.44:0/2745387727 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55654e876130 con 0x55654e967b60 2023-11-16T16:57:14.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.402+0000 7f0fceff5700 1 -- 172.21.15.44:0/2745387727 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55654e951960 con 0x55654e967b60 2023-11-16T16:57:14.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.402+0000 7f0fceff5700 1 -- 172.21.15.44:0/2745387727 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0fe0021070 con 0x55654e967b60 2023-11-16T16:57:14.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.402+0000 7f1004cbfb80 10 client.26797 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:57:14.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.402+0000 7f1004cbfb80 10 client.26797 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:57:14.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.402+0000 7f1004cbfb80 20 client.26797 populate_metadata read hostname 'smithi044' 2023-11-16T16:57:14.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.402+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/2745387727 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55654e9806a0 0x55654e982ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.402+0000 7f1004cbfb80 4 client.26797 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:57:14.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.402+0000 7f1004cbfb80 1 -- 172.21.15.44:0/2745387727 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55654e872040 con 0x55654e9806a0 2023-11-16T16:57:14.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.403+0000 7f0feaffd700 1 --2- 172.21.15.44:0/2745387727 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55654e9806a0 0x55654e982ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:14.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.405+0000 7f0fceff5700 10 client.26797 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:14.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.405+0000 7f0feaffd700 1 --2- 172.21.15.44:0/2745387727 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55654e9806a0 0x55654e982ab0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb1654ca01daad7c server_cookie=42e2f0d1f6d65de9 in_seq=0 out_seq=0 2023-11-16T16:57:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.467+0000 7f0fceff5700 1 -- 172.21.15.44:0/2745387727 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55654e872040 con 0x55654e9806a0 2023-11-16T16:57:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.467+0000 7f0fceff5700 10 client.26797 handle_command_reply: tid=0 2023-11-16T16:57:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.468+0000 7f1004cbfb80 1 client.26797 shutdown 2023-11-16T16:57:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.469+0000 7f1004cbfb80 1 -- 172.21.15.44:0/2745387727 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e967b60 msgr2=0x55654e970380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.469+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/2745387727 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e967b60 0x55654e970380 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0011a30 tx=0x7f0fe0011df0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.470+0000 7f1004cbfb80 1 -- 172.21.15.44:0/2745387727 shutdown_connections 2023-11-16T16:57:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.470+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/2745387727 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55654e9806a0 0x55654e982ab0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.470+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/2745387727 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e967b60 0x55654e970380 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.470+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/2745387727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e7ba6b0 0x55654e976130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.470+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/2745387727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e9676f0 0x55654e976670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.470+0000 7f1004cbfb80 1 -- 172.21.15.44:0/2745387727 >> 172.21.15.44:0/2745387727 conn(0x55654e9541a0 msgr2=0x55654e96d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:14.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.470+0000 7f0fceff5700 0 client.26797 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:14.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.470+0000 7f1004cbfb80 1 -- 172.21.15.44:0/2745387727 shutdown_connections 2023-11-16T16:57:14.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.470+0000 7f1004cbfb80 1 -- 172.21.15.44:0/2745387727 wait complete. 2023-11-16T16:57:14.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.471+0000 7f1004cbfb80 20 client.26797 trim_cache size 0 max 16384 2023-11-16T16:57:14.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.504+0000 7f1004cbfb80 1 Processor -- start 2023-11-16T16:57:14.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.506+0000 7f1004cbfb80 1 -- start start 2023-11-16T16:57:14.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.506+0000 7f1004cbfb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e7ba6b0 0x55654e959180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.506+0000 7f1004cbfb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 0x55654e96fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.506+0000 7f1004cbfb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e967b60 0x55654e970160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.506+0000 7f1004cbfb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654e96e9c0 con 0x55654e967b60 2023-11-16T16:57:14.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.506+0000 7f1004cbfb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654e96ace0 con 0x55654e7ba6b0 2023-11-16T16:57:14.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.506+0000 7f1004cbfb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654e95a130 con 0x55654e9676f0 2023-11-16T16:57:14.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.507+0000 7f0fea7fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 0x55654e96fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:14.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.507+0000 7f0fea7fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 0x55654e96fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:60522/0 (socket says 172.21.15.44:60522) 2023-11-16T16:57:14.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.507+0000 7f0fea7fc700 1 -- 172.21.15.44:0/1548094491 learned_addr learned my addr 172.21.15.44:0/1548094491 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:14.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.507+0000 7f0feaffd700 1 --2- 172.21.15.44:0/1548094491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e7ba6b0 0x55654e959180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:14.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.508+0000 7f0fea7fc700 1 -- 172.21.15.44:0/1548094491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e7ba6b0 msgr2=0x55654e959180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:14.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.508+0000 7f0fea7fc700 1 --2- 172.21.15.44:0/1548094491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e7ba6b0 0x55654e959180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.508+0000 7f0fea7fc700 1 -- 172.21.15.44:0/1548094491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e967b60 msgr2=0x55654e970160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:14.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.508+0000 7f0fea7fc700 1 --2- 172.21.15.44:0/1548094491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e967b60 0x55654e970160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.508+0000 7f0fea7fc700 1 -- 172.21.15.44:0/1548094491 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55654e955940 con 0x55654e9676f0 2023-11-16T16:57:14.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.508+0000 7f0fea7fc700 1 --2- 172.21.15.44:0/1548094491 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 0x55654e96fc20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x55654e95a100 tx=0x7f0fdc011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ec1266d3a81928a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:14.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.509+0000 7f0fd1ffb700 1 -- 172.21.15.44:0/1548094491 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc022070 con 0x55654e9676f0 2023-11-16T16:57:14.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.509+0000 7f0fd1ffb700 1 -- 172.21.15.44:0/1548094491 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fdc00ed80 con 0x55654e9676f0 2023-11-16T16:57:14.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.509+0000 7f0fd1ffb700 1 -- 172.21.15.44:0/1548094491 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc005060 con 0x55654e9676f0 2023-11-16T16:57:14.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.510+0000 7f1004cbfb80 1 -- 172.21.15.44:0/1548094491 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 msgr2=0x55654e96fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:14.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.510+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/1548094491 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 0x55654e96fc20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x55654e95a100 tx=0x7f0fdc011000 comp rx=0 tx=0).stop 2023-11-16T16:57:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.511+0000 7f1004cbfb80 1 -- 172.21.15.44:0/1548094491 shutdown_connections 2023-11-16T16:57:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.511+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/1548094491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e967b60 0x55654e970160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.511+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/1548094491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e7ba6b0 0x55654e959180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.511+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/1548094491 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 0x55654e96fc20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.511+0000 7f1004cbfb80 1 -- 172.21.15.44:0/1548094491 >> 172.21.15.44:0/1548094491 conn(0x55654e9541a0 msgr2=0x55654e96a170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.511+0000 7f1004cbfb80 1 -- 172.21.15.44:0/1548094491 shutdown_connections 2023-11-16T16:57:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.512+0000 7f1004cbfb80 1 -- 172.21.15.44:0/1548094491 wait complete. 2023-11-16T16:57:14.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.512+0000 7f1004cbfb80 1 -- start start 2023-11-16T16:57:14.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.513+0000 7f1004cbfb80 1 Processor -- start 2023-11-16T16:57:14.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.514+0000 7f1004cbfb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e7ba6b0 0x55654e9cd540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.514+0000 7f1004cbfb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 0x55654e9cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.514+0000 7f1004cbfb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e967b60 0x55654e9c84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.514+0000 7f1004cbfb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654e9ce1d0 con 0x55654e7ba6b0 2023-11-16T16:57:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.514+0000 7f1004cbfb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654e9ce340 con 0x55654e967b60 2023-11-16T16:57:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.514+0000 7f1004cbfb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654e9c8a30 con 0x55654e9676f0 2023-11-16T16:57:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.514+0000 7f1004cbfb80 10 client.0 _ll_register_callbacks cb 0x7f0fe000cd10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:57:14.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.515+0000 7f0feb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e967b60 0x55654e9c84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:14.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.515+0000 7f0feaffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e7ba6b0 0x55654e9cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:14.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.515+0000 7f0feb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e967b60 0x55654e9c84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:55628/0 (socket says 172.21.15.44:55628) 2023-11-16T16:57:14.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.515+0000 7f0feb7fe700 1 -- 172.21.15.44:0/700790783 learned_addr learned my addr 172.21.15.44:0/700790783 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:14.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.515+0000 7f0fea7fc700 1 --2- 172.21.15.44:0/700790783 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 0x55654e9cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:14.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.515+0000 7f0feb7fe700 1 -- 172.21.15.44:0/700790783 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 msgr2=0x55654e9cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:14.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.515+0000 7f0feb7fe700 1 --2- 172.21.15.44:0/700790783 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 0x55654e9cda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.515+0000 7f0feb7fe700 1 -- 172.21.15.44:0/700790783 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e7ba6b0 msgr2=0x55654e9cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:14.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.515+0000 7f0feb7fe700 1 --2- 172.21.15.44:0/700790783 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e7ba6b0 0x55654e9cd540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:14.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.515+0000 7f0feb7fe700 1 -- 172.21.15.44:0/700790783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fdc00ea60 con 0x55654e967b60 2023-11-16T16:57:14.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.516+0000 7f0feb7fe700 1 --2- 172.21.15.44:0/700790783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e967b60 0x55654e9c84f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0022830 tx=0x7f0fe000fcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2c4ef69f3733734 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:14.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.516+0000 7f0fceff5700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:57:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.516+0000 7f0fceff5700 1 -- 172.21.15.44:0/700790783 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe000a750 con 0x55654e967b60 2023-11-16T16:57:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.516+0000 7f0fceff5700 1 -- 172.21.15.44:0/700790783 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f0fe0000e40 con 0x55654e967b60 2023-11-16T16:57:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.516+0000 7f1004cbfb80 1 -- 172.21.15.44:0/700790783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0fec10dca0 con 0x55654e967b60 2023-11-16T16:57:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.521+0000 7f0fceff5700 1 -- 172.21.15.44:0/700790783 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fe000c7f0 con 0x55654e967b60 2023-11-16T16:57:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.521+0000 7f1004cbfb80 10 client.26843 fetch_fsmap learned FSMap version 21 2023-11-16T16:57:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.521+0000 7f1004cbfb80 1 -- 172.21.15.44:0/700790783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55654e6ddd80 con 0x55654e967b60 2023-11-16T16:57:14.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.523+0000 7f0fceff5700 1 -- 172.21.15.44:0/700790783 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fec10dca0 con 0x55654e967b60 2023-11-16T16:57:14.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.524+0000 7f0fceff5700 1 -- 172.21.15.44:0/700790783 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0fe0021070 con 0x55654e967b60 2023-11-16T16:57:14.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.524+0000 7f1004cbfb80 10 client.26843 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:57:14.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.524+0000 7f1004cbfb80 10 client.26843 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:57:14.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.524+0000 7f1004cbfb80 20 client.26843 populate_metadata read hostname 'smithi044' 2023-11-16T16:57:14.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.524+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/700790783 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55654e9e61e0 0x55654e9e85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:14.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.524+0000 7f1004cbfb80 4 client.26843 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:57:14.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.524+0000 7f1004cbfb80 1 -- 172.21.15.44:0/700790783 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55654e87e600 con 0x55654e9e61e0 2023-11-16T16:57:14.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.525+0000 7f0feaffd700 1 --2- 172.21.15.44:0/700790783 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55654e9e61e0 0x55654e9e85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:14.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.525+0000 7f0feaffd700 1 --2- 172.21.15.44:0/700790783 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55654e9e61e0 0x55654e9e85f0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b74944c9797a9ab server_cookie=3d9c625bd58a4517 in_seq=0 out_seq=0 2023-11-16T16:57:14.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:14.525+0000 7f0fceff5700 10 client.26843 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:14.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:14 smithi077 ceph-mon[131204]: osdmap e95: 12 total, 12 up, 12 in 2023-11-16T16:57:14.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:14 smithi044 ceph-mon[128750]: osdmap e95: 12 total, 12 up, 12 in 2023-11-16T16:57:14.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:14 smithi176 ceph-mon[131630]: osdmap e95: 12 total, 12 up, 12 in 2023-11-16T16:57:15.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:15 smithi077 ceph-mon[131204]: pgmap v225: 129 pgs: 129 active+clean; 34 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 1.75k op/s 2023-11-16T16:57:15.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:15 smithi044 ceph-mon[128750]: pgmap v225: 129 pgs: 129 active+clean; 34 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 1.75k op/s 2023-11-16T16:57:15.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:15 smithi176 ceph-mon[131630]: pgmap v225: 129 pgs: 129 active+clean; 34 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 1.75k op/s 2023-11-16T16:57:17.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:17 smithi077 ceph-mon[131204]: pgmap v226: 129 pgs: 129 active+clean; 34 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 1.75k op/s 2023-11-16T16:57:17.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:17 smithi044 ceph-mon[128750]: pgmap v226: 129 pgs: 129 active+clean; 34 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 1.75k op/s 2023-11-16T16:57:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:17 smithi176 ceph-mon[131630]: pgmap v226: 129 pgs: 129 active+clean; 34 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 1.75k op/s 2023-11-16T16:57:19.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:19 smithi077 ceph-mon[131204]: pgmap v227: 129 pgs: 129 active+clean; 50 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 1.29k op/s 2023-11-16T16:57:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:19 smithi044 ceph-mon[128750]: pgmap v227: 129 pgs: 129 active+clean; 50 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 1.29k op/s 2023-11-16T16:57:19.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:19 smithi176 ceph-mon[131630]: pgmap v227: 129 pgs: 129 active+clean; 50 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 1.29k op/s 2023-11-16T16:57:21.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:21 smithi044 ceph-mon[128750]: pgmap v228: 129 pgs: 129 active+clean; 50 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 1.25k op/s 2023-11-16T16:57:21.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:21 smithi176 ceph-mon[131630]: pgmap v228: 129 pgs: 129 active+clean; 50 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 1.25k op/s 2023-11-16T16:57:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:21 smithi077 ceph-mon[131204]: pgmap v228: 129 pgs: 129 active+clean; 50 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 1.25k op/s 2023-11-16T16:57:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.427+0000 7f0fceff5700 1 -- 172.21.15.44:0/700790783 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55654e87e600 con 0x55654e9e61e0 2023-11-16T16:57:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.427+0000 7f0fceff5700 10 client.26843 handle_command_reply: tid=0 2023-11-16T16:57:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.428+0000 7f1004cbfb80 1 client.26843 shutdown 2023-11-16T16:57:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.429+0000 7f1004cbfb80 1 -- 172.21.15.44:0/700790783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e967b60 msgr2=0x55654e9c84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.430+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/700790783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e967b60 0x55654e9c84f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0022830 tx=0x7f0fe000fcf0 comp rx=0 tx=0).stop 2023-11-16T16:57:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.430+0000 7f1004cbfb80 1 -- 172.21.15.44:0/700790783 shutdown_connections 2023-11-16T16:57:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.430+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/700790783 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55654e9e61e0 0x55654e9e85f0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.430+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/700790783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55654e967b60 0x55654e9c84f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.430+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/700790783 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654e7ba6b0 0x55654e9cd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.430+0000 7f1004cbfb80 1 --2- 172.21.15.44:0/700790783 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55654e9676f0 0x55654e9cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.430+0000 7f1004cbfb80 1 -- 172.21.15.44:0/700790783 >> 172.21.15.44:0/700790783 conn(0x55654e9541a0 msgr2=0x55654e95d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:23.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.430+0000 7f0fceff5700 0 client.26843 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:23.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.432+0000 7f1004cbfb80 1 -- 172.21.15.44:0/700790783 shutdown_connections 2023-11-16T16:57:23.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.432+0000 7f1004cbfb80 1 -- 172.21.15.44:0/700790783 wait complete. 2023-11-16T16:57:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.435+0000 7f1004cbfb80 20 client.26843 trim_cache size 0 max 16384 2023-11-16T16:57:23.436 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:57:23.436 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:57:23.436 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:57:23.436 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:57:23.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.436+0000 7f0febfff700 1 -- 172.21.15.44:0/3824600941 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0fd40a1340 msgr2=0x7f0fd40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:23.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.436+0000 7f0febfff700 1 --2- 172.21.15.44:0/3824600941 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0fd40a1340 0x7f0fd40a37f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc005950 tx=0x7f0fdc0058e0 comp rx=0 tx=0).stop 2023-11-16T16:57:23.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.436+0000 7f0febfff700 1 -- 172.21.15.44:0/3824600941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe40975c0 msgr2=0x7f0fe412d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:23.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.436+0000 7f0febfff700 1 --2- 172.21.15.44:0/3824600941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe40975c0 0x7f0fe412d670 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0fec04fde0 tx=0x7f0fec07e9a0 comp rx=0 tx=0).stop 2023-11-16T16:57:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.437+0000 7f0febfff700 1 -- 172.21.15.44:0/3824600941 shutdown_connections 2023-11-16T16:57:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.437+0000 7f0febfff700 1 --2- 172.21.15.44:0/3824600941 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0fd40a1340 0x7f0fd40a37f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.437+0000 7f0febfff700 1 --2- 172.21.15.44:0/3824600941 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fe409d5f0 0x7f0fe4132010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.437+0000 7f0febfff700 1 --2- 172.21.15.44:0/3824600941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe40975c0 0x7f0fe412d670 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.437+0000 7f0febfff700 1 --2- 172.21.15.44:0/3824600941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0fe4097f70 0x7f0fe412dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.437+0000 7f0febfff700 1 -- 172.21.15.44:0/3824600941 >> 172.21.15.44:0/3824600941 conn(0x7f0fe4093080 msgr2=0x7f0fe4094650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.437+0000 7f0febfff700 1 -- 172.21.15.44:0/3824600941 shutdown_connections 2023-11-16T16:57:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:23.437+0000 7f0febfff700 1 -- 172.21.15.44:0/3824600941 wait complete. 2023-11-16T16:57:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:23 smithi044 ceph-mon[128750]: pgmap v229: 129 pgs: 129 active+clean; 62 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 1.22k op/s 2023-11-16T16:57:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:23 smithi044 ceph-mon[128750]: osdmap e96: 12 total, 12 up, 12 in 2023-11-16T16:57:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:23 smithi176 ceph-mon[131630]: pgmap v229: 129 pgs: 129 active+clean; 62 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 1.22k op/s 2023-11-16T16:57:23.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:23 smithi176 ceph-mon[131630]: osdmap e96: 12 total, 12 up, 12 in 2023-11-16T16:57:23.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:23 smithi077 ceph-mon[131204]: pgmap v229: 129 pgs: 129 active+clean; 62 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 1.22k op/s 2023-11-16T16:57:23.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:23 smithi077 ceph-mon[131204]: osdmap e96: 12 total, 12 up, 12 in 2023-11-16T16:57:25.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:25 smithi044 ceph-mon[128750]: pgmap v231: 129 pgs: 129 active+clean; 38 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 1.08k op/s 2023-11-16T16:57:25.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:25 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:57:25.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:25 smithi176 ceph-mon[131630]: pgmap v231: 129 pgs: 129 active+clean; 38 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 1.08k op/s 2023-11-16T16:57:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:25 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:57:26.055 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:25 smithi077 ceph-mon[131204]: pgmap v231: 129 pgs: 129 active+clean; 38 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 1.08k op/s 2023-11-16T16:57:26.055 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:25 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:57:27.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:27 smithi044 ceph-mon[128750]: pgmap v232: 129 pgs: 129 active+clean; 38 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 1.08k op/s 2023-11-16T16:57:27.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:27 smithi176 ceph-mon[131630]: pgmap v232: 129 pgs: 129 active+clean; 38 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 1.08k op/s 2023-11-16T16:57:28.113 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:27 smithi077 ceph-mon[131204]: pgmap v232: 129 pgs: 129 active+clean; 38 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 1.08k op/s 2023-11-16T16:57:29.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:29 smithi044 ceph-mon[128750]: pgmap v233: 129 pgs: 129 active+clean; 35 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 775 op/s 2023-11-16T16:57:29.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:29 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:57:29.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:29 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:57:29.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:29 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:57:29.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:29 smithi176 ceph-mon[131630]: pgmap v233: 129 pgs: 129 active+clean; 35 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 775 op/s 2023-11-16T16:57:29.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:29 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:57:29.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:29 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:57:29.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:29 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:57:29.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:29 smithi077 ceph-mon[131204]: pgmap v233: 129 pgs: 129 active+clean; 35 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 775 op/s 2023-11-16T16:57:29.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:29 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:57:29.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:29 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:57:29.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:29 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:57:30.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:30 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:57:30.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:30 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T16:57:30.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:30 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:57:30.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:30 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:57:30.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:30 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:57:30.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:30 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T16:57:30.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:30 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:57:30.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:30 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:57:30.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:30 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:57:30.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:30 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T16:57:30.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:30 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:57:30.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:30 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:57:31.147 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:57:31.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.505+0000 7f7fd0dff700 1 -- 172.21.15.44:0/708119239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fcc100cc0 msgr2=0x7f7fcc101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:31.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.505+0000 7f7fd0dff700 1 --2- 172.21.15.44:0/708119239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fcc100cc0 0x7f7fcc101090 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0099c0 tx=0x7f7fbc009cd0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.505+0000 7f7fd0dff700 1 -- 172.21.15.44:0/708119239 shutdown_connections 2023-11-16T16:57:31.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.505+0000 7f7fd0dff700 1 --2- 172.21.15.44:0/708119239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fcc105620 0x7f7fcc10d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.505+0000 7f7fd0dff700 1 --2- 172.21.15.44:0/708119239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fcc100cc0 0x7f7fcc101090 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.505+0000 7f7fd0dff700 1 --2- 172.21.15.44:0/708119239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7fcc101660 0x7f7fcc104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.505+0000 7f7fd0dff700 1 -- 172.21.15.44:0/708119239 >> 172.21.15.44:0/708119239 conn(0x7f7fcc077560 msgr2=0x7f7fcc077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:31.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.505+0000 7f7fd0dff700 1 -- 172.21.15.44:0/708119239 shutdown_connections 2023-11-16T16:57:31.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.506+0000 7f7fd0dff700 1 -- 172.21.15.44:0/708119239 wait complete. 2023-11-16T16:57:31.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.507+0000 7f7fd0dff700 1 Processor -- start 2023-11-16T16:57:31.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.507+0000 7f7fd0dff700 1 -- start start 2023-11-16T16:57:31.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.507+0000 7f7fd0dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fcc100cc0 0x7f7fcc1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.507+0000 7f7fd0dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fcc101660 0x7f7fcc198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.507+0000 7f7fd0dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7fcc105620 0x7f7fcc19bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.507+0000 7f7fd0dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc19c5e0 con 0x7f7fcc100cc0 2023-11-16T16:57:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.507+0000 7f7fd0dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc19c750 con 0x7f7fcc101660 2023-11-16T16:57:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.507+0000 7f7fd0dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc19c910 con 0x7f7fcc105620 2023-11-16T16:57:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.507+0000 7f7fcad9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7fcc105620 0x7f7fcc19bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.507+0000 7f7fc9d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fcc101660 0x7f7fcc198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.507+0000 7f7fca59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fcc100cc0 0x7f7fcc1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.508+0000 7f7fca59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fcc100cc0 0x7f7fcc1986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59558/0 (socket says 172.21.15.44:59558) 2023-11-16T16:57:31.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.508+0000 7f7fca59c700 1 -- 172.21.15.44:0/2059587421 learned_addr learned my addr 172.21.15.44:0/2059587421 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:31.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.508+0000 7f7fca59c700 1 -- 172.21.15.44:0/2059587421 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7fcc105620 msgr2=0x7f7fcc19bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:31.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.508+0000 7f7fca59c700 1 --2- 172.21.15.44:0/2059587421 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7fcc105620 0x7f7fcc19bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.508+0000 7f7fca59c700 1 -- 172.21.15.44:0/2059587421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fcc101660 msgr2=0x7f7fcc198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:31.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.508+0000 7f7fca59c700 1 --2- 172.21.15.44:0/2059587421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fcc101660 0x7f7fcc198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.508+0000 7f7fca59c700 1 -- 172.21.15.44:0/2059587421 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc0009520 con 0x7f7fcc100cc0 2023-11-16T16:57:31.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.508+0000 7f7fc9d9b700 1 --2- 172.21.15.44:0/2059587421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fcc101660 0x7f7fcc198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:57:31.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.508+0000 7f7fcad9d700 1 --2- 172.21.15.44:0/2059587421 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7fcc105620 0x7f7fcc19bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:57:31.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.508+0000 7f7fca59c700 1 --2- 172.21.15.44:0/2059587421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fcc100cc0 0x7f7fcc1986f0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc00b580 tx=0x7f7fbc00bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:31.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.509+0000 7f7fbb7fe700 1 -- 172.21.15.44:0/2059587421 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc01d070 con 0x7f7fcc100cc0 2023-11-16T16:57:31.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.509+0000 7f7fd0dff700 1 -- 172.21.15.44:0/2059587421 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fbc0095f0 con 0x7f7fcc100cc0 2023-11-16T16:57:31.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.509+0000 7f7fd0dff700 1 -- 172.21.15.44:0/2059587421 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fcc1103d0 con 0x7f7fcc100cc0 2023-11-16T16:57:31.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.509+0000 7f7fbb7fe700 1 -- 172.21.15.44:0/2059587421 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7fbc005050 con 0x7f7fcc100cc0 2023-11-16T16:57:31.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.509+0000 7f7fbb7fe700 1 -- 172.21.15.44:0/2059587421 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc021650 con 0x7f7fcc100cc0 2023-11-16T16:57:31.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.513+0000 7f7fbb7fe700 1 -- 172.21.15.44:0/2059587421 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f7fbc00fac0 con 0x7f7fcc100cc0 2023-11-16T16:57:31.517 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.514+0000 7f7fe46a2b80 1 Processor -- start 2023-11-16T16:57:31.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.514+0000 7f7fbb7fe700 1 --2- 172.21.15.44:0/2059587421 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7fb40a1340 0x7f7fb40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.514+0000 7f7fe46a2b80 1 -- start start 2023-11-16T16:57:31.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.514+0000 7f7fbb7fe700 1 -- 172.21.15.44:0/2059587421 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f7fbc0cd9c0 con 0x7f7fcc100cc0 2023-11-16T16:57:31.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.515+0000 7f7fcad9d700 1 --2- 172.21.15.44:0/2059587421 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7fb40a1340 0x7f7fb40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:31.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.515+0000 7f7fe46a2b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da64740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.515+0000 7f7fe46a2b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da64c80 0x561b3da68b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.515+0000 7f7fe46a2b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da690c0 0x561b3da69510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.515+0000 7f7fe46a2b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b3d961060 con 0x561b3da64c80 2023-11-16T16:57:31.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.515+0000 7f7fe46a2b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b3d9b38a0 con 0x561b3da64370 2023-11-16T16:57:31.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.515+0000 7f7fe46a2b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b3d9acd30 con 0x561b3da690c0 2023-11-16T16:57:31.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.515+0000 7f7fca59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da64740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:31.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.515+0000 7f7fca59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da64740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:39370/0 (socket says 172.21.15.44:39370) 2023-11-16T16:57:31.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.515+0000 7f7fca59c700 1 -- 172.21.15.44:0/4173880334 learned_addr learned my addr 172.21.15.44:0/4173880334 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:31.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.516+0000 7f7fc9d9b700 1 --2- 172.21.15.44:0/4173880334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da64c80 0x561b3da68b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:31.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.516+0000 7f7fcad9d700 1 --2- 172.21.15.44:0/2059587421 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7fb40a1340 0x7f7fb40a37f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4003e30 tx=0x7f7fc400b040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:31.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.516+0000 7f7fca59c700 1 -- 172.21.15.44:0/4173880334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da690c0 msgr2=0x561b3da69510 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T16:57:31.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.516+0000 7f7fca59c700 1 --2- 172.21.15.44:0/4173880334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da690c0 0x561b3da69510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.516+0000 7f7fca59c700 1 -- 172.21.15.44:0/4173880334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da64c80 msgr2=0x561b3da68b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:31.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.516+0000 7f7fca59c700 1 --2- 172.21.15.44:0/4173880334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da64c80 0x561b3da68b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.516+0000 7f7fca59c700 1 -- 172.21.15.44:0/4173880334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b3d82e630 con 0x561b3da64370 2023-11-16T16:57:31.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.516+0000 7f7fca59c700 1 --2- 172.21.15.44:0/4173880334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da64740 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc09a3e0 tx=0x7f7fbc09a6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc4a6019fcb0ea70 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:31.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.516+0000 7f7fb8ff9700 1 -- 172.21.15.44:0/4173880334 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc098a80 con 0x561b3da64370 2023-11-16T16:57:31.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.516+0000 7f7fb8ff9700 1 -- 172.21.15.44:0/4173880334 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7fbc09a940 con 0x561b3da64370 2023-11-16T16:57:31.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.516+0000 7f7fb8ff9700 1 -- 172.21.15.44:0/4173880334 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc0a2870 con 0x561b3da64370 2023-11-16T16:57:31.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.517+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/4173880334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 msgr2=0x561b3da64740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:31.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.517+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/4173880334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da64740 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc09a3e0 tx=0x7f7fbc09a6e0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.517+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/4173880334 shutdown_connections 2023-11-16T16:57:31.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.518+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/4173880334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da690c0 0x561b3da69510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.518+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/4173880334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da64740 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.518+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/4173880334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da64c80 0x561b3da68b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.518+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/4173880334 >> 172.21.15.44:0/4173880334 conn(0x561b3da51360 msgr2=0x561b3da56b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:31.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.518+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/4173880334 shutdown_connections 2023-11-16T16:57:31.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.518+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/4173880334 wait complete. 2023-11-16T16:57:31.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.518+0000 7f7fe46a2b80 1 -- start start 2023-11-16T16:57:31.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.519+0000 7f7fe46a2b80 1 Processor -- start 2023-11-16T16:57:31.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.519+0000 7f7fe46a2b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.519+0000 7f7fe46a2b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da64c80 0x561b3da6db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.519+0000 7f7fe46a2b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da690c0 0x561b3da6e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.519+0000 7f7fe46a2b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b3d94e8e0 con 0x561b3da690c0 2023-11-16T16:57:31.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.519+0000 7f7fe46a2b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b3d94e580 con 0x561b3da64370 2023-11-16T16:57:31.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.519+0000 7f7fe46a2b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b3d830920 con 0x561b3da64c80 2023-11-16T16:57:31.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.519+0000 7f7fe46a2b80 10 client.0 _ll_register_callbacks cb 0x561b3d96c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:57:31.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.519+0000 7f7fca59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:31.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.519+0000 7f7fca59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:39374/0 (socket says 172.21.15.44:39374) 2023-11-16T16:57:31.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.519+0000 7f7fca59c700 1 -- 172.21.15.44:0/663071945 learned_addr learned my addr 172.21.15.44:0/663071945 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:31.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.520+0000 7f7fca59c700 1 -- 172.21.15.44:0/663071945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da64c80 msgr2=0x561b3da6db60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:57:31.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.520+0000 7f7fca59c700 1 --2- 172.21.15.44:0/663071945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da64c80 0x561b3da6db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.520+0000 7f7fca59c700 1 -- 172.21.15.44:0/663071945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da690c0 msgr2=0x561b3da6e2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:57:31.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.520+0000 7f7fcad9d700 1 --2- 172.21.15.44:0/663071945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da690c0 0x561b3da6e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:31.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.520+0000 7f7fca59c700 1 --2- 172.21.15.44:0/663071945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da690c0 0x561b3da6e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.520+0000 7f7fca59c700 1 -- 172.21.15.44:0/663071945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fcc1103d0 con 0x561b3da64370 2023-11-16T16:57:31.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.520+0000 7f7fcad9d700 1 --2- 172.21.15.44:0/663071945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da690c0 0x561b3da6e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:57:31.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.520+0000 7f7fca59c700 1 --2- 172.21.15.44:0/663071945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da708b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0a2570 tx=0x7f7fbc09e870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59f0b355ab5a3bb7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:31.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.520+0000 7f7fb1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:57:31.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.521+0000 7f7fb1ffb700 1 -- 172.21.15.44:0/663071945 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc0a2810 con 0x561b3da64370 2023-11-16T16:57:31.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.521+0000 7f7fb1ffb700 1 -- 172.21.15.44:0/663071945 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f7fbc098b40 con 0x561b3da64370 2023-11-16T16:57:31.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.521+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/663071945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561b3da4eb20 con 0x561b3da64370 2023-11-16T16:57:31.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.521+0000 7f7fb1ffb700 1 -- 172.21.15.44:0/663071945 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7fbc0850b0 con 0x561b3da64370 2023-11-16T16:57:31.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.521+0000 7f7fe46a2b80 10 client.26855 fetch_fsmap learned FSMap version 21 2023-11-16T16:57:31.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.521+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/663071945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x561b3d973190 con 0x561b3da64370 2023-11-16T16:57:31.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.521+0000 7f7fb1ffb700 1 -- 172.21.15.44:0/663071945 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561b3da4eb20 con 0x561b3da64370 2023-11-16T16:57:31.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.521+0000 7f7fb1ffb700 1 -- 172.21.15.44:0/663071945 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7fbc09e170 con 0x561b3da64370 2023-11-16T16:57:31.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.521+0000 7f7fe46a2b80 10 client.26855 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:57:31.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.521+0000 7f7fe46a2b80 10 client.26855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:57:31.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.522+0000 7f7fe46a2b80 20 client.26855 populate_metadata read hostname 'smithi044' 2023-11-16T16:57:31.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.522+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/663071945 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561b3da5f2b0 0x561b3da616c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.522+0000 7f7fe46a2b80 4 client.26855 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:57:31.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.522+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/663071945 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b3d96f0a0 con 0x561b3da5f2b0 2023-11-16T16:57:31.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.522+0000 7f7fc9d9b700 1 --2- 172.21.15.44:0/663071945 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561b3da5f2b0 0x561b3da616c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:31.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.522+0000 7f7fc9d9b700 1 --2- 172.21.15.44:0/663071945 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561b3da5f2b0 0x561b3da616c0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70796eb426c1d7a5 server_cookie=2aae8f6129d4c41e in_seq=0 out_seq=0 2023-11-16T16:57:31.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.522+0000 7f7fb1ffb700 10 client.26855 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:31.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.525+0000 7f7fb1ffb700 1 -- 172.21.15.44:0/663071945 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x561b3d96f0a0 con 0x561b3da5f2b0 2023-11-16T16:57:31.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.525+0000 7f7fb1ffb700 10 client.26855 handle_command_reply: tid=0 2023-11-16T16:57:31.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.525+0000 7f7fe46a2b80 1 client.26855 shutdown 2023-11-16T16:57:31.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.526+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/663071945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 msgr2=0x561b3da708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:31.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.526+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/663071945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da708b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0a2570 tx=0x7f7fbc09e870 comp rx=0 tx=0).stop 2023-11-16T16:57:31.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.526+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/663071945 shutdown_connections 2023-11-16T16:57:31.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.526+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/663071945 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561b3da5f2b0 0x561b3da616c0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.526+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/663071945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da690c0 0x561b3da6e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.526+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/663071945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da708b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.526+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/663071945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da64c80 0x561b3da6db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.526+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/663071945 >> 172.21.15.44:0/663071945 conn(0x561b3da51360 msgr2=0x561b3da6a570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:31.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.526+0000 7f7fb1ffb700 0 client.26855 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:31.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.526+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/663071945 shutdown_connections 2023-11-16T16:57:31.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.526+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/663071945 wait complete. 2023-11-16T16:57:31.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.526+0000 7f7fe46a2b80 20 client.26855 trim_cache size 0 max 16384 2023-11-16T16:57:31.546 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.546+0000 7f7fe46a2b80 1 Processor -- start 2023-11-16T16:57:31.546 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.546+0000 7f7fe46a2b80 1 -- start start 2023-11-16T16:57:31.547 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.546+0000 7f7fe46a2b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da64370 0x561b3da74ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.546+0000 7f7fe46a2b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da64c80 0x561b3da70370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.546+0000 7f7fe46a2b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da690c0 0x561b3da708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.546+0000 7f7fe46a2b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b3da51170 con 0x561b3da64370 2023-11-16T16:57:31.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.546+0000 7f7fe46a2b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b3da67d60 con 0x561b3da690c0 2023-11-16T16:57:31.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.546+0000 7f7fe46a2b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b3da75420 con 0x561b3da64c80 2023-11-16T16:57:31.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.547+0000 7f7fc9d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da64c80 0x561b3da70370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:31.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.547+0000 7f7fcad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da690c0 0x561b3da708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:31.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.547+0000 7f7fc9d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da64c80 0x561b3da70370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:40142/0 (socket says 172.21.15.44:40142) 2023-11-16T16:57:31.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.547+0000 7f7fc9d9b700 1 -- 172.21.15.44:0/228879719 learned_addr learned my addr 172.21.15.44:0/228879719 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:31.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.547+0000 7f7fca59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da64370 0x561b3da74ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:31.548 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.547+0000 7f7fc9d9b700 1 -- 172.21.15.44:0/228879719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da690c0 msgr2=0x561b3da708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:31.549 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.547+0000 7f7fc9d9b700 1 --2- 172.21.15.44:0/228879719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da690c0 0x561b3da708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.549 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.547+0000 7f7fc9d9b700 1 -- 172.21.15.44:0/228879719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da64370 msgr2=0x561b3da74ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:31.549 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.547+0000 7f7fc9d9b700 1 --2- 172.21.15.44:0/228879719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da64370 0x561b3da74ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.549 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.547+0000 7f7fc9d9b700 1 -- 172.21.15.44:0/228879719 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b3d972ee0 con 0x561b3da64c80 2023-11-16T16:57:31.549 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.547+0000 7f7fc9d9b700 1 --2- 172.21.15.44:0/228879719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da64c80 0x561b3da70370 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc06a350 tx=0x7f7fc001ab40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e0d7db8b73d79f6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:31.551 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.547+0000 7f7fb97fa700 1 -- 172.21.15.44:0/228879719 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0018040 con 0x561b3da64c80 2023-11-16T16:57:31.551 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.547+0000 7f7fb97fa700 1 -- 172.21.15.44:0/228879719 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7fc00044e0 con 0x561b3da64c80 2023-11-16T16:57:31.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.548+0000 7f7fb97fa700 1 -- 172.21.15.44:0/228879719 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0003ea0 con 0x561b3da64c80 2023-11-16T16:57:31.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.548+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/228879719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da64c80 msgr2=0x561b3da70370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:31.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.548+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/228879719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da64c80 0x561b3da70370 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc06a350 tx=0x7f7fc001ab40 comp rx=0 tx=0).stop 2023-11-16T16:57:31.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.548+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/228879719 shutdown_connections 2023-11-16T16:57:31.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.548+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/228879719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da690c0 0x561b3da708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.548+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/228879719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da64370 0x561b3da74ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.548+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/228879719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da64c80 0x561b3da70370 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.548+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/228879719 >> 172.21.15.44:0/228879719 conn(0x561b3da51360 msgr2=0x561b3da67220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:31.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.548+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/228879719 shutdown_connections 2023-11-16T16:57:31.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.549+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/228879719 wait complete. 2023-11-16T16:57:31.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.549+0000 7f7fe46a2b80 1 -- start start 2023-11-16T16:57:31.552 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.549+0000 7f7fe46a2b80 1 Processor -- start 2023-11-16T16:57:31.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fe46a2b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da72e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fe46a2b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da64c80 0x561b3da73370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fe46a2b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da690c0 0x561b3da738b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fe46a2b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b3dad9200 con 0x561b3da64c80 2023-11-16T16:57:31.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fe46a2b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b3dad9370 con 0x561b3da64370 2023-11-16T16:57:31.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fe46a2b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b3dad9530 con 0x561b3da690c0 2023-11-16T16:57:31.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fe46a2b80 10 client.0 _ll_register_callbacks cb 0x7f7fbc09a2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:57:31.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fca59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da72e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:31.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fca59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da72e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:39380/0 (socket says 172.21.15.44:39380) 2023-11-16T16:57:31.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fca59c700 1 -- 172.21.15.44:0/516317698 learned_addr learned my addr 172.21.15.44:0/516317698 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:31.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fc9d9b700 1 --2- 172.21.15.44:0/516317698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da64c80 0x561b3da73370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:31.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fca59c700 1 -- 172.21.15.44:0/516317698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da690c0 msgr2=0x561b3da738b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T16:57:31.553 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fca59c700 1 --2- 172.21.15.44:0/516317698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da690c0 0x561b3da738b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.554 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fca59c700 1 -- 172.21.15.44:0/516317698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da64c80 msgr2=0x561b3da73370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:31.554 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fca59c700 1 --2- 172.21.15.44:0/516317698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da64c80 0x561b3da73370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.554 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fca59c700 1 -- 172.21.15.44:0/516317698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc0009e50 con 0x561b3da64370 2023-11-16T16:57:31.554 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fca59c700 1 --2- 172.21.15.44:0/516317698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da72e30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0036c0 tx=0x7f7fbc0967b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c48cb0ab4f15ec87 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:31.554 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.550+0000 7f7fb1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:57:31.554 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.551+0000 7f7fb1ffb700 1 -- 172.21.15.44:0/516317698 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc096870 con 0x561b3da64370 2023-11-16T16:57:31.554 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.551+0000 7f7fb1ffb700 1 -- 172.21.15.44:0/516317698 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f7fbc098b40 con 0x561b3da64370 2023-11-16T16:57:31.554 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.551+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/516317698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7fbc09a940 con 0x561b3da64370 2023-11-16T16:57:31.554 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.551+0000 7f7fb1ffb700 1 -- 172.21.15.44:0/516317698 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7fbc0988d0 con 0x561b3da64370 2023-11-16T16:57:31.554 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.551+0000 7f7fe46a2b80 10 client.26861 fetch_fsmap learned FSMap version 21 2023-11-16T16:57:31.554 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.551+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/516317698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x561b3d7c3c60 con 0x561b3da64370 2023-11-16T16:57:31.555 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.551+0000 7f7fb1ffb700 1 -- 172.21.15.44:0/516317698 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7fbc09a940 con 0x561b3da64370 2023-11-16T16:57:31.555 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.551+0000 7f7fb1ffb700 1 -- 172.21.15.44:0/516317698 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7fbc0850b0 con 0x561b3da64370 2023-11-16T16:57:31.555 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.551+0000 7f7fe46a2b80 10 client.26861 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:57:31.555 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.551+0000 7f7fe46a2b80 10 client.26861 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:57:31.555 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.551+0000 7f7fe46a2b80 20 client.26861 populate_metadata read hostname 'smithi044' 2023-11-16T16:57:31.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.551+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/516317698 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561b3dada790 0x561b3dadcb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:31.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.551+0000 7f7fe46a2b80 4 client.26861 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:57:31.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.551+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/516317698 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561b3d97b660 con 0x561b3dada790 2023-11-16T16:57:31.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.552+0000 7f7fc9d9b700 1 --2- 172.21.15.44:0/516317698 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561b3dada790 0x561b3dadcb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:31.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.552+0000 7f7fc9d9b700 1 --2- 172.21.15.44:0/516317698 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561b3dada790 0x561b3dadcb70 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dbe41b5149f4f8a server_cookie=45a32df1a1af2a56 in_seq=0 out_seq=0 2023-11-16T16:57:31.556 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.552+0000 7f7fb1ffb700 10 client.26861 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:31.559 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.558+0000 7f7fb1ffb700 1 -- 172.21.15.44:0/516317698 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x561b3d97b660 con 0x561b3dada790 2023-11-16T16:57:31.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.558+0000 7f7fb1ffb700 10 client.26861 handle_command_reply: tid=0 2023-11-16T16:57:31.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.559+0000 7f7fe46a2b80 1 client.26861 shutdown 2023-11-16T16:57:31.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.559+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/516317698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 msgr2=0x561b3da72e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:31.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.559+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/516317698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da72e30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0036c0 tx=0x7f7fbc0967b0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.559+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/516317698 shutdown_connections 2023-11-16T16:57:31.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.559+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/516317698 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561b3dada790 0x561b3dadcb70 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.559+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/516317698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561b3da690c0 0x561b3da738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.559+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/516317698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b3da64370 0x561b3da72e30 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.559+0000 7f7fb1ffb700 0 client.26861 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:31.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.559+0000 7f7fe46a2b80 1 --2- 172.21.15.44:0/516317698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b3da64c80 0x561b3da73370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.559+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/516317698 >> 172.21.15.44:0/516317698 conn(0x561b3da51360 msgr2=0x561b3da612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:31.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.559+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/516317698 shutdown_connections 2023-11-16T16:57:31.560 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.559+0000 7f7fe46a2b80 1 -- 172.21.15.44:0/516317698 wait complete. 2023-11-16T16:57:31.561 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.559+0000 7f7fe46a2b80 20 client.26861 trim_cache size 0 max 16384 2023-11-16T16:57:31.561 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:57:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "scrub active (1668 inodes in the stack)", 2023-11-16T16:57:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2023-11-16T16:57:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: "58fab0cd-3198-4023-82f3-75a84820f59b": { 2023-11-16T16:57:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2023-11-16T16:57:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "58fab0cd-3198-4023-82f3-75a84820f59b", 2023-11-16T16:57:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2023-11-16T16:57:31.561 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:57:31.562 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:57:31.562 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:57:31.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.560+0000 7f7fd0dff700 1 -- 172.21.15.44:0/2059587421 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7fb40a1340 msgr2=0x7f7fb40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:31.562 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.560+0000 7f7fd0dff700 1 --2- 172.21.15.44:0/2059587421 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7fb40a1340 0x7f7fb40a37f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4003e30 tx=0x7f7fc400b040 comp rx=0 tx=0).stop 2023-11-16T16:57:31.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.560+0000 7f7fd0dff700 1 -- 172.21.15.44:0/2059587421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fcc100cc0 msgr2=0x7f7fcc1986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:31.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.560+0000 7f7fd0dff700 1 --2- 172.21.15.44:0/2059587421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fcc100cc0 0x7f7fcc1986f0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc00b580 tx=0x7f7fbc00bfd0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.560+0000 7f7fd0dff700 1 -- 172.21.15.44:0/2059587421 shutdown_connections 2023-11-16T16:57:31.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.560+0000 7f7fd0dff700 1 --2- 172.21.15.44:0/2059587421 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7fb40a1340 0x7f7fb40a37f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.560+0000 7f7fd0dff700 1 --2- 172.21.15.44:0/2059587421 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7fcc105620 0x7f7fcc19bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.560+0000 7f7fd0dff700 1 --2- 172.21.15.44:0/2059587421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fcc100cc0 0x7f7fcc1986f0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.560+0000 7f7fd0dff700 1 --2- 172.21.15.44:0/2059587421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fcc101660 0x7f7fcc198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:31.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.560+0000 7f7fd0dff700 1 -- 172.21.15.44:0/2059587421 >> 172.21.15.44:0/2059587421 conn(0x7f7fcc077560 msgr2=0x7f7fcc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:31.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.560+0000 7f7fd0dff700 1 -- 172.21.15.44:0/2059587421 shutdown_connections 2023-11-16T16:57:31.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:57:31.560+0000 7f7fd0dff700 1 -- 172.21.15.44:0/2059587421 wait complete. 2023-11-16T16:57:31.578 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:31 smithi044 ceph-mon[128750]: pgmap v234: 129 pgs: 129 active+clean; 35 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 745 op/s 2023-11-16T16:57:31.581 INFO:tasks.cephfs.filesystem:scrub status for tag:58fab0cd-3198-4023-82f3-75a84820f59b - {'path': '/', 'tag': '58fab0cd-3198-4023-82f3-75a84820f59b', 'options': 'recursive,force'} 2023-11-16T16:57:31.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:31 smithi176 ceph-mon[131630]: pgmap v234: 129 pgs: 129 active+clean; 35 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 745 op/s 2023-11-16T16:57:32.255 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:31 smithi077 ceph-mon[131204]: pgmap v234: 129 pgs: 129 active+clean; 35 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 745 op/s 2023-11-16T16:57:33.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:33 smithi044 ceph-mon[128750]: pgmap v235: 129 pgs: 129 active+clean; 43 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 752 op/s 2023-11-16T16:57:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:33 smithi176 ceph-mon[131630]: pgmap v235: 129 pgs: 129 active+clean; 43 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 752 op/s 2023-11-16T16:57:33.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:33 smithi077 ceph-mon[131204]: pgmap v235: 129 pgs: 129 active+clean; 43 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 752 op/s 2023-11-16T16:57:35.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:35 smithi044 ceph-mon[128750]: pgmap v236: 129 pgs: 129 active+clean; 55 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 750 op/s 2023-11-16T16:57:35.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:35 smithi176 ceph-mon[131630]: pgmap v236: 129 pgs: 129 active+clean; 55 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 750 op/s 2023-11-16T16:57:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:35 smithi077 ceph-mon[131204]: pgmap v236: 129 pgs: 129 active+clean; 55 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 750 op/s 2023-11-16T16:57:36.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.629+0000 7f8737bff700 1 -- 172.21.15.44:0/2214630701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8730100f00 msgr2=0x7f8730104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:36.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.629+0000 7f8737bff700 1 --2- 172.21.15.44:0/2214630701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8730100f00 0x7f8730104ec0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8720009990 tx=0x7f8720009ca0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.629+0000 7f8737bff700 1 -- 172.21.15.44:0/2214630701 shutdown_connections 2023-11-16T16:57:36.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.629+0000 7f8737bff700 1 --2- 172.21.15.44:0/2214630701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87301055a0 0x7f873010d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.629+0000 7f8737bff700 1 --2- 172.21.15.44:0/2214630701 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8730100560 0x7f8730100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.629+0000 7f8737bff700 1 --2- 172.21.15.44:0/2214630701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8730100f00 0x7f8730104ec0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.629+0000 7f8737bff700 1 -- 172.21.15.44:0/2214630701 >> 172.21.15.44:0/2214630701 conn(0x7f87300781b0 msgr2=0x7f87300785b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:36.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.630+0000 7f8737bff700 1 -- 172.21.15.44:0/2214630701 shutdown_connections 2023-11-16T16:57:36.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.630+0000 7f8737bff700 1 -- 172.21.15.44:0/2214630701 wait complete. 2023-11-16T16:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.631+0000 7f8737bff700 1 Processor -- start 2023-11-16T16:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.631+0000 7f8737bff700 1 -- start start 2023-11-16T16:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f8737bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8730100560 0x7f8730196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f8737bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8730100f00 0x7f8730196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f8737bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87301055a0 0x7f873019ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f8737bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873019b420 con 0x7f87301055a0 2023-11-16T16:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f8737bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873019b590 con 0x7f8730100560 2023-11-16T16:57:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f8737bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873019b750 con 0x7f8730100f00 2023-11-16T16:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f873599b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8730100560 0x7f8730196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f873619c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87301055a0 0x7f873019ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f873599b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8730100560 0x7f8730196440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:46902/0 (socket says 172.21.15.44:46902) 2023-11-16T16:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f873599b700 1 -- 172.21.15.44:0/2277317836 learned_addr learned my addr 172.21.15.44:0/2277317836 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f873519a700 1 --2- 172.21.15.44:0/2277317836 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8730100f00 0x7f8730196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f873599b700 1 -- 172.21.15.44:0/2277317836 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8730100f00 msgr2=0x7f8730196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f873599b700 1 --2- 172.21.15.44:0/2277317836 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8730100f00 0x7f8730196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f873599b700 1 -- 172.21.15.44:0/2277317836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87301055a0 msgr2=0x7f873019ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f873599b700 1 --2- 172.21.15.44:0/2277317836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87301055a0 0x7f873019ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f873599b700 1 -- 172.21.15.44:0/2277317836 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87200095f0 con 0x7f8730100560 2023-11-16T16:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.632+0000 7f873599b700 1 --2- 172.21.15.44:0/2277317836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8730100560 0x7f8730196440 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f872c00e9a0 tx=0x7f872c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.633+0000 7f8726ffd700 1 -- 172.21.15.44:0/2277317836 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c009790 con 0x7f8730100560 2023-11-16T16:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.633+0000 7f8726ffd700 1 -- 172.21.15.44:0/2277317836 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f872c010430 con 0x7f8730100560 2023-11-16T16:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.633+0000 7f8737bff700 1 -- 172.21.15.44:0/2277317836 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8730110370 con 0x7f8730100560 2023-11-16T16:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.633+0000 7f8726ffd700 1 -- 172.21.15.44:0/2277317836 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c018680 con 0x7f8730100560 2023-11-16T16:57:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.633+0000 7f8737bff700 1 -- 172.21.15.44:0/2277317836 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87301101e0 con 0x7f8730100560 2023-11-16T16:57:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.638+0000 7f8726ffd700 1 -- 172.21.15.44:0/2277317836 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f872c0105a0 con 0x7f8730100560 2023-11-16T16:57:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.639+0000 7f874b5bdb80 1 Processor -- start 2023-11-16T16:57:36.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.639+0000 7f874b5bdb80 1 -- start start 2023-11-16T16:57:36.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.639+0000 7f874b5bdb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa10b6b0 0x5600aa2b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.639+0000 7f874b5bdb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa2b9b20 0x5600aa2b3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.639+0000 7f874b5bdb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9f90 0x5600aa2b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.639+0000 7f874b5bdb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600aa1b5000 con 0x5600aa2b9f90 2023-11-16T16:57:36.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.639+0000 7f874b5bdb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600aa207840 con 0x5600aa10b6b0 2023-11-16T16:57:36.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.639+0000 7f874b5bdb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600aa200cd0 con 0x5600aa2b9b20 2023-11-16T16:57:36.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.639+0000 7f8726ffd700 1 --2- 172.21.15.44:0/2277317836 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f871c0a1340 0x7f871c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.640+0000 7f873619c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa10b6b0 0x5600aa2b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.640+0000 7f873619c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa10b6b0 0x5600aa2b95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:46910/0 (socket says 172.21.15.44:46910) 2023-11-16T16:57:36.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.640+0000 7f873619c700 1 -- 172.21.15.44:0/2528816921 learned_addr learned my addr 172.21.15.44:0/2528816921 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:36.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.640+0000 7f8726ffd700 1 -- 172.21.15.44:0/2277317836 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f872c014070 con 0x7f8730100560 2023-11-16T16:57:36.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.640+0000 7f873599b700 1 --2- 172.21.15.44:0/2528816921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa2b9b20 0x5600aa2b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.640+0000 7f873519a700 1 --2- 172.21.15.44:0/2528816921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9f90 0x5600aa2b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.640+0000 7f873619c700 1 --2- 172.21.15.44:0/2277317836 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f871c0a1340 0x7f871c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.640+0000 7f873599b700 1 -- 172.21.15.44:0/2528816921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa10b6b0 msgr2=0x5600aa2b95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:36.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.640+0000 7f873599b700 1 --2- 172.21.15.44:0/2528816921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa10b6b0 0x5600aa2b95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.640+0000 7f873599b700 1 -- 172.21.15.44:0/2528816921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9f90 msgr2=0x5600aa2b3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:36.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.640+0000 7f873599b700 1 --2- 172.21.15.44:0/2528816921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9f90 0x5600aa2b3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.640+0000 7f873599b700 1 -- 172.21.15.44:0/2528816921 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600aa0825d0 con 0x5600aa2b9b20 2023-11-16T16:57:36.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.641+0000 7f873599b700 1 --2- 172.21.15.44:0/2528816921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa2b9b20 0x5600aa2b3970 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f872c0ca5e0 tx=0x7f872c01c3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edc3d2c86f7b4d7b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:36.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.641+0000 7f873619c700 1 --2- 172.21.15.44:0/2277317836 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f871c0a1340 0x7f871c0a37f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8728009790 tx=0x7f8728004b40 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:36.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.641+0000 7f871bfff700 1 -- 172.21.15.44:0/2528816921 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c099f60 con 0x5600aa2b9b20 2023-11-16T16:57:36.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.641+0000 7f871bfff700 1 -- 172.21.15.44:0/2528816921 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f872c09df90 con 0x5600aa2b9b20 2023-11-16T16:57:36.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.641+0000 7f871bfff700 1 -- 172.21.15.44:0/2528816921 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c09a420 con 0x5600aa2b9b20 2023-11-16T16:57:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.642+0000 7f874b5bdb80 1 -- 172.21.15.44:0/2528816921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa2b9b20 msgr2=0x5600aa2b3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.642+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/2528816921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa2b9b20 0x5600aa2b3970 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f872c0ca5e0 tx=0x7f872c01c3b0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.642+0000 7f874b5bdb80 1 -- 172.21.15.44:0/2528816921 shutdown_connections 2023-11-16T16:57:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.642+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/2528816921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9f90 0x5600aa2b3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.642+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/2528816921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa10b6b0 0x5600aa2b95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.642+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/2528816921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa2b9b20 0x5600aa2b3970 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.642+0000 7f874b5bdb80 1 -- 172.21.15.44:0/2528816921 >> 172.21.15.44:0/2528816921 conn(0x5600aa2a4f50 msgr2=0x5600aa2aa270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.642+0000 7f874b5bdb80 1 -- 172.21.15.44:0/2528816921 shutdown_connections 2023-11-16T16:57:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.642+0000 7f874b5bdb80 1 -- 172.21.15.44:0/2528816921 wait complete. 2023-11-16T16:57:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.643+0000 7f874b5bdb80 1 -- start start 2023-11-16T16:57:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.643+0000 7f874b5bdb80 1 Processor -- start 2023-11-16T16:57:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.643+0000 7f874b5bdb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa10b6b0 0x5600aa2bfef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.643+0000 7f874b5bdb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa2b9b20 0x5600aa2bd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.643+0000 7f874b5bdb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9f90 0x5600aa2bd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.643+0000 7f874b5bdb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600aa1a2880 con 0x5600aa10b6b0 2023-11-16T16:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.643+0000 7f874b5bdb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600aa1a2520 con 0x5600aa2b9f90 2023-11-16T16:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.643+0000 7f874b5bdb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600aa0848c0 con 0x5600aa2b9b20 2023-11-16T16:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.643+0000 7f874b5bdb80 10 client.0 _ll_register_callbacks cb 0x5600aa1c0720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.644+0000 7f873599b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa10b6b0 0x5600aa2bfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.644+0000 7f873519a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa2b9b20 0x5600aa2bd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.644+0000 7f873519a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa2b9b20 0x5600aa2bd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:55538/0 (socket says 172.21.15.44:55538) 2023-11-16T16:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.644+0000 7f873519a700 1 -- 172.21.15.44:0/1687884310 learned_addr learned my addr 172.21.15.44:0/1687884310 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.644+0000 7f873619c700 1 --2- 172.21.15.44:0/1687884310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9f90 0x5600aa2bd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.644+0000 7f873519a700 1 -- 172.21.15.44:0/1687884310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9f90 msgr2=0x5600aa2bd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.644+0000 7f873519a700 1 --2- 172.21.15.44:0/1687884310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9f90 0x5600aa2bd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.644+0000 7f873519a700 1 -- 172.21.15.44:0/1687884310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa10b6b0 msgr2=0x5600aa2bfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.644+0000 7f873519a700 1 --2- 172.21.15.44:0/1687884310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa10b6b0 0x5600aa2bfef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.644+0000 7f873519a700 1 -- 172.21.15.44:0/1687884310 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87301101e0 con 0x5600aa2b9b20 2023-11-16T16:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.645+0000 7f873519a700 1 --2- 172.21.15.44:0/1687884310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa2b9b20 0x5600aa2bd100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8720009990 tx=0x7f872000b650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=129205e3973972f6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.645+0000 7f87197fa700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.645+0000 7f87197fa700 1 -- 172.21.15.44:0/1687884310 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872001d030 con 0x5600aa2b9b20 2023-11-16T16:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.645+0000 7f874b5bdb80 1 -- 172.21.15.44:0/1687884310 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5600aa2a2710 con 0x5600aa2b9b20 2023-11-16T16:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.645+0000 7f87197fa700 1 -- 172.21.15.44:0/1687884310 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f8720004500 con 0x5600aa2b9b20 2023-11-16T16:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.645+0000 7f87197fa700 1 -- 172.21.15.44:0/1687884310 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87200046c0 con 0x5600aa2b9b20 2023-11-16T16:57:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.645+0000 7f874b5bdb80 10 client.26824 fetch_fsmap learned FSMap version 21 2023-11-16T16:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.645+0000 7f874b5bdb80 1 -- 172.21.15.44:0/1687884310 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5600aa1c7130 con 0x5600aa2b9b20 2023-11-16T16:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.646+0000 7f87197fa700 1 -- 172.21.15.44:0/1687884310 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5600aa2a2710 con 0x5600aa2b9b20 2023-11-16T16:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.646+0000 7f87197fa700 1 -- 172.21.15.44:0/1687884310 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f872000f940 con 0x5600aa2b9b20 2023-11-16T16:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.646+0000 7f874b5bdb80 10 client.26824 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.646+0000 7f874b5bdb80 10 client.26824 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.646+0000 7f874b5bdb80 20 client.26824 populate_metadata read hostname 'smithi044' 2023-11-16T16:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.646+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/1687884310 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5600aa2cd4b0 0x5600aa2cf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.646+0000 7f874b5bdb80 4 client.26824 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:57:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.646+0000 7f874b5bdb80 1 -- 172.21.15.44:0/1687884310 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600aa1c3040 con 0x5600aa2cd4b0 2023-11-16T16:57:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.646+0000 7f873599b700 1 --2- 172.21.15.44:0/1687884310 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5600aa2cd4b0 0x5600aa2cf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.647+0000 7f873599b700 1 --2- 172.21.15.44:0/1687884310 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5600aa2cd4b0 0x5600aa2cf8c0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f63ae9f352c8ade server_cookie=c7e05d2d168c5385 in_seq=0 out_seq=0 2023-11-16T16:57:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.647+0000 7f87197fa700 10 client.26824 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.650+0000 7f87197fa700 1 -- 172.21.15.44:0/1687884310 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5600aa1c3040 con 0x5600aa2cd4b0 2023-11-16T16:57:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.650+0000 7f87197fa700 10 client.26824 handle_command_reply: tid=0 2023-11-16T16:57:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.650+0000 7f874b5bdb80 1 client.26824 shutdown 2023-11-16T16:57:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.650+0000 7f874b5bdb80 1 -- 172.21.15.44:0/1687884310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa2b9b20 msgr2=0x5600aa2bd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.650+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/1687884310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa2b9b20 0x5600aa2bd100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8720009990 tx=0x7f872000b650 comp rx=0 tx=0).stop 2023-11-16T16:57:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.650+0000 7f874b5bdb80 1 -- 172.21.15.44:0/1687884310 shutdown_connections 2023-11-16T16:57:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.650+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/1687884310 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5600aa2cd4b0 0x5600aa2cf8c0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.650+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/1687884310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9f90 0x5600aa2bd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.650+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/1687884310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa10b6b0 0x5600aa2bfef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.650+0000 7f87197fa700 0 client.26824 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.650+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/1687884310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa2b9b20 0x5600aa2bd100 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.650+0000 7f874b5bdb80 1 -- 172.21.15.44:0/1687884310 >> 172.21.15.44:0/1687884310 conn(0x5600aa2a4f50 msgr2=0x5600aa2ae550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.650+0000 7f874b5bdb80 1 -- 172.21.15.44:0/1687884310 shutdown_connections 2023-11-16T16:57:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.650+0000 7f874b5bdb80 1 -- 172.21.15.44:0/1687884310 wait complete. 2023-11-16T16:57:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.651+0000 7f874b5bdb80 20 client.26824 trim_cache size 0 max 16384 2023-11-16T16:57:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.668+0000 7f874b5bdb80 1 Processor -- start 2023-11-16T16:57:36.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.668+0000 7f874b5bdb80 1 -- start start 2023-11-16T16:57:36.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f874b5bdb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa10b6b0 0x5600aa2bd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f874b5bdb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9b20 0x5600aa2bc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f874b5bdb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9f90 0x5600aa2c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f874b5bdb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600aa2bcee0 con 0x5600aa2b9f90 2023-11-16T16:57:36.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f874b5bdb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600aa2ab2b0 con 0x5600aa2b9b20 2023-11-16T16:57:36.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f874b5bdb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600aa2b7df0 con 0x5600aa10b6b0 2023-11-16T16:57:36.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f873519a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9b20 0x5600aa2bc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f873519a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9b20 0x5600aa2bc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:46926/0 (socket says 172.21.15.44:46926) 2023-11-16T16:57:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f873519a700 1 -- 172.21.15.44:0/2854946637 learned_addr learned my addr 172.21.15.44:0/2854946637 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f873619c700 1 --2- 172.21.15.44:0/2854946637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9f90 0x5600aa2c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f873599b700 1 --2- 172.21.15.44:0/2854946637 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa10b6b0 0x5600aa2bd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f873599b700 1 -- 172.21.15.44:0/2854946637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9b20 msgr2=0x5600aa2bc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f873599b700 1 --2- 172.21.15.44:0/2854946637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9b20 0x5600aa2bc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f873599b700 1 -- 172.21.15.44:0/2854946637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9f90 msgr2=0x5600aa2c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f873599b700 1 --2- 172.21.15.44:0/2854946637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9f90 0x5600aa2c2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f873599b700 1 -- 172.21.15.44:0/2854946637 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600aa1c6e80 con 0x5600aa10b6b0 2023-11-16T16:57:36.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.669+0000 7f873519a700 1 --2- 172.21.15.44:0/2854946637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9b20 0x5600aa2bc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:57:36.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.670+0000 7f873599b700 1 --2- 172.21.15.44:0/2854946637 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa10b6b0 0x5600aa2bd0c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x5600aa2b4670 tx=0x7f872c01f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ab504dcb9d19077 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:36.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.670+0000 7f8724bff700 1 -- 172.21.15.44:0/2854946637 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c099f60 con 0x5600aa10b6b0 2023-11-16T16:57:36.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.670+0000 7f8724bff700 1 -- 172.21.15.44:0/2854946637 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f872c010880 con 0x5600aa10b6b0 2023-11-16T16:57:36.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.670+0000 7f8724bff700 1 -- 172.21.15.44:0/2854946637 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c09a420 con 0x5600aa10b6b0 2023-11-16T16:57:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.671+0000 7f874b5bdb80 1 -- 172.21.15.44:0/2854946637 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa10b6b0 msgr2=0x5600aa2bd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.671+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/2854946637 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa10b6b0 0x5600aa2bd0c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x5600aa2b4670 tx=0x7f872c01f000 comp rx=0 tx=0).stop 2023-11-16T16:57:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.671+0000 7f874b5bdb80 1 -- 172.21.15.44:0/2854946637 shutdown_connections 2023-11-16T16:57:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.671+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/2854946637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9f90 0x5600aa2c2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.671+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/2854946637 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa10b6b0 0x5600aa2bd0c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.671+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/2854946637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9b20 0x5600aa2bc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.671+0000 7f874b5bdb80 1 -- 172.21.15.44:0/2854946637 >> 172.21.15.44:0/2854946637 conn(0x5600aa2a4f50 msgr2=0x5600aa2d41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.671+0000 7f874b5bdb80 1 -- 172.21.15.44:0/2854946637 shutdown_connections 2023-11-16T16:57:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.671+0000 7f874b5bdb80 1 -- 172.21.15.44:0/2854946637 wait complete. 2023-11-16T16:57:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.671+0000 7f874b5bdb80 1 -- start start 2023-11-16T16:57:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.672+0000 7f874b5bdb80 1 Processor -- start 2023-11-16T16:57:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.672+0000 7f874b5bdb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa10b6b0 0x5600aa2c22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.672+0000 7f874b5bdb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9b20 0x5600aa2c2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.672+0000 7f874b5bdb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9f90 0x5600aa318a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.672+0000 7f874b5bdb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600aa318fb0 con 0x5600aa2b9b20 2023-11-16T16:57:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.672+0000 7f874b5bdb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600aa319120 con 0x5600aa2b9f90 2023-11-16T16:57:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.672+0000 7f874b5bdb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600aa3192e0 con 0x5600aa10b6b0 2023-11-16T16:57:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.672+0000 7f874b5bdb80 10 client.0 _ll_register_callbacks cb 0x5600aa1c0720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:57:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.673+0000 7f873519a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9b20 0x5600aa2c2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.673+0000 7f873599b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa10b6b0 0x5600aa2c22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.673+0000 7f873599b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa10b6b0 0x5600aa2c22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:55562/0 (socket says 172.21.15.44:55562) 2023-11-16T16:57:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.673+0000 7f873519a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9b20 0x5600aa2c2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46676/0 (socket says 172.21.15.44:46676) 2023-11-16T16:57:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.673+0000 7f873599b700 1 -- 172.21.15.44:0/4174618779 learned_addr learned my addr 172.21.15.44:0/4174618779 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.673+0000 7f873619c700 1 --2- 172.21.15.44:0/4174618779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9f90 0x5600aa318a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.673+0000 7f873519a700 1 -- 172.21.15.44:0/4174618779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa10b6b0 msgr2=0x5600aa2c22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.673+0000 7f873519a700 1 --2- 172.21.15.44:0/4174618779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa10b6b0 0x5600aa2c22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.673+0000 7f873519a700 1 -- 172.21.15.44:0/4174618779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9f90 msgr2=0x5600aa318a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.673+0000 7f873519a700 1 --2- 172.21.15.44:0/4174618779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9f90 0x5600aa318a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.673+0000 7f873519a700 1 -- 172.21.15.44:0/4174618779 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600aa1c7130 con 0x5600aa2b9b20 2023-11-16T16:57:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.673+0000 7f873619c700 1 --2- 172.21.15.44:0/4174618779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9f90 0x5600aa318a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:57:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.673+0000 7f873519a700 1 --2- 172.21.15.44:0/4174618779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9b20 0x5600aa2c2800 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8720001eb0 tx=0x7f87200244e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4787ae579b707bc3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.673+0000 7f87197fa700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:57:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.674+0000 7f87197fa700 1 -- 172.21.15.44:0/4174618779 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872001d040 con 0x5600aa2b9b20 2023-11-16T16:57:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.674+0000 7f87197fa700 1 -- 172.21.15.44:0/4174618779 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f8720000e40 con 0x5600aa2b9b20 2023-11-16T16:57:36.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.674+0000 7f874b5bdb80 1 -- 172.21.15.44:0/4174618779 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f872c09df90 con 0x5600aa2b9b20 2023-11-16T16:57:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.674+0000 7f87197fa700 1 -- 172.21.15.44:0/4174618779 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f872000f460 con 0x5600aa2b9b20 2023-11-16T16:57:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.674+0000 7f874b5bdb80 10 client.16800 fetch_fsmap learned FSMap version 21 2023-11-16T16:57:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.674+0000 7f874b5bdb80 1 -- 172.21.15.44:0/4174618779 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5600aa0825d0 con 0x5600aa2b9b20 2023-11-16T16:57:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.674+0000 7f87197fa700 1 -- 172.21.15.44:0/4174618779 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f872c09df90 con 0x5600aa2b9b20 2023-11-16T16:57:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.674+0000 7f87197fa700 1 -- 172.21.15.44:0/4174618779 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8720017b90 con 0x5600aa2b9b20 2023-11-16T16:57:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.674+0000 7f874b5bdb80 10 client.16800 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:57:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.675+0000 7f874b5bdb80 10 client.16800 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:57:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.675+0000 7f874b5bdb80 20 client.16800 populate_metadata read hostname 'smithi044' 2023-11-16T16:57:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.675+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/4174618779 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5600aa321860 0x5600aa323c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.675+0000 7f874b5bdb80 4 client.16800 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:57:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.675+0000 7f874b5bdb80 1 -- 172.21.15.44:0/4174618779 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5600aa1cf600 con 0x5600aa321860 2023-11-16T16:57:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.675+0000 7f873599b700 1 --2- 172.21.15.44:0/4174618779 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5600aa321860 0x5600aa323c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.675+0000 7f873599b700 1 --2- 172.21.15.44:0/4174618779 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5600aa321860 0x5600aa323c40 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=723e0116c080289 server_cookie=a191adf140214d95 in_seq=0 out_seq=0 2023-11-16T16:57:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:36.675+0000 7f87197fa700 10 client.16800 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:37.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:37 smithi044 ceph-mon[128750]: pgmap v237: 129 pgs: 129 active+clean; 55 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 650 op/s 2023-11-16T16:57:37.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:37 smithi176 ceph-mon[131630]: pgmap v237: 129 pgs: 129 active+clean; 55 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 650 op/s 2023-11-16T16:57:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:37 smithi077 ceph-mon[131204]: pgmap v237: 129 pgs: 129 active+clean; 55 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 650 op/s 2023-11-16T16:57:39.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:39 smithi044 ceph-mon[128750]: pgmap v238: 129 pgs: 129 active+clean; 68 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 1.03k op/s 2023-11-16T16:57:39.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:39 smithi176 ceph-mon[131630]: pgmap v238: 129 pgs: 129 active+clean; 68 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 1.03k op/s 2023-11-16T16:57:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:39 smithi077 ceph-mon[131204]: pgmap v238: 129 pgs: 129 active+clean; 68 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 1.03k op/s 2023-11-16T16:57:40.779 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.777+0000 7f87197fa700 1 -- 172.21.15.44:0/4174618779 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5600aa1cf600 con 0x5600aa321860 2023-11-16T16:57:40.779 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.777+0000 7f87197fa700 10 client.16800 handle_command_reply: tid=0 2023-11-16T16:57:40.779 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.777+0000 7f874b5bdb80 1 client.16800 shutdown 2023-11-16T16:57:40.780 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.778+0000 7f874b5bdb80 1 -- 172.21.15.44:0/4174618779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9b20 msgr2=0x5600aa2c2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:40.780 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.778+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/4174618779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9b20 0x5600aa2c2800 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8720001eb0 tx=0x7f87200244e0 comp rx=0 tx=0).stop 2023-11-16T16:57:40.780 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.778+0000 7f874b5bdb80 1 -- 172.21.15.44:0/4174618779 shutdown_connections 2023-11-16T16:57:40.780 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.778+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/4174618779 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5600aa321860 0x5600aa323c40 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:40.780 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.778+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/4174618779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600aa2b9f90 0x5600aa318a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:40.781 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.778+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/4174618779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5600aa10b6b0 0x5600aa2c22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:40.781 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.778+0000 7f874b5bdb80 1 --2- 172.21.15.44:0/4174618779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5600aa2b9b20 0x5600aa2c2800 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:40.781 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.778+0000 7f874b5bdb80 1 -- 172.21.15.44:0/4174618779 >> 172.21.15.44:0/4174618779 conn(0x5600aa2a4f50 msgr2=0x5600aa2b3970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:40.781 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.778+0000 7f87197fa700 0 client.16800 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:40.781 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.779+0000 7f874b5bdb80 1 -- 172.21.15.44:0/4174618779 shutdown_connections 2023-11-16T16:57:40.781 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.779+0000 7f874b5bdb80 1 -- 172.21.15.44:0/4174618779 wait complete. 2023-11-16T16:57:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.780+0000 7f874b5bdb80 20 client.16800 trim_cache size 0 max 16384 2023-11-16T16:57:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:57:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:57:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:57:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:57:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.784+0000 7f8737bff700 1 -- 172.21.15.44:0/2277317836 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f871c0a1340 msgr2=0x7f871c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.784+0000 7f8737bff700 1 --2- 172.21.15.44:0/2277317836 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f871c0a1340 0x7f871c0a37f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8728009790 tx=0x7f8728004b40 comp rx=0 tx=0).stop 2023-11-16T16:57:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.784+0000 7f8737bff700 1 -- 172.21.15.44:0/2277317836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8730100560 msgr2=0x7f8730196440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.784+0000 7f8737bff700 1 --2- 172.21.15.44:0/2277317836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8730100560 0x7f8730196440 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f872c00e9a0 tx=0x7f872c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:57:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.784+0000 7f8737bff700 1 -- 172.21.15.44:0/2277317836 shutdown_connections 2023-11-16T16:57:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.784+0000 7f8737bff700 1 --2- 172.21.15.44:0/2277317836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87301055a0 0x7f873019ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.785+0000 7f8737bff700 1 --2- 172.21.15.44:0/2277317836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8730100560 0x7f8730196440 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.785+0000 7f8737bff700 1 --2- 172.21.15.44:0/2277317836 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f871c0a1340 0x7f871c0a37f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.785+0000 7f8737bff700 1 --2- 172.21.15.44:0/2277317836 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8730100f00 0x7f8730196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.785+0000 7f8737bff700 1 -- 172.21.15.44:0/2277317836 >> 172.21.15.44:0/2277317836 conn(0x7f87300781b0 msgr2=0x7f87300fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.785+0000 7f8737bff700 1 -- 172.21.15.44:0/2277317836 shutdown_connections 2023-11-16T16:57:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:40.785+0000 7f8737bff700 1 -- 172.21.15.44:0/2277317836 wait complete. 2023-11-16T16:57:41.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:41 smithi044 ceph-mon[128750]: pgmap v239: 129 pgs: 129 active+clean; 68 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 696 op/s 2023-11-16T16:57:41.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:41 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:57:41.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:41 smithi176 ceph-mon[131630]: pgmap v239: 129 pgs: 129 active+clean; 68 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 696 op/s 2023-11-16T16:57:41.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:41 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:57:41.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:41 smithi077 ceph-mon[131204]: pgmap v239: 129 pgs: 129 active+clean; 68 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 696 op/s 2023-11-16T16:57:41.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:41 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:57:43.514 INFO:tasks.workunit.client.0.smithi044.stdout:cleaned up hierarchy 2023-11-16T16:57:43.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:43 smithi044 ceph-mon[128750]: pgmap v240: 129 pgs: 129 active+clean; 44 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 808 op/s 2023-11-16T16:57:43.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:43 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:57:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:43 smithi176 ceph-mon[131630]: pgmap v240: 129 pgs: 129 active+clean; 44 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 808 op/s 2023-11-16T16:57:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:43 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:57:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:43 smithi077 ceph-mon[131204]: pgmap v240: 129 pgs: 129 active+clean; 44 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 808 op/s 2023-11-16T16:57:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:43 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:57:45.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:45 smithi044 ceph-mon[128750]: pgmap v241: 129 pgs: 129 active+clean; 11 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 1.39k op/s 2023-11-16T16:57:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:45 smithi176 ceph-mon[131630]: pgmap v241: 129 pgs: 129 active+clean; 11 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 1.39k op/s 2023-11-16T16:57:45.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:45 smithi077 ceph-mon[131204]: pgmap v241: 129 pgs: 129 active+clean; 11 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 1.39k op/s 2023-11-16T16:57:46.998 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T16:57:46.998 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T16:57:47.075 INFO:tasks.workunit:Running workunit fs/misc/filelock_deadlock.py... 2023-11-16T16:57:47.075 DEBUG:teuthology.orchestra.run.smithi044:workunit test fs/misc/filelock_deadlock.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/filelock_deadlock.py 2023-11-16T16:57:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:47 smithi044 ceph-mon[128750]: pgmap v242: 129 pgs: 129 active+clean; 11 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 1.17k op/s 2023-11-16T16:57:47.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:47 smithi176 ceph-mon[131630]: pgmap v242: 129 pgs: 129 active+clean; 11 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 1.17k op/s 2023-11-16T16:57:47.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:47 smithi077 ceph-mon[131204]: pgmap v242: 129 pgs: 129 active+clean; 11 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 1.17k op/s 2023-11-16T16:57:49.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:49 smithi044 ceph-mon[128750]: pgmap v243: 129 pgs: 129 active+clean; 17 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 1.33k op/s 2023-11-16T16:57:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:49 smithi176 ceph-mon[131630]: pgmap v243: 129 pgs: 129 active+clean; 17 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 1.33k op/s 2023-11-16T16:57:49.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:49 smithi077 ceph-mon[131204]: pgmap v243: 129 pgs: 129 active+clean; 17 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 1.33k op/s 2023-11-16T16:57:51.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:51 smithi044 ceph-mon[128750]: pgmap v244: 129 pgs: 129 active+clean; 17 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 962 op/s 2023-11-16T16:57:51.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:51 smithi176 ceph-mon[131630]: pgmap v244: 129 pgs: 129 active+clean; 17 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 962 op/s 2023-11-16T16:57:51.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:51 smithi077 ceph-mon[131204]: pgmap v244: 129 pgs: 129 active+clean; 17 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 962 op/s 2023-11-16T16:57:53.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:53 smithi044 ceph-mon[128750]: pgmap v245: 129 pgs: 129 active+clean; 17 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 932 op/s 2023-11-16T16:57:53.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:53 smithi176 ceph-mon[131630]: pgmap v245: 129 pgs: 129 active+clean; 17 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 932 op/s 2023-11-16T16:57:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.963+0000 7f4e239ff700 1 -- 172.21.15.44:0/314512889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c1015f0 msgr2=0x7f4e1c104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.963+0000 7f4e239ff700 1 --2- 172.21.15.44:0/314512889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c1015f0 0x7f4e1c104e80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c0099c0 tx=0x7f4e0c009cd0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.963+0000 7f4e239ff700 1 -- 172.21.15.44:0/314512889 shutdown_connections 2023-11-16T16:57:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.963+0000 7f4e239ff700 1 --2- 172.21.15.44:0/314512889 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e1c105560 0x7f4e1c10d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.963+0000 7f4e239ff700 1 --2- 172.21.15.44:0/314512889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e1c100c50 0x7f4e1c101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.963+0000 7f4e239ff700 1 --2- 172.21.15.44:0/314512889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c1015f0 0x7f4e1c104e80 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.963+0000 7f4e239ff700 1 -- 172.21.15.44:0/314512889 >> 172.21.15.44:0/314512889 conn(0x7f4e1c077550 msgr2=0x7f4e1c077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.963+0000 7f4e239ff700 1 -- 172.21.15.44:0/314512889 shutdown_connections 2023-11-16T16:57:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.963+0000 7f4e239ff700 1 -- 172.21.15.44:0/314512889 wait complete. 2023-11-16T16:57:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.964+0000 7f4e239ff700 1 Processor -- start 2023-11-16T16:57:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.965+0000 7f4e239ff700 1 -- start start 2023-11-16T16:57:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.965+0000 7f4e239ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e1c100c50 0x7f4e1c198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.965+0000 7f4e239ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e1c1015f0 0x7f4e1c198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.965+0000 7f4e239ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c105560 0x7f4e1c19cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.965+0000 7f4e239ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c19d570 con 0x7f4e1c105560 2023-11-16T16:57:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.965+0000 7f4e239ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c19d6e0 con 0x7f4e1c100c50 2023-11-16T16:57:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.965+0000 7f4e239ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c19d8a0 con 0x7f4e1c1015f0 2023-11-16T16:57:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.965+0000 7f4e21f9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c105560 0x7f4e1c19cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.965+0000 7f4e2179b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e1c100c50 0x7f4e1c198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.965+0000 7f4e20f9a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e1c1015f0 0x7f4e1c198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.965+0000 7f4e21f9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c105560 0x7f4e1c19cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60566/0 (socket says 172.21.15.44:60566) 2023-11-16T16:57:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.965+0000 7f4e21f9c700 1 -- 172.21.15.44:0/105890510 learned_addr learned my addr 172.21.15.44:0/105890510 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.966+0000 7f4e2179b700 1 -- 172.21.15.44:0/105890510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e1c1015f0 msgr2=0x7f4e1c198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.966+0000 7f4e2179b700 1 --2- 172.21.15.44:0/105890510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e1c1015f0 0x7f4e1c198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.966+0000 7f4e2179b700 1 -- 172.21.15.44:0/105890510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c105560 msgr2=0x7f4e1c19cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.966+0000 7f4e2179b700 1 --2- 172.21.15.44:0/105890510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c105560 0x7f4e1c19cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.966+0000 7f4e2179b700 1 -- 172.21.15.44:0/105890510 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c0095f0 con 0x7f4e1c100c50 2023-11-16T16:57:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.966+0000 7f4e21f9c700 1 --2- 172.21.15.44:0/105890510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c105560 0x7f4e1c19cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:57:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.966+0000 7f4e2179b700 1 --2- 172.21.15.44:0/105890510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e1c100c50 0x7f4e1c198600 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800d750 tx=0x7f4e1800da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.966+0000 7f4e127fc700 1 -- 172.21.15.44:0/105890510 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18004020 con 0x7f4e1c100c50 2023-11-16T16:57:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.966+0000 7f4e127fc700 1 -- 172.21.15.44:0/105890510 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e18004830 con 0x7f4e1c100c50 2023-11-16T16:57:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.966+0000 7f4e239ff700 1 -- 172.21.15.44:0/105890510 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e1c110370 con 0x7f4e1c100c50 2023-11-16T16:57:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.966+0000 7f4e127fc700 1 -- 172.21.15.44:0/105890510 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e180054c0 con 0x7f4e1c100c50 2023-11-16T16:57:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.967+0000 7f4e239ff700 1 -- 172.21.15.44:0/105890510 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e1c1101e0 con 0x7f4e1c100c50 2023-11-16T16:57:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.970+0000 7f4e127fc700 1 -- 172.21.15.44:0/105890510 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f4e180049a0 con 0x7f4e1c100c50 2023-11-16T16:57:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.971+0000 7f4e373b2b80 1 Processor -- start 2023-11-16T16:57:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.971+0000 7f4e373b2b80 1 -- start start 2023-11-16T16:57:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.971+0000 7f4e127fc700 1 --2- 172.21.15.44:0/105890510 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4e080a12c0 0x7f4e080a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.971+0000 7f4e127fc700 1 -- 172.21.15.44:0/105890510 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f4e180cca30 con 0x7f4e1c100c50 2023-11-16T16:57:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.971+0000 7f4e373b2b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5652307a56b0 0x56523094f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.971+0000 7f4e373b2b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094f720 0x565230949560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.971+0000 7f4e373b2b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56523094fb90 0x565230949b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.971+0000 7f4e21f9c700 1 --2- 172.21.15.44:0/105890510 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4e080a12c0 0x7f4e080a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.971+0000 7f4e373b2b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56523084f000 con 0x5652307a56b0 2023-11-16T16:57:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.971+0000 7f4e373b2b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652308a1840 con 0x56523094f720 2023-11-16T16:57:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.971+0000 7f4e373b2b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56523089acd0 con 0x56523094fb90 2023-11-16T16:57:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.972+0000 7f4e2179b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5652307a56b0 0x56523094f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.972+0000 7f4e20f9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094f720 0x565230949560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.972+0000 7f4e2179b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5652307a56b0 0x56523094f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60574/0 (socket says 172.21.15.44:60574) 2023-11-16T16:57:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.972+0000 7f4e2179b700 1 -- 172.21.15.44:0/3064202546 learned_addr learned my addr 172.21.15.44:0/3064202546 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.972+0000 7f4e20f9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094f720 0x565230949560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:58156/0 (socket says 172.21.15.44:58156) 2023-11-16T16:57:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.972+0000 7f4e21f9c700 1 --2- 172.21.15.44:0/3064202546 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56523094fb90 0x565230949b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.972+0000 7f4e2179b700 1 -- 172.21.15.44:0/3064202546 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56523094fb90 msgr2=0x565230949b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.972+0000 7f4e2179b700 1 --2- 172.21.15.44:0/3064202546 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56523094fb90 0x565230949b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.972+0000 7f4e2179b700 1 -- 172.21.15.44:0/3064202546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094f720 msgr2=0x565230949560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.972+0000 7f4e2179b700 1 --2- 172.21.15.44:0/3064202546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094f720 0x565230949560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.972+0000 7f4e2179b700 1 -- 172.21.15.44:0/3064202546 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56523071c5d0 con 0x5652307a56b0 2023-11-16T16:57:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.972+0000 7f4e20f9a700 1 --2- 172.21.15.44:0/3064202546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094f720 0x565230949560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:57:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.972+0000 7f4e2179b700 1 --2- 172.21.15.44:0/3064202546 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5652307a56b0 0x56523094f1e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4e180a15e0 tx=0x7f4e180996c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c19a5117ee37319 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.972+0000 7f4e21f9c700 1 --2- 172.21.15.44:0/105890510 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4e080a12c0 0x7f4e080a3770 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009550 tx=0x7f4e14007ed0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.973+0000 7f4e077fe700 1 -- 172.21.15.44:0/3064202546 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e180a1b10 con 0x5652307a56b0 2023-11-16T16:57:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.973+0000 7f4e077fe700 1 -- 172.21.15.44:0/3064202546 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e180b90d0 con 0x5652307a56b0 2023-11-16T16:57:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.973+0000 7f4e077fe700 1 -- 172.21.15.44:0/3064202546 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18099b40 con 0x5652307a56b0 2023-11-16T16:57:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.973+0000 7f4e373b2b80 1 -- 172.21.15.44:0/3064202546 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5652307a56b0 msgr2=0x56523094f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.973+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/3064202546 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5652307a56b0 0x56523094f1e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4e180a15e0 tx=0x7f4e180996c0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.973+0000 7f4e373b2b80 1 -- 172.21.15.44:0/3064202546 shutdown_connections 2023-11-16T16:57:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.973+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/3064202546 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56523094fb90 0x565230949b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.973+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/3064202546 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5652307a56b0 0x56523094f1e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.973+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/3064202546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094f720 0x565230949560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.973+0000 7f4e373b2b80 1 -- 172.21.15.44:0/3064202546 >> 172.21.15.44:0/3064202546 conn(0x56523093ef50 msgr2=0x565230944270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.973+0000 7f4e373b2b80 1 -- 172.21.15.44:0/3064202546 shutdown_connections 2023-11-16T16:57:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.974+0000 7f4e373b2b80 1 -- 172.21.15.44:0/3064202546 wait complete. 2023-11-16T16:57:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.974+0000 7f4e373b2b80 1 -- start start 2023-11-16T16:57:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.974+0000 7f4e373b2b80 1 Processor -- start 2023-11-16T16:57:53.981 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:53 smithi077 ceph-mon[131204]: pgmap v245: 129 pgs: 129 active+clean; 17 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 932 op/s 2023-11-16T16:57:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e373b2b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5652307a56b0 0x565230955b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e373b2b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094f720 0x565230952d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e373b2b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56523094fb90 0x5652309534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e373b2b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56523083c880 con 0x5652307a56b0 2023-11-16T16:57:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e373b2b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56523083c520 con 0x56523094f720 2023-11-16T16:57:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e373b2b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56523071e8c0 con 0x56523094fb90 2023-11-16T16:57:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e373b2b80 10 client.0 _ll_register_callbacks cb 0x56523085a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:57:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e2179b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5652307a56b0 0x565230955b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e2179b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5652307a56b0 0x565230955b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60590/0 (socket says 172.21.15.44:60590) 2023-11-16T16:57:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e2179b700 1 -- 172.21.15.44:0/1614765688 learned_addr learned my addr 172.21.15.44:0/1614765688 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e20f9a700 1 --2- 172.21.15.44:0/1614765688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094f720 0x565230952d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e21f9c700 1 --2- 172.21.15.44:0/1614765688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56523094fb90 0x5652309534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e2179b700 1 -- 172.21.15.44:0/1614765688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56523094fb90 msgr2=0x5652309534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e2179b700 1 --2- 172.21.15.44:0/1614765688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56523094fb90 0x5652309534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e2179b700 1 -- 172.21.15.44:0/1614765688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094f720 msgr2=0x565230952d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e2179b700 1 --2- 172.21.15.44:0/1614765688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094f720 0x565230952d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.975+0000 7f4e2179b700 1 -- 172.21.15.44:0/1614765688 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e1c1101e0 con 0x5652307a56b0 2023-11-16T16:57:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.976+0000 7f4e2179b700 1 --2- 172.21.15.44:0/1614765688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5652307a56b0 0x565230955b00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800ddb0 tx=0x7f4e18099490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3440410c7bdcec02 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.976+0000 7f4e04ff9700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:57:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.976+0000 7f4e04ff9700 1 -- 172.21.15.44:0/1614765688 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18098bb0 con 0x5652307a56b0 2023-11-16T16:57:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.976+0000 7f4e04ff9700 1 -- 172.21.15.44:0/1614765688 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f4e18098d10 con 0x5652307a56b0 2023-11-16T16:57:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.976+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1614765688 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56523093c710 con 0x5652307a56b0 2023-11-16T16:57:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.976+0000 7f4e04ff9700 1 -- 172.21.15.44:0/1614765688 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e18099920 con 0x5652307a56b0 2023-11-16T16:57:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.976+0000 7f4e373b2b80 10 client.16812 fetch_fsmap learned FSMap version 21 2023-11-16T16:57:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.976+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1614765688 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x565230861130 con 0x5652307a56b0 2023-11-16T16:57:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.976+0000 7f4e04ff9700 1 -- 172.21.15.44:0/1614765688 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e18063020 con 0x5652307a56b0 2023-11-16T16:57:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.977+0000 7f4e04ff9700 1 -- 172.21.15.44:0/1614765688 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4e180840c0 con 0x5652307a56b0 2023-11-16T16:57:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.977+0000 7f4e373b2b80 10 client.16812 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:57:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.977+0000 7f4e373b2b80 10 client.16812 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:57:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.977+0000 7f4e373b2b80 20 client.16812 populate_metadata read hostname 'smithi044' 2023-11-16T16:57:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.977+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/1614765688 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x565230962af0 0x565230964ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.977+0000 7f4e373b2b80 4 client.16812 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:57:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.977+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1614765688 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56523085d040 con 0x565230962af0 2023-11-16T16:57:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.977+0000 7f4e20f9a700 1 --2- 172.21.15.44:0/1614765688 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x565230962af0 0x565230964ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.978+0000 7f4e20f9a700 1 --2- 172.21.15.44:0/1614765688 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x565230962af0 0x565230964ed0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3ec90ec108e748f server_cookie=c9952b939a6e7266 in_seq=0 out_seq=0 2023-11-16T16:57:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.978+0000 7f4e04ff9700 10 client.16812 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.978+0000 7f4e04ff9700 1 -- 172.21.15.44:0/1614765688 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56523085d040 con 0x565230962af0 2023-11-16T16:57:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.978+0000 7f4e04ff9700 10 client.16812 handle_command_reply: tid=0 2023-11-16T16:57:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.978+0000 7f4e373b2b80 1 client.16812 shutdown 2023-11-16T16:57:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.979+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1614765688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5652307a56b0 msgr2=0x565230955b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.979+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/1614765688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5652307a56b0 0x565230955b00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800ddb0 tx=0x7f4e18099490 comp rx=0 tx=0).stop 2023-11-16T16:57:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.979+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1614765688 shutdown_connections 2023-11-16T16:57:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.979+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/1614765688 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x565230962af0 0x565230964ed0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.979+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/1614765688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56523094fb90 0x5652309534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.979+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/1614765688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5652307a56b0 0x565230955b00 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.979+0000 7f4e04ff9700 0 client.16812 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.979+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/1614765688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094f720 0x565230952d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.979+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1614765688 >> 172.21.15.44:0/1614765688 conn(0x56523093ef50 msgr2=0x5652309437e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.979+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1614765688 shutdown_connections 2023-11-16T16:57:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.979+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1614765688 wait complete. 2023-11-16T16:57:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.979+0000 7f4e373b2b80 20 client.16812 trim_cache size 0 max 16384 2023-11-16T16:57:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.994+0000 7f4e373b2b80 1 Processor -- start 2023-11-16T16:57:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.994+0000 7f4e373b2b80 1 -- start start 2023-11-16T16:57:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e373b2b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5652307a56b0 0x56523095a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e373b2b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56523094f720 0x5652309556a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e373b2b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094fb90 0x565230952580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e373b2b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56523093eba0 con 0x56523094f720 2023-11-16T16:57:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e373b2b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x565230954340 con 0x56523094fb90 2023-11-16T16:57:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e373b2b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x565230945340 con 0x5652307a56b0 2023-11-16T16:57:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e20f9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56523094f720 0x5652309556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e2179b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5652307a56b0 0x56523095a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e2179b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5652307a56b0 0x56523095a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:60534/0 (socket says 172.21.15.44:60534) 2023-11-16T16:57:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e2179b700 1 -- 172.21.15.44:0/1468682400 learned_addr learned my addr 172.21.15.44:0/1468682400 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e20f9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56523094f720 0x5652309556a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60604/0 (socket says 172.21.15.44:60604) 2023-11-16T16:57:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e21f9c700 1 --2- 172.21.15.44:0/1468682400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094fb90 0x565230952580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e20f9a700 1 -- 172.21.15.44:0/1468682400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5652307a56b0 msgr2=0x56523095a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e20f9a700 1 --2- 172.21.15.44:0/1468682400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5652307a56b0 0x56523095a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e20f9a700 1 -- 172.21.15.44:0/1468682400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094fb90 msgr2=0x565230952580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e20f9a700 1 --2- 172.21.15.44:0/1468682400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094fb90 0x565230952580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e20f9a700 1 -- 172.21.15.44:0/1468682400 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652309406f0 con 0x56523094f720 2023-11-16T16:57:54.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.995+0000 7f4e2179b700 1 --2- 172.21.15.44:0/1468682400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5652307a56b0 0x56523095a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:57:54.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.996+0000 7f4e20f9a700 1 --2- 172.21.15.44:0/1468682400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56523094f720 0x5652309556a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c0098e0 tx=0x7f4e0c009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8950b35f7d201080 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:54.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.996+0000 7f4e07fff700 1 -- 172.21.15.44:0/1468682400 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c02a070 con 0x56523094f720 2023-11-16T16:57:54.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.996+0000 7f4e07fff700 1 -- 172.21.15.44:0/1468682400 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e0c01e040 con 0x56523094f720 2023-11-16T16:57:54.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.996+0000 7f4e07fff700 1 -- 172.21.15.44:0/1468682400 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c01d040 con 0x56523094f720 2023-11-16T16:57:54.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.996+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1468682400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56523094f720 msgr2=0x5652309556a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:54.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.996+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/1468682400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56523094f720 0x5652309556a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c0098e0 tx=0x7f4e0c009530 comp rx=0 tx=0).stop 2023-11-16T16:57:54.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.997+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1468682400 shutdown_connections 2023-11-16T16:57:54.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.997+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/1468682400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56523094fb90 0x565230952580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.997+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/1468682400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5652307a56b0 0x56523095a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.997+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/1468682400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56523094f720 0x5652309556a0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.997+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1468682400 >> 172.21.15.44:0/1468682400 conn(0x56523093ef50 msgr2=0x56523094cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.997+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1468682400 shutdown_connections 2023-11-16T16:57:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.997+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1468682400 wait complete. 2023-11-16T16:57:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.997+0000 7f4e373b2b80 1 -- start start 2023-11-16T16:57:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.997+0000 7f4e373b2b80 1 Processor -- start 2023-11-16T16:57:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.998+0000 7f4e373b2b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652307a56b0 0x56523094bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.998+0000 7f4e373b2b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56523094f720 0x56523094c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.998+0000 7f4e373b2b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56523094fb90 0x56523094ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.998+0000 7f4e373b2b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56523094c7b0 con 0x56523094f720 2023-11-16T16:57:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.998+0000 7f4e373b2b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56523094c920 con 0x5652307a56b0 2023-11-16T16:57:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.998+0000 7f4e373b2b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56523094b1a0 con 0x56523094fb90 2023-11-16T16:57:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.998+0000 7f4e373b2b80 10 client.0 _ll_register_callbacks cb 0x7f4e0c009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:57:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.998+0000 7f4e20f9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56523094f720 0x56523094c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.998+0000 7f4e20f9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56523094f720 0x56523094c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60606/0 (socket says 172.21.15.44:60606) 2023-11-16T16:57:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.998+0000 7f4e20f9a700 1 -- 172.21.15.44:0/1069707674 learned_addr learned my addr 172.21.15.44:0/1069707674 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:57:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.998+0000 7f4e2179b700 1 --2- 172.21.15.44:0/1069707674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652307a56b0 0x56523094bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.998+0000 7f4e21f9c700 1 --2- 172.21.15.44:0/1069707674 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56523094fb90 0x56523094ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.999+0000 7f4e2179b700 1 -- 172.21.15.44:0/1069707674 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56523094fb90 msgr2=0x56523094ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.999+0000 7f4e2179b700 1 --2- 172.21.15.44:0/1069707674 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56523094fb90 0x56523094ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.999+0000 7f4e2179b700 1 -- 172.21.15.44:0/1069707674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56523094f720 msgr2=0x56523094c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:57:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.999+0000 7f4e2179b700 1 --2- 172.21.15.44:0/1069707674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56523094f720 0x56523094c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.999+0000 7f4e2179b700 1 -- 172.21.15.44:0/1069707674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565230861130 con 0x5652307a56b0 2023-11-16T16:57:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.999+0000 7f4e20f9a700 1 --2- 172.21.15.44:0/1069707674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56523094f720 0x56523094c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:57:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.999+0000 7f4e2179b700 1 --2- 172.21.15.44:0/1069707674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652307a56b0 0x56523094bce0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4e180993c0 tx=0x7f4e180996c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c093a81139beed1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:57:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.999+0000 7f4e04ff9700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:57:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.999+0000 7f4e04ff9700 1 -- 172.21.15.44:0/1069707674 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e180654b0 con 0x5652307a56b0 2023-11-16T16:57:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.999+0000 7f4e04ff9700 1 -- 172.21.15.44:0/1069707674 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f4e18064080 con 0x5652307a56b0 2023-11-16T16:57:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.999+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1069707674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e180b90d0 con 0x5652307a56b0 2023-11-16T16:57:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.999+0000 7f4e04ff9700 1 -- 172.21.15.44:0/1069707674 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e18084250 con 0x5652307a56b0 2023-11-16T16:57:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.999+0000 7f4e373b2b80 10 client.26897 fetch_fsmap learned FSMap version 21 2023-11-16T16:57:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:53.999+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1069707674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x7f4e0c005300 con 0x5652307a56b0 2023-11-16T16:57:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.000+0000 7f4e04ff9700 1 -- 172.21.15.44:0/1069707674 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e180b90d0 con 0x5652307a56b0 2023-11-16T16:57:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.000+0000 7f4e04ff9700 1 -- 172.21.15.44:0/1069707674 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4e18098bb0 con 0x5652307a56b0 2023-11-16T16:57:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.000+0000 7f4e373b2b80 10 client.26897 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:57:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.000+0000 7f4e373b2b80 10 client.26897 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:57:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.000+0000 7f4e373b2b80 20 client.26897 populate_metadata read hostname 'smithi044' 2023-11-16T16:57:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.000+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/1069707674 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5652309b7000 0x5652309b9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:57:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.000+0000 7f4e373b2b80 4 client.26897 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:57:54.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.000+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1069707674 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x565230869600 con 0x5652309b7000 2023-11-16T16:57:54.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.000+0000 7f4e20f9a700 1 --2- 172.21.15.44:0/1069707674 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5652309b7000 0x5652309b9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:57:54.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.001+0000 7f4e20f9a700 1 --2- 172.21.15.44:0/1069707674 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5652309b7000 0x5652309b9410 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6631750f59343467 server_cookie=8b110e6ded52cffd in_seq=0 out_seq=0 2023-11-16T16:57:54.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.001+0000 7f4e04ff9700 10 client.26897 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:54.354 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.351+0000 7f4e04ff9700 1 -- 172.21.15.44:0/1069707674 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x565230869600 con 0x5652309b7000 2023-11-16T16:57:54.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.351+0000 7f4e04ff9700 10 client.26897 handle_command_reply: tid=0 2023-11-16T16:57:54.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.352+0000 7f4e373b2b80 1 client.26897 shutdown 2023-11-16T16:57:54.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.352+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1069707674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652307a56b0 msgr2=0x56523094bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:54.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.352+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/1069707674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652307a56b0 0x56523094bce0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4e180993c0 tx=0x7f4e180996c0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.352+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1069707674 shutdown_connections 2023-11-16T16:57:54.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.352+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/1069707674 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5652309b7000 0x5652309b9410 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.352+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/1069707674 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56523094fb90 0x56523094ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.352+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/1069707674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652307a56b0 0x56523094bce0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.352+0000 7f4e04ff9700 0 client.26897 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:57:54.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.352+0000 7f4e373b2b80 1 --2- 172.21.15.44:0/1069707674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56523094f720 0x56523094c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.352+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1069707674 >> 172.21.15.44:0/1069707674 conn(0x56523093ef50 msgr2=0x56523094de60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:54.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.352+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1069707674 shutdown_connections 2023-11-16T16:57:54.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.352+0000 7f4e373b2b80 1 -- 172.21.15.44:0/1069707674 wait complete. 2023-11-16T16:57:54.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.353+0000 7f4e373b2b80 20 client.26897 trim_cache size 0 max 16384 2023-11-16T16:57:54.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:57:54.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:57:54.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:57:54.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:57:54.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.353+0000 7f4e239ff700 1 -- 172.21.15.44:0/105890510 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4e080a12c0 msgr2=0x7f4e080a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:54.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.353+0000 7f4e239ff700 1 --2- 172.21.15.44:0/105890510 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4e080a12c0 0x7f4e080a3770 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009550 tx=0x7f4e14007ed0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.353+0000 7f4e239ff700 1 -- 172.21.15.44:0/105890510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e1c100c50 msgr2=0x7f4e1c198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:57:54.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.353+0000 7f4e239ff700 1 --2- 172.21.15.44:0/105890510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e1c100c50 0x7f4e1c198600 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800d750 tx=0x7f4e1800da60 comp rx=0 tx=0).stop 2023-11-16T16:57:54.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.353+0000 7f4e239ff700 1 -- 172.21.15.44:0/105890510 shutdown_connections 2023-11-16T16:57:54.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.353+0000 7f4e239ff700 1 --2- 172.21.15.44:0/105890510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c105560 0x7f4e1c19cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.353+0000 7f4e239ff700 1 --2- 172.21.15.44:0/105890510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e1c100c50 0x7f4e1c198600 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.353+0000 7f4e239ff700 1 --2- 172.21.15.44:0/105890510 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4e080a12c0 0x7f4e080a3770 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.353+0000 7f4e239ff700 1 --2- 172.21.15.44:0/105890510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e1c1015f0 0x7f4e1c198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:57:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.353+0000 7f4e239ff700 1 -- 172.21.15.44:0/105890510 >> 172.21.15.44:0/105890510 conn(0x7f4e1c077550 msgr2=0x7f4e1c0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:57:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.354+0000 7f4e239ff700 1 -- 172.21.15.44:0/105890510 shutdown_connections 2023-11-16T16:57:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:57:54.354+0000 7f4e239ff700 1 -- 172.21.15.44:0/105890510 wait complete. 2023-11-16T16:57:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:55 smithi044 ceph-mon[128750]: pgmap v246: 129 pgs: 129 active+clean; 17 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 789 op/s 2023-11-16T16:57:55.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:55 smithi176 ceph-mon[131630]: pgmap v246: 129 pgs: 129 active+clean; 17 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 789 op/s 2023-11-16T16:57:55.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:55 smithi077 ceph-mon[131204]: pgmap v246: 129 pgs: 129 active+clean; 17 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 789 op/s 2023-11-16T16:57:57.220 INFO:tasks.workunit.client.0.smithi044.stdout:ok 2023-11-16T16:57:57.226 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T16:57:57.226 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T16:57:57.324 INFO:tasks.workunit:Running workunit fs/misc/filelock_interrupt.py... 2023-11-16T16:57:57.325 DEBUG:teuthology.orchestra.run.smithi044:workunit test fs/misc/filelock_interrupt.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/filelock_interrupt.py 2023-11-16T16:57:57.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:57 smithi044 ceph-mon[128750]: pgmap v247: 129 pgs: 129 active+clean; 17 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 520 KiB/s wr, 169 op/s 2023-11-16T16:57:57.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:57 smithi176 ceph-mon[131630]: pgmap v247: 129 pgs: 129 active+clean; 17 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 520 KiB/s wr, 169 op/s 2023-11-16T16:57:57.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:57 smithi077 ceph-mon[131204]: pgmap v247: 129 pgs: 129 active+clean; 17 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 520 KiB/s wr, 169 op/s 2023-11-16T16:57:59.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:57:59 smithi044 ceph-mon[128750]: pgmap v248: 129 pgs: 129 active+clean; 5.8 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 610 KiB/s wr, 423 op/s 2023-11-16T16:57:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:57:59 smithi176 ceph-mon[131630]: pgmap v248: 129 pgs: 129 active+clean; 5.8 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 610 KiB/s wr, 423 op/s 2023-11-16T16:57:59.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:57:59 smithi077 ceph-mon[131204]: pgmap v248: 129 pgs: 129 active+clean; 5.8 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 610 KiB/s wr, 423 op/s 2023-11-16T16:58:01.582 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:58:01.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.801+0000 7fbed89ff700 1 -- 172.21.15.44:0/2019666114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed4100cc0 msgr2=0x7fbed4101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:01.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.801+0000 7fbed89ff700 1 --2- 172.21.15.44:0/2019666114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed4100cc0 0x7fbed4101090 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fbec40099c0 tx=0x7fbec4009cd0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.801+0000 7fbed89ff700 1 -- 172.21.15.44:0/2019666114 shutdown_connections 2023-11-16T16:58:01.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.801+0000 7fbed89ff700 1 --2- 172.21.15.44:0/2019666114 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbed4105620 0x7fbed410d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.801+0000 7fbed89ff700 1 --2- 172.21.15.44:0/2019666114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed4100cc0 0x7fbed4101090 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.801+0000 7fbed89ff700 1 --2- 172.21.15.44:0/2019666114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbed4101660 0x7fbed4104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.801+0000 7fbed89ff700 1 -- 172.21.15.44:0/2019666114 >> 172.21.15.44:0/2019666114 conn(0x7fbed4077560 msgr2=0x7fbed4077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:01.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.801+0000 7fbed89ff700 1 -- 172.21.15.44:0/2019666114 shutdown_connections 2023-11-16T16:58:01.804 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.801+0000 7fbed89ff700 1 -- 172.21.15.44:0/2019666114 wait complete. 2023-11-16T16:58:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.802+0000 7fbed89ff700 1 Processor -- start 2023-11-16T16:58:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.802+0000 7fbed89ff700 1 -- start start 2023-11-16T16:58:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.803+0000 7fbed89ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbed4100cc0 0x7fbed41964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.803+0000 7fbed89ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbed4101660 0x7fbed41969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.803+0000 7fbed89ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed4105620 0x7fbed419ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.803+0000 7fbed89ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed419b490 con 0x7fbed4105620 2023-11-16T16:58:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.803+0000 7fbed89ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed419b600 con 0x7fbed4101660 2023-11-16T16:58:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.803+0000 7fbed89ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed419b7c0 con 0x7fbed4100cc0 2023-11-16T16:58:01.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.803+0000 7fbed2d9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed4105620 0x7fbed419ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.803+0000 7fbed2d9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed4105620 0x7fbed419ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51194/0 (socket says 172.21.15.44:51194) 2023-11-16T16:58:01.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.803+0000 7fbed2d9d700 1 -- 172.21.15.44:0/85374050 learned_addr learned my addr 172.21.15.44:0/85374050 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:01.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.803+0000 7fbecbfff700 1 --2- 172.21.15.44:0/85374050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbed4101660 0x7fbed41969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.803+0000 7fbed259c700 1 --2- 172.21.15.44:0/85374050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbed4100cc0 0x7fbed41964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.803+0000 7fbed2d9d700 1 -- 172.21.15.44:0/85374050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbed4100cc0 msgr2=0x7fbed41964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:01.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.803+0000 7fbed2d9d700 1 --2- 172.21.15.44:0/85374050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbed4100cc0 0x7fbed41964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.803+0000 7fbed2d9d700 1 -- 172.21.15.44:0/85374050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbed4101660 msgr2=0x7fbed41969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:01.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.803+0000 7fbed2d9d700 1 --2- 172.21.15.44:0/85374050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbed4101660 0x7fbed41969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.803+0000 7fbed2d9d700 1 -- 172.21.15.44:0/85374050 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbebc009520 con 0x7fbed4105620 2023-11-16T16:58:01.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.804+0000 7fbed2d9d700 1 --2- 172.21.15.44:0/85374050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed4105620 0x7fbed419ae50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fbecc00ef30 tx=0x7fbecc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:01.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.804+0000 7fbecb7fe700 1 -- 172.21.15.44:0/85374050 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc018070 con 0x7fbed4105620 2023-11-16T16:58:01.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.804+0000 7fbecb7fe700 1 -- 172.21.15.44:0/85374050 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbecc00f040 con 0x7fbed4105620 2023-11-16T16:58:01.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.804+0000 7fbecb7fe700 1 -- 172.21.15.44:0/85374050 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc00cd00 con 0x7fbed4105620 2023-11-16T16:58:01.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.804+0000 7fbed89ff700 1 -- 172.21.15.44:0/85374050 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec40095f0 con 0x7fbed4105620 2023-11-16T16:58:01.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.804+0000 7fbed89ff700 1 -- 172.21.15.44:0/85374050 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed4110400 con 0x7fbed4105620 2023-11-16T16:58:01.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.807+0000 7fbecb7fe700 1 -- 172.21.15.44:0/85374050 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fbecc003a00 con 0x7fbed4105620 2023-11-16T16:58:01.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.808+0000 7fbeec28bb80 1 Processor -- start 2023-11-16T16:58:01.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.808+0000 7fbecb7fe700 1 --2- 172.21.15.44:0/85374050 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbec00a1340 0x7fbec00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.809+0000 7fbeec28bb80 1 -- start start 2023-11-16T16:58:01.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.809+0000 7fbecb7fe700 1 -- 172.21.15.44:0/85374050 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fbecc0ccbb0 con 0x7fbed4105620 2023-11-16T16:58:01.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.809+0000 7fbecbfff700 1 --2- 172.21.15.44:0/85374050 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbec00a1340 0x7fbec00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.809+0000 7fbeec28bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557687309b60 0x557687309f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.809+0000 7fbeec28bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55768730a470 0x55768730e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.809+0000 7fbeec28bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730e8b0 0x55768730ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.809+0000 7fbeec28bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55768720f060 con 0x55768730e8b0 2023-11-16T16:58:01.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.809+0000 7fbeec28bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576872618a0 con 0x557687309b60 2023-11-16T16:58:01.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.809+0000 7fbeec28bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55768725ad30 con 0x55768730a470 2023-11-16T16:58:01.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.809+0000 7fbed259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557687309b60 0x557687309f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.809+0000 7fbed259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557687309b60 0x557687309f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:49494/0 (socket says 172.21.15.44:49494) 2023-11-16T16:58:01.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.809+0000 7fbed259c700 1 -- 172.21.15.44:0/2490582752 learned_addr learned my addr 172.21.15.44:0/2490582752 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:01.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.809+0000 7fbecbfff700 1 --2- 172.21.15.44:0/2490582752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55768730a470 0x55768730e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.809+0000 7fbed2d9d700 1 --2- 172.21.15.44:0/2490582752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730e8b0 0x55768730ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.810+0000 7fbed259c700 1 -- 172.21.15.44:0/2490582752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55768730a470 msgr2=0x55768730e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:01.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.810+0000 7fbed259c700 1 --2- 172.21.15.44:0/2490582752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55768730a470 0x55768730e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.810+0000 7fbed259c700 1 -- 172.21.15.44:0/2490582752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730e8b0 msgr2=0x55768730ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:01.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.810+0000 7fbed259c700 1 --2- 172.21.15.44:0/2490582752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730e8b0 0x55768730ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.810+0000 7fbed259c700 1 -- 172.21.15.44:0/2490582752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576870dc630 con 0x557687309b60 2023-11-16T16:58:01.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.810+0000 7fbecbfff700 1 --2- 172.21.15.44:0/85374050 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbec00a1340 0x7fbec00a37f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbed4197ac0 tx=0x7fbebc011a80 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:01.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.810+0000 7fbed259c700 1 --2- 172.21.15.44:0/2490582752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557687309b60 0x557687309f30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fbec40058e0 tx=0x7fbec4005f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a387a185ad02de8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:01.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.810+0000 7fbec8ff9700 1 -- 172.21.15.44:0/2490582752 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec4005390 con 0x557687309b60 2023-11-16T16:58:01.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.810+0000 7fbec8ff9700 1 -- 172.21.15.44:0/2490582752 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbec401e040 con 0x557687309b60 2023-11-16T16:58:01.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.810+0000 7fbec8ff9700 1 -- 172.21.15.44:0/2490582752 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec400bb10 con 0x557687309b60 2023-11-16T16:58:01.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.811+0000 7fbeec28bb80 1 -- 172.21.15.44:0/2490582752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557687309b60 msgr2=0x557687309f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:01.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.811+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/2490582752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557687309b60 0x557687309f30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fbec40058e0 tx=0x7fbec4005f00 comp rx=0 tx=0).stop 2023-11-16T16:58:01.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.811+0000 7fbeec28bb80 1 -- 172.21.15.44:0/2490582752 shutdown_connections 2023-11-16T16:58:01.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.811+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/2490582752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730e8b0 0x55768730ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.811+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/2490582752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557687309b60 0x557687309f30 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.811+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/2490582752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55768730a470 0x55768730e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.811+0000 7fbeec28bb80 1 -- 172.21.15.44:0/2490582752 >> 172.21.15.44:0/2490582752 conn(0x5576872ff360 msgr2=0x557687304b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:01.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.811+0000 7fbeec28bb80 1 -- 172.21.15.44:0/2490582752 shutdown_connections 2023-11-16T16:58:01.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.811+0000 7fbeec28bb80 1 -- 172.21.15.44:0/2490582752 wait complete. 2023-11-16T16:58:01.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.812+0000 7fbeec28bb80 1 -- start start 2023-11-16T16:58:01.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.812+0000 7fbeec28bb80 1 Processor -- start 2023-11-16T16:58:01.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.813+0000 7fbeec28bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557687309b60 0x557687316030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.813+0000 7fbeec28bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768730a470 0x5576873132b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.813+0000 7fbeec28bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730e8b0 0x557687313910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.813+0000 7fbeec28bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576871fc8e0 con 0x55768730e8b0 2023-11-16T16:58:01.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.813+0000 7fbeec28bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576871fc580 con 0x55768730a470 2023-11-16T16:58:01.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.813+0000 7fbeec28bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576870de920 con 0x557687309b60 2023-11-16T16:58:01.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.813+0000 7fbeec28bb80 10 client.0 _ll_register_callbacks cb 0x55768721a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:01.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.813+0000 7fbecbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768730a470 0x5576873132b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.813+0000 7fbed2d9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730e8b0 0x557687313910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.813+0000 7fbed259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557687309b60 0x557687316030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.813+0000 7fbecbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768730a470 0x5576873132b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:49508/0 (socket says 172.21.15.44:49508) 2023-11-16T16:58:01.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.813+0000 7fbecbfff700 1 -- 172.21.15.44:0/1002217724 learned_addr learned my addr 172.21.15.44:0/1002217724 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:01.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.814+0000 7fbecbfff700 1 -- 172.21.15.44:0/1002217724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557687309b60 msgr2=0x557687316030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:01.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.814+0000 7fbecbfff700 1 --2- 172.21.15.44:0/1002217724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557687309b60 0x557687316030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.814+0000 7fbecbfff700 1 -- 172.21.15.44:0/1002217724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730e8b0 msgr2=0x557687313910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:01.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.814+0000 7fbecbfff700 1 --2- 172.21.15.44:0/1002217724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730e8b0 0x557687313910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.814+0000 7fbecbfff700 1 -- 172.21.15.44:0/1002217724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed410d9b0 con 0x55768730a470 2023-11-16T16:58:01.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.814+0000 7fbed2d9d700 1 --2- 172.21.15.44:0/1002217724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730e8b0 0x557687313910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:01.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.814+0000 7fbecbfff700 1 --2- 172.21.15.44:0/1002217724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768730a470 0x5576873132b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbebc004570 tx=0x7fbebc0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e4f99e0fec0fec8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:01.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.814+0000 7fbeb9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:58:01.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.814+0000 7fbeb9ffb700 1 -- 172.21.15.44:0/1002217724 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc00f730 con 0x55768730a470 2023-11-16T16:58:01.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.814+0000 7fbeec28bb80 1 -- 172.21.15.44:0/1002217724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5576872fcb20 con 0x55768730a470 2023-11-16T16:58:01.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.814+0000 7fbeb9ffb700 1 -- 172.21.15.44:0/1002217724 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fbebc00f890 con 0x55768730a470 2023-11-16T16:58:01.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.814+0000 7fbeb9ffb700 1 -- 172.21.15.44:0/1002217724 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbebc0045a0 con 0x55768730a470 2023-11-16T16:58:01.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.814+0000 7fbeec28bb80 10 client.26909 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:01.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.814+0000 7fbeec28bb80 1 -- 172.21.15.44:0/1002217724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x557687221190 con 0x55768730a470 2023-11-16T16:58:01.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.815+0000 7fbeb9ffb700 1 -- 172.21.15.44:0/1002217724 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5576872fcb20 con 0x55768730a470 2023-11-16T16:58:01.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.815+0000 7fbeb9ffb700 1 -- 172.21.15.44:0/1002217724 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbebc020b70 con 0x55768730a470 2023-11-16T16:58:01.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.815+0000 7fbeec28bb80 10 client.26909 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:01.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.815+0000 7fbeec28bb80 10 client.26909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:01.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.815+0000 7fbeec28bb80 20 client.26909 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:01.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.815+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/1002217724 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557687323940 0x557687325d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.815+0000 7fbeec28bb80 4 client.26909 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:58:01.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.815+0000 7fbeec28bb80 1 -- 172.21.15.44:0/1002217724 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55768721d0a0 con 0x557687323940 2023-11-16T16:58:01.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.815+0000 7fbed259c700 1 --2- 172.21.15.44:0/1002217724 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557687323940 0x557687325d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.816+0000 7fbed259c700 1 --2- 172.21.15.44:0/1002217724 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557687323940 0x557687325d50 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df06ff750cc26dba server_cookie=236059378781d821 in_seq=0 out_seq=0 2023-11-16T16:58:01.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.816+0000 7fbeb9ffb700 10 client.26909 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:01.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.817+0000 7fbeb9ffb700 1 -- 172.21.15.44:0/1002217724 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55768721d0a0 con 0x557687323940 2023-11-16T16:58:01.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.817+0000 7fbeb9ffb700 10 client.26909 handle_command_reply: tid=0 2023-11-16T16:58:01.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.817+0000 7fbeec28bb80 1 client.26909 shutdown 2023-11-16T16:58:01.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.817+0000 7fbeec28bb80 1 -- 172.21.15.44:0/1002217724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768730a470 msgr2=0x5576873132b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:01.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.817+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/1002217724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768730a470 0x5576873132b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbebc004570 tx=0x7fbebc0047c0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.817+0000 7fbeec28bb80 1 -- 172.21.15.44:0/1002217724 shutdown_connections 2023-11-16T16:58:01.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.817+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/1002217724 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557687323940 0x557687325d50 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.817+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/1002217724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730e8b0 0x557687313910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.817+0000 7fbeb9ffb700 0 client.26909 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:01.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.817+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/1002217724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557687309b60 0x557687316030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.817+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/1002217724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768730a470 0x5576873132b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.817+0000 7fbeec28bb80 1 -- 172.21.15.44:0/1002217724 >> 172.21.15.44:0/1002217724 conn(0x5576872ff360 msgr2=0x55768730fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:01.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.817+0000 7fbeec28bb80 1 -- 172.21.15.44:0/1002217724 shutdown_connections 2023-11-16T16:58:01.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.817+0000 7fbeec28bb80 1 -- 172.21.15.44:0/1002217724 wait complete. 2023-11-16T16:58:01.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.817+0000 7fbeec28bb80 20 client.26909 trim_cache size 0 max 16384 2023-11-16T16:58:01.837 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.835+0000 7fbeec28bb80 1 Processor -- start 2023-11-16T16:58:01.837 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.835+0000 7fbeec28bb80 1 -- start start 2023-11-16T16:58:01.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.835+0000 7fbeec28bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557687309b60 0x55768731a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.838 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.835+0000 7fbeec28bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730a470 0x55768730c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.835+0000 7fbeec28bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55768730e8b0 0x55768730ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.835+0000 7fbeec28bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55768730abc0 con 0x55768730a470 2023-11-16T16:58:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.835+0000 7fbeec28bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55768731f800 con 0x557687309b60 2023-11-16T16:58:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.835+0000 7fbeec28bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x557687311810 con 0x55768730e8b0 2023-11-16T16:58:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.836+0000 7fbed2d9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55768730e8b0 0x55768730ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.836+0000 7fbecbfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730a470 0x55768730c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.836+0000 7fbed2d9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55768730e8b0 0x55768730ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:40180/0 (socket says 172.21.15.44:40180) 2023-11-16T16:58:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.836+0000 7fbed2d9d700 1 -- 172.21.15.44:0/657316296 learned_addr learned my addr 172.21.15.44:0/657316296 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.836+0000 7fbed259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557687309b60 0x55768731a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.836+0000 7fbed2d9d700 1 -- 172.21.15.44:0/657316296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557687309b60 msgr2=0x55768731a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:01.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.836+0000 7fbed2d9d700 1 --2- 172.21.15.44:0/657316296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557687309b60 0x55768731a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.836+0000 7fbed2d9d700 1 -- 172.21.15.44:0/657316296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730a470 msgr2=0x55768730c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:01.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.836+0000 7fbed2d9d700 1 --2- 172.21.15.44:0/657316296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730a470 0x55768730c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.836+0000 7fbed2d9d700 1 -- 172.21.15.44:0/657316296 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557687300b00 con 0x55768730e8b0 2023-11-16T16:58:01.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.836+0000 7fbed259c700 1 --2- 172.21.15.44:0/657316296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557687309b60 0x55768731a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:01.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.836+0000 7fbed2d9d700 1 --2- 172.21.15.44:0/657316296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55768730e8b0 0x55768730ce30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbecc00ef00 tx=0x7fbecc09f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1a9aee22f8cf6c1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:01.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.837+0000 7fbec97fa700 1 -- 172.21.15.44:0/657316296 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc004ad0 con 0x55768730e8b0 2023-11-16T16:58:01.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.837+0000 7fbec97fa700 1 -- 172.21.15.44:0/657316296 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbecc00ce60 con 0x55768730e8b0 2023-11-16T16:58:01.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.837+0000 7fbec97fa700 1 -- 172.21.15.44:0/657316296 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc099b30 con 0x55768730e8b0 2023-11-16T16:58:01.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.837+0000 7fbeec28bb80 1 -- 172.21.15.44:0/657316296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55768730e8b0 msgr2=0x55768730ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:01.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.837+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/657316296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55768730e8b0 0x55768730ce30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbecc00ef00 tx=0x7fbecc09f040 comp rx=0 tx=0).stop 2023-11-16T16:58:01.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.837+0000 7fbeec28bb80 1 -- 172.21.15.44:0/657316296 shutdown_connections 2023-11-16T16:58:01.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.837+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/657316296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55768730e8b0 0x55768730ce30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.837+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/657316296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557687309b60 0x55768731a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.837+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/657316296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730a470 0x55768730c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.837+0000 7fbeec28bb80 1 -- 172.21.15.44:0/657316296 >> 172.21.15.44:0/657316296 conn(0x5576872ff360 msgr2=0x55768731f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.837+0000 7fbeec28bb80 1 -- 172.21.15.44:0/657316296 shutdown_connections 2023-11-16T16:58:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.837+0000 7fbeec28bb80 1 -- 172.21.15.44:0/657316296 wait complete. 2023-11-16T16:58:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.838+0000 7fbeec28bb80 1 -- start start 2023-11-16T16:58:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.838+0000 7fbeec28bb80 1 Processor -- start 2023-11-16T16:58:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.839+0000 7fbeec28bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557687309b60 0x557687318b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.839+0000 7fbeec28bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730a470 0x557687319060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.839+0000 7fbeec28bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768730e8b0 0x557687376800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.839+0000 7fbeec28bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x557687376d40 con 0x55768730a470 2023-11-16T16:58:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.839+0000 7fbeec28bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x557687376e80 con 0x55768730e8b0 2023-11-16T16:58:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.839+0000 7fbeec28bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x557687376fc0 con 0x557687309b60 2023-11-16T16:58:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.839+0000 7fbeec28bb80 10 client.0 _ll_register_callbacks cb 0x7fbebc004340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.839+0000 7fbed2d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768730e8b0 0x557687376800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.839+0000 7fbecbfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730a470 0x557687319060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.839+0000 7fbed259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557687309b60 0x557687318b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.839+0000 7fbed2d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768730e8b0 0x557687376800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:49524/0 (socket says 172.21.15.44:49524) 2023-11-16T16:58:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.839+0000 7fbed2d9d700 1 -- 172.21.15.44:0/851582574 learned_addr learned my addr 172.21.15.44:0/851582574 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.840+0000 7fbed2d9d700 1 -- 172.21.15.44:0/851582574 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557687309b60 msgr2=0x557687318b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.840+0000 7fbed2d9d700 1 --2- 172.21.15.44:0/851582574 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557687309b60 0x557687318b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.840+0000 7fbed2d9d700 1 -- 172.21.15.44:0/851582574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730a470 msgr2=0x557687319060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:01.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.840+0000 7fbed2d9d700 1 --2- 172.21.15.44:0/851582574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730a470 0x557687319060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.840+0000 7fbed2d9d700 1 -- 172.21.15.44:0/851582574 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5576870dc630 con 0x55768730e8b0 2023-11-16T16:58:01.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.840+0000 7fbecbfff700 1 --2- 172.21.15.44:0/851582574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730a470 0x557687319060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:01.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.840+0000 7fbed259c700 1 --2- 172.21.15.44:0/851582574 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557687309b60 0x557687318b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:01.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.840+0000 7fbed2d9d700 1 --2- 172.21.15.44:0/851582574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768730e8b0 0x557687376800 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0b9010 tx=0x7fbecc004dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=addb8723979efa49 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:01.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.840+0000 7fbeb9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:58:01.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.840+0000 7fbeb9ffb700 1 -- 172.21.15.44:0/851582574 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc09d900 con 0x55768730e8b0 2023-11-16T16:58:01.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.840+0000 7fbeb9ffb700 1 -- 172.21.15.44:0/851582574 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fbecc0a1c90 con 0x55768730e8b0 2023-11-16T16:58:01.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.840+0000 7fbeec28bb80 1 -- 172.21.15.44:0/851582574 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbec401e040 con 0x55768730e8b0 2023-11-16T16:58:01.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.840+0000 7fbeb9ffb700 1 -- 172.21.15.44:0/851582574 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbecc0998d0 con 0x55768730e8b0 2023-11-16T16:58:01.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.840+0000 7fbeec28bb80 10 client.26915 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:01.848 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.840+0000 7fbeec28bb80 1 -- 172.21.15.44:0/851582574 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x7fbecc0b7e70 con 0x55768730e8b0 2023-11-16T16:58:01.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.841+0000 7fbeb9ffb700 1 -- 172.21.15.44:0/851582574 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbec401e040 con 0x55768730e8b0 2023-11-16T16:58:01.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.841+0000 7fbeb9ffb700 1 -- 172.21.15.44:0/851582574 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbecc09db60 con 0x55768730e8b0 2023-11-16T16:58:01.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.841+0000 7fbeec28bb80 10 client.26915 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:01.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.841+0000 7fbeec28bb80 10 client.26915 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:01.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.841+0000 7fbeec28bb80 20 client.26915 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:01.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.841+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/851582574 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557687378340 0x55768737a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:01.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.841+0000 7fbeec28bb80 4 client.26915 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:58:01.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.841+0000 7fbeec28bb80 1 -- 172.21.15.44:0/851582574 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557687229660 con 0x557687378340 2023-11-16T16:58:01.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.841+0000 7fbed259c700 1 --2- 172.21.15.44:0/851582574 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557687378340 0x55768737a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:01.849 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.841+0000 7fbed259c700 1 --2- 172.21.15.44:0/851582574 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557687378340 0x55768737a720 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=678447d61d00f849 server_cookie=abc1d535d62814d in_seq=0 out_seq=0 2023-11-16T16:58:01.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.841+0000 7fbeb9ffb700 10 client.26915 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:01.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.842+0000 7fbeb9ffb700 1 -- 172.21.15.44:0/851582574 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x557687229660 con 0x557687378340 2023-11-16T16:58:01.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.842+0000 7fbeb9ffb700 10 client.26915 handle_command_reply: tid=0 2023-11-16T16:58:01.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.842+0000 7fbeec28bb80 1 client.26915 shutdown 2023-11-16T16:58:01.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.842+0000 7fbeec28bb80 1 -- 172.21.15.44:0/851582574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768730e8b0 msgr2=0x557687376800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:01.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.842+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/851582574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768730e8b0 0x557687376800 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0b9010 tx=0x7fbecc004dc0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.842+0000 7fbeec28bb80 1 -- 172.21.15.44:0/851582574 shutdown_connections 2023-11-16T16:58:01.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.842+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/851582574 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557687378340 0x55768737a720 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.842+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/851582574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768730e8b0 0x557687376800 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.842+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/851582574 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557687309b60 0x557687318b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.842+0000 7fbeb9ffb700 0 client.26915 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:01.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.842+0000 7fbeec28bb80 1 --2- 172.21.15.44:0/851582574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55768730a470 0x557687319060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.842+0000 7fbeec28bb80 1 -- 172.21.15.44:0/851582574 >> 172.21.15.44:0/851582574 conn(0x5576872ff360 msgr2=0x55768730c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:01.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.842+0000 7fbeec28bb80 1 -- 172.21.15.44:0/851582574 shutdown_connections 2023-11-16T16:58:01.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.842+0000 7fbeec28bb80 1 -- 172.21.15.44:0/851582574 wait complete. 2023-11-16T16:58:01.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.842+0000 7fbeec28bb80 20 client.26915 trim_cache size 0 max 16384 2023-11-16T16:58:01.851 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:58:01.851 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T16:58:01.851 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T16:58:01.852 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:58:01.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.843+0000 7fbed89ff700 1 -- 172.21.15.44:0/85374050 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbec00a1340 msgr2=0x7fbec00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:01.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.843+0000 7fbed89ff700 1 --2- 172.21.15.44:0/85374050 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbec00a1340 0x7fbec00a37f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbed4197ac0 tx=0x7fbebc011a80 comp rx=0 tx=0).stop 2023-11-16T16:58:01.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.843+0000 7fbed89ff700 1 -- 172.21.15.44:0/85374050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed4105620 msgr2=0x7fbed419ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:01.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.843+0000 7fbed89ff700 1 --2- 172.21.15.44:0/85374050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed4105620 0x7fbed419ae50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fbecc00ef30 tx=0x7fbecc00c3a0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.843+0000 7fbed89ff700 1 -- 172.21.15.44:0/85374050 shutdown_connections 2023-11-16T16:58:01.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.843+0000 7fbed89ff700 1 --2- 172.21.15.44:0/85374050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed4105620 0x7fbed419ae50 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.843+0000 7fbed89ff700 1 --2- 172.21.15.44:0/85374050 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fbec00a1340 0x7fbec00a37f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.843+0000 7fbed89ff700 1 --2- 172.21.15.44:0/85374050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbed4100cc0 0x7fbed41964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.843+0000 7fbed89ff700 1 --2- 172.21.15.44:0/85374050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbed4101660 0x7fbed41969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:01.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.843+0000 7fbed89ff700 1 -- 172.21.15.44:0/85374050 >> 172.21.15.44:0/85374050 conn(0x7fbed4077560 msgr2=0x7fbed40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:01.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.843+0000 7fbed89ff700 1 -- 172.21.15.44:0/85374050 shutdown_connections 2023-11-16T16:58:01.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:01.843+0000 7fbed89ff700 1 -- 172.21.15.44:0/85374050 wait complete. 2023-11-16T16:58:01.861 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:01 smithi044 ceph-mon[128750]: pgmap v249: 129 pgs: 129 active+clean; 5.8 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 110 KiB/s wr, 296 op/s 2023-11-16T16:58:01.863 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T16:58:01.864 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:58:01.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:01 smithi176 ceph-mon[131630]: pgmap v249: 129 pgs: 129 active+clean; 5.8 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 110 KiB/s wr, 296 op/s 2023-11-16T16:58:01.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:01 smithi077 ceph-mon[131204]: pgmap v249: 129 pgs: 129 active+clean; 5.8 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 110 KiB/s wr, 296 op/s 2023-11-16T16:58:02.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.067+0000 7f277b1ff700 1 -- 172.21.15.44:0/2990931474 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2774072e90 msgr2=0x7f2774073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:02.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.067+0000 7f277b1ff700 1 --2- 172.21.15.44:0/2990931474 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2774072e90 0x7f2774073260 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f2768009990 tx=0x7f2768009ca0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.069 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.067+0000 7f277b1ff700 1 -- 172.21.15.44:0/2990931474 shutdown_connections 2023-11-16T16:58:02.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.067+0000 7f277b1ff700 1 --2- 172.21.15.44:0/2990931474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27741112b0 0x7f2774113690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.067+0000 7f277b1ff700 1 --2- 172.21.15.44:0/2990931474 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2774072e90 0x7f2774073260 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.067+0000 7f277b1ff700 1 --2- 172.21.15.44:0/2990931474 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27740737a0 0x7f2774110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.067+0000 7f277b1ff700 1 -- 172.21.15.44:0/2990931474 >> 172.21.15.44:0/2990931474 conn(0x7f2774077050 msgr2=0x7f2774077450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:02.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.067+0000 7f277b1ff700 1 -- 172.21.15.44:0/2990931474 shutdown_connections 2023-11-16T16:58:02.070 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.067+0000 7f277b1ff700 1 -- 172.21.15.44:0/2990931474 wait complete. 2023-11-16T16:58:02.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.068+0000 7f277b1ff700 1 Processor -- start 2023-11-16T16:58:02.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.069+0000 7f277b1ff700 1 -- start start 2023-11-16T16:58:02.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.069+0000 7f277b1ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2774072e90 0x7f277419c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.069+0000 7f277b1ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27740737a0 0x7f277419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.069+0000 7f277b1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27741112b0 0x7f27741a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.069+0000 7f277b1ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27741a1900 con 0x7f2774072e90 2023-11-16T16:58:02.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.069+0000 7f277b1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27741a1a70 con 0x7f27741112b0 2023-11-16T16:58:02.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.069+0000 7f277b1ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27741a1c30 con 0x7f27740737a0 2023-11-16T16:58:02.072 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.069+0000 7f2778f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2774072e90 0x7f277419c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.069+0000 7f2778f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2774072e90 0x7f277419c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51242/0 (socket says 172.21.15.44:51242) 2023-11-16T16:58:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.069+0000 7f2778f9b700 1 -- 172.21.15.44:0/3738674150 learned_addr learned my addr 172.21.15.44:0/3738674150 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.069+0000 7f2773fff700 1 --2- 172.21.15.44:0/3738674150 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27740737a0 0x7f277419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.069+0000 7f277979c700 1 --2- 172.21.15.44:0/3738674150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27741112b0 0x7f27741a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.070+0000 7f2778f9b700 1 -- 172.21.15.44:0/3738674150 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27740737a0 msgr2=0x7f277419cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.070+0000 7f2778f9b700 1 --2- 172.21.15.44:0/3738674150 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27740737a0 0x7f277419cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.070+0000 7f2778f9b700 1 -- 172.21.15.44:0/3738674150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27741112b0 msgr2=0x7f27741a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.070+0000 7f2778f9b700 1 --2- 172.21.15.44:0/3738674150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27741112b0 0x7f27741a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.070+0000 7f2778f9b700 1 -- 172.21.15.44:0/3738674150 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27680095f0 con 0x7f2774072e90 2023-11-16T16:58:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.070+0000 7f2778f9b700 1 --2- 172.21.15.44:0/3738674150 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2774072e90 0x7f277419c960 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2768005910 tx=0x7f2768005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.070+0000 7f2771ffb700 1 -- 172.21.15.44:0/3738674150 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276801d070 con 0x7f2774072e90 2023-11-16T16:58:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.070+0000 7f2771ffb700 1 -- 172.21.15.44:0/3738674150 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2768022470 con 0x7f2774072e90 2023-11-16T16:58:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.070+0000 7f277b1ff700 1 -- 172.21.15.44:0/3738674150 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2774116930 con 0x7f2774072e90 2023-11-16T16:58:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.070+0000 7f2771ffb700 1 -- 172.21.15.44:0/3738674150 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2768017650 con 0x7f2774072e90 2023-11-16T16:58:02.074 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.070+0000 7f277b1ff700 1 -- 172.21.15.44:0/3738674150 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27741167a0 con 0x7f2774072e90 2023-11-16T16:58:02.074 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.071+0000 7f277b1ff700 1 -- 172.21.15.44:0/3738674150 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2774116ac0 con 0x7f2774072e90 2023-11-16T16:58:02.075 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.073+0000 7f2771ffb700 1 -- 172.21.15.44:0/3738674150 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f27680225e0 con 0x7f2774072e90 2023-11-16T16:58:02.076 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.075+0000 7f2771ffb700 1 --2- 172.21.15.44:0/3738674150 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f27640a12c0 0x7f27640a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.076 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.075+0000 7f2771ffb700 1 -- 172.21.15.44:0/3738674150 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f27680ce460 con 0x7f2774072e90 2023-11-16T16:58:02.077 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.075+0000 7f2773fff700 1 --2- 172.21.15.44:0/3738674150 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f27640a12c0 0x7f27640a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.077 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.076+0000 7f2773fff700 1 --2- 172.21.15.44:0/3738674150 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f27640a12c0 0x7f27640a3770 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f277419df70 tx=0x7f2760009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:02.080 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.079+0000 7f2771ffb700 1 -- 172.21.15.44:0/3738674150 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f27680974c0 con 0x7f2774072e90 2023-11-16T16:58:02.334 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.332+0000 7f277b1ff700 1 -- 172.21.15.44:0/3738674150 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2774102ef0 con 0x7f2774072e90 2023-11-16T16:58:02.334 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.333+0000 7f2771ffb700 1 -- 172.21.15.44:0/3738674150 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f276809aae0 con 0x7f2774072e90 2023-11-16T16:58:02.335 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:58:02.335 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:58:02.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.335+0000 7f277b1ff700 1 -- 172.21.15.44:0/3738674150 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f27640a12c0 msgr2=0x7f27640a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:02.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.335+0000 7f277b1ff700 1 --2- 172.21.15.44:0/3738674150 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f27640a12c0 0x7f27640a3770 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f277419df70 tx=0x7f2760009260 comp rx=0 tx=0).stop 2023-11-16T16:58:02.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.335+0000 7f277b1ff700 1 -- 172.21.15.44:0/3738674150 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2774072e90 msgr2=0x7f277419c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:02.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.335+0000 7f277b1ff700 1 --2- 172.21.15.44:0/3738674150 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2774072e90 0x7f277419c960 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2768005910 tx=0x7f2768005720 comp rx=0 tx=0).stop 2023-11-16T16:58:02.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.335+0000 7f277b1ff700 1 -- 172.21.15.44:0/3738674150 shutdown_connections 2023-11-16T16:58:02.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.335+0000 7f277b1ff700 1 --2- 172.21.15.44:0/3738674150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27741112b0 0x7f27741a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.335+0000 7f277b1ff700 1 --2- 172.21.15.44:0/3738674150 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2774072e90 0x7f277419c960 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.335+0000 7f277b1ff700 1 --2- 172.21.15.44:0/3738674150 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f27640a12c0 0x7f27640a3770 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.335+0000 7f277b1ff700 1 --2- 172.21.15.44:0/3738674150 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27740737a0 0x7f277419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.335+0000 7f277b1ff700 1 -- 172.21.15.44:0/3738674150 >> 172.21.15.44:0/3738674150 conn(0x7f2774077050 msgr2=0x7f2774100030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:02.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.335+0000 7f277b1ff700 1 -- 172.21.15.44:0/3738674150 shutdown_connections 2023-11-16T16:58:02.338 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.335+0000 7f277b1ff700 1 -- 172.21.15.44:0/3738674150 wait complete. 2023-11-16T16:58:02.338 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:58:02.356 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T16:58:02.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.561+0000 7ff1533ff700 1 -- 172.21.15.44:0/1648481488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff14c101660 msgr2=0x7ff14c104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:02.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.561+0000 7ff1533ff700 1 --2- 172.21.15.44:0/1648481488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff14c101660 0x7ff14c104ef0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff13c0099c0 tx=0x7ff13c009cd0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.561+0000 7ff1533ff700 1 -- 172.21.15.44:0/1648481488 shutdown_connections 2023-11-16T16:58:02.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.561+0000 7ff1533ff700 1 --2- 172.21.15.44:0/1648481488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff14c105620 0x7ff14c10d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.563 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.561+0000 7ff1533ff700 1 --2- 172.21.15.44:0/1648481488 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff14c100cc0 0x7ff14c101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.561+0000 7ff1533ff700 1 --2- 172.21.15.44:0/1648481488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff14c101660 0x7ff14c104ef0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.561+0000 7ff1533ff700 1 -- 172.21.15.44:0/1648481488 >> 172.21.15.44:0/1648481488 conn(0x7ff14c077560 msgr2=0x7ff14c077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:02.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.561+0000 7ff1533ff700 1 -- 172.21.15.44:0/1648481488 shutdown_connections 2023-11-16T16:58:02.564 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.561+0000 7ff1533ff700 1 -- 172.21.15.44:0/1648481488 wait complete. 2023-11-16T16:58:02.565 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.562+0000 7ff1533ff700 1 Processor -- start 2023-11-16T16:58:02.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.563+0000 7ff1533ff700 1 -- start start 2023-11-16T16:58:02.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.563+0000 7ff1533ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff14c100cc0 0x7ff14c072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.563+0000 7ff1533ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff14c101660 0x7ff14c073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.563+0000 7ff1533ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff14c105620 0x7ff14c1a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.563+0000 7ff1533ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c073b60 con 0x7ff14c105620 2023-11-16T16:58:02.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.563+0000 7ff1533ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c1a4bc0 con 0x7ff14c100cc0 2023-11-16T16:58:02.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.563+0000 7ff1533ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c1a4d00 con 0x7ff14c101660 2023-11-16T16:58:02.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.563+0000 7ff15119b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff14c100cc0 0x7ff14c072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.563+0000 7ff15199c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff14c105620 0x7ff14c1a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.563+0000 7ff15119b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff14c100cc0 0x7ff14c072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:49558/0 (socket says 172.21.15.44:49558) 2023-11-16T16:58:02.566 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.563+0000 7ff15119b700 1 -- 172.21.15.44:0/2923964688 learned_addr learned my addr 172.21.15.44:0/2923964688 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:02.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.564+0000 7ff15099a700 1 --2- 172.21.15.44:0/2923964688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff14c101660 0x7ff14c073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.564+0000 7ff15119b700 1 -- 172.21.15.44:0/2923964688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff14c101660 msgr2=0x7ff14c073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:02.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.564+0000 7ff15119b700 1 --2- 172.21.15.44:0/2923964688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff14c101660 0x7ff14c073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.564+0000 7ff15119b700 1 -- 172.21.15.44:0/2923964688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff14c105620 msgr2=0x7ff14c1a4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:02.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.564+0000 7ff15119b700 1 --2- 172.21.15.44:0/2923964688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff14c105620 0x7ff14c1a4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.564+0000 7ff15119b700 1 -- 172.21.15.44:0/2923964688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c0095f0 con 0x7ff14c100cc0 2023-11-16T16:58:02.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.564+0000 7ff15199c700 1 --2- 172.21.15.44:0/2923964688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff14c105620 0x7ff14c1a4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:02.567 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.564+0000 7ff15119b700 1 --2- 172.21.15.44:0/2923964688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff14c100cc0 0x7ff14c072df0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff14800e9a0 tx=0x7ff14800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:02.568 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.564+0000 7ff1427fc700 1 -- 172.21.15.44:0/2923964688 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff148009790 con 0x7ff14c100cc0 2023-11-16T16:58:02.568 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.565+0000 7ff1533ff700 1 -- 172.21.15.44:0/2923964688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14c110370 con 0x7ff14c100cc0 2023-11-16T16:58:02.568 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.565+0000 7ff1427fc700 1 -- 172.21.15.44:0/2923964688 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff148010430 con 0x7ff14c100cc0 2023-11-16T16:58:02.568 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.565+0000 7ff1427fc700 1 -- 172.21.15.44:0/2923964688 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff148018680 con 0x7ff14c100cc0 2023-11-16T16:58:02.568 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.565+0000 7ff1533ff700 1 -- 172.21.15.44:0/2923964688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c1101e0 con 0x7ff14c100cc0 2023-11-16T16:58:02.570 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.568+0000 7ff1427fc700 1 -- 172.21.15.44:0/2923964688 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7ff1480105a0 con 0x7ff14c100cc0 2023-11-16T16:58:02.571 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.569+0000 7ff166d9db80 1 Processor -- start 2023-11-16T16:58:02.571 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.570+0000 7ff166d9db80 1 -- start start 2023-11-16T16:58:02.573 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.570+0000 7ff166d9db80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb48b60 0x56119eb48f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.573 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.570+0000 7ff166d9db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb49470 0x56119eb4d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.573 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.570+0000 7ff166d9db80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb4d8b0 0x56119eb4dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.573 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.570+0000 7ff1427fc700 1 --2- 172.21.15.44:0/2923964688 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff1380a12c0 0x7ff1380a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.573 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.570+0000 7ff166d9db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56119ea4e060 con 0x56119eb49470 2023-11-16T16:58:02.573 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.570+0000 7ff166d9db80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56119eaa08a0 con 0x56119eb48b60 2023-11-16T16:58:02.573 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.570+0000 7ff166d9db80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56119ea99d30 con 0x56119eb4d8b0 2023-11-16T16:58:02.574 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.570+0000 7ff1427fc700 1 -- 172.21.15.44:0/2923964688 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7ff148014070 con 0x7ff14c100cc0 2023-11-16T16:58:02.574 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.570+0000 7ff15199c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb48b60 0x56119eb48f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.574 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.570+0000 7ff15119b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb49470 0x56119eb4d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.574 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.570+0000 7ff15199c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb48b60 0x56119eb48f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:49560/0 (socket says 172.21.15.44:49560) 2023-11-16T16:58:02.574 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.570+0000 7ff15199c700 1 -- 172.21.15.44:0/115787388 learned_addr learned my addr 172.21.15.44:0/115787388 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:02.574 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.570+0000 7ff15099a700 1 --2- 172.21.15.44:0/115787388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb4d8b0 0x56119eb4dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.574 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.570+0000 7ff15119b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb49470 0x56119eb4d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51272/0 (socket says 172.21.15.44:51272) 2023-11-16T16:58:02.574 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.570+0000 7ff15199c700 1 --2- 172.21.15.44:0/2923964688 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff1380a12c0 0x7ff1380a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.574 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.571+0000 7ff15119b700 1 -- 172.21.15.44:0/115787388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb4d8b0 msgr2=0x56119eb4dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:02.574 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.571+0000 7ff15119b700 1 --2- 172.21.15.44:0/115787388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb4d8b0 0x56119eb4dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.574 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.571+0000 7ff15119b700 1 -- 172.21.15.44:0/115787388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb48b60 msgr2=0x56119eb48f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:02.574 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.571+0000 7ff15119b700 1 --2- 172.21.15.44:0/115787388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb48b60 0x56119eb48f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.571+0000 7ff15119b700 1 -- 172.21.15.44:0/115787388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56119e91b630 con 0x56119eb49470 2023-11-16T16:58:02.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.571+0000 7ff15199c700 1 --2- 172.21.15.44:0/115787388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb48b60 0x56119eb48f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:02.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.571+0000 7ff15119b700 1 --2- 172.21.15.44:0/115787388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb49470 0x56119eb4d370 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff1480ca620 tx=0x7ff148099900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2b8ef0d8be5ce4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:02.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.571+0000 7ff15199c700 1 --2- 172.21.15.44:0/2923964688 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff1380a12c0 0x7ff1380a3770 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff144008ea0 tx=0x7ff14400b040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:02.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.571+0000 7ff1377fe700 1 -- 172.21.15.44:0/115787388 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14809dfd0 con 0x56119eb49470 2023-11-16T16:58:02.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.571+0000 7ff1377fe700 1 -- 172.21.15.44:0/115787388 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1480a23f0 con 0x56119eb49470 2023-11-16T16:58:02.575 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.571+0000 7ff1377fe700 1 -- 172.21.15.44:0/115787388 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1480996d0 con 0x56119eb49470 2023-11-16T16:58:02.579 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.572+0000 7ff166d9db80 1 -- 172.21.15.44:0/115787388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb49470 msgr2=0x56119eb4d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:02.579 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.572+0000 7ff166d9db80 1 --2- 172.21.15.44:0/115787388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb49470 0x56119eb4d370 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff1480ca620 tx=0x7ff148099900 comp rx=0 tx=0).stop 2023-11-16T16:58:02.579 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.572+0000 7ff166d9db80 1 -- 172.21.15.44:0/115787388 shutdown_connections 2023-11-16T16:58:02.580 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.572+0000 7ff166d9db80 1 --2- 172.21.15.44:0/115787388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb4d8b0 0x56119eb4dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.580 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.572+0000 7ff166d9db80 1 --2- 172.21.15.44:0/115787388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb48b60 0x56119eb48f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.580 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.572+0000 7ff166d9db80 1 --2- 172.21.15.44:0/115787388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb49470 0x56119eb4d370 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.580 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.572+0000 7ff166d9db80 1 -- 172.21.15.44:0/115787388 >> 172.21.15.44:0/115787388 conn(0x56119eb3e360 msgr2=0x56119eb43b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:02.580 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.572+0000 7ff166d9db80 1 -- 172.21.15.44:0/115787388 shutdown_connections 2023-11-16T16:58:02.580 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.572+0000 7ff166d9db80 1 -- 172.21.15.44:0/115787388 wait complete. 2023-11-16T16:58:02.580 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.573+0000 7ff166d9db80 1 -- start start 2023-11-16T16:58:02.580 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.573+0000 7ff166d9db80 1 Processor -- start 2023-11-16T16:58:02.580 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.573+0000 7ff166d9db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb48b60 0x56119eb55030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.580 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.573+0000 7ff166d9db80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb49470 0x56119eb522b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.580 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.573+0000 7ff166d9db80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb4d8b0 0x56119eb52890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.580 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.573+0000 7ff166d9db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56119ea3b8e0 con 0x56119eb48b60 2023-11-16T16:58:02.581 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.573+0000 7ff166d9db80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56119ea3b580 con 0x56119eb4d8b0 2023-11-16T16:58:02.581 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.573+0000 7ff166d9db80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56119e91d920 con 0x56119eb49470 2023-11-16T16:58:02.581 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.574+0000 7ff166d9db80 10 client.0 _ll_register_callbacks cb 0x56119ea59780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:02.581 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.574+0000 7ff15119b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb48b60 0x56119eb55030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.581 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.574+0000 7ff15099a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb49470 0x56119eb522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.581 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.574+0000 7ff15099a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb49470 0x56119eb522b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:40224/0 (socket says 172.21.15.44:40224) 2023-11-16T16:58:02.581 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.574+0000 7ff15099a700 1 -- 172.21.15.44:0/176586281 learned_addr learned my addr 172.21.15.44:0/176586281 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:02.581 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.574+0000 7ff15199c700 1 --2- 172.21.15.44:0/176586281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb4d8b0 0x56119eb52890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.581 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.574+0000 7ff15099a700 1 -- 172.21.15.44:0/176586281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb4d8b0 msgr2=0x56119eb52890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:02.581 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.574+0000 7ff15099a700 1 --2- 172.21.15.44:0/176586281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb4d8b0 0x56119eb52890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.581 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.574+0000 7ff15099a700 1 -- 172.21.15.44:0/176586281 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb48b60 msgr2=0x56119eb55030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:02.581 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.574+0000 7ff15099a700 1 --2- 172.21.15.44:0/176586281 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb48b60 0x56119eb55030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.581 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.574+0000 7ff15099a700 1 -- 172.21.15.44:0/176586281 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c1101e0 con 0x56119eb49470 2023-11-16T16:58:02.582 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.574+0000 7ff15119b700 1 --2- 172.21.15.44:0/176586281 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb48b60 0x56119eb55030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:02.582 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.574+0000 7ff15099a700 1 --2- 172.21.15.44:0/176586281 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb49470 0x56119eb522b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff13c0098e0 tx=0x7ff13c003ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c518474c6583576 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:02.582 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.574+0000 7ff134ff9700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:58:02.582 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.575+0000 7ff134ff9700 1 -- 172.21.15.44:0/176586281 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c005610 con 0x56119eb49470 2023-11-16T16:58:02.582 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.575+0000 7ff166d9db80 1 -- 172.21.15.44:0/176586281 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56119eb3bb20 con 0x56119eb49470 2023-11-16T16:58:02.582 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.575+0000 7ff134ff9700 1 -- 172.21.15.44:0/176586281 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7ff13c00b790 con 0x56119eb49470 2023-11-16T16:58:02.582 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.575+0000 7ff134ff9700 1 -- 172.21.15.44:0/176586281 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff13c00b990 con 0x56119eb49470 2023-11-16T16:58:02.582 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.575+0000 7ff166d9db80 10 client.26884 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:02.582 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.575+0000 7ff166d9db80 1 -- 172.21.15.44:0/176586281 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x56119ea60190 con 0x56119eb49470 2023-11-16T16:58:02.582 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.575+0000 7ff134ff9700 1 -- 172.21.15.44:0/176586281 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56119eb3bb20 con 0x56119eb49470 2023-11-16T16:58:02.582 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.575+0000 7ff134ff9700 1 -- 172.21.15.44:0/176586281 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff13c00fb50 con 0x56119eb49470 2023-11-16T16:58:02.582 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.575+0000 7ff166d9db80 10 client.26884 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:02.582 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.575+0000 7ff166d9db80 10 client.26884 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:02.583 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.576+0000 7ff166d9db80 20 client.26884 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:02.583 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.576+0000 7ff166d9db80 1 --2- 172.21.15.44:0/176586281 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56119eb62750 0x56119eb64b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.583 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.576+0000 7ff166d9db80 4 client.26884 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:58:02.583 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.576+0000 7ff166d9db80 1 -- 172.21.15.44:0/176586281 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56119ea5c0a0 con 0x56119eb62750 2023-11-16T16:58:02.583 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.576+0000 7ff15119b700 1 --2- 172.21.15.44:0/176586281 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56119eb62750 0x56119eb64b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.583 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.576+0000 7ff15119b700 1 --2- 172.21.15.44:0/176586281 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56119eb62750 0x56119eb64b60 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=755f6921406248d3 server_cookie=9f4a0b9eef5c27e5 in_seq=0 out_seq=0 2023-11-16T16:58:02.583 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.576+0000 7ff134ff9700 10 client.26884 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:02.583 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.577+0000 7ff134ff9700 1 -- 172.21.15.44:0/176586281 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56119ea5c0a0 con 0x56119eb62750 2023-11-16T16:58:02.583 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.577+0000 7ff134ff9700 10 client.26884 handle_command_reply: tid=0 2023-11-16T16:58:02.583 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.577+0000 7ff166d9db80 1 client.26884 shutdown 2023-11-16T16:58:02.583 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.577+0000 7ff166d9db80 1 -- 172.21.15.44:0/176586281 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb49470 msgr2=0x56119eb522b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:02.583 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.577+0000 7ff166d9db80 1 --2- 172.21.15.44:0/176586281 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb49470 0x56119eb522b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff13c0098e0 tx=0x7ff13c003ed0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.583 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.578+0000 7ff166d9db80 1 -- 172.21.15.44:0/176586281 shutdown_connections 2023-11-16T16:58:02.584 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.578+0000 7ff166d9db80 1 --2- 172.21.15.44:0/176586281 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56119eb62750 0x56119eb64b60 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.584 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.578+0000 7ff166d9db80 1 --2- 172.21.15.44:0/176586281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb4d8b0 0x56119eb52890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.584 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.578+0000 7ff166d9db80 1 --2- 172.21.15.44:0/176586281 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb48b60 0x56119eb55030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.584 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.578+0000 7ff166d9db80 1 --2- 172.21.15.44:0/176586281 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb49470 0x56119eb522b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.584 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.578+0000 7ff134ff9700 0 client.26884 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:02.584 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.578+0000 7ff166d9db80 1 -- 172.21.15.44:0/176586281 >> 172.21.15.44:0/176586281 conn(0x56119eb3e360 msgr2=0x56119eb4ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:02.584 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.578+0000 7ff166d9db80 1 -- 172.21.15.44:0/176586281 shutdown_connections 2023-11-16T16:58:02.584 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.578+0000 7ff166d9db80 1 -- 172.21.15.44:0/176586281 wait complete. 2023-11-16T16:58:02.584 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.578+0000 7ff166d9db80 20 client.26884 trim_cache size 0 max 16384 2023-11-16T16:58:02.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.591+0000 7ff166d9db80 1 Processor -- start 2023-11-16T16:58:02.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.591+0000 7ff166d9db80 1 -- start start 2023-11-16T16:58:02.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.592+0000 7ff166d9db80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb48b60 0x56119eb599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.592+0000 7ff166d9db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb49470 0x56119eb5eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.592+0000 7ff166d9db80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb4d8b0 0x56119eb54bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.592+0000 7ff166d9db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56119eb3ba00 con 0x56119eb49470 2023-11-16T16:58:02.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.592+0000 7ff166d9db80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56119eb4c420 con 0x56119eb48b60 2023-11-16T16:58:02.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.592+0000 7ff166d9db80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56119eb44800 con 0x56119eb4d8b0 2023-11-16T16:58:02.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.592+0000 7ff15119b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb48b60 0x56119eb599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.592+0000 7ff15119b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb48b60 0x56119eb599a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:49574/0 (socket says 172.21.15.44:49574) 2023-11-16T16:58:02.594 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.592+0000 7ff15119b700 1 -- 172.21.15.44:0/2039636540 learned_addr learned my addr 172.21.15.44:0/2039636540 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:02.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.592+0000 7ff15199c700 1 --2- 172.21.15.44:0/2039636540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb4d8b0 0x56119eb54bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.592+0000 7ff15099a700 1 --2- 172.21.15.44:0/2039636540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb49470 0x56119eb5eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.592+0000 7ff15119b700 1 -- 172.21.15.44:0/2039636540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb4d8b0 msgr2=0x56119eb54bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:02.595 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.592+0000 7ff15119b700 1 --2- 172.21.15.44:0/2039636540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb4d8b0 0x56119eb54bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.596 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.592+0000 7ff15119b700 1 -- 172.21.15.44:0/2039636540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb49470 msgr2=0x56119eb5eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:02.596 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.592+0000 7ff15119b700 1 --2- 172.21.15.44:0/2039636540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb49470 0x56119eb5eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.596 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.593+0000 7ff15119b700 1 -- 172.21.15.44:0/2039636540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56119eb45490 con 0x56119eb48b60 2023-11-16T16:58:02.596 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.593+0000 7ff15119b700 1 --2- 172.21.15.44:0/2039636540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb48b60 0x56119eb599a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff1480982f0 tx=0x7ff1480092d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=843eb634515f9a39 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:02.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.594+0000 7ff137fff700 1 -- 172.21.15.44:0/2039636540 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff148010880 con 0x56119eb48b60 2023-11-16T16:58:02.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.594+0000 7ff137fff700 1 -- 172.21.15.44:0/2039636540 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff14809a200 con 0x56119eb48b60 2023-11-16T16:58:02.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.594+0000 7ff137fff700 1 -- 172.21.15.44:0/2039636540 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff148099160 con 0x56119eb48b60 2023-11-16T16:58:02.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.595+0000 7ff166d9db80 1 -- 172.21.15.44:0/2039636540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb48b60 msgr2=0x56119eb599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:02.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.595+0000 7ff166d9db80 1 --2- 172.21.15.44:0/2039636540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb48b60 0x56119eb599a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff1480982f0 tx=0x7ff1480092d0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.595+0000 7ff166d9db80 1 -- 172.21.15.44:0/2039636540 shutdown_connections 2023-11-16T16:58:02.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.595+0000 7ff166d9db80 1 --2- 172.21.15.44:0/2039636540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb4d8b0 0x56119eb54bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.595+0000 7ff166d9db80 1 --2- 172.21.15.44:0/2039636540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb48b60 0x56119eb599a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.595+0000 7ff166d9db80 1 --2- 172.21.15.44:0/2039636540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb49470 0x56119eb5eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.597 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.595+0000 7ff166d9db80 1 -- 172.21.15.44:0/2039636540 >> 172.21.15.44:0/2039636540 conn(0x56119eb3e360 msgr2=0x56119eb42550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:02.598 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.595+0000 7ff166d9db80 1 -- 172.21.15.44:0/2039636540 shutdown_connections 2023-11-16T16:58:02.598 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.595+0000 7ff166d9db80 1 -- 172.21.15.44:0/2039636540 wait complete. 2023-11-16T16:58:02.600 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.596+0000 7ff166d9db80 1 -- start start 2023-11-16T16:58:02.600 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.596+0000 7ff166d9db80 1 Processor -- start 2023-11-16T16:58:02.600 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.596+0000 7ff166d9db80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb48b60 0x56119eb57990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.600 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.596+0000 7ff166d9db80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb49470 0x56119eb57ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.600 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.596+0000 7ff166d9db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb4d8b0 0x56119eb58410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.600 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.596+0000 7ff166d9db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56119ebb5720 con 0x56119eb4d8b0 2023-11-16T16:58:02.600 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.596+0000 7ff166d9db80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56119ebb5890 con 0x56119eb49470 2023-11-16T16:58:02.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.596+0000 7ff166d9db80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x56119ebb5a50 con 0x56119eb48b60 2023-11-16T16:58:02.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.596+0000 7ff166d9db80 10 client.0 _ll_register_callbacks cb 0x7ff13c003ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:02.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.597+0000 7ff15119b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb48b60 0x56119eb57990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.597+0000 7ff15119b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb48b60 0x56119eb57990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:40254/0 (socket says 172.21.15.44:40254) 2023-11-16T16:58:02.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.597+0000 7ff15119b700 1 -- 172.21.15.44:0/931644657 learned_addr learned my addr 172.21.15.44:0/931644657 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:02.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.597+0000 7ff15099a700 1 --2- 172.21.15.44:0/931644657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb49470 0x56119eb57ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.597+0000 7ff15199c700 1 --2- 172.21.15.44:0/931644657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb4d8b0 0x56119eb58410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.597+0000 7ff15119b700 1 -- 172.21.15.44:0/931644657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb49470 msgr2=0x56119eb57ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:02.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.597+0000 7ff15119b700 1 --2- 172.21.15.44:0/931644657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb49470 0x56119eb57ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.597+0000 7ff15119b700 1 -- 172.21.15.44:0/931644657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb4d8b0 msgr2=0x56119eb58410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:02.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.597+0000 7ff15119b700 1 --2- 172.21.15.44:0/931644657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb4d8b0 0x56119eb58410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.601 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.597+0000 7ff15119b700 1 -- 172.21.15.44:0/931644657 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56119e91b630 con 0x56119eb48b60 2023-11-16T16:58:02.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.597+0000 7ff15099a700 1 --2- 172.21.15.44:0/931644657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb49470 0x56119eb57ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:02.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.597+0000 7ff15119b700 1 --2- 172.21.15.44:0/931644657 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb48b60 0x56119eb57990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff148096a10 tx=0x7ff14800c4c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac6b6b2ccee7c00f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:02.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.597+0000 7ff134ff9700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:58:02.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.598+0000 7ff134ff9700 1 -- 172.21.15.44:0/931644657 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff148010880 con 0x56119eb48b60 2023-11-16T16:58:02.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.598+0000 7ff134ff9700 1 -- 172.21.15.44:0/931644657 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7ff1480a1f60 con 0x56119eb48b60 2023-11-16T16:58:02.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.598+0000 7ff166d9db80 1 -- 172.21.15.44:0/931644657 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff1480a23f0 con 0x56119eb48b60 2023-11-16T16:58:02.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.598+0000 7ff134ff9700 1 -- 172.21.15.44:0/931644657 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1480109e0 con 0x56119eb48b60 2023-11-16T16:58:02.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.598+0000 7ff166d9db80 10 client.26890 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:02.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.598+0000 7ff166d9db80 1 -- 172.21.15.44:0/931644657 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x56119e8c7de0 con 0x56119eb48b60 2023-11-16T16:58:02.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.598+0000 7ff134ff9700 1 -- 172.21.15.44:0/931644657 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1480a23f0 con 0x56119eb48b60 2023-11-16T16:58:02.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.598+0000 7ff134ff9700 1 -- 172.21.15.44:0/931644657 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff148098f70 con 0x56119eb48b60 2023-11-16T16:58:02.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.598+0000 7ff166d9db80 10 client.26890 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:02.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.598+0000 7ff166d9db80 10 client.26890 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:02.602 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.598+0000 7ff166d9db80 20 client.26890 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:02.603 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.598+0000 7ff166d9db80 1 --2- 172.21.15.44:0/931644657 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56119ebb6c60 0x56119ebb9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:02.603 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.598+0000 7ff166d9db80 4 client.26890 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T16:58:02.603 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.598+0000 7ff166d9db80 1 -- 172.21.15.44:0/931644657 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56119ea68660 con 0x56119ebb6c60 2023-11-16T16:58:02.603 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.599+0000 7ff15099a700 1 --2- 172.21.15.44:0/931644657 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56119ebb6c60 0x56119ebb9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:02.604 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.599+0000 7ff15099a700 1 --2- 172.21.15.44:0/931644657 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56119ebb6c60 0x56119ebb9040 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d37ede8b0ecc5e server_cookie=ff1aa667e0df29df in_seq=0 out_seq=0 2023-11-16T16:58:02.604 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.599+0000 7ff134ff9700 10 client.26890 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:02.604 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.599+0000 7ff134ff9700 1 -- 172.21.15.44:0/931644657 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56119ea68660 con 0x56119ebb6c60 2023-11-16T16:58:02.604 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.599+0000 7ff134ff9700 10 client.26890 handle_command_reply: tid=0 2023-11-16T16:58:02.604 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.599+0000 7ff166d9db80 1 client.26890 shutdown 2023-11-16T16:58:02.604 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.599+0000 7ff166d9db80 1 -- 172.21.15.44:0/931644657 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb48b60 msgr2=0x56119eb57990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:02.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.599+0000 7ff166d9db80 1 --2- 172.21.15.44:0/931644657 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb48b60 0x56119eb57990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff148096a10 tx=0x7ff14800c4c0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.600+0000 7ff166d9db80 1 -- 172.21.15.44:0/931644657 shutdown_connections 2023-11-16T16:58:02.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.600+0000 7ff166d9db80 1 --2- 172.21.15.44:0/931644657 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56119ebb6c60 0x56119ebb9040 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.600+0000 7ff166d9db80 1 --2- 172.21.15.44:0/931644657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56119eb4d8b0 0x56119eb58410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.600+0000 7ff166d9db80 1 --2- 172.21.15.44:0/931644657 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56119eb48b60 0x56119eb57990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.600+0000 7ff134ff9700 0 client.26890 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:02.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.600+0000 7ff166d9db80 1 --2- 172.21.15.44:0/931644657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56119eb49470 0x56119eb57ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.600+0000 7ff166d9db80 1 -- 172.21.15.44:0/931644657 >> 172.21.15.44:0/931644657 conn(0x56119eb3e360 msgr2=0x56119eb4b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:02.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.600+0000 7ff166d9db80 1 -- 172.21.15.44:0/931644657 shutdown_connections 2023-11-16T16:58:02.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.600+0000 7ff166d9db80 1 -- 172.21.15.44:0/931644657 wait complete. 2023-11-16T16:58:02.605 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.600+0000 7ff166d9db80 20 client.26890 trim_cache size 0 max 16384 2023-11-16T16:58:02.606 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T16:58:02.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.600+0000 7ff1533ff700 1 -- 172.21.15.44:0/2923964688 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff1380a12c0 msgr2=0x7ff1380a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:02.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.600+0000 7ff1533ff700 1 --2- 172.21.15.44:0/2923964688 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff1380a12c0 0x7ff1380a3770 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff144008ea0 tx=0x7ff14400b040 comp rx=0 tx=0).stop 2023-11-16T16:58:02.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.600+0000 7ff1533ff700 1 -- 172.21.15.44:0/2923964688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff14c100cc0 msgr2=0x7ff14c072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:02.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.600+0000 7ff1533ff700 1 --2- 172.21.15.44:0/2923964688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff14c100cc0 0x7ff14c072df0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff14800e9a0 tx=0x7ff14800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.606 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.601+0000 7ff1533ff700 1 -- 172.21.15.44:0/2923964688 shutdown_connections 2023-11-16T16:58:02.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.601+0000 7ff1533ff700 1 --2- 172.21.15.44:0/2923964688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff14c105620 0x7ff14c1a4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.601+0000 7ff1533ff700 1 --2- 172.21.15.44:0/2923964688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff14c100cc0 0x7ff14c072df0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.601+0000 7ff1533ff700 1 --2- 172.21.15.44:0/2923964688 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff1380a12c0 0x7ff1380a3770 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.601+0000 7ff1533ff700 1 --2- 172.21.15.44:0/2923964688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff14c101660 0x7ff14c073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:02.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.601+0000 7ff1533ff700 1 -- 172.21.15.44:0/2923964688 >> 172.21.15.44:0/2923964688 conn(0x7ff14c077560 msgr2=0x7ff14c0ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:02.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.601+0000 7ff1533ff700 1 -- 172.21.15.44:0/2923964688 shutdown_connections 2023-11-16T16:58:02.607 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:02.601+0000 7ff1533ff700 1 -- 172.21.15.44:0/2923964688 wait complete. 2023-11-16T16:58:02.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:02 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3738674150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:58:02.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:02 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3738674150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:58:02.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:02 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3738674150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:58:03.621 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:58:03.621 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:58:03.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.838+0000 7f6531dff700 1 -- 172.21.15.44:0/3493707407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f652c105620 msgr2=0x7f652c10d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:03.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.838+0000 7f6531dff700 1 --2- 172.21.15.44:0/3493707407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f652c105620 0x7f652c10d1a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f652400b0d0 tx=0x7f652400b3e0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.839+0000 7f6531dff700 1 -- 172.21.15.44:0/3493707407 shutdown_connections 2023-11-16T16:58:03.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.839+0000 7f6531dff700 1 --2- 172.21.15.44:0/3493707407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f652c105620 0x7f652c10d1a0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.839+0000 7f6531dff700 1 --2- 172.21.15.44:0/3493707407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f652c100cc0 0x7f652c101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.839+0000 7f6531dff700 1 --2- 172.21.15.44:0/3493707407 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f652c101660 0x7f652c104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.839+0000 7f6531dff700 1 -- 172.21.15.44:0/3493707407 >> 172.21.15.44:0/3493707407 conn(0x7f652c077560 msgr2=0x7f652c077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:03.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.839+0000 7f6531dff700 1 -- 172.21.15.44:0/3493707407 shutdown_connections 2023-11-16T16:58:03.842 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.839+0000 7f6531dff700 1 -- 172.21.15.44:0/3493707407 wait complete. 2023-11-16T16:58:03.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.840+0000 7f6531dff700 1 Processor -- start 2023-11-16T16:58:03.844 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.840+0000 7f6531dff700 1 -- start start 2023-11-16T16:58:03.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.840+0000 7f6531dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f652c100cc0 0x7f652c19e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.840+0000 7f6531dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f652c101660 0x7f652c19eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.840+0000 7f6531dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f652c105620 0x7f652c198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.840+0000 7f6531dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652c198c90 con 0x7f652c101660 2023-11-16T16:58:03.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.840+0000 7f6531dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652c198e00 con 0x7f652c100cc0 2023-11-16T16:58:03.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.840+0000 7f6531dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652c198fc0 con 0x7f652c105620 2023-11-16T16:58:03.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.841+0000 7f652affd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f652c101660 0x7f652c19eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.841+0000 7f652b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f652c100cc0 0x7f652c19e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.841+0000 7f652affd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f652c101660 0x7f652c19eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51338/0 (socket says 172.21.15.44:51338) 2023-11-16T16:58:03.845 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.841+0000 7f652affd700 1 -- 172.21.15.44:0/3408763621 learned_addr learned my addr 172.21.15.44:0/3408763621 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.841+0000 7f652bfff700 1 --2- 172.21.15.44:0/3408763621 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f652c105620 0x7f652c198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.841+0000 7f652affd700 1 -- 172.21.15.44:0/3408763621 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f652c105620 msgr2=0x7f652c198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.841+0000 7f652affd700 1 --2- 172.21.15.44:0/3408763621 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f652c105620 0x7f652c198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.841+0000 7f652affd700 1 -- 172.21.15.44:0/3408763621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f652c100cc0 msgr2=0x7f652c19e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.841+0000 7f652affd700 1 --2- 172.21.15.44:0/3408763621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f652c100cc0 0x7f652c19e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.841+0000 7f652affd700 1 -- 172.21.15.44:0/3408763621 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6524009d00 con 0x7f652c101660 2023-11-16T16:58:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.841+0000 7f652b7fe700 1 --2- 172.21.15.44:0/3408763621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f652c100cc0 0x7f652c19e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.841+0000 7f652affd700 1 --2- 172.21.15.44:0/3408763621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f652c101660 0x7f652c19eaf0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f652000d980 tx=0x7f652000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:03.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.842+0000 7f6528ff9700 1 -- 172.21.15.44:0/3408763621 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652000f890 con 0x7f652c101660 2023-11-16T16:58:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.842+0000 7f6531dff700 1 -- 172.21.15.44:0/3408763621 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f652c1103e0 con 0x7f652c101660 2023-11-16T16:58:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.842+0000 7f6528ff9700 1 -- 172.21.15.44:0/3408763621 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6520010460 con 0x7f652c101660 2023-11-16T16:58:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.842+0000 7f6528ff9700 1 -- 172.21.15.44:0/3408763621 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652000e610 con 0x7f652c101660 2023-11-16T16:58:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.842+0000 7f6531dff700 1 -- 172.21.15.44:0/3408763621 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f652c110250 con 0x7f652c101660 2023-11-16T16:58:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.845+0000 7f6528ff9700 1 -- 172.21.15.44:0/3408763621 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f65200105d0 con 0x7f652c101660 2023-11-16T16:58:03.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.846+0000 7f65456a3b80 1 Processor -- start 2023-11-16T16:58:03.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.846+0000 7f6528ff9700 1 --2- 172.21.15.44:0/3408763621 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f65140a1340 0x7f65140a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.850 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.846+0000 7f6528ff9700 1 -- 172.21.15.44:0/3408763621 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f65200ccb00 con 0x7f652c101660 2023-11-16T16:58:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.846+0000 7f65456a3b80 1 -- start start 2023-11-16T16:58:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.847+0000 7f652bfff700 1 --2- 172.21.15.44:0/3408763621 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f65140a1340 0x7f65140a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.847+0000 7f65456a3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1a1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.847+0000 7f65456a3b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a24f0 0x559afb1a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.847+0000 7f65456a3b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a6930 0x559afb1a6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.847+0000 7f65456a3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559afb0a70e0 con 0x559afb1a1be0 2023-11-16T16:58:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.847+0000 7f65456a3b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559afb0f9920 con 0x559afb1a6930 2023-11-16T16:58:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.847+0000 7f65456a3b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559afb0f2db0 con 0x559afb1a24f0 2023-11-16T16:58:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.847+0000 7f652b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1a1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.847+0000 7f652bfff700 1 --2- 172.21.15.44:0/3408763621 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f65140a1340 0x7f65140a37f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f652c19f2b0 tx=0x7f652400b040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.847+0000 7f652b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1a1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51346/0 (socket says 172.21.15.44:51346) 2023-11-16T16:58:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.847+0000 7f652b7fe700 1 -- 172.21.15.44:0/630033914 learned_addr learned my addr 172.21.15.44:0/630033914 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.847+0000 7f652bfff700 1 --2- 172.21.15.44:0/630033914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a6930 0x559afb1a6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.847+0000 7f652affd700 1 --2- 172.21.15.44:0/630033914 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a24f0 0x559afb1a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.848+0000 7f652b7fe700 1 -- 172.21.15.44:0/630033914 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a24f0 msgr2=0x559afb1a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.848+0000 7f652b7fe700 1 --2- 172.21.15.44:0/630033914 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a24f0 0x559afb1a63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.848+0000 7f652b7fe700 1 -- 172.21.15.44:0/630033914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a6930 msgr2=0x559afb1a6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.848+0000 7f652b7fe700 1 --2- 172.21.15.44:0/630033914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a6930 0x559afb1a6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.848+0000 7f652b7fe700 1 -- 172.21.15.44:0/630033914 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559afaf746b0 con 0x559afb1a1be0 2023-11-16T16:58:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.848+0000 7f652b7fe700 1 --2- 172.21.15.44:0/630033914 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1a1fb0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f651c0115f0 tx=0x7f651c011900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c402c72e16f640 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:03.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.848+0000 7f6511ffb700 1 -- 172.21.15.44:0/630033914 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651c019630 con 0x559afb1a1be0 2023-11-16T16:58:03.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.848+0000 7f6511ffb700 1 -- 172.21.15.44:0/630033914 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f651c019c50 con 0x559afb1a1be0 2023-11-16T16:58:03.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.848+0000 7f6511ffb700 1 -- 172.21.15.44:0/630033914 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651c00eb70 con 0x559afb1a1be0 2023-11-16T16:58:03.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.849+0000 7f65456a3b80 1 -- 172.21.15.44:0/630033914 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 msgr2=0x559afb1a1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:03.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.849+0000 7f65456a3b80 1 --2- 172.21.15.44:0/630033914 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1a1fb0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f651c0115f0 tx=0x7f651c011900 comp rx=0 tx=0).stop 2023-11-16T16:58:03.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.849+0000 7f65456a3b80 1 -- 172.21.15.44:0/630033914 shutdown_connections 2023-11-16T16:58:03.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.849+0000 7f65456a3b80 1 --2- 172.21.15.44:0/630033914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a6930 0x559afb1a6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.849+0000 7f65456a3b80 1 --2- 172.21.15.44:0/630033914 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1a1fb0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.849+0000 7f65456a3b80 1 --2- 172.21.15.44:0/630033914 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a24f0 0x559afb1a63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.849+0000 7f65456a3b80 1 -- 172.21.15.44:0/630033914 >> 172.21.15.44:0/630033914 conn(0x559afb1973e0 msgr2=0x559afb19cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:03.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.849+0000 7f65456a3b80 1 -- 172.21.15.44:0/630033914 shutdown_connections 2023-11-16T16:58:03.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.849+0000 7f65456a3b80 1 -- 172.21.15.44:0/630033914 wait complete. 2023-11-16T16:58:03.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.850+0000 7f65456a3b80 1 -- start start 2023-11-16T16:58:03.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.850+0000 7f65456a3b80 1 Processor -- start 2023-11-16T16:58:03.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.850+0000 7f65456a3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.850+0000 7f65456a3b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a24f0 0x559afb1ab2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.850+0000 7f65456a3b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a6930 0x559afb1ab920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.850+0000 7f65456a3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559afb094960 con 0x559afb1a1be0 2023-11-16T16:58:03.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.850+0000 7f65456a3b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559afb094600 con 0x559afb1a6930 2023-11-16T16:58:03.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.850+0000 7f65456a3b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559afaf769a0 con 0x559afb1a24f0 2023-11-16T16:58:03.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.851+0000 7f65456a3b80 10 client.0 _ll_register_callbacks cb 0x559afb0b2800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:03.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.851+0000 7f652b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.851+0000 7f652bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a6930 0x559afb1ab920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.851+0000 7f652b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1ae0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51350/0 (socket says 172.21.15.44:51350) 2023-11-16T16:58:03.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.851+0000 7f652b7fe700 1 -- 172.21.15.44:0/151520458 learned_addr learned my addr 172.21.15.44:0/151520458 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:03.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.851+0000 7f652affd700 1 --2- 172.21.15.44:0/151520458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a24f0 0x559afb1ab2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.851+0000 7f652b7fe700 1 -- 172.21.15.44:0/151520458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a24f0 msgr2=0x559afb1ab2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:03.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.851+0000 7f652b7fe700 1 --2- 172.21.15.44:0/151520458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a24f0 0x559afb1ab2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.851+0000 7f652b7fe700 1 -- 172.21.15.44:0/151520458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a6930 msgr2=0x559afb1ab920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:03.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.851+0000 7f652b7fe700 1 --2- 172.21.15.44:0/151520458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a6930 0x559afb1ab920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.851+0000 7f652b7fe700 1 -- 172.21.15.44:0/151520458 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f651c009dd0 con 0x559afb1a1be0 2023-11-16T16:58:03.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.851+0000 7f652b7fe700 1 --2- 172.21.15.44:0/151520458 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1ae0e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f651c000c00 tx=0x7f651c00e450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed9a6520b0815b6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:03.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.852+0000 7f650affd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:58:03.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.852+0000 7f650affd700 1 -- 172.21.15.44:0/151520458 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651c00ec00 con 0x559afb1a1be0 2023-11-16T16:58:03.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.852+0000 7f650affd700 1 -- 172.21.15.44:0/151520458 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f651c00ed60 con 0x559afb1a1be0 2023-11-16T16:58:03.861 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.852+0000 7f65456a3b80 1 -- 172.21.15.44:0/151520458 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559afb194ba0 con 0x559afb1a1be0 2023-11-16T16:58:03.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.852+0000 7f650affd700 1 -- 172.21.15.44:0/151520458 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f651c019430 con 0x559afb1a1be0 2023-11-16T16:58:03.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.852+0000 7f65456a3b80 10 client.16872 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:03.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.852+0000 7f65456a3b80 1 -- 172.21.15.44:0/151520458 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x559afb0b9210 con 0x559afb1a1be0 2023-11-16T16:58:03.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.852+0000 7f650affd700 1 -- 172.21.15.44:0/151520458 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559afb194ba0 con 0x559afb1a1be0 2023-11-16T16:58:03.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.853+0000 7f650affd700 1 -- 172.21.15.44:0/151520458 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f651c003b60 con 0x559afb1a1be0 2023-11-16T16:58:03.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.853+0000 7f65456a3b80 10 client.16872 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:03.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.853+0000 7f65456a3b80 10 client.16872 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:03.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.853+0000 7f65456a3b80 20 client.16872 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:03.862 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.853+0000 7f65456a3b80 1 --2- 172.21.15.44:0/151520458 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559afb1bb770 0x559afb1bdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.853+0000 7f65456a3b80 4 client.16872 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:58:03.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.853+0000 7f65456a3b80 1 -- 172.21.15.44:0/151520458 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559afb0b5120 con 0x559afb1bb770 2023-11-16T16:58:03.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.853+0000 7f652affd700 1 --2- 172.21.15.44:0/151520458 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559afb1bb770 0x559afb1bdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.854+0000 7f652affd700 1 --2- 172.21.15.44:0/151520458 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559afb1bb770 0x559afb1bdb80 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe336d7ccbe87404 server_cookie=2e713239d96ce5b4 in_seq=0 out_seq=0 2023-11-16T16:58:03.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.854+0000 7f650affd700 10 client.16872 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:03.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.854+0000 7f650affd700 1 -- 172.21.15.44:0/151520458 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559afb0b5120 con 0x559afb1bb770 2023-11-16T16:58:03.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.854+0000 7f650affd700 10 client.16872 handle_command_reply: tid=0 2023-11-16T16:58:03.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.854+0000 7f65456a3b80 1 client.16872 shutdown 2023-11-16T16:58:03.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.855+0000 7f65456a3b80 1 -- 172.21.15.44:0/151520458 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 msgr2=0x559afb1ae0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:03.863 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.855+0000 7f65456a3b80 1 --2- 172.21.15.44:0/151520458 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1ae0e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f651c000c00 tx=0x7f651c00e450 comp rx=0 tx=0).stop 2023-11-16T16:58:03.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.855+0000 7f65456a3b80 1 -- 172.21.15.44:0/151520458 shutdown_connections 2023-11-16T16:58:03.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.855+0000 7f65456a3b80 1 --2- 172.21.15.44:0/151520458 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559afb1bb770 0x559afb1bdb80 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.855+0000 7f65456a3b80 1 --2- 172.21.15.44:0/151520458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a6930 0x559afb1ab920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.855+0000 7f65456a3b80 1 --2- 172.21.15.44:0/151520458 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1ae0e0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.855+0000 7f65456a3b80 1 --2- 172.21.15.44:0/151520458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a24f0 0x559afb1ab2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.855+0000 7f650affd700 0 client.16872 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:03.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.855+0000 7f65456a3b80 1 -- 172.21.15.44:0/151520458 >> 172.21.15.44:0/151520458 conn(0x559afb1973e0 msgr2=0x559afb1a7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:03.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.855+0000 7f65456a3b80 1 -- 172.21.15.44:0/151520458 shutdown_connections 2023-11-16T16:58:03.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.855+0000 7f65456a3b80 1 -- 172.21.15.44:0/151520458 wait complete. 2023-11-16T16:58:03.864 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.855+0000 7f65456a3b80 20 client.16872 trim_cache size 0 max 16384 2023-11-16T16:58:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:03 smithi044 ceph-mon[128750]: pgmap v250: 129 pgs: 129 active+clean; 5.8 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 110 KiB/s wr, 287 op/s 2023-11-16T16:58:03.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.873+0000 7f65456a3b80 1 Processor -- start 2023-11-16T16:58:03.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.873+0000 7f65456a3b80 1 -- start start 2023-11-16T16:58:03.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.873+0000 7f65456a3b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a1be0 0x559afb1ab2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.873+0000 7f65456a3b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a24f0 0x559afb1b2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.873+0000 7f65456a3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a6930 0x559afb1adc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.873+0000 7f65456a3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559afb1a5280 con 0x559afb1a6930 2023-11-16T16:58:03.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.873+0000 7f65456a3b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559afb1a9850 con 0x559afb1a24f0 2023-11-16T16:58:03.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.873+0000 7f65456a3b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559afb1b82e0 con 0x559afb1a1be0 2023-11-16T16:58:03.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.874+0000 7f652bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a6930 0x559afb1adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.874+0000 7f652bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a6930 0x559afb1adc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51366/0 (socket says 172.21.15.44:51366) 2023-11-16T16:58:03.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.874+0000 7f652b7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a1be0 0x559afb1ab2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.877 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.874+0000 7f652bfff700 1 -- 172.21.15.44:0/617013679 learned_addr learned my addr 172.21.15.44:0/617013679 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:03.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.874+0000 7f652affd700 1 --2- 172.21.15.44:0/617013679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a24f0 0x559afb1b2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.874+0000 7f652b7fe700 1 -- 172.21.15.44:0/617013679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a24f0 msgr2=0x559afb1b2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:03.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.874+0000 7f652b7fe700 1 --2- 172.21.15.44:0/617013679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a24f0 0x559afb1b2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.874+0000 7f652b7fe700 1 -- 172.21.15.44:0/617013679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a6930 msgr2=0x559afb1adc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:03.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.874+0000 7f652b7fe700 1 --2- 172.21.15.44:0/617013679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a6930 0x559afb1adc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.874+0000 7f652b7fe700 1 -- 172.21.15.44:0/617013679 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559afb0b8f60 con 0x559afb1a1be0 2023-11-16T16:58:03.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.874+0000 7f652bfff700 1 --2- 172.21.15.44:0/617013679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a6930 0x559afb1adc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:03.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.874+0000 7f652b7fe700 1 --2- 172.21.15.44:0/617013679 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a1be0 0x559afb1ab2a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f651c011510 tx=0x7f651c005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff72bbc78d4577fa server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:03.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.875+0000 7f65127fc700 1 -- 172.21.15.44:0/617013679 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651c022550 con 0x559afb1a1be0 2023-11-16T16:58:03.878 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.875+0000 7f65127fc700 1 -- 172.21.15.44:0/617013679 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f651c00eec0 con 0x559afb1a1be0 2023-11-16T16:58:03.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.875+0000 7f65127fc700 1 -- 172.21.15.44:0/617013679 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651c019ae0 con 0x559afb1a1be0 2023-11-16T16:58:03.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.875+0000 7f65456a3b80 1 -- 172.21.15.44:0/617013679 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a1be0 msgr2=0x559afb1ab2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:03.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.875+0000 7f65456a3b80 1 --2- 172.21.15.44:0/617013679 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a1be0 0x559afb1ab2a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f651c011510 tx=0x7f651c005e50 comp rx=0 tx=0).stop 2023-11-16T16:58:03.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.875+0000 7f65456a3b80 1 -- 172.21.15.44:0/617013679 shutdown_connections 2023-11-16T16:58:03.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.875+0000 7f65456a3b80 1 --2- 172.21.15.44:0/617013679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a6930 0x559afb1adc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.875+0000 7f65456a3b80 1 --2- 172.21.15.44:0/617013679 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a1be0 0x559afb1ab2a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.875+0000 7f65456a3b80 1 --2- 172.21.15.44:0/617013679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a24f0 0x559afb1b2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.875+0000 7f65456a3b80 1 -- 172.21.15.44:0/617013679 >> 172.21.15.44:0/617013679 conn(0x559afb1973e0 msgr2=0x559afb1aab90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.875+0000 7f65456a3b80 1 -- 172.21.15.44:0/617013679 shutdown_connections 2023-11-16T16:58:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.875+0000 7f65456a3b80 1 -- 172.21.15.44:0/617013679 wait complete. 2023-11-16T16:58:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.876+0000 7f65456a3b80 1 -- start start 2023-11-16T16:58:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.876+0000 7f65456a3b80 1 Processor -- start 2023-11-16T16:58:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.876+0000 7f65456a3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1b1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f65456a3b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a24f0 0x559afb1b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f65456a3b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a6930 0x559afb20f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f65456a3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x559afb1b2040 con 0x559afb1a1be0 2023-11-16T16:58:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f65456a3b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559afb1b21b0 con 0x559afb1a24f0 2023-11-16T16:58:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f65456a3b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x559afb20fb80 con 0x559afb1a6930 2023-11-16T16:58:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f65456a3b80 10 client.0 _ll_register_callbacks cb 0x7f6524020070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:03.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f652b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1b1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f652b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1b1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51380/0 (socket says 172.21.15.44:51380) 2023-11-16T16:58:03.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f652b7fe700 1 -- 172.21.15.44:0/3752958849 learned_addr learned my addr 172.21.15.44:0/3752958849 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:03.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f652affd700 1 --2- 172.21.15.44:0/3752958849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a24f0 0x559afb1b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f652bfff700 1 --2- 172.21.15.44:0/3752958849 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a6930 0x559afb20f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f652b7fe700 1 -- 172.21.15.44:0/3752958849 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a6930 msgr2=0x559afb20f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:03.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f652b7fe700 1 --2- 172.21.15.44:0/3752958849 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a6930 0x559afb20f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f652b7fe700 1 -- 172.21.15.44:0/3752958849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a24f0 msgr2=0x559afb1b18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:03.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f652b7fe700 1 --2- 172.21.15.44:0/3752958849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a24f0 0x559afb1b18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f652b7fe700 1 -- 172.21.15.44:0/3752958849 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f652c10d9b0 con 0x559afb1a1be0 2023-11-16T16:58:03.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f652bfff700 1 --2- 172.21.15.44:0/3752958849 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a6930 0x559afb20f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:58:03.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f652b7fe700 1 --2- 172.21.15.44:0/3752958849 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1b1370 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f651c003f20 tx=0x7f651c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4f7e9df0c6db02 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:03.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.877+0000 7f650affd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:58:03.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.878+0000 7f650affd700 1 -- 172.21.15.44:0/3752958849 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651c00ec10 con 0x559afb1a1be0 2023-11-16T16:58:03.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.878+0000 7f650affd700 1 -- 172.21.15.44:0/3752958849 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f651c000e40 con 0x559afb1a1be0 2023-11-16T16:58:03.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.878+0000 7f65456a3b80 1 -- 172.21.15.44:0/3752958849 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559afb0c16e0 con 0x559afb1a1be0 2023-11-16T16:58:03.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.878+0000 7f650affd700 1 -- 172.21.15.44:0/3752958849 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f651c00ed70 con 0x559afb1a1be0 2023-11-16T16:58:03.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.878+0000 7f65456a3b80 10 client.16878 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:03.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.878+0000 7f65456a3b80 1 -- 172.21.15.44:0/3752958849 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x559afb0b9210 con 0x559afb1a1be0 2023-11-16T16:58:03.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.878+0000 7f650affd700 1 -- 172.21.15.44:0/3752958849 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559afb0c16e0 con 0x559afb1a1be0 2023-11-16T16:58:03.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.878+0000 7f650affd700 1 -- 172.21.15.44:0/3752958849 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f651c0039a0 con 0x559afb1a1be0 2023-11-16T16:58:03.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.879+0000 7f65456a3b80 10 client.16878 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:03.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.879+0000 7f65456a3b80 10 client.16878 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:03.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.879+0000 7f65456a3b80 20 client.16878 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:03.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.879+0000 7f65456a3b80 1 --2- 172.21.15.44:0/3752958849 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559afb210e60 0x559afb213270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:03.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.879+0000 7f65456a3b80 4 client.16878 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:58:03.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.879+0000 7f65456a3b80 1 -- 172.21.15.44:0/3752958849 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x559afaf41ff0 con 0x559afb210e60 2023-11-16T16:58:03.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.879+0000 7f652affd700 1 --2- 172.21.15.44:0/3752958849 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559afb210e60 0x559afb213270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:03.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.879+0000 7f650affd700 10 client.16878 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:03.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.879+0000 7f652affd700 1 --2- 172.21.15.44:0/3752958849 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559afb210e60 0x559afb213270 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cc21148def21bb9 server_cookie=56f7c2396c02f0ed in_seq=0 out_seq=0 2023-11-16T16:58:03.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.880+0000 7f650affd700 1 -- 172.21.15.44:0/3752958849 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x559afaf41ff0 con 0x559afb210e60 2023-11-16T16:58:03.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.880+0000 7f650affd700 10 client.16878 handle_command_reply: tid=0 2023-11-16T16:58:03.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.880+0000 7f65456a3b80 1 client.16878 shutdown 2023-11-16T16:58:03.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.881+0000 7f65456a3b80 1 -- 172.21.15.44:0/3752958849 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 msgr2=0x559afb1b1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:03.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.881+0000 7f65456a3b80 1 --2- 172.21.15.44:0/3752958849 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1b1370 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f651c003f20 tx=0x7f651c009260 comp rx=0 tx=0).stop 2023-11-16T16:58:03.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.881+0000 7f65456a3b80 1 -- 172.21.15.44:0/3752958849 shutdown_connections 2023-11-16T16:58:03.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.881+0000 7f65456a3b80 1 --2- 172.21.15.44:0/3752958849 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x559afb210e60 0x559afb213270 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.881+0000 7f65456a3b80 1 --2- 172.21.15.44:0/3752958849 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559afb1a6930 0x559afb20f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.881+0000 7f65456a3b80 1 --2- 172.21.15.44:0/3752958849 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559afb1a1be0 0x559afb1b1370 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.881+0000 7f65456a3b80 1 --2- 172.21.15.44:0/3752958849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559afb1a24f0 0x559afb1b18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.881+0000 7f65456a3b80 1 -- 172.21.15.44:0/3752958849 >> 172.21.15.44:0/3752958849 conn(0x559afb1973e0 msgr2=0x559afb1a58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:03.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.881+0000 7f650affd700 0 client.16878 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:03.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.881+0000 7f65456a3b80 1 -- 172.21.15.44:0/3752958849 shutdown_connections 2023-11-16T16:58:03.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.881+0000 7f65456a3b80 1 -- 172.21.15.44:0/3752958849 wait complete. 2023-11-16T16:58:03.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.881+0000 7f65456a3b80 20 client.16878 trim_cache size 0 max 16384 2023-11-16T16:58:03.890 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:58:03.890 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:58:03.890 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "31904582-88fd-42ca-baf6-5526687f968e", 2023-11-16T16:58:03.891 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:58:03.891 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:58:03.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.882+0000 7f6531dff700 1 -- 172.21.15.44:0/3408763621 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f65140a1340 msgr2=0x7f65140a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:03.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.882+0000 7f6531dff700 1 --2- 172.21.15.44:0/3408763621 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f65140a1340 0x7f65140a37f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f652c19f2b0 tx=0x7f652400b040 comp rx=0 tx=0).stop 2023-11-16T16:58:03.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.882+0000 7f6531dff700 1 -- 172.21.15.44:0/3408763621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f652c101660 msgr2=0x7f652c19eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:03.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.882+0000 7f6531dff700 1 --2- 172.21.15.44:0/3408763621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f652c101660 0x7f652c19eaf0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f652000d980 tx=0x7f652000dc90 comp rx=0 tx=0).stop 2023-11-16T16:58:03.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.882+0000 7f6531dff700 1 -- 172.21.15.44:0/3408763621 shutdown_connections 2023-11-16T16:58:03.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.882+0000 7f6531dff700 1 --2- 172.21.15.44:0/3408763621 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f65140a1340 0x7f65140a37f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.882+0000 7f6531dff700 1 --2- 172.21.15.44:0/3408763621 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f652c105620 0x7f652c198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.882+0000 7f6531dff700 1 --2- 172.21.15.44:0/3408763621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f652c100cc0 0x7f652c19e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.882+0000 7f6531dff700 1 --2- 172.21.15.44:0/3408763621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f652c101660 0x7f652c19eaf0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:03.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.882+0000 7f6531dff700 1 -- 172.21.15.44:0/3408763621 >> 172.21.15.44:0/3408763621 conn(0x7f652c077560 msgr2=0x7f652c0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:03.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.882+0000 7f6531dff700 1 -- 172.21.15.44:0/3408763621 shutdown_connections 2023-11-16T16:58:03.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:03.882+0000 7f6531dff700 1 -- 172.21.15.44:0/3408763621 wait complete. 2023-11-16T16:58:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:03 smithi176 ceph-mon[131630]: pgmap v250: 129 pgs: 129 active+clean; 5.8 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 110 KiB/s wr, 287 op/s 2023-11-16T16:58:03.902 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:58:03.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:03 smithi077 ceph-mon[131204]: pgmap v250: 129 pgs: 129 active+clean; 5.8 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 110 KiB/s wr, 287 op/s 2023-11-16T16:58:04.110 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.108+0000 7fadf59ff700 1 -- 172.21.15.44:0/48518644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fadf0100cc0 msgr2=0x7fadf0101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:04.110 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.108+0000 7fadf59ff700 1 --2- 172.21.15.44:0/48518644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fadf0100cc0 0x7fadf0101090 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fade00099c0 tx=0x7fade0009cd0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.110 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.108+0000 7fadf59ff700 1 -- 172.21.15.44:0/48518644 shutdown_connections 2023-11-16T16:58:04.110 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.108+0000 7fadf59ff700 1 --2- 172.21.15.44:0/48518644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf0105620 0x7fadf010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.110 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.108+0000 7fadf59ff700 1 --2- 172.21.15.44:0/48518644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fadf0100cc0 0x7fadf0101090 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.108+0000 7fadf59ff700 1 --2- 172.21.15.44:0/48518644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fadf0101660 0x7fadf0104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.108+0000 7fadf59ff700 1 -- 172.21.15.44:0/48518644 >> 172.21.15.44:0/48518644 conn(0x7fadf0077560 msgr2=0x7fadf0077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:04.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.108+0000 7fadf59ff700 1 -- 172.21.15.44:0/48518644 shutdown_connections 2023-11-16T16:58:04.111 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.108+0000 7fadf59ff700 1 -- 172.21.15.44:0/48518644 wait complete. 2023-11-16T16:58:04.113 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.109+0000 7fadf59ff700 1 Processor -- start 2023-11-16T16:58:04.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.109+0000 7fadf59ff700 1 -- start start 2023-11-16T16:58:04.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.109+0000 7fadf59ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fadf0100cc0 0x7fadf0072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.109+0000 7fadf59ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf0101660 0x7fadf0073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.109+0000 7fadf59ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fadf0105620 0x7fadf0073870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.110+0000 7fadf59ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf01a49c0 con 0x7fadf0100cc0 2023-11-16T16:58:04.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.110+0000 7fadf59ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf01a4b30 con 0x7fadf0101660 2023-11-16T16:58:04.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.110+0000 7fadf59ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf01a4cf0 con 0x7fadf0105620 2023-11-16T16:58:04.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.110+0000 7fadeeffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fadf0100cc0 0x7fadf0072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.114 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.110+0000 7fadeeffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fadf0100cc0 0x7fadf0072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51394/0 (socket says 172.21.15.44:51394) 2023-11-16T16:58:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.110+0000 7fadeeffd700 1 -- 172.21.15.44:0/163279406 learned_addr learned my addr 172.21.15.44:0/163279406 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.110+0000 7fade7fff700 1 --2- 172.21.15.44:0/163279406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf0101660 0x7fadf0073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.110+0000 7fadef7fe700 1 --2- 172.21.15.44:0/163279406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fadf0105620 0x7fadf0073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.110+0000 7fadeeffd700 1 -- 172.21.15.44:0/163279406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fadf0105620 msgr2=0x7fadf0073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.110+0000 7fadeeffd700 1 --2- 172.21.15.44:0/163279406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fadf0105620 0x7fadf0073870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.110+0000 7fadeeffd700 1 -- 172.21.15.44:0/163279406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf0101660 msgr2=0x7fadf0073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.110+0000 7fadeeffd700 1 --2- 172.21.15.44:0/163279406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf0101660 0x7fadf0073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.110+0000 7fadeeffd700 1 -- 172.21.15.44:0/163279406 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade00095f0 con 0x7fadf0100cc0 2023-11-16T16:58:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.110+0000 7fade7fff700 1 --2- 172.21.15.44:0/163279406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf0101660 0x7fadf0073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:58:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.111+0000 7fadeeffd700 1 --2- 172.21.15.44:0/163279406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fadf0100cc0 0x7fadf0072df0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fade000b580 tx=0x7fade000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.111+0000 7fadf49fd700 1 -- 172.21.15.44:0/163279406 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade001d070 con 0x7fadf0100cc0 2023-11-16T16:58:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.111+0000 7fadf49fd700 1 -- 172.21.15.44:0/163279406 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fade0005050 con 0x7fadf0100cc0 2023-11-16T16:58:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.111+0000 7fadf59ff700 1 -- 172.21.15.44:0/163279406 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf01103d0 con 0x7fadf0100cc0 2023-11-16T16:58:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.111+0000 7fadf49fd700 1 -- 172.21.15.44:0/163279406 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade0004a40 con 0x7fadf0100cc0 2023-11-16T16:58:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.111+0000 7fadf59ff700 1 -- 172.21.15.44:0/163279406 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf0110240 con 0x7fadf0100cc0 2023-11-16T16:58:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.114+0000 7fadf49fd700 1 -- 172.21.15.44:0/163279406 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fade00051c0 con 0x7fadf0100cc0 2023-11-16T16:58:04.118 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.116+0000 7fae09237b80 1 Processor -- start 2023-11-16T16:58:04.118 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.116+0000 7fae09237b80 1 -- start start 2023-11-16T16:58:04.121 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.117+0000 7fae09237b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c013036b60 0x55c013036f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.117+0000 7fae09237b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013037470 0x55c01303b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.117+0000 7fae09237b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c01303b8b0 0x55c01303bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.117+0000 7fae09237b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c012f3c060 con 0x55c013036b60 2023-11-16T16:58:04.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.117+0000 7fae09237b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c012f8e8a0 con 0x55c01303b8b0 2023-11-16T16:58:04.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.117+0000 7fae09237b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c012f87d30 con 0x55c013037470 2023-11-16T16:58:04.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.117+0000 7fadf49fd700 1 --2- 172.21.15.44:0/163279406 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7faddc0a1340 0x7faddc0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.117+0000 7fadef7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c013036b60 0x55c013036f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.117+0000 7fadeeffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013037470 0x55c01303b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.122 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.117+0000 7fadf49fd700 1 -- 172.21.15.44:0/163279406 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fade00cdec0 con 0x7fadf0100cc0 2023-11-16T16:58:04.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.117+0000 7fade7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c01303b8b0 0x55c01303bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.117+0000 7fadeeffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013037470 0x55c01303b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:40338/0 (socket says 172.21.15.44:40338) 2023-11-16T16:58:04.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.117+0000 7fade7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c01303b8b0 0x55c01303bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:49668/0 (socket says 172.21.15.44:49668) 2023-11-16T16:58:04.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.117+0000 7fadef7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c013036b60 0x55c013036f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51410/0 (socket says 172.21.15.44:51410) 2023-11-16T16:58:04.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.117+0000 7fadeeffd700 1 -- 172.21.15.44:0/3892287933 learned_addr learned my addr 172.21.15.44:0/3892287933 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:04.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.117+0000 7fadef7fe700 1 --2- 172.21.15.44:0/163279406 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7faddc0a1340 0x7faddc0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.118+0000 7fade7fff700 1 -- 172.21.15.44:0/3892287933 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013037470 msgr2=0x55c01303b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:04.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.118+0000 7fade7fff700 1 --2- 172.21.15.44:0/3892287933 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013037470 0x55c01303b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.118+0000 7fade7fff700 1 -- 172.21.15.44:0/3892287933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c013036b60 msgr2=0x55c013036f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:04.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.118+0000 7fade7fff700 1 --2- 172.21.15.44:0/3892287933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c013036b60 0x55c013036f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.118+0000 7fade7fff700 1 -- 172.21.15.44:0/3892287933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c012e09630 con 0x55c01303b8b0 2023-11-16T16:58:04.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.118+0000 7fadeeffd700 1 --2- 172.21.15.44:0/3892287933 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013037470 0x55c01303b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:04.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.118+0000 7fadef7fe700 1 --2- 172.21.15.44:0/3892287933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c013036b60 0x55c013036f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:58:04.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.118+0000 7fade7fff700 1 --2- 172.21.15.44:0/3892287933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c01303b8b0 0x55c01303bd00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fadf006a350 tx=0x7fadd8009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=622ab3cdba5bced7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:04.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.118+0000 7fadef7fe700 1 --2- 172.21.15.44:0/163279406 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7faddc0a1340 0x7faddc0a37f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fade8009ec0 tx=0x7fade800b040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:04.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.118+0000 7fade5ffb700 1 -- 172.21.15.44:0/3892287933 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd8011bc0 con 0x55c01303b8b0 2023-11-16T16:58:04.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.118+0000 7fade5ffb700 1 -- 172.21.15.44:0/3892287933 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fadd8011d20 con 0x55c01303b8b0 2023-11-16T16:58:04.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.118+0000 7fade5ffb700 1 -- 172.21.15.44:0/3892287933 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd800f6a0 con 0x55c01303b8b0 2023-11-16T16:58:04.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.119+0000 7fae09237b80 1 -- 172.21.15.44:0/3892287933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c01303b8b0 msgr2=0x55c01303bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:04.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.119+0000 7fae09237b80 1 --2- 172.21.15.44:0/3892287933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c01303b8b0 0x55c01303bd00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fadf006a350 tx=0x7fadd8009f70 comp rx=0 tx=0).stop 2023-11-16T16:58:04.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.119+0000 7fae09237b80 1 -- 172.21.15.44:0/3892287933 shutdown_connections 2023-11-16T16:58:04.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.119+0000 7fae09237b80 1 --2- 172.21.15.44:0/3892287933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c01303b8b0 0x55c01303bd00 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.119+0000 7fae09237b80 1 --2- 172.21.15.44:0/3892287933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c013036b60 0x55c013036f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.119+0000 7fae09237b80 1 --2- 172.21.15.44:0/3892287933 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013037470 0x55c01303b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.119+0000 7fae09237b80 1 -- 172.21.15.44:0/3892287933 >> 172.21.15.44:0/3892287933 conn(0x55c01302c360 msgr2=0x55c013031b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:04.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.119+0000 7fae09237b80 1 -- 172.21.15.44:0/3892287933 shutdown_connections 2023-11-16T16:58:04.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.119+0000 7fae09237b80 1 -- 172.21.15.44:0/3892287933 wait complete. 2023-11-16T16:58:04.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.120+0000 7fae09237b80 1 -- start start 2023-11-16T16:58:04.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.120+0000 7fae09237b80 1 Processor -- start 2023-11-16T16:58:04.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.120+0000 7fae09237b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013036b60 0x55c013042f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.120+0000 7fae09237b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013037470 0x55c0130401d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.129 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.120+0000 7fae09237b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c01303b8b0 0x55c013040750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.120+0000 7fae09237b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c012f298e0 con 0x55c01303b8b0 2023-11-16T16:58:04.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.120+0000 7fae09237b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c012f29580 con 0x55c013037470 2023-11-16T16:58:04.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.120+0000 7fae09237b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c012e0b920 con 0x55c013036b60 2023-11-16T16:58:04.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.120+0000 7fae09237b80 10 client.0 _ll_register_callbacks cb 0x55c012f47780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:04.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.121+0000 7fadeeffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013036b60 0x55c013042f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.121+0000 7fade7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013037470 0x55c0130401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.121+0000 7fadef7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c01303b8b0 0x55c013040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.121+0000 7fadeeffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013036b60 0x55c013042f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:40346/0 (socket says 172.21.15.44:40346) 2023-11-16T16:58:04.130 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.121+0000 7fadeeffd700 1 -- 172.21.15.44:0/2960620512 learned_addr learned my addr 172.21.15.44:0/2960620512 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:04.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.121+0000 7fade7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013037470 0x55c0130401d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:49672/0 (socket says 172.21.15.44:49672) 2023-11-16T16:58:04.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.121+0000 7fadeeffd700 1 -- 172.21.15.44:0/2960620512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013037470 msgr2=0x55c0130401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:04.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.121+0000 7fadeeffd700 1 --2- 172.21.15.44:0/2960620512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013037470 0x55c0130401d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.121+0000 7fadeeffd700 1 -- 172.21.15.44:0/2960620512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c01303b8b0 msgr2=0x55c013040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:04.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.121+0000 7fadeeffd700 1 --2- 172.21.15.44:0/2960620512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c01303b8b0 0x55c013040750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.121+0000 7fadeeffd700 1 -- 172.21.15.44:0/2960620512 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd80098f0 con 0x55c013036b60 2023-11-16T16:58:04.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.121+0000 7fadeeffd700 1 --2- 172.21.15.44:0/2960620512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013036b60 0x55c013042f50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fade00a2a70 tx=0x7fade009acd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50f8388becb1561f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:04.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.121+0000 7fadceffd700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:58:04.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.121+0000 7fadceffd700 1 -- 172.21.15.44:0/2960620512 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade009a0d0 con 0x55c013036b60 2023-11-16T16:58:04.131 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.121+0000 7fae09237b80 1 -- 172.21.15.44:0/2960620512 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c013029b20 con 0x55c013036b60 2023-11-16T16:58:04.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.121+0000 7fadceffd700 1 -- 172.21.15.44:0/2960620512 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fade009a230 con 0x55c013036b60 2023-11-16T16:58:04.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.122+0000 7fadceffd700 1 -- 172.21.15.44:0/2960620512 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fade00b3260 con 0x55c013036b60 2023-11-16T16:58:04.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.122+0000 7fae09237b80 10 client.26917 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:04.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.122+0000 7fae09237b80 1 -- 172.21.15.44:0/2960620512 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55c012f4e190 con 0x55c013036b60 2023-11-16T16:58:04.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.122+0000 7fadceffd700 1 -- 172.21.15.44:0/2960620512 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c013029b20 con 0x55c013036b60 2023-11-16T16:58:04.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.122+0000 7fadceffd700 1 -- 172.21.15.44:0/2960620512 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fade0004c90 con 0x55c013036b60 2023-11-16T16:58:04.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.122+0000 7fae09237b80 10 client.26917 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:04.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.122+0000 7fae09237b80 10 client.26917 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:04.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.122+0000 7fae09237b80 20 client.26917 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:04.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.122+0000 7fae09237b80 1 --2- 172.21.15.44:0/2960620512 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c013050340 0x55c013052750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.122+0000 7fae09237b80 4 client.26917 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:58:04.132 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.122+0000 7fae09237b80 1 -- 172.21.15.44:0/2960620512 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c012f4a0a0 con 0x55c013050340 2023-11-16T16:58:04.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.123+0000 7fade7fff700 1 --2- 172.21.15.44:0/2960620512 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c013050340 0x55c013052750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.123+0000 7fade7fff700 1 --2- 172.21.15.44:0/2960620512 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c013050340 0x55c013052750 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2651138a90a9e31 server_cookie=98c2bbbfce5fedbd in_seq=0 out_seq=0 2023-11-16T16:58:04.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.123+0000 7fadceffd700 10 client.26917 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:04.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.124+0000 7fadceffd700 1 -- 172.21.15.44:0/2960620512 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55c012f4a0a0 con 0x55c013050340 2023-11-16T16:58:04.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.124+0000 7fadceffd700 10 client.26917 handle_command_reply: tid=0 2023-11-16T16:58:04.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.124+0000 7fae09237b80 1 client.26917 shutdown 2023-11-16T16:58:04.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.124+0000 7fae09237b80 1 -- 172.21.15.44:0/2960620512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013036b60 msgr2=0x55c013042f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:04.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.124+0000 7fae09237b80 1 --2- 172.21.15.44:0/2960620512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013036b60 0x55c013042f50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fade00a2a70 tx=0x7fade009acd0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.124+0000 7fae09237b80 1 -- 172.21.15.44:0/2960620512 shutdown_connections 2023-11-16T16:58:04.133 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.124+0000 7fae09237b80 1 --2- 172.21.15.44:0/2960620512 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c013050340 0x55c013052750 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.124+0000 7fae09237b80 1 --2- 172.21.15.44:0/2960620512 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c01303b8b0 0x55c013040750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.124+0000 7fae09237b80 1 --2- 172.21.15.44:0/2960620512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013036b60 0x55c013042f50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.124+0000 7fae09237b80 1 --2- 172.21.15.44:0/2960620512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013037470 0x55c0130401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.124+0000 7fadceffd700 0 client.26917 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:04.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.124+0000 7fae09237b80 1 -- 172.21.15.44:0/2960620512 >> 172.21.15.44:0/2960620512 conn(0x55c01302c360 msgr2=0x55c01303cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:04.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.124+0000 7fae09237b80 1 -- 172.21.15.44:0/2960620512 shutdown_connections 2023-11-16T16:58:04.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.124+0000 7fae09237b80 1 -- 172.21.15.44:0/2960620512 wait complete. 2023-11-16T16:58:04.134 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.125+0000 7fae09237b80 20 client.26917 trim_cache size 0 max 16384 2023-11-16T16:58:04.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.142+0000 7fae09237b80 1 Processor -- start 2023-11-16T16:58:04.146 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.142+0000 7fae09237b80 1 -- start start 2023-11-16T16:58:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fae09237b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c013036b60 0x55c013031ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fae09237b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013037470 0x55c0130479d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fae09237b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c01303b8b0 0x55c013042af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fae09237b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0130379c0 con 0x55c013036b60 2023-11-16T16:58:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fae09237b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c01303a140 con 0x55c013037470 2023-11-16T16:58:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fae09237b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0130417b0 con 0x55c01303b8b0 2023-11-16T16:58:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fadeeffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c013036b60 0x55c013031ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fadeeffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c013036b60 0x55c013031ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51436/0 (socket says 172.21.15.44:51436) 2023-11-16T16:58:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fadeeffd700 1 -- 172.21.15.44:0/764017986 learned_addr learned my addr 172.21.15.44:0/764017986 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fade7fff700 1 --2- 172.21.15.44:0/764017986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013037470 0x55c0130479d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fadef7fe700 1 --2- 172.21.15.44:0/764017986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c01303b8b0 0x55c013042af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fadeeffd700 1 -- 172.21.15.44:0/764017986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c01303b8b0 msgr2=0x55c013042af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fadeeffd700 1 --2- 172.21.15.44:0/764017986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c01303b8b0 0x55c013042af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fadeeffd700 1 -- 172.21.15.44:0/764017986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013037470 msgr2=0x55c0130479d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fadeeffd700 1 --2- 172.21.15.44:0/764017986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013037470 0x55c0130479d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fadeeffd700 1 -- 172.21.15.44:0/764017986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c012f4dee0 con 0x55c013036b60 2023-11-16T16:58:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.143+0000 7fadef7fe700 1 --2- 172.21.15.44:0/764017986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c01303b8b0 0x55c013042af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:58:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.144+0000 7fadeeffd700 1 --2- 172.21.15.44:0/764017986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c013036b60 0x55c013031ce0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fade0098f30 tx=0x7fade00cb2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f23bd56aa384faf server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.144+0000 7fade67fc700 1 -- 172.21.15.44:0/764017986 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade0096ac0 con 0x55c013036b60 2023-11-16T16:58:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.144+0000 7fade67fc700 1 -- 172.21.15.44:0/764017986 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fade009b050 con 0x55c013036b60 2023-11-16T16:58:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.144+0000 7fade67fc700 1 -- 172.21.15.44:0/764017986 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade009a330 con 0x55c013036b60 2023-11-16T16:58:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.144+0000 7fae09237b80 1 -- 172.21.15.44:0/764017986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c013036b60 msgr2=0x55c013031ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.144+0000 7fae09237b80 1 --2- 172.21.15.44:0/764017986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c013036b60 0x55c013031ce0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fade0098f30 tx=0x7fade00cb2b0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.144+0000 7fae09237b80 1 -- 172.21.15.44:0/764017986 shutdown_connections 2023-11-16T16:58:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.144+0000 7fae09237b80 1 --2- 172.21.15.44:0/764017986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c01303b8b0 0x55c013042af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.144+0000 7fae09237b80 1 --2- 172.21.15.44:0/764017986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c013036b60 0x55c013031ce0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.144+0000 7fae09237b80 1 --2- 172.21.15.44:0/764017986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013037470 0x55c0130479d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.144+0000 7fae09237b80 1 -- 172.21.15.44:0/764017986 >> 172.21.15.44:0/764017986 conn(0x55c01302c360 msgr2=0x55c01304cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.144+0000 7fae09237b80 1 -- 172.21.15.44:0/764017986 shutdown_connections 2023-11-16T16:58:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.145+0000 7fae09237b80 1 -- 172.21.15.44:0/764017986 wait complete. 2023-11-16T16:58:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.145+0000 7fae09237b80 1 -- start start 2023-11-16T16:58:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.145+0000 7fae09237b80 1 Processor -- start 2023-11-16T16:58:04.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.145+0000 7fae09237b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013036b60 0x55c01303dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fae09237b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013037470 0x55c01303c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fae09237b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c01303b8b0 0x55c01303c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fae09237b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c01303e210 con 0x55c01303b8b0 2023-11-16T16:58:04.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fae09237b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c01303cee0 con 0x55c013036b60 2023-11-16T16:58:04.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fae09237b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c01303d0a0 con 0x55c013037470 2023-11-16T16:58:04.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fae09237b80 10 client.0 _ll_register_callbacks cb 0x7fade009a5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:04.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fadeeffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013036b60 0x55c01303dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fadeeffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013036b60 0x55c01303dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:49690/0 (socket says 172.21.15.44:49690) 2023-11-16T16:58:04.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fadeeffd700 1 -- 172.21.15.44:0/1923835843 learned_addr learned my addr 172.21.15.44:0/1923835843 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:04.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fadef7fe700 1 --2- 172.21.15.44:0/1923835843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c01303b8b0 0x55c01303c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fade7fff700 1 --2- 172.21.15.44:0/1923835843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013037470 0x55c01303c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fadeeffd700 1 -- 172.21.15.44:0/1923835843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013037470 msgr2=0x55c01303c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:04.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fadeeffd700 1 --2- 172.21.15.44:0/1923835843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013037470 0x55c01303c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fadeeffd700 1 -- 172.21.15.44:0/1923835843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c01303b8b0 msgr2=0x55c01303c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:04.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fadeeffd700 1 --2- 172.21.15.44:0/1923835843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c01303b8b0 0x55c01303c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.155 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fadeeffd700 1 -- 172.21.15.44:0/1923835843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c012f4e190 con 0x55c013036b60 2023-11-16T16:58:04.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fade7fff700 1 --2- 172.21.15.44:0/1923835843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013037470 0x55c01303c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:58:04.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fadeeffd700 1 --2- 172.21.15.44:0/1923835843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013036b60 0x55c01303dbf0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fade00970e0 tx=0x7fade009a9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba82a2eac324256f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:04.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.146+0000 7fadceffd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:58:04.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.147+0000 7fadceffd700 1 -- 172.21.15.44:0/1923835843 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade0096ac0 con 0x55c013036b60 2023-11-16T16:58:04.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.147+0000 7fadceffd700 1 -- 172.21.15.44:0/1923835843 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fade0004bc0 con 0x55c013036b60 2023-11-16T16:58:04.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.147+0000 7fae09237b80 1 -- 172.21.15.44:0/1923835843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fadd8011d20 con 0x55c013036b60 2023-11-16T16:58:04.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.147+0000 7fadceffd700 1 -- 172.21.15.44:0/1923835843 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fade0084170 con 0x55c013036b60 2023-11-16T16:58:04.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.147+0000 7fae09237b80 10 client.26975 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:04.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.147+0000 7fae09237b80 1 -- 172.21.15.44:0/1923835843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55c012d9ec60 con 0x55c013036b60 2023-11-16T16:58:04.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.147+0000 7fadceffd700 1 -- 172.21.15.44:0/1923835843 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fade00be080 con 0x55c013036b60 2023-11-16T16:58:04.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.147+0000 7fadceffd700 1 -- 172.21.15.44:0/1923835843 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fade00ba1f0 con 0x55c013036b60 2023-11-16T16:58:04.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.147+0000 7fae09237b80 10 client.26975 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:04.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.147+0000 7fae09237b80 10 client.26975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:04.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.147+0000 7fae09237b80 20 client.26975 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:04.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.147+0000 7fae09237b80 1 --2- 172.21.15.44:0/1923835843 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c0130a4fb0 0x55c0130a7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:04.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.147+0000 7fae09237b80 4 client.26975 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:58:04.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.147+0000 7fae09237b80 1 -- 172.21.15.44:0/1923835843 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c012f56660 con 0x55c0130a4fb0 2023-11-16T16:58:04.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.148+0000 7fade7fff700 1 --2- 172.21.15.44:0/1923835843 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c0130a4fb0 0x55c0130a7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:04.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.148+0000 7fade7fff700 1 --2- 172.21.15.44:0/1923835843 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c0130a4fb0 0x55c0130a7390 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8094b91945eaa3c server_cookie=cf664435487a7d32 in_seq=0 out_seq=0 2023-11-16T16:58:04.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.148+0000 7fadceffd700 10 client.26975 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:04.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.148+0000 7fadceffd700 1 -- 172.21.15.44:0/1923835843 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c012f56660 con 0x55c0130a4fb0 2023-11-16T16:58:04.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.148+0000 7fadceffd700 10 client.26975 handle_command_reply: tid=0 2023-11-16T16:58:04.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.148+0000 7fae09237b80 1 client.26975 shutdown 2023-11-16T16:58:04.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.149+0000 7fae09237b80 1 -- 172.21.15.44:0/1923835843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013036b60 msgr2=0x55c01303dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:04.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.149+0000 7fae09237b80 1 --2- 172.21.15.44:0/1923835843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013036b60 0x55c01303dbf0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fade00970e0 tx=0x7fade009a9b0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.149+0000 7fae09237b80 1 -- 172.21.15.44:0/1923835843 shutdown_connections 2023-11-16T16:58:04.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.149+0000 7fae09237b80 1 --2- 172.21.15.44:0/1923835843 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c0130a4fb0 0x55c0130a7390 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.149+0000 7fae09237b80 1 --2- 172.21.15.44:0/1923835843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c01303b8b0 0x55c01303c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.149+0000 7fae09237b80 1 --2- 172.21.15.44:0/1923835843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c013036b60 0x55c01303dbf0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.149+0000 7fae09237b80 1 --2- 172.21.15.44:0/1923835843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c013037470 0x55c01303c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.149+0000 7fadceffd700 0 client.26975 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.149+0000 7fae09237b80 1 -- 172.21.15.44:0/1923835843 >> 172.21.15.44:0/1923835843 conn(0x55c01302c360 msgr2=0x55c013035960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.149+0000 7fae09237b80 1 -- 172.21.15.44:0/1923835843 shutdown_connections 2023-11-16T16:58:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.149+0000 7fae09237b80 1 -- 172.21.15.44:0/1923835843 wait complete. 2023-11-16T16:58:04.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.149+0000 7fae09237b80 20 client.26975 trim_cache size 0 max 16384 2023-11-16T16:58:04.159 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:58:04.159 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2023-11-16T16:58:04.159 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2023-11-16T16:58:04.160 INFO:teuthology.orchestra.run.smithi044.stdout: "31904582-88fd-42ca-baf6-5526687f968e": { 2023-11-16T16:58:04.160 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2023-11-16T16:58:04.160 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "31904582-88fd-42ca-baf6-5526687f968e", 2023-11-16T16:58:04.160 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2023-11-16T16:58:04.160 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:58:04.160 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:58:04.160 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:58:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.149+0000 7fadf59ff700 1 -- 172.21.15.44:0/163279406 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7faddc0a1340 msgr2=0x7faddc0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.150+0000 7fadf59ff700 1 --2- 172.21.15.44:0/163279406 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7faddc0a1340 0x7faddc0a37f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fade8009ec0 tx=0x7fade800b040 comp rx=0 tx=0).stop 2023-11-16T16:58:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.150+0000 7fadf59ff700 1 -- 172.21.15.44:0/163279406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fadf0100cc0 msgr2=0x7fadf0072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.150+0000 7fadf59ff700 1 --2- 172.21.15.44:0/163279406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fadf0100cc0 0x7fadf0072df0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fade000b580 tx=0x7fade000bfd0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.150+0000 7fadf59ff700 1 -- 172.21.15.44:0/163279406 shutdown_connections 2023-11-16T16:58:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.150+0000 7fadf59ff700 1 --2- 172.21.15.44:0/163279406 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7faddc0a1340 0x7faddc0a37f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.150+0000 7fadf59ff700 1 --2- 172.21.15.44:0/163279406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fadf0105620 0x7fadf0073870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.161 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.150+0000 7fadf59ff700 1 --2- 172.21.15.44:0/163279406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fadf0100cc0 0x7fadf0072df0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.150+0000 7fadf59ff700 1 --2- 172.21.15.44:0/163279406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf0101660 0x7fadf0073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:04.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.150+0000 7fadf59ff700 1 -- 172.21.15.44:0/163279406 >> 172.21.15.44:0/163279406 conn(0x7fadf0077560 msgr2=0x7fadf00fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:04.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.150+0000 7fadf59ff700 1 -- 172.21.15.44:0/163279406 shutdown_connections 2023-11-16T16:58:04.162 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:04.150+0000 7fadf59ff700 1 -- 172.21.15.44:0/163279406 wait complete. 2023-11-16T16:58:04.170 INFO:tasks.cephfs.filesystem:scrub status for tag:31904582-88fd-42ca-baf6-5526687f968e - {'path': '/', 'tag': '31904582-88fd-42ca-baf6-5526687f968e', 'options': 'recursive,force'} 2023-11-16T16:58:04.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:04 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:58:04.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:04 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:58:04.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:04 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:58:04.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:04 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:58:04.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:04 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:58:04.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:04 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:58:04.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:04 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:58:04.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:04 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:58:04.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:04 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:58:04.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:04 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:58:04.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:04 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:58:04.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:04 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:58:05.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:05 smithi044 ceph-mon[128750]: pgmap v251: 129 pgs: 129 active+clean; 5.9 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 163 KiB/s wr, 305 op/s 2023-11-16T16:58:05.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:05 smithi176 ceph-mon[131630]: pgmap v251: 129 pgs: 129 active+clean; 5.9 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 163 KiB/s wr, 305 op/s 2023-11-16T16:58:05.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:05 smithi077 ceph-mon[131204]: pgmap v251: 129 pgs: 129 active+clean; 5.9 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 163 KiB/s wr, 305 op/s 2023-11-16T16:58:07.419 INFO:tasks.workunit.client.0.smithi044.stdout:ok 2023-11-16T16:58:07.424 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T16:58:07.424 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T16:58:07.517 INFO:tasks.workunit:Running workunit fs/misc/general_vxattrs.sh... 2023-11-16T16:58:07.517 DEBUG:teuthology.orchestra.run.smithi044:workunit test fs/misc/general_vxattrs.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh 2023-11-16T16:58:07.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.520+0000 7fb0855ff700 1 -- 172.21.15.44:0/2061568872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb080100c50 msgr2=0x7fb080101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:07.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.520+0000 7fb0855ff700 1 --2- 172.21.15.44:0/2061568872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb080100c50 0x7fb080101020 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb0700099c0 tx=0x7fb070009cd0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.520+0000 7fb0855ff700 1 -- 172.21.15.44:0/2061568872 shutdown_connections 2023-11-16T16:58:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.520+0000 7fb0855ff700 1 --2- 172.21.15.44:0/2061568872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb080105560 0x7fb08010d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.520+0000 7fb0855ff700 1 --2- 172.21.15.44:0/2061568872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb080100c50 0x7fb080101020 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.520+0000 7fb0855ff700 1 --2- 172.21.15.44:0/2061568872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0801015f0 0x7fb080104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.520+0000 7fb0855ff700 1 -- 172.21.15.44:0/2061568872 >> 172.21.15.44:0/2061568872 conn(0x7fb080077550 msgr2=0x7fb080077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.521+0000 7fb0855ff700 1 -- 172.21.15.44:0/2061568872 shutdown_connections 2023-11-16T16:58:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.522+0000 7fb0855ff700 1 -- 172.21.15.44:0/2061568872 wait complete. 2023-11-16T16:58:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.523+0000 7fb0855ff700 1 Processor -- start 2023-11-16T16:58:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.523+0000 7fb0855ff700 1 -- start start 2023-11-16T16:58:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.523+0000 7fb0855ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb080100c50 0x7fb080072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.523+0000 7fb0855ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0801015f0 0x7fb080073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.523+0000 7fb0855ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb080105560 0x7fb0801a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.523+0000 7fb0855ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb080073b60 con 0x7fb0801015f0 2023-11-16T16:58:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.523+0000 7fb0855ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0801a4ac0 con 0x7fb080105560 2023-11-16T16:58:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.523+0000 7fb0855ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0801a4c00 con 0x7fb080100c50 2023-11-16T16:58:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.524+0000 7fb07e7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0801015f0 0x7fb080073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.524+0000 7fb07effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb080100c50 0x7fb080072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.524+0000 7fb07e7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0801015f0 0x7fb080073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57270/0 (socket says 172.21.15.44:57270) 2023-11-16T16:58:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.524+0000 7fb07effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb080100c50 0x7fb080072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:46434/0 (socket says 172.21.15.44:46434) 2023-11-16T16:58:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.524+0000 7fb07e7fc700 1 -- 172.21.15.44:0/1457826444 learned_addr learned my addr 172.21.15.44:0/1457826444 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.524+0000 7fb07f7fe700 1 --2- 172.21.15.44:0/1457826444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb080105560 0x7fb0801a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.524+0000 7fb07effd700 1 -- 172.21.15.44:0/1457826444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb080105560 msgr2=0x7fb0801a4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.524+0000 7fb07effd700 1 --2- 172.21.15.44:0/1457826444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb080105560 0x7fb0801a4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.524+0000 7fb07effd700 1 -- 172.21.15.44:0/1457826444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0801015f0 msgr2=0x7fb080073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.524+0000 7fb07effd700 1 --2- 172.21.15.44:0/1457826444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0801015f0 0x7fb080073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.524+0000 7fb07effd700 1 -- 172.21.15.44:0/1457826444 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0700095f0 con 0x7fb080100c50 2023-11-16T16:58:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.524+0000 7fb07e7fc700 1 --2- 172.21.15.44:0/1457826444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0801015f0 0x7fb080073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.524+0000 7fb07effd700 1 --2- 172.21.15.44:0/1457826444 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb080100c50 0x7fb080072df0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb07000b580 tx=0x7fb07000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.525+0000 7fb067fff700 1 -- 172.21.15.44:0/1457826444 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07001d070 con 0x7fb080100c50 2023-11-16T16:58:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.525+0000 7fb0855ff700 1 -- 172.21.15.44:0/1457826444 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0801103b0 con 0x7fb080100c50 2023-11-16T16:58:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.525+0000 7fb067fff700 1 -- 172.21.15.44:0/1457826444 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb070022470 con 0x7fb080100c50 2023-11-16T16:58:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.525+0000 7fb067fff700 1 -- 172.21.15.44:0/1457826444 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb070017650 con 0x7fb080100c50 2023-11-16T16:58:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.525+0000 7fb0855ff700 1 -- 172.21.15.44:0/1457826444 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb080110220 con 0x7fb080100c50 2023-11-16T16:58:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.530+0000 7fb067fff700 1 -- 172.21.15.44:0/1457826444 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fb070022850 con 0x7fb080100c50 2023-11-16T16:58:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.531+0000 7fb098e6db80 1 Processor -- start 2023-11-16T16:58:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.531+0000 7fb098e6db80 1 -- start start 2023-11-16T16:58:07.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.531+0000 7fb098e6db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15d9a6b0 0x557a15f441e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.531+0000 7fb098e6db80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 0x557a15f3e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.531+0000 7fb098e6db80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15f44b90 0x557a15f3eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.531+0000 7fb098e6db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a15e44000 con 0x557a15d9a6b0 2023-11-16T16:58:07.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.531+0000 7fb098e6db80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a15e96840 con 0x557a15f44720 2023-11-16T16:58:07.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.531+0000 7fb098e6db80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a15e8fcd0 con 0x557a15f44b90 2023-11-16T16:58:07.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.532+0000 7fb07f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15d9a6b0 0x557a15f441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.532+0000 7fb07effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 0x557a15f3e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.532+0000 7fb07f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15d9a6b0 0x557a15f441e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57280/0 (socket says 172.21.15.44:57280) 2023-11-16T16:58:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.532+0000 7fb07f7fe700 1 -- 172.21.15.44:0/465166700 learned_addr learned my addr 172.21.15.44:0/465166700 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.532+0000 7fb07e7fc700 1 --2- 172.21.15.44:0/465166700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15f44b90 0x557a15f3eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.532+0000 7fb07f7fe700 1 -- 172.21.15.44:0/465166700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15f44b90 msgr2=0x557a15f3eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.532+0000 7fb07f7fe700 1 --2- 172.21.15.44:0/465166700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15f44b90 0x557a15f3eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.532+0000 7fb07f7fe700 1 -- 172.21.15.44:0/465166700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 msgr2=0x557a15f3e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.532+0000 7fb07f7fe700 1 --2- 172.21.15.44:0/465166700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 0x557a15f3e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.532+0000 7fb07f7fe700 1 -- 172.21.15.44:0/465166700 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a15d115d0 con 0x557a15d9a6b0 2023-11-16T16:58:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.532+0000 7fb067fff700 1 --2- 172.21.15.44:0/1457826444 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fb0680a12c0 0x7fb0680a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.532+0000 7fb07f7fe700 1 --2- 172.21.15.44:0/465166700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15d9a6b0 0x557a15f441e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb078011600 tx=0x7fb078009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65274d6b59fbd1ee server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.532+0000 7fb067fff700 1 -- 172.21.15.44:0/1457826444 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fb0700cdf30 con 0x7fb080100c50 2023-11-16T16:58:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.533+0000 7fb07effd700 1 --2- 172.21.15.44:0/1457826444 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fb0680a12c0 0x7fb0680a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.533+0000 7fb0657fa700 1 -- 172.21.15.44:0/465166700 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07800e040 con 0x557a15d9a6b0 2023-11-16T16:58:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.533+0000 7fb0657fa700 1 -- 172.21.15.44:0/465166700 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb07800f040 con 0x557a15d9a6b0 2023-11-16T16:58:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.533+0000 7fb0657fa700 1 -- 172.21.15.44:0/465166700 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0780046a0 con 0x557a15d9a6b0 2023-11-16T16:58:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.534+0000 7fb07effd700 1 --2- 172.21.15.44:0/1457826444 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fb0680a12c0 0x7fb0680a3770 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb0700cb350 tx=0x7fb0700d1a40 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.534+0000 7fb098e6db80 1 -- 172.21.15.44:0/465166700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15d9a6b0 msgr2=0x557a15f441e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.534+0000 7fb098e6db80 1 --2- 172.21.15.44:0/465166700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15d9a6b0 0x557a15f441e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb078011600 tx=0x7fb078009920 comp rx=0 tx=0).stop 2023-11-16T16:58:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.534+0000 7fb098e6db80 1 -- 172.21.15.44:0/465166700 shutdown_connections 2023-11-16T16:58:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.534+0000 7fb098e6db80 1 --2- 172.21.15.44:0/465166700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15f44b90 0x557a15f3eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.534+0000 7fb098e6db80 1 --2- 172.21.15.44:0/465166700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15d9a6b0 0x557a15f441e0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.534+0000 7fb098e6db80 1 --2- 172.21.15.44:0/465166700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 0x557a15f3e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.534+0000 7fb098e6db80 1 -- 172.21.15.44:0/465166700 >> 172.21.15.44:0/465166700 conn(0x557a15f33f50 msgr2=0x557a15f39270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.534+0000 7fb098e6db80 1 -- 172.21.15.44:0/465166700 shutdown_connections 2023-11-16T16:58:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.534+0000 7fb098e6db80 1 -- 172.21.15.44:0/465166700 wait complete. 2023-11-16T16:58:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.535+0000 7fb098e6db80 1 -- start start 2023-11-16T16:58:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.535+0000 7fb098e6db80 1 Processor -- start 2023-11-16T16:58:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.535+0000 7fb098e6db80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15d9a6b0 0x557a15f4aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.535+0000 7fb098e6db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15f44720 0x557a15f47c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.535+0000 7fb098e6db80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15f44b90 0x557a15f482c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.535+0000 7fb098e6db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a15e31880 con 0x557a15f44720 2023-11-16T16:58:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.535+0000 7fb098e6db80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a15e31520 con 0x557a15d9a6b0 2023-11-16T16:58:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.535+0000 7fb098e6db80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a15d138c0 con 0x557a15f44b90 2023-11-16T16:58:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.535+0000 7fb098e6db80 10 client.0 _ll_register_callbacks cb 0x557a15e4f720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.536+0000 7fb07e7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15f44720 0x557a15f47c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.536+0000 7fb07f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15d9a6b0 0x557a15f4aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.536+0000 7fb07e7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15f44720 0x557a15f47c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57296/0 (socket says 172.21.15.44:57296) 2023-11-16T16:58:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.536+0000 7fb07e7fc700 1 -- 172.21.15.44:0/966096532 learned_addr learned my addr 172.21.15.44:0/966096532 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.536+0000 7fb07f7fe700 1 --2- 172.21.15.44:0/966096532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15f44b90 0x557a15f482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.536+0000 7fb07f7fe700 1 -- 172.21.15.44:0/966096532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15f44b90 msgr2=0x557a15f482c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.536+0000 7fb07f7fe700 1 --2- 172.21.15.44:0/966096532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15f44b90 0x557a15f482c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.536+0000 7fb07f7fe700 1 -- 172.21.15.44:0/966096532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15f44720 msgr2=0x557a15f47c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.536+0000 7fb07f7fe700 1 --2- 172.21.15.44:0/966096532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15f44720 0x557a15f47c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.536+0000 7fb07f7fe700 1 -- 172.21.15.44:0/966096532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0780112b0 con 0x557a15d9a6b0 2023-11-16T16:58:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.536+0000 7fb07e7fc700 1 --2- 172.21.15.44:0/966096532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15f44720 0x557a15f47c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.536+0000 7fb07f7fe700 1 --2- 172.21.15.44:0/966096532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15d9a6b0 0x557a15f4aa10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb0780115d0 tx=0x7fb0780041a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f5ac03600989a04 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.537+0000 7fb062ff5700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:58:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.537+0000 7fb062ff5700 1 -- 172.21.15.44:0/966096532 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb078026070 con 0x557a15d9a6b0 2023-11-16T16:58:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.537+0000 7fb098e6db80 1 -- 172.21.15.44:0/966096532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557a15f31710 con 0x557a15d9a6b0 2023-11-16T16:58:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.537+0000 7fb062ff5700 1 -- 172.21.15.44:0/966096532 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fb078004240 con 0x557a15d9a6b0 2023-11-16T16:58:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.537+0000 7fb062ff5700 1 -- 172.21.15.44:0/966096532 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb078003e90 con 0x557a15d9a6b0 2023-11-16T16:58:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.537+0000 7fb098e6db80 10 client.26987 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.537+0000 7fb098e6db80 1 -- 172.21.15.44:0/966096532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x557a15e56130 con 0x557a15d9a6b0 2023-11-16T16:58:07.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.537+0000 7fb062ff5700 1 -- 172.21.15.44:0/966096532 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557a15f31710 con 0x557a15d9a6b0 2023-11-16T16:58:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.538+0000 7fb062ff5700 1 -- 172.21.15.44:0/966096532 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb078015450 con 0x557a15d9a6b0 2023-11-16T16:58:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.538+0000 7fb098e6db80 10 client.26987 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.538+0000 7fb098e6db80 10 client.26987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.538+0000 7fb098e6db80 20 client.26987 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.538+0000 7fb098e6db80 1 --2- 172.21.15.44:0/966096532 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557a15f57f40 0x557a15f5a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.538+0000 7fb098e6db80 4 client.26987 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:58:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.538+0000 7fb098e6db80 1 -- 172.21.15.44:0/966096532 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557a15e52040 con 0x557a15f57f40 2023-11-16T16:58:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.538+0000 7fb07e7fc700 1 --2- 172.21.15.44:0/966096532 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557a15f57f40 0x557a15f5a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.539+0000 7fb07e7fc700 1 --2- 172.21.15.44:0/966096532 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557a15f57f40 0x557a15f5a350 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=633e1e2d7967afc server_cookie=f465d9ea43f7ca1d in_seq=0 out_seq=0 2023-11-16T16:58:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.539+0000 7fb062ff5700 10 client.26987 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.539+0000 7fb062ff5700 1 -- 172.21.15.44:0/966096532 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557a15e52040 con 0x557a15f57f40 2023-11-16T16:58:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.539+0000 7fb062ff5700 10 client.26987 handle_command_reply: tid=0 2023-11-16T16:58:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.539+0000 7fb098e6db80 1 client.26987 shutdown 2023-11-16T16:58:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.540+0000 7fb098e6db80 1 -- 172.21.15.44:0/966096532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15d9a6b0 msgr2=0x557a15f4aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.540+0000 7fb098e6db80 1 --2- 172.21.15.44:0/966096532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15d9a6b0 0x557a15f4aa10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb0780115d0 tx=0x7fb0780041a0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.540+0000 7fb098e6db80 1 -- 172.21.15.44:0/966096532 shutdown_connections 2023-11-16T16:58:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.540+0000 7fb098e6db80 1 --2- 172.21.15.44:0/966096532 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557a15f57f40 0x557a15f5a350 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.540+0000 7fb098e6db80 1 --2- 172.21.15.44:0/966096532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15f44b90 0x557a15f482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.540+0000 7fb098e6db80 1 --2- 172.21.15.44:0/966096532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15d9a6b0 0x557a15f4aa10 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.540+0000 7fb098e6db80 1 --2- 172.21.15.44:0/966096532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15f44720 0x557a15f47c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.540+0000 7fb062ff5700 0 client.26987 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.540+0000 7fb098e6db80 1 -- 172.21.15.44:0/966096532 >> 172.21.15.44:0/966096532 conn(0x557a15f33f50 msgr2=0x557a15f387e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.540+0000 7fb098e6db80 1 -- 172.21.15.44:0/966096532 shutdown_connections 2023-11-16T16:58:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.540+0000 7fb098e6db80 1 -- 172.21.15.44:0/966096532 wait complete. 2023-11-16T16:58:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.540+0000 7fb098e6db80 20 client.26987 trim_cache size 0 max 16384 2023-11-16T16:58:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.558+0000 7fb098e6db80 1 Processor -- start 2023-11-16T16:58:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.558+0000 7fb098e6db80 1 -- start start 2023-11-16T16:58:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.558+0000 7fb098e6db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15d9a6b0 0x557a15f41260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.558+0000 7fb098e6db80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 0x557a15f54280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.558+0000 7fb098e6db80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15f44b90 0x557a15f38f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.558+0000 7fb098e6db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a15f42940 con 0x557a15d9a6b0 2023-11-16T16:58:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.558+0000 7fb098e6db80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a15f3a2b0 con 0x557a15f44720 2023-11-16T16:58:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.558+0000 7fb098e6db80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a15f43d60 con 0x557a15f44b90 2023-11-16T16:58:07.567 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.559+0000 7fb07f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15d9a6b0 0x557a15f41260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.567 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.559+0000 7fb07e7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 0x557a15f54280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.567 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.559+0000 7fb07f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15d9a6b0 0x557a15f41260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57304/0 (socket says 172.21.15.44:57304) 2023-11-16T16:58:07.567 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.559+0000 7fb07f7fe700 1 -- 172.21.15.44:0/2242245610 learned_addr learned my addr 172.21.15.44:0/2242245610 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.559+0000 7fb07e7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 0x557a15f54280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:45584/0 (socket says 172.21.15.44:45584) 2023-11-16T16:58:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.559+0000 7fb07f7fe700 1 --2- 172.21.15.44:0/2242245610 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15f44b90 0x557a15f38f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.559+0000 7fb07e7fc700 1 -- 172.21.15.44:0/2242245610 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15f44b90 msgr2=0x557a15f38f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.559+0000 7fb07e7fc700 1 --2- 172.21.15.44:0/2242245610 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15f44b90 0x557a15f38f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.559+0000 7fb07e7fc700 1 -- 172.21.15.44:0/2242245610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15d9a6b0 msgr2=0x557a15f41260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.559+0000 7fb07e7fc700 1 --2- 172.21.15.44:0/2242245610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15d9a6b0 0x557a15f41260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.559+0000 7fb07e7fc700 1 -- 172.21.15.44:0/2242245610 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a15f356f0 con 0x557a15f44720 2023-11-16T16:58:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.559+0000 7fb07f7fe700 1 --2- 172.21.15.44:0/2242245610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15d9a6b0 0x557a15f41260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.560+0000 7fb07e7fc700 1 --2- 172.21.15.44:0/2242245610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 0x557a15f54280 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x557a15f5abc0 tx=0x7fb074009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=279175c95f5292fc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.560+0000 7fb065ffb700 1 -- 172.21.15.44:0/2242245610 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb074016040 con 0x557a15f44720 2023-11-16T16:58:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.560+0000 7fb065ffb700 1 -- 172.21.15.44:0/2242245610 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb074017040 con 0x557a15f44720 2023-11-16T16:58:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.560+0000 7fb065ffb700 1 -- 172.21.15.44:0/2242245610 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb074004cd0 con 0x557a15f44720 2023-11-16T16:58:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.561+0000 7fb098e6db80 1 -- 172.21.15.44:0/2242245610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 msgr2=0x557a15f54280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.561+0000 7fb098e6db80 1 --2- 172.21.15.44:0/2242245610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 0x557a15f54280 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x557a15f5abc0 tx=0x7fb074009260 comp rx=0 tx=0).stop 2023-11-16T16:58:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.561+0000 7fb098e6db80 1 -- 172.21.15.44:0/2242245610 shutdown_connections 2023-11-16T16:58:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.561+0000 7fb098e6db80 1 --2- 172.21.15.44:0/2242245610 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15f44b90 0x557a15f38f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.561+0000 7fb098e6db80 1 --2- 172.21.15.44:0/2242245610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15d9a6b0 0x557a15f41260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.561+0000 7fb098e6db80 1 --2- 172.21.15.44:0/2242245610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 0x557a15f54280 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.561+0000 7fb098e6db80 1 -- 172.21.15.44:0/2242245610 >> 172.21.15.44:0/2242245610 conn(0x557a15f33f50 msgr2=0x557a15f3aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.561+0000 7fb098e6db80 1 -- 172.21.15.44:0/2242245610 shutdown_connections 2023-11-16T16:58:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.561+0000 7fb098e6db80 1 -- 172.21.15.44:0/2242245610 wait complete. 2023-11-16T16:58:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.561+0000 7fb098e6db80 1 -- start start 2023-11-16T16:58:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.562+0000 7fb098e6db80 1 Processor -- start 2023-11-16T16:58:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.562+0000 7fb098e6db80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15d9a6b0 0x557a15f4f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.562+0000 7fb098e6db80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 0x557a15f4d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.562+0000 7fb098e6db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15f44b90 0x557a15f4d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.562+0000 7fb098e6db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a15f4dda0 con 0x557a15f44b90 2023-11-16T16:58:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.562+0000 7fb098e6db80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a15f4df10 con 0x557a15f44720 2023-11-16T16:58:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.562+0000 7fb098e6db80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a15faadc0 con 0x557a15d9a6b0 2023-11-16T16:58:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.562+0000 7fb098e6db80 10 client.0 _ll_register_callbacks cb 0x7fb0700d1910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.563+0000 7fb07e7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 0x557a15f4d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.563+0000 7fb07f7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15d9a6b0 0x557a15f4f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.563+0000 7fb07f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15f44b90 0x557a15f4d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.563+0000 7fb07e7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 0x557a15f4d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:45600/0 (socket says 172.21.15.44:45600) 2023-11-16T16:58:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.563+0000 7fb07e7fc700 1 -- 172.21.15.44:0/60979079 learned_addr learned my addr 172.21.15.44:0/60979079 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.563+0000 7fb07e7fc700 1 -- 172.21.15.44:0/60979079 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15d9a6b0 msgr2=0x557a15f4f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.563+0000 7fb07e7fc700 1 --2- 172.21.15.44:0/60979079 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15d9a6b0 0x557a15f4f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.563+0000 7fb07e7fc700 1 -- 172.21.15.44:0/60979079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15f44b90 msgr2=0x557a15f4d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.563+0000 7fb07e7fc700 1 --2- 172.21.15.44:0/60979079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15f44b90 0x557a15f4d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.563+0000 7fb07e7fc700 1 -- 172.21.15.44:0/60979079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557a15e56130 con 0x557a15f44720 2023-11-16T16:58:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.563+0000 7fb07f7fe700 1 --2- 172.21.15.44:0/60979079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15f44b90 0x557a15f4d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.564+0000 7fb07e7fc700 1 --2- 172.21.15.44:0/60979079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 0x557a15f4d110 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb074005920 tx=0x7fb074011c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab57fa020c600786 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.564+0000 7fb062ff5700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:58:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.564+0000 7fb062ff5700 1 -- 172.21.15.44:0/60979079 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0740047c0 con 0x557a15f44720 2023-11-16T16:58:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.564+0000 7fb098e6db80 1 -- 172.21.15.44:0/60979079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb07800f040 con 0x557a15f44720 2023-11-16T16:58:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.564+0000 7fb062ff5700 1 -- 172.21.15.44:0/60979079 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fb074004920 con 0x557a15f44720 2023-11-16T16:58:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.564+0000 7fb062ff5700 1 -- 172.21.15.44:0/60979079 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb074004ae0 con 0x557a15f44720 2023-11-16T16:58:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.564+0000 7fb098e6db80 10 client.26993 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.564+0000 7fb098e6db80 1 -- 172.21.15.44:0/60979079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x7fb074004520 con 0x557a15f44720 2023-11-16T16:58:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.564+0000 7fb062ff5700 1 -- 172.21.15.44:0/60979079 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb07800f040 con 0x557a15f44720 2023-11-16T16:58:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.565+0000 7fb062ff5700 1 -- 172.21.15.44:0/60979079 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb074003710 con 0x557a15f44720 2023-11-16T16:58:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.565+0000 7fb098e6db80 10 client.26993 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.565+0000 7fb098e6db80 10 client.26993 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.565+0000 7fb098e6db80 20 client.26993 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.565+0000 7fb098e6db80 1 --2- 172.21.15.44:0/60979079 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557a15fac390 0x557a15fae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.565+0000 7fb098e6db80 4 client.26993 mds_command: new command op to 24551 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:58:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.565+0000 7fb098e6db80 1 -- 172.21.15.44:0/60979079 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557a15e5e600 con 0x557a15fac390 2023-11-16T16:58:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.565+0000 7fb07f7fe700 1 --2- 172.21.15.44:0/60979079 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557a15fac390 0x557a15fae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.566+0000 7fb062ff5700 10 client.26993 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.565+0000 7fb07f7fe700 1 --2- 172.21.15.44:0/60979079 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557a15fac390 0x557a15fae7a0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aad64a8c0a19ece5 server_cookie=23a84506e9f6ef40 in_seq=0 out_seq=0 2023-11-16T16:58:07.607 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir -p dir 2023-11-16T16:58:07.611 INFO:tasks.workunit.client.0.smithi044.stderr:+ getfattr -n ceph.dir.pin dir 2023-11-16T16:58:07.612 INFO:tasks.workunit.client.0.smithi044.stderr:+ grep 'ceph.dir.pin="-1"' 2023-11-16T16:58:07.614 INFO:tasks.workunit.client.0.smithi044.stdout:ceph.dir.pin="-1" 2023-11-16T16:58:07.618 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -n ceph.dir.pin dir 2023-11-16T16:58:07.618 INFO:tasks.workunit.client.0.smithi044.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-11-16T16:58:07.621 INFO:tasks.workunit.client.0.smithi044.stdout:setfattr: dir: Invalid argument 2023-11-16T16:58:07.621 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -n ceph.dir.pin -v 1 dir 2023-11-16T16:58:07.630 INFO:tasks.workunit.client.0.smithi044.stderr:+ getfattr -n ceph.dir.pin dir 2023-11-16T16:58:07.630 INFO:tasks.workunit.client.0.smithi044.stderr:+ grep 'ceph.dir.pin="1"' 2023-11-16T16:58:07.632 INFO:tasks.workunit.client.0.smithi044.stdout:ceph.dir.pin="1" 2023-11-16T16:58:07.632 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -x ceph.dir.pin dir 2023-11-16T16:58:07.638 INFO:tasks.workunit.client.0.smithi044.stderr:+ getfattr -n ceph.dir.pin dir 2023-11-16T16:58:07.638 INFO:tasks.workunit.client.0.smithi044.stderr:+ grep 'ceph.dir.pin="-1"' 2023-11-16T16:58:07.642 INFO:tasks.workunit.client.0.smithi044.stdout:ceph.dir.pin="-1" 2023-11-16T16:58:07.643 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -n ceph.dir.pin.distributed dir 2023-11-16T16:58:07.644 INFO:tasks.workunit.client.0.smithi044.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-11-16T16:58:07.646 INFO:tasks.workunit.client.0.smithi044.stdout:setfattr: dir: Invalid argument 2023-11-16T16:58:07.646 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -n ceph.dir.pin.distributed -v 1 dir 2023-11-16T16:58:07.649 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -x ceph.dir.pin.distributed dir 2023-11-16T16:58:07.653 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -n ceph.dir.pin.random dir 2023-11-16T16:58:07.653 INFO:tasks.workunit.client.0.smithi044.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-11-16T16:58:07.655 INFO:tasks.workunit.client.0.smithi044.stdout:setfattr: dir: Invalid argument 2023-11-16T16:58:07.656 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -n ceph.dir.pin.random -v 0.01 dir 2023-11-16T16:58:07.659 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -x ceph.dir.pin.random dir 2023-11-16T16:58:07.662 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -n ceph.dir.subvolume dir 2023-11-16T16:58:07.662 INFO:tasks.workunit.client.0.smithi044.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-11-16T16:58:07.664 INFO:tasks.workunit.client.0.smithi044.stdout:setfattr: dir: Invalid argument 2023-11-16T16:58:07.665 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -n ceph.dir.subvolume -v 1 dir 2023-11-16T16:58:07.674 INFO:tasks.workunit.client.0.smithi044.stderr:setfattr: dir: Invalid argument 2023-11-16T16:58:07.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T16:58:07.675 INFO:tasks.workunit:Stopping ['fs/misc'] on client.0... 2023-11-16T16:58:07.675 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-11-16T16:58:07.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.693+0000 7fb062ff5700 1 -- 172.21.15.44:0/60979079 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557a15e5e600 con 0x557a15fac390 2023-11-16T16:58:07.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.693+0000 7fb062ff5700 10 client.26993 handle_command_reply: tid=0 2023-11-16T16:58:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.693+0000 7fb098e6db80 1 client.26993 shutdown 2023-11-16T16:58:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.694+0000 7fb098e6db80 1 -- 172.21.15.44:0/60979079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 msgr2=0x557a15f4d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.694+0000 7fb098e6db80 1 --2- 172.21.15.44:0/60979079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 0x557a15f4d110 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb074005920 tx=0x7fb074011c10 comp rx=0 tx=0).stop 2023-11-16T16:58:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2023-11-16T16:58:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2023-11-16T16:58:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2023-11-16T16:58:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2023-11-16T16:58:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.694+0000 7fb098e6db80 1 -- 172.21.15.44:0/60979079 shutdown_connections 2023-11-16T16:58:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.694+0000 7fb098e6db80 1 --2- 172.21.15.44:0/60979079 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557a15fac390 0x557a15fae7a0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.694+0000 7fb098e6db80 1 --2- 172.21.15.44:0/60979079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557a15f44b90 0x557a15f4d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.694+0000 7fb098e6db80 1 --2- 172.21.15.44:0/60979079 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x557a15d9a6b0 0x557a15f4f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.694+0000 7fb098e6db80 1 --2- 172.21.15.44:0/60979079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557a15f44720 0x557a15f4d110 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.694+0000 7fb062ff5700 0 client.26993 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.694+0000 7fb098e6db80 1 -- 172.21.15.44:0/60979079 >> 172.21.15.44:0/60979079 conn(0x557a15f33f50 msgr2=0x557a15f54280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.694+0000 7fb098e6db80 1 -- 172.21.15.44:0/60979079 shutdown_connections 2023-11-16T16:58:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.694+0000 7fb098e6db80 1 -- 172.21.15.44:0/60979079 wait complete. 2023-11-16T16:58:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.694+0000 7fb098e6db80 20 client.26993 trim_cache size 0 max 16384 2023-11-16T16:58:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.695+0000 7fb0855ff700 1 -- 172.21.15.44:0/1457826444 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fb0680a12c0 msgr2=0x7fb0680a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.695+0000 7fb0855ff700 1 --2- 172.21.15.44:0/1457826444 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fb0680a12c0 0x7fb0680a3770 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb0700cb350 tx=0x7fb0700d1a40 comp rx=0 tx=0).stop 2023-11-16T16:58:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.695+0000 7fb0855ff700 1 -- 172.21.15.44:0/1457826444 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb080100c50 msgr2=0x7fb080072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.695+0000 7fb0855ff700 1 --2- 172.21.15.44:0/1457826444 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb080100c50 0x7fb080072df0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb07000b580 tx=0x7fb07000f690 comp rx=0 tx=0).stop 2023-11-16T16:58:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.695+0000 7fb0855ff700 1 -- 172.21.15.44:0/1457826444 shutdown_connections 2023-11-16T16:58:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.695+0000 7fb0855ff700 1 --2- 172.21.15.44:0/1457826444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb080105560 0x7fb0801a4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.695+0000 7fb0855ff700 1 --2- 172.21.15.44:0/1457826444 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fb0680a12c0 0x7fb0680a3770 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.695+0000 7fb0855ff700 1 --2- 172.21.15.44:0/1457826444 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb080100c50 0x7fb080072df0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.695+0000 7fb0855ff700 1 --2- 172.21.15.44:0/1457826444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0801015f0 0x7fb080073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.695+0000 7fb0855ff700 1 -- 172.21.15.44:0/1457826444 >> 172.21.15.44:0/1457826444 conn(0x7fb080077550 msgr2=0x7fb0800fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.696+0000 7fb0855ff700 1 -- 172.21.15.44:0/1457826444 shutdown_connections 2023-11-16T16:58:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2023-11-16T16:58:07.696+0000 7fb0855ff700 1 -- 172.21.15.44:0/1457826444 wait complete. 2023-11-16T16:58:07.759 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:07 smithi044 ceph-mon[128750]: pgmap v252: 129 pgs: 129 active+clean; 5.9 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 149 KiB/s wr, 301 op/s 2023-11-16T16:58:07.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:07 smithi176 ceph-mon[131630]: pgmap v252: 129 pgs: 129 active+clean; 5.9 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 149 KiB/s wr, 301 op/s 2023-11-16T16:58:07.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:07 smithi077 ceph-mon[131204]: pgmap v252: 129 pgs: 129 active+clean; 5.9 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 149 KiB/s wr, 301 op/s 2023-11-16T16:58:08.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:08 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:58:08.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:08 smithi176 ceph-mon[131630]: osdmap e97: 12 total, 12 up, 12 in 2023-11-16T16:58:08.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:08 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:58:08.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:08 smithi077 ceph-mon[131204]: osdmap e97: 12 total, 12 up, 12 in 2023-11-16T16:58:09.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:08 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:58:09.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:08 smithi044 ceph-mon[128750]: osdmap e97: 12 total, 12 up, 12 in 2023-11-16T16:58:09.149 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh' 2023-11-16T16:58:09.237 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7a0a097208514f73992dbf62c029bd9c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh' 2023-11-16T16:58:09.239 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2023-11-16T16:58:09.254 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi044.front.sepia.ceph.com 2023-11-16T16:58:09.266 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2023-11-16T16:58:09.278 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2023-11-16T16:58:09.278 INFO:teuthology.task.exec:Executing custom commands... 2023-11-16T16:58:09.279 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2023-11-16T16:58:09.279 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2023-11-16T16:58:09.394 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2023-11-16T16:58:09.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:09 smithi044 ceph-mon[128750]: pgmap v254: 129 pgs: 129 active+clean; 5.9 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 74 KiB/s wr, 60 op/s 2023-11-16T16:58:09.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:09 smithi176 ceph-mon[131630]: pgmap v254: 129 pgs: 129 active+clean; 5.9 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 74 KiB/s wr, 60 op/s 2023-11-16T16:58:09.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:09 smithi077 ceph-mon[131204]: pgmap v254: 129 pgs: 129 active+clean; 5.9 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 74 KiB/s wr, 60 op/s 2023-11-16T16:58:11.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:11 smithi176 ceph-mon[131630]: pgmap v255: 129 pgs: 129 active+clean; 5.9 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-11-16T16:58:11.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:11 smithi077 ceph-mon[131204]: pgmap v255: 129 pgs: 129 active+clean; 5.9 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-11-16T16:58:12.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:11 smithi044 ceph-mon[128750]: pgmap v255: 129 pgs: 129 active+clean; 5.9 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-11-16T16:58:13.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:13 smithi176 ceph-mon[131630]: pgmap v256: 129 pgs: 129 active+clean; 5.9 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 69 KiB/s wr, 26 op/s 2023-11-16T16:58:13.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:13 smithi077 ceph-mon[131204]: pgmap v256: 129 pgs: 129 active+clean; 5.9 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 69 KiB/s wr, 26 op/s 2023-11-16T16:58:14.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:13 smithi044 ceph-mon[128750]: pgmap v256: 129 pgs: 129 active+clean; 5.9 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 69 KiB/s wr, 26 op/s 2023-11-16T16:58:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:15 smithi176 ceph-mon[131630]: pgmap v257: 129 pgs: 129 active+clean; 6.0 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 10 op/s 2023-11-16T16:58:15.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:15 smithi077 ceph-mon[131204]: pgmap v257: 129 pgs: 129 active+clean; 6.0 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 10 op/s 2023-11-16T16:58:16.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:15 smithi044 ceph-mon[128750]: pgmap v257: 129 pgs: 129 active+clean; 6.0 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 10 op/s 2023-11-16T16:58:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:17 smithi176 ceph-mon[131630]: pgmap v258: 129 pgs: 129 active+clean; 6.0 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 10 op/s 2023-11-16T16:58:17.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:17 smithi077 ceph-mon[131204]: pgmap v258: 129 pgs: 129 active+clean; 6.0 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 10 op/s 2023-11-16T16:58:18.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:17 smithi044 ceph-mon[128750]: pgmap v258: 129 pgs: 129 active+clean; 6.0 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 10 op/s 2023-11-16T16:58:19.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:19 smithi176 ceph-mon[131630]: pgmap v259: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-11-16T16:58:19.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:19 smithi176 ceph-mon[131630]: osdmap e98: 12 total, 12 up, 12 in 2023-11-16T16:58:19.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:19 smithi077 ceph-mon[131204]: pgmap v259: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-11-16T16:58:19.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:19 smithi077 ceph-mon[131204]: osdmap e98: 12 total, 12 up, 12 in 2023-11-16T16:58:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:19 smithi044 ceph-mon[128750]: pgmap v259: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-11-16T16:58:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:19 smithi044 ceph-mon[128750]: osdmap e98: 12 total, 12 up, 12 in 2023-11-16T16:58:21.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:21 smithi176 ceph-mon[131630]: pgmap v261: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-11-16T16:58:21.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:21 smithi077 ceph-mon[131204]: pgmap v261: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-11-16T16:58:21.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:21 smithi044 ceph-mon[128750]: pgmap v261: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-11-16T16:58:23.645 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:23 smithi176 ceph-mon[131630]: pgmap v262: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-11-16T16:58:23.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:23 smithi077 ceph-mon[131204]: pgmap v262: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-11-16T16:58:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:23 smithi044 ceph-mon[128750]: pgmap v262: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-11-16T16:58:25.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:25 smithi077 ceph-mon[131204]: pgmap v263: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.4 KiB/s wr, 2 op/s 2023-11-16T16:58:25.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:25 smithi044 ceph-mon[128750]: pgmap v263: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.4 KiB/s wr, 2 op/s 2023-11-16T16:58:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:25 smithi176 ceph-mon[131630]: pgmap v263: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.4 KiB/s wr, 2 op/s 2023-11-16T16:58:27.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:27 smithi077 ceph-mon[131204]: pgmap v264: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.4 KiB/s wr, 2 op/s 2023-11-16T16:58:27.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:27 smithi044 ceph-mon[128750]: pgmap v264: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.4 KiB/s wr, 2 op/s 2023-11-16T16:58:27.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:27 smithi176 ceph-mon[131630]: pgmap v264: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.4 KiB/s wr, 2 op/s 2023-11-16T16:58:29.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:29 smithi077 ceph-mon[131204]: pgmap v265: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-11-16T16:58:29.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:29 smithi044 ceph-mon[128750]: pgmap v265: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-11-16T16:58:29.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:29 smithi176 ceph-mon[131630]: pgmap v265: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-11-16T16:58:31.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:31 smithi077 ceph-mon[131204]: pgmap v266: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 1 op/s 2023-11-16T16:58:31.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:31 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:58:31.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:31 smithi044 ceph-mon[128750]: pgmap v266: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 1 op/s 2023-11-16T16:58:31.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:31 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:58:31.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:31 smithi176 ceph-mon[131630]: pgmap v266: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 1 op/s 2023-11-16T16:58:31.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:31 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:58:32.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:32 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:58:32.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:32 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:58:32.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:32 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:58:32.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:32 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:58:32.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:32 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:58:32.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:32 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:58:32.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:32 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:58:32.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:32 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:58:32.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:32 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:58:33.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:33 smithi077 ceph-mon[131204]: pgmap v267: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1023 B/s wr, 1 op/s 2023-11-16T16:58:33.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:33 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:58:33.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:33 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T16:58:33.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:33 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:58:33.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:33 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:58:33.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:33 smithi044 ceph-mon[128750]: pgmap v267: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1023 B/s wr, 1 op/s 2023-11-16T16:58:33.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:33 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:58:33.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:33 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T16:58:33.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:33 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:58:33.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:33 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:58:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:33 smithi176 ceph-mon[131630]: pgmap v267: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1023 B/s wr, 1 op/s 2023-11-16T16:58:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:33 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:58:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:33 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T16:58:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:33 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:58:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:33 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:58:34.170 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:58:34.409 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.407+0000 7fed0b1ff700 1 -- 172.21.15.44:0/3201759070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed04101660 msgr2=0x7fed04104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:34.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.407+0000 7fed0b1ff700 1 --2- 172.21.15.44:0/3201759070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed04101660 0x7fed04104ef0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fecf00099c0 tx=0x7fecf0009cd0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.407+0000 7fed0b1ff700 1 -- 172.21.15.44:0/3201759070 shutdown_connections 2023-11-16T16:58:34.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.407+0000 7fed0b1ff700 1 --2- 172.21.15.44:0/3201759070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed04105620 0x7fed0410d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.407+0000 7fed0b1ff700 1 --2- 172.21.15.44:0/3201759070 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed04100cc0 0x7fed04101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.407+0000 7fed0b1ff700 1 --2- 172.21.15.44:0/3201759070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed04101660 0x7fed04104ef0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.407+0000 7fed0b1ff700 1 -- 172.21.15.44:0/3201759070 >> 172.21.15.44:0/3201759070 conn(0x7fed04077560 msgr2=0x7fed04077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:34.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.408+0000 7fed0b1ff700 1 -- 172.21.15.44:0/3201759070 shutdown_connections 2023-11-16T16:58:34.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.408+0000 7fed0b1ff700 1 -- 172.21.15.44:0/3201759070 wait complete. 2023-11-16T16:58:34.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.408+0000 7fed0b1ff700 1 Processor -- start 2023-11-16T16:58:34.412 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.409+0000 7fed0b1ff700 1 -- start start 2023-11-16T16:58:34.415 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.409+0000 7fed0b1ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed04100cc0 0x7fed04072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.415 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.409+0000 7fed0b1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed04101660 0x7fed04073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.415 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.409+0000 7fed0b1ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed04105620 0x7fed041a4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.415 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.409+0000 7fed0b1ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed04073b60 con 0x7fed04100cc0 2023-11-16T16:58:34.415 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.409+0000 7fed0b1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed041a4c40 con 0x7fed04101660 2023-11-16T16:58:34.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.409+0000 7fed0b1ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed041a4d80 con 0x7fed04105620 2023-11-16T16:58:34.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.409+0000 7fed08f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed04100cc0 0x7fed04072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.409+0000 7fed08f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed04100cc0 0x7fed04072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43362/0 (socket says 172.21.15.44:43362) 2023-11-16T16:58:34.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.409+0000 7fed08f9b700 1 -- 172.21.15.44:0/1408316901 learned_addr learned my addr 172.21.15.44:0/1408316901 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:34.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.409+0000 7fed03fff700 1 --2- 172.21.15.44:0/1408316901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed04101660 0x7fed04073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.410+0000 7fed0979c700 1 --2- 172.21.15.44:0/1408316901 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed04105620 0x7fed041a4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.410+0000 7fed08f9b700 1 -- 172.21.15.44:0/1408316901 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed04105620 msgr2=0x7fed041a4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:34.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.410+0000 7fed08f9b700 1 --2- 172.21.15.44:0/1408316901 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed04105620 0x7fed041a4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.410+0000 7fed08f9b700 1 -- 172.21.15.44:0/1408316901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed04101660 msgr2=0x7fed04073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:34.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.410+0000 7fed08f9b700 1 --2- 172.21.15.44:0/1408316901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed04101660 0x7fed04073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.410+0000 7fed08f9b700 1 -- 172.21.15.44:0/1408316901 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc009c40 con 0x7fed04100cc0 2023-11-16T16:58:34.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.410+0000 7fed08f9b700 1 --2- 172.21.15.44:0/1408316901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed04100cc0 0x7fed04072df0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fecf800e9a0 tx=0x7fecf800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:34.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.410+0000 7fed01ffb700 1 -- 172.21.15.44:0/1408316901 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf8009790 con 0x7fed04100cc0 2023-11-16T16:58:34.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.410+0000 7fed01ffb700 1 -- 172.21.15.44:0/1408316901 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fecf800cbd0 con 0x7fed04100cc0 2023-11-16T16:58:34.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.410+0000 7fed0b1ff700 1 -- 172.21.15.44:0/1408316901 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf00095f0 con 0x7fed04100cc0 2023-11-16T16:58:34.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.410+0000 7fed01ffb700 1 -- 172.21.15.44:0/1408316901 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf8018680 con 0x7fed04100cc0 2023-11-16T16:58:34.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.410+0000 7fed0b1ff700 1 -- 172.21.15.44:0/1408316901 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed04110370 con 0x7fed04100cc0 2023-11-16T16:58:34.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.414+0000 7fed01ffb700 1 -- 172.21.15.44:0/1408316901 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fecf800cd40 con 0x7fed04100cc0 2023-11-16T16:58:34.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.415+0000 7fed01ffb700 1 --2- 172.21.15.44:0/1408316901 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fecf40a12c0 0x7fecf40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.415+0000 7fed1eabeb80 1 Processor -- start 2023-11-16T16:58:34.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.415+0000 7fed1eabeb80 1 -- start start 2023-11-16T16:58:34.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.415+0000 7fed01ffb700 1 -- 172.21.15.44:0/1408316901 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fecf8014070 con 0x7fed04100cc0 2023-11-16T16:58:34.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.415+0000 7fed03fff700 1 --2- 172.21.15.44:0/1408316901 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fecf40a12c0 0x7fecf40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.416+0000 7fed1eabeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9d91f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.416+0000 7fed1eabeb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d92470 0x5633a9d96370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.416+0000 7fed1eabeb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d968b0 0x5633a9d96d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.416+0000 7fed1eabeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633a9c97060 con 0x5633a9d91b60 2023-11-16T16:58:34.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.416+0000 7fed1eabeb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633a9ce98a0 con 0x5633a9d968b0 2023-11-16T16:58:34.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.416+0000 7fed1eabeb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633a9ce2d30 con 0x5633a9d92470 2023-11-16T16:58:34.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.416+0000 7fed08f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9d91f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.416+0000 7fed08f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9d91f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43366/0 (socket says 172.21.15.44:43366) 2023-11-16T16:58:34.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.416+0000 7fed08f9b700 1 -- 172.21.15.44:0/4011468568 learned_addr learned my addr 172.21.15.44:0/4011468568 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:34.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.416+0000 7fed0979c700 1 --2- 172.21.15.44:0/4011468568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d968b0 0x5633a9d96d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.416+0000 7fed08f9b700 1 -- 172.21.15.44:0/4011468568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d92470 msgr2=0x5633a9d96370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:34.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.416+0000 7fed03fff700 1 --2- 172.21.15.44:0/4011468568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d92470 0x5633a9d96370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.417+0000 7fed08f9b700 1 --2- 172.21.15.44:0/4011468568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d92470 0x5633a9d96370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.417+0000 7fed08f9b700 1 -- 172.21.15.44:0/4011468568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d968b0 msgr2=0x5633a9d96d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:34.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.417+0000 7fed08f9b700 1 --2- 172.21.15.44:0/4011468568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d968b0 0x5633a9d96d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.417+0000 7fed08f9b700 1 -- 172.21.15.44:0/4011468568 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5633a9b64630 con 0x5633a9d91b60 2023-11-16T16:58:34.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.417+0000 7fed03fff700 1 --2- 172.21.15.44:0/4011468568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d92470 0x5633a9d96370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:58:34.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.417+0000 7fed0979c700 1 --2- 172.21.15.44:0/4011468568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d968b0 0x5633a9d96d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:34.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.417+0000 7fed03fff700 1 --2- 172.21.15.44:0/1408316901 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fecf40a12c0 0x7fecf40a3770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fecf000b580 tx=0x7fecf001a040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:34.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.417+0000 7fed08f9b700 1 --2- 172.21.15.44:0/4011468568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9d91f30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fecf800f010 tx=0x7fecf809a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ff996dcf6913e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:34.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.418+0000 7feceeffd700 1 -- 172.21.15.44:0/4011468568 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf80cd160 con 0x5633a9d91b60 2023-11-16T16:58:34.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.418+0000 7feceeffd700 1 -- 172.21.15.44:0/4011468568 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fecf8096820 con 0x5633a9d91b60 2023-11-16T16:58:34.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.418+0000 7feceeffd700 1 -- 172.21.15.44:0/4011468568 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf8099860 con 0x5633a9d91b60 2023-11-16T16:58:34.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.419+0000 7fed1eabeb80 1 -- 172.21.15.44:0/4011468568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 msgr2=0x5633a9d91f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:34.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.419+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/4011468568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9d91f30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fecf800f010 tx=0x7fecf809a1e0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.419+0000 7fed1eabeb80 1 -- 172.21.15.44:0/4011468568 shutdown_connections 2023-11-16T16:58:34.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.419+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/4011468568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d968b0 0x5633a9d96d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.419+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/4011468568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9d91f30 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.419+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/4011468568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d92470 0x5633a9d96370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.419+0000 7fed1eabeb80 1 -- 172.21.15.44:0/4011468568 >> 172.21.15.44:0/4011468568 conn(0x5633a9d87360 msgr2=0x5633a9d8cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:34.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.419+0000 7fed1eabeb80 1 -- 172.21.15.44:0/4011468568 shutdown_connections 2023-11-16T16:58:34.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.419+0000 7fed1eabeb80 1 -- 172.21.15.44:0/4011468568 wait complete. 2023-11-16T16:58:34.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.420+0000 7fed1eabeb80 1 -- start start 2023-11-16T16:58:34.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.420+0000 7fed1eabeb80 1 Processor -- start 2023-11-16T16:58:34.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed1eabeb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d91b60 0x5633a9d9e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed1eabeb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d92470 0x5633a9d9b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed1eabeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d968b0 0x5633a9d9ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed1eabeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633a9c848e0 con 0x5633a9d968b0 2023-11-16T16:58:34.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed1eabeb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633a9c84580 con 0x5633a9d91b60 2023-11-16T16:58:34.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed1eabeb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633a9b66920 con 0x5633a9d92470 2023-11-16T16:58:34.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed1eabeb80 10 client.0 _ll_register_callbacks cb 0x5633a9ca2780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:34.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed08f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d91b60 0x5633a9d9e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed08f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d91b60 0x5633a9d9e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:41648/0 (socket says 172.21.15.44:41648) 2023-11-16T16:58:34.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed03fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d92470 0x5633a9d9b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed08f9b700 1 -- 172.21.15.44:0/1667450762 learned_addr learned my addr 172.21.15.44:0/1667450762 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:34.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed0979c700 1 --2- 172.21.15.44:0/1667450762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d968b0 0x5633a9d9ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed08f9b700 1 -- 172.21.15.44:0/1667450762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d92470 msgr2=0x5633a9d9b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:34.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed08f9b700 1 --2- 172.21.15.44:0/1667450762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d92470 0x5633a9d9b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed08f9b700 1 -- 172.21.15.44:0/1667450762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d968b0 msgr2=0x5633a9d9ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:34.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed08f9b700 1 --2- 172.21.15.44:0/1667450762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d968b0 0x5633a9d9ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.421+0000 7fed08f9b700 1 -- 172.21.15.44:0/1667450762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed04110370 con 0x5633a9d91b60 2023-11-16T16:58:34.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.422+0000 7fed08f9b700 1 --2- 172.21.15.44:0/1667450762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d91b60 0x5633a9d9e0d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fecf80cad10 tx=0x7fecf8099d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1dc1a7b2b2fefe5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:34.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.422+0000 7fece3fff700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:58:34.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.422+0000 7fece3fff700 1 -- 172.21.15.44:0/1667450762 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf809a580 con 0x5633a9d91b60 2023-11-16T16:58:34.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.422+0000 7fece3fff700 1 -- 172.21.15.44:0/1667450762 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fecf80a20d0 con 0x5633a9d91b60 2023-11-16T16:58:34.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.422+0000 7fed1eabeb80 1 -- 172.21.15.44:0/1667450762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5633a9d84b20 con 0x5633a9d91b60 2023-11-16T16:58:34.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.422+0000 7fece3fff700 1 -- 172.21.15.44:0/1667450762 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecf80a2290 con 0x5633a9d91b60 2023-11-16T16:58:34.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.422+0000 7fed1eabeb80 10 client.27005 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:34.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.422+0000 7fed1eabeb80 1 -- 172.21.15.44:0/1667450762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5633a9ca9190 con 0x5633a9d91b60 2023-11-16T16:58:34.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.422+0000 7fece3fff700 1 -- 172.21.15.44:0/1667450762 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5633a9d84b20 con 0x5633a9d91b60 2023-11-16T16:58:34.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.422+0000 7fece3fff700 1 -- 172.21.15.44:0/1667450762 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fecf8099070 con 0x5633a9d91b60 2023-11-16T16:58:34.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.422+0000 7fed1eabeb80 10 client.27005 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:34.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.422+0000 7fed1eabeb80 10 client.27005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:34.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.423+0000 7fed1eabeb80 20 client.27005 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:34.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.423+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/1667450762 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5633a9dab440 0x5633a9dad820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.423+0000 7fed1eabeb80 4 client.27005 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:58:34.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.423+0000 7fed1eabeb80 1 -- 172.21.15.44:0/1667450762 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5633a9ca50a0 con 0x5633a9dab440 2023-11-16T16:58:34.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.423+0000 7fed03fff700 1 --2- 172.21.15.44:0/1667450762 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5633a9dab440 0x5633a9dad820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.423+0000 7fed03fff700 1 --2- 172.21.15.44:0/1667450762 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5633a9dab440 0x5633a9dad820 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9e40f496fc2da35 server_cookie=8b516bd5efba084f in_seq=0 out_seq=0 2023-11-16T16:58:34.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.423+0000 7fece3fff700 10 client.27005 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:34.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.424+0000 7fece3fff700 1 -- 172.21.15.44:0/1667450762 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5633a9ca50a0 con 0x5633a9dab440 2023-11-16T16:58:34.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.424+0000 7fece3fff700 10 client.27005 handle_command_reply: tid=0 2023-11-16T16:58:34.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.424+0000 7fed1eabeb80 1 client.27005 shutdown 2023-11-16T16:58:34.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.424+0000 7fed1eabeb80 1 -- 172.21.15.44:0/1667450762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d91b60 msgr2=0x5633a9d9e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:34.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.424+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/1667450762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d91b60 0x5633a9d9e0d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fecf80cad10 tx=0x7fecf8099d80 comp rx=0 tx=0).stop 2023-11-16T16:58:34.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.424+0000 7fed1eabeb80 1 -- 172.21.15.44:0/1667450762 shutdown_connections 2023-11-16T16:58:34.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.424+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/1667450762 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5633a9dab440 0x5633a9dad820 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.424+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/1667450762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d968b0 0x5633a9d9ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.424+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/1667450762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d91b60 0x5633a9d9e0d0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.424+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/1667450762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d92470 0x5633a9d9b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.424+0000 7fed1eabeb80 1 -- 172.21.15.44:0/1667450762 >> 172.21.15.44:0/1667450762 conn(0x5633a9d87360 msgr2=0x5633a9d97d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:34.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.424+0000 7fece3fff700 0 client.27005 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:34.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.425+0000 7fed1eabeb80 1 -- 172.21.15.44:0/1667450762 shutdown_connections 2023-11-16T16:58:34.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.425+0000 7fed1eabeb80 1 -- 172.21.15.44:0/1667450762 wait complete. 2023-11-16T16:58:34.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.425+0000 7fed1eabeb80 20 client.27005 trim_cache size 0 max 16384 2023-11-16T16:58:34.444 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.442+0000 7fed1eabeb80 1 Processor -- start 2023-11-16T16:58:34.445 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.442+0000 7fed1eabeb80 1 -- start start 2023-11-16T16:58:34.445 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.443+0000 7fed1eabeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9d9dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.445 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.443+0000 7fed1eabeb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d92470 0x5633a9d8c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.445 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.443+0000 7fed1eabeb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d968b0 0x5633a9d8caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.445 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.443+0000 7fed1eabeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633a9d92bc0 con 0x5633a9d91b60 2023-11-16T16:58:34.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.443+0000 7fed1eabeb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633a9d9c930 con 0x5633a9d968b0 2023-11-16T16:58:34.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.443+0000 7fed1eabeb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633a9d99400 con 0x5633a9d92470 2023-11-16T16:58:34.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.443+0000 7fed08f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9d9dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.443+0000 7fed08f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9d9dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43384/0 (socket says 172.21.15.44:43384) 2023-11-16T16:58:34.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.443+0000 7fed08f9b700 1 -- 172.21.15.44:0/3767014938 learned_addr learned my addr 172.21.15.44:0/3767014938 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:34.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.443+0000 7fed03fff700 1 --2- 172.21.15.44:0/3767014938 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d92470 0x5633a9d8c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.443+0000 7fed0979c700 1 --2- 172.21.15.44:0/3767014938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d968b0 0x5633a9d8caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.443+0000 7fed08f9b700 1 -- 172.21.15.44:0/3767014938 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d92470 msgr2=0x5633a9d8c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:34.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.443+0000 7fed08f9b700 1 --2- 172.21.15.44:0/3767014938 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d92470 0x5633a9d8c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.443+0000 7fed08f9b700 1 -- 172.21.15.44:0/3767014938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d968b0 msgr2=0x5633a9d8caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:34.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.443+0000 7fed08f9b700 1 --2- 172.21.15.44:0/3767014938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d968b0 0x5633a9d8caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.443+0000 7fed08f9b700 1 -- 172.21.15.44:0/3767014938 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5633a9d96f50 con 0x5633a9d91b60 2023-11-16T16:58:34.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.444+0000 7fed08f9b700 1 --2- 172.21.15.44:0/3767014938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9d9dc70 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fecf800f010 tx=0x7fecf801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf83b763c2f4282 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:34.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.444+0000 7fecef7fe700 1 -- 172.21.15.44:0/3767014938 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf80962c0 con 0x5633a9d91b60 2023-11-16T16:58:34.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.444+0000 7fecef7fe700 1 -- 172.21.15.44:0/3767014938 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fecf8096990 con 0x5633a9d91b60 2023-11-16T16:58:34.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.444+0000 7fecef7fe700 1 -- 172.21.15.44:0/3767014938 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf80b82d0 con 0x5633a9d91b60 2023-11-16T16:58:34.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.444+0000 7fed1eabeb80 1 -- 172.21.15.44:0/3767014938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 msgr2=0x5633a9d9dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:34.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.444+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/3767014938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9d9dc70 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fecf800f010 tx=0x7fecf801f000 comp rx=0 tx=0).stop 2023-11-16T16:58:34.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.445+0000 7fed1eabeb80 1 -- 172.21.15.44:0/3767014938 shutdown_connections 2023-11-16T16:58:34.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.445+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/3767014938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d968b0 0x5633a9d8caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.445+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/3767014938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9d9dc70 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.445+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/3767014938 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d92470 0x5633a9d8c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.445+0000 7fed1eabeb80 1 -- 172.21.15.44:0/3767014938 >> 172.21.15.44:0/3767014938 conn(0x5633a9d87360 msgr2=0x5633a9d949d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:34.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.445+0000 7fed1eabeb80 1 -- 172.21.15.44:0/3767014938 shutdown_connections 2023-11-16T16:58:34.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.445+0000 7fed1eabeb80 1 -- 172.21.15.44:0/3767014938 wait complete. 2023-11-16T16:58:34.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.445+0000 7fed1eabeb80 1 -- start start 2023-11-16T16:58:34.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.445+0000 7fed1eabeb80 1 Processor -- start 2023-11-16T16:58:34.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed1eabeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9da0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed1eabeb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d92470 0x5633a9da0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed1eabeb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d968b0 0x5633a9da13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed1eabeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633a9dfe2d0 con 0x5633a9d91b60 2023-11-16T16:58:34.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed1eabeb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633a9dfe440 con 0x5633a9d92470 2023-11-16T16:58:34.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed1eabeb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633a9dfe600 con 0x5633a9d968b0 2023-11-16T16:58:34.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed1eabeb80 10 client.0 _ll_register_callbacks cb 0x7fecf8099950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:34.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed08f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9da0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed08f9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9da0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43394/0 (socket says 172.21.15.44:43394) 2023-11-16T16:58:34.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed08f9b700 1 -- 172.21.15.44:0/1399635067 learned_addr learned my addr 172.21.15.44:0/1399635067 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:34.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed03fff700 1 --2- 172.21.15.44:0/1399635067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d92470 0x5633a9da0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed0979c700 1 --2- 172.21.15.44:0/1399635067 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d968b0 0x5633a9da13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed08f9b700 1 -- 172.21.15.44:0/1399635067 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d968b0 msgr2=0x5633a9da13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:34.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed08f9b700 1 --2- 172.21.15.44:0/1399635067 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d968b0 0x5633a9da13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed08f9b700 1 -- 172.21.15.44:0/1399635067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d92470 msgr2=0x5633a9da0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:34.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed08f9b700 1 --2- 172.21.15.44:0/1399635067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d92470 0x5633a9da0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.446+0000 7fed08f9b700 1 -- 172.21.15.44:0/1399635067 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5633a9ca9190 con 0x5633a9d91b60 2023-11-16T16:58:34.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.447+0000 7fed08f9b700 1 --2- 172.21.15.44:0/1399635067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9da0930 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fecf80965c0 tx=0x7fecf800efa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aefc72bf43b3a8e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:34.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.447+0000 7fece3fff700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:58:34.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.447+0000 7fece3fff700 1 -- 172.21.15.44:0/1399635067 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf80a22d0 con 0x5633a9d91b60 2023-11-16T16:58:34.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.447+0000 7fece3fff700 1 -- 172.21.15.44:0/1399635067 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fecf8099070 con 0x5633a9d91b60 2023-11-16T16:58:34.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.447+0000 7fed1eabeb80 1 -- 172.21.15.44:0/1399635067 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5633a9cb1660 con 0x5633a9d91b60 2023-11-16T16:58:34.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.447+0000 7fece3fff700 1 -- 172.21.15.44:0/1399635067 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecf80b84e0 con 0x5633a9d91b60 2023-11-16T16:58:34.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.447+0000 7fed1eabeb80 10 client.16932 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:34.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.447+0000 7fed1eabeb80 1 -- 172.21.15.44:0/1399635067 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5633a9b10de0 con 0x5633a9d91b60 2023-11-16T16:58:34.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.447+0000 7fece3fff700 1 -- 172.21.15.44:0/1399635067 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5633a9cb1660 con 0x5633a9d91b60 2023-11-16T16:58:34.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.447+0000 7fece3fff700 1 -- 172.21.15.44:0/1399635067 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fecf8083a20 con 0x5633a9d91b60 2023-11-16T16:58:34.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.448+0000 7fed1eabeb80 10 client.16932 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:34.458 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.448+0000 7fed1eabeb80 10 client.16932 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:34.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.448+0000 7fed1eabeb80 20 client.16932 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:34.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.448+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/1399635067 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5633a9dff840 0x5633a9e01c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.448+0000 7fed1eabeb80 4 client.16932 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:58:34.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.448+0000 7fed1eabeb80 1 -- 172.21.15.44:0/1399635067 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5633a9b31f70 con 0x5633a9dff840 2023-11-16T16:58:34.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.448+0000 7fed03fff700 1 --2- 172.21.15.44:0/1399635067 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5633a9dff840 0x5633a9e01c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.448+0000 7fed03fff700 1 --2- 172.21.15.44:0/1399635067 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5633a9dff840 0x5633a9e01c20 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5c680dbff548ad4 server_cookie=de3a08c46f468abd in_seq=0 out_seq=0 2023-11-16T16:58:34.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.448+0000 7fece3fff700 10 client.16932 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:34.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.449+0000 7fece3fff700 1 -- 172.21.15.44:0/1399635067 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5633a9b31f70 con 0x5633a9dff840 2023-11-16T16:58:34.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.449+0000 7fece3fff700 10 client.16932 handle_command_reply: tid=0 2023-11-16T16:58:34.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.449+0000 7fed1eabeb80 1 client.16932 shutdown 2023-11-16T16:58:34.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.449+0000 7fed1eabeb80 1 -- 172.21.15.44:0/1399635067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 msgr2=0x5633a9da0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:34.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.449+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/1399635067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9da0930 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fecf80965c0 tx=0x7fecf800efa0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.449+0000 7fed1eabeb80 1 -- 172.21.15.44:0/1399635067 shutdown_connections 2023-11-16T16:58:34.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.449+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/1399635067 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5633a9dff840 0x5633a9e01c20 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.449+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/1399635067 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5633a9d968b0 0x5633a9da13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.449+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/1399635067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633a9d91b60 0x5633a9da0930 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.449+0000 7fed1eabeb80 1 --2- 172.21.15.44:0/1399635067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5633a9d92470 0x5633a9da0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.449+0000 7fece3fff700 0 client.16932 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:34.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.449+0000 7fed1eabeb80 1 -- 172.21.15.44:0/1399635067 >> 172.21.15.44:0/1399635067 conn(0x5633a9d87360 msgr2=0x5633a9d958f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:34.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.449+0000 7fed1eabeb80 1 -- 172.21.15.44:0/1399635067 shutdown_connections 2023-11-16T16:58:34.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.449+0000 7fed1eabeb80 1 -- 172.21.15.44:0/1399635067 wait complete. 2023-11-16T16:58:34.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.449+0000 7fed1eabeb80 20 client.16932 trim_cache size 0 max 16384 2023-11-16T16:58:34.462 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:58:34.462 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T16:58:34.462 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T16:58:34.463 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:58:34.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.450+0000 7fed0b1ff700 1 -- 172.21.15.44:0/1408316901 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fecf40a12c0 msgr2=0x7fecf40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:34.464 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.450+0000 7fed0b1ff700 1 --2- 172.21.15.44:0/1408316901 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fecf40a12c0 0x7fecf40a3770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fecf000b580 tx=0x7fecf001a040 comp rx=0 tx=0).stop 2023-11-16T16:58:34.464 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.450+0000 7fed0b1ff700 1 -- 172.21.15.44:0/1408316901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed04100cc0 msgr2=0x7fed04072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:34.464 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.450+0000 7fed0b1ff700 1 --2- 172.21.15.44:0/1408316901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed04100cc0 0x7fed04072df0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fecf800e9a0 tx=0x7fecf800ed60 comp rx=0 tx=0).stop 2023-11-16T16:58:34.464 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.450+0000 7fed0b1ff700 1 -- 172.21.15.44:0/1408316901 shutdown_connections 2023-11-16T16:58:34.464 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.450+0000 7fed0b1ff700 1 --2- 172.21.15.44:0/1408316901 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fecf40a12c0 0x7fecf40a3770 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.450+0000 7fed0b1ff700 1 --2- 172.21.15.44:0/1408316901 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed04105620 0x7fed041a4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.450+0000 7fed0b1ff700 1 --2- 172.21.15.44:0/1408316901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed04100cc0 0x7fed04072df0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.450+0000 7fed0b1ff700 1 --2- 172.21.15.44:0/1408316901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed04101660 0x7fed04073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.450+0000 7fed0b1ff700 1 -- 172.21.15.44:0/1408316901 >> 172.21.15.44:0/1408316901 conn(0x7fed04077560 msgr2=0x7fed040ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.450+0000 7fed0b1ff700 1 -- 172.21.15.44:0/1408316901 shutdown_connections 2023-11-16T16:58:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.450+0000 7fed0b1ff700 1 -- 172.21.15.44:0/1408316901 wait complete. 2023-11-16T16:58:34.469 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T16:58:34.469 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:58:34.674 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.672+0000 7f7c0b9ff700 1 -- 172.21.15.44:0/1268896085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c04100ef0 msgr2=0x7f7c04104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:34.674 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.672+0000 7f7c0b9ff700 1 --2- 172.21.15.44:0/1268896085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c04100ef0 0x7f7c04104e60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40099c0 tx=0x7f7bf4009cd0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.674 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.673+0000 7f7c0b9ff700 1 -- 172.21.15.44:0/1268896085 shutdown_connections 2023-11-16T16:58:34.674 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.673+0000 7f7c0b9ff700 1 --2- 172.21.15.44:0/1268896085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c04105540 0x7f7c0410d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.674 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.673+0000 7f7c0b9ff700 1 --2- 172.21.15.44:0/1268896085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c04100550 0x7f7c04100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.675 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.673+0000 7f7c0b9ff700 1 --2- 172.21.15.44:0/1268896085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c04100ef0 0x7f7c04104e60 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.675 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.673+0000 7f7c0b9ff700 1 -- 172.21.15.44:0/1268896085 >> 172.21.15.44:0/1268896085 conn(0x7f7c04077020 msgr2=0x7f7c04077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:34.675 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.673+0000 7f7c0b9ff700 1 -- 172.21.15.44:0/1268896085 shutdown_connections 2023-11-16T16:58:34.675 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.673+0000 7f7c0b9ff700 1 -- 172.21.15.44:0/1268896085 wait complete. 2023-11-16T16:58:34.677 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.674+0000 7f7c0b9ff700 1 Processor -- start 2023-11-16T16:58:34.677 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.674+0000 7f7c0b9ff700 1 -- start start 2023-11-16T16:58:34.677 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.675+0000 7f7c0b9ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c04100550 0x7f7c04072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.677 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.675+0000 7f7c0b9ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c04100ef0 0x7f7c04073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.675+0000 7f7c0b9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c04105540 0x7f7c041a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.675+0000 7f7c0b9ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c04073b60 con 0x7f7c04100ef0 2023-11-16T16:58:34.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.675+0000 7f7c0b9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c041a4af0 con 0x7f7c04105540 2023-11-16T16:58:34.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.675+0000 7f7c0b9ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c041a4c30 con 0x7f7c04100550 2023-11-16T16:58:34.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.675+0000 7f7c08f9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c04100ef0 0x7f7c04073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.675+0000 7f7c0979b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c04100550 0x7f7c04072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.675+0000 7f7c08f9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c04100ef0 0x7f7c04073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43406/0 (socket says 172.21.15.44:43406) 2023-11-16T16:58:34.678 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.675+0000 7f7c08f9a700 1 -- 172.21.15.44:0/3922986031 learned_addr learned my addr 172.21.15.44:0/3922986031 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:34.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.676+0000 7f7c09f9c700 1 --2- 172.21.15.44:0/3922986031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c04105540 0x7f7c041a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.676+0000 7f7c08f9a700 1 -- 172.21.15.44:0/3922986031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c04100550 msgr2=0x7f7c04072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:34.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.676+0000 7f7c08f9a700 1 --2- 172.21.15.44:0/3922986031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c04100550 0x7f7c04072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.676+0000 7f7c08f9a700 1 -- 172.21.15.44:0/3922986031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c04105540 msgr2=0x7f7c041a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:34.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.676+0000 7f7c08f9a700 1 --2- 172.21.15.44:0/3922986031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c04105540 0x7f7c041a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.676+0000 7f7c08f9a700 1 -- 172.21.15.44:0/3922986031 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf40095f0 con 0x7f7c04100ef0 2023-11-16T16:58:34.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.676+0000 7f7c0979b700 1 --2- 172.21.15.44:0/3922986031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c04100550 0x7f7c04072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:34.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.676+0000 7f7c08f9a700 1 --2- 172.21.15.44:0/3922986031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c04100ef0 0x7f7c04073330 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40098e0 tx=0x7f7bf400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.676+0000 7f7bfa7fc700 1 -- 172.21.15.44:0/3922986031 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf401d070 con 0x7f7c04100ef0 2023-11-16T16:58:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.676+0000 7f7c0b9ff700 1 -- 172.21.15.44:0/3922986031 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c04110310 con 0x7f7c04100ef0 2023-11-16T16:58:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.676+0000 7f7bfa7fc700 1 -- 172.21.15.44:0/3922986031 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7bf4022470 con 0x7f7c04100ef0 2023-11-16T16:58:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.676+0000 7f7bfa7fc700 1 -- 172.21.15.44:0/3922986031 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4017650 con 0x7f7c04100ef0 2023-11-16T16:58:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.676+0000 7f7c0b9ff700 1 -- 172.21.15.44:0/3922986031 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c04110180 con 0x7f7c04100ef0 2023-11-16T16:58:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.678+0000 7f7c0b9ff700 1 -- 172.21.15.44:0/3922986031 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c041104a0 con 0x7f7c04100ef0 2023-11-16T16:58:34.681 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.679+0000 7f7bfa7fc700 1 -- 172.21.15.44:0/3922986031 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f7bf4022850 con 0x7f7c04100ef0 2023-11-16T16:58:34.681 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.681+0000 7f7bfa7fc700 1 --2- 172.21.15.44:0/3922986031 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7bf00a12c0 0x7f7bf00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:34.682 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.681+0000 7f7bfa7fc700 1 -- 172.21.15.44:0/3922986031 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f7bf40cdff0 con 0x7f7c04100ef0 2023-11-16T16:58:34.682 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.681+0000 7f7c0979b700 1 --2- 172.21.15.44:0/3922986031 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7bf00a12c0 0x7f7bf00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:34.682 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.682+0000 7f7c0979b700 1 --2- 172.21.15.44:0/3922986031 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7bf00a12c0 0x7f7bf00a3770 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7c00011490 tx=0x7f7c00009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:34.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.685+0000 7f7bfa7fc700 1 -- 172.21.15.44:0/3922986031 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7bf4097070 con 0x7f7c04100ef0 2023-11-16T16:58:34.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.945+0000 7f7c0b9ff700 1 -- 172.21.15.44:0/3922986031 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c04067a60 con 0x7f7c04100ef0 2023-11-16T16:58:34.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.945+0000 7f7bfa7fc700 1 -- 172.21.15.44:0/3922986031 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f7bf409a690 con 0x7f7c04100ef0 2023-11-16T16:58:34.946 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:58:34.947 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:58:34.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.948+0000 7f7c0b9ff700 1 -- 172.21.15.44:0/3922986031 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7bf00a12c0 msgr2=0x7f7bf00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:34.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.948+0000 7f7c0b9ff700 1 --2- 172.21.15.44:0/3922986031 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7bf00a12c0 0x7f7bf00a3770 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7c00011490 tx=0x7f7c00009260 comp rx=0 tx=0).stop 2023-11-16T16:58:34.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.948+0000 7f7c0b9ff700 1 -- 172.21.15.44:0/3922986031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c04100ef0 msgr2=0x7f7c04073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:34.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.948+0000 7f7c0b9ff700 1 --2- 172.21.15.44:0/3922986031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c04100ef0 0x7f7c04073330 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40098e0 tx=0x7f7bf400f690 comp rx=0 tx=0).stop 2023-11-16T16:58:34.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.948+0000 7f7c0b9ff700 1 -- 172.21.15.44:0/3922986031 shutdown_connections 2023-11-16T16:58:34.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.948+0000 7f7c0b9ff700 1 --2- 172.21.15.44:0/3922986031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c04105540 0x7f7c041a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.948+0000 7f7c0b9ff700 1 --2- 172.21.15.44:0/3922986031 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7bf00a12c0 0x7f7bf00a3770 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.948+0000 7f7c0b9ff700 1 --2- 172.21.15.44:0/3922986031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c04100550 0x7f7c04072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.948+0000 7f7c0b9ff700 1 --2- 172.21.15.44:0/3922986031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c04100ef0 0x7f7c04073330 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:34.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.948+0000 7f7c0b9ff700 1 -- 172.21.15.44:0/3922986031 >> 172.21.15.44:0/3922986031 conn(0x7f7c04077020 msgr2=0x7f7c040fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:34.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.948+0000 7f7c0b9ff700 1 -- 172.21.15.44:0/3922986031 shutdown_connections 2023-11-16T16:58:34.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:34.948+0000 7f7c0b9ff700 1 -- 172.21.15.44:0/3922986031 wait complete. 2023-11-16T16:58:34.951 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:58:34.968 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T16:58:35.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.175+0000 7fc094bff700 1 -- 172.21.15.44:0/1207425345 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc090101660 msgr2=0x7fc090104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:35.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.175+0000 7fc094bff700 1 --2- 172.21.15.44:0/1207425345 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc090101660 0x7fc090104ef0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc0780099c0 tx=0x7fc078009cd0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.175+0000 7fc094bff700 1 -- 172.21.15.44:0/1207425345 shutdown_connections 2023-11-16T16:58:35.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.175+0000 7fc094bff700 1 --2- 172.21.15.44:0/1207425345 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc090105620 0x7fc09010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.175+0000 7fc094bff700 1 --2- 172.21.15.44:0/1207425345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc090100cc0 0x7fc090101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.175+0000 7fc094bff700 1 --2- 172.21.15.44:0/1207425345 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc090101660 0x7fc090104ef0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.175+0000 7fc094bff700 1 -- 172.21.15.44:0/1207425345 >> 172.21.15.44:0/1207425345 conn(0x7fc090077560 msgr2=0x7fc090077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:35.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.175+0000 7fc094bff700 1 -- 172.21.15.44:0/1207425345 shutdown_connections 2023-11-16T16:58:35.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.175+0000 7fc094bff700 1 -- 172.21.15.44:0/1207425345 wait complete. 2023-11-16T16:58:35.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.176+0000 7fc094bff700 1 Processor -- start 2023-11-16T16:58:35.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.176+0000 7fc094bff700 1 -- start start 2023-11-16T16:58:35.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.176+0000 7fc094bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc090100cc0 0x7fc090072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.176+0000 7fc094bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc090101660 0x7fc090073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.177+0000 7fc094bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc090105620 0x7fc0901a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.177+0000 7fc094bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc090073b60 con 0x7fc090100cc0 2023-11-16T16:58:35.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.177+0000 7fc094bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0901a4cd0 con 0x7fc090101660 2023-11-16T16:58:35.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.177+0000 7fc094bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0901a4e10 con 0x7fc090105620 2023-11-16T16:58:35.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.177+0000 7fc08e59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc090100cc0 0x7fc090072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.177+0000 7fc08e59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc090100cc0 0x7fc090072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47020/0 (socket says 172.21.15.44:47020) 2023-11-16T16:58:35.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.177+0000 7fc08e59c700 1 -- 172.21.15.44:0/647556997 learned_addr learned my addr 172.21.15.44:0/647556997 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:35.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.177+0000 7fc087fff700 1 --2- 172.21.15.44:0/647556997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc090101660 0x7fc090073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.177+0000 7fc08ed9d700 1 --2- 172.21.15.44:0/647556997 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc090105620 0x7fc0901a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.177+0000 7fc08e59c700 1 -- 172.21.15.44:0/647556997 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc090105620 msgr2=0x7fc0901a4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.177+0000 7fc08e59c700 1 --2- 172.21.15.44:0/647556997 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc090105620 0x7fc0901a4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.177+0000 7fc08e59c700 1 -- 172.21.15.44:0/647556997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc090101660 msgr2=0x7fc090073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.177+0000 7fc08e59c700 1 --2- 172.21.15.44:0/647556997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc090101660 0x7fc090073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.177+0000 7fc08e59c700 1 -- 172.21.15.44:0/647556997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc080009520 con 0x7fc090100cc0 2023-11-16T16:58:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.177+0000 7fc08e59c700 1 --2- 172.21.15.44:0/647556997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc090100cc0 0x7fc090072df0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc08000eb30 tx=0x7fc08000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.178+0000 7fc0877fe700 1 -- 172.21.15.44:0/647556997 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc080009880 con 0x7fc090100cc0 2023-11-16T16:58:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.178+0000 7fc0877fe700 1 -- 172.21.15.44:0/647556997 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0800044e0 con 0x7fc090100cc0 2023-11-16T16:58:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.178+0000 7fc094bff700 1 -- 172.21.15.44:0/647556997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0780095f0 con 0x7fc090100cc0 2023-11-16T16:58:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.178+0000 7fc0877fe700 1 -- 172.21.15.44:0/647556997 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc080003e90 con 0x7fc090100cc0 2023-11-16T16:58:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.178+0000 7fc094bff700 1 -- 172.21.15.44:0/647556997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc090110370 con 0x7fc090100cc0 2023-11-16T16:58:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.181+0000 7fc0877fe700 1 -- 172.21.15.44:0/647556997 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fc080004000 con 0x7fc090100cc0 2023-11-16T16:58:35.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.182+0000 7fc0a848cb80 1 Processor -- start 2023-11-16T16:58:35.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.182+0000 7fc0a848cb80 1 -- start start 2023-11-16T16:58:35.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.183+0000 7fc0877fe700 1 --2- 172.21.15.44:0/647556997 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc07c0a12c0 0x7fc07c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.183+0000 7fc0a848cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d3db60 0x560164d3df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.183+0000 7fc0a848cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3e470 0x560164d42370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.183+0000 7fc0a848cb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d428b0 0x560164d42d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.183+0000 7fc0a848cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x560164c43060 con 0x560164d3db60 2023-11-16T16:58:35.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.183+0000 7fc0a848cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560164c958a0 con 0x560164d3e470 2023-11-16T16:58:35.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.183+0000 7fc0a848cb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x560164c8ed30 con 0x560164d428b0 2023-11-16T16:58:35.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.183+0000 7fc0877fe700 1 -- 172.21.15.44:0/647556997 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fc080014070 con 0x7fc090100cc0 2023-11-16T16:58:35.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.183+0000 7fc08ed9d700 1 --2- 172.21.15.44:0/647556997 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc07c0a12c0 0x7fc07c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.183+0000 7fc08e59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d3db60 0x560164d3df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.183+0000 7fc087fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3e470 0x560164d42370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.183+0000 7fc08e59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d3db60 0x560164d3df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47026/0 (socket says 172.21.15.44:47026) 2023-11-16T16:58:35.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.183+0000 7fc08e59c700 1 -- 172.21.15.44:0/3636643769 learned_addr learned my addr 172.21.15.44:0/3636643769 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:35.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.183+0000 7fc08ed9d700 1 --2- 172.21.15.44:0/3636643769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d428b0 0x560164d42d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.184+0000 7fc08e59c700 1 -- 172.21.15.44:0/3636643769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d428b0 msgr2=0x560164d42d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:35.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.184+0000 7fc08e59c700 1 --2- 172.21.15.44:0/3636643769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d428b0 0x560164d42d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.184+0000 7fc08e59c700 1 -- 172.21.15.44:0/3636643769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3e470 msgr2=0x560164d42370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:35.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.184+0000 7fc08e59c700 1 --2- 172.21.15.44:0/3636643769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3e470 0x560164d42370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.184+0000 7fc08e59c700 1 -- 172.21.15.44:0/3636643769 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560164b10630 con 0x560164d3db60 2023-11-16T16:58:35.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.184+0000 7fc087fff700 1 --2- 172.21.15.44:0/3636643769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3e470 0x560164d42370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:35.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.184+0000 7fc08ed9d700 1 --2- 172.21.15.44:0/647556997 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc07c0a12c0 0x7fc07c0a3770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc090074f00 tx=0x7fc088009200 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:35.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.184+0000 7fc08e59c700 1 --2- 172.21.15.44:0/3636643769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d3db60 0x560164d3df30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc08000eb00 tx=0x7fc080099e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e56320ee0898cb3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:35.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.184+0000 7fc084ff9700 1 -- 172.21.15.44:0/3636643769 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08000cc90 con 0x560164d3db60 2023-11-16T16:58:35.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.184+0000 7fc084ff9700 1 -- 172.21.15.44:0/3636643769 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc080013370 con 0x560164d3db60 2023-11-16T16:58:35.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.184+0000 7fc084ff9700 1 -- 172.21.15.44:0/3636643769 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0800cab40 con 0x560164d3db60 2023-11-16T16:58:35.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.185+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3636643769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d3db60 msgr2=0x560164d3df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:35.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.185+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/3636643769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d3db60 0x560164d3df30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc08000eb00 tx=0x7fc080099e00 comp rx=0 tx=0).stop 2023-11-16T16:58:35.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.185+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3636643769 shutdown_connections 2023-11-16T16:58:35.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.185+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/3636643769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d428b0 0x560164d42d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.185+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/3636643769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d3db60 0x560164d3df30 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.185+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/3636643769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3e470 0x560164d42370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.185+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3636643769 >> 172.21.15.44:0/3636643769 conn(0x560164d33360 msgr2=0x560164d38b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:35.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.185+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3636643769 shutdown_connections 2023-11-16T16:58:35.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.185+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3636643769 wait complete. 2023-11-16T16:58:35.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.185+0000 7fc0a848cb80 1 -- start start 2023-11-16T16:58:35.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.186+0000 7fc0a848cb80 1 Processor -- start 2023-11-16T16:58:35.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.186+0000 7fc0a848cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3db60 0x560164d4a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.186+0000 7fc0a848cb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d3e470 0x560164d472a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.186+0000 7fc0a848cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d428b0 0x560164d478b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.186+0000 7fc0a848cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x560164c308e0 con 0x560164d428b0 2023-11-16T16:58:35.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.186+0000 7fc0a848cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560164c30580 con 0x560164d3db60 2023-11-16T16:58:35.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.186+0000 7fc0a848cb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x560164b12920 con 0x560164d3e470 2023-11-16T16:58:35.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.186+0000 7fc0a848cb80 10 client.0 _ll_register_callbacks cb 0x560164c4e780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:35.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.186+0000 7fc08ed9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d428b0 0x560164d478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.186+0000 7fc08e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3db60 0x560164d4a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.187+0000 7fc087fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d3e470 0x560164d472a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.186+0000 7fc08ed9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d428b0 0x560164d478b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47042/0 (socket says 172.21.15.44:47042) 2023-11-16T16:58:35.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.187+0000 7fc08ed9d700 1 -- 172.21.15.44:0/3782542706 learned_addr learned my addr 172.21.15.44:0/3782542706 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:35.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.187+0000 7fc08e59c700 1 -- 172.21.15.44:0/3782542706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d3e470 msgr2=0x560164d472a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:35.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.187+0000 7fc08e59c700 1 --2- 172.21.15.44:0/3782542706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d3e470 0x560164d472a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.187+0000 7fc08e59c700 1 -- 172.21.15.44:0/3782542706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d428b0 msgr2=0x560164d478b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:35.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.187+0000 7fc08e59c700 1 --2- 172.21.15.44:0/3782542706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d428b0 0x560164d478b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.187+0000 7fc08e59c700 1 -- 172.21.15.44:0/3782542706 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc090110370 con 0x560164d3db60 2023-11-16T16:58:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.187+0000 7fc08e59c700 1 --2- 172.21.15.44:0/3782542706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3db60 0x560164d4a090 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc08009a1d0 tx=0x7fc0800966b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77077f80be6ba0a1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.187+0000 7fc075ffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:58:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.187+0000 7fc075ffb700 1 -- 172.21.15.44:0/3782542706 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08000cc90 con 0x560164d3db60 2023-11-16T16:58:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.187+0000 7fc075ffb700 1 -- 172.21.15.44:0/3782542706 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fc08009a240 con 0x560164d3db60 2023-11-16T16:58:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.187+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3782542706 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560164d30b20 con 0x560164d3db60 2023-11-16T16:58:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.188+0000 7fc075ffb700 1 -- 172.21.15.44:0/3782542706 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc08000ce90 con 0x560164d3db60 2023-11-16T16:58:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.188+0000 7fc0a848cb80 10 client.27029 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.188+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3782542706 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x560164c55190 con 0x560164d3db60 2023-11-16T16:58:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.188+0000 7fc075ffb700 1 -- 172.21.15.44:0/3782542706 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0800c8080 con 0x560164d3db60 2023-11-16T16:58:35.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.188+0000 7fc075ffb700 1 -- 172.21.15.44:0/3782542706 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc08009a510 con 0x560164d3db60 2023-11-16T16:58:35.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.188+0000 7fc0a848cb80 10 client.27029 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:35.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.188+0000 7fc0a848cb80 10 client.27029 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:35.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.188+0000 7fc0a848cb80 20 client.27029 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:35.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.188+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/3782542706 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560164d574d0 0x560164d598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.188+0000 7fc0a848cb80 4 client.27029 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:58:35.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.188+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3782542706 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560164c510a0 con 0x560164d574d0 2023-11-16T16:58:35.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.188+0000 7fc087fff700 1 --2- 172.21.15.44:0/3782542706 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560164d574d0 0x560164d598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.189+0000 7fc087fff700 1 --2- 172.21.15.44:0/3782542706 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560164d574d0 0x560164d598b0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b52e90f0b3a3ffb4 server_cookie=dd0dbe6e6e6f0aad in_seq=0 out_seq=0 2023-11-16T16:58:35.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.189+0000 7fc075ffb700 10 client.27029 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:35.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.190+0000 7fc075ffb700 1 -- 172.21.15.44:0/3782542706 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x560164c510a0 con 0x560164d574d0 2023-11-16T16:58:35.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.190+0000 7fc075ffb700 10 client.27029 handle_command_reply: tid=0 2023-11-16T16:58:35.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.190+0000 7fc0a848cb80 1 client.27029 shutdown 2023-11-16T16:58:35.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.190+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3782542706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3db60 msgr2=0x560164d4a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:35.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.190+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/3782542706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3db60 0x560164d4a090 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc08009a1d0 tx=0x7fc0800966b0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.190+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3782542706 shutdown_connections 2023-11-16T16:58:35.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.190+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/3782542706 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560164d574d0 0x560164d598b0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.190+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/3782542706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d428b0 0x560164d478b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.190+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/3782542706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3db60 0x560164d4a090 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.190+0000 7fc075ffb700 0 client.27029 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:35.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.190+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/3782542706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d3e470 0x560164d472a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.190+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3782542706 >> 172.21.15.44:0/3782542706 conn(0x560164d33360 msgr2=0x560164d43d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:35.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.190+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3782542706 shutdown_connections 2023-11-16T16:58:35.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.190+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3782542706 wait complete. 2023-11-16T16:58:35.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.190+0000 7fc0a848cb80 20 client.27029 trim_cache size 0 max 16384 2023-11-16T16:58:35.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.204+0000 7fc0a848cb80 1 Processor -- start 2023-11-16T16:58:35.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.204+0000 7fc0a848cb80 1 -- start start 2023-11-16T16:58:35.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.204+0000 7fc0a848cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3db60 0x560164d401a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.204+0000 7fc0a848cb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d3e470 0x560164d53b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.204+0000 7fc0a848cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d428b0 0x560164d40a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.204+0000 7fc0a848cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x560164d399a0 con 0x560164d428b0 2023-11-16T16:58:35.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.204+0000 7fc0a848cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560164d41430 con 0x560164d3db60 2023-11-16T16:58:35.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.204+0000 7fc0a848cb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x560164d59ee0 con 0x560164d3e470 2023-11-16T16:58:35.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.205+0000 7fc08ed9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d428b0 0x560164d40a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.205+0000 7fc08e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3db60 0x560164d401a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.205+0000 7fc08e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3db60 0x560164d401a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:36930/0 (socket says 172.21.15.44:36930) 2023-11-16T16:58:35.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.205+0000 7fc08e59c700 1 -- 172.21.15.44:0/3757072065 learned_addr learned my addr 172.21.15.44:0/3757072065 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:35.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.205+0000 7fc08ed9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d428b0 0x560164d40a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47046/0 (socket says 172.21.15.44:47046) 2023-11-16T16:58:35.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.205+0000 7fc087fff700 1 --2- 172.21.15.44:0/3757072065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d3e470 0x560164d53b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.205+0000 7fc08e59c700 1 -- 172.21.15.44:0/3757072065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d3e470 msgr2=0x560164d53b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:35.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.205+0000 7fc08e59c700 1 --2- 172.21.15.44:0/3757072065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d3e470 0x560164d53b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.205+0000 7fc08e59c700 1 -- 172.21.15.44:0/3757072065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d428b0 msgr2=0x560164d40a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:35.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.205+0000 7fc08e59c700 1 --2- 172.21.15.44:0/3757072065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d428b0 0x560164d40a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.205+0000 7fc08e59c700 1 -- 172.21.15.44:0/3757072065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560164d34b00 con 0x560164d3db60 2023-11-16T16:58:35.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.205+0000 7fc08e59c700 1 --2- 172.21.15.44:0/3757072065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3db60 0x560164d401a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc080096cb0 tx=0x7fc0800987f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99969d68f818a470 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:35.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.206+0000 7fc0857fa700 1 -- 172.21.15.44:0/3757072065 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08000cc90 con 0x560164d3db60 2023-11-16T16:58:35.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.206+0000 7fc0857fa700 1 -- 172.21.15.44:0/3757072065 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560164d30b20 con 0x560164d3db60 2023-11-16T16:58:35.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.206+0000 7fc0857fa700 1 -- 172.21.15.44:0/3757072065 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08009e270 con 0x560164d3db60 2023-11-16T16:58:35.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.206+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3757072065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3db60 msgr2=0x560164d401a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:35.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.206+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/3757072065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3db60 0x560164d401a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc080096cb0 tx=0x7fc0800987f0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.206+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3757072065 shutdown_connections 2023-11-16T16:58:35.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.206+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/3757072065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d428b0 0x560164d40a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.206+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/3757072065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d3db60 0x560164d401a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.206+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/3757072065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d3e470 0x560164d53b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.206+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3757072065 >> 172.21.15.44:0/3757072065 conn(0x560164d33360 msgr2=0x560164d44c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:35.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.206+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3757072065 shutdown_connections 2023-11-16T16:58:35.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.206+0000 7fc0a848cb80 1 -- 172.21.15.44:0/3757072065 wait complete. 2023-11-16T16:58:35.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.207+0000 7fc0a848cb80 1 -- start start 2023-11-16T16:58:35.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.207+0000 7fc0a848cb80 1 Processor -- start 2023-11-16T16:58:35.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.207+0000 7fc0a848cb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d3db60 0x560164da3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.207+0000 7fc0a848cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d3e470 0x560164da3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.207+0000 7fc0a848cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d428b0 0x560164d4dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.207+0000 7fc0a848cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x560164d4e2a0 con 0x560164d3e470 2023-11-16T16:58:35.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.207+0000 7fc0a848cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560164d4e410 con 0x560164d428b0 2023-11-16T16:58:35.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.207+0000 7fc0a848cb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x560164d4e5d0 con 0x560164d3db60 2023-11-16T16:58:35.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.207+0000 7fc0a848cb80 10 client.0 _ll_register_callbacks cb 0x7fc080005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:35.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.208+0000 7fc087fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d3e470 0x560164da3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.208+0000 7fc087fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d3e470 0x560164da3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47054/0 (socket says 172.21.15.44:47054) 2023-11-16T16:58:35.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.208+0000 7fc087fff700 1 -- 172.21.15.44:0/4021432171 learned_addr learned my addr 172.21.15.44:0/4021432171 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:35.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.208+0000 7fc08ed9d700 1 --2- 172.21.15.44:0/4021432171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d428b0 0x560164d4dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.208+0000 7fc08e59c700 1 --2- 172.21.15.44:0/4021432171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d3db60 0x560164da3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.208+0000 7fc087fff700 1 -- 172.21.15.44:0/4021432171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d3db60 msgr2=0x560164da3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:35.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.208+0000 7fc087fff700 1 --2- 172.21.15.44:0/4021432171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d3db60 0x560164da3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.208+0000 7fc087fff700 1 -- 172.21.15.44:0/4021432171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d428b0 msgr2=0x560164d4dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:35.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.208+0000 7fc087fff700 1 --2- 172.21.15.44:0/4021432171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d428b0 0x560164d4dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.208+0000 7fc087fff700 1 -- 172.21.15.44:0/4021432171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560164c55190 con 0x560164d3e470 2023-11-16T16:58:35.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.208+0000 7fc08ed9d700 1 --2- 172.21.15.44:0/4021432171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d428b0 0x560164d4dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:58:35.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.208+0000 7fc087fff700 1 --2- 172.21.15.44:0/4021432171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d3e470 0x560164da3f80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc07800b580 tx=0x7fc078009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994158f0bd697169 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:35.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.208+0000 7fc075ffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:58:35.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.208+0000 7fc075ffb700 1 -- 172.21.15.44:0/4021432171 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0780280b0 con 0x560164d3e470 2023-11-16T16:58:35.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.209+0000 7fc0a848cb80 1 -- 172.21.15.44:0/4021432171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc080013370 con 0x560164d3e470 2023-11-16T16:58:35.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.209+0000 7fc075ffb700 1 -- 172.21.15.44:0/4021432171 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fc078028210 con 0x560164d3e470 2023-11-16T16:58:35.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.209+0000 7fc075ffb700 1 -- 172.21.15.44:0/4021432171 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0780283d0 con 0x560164d3e470 2023-11-16T16:58:35.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.209+0000 7fc0a848cb80 10 client.16956 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:35.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.209+0000 7fc0a848cb80 1 -- 172.21.15.44:0/4021432171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x560164abcde0 con 0x560164d3e470 2023-11-16T16:58:35.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.209+0000 7fc075ffb700 1 -- 172.21.15.44:0/4021432171 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc080013370 con 0x560164d3e470 2023-11-16T16:58:35.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.209+0000 7fc075ffb700 1 -- 172.21.15.44:0/4021432171 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc07800baf0 con 0x560164d3e470 2023-11-16T16:58:35.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.209+0000 7fc0a848cb80 10 client.16956 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:35.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.209+0000 7fc0a848cb80 10 client.16956 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:35.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.209+0000 7fc0a848cb80 20 client.16956 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:35.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.209+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/4021432171 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560164dab8d0 0x560164dadce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:35.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.209+0000 7fc0a848cb80 4 client.16956 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T16:58:35.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.209+0000 7fc0a848cb80 1 -- 172.21.15.44:0/4021432171 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560164c5d660 con 0x560164dab8d0 2023-11-16T16:58:35.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.210+0000 7fc08e59c700 1 --2- 172.21.15.44:0/4021432171 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560164dab8d0 0x560164dadce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:35.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.210+0000 7fc08e59c700 1 --2- 172.21.15.44:0/4021432171 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560164dab8d0 0x560164dadce0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf0d31c01a49065f server_cookie=38340ce561a5e7ca in_seq=0 out_seq=0 2023-11-16T16:58:35.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.210+0000 7fc075ffb700 10 client.16956 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:35.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.210+0000 7fc075ffb700 1 -- 172.21.15.44:0/4021432171 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560164c5d660 con 0x560164dab8d0 2023-11-16T16:58:35.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.210+0000 7fc075ffb700 10 client.16956 handle_command_reply: tid=0 2023-11-16T16:58:35.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.210+0000 7fc0a848cb80 1 client.16956 shutdown 2023-11-16T16:58:35.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.210+0000 7fc0a848cb80 1 -- 172.21.15.44:0/4021432171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d3e470 msgr2=0x560164da3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:35.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.210+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/4021432171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d3e470 0x560164da3f80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc07800b580 tx=0x7fc078009fd0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.210+0000 7fc0a848cb80 1 -- 172.21.15.44:0/4021432171 shutdown_connections 2023-11-16T16:58:35.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.210+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/4021432171 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560164dab8d0 0x560164dadce0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.210+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/4021432171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560164d428b0 0x560164d4dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.210+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/4021432171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560164d3db60 0x560164da3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.210+0000 7fc075ffb700 0 client.16956 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:35.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.210+0000 7fc0a848cb80 1 --2- 172.21.15.44:0/4021432171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560164d3e470 0x560164da3f80 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.211+0000 7fc0a848cb80 1 -- 172.21.15.44:0/4021432171 >> 172.21.15.44:0/4021432171 conn(0x560164d33360 msgr2=0x560164d53150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:35.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.211+0000 7fc0a848cb80 1 -- 172.21.15.44:0/4021432171 shutdown_connections 2023-11-16T16:58:35.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.211+0000 7fc0a848cb80 1 -- 172.21.15.44:0/4021432171 wait complete. 2023-11-16T16:58:35.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.211+0000 7fc0a848cb80 20 client.16956 trim_cache size 0 max 16384 2023-11-16T16:58:35.218 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T16:58:35.219 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.211+0000 7fc094bff700 1 -- 172.21.15.44:0/647556997 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc07c0a12c0 msgr2=0x7fc07c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:35.219 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.211+0000 7fc094bff700 1 --2- 172.21.15.44:0/647556997 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc07c0a12c0 0x7fc07c0a3770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc090074f00 tx=0x7fc088009200 comp rx=0 tx=0).stop 2023-11-16T16:58:35.219 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.211+0000 7fc094bff700 1 -- 172.21.15.44:0/647556997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc090100cc0 msgr2=0x7fc090072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:35.219 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.211+0000 7fc094bff700 1 --2- 172.21.15.44:0/647556997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc090100cc0 0x7fc090072df0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc08000eb30 tx=0x7fc08000ee40 comp rx=0 tx=0).stop 2023-11-16T16:58:35.219 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.211+0000 7fc094bff700 1 -- 172.21.15.44:0/647556997 shutdown_connections 2023-11-16T16:58:35.219 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.211+0000 7fc094bff700 1 --2- 172.21.15.44:0/647556997 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc07c0a12c0 0x7fc07c0a3770 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.211+0000 7fc094bff700 1 --2- 172.21.15.44:0/647556997 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc090105620 0x7fc0901a4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.211+0000 7fc094bff700 1 --2- 172.21.15.44:0/647556997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc090100cc0 0x7fc090072df0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.211+0000 7fc094bff700 1 --2- 172.21.15.44:0/647556997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc090101660 0x7fc090073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:35.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.211+0000 7fc094bff700 1 -- 172.21.15.44:0/647556997 >> 172.21.15.44:0/647556997 conn(0x7fc090077560 msgr2=0x7fc0900ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:35.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.212+0000 7fc094bff700 1 -- 172.21.15.44:0/647556997 shutdown_connections 2023-11-16T16:58:35.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:35.212+0000 7fc094bff700 1 -- 172.21.15.44:0/647556997 wait complete. 2023-11-16T16:58:35.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:35 smithi044 ceph-mon[128750]: pgmap v268: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-11-16T16:58:35.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:35 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3922986031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:58:35.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:35 smithi077 ceph-mon[131204]: pgmap v268: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-11-16T16:58:35.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:35 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3922986031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:58:35.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:35 smithi176 ceph-mon[131630]: pgmap v268: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-11-16T16:58:35.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:35 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3922986031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:58:36.232 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:58:36.233 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:58:36.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.454+0000 7f6226dff700 1 -- 172.21.15.44:0/3007448659 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6220100cc0 msgr2=0x7f6220101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.454+0000 7f6226dff700 1 --2- 172.21.15.44:0/3007448659 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6220100cc0 0x7f6220101090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6214009990 tx=0x7f6214009ca0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.454+0000 7f6226dff700 1 -- 172.21.15.44:0/3007448659 shutdown_connections 2023-11-16T16:58:36.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.454+0000 7f6226dff700 1 --2- 172.21.15.44:0/3007448659 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6220105620 0x7f622010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.454+0000 7f6226dff700 1 --2- 172.21.15.44:0/3007448659 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6220100cc0 0x7f6220101090 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.454+0000 7f6226dff700 1 --2- 172.21.15.44:0/3007448659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6220101660 0x7f6220104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.454+0000 7f6226dff700 1 -- 172.21.15.44:0/3007448659 >> 172.21.15.44:0/3007448659 conn(0x7f6220077560 msgr2=0x7f6220077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:36.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.454+0000 7f6226dff700 1 -- 172.21.15.44:0/3007448659 shutdown_connections 2023-11-16T16:58:36.457 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.454+0000 7f6226dff700 1 -- 172.21.15.44:0/3007448659 wait complete. 2023-11-16T16:58:36.459 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.455+0000 7f6226dff700 1 Processor -- start 2023-11-16T16:58:36.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.456+0000 7f6226dff700 1 -- start start 2023-11-16T16:58:36.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.456+0000 7f6226dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6220100cc0 0x7f622019e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.456+0000 7f6226dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6220101660 0x7f622019ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.456+0000 7f6226dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6220105620 0x7f6220198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.456+0000 7f6226dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6220198c00 con 0x7f6220105620 2023-11-16T16:58:36.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.456+0000 7f6226dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6220198d70 con 0x7f6220101660 2023-11-16T16:58:36.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.456+0000 7f6226dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6220198f30 con 0x7f6220100cc0 2023-11-16T16:58:36.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.456+0000 7f622539c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6220105620 0x7f6220198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.456+0000 7f621ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6220101660 0x7f622019ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.456+0000 7f622539c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6220105620 0x7f6220198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47070/0 (socket says 172.21.15.44:47070) 2023-11-16T16:58:36.461 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.456+0000 7f6224b9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6220100cc0 0x7f622019e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.456+0000 7f622539c700 1 -- 172.21.15.44:0/838036404 learned_addr learned my addr 172.21.15.44:0/838036404 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:36.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.457+0000 7f622539c700 1 -- 172.21.15.44:0/838036404 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6220100cc0 msgr2=0x7f622019e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.457+0000 7f622539c700 1 --2- 172.21.15.44:0/838036404 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6220100cc0 0x7f622019e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.457+0000 7f622539c700 1 -- 172.21.15.44:0/838036404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6220101660 msgr2=0x7f622019ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.457+0000 7f622539c700 1 --2- 172.21.15.44:0/838036404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6220101660 0x7f622019ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.457+0000 7f622539c700 1 -- 172.21.15.44:0/838036404 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62140095f0 con 0x7f6220105620 2023-11-16T16:58:36.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.457+0000 7f6224b9b700 1 --2- 172.21.15.44:0/838036404 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6220100cc0 0x7f622019e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:36.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.457+0000 7f622539c700 1 --2- 172.21.15.44:0/838036404 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6220105620 0x7f6220198590 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f621800ef30 tx=0x7f621800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:36.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.457+0000 7f621dffb700 1 -- 172.21.15.44:0/838036404 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6218018070 con 0x7f6220105620 2023-11-16T16:58:36.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.457+0000 7f6226dff700 1 -- 172.21.15.44:0/838036404 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62201103d0 con 0x7f6220105620 2023-11-16T16:58:36.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.457+0000 7f621dffb700 1 -- 172.21.15.44:0/838036404 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f621800f040 con 0x7f6220105620 2023-11-16T16:58:36.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.457+0000 7f621dffb700 1 -- 172.21.15.44:0/838036404 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621800cca0 con 0x7f6220105620 2023-11-16T16:58:36.464 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.457+0000 7f6226dff700 1 -- 172.21.15.44:0/838036404 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6220110240 con 0x7f6220105620 2023-11-16T16:58:36.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.460+0000 7f621dffb700 1 -- 172.21.15.44:0/838036404 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f6218003a00 con 0x7f6220105620 2023-11-16T16:58:36.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.461+0000 7f623a7e7b80 1 Processor -- start 2023-11-16T16:58:36.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.462+0000 7f623a7e7b80 1 -- start start 2023-11-16T16:58:36.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.462+0000 7f621dffb700 1 --2- 172.21.15.44:0/838036404 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f62100a12c0 0x7f62100a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.462+0000 7f623a7e7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e2ebe0 0x562529e2efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.462+0000 7f621dffb700 1 -- 172.21.15.44:0/838036404 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f62180cc980 con 0x7f6220105620 2023-11-16T16:58:36.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.462+0000 7f623a7e7b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e2f4f0 0x562529e333f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.462+0000 7f623a7e7b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e33930 0x562529e33d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.462+0000 7f623a7e7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562529d340e0 con 0x562529e2ebe0 2023-11-16T16:58:36.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.462+0000 7f623a7e7b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562529d86920 con 0x562529e2f4f0 2023-11-16T16:58:36.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.462+0000 7f623a7e7b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x562529d7fdb0 con 0x562529e33930 2023-11-16T16:58:36.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.462+0000 7f621ffff700 1 --2- 172.21.15.44:0/838036404 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f62100a12c0 0x7f62100a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.462+0000 7f6224b9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e2ebe0 0x562529e2efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.462+0000 7f6224b9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e2ebe0 0x562529e2efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47078/0 (socket says 172.21.15.44:47078) 2023-11-16T16:58:36.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.462+0000 7f6224b9b700 1 -- 172.21.15.44:0/2069180393 learned_addr learned my addr 172.21.15.44:0/2069180393 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:36.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.463+0000 7f621ffff700 1 --2- 172.21.15.44:0/2069180393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e2f4f0 0x562529e333f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.463+0000 7f622539c700 1 --2- 172.21.15.44:0/2069180393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e33930 0x562529e33d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.463+0000 7f6224b9b700 1 -- 172.21.15.44:0/2069180393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e33930 msgr2=0x562529e33d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:36.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.463+0000 7f6224b9b700 1 --2- 172.21.15.44:0/2069180393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e33930 0x562529e33d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.463+0000 7f6224b9b700 1 -- 172.21.15.44:0/2069180393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e2f4f0 msgr2=0x562529e333f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:36.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.463+0000 7f6224b9b700 1 --2- 172.21.15.44:0/2069180393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e2f4f0 0x562529e333f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.463+0000 7f6224b9b700 1 -- 172.21.15.44:0/2069180393 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562529c016b0 con 0x562529e2ebe0 2023-11-16T16:58:36.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.463+0000 7f622539c700 1 --2- 172.21.15.44:0/2069180393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e33930 0x562529e33d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:58:36.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.463+0000 7f621ffff700 1 --2- 172.21.15.44:0/838036404 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f62100a12c0 0x7f62100a3770 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6220106a90 tx=0x7f620c009f90 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:36.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.463+0000 7f6224b9b700 1 --2- 172.21.15.44:0/2069180393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e2ebe0 0x562529e2efb0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6214005ea0 tx=0x7f6214005ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938041ad44b1277f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:36.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.463+0000 7f620affd700 1 -- 172.21.15.44:0/2069180393 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621401d070 con 0x562529e2ebe0 2023-11-16T16:58:36.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.463+0000 7f620affd700 1 -- 172.21.15.44:0/2069180393 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f621401e040 con 0x562529e2ebe0 2023-11-16T16:58:36.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.463+0000 7f620affd700 1 -- 172.21.15.44:0/2069180393 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6214005600 con 0x562529e2ebe0 2023-11-16T16:58:36.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.464+0000 7f623a7e7b80 1 -- 172.21.15.44:0/2069180393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e2ebe0 msgr2=0x562529e2efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.464+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/2069180393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e2ebe0 0x562529e2efb0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6214005ea0 tx=0x7f6214005ed0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.464+0000 7f623a7e7b80 1 -- 172.21.15.44:0/2069180393 shutdown_connections 2023-11-16T16:58:36.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.464+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/2069180393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e33930 0x562529e33d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.464+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/2069180393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e2ebe0 0x562529e2efb0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.464+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/2069180393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e2f4f0 0x562529e333f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.464+0000 7f623a7e7b80 1 -- 172.21.15.44:0/2069180393 >> 172.21.15.44:0/2069180393 conn(0x562529e243e0 msgr2=0x562529e29bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:36.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.464+0000 7f623a7e7b80 1 -- 172.21.15.44:0/2069180393 shutdown_connections 2023-11-16T16:58:36.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.464+0000 7f623a7e7b80 1 -- 172.21.15.44:0/2069180393 wait complete. 2023-11-16T16:58:36.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.465+0000 7f623a7e7b80 1 -- start start 2023-11-16T16:58:36.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.465+0000 7f623a7e7b80 1 Processor -- start 2023-11-16T16:58:36.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f623a7e7b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 0x562529e3b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f623a7e7b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e2f4f0 0x562529e382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f623a7e7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e33930 0x562529e38920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f623a7e7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562529d21960 con 0x562529e33930 2023-11-16T16:58:36.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f623a7e7b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562529d21600 con 0x562529e2f4f0 2023-11-16T16:58:36.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f623a7e7b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x562529c039a0 con 0x562529e2ebe0 2023-11-16T16:58:36.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f623a7e7b80 10 client.0 _ll_register_callbacks cb 0x562529d3f800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:36.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f622539c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e33930 0x562529e38920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f622539c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e33930 0x562529e38920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47082/0 (socket says 172.21.15.44:47082) 2023-11-16T16:58:36.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f622539c700 1 -- 172.21.15.44:0/570359875 learned_addr learned my addr 172.21.15.44:0/570359875 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:36.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f621ffff700 1 --2- 172.21.15.44:0/570359875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e2f4f0 0x562529e382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f6224b9b700 1 --2- 172.21.15.44:0/570359875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 0x562529e3b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f622539c700 1 -- 172.21.15.44:0/570359875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 msgr2=0x562529e3b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:36.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f622539c700 1 --2- 172.21.15.44:0/570359875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 0x562529e3b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f622539c700 1 -- 172.21.15.44:0/570359875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e2f4f0 msgr2=0x562529e382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:36.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f622539c700 1 --2- 172.21.15.44:0/570359875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e2f4f0 0x562529e382f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f622539c700 1 -- 172.21.15.44:0/570359875 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f622010d9b0 con 0x562529e33930 2023-11-16T16:58:36.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f621ffff700 1 --2- 172.21.15.44:0/570359875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e2f4f0 0x562529e382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:58:36.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.466+0000 7f622539c700 1 --2- 172.21.15.44:0/570359875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e33930 0x562529e38920 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6218096110 tx=0x7f6218099000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f740ce0aa22ebe4a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:36.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.467+0000 7f61fffff700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:58:36.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.467+0000 7f61fffff700 1 -- 172.21.15.44:0/570359875 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6218082ea0 con 0x562529e33930 2023-11-16T16:58:36.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.467+0000 7f61fffff700 1 -- 172.21.15.44:0/570359875 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f62180996a0 con 0x562529e33930 2023-11-16T16:58:36.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.467+0000 7f623a7e7b80 1 -- 172.21.15.44:0/570359875 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562529e21ba0 con 0x562529e33930 2023-11-16T16:58:36.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.467+0000 7f61fffff700 1 -- 172.21.15.44:0/570359875 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f621809d6d0 con 0x562529e33930 2023-11-16T16:58:36.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.467+0000 7f623a7e7b80 10 client.16968 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:36.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.467+0000 7f623a7e7b80 1 -- 172.21.15.44:0/570359875 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x562529d46210 con 0x562529e33930 2023-11-16T16:58:36.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.467+0000 7f61fffff700 1 -- 172.21.15.44:0/570359875 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562529e21ba0 con 0x562529e33930 2023-11-16T16:58:36.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.467+0000 7f61fffff700 1 -- 172.21.15.44:0/570359875 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f62180a15d0 con 0x562529e33930 2023-11-16T16:58:36.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.467+0000 7f623a7e7b80 10 client.16968 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:36.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.467+0000 7f623a7e7b80 10 client.16968 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:36.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.468+0000 7f623a7e7b80 20 client.16968 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:36.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.468+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/570359875 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562529e48840 0x562529e4ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.468+0000 7f623a7e7b80 4 client.16968 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:58:36.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.468+0000 7f623a7e7b80 1 -- 172.21.15.44:0/570359875 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562529d42120 con 0x562529e48840 2023-11-16T16:58:36.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.468+0000 7f6224b9b700 1 --2- 172.21.15.44:0/570359875 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562529e48840 0x562529e4ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.468+0000 7f6224b9b700 1 --2- 172.21.15.44:0/570359875 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562529e48840 0x562529e4ac50 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b8d505786f6c02d server_cookie=f006fd57f5ad0bc8 in_seq=0 out_seq=0 2023-11-16T16:58:36.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.468+0000 7f61fffff700 10 client.16968 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:36.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.469+0000 7f61fffff700 1 -- 172.21.15.44:0/570359875 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562529d42120 con 0x562529e48840 2023-11-16T16:58:36.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.469+0000 7f61fffff700 10 client.16968 handle_command_reply: tid=0 2023-11-16T16:58:36.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.469+0000 7f623a7e7b80 1 client.16968 shutdown 2023-11-16T16:58:36.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.469+0000 7f623a7e7b80 1 -- 172.21.15.44:0/570359875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e33930 msgr2=0x562529e38920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.469+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/570359875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e33930 0x562529e38920 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6218096110 tx=0x7f6218099000 comp rx=0 tx=0).stop 2023-11-16T16:58:36.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.469+0000 7f623a7e7b80 1 -- 172.21.15.44:0/570359875 shutdown_connections 2023-11-16T16:58:36.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.469+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/570359875 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562529e48840 0x562529e4ac50 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.469+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/570359875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e33930 0x562529e38920 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.469+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/570359875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 0x562529e3b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.469+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/570359875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e2f4f0 0x562529e382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.469+0000 7f61fffff700 0 client.16968 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:36.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.469+0000 7f623a7e7b80 1 -- 172.21.15.44:0/570359875 >> 172.21.15.44:0/570359875 conn(0x562529e243e0 msgr2=0x562529e28e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:36.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.470+0000 7f623a7e7b80 1 -- 172.21.15.44:0/570359875 shutdown_connections 2023-11-16T16:58:36.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.470+0000 7f623a7e7b80 1 -- 172.21.15.44:0/570359875 wait complete. 2023-11-16T16:58:36.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.470+0000 7f623a7e7b80 20 client.16968 trim_cache size 0 max 16384 2023-11-16T16:58:36.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.488+0000 7f623a7e7b80 1 Processor -- start 2023-11-16T16:58:36.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.488+0000 7f623a7e7b80 1 -- start start 2023-11-16T16:58:36.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.488+0000 7f623a7e7b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 0x562529e3fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.488+0000 7f623a7e7b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e2f4f0 0x562529e2e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.488+0000 7f623a7e7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e33930 0x562529e295e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.488+0000 7f623a7e7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562529e2fc40 con 0x562529e33930 2023-11-16T16:58:36.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.488+0000 7f623a7e7b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562529e23f60 con 0x562529e2f4f0 2023-11-16T16:58:36.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.488+0000 7f623a7e7b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x562529e2b4b0 con 0x562529e2ebe0 2023-11-16T16:58:36.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.489+0000 7f622539c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e33930 0x562529e295e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.489+0000 7f622539c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e33930 0x562529e295e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47094/0 (socket says 172.21.15.44:47094) 2023-11-16T16:58:36.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.489+0000 7f622539c700 1 -- 172.21.15.44:0/3450636400 learned_addr learned my addr 172.21.15.44:0/3450636400 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:36.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.489+0000 7f621ffff700 1 --2- 172.21.15.44:0/3450636400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e2f4f0 0x562529e2e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.489+0000 7f6224b9b700 1 --2- 172.21.15.44:0/3450636400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 0x562529e3fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.489+0000 7f622539c700 1 -- 172.21.15.44:0/3450636400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 msgr2=0x562529e3fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:36.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.489+0000 7f622539c700 1 --2- 172.21.15.44:0/3450636400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 0x562529e3fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.489+0000 7f622539c700 1 -- 172.21.15.44:0/3450636400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e2f4f0 msgr2=0x562529e2e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:36.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.489+0000 7f622539c700 1 --2- 172.21.15.44:0/3450636400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e2f4f0 0x562529e2e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.489+0000 7f622539c700 1 -- 172.21.15.44:0/3450636400 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562529e33fd0 con 0x562529e33930 2023-11-16T16:58:36.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.489+0000 7f622539c700 1 --2- 172.21.15.44:0/3450636400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e33930 0x562529e295e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f621800ef00 tx=0x7f6218004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=649e0736f360ea5a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:36.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.489+0000 7f620b7fe700 1 -- 172.21.15.44:0/3450636400 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6218082ea0 con 0x562529e33930 2023-11-16T16:58:36.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.489+0000 7f620b7fe700 1 -- 172.21.15.44:0/3450636400 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6218095490 con 0x562529e33930 2023-11-16T16:58:36.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.489+0000 7f620b7fe700 1 -- 172.21.15.44:0/3450636400 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62180a1ad0 con 0x562529e33930 2023-11-16T16:58:36.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.490+0000 7f623a7e7b80 1 -- 172.21.15.44:0/3450636400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e33930 msgr2=0x562529e295e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.490+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/3450636400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e33930 0x562529e295e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f621800ef00 tx=0x7f6218004dc0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.490+0000 7f623a7e7b80 1 -- 172.21.15.44:0/3450636400 shutdown_connections 2023-11-16T16:58:36.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.490+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/3450636400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e33930 0x562529e295e0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.490+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/3450636400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 0x562529e3fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.490+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/3450636400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e2f4f0 0x562529e2e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.490+0000 7f623a7e7b80 1 -- 172.21.15.44:0/3450636400 >> 172.21.15.44:0/3450636400 conn(0x562529e243e0 msgr2=0x562529e2d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:36.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.490+0000 7f623a7e7b80 1 -- 172.21.15.44:0/3450636400 shutdown_connections 2023-11-16T16:58:36.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.490+0000 7f623a7e7b80 1 -- 172.21.15.44:0/3450636400 wait complete. 2023-11-16T16:58:36.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.490+0000 7f623a7e7b80 1 -- start start 2023-11-16T16:58:36.496 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.491+0000 7f623a7e7b80 1 Processor -- start 2023-11-16T16:58:36.499 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f623a7e7b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 0x562529e966b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f623a7e7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e2f4f0 0x562529e96bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f623a7e7b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e33930 0x562529e90a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f623a7e7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562529e91040 con 0x562529e2f4f0 2023-11-16T16:58:36.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f623a7e7b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562529e911b0 con 0x562529e33930 2023-11-16T16:58:36.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f623a7e7b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x562529e91370 con 0x562529e2ebe0 2023-11-16T16:58:36.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f623a7e7b80 10 client.0 _ll_register_callbacks cb 0x7f6218095ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:36.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f621ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e2f4f0 0x562529e96bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f6224b9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 0x562529e966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f6224b9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 0x562529e966b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:49940/0 (socket says 172.21.15.44:49940) 2023-11-16T16:58:36.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f6224b9b700 1 -- 172.21.15.44:0/2719310488 learned_addr learned my addr 172.21.15.44:0/2719310488 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:36.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f621ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e2f4f0 0x562529e96bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47096/0 (socket says 172.21.15.44:47096) 2023-11-16T16:58:36.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f622539c700 1 --2- 172.21.15.44:0/2719310488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e33930 0x562529e90a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f6224b9b700 1 -- 172.21.15.44:0/2719310488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e33930 msgr2=0x562529e90a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:36.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f6224b9b700 1 --2- 172.21.15.44:0/2719310488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e33930 0x562529e90a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f6224b9b700 1 -- 172.21.15.44:0/2719310488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e2f4f0 msgr2=0x562529e96bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:36.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f6224b9b700 1 --2- 172.21.15.44:0/2719310488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e2f4f0 0x562529e96bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.492+0000 7f6224b9b700 1 -- 172.21.15.44:0/2719310488 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562529d46210 con 0x562529e2ebe0 2023-11-16T16:58:36.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.493+0000 7f6224b9b700 1 --2- 172.21.15.44:0/2719310488 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 0x562529e966b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6214025040 tx=0x7f6214005a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f31dce5a8750524 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:36.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.493+0000 7f61fffff700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:58:36.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.493+0000 7f61fffff700 1 -- 172.21.15.44:0/2719310488 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621401d040 con 0x562529e2ebe0 2023-11-16T16:58:36.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.493+0000 7f623a7e7b80 1 -- 172.21.15.44:0/2719310488 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f621401e040 con 0x562529e2ebe0 2023-11-16T16:58:36.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.493+0000 7f61fffff700 1 -- 172.21.15.44:0/2719310488 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f6214031130 con 0x562529e2ebe0 2023-11-16T16:58:36.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.493+0000 7f61fffff700 1 -- 172.21.15.44:0/2719310488 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62140312f0 con 0x562529e2ebe0 2023-11-16T16:58:36.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.493+0000 7f623a7e7b80 10 client.26998 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:36.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.493+0000 7f623a7e7b80 1 -- 172.21.15.44:0/2719310488 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x562529bade60 con 0x562529e2ebe0 2023-11-16T16:58:36.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.494+0000 7f61fffff700 1 -- 172.21.15.44:0/2719310488 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f621401e040 con 0x562529e2ebe0 2023-11-16T16:58:36.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.494+0000 7f61fffff700 1 -- 172.21.15.44:0/2719310488 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6214004b30 con 0x562529e2ebe0 2023-11-16T16:58:36.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.494+0000 7f623a7e7b80 10 client.26998 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:36.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.494+0000 7f623a7e7b80 10 client.26998 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:36.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.494+0000 7f623a7e7b80 20 client.26998 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:36.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.494+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/2719310488 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562529e9e080 0x562529ea0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.494+0000 7f623a7e7b80 4 client.26998 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:58:36.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.494+0000 7f623a7e7b80 1 -- 172.21.15.44:0/2719310488 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562529d4e6e0 con 0x562529e9e080 2023-11-16T16:58:36.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.494+0000 7f621ffff700 1 --2- 172.21.15.44:0/2719310488 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562529e9e080 0x562529ea0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.495+0000 7f621ffff700 1 --2- 172.21.15.44:0/2719310488 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562529e9e080 0x562529ea0460 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b23c571666047405 server_cookie=4b5bc2653d013417 in_seq=0 out_seq=0 2023-11-16T16:58:36.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.495+0000 7f61fffff700 10 client.26998 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:36.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.496+0000 7f61fffff700 1 -- 172.21.15.44:0/2719310488 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562529d4e6e0 con 0x562529e9e080 2023-11-16T16:58:36.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.496+0000 7f61fffff700 10 client.26998 handle_command_reply: tid=0 2023-11-16T16:58:36.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.496+0000 7f623a7e7b80 1 client.26998 shutdown 2023-11-16T16:58:36.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.496+0000 7f623a7e7b80 1 -- 172.21.15.44:0/2719310488 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 msgr2=0x562529e966b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.496+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/2719310488 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 0x562529e966b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6214025040 tx=0x7f6214005a40 comp rx=0 tx=0).stop 2023-11-16T16:58:36.505 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.496+0000 7f623a7e7b80 1 -- 172.21.15.44:0/2719310488 shutdown_connections 2023-11-16T16:58:36.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.496+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/2719310488 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562529e9e080 0x562529ea0460 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.496+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/2719310488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562529e33930 0x562529e90a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.496+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/2719310488 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562529e2ebe0 0x562529e966b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.496+0000 7f623a7e7b80 1 --2- 172.21.15.44:0/2719310488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562529e2f4f0 0x562529e96bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.496+0000 7f61fffff700 0 client.26998 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:36.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.496+0000 7f623a7e7b80 1 -- 172.21.15.44:0/2719310488 >> 172.21.15.44:0/2719310488 conn(0x562529e243e0 msgr2=0x562529e2e270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:36.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.496+0000 7f623a7e7b80 1 -- 172.21.15.44:0/2719310488 shutdown_connections 2023-11-16T16:58:36.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.496+0000 7f623a7e7b80 1 -- 172.21.15.44:0/2719310488 wait complete. 2023-11-16T16:58:36.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.497+0000 7f623a7e7b80 20 client.26998 trim_cache size 0 max 16384 2023-11-16T16:58:36.507 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:58:36.507 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:58:36.507 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "2badeab4-5eb5-48b9-b795-8d10d6e8b89a", 2023-11-16T16:58:36.507 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:58:36.508 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:58:36.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.497+0000 7f6226dff700 1 -- 172.21.15.44:0/838036404 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f62100a12c0 msgr2=0x7f62100a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.497+0000 7f6226dff700 1 --2- 172.21.15.44:0/838036404 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f62100a12c0 0x7f62100a3770 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6220106a90 tx=0x7f620c009f90 comp rx=0 tx=0).stop 2023-11-16T16:58:36.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.497+0000 7f6226dff700 1 -- 172.21.15.44:0/838036404 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6220105620 msgr2=0x7f6220198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.497+0000 7f6226dff700 1 --2- 172.21.15.44:0/838036404 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6220105620 0x7f6220198590 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f621800ef30 tx=0x7f621800c3a0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.497+0000 7f6226dff700 1 -- 172.21.15.44:0/838036404 shutdown_connections 2023-11-16T16:58:36.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.497+0000 7f6226dff700 1 --2- 172.21.15.44:0/838036404 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6220105620 0x7f6220198590 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.497+0000 7f6226dff700 1 --2- 172.21.15.44:0/838036404 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f62100a12c0 0x7f62100a3770 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.497+0000 7f6226dff700 1 --2- 172.21.15.44:0/838036404 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6220100cc0 0x7f622019e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.497+0000 7f6226dff700 1 --2- 172.21.15.44:0/838036404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6220101660 0x7f622019ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.497+0000 7f6226dff700 1 -- 172.21.15.44:0/838036404 >> 172.21.15.44:0/838036404 conn(0x7f6220077560 msgr2=0x7f62200fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:36.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.497+0000 7f6226dff700 1 -- 172.21.15.44:0/838036404 shutdown_connections 2023-11-16T16:58:36.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.497+0000 7f6226dff700 1 -- 172.21.15.44:0/838036404 wait complete. 2023-11-16T16:58:36.516 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:58:36.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.723+0000 7f4fd59ff700 1 -- 172.21.15.44:0/3441896620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fd0105620 msgr2=0x7f4fd010d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.723+0000 7f4fd59ff700 1 --2- 172.21.15.44:0/3441896620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fd0105620 0x7f4fd010d1a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8009fc0 tx=0x7f4fc800b330 comp rx=0 tx=0).stop 2023-11-16T16:58:36.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.723+0000 7f4fd59ff700 1 -- 172.21.15.44:0/3441896620 shutdown_connections 2023-11-16T16:58:36.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.723+0000 7f4fd59ff700 1 --2- 172.21.15.44:0/3441896620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fd0105620 0x7f4fd010d1a0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.724 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.723+0000 7f4fd59ff700 1 --2- 172.21.15.44:0/3441896620 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd0100cc0 0x7f4fd0101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.723+0000 7f4fd59ff700 1 --2- 172.21.15.44:0/3441896620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4fd0101660 0x7f4fd0104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.723+0000 7f4fd59ff700 1 -- 172.21.15.44:0/3441896620 >> 172.21.15.44:0/3441896620 conn(0x7f4fd0077560 msgr2=0x7f4fd0077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:36.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.723+0000 7f4fd59ff700 1 -- 172.21.15.44:0/3441896620 shutdown_connections 2023-11-16T16:58:36.725 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.723+0000 7f4fd59ff700 1 -- 172.21.15.44:0/3441896620 wait complete. 2023-11-16T16:58:36.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.724+0000 7f4fd59ff700 1 Processor -- start 2023-11-16T16:58:36.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.724+0000 7f4fd59ff700 1 -- start start 2023-11-16T16:58:36.727 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.724+0000 7f4fd59ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fd0100cc0 0x7f4fd0072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.724+0000 7f4fd59ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd0101660 0x7f4fd0073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fd59ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4fd0105620 0x7f4fd01a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fd59ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd0073b60 con 0x7f4fd0101660 2023-11-16T16:58:36.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fd59ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd01a4c60 con 0x7f4fd0100cc0 2023-11-16T16:58:36.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fd59ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd01a4da0 con 0x7f4fd0105620 2023-11-16T16:58:36.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fce7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd0101660 0x7f4fd0073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.728 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fceffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fd0100cc0 0x7f4fd0072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fceffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fd0100cc0 0x7f4fd0072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:37020/0 (socket says 172.21.15.44:37020) 2023-11-16T16:58:36.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fce7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd0101660 0x7f4fd0073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47118/0 (socket says 172.21.15.44:47118) 2023-11-16T16:58:36.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fceffd700 1 -- 172.21.15.44:0/3941154720 learned_addr learned my addr 172.21.15.44:0/3941154720 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:36.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fcf7fe700 1 --2- 172.21.15.44:0/3941154720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4fd0105620 0x7f4fd01a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fceffd700 1 -- 172.21.15.44:0/3941154720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4fd0105620 msgr2=0x7f4fd01a4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.729 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fceffd700 1 --2- 172.21.15.44:0/3941154720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4fd0105620 0x7f4fd01a4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fceffd700 1 -- 172.21.15.44:0/3941154720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd0101660 msgr2=0x7f4fd0073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fceffd700 1 --2- 172.21.15.44:0/3941154720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd0101660 0x7f4fd0073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fceffd700 1 -- 172.21.15.44:0/3941154720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc00095f0 con 0x7f4fd0100cc0 2023-11-16T16:58:36.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fce7fc700 1 --2- 172.21.15.44:0/3941154720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd0101660 0x7f4fd0073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:36.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.725+0000 7f4fceffd700 1 --2- 172.21.15.44:0/3941154720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fd0100cc0 0x7f4fd0072df0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000ec10 tx=0x7f4fc000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:36.730 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.726+0000 7f4fd49fd700 1 -- 172.21.15.44:0/3941154720 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc000cd40 con 0x7f4fd0100cc0 2023-11-16T16:58:36.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.726+0000 7f4fd49fd700 1 -- 172.21.15.44:0/3941154720 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4fc00044e0 con 0x7f4fd0100cc0 2023-11-16T16:58:36.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.726+0000 7f4fd59ff700 1 -- 172.21.15.44:0/3941154720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc8009c40 con 0x7f4fd0100cc0 2023-11-16T16:58:36.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.726+0000 7f4fd49fd700 1 -- 172.21.15.44:0/3941154720 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc0010690 con 0x7f4fd0100cc0 2023-11-16T16:58:36.731 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.726+0000 7f4fd59ff700 1 -- 172.21.15.44:0/3941154720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fd0110390 con 0x7f4fd0100cc0 2023-11-16T16:58:36.734 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.729+0000 7f4fd49fd700 1 -- 172.21.15.44:0/3941154720 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f4fc0003710 con 0x7f4fd0100cc0 2023-11-16T16:58:36.734 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.730+0000 7f4fe931bb80 1 Processor -- start 2023-11-16T16:58:36.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.730+0000 7f4fe931bb80 1 -- start start 2023-11-16T16:58:36.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fe931bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a42b60 0x55c823a42f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fe931bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a43470 0x55c823a47370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fe931bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a47d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fe931bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c823948060 con 0x55c823a478b0 2023-11-16T16:58:36.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fe931bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c82399a8a0 con 0x55c823a43470 2023-11-16T16:58:36.735 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fe931bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c823993d30 con 0x55c823a42b60 2023-11-16T16:58:36.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fce7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a47d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fce7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a47d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47132/0 (socket says 172.21.15.44:47132) 2023-11-16T16:58:36.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fce7fc700 1 -- 172.21.15.44:0/4174488521 learned_addr learned my addr 172.21.15.44:0/4174488521 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:36.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fceffd700 1 --2- 172.21.15.44:0/4174488521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a43470 0x55c823a47370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fcf7fe700 1 --2- 172.21.15.44:0/4174488521 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a42b60 0x55c823a42f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fd49fd700 1 --2- 172.21.15.44:0/3941154720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4fb80a12c0 0x7f4fb80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.736 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fce7fc700 1 -- 172.21.15.44:0/4174488521 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a42b60 msgr2=0x55c823a42f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:36.737 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fce7fc700 1 --2- 172.21.15.44:0/4174488521 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a42b60 0x55c823a42f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.737 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fce7fc700 1 -- 172.21.15.44:0/4174488521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a43470 msgr2=0x55c823a47370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:36.737 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fce7fc700 1 --2- 172.21.15.44:0/4174488521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a43470 0x55c823a47370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.737 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.731+0000 7f4fce7fc700 1 -- 172.21.15.44:0/4174488521 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c823815630 con 0x55c823a478b0 2023-11-16T16:58:36.737 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.732+0000 7f4fd49fd700 1 -- 172.21.15.44:0/3941154720 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f4fc0014070 con 0x7f4fd0100cc0 2023-11-16T16:58:36.737 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.732+0000 7f4fcf7fe700 1 --2- 172.21.15.44:0/3941154720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4fb80a12c0 0x7f4fb80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.737 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.732+0000 7f4fce7fc700 1 --2- 172.21.15.44:0/4174488521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a47d00 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4011440 tx=0x7f4fc4011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e41058d04a42909 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:36.737 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.732+0000 7f4fb5ffb700 1 -- 172.21.15.44:0/4174488521 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4004230 con 0x55c823a478b0 2023-11-16T16:58:36.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.732+0000 7f4fb5ffb700 1 -- 172.21.15.44:0/4174488521 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4fc4004cf0 con 0x55c823a478b0 2023-11-16T16:58:36.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.732+0000 7f4fb5ffb700 1 -- 172.21.15.44:0/4174488521 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4005700 con 0x55c823a478b0 2023-11-16T16:58:36.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.733+0000 7f4fe931bb80 1 -- 172.21.15.44:0/4174488521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 msgr2=0x55c823a47d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.733+0000 7f4fcf7fe700 1 --2- 172.21.15.44:0/3941154720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4fb80a12c0 0x7f4fb80a3770 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8006130 tx=0x7f4fc8009920 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:36.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.733+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/4174488521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a47d00 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4011440 tx=0x7f4fc4011750 comp rx=0 tx=0).stop 2023-11-16T16:58:36.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.733+0000 7f4fe931bb80 1 -- 172.21.15.44:0/4174488521 shutdown_connections 2023-11-16T16:58:36.738 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.733+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/4174488521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a47d00 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.733+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/4174488521 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a42b60 0x55c823a42f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.733+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/4174488521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a43470 0x55c823a47370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.733+0000 7f4fe931bb80 1 -- 172.21.15.44:0/4174488521 >> 172.21.15.44:0/4174488521 conn(0x55c823a38360 msgr2=0x55c823a3db40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:36.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.733+0000 7f4fe931bb80 1 -- 172.21.15.44:0/4174488521 shutdown_connections 2023-11-16T16:58:36.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.733+0000 7f4fe931bb80 1 -- 172.21.15.44:0/4174488521 wait complete. 2023-11-16T16:58:36.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.733+0000 7f4fe931bb80 1 -- start start 2023-11-16T16:58:36.739 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.734+0000 7f4fe931bb80 1 Processor -- start 2023-11-16T16:58:36.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.734+0000 7f4fe931bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a42b60 0x55c823a4eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.734+0000 7f4fe931bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a43470 0x55c823a4c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.743 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.734+0000 7f4fe931bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a4c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.734+0000 7f4fe931bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8239358e0 con 0x55c823a478b0 2023-11-16T16:58:36.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.734+0000 7f4fe931bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c823935580 con 0x55c823a42b60 2023-11-16T16:58:36.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.734+0000 7f4fe931bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c823817920 con 0x55c823a43470 2023-11-16T16:58:36.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.734+0000 7f4fe931bb80 10 client.0 _ll_register_callbacks cb 0x55c823953780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:36.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.735+0000 7f4fcf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a4c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.735+0000 7f4fceffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a42b60 0x55c823a4eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.735+0000 7f4fcf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a4c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47142/0 (socket says 172.21.15.44:47142) 2023-11-16T16:58:36.744 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.735+0000 7f4fcf7fe700 1 -- 172.21.15.44:0/1236076339 learned_addr learned my addr 172.21.15.44:0/1236076339 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:36.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.735+0000 7f4fce7fc700 1 --2- 172.21.15.44:0/1236076339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a43470 0x55c823a4c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.735+0000 7f4fcf7fe700 1 -- 172.21.15.44:0/1236076339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a43470 msgr2=0x55c823a4c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:36.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.735+0000 7f4fcf7fe700 1 --2- 172.21.15.44:0/1236076339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a43470 0x55c823a4c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.735+0000 7f4fcf7fe700 1 -- 172.21.15.44:0/1236076339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a42b60 msgr2=0x55c823a4eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:36.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.735+0000 7f4fcf7fe700 1 --2- 172.21.15.44:0/1236076339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a42b60 0x55c823a4eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.735+0000 7f4fcf7fe700 1 -- 172.21.15.44:0/1236076339 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc4009c90 con 0x55c823a478b0 2023-11-16T16:58:36.745 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.735+0000 7f4fceffd700 1 --2- 172.21.15.44:0/1236076339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a42b60 0x55c823a4eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:36.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.735+0000 7f4fcf7fe700 1 --2- 172.21.15.44:0/1236076339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a4c910 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8004f50 tx=0x7f4fc800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=140eea516e3ada2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:36.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.735+0000 7f4faeffd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:58:36.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.736+0000 7f4faeffd700 1 -- 172.21.15.44:0/1236076339 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc800e040 con 0x55c823a478b0 2023-11-16T16:58:36.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.736+0000 7f4faeffd700 1 -- 172.21.15.44:0/1236076339 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f4fc800be00 con 0x55c823a478b0 2023-11-16T16:58:36.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.736+0000 7f4fe931bb80 1 -- 172.21.15.44:0/1236076339 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c823a35b20 con 0x55c823a478b0 2023-11-16T16:58:36.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.736+0000 7f4faeffd700 1 -- 172.21.15.44:0/1236076339 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fc80077f0 con 0x55c823a478b0 2023-11-16T16:58:36.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.736+0000 7f4fe931bb80 10 client.16986 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:36.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.736+0000 7f4fe931bb80 1 -- 172.21.15.44:0/1236076339 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55c82395a190 con 0x55c823a478b0 2023-11-16T16:58:36.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.736+0000 7f4faeffd700 1 -- 172.21.15.44:0/1236076339 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c823a35b20 con 0x55c823a478b0 2023-11-16T16:58:36.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.736+0000 7f4faeffd700 1 -- 172.21.15.44:0/1236076339 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4fc802e030 con 0x55c823a478b0 2023-11-16T16:58:36.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.736+0000 7f4fe931bb80 10 client.16986 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:36.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.737+0000 7f4fe931bb80 10 client.16986 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:36.747 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.737+0000 7f4fe931bb80 20 client.16986 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:36.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.737+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/1236076339 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c823a5c800 0x55c823a5ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.737+0000 7f4fe931bb80 4 client.16986 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:58:36.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.737+0000 7f4fe931bb80 1 -- 172.21.15.44:0/1236076339 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c8239560a0 con 0x55c823a5c800 2023-11-16T16:58:36.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.737+0000 7f4fceffd700 1 --2- 172.21.15.44:0/1236076339 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c823a5c800 0x55c823a5ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.737+0000 7f4fceffd700 1 --2- 172.21.15.44:0/1236076339 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c823a5c800 0x55c823a5ec10 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=948d00f5184209bd server_cookie=566d5dbc7295ed3d in_seq=0 out_seq=0 2023-11-16T16:58:36.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.737+0000 7f4faeffd700 10 client.16986 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:36.748 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.738+0000 7f4faeffd700 1 -- 172.21.15.44:0/1236076339 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55c8239560a0 con 0x55c823a5c800 2023-11-16T16:58:36.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.738+0000 7f4faeffd700 10 client.16986 handle_command_reply: tid=0 2023-11-16T16:58:36.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.738+0000 7f4fe931bb80 1 client.16986 shutdown 2023-11-16T16:58:36.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.738+0000 7f4fe931bb80 1 -- 172.21.15.44:0/1236076339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 msgr2=0x55c823a4c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.738+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/1236076339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a4c910 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8004f50 tx=0x7f4fc800b330 comp rx=0 tx=0).stop 2023-11-16T16:58:36.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.738+0000 7f4fe931bb80 1 -- 172.21.15.44:0/1236076339 shutdown_connections 2023-11-16T16:58:36.749 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.738+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/1236076339 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c823a5c800 0x55c823a5ec10 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.738+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/1236076339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a4c910 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.738+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/1236076339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a42b60 0x55c823a4eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.738+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/1236076339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a43470 0x55c823a4c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.738+0000 7f4fe931bb80 1 -- 172.21.15.44:0/1236076339 >> 172.21.15.44:0/1236076339 conn(0x55c823a38360 msgr2=0x55c823a48d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:36.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.738+0000 7f4faeffd700 0 client.16986 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:36.750 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.739+0000 7f4fe931bb80 1 -- 172.21.15.44:0/1236076339 shutdown_connections 2023-11-16T16:58:36.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.739+0000 7f4fe931bb80 1 -- 172.21.15.44:0/1236076339 wait complete. 2023-11-16T16:58:36.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.739+0000 7f4fe931bb80 20 client.16986 trim_cache size 0 max 16384 2023-11-16T16:58:36.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.756+0000 7f4fe931bb80 1 Processor -- start 2023-11-16T16:58:36.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.756+0000 7f4fe931bb80 1 -- start start 2023-11-16T16:58:36.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.757+0000 7f4fe931bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a42b60 0x55c823a539b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.757+0000 7f4fe931bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a43470 0x55c823a58c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.757+0000 7f4fe931bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a45970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.757+0000 7f4fe931bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c823a43bc0 con 0x55c823a478b0 2023-11-16T16:58:36.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.757+0000 7f4fe931bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c823a4d850 con 0x55c823a42b60 2023-11-16T16:58:36.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.757+0000 7f4fe931bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c823a46410 con 0x55c823a43470 2023-11-16T16:58:36.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.757+0000 7f4fceffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a42b60 0x55c823a539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.757+0000 7f4fcf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a45970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.757+0000 7f4fcf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a45970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47148/0 (socket says 172.21.15.44:47148) 2023-11-16T16:58:36.763 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.757+0000 7f4fcf7fe700 1 -- 172.21.15.44:0/1309375564 learned_addr learned my addr 172.21.15.44:0/1309375564 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:36.764 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.757+0000 7f4fceffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a42b60 0x55c823a539b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:37040/0 (socket says 172.21.15.44:37040) 2023-11-16T16:58:36.764 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.757+0000 7f4fce7fc700 1 --2- 172.21.15.44:0/1309375564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a43470 0x55c823a58c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.764 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.758+0000 7f4fcf7fe700 1 -- 172.21.15.44:0/1309375564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a43470 msgr2=0x55c823a58c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:36.764 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.758+0000 7f4fcf7fe700 1 --2- 172.21.15.44:0/1309375564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a43470 0x55c823a58c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.764 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.758+0000 7f4fcf7fe700 1 -- 172.21.15.44:0/1309375564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a42b60 msgr2=0x55c823a539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:36.764 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.758+0000 7f4fcf7fe700 1 --2- 172.21.15.44:0/1309375564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a42b60 0x55c823a539b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.764 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.758+0000 7f4fcf7fe700 1 -- 172.21.15.44:0/1309375564 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c823a3f490 con 0x55c823a478b0 2023-11-16T16:58:36.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.758+0000 7f4fcf7fe700 1 --2- 172.21.15.44:0/1309375564 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a45970 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4fc80287a0 tx=0x7f4fc80287d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd456f4bde14384 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:36.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.758+0000 7f4fb67fc700 1 -- 172.21.15.44:0/1309375564 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc800e040 con 0x55c823a478b0 2023-11-16T16:58:36.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.758+0000 7f4fb67fc700 1 -- 172.21.15.44:0/1309375564 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4fc80288f0 con 0x55c823a478b0 2023-11-16T16:58:36.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.758+0000 7f4fb67fc700 1 -- 172.21.15.44:0/1309375564 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc801d910 con 0x55c823a478b0 2023-11-16T16:58:36.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.759+0000 7f4fe931bb80 1 -- 172.21.15.44:0/1309375564 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 msgr2=0x55c823a45970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.759+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/1309375564 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a45970 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4fc80287a0 tx=0x7f4fc80287d0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.765 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.759+0000 7f4fe931bb80 1 -- 172.21.15.44:0/1309375564 shutdown_connections 2023-11-16T16:58:36.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.759+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/1309375564 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a478b0 0x55c823a45970 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.759+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/1309375564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a42b60 0x55c823a539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.759+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/1309375564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a43470 0x55c823a58c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.759+0000 7f4fe931bb80 1 -- 172.21.15.44:0/1309375564 >> 172.21.15.44:0/1309375564 conn(0x55c823a38360 msgr2=0x55c823a421f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:36.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.759+0000 7f4fe931bb80 1 -- 172.21.15.44:0/1309375564 shutdown_connections 2023-11-16T16:58:36.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.759+0000 7f4fe931bb80 1 -- 172.21.15.44:0/1309375564 wait complete. 2023-11-16T16:58:36.766 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.759+0000 7f4fe931bb80 1 -- start start 2023-11-16T16:58:36.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.760+0000 7f4fe931bb80 1 Processor -- start 2023-11-16T16:58:36.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.760+0000 7f4fe931bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a42b60 0x55c823aa4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.760+0000 7f4fe931bb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a43470 0x55c823aa4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.760+0000 7f4fe931bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a478b0 0x55c823aaf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.760+0000 7f4fe931bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c823aa5510 con 0x55c823a42b60 2023-11-16T16:58:36.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.760+0000 7f4fe931bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c823aafc40 con 0x55c823a478b0 2023-11-16T16:58:36.767 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.760+0000 7f4fe931bb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c823aafe00 con 0x55c823a43470 2023-11-16T16:58:36.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.760+0000 7f4fe931bb80 10 client.0 _ll_register_callbacks cb 0x7f4fc8004d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:58:36.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.760+0000 7f4fceffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a42b60 0x55c823aa4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.761+0000 7f4fce7fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a43470 0x55c823aa4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.761+0000 7f4fceffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a42b60 0x55c823aa4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47158/0 (socket says 172.21.15.44:47158) 2023-11-16T16:58:36.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.761+0000 7f4fceffd700 1 -- 172.21.15.44:0/713069465 learned_addr learned my addr 172.21.15.44:0/713069465 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:58:36.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.761+0000 7f4fcf7fe700 1 --2- 172.21.15.44:0/713069465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a478b0 0x55c823aaf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.761+0000 7f4fceffd700 1 -- 172.21.15.44:0/713069465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a43470 msgr2=0x55c823aa4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:36.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.761+0000 7f4fceffd700 1 --2- 172.21.15.44:0/713069465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a43470 0x55c823aa4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.769 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.761+0000 7f4fceffd700 1 -- 172.21.15.44:0/713069465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a478b0 msgr2=0x55c823aaf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:58:36.769 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.761+0000 7f4fceffd700 1 --2- 172.21.15.44:0/713069465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a478b0 0x55c823aaf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.769 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.761+0000 7f4fceffd700 1 -- 172.21.15.44:0/713069465 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c82395a190 con 0x55c823a42b60 2023-11-16T16:58:36.769 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.761+0000 7f4fce7fc700 1 --2- 172.21.15.44:0/713069465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a43470 0x55c823aa4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:36.769 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.761+0000 7f4fcf7fe700 1 --2- 172.21.15.44:0/713069465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a478b0 0x55c823aaf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:58:36.769 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.761+0000 7f4fceffd700 1 --2- 172.21.15.44:0/713069465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a42b60 0x55c823aa4840 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000c660 tx=0x7f4fc00cd810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f0a0f7b7ed603df server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:58:36.769 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.761+0000 7f4faeffd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:58:36.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.761+0000 7f4faeffd700 1 -- 172.21.15.44:0/713069465 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc009a7d0 con 0x55c823a42b60 2023-11-16T16:58:36.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.761+0000 7f4faeffd700 1 -- 172.21.15.44:0/713069465 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f4fc009e170 con 0x55c823a42b60 2023-11-16T16:58:36.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.761+0000 7f4fe931bb80 1 -- 172.21.15.44:0/713069465 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4fc4004cf0 con 0x55c823a42b60 2023-11-16T16:58:36.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.762+0000 7f4faeffd700 1 -- 172.21.15.44:0/713069465 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fc00966b0 con 0x55c823a42b60 2023-11-16T16:58:36.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.762+0000 7f4fe931bb80 10 client.16992 fetch_fsmap learned FSMap version 21 2023-11-16T16:58:36.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.762+0000 7f4fe931bb80 1 -- 172.21.15.44:0/713069465 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55c8237c1de0 con 0x55c823a42b60 2023-11-16T16:58:36.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.762+0000 7f4faeffd700 1 -- 172.21.15.44:0/713069465 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4fc4004cf0 con 0x55c823a42b60 2023-11-16T16:58:36.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.762+0000 7f4faeffd700 1 -- 172.21.15.44:0/713069465 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4fc0099860 con 0x55c823a42b60 2023-11-16T16:58:36.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.762+0000 7f4fe931bb80 10 client.16992 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:58:36.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.762+0000 7f4fe931bb80 10 client.16992 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:58:36.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.762+0000 7f4fe931bb80 20 client.16992 populate_metadata read hostname 'smithi044' 2023-11-16T16:58:36.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.762+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/713069465 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c823ab1560 0x55c823ab3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:58:36.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.762+0000 7f4fe931bb80 4 client.16992 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:58:36.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.762+0000 7f4fe931bb80 1 -- 172.21.15.44:0/713069465 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c823962660 con 0x55c823ab1560 2023-11-16T16:58:36.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.763+0000 7f4fce7fc700 1 --2- 172.21.15.44:0/713069465 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c823ab1560 0x55c823ab3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:58:36.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.763+0000 7f4fce7fc700 1 --2- 172.21.15.44:0/713069465 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c823ab1560 0x55c823ab3940 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=491d5ec129fb7862 server_cookie=cb0ef3c9d916372b in_seq=0 out_seq=0 2023-11-16T16:58:36.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.763+0000 7f4faeffd700 10 client.16992 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:36.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.763+0000 7f4faeffd700 1 -- 172.21.15.44:0/713069465 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c823962660 con 0x55c823ab1560 2023-11-16T16:58:36.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.763+0000 7f4faeffd700 10 client.16992 handle_command_reply: tid=0 2023-11-16T16:58:36.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.763+0000 7f4fe931bb80 1 client.16992 shutdown 2023-11-16T16:58:36.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.763+0000 7f4fe931bb80 1 -- 172.21.15.44:0/713069465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a42b60 msgr2=0x55c823aa4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.763+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/713069465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a42b60 0x55c823aa4840 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000c660 tx=0x7f4fc00cd810 comp rx=0 tx=0).stop 2023-11-16T16:58:36.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.764+0000 7f4fe931bb80 1 -- 172.21.15.44:0/713069465 shutdown_connections 2023-11-16T16:58:36.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.764+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/713069465 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55c823ab1560 0x55c823ab3940 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.764+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/713069465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c823a478b0 0x55c823aaf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.764+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/713069465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c823a42b60 0x55c823aa4840 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.764+0000 7f4fe931bb80 1 --2- 172.21.15.44:0/713069465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c823a43470 0x55c823aa4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.764+0000 7f4faeffd700 0 client.16992 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:58:36.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.764+0000 7f4fe931bb80 1 -- 172.21.15.44:0/713069465 >> 172.21.15.44:0/713069465 conn(0x55c823a38360 msgr2=0x55c823a41960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:36.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.764+0000 7f4fe931bb80 1 -- 172.21.15.44:0/713069465 shutdown_connections 2023-11-16T16:58:36.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.764+0000 7f4fe931bb80 1 -- 172.21.15.44:0/713069465 wait complete. 2023-11-16T16:58:36.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.764+0000 7f4fe931bb80 20 client.16992 trim_cache size 0 max 16384 2023-11-16T16:58:36.775 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:58:36.775 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2023-11-16T16:58:36.775 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2023-11-16T16:58:36.776 INFO:teuthology.orchestra.run.smithi044.stdout: "2badeab4-5eb5-48b9-b795-8d10d6e8b89a": { 2023-11-16T16:58:36.776 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2023-11-16T16:58:36.776 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "2badeab4-5eb5-48b9-b795-8d10d6e8b89a", 2023-11-16T16:58:36.776 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2023-11-16T16:58:36.776 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:58:36.776 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:58:36.776 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:58:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.764+0000 7f4fd59ff700 1 -- 172.21.15.44:0/3941154720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4fb80a12c0 msgr2=0x7f4fb80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.764+0000 7f4fd59ff700 1 --2- 172.21.15.44:0/3941154720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4fb80a12c0 0x7f4fb80a3770 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8006130 tx=0x7f4fc8009920 comp rx=0 tx=0).stop 2023-11-16T16:58:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.764+0000 7f4fd59ff700 1 -- 172.21.15.44:0/3941154720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fd0100cc0 msgr2=0x7f4fd0072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:58:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.764+0000 7f4fd59ff700 1 --2- 172.21.15.44:0/3941154720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fd0100cc0 0x7f4fd0072df0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000ec10 tx=0x7f4fc000ef20 comp rx=0 tx=0).stop 2023-11-16T16:58:36.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.765+0000 7f4fd59ff700 1 -- 172.21.15.44:0/3941154720 shutdown_connections 2023-11-16T16:58:36.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.765+0000 7f4fd59ff700 1 --2- 172.21.15.44:0/3941154720 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f4fb80a12c0 0x7f4fb80a3770 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.765+0000 7f4fd59ff700 1 --2- 172.21.15.44:0/3941154720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4fd0105620 0x7f4fd01a4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.765+0000 7f4fd59ff700 1 --2- 172.21.15.44:0/3941154720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fd0100cc0 0x7f4fd0072df0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.765+0000 7f4fd59ff700 1 --2- 172.21.15.44:0/3941154720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fd0101660 0x7f4fd0073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:58:36.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.765+0000 7f4fd59ff700 1 -- 172.21.15.44:0/3941154720 >> 172.21.15.44:0/3941154720 conn(0x7f4fd0077560 msgr2=0x7f4fd00feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:58:36.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.765+0000 7f4fd59ff700 1 -- 172.21.15.44:0/3941154720 shutdown_connections 2023-11-16T16:58:36.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:58:36.765+0000 7f4fd59ff700 1 -- 172.21.15.44:0/3941154720 wait complete. 2023-11-16T16:58:36.784 INFO:tasks.cephfs.filesystem:scrub status for tag:2badeab4-5eb5-48b9-b795-8d10d6e8b89a - {'path': '/', 'tag': '2badeab4-5eb5-48b9-b795-8d10d6e8b89a', 'options': 'recursive,force'} 2023-11-16T16:58:37.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:37 smithi077 ceph-mon[131204]: pgmap v269: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1023 B/s wr, 1 op/s 2023-11-16T16:58:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:37 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:58:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:37 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:58:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:37 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:58:37.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:37 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:58:37.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:37 smithi044 ceph-mon[128750]: pgmap v269: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1023 B/s wr, 1 op/s 2023-11-16T16:58:37.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:37 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:58:37.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:37 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:58:37.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:37 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:58:37.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:37 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:58:37.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:37 smithi176 ceph-mon[131630]: pgmap v269: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1023 B/s wr, 1 op/s 2023-11-16T16:58:37.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:37 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:58:37.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:37 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:58:37.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:37 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:58:37.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:37 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:58:38.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:38 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:58:38.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:38 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:58:38.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:38 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:58:39.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:39 smithi077 ceph-mon[131204]: pgmap v270: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 3 op/s 2023-11-16T16:58:39.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:39 smithi044 ceph-mon[128750]: pgmap v270: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 3 op/s 2023-11-16T16:58:39.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:39 smithi176 ceph-mon[131630]: pgmap v270: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 3 op/s 2023-11-16T16:58:41.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:41 smithi077 ceph-mon[131204]: pgmap v271: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2 op/s 2023-11-16T16:58:41.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:41 smithi044 ceph-mon[128750]: pgmap v271: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2 op/s 2023-11-16T16:58:41.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:41 smithi176 ceph-mon[131630]: pgmap v271: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2 op/s 2023-11-16T16:58:43.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:43 smithi077 ceph-mon[131204]: pgmap v272: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2 op/s 2023-11-16T16:58:43.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:43 smithi044 ceph-mon[128750]: pgmap v272: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2 op/s 2023-11-16T16:58:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:43 smithi176 ceph-mon[131630]: pgmap v272: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2 op/s 2023-11-16T16:58:45.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:45 smithi077 ceph-mon[131204]: pgmap v273: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3 op/s 2023-11-16T16:58:45.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:45 smithi044 ceph-mon[128750]: pgmap v273: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3 op/s 2023-11-16T16:58:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:45 smithi176 ceph-mon[131630]: pgmap v273: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3 op/s 2023-11-16T16:58:47.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:47 smithi077 ceph-mon[131204]: pgmap v274: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2 op/s 2023-11-16T16:58:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:47 smithi044 ceph-mon[128750]: pgmap v274: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2 op/s 2023-11-16T16:58:47.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:47 smithi176 ceph-mon[131630]: pgmap v274: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2 op/s 2023-11-16T16:58:49.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:49 smithi077 ceph-mon[131204]: pgmap v275: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3 op/s 2023-11-16T16:58:49.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:49 smithi044 ceph-mon[128750]: pgmap v275: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3 op/s 2023-11-16T16:58:49.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:49 smithi176 ceph-mon[131630]: pgmap v275: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3 op/s 2023-11-16T16:58:51.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:51 smithi077 ceph-mon[131204]: pgmap v276: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1 op/s 2023-11-16T16:58:51.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:51 smithi044 ceph-mon[128750]: pgmap v276: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1 op/s 2023-11-16T16:58:51.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:51 smithi176 ceph-mon[131630]: pgmap v276: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1 op/s 2023-11-16T16:58:53.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:53 smithi077 ceph-mon[131204]: pgmap v277: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:58:53.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:53 smithi044 ceph-mon[128750]: pgmap v277: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:58:53.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:53 smithi176 ceph-mon[131630]: pgmap v277: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:58:55.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:55 smithi077 ceph-mon[131204]: pgmap v278: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:58:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:55 smithi044 ceph-mon[128750]: pgmap v278: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:58:55.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:55 smithi176 ceph-mon[131630]: pgmap v278: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:58:57.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:57 smithi077 ceph-mon[131204]: pgmap v279: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:58:57.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:57 smithi044 ceph-mon[128750]: pgmap v279: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:58:57.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:57 smithi176 ceph-mon[131630]: pgmap v279: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:58:59.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:58:59 smithi077 ceph-mon[131204]: pgmap v280: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:58:59.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:58:59 smithi044 ceph-mon[128750]: pgmap v280: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:58:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:58:59 smithi176 ceph-mon[131630]: pgmap v280: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:59:01.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:01 smithi077 ceph-mon[131204]: pgmap v281: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:59:01.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:01 smithi044 ceph-mon[128750]: pgmap v281: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:59:01.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:01 smithi176 ceph-mon[131630]: pgmap v281: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:59:03.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:03 smithi077 ceph-mon[131204]: pgmap v282: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:59:03.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:03 smithi044 ceph-mon[128750]: pgmap v282: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:59:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:03 smithi176 ceph-mon[131630]: pgmap v282: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:59:05.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:05 smithi077 ceph-mon[131204]: pgmap v283: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 45 KiB/s wr, 17 op/s 2023-11-16T16:59:05.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:05 smithi044 ceph-mon[128750]: pgmap v283: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 45 KiB/s wr, 17 op/s 2023-11-16T16:59:05.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:05 smithi176 ceph-mon[131630]: pgmap v283: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 45 KiB/s wr, 17 op/s 2023-11-16T16:59:06.785 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:59:07.007 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.005+0000 7fc7f1bff700 1 -- 172.21.15.44:0/1170699601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7ec1063f0 msgr2=0x7fc7ec1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.007 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.005+0000 7fc7f1bff700 1 --2- 172.21.15.44:0/1170699601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7ec1063f0 0x7fc7ec1067c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc009990 tx=0x7fc7dc009ca0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.007 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.005+0000 7fc7f1bff700 1 -- 172.21.15.44:0/1170699601 shutdown_connections 2023-11-16T16:59:07.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.005+0000 7fc7f1bff700 1 --2- 172.21.15.44:0/1170699601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7ec101400 0x7fc7ec10d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.005+0000 7fc7f1bff700 1 --2- 172.21.15.44:0/1170699601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7ec1063f0 0x7fc7ec1067c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.005+0000 7fc7f1bff700 1 --2- 172.21.15.44:0/1170699601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ec100a50 0x7fc7ec100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.005+0000 7fc7f1bff700 1 -- 172.21.15.44:0/1170699601 >> 172.21.15.44:0/1170699601 conn(0x7fc7ec077590 msgr2=0x7fc7ec077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:07.008 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.005+0000 7fc7f1bff700 1 -- 172.21.15.44:0/1170699601 shutdown_connections 2023-11-16T16:59:07.009 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.005+0000 7fc7f1bff700 1 -- 172.21.15.44:0/1170699601 wait complete. 2023-11-16T16:59:07.011 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.006+0000 7fc7f1bff700 1 Processor -- start 2023-11-16T16:59:07.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.006+0000 7fc7f1bff700 1 -- start start 2023-11-16T16:59:07.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.006+0000 7fc7f1bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ec100a50 0x7fc7ec072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.006+0000 7fc7f1bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7ec101400 0x7fc7ec073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.012 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.006+0000 7fc7f1bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7ec1063f0 0x7fc7ec1a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.006+0000 7fc7f1bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec073b60 con 0x7fc7ec1063f0 2023-11-16T16:59:07.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.006+0000 7fc7f1bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec1a4b40 con 0x7fc7ec100a50 2023-11-16T16:59:07.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.006+0000 7fc7f1bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec1a4c80 con 0x7fc7ec101400 2023-11-16T16:59:07.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.007+0000 7fc7ebfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7ec1063f0 0x7fc7ec1a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.013 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.007+0000 7fc7ebfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7ec1063f0 0x7fc7ec1a4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35400/0 (socket says 172.21.15.44:35400) 2023-11-16T16:59:07.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.007+0000 7fc7ebfff700 1 -- 172.21.15.44:0/3642113372 learned_addr learned my addr 172.21.15.44:0/3642113372 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:07.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.007+0000 7fc7eaffd700 1 --2- 172.21.15.44:0/3642113372 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7ec101400 0x7fc7ec073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.007+0000 7fc7eb7fe700 1 --2- 172.21.15.44:0/3642113372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ec100a50 0x7fc7ec072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.007+0000 7fc7ebfff700 1 -- 172.21.15.44:0/3642113372 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7ec101400 msgr2=0x7fc7ec073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.014 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.007+0000 7fc7ebfff700 1 --2- 172.21.15.44:0/3642113372 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7ec101400 0x7fc7ec073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.007+0000 7fc7ebfff700 1 -- 172.21.15.44:0/3642113372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ec100a50 msgr2=0x7fc7ec072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.007+0000 7fc7ebfff700 1 --2- 172.21.15.44:0/3642113372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ec100a50 0x7fc7ec072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.007+0000 7fc7ebfff700 1 -- 172.21.15.44:0/3642113372 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc0095f0 con 0x7fc7ec1063f0 2023-11-16T16:59:07.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.007+0000 7fc7ebfff700 1 --2- 172.21.15.44:0/3642113372 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7ec1063f0 0x7fc7ec1a4600 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400ef30 tx=0x7fc7e400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:07.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.008+0000 7fc7e8ff9700 1 -- 172.21.15.44:0/3642113372 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4018070 con 0x7fc7ec1063f0 2023-11-16T16:59:07.015 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.008+0000 7fc7f1bff700 1 -- 172.21.15.44:0/3642113372 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ec1103d0 con 0x7fc7ec1063f0 2023-11-16T16:59:07.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.008+0000 7fc7e8ff9700 1 -- 172.21.15.44:0/3642113372 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc7e400f040 con 0x7fc7ec1063f0 2023-11-16T16:59:07.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.008+0000 7fc7e8ff9700 1 -- 172.21.15.44:0/3642113372 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e400cd00 con 0x7fc7ec1063f0 2023-11-16T16:59:07.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.008+0000 7fc7f1bff700 1 -- 172.21.15.44:0/3642113372 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ec110240 con 0x7fc7ec1063f0 2023-11-16T16:59:07.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.011+0000 7fc7e8ff9700 1 -- 172.21.15.44:0/3642113372 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fc7e4003a00 con 0x7fc7ec1063f0 2023-11-16T16:59:07.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.012+0000 7fc805529b80 1 Processor -- start 2023-11-16T16:59:07.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.012+0000 7fc7e8ff9700 1 --2- 172.21.15.44:0/3642113372 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc7d40a12c0 0x7fc7d40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.012+0000 7fc805529b80 1 -- start start 2023-11-16T16:59:07.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.012+0000 7fc7e8ff9700 1 -- 172.21.15.44:0/3642113372 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fc7e40ccc10 con 0x7fc7ec1063f0 2023-11-16T16:59:07.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.012+0000 7fc7eaffd700 1 --2- 172.21.15.44:0/3642113372 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc7d40a12c0 0x7fc7d40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc805529b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x5576949f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc805529b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f4470 0x5576949f8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc805529b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 0x5576949f8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc805529b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576948f9060 con 0x5576949f3b60 2023-11-16T16:59:07.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc805529b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55769494b8a0 con 0x5576949f4470 2023-11-16T16:59:07.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc805529b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x557694944d30 con 0x5576949f88b0 2023-11-16T16:59:07.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc7eb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x5576949f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc7eb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x5576949f3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35408/0 (socket says 172.21.15.44:35408) 2023-11-16T16:59:07.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc7eb7fe700 1 -- 172.21.15.44:0/1914919242 learned_addr learned my addr 172.21.15.44:0/1914919242 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:07.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc7ebfff700 1 --2- 172.21.15.44:0/1914919242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 0x5576949f8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc7eaffd700 1 --2- 172.21.15.44:0/1914919242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f4470 0x5576949f8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc7eaffd700 1 --2- 172.21.15.44:0/3642113372 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc7d40a12c0 0x7fc7d40a3770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec0745e0 tx=0x7fc7e0011440 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:07.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc7eb7fe700 1 -- 172.21.15.44:0/1914919242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 msgr2=0x5576949f8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:07.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc7eb7fe700 1 --2- 172.21.15.44:0/1914919242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 0x5576949f8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc7eb7fe700 1 -- 172.21.15.44:0/1914919242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f4470 msgr2=0x5576949f8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:07.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc7eb7fe700 1 --2- 172.21.15.44:0/1914919242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f4470 0x5576949f8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc7eb7fe700 1 -- 172.21.15.44:0/1914919242 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576947c6630 con 0x5576949f3b60 2023-11-16T16:59:07.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.013+0000 7fc7ebfff700 1 --2- 172.21.15.44:0/1914919242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 0x5576949f8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:07.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.014+0000 7fc7eb7fe700 1 --2- 172.21.15.44:0/1914919242 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x5576949f3f30 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc005ea0 tx=0x7fc7dc005ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771deebbbfd2f8c6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:07.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.014+0000 7fc7d1ffb700 1 -- 172.21.15.44:0/1914919242 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc01d070 con 0x5576949f3b60 2023-11-16T16:59:07.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.014+0000 7fc7d1ffb700 1 -- 172.21.15.44:0/1914919242 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc7dc01e040 con 0x5576949f3b60 2023-11-16T16:59:07.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.014+0000 7fc7d1ffb700 1 -- 172.21.15.44:0/1914919242 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc004920 con 0x5576949f3b60 2023-11-16T16:59:07.029 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.015+0000 7fc805529b80 1 -- 172.21.15.44:0/1914919242 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 msgr2=0x5576949f3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.015+0000 7fc805529b80 1 --2- 172.21.15.44:0/1914919242 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x5576949f3f30 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc005ea0 tx=0x7fc7dc005ed0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.015+0000 7fc805529b80 1 -- 172.21.15.44:0/1914919242 shutdown_connections 2023-11-16T16:59:07.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.015+0000 7fc805529b80 1 --2- 172.21.15.44:0/1914919242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 0x5576949f8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.015+0000 7fc805529b80 1 --2- 172.21.15.44:0/1914919242 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x5576949f3f30 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.015+0000 7fc805529b80 1 --2- 172.21.15.44:0/1914919242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f4470 0x5576949f8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.015+0000 7fc805529b80 1 -- 172.21.15.44:0/1914919242 >> 172.21.15.44:0/1914919242 conn(0x5576949e9360 msgr2=0x5576949eeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:07.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.015+0000 7fc805529b80 1 -- 172.21.15.44:0/1914919242 shutdown_connections 2023-11-16T16:59:07.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.015+0000 7fc805529b80 1 -- 172.21.15.44:0/1914919242 wait complete. 2023-11-16T16:59:07.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.015+0000 7fc805529b80 1 -- start start 2023-11-16T16:59:07.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.016+0000 7fc805529b80 1 Processor -- start 2023-11-16T16:59:07.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.016+0000 7fc805529b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f3b60 0x5576949fff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.016+0000 7fc805529b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f4470 0x5576949fd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.016+0000 7fc805529b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 0x5576949fd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.016+0000 7fc805529b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576948e68e0 con 0x5576949f4470 2023-11-16T16:59:07.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.016+0000 7fc805529b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576948e6580 con 0x5576949f3b60 2023-11-16T16:59:07.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.016+0000 7fc805529b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576947c8920 con 0x5576949f88b0 2023-11-16T16:59:07.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.016+0000 7fc805529b80 10 client.0 _ll_register_callbacks cb 0x557694904780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:59:07.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.016+0000 7fc7eaffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f4470 0x5576949fd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.017+0000 7fc7eaffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f4470 0x5576949fd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35416/0 (socket says 172.21.15.44:35416) 2023-11-16T16:59:07.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.017+0000 7fc7eaffd700 1 -- 172.21.15.44:0/682187191 learned_addr learned my addr 172.21.15.44:0/682187191 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:07.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.017+0000 7fc7eb7fe700 1 --2- 172.21.15.44:0/682187191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f3b60 0x5576949fff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.017+0000 7fc7ebfff700 1 --2- 172.21.15.44:0/682187191 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 0x5576949fd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.017+0000 7fc7eaffd700 1 -- 172.21.15.44:0/682187191 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 msgr2=0x5576949fd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:07.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.017+0000 7fc7eaffd700 1 --2- 172.21.15.44:0/682187191 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 0x5576949fd790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.017+0000 7fc7eaffd700 1 -- 172.21.15.44:0/682187191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f3b60 msgr2=0x5576949fff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:07.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.017+0000 7fc7eaffd700 1 --2- 172.21.15.44:0/682187191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f3b60 0x5576949fff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.017+0000 7fc7eaffd700 1 -- 172.21.15.44:0/682187191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ec10da50 con 0x5576949f4470 2023-11-16T16:59:07.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.017+0000 7fc7eaffd700 1 --2- 172.21.15.44:0/682187191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f4470 0x5576949fd1f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000fac0 tx=0x7fc7e00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d57209028d964da7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:07.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.017+0000 7fc7caffd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:59:07.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.017+0000 7fc7caffd700 1 -- 172.21.15.44:0/682187191 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e001ec70 con 0x5576949f4470 2023-11-16T16:59:07.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.018+0000 7fc7caffd700 1 -- 172.21.15.44:0/682187191 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fc7e001edd0 con 0x5576949f4470 2023-11-16T16:59:07.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.017+0000 7fc805529b80 1 -- 172.21.15.44:0/682187191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5576949e6b20 con 0x5576949f4470 2023-11-16T16:59:07.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.018+0000 7fc7caffd700 1 -- 172.21.15.44:0/682187191 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7e000fe90 con 0x5576949f4470 2023-11-16T16:59:07.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.018+0000 7fc805529b80 10 client.17004 fetch_fsmap learned FSMap version 21 2023-11-16T16:59:07.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.018+0000 7fc805529b80 1 -- 172.21.15.44:0/682187191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55769490b190 con 0x5576949f4470 2023-11-16T16:59:07.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.018+0000 7fc7caffd700 1 -- 172.21.15.44:0/682187191 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5576949e6b20 con 0x5576949f4470 2023-11-16T16:59:07.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.018+0000 7fc7caffd700 1 -- 172.21.15.44:0/682187191 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc7e000e440 con 0x5576949f4470 2023-11-16T16:59:07.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.018+0000 7fc805529b80 10 client.17004 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:59:07.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.018+0000 7fc805529b80 10 client.17004 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:59:07.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.018+0000 7fc805529b80 20 client.17004 populate_metadata read hostname 'smithi044' 2023-11-16T16:59:07.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.018+0000 7fc805529b80 1 --2- 172.21.15.44:0/682187191 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557694a0d840 0x557694a0fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.018+0000 7fc805529b80 4 client.17004 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:59:07.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.018+0000 7fc805529b80 1 -- 172.21.15.44:0/682187191 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5576949070a0 con 0x557694a0d840 2023-11-16T16:59:07.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.019+0000 7fc7eb7fe700 1 --2- 172.21.15.44:0/682187191 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557694a0d840 0x557694a0fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.019+0000 7fc7caffd700 10 client.17004 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:07.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.019+0000 7fc7eb7fe700 1 --2- 172.21.15.44:0/682187191 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557694a0d840 0x557694a0fc50 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d5d54c43f5b6e13 server_cookie=ade356e31930062c in_seq=0 out_seq=0 2023-11-16T16:59:07.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.020+0000 7fc7caffd700 1 -- 172.21.15.44:0/682187191 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5576949070a0 con 0x557694a0d840 2023-11-16T16:59:07.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.020+0000 7fc7caffd700 10 client.17004 handle_command_reply: tid=0 2023-11-16T16:59:07.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.020+0000 7fc805529b80 1 client.17004 shutdown 2023-11-16T16:59:07.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.020+0000 7fc805529b80 1 -- 172.21.15.44:0/682187191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f4470 msgr2=0x5576949fd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.020+0000 7fc805529b80 1 --2- 172.21.15.44:0/682187191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f4470 0x5576949fd1f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000fac0 tx=0x7fc7e00047c0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.038 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.021+0000 7fc805529b80 1 -- 172.21.15.44:0/682187191 shutdown_connections 2023-11-16T16:59:07.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.021+0000 7fc805529b80 1 --2- 172.21.15.44:0/682187191 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557694a0d840 0x557694a0fc50 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.021+0000 7fc805529b80 1 --2- 172.21.15.44:0/682187191 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 0x5576949fd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.021+0000 7fc805529b80 1 --2- 172.21.15.44:0/682187191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f3b60 0x5576949fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.021+0000 7fc805529b80 1 --2- 172.21.15.44:0/682187191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f4470 0x5576949fd1f0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.021+0000 7fc805529b80 1 -- 172.21.15.44:0/682187191 >> 172.21.15.44:0/682187191 conn(0x5576949e9360 msgr2=0x5576949edde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:07.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.021+0000 7fc7caffd700 0 client.17004 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:07.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.021+0000 7fc805529b80 1 -- 172.21.15.44:0/682187191 shutdown_connections 2023-11-16T16:59:07.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.021+0000 7fc805529b80 1 -- 172.21.15.44:0/682187191 wait complete. 2023-11-16T16:59:07.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.021+0000 7fc805529b80 20 client.17004 trim_cache size 0 max 16384 2023-11-16T16:59:07.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.038+0000 7fc805529b80 1 Processor -- start 2023-11-16T16:59:07.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.038+0000 7fc805529b80 1 -- start start 2023-11-16T16:59:07.046 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.039+0000 7fc805529b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x557694a04950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.039+0000 7fc805529b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f4470 0x5576949ee560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.039+0000 7fc805529b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 0x5576949eeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.039+0000 7fc805529b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x557694a0aec0 con 0x5576949f3b60 2023-11-16T16:59:07.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.039+0000 7fc805529b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576949e8ee0 con 0x5576949f4470 2023-11-16T16:59:07.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.039+0000 7fc805529b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x557694a04e90 con 0x5576949f88b0 2023-11-16T16:59:07.047 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.039+0000 7fc7eb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x557694a04950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.048 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.039+0000 7fc7eb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x557694a04950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35430/0 (socket says 172.21.15.44:35430) 2023-11-16T16:59:07.048 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.039+0000 7fc7eb7fe700 1 -- 172.21.15.44:0/3595757717 learned_addr learned my addr 172.21.15.44:0/3595757717 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:07.048 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.039+0000 7fc7eaffd700 1 --2- 172.21.15.44:0/3595757717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f4470 0x5576949ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.048 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.039+0000 7fc7ebfff700 1 --2- 172.21.15.44:0/3595757717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 0x5576949eeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.048 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.039+0000 7fc7eb7fe700 1 -- 172.21.15.44:0/3595757717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 msgr2=0x5576949eeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:07.048 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.039+0000 7fc7eb7fe700 1 --2- 172.21.15.44:0/3595757717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 0x5576949eeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.039+0000 7fc7eb7fe700 1 -- 172.21.15.44:0/3595757717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f4470 msgr2=0x5576949ee560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:07.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.039+0000 7fc7eb7fe700 1 --2- 172.21.15.44:0/3595757717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f4470 0x5576949ee560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.039+0000 7fc7eb7fe700 1 -- 172.21.15.44:0/3595757717 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55769490aee0 con 0x5576949f3b60 2023-11-16T16:59:07.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.040+0000 7fc7ebfff700 1 --2- 172.21.15.44:0/3595757717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 0x5576949eeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:59:07.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.040+0000 7fc7eb7fe700 1 --2- 172.21.15.44:0/3595757717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x557694a04950 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc005ea0 tx=0x7fc7dc004500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40bd48ac92e8905e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:07.049 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.040+0000 7fc7d27fc700 1 -- 172.21.15.44:0/3595757717 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc01d040 con 0x5576949f3b60 2023-11-16T16:59:07.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.040+0000 7fc7d27fc700 1 -- 172.21.15.44:0/3595757717 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc7dc005950 con 0x5576949f3b60 2023-11-16T16:59:07.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.040+0000 7fc7d27fc700 1 -- 172.21.15.44:0/3595757717 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc033cb0 con 0x5576949f3b60 2023-11-16T16:59:07.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.041+0000 7fc805529b80 1 -- 172.21.15.44:0/3595757717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 msgr2=0x557694a04950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.041+0000 7fc805529b80 1 --2- 172.21.15.44:0/3595757717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x557694a04950 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc005ea0 tx=0x7fc7dc004500 comp rx=0 tx=0).stop 2023-11-16T16:59:07.050 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.041+0000 7fc805529b80 1 -- 172.21.15.44:0/3595757717 shutdown_connections 2023-11-16T16:59:07.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.041+0000 7fc805529b80 1 --2- 172.21.15.44:0/3595757717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f88b0 0x5576949eeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.041+0000 7fc805529b80 1 --2- 172.21.15.44:0/3595757717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x557694a04950 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.041+0000 7fc805529b80 1 --2- 172.21.15.44:0/3595757717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f4470 0x5576949ee560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.041+0000 7fc805529b80 1 -- 172.21.15.44:0/3595757717 >> 172.21.15.44:0/3595757717 conn(0x5576949e9360 msgr2=0x5576949f7910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:07.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.041+0000 7fc805529b80 1 -- 172.21.15.44:0/3595757717 shutdown_connections 2023-11-16T16:59:07.051 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.041+0000 7fc805529b80 1 -- 172.21.15.44:0/3595757717 wait complete. 2023-11-16T16:59:07.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.041+0000 7fc805529b80 1 -- start start 2023-11-16T16:59:07.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.041+0000 7fc805529b80 1 Processor -- start 2023-11-16T16:59:07.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc805529b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x557694a02190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc805529b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f4470 0x557694a026d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.052 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc805529b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f88b0 0x557694a606c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc805529b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x557694a03010 con 0x5576949f3b60 2023-11-16T16:59:07.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc805529b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x557694a60c00 con 0x5576949f88b0 2023-11-16T16:59:07.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc805529b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x557694a60d40 con 0x5576949f4470 2023-11-16T16:59:07.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc805529b80 10 client.0 _ll_register_callbacks cb 0x7fc7e000f880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:59:07.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc7eb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x557694a02190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.053 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc7eb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x557694a02190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35444/0 (socket says 172.21.15.44:35444) 2023-11-16T16:59:07.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc7eb7fe700 1 -- 172.21.15.44:0/1643475220 learned_addr learned my addr 172.21.15.44:0/1643475220 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:07.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc7eaffd700 1 --2- 172.21.15.44:0/1643475220 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f4470 0x557694a026d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc7ebfff700 1 --2- 172.21.15.44:0/1643475220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f88b0 0x557694a606c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc7eb7fe700 1 -- 172.21.15.44:0/1643475220 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f4470 msgr2=0x557694a026d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:07.054 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc7eb7fe700 1 --2- 172.21.15.44:0/1643475220 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f4470 0x557694a026d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc7eb7fe700 1 -- 172.21.15.44:0/1643475220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f88b0 msgr2=0x557694a606c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:07.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc7eb7fe700 1 --2- 172.21.15.44:0/1643475220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f88b0 0x557694a606c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc7eb7fe700 1 -- 172.21.15.44:0/1643475220 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5576947c6630 con 0x5576949f3b60 2023-11-16T16:59:07.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.042+0000 7fc7ebfff700 1 --2- 172.21.15.44:0/1643475220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f88b0 0x557694a606c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:59:07.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.043+0000 7fc7eb7fe700 1 --2- 172.21.15.44:0/1643475220 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x557694a02190 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc00b360 tx=0x7fc7dc00bbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0940be56cf9792 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:07.055 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.043+0000 7fc7caffd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:59:07.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.043+0000 7fc7caffd700 1 -- 172.21.15.44:0/1643475220 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc01d070 con 0x5576949f3b60 2023-11-16T16:59:07.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.043+0000 7fc805529b80 1 -- 172.21.15.44:0/1643475220 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc7dc01e040 con 0x5576949f3b60 2023-11-16T16:59:07.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.043+0000 7fc7caffd700 1 -- 172.21.15.44:0/1643475220 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fc7dc033b30 con 0x5576949f3b60 2023-11-16T16:59:07.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.043+0000 7fc7caffd700 1 -- 172.21.15.44:0/1643475220 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7dc0336e0 con 0x5576949f3b60 2023-11-16T16:59:07.056 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.043+0000 7fc805529b80 10 client.17010 fetch_fsmap learned FSMap version 21 2023-11-16T16:59:07.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.043+0000 7fc805529b80 1 -- 172.21.15.44:0/1643475220 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55769475bc60 con 0x5576949f3b60 2023-11-16T16:59:07.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.043+0000 7fc7caffd700 1 -- 172.21.15.44:0/1643475220 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc7dc01e040 con 0x5576949f3b60 2023-11-16T16:59:07.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.044+0000 7fc7caffd700 1 -- 172.21.15.44:0/1643475220 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc7dc004840 con 0x5576949f3b60 2023-11-16T16:59:07.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.044+0000 7fc805529b80 10 client.17010 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:59:07.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.044+0000 7fc805529b80 10 client.17010 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:59:07.057 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.044+0000 7fc805529b80 20 client.17010 populate_metadata read hostname 'smithi044' 2023-11-16T16:59:07.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.044+0000 7fc805529b80 1 --2- 172.21.15.44:0/1643475220 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557694a61f60 0x557694a64340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.044+0000 7fc805529b80 4 client.17010 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:59:07.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.044+0000 7fc805529b80 1 -- 172.21.15.44:0/1643475220 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557694913660 con 0x557694a61f60 2023-11-16T16:59:07.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.044+0000 7fc7eaffd700 1 --2- 172.21.15.44:0/1643475220 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557694a61f60 0x557694a64340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.058 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.045+0000 7fc7eaffd700 1 --2- 172.21.15.44:0/1643475220 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557694a61f60 0x557694a64340 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f49d60cdbe0b3299 server_cookie=a69ce4ce72728749 in_seq=0 out_seq=0 2023-11-16T16:59:07.059 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.045+0000 7fc7caffd700 10 client.17010 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:07.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.045+0000 7fc7caffd700 1 -- 172.21.15.44:0/1643475220 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x557694913660 con 0x557694a61f60 2023-11-16T16:59:07.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.045+0000 7fc7caffd700 10 client.17010 handle_command_reply: tid=0 2023-11-16T16:59:07.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.045+0000 7fc805529b80 1 client.17010 shutdown 2023-11-16T16:59:07.060 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.045+0000 7fc805529b80 1 -- 172.21.15.44:0/1643475220 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 msgr2=0x557694a02190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.045+0000 7fc805529b80 1 --2- 172.21.15.44:0/1643475220 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x557694a02190 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc00b360 tx=0x7fc7dc00bbe0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.045+0000 7fc805529b80 1 -- 172.21.15.44:0/1643475220 shutdown_connections 2023-11-16T16:59:07.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.045+0000 7fc805529b80 1 --2- 172.21.15.44:0/1643475220 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x557694a61f60 0x557694a64340 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.045+0000 7fc805529b80 1 --2- 172.21.15.44:0/1643475220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576949f88b0 0x557694a606c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.045+0000 7fc805529b80 1 --2- 172.21.15.44:0/1643475220 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576949f3b60 0x557694a02190 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.061 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.045+0000 7fc805529b80 1 --2- 172.21.15.44:0/1643475220 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5576949f4470 0x557694a026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.045+0000 7fc7caffd700 0 client.17010 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:07.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.045+0000 7fc805529b80 1 -- 172.21.15.44:0/1643475220 >> 172.21.15.44:0/1643475220 conn(0x5576949e9360 msgr2=0x5576949f2960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:07.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.045+0000 7fc805529b80 1 -- 172.21.15.44:0/1643475220 shutdown_connections 2023-11-16T16:59:07.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.045+0000 7fc805529b80 1 -- 172.21.15.44:0/1643475220 wait complete. 2023-11-16T16:59:07.062 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.046+0000 7fc805529b80 20 client.17010 trim_cache size 0 max 16384 2023-11-16T16:59:07.063 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:59:07.063 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T16:59:07.063 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T16:59:07.063 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:59:07.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.046+0000 7fc7f1bff700 1 -- 172.21.15.44:0/3642113372 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc7d40a12c0 msgr2=0x7fc7d40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.046+0000 7fc7f1bff700 1 --2- 172.21.15.44:0/3642113372 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc7d40a12c0 0x7fc7d40a3770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec0745e0 tx=0x7fc7e0011440 comp rx=0 tx=0).stop 2023-11-16T16:59:07.064 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.046+0000 7fc7f1bff700 1 -- 172.21.15.44:0/3642113372 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7ec1063f0 msgr2=0x7fc7ec1a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.046+0000 7fc7f1bff700 1 --2- 172.21.15.44:0/3642113372 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7ec1063f0 0x7fc7ec1a4600 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400ef30 tx=0x7fc7e400c3a0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.046+0000 7fc7f1bff700 1 -- 172.21.15.44:0/3642113372 shutdown_connections 2023-11-16T16:59:07.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.046+0000 7fc7f1bff700 1 --2- 172.21.15.44:0/3642113372 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7ec1063f0 0x7fc7ec1a4600 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.046+0000 7fc7f1bff700 1 --2- 172.21.15.44:0/3642113372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ec100a50 0x7fc7ec072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.046+0000 7fc7f1bff700 1 --2- 172.21.15.44:0/3642113372 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fc7d40a12c0 0x7fc7d40a3770 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.065 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.046+0000 7fc7f1bff700 1 --2- 172.21.15.44:0/3642113372 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7ec101400 0x7fc7ec073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.046+0000 7fc7f1bff700 1 -- 172.21.15.44:0/3642113372 >> 172.21.15.44:0/3642113372 conn(0x7fc7ec077590 msgr2=0x7fc7ec0ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:07.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.046+0000 7fc7f1bff700 1 -- 172.21.15.44:0/3642113372 shutdown_connections 2023-11-16T16:59:07.066 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.046+0000 7fc7f1bff700 1 -- 172.21.15.44:0/3642113372 wait complete. 2023-11-16T16:59:07.067 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T16:59:07.067 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:59:07.273 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.270+0000 7efef91ff700 1 -- 172.21.15.44:0/1844189684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef4100560 msgr2=0x7efef4100930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.274 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.270+0000 7efef91ff700 1 --2- 172.21.15.44:0/1844189684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef4100560 0x7efef4100930 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7efee4009990 tx=0x7efee4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.274 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.271+0000 7efef91ff700 1 -- 172.21.15.44:0/1844189684 shutdown_connections 2023-11-16T16:59:07.274 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.271+0000 7efef91ff700 1 --2- 172.21.15.44:0/1844189684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efef41055a0 0x7efef410d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.274 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.271+0000 7efef91ff700 1 --2- 172.21.15.44:0/1844189684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef4100560 0x7efef4100930 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.274 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.271+0000 7efef91ff700 1 --2- 172.21.15.44:0/1844189684 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efef4100f00 0x7efef4104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.275 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.271+0000 7efef91ff700 1 -- 172.21.15.44:0/1844189684 >> 172.21.15.44:0/1844189684 conn(0x7efef4077080 msgr2=0x7efef4077480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:07.275 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.271+0000 7efef91ff700 1 -- 172.21.15.44:0/1844189684 shutdown_connections 2023-11-16T16:59:07.275 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.271+0000 7efef91ff700 1 -- 172.21.15.44:0/1844189684 wait complete. 2023-11-16T16:59:07.275 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.272+0000 7efef91ff700 1 Processor -- start 2023-11-16T16:59:07.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.272+0000 7efef91ff700 1 -- start start 2023-11-16T16:59:07.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.272+0000 7efef91ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efef4100560 0x7efef4196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.272+0000 7efef91ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef4100f00 0x7efef41968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.272+0000 7efef91ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efef41055a0 0x7efef419ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.278 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.272+0000 7efef91ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef419b350 con 0x7efef41055a0 2023-11-16T16:59:07.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.272+0000 7efef91ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef419b4c0 con 0x7efef4100f00 2023-11-16T16:59:07.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.272+0000 7efef91ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef419b680 con 0x7efef4100560 2023-11-16T16:59:07.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.273+0000 7efef359e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efef41055a0 0x7efef419ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.273+0000 7efef259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef4100f00 0x7efef41968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.279 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.273+0000 7efef259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef4100f00 0x7efef41968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:35284/0 (socket says 172.21.15.44:35284) 2023-11-16T16:59:07.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.273+0000 7efef359e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efef41055a0 0x7efef419ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35454/0 (socket says 172.21.15.44:35454) 2023-11-16T16:59:07.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.273+0000 7efef259c700 1 -- 172.21.15.44:0/2315715925 learned_addr learned my addr 172.21.15.44:0/2315715925 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:07.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.273+0000 7efef2d9d700 1 --2- 172.21.15.44:0/2315715925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efef4100560 0x7efef4196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.273+0000 7efef259c700 1 -- 172.21.15.44:0/2315715925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efef4100560 msgr2=0x7efef4196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.280 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.273+0000 7efef259c700 1 --2- 172.21.15.44:0/2315715925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efef4100560 0x7efef4196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.273+0000 7efef259c700 1 -- 172.21.15.44:0/2315715925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efef41055a0 msgr2=0x7efef419ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.273+0000 7efef259c700 1 --2- 172.21.15.44:0/2315715925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efef41055a0 0x7efef419ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.273+0000 7efef259c700 1 -- 172.21.15.44:0/2315715925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee40095f0 con 0x7efef4100f00 2023-11-16T16:59:07.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.273+0000 7efef259c700 1 --2- 172.21.15.44:0/2315715925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef4100f00 0x7efef41968b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7efee800e940 tx=0x7efee800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:07.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.273+0000 7efedbfff700 1 -- 172.21.15.44:0/2315715925 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8009730 con 0x7efef4100f00 2023-11-16T16:59:07.281 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.273+0000 7efef91ff700 1 -- 172.21.15.44:0/2315715925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef41103b0 con 0x7efef4100f00 2023-11-16T16:59:07.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.274+0000 7efedbfff700 1 -- 172.21.15.44:0/2315715925 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efee8010430 con 0x7efef4100f00 2023-11-16T16:59:07.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.274+0000 7efedbfff700 1 -- 172.21.15.44:0/2315715925 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee800cea0 con 0x7efef4100f00 2023-11-16T16:59:07.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.274+0000 7efef91ff700 1 -- 172.21.15.44:0/2315715925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef4110220 con 0x7efef4100f00 2023-11-16T16:59:07.282 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.275+0000 7efef91ff700 1 -- 172.21.15.44:0/2315715925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef4110540 con 0x7efef4100f00 2023-11-16T16:59:07.283 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.277+0000 7efedbfff700 1 -- 172.21.15.44:0/2315715925 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7efee8010810 con 0x7efef4100f00 2023-11-16T16:59:07.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.278+0000 7efedbfff700 1 --2- 172.21.15.44:0/2315715925 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7efedc0a12c0 0x7efedc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.279+0000 7efedbfff700 1 -- 172.21.15.44:0/2315715925 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7efee8014070 con 0x7efef4100f00 2023-11-16T16:59:07.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.279+0000 7efef2d9d700 1 --2- 172.21.15.44:0/2315715925 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7efedc0a12c0 0x7efedc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.279+0000 7efef2d9d700 1 --2- 172.21.15.44:0/2315715925 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7efedc0a12c0 0x7efedc0a3770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efee400b2e0 tx=0x7efee4005fb0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:07.284 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.280+0000 7efedbfff700 1 -- 172.21.15.44:0/2315715925 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efee8096460 con 0x7efef4100f00 2023-11-16T16:59:07.539 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.537+0000 7efef91ff700 1 -- 172.21.15.44:0/2315715925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efef419bb50 con 0x7efef4100f00 2023-11-16T16:59:07.540 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.539+0000 7efedbfff700 1 -- 172.21.15.44:0/2315715925 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7efee8099a80 con 0x7efef4100f00 2023-11-16T16:59:07.540 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:59:07.541 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:59:07.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.540+0000 7efef91ff700 1 -- 172.21.15.44:0/2315715925 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7efedc0a12c0 msgr2=0x7efedc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.540+0000 7efef91ff700 1 --2- 172.21.15.44:0/2315715925 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7efedc0a12c0 0x7efedc0a3770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efee400b2e0 tx=0x7efee4005fb0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.543 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.541+0000 7efef91ff700 1 -- 172.21.15.44:0/2315715925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef4100f00 msgr2=0x7efef41968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.544 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.541+0000 7efef91ff700 1 --2- 172.21.15.44:0/2315715925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef4100f00 0x7efef41968b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7efee800e940 tx=0x7efee800ec50 comp rx=0 tx=0).stop 2023-11-16T16:59:07.544 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.541+0000 7efef91ff700 1 -- 172.21.15.44:0/2315715925 shutdown_connections 2023-11-16T16:59:07.544 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.541+0000 7efef91ff700 1 --2- 172.21.15.44:0/2315715925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efef41055a0 0x7efef419ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.544 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.541+0000 7efef91ff700 1 --2- 172.21.15.44:0/2315715925 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7efedc0a12c0 0x7efedc0a3770 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.544 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.541+0000 7efef91ff700 1 --2- 172.21.15.44:0/2315715925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efef4100560 0x7efef4196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.541+0000 7efef91ff700 1 --2- 172.21.15.44:0/2315715925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef4100f00 0x7efef41968b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.541+0000 7efef91ff700 1 -- 172.21.15.44:0/2315715925 >> 172.21.15.44:0/2315715925 conn(0x7efef4077080 msgr2=0x7efef40fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:07.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.541+0000 7efef91ff700 1 -- 172.21.15.44:0/2315715925 shutdown_connections 2023-11-16T16:59:07.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.541+0000 7efef91ff700 1 -- 172.21.15.44:0/2315715925 wait complete. 2023-11-16T16:59:07.546 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:59:07.562 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T16:59:07.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.768+0000 7f0a95bff700 1 -- 172.21.15.44:0/687993045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a90101660 msgr2=0x7f0a90104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.770 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.768+0000 7f0a95bff700 1 --2- 172.21.15.44:0/687993045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a90101660 0x7f0a90104ef0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0a780099c0 tx=0x7f0a78009cd0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.768+0000 7f0a95bff700 1 -- 172.21.15.44:0/687993045 shutdown_connections 2023-11-16T16:59:07.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.768+0000 7f0a95bff700 1 --2- 172.21.15.44:0/687993045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a90105620 0x7f0a9010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.768+0000 7f0a95bff700 1 --2- 172.21.15.44:0/687993045 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a90100cc0 0x7f0a90101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.768+0000 7f0a95bff700 1 --2- 172.21.15.44:0/687993045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a90101660 0x7f0a90104ef0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.771 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.768+0000 7f0a95bff700 1 -- 172.21.15.44:0/687993045 >> 172.21.15.44:0/687993045 conn(0x7f0a90077560 msgr2=0x7f0a90077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:07.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.768+0000 7f0a95bff700 1 -- 172.21.15.44:0/687993045 shutdown_connections 2023-11-16T16:59:07.772 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.768+0000 7f0a95bff700 1 -- 172.21.15.44:0/687993045 wait complete. 2023-11-16T16:59:07.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.769+0000 7f0a95bff700 1 Processor -- start 2023-11-16T16:59:07.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.769+0000 7f0a95bff700 1 -- start start 2023-11-16T16:59:07.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.770+0000 7f0a95bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a90100cc0 0x7f0a90072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.775 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.770+0000 7f0a95bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a90101660 0x7f0a90073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.770+0000 7f0a95bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a90105620 0x7f0a901a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.770+0000 7f0a95bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a90073b60 con 0x7f0a90100cc0 2023-11-16T16:59:07.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.770+0000 7f0a95bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a901a4be0 con 0x7f0a90101660 2023-11-16T16:59:07.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.770+0000 7f0a95bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a901a4d20 con 0x7f0a90105620 2023-11-16T16:59:07.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.770+0000 7f0a8f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a90100cc0 0x7f0a90072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.770+0000 7f0a8f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a90100cc0 0x7f0a90072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35480/0 (socket says 172.21.15.44:35480) 2023-11-16T16:59:07.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.770+0000 7f0a8f7fe700 1 -- 172.21.15.44:0/361531079 learned_addr learned my addr 172.21.15.44:0/361531079 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:07.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.770+0000 7f0a87fff700 1 --2- 172.21.15.44:0/361531079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a90101660 0x7f0a90073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.770+0000 7f0a8ffff700 1 --2- 172.21.15.44:0/361531079 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a90105620 0x7f0a901a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.770+0000 7f0a8f7fe700 1 -- 172.21.15.44:0/361531079 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a90105620 msgr2=0x7f0a901a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.770+0000 7f0a8f7fe700 1 --2- 172.21.15.44:0/361531079 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a90105620 0x7f0a901a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.770+0000 7f0a8f7fe700 1 -- 172.21.15.44:0/361531079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a90101660 msgr2=0x7f0a90073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.770+0000 7f0a8f7fe700 1 --2- 172.21.15.44:0/361531079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a90101660 0x7f0a90073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.770+0000 7f0a8f7fe700 1 -- 172.21.15.44:0/361531079 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a780095f0 con 0x7f0a90100cc0 2023-11-16T16:59:07.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.771+0000 7f0a8f7fe700 1 --2- 172.21.15.44:0/361531079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a90100cc0 0x7f0a90072df0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000e9a0 tx=0x7f0a8000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:07.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.771+0000 7f0a8d7fa700 1 -- 172.21.15.44:0/361531079 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80009790 con 0x7f0a90100cc0 2023-11-16T16:59:07.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.771+0000 7f0a8d7fa700 1 -- 172.21.15.44:0/361531079 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a80010430 con 0x7f0a90100cc0 2023-11-16T16:59:07.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.771+0000 7f0a8d7fa700 1 -- 172.21.15.44:0/361531079 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80018680 con 0x7f0a90100cc0 2023-11-16T16:59:07.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.771+0000 7f0a95bff700 1 -- 172.21.15.44:0/361531079 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a90110370 con 0x7f0a90100cc0 2023-11-16T16:59:07.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.771+0000 7f0a95bff700 1 -- 172.21.15.44:0/361531079 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a901101e0 con 0x7f0a90100cc0 2023-11-16T16:59:07.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.776+0000 7f0a8d7fa700 1 -- 172.21.15.44:0/361531079 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f0a800105a0 con 0x7f0a90100cc0 2023-11-16T16:59:07.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.777+0000 7f0aa95ddb80 1 Processor -- start 2023-11-16T16:59:07.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.777+0000 7f0aa95ddb80 1 -- start start 2023-11-16T16:59:07.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.777+0000 7f0aa95ddb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad4b60 0x55e53dad4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.777+0000 7f0aa95ddb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad5470 0x55e53dad9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.777+0000 7f0aa95ddb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad98b0 0x55e53dad9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.777+0000 7f0aa95ddb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e53d9da060 con 0x55e53dad98b0 2023-11-16T16:59:07.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.777+0000 7f0aa95ddb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e53da2c8a0 con 0x55e53dad5470 2023-11-16T16:59:07.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.777+0000 7f0aa95ddb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e53da25d30 con 0x55e53dad4b60 2023-11-16T16:59:07.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.777+0000 7f0a8d7fa700 1 --2- 172.21.15.44:0/361531079 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0a7c0a12c0 0x7f0a7c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.777+0000 7f0a87fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad98b0 0x55e53dad9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.777+0000 7f0a8d7fa700 1 -- 172.21.15.44:0/361531079 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f0a80014070 con 0x7f0a90100cc0 2023-11-16T16:59:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.777+0000 7f0a8f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad5470 0x55e53dad9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.778+0000 7f0a87fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad98b0 0x55e53dad9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35496/0 (socket says 172.21.15.44:35496) 2023-11-16T16:59:07.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.778+0000 7f0a87fff700 1 -- 172.21.15.44:0/3801286882 learned_addr learned my addr 172.21.15.44:0/3801286882 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.778+0000 7f0a8f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad5470 0x55e53dad9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:35306/0 (socket says 172.21.15.44:35306) 2023-11-16T16:59:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.778+0000 7f0a8ffff700 1 --2- 172.21.15.44:0/3801286882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad4b60 0x55e53dad4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.778+0000 7f0a8ffff700 1 --2- 172.21.15.44:0/361531079 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0a7c0a12c0 0x7f0a7c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.778+0000 7f0a8f7fe700 1 -- 172.21.15.44:0/3801286882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad4b60 msgr2=0x55e53dad4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.778+0000 7f0a8f7fe700 1 --2- 172.21.15.44:0/3801286882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad4b60 0x55e53dad4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.778+0000 7f0a8f7fe700 1 -- 172.21.15.44:0/3801286882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad98b0 msgr2=0x55e53dad9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:07.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.778+0000 7f0a8f7fe700 1 --2- 172.21.15.44:0/3801286882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad98b0 0x55e53dad9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.778+0000 7f0a8f7fe700 1 -- 172.21.15.44:0/3801286882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e53d8a7630 con 0x55e53dad5470 2023-11-16T16:59:07.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.778+0000 7f0a8f7fe700 1 --2- 172.21.15.44:0/3801286882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad5470 0x55e53dad9370 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0a800ca630 tx=0x7f0a80099bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b4e46d372e1603 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:07.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.778+0000 7f0a85ffb700 1 -- 172.21.15.44:0/3801286882 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8009dfe0 con 0x55e53dad5470 2023-11-16T16:59:07.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.778+0000 7f0a85ffb700 1 -- 172.21.15.44:0/3801286882 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a800a2400 con 0x55e53dad5470 2023-11-16T16:59:07.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.778+0000 7f0a85ffb700 1 -- 172.21.15.44:0/3801286882 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80099700 con 0x55e53dad5470 2023-11-16T16:59:07.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.779+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/3801286882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad5470 msgr2=0x55e53dad9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.779+0000 7f0a8ffff700 1 --2- 172.21.15.44:0/361531079 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0a7c0a12c0 0x7f0a7c0a3770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009790 tx=0x7f0a88009130 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:07.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.779+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/3801286882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad5470 0x55e53dad9370 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0a800ca630 tx=0x7f0a80099bc0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.779+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/3801286882 shutdown_connections 2023-11-16T16:59:07.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.779+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/3801286882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad98b0 0x55e53dad9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.779+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/3801286882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad4b60 0x55e53dad4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.779+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/3801286882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad5470 0x55e53dad9370 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.779+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/3801286882 >> 172.21.15.44:0/3801286882 conn(0x55e53daca360 msgr2=0x55e53dacfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:07.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.779+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/3801286882 shutdown_connections 2023-11-16T16:59:07.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.779+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/3801286882 wait complete. 2023-11-16T16:59:07.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.780+0000 7f0aa95ddb80 1 -- start start 2023-11-16T16:59:07.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.780+0000 7f0aa95ddb80 1 Processor -- start 2023-11-16T16:59:07.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.781+0000 7f0aa95ddb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad4b60 0x55e53dae1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.781+0000 7f0aa95ddb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad5470 0x55e53dade280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.781+0000 7f0aa95ddb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad98b0 0x55e53dade8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.781+0000 7f0aa95ddb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e53d9c78e0 con 0x55e53dad5470 2023-11-16T16:59:07.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.781+0000 7f0aa95ddb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e53d9c7580 con 0x55e53dad4b60 2023-11-16T16:59:07.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.781+0000 7f0aa95ddb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e53d8a9920 con 0x55e53dad98b0 2023-11-16T16:59:07.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.781+0000 7f0aa95ddb80 10 client.0 _ll_register_callbacks cb 0x55e53d9e5780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:59:07.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.781+0000 7f0a87fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad5470 0x55e53dade280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.781+0000 7f0a8f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad4b60 0x55e53dae1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.781+0000 7f0a87fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad5470 0x55e53dade280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35502/0 (socket says 172.21.15.44:35502) 2023-11-16T16:59:07.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.782+0000 7f0a8f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad4b60 0x55e53dae1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:35312/0 (socket says 172.21.15.44:35312) 2023-11-16T16:59:07.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.782+0000 7f0a87fff700 1 -- 172.21.15.44:0/2447440091 learned_addr learned my addr 172.21.15.44:0/2447440091 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:07.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.782+0000 7f0a8ffff700 1 --2- 172.21.15.44:0/2447440091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad98b0 0x55e53dade8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.782+0000 7f0a87fff700 1 -- 172.21.15.44:0/2447440091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad98b0 msgr2=0x55e53dade8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:07.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.782+0000 7f0a87fff700 1 --2- 172.21.15.44:0/2447440091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad98b0 0x55e53dade8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.782+0000 7f0a87fff700 1 -- 172.21.15.44:0/2447440091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad4b60 msgr2=0x55e53dae1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:07.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.782+0000 7f0a87fff700 1 --2- 172.21.15.44:0/2447440091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad4b60 0x55e53dae1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.782+0000 7f0a87fff700 1 -- 172.21.15.44:0/2447440091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a901101e0 con 0x55e53dad5470 2023-11-16T16:59:07.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.782+0000 7f0a8f7fe700 1 --2- 172.21.15.44:0/2447440091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad4b60 0x55e53dae1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:07.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.782+0000 7f0a87fff700 1 --2- 172.21.15.44:0/2447440091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad5470 0x55e53dade280 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0a78000c00 tx=0x7f0a78005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606ed25f95b15944 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:07.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.782+0000 7f0a6effd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:59:07.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.783+0000 7f0a6effd700 1 -- 172.21.15.44:0/2447440091 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a78005670 con 0x55e53dad5470 2023-11-16T16:59:07.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.783+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2447440091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e53dac7b20 con 0x55e53dad5470 2023-11-16T16:59:07.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.783+0000 7f0a6effd700 1 -- 172.21.15.44:0/2447440091 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f0a7800b790 con 0x55e53dad5470 2023-11-16T16:59:07.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.783+0000 7f0a6effd700 1 -- 172.21.15.44:0/2447440091 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a7800b950 con 0x55e53dad5470 2023-11-16T16:59:07.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.783+0000 7f0aa95ddb80 10 client.17028 fetch_fsmap learned FSMap version 21 2023-11-16T16:59:07.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.783+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2447440091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55e53d9ec190 con 0x55e53dad5470 2023-11-16T16:59:07.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.783+0000 7f0a6effd700 1 -- 172.21.15.44:0/2447440091 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e53dac7b20 con 0x55e53dad5470 2023-11-16T16:59:07.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.783+0000 7f0a6effd700 1 -- 172.21.15.44:0/2447440091 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0a78024bc0 con 0x55e53dad5470 2023-11-16T16:59:07.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.783+0000 7f0aa95ddb80 10 client.17028 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:59:07.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.783+0000 7f0aa95ddb80 10 client.17028 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:59:07.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.783+0000 7f0aa95ddb80 20 client.17028 populate_metadata read hostname 'smithi044' 2023-11-16T16:59:07.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.783+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/2447440091 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e53daee820 0x55e53daf0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.783+0000 7f0aa95ddb80 4 client.17028 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:59:07.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.783+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2447440091 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e53d9e80a0 con 0x55e53daee820 2023-11-16T16:59:07.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.784+0000 7f0a8f7fe700 1 --2- 172.21.15.44:0/2447440091 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e53daee820 0x55e53daf0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.784+0000 7f0a6effd700 10 client.17028 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:07.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.784+0000 7f0a8f7fe700 1 --2- 172.21.15.44:0/2447440091 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e53daee820 0x55e53daf0c30 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15a894bf06d91414 server_cookie=e519a552d906be7b in_seq=0 out_seq=0 2023-11-16T16:59:07.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.785+0000 7f0a6effd700 1 -- 172.21.15.44:0/2447440091 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e53d9e80a0 con 0x55e53daee820 2023-11-16T16:59:07.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.785+0000 7f0a6effd700 10 client.17028 handle_command_reply: tid=0 2023-11-16T16:59:07.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.785+0000 7f0aa95ddb80 1 client.17028 shutdown 2023-11-16T16:59:07.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.785+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2447440091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad5470 msgr2=0x55e53dade280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.785+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/2447440091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad5470 0x55e53dade280 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0a78000c00 tx=0x7f0a78005040 comp rx=0 tx=0).stop 2023-11-16T16:59:07.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.785+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2447440091 shutdown_connections 2023-11-16T16:59:07.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.785+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/2447440091 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e53daee820 0x55e53daf0c30 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.785+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/2447440091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad98b0 0x55e53dade8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.785+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/2447440091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad4b60 0x55e53dae1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.785+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/2447440091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad5470 0x55e53dade280 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.785+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2447440091 >> 172.21.15.44:0/2447440091 conn(0x55e53daca360 msgr2=0x55e53dadad60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:07.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.785+0000 7f0a6effd700 0 client.17028 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:07.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.785+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2447440091 shutdown_connections 2023-11-16T16:59:07.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.785+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2447440091 wait complete. 2023-11-16T16:59:07.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.786+0000 7f0aa95ddb80 20 client.17028 trim_cache size 0 max 16384 2023-11-16T16:59:07.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.799+0000 7f0aa95ddb80 1 Processor -- start 2023-11-16T16:59:07.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.799+0000 7f0aa95ddb80 1 -- start start 2023-11-16T16:59:07.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.799+0000 7f0aa95ddb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad4b60 0x55e53dae5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.810 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.799+0000 7f0aa95ddb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad5470 0x55e53daf0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.800+0000 7f0aa95ddb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad98b0 0x55e53dae0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.800+0000 7f0aa95ddb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e53dac7a10 con 0x55e53dad4b60 2023-11-16T16:59:07.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.800+0000 7f0aa95ddb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e53dad11c0 con 0x55e53dad98b0 2023-11-16T16:59:07.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.800+0000 7f0aa95ddb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e53dadc800 con 0x55e53dad5470 2023-11-16T16:59:07.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.800+0000 7f0a8f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad4b60 0x55e53dae5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.800+0000 7f0a87fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad5470 0x55e53daf0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.800+0000 7f0a87fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad5470 0x55e53daf0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:53004/0 (socket says 172.21.15.44:53004) 2023-11-16T16:59:07.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.800+0000 7f0a8f7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad4b60 0x55e53dae5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35506/0 (socket says 172.21.15.44:35506) 2023-11-16T16:59:07.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.800+0000 7f0a87fff700 1 -- 172.21.15.44:0/2531918894 learned_addr learned my addr 172.21.15.44:0/2531918894 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:07.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.800+0000 7f0a8ffff700 1 --2- 172.21.15.44:0/2531918894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad98b0 0x55e53dae0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.800+0000 7f0a87fff700 1 -- 172.21.15.44:0/2531918894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad98b0 msgr2=0x55e53dae0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:07.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.800+0000 7f0a87fff700 1 --2- 172.21.15.44:0/2531918894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad98b0 0x55e53dae0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.800+0000 7f0a87fff700 1 -- 172.21.15.44:0/2531918894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad4b60 msgr2=0x55e53dae5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:07.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.800+0000 7f0a87fff700 1 --2- 172.21.15.44:0/2531918894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad4b60 0x55e53dae5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.800+0000 7f0a87fff700 1 -- 172.21.15.44:0/2531918894 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e53d9ebee0 con 0x55e53dad5470 2023-11-16T16:59:07.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.800+0000 7f0a87fff700 1 --2- 172.21.15.44:0/2531918894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad5470 0x55e53daf0830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0a78024f80 tx=0x7f0a7800f4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f125389340b99ea1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:07.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.801+0000 7f0a867fc700 1 -- 172.21.15.44:0/2531918894 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a78038030 con 0x55e53dad5470 2023-11-16T16:59:07.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.801+0000 7f0a867fc700 1 -- 172.21.15.44:0/2531918894 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a7801e040 con 0x55e53dad5470 2023-11-16T16:59:07.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.801+0000 7f0a867fc700 1 -- 172.21.15.44:0/2531918894 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7800f7f0 con 0x55e53dad5470 2023-11-16T16:59:07.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.801+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2531918894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad5470 msgr2=0x55e53daf0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.801+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/2531918894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad5470 0x55e53daf0830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0a78024f80 tx=0x7f0a7800f4e0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.802+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2531918894 shutdown_connections 2023-11-16T16:59:07.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.802+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/2531918894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad98b0 0x55e53dae0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.802+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/2531918894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad4b60 0x55e53dae5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.802+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/2531918894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad5470 0x55e53daf0830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.802+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2531918894 >> 172.21.15.44:0/2531918894 conn(0x55e53daca360 msgr2=0x55e53dace550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:07.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.802+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2531918894 shutdown_connections 2023-11-16T16:59:07.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.802+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2531918894 wait complete. 2023-11-16T16:59:07.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.802+0000 7f0aa95ddb80 1 -- start start 2023-11-16T16:59:07.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.802+0000 7f0aa95ddb80 1 Processor -- start 2023-11-16T16:59:07.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.803+0000 7f0aa95ddb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad4b60 0x55e53daddf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.803+0000 7f0aa95ddb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad5470 0x55e53dae01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.803+0000 7f0aa95ddb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad98b0 0x55e53dae0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.803+0000 7f0aa95ddb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e53dade510 con 0x55e53dad98b0 2023-11-16T16:59:07.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.803+0000 7f0aa95ddb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e53dade680 con 0x55e53dad5470 2023-11-16T16:59:07.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.803+0000 7f0aa95ddb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e53dae0c70 con 0x55e53dad4b60 2023-11-16T16:59:07.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.803+0000 7f0aa95ddb80 10 client.0 _ll_register_callbacks cb 0x55e53d9e5780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:59:07.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.803+0000 7f0a8ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad98b0 0x55e53dae0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.803+0000 7f0a87fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad5470 0x55e53dae01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.804+0000 7f0a8ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad98b0 0x55e53dae0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35516/0 (socket says 172.21.15.44:35516) 2023-11-16T16:59:07.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.804+0000 7f0a87fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad5470 0x55e53dae01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:35328/0 (socket says 172.21.15.44:35328) 2023-11-16T16:59:07.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.804+0000 7f0a8ffff700 1 -- 172.21.15.44:0/2675586424 learned_addr learned my addr 172.21.15.44:0/2675586424 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:07.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.804+0000 7f0a8f7fe700 1 --2- 172.21.15.44:0/2675586424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad4b60 0x55e53daddf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.804+0000 7f0a87fff700 1 -- 172.21.15.44:0/2675586424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad4b60 msgr2=0x55e53daddf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:07.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.804+0000 7f0a87fff700 1 --2- 172.21.15.44:0/2675586424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad4b60 0x55e53daddf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.804+0000 7f0a87fff700 1 -- 172.21.15.44:0/2675586424 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad98b0 msgr2=0x55e53dae0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:07.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.804+0000 7f0a87fff700 1 --2- 172.21.15.44:0/2675586424 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad98b0 0x55e53dae0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.804+0000 7f0a87fff700 1 -- 172.21.15.44:0/2675586424 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e53d9ec190 con 0x55e53dad5470 2023-11-16T16:59:07.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.804+0000 7f0a8ffff700 1 --2- 172.21.15.44:0/2675586424 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad98b0 0x55e53dae0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:07.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.804+0000 7f0a87fff700 1 --2- 172.21.15.44:0/2675586424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad5470 0x55e53dae01f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x55e53dad5d40 tx=0x7f0a78005d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ba98fd3c39d6d4f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:07.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.804+0000 7f0a6effd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:59:07.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.805+0000 7f0a6effd700 1 -- 172.21.15.44:0/2675586424 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a78038070 con 0x55e53dad5470 2023-11-16T16:59:07.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.805+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2675586424 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0a800a2400 con 0x55e53dad5470 2023-11-16T16:59:07.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.805+0000 7f0a6effd700 1 -- 172.21.15.44:0/2675586424 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f0a78039070 con 0x55e53dad5470 2023-11-16T16:59:07.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.805+0000 7f0a6effd700 1 -- 172.21.15.44:0/2675586424 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a78017600 con 0x55e53dad5470 2023-11-16T16:59:07.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.805+0000 7f0aa95ddb80 10 client.27113 fetch_fsmap learned FSMap version 21 2023-11-16T16:59:07.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.805+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2675586424 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55e53d83cc60 con 0x55e53dad5470 2023-11-16T16:59:07.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.805+0000 7f0a6effd700 1 -- 172.21.15.44:0/2675586424 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a800a2400 con 0x55e53dad5470 2023-11-16T16:59:07.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.805+0000 7f0a6effd700 1 -- 172.21.15.44:0/2675586424 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0a7802c070 con 0x55e53dad5470 2023-11-16T16:59:07.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.805+0000 7f0aa95ddb80 10 client.27113 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:59:07.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.805+0000 7f0aa95ddb80 10 client.27113 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:59:07.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.805+0000 7f0aa95ddb80 20 client.27113 populate_metadata read hostname 'smithi044' 2023-11-16T16:59:07.820 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.805+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/2675586424 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e53db42400 0x55e53db44810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:07.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.805+0000 7f0aa95ddb80 4 client.27113 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T16:59:07.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.805+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2675586424 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e53d9f4660 con 0x55e53db42400 2023-11-16T16:59:07.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.806+0000 7f0a8f7fe700 1 --2- 172.21.15.44:0/2675586424 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e53db42400 0x55e53db44810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:07.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.806+0000 7f0a8f7fe700 1 --2- 172.21.15.44:0/2675586424 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e53db42400 0x55e53db44810 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3eb0093433c6ae1 server_cookie=4cd843a1e120c840 in_seq=0 out_seq=0 2023-11-16T16:59:07.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.806+0000 7f0a6effd700 10 client.27113 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:07.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.806+0000 7f0a6effd700 1 -- 172.21.15.44:0/2675586424 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e53d9f4660 con 0x55e53db42400 2023-11-16T16:59:07.821 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.806+0000 7f0a6effd700 10 client.27113 handle_command_reply: tid=0 2023-11-16T16:59:07.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.806+0000 7f0aa95ddb80 1 client.27113 shutdown 2023-11-16T16:59:07.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.806+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2675586424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad5470 msgr2=0x55e53dae01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.806+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/2675586424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad5470 0x55e53dae01f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x55e53dad5d40 tx=0x7f0a78005d30 comp rx=0 tx=0).stop 2023-11-16T16:59:07.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.807+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2675586424 shutdown_connections 2023-11-16T16:59:07.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.807+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/2675586424 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55e53db42400 0x55e53db44810 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.807+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/2675586424 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e53dad98b0 0x55e53dae0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.807+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/2675586424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55e53dad4b60 0x55e53daddf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.807+0000 7f0aa95ddb80 1 --2- 172.21.15.44:0/2675586424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e53dad5470 0x55e53dae01f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.807+0000 7f0a6effd700 0 client.27113 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:07.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.807+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2675586424 >> 172.21.15.44:0/2675586424 conn(0x55e53daca360 msgr2=0x55e53daea0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:07.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.807+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2675586424 shutdown_connections 2023-11-16T16:59:07.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.807+0000 7f0aa95ddb80 1 -- 172.21.15.44:0/2675586424 wait complete. 2023-11-16T16:59:07.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.807+0000 7f0aa95ddb80 20 client.27113 trim_cache size 0 max 16384 2023-11-16T16:59:07.824 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T16:59:07.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.807+0000 7f0a95bff700 1 -- 172.21.15.44:0/361531079 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0a7c0a12c0 msgr2=0x7f0a7c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.807+0000 7f0a95bff700 1 --2- 172.21.15.44:0/361531079 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0a7c0a12c0 0x7f0a7c0a3770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009790 tx=0x7f0a88009130 comp rx=0 tx=0).stop 2023-11-16T16:59:07.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.807+0000 7f0a95bff700 1 -- 172.21.15.44:0/361531079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a90100cc0 msgr2=0x7f0a90072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:07.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.807+0000 7f0a95bff700 1 --2- 172.21.15.44:0/361531079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a90100cc0 0x7f0a90072df0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000e9a0 tx=0x7f0a8000ecb0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.826 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.808+0000 7f0a95bff700 1 -- 172.21.15.44:0/361531079 shutdown_connections 2023-11-16T16:59:07.826 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.808+0000 7f0a95bff700 1 --2- 172.21.15.44:0/361531079 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0a7c0a12c0 0x7f0a7c0a3770 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.826 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.808+0000 7f0a95bff700 1 --2- 172.21.15.44:0/361531079 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a90105620 0x7f0a901a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.826 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.808+0000 7f0a95bff700 1 --2- 172.21.15.44:0/361531079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a90100cc0 0x7f0a90072df0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.826 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.808+0000 7f0a95bff700 1 --2- 172.21.15.44:0/361531079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a90101660 0x7f0a90073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:07.826 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.808+0000 7f0a95bff700 1 -- 172.21.15.44:0/361531079 >> 172.21.15.44:0/361531079 conn(0x7f0a90077560 msgr2=0x7f0a90100210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:07.827 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.808+0000 7f0a95bff700 1 -- 172.21.15.44:0/361531079 shutdown_connections 2023-11-16T16:59:07.827 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:07.808+0000 7f0a95bff700 1 -- 172.21.15.44:0/361531079 wait complete. 2023-11-16T16:59:07.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:07 smithi044 ceph-mon[128750]: pgmap v284: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 45 KiB/s wr, 16 op/s 2023-11-16T16:59:07.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:07 smithi176 ceph-mon[131630]: pgmap v284: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 45 KiB/s wr, 16 op/s 2023-11-16T16:59:07.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:07 smithi077 ceph-mon[131204]: pgmap v284: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 45 KiB/s wr, 16 op/s 2023-11-16T16:59:08.829 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:59:08.829 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:59:08.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:08 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2315715925' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:59:08.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:08 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2315715925' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:59:08.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:08 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2315715925' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:59:09.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.222+0000 7f32f55ff700 1 -- 172.21.15.44:0/210885456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32f0107af0 msgr2=0x7f32f0107ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.222+0000 7f32f55ff700 1 --2- 172.21.15.44:0/210885456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32f0107af0 0x7f32f0107ec0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f32e00098f0 tx=0x7f32e0009c00 comp rx=0 tx=0).stop 2023-11-16T16:59:09.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.223+0000 7f32f55ff700 1 -- 172.21.15.44:0/210885456 shutdown_connections 2023-11-16T16:59:09.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.223+0000 7f32f55ff700 1 --2- 172.21.15.44:0/210885456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32f01133b0 0x7f32f0115790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.223+0000 7f32f55ff700 1 --2- 172.21.15.44:0/210885456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32f0107af0 0x7f32f0107ec0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.223+0000 7f32f55ff700 1 --2- 172.21.15.44:0/210885456 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32f0108400 0x7f32f0112e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.223+0000 7f32f55ff700 1 -- 172.21.15.44:0/210885456 >> 172.21.15.44:0/210885456 conn(0x7f32f006c7b0 msgr2=0x7f32f0071770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:09.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.223+0000 7f32f55ff700 1 -- 172.21.15.44:0/210885456 shutdown_connections 2023-11-16T16:59:09.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.223+0000 7f32f55ff700 1 -- 172.21.15.44:0/210885456 wait complete. 2023-11-16T16:59:09.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.226+0000 7f32f55ff700 1 Processor -- start 2023-11-16T16:59:09.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.227+0000 7f32f55ff700 1 -- start start 2023-11-16T16:59:09.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.228+0000 7f32f55ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32f0107af0 0x7f32f0112b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.228+0000 7f32f55ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32f0108400 0x7f32f010fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.228+0000 7f32f55ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32f01133b0 0x7f32f0110070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.228+0000 7f32f55ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f01106d0 con 0x7f32f0107af0 2023-11-16T16:59:09.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.228+0000 7f32f55ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f0110840 con 0x7f32f01133b0 2023-11-16T16:59:09.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.228+0000 7f32f55ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f0110a00 con 0x7f32f0108400 2023-11-16T16:59:09.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.228+0000 7f32eeffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32f0107af0 0x7f32f0112b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.228+0000 7f32eeffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32f0107af0 0x7f32f0112b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35522/0 (socket says 172.21.15.44:35522) 2023-11-16T16:59:09.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.228+0000 7f32eeffd700 1 -- 172.21.15.44:0/2371374531 learned_addr learned my addr 172.21.15.44:0/2371374531 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:09.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.229+0000 7f32eeffd700 1 -- 172.21.15.44:0/2371374531 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32f0108400 msgr2=0x7f32f010fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.229+0000 7f32eeffd700 1 --2- 172.21.15.44:0/2371374531 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32f0108400 0x7f32f010fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.229+0000 7f32eeffd700 1 -- 172.21.15.44:0/2371374531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32f01133b0 msgr2=0x7f32f0110070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.229+0000 7f32eeffd700 1 --2- 172.21.15.44:0/2371374531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32f01133b0 0x7f32f0110070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.229+0000 7f32eeffd700 1 -- 172.21.15.44:0/2371374531 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e0009520 con 0x7f32f0107af0 2023-11-16T16:59:09.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.230+0000 7f32eeffd700 1 --2- 172.21.15.44:0/2371374531 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32f0107af0 0x7f32f0112b20 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f32e0000c00 tx=0x7f32e000b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:09.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.231+0000 7f32d7fff700 1 -- 172.21.15.44:0/2371374531 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e001c070 con 0x7f32f0107af0 2023-11-16T16:59:09.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.231+0000 7f32f55ff700 1 -- 172.21.15.44:0/2371374531 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f010a7e0 con 0x7f32f0107af0 2023-11-16T16:59:09.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.231+0000 7f32f55ff700 1 -- 172.21.15.44:0/2371374531 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f010a650 con 0x7f32f0107af0 2023-11-16T16:59:09.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.231+0000 7f32d7fff700 1 -- 172.21.15.44:0/2371374531 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f32e00050d0 con 0x7f32f0107af0 2023-11-16T16:59:09.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.231+0000 7f32d7fff700 1 -- 172.21.15.44:0/2371374531 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0003ca0 con 0x7f32f0107af0 2023-11-16T16:59:09.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.239+0000 7f3308e9fb80 1 Processor -- start 2023-11-16T16:59:09.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.240+0000 7f3308e9fb80 1 -- start start 2023-11-16T16:59:09.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.240+0000 7f3308e9fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df93be0 0x55b23df93fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.240+0000 7f3308e9fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df944f0 0x55b23df983f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.240+0000 7f3308e9fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df98930 0x55b23df98d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.243 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.241+0000 7f3308e9fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23de990e0 con 0x55b23df98930 2023-11-16T16:59:09.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.241+0000 7f3308e9fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23deeb920 con 0x55b23df944f0 2023-11-16T16:59:09.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.241+0000 7f3308e9fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23dee4db0 con 0x55b23df93be0 2023-11-16T16:59:09.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.241+0000 7f32ee7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df98930 0x55b23df98d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.242+0000 7f32d7fff700 1 -- 172.21.15.44:0/2371374531 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f32e0020070 con 0x7f32f0107af0 2023-11-16T16:59:09.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.242+0000 7f32eeffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df944f0 0x55b23df983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.242+0000 7f32eeffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df944f0 0x55b23df983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:35348/0 (socket says 172.21.15.44:35348) 2023-11-16T16:59:09.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.242+0000 7f32eeffd700 1 -- 172.21.15.44:0/3210451715 learned_addr learned my addr 172.21.15.44:0/3210451715 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:09.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.243+0000 7f32eeffd700 1 -- 172.21.15.44:0/3210451715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df93be0 msgr2=0x55b23df93fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:09.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.243+0000 7f32eeffd700 1 --2- 172.21.15.44:0/3210451715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df93be0 0x55b23df93fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.243+0000 7f32eeffd700 1 -- 172.21.15.44:0/3210451715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df98930 msgr2=0x55b23df98d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:09.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.243+0000 7f32eeffd700 1 --2- 172.21.15.44:0/3210451715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df98930 0x55b23df98d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.243+0000 7f32eeffd700 1 -- 172.21.15.44:0/3210451715 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b23dd666b0 con 0x55b23df944f0 2023-11-16T16:59:09.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.244+0000 7f32eeffd700 1 --2- 172.21.15.44:0/3210451715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df944f0 0x55b23df983f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f32e00cb430 tx=0x7f32e00d6310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a397071d9273048 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:09.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.244+0000 7f32d7fff700 1 --2- 172.21.15.44:0/2371374531 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f32d80a1340 0x7f32d80a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.244+0000 7f32d7fff700 1 -- 172.21.15.44:0/2371374531 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f32e00ce1a0 con 0x7f32f0107af0 2023-11-16T16:59:09.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.244+0000 7f32ef7fe700 1 --2- 172.21.15.44:0/2371374531 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f32d80a1340 0x7f32d80a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.245+0000 7f32d57fa700 1 -- 172.21.15.44:0/3210451715 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e00d6a90 con 0x55b23df944f0 2023-11-16T16:59:09.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.245+0000 7f32d57fa700 1 -- 172.21.15.44:0/3210451715 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f32e00d6bf0 con 0x55b23df944f0 2023-11-16T16:59:09.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.245+0000 7f32d57fa700 1 -- 172.21.15.44:0/3210451715 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e00dc620 con 0x55b23df944f0 2023-11-16T16:59:09.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.246+0000 7f3308e9fb80 1 -- 172.21.15.44:0/3210451715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df944f0 msgr2=0x55b23df983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.246+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/3210451715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df944f0 0x55b23df983f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f32e00cb430 tx=0x7f32e00d6310 comp rx=0 tx=0).stop 2023-11-16T16:59:09.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.246+0000 7f32ef7fe700 1 --2- 172.21.15.44:0/2371374531 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f32d80a1340 0x7f32d80a37f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f32f006a380 tx=0x7f32e8009250 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:09.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.246+0000 7f3308e9fb80 1 -- 172.21.15.44:0/3210451715 shutdown_connections 2023-11-16T16:59:09.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.246+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/3210451715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df98930 0x55b23df98d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.246+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/3210451715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df93be0 0x55b23df93fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.246+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/3210451715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df944f0 0x55b23df983f0 secure :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f32e00cb430 tx=0x7f32e00d6310 comp rx=0 tx=0).stop 2023-11-16T16:59:09.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.246+0000 7f3308e9fb80 1 -- 172.21.15.44:0/3210451715 >> 172.21.15.44:0/3210451715 conn(0x55b23df893e0 msgr2=0x55b23df8ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:09.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.246+0000 7f3308e9fb80 1 -- 172.21.15.44:0/3210451715 shutdown_connections 2023-11-16T16:59:09.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.247+0000 7f3308e9fb80 1 -- 172.21.15.44:0/3210451715 wait complete. 2023-11-16T16:59:09.251 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.248+0000 7f3308e9fb80 1 -- start start 2023-11-16T16:59:09.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.248+0000 7f3308e9fb80 1 Processor -- start 2023-11-16T16:59:09.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.249+0000 7f3308e9fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df93be0 0x55b23df9d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.249+0000 7f3308e9fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df98930 0x55b23df9daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.249+0000 7f3308e9fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df9e030 0x55b23dfa9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.249+0000 7f3308e9fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23de86960 con 0x55b23df9e030 2023-11-16T16:59:09.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.249+0000 7f3308e9fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23de86600 con 0x55b23df98930 2023-11-16T16:59:09.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.249+0000 7f3308e9fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23dd689a0 con 0x55b23df93be0 2023-11-16T16:59:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.249+0000 7f3308e9fb80 10 client.0 _ll_register_callbacks cb 0x55b23dea4800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:59:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.250+0000 7f32ef7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df9e030 0x55b23dfa9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.250+0000 7f32eeffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df93be0 0x55b23df9d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.250+0000 7f32ef7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df9e030 0x55b23dfa9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35532/0 (socket says 172.21.15.44:35532) 2023-11-16T16:59:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.250+0000 7f32ef7fe700 1 -- 172.21.15.44:0/365270137 learned_addr learned my addr 172.21.15.44:0/365270137 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.250+0000 7f32ee7fc700 1 --2- 172.21.15.44:0/365270137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df98930 0x55b23df9daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.251+0000 7f32ef7fe700 1 -- 172.21.15.44:0/365270137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df93be0 msgr2=0x55b23df9d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:09.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.251+0000 7f32ef7fe700 1 --2- 172.21.15.44:0/365270137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df93be0 0x55b23df9d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.251+0000 7f32ef7fe700 1 -- 172.21.15.44:0/365270137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df98930 msgr2=0x55b23df9daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:09.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.251+0000 7f32ef7fe700 1 --2- 172.21.15.44:0/365270137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df98930 0x55b23df9daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.251+0000 7f32ef7fe700 1 -- 172.21.15.44:0/365270137 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f010a650 con 0x55b23df9e030 2023-11-16T16:59:09.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.251+0000 7f32d2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:59:09.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.251+0000 7f32ef7fe700 1 --2- 172.21.15.44:0/365270137 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df9e030 0x55b23dfa9870 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f32e800d7a0 tx=0x7f32e8004cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a49cb46dc5b7abc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:09.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.252+0000 7f32d2ff5700 1 -- 172.21.15.44:0/365270137 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e801a8f0 con 0x55b23df9e030 2023-11-16T16:59:09.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.252+0000 7f32d2ff5700 1 -- 172.21.15.44:0/365270137 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f32e8013070 con 0x55b23df9e030 2023-11-16T16:59:09.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.252+0000 7f3308e9fb80 1 -- 172.21.15.44:0/365270137 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b23df86ba0 con 0x55b23df9e030 2023-11-16T16:59:09.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.259+0000 7f32d2ff5700 1 -- 172.21.15.44:0/365270137 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32e801aa50 con 0x55b23df9e030 2023-11-16T16:59:09.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.259+0000 7f3308e9fb80 10 client.17040 fetch_fsmap learned FSMap version 21 2023-11-16T16:59:09.261 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.260+0000 7f3308e9fb80 1 -- 172.21.15.44:0/365270137 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55b23deab210 con 0x55b23df9e030 2023-11-16T16:59:09.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.260+0000 7f32d2ff5700 1 -- 172.21.15.44:0/365270137 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b23df86ba0 con 0x55b23df9e030 2023-11-16T16:59:09.262 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.260+0000 7f32d2ff5700 1 -- 172.21.15.44:0/365270137 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f32e8020020 con 0x55b23df9e030 2023-11-16T16:59:09.263 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.261+0000 7f3308e9fb80 10 client.17040 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:59:09.263 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.261+0000 7f3308e9fb80 10 client.17040 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:59:09.263 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.261+0000 7f3308e9fb80 20 client.17040 populate_metadata read hostname 'smithi044' 2023-11-16T16:59:09.263 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.261+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/365270137 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b23dfadc50 0x55b23dfb0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.264 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.261+0000 7f3308e9fb80 4 client.17040 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:59:09.264 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.261+0000 7f3308e9fb80 1 -- 172.21.15.44:0/365270137 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b23dea7120 con 0x55b23dfadc50 2023-11-16T16:59:09.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.263+0000 7f32eeffd700 1 --2- 172.21.15.44:0/365270137 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b23dfadc50 0x55b23dfb0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.265 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.264+0000 7f32eeffd700 1 --2- 172.21.15.44:0/365270137 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b23dfadc50 0x55b23dfb0060 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a8e258e6a4fa6f server_cookie=c50ffdaea572fa91 in_seq=0 out_seq=0 2023-11-16T16:59:09.266 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.264+0000 7f32d2ff5700 10 client.17040 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:09.266 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.265+0000 7f32d2ff5700 1 -- 172.21.15.44:0/365270137 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55b23dea7120 con 0x55b23dfadc50 2023-11-16T16:59:09.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.265+0000 7f32d2ff5700 10 client.17040 handle_command_reply: tid=0 2023-11-16T16:59:09.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.265+0000 7f3308e9fb80 1 client.17040 shutdown 2023-11-16T16:59:09.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.266+0000 7f3308e9fb80 1 -- 172.21.15.44:0/365270137 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df9e030 msgr2=0x55b23dfa9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.267 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.266+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/365270137 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df9e030 0x55b23dfa9870 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f32e800d7a0 tx=0x7f32e8004cc0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.266+0000 7f3308e9fb80 1 -- 172.21.15.44:0/365270137 shutdown_connections 2023-11-16T16:59:09.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.266+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/365270137 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b23dfadc50 0x55b23dfb0060 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.266+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/365270137 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df9e030 0x55b23dfa9870 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.266+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/365270137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df93be0 0x55b23df9d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.266+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/365270137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df98930 0x55b23df9daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.268 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.266+0000 7f3308e9fb80 1 -- 172.21.15.44:0/365270137 >> 172.21.15.44:0/365270137 conn(0x55b23df893e0 msgr2=0x55b23df99de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:09.269 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.267+0000 7f32d2ff5700 0 client.17040 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:09.269 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.267+0000 7f3308e9fb80 1 -- 172.21.15.44:0/365270137 shutdown_connections 2023-11-16T16:59:09.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.268+0000 7f3308e9fb80 1 -- 172.21.15.44:0/365270137 wait complete. 2023-11-16T16:59:09.270 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.268+0000 7f3308e9fb80 20 client.17040 trim_cache size 0 max 16384 2023-11-16T16:59:09.306 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.304+0000 7f3308e9fb80 1 Processor -- start 2023-11-16T16:59:09.307 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.304+0000 7f3308e9fb80 1 -- start start 2023-11-16T16:59:09.307 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.305+0000 7f3308e9fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df93be0 0x55b23df8ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.307 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.305+0000 7f3308e9fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df98930 0x55b23df96220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.307 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.305+0000 7f3308e9fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df9e030 0x55b23dfa4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.305+0000 7f3308e9fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23dee4db0 con 0x55b23df93be0 2023-11-16T16:59:09.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.305+0000 7f3308e9fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23dea7120 con 0x55b23df98930 2023-11-16T16:59:09.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.305+0000 7f32ee7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df98930 0x55b23df96220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.305+0000 7f32ef7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df9e030 0x55b23dfa4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.305+0000 7f32ee7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df98930 0x55b23df96220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:35378/0 (socket says 172.21.15.44:35378) 2023-11-16T16:59:09.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.305+0000 7f32ee7fc700 1 -- 172.21.15.44:0/2099453934 learned_addr learned my addr 172.21.15.44:0/2099453934 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:09.308 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.306+0000 7f3308e9fb80 1 -- 172.21.15.44:0/2099453934 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23dfa0140 con 0x55b23df9e030 2023-11-16T16:59:09.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.305+0000 7f32eeffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df93be0 0x55b23df8ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.307+0000 7f32ee7fc700 1 -- 172.21.15.44:0/2099453934 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df9e030 msgr2=0x55b23dfa4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:09.309 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.307+0000 7f32ee7fc700 1 --2- 172.21.15.44:0/2099453934 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df9e030 0x55b23dfa4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.307+0000 7f32ee7fc700 1 -- 172.21.15.44:0/2099453934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df93be0 msgr2=0x55b23df8ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:09.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.307+0000 7f32ee7fc700 1 --2- 172.21.15.44:0/2099453934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df93be0 0x55b23df8ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.308+0000 7f32ee7fc700 1 -- 172.21.15.44:0/2099453934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b23df99220 con 0x55b23df98930 2023-11-16T16:59:09.310 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.308+0000 7f32ef7fe700 1 --2- 172.21.15.44:0/2099453934 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df9e030 0x55b23dfa4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:09.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.308+0000 7f32eeffd700 1 --2- 172.21.15.44:0/2099453934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df93be0 0x55b23df8ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:59:09.311 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.309+0000 7f32ee7fc700 1 --2- 172.21.15.44:0/2099453934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df98930 0x55b23df96220 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f32f0114780 tx=0x7f32e4009d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6725f572bd943ca5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:09.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.310+0000 7f32d5ffb700 1 -- 172.21.15.44:0/2099453934 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e4004230 con 0x55b23df98930 2023-11-16T16:59:09.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.310+0000 7f32d5ffb700 1 -- 172.21.15.44:0/2099453934 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f32e4010d20 con 0x55b23df98930 2023-11-16T16:59:09.312 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.310+0000 7f32d5ffb700 1 -- 172.21.15.44:0/2099453934 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e4003ce0 con 0x55b23df98930 2023-11-16T16:59:09.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.311+0000 7f3308e9fb80 1 -- 172.21.15.44:0/2099453934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df98930 msgr2=0x55b23df96220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.311+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/2099453934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df98930 0x55b23df96220 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f32f0114780 tx=0x7f32e4009d30 comp rx=0 tx=0).stop 2023-11-16T16:59:09.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.312+0000 7f3308e9fb80 1 -- 172.21.15.44:0/2099453934 shutdown_connections 2023-11-16T16:59:09.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.312+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/2099453934 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df9e030 0x55b23dfa4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.312+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/2099453934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df93be0 0x55b23df8ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.312+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/2099453934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df98930 0x55b23df96220 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.314 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.312+0000 7f3308e9fb80 1 -- 172.21.15.44:0/2099453934 >> 172.21.15.44:0/2099453934 conn(0x55b23df893e0 msgr2=0x55b23df8e600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:09.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.312+0000 7f3308e9fb80 1 -- 172.21.15.44:0/2099453934 shutdown_connections 2023-11-16T16:59:09.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.312+0000 7f3308e9fb80 1 -- 172.21.15.44:0/2099453934 wait complete. 2023-11-16T16:59:09.315 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.313+0000 7f3308e9fb80 1 -- start start 2023-11-16T16:59:09.317 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.313+0000 7f3308e9fb80 1 Processor -- start 2023-11-16T16:59:09.318 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.316+0000 7f3308e9fb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df93be0 0x55b23dfa11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.316+0000 7f3308e9fb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df98930 0x55b23dfa16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.316+0000 7f3308e9fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df9e030 0x55b23dfa1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.316+0000 7f3308e9fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23e001c90 con 0x55b23df9e030 2023-11-16T16:59:09.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.316+0000 7f3308e9fb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23e001e00 con 0x55b23df93be0 2023-11-16T16:59:09.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.316+0000 7f3308e9fb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23e001fc0 con 0x55b23df98930 2023-11-16T16:59:09.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.316+0000 7f3308e9fb80 10 client.0 _ll_register_callbacks cb 0x7f32e800ccc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:59:09.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.317+0000 7f32eeffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df93be0 0x55b23dfa11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.319 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.317+0000 7f32eeffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df93be0 0x55b23dfa11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:35382/0 (socket says 172.21.15.44:35382) 2023-11-16T16:59:09.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.317+0000 7f32ef7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df9e030 0x55b23dfa1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.317+0000 7f32ee7fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df98930 0x55b23dfa16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.320 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.317+0000 7f32eeffd700 1 -- 172.21.15.44:0/3502225942 learned_addr learned my addr 172.21.15.44:0/3502225942 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:09.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.318+0000 7f32ee7fc700 1 -- 172.21.15.44:0/3502225942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df93be0 msgr2=0x55b23dfa11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:09.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.318+0000 7f32ee7fc700 1 --2- 172.21.15.44:0/3502225942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df93be0 0x55b23dfa11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.318+0000 7f32ee7fc700 1 -- 172.21.15.44:0/3502225942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df9e030 msgr2=0x55b23dfa1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:09.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.318+0000 7f32ee7fc700 1 --2- 172.21.15.44:0/3502225942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df9e030 0x55b23dfa1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.318+0000 7f32ee7fc700 1 -- 172.21.15.44:0/3502225942 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b23deab210 con 0x55b23df98930 2023-11-16T16:59:09.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.318+0000 7f32ee7fc700 1 --2- 172.21.15.44:0/3502225942 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df98930 0x55b23dfa16f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x55b23df9eaa0 tx=0x7f32e4010a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c01392e989da0e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:09.321 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.318+0000 7f32d2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:59:09.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.319+0000 7f32d2ff5700 1 -- 172.21.15.44:0/3502225942 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e4012860 con 0x55b23df98930 2023-11-16T16:59:09.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.320+0000 7f32d2ff5700 1 -- 172.21.15.44:0/3502225942 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f32e40129c0 con 0x55b23df98930 2023-11-16T16:59:09.322 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.320+0000 7f3308e9fb80 1 -- 172.21.15.44:0/3502225942 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f32e00d6bf0 con 0x55b23df98930 2023-11-16T16:59:09.324 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.322+0000 7f32d2ff5700 1 -- 172.21.15.44:0/3502225942 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32e4009ec0 con 0x55b23df98930 2023-11-16T16:59:09.324 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.322+0000 7f3308e9fb80 10 client.27070 fetch_fsmap learned FSMap version 21 2023-11-16T16:59:09.324 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.322+0000 7f3308e9fb80 1 -- 172.21.15.44:0/3502225942 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x7f32e40096b0 con 0x55b23df98930 2023-11-16T16:59:09.324 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.323+0000 7f32d2ff5700 1 -- 172.21.15.44:0/3502225942 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f32e00d6bf0 con 0x55b23df98930 2023-11-16T16:59:09.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.324+0000 7f32d2ff5700 1 -- 172.21.15.44:0/3502225942 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f32e402b050 con 0x55b23df98930 2023-11-16T16:59:09.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.324+0000 7f3308e9fb80 10 client.27070 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:59:09.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.324+0000 7f3308e9fb80 10 client.27070 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:59:09.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.324+0000 7f3308e9fb80 20 client.27070 populate_metadata read hostname 'smithi044' 2023-11-16T16:59:09.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.324+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/3502225942 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b23e0035c0 0x55b23e0059a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.326 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.324+0000 7f3308e9fb80 4 client.27070 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:59:09.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.324+0000 7f3308e9fb80 1 -- 172.21.15.44:0/3502225942 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55b23deb36e0 con 0x55b23e0035c0 2023-11-16T16:59:09.327 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.326+0000 7f32eeffd700 1 --2- 172.21.15.44:0/3502225942 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b23e0035c0 0x55b23e0059a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.327+0000 7f32eeffd700 1 --2- 172.21.15.44:0/3502225942 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b23e0035c0 0x55b23e0059a0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f1bb4e08b9388bb server_cookie=939ec674eb755e3 in_seq=0 out_seq=0 2023-11-16T16:59:09.328 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.327+0000 7f32d2ff5700 10 client.27070 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:09.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.329+0000 7f32d2ff5700 1 -- 172.21.15.44:0/3502225942 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55b23deb36e0 con 0x55b23e0035c0 2023-11-16T16:59:09.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.329+0000 7f32d2ff5700 10 client.27070 handle_command_reply: tid=0 2023-11-16T16:59:09.331 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.329+0000 7f3308e9fb80 1 client.27070 shutdown 2023-11-16T16:59:09.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.330+0000 7f3308e9fb80 1 -- 172.21.15.44:0/3502225942 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df98930 msgr2=0x55b23dfa16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.330+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/3502225942 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df98930 0x55b23dfa16f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x55b23df9eaa0 tx=0x7f32e4010a90 comp rx=0 tx=0).stop 2023-11-16T16:59:09.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.330+0000 7f3308e9fb80 1 -- 172.21.15.44:0/3502225942 shutdown_connections 2023-11-16T16:59:09.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.330+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/3502225942 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55b23e0035c0 0x55b23e0059a0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.330+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/3502225942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b23df9e030 0x55b23dfa1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.332 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.330+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/3502225942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b23df93be0 0x55b23dfa11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.333 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.330+0000 7f3308e9fb80 1 --2- 172.21.15.44:0/3502225942 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b23df98930 0x55b23dfa16f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.333 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.330+0000 7f32d2ff5700 0 client.27070 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:09.333 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.330+0000 7f3308e9fb80 1 -- 172.21.15.44:0/3502225942 >> 172.21.15.44:0/3502225942 conn(0x55b23df893e0 msgr2=0x55b23dfa8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:09.333 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.331+0000 7f3308e9fb80 1 -- 172.21.15.44:0/3502225942 shutdown_connections 2023-11-16T16:59:09.334 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.331+0000 7f3308e9fb80 1 -- 172.21.15.44:0/3502225942 wait complete. 2023-11-16T16:59:09.334 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.331+0000 7f3308e9fb80 20 client.27070 trim_cache size 0 max 16384 2023-11-16T16:59:09.334 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:59:09.334 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:59:09.334 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "0e48cc0a-b877-4033-b782-c99e97ce9624", 2023-11-16T16:59:09.335 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:59:09.335 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:59:09.335 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.332+0000 7f32f55ff700 1 -- 172.21.15.44:0/2371374531 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f32d80a1340 msgr2=0x7f32d80a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.335 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.332+0000 7f32f55ff700 1 --2- 172.21.15.44:0/2371374531 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f32d80a1340 0x7f32d80a37f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f32f006a380 tx=0x7f32e8009250 comp rx=0 tx=0).stop 2023-11-16T16:59:09.335 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.332+0000 7f32f55ff700 1 -- 172.21.15.44:0/2371374531 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32f0107af0 msgr2=0x7f32f0112b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.332+0000 7f32f55ff700 1 --2- 172.21.15.44:0/2371374531 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32f0107af0 0x7f32f0112b20 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f32e0000c00 tx=0x7f32e000b610 comp rx=0 tx=0).stop 2023-11-16T16:59:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.332+0000 7f32f55ff700 1 -- 172.21.15.44:0/2371374531 shutdown_connections 2023-11-16T16:59:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.332+0000 7f32f55ff700 1 --2- 172.21.15.44:0/2371374531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32f01133b0 0x7f32f0110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.332+0000 7f32f55ff700 1 --2- 172.21.15.44:0/2371374531 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32f0107af0 0x7f32f0112b20 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.332+0000 7f32f55ff700 1 --2- 172.21.15.44:0/2371374531 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f32d80a1340 0x7f32d80a37f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.332+0000 7f32f55ff700 1 --2- 172.21.15.44:0/2371374531 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32f0108400 0x7f32f010fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.336 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.332+0000 7f32f55ff700 1 -- 172.21.15.44:0/2371374531 >> 172.21.15.44:0/2371374531 conn(0x7f32f006c7b0 msgr2=0x7f32f006ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:09.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.334+0000 7f32f55ff700 1 -- 172.21.15.44:0/2371374531 shutdown_connections 2023-11-16T16:59:09.337 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.335+0000 7f32f55ff700 1 -- 172.21.15.44:0/2371374531 wait complete. 2023-11-16T16:59:09.369 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:59:09.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.793+0000 7f10349ff700 1 -- 172.21.15.44:0/1660247881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10301084b0 msgr2=0x7f10301109a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.793+0000 7f10349ff700 1 --2- 172.21.15.44:0/1660247881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10301084b0 0x7f10301109a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1024009990 tx=0x7f1024009ca0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.793+0000 7f10349ff700 1 -- 172.21.15.44:0/1660247881 shutdown_connections 2023-11-16T16:59:09.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.793+0000 7f10349ff700 1 --2- 172.21.15.44:0/1660247881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1030110f70 0x7f1030113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.793+0000 7f10349ff700 1 --2- 172.21.15.44:0/1660247881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1030107ba0 0x7f1030107f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.793+0000 7f10349ff700 1 --2- 172.21.15.44:0/1660247881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10301084b0 0x7f10301109a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.793+0000 7f10349ff700 1 -- 172.21.15.44:0/1660247881 >> 172.21.15.44:0/1660247881 conn(0x7f1030076120 msgr2=0x7f1030078530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:09.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.793+0000 7f10349ff700 1 -- 172.21.15.44:0/1660247881 shutdown_connections 2023-11-16T16:59:09.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.793+0000 7f10349ff700 1 -- 172.21.15.44:0/1660247881 wait complete. 2023-11-16T16:59:09.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.795+0000 7f10349ff700 1 Processor -- start 2023-11-16T16:59:09.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.796+0000 7f10349ff700 1 -- start start 2023-11-16T16:59:09.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.797+0000 7f10349ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1030107ba0 0x7f10301a2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.798+0000 7f10349ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10301084b0 0x7f10301a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.798+0000 7f10349ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1030110f70 0x7f103019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.798+0000 7f10349ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103019cf10 con 0x7f1030107ba0 2023-11-16T16:59:09.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.798+0000 7f10349ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103019d080 con 0x7f10301084b0 2023-11-16T16:59:09.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.798+0000 7f10349ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103019d240 con 0x7f1030110f70 2023-11-16T16:59:09.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.798+0000 7f102dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10301084b0 0x7f10301a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.798+0000 7f102e59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1030107ba0 0x7f10301a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.798+0000 7f102dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10301084b0 0x7f10301a2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:35400/0 (socket says 172.21.15.44:35400) 2023-11-16T16:59:09.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.798+0000 7f102dd9b700 1 -- 172.21.15.44:0/480514498 learned_addr learned my addr 172.21.15.44:0/480514498 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:09.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.799+0000 7f102dd9b700 1 -- 172.21.15.44:0/480514498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1030110f70 msgr2=0x7f103019c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.799+0000 7f102dd9b700 1 --2- 172.21.15.44:0/480514498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1030110f70 0x7f103019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.799+0000 7f102dd9b700 1 -- 172.21.15.44:0/480514498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1030107ba0 msgr2=0x7f10301a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.799+0000 7f102dd9b700 1 --2- 172.21.15.44:0/480514498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1030107ba0 0x7f10301a2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.799+0000 7f102dd9b700 1 -- 172.21.15.44:0/480514498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10240095f0 con 0x7f10301084b0 2023-11-16T16:59:09.803 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.799+0000 7f102e59c700 1 --2- 172.21.15.44:0/480514498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1030107ba0 0x7f10301a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:59:09.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.800+0000 7f102dd9b700 1 --2- 172.21.15.44:0/480514498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10301084b0 0x7f10301a2d70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f10240098e0 tx=0x7f1024005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:09.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.800+0000 7f101f7fe700 1 -- 172.21.15.44:0/480514498 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102401d070 con 0x7f10301084b0 2023-11-16T16:59:09.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.800+0000 7f101f7fe700 1 -- 172.21.15.44:0/480514498 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f10240047e0 con 0x7f10301084b0 2023-11-16T16:59:09.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.800+0000 7f101f7fe700 1 -- 172.21.15.44:0/480514498 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1024005490 con 0x7f10301084b0 2023-11-16T16:59:09.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.800+0000 7f10349ff700 1 -- 172.21.15.44:0/480514498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10301169e0 con 0x7f10301084b0 2023-11-16T16:59:09.809 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.801+0000 7f10349ff700 1 -- 172.21.15.44:0/480514498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1030116850 con 0x7f10301084b0 2023-11-16T16:59:09.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.809+0000 7f101f7fe700 1 -- 172.21.15.44:0/480514498 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f1024004950 con 0x7f10301084b0 2023-11-16T16:59:09.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.811+0000 7f101f7fe700 1 --2- 172.21.15.44:0/480514498 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f10180a1490 0x7f10180a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.813 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.812+0000 7f101f7fe700 1 -- 172.21.15.44:0/480514498 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f10240ce2a0 con 0x7f10301084b0 2023-11-16T16:59:09.814 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.813+0000 7f102e59c700 1 --2- 172.21.15.44:0/480514498 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f10180a1490 0x7f10180a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.814+0000 7f10482d9b80 1 Processor -- start 2023-11-16T16:59:09.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.814+0000 7f10482d9b80 1 -- start start 2023-11-16T16:59:09.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.815+0000 7f10482d9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b448eb60 0x5641b448ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.815+0000 7f10482d9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448f470 0x5641b4493370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.815+0000 7f10482d9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 0x5641b4493d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.815+0000 7f10482d9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b4394060 con 0x5641b448f470 2023-11-16T16:59:09.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.815+0000 7f10482d9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b43e68a0 con 0x5641b448eb60 2023-11-16T16:59:09.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.815+0000 7f10482d9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b43dfd30 con 0x5641b44938b0 2023-11-16T16:59:09.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.815+0000 7f102ed9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 0x5641b4493d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.815+0000 7f102e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b448eb60 0x5641b448ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.815+0000 7f102ed9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 0x5641b4493d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:53096/0 (socket says 172.21.15.44:53096) 2023-11-16T16:59:09.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.815+0000 7f102ed9d700 1 -- 172.21.15.44:0/67798252 learned_addr learned my addr 172.21.15.44:0/67798252 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:09.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.816+0000 7f102ed9d700 1 -- 172.21.15.44:0/67798252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b448eb60 msgr2=0x5641b448ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:09.822 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.816+0000 7f102ed9d700 1 --2- 172.21.15.44:0/67798252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b448eb60 0x5641b448ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.816+0000 7f102e59c700 1 --2- 172.21.15.44:0/480514498 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f10180a1490 0x7f10180a3940 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f102000bc10 tx=0x7f102000b5f0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:09.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.816+0000 7f102ed9d700 1 -- 172.21.15.44:0/67798252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448f470 msgr2=0x5641b4493370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:09.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.816+0000 7f102ed9d700 1 --2- 172.21.15.44:0/67798252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448f470 0x5641b4493370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.816+0000 7f102ed9d700 1 -- 172.21.15.44:0/67798252 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641b4261630 con 0x5641b44938b0 2023-11-16T16:59:09.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.816+0000 7f102ed9d700 1 --2- 172.21.15.44:0/67798252 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 0x5641b4493d00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1028009fd0 tx=0x7f102800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc6fa3efd92ca5fa server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:09.823 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.817+0000 7f101cff9700 1 -- 172.21.15.44:0/67798252 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1028018070 con 0x5641b44938b0 2023-11-16T16:59:09.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.817+0000 7f101cff9700 1 -- 172.21.15.44:0/67798252 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f102800ee60 con 0x5641b44938b0 2023-11-16T16:59:09.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.817+0000 7f101cff9700 1 -- 172.21.15.44:0/67798252 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1028018070 con 0x5641b44938b0 2023-11-16T16:59:09.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.818+0000 7f10482d9b80 1 -- 172.21.15.44:0/67798252 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 msgr2=0x5641b4493d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.818+0000 7f10482d9b80 1 --2- 172.21.15.44:0/67798252 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 0x5641b4493d00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1028009fd0 tx=0x7f102800b040 comp rx=0 tx=0).stop 2023-11-16T16:59:09.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.818+0000 7f10482d9b80 1 -- 172.21.15.44:0/67798252 shutdown_connections 2023-11-16T16:59:09.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.818+0000 7f10482d9b80 1 --2- 172.21.15.44:0/67798252 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 0x5641b4493d00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.818+0000 7f10482d9b80 1 --2- 172.21.15.44:0/67798252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b448eb60 0x5641b448ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.818+0000 7f10482d9b80 1 --2- 172.21.15.44:0/67798252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448f470 0x5641b4493370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.818+0000 7f10482d9b80 1 -- 172.21.15.44:0/67798252 >> 172.21.15.44:0/67798252 conn(0x5641b4484360 msgr2=0x5641b4489b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:09.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.819+0000 7f10482d9b80 1 -- 172.21.15.44:0/67798252 shutdown_connections 2023-11-16T16:59:09.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.819+0000 7f10482d9b80 1 -- 172.21.15.44:0/67798252 wait complete. 2023-11-16T16:59:09.825 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.820+0000 7f10482d9b80 1 -- start start 2023-11-16T16:59:09.826 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.820+0000 7f10482d9b80 1 Processor -- start 2023-11-16T16:59:09.827 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.821+0000 7f10482d9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.827 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.821+0000 7f10482d9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b448f470 0x5641b4498400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.827 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.821+0000 7f10482d9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 0x5641b4498b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.827 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.821+0000 7f10482d9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b43818e0 con 0x5641b448eb60 2023-11-16T16:59:09.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.821+0000 7f10482d9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b4381580 con 0x5641b448f470 2023-11-16T16:59:09.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.821+0000 7f10482d9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b4263920 con 0x5641b44938b0 2023-11-16T16:59:09.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.821+0000 7f10482d9b80 10 client.0 _ll_register_callbacks cb 0x5641b439f780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:59:09.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.822+0000 7f102e59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.822+0000 7f102e59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35592/0 (socket says 172.21.15.44:35592) 2023-11-16T16:59:09.828 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.822+0000 7f102e59c700 1 -- 172.21.15.44:0/480716091 learned_addr learned my addr 172.21.15.44:0/480716091 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:09.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.823+0000 7f102ed9d700 1 --2- 172.21.15.44:0/480716091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 0x5641b4498b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.823+0000 7f102e59c700 1 -- 172.21.15.44:0/480716091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 msgr2=0x5641b4498b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:09.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.823+0000 7f102e59c700 1 --2- 172.21.15.44:0/480716091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 0x5641b4498b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.823+0000 7f102e59c700 1 -- 172.21.15.44:0/480716091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b448f470 msgr2=0x5641b4498400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:09.829 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.823+0000 7f102e59c700 1 --2- 172.21.15.44:0/480716091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b448f470 0x5641b4498400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.830 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.823+0000 7f102e59c700 1 -- 172.21.15.44:0/480716091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102800e0c0 con 0x5641b448eb60 2023-11-16T16:59:09.830 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.823+0000 7f102ed9d700 1 --2- 172.21.15.44:0/480716091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 0x5641b4498b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:59:09.830 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.825+0000 7f102e59c700 1 --2- 172.21.15.44:0/480716091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449b110 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1020011470 tx=0x7f1020004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7ef577f45331d2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:09.830 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.825+0000 7f1015ffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:59:09.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.826+0000 7f1015ffb700 1 -- 172.21.15.44:0/480716091 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102000fa30 con 0x5641b448eb60 2023-11-16T16:59:09.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.826+0000 7f10482d9b80 1 -- 172.21.15.44:0/480716091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5641b4481b20 con 0x5641b448eb60 2023-11-16T16:59:09.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.827+0000 7f1015ffb700 1 -- 172.21.15.44:0/480716091 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f1020028670 con 0x5641b448eb60 2023-11-16T16:59:09.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.827+0000 7f1015ffb700 1 -- 172.21.15.44:0/480716091 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1020028870 con 0x5641b448eb60 2023-11-16T16:59:09.832 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.827+0000 7f10482d9b80 10 client.17055 fetch_fsmap learned FSMap version 21 2023-11-16T16:59:09.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.827+0000 7f10482d9b80 1 -- 172.21.15.44:0/480716091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5641b43a6190 con 0x5641b448eb60 2023-11-16T16:59:09.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.828+0000 7f1015ffb700 1 -- 172.21.15.44:0/480716091 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5641b4481b20 con 0x5641b448eb60 2023-11-16T16:59:09.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.828+0000 7f1015ffb700 1 -- 172.21.15.44:0/480716091 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f102001fc50 con 0x5641b448eb60 2023-11-16T16:59:09.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.828+0000 7f10482d9b80 10 client.17055 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:59:09.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.828+0000 7f10482d9b80 10 client.17055 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:59:09.833 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.828+0000 7f10482d9b80 20 client.17055 populate_metadata read hostname 'smithi044' 2023-11-16T16:59:09.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.828+0000 7f10482d9b80 1 --2- 172.21.15.44:0/480716091 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5641b44b9940 0x5641b44bbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.828+0000 7f10482d9b80 4 client.17055 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:59:09.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.828+0000 7f10482d9b80 1 -- 172.21.15.44:0/480716091 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5641b43a20a0 con 0x5641b44b9940 2023-11-16T16:59:09.834 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.829+0000 7f102dd9b700 1 --2- 172.21.15.44:0/480716091 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5641b44b9940 0x5641b44bbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.831+0000 7f102dd9b700 1 --2- 172.21.15.44:0/480716091 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5641b44b9940 0x5641b44bbd50 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fb435dce1a911a6 server_cookie=d1e41c47e66c6c99 in_seq=0 out_seq=0 2023-11-16T16:59:09.835 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.831+0000 7f1015ffb700 10 client.17055 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:09.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.834+0000 7f1015ffb700 1 -- 172.21.15.44:0/480716091 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5641b43a20a0 con 0x5641b44b9940 2023-11-16T16:59:09.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.834+0000 7f1015ffb700 10 client.17055 handle_command_reply: tid=0 2023-11-16T16:59:09.836 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.834+0000 7f10482d9b80 1 client.17055 shutdown 2023-11-16T16:59:09.837 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.836+0000 7f10482d9b80 1 -- 172.21.15.44:0/480716091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 msgr2=0x5641b449b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.837 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.836+0000 7f10482d9b80 1 --2- 172.21.15.44:0/480716091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449b110 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1020011470 tx=0x7f1020004750 comp rx=0 tx=0).stop 2023-11-16T16:59:09.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.837+0000 7f10482d9b80 1 -- 172.21.15.44:0/480716091 shutdown_connections 2023-11-16T16:59:09.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.837+0000 7f10482d9b80 1 --2- 172.21.15.44:0/480716091 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5641b44b9940 0x5641b44bbd50 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.837+0000 7f1015ffb700 0 client.17055 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:09.839 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.837+0000 7f10482d9b80 1 --2- 172.21.15.44:0/480716091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 0x5641b4498b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.837+0000 7f10482d9b80 1 --2- 172.21.15.44:0/480716091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449b110 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.837+0000 7f10482d9b80 1 --2- 172.21.15.44:0/480716091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b448f470 0x5641b4498400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.840 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.837+0000 7f10482d9b80 1 -- 172.21.15.44:0/480716091 >> 172.21.15.44:0/480716091 conn(0x5641b4484360 msgr2=0x5641b4488de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:09.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.838+0000 7f10482d9b80 1 -- 172.21.15.44:0/480716091 shutdown_connections 2023-11-16T16:59:09.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.838+0000 7f10482d9b80 1 -- 172.21.15.44:0/480716091 wait complete. 2023-11-16T16:59:09.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.839+0000 7f10482d9b80 20 client.17055 trim_cache size 0 max 16384 2023-11-16T16:59:09.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:09 smithi044 ceph-mon[128750]: pgmap v285: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 45 KiB/s wr, 17 op/s 2023-11-16T16:59:09.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:09 smithi044 ceph-mon[128750]: osdmap e99: 12 total, 12 up, 12 in 2023-11-16T16:59:09.876 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.875+0000 7f10482d9b80 1 Processor -- start 2023-11-16T16:59:09.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.875+0000 7f10482d9b80 1 -- start start 2023-11-16T16:59:09.879 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.875+0000 7f10482d9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.875+0000 7f10482d9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b448f470 0x5641b449acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.875+0000 7f10482d9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 0x5641b4489560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.876+0000 7f10482d9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b4499380 con 0x5641b448eb60 2023-11-16T16:59:09.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.876+0000 7f10482d9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b448a950 con 0x5641b448f470 2023-11-16T16:59:09.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.876+0000 7f10482d9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b44b7040 con 0x5641b44938b0 2023-11-16T16:59:09.880 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.876+0000 7f102dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b448f470 0x5641b449acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.881 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.876+0000 7f102dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b448f470 0x5641b449acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:35440/0 (socket says 172.21.15.44:35440) 2023-11-16T16:59:09.881 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.876+0000 7f102dd9b700 1 -- 172.21.15.44:0/528103601 learned_addr learned my addr 172.21.15.44:0/528103601 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:09.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.878+0000 7f102e59c700 1 --2- 172.21.15.44:0/528103601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.878+0000 7f102ed9d700 1 --2- 172.21.15.44:0/528103601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 0x5641b4489560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.879+0000 7f102e59c700 1 -- 172.21.15.44:0/528103601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 msgr2=0x5641b4489560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:09.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.879+0000 7f102e59c700 1 --2- 172.21.15.44:0/528103601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 0x5641b4489560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.882 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.879+0000 7f102e59c700 1 -- 172.21.15.44:0/528103601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b448f470 msgr2=0x5641b449acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:09.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.879+0000 7f102e59c700 1 --2- 172.21.15.44:0/528103601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b448f470 0x5641b449acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.879+0000 7f102e59c700 1 -- 172.21.15.44:0/528103601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641b44941a0 con 0x5641b448eb60 2023-11-16T16:59:09.883 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.879+0000 7f102e59c700 1 --2- 172.21.15.44:0/528103601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449f940 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1020022040 tx=0x7f102000b3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5276f443b5628132 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:09.884 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.881+0000 7f101d7fa700 1 -- 172.21.15.44:0/528103601 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102000fa30 con 0x5641b448eb60 2023-11-16T16:59:09.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.881+0000 7f101d7fa700 1 -- 172.21.15.44:0/528103601 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1020011bc0 con 0x5641b448eb60 2023-11-16T16:59:09.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.881+0000 7f101d7fa700 1 -- 172.21.15.44:0/528103601 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102001ece0 con 0x5641b448eb60 2023-11-16T16:59:09.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.882+0000 7f10482d9b80 1 -- 172.21.15.44:0/528103601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 msgr2=0x5641b449f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.882+0000 7f10482d9b80 1 --2- 172.21.15.44:0/528103601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449f940 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1020022040 tx=0x7f102000b3c0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.885 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.883+0000 7f10482d9b80 1 -- 172.21.15.44:0/528103601 shutdown_connections 2023-11-16T16:59:09.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.883+0000 7f10482d9b80 1 --2- 172.21.15.44:0/528103601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b44938b0 0x5641b4489560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.883+0000 7f10482d9b80 1 --2- 172.21.15.44:0/528103601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449f940 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.883+0000 7f10482d9b80 1 --2- 172.21.15.44:0/528103601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b448f470 0x5641b449acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.883+0000 7f10482d9b80 1 -- 172.21.15.44:0/528103601 >> 172.21.15.44:0/528103601 conn(0x5641b4484360 msgr2=0x5641b44919d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:09.886 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.883+0000 7f10482d9b80 1 -- 172.21.15.44:0/528103601 shutdown_connections 2023-11-16T16:59:09.887 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.883+0000 7f10482d9b80 1 -- 172.21.15.44:0/528103601 wait complete. 2023-11-16T16:59:09.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.884+0000 7f10482d9b80 1 -- start start 2023-11-16T16:59:09.888 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.884+0000 7f10482d9b80 1 Processor -- start 2023-11-16T16:59:09.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.885+0000 7f10482d9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.885+0000 7f10482d9b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b448f470 0x5641b449be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.885+0000 7f10482d9b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b44938b0 0x5641b450c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.885+0000 7f10482d9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b449c7a0 con 0x5641b448eb60 2023-11-16T16:59:09.889 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.885+0000 7f10482d9b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b450cd50 con 0x5641b44938b0 2023-11-16T16:59:09.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.885+0000 7f10482d9b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b450ce90 con 0x5641b448f470 2023-11-16T16:59:09.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.885+0000 7f10482d9b80 10 client.0 _ll_register_callbacks cb 0x7f102000fc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:59:09.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.886+0000 7f102e59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.886+0000 7f102e59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35600/0 (socket says 172.21.15.44:35600) 2023-11-16T16:59:09.890 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.886+0000 7f102e59c700 1 -- 172.21.15.44:0/160527902 learned_addr learned my addr 172.21.15.44:0/160527902 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:09.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.886+0000 7f102ed9d700 1 --2- 172.21.15.44:0/160527902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b44938b0 0x5641b450c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.886+0000 7f102e59c700 1 -- 172.21.15.44:0/160527902 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b448f470 msgr2=0x5641b449be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:09.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.886+0000 7f102e59c700 1 --2- 172.21.15.44:0/160527902 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b448f470 0x5641b449be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.886+0000 7f102e59c700 1 -- 172.21.15.44:0/160527902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b44938b0 msgr2=0x5641b450c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:09.891 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.886+0000 7f102e59c700 1 --2- 172.21.15.44:0/160527902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b44938b0 0x5641b450c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.892 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.886+0000 7f102e59c700 1 -- 172.21.15.44:0/160527902 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102002d040 con 0x5641b448eb60 2023-11-16T16:59:09.894 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.887+0000 7f102e59c700 1 --2- 172.21.15.44:0/160527902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449b920 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f102001e550 tx=0x7f102001fd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea1754179acca81 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:09.894 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.887+0000 7f1015ffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:59:09.894 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.887+0000 7f1015ffb700 1 -- 172.21.15.44:0/160527902 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10200042f0 con 0x5641b448eb60 2023-11-16T16:59:09.894 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.888+0000 7f1015ffb700 1 -- 172.21.15.44:0/160527902 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f1020004450 con 0x5641b448eb60 2023-11-16T16:59:09.895 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.888+0000 7f10482d9b80 1 -- 172.21.15.44:0/160527902 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f102800ee60 con 0x5641b448eb60 2023-11-16T16:59:09.895 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.888+0000 7f1015ffb700 1 -- 172.21.15.44:0/160527902 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f102001e960 con 0x5641b448eb60 2023-11-16T16:59:09.895 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.888+0000 7f10482d9b80 10 client.17061 fetch_fsmap learned FSMap version 21 2023-11-16T16:59:09.895 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.888+0000 7f10482d9b80 1 -- 172.21.15.44:0/160527902 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5641b420dde0 con 0x5641b448eb60 2023-11-16T16:59:09.895 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.889+0000 7f1015ffb700 1 -- 172.21.15.44:0/160527902 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f102800ee60 con 0x5641b448eb60 2023-11-16T16:59:09.895 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.889+0000 7f1015ffb700 1 -- 172.21.15.44:0/160527902 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f102003b030 con 0x5641b448eb60 2023-11-16T16:59:09.896 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.889+0000 7f10482d9b80 10 client.17061 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:59:09.896 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.889+0000 7f10482d9b80 10 client.17061 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:59:09.896 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.889+0000 7f10482d9b80 20 client.17061 populate_metadata read hostname 'smithi044' 2023-11-16T16:59:09.896 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.889+0000 7f10482d9b80 1 --2- 172.21.15.44:0/160527902 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5641b44b9940 0x5641b4510190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:09.896 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.889+0000 7f10482d9b80 4 client.17061 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:59:09.896 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.889+0000 7f10482d9b80 1 -- 172.21.15.44:0/160527902 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5641b43ae660 con 0x5641b44b9940 2023-11-16T16:59:09.896 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.889+0000 7f102dd9b700 1 --2- 172.21.15.44:0/160527902 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5641b44b9940 0x5641b4510190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:09.897 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.890+0000 7f102dd9b700 1 --2- 172.21.15.44:0/160527902 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5641b44b9940 0x5641b4510190 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6d88800283360fc server_cookie=a702fe09cedf03bf in_seq=0 out_seq=0 2023-11-16T16:59:09.897 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.891+0000 7f1015ffb700 10 client.17061 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:09.897 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.891+0000 7f1015ffb700 1 -- 172.21.15.44:0/160527902 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5641b43ae660 con 0x5641b44b9940 2023-11-16T16:59:09.897 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.891+0000 7f1015ffb700 10 client.17061 handle_command_reply: tid=0 2023-11-16T16:59:09.897 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.891+0000 7f10482d9b80 1 client.17061 shutdown 2023-11-16T16:59:09.897 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.891+0000 7f10482d9b80 1 -- 172.21.15.44:0/160527902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 msgr2=0x5641b449b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.897 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.891+0000 7f10482d9b80 1 --2- 172.21.15.44:0/160527902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449b920 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f102001e550 tx=0x7f102001fd00 comp rx=0 tx=0).stop 2023-11-16T16:59:09.898 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.891+0000 7f10482d9b80 1 -- 172.21.15.44:0/160527902 shutdown_connections 2023-11-16T16:59:09.898 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.891+0000 7f10482d9b80 1 --2- 172.21.15.44:0/160527902 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5641b44b9940 0x5641b4510190 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.898 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.891+0000 7f10482d9b80 1 --2- 172.21.15.44:0/160527902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5641b44938b0 0x5641b450c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.898 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.891+0000 7f10482d9b80 1 --2- 172.21.15.44:0/160527902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5641b448eb60 0x5641b449b920 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.898 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.891+0000 7f10482d9b80 1 --2- 172.21.15.44:0/160527902 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641b448f470 0x5641b449be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.898 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.891+0000 7f1015ffb700 0 client.17061 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:09.899 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.891+0000 7f10482d9b80 1 -- 172.21.15.44:0/160527902 >> 172.21.15.44:0/160527902 conn(0x5641b4484360 msgr2=0x5641b448d960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:09.899 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.892+0000 7f10482d9b80 1 -- 172.21.15.44:0/160527902 shutdown_connections 2023-11-16T16:59:09.899 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.892+0000 7f10482d9b80 1 -- 172.21.15.44:0/160527902 wait complete. 2023-11-16T16:59:09.899 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.892+0000 7f10482d9b80 20 client.17061 trim_cache size 0 max 16384 2023-11-16T16:59:09.899 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:59:09.900 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2023-11-16T16:59:09.900 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2023-11-16T16:59:09.900 INFO:teuthology.orchestra.run.smithi044.stdout: "0e48cc0a-b877-4033-b782-c99e97ce9624": { 2023-11-16T16:59:09.900 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2023-11-16T16:59:09.900 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "0e48cc0a-b877-4033-b782-c99e97ce9624", 2023-11-16T16:59:09.900 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2023-11-16T16:59:09.901 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:59:09.901 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:59:09.901 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:59:09.902 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.893+0000 7f10349ff700 1 -- 172.21.15.44:0/480514498 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f10180a1490 msgr2=0x7f10180a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.902 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.893+0000 7f10349ff700 1 --2- 172.21.15.44:0/480514498 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f10180a1490 0x7f10180a3940 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f102000bc10 tx=0x7f102000b5f0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.902 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.893+0000 7f10349ff700 1 -- 172.21.15.44:0/480514498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10301084b0 msgr2=0x7f10301a2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:09.902 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.893+0000 7f10349ff700 1 --2- 172.21.15.44:0/480514498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10301084b0 0x7f10301a2d70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f10240098e0 tx=0x7f1024005ef0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.902 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.893+0000 7f10349ff700 1 -- 172.21.15.44:0/480514498 shutdown_connections 2023-11-16T16:59:09.902 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.893+0000 7f10349ff700 1 --2- 172.21.15.44:0/480514498 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f10180a1490 0x7f10180a3940 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.903 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.893+0000 7f10349ff700 1 --2- 172.21.15.44:0/480514498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1030110f70 0x7f103019c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.903 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.893+0000 7f10349ff700 1 --2- 172.21.15.44:0/480514498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1030107ba0 0x7f10301a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.903 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.893+0000 7f10349ff700 1 --2- 172.21.15.44:0/480514498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10301084b0 0x7f10301a2d70 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:09.903 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.893+0000 7f10349ff700 1 -- 172.21.15.44:0/480514498 >> 172.21.15.44:0/480514498 conn(0x7f1030076120 msgr2=0x7f10300766f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:09.903 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.893+0000 7f10349ff700 1 -- 172.21.15.44:0/480514498 shutdown_connections 2023-11-16T16:59:09.903 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:09.893+0000 7f10349ff700 1 -- 172.21.15.44:0/480514498 wait complete. 2023-11-16T16:59:09.904 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:09 smithi176 ceph-mon[131630]: pgmap v285: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 45 KiB/s wr, 17 op/s 2023-11-16T16:59:09.904 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:09 smithi176 ceph-mon[131630]: osdmap e99: 12 total, 12 up, 12 in 2023-11-16T16:59:09.931 INFO:tasks.cephfs.filesystem:scrub status for tag:0e48cc0a-b877-4033-b782-c99e97ce9624 - {'path': '/', 'tag': '0e48cc0a-b877-4033-b782-c99e97ce9624', 'options': 'recursive,force'} 2023-11-16T16:59:09.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:09 smithi077 ceph-mon[131204]: pgmap v285: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 45 KiB/s wr, 17 op/s 2023-11-16T16:59:09.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:09 smithi077 ceph-mon[131204]: osdmap e99: 12 total, 12 up, 12 in 2023-11-16T16:59:10.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:10 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:59:10.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:10 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:59:10.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:10 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:59:10.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:10 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:59:10.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:10 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:59:10.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:10 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:59:10.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:10 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:59:10.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:10 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:59:10.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:10 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:59:10.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:10 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:59:10.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:10 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:59:10.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:10 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:59:11.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:11 smithi044 ceph-mon[128750]: pgmap v287: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 54 KiB/s wr, 19 op/s 2023-11-16T16:59:11.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:11 smithi176 ceph-mon[131630]: pgmap v287: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 54 KiB/s wr, 19 op/s 2023-11-16T16:59:11.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:11 smithi077 ceph-mon[131204]: pgmap v287: 129 pgs: 129 active+clean; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 54 KiB/s wr, 19 op/s 2023-11-16T16:59:13.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:13 smithi044 ceph-mon[128750]: pgmap v288: 129 pgs: 3 active+clean+snaptrim, 118 active+clean, 8 active+clean+snaptrim_wait; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 54 KiB/s wr, 19 op/s 2023-11-16T16:59:13.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:13 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:59:13.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:13 smithi176 ceph-mon[131630]: pgmap v288: 129 pgs: 3 active+clean+snaptrim, 118 active+clean, 8 active+clean+snaptrim_wait; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 54 KiB/s wr, 19 op/s 2023-11-16T16:59:13.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:13 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:59:13.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:13 smithi077 ceph-mon[131204]: pgmap v288: 129 pgs: 3 active+clean+snaptrim, 118 active+clean, 8 active+clean+snaptrim_wait; 6.1 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 54 KiB/s wr, 19 op/s 2023-11-16T16:59:13.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:13 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:59:15.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:15 smithi044 ceph-mon[128750]: pgmap v289: 129 pgs: 19 active+clean+snaptrim, 54 active+clean, 56 active+clean+snaptrim_wait; 6.1 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 307 B/s wr, 3 op/s 2023-11-16T16:59:15.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:15 smithi176 ceph-mon[131630]: pgmap v289: 129 pgs: 19 active+clean+snaptrim, 54 active+clean, 56 active+clean+snaptrim_wait; 6.1 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 307 B/s wr, 3 op/s 2023-11-16T16:59:15.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:15 smithi077 ceph-mon[131204]: pgmap v289: 129 pgs: 19 active+clean+snaptrim, 54 active+clean, 56 active+clean+snaptrim_wait; 6.1 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 307 B/s wr, 3 op/s 2023-11-16T16:59:17.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:17 smithi044 ceph-mon[128750]: pgmap v290: 129 pgs: 19 active+clean+snaptrim, 54 active+clean, 56 active+clean+snaptrim_wait; 6.1 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 307 B/s wr, 3 op/s 2023-11-16T16:59:17.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:17 smithi176 ceph-mon[131630]: pgmap v290: 129 pgs: 19 active+clean+snaptrim, 54 active+clean, 56 active+clean+snaptrim_wait; 6.1 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 307 B/s wr, 3 op/s 2023-11-16T16:59:17.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:17 smithi077 ceph-mon[131204]: pgmap v290: 129 pgs: 19 active+clean+snaptrim, 54 active+clean, 56 active+clean+snaptrim_wait; 6.1 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 307 B/s wr, 3 op/s 2023-11-16T16:59:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:19 smithi044 ceph-mon[128750]: pgmap v291: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 6.1 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 307 B/s wr, 3 op/s 2023-11-16T16:59:19.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:19 smithi176 ceph-mon[131630]: pgmap v291: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 6.1 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 307 B/s wr, 3 op/s 2023-11-16T16:59:19.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:19 smithi077 ceph-mon[131204]: pgmap v291: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 6.1 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 307 B/s wr, 3 op/s 2023-11-16T16:59:21.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:21 smithi044 ceph-mon[128750]: pgmap v292: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 271 B/s wr, 3 op/s 2023-11-16T16:59:21.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:21 smithi176 ceph-mon[131630]: pgmap v292: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 271 B/s wr, 3 op/s 2023-11-16T16:59:21.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:21 smithi077 ceph-mon[131204]: pgmap v292: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 271 B/s wr, 3 op/s 2023-11-16T16:59:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:23 smithi044 ceph-mon[128750]: pgmap v293: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 255 B/s wr, 3 op/s 2023-11-16T16:59:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:23 smithi044 ceph-mon[128750]: osdmap e100: 12 total, 12 up, 12 in 2023-11-16T16:59:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:23 smithi176 ceph-mon[131630]: pgmap v293: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 255 B/s wr, 3 op/s 2023-11-16T16:59:23.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:23 smithi176 ceph-mon[131630]: osdmap e100: 12 total, 12 up, 12 in 2023-11-16T16:59:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:23 smithi077 ceph-mon[131204]: pgmap v293: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 255 B/s wr, 3 op/s 2023-11-16T16:59:23.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:23 smithi077 ceph-mon[131204]: osdmap e100: 12 total, 12 up, 12 in 2023-11-16T16:59:25.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:25 smithi044 ceph-mon[128750]: pgmap v295: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2023-11-16T16:59:25.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:25 smithi176 ceph-mon[131630]: pgmap v295: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2023-11-16T16:59:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:25 smithi077 ceph-mon[131204]: pgmap v295: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2023-11-16T16:59:27.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:27 smithi044 ceph-mon[128750]: pgmap v296: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2023-11-16T16:59:27.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:27 smithi176 ceph-mon[131630]: pgmap v296: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2023-11-16T16:59:27.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:27 smithi077 ceph-mon[131204]: pgmap v296: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2023-11-16T16:59:29.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:29 smithi044 ceph-mon[128750]: pgmap v297: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2023-11-16T16:59:29.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:29 smithi176 ceph-mon[131630]: pgmap v297: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2023-11-16T16:59:29.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:29 smithi077 ceph-mon[131204]: pgmap v297: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2023-11-16T16:59:31.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:31 smithi044 ceph-mon[128750]: pgmap v298: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2023-11-16T16:59:31.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:31 smithi176 ceph-mon[131630]: pgmap v298: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2023-11-16T16:59:31.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:31 smithi077 ceph-mon[131204]: pgmap v298: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2023-11-16T16:59:32.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:32 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:59:32.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:32 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:59:32.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:32 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:59:33.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:33 smithi044 ceph-mon[128750]: pgmap v299: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2023-11-16T16:59:33.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:33 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:59:33.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:33 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:59:33.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:33 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:59:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:33 smithi176 ceph-mon[131630]: pgmap v299: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2023-11-16T16:59:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:33 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:59:33.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:33 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:59:33.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:33 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:59:33.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:33 smithi077 ceph-mon[131204]: pgmap v299: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2023-11-16T16:59:33.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:33 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:59:33.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:33 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:59:33.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:33 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T16:59:34.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:34 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:59:34.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:34 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T16:59:34.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:34 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T16:59:34.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:34 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T16:59:34.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:34 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:59:34.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:34 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T16:59:34.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:34 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T16:59:34.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:34 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T16:59:34.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:34 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:59:34.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:34 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T16:59:34.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:34 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T16:59:34.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:34 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T16:59:35.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:35 smithi044 ceph-mon[128750]: pgmap v300: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 984 B/s rd, 1 op/s 2023-11-16T16:59:35.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:35 smithi176 ceph-mon[131630]: pgmap v300: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 984 B/s rd, 1 op/s 2023-11-16T16:59:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:35 smithi077 ceph-mon[131204]: pgmap v300: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 984 B/s rd, 1 op/s 2023-11-16T16:59:37.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:37 smithi044 ceph-mon[128750]: pgmap v301: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:59:37.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:37 smithi176 ceph-mon[131630]: pgmap v301: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:59:37.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:37 smithi077 ceph-mon[131204]: pgmap v301: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:59:39.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:39 smithi044 ceph-mon[128750]: pgmap v302: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:59:39.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:39 smithi176 ceph-mon[131630]: pgmap v302: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:59:39.932 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:59:39.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:39 smithi077 ceph-mon[131204]: pgmap v302: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:59:40.151 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.150+0000 7f43e7fff700 1 -- 172.21.15.44:0/2631409988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43e0101390 msgr2=0x7f43e010d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.152 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.150+0000 7f43e7fff700 1 --2- 172.21.15.44:0/2631409988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43e0101390 0x7f43e010d020 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f43d800b0d0 tx=0x7f43d800b3e0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.152 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.150+0000 7f43e7fff700 1 -- 172.21.15.44:0/2631409988 shutdown_connections 2023-11-16T16:59:40.152 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.150+0000 7f43e7fff700 1 --2- 172.21.15.44:0/2631409988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43e0101390 0x7f43e010d020 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.150+0000 7f43e7fff700 1 --2- 172.21.15.44:0/2631409988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43e01051b0 0x7f43e0105580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.150+0000 7f43e7fff700 1 --2- 172.21.15.44:0/2631409988 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43e01009e0 0x7f43e0100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.150+0000 7f43e7fff700 1 -- 172.21.15.44:0/2631409988 >> 172.21.15.44:0/2631409988 conn(0x7f43e0077530 msgr2=0x7f43e0077930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:40.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.150+0000 7f43e7fff700 1 -- 172.21.15.44:0/2631409988 shutdown_connections 2023-11-16T16:59:40.153 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.150+0000 7f43e7fff700 1 -- 172.21.15.44:0/2631409988 wait complete. 2023-11-16T16:59:40.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.151+0000 7f43e7fff700 1 Processor -- start 2023-11-16T16:59:40.154 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.151+0000 7f43e7fff700 1 -- start start 2023-11-16T16:59:40.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.151+0000 7f43e7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43e01009e0 0x7f43e01964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.156 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.151+0000 7f43e7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43e0101390 0x7f43e0196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.151+0000 7f43e7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43e01051b0 0x7f43e019ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.152+0000 7f43e7fff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e019b4a0 con 0x7f43e01051b0 2023-11-16T16:59:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.152+0000 7f43e7fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e019b610 con 0x7f43e01009e0 2023-11-16T16:59:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.152+0000 7f43e7fff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e019b7d0 con 0x7f43e0101390 2023-11-16T16:59:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.152+0000 7f43e659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43e01051b0 0x7f43e019ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.157 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.152+0000 7f43e5d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43e01009e0 0x7f43e01964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.152+0000 7f43e659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43e01051b0 0x7f43e019ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44146/0 (socket says 172.21.15.44:44146) 2023-11-16T16:59:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.152+0000 7f43e659c700 1 -- 172.21.15.44:0/3790579099 learned_addr learned my addr 172.21.15.44:0/3790579099 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.152+0000 7f43e559a700 1 --2- 172.21.15.44:0/3790579099 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43e0101390 0x7f43e0196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.152+0000 7f43e659c700 1 -- 172.21.15.44:0/3790579099 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43e0101390 msgr2=0x7f43e0196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.158 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.152+0000 7f43e659c700 1 --2- 172.21.15.44:0/3790579099 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43e0101390 0x7f43e0196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.152+0000 7f43e659c700 1 -- 172.21.15.44:0/3790579099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43e01009e0 msgr2=0x7f43e01964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.152+0000 7f43e659c700 1 --2- 172.21.15.44:0/3790579099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43e01009e0 0x7f43e01964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.152+0000 7f43e659c700 1 -- 172.21.15.44:0/3790579099 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d8009d00 con 0x7f43e01051b0 2023-11-16T16:59:40.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.152+0000 7f43e659c700 1 --2- 172.21.15.44:0/3790579099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43e01051b0 0x7f43e019ae60 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f43d8007820 tx=0x7f43d8007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:40.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.153+0000 7f43d6ffd700 1 -- 172.21.15.44:0/3790579099 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d800e040 con 0x7f43e01051b0 2023-11-16T16:59:40.159 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.153+0000 7f43d6ffd700 1 -- 172.21.15.44:0/3790579099 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43d800bcb0 con 0x7f43e01051b0 2023-11-16T16:59:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.153+0000 7f43d6ffd700 1 -- 172.21.15.44:0/3790579099 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d80058a0 con 0x7f43e01051b0 2023-11-16T16:59:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.153+0000 7f43e7fff700 1 -- 172.21.15.44:0/3790579099 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e01103b0 con 0x7f43e01051b0 2023-11-16T16:59:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.153+0000 7f43e7fff700 1 -- 172.21.15.44:0/3790579099 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e0110220 con 0x7f43e01051b0 2023-11-16T16:59:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.156+0000 7f43d6ffd700 1 -- 172.21.15.44:0/3790579099 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f43d8004a50 con 0x7f43e01051b0 2023-11-16T16:59:40.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.157+0000 7f43fb893b80 1 Processor -- start 2023-11-16T16:59:40.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.157+0000 7f43d6ffd700 1 --2- 172.21.15.44:0/3790579099 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f43cc0a1340 0x7f43cc0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.167 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.157+0000 7f43fb893b80 1 -- start start 2023-11-16T16:59:40.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.157+0000 7f43d6ffd700 1 -- 172.21.15.44:0/3790579099 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f43d80ce230 con 0x7f43e01051b0 2023-11-16T16:59:40.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43fb893b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 0x562b44ebf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43fb893b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ebf880 0x562b44ec3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43fb893b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ec3cc0 0x562b44ec4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43e559a700 1 --2- 172.21.15.44:0/3790579099 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f43cc0a1340 0x7f43cc0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.168 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43fb893b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b44dc0060 con 0x562b44ebef70 2023-11-16T16:59:40.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43fb893b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b44e128a0 con 0x562b44ec3cc0 2023-11-16T16:59:40.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43fb893b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b44e0bd30 con 0x562b44ebf880 2023-11-16T16:59:40.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43e5d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 0x562b44ebf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43e5d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 0x562b44ebf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44156/0 (socket says 172.21.15.44:44156) 2023-11-16T16:59:40.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43e5d9b700 1 -- 172.21.15.44:0/80423828 learned_addr learned my addr 172.21.15.44:0/80423828 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:40.169 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43e659c700 1 --2- 172.21.15.44:0/80423828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ec3cc0 0x562b44ec4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43e559a700 1 --2- 172.21.15.44:0/80423828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ebf880 0x562b44ec3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43e659c700 1 -- 172.21.15.44:0/80423828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ebf880 msgr2=0x562b44ec3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:40.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43e659c700 1 --2- 172.21.15.44:0/80423828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ebf880 0x562b44ec3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43e659c700 1 -- 172.21.15.44:0/80423828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 msgr2=0x562b44ebf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:40.170 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43e659c700 1 --2- 172.21.15.44:0/80423828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 0x562b44ebf340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43e659c700 1 -- 172.21.15.44:0/80423828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b44c8d630 con 0x562b44ec3cc0 2023-11-16T16:59:40.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.158+0000 7f43e559a700 1 --2- 172.21.15.44:0/3790579099 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f43cc0a1340 0x7f43cc0a37f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f43e0197ad0 tx=0x7f43d0011440 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:40.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.159+0000 7f43e659c700 1 --2- 172.21.15.44:0/80423828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ec3cc0 0x562b44ec4110 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f43d80cb540 tx=0x7f43d8097310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c61f82d5e5126ded server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:40.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.159+0000 7f43cbfff700 1 -- 172.21.15.44:0/80423828 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d809a8b0 con 0x562b44ec3cc0 2023-11-16T16:59:40.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.159+0000 7f43cbfff700 1 -- 172.21.15.44:0/80423828 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43d80ce020 con 0x562b44ec3cc0 2023-11-16T16:59:40.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.159+0000 7f43cbfff700 1 -- 172.21.15.44:0/80423828 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d80993d0 con 0x562b44ec3cc0 2023-11-16T16:59:40.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.160+0000 7f43fb893b80 1 -- 172.21.15.44:0/80423828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ec3cc0 msgr2=0x562b44ec4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.160+0000 7f43fb893b80 1 --2- 172.21.15.44:0/80423828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ec3cc0 0x562b44ec4110 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f43d80cb540 tx=0x7f43d8097310 comp rx=0 tx=0).stop 2023-11-16T16:59:40.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.160+0000 7f43fb893b80 1 -- 172.21.15.44:0/80423828 shutdown_connections 2023-11-16T16:59:40.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.160+0000 7f43fb893b80 1 --2- 172.21.15.44:0/80423828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ec3cc0 0x562b44ec4110 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.160+0000 7f43fb893b80 1 --2- 172.21.15.44:0/80423828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 0x562b44ebf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.160+0000 7f43fb893b80 1 --2- 172.21.15.44:0/80423828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ebf880 0x562b44ec3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.160+0000 7f43fb893b80 1 -- 172.21.15.44:0/80423828 >> 172.21.15.44:0/80423828 conn(0x562b44eb0360 msgr2=0x562b44eb5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:40.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.160+0000 7f43fb893b80 1 -- 172.21.15.44:0/80423828 shutdown_connections 2023-11-16T16:59:40.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.160+0000 7f43fb893b80 1 -- 172.21.15.44:0/80423828 wait complete. 2023-11-16T16:59:40.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.160+0000 7f43fb893b80 1 -- start start 2023-11-16T16:59:40.173 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.161+0000 7f43fb893b80 1 Processor -- start 2023-11-16T16:59:40.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.161+0000 7f43fb893b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 0x562b44ecb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.161+0000 7f43fb893b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ebf880 0x562b44ec8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.161+0000 7f43fb893b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ec3cc0 0x562b44ec8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.161+0000 7f43fb893b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b44dad8e0 con 0x562b44ebef70 2023-11-16T16:59:40.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.161+0000 7f43fb893b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b44dad580 con 0x562b44ec3cc0 2023-11-16T16:59:40.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.161+0000 7f43fb893b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b44c8f920 con 0x562b44ebf880 2023-11-16T16:59:40.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.161+0000 7f43fb893b80 10 client.0 _ll_register_callbacks cb 0x562b44dcb780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:59:40.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.161+0000 7f43e5d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 0x562b44ecb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.161+0000 7f43e659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ec3cc0 0x562b44ec8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.161+0000 7f43e5d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 0x562b44ecb450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44164/0 (socket says 172.21.15.44:44164) 2023-11-16T16:59:40.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.161+0000 7f43e5d9b700 1 -- 172.21.15.44:0/1958601120 learned_addr learned my addr 172.21.15.44:0/1958601120 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:40.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.162+0000 7f43e559a700 1 --2- 172.21.15.44:0/1958601120 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ebf880 0x562b44ec8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.162+0000 7f43e659c700 1 -- 172.21.15.44:0/1958601120 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ebf880 msgr2=0x562b44ec8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:40.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.162+0000 7f43e659c700 1 --2- 172.21.15.44:0/1958601120 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ebf880 0x562b44ec8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.162+0000 7f43e659c700 1 -- 172.21.15.44:0/1958601120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 msgr2=0x562b44ecb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:40.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.162+0000 7f43e659c700 1 --2- 172.21.15.44:0/1958601120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 0x562b44ecb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.162+0000 7f43e659c700 1 -- 172.21.15.44:0/1958601120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e0110220 con 0x562b44ec3cc0 2023-11-16T16:59:40.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.162+0000 7f43e559a700 1 --2- 172.21.15.44:0/1958601120 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ebf880 0x562b44ec8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:40.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.162+0000 7f43e659c700 1 --2- 172.21.15.44:0/1958601120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ec3cc0 0x562b44ec8d10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f43d809b110 tx=0x7f43d809a930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=808ccc6d35512de9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:40.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.162+0000 7f43c97fa700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:59:40.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.162+0000 7f43c97fa700 1 -- 172.21.15.44:0/1958601120 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d800be20 con 0x562b44ec3cc0 2023-11-16T16:59:40.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.162+0000 7f43c97fa700 1 -- 172.21.15.44:0/1958601120 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f43d809ef80 con 0x562b44ec3cc0 2023-11-16T16:59:40.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.162+0000 7f43fb893b80 1 -- 172.21.15.44:0/1958601120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562b44eadb20 con 0x562b44ec3cc0 2023-11-16T16:59:40.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.163+0000 7f43c97fa700 1 -- 172.21.15.44:0/1958601120 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43d80b9300 con 0x562b44ec3cc0 2023-11-16T16:59:40.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.163+0000 7f43fb893b80 10 client.27155 fetch_fsmap learned FSMap version 21 2023-11-16T16:59:40.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.163+0000 7f43fb893b80 1 -- 172.21.15.44:0/1958601120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x562b44dd2190 con 0x562b44ec3cc0 2023-11-16T16:59:40.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.163+0000 7f43c97fa700 1 -- 172.21.15.44:0/1958601120 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562b44eadb20 con 0x562b44ec3cc0 2023-11-16T16:59:40.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.163+0000 7f43c97fa700 1 -- 172.21.15.44:0/1958601120 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f43d809a440 con 0x562b44ec3cc0 2023-11-16T16:59:40.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.163+0000 7f43fb893b80 10 client.27155 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:59:40.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.163+0000 7f43fb893b80 10 client.27155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:59:40.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.163+0000 7f43fb893b80 20 client.27155 populate_metadata read hostname 'smithi044' 2023-11-16T16:59:40.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.163+0000 7f43fb893b80 1 --2- 172.21.15.44:0/1958601120 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562b44ed8640 0x562b44edaa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.163+0000 7f43fb893b80 4 client.27155 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:59:40.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.163+0000 7f43fb893b80 1 -- 172.21.15.44:0/1958601120 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562b44dce0a0 con 0x562b44ed8640 2023-11-16T16:59:40.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.164+0000 7f43e5d9b700 1 --2- 172.21.15.44:0/1958601120 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562b44ed8640 0x562b44edaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.164+0000 7f43e5d9b700 1 --2- 172.21.15.44:0/1958601120 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562b44ed8640 0x562b44edaa50 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89c701e7a9825c49 server_cookie=2b09ee7a85ea9d4 in_seq=0 out_seq=0 2023-11-16T16:59:40.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.164+0000 7f43c97fa700 10 client.27155 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:40.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.165+0000 7f43c97fa700 1 -- 172.21.15.44:0/1958601120 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562b44dce0a0 con 0x562b44ed8640 2023-11-16T16:59:40.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.165+0000 7f43c97fa700 10 client.27155 handle_command_reply: tid=0 2023-11-16T16:59:40.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.165+0000 7f43fb893b80 1 client.27155 shutdown 2023-11-16T16:59:40.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.165+0000 7f43fb893b80 1 -- 172.21.15.44:0/1958601120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ec3cc0 msgr2=0x562b44ec8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.165+0000 7f43fb893b80 1 --2- 172.21.15.44:0/1958601120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ec3cc0 0x562b44ec8d10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f43d809b110 tx=0x7f43d809a930 comp rx=0 tx=0).stop 2023-11-16T16:59:40.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.165+0000 7f43fb893b80 1 -- 172.21.15.44:0/1958601120 shutdown_connections 2023-11-16T16:59:40.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.165+0000 7f43fb893b80 1 --2- 172.21.15.44:0/1958601120 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562b44ed8640 0x562b44edaa50 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.165+0000 7f43fb893b80 1 --2- 172.21.15.44:0/1958601120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ec3cc0 0x562b44ec8d10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.165+0000 7f43fb893b80 1 --2- 172.21.15.44:0/1958601120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 0x562b44ecb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.165+0000 7f43c97fa700 0 client.27155 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:40.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.165+0000 7f43fb893b80 1 --2- 172.21.15.44:0/1958601120 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ebf880 0x562b44ec8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.165+0000 7f43fb893b80 1 -- 172.21.15.44:0/1958601120 >> 172.21.15.44:0/1958601120 conn(0x562b44eb0360 msgr2=0x562b44ec5170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:40.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.165+0000 7f43fb893b80 1 -- 172.21.15.44:0/1958601120 shutdown_connections 2023-11-16T16:59:40.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.165+0000 7f43fb893b80 1 -- 172.21.15.44:0/1958601120 wait complete. 2023-11-16T16:59:40.182 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.165+0000 7f43fb893b80 20 client.27155 trim_cache size 0 max 16384 2023-11-16T16:59:40.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.183+0000 7f43fb893b80 1 Processor -- start 2023-11-16T16:59:40.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.183+0000 7f43fb893b80 1 -- start start 2023-11-16T16:59:40.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.183+0000 7f43fb893b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ebef70 0x562b44eb5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.183+0000 7f43fb893b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebf880 0x562b44ecfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.183+0000 7f43fb893b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ec3cc0 0x562b44ecaf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.183+0000 7f43fb893b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b44dce0d0 con 0x562b44ebf880 2023-11-16T16:59:40.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.183+0000 7f43fb893b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b44eb68e0 con 0x562b44ebef70 2023-11-16T16:59:40.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.183+0000 7f43fb893b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b44ec6cb0 con 0x562b44ec3cc0 2023-11-16T16:59:40.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.183+0000 7f43e559a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebf880 0x562b44ecfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.183+0000 7f43e559a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebf880 0x562b44ecfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44172/0 (socket says 172.21.15.44:44172) 2023-11-16T16:59:40.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.183+0000 7f43e559a700 1 -- 172.21.15.44:0/1612069655 learned_addr learned my addr 172.21.15.44:0/1612069655 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:40.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.183+0000 7f43e5d9b700 1 --2- 172.21.15.44:0/1612069655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ebef70 0x562b44eb5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.183+0000 7f43e659c700 1 --2- 172.21.15.44:0/1612069655 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ec3cc0 0x562b44ecaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.184+0000 7f43e559a700 1 -- 172.21.15.44:0/1612069655 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ec3cc0 msgr2=0x562b44ecaf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:40.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.184+0000 7f43e559a700 1 --2- 172.21.15.44:0/1612069655 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ec3cc0 0x562b44ecaf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.184+0000 7f43e559a700 1 -- 172.21.15.44:0/1612069655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ebef70 msgr2=0x562b44eb5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:40.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.184+0000 7f43e559a700 1 --2- 172.21.15.44:0/1612069655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ebef70 0x562b44eb5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.184+0000 7f43e559a700 1 -- 172.21.15.44:0/1612069655 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b44eb1b00 con 0x562b44ebf880 2023-11-16T16:59:40.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.184+0000 7f43e5d9b700 1 --2- 172.21.15.44:0/1612069655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ebef70 0x562b44eb5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:40.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.184+0000 7f43e559a700 1 --2- 172.21.15.44:0/1612069655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebf880 0x562b44ecfc50 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f43d001d7b0 tx=0x7f43d001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd4b61c2cadac74 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:40.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.184+0000 7f43d4ff9700 1 -- 172.21.15.44:0/1612069655 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d0023040 con 0x562b44ebf880 2023-11-16T16:59:40.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.184+0000 7f43d4ff9700 1 -- 172.21.15.44:0/1612069655 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43d0028b40 con 0x562b44ebf880 2023-11-16T16:59:40.188 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.184+0000 7f43d4ff9700 1 -- 172.21.15.44:0/1612069655 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d000f9f0 con 0x562b44ebf880 2023-11-16T16:59:40.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.185+0000 7f43fb893b80 1 -- 172.21.15.44:0/1612069655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebf880 msgr2=0x562b44ecfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.185+0000 7f43fb893b80 1 --2- 172.21.15.44:0/1612069655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebf880 0x562b44ecfc50 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f43d001d7b0 tx=0x7f43d001d000 comp rx=0 tx=0).stop 2023-11-16T16:59:40.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.185+0000 7f43fb893b80 1 -- 172.21.15.44:0/1612069655 shutdown_connections 2023-11-16T16:59:40.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.185+0000 7f43fb893b80 1 --2- 172.21.15.44:0/1612069655 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ec3cc0 0x562b44ecaf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.185+0000 7f43fb893b80 1 --2- 172.21.15.44:0/1612069655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ebef70 0x562b44eb5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.185+0000 7f43fb893b80 1 --2- 172.21.15.44:0/1612069655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebf880 0x562b44ecfc50 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.185+0000 7f43fb893b80 1 -- 172.21.15.44:0/1612069655 >> 172.21.15.44:0/1612069655 conn(0x562b44eb0360 msgr2=0x562b44ec1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:40.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.185+0000 7f43fb893b80 1 -- 172.21.15.44:0/1612069655 shutdown_connections 2023-11-16T16:59:40.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.185+0000 7f43fb893b80 1 -- 172.21.15.44:0/1612069655 wait complete. 2023-11-16T16:59:40.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.185+0000 7f43fb893b80 1 -- start start 2023-11-16T16:59:40.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.186+0000 7f43fb893b80 1 Processor -- start 2023-11-16T16:59:40.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.186+0000 7f43fb893b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 0x562b44ecd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.194 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.186+0000 7f43fb893b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ebf880 0x562b44ecdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.186+0000 7f43fb893b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ec3cc0 0x562b44f2b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.186+0000 7f43fb893b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b44ece3f0 con 0x562b44ebef70 2023-11-16T16:59:40.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.186+0000 7f43fb893b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b44f2ba00 con 0x562b44ebf880 2023-11-16T16:59:40.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.186+0000 7f43fb893b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b44f2bb40 con 0x562b44ec3cc0 2023-11-16T16:59:40.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.186+0000 7f43fb893b80 10 client.0 _ll_register_callbacks cb 0x562b44dcb780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:59:40.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.186+0000 7f43e5d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 0x562b44ecd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.186+0000 7f43e659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ec3cc0 0x562b44f2b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.186+0000 7f43e5d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 0x562b44ecd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44188/0 (socket says 172.21.15.44:44188) 2023-11-16T16:59:40.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.186+0000 7f43e559a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ebf880 0x562b44ecdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.186+0000 7f43e5d9b700 1 -- 172.21.15.44:0/607021387 learned_addr learned my addr 172.21.15.44:0/607021387 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:40.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.187+0000 7f43e5d9b700 1 -- 172.21.15.44:0/607021387 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ec3cc0 msgr2=0x562b44f2b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:40.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.187+0000 7f43e5d9b700 1 --2- 172.21.15.44:0/607021387 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ec3cc0 0x562b44f2b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.187+0000 7f43e5d9b700 1 -- 172.21.15.44:0/607021387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ebf880 msgr2=0x562b44ecdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:40.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.187+0000 7f43e5d9b700 1 --2- 172.21.15.44:0/607021387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ebf880 0x562b44ecdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.187+0000 7f43e5d9b700 1 -- 172.21.15.44:0/607021387 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43dc004040 con 0x562b44ebef70 2023-11-16T16:59:40.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.187+0000 7f43e659c700 1 --2- 172.21.15.44:0/607021387 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ec3cc0 0x562b44f2b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:40.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.187+0000 7f43e559a700 1 --2- 172.21.15.44:0/607021387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ebf880 0x562b44ecdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:40.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.187+0000 7f43e5d9b700 1 --2- 172.21.15.44:0/607021387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 0x562b44ecd670 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x562b44edb2c0 tx=0x7f43dc0044b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11552cb81881689e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:40.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.187+0000 7f43c97fa700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:59:40.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.187+0000 7f43c97fa700 1 -- 172.21.15.44:0/607021387 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc017070 con 0x562b44ebef70 2023-11-16T16:59:40.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.187+0000 7f43c97fa700 1 -- 172.21.15.44:0/607021387 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f43dc021a50 con 0x562b44ebef70 2023-11-16T16:59:40.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.187+0000 7f43fb893b80 1 -- 172.21.15.44:0/607021387 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f43d80ce020 con 0x562b44ebef70 2023-11-16T16:59:40.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.188+0000 7f43c97fa700 1 -- 172.21.15.44:0/607021387 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43dc021610 con 0x562b44ebef70 2023-11-16T16:59:40.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.188+0000 7f43fb893b80 10 client.17079 fetch_fsmap learned FSMap version 21 2023-11-16T16:59:40.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.188+0000 7f43fb893b80 1 -- 172.21.15.44:0/607021387 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x7f43d001d4f0 con 0x562b44ebef70 2023-11-16T16:59:40.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.188+0000 7f43c97fa700 1 -- 172.21.15.44:0/607021387 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43d80ce020 con 0x562b44ebef70 2023-11-16T16:59:40.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.188+0000 7f43c97fa700 1 -- 172.21.15.44:0/607021387 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f43dc012530 con 0x562b44ebef70 2023-11-16T16:59:40.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.188+0000 7f43fb893b80 10 client.17079 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:59:40.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.188+0000 7f43fb893b80 10 client.17079 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:59:40.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.188+0000 7f43fb893b80 20 client.17079 populate_metadata read hostname 'smithi044' 2023-11-16T16:59:40.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.188+0000 7f43fb893b80 1 --2- 172.21.15.44:0/607021387 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562b44f2d240 0x562b44f2f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.188+0000 7f43fb893b80 4 client.17079 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:59:40.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.188+0000 7f43fb893b80 1 -- 172.21.15.44:0/607021387 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562b44dda660 con 0x562b44f2d240 2023-11-16T16:59:40.199 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.189+0000 7f43e559a700 1 --2- 172.21.15.44:0/607021387 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562b44f2d240 0x562b44f2f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.189+0000 7f43e559a700 1 --2- 172.21.15.44:0/607021387 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562b44f2d240 0x562b44f2f650 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c6a10363cf55486 server_cookie=8c5ae5f695f5f844 in_seq=0 out_seq=0 2023-11-16T16:59:40.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.189+0000 7f43c97fa700 10 client.17079 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:40.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.189+0000 7f43c97fa700 1 -- 172.21.15.44:0/607021387 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x562b44dda660 con 0x562b44f2d240 2023-11-16T16:59:40.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.189+0000 7f43c97fa700 10 client.17079 handle_command_reply: tid=0 2023-11-16T16:59:40.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.189+0000 7f43fb893b80 1 client.17079 shutdown 2023-11-16T16:59:40.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.190+0000 7f43fb893b80 1 -- 172.21.15.44:0/607021387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 msgr2=0x562b44ecd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.190+0000 7f43fb893b80 1 --2- 172.21.15.44:0/607021387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 0x562b44ecd670 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x562b44edb2c0 tx=0x7f43dc0044b0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.190+0000 7f43fb893b80 1 -- 172.21.15.44:0/607021387 shutdown_connections 2023-11-16T16:59:40.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.190+0000 7f43fb893b80 1 --2- 172.21.15.44:0/607021387 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x562b44f2d240 0x562b44f2f650 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.190+0000 7f43fb893b80 1 --2- 172.21.15.44:0/607021387 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x562b44ec3cc0 0x562b44f2b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.190+0000 7f43fb893b80 1 --2- 172.21.15.44:0/607021387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562b44ebef70 0x562b44ecd670 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.190+0000 7f43c97fa700 0 client.17079 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:40.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.190+0000 7f43fb893b80 1 --2- 172.21.15.44:0/607021387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562b44ebf880 0x562b44ecdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.190+0000 7f43fb893b80 1 -- 172.21.15.44:0/607021387 >> 172.21.15.44:0/607021387 conn(0x562b44eb0360 msgr2=0x562b44eb9960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:40.202 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.190+0000 7f43fb893b80 1 -- 172.21.15.44:0/607021387 shutdown_connections 2023-11-16T16:59:40.202 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.190+0000 7f43fb893b80 1 -- 172.21.15.44:0/607021387 wait complete. 2023-11-16T16:59:40.202 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.190+0000 7f43fb893b80 20 client.17079 trim_cache size 0 max 16384 2023-11-16T16:59:40.202 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:59:40.203 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T16:59:40.203 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T16:59:40.203 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:59:40.204 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.190+0000 7f43e7fff700 1 -- 172.21.15.44:0/3790579099 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f43cc0a1340 msgr2=0x7f43cc0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.204 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.190+0000 7f43e7fff700 1 --2- 172.21.15.44:0/3790579099 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f43cc0a1340 0x7f43cc0a37f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f43e0197ad0 tx=0x7f43d0011440 comp rx=0 tx=0).stop 2023-11-16T16:59:40.204 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.190+0000 7f43e7fff700 1 -- 172.21.15.44:0/3790579099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43e01051b0 msgr2=0x7f43e019ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.204 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.190+0000 7f43e7fff700 1 --2- 172.21.15.44:0/3790579099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43e01051b0 0x7f43e019ae60 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f43d8007820 tx=0x7f43d8007850 comp rx=0 tx=0).stop 2023-11-16T16:59:40.204 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.191+0000 7f43e7fff700 1 -- 172.21.15.44:0/3790579099 shutdown_connections 2023-11-16T16:59:40.204 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.191+0000 7f43e7fff700 1 --2- 172.21.15.44:0/3790579099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43e01051b0 0x7f43e019ae60 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.191+0000 7f43e7fff700 1 --2- 172.21.15.44:0/3790579099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43e01009e0 0x7f43e01964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.191+0000 7f43e7fff700 1 --2- 172.21.15.44:0/3790579099 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f43cc0a1340 0x7f43cc0a37f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.191+0000 7f43e7fff700 1 --2- 172.21.15.44:0/3790579099 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43e0101390 0x7f43e0196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.191+0000 7f43e7fff700 1 -- 172.21.15.44:0/3790579099 >> 172.21.15.44:0/3790579099 conn(0x7f43e0077530 msgr2=0x7f43e00fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:40.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.191+0000 7f43e7fff700 1 -- 172.21.15.44:0/3790579099 shutdown_connections 2023-11-16T16:59:40.205 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.191+0000 7f43e7fff700 1 -- 172.21.15.44:0/3790579099 wait complete. 2023-11-16T16:59:40.209 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T16:59:40.210 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:59:40.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.415+0000 7f018d3ff700 1 -- 172.21.15.44:0/1188823632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0188100550 msgr2=0x7f0188100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.415+0000 7f018d3ff700 1 --2- 172.21.15.44:0/1188823632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0188100550 0x7f0188100920 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f01780099c0 tx=0x7f0178009cd0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.415+0000 7f018d3ff700 1 -- 172.21.15.44:0/1188823632 shutdown_connections 2023-11-16T16:59:40.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.415+0000 7f018d3ff700 1 --2- 172.21.15.44:0/1188823632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0188105540 0x7f018810d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.415+0000 7f018d3ff700 1 --2- 172.21.15.44:0/1188823632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0188100550 0x7f0188100920 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.415+0000 7f018d3ff700 1 --2- 172.21.15.44:0/1188823632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0188100ef0 0x7f0188104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.415+0000 7f018d3ff700 1 -- 172.21.15.44:0/1188823632 >> 172.21.15.44:0/1188823632 conn(0x7f0188077020 msgr2=0x7f0188077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:40.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.415+0000 7f018d3ff700 1 -- 172.21.15.44:0/1188823632 shutdown_connections 2023-11-16T16:59:40.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.415+0000 7f018d3ff700 1 -- 172.21.15.44:0/1188823632 wait complete. 2023-11-16T16:59:40.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.416+0000 7f018d3ff700 1 Processor -- start 2023-11-16T16:59:40.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.416+0000 7f018d3ff700 1 -- start start 2023-11-16T16:59:40.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.417+0000 7f018d3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0188100550 0x7f0188198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.417+0000 7f018d3ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0188100ef0 0x7f0188198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.417+0000 7f018d3ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0188105540 0x7f018819cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.417+0000 7f018d3ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018819d610 con 0x7f0188100ef0 2023-11-16T16:59:40.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.417+0000 7f018d3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018819d780 con 0x7f0188100550 2023-11-16T16:59:40.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.417+0000 7f018d3ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018819d940 con 0x7f0188105540 2023-11-16T16:59:40.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.417+0000 7f01867fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0188100ef0 0x7f0188198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.417+0000 7f01867fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0188100ef0 0x7f0188198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44206/0 (socket says 172.21.15.44:44206) 2023-11-16T16:59:40.423 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.417+0000 7f01867fc700 1 -- 172.21.15.44:0/1433575098 learned_addr learned my addr 172.21.15.44:0/1433575098 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:40.423 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.417+0000 7f0186ffd700 1 --2- 172.21.15.44:0/1433575098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0188100550 0x7f0188198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.423 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.417+0000 7f01877fe700 1 --2- 172.21.15.44:0/1433575098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0188105540 0x7f018819cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.423 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.417+0000 7f01867fc700 1 -- 172.21.15.44:0/1433575098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0188105540 msgr2=0x7f018819cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.423 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.418+0000 7f01867fc700 1 --2- 172.21.15.44:0/1433575098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0188105540 0x7f018819cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.423 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.418+0000 7f01867fc700 1 -- 172.21.15.44:0/1433575098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0188100550 msgr2=0x7f0188198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.418+0000 7f01867fc700 1 --2- 172.21.15.44:0/1433575098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0188100550 0x7f0188198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.418+0000 7f01867fc700 1 -- 172.21.15.44:0/1433575098 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017c009520 con 0x7f0188100ef0 2023-11-16T16:59:40.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.418+0000 7f0186ffd700 1 --2- 172.21.15.44:0/1433575098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0188100550 0x7f0188198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:40.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.418+0000 7f01867fc700 1 --2- 172.21.15.44:0/1433575098 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0188100ef0 0x7f0188198b70 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f017c009fd0 tx=0x7f017c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:40.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.418+0000 7f016ffff700 1 -- 172.21.15.44:0/1433575098 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c00cbd0 con 0x7f0188100ef0 2023-11-16T16:59:40.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.418+0000 7f018d3ff700 1 -- 172.21.15.44:0/1433575098 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01780095f0 con 0x7f0188100ef0 2023-11-16T16:59:40.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.418+0000 7f018d3ff700 1 -- 172.21.15.44:0/1433575098 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0188110350 con 0x7f0188100ef0 2023-11-16T16:59:40.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.418+0000 7f016ffff700 1 -- 172.21.15.44:0/1433575098 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f017c00eea0 con 0x7f0188100ef0 2023-11-16T16:59:40.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.418+0000 7f016ffff700 1 -- 172.21.15.44:0/1433575098 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c003cf0 con 0x7f0188100ef0 2023-11-16T16:59:40.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.419+0000 7f018d3ff700 1 -- 172.21.15.44:0/1433575098 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0174005310 con 0x7f0188100ef0 2023-11-16T16:59:40.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.422+0000 7f016ffff700 1 -- 172.21.15.44:0/1433575098 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f017c00cd30 con 0x7f0188100ef0 2023-11-16T16:59:40.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.423+0000 7f016ffff700 1 --2- 172.21.15.44:0/1433575098 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f01700a1340 0x7f01700a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.423+0000 7f016ffff700 1 -- 172.21.15.44:0/1433575098 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f017c014070 con 0x7f0188100ef0 2023-11-16T16:59:40.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.423+0000 7f0186ffd700 1 --2- 172.21.15.44:0/1433575098 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f01700a1340 0x7f01700a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.424+0000 7f0186ffd700 1 --2- 172.21.15.44:0/1433575098 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f01700a1340 0x7f01700a37f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0178009990 tx=0x7f017801a040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:40.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.425+0000 7f016ffff700 1 -- 172.21.15.44:0/1433575098 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f017c0963c0 con 0x7f0188100ef0 2023-11-16T16:59:40.682 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.681+0000 7f018d3ff700 1 -- 172.21.15.44:0/1433575098 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0174005180 con 0x7f0188100ef0 2023-11-16T16:59:40.683 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.682+0000 7f016ffff700 1 -- 172.21.15.44:0/1433575098 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f017c0999e0 con 0x7f0188100ef0 2023-11-16T16:59:40.683 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T16:59:40.684 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:59:40.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.684+0000 7f018d3ff700 1 -- 172.21.15.44:0/1433575098 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f01700a1340 msgr2=0x7f01700a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.684+0000 7f018d3ff700 1 --2- 172.21.15.44:0/1433575098 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f01700a1340 0x7f01700a37f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0178009990 tx=0x7f017801a040 comp rx=0 tx=0).stop 2023-11-16T16:59:40.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.684+0000 7f018d3ff700 1 -- 172.21.15.44:0/1433575098 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0188100ef0 msgr2=0x7f0188198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.686 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.684+0000 7f018d3ff700 1 --2- 172.21.15.44:0/1433575098 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0188100ef0 0x7f0188198b70 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f017c009fd0 tx=0x7f017c00ed10 comp rx=0 tx=0).stop 2023-11-16T16:59:40.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.685+0000 7f018d3ff700 1 -- 172.21.15.44:0/1433575098 shutdown_connections 2023-11-16T16:59:40.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.685+0000 7f018d3ff700 1 --2- 172.21.15.44:0/1433575098 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f01700a1340 0x7f01700a37f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.685+0000 7f018d3ff700 1 --2- 172.21.15.44:0/1433575098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0188105540 0x7f018819cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.685+0000 7f018d3ff700 1 --2- 172.21.15.44:0/1433575098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0188100550 0x7f0188198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.687 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.685+0000 7f018d3ff700 1 --2- 172.21.15.44:0/1433575098 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0188100ef0 0x7f0188198b70 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.685+0000 7f018d3ff700 1 -- 172.21.15.44:0/1433575098 >> 172.21.15.44:0/1433575098 conn(0x7f0188077020 msgr2=0x7f01880fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:40.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.685+0000 7f018d3ff700 1 -- 172.21.15.44:0/1433575098 shutdown_connections 2023-11-16T16:59:40.688 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.685+0000 7f018d3ff700 1 -- 172.21.15.44:0/1433575098 wait complete. 2023-11-16T16:59:40.689 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T16:59:40.704 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T16:59:40.913 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.912+0000 7f48b8bff700 1 -- 172.21.15.44:0/2499415850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48b4100cc0 msgr2=0x7f48b4101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.914 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.912+0000 7f48b8bff700 1 --2- 172.21.15.44:0/2499415850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48b4100cc0 0x7f48b4101090 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f48a40099c0 tx=0x7f48a4009cd0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.914 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.912+0000 7f48b8bff700 1 -- 172.21.15.44:0/2499415850 shutdown_connections 2023-11-16T16:59:40.914 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.912+0000 7f48b8bff700 1 --2- 172.21.15.44:0/2499415850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b4105620 0x7f48b410d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.914 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.912+0000 7f48b8bff700 1 --2- 172.21.15.44:0/2499415850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48b4100cc0 0x7f48b4101090 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.914 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.912+0000 7f48b8bff700 1 --2- 172.21.15.44:0/2499415850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48b4101660 0x7f48b4104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.912+0000 7f48b8bff700 1 -- 172.21.15.44:0/2499415850 >> 172.21.15.44:0/2499415850 conn(0x7f48b4077560 msgr2=0x7f48b4077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:40.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.912+0000 7f48b8bff700 1 -- 172.21.15.44:0/2499415850 shutdown_connections 2023-11-16T16:59:40.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.913+0000 7f48b8bff700 1 -- 172.21.15.44:0/2499415850 wait complete. 2023-11-16T16:59:40.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.914+0000 7f48b8bff700 1 Processor -- start 2023-11-16T16:59:40.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.914+0000 7f48b8bff700 1 -- start start 2023-11-16T16:59:40.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.914+0000 7f48b8bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48b4100cc0 0x7f48b41965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.914+0000 7f48b8bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b4101660 0x7f48b4196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.914+0000 7f48b8bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48b4105620 0x7f48b419af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.914+0000 7f48b8bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b419b540 con 0x7f48b4105620 2023-11-16T16:59:40.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.914+0000 7f48b8bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b419b6b0 con 0x7f48b4101660 2023-11-16T16:59:40.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.914+0000 7f48b8bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b419b870 con 0x7f48b4100cc0 2023-11-16T16:59:40.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.914+0000 7f48b2d9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48b4105620 0x7f48b419af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.914+0000 7f48b259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48b4100cc0 0x7f48b41965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.914+0000 7f48b259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48b4100cc0 0x7f48b41965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:51860/0 (socket says 172.21.15.44:51860) 2023-11-16T16:59:40.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.914+0000 7f48b259c700 1 -- 172.21.15.44:0/2849634139 learned_addr learned my addr 172.21.15.44:0/2849634139 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:40.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.915+0000 7f48b1d9b700 1 --2- 172.21.15.44:0/2849634139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b4101660 0x7f48b4196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.915+0000 7f48b259c700 1 -- 172.21.15.44:0/2849634139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b4101660 msgr2=0x7f48b4196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.915+0000 7f48b259c700 1 --2- 172.21.15.44:0/2849634139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b4101660 0x7f48b4196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.915+0000 7f48b259c700 1 -- 172.21.15.44:0/2849634139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48b4105620 msgr2=0x7f48b419af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.915+0000 7f48b259c700 1 --2- 172.21.15.44:0/2849634139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48b4105620 0x7f48b419af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.915+0000 7f48b259c700 1 -- 172.21.15.44:0/2849634139 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a40095f0 con 0x7f48b4100cc0 2023-11-16T16:59:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.915+0000 7f48b259c700 1 --2- 172.21.15.44:0/2849634139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48b4100cc0 0x7f48b41965b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f48a400b580 tx=0x7f48a400bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.915+0000 7f48a37fe700 1 -- 172.21.15.44:0/2849634139 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a401d070 con 0x7f48b4100cc0 2023-11-16T16:59:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.916+0000 7f48a37fe700 1 -- 172.21.15.44:0/2849634139 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48a4005050 con 0x7f48b4100cc0 2023-11-16T16:59:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.916+0000 7f48a37fe700 1 -- 172.21.15.44:0/2849634139 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a4004a40 con 0x7f48b4100cc0 2023-11-16T16:59:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.916+0000 7f48b8bff700 1 -- 172.21.15.44:0/2849634139 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b41103d0 con 0x7f48b4100cc0 2023-11-16T16:59:40.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.916+0000 7f48b8bff700 1 -- 172.21.15.44:0/2849634139 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b4110240 con 0x7f48b4100cc0 2023-11-16T16:59:40.928 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.919+0000 7f48a37fe700 1 -- 172.21.15.44:0/2849634139 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f48a40051c0 con 0x7f48b4100cc0 2023-11-16T16:59:40.928 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.920+0000 7f48cc52cb80 1 Processor -- start 2023-11-16T16:59:40.928 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.920+0000 7f48cc52cb80 1 -- start start 2023-11-16T16:59:40.928 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.920+0000 7f48a37fe700 1 --2- 172.21.15.44:0/2849634139 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f489c0a1340 0x7f489c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.920+0000 7f48cc52cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1fefd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.920+0000 7f48cc52cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefd880 0x560f1ff01780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.920+0000 7f48cc52cb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1ff01cc0 0x560f1ff02110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.921+0000 7f48cc52cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1fdfe060 con 0x560f1fefcf70 2023-11-16T16:59:40.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.921+0000 7f48a37fe700 1 -- 172.21.15.44:0/2849634139 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f48a40cde40 con 0x7f48b4100cc0 2023-11-16T16:59:40.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.921+0000 7f48cc52cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1fe508a0 con 0x560f1fefd880 2023-11-16T16:59:40.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.921+0000 7f48cc52cb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1fe49d30 con 0x560f1ff01cc0 2023-11-16T16:59:40.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.921+0000 7f48b2d9d700 1 --2- 172.21.15.44:0/2849634139 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f489c0a1340 0x7f489c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.921+0000 7f48b1d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefd880 0x560f1ff01780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.921+0000 7f48b2d9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1ff01cc0 0x560f1ff02110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.930 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.921+0000 7f48b1d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefd880 0x560f1ff01780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:34452/0 (socket says 172.21.15.44:34452) 2023-11-16T16:59:40.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.921+0000 7f48b1d9b700 1 -- 172.21.15.44:0/785392056 learned_addr learned my addr 172.21.15.44:0/785392056 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:40.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.921+0000 7f48b259c700 1 --2- 172.21.15.44:0/785392056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1fefd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.921+0000 7f48b1d9b700 1 -- 172.21.15.44:0/785392056 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1ff01cc0 msgr2=0x560f1ff02110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:40.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.921+0000 7f48b1d9b700 1 --2- 172.21.15.44:0/785392056 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1ff01cc0 0x560f1ff02110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.921+0000 7f48b1d9b700 1 -- 172.21.15.44:0/785392056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 msgr2=0x560f1fefd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:40.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.921+0000 7f48b1d9b700 1 --2- 172.21.15.44:0/785392056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1fefd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.921+0000 7f48b1d9b700 1 -- 172.21.15.44:0/785392056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f1fccb630 con 0x560f1fefd880 2023-11-16T16:59:40.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.921+0000 7f48b259c700 1 --2- 172.21.15.44:0/785392056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1fefd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:59:40.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.922+0000 7f48b2d9d700 1 --2- 172.21.15.44:0/2849634139 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f489c0a1340 0x7f489c0a37f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f48ac009ec0 tx=0x7f48ac0134a0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:40.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.922+0000 7f48b1d9b700 1 --2- 172.21.15.44:0/785392056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefd880 0x560f1ff01780 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f48a8009cb0 tx=0x7f48a8011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3f27696f6c9489a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:40.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.922+0000 7f48a0ff9700 1 -- 172.21.15.44:0/785392056 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a8011d20 con 0x560f1fefd880 2023-11-16T16:59:40.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.922+0000 7f48a0ff9700 1 -- 172.21.15.44:0/785392056 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48a8011e80 con 0x560f1fefd880 2023-11-16T16:59:40.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.922+0000 7f48a0ff9700 1 -- 172.21.15.44:0/785392056 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a80193e0 con 0x560f1fefd880 2023-11-16T16:59:40.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.923+0000 7f48cc52cb80 1 -- 172.21.15.44:0/785392056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefd880 msgr2=0x560f1ff01780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.923+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/785392056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefd880 0x560f1ff01780 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f48a8009cb0 tx=0x7f48a8011440 comp rx=0 tx=0).stop 2023-11-16T16:59:40.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.923+0000 7f48cc52cb80 1 -- 172.21.15.44:0/785392056 shutdown_connections 2023-11-16T16:59:40.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.923+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/785392056 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1ff01cc0 0x560f1ff02110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.934 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.923+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/785392056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1fefd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.934 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.923+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/785392056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefd880 0x560f1ff01780 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.934 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.923+0000 7f48cc52cb80 1 -- 172.21.15.44:0/785392056 >> 172.21.15.44:0/785392056 conn(0x560f1feee360 msgr2=0x560f1fef3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:40.934 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.923+0000 7f48cc52cb80 1 -- 172.21.15.44:0/785392056 shutdown_connections 2023-11-16T16:59:40.934 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.923+0000 7f48cc52cb80 1 -- 172.21.15.44:0/785392056 wait complete. 2023-11-16T16:59:40.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.923+0000 7f48cc52cb80 1 -- start start 2023-11-16T16:59:40.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.923+0000 7f48cc52cb80 1 Processor -- start 2023-11-16T16:59:40.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.924+0000 7f48cc52cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefcf70 0x560f1ff09430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.924+0000 7f48cc52cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefd880 0x560f1ff06640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.924+0000 7f48cc52cb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1ff01cc0 0x560f1ff06c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.924+0000 7f48cc52cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1fdeb8e0 con 0x560f1fefd880 2023-11-16T16:59:40.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.924+0000 7f48cc52cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1fdeb580 con 0x560f1fefcf70 2023-11-16T16:59:40.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.924+0000 7f48cc52cb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1fccd920 con 0x560f1ff01cc0 2023-11-16T16:59:40.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.924+0000 7f48cc52cb80 10 client.0 _ll_register_callbacks cb 0x560f1fe09780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:59:40.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.924+0000 7f48b1d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefd880 0x560f1ff06640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.924+0000 7f48b1d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefd880 0x560f1ff06640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44238/0 (socket says 172.21.15.44:44238) 2023-11-16T16:59:40.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.924+0000 7f48b1d9b700 1 -- 172.21.15.44:0/3651646473 learned_addr learned my addr 172.21.15.44:0/3651646473 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:40.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.924+0000 7f48b259c700 1 --2- 172.21.15.44:0/3651646473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefcf70 0x560f1ff09430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.924+0000 7f48b2d9d700 1 --2- 172.21.15.44:0/3651646473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1ff01cc0 0x560f1ff06c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.925+0000 7f48b259c700 1 -- 172.21.15.44:0/3651646473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1ff01cc0 msgr2=0x560f1ff06c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:40.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.925+0000 7f48b259c700 1 --2- 172.21.15.44:0/3651646473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1ff01cc0 0x560f1ff06c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.925+0000 7f48b259c700 1 -- 172.21.15.44:0/3651646473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefd880 msgr2=0x560f1ff06640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:40.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.925+0000 7f48b259c700 1 --2- 172.21.15.44:0/3651646473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefd880 0x560f1ff06640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.925+0000 7f48b259c700 1 -- 172.21.15.44:0/3651646473 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a80098e0 con 0x560f1fefcf70 2023-11-16T16:59:40.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.925+0000 7f48b2d9d700 1 --2- 172.21.15.44:0/3651646473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1ff01cc0 0x560f1ff06c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:40.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.925+0000 7f48b1d9b700 1 --2- 172.21.15.44:0/3651646473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefd880 0x560f1ff06640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:40.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.925+0000 7f48b259c700 1 --2- 172.21.15.44:0/3651646473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefcf70 0x560f1ff09430 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f48a40cb150 tx=0x7f48a409ac50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=522e4191ecee0519 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:40.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.925+0000 7f4899ffb700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:59:40.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.925+0000 7f4899ffb700 1 -- 172.21.15.44:0/3651646473 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a409afc0 con 0x560f1fefcf70 2023-11-16T16:59:40.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.925+0000 7f4899ffb700 1 -- 172.21.15.44:0/3651646473 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f48a409a050 con 0x560f1fefcf70 2023-11-16T16:59:40.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.925+0000 7f48cc52cb80 1 -- 172.21.15.44:0/3651646473 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560f1fe0c0a0 con 0x560f1fefcf70 2023-11-16T16:59:40.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.925+0000 7f4899ffb700 1 -- 172.21.15.44:0/3651646473 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48a40b3250 con 0x560f1fefcf70 2023-11-16T16:59:40.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.925+0000 7f48cc52cb80 10 client.27179 fetch_fsmap learned FSMap version 21 2023-11-16T16:59:40.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.925+0000 7f48cc52cb80 1 -- 172.21.15.44:0/3651646473 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x560f1fe10190 con 0x560f1fefcf70 2023-11-16T16:59:40.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.926+0000 7f4899ffb700 1 -- 172.21.15.44:0/3651646473 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560f1fe0c0a0 con 0x560f1fefcf70 2023-11-16T16:59:40.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.926+0000 7f4899ffb700 1 -- 172.21.15.44:0/3651646473 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f48a4004ba0 con 0x560f1fefcf70 2023-11-16T16:59:40.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.926+0000 7f48cc52cb80 10 client.27179 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:59:40.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.926+0000 7f48cc52cb80 10 client.27179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:59:40.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.926+0000 7f48cc52cb80 20 client.27179 populate_metadata read hostname 'smithi044' 2023-11-16T16:59:40.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.926+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/3651646473 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560f1ff168f0 0x560f1ff18d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.926+0000 7f48cc52cb80 4 client.27179 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:59:40.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.926+0000 7f48cc52cb80 1 -- 172.21.15.44:0/3651646473 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560f1fe18660 con 0x560f1ff168f0 2023-11-16T16:59:40.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.926+0000 7f48b1d9b700 1 --2- 172.21.15.44:0/3651646473 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560f1ff168f0 0x560f1ff18d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.927+0000 7f48b1d9b700 1 --2- 172.21.15.44:0/3651646473 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560f1ff168f0 0x560f1ff18d00 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66022e2e1a44942d server_cookie=db547cdc22545eb5 in_seq=0 out_seq=0 2023-11-16T16:59:40.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.927+0000 7f4899ffb700 10 client.27179 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:40.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.928+0000 7f4899ffb700 1 -- 172.21.15.44:0/3651646473 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x560f1fe18660 con 0x560f1ff168f0 2023-11-16T16:59:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.928+0000 7f4899ffb700 10 client.27179 handle_command_reply: tid=0 2023-11-16T16:59:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.928+0000 7f48cc52cb80 1 client.27179 shutdown 2023-11-16T16:59:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.928+0000 7f48cc52cb80 1 -- 172.21.15.44:0/3651646473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefcf70 msgr2=0x560f1ff09430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.928+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/3651646473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefcf70 0x560f1ff09430 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f48a40cb150 tx=0x7f48a409ac50 comp rx=0 tx=0).stop 2023-11-16T16:59:40.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.928+0000 7f48cc52cb80 1 -- 172.21.15.44:0/3651646473 shutdown_connections 2023-11-16T16:59:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.928+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/3651646473 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560f1ff168f0 0x560f1ff18d00 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.928+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/3651646473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1ff01cc0 0x560f1ff06c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.928+0000 7f4899ffb700 0 client.27179 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.928+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/3651646473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefcf70 0x560f1ff09430 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.928+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/3651646473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefd880 0x560f1ff06640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.928+0000 7f48cc52cb80 1 -- 172.21.15.44:0/3651646473 >> 172.21.15.44:0/3651646473 conn(0x560f1feee360 msgr2=0x560f1ff03170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.928+0000 7f48cc52cb80 1 -- 172.21.15.44:0/3651646473 shutdown_connections 2023-11-16T16:59:40.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.928+0000 7f48cc52cb80 1 -- 172.21.15.44:0/3651646473 wait complete. 2023-11-16T16:59:40.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.929+0000 7f48cc52cb80 20 client.27179 trim_cache size 0 max 16384 2023-11-16T16:59:40.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.942+0000 7f48cc52cb80 1 Processor -- start 2023-11-16T16:59:40.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.942+0000 7f48cc52cb80 1 -- start start 2023-11-16T16:59:40.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.942+0000 7f48cc52cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1ff0ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.942+0000 7f48cc52cb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1fefd880 0x560f1ff08f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.942+0000 7f48cc52cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1ff01cc0 0x560f1fef3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.942+0000 7f48cc52cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1fef54a0 con 0x560f1fefcf70 2023-11-16T16:59:40.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.942+0000 7f48cc52cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1feeb660 con 0x560f1ff01cc0 2023-11-16T16:59:40.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.942+0000 7f48cc52cb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1fef51c0 con 0x560f1fefd880 2023-11-16T16:59:40.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.943+0000 7f48b259c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1ff0ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.943+0000 7f48b259c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1ff0ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44244/0 (socket says 172.21.15.44:44244) 2023-11-16T16:59:40.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.943+0000 7f48b259c700 1 -- 172.21.15.44:0/1468527812 learned_addr learned my addr 172.21.15.44:0/1468527812 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:40.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.943+0000 7f48b1d9b700 1 --2- 172.21.15.44:0/1468527812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1fefd880 0x560f1ff08f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.943+0000 7f48b2d9d700 1 --2- 172.21.15.44:0/1468527812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1ff01cc0 0x560f1fef3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.943+0000 7f48b259c700 1 -- 172.21.15.44:0/1468527812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1fefd880 msgr2=0x560f1ff08f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:40.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.943+0000 7f48b259c700 1 --2- 172.21.15.44:0/1468527812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1fefd880 0x560f1ff08f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.943+0000 7f48b259c700 1 -- 172.21.15.44:0/1468527812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1ff01cc0 msgr2=0x560f1fef3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:40.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.943+0000 7f48b259c700 1 --2- 172.21.15.44:0/1468527812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1ff01cc0 0x560f1fef3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.943+0000 7f48b259c700 1 -- 172.21.15.44:0/1468527812 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f1fe0fee0 con 0x560f1fefcf70 2023-11-16T16:59:40.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.943+0000 7f48b259c700 1 --2- 172.21.15.44:0/1468527812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1ff0ddb0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f48a40975d0 tx=0x7f48a409a9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=361661351637f9c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:40.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.943+0000 7f48a17fa700 1 -- 172.21.15.44:0/1468527812 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a4096950 con 0x560f1fefcf70 2023-11-16T16:59:40.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.944+0000 7f48a17fa700 1 -- 172.21.15.44:0/1468527812 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48a4004ba0 con 0x560f1fefcf70 2023-11-16T16:59:40.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.944+0000 7f48a17fa700 1 -- 172.21.15.44:0/1468527812 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a40a2a90 con 0x560f1fefcf70 2023-11-16T16:59:40.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.944+0000 7f48cc52cb80 1 -- 172.21.15.44:0/1468527812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 msgr2=0x560f1ff0ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.944+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/1468527812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1ff0ddb0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f48a40975d0 tx=0x7f48a409a9d0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.944+0000 7f48cc52cb80 1 -- 172.21.15.44:0/1468527812 shutdown_connections 2023-11-16T16:59:40.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.944+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/1468527812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1ff01cc0 0x560f1fef3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.944+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/1468527812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1ff0ddb0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.944+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/1468527812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1fefd880 0x560f1ff08f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.944+0000 7f48cc52cb80 1 -- 172.21.15.44:0/1468527812 >> 172.21.15.44:0/1468527812 conn(0x560f1feee360 msgr2=0x560f1feffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:40.953 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.944+0000 7f48cc52cb80 1 -- 172.21.15.44:0/1468527812 shutdown_connections 2023-11-16T16:59:40.953 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.944+0000 7f48cc52cb80 1 -- 172.21.15.44:0/1468527812 wait complete. 2023-11-16T16:59:40.953 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.945+0000 7f48cc52cb80 1 -- start start 2023-11-16T16:59:40.953 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.945+0000 7f48cc52cb80 1 Processor -- start 2023-11-16T16:59:40.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.945+0000 7f48cc52cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1ff09d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.945+0000 7f48cc52cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefd880 0x560f1ff0a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.945+0000 7f48cc52cb80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1ff01cc0 0x560f1ff0a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.945+0000 7f48cc52cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1ff0e3b0 con 0x560f1fefcf70 2023-11-16T16:59:40.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.945+0000 7f48cc52cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1ff697a0 con 0x560f1fefd880 2023-11-16T16:59:40.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.945+0000 7f48cc52cb80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1ff69910 con 0x560f1ff01cc0 2023-11-16T16:59:40.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.946+0000 7f48cc52cb80 10 client.0 _ll_register_callbacks cb 0x7f48a409a4c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:59:40.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.946+0000 7f48b259c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1ff09d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.946+0000 7f48b259c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1ff09d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44252/0 (socket says 172.21.15.44:44252) 2023-11-16T16:59:40.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.946+0000 7f48b259c700 1 -- 172.21.15.44:0/4257545166 learned_addr learned my addr 172.21.15.44:0/4257545166 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:40.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.946+0000 7f48b1d9b700 1 --2- 172.21.15.44:0/4257545166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefd880 0x560f1ff0a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.946+0000 7f48b2d9d700 1 --2- 172.21.15.44:0/4257545166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1ff01cc0 0x560f1ff0a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.946+0000 7f48b259c700 1 -- 172.21.15.44:0/4257545166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1ff01cc0 msgr2=0x560f1ff0a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:40.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.946+0000 7f48b259c700 1 --2- 172.21.15.44:0/4257545166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1ff01cc0 0x560f1ff0a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.946+0000 7f48b259c700 1 -- 172.21.15.44:0/4257545166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefd880 msgr2=0x560f1ff0a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:40.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.946+0000 7f48b259c700 1 --2- 172.21.15.44:0/4257545166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefd880 0x560f1ff0a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.946+0000 7f48b259c700 1 -- 172.21.15.44:0/4257545166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f1fe10190 con 0x560f1fefcf70 2023-11-16T16:59:40.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.946+0000 7f48b259c700 1 --2- 172.21.15.44:0/4257545166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1ff09d90 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f48a4000c00 tx=0x7f48a40a2d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c36d6e537d2f407 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:40.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.946+0000 7f4899ffb700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:59:40.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.947+0000 7f4899ffb700 1 -- 172.21.15.44:0/4257545166 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a4004d10 con 0x560f1fefcf70 2023-11-16T16:59:40.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.947+0000 7f4899ffb700 1 -- 172.21.15.44:0/4257545166 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f48a409ab60 con 0x560f1fefcf70 2023-11-16T16:59:40.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.947+0000 7f48cc52cb80 1 -- 172.21.15.44:0/4257545166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560f1feebb20 con 0x560f1fefcf70 2023-11-16T16:59:40.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.947+0000 7f4899ffb700 1 -- 172.21.15.44:0/4257545166 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48a409afd0 con 0x560f1fefcf70 2023-11-16T16:59:40.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.947+0000 7f48cc52cb80 10 client.17103 fetch_fsmap learned FSMap version 21 2023-11-16T16:59:40.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.947+0000 7f48cc52cb80 1 -- 172.21.15.44:0/4257545166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x560f1fc60c60 con 0x560f1fefcf70 2023-11-16T16:59:40.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.947+0000 7f4899ffb700 1 -- 172.21.15.44:0/4257545166 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48a407f080 con 0x560f1fefcf70 2023-11-16T16:59:40.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.947+0000 7f4899ffb700 1 -- 172.21.15.44:0/4257545166 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f48a40ba170 con 0x560f1fefcf70 2023-11-16T16:59:40.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.947+0000 7f48cc52cb80 10 client.17103 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:59:40.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.947+0000 7f48cc52cb80 10 client.17103 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:59:40.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.947+0000 7f48cc52cb80 20 client.17103 populate_metadata read hostname 'smithi044' 2023-11-16T16:59:40.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.947+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/4257545166 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560f1ff6ab00 0x560f1ff6cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:40.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.947+0000 7f48cc52cb80 4 client.17103 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T16:59:40.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.947+0000 7f48cc52cb80 1 -- 172.21.15.44:0/4257545166 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560f1fc98f70 con 0x560f1ff6ab00 2023-11-16T16:59:40.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.948+0000 7f48b1d9b700 1 --2- 172.21.15.44:0/4257545166 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560f1ff6ab00 0x560f1ff6cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:40.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.948+0000 7f48b1d9b700 1 --2- 172.21.15.44:0/4257545166 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560f1ff6ab00 0x560f1ff6cee0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4f61138f9f49069 server_cookie=70476f2c028b6222 in_seq=0 out_seq=0 2023-11-16T16:59:40.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.948+0000 7f4899ffb700 10 client.17103 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:40.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.948+0000 7f4899ffb700 1 -- 172.21.15.44:0/4257545166 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560f1fc98f70 con 0x560f1ff6ab00 2023-11-16T16:59:40.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.948+0000 7f4899ffb700 10 client.17103 handle_command_reply: tid=0 2023-11-16T16:59:40.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.948+0000 7f48cc52cb80 1 client.17103 shutdown 2023-11-16T16:59:40.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.948+0000 7f48cc52cb80 1 -- 172.21.15.44:0/4257545166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 msgr2=0x560f1ff09d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.948+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/4257545166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1ff09d90 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f48a4000c00 tx=0x7f48a40a2d40 comp rx=0 tx=0).stop 2023-11-16T16:59:40.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.949+0000 7f48cc52cb80 1 -- 172.21.15.44:0/4257545166 shutdown_connections 2023-11-16T16:59:40.963 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.949+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/4257545166 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x560f1ff6ab00 0x560f1ff6cee0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.949+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/4257545166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560f1ff01cc0 0x560f1ff0a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.949+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/4257545166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560f1fefcf70 0x560f1ff09d90 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.949+0000 7f4899ffb700 0 client.17103 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.949+0000 7f48cc52cb80 1 --2- 172.21.15.44:0/4257545166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f1fefd880 0x560f1ff0a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.949+0000 7f48cc52cb80 1 -- 172.21.15.44:0/4257545166 >> 172.21.15.44:0/4257545166 conn(0x560f1feee360 msgr2=0x560f1ff05ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.949+0000 7f48cc52cb80 1 -- 172.21.15.44:0/4257545166 shutdown_connections 2023-11-16T16:59:40.964 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.949+0000 7f48cc52cb80 1 -- 172.21.15.44:0/4257545166 wait complete. 2023-11-16T16:59:40.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.949+0000 7f48cc52cb80 20 client.17103 trim_cache size 0 max 16384 2023-11-16T16:59:40.965 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T16:59:40.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.949+0000 7f48b8bff700 1 -- 172.21.15.44:0/2849634139 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f489c0a1340 msgr2=0x7f489c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.949+0000 7f48b8bff700 1 --2- 172.21.15.44:0/2849634139 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f489c0a1340 0x7f489c0a37f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f48ac009ec0 tx=0x7f48ac0134a0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.949+0000 7f48b8bff700 1 -- 172.21.15.44:0/2849634139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48b4100cc0 msgr2=0x7f48b41965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:40.966 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.949+0000 7f48b8bff700 1 --2- 172.21.15.44:0/2849634139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48b4100cc0 0x7f48b41965b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f48a400b580 tx=0x7f48a400bfd0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.950+0000 7f48b8bff700 1 -- 172.21.15.44:0/2849634139 shutdown_connections 2023-11-16T16:59:40.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.950+0000 7f48b8bff700 1 --2- 172.21.15.44:0/2849634139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48b4105620 0x7f48b419af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.950+0000 7f48b8bff700 1 --2- 172.21.15.44:0/2849634139 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f489c0a1340 0x7f489c0a37f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.950+0000 7f48b8bff700 1 --2- 172.21.15.44:0/2849634139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48b4100cc0 0x7f48b41965b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.950+0000 7f48b8bff700 1 --2- 172.21.15.44:0/2849634139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b4101660 0x7f48b4196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:40.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.950+0000 7f48b8bff700 1 -- 172.21.15.44:0/2849634139 >> 172.21.15.44:0/2849634139 conn(0x7f48b4077560 msgr2=0x7f48b40fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:40.967 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.950+0000 7f48b8bff700 1 -- 172.21.15.44:0/2849634139 shutdown_connections 2023-11-16T16:59:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:40.950+0000 7f48b8bff700 1 -- 172.21.15.44:0/2849634139 wait complete. 2023-11-16T16:59:41.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:41 smithi044 ceph-mon[128750]: pgmap v303: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:59:41.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:41 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1433575098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:59:41.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:41 smithi176 ceph-mon[131630]: pgmap v303: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:59:41.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:41 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1433575098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:59:41.970 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-11-16T16:59:41.970 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-11-16T16:59:41.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:41 smithi077 ceph-mon[131204]: pgmap v303: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:59:41.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:41 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1433575098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:59:42.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.190+0000 7f81913ff700 1 -- 172.21.15.44:0/3968200815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f818c10cec0 msgr2=0x7f818c10f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.190+0000 7f81913ff700 1 --2- 172.21.15.44:0/3968200815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f818c10cec0 0x7f818c10f2a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f818400b0a0 tx=0x7f818400b3b0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.190+0000 7f81913ff700 1 -- 172.21.15.44:0/3968200815 shutdown_connections 2023-11-16T16:59:42.191 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.190+0000 7f81913ff700 1 --2- 172.21.15.44:0/3968200815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f818c10cec0 0x7f818c10f2a0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.190+0000 7f81913ff700 1 --2- 172.21.15.44:0/3968200815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c0746f0 0x7f818c072d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.190+0000 7f81913ff700 1 --2- 172.21.15.44:0/3968200815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f818c074ac0 0x7f818c073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.190+0000 7f81913ff700 1 -- 172.21.15.44:0/3968200815 >> 172.21.15.44:0/3968200815 conn(0x7f818c077560 msgr2=0x7f818c077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.190+0000 7f81913ff700 1 -- 172.21.15.44:0/3968200815 shutdown_connections 2023-11-16T16:59:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.190+0000 7f81913ff700 1 -- 172.21.15.44:0/3968200815 wait complete. 2023-11-16T16:59:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.192+0000 7f81913ff700 1 Processor -- start 2023-11-16T16:59:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.192+0000 7f81913ff700 1 -- start start 2023-11-16T16:59:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f81913ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f818c0746f0 0x7f818c19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f81913ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f818c074ac0 0x7f818c19eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f81913ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c10cec0 0x7f818c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f81913ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818c198cc0 con 0x7f818c0746f0 2023-11-16T16:59:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f81913ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818c198e30 con 0x7f818c10cec0 2023-11-16T16:59:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f81913ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818c198ff0 con 0x7f818c074ac0 2023-11-16T16:59:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f818affd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f818c0746f0 0x7f818c19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f818affd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f818c0746f0 0x7f818c19e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44278/0 (socket says 172.21.15.44:44278) 2023-11-16T16:59:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f818affd700 1 -- 172.21.15.44:0/136217448 learned_addr learned my addr 172.21.15.44:0/136217448 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f818b7fe700 1 --2- 172.21.15.44:0/136217448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c10cec0 0x7f818c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f818a7fc700 1 --2- 172.21.15.44:0/136217448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f818c074ac0 0x7f818c19eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f818affd700 1 -- 172.21.15.44:0/136217448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f818c074ac0 msgr2=0x7f818c19eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f818affd700 1 --2- 172.21.15.44:0/136217448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f818c074ac0 0x7f818c19eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f818affd700 1 -- 172.21.15.44:0/136217448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c10cec0 msgr2=0x7f818c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.196 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f818affd700 1 --2- 172.21.15.44:0/136217448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c10cec0 0x7f818c198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f818affd700 1 -- 172.21.15.44:0/136217448 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8180009520 con 0x7f818c0746f0 2023-11-16T16:59:42.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f818b7fe700 1 --2- 172.21.15.44:0/136217448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c10cec0 0x7f818c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:59:42.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.193+0000 7f818affd700 1 --2- 172.21.15.44:0/136217448 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f818c0746f0 0x7f818c19e5e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f817c00e9a0 tx=0x7f817c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:42.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.194+0000 7f8173fff700 1 -- 172.21.15.44:0/136217448 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c009790 con 0x7f818c0746f0 2023-11-16T16:59:42.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.194+0000 7f8173fff700 1 -- 172.21.15.44:0/136217448 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f817c00cbd0 con 0x7f818c0746f0 2023-11-16T16:59:42.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.194+0000 7f8173fff700 1 -- 172.21.15.44:0/136217448 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c018680 con 0x7f818c0746f0 2023-11-16T16:59:42.197 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.194+0000 7f81913ff700 1 -- 172.21.15.44:0/136217448 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8184009d00 con 0x7f818c0746f0 2023-11-16T16:59:42.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.194+0000 7f81913ff700 1 -- 172.21.15.44:0/136217448 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f818c1125c0 con 0x7f818c0746f0 2023-11-16T16:59:42.198 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.197+0000 7f8173fff700 1 -- 172.21.15.44:0/136217448 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f817c00cd40 con 0x7f818c0746f0 2023-11-16T16:59:42.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.198+0000 7f81a4d94b80 1 Processor -- start 2023-11-16T16:59:42.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.198+0000 7f8173fff700 1 --2- 172.21.15.44:0/136217448 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f81740a1340 0x7f81740a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.199+0000 7f81a4d94b80 1 -- start start 2023-11-16T16:59:42.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.199+0000 7f8173fff700 1 -- 172.21.15.44:0/136217448 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f817c014070 con 0x7f818c0746f0 2023-11-16T16:59:42.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.199+0000 7f818b7fe700 1 --2- 172.21.15.44:0/136217448 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f81740a1340 0x7f81740a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.200 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.199+0000 7f81a4d94b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 0x560778241fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.199+0000 7f81a4d94b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5607782424f0 0x5607782463f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.199+0000 7f81a4d94b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560778246930 0x560778246d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.199+0000 7f81a4d94b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607781470e0 con 0x5607782424f0 2023-11-16T16:59:42.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.199+0000 7f81a4d94b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560778199920 con 0x560778246930 2023-11-16T16:59:42.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.199+0000 7f81a4d94b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x560778192db0 con 0x560778241be0 2023-11-16T16:59:42.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.199+0000 7f818affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 0x560778241fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.199+0000 7f818a7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5607782424f0 0x5607782463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.201 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.199+0000 7f818affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 0x560778241fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:51922/0 (socket says 172.21.15.44:51922) 2023-11-16T16:59:42.202 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.199+0000 7f818affd700 1 -- 172.21.15.44:0/1674106759 learned_addr learned my addr 172.21.15.44:0/1674106759 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:42.202 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.199+0000 7f818b7fe700 1 --2- 172.21.15.44:0/1674106759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560778246930 0x560778246d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.200+0000 7f818a7fc700 1 -- 172.21.15.44:0/1674106759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 msgr2=0x560778241fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:42.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.200+0000 7f818a7fc700 1 --2- 172.21.15.44:0/1674106759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 0x560778241fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.200+0000 7f818a7fc700 1 -- 172.21.15.44:0/1674106759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560778246930 msgr2=0x560778246d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:42.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.200+0000 7f818a7fc700 1 --2- 172.21.15.44:0/1674106759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560778246930 0x560778246d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.200+0000 7f818a7fc700 1 -- 172.21.15.44:0/1674106759 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607780146b0 con 0x5607782424f0 2023-11-16T16:59:42.206 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.200+0000 7f818b7fe700 1 --2- 172.21.15.44:0/136217448 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f81740a1340 0x7f81740a37f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8184009fd0 tx=0x7f8184007ed0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:42.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.200+0000 7f818a7fc700 1 --2- 172.21.15.44:0/1674106759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5607782424f0 0x5607782463f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f818c06a120 tx=0x7f8180011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eeb866235da03d0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:42.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.200+0000 7f81717fa700 1 -- 172.21.15.44:0/1674106759 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180019630 con 0x5607782424f0 2023-11-16T16:59:42.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.200+0000 7f81717fa700 1 -- 172.21.15.44:0/1674106759 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8180019c50 con 0x5607782424f0 2023-11-16T16:59:42.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.200+0000 7f81717fa700 1 -- 172.21.15.44:0/1674106759 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818000ec50 con 0x5607782424f0 2023-11-16T16:59:42.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.201+0000 7f81a4d94b80 1 -- 172.21.15.44:0/1674106759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5607782424f0 msgr2=0x5607782463f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.201+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/1674106759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5607782424f0 0x5607782463f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f818c06a120 tx=0x7f8180011720 comp rx=0 tx=0).stop 2023-11-16T16:59:42.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.201+0000 7f81a4d94b80 1 -- 172.21.15.44:0/1674106759 shutdown_connections 2023-11-16T16:59:42.207 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.201+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/1674106759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560778246930 0x560778246d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.201+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/1674106759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 0x560778241fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.201+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/1674106759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5607782424f0 0x5607782463f0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.201+0000 7f81a4d94b80 1 -- 172.21.15.44:0/1674106759 >> 172.21.15.44:0/1674106759 conn(0x5607782373e0 msgr2=0x56077823cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:42.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.201+0000 7f81a4d94b80 1 -- 172.21.15.44:0/1674106759 shutdown_connections 2023-11-16T16:59:42.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.201+0000 7f81a4d94b80 1 -- 172.21.15.44:0/1674106759 wait complete. 2023-11-16T16:59:42.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.202+0000 7f81a4d94b80 1 -- start start 2023-11-16T16:59:42.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.202+0000 7f81a4d94b80 1 Processor -- start 2023-11-16T16:59:42.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.202+0000 7f81a4d94b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 0x56077824e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.202+0000 7f81a4d94b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5607782424f0 0x56077824b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.202+0000 7f81a4d94b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560778246930 0x56077824b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.202+0000 7f81a4d94b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x560778134960 con 0x560778246930 2023-11-16T16:59:42.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.202+0000 7f81a4d94b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560778134600 con 0x5607782424f0 2023-11-16T16:59:42.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.202+0000 7f81a4d94b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607780169a0 con 0x560778241be0 2023-11-16T16:59:42.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.202+0000 7f81a4d94b80 10 client.0 _ll_register_callbacks cb 0x560778152800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:59:42.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.203+0000 7f818b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560778246930 0x56077824b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.203+0000 7f818b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560778246930 0x56077824b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44294/0 (socket says 172.21.15.44:44294) 2023-11-16T16:59:42.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.203+0000 7f818b7fe700 1 -- 172.21.15.44:0/3150411383 learned_addr learned my addr 172.21.15.44:0/3150411383 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:42.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.203+0000 7f818affd700 1 --2- 172.21.15.44:0/3150411383 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 0x56077824e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.203+0000 7f818a7fc700 1 --2- 172.21.15.44:0/3150411383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5607782424f0 0x56077824b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.203+0000 7f818b7fe700 1 -- 172.21.15.44:0/3150411383 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 msgr2=0x56077824e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:42.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.203+0000 7f818b7fe700 1 --2- 172.21.15.44:0/3150411383 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 0x56077824e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.210 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.203+0000 7f818b7fe700 1 -- 172.21.15.44:0/3150411383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5607782424f0 msgr2=0x56077824b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:42.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.203+0000 7f818b7fe700 1 --2- 172.21.15.44:0/3150411383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5607782424f0 0x56077824b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.203+0000 7f818b7fe700 1 -- 172.21.15.44:0/3150411383 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8180009a70 con 0x560778246930 2023-11-16T16:59:42.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.203+0000 7f818affd700 1 --2- 172.21.15.44:0/3150411383 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 0x56077824e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:59:42.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.203+0000 7f818a7fc700 1 --2- 172.21.15.44:0/3150411383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5607782424f0 0x56077824b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:59:42.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.203+0000 7f818b7fe700 1 --2- 172.21.15.44:0/3150411383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560778246930 0x56077824b890 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8184015000 tx=0x7f818400b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1720b05869db2e2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:42.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.203+0000 7f816eff5700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:59:42.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.204+0000 7f816eff5700 1 -- 172.21.15.44:0/3150411383 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818400e070 con 0x560778246930 2023-11-16T16:59:42.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.204+0000 7f816eff5700 1 -- 172.21.15.44:0/3150411383 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f81840054e0 con 0x560778246930 2023-11-16T16:59:42.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.204+0000 7f81a4d94b80 1 -- 172.21.15.44:0/3150411383 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560778234ba0 con 0x560778246930 2023-11-16T16:59:42.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.204+0000 7f816eff5700 1 -- 172.21.15.44:0/3150411383 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81840094b0 con 0x560778246930 2023-11-16T16:59:42.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.204+0000 7f81a4d94b80 10 client.17115 fetch_fsmap learned FSMap version 21 2023-11-16T16:59:42.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.204+0000 7f81a4d94b80 1 -- 172.21.15.44:0/3150411383 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x560778159210 con 0x560778246930 2023-11-16T16:59:42.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.204+0000 7f816eff5700 1 -- 172.21.15.44:0/3150411383 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560778234ba0 con 0x560778246930 2023-11-16T16:59:42.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.204+0000 7f816eff5700 1 -- 172.21.15.44:0/3150411383 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8184028030 con 0x560778246930 2023-11-16T16:59:42.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.204+0000 7f81a4d94b80 10 client.17115 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:59:42.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.204+0000 7f81a4d94b80 10 client.17115 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:59:42.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.204+0000 7f81a4d94b80 20 client.17115 populate_metadata read hostname 'smithi044' 2023-11-16T16:59:42.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.205+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/3150411383 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56077825b8f0 0x56077825dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.205+0000 7f81a4d94b80 4 client.17115 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:59:42.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.205+0000 7f81a4d94b80 1 -- 172.21.15.44:0/3150411383 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560778155120 con 0x56077825b8f0 2023-11-16T16:59:42.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.205+0000 7f818affd700 1 --2- 172.21.15.44:0/3150411383 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56077825b8f0 0x56077825dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.205+0000 7f818affd700 1 --2- 172.21.15.44:0/3150411383 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56077825b8f0 0x56077825dd00 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e786d2d2eadea99e server_cookie=2a54ad8c11497d30 in_seq=0 out_seq=0 2023-11-16T16:59:42.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.205+0000 7f816eff5700 10 client.17115 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:42.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.206+0000 7f816eff5700 1 -- 172.21.15.44:0/3150411383 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x560778155120 con 0x56077825b8f0 2023-11-16T16:59:42.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.206+0000 7f816eff5700 10 client.17115 handle_command_reply: tid=0 2023-11-16T16:59:42.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.206+0000 7f81a4d94b80 1 client.17115 shutdown 2023-11-16T16:59:42.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.206+0000 7f81a4d94b80 1 -- 172.21.15.44:0/3150411383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560778246930 msgr2=0x56077824b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.206+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/3150411383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560778246930 0x56077824b890 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8184015000 tx=0x7f818400b3b0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.206+0000 7f81a4d94b80 1 -- 172.21.15.44:0/3150411383 shutdown_connections 2023-11-16T16:59:42.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.206+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/3150411383 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x56077825b8f0 0x56077825dd00 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.206+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/3150411383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560778246930 0x56077824b890 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.206+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/3150411383 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 0x56077824e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.206+0000 7f816eff5700 0 client.17115 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:42.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.206+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/3150411383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5607782424f0 0x56077824b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.206+0000 7f81a4d94b80 1 -- 172.21.15.44:0/3150411383 >> 172.21.15.44:0/3150411383 conn(0x5607782373e0 msgr2=0x56077823be60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:42.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.207+0000 7f81a4d94b80 1 -- 172.21.15.44:0/3150411383 shutdown_connections 2023-11-16T16:59:42.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.207+0000 7f81a4d94b80 1 -- 172.21.15.44:0/3150411383 wait complete. 2023-11-16T16:59:42.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.207+0000 7f81a4d94b80 20 client.17115 trim_cache size 0 max 16384 2023-11-16T16:59:42.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.225+0000 7f81a4d94b80 1 Processor -- start 2023-11-16T16:59:42.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.225+0000 7f81a4d94b80 1 -- start start 2023-11-16T16:59:42.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.225+0000 7f81a4d94b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560778241be0 0x560778252a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.225+0000 7f81a4d94b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5607782424f0 0x56077824d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.225+0000 7f81a4d94b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778246930 0x56077823c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.225+0000 7f81a4d94b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x56077823e240 con 0x5607782424f0 2023-11-16T16:59:42.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.225+0000 7f81a4d94b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560778246300 con 0x560778241be0 2023-11-16T16:59:42.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.225+0000 7f81a4d94b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x560778236f90 con 0x560778246930 2023-11-16T16:59:42.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.226+0000 7f818affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560778241be0 0x560778252a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.226+0000 7f818b7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778246930 0x56077823c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.226+0000 7f818b7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778246930 0x56077823c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:51938/0 (socket says 172.21.15.44:51938) 2023-11-16T16:59:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.226+0000 7f818b7fe700 1 -- 172.21.15.44:0/723474848 learned_addr learned my addr 172.21.15.44:0/723474848 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.226+0000 7f818affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560778241be0 0x560778252a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:34506/0 (socket says 172.21.15.44:34506) 2023-11-16T16:59:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.226+0000 7f818a7fc700 1 --2- 172.21.15.44:0/723474848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5607782424f0 0x56077824d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.226+0000 7f818b7fe700 1 -- 172.21.15.44:0/723474848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560778241be0 msgr2=0x560778252a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:42.229 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.226+0000 7f818b7fe700 1 --2- 172.21.15.44:0/723474848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560778241be0 0x560778252a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.230 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.226+0000 7f818b7fe700 1 -- 172.21.15.44:0/723474848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5607782424f0 msgr2=0x56077824d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:42.230 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.226+0000 7f818b7fe700 1 --2- 172.21.15.44:0/723474848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5607782424f0 0x56077824d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.230 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.226+0000 7f818b7fe700 1 -- 172.21.15.44:0/723474848 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560778247220 con 0x560778246930 2023-11-16T16:59:42.230 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.226+0000 7f818b7fe700 1 --2- 172.21.15.44:0/723474848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778246930 0x56077823c5e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f818400b800 tx=0x7f8184007d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efa34af2d491cb30 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:42.230 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.227+0000 7f8171ffb700 1 -- 172.21.15.44:0/723474848 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818400e070 con 0x560778246930 2023-11-16T16:59:42.230 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.227+0000 7f8171ffb700 1 -- 172.21.15.44:0/723474848 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8184005330 con 0x560778246930 2023-11-16T16:59:42.230 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.227+0000 7f8171ffb700 1 -- 172.21.15.44:0/723474848 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81840094b0 con 0x560778246930 2023-11-16T16:59:42.230 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.227+0000 7f81a4d94b80 1 -- 172.21.15.44:0/723474848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778246930 msgr2=0x56077823c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.227+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/723474848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778246930 0x56077823c5e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f818400b800 tx=0x7f8184007d80 comp rx=0 tx=0).stop 2023-11-16T16:59:42.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.227+0000 7f81a4d94b80 1 -- 172.21.15.44:0/723474848 shutdown_connections 2023-11-16T16:59:42.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.227+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/723474848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778246930 0x56077823c5e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.227+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/723474848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560778241be0 0x560778252a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.227+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/723474848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5607782424f0 0x56077824d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.227+0000 7f81a4d94b80 1 -- 172.21.15.44:0/723474848 >> 172.21.15.44:0/723474848 conn(0x5607782373e0 msgr2=0x56077824cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:42.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.227+0000 7f81a4d94b80 1 -- 172.21.15.44:0/723474848 shutdown_connections 2023-11-16T16:59:42.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.227+0000 7f81a4d94b80 1 -- 172.21.15.44:0/723474848 wait complete. 2023-11-16T16:59:42.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.228+0000 7f81a4d94b80 1 -- start start 2023-11-16T16:59:42.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.228+0000 7f81a4d94b80 1 Processor -- start 2023-11-16T16:59:42.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f81a4d94b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 0x5607782a95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f81a4d94b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5607782424f0 0x5607782a9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f81a4d94b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560778246930 0x5607782a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f81a4d94b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607782a3f20 con 0x560778246930 2023-11-16T16:59:42.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f81a4d94b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607782a4090 con 0x5607782424f0 2023-11-16T16:59:42.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f81a4d94b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607782a4250 con 0x560778241be0 2023-11-16T16:59:42.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f81a4d94b80 10 client.0 _ll_register_callbacks cb 0x7f81840078f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:59:42.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f818affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 0x5607782a95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f818affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 0x5607782a95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:51954/0 (socket says 172.21.15.44:51954) 2023-11-16T16:59:42.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f818a7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5607782424f0 0x5607782a9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f818affd700 1 -- 172.21.15.44:0/2387641148 learned_addr learned my addr 172.21.15.44:0/2387641148 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:42.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f818b7fe700 1 --2- 172.21.15.44:0/2387641148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560778246930 0x5607782a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f818affd700 1 -- 172.21.15.44:0/2387641148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5607782424f0 msgr2=0x5607782a9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:42.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f818affd700 1 --2- 172.21.15.44:0/2387641148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5607782424f0 0x5607782a9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f818affd700 1 -- 172.21.15.44:0/2387641148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560778246930 msgr2=0x5607782a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:42.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f818affd700 1 --2- 172.21.15.44:0/2387641148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560778246930 0x5607782a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f818affd700 1 -- 172.21.15.44:0/2387641148 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560778159210 con 0x560778241be0 2023-11-16T16:59:42.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f818b7fe700 1 --2- 172.21.15.44:0/2387641148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560778246930 0x5607782a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:42.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.229+0000 7f818a7fc700 1 --2- 172.21.15.44:0/2387641148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5607782424f0 0x5607782a9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:42.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.230+0000 7f818affd700 1 --2- 172.21.15.44:0/2387641148 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 0x5607782a95a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f817c099f50 tx=0x7f817c0cc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4c6e33e47ca1650 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:42.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.230+0000 7f816eff5700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T16:59:42.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.230+0000 7f816eff5700 1 -- 172.21.15.44:0/2387641148 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c09dff0 con 0x560778241be0 2023-11-16T16:59:42.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.230+0000 7f816eff5700 1 -- 172.21.15.44:0/2387641148 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f817c0a2410 con 0x560778241be0 2023-11-16T16:59:42.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.230+0000 7f81a4d94b80 1 -- 172.21.15.44:0/2387641148 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8180019c50 con 0x560778241be0 2023-11-16T16:59:42.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.230+0000 7f816eff5700 1 -- 172.21.15.44:0/2387641148 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f817c09a220 con 0x560778241be0 2023-11-16T16:59:42.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.230+0000 7f81a4d94b80 10 client.27139 fetch_fsmap learned FSMap version 21 2023-11-16T16:59:42.239 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.230+0000 7f81a4d94b80 1 -- 172.21.15.44:0/2387641148 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x560777fc0e60 con 0x560778241be0 2023-11-16T16:59:42.239 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.230+0000 7f816eff5700 1 -- 172.21.15.44:0/2387641148 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f817c07c020 con 0x560778241be0 2023-11-16T16:59:42.239 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.231+0000 7f816eff5700 1 -- 172.21.15.44:0/2387641148 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f817c094bd0 con 0x560778241be0 2023-11-16T16:59:42.239 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.231+0000 7f81a4d94b80 10 client.27139 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:59:42.239 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.231+0000 7f81a4d94b80 10 client.27139 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:59:42.239 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.231+0000 7f81a4d94b80 20 client.27139 populate_metadata read hostname 'smithi044' 2023-11-16T16:59:42.239 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.231+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/2387641148 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5607782b1470 0x5607782b3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.231+0000 7f81a4d94b80 4 client.27139 mds_command: new command op to 24551 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-11-16T16:59:42.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.231+0000 7f81a4d94b80 1 -- 172.21.15.44:0/2387641148 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5607781616e0 con 0x5607782b1470 2023-11-16T16:59:42.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.231+0000 7f818a7fc700 1 --2- 172.21.15.44:0/2387641148 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5607782b1470 0x5607782b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.231+0000 7f818a7fc700 1 --2- 172.21.15.44:0/2387641148 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5607782b1470 0x5607782b3880 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d80b93165fa3cb9 server_cookie=d62918e1efa0437e in_seq=0 out_seq=0 2023-11-16T16:59:42.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.231+0000 7f816eff5700 10 client.27139 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:42.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.233+0000 7f816eff5700 1 -- 172.21.15.44:0/2387641148 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5607781616e0 con 0x5607782b1470 2023-11-16T16:59:42.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.233+0000 7f816eff5700 10 client.27139 handle_command_reply: tid=0 2023-11-16T16:59:42.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.233+0000 7f81a4d94b80 1 client.27139 shutdown 2023-11-16T16:59:42.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.233+0000 7f81a4d94b80 1 -- 172.21.15.44:0/2387641148 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 msgr2=0x5607782a95a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.233+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/2387641148 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 0x5607782a95a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f817c099f50 tx=0x7f817c0cc040 comp rx=0 tx=0).stop 2023-11-16T16:59:42.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.233+0000 7f81a4d94b80 1 -- 172.21.15.44:0/2387641148 shutdown_connections 2023-11-16T16:59:42.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.233+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/2387641148 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5607782b1470 0x5607782b3880 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.233+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/2387641148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560778246930 0x5607782a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.233+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/2387641148 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x560778241be0 0x5607782a95a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.233+0000 7f816eff5700 0 client.27139 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:42.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.233+0000 7f81a4d94b80 1 --2- 172.21.15.44:0/2387641148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5607782424f0 0x5607782a9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.233+0000 7f81a4d94b80 1 -- 172.21.15.44:0/2387641148 >> 172.21.15.44:0/2387641148 conn(0x5607782373e0 msgr2=0x560778241270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:42.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.233+0000 7f81a4d94b80 1 -- 172.21.15.44:0/2387641148 shutdown_connections 2023-11-16T16:59:42.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.233+0000 7f81a4d94b80 1 -- 172.21.15.44:0/2387641148 wait complete. 2023-11-16T16:59:42.242 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.233+0000 7f81a4d94b80 20 client.27139 trim_cache size 0 max 16384 2023-11-16T16:59:42.242 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:59:42.242 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2023-11-16T16:59:42.243 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "c64f63b3-8976-493c-bda7-faf84f6f8650", 2023-11-16T16:59:42.243 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2023-11-16T16:59:42.243 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:59:42.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.234+0000 7f81913ff700 1 -- 172.21.15.44:0/136217448 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f81740a1340 msgr2=0x7f81740a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.234+0000 7f81913ff700 1 --2- 172.21.15.44:0/136217448 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f81740a1340 0x7f81740a37f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8184009fd0 tx=0x7f8184007ed0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.234+0000 7f81913ff700 1 -- 172.21.15.44:0/136217448 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f818c0746f0 msgr2=0x7f818c19e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.234+0000 7f81913ff700 1 --2- 172.21.15.44:0/136217448 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f818c0746f0 0x7f818c19e5e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f817c00e9a0 tx=0x7f817c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.234+0000 7f81913ff700 1 -- 172.21.15.44:0/136217448 shutdown_connections 2023-11-16T16:59:42.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.234+0000 7f81913ff700 1 --2- 172.21.15.44:0/136217448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c10cec0 0x7f818c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.234+0000 7f81913ff700 1 --2- 172.21.15.44:0/136217448 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f818c0746f0 0x7f818c19e5e0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.234+0000 7f81913ff700 1 --2- 172.21.15.44:0/136217448 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f81740a1340 0x7f81740a37f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.234+0000 7f81913ff700 1 --2- 172.21.15.44:0/136217448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f818c074ac0 0x7f818c19eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.234+0000 7f81913ff700 1 -- 172.21.15.44:0/136217448 >> 172.21.15.44:0/136217448 conn(0x7f818c077560 msgr2=0x7f818c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:42.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.234+0000 7f81913ff700 1 -- 172.21.15.44:0/136217448 shutdown_connections 2023-11-16T16:59:42.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.234+0000 7f81913ff700 1 -- 172.21.15.44:0/136217448 wait complete. 2023-11-16T16:59:42.254 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T16:59:42.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.465+0000 7f3d7d9ff700 1 -- 172.21.15.44:0/2573855928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d78101660 msgr2=0x7f3d78104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.465+0000 7f3d7d9ff700 1 --2- 172.21.15.44:0/2573855928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d78101660 0x7f3d78104ef0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3d600099c0 tx=0x7f3d60009cd0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.465+0000 7f3d7d9ff700 1 -- 172.21.15.44:0/2573855928 shutdown_connections 2023-11-16T16:59:42.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.465+0000 7f3d7d9ff700 1 --2- 172.21.15.44:0/2573855928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d78105620 0x7f3d7810d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.466 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.465+0000 7f3d7d9ff700 1 --2- 172.21.15.44:0/2573855928 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d78100cc0 0x7f3d78101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.465+0000 7f3d7d9ff700 1 --2- 172.21.15.44:0/2573855928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d78101660 0x7f3d78104ef0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.465+0000 7f3d7d9ff700 1 -- 172.21.15.44:0/2573855928 >> 172.21.15.44:0/2573855928 conn(0x7f3d78077560 msgr2=0x7f3d78077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:42.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.465+0000 7f3d7d9ff700 1 -- 172.21.15.44:0/2573855928 shutdown_connections 2023-11-16T16:59:42.467 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.465+0000 7f3d7d9ff700 1 -- 172.21.15.44:0/2573855928 wait complete. 2023-11-16T16:59:42.469 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.466+0000 7f3d7d9ff700 1 Processor -- start 2023-11-16T16:59:42.469 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.466+0000 7f3d7d9ff700 1 -- start start 2023-11-16T16:59:42.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.466+0000 7f3d7d9ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d78100cc0 0x7f3d781986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.467+0000 7f3d7d9ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d78101660 0x7f3d78198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.467+0000 7f3d7d9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d78105620 0x7f3d7819bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.467+0000 7f3d7d9ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7819c590 con 0x7f3d78100cc0 2023-11-16T16:59:42.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.467+0000 7f3d7d9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7819c700 con 0x7f3d78105620 2023-11-16T16:59:42.470 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.467+0000 7f3d7d9ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7819c8c0 con 0x7f3d78101660 2023-11-16T16:59:42.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.467+0000 7f3d76ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d78100cc0 0x7f3d781986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.467+0000 7f3d76ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d78100cc0 0x7f3d781986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44324/0 (socket says 172.21.15.44:44324) 2023-11-16T16:59:42.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.467+0000 7f3d76ffd700 1 -- 172.21.15.44:0/3158516788 learned_addr learned my addr 172.21.15.44:0/3158516788 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:42.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.467+0000 7f3d6ffff700 1 --2- 172.21.15.44:0/3158516788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d78101660 0x7f3d78198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.467+0000 7f3d777fe700 1 --2- 172.21.15.44:0/3158516788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d78105620 0x7f3d7819bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.467+0000 7f3d76ffd700 1 -- 172.21.15.44:0/3158516788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d78101660 msgr2=0x7f3d78198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.471 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.467+0000 7f3d76ffd700 1 --2- 172.21.15.44:0/3158516788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d78101660 0x7f3d78198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.467+0000 7f3d76ffd700 1 -- 172.21.15.44:0/3158516788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d78105620 msgr2=0x7f3d7819bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.467+0000 7f3d76ffd700 1 --2- 172.21.15.44:0/3158516788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d78105620 0x7f3d7819bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.467+0000 7f3d76ffd700 1 -- 172.21.15.44:0/3158516788 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d600095f0 con 0x7f3d78100cc0 2023-11-16T16:59:42.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.467+0000 7f3d6ffff700 1 --2- 172.21.15.44:0/3158516788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d78101660 0x7f3d78198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:42.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.468+0000 7f3d76ffd700 1 --2- 172.21.15.44:0/3158516788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d78100cc0 0x7f3d781986a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800e9a0 tx=0x7f3d6800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:42.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.468+0000 7f3d7c9fd700 1 -- 172.21.15.44:0/3158516788 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68009790 con 0x7f3d78100cc0 2023-11-16T16:59:42.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.468+0000 7f3d7c9fd700 1 -- 172.21.15.44:0/3158516788 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d68010430 con 0x7f3d78100cc0 2023-11-16T16:59:42.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.468+0000 7f3d7c9fd700 1 -- 172.21.15.44:0/3158516788 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68018680 con 0x7f3d78100cc0 2023-11-16T16:59:42.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.468+0000 7f3d7d9ff700 1 -- 172.21.15.44:0/3158516788 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d78110370 con 0x7f3d78100cc0 2023-11-16T16:59:42.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.468+0000 7f3d7d9ff700 1 -- 172.21.15.44:0/3158516788 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d781101e0 con 0x7f3d78100cc0 2023-11-16T16:59:42.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.471+0000 7f3d7c9fd700 1 -- 172.21.15.44:0/3158516788 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f3d680105a0 con 0x7f3d78100cc0 2023-11-16T16:59:42.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.472+0000 7f3d913bab80 1 Processor -- start 2023-11-16T16:59:42.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.473+0000 7f3d913bab80 1 -- start start 2023-11-16T16:59:42.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.473+0000 7f3d913bab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b3529b60 0x5558b3529f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.473+0000 7f3d913bab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b352e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.473+0000 7f3d913bab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b352e8b0 0x5558b352ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.473+0000 7f3d913bab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b342f060 con 0x5558b352a470 2023-11-16T16:59:42.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.473+0000 7f3d913bab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b34818a0 con 0x5558b3529b60 2023-11-16T16:59:42.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.473+0000 7f3d913bab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b347ad30 con 0x5558b352e8b0 2023-11-16T16:59:42.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.473+0000 7f3d7c9fd700 1 --2- 172.21.15.44:0/3158516788 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3d640a1340 0x7f3d640a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.473+0000 7f3d76ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b352e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.473+0000 7f3d76ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b352e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44332/0 (socket says 172.21.15.44:44332) 2023-11-16T16:59:42.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.473+0000 7f3d777fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b3529b60 0x5558b3529f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.473+0000 7f3d76ffd700 1 -- 172.21.15.44:0/2833474229 learned_addr learned my addr 172.21.15.44:0/2833474229 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:42.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.473+0000 7f3d6ffff700 1 --2- 172.21.15.44:0/2833474229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b352e8b0 0x5558b352ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.474+0000 7f3d777fe700 1 --2- 172.21.15.44:0/3158516788 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3d640a1340 0x7f3d640a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.474+0000 7f3d7c9fd700 1 -- 172.21.15.44:0/3158516788 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f3d68014070 con 0x7f3d78100cc0 2023-11-16T16:59:42.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.474+0000 7f3d76ffd700 1 -- 172.21.15.44:0/2833474229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b352e8b0 msgr2=0x5558b352ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:42.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.474+0000 7f3d76ffd700 1 --2- 172.21.15.44:0/2833474229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b352e8b0 0x5558b352ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.474+0000 7f3d76ffd700 1 -- 172.21.15.44:0/2833474229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b3529b60 msgr2=0x5558b3529f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:42.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.474+0000 7f3d76ffd700 1 --2- 172.21.15.44:0/2833474229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b3529b60 0x5558b3529f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.474+0000 7f3d76ffd700 1 -- 172.21.15.44:0/2833474229 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558b32fc630 con 0x5558b352a470 2023-11-16T16:59:42.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.474+0000 7f3d76ffd700 1 --2- 172.21.15.44:0/2833474229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b352e370 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f3d68096460 tx=0x7f3d6801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806abe9e3e6842f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:42.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.474+0000 7f3d777fe700 1 --2- 172.21.15.44:0/3158516788 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3d640a1340 0x7f3d640a37f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009ec0 tx=0x7f3d7000b040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:42.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.474+0000 7f3d6dffb700 1 -- 172.21.15.44:0/2833474229 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68099f60 con 0x5558b352a470 2023-11-16T16:59:42.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.474+0000 7f3d6dffb700 1 -- 172.21.15.44:0/2833474229 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d680c8e90 con 0x5558b352a470 2023-11-16T16:59:42.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.474+0000 7f3d6dffb700 1 -- 172.21.15.44:0/2833474229 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d680ca860 con 0x5558b352a470 2023-11-16T16:59:42.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.475+0000 7f3d913bab80 1 -- 172.21.15.44:0/2833474229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 msgr2=0x5558b352e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.475+0000 7f3d913bab80 1 --2- 172.21.15.44:0/2833474229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b352e370 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f3d68096460 tx=0x7f3d6801f000 comp rx=0 tx=0).stop 2023-11-16T16:59:42.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.475+0000 7f3d913bab80 1 -- 172.21.15.44:0/2833474229 shutdown_connections 2023-11-16T16:59:42.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.475+0000 7f3d913bab80 1 --2- 172.21.15.44:0/2833474229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b352e8b0 0x5558b352ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.475+0000 7f3d913bab80 1 --2- 172.21.15.44:0/2833474229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b3529b60 0x5558b3529f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.475+0000 7f3d913bab80 1 --2- 172.21.15.44:0/2833474229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b352e370 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.475+0000 7f3d913bab80 1 -- 172.21.15.44:0/2833474229 >> 172.21.15.44:0/2833474229 conn(0x5558b351f360 msgr2=0x5558b3524b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:42.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.475+0000 7f3d913bab80 1 -- 172.21.15.44:0/2833474229 shutdown_connections 2023-11-16T16:59:42.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.475+0000 7f3d913bab80 1 -- 172.21.15.44:0/2833474229 wait complete. 2023-11-16T16:59:42.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.476+0000 7f3d913bab80 1 -- start start 2023-11-16T16:59:42.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.476+0000 7f3d913bab80 1 Processor -- start 2023-11-16T16:59:42.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.476+0000 7f3d913bab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b3529b60 0x5558b3536020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.476+0000 7f3d913bab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b35332a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.476+0000 7f3d913bab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 0x5558b3533920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.477+0000 7f3d913bab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b341c8e0 con 0x5558b352a470 2023-11-16T16:59:42.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.477+0000 7f3d913bab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b341c580 con 0x5558b352e8b0 2023-11-16T16:59:42.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.477+0000 7f3d913bab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b32fe920 con 0x5558b3529b60 2023-11-16T16:59:42.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.477+0000 7f3d913bab80 10 client.0 _ll_register_callbacks cb 0x5558b343a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:59:42.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.477+0000 7f3d6ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b35332a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.477+0000 7f3d777fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 0x5558b3533920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.477+0000 7f3d6ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b35332a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44342/0 (socket says 172.21.15.44:44342) 2023-11-16T16:59:42.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.477+0000 7f3d6ffff700 1 -- 172.21.15.44:0/2483336885 learned_addr learned my addr 172.21.15.44:0/2483336885 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:42.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.477+0000 7f3d777fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 0x5558b3533920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:34548/0 (socket says 172.21.15.44:34548) 2023-11-16T16:59:42.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.477+0000 7f3d76ffd700 1 --2- 172.21.15.44:0/2483336885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b3529b60 0x5558b3536020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.477+0000 7f3d777fe700 1 -- 172.21.15.44:0/2483336885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b3529b60 msgr2=0x5558b3536020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:42.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.477+0000 7f3d777fe700 1 --2- 172.21.15.44:0/2483336885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b3529b60 0x5558b3536020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.477+0000 7f3d777fe700 1 -- 172.21.15.44:0/2483336885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 msgr2=0x5558b35332a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:42.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.477+0000 7f3d777fe700 1 --2- 172.21.15.44:0/2483336885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b35332a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.477+0000 7f3d777fe700 1 -- 172.21.15.44:0/2483336885 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d781101e0 con 0x5558b352e8b0 2023-11-16T16:59:42.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.477+0000 7f3d6ffff700 1 --2- 172.21.15.44:0/2483336885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b35332a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:42.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.478+0000 7f3d777fe700 1 --2- 172.21.15.44:0/2483336885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 0x5558b3533920 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3d70003f80 tx=0x7f3d70004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1269eb860623fb46 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:42.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.478+0000 7f3d56ffd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T16:59:42.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.478+0000 7f3d56ffd700 1 -- 172.21.15.44:0/2483336885 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d70022070 con 0x5558b352e8b0 2023-11-16T16:59:42.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.478+0000 7f3d913bab80 1 -- 172.21.15.44:0/2483336885 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5558b351cb20 con 0x5558b352e8b0 2023-11-16T16:59:42.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.478+0000 7f3d56ffd700 1 -- 172.21.15.44:0/2483336885 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f3d70008240 con 0x5558b352e8b0 2023-11-16T16:59:42.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.478+0000 7f3d56ffd700 1 -- 172.21.15.44:0/2483336885 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d7000e070 con 0x5558b352e8b0 2023-11-16T16:59:42.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.478+0000 7f3d913bab80 10 client.27215 fetch_fsmap learned FSMap version 21 2023-11-16T16:59:42.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.478+0000 7f3d913bab80 1 -- 172.21.15.44:0/2483336885 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5558b3441190 con 0x5558b352e8b0 2023-11-16T16:59:42.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.478+0000 7f3d56ffd700 1 -- 172.21.15.44:0/2483336885 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5558b351cb20 con 0x5558b352e8b0 2023-11-16T16:59:42.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.479+0000 7f3d56ffd700 1 -- 172.21.15.44:0/2483336885 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3d70007500 con 0x5558b352e8b0 2023-11-16T16:59:42.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.479+0000 7f3d913bab80 10 client.27215 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:59:42.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.479+0000 7f3d913bab80 10 client.27215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:59:42.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.479+0000 7f3d913bab80 20 client.27215 populate_metadata read hostname 'smithi044' 2023-11-16T16:59:42.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.479+0000 7f3d913bab80 1 --2- 172.21.15.44:0/2483336885 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5558b35438c0 0x5558b3545cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.479+0000 7f3d913bab80 4 client.27215 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:59:42.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.479+0000 7f3d913bab80 1 -- 172.21.15.44:0/2483336885 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5558b343d0a0 con 0x5558b35438c0 2023-11-16T16:59:42.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.479+0000 7f3d76ffd700 1 --2- 172.21.15.44:0/2483336885 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5558b35438c0 0x5558b3545cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.480+0000 7f3d76ffd700 1 --2- 172.21.15.44:0/2483336885 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5558b35438c0 0x5558b3545cd0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7a1af57517a45f0 server_cookie=3484726ff4d91d8b in_seq=0 out_seq=0 2023-11-16T16:59:42.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.480+0000 7f3d56ffd700 10 client.27215 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:42.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.480+0000 7f3d56ffd700 1 -- 172.21.15.44:0/2483336885 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5558b343d0a0 con 0x5558b35438c0 2023-11-16T16:59:42.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.480+0000 7f3d56ffd700 10 client.27215 handle_command_reply: tid=0 2023-11-16T16:59:42.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.480+0000 7f3d913bab80 1 client.27215 shutdown 2023-11-16T16:59:42.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.481+0000 7f3d913bab80 1 -- 172.21.15.44:0/2483336885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 msgr2=0x5558b3533920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.481+0000 7f3d913bab80 1 --2- 172.21.15.44:0/2483336885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 0x5558b3533920 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3d70003f80 tx=0x7f3d70004ad0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.481+0000 7f3d913bab80 1 -- 172.21.15.44:0/2483336885 shutdown_connections 2023-11-16T16:59:42.491 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.481+0000 7f3d913bab80 1 --2- 172.21.15.44:0/2483336885 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5558b35438c0 0x5558b3545cd0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.481+0000 7f3d913bab80 1 --2- 172.21.15.44:0/2483336885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 0x5558b3533920 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.481+0000 7f3d913bab80 1 --2- 172.21.15.44:0/2483336885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b3529b60 0x5558b3536020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.481+0000 7f3d913bab80 1 --2- 172.21.15.44:0/2483336885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b35332a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.481+0000 7f3d56ffd700 0 client.27215 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:42.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.481+0000 7f3d913bab80 1 -- 172.21.15.44:0/2483336885 >> 172.21.15.44:0/2483336885 conn(0x5558b351f360 msgr2=0x5558b352fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:42.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.481+0000 7f3d913bab80 1 -- 172.21.15.44:0/2483336885 shutdown_connections 2023-11-16T16:59:42.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.481+0000 7f3d913bab80 1 -- 172.21.15.44:0/2483336885 wait complete. 2023-11-16T16:59:42.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.481+0000 7f3d913bab80 20 client.27215 trim_cache size 0 max 16384 2023-11-16T16:59:42.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.498+0000 7f3d913bab80 1 Processor -- start 2023-11-16T16:59:42.500 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.498+0000 7f3d913bab80 1 -- start start 2023-11-16T16:59:42.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d913bab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b3529b60 0x5558b352c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d913bab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b352a470 0x5558b3528960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d913bab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 0x5558b3528ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d913bab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b352abc0 con 0x5558b3529b60 2023-11-16T16:59:42.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d913bab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b3533cb0 con 0x5558b352e8b0 2023-11-16T16:59:42.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d913bab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b3525780 con 0x5558b352a470 2023-11-16T16:59:42.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d76ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b3529b60 0x5558b352c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d76ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b3529b60 0x5558b352c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44346/0 (socket says 172.21.15.44:44346) 2023-11-16T16:59:42.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d76ffd700 1 -- 172.21.15.44:0/1271338756 learned_addr learned my addr 172.21.15.44:0/1271338756 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:42.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d6ffff700 1 --2- 172.21.15.44:0/1271338756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b352a470 0x5558b3528960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d777fe700 1 --2- 172.21.15.44:0/1271338756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 0x5558b3528ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d76ffd700 1 -- 172.21.15.44:0/1271338756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b352a470 msgr2=0x5558b3528960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:42.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d76ffd700 1 --2- 172.21.15.44:0/1271338756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b352a470 0x5558b3528960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d76ffd700 1 -- 172.21.15.44:0/1271338756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 msgr2=0x5558b3528ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:42.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d76ffd700 1 --2- 172.21.15.44:0/1271338756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 0x5558b3528ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d76ffd700 1 -- 172.21.15.44:0/1271338756 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558b3440ee0 con 0x5558b3529b60 2023-11-16T16:59:42.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d6ffff700 1 --2- 172.21.15.44:0/1271338756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b352a470 0x5558b3528960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:59:42.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.499+0000 7f3d76ffd700 1 --2- 172.21.15.44:0/1271338756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b3529b60 0x5558b352c930 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3d68099ef0 tx=0x7f3d680ca580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a40cd3d5fc1e7353 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:42.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.500+0000 7f3d6e7fc700 1 -- 172.21.15.44:0/1271338756 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68099fe0 con 0x5558b3529b60 2023-11-16T16:59:42.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.500+0000 7f3d6e7fc700 1 -- 172.21.15.44:0/1271338756 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d68010880 con 0x5558b3529b60 2023-11-16T16:59:42.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.500+0000 7f3d6e7fc700 1 -- 172.21.15.44:0/1271338756 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68099450 con 0x5558b3529b60 2023-11-16T16:59:42.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.500+0000 7f3d913bab80 1 -- 172.21.15.44:0/1271338756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b3529b60 msgr2=0x5558b352c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.500+0000 7f3d913bab80 1 --2- 172.21.15.44:0/1271338756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b3529b60 0x5558b352c930 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3d68099ef0 tx=0x7f3d680ca580 comp rx=0 tx=0).stop 2023-11-16T16:59:42.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.500+0000 7f3d913bab80 1 -- 172.21.15.44:0/1271338756 shutdown_connections 2023-11-16T16:59:42.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.500+0000 7f3d913bab80 1 --2- 172.21.15.44:0/1271338756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 0x5558b3528ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.500+0000 7f3d913bab80 1 --2- 172.21.15.44:0/1271338756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b3529b60 0x5558b352c930 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.500+0000 7f3d913bab80 1 --2- 172.21.15.44:0/1271338756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b352a470 0x5558b3528960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.500+0000 7f3d913bab80 1 -- 172.21.15.44:0/1271338756 >> 172.21.15.44:0/1271338756 conn(0x5558b351f360 msgr2=0x5558b353fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:42.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.500+0000 7f3d913bab80 1 -- 172.21.15.44:0/1271338756 shutdown_connections 2023-11-16T16:59:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.500+0000 7f3d913bab80 1 -- 172.21.15.44:0/1271338756 wait complete. 2023-11-16T16:59:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.501+0000 7f3d913bab80 1 -- start start 2023-11-16T16:59:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.501+0000 7f3d913bab80 1 Processor -- start 2023-11-16T16:59:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.501+0000 7f3d913bab80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b3529b60 0x5558b358b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.501+0000 7f3d913bab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b358bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.501+0000 7f3d913bab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 0x5558b358c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.501+0000 7f3d913bab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b35968c0 con 0x5558b352a470 2023-11-16T16:59:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.501+0000 7f3d913bab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b3596a00 con 0x5558b352e8b0 2023-11-16T16:59:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.501+0000 7f3d913bab80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b3596bc0 con 0x5558b3529b60 2023-11-16T16:59:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.502+0000 7f3d913bab80 10 client.0 _ll_register_callbacks cb 0x5558b343a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:59:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.502+0000 7f3d6ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b358bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.502+0000 7f3d6ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b358bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44352/0 (socket says 172.21.15.44:44352) 2023-11-16T16:59:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.502+0000 7f3d6ffff700 1 -- 172.21.15.44:0/3796618428 learned_addr learned my addr 172.21.15.44:0/3796618428 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.502+0000 7f3d777fe700 1 --2- 172.21.15.44:0/3796618428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 0x5558b358c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.502+0000 7f3d76ffd700 1 --2- 172.21.15.44:0/3796618428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b3529b60 0x5558b358b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.502+0000 7f3d6ffff700 1 -- 172.21.15.44:0/3796618428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b3529b60 msgr2=0x5558b358b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.502+0000 7f3d6ffff700 1 --2- 172.21.15.44:0/3796618428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b3529b60 0x5558b358b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.502+0000 7f3d6ffff700 1 -- 172.21.15.44:0/3796618428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 msgr2=0x5558b358c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:59:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.502+0000 7f3d6ffff700 1 --2- 172.21.15.44:0/3796618428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 0x5558b358c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.502+0000 7f3d6ffff700 1 -- 172.21.15.44:0/3796618428 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558b32fc630 con 0x5558b352a470 2023-11-16T16:59:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.502+0000 7f3d777fe700 1 --2- 172.21.15.44:0/3796618428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 0x5558b358c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.502+0000 7f3d6ffff700 1 --2- 172.21.15.44:0/3796618428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b358bc50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3d600058e0 tx=0x7f3d6000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36c37010d6d7d363 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:42.512 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.502+0000 7f3d56ffd700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2023-11-16T16:59:42.512 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.503+0000 7f3d56ffd700 1 -- 172.21.15.44:0/3796618428 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d60005450 con 0x5558b352a470 2023-11-16T16:59:42.512 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.503+0000 7f3d913bab80 1 -- 172.21.15.44:0/3796618428 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d680c8e90 con 0x5558b352a470 2023-11-16T16:59:42.512 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.503+0000 7f3d56ffd700 1 -- 172.21.15.44:0/3796618428 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f3d600055b0 con 0x5558b352a470 2023-11-16T16:59:42.512 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.503+0000 7f3d56ffd700 1 -- 172.21.15.44:0/3796618428 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d6000f460 con 0x5558b352a470 2023-11-16T16:59:42.512 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.503+0000 7f3d913bab80 10 client.17139 fetch_fsmap learned FSMap version 21 2023-11-16T16:59:42.512 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.503+0000 7f3d913bab80 1 -- 172.21.15.44:0/3796618428 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5558b3291c60 con 0x5558b352a470 2023-11-16T16:59:42.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.503+0000 7f3d56ffd700 1 -- 172.21.15.44:0/3796618428 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d680c8e90 con 0x5558b352a470 2023-11-16T16:59:42.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.503+0000 7f3d56ffd700 1 -- 172.21.15.44:0/3796618428 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3d60023050 con 0x5558b352a470 2023-11-16T16:59:42.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.503+0000 7f3d913bab80 10 client.17139 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T16:59:42.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.503+0000 7f3d913bab80 10 client.17139 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T16:59:42.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.503+0000 7f3d913bab80 20 client.17139 populate_metadata read hostname 'smithi044' 2023-11-16T16:59:42.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.503+0000 7f3d913bab80 1 --2- 172.21.15.44:0/3796618428 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5558b3598270 0x5558b359a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:42.513 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.503+0000 7f3d913bab80 4 client.17139 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T16:59:42.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.503+0000 7f3d913bab80 1 -- 172.21.15.44:0/3796618428 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5558b3449660 con 0x5558b3598270 2023-11-16T16:59:42.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.504+0000 7f3d76ffd700 1 --2- 172.21.15.44:0/3796618428 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5558b3598270 0x5558b359a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:42.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.504+0000 7f3d76ffd700 1 --2- 172.21.15.44:0/3796618428 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5558b3598270 0x5558b359a680 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=974bda778cbd445 server_cookie=e0aca725cb7439ef in_seq=0 out_seq=0 2023-11-16T16:59:42.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.504+0000 7f3d56ffd700 10 client.17139 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:42.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.504+0000 7f3d56ffd700 1 -- 172.21.15.44:0/3796618428 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5558b3449660 con 0x5558b3598270 2023-11-16T16:59:42.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.505+0000 7f3d56ffd700 10 client.17139 handle_command_reply: tid=0 2023-11-16T16:59:42.514 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.505+0000 7f3d913bab80 1 client.17139 shutdown 2023-11-16T16:59:42.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.505+0000 7f3d913bab80 1 -- 172.21.15.44:0/3796618428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 msgr2=0x5558b358bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.505+0000 7f3d913bab80 1 --2- 172.21.15.44:0/3796618428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b358bc50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3d600058e0 tx=0x7f3d6000be50 comp rx=0 tx=0).stop 2023-11-16T16:59:42.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.505+0000 7f3d913bab80 1 -- 172.21.15.44:0/3796618428 shutdown_connections 2023-11-16T16:59:42.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.505+0000 7f3d913bab80 1 --2- 172.21.15.44:0/3796618428 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x5558b3598270 0x5558b359a680 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.505+0000 7f3d913bab80 1 --2- 172.21.15.44:0/3796618428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558b352e8b0 0x5558b358c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.505+0000 7f3d913bab80 1 --2- 172.21.15.44:0/3796618428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5558b3529b60 0x5558b358b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.515 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.505+0000 7f3d913bab80 1 --2- 172.21.15.44:0/3796618428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5558b352a470 0x5558b358bc50 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.505+0000 7f3d913bab80 1 -- 172.21.15.44:0/3796618428 >> 172.21.15.44:0/3796618428 conn(0x5558b351f360 msgr2=0x5558b3528960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:42.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.505+0000 7f3d56ffd700 0 client.17139 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T16:59:42.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.505+0000 7f3d913bab80 1 -- 172.21.15.44:0/3796618428 shutdown_connections 2023-11-16T16:59:42.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.505+0000 7f3d913bab80 1 -- 172.21.15.44:0/3796618428 wait complete. 2023-11-16T16:59:42.516 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.505+0000 7f3d913bab80 20 client.17139 trim_cache size 0 max 16384 2023-11-16T16:59:42.516 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T16:59:42.517 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2023-11-16T16:59:42.517 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2023-11-16T16:59:42.517 INFO:teuthology.orchestra.run.smithi044.stdout: "c64f63b3-8976-493c-bda7-faf84f6f8650": { 2023-11-16T16:59:42.517 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2023-11-16T16:59:42.517 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "c64f63b3-8976-493c-bda7-faf84f6f8650", 2023-11-16T16:59:42.517 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2023-11-16T16:59:42.517 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:59:42.518 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T16:59:42.518 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T16:59:42.518 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.506+0000 7f3d7d9ff700 1 -- 172.21.15.44:0/3158516788 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3d640a1340 msgr2=0x7f3d640a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.506+0000 7f3d7d9ff700 1 --2- 172.21.15.44:0/3158516788 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3d640a1340 0x7f3d640a37f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009ec0 tx=0x7f3d7000b040 comp rx=0 tx=0).stop 2023-11-16T16:59:42.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.506+0000 7f3d7d9ff700 1 -- 172.21.15.44:0/3158516788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d78100cc0 msgr2=0x7f3d781986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:42.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.506+0000 7f3d7d9ff700 1 --2- 172.21.15.44:0/3158516788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d78100cc0 0x7f3d781986a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800e9a0 tx=0x7f3d6800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.506+0000 7f3d7d9ff700 1 -- 172.21.15.44:0/3158516788 shutdown_connections 2023-11-16T16:59:42.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.506+0000 7f3d7d9ff700 1 --2- 172.21.15.44:0/3158516788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d78105620 0x7f3d7819bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.506+0000 7f3d7d9ff700 1 --2- 172.21.15.44:0/3158516788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d78100cc0 0x7f3d781986a0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.506+0000 7f3d7d9ff700 1 --2- 172.21.15.44:0/3158516788 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3d640a1340 0x7f3d640a37f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.506+0000 7f3d7d9ff700 1 --2- 172.21.15.44:0/3158516788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d78101660 0x7f3d78198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:42.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.506+0000 7f3d7d9ff700 1 -- 172.21.15.44:0/3158516788 >> 172.21.15.44:0/3158516788 conn(0x7f3d78077560 msgr2=0x7f3d780fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:42.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.506+0000 7f3d7d9ff700 1 -- 172.21.15.44:0/3158516788 shutdown_connections 2023-11-16T16:59:42.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:42.506+0000 7f3d7d9ff700 1 -- 172.21.15.44:0/3158516788 wait complete. 2023-11-16T16:59:42.525 INFO:tasks.cephfs.filesystem:scrub status for tag:c64f63b3-8976-493c-bda7-faf84f6f8650 - {'path': '/', 'tag': 'c64f63b3-8976-493c-bda7-faf84f6f8650', 'options': 'recursive,force'} 2023-11-16T16:59:42.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:42 smithi044 ceph-mon[128750]: scrub queued for path: / 2023-11-16T16:59:42.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:42 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:59:42.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:42 smithi044 ceph-mon[128750]: scrub summary: active paths [/] 2023-11-16T16:59:42.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:42 smithi044 ceph-mon[128750]: scrub summary: idle+waiting paths [/] 2023-11-16T16:59:42.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:42 smithi176 ceph-mon[131630]: scrub queued for path: / 2023-11-16T16:59:42.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:42 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:59:42.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:42 smithi176 ceph-mon[131630]: scrub summary: active paths [/] 2023-11-16T16:59:42.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:42 smithi176 ceph-mon[131630]: scrub summary: idle+waiting paths [/] 2023-11-16T16:59:42.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:42 smithi077 ceph-mon[131204]: scrub queued for path: / 2023-11-16T16:59:42.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:42 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:59:42.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:42 smithi077 ceph-mon[131204]: scrub summary: active paths [/] 2023-11-16T16:59:42.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:42 smithi077 ceph-mon[131204]: scrub summary: idle+waiting paths [/] 2023-11-16T16:59:43.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:43 smithi044 ceph-mon[128750]: pgmap v304: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:59:43.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:43 smithi044 ceph-mon[128750]: scrub summary: idle 2023-11-16T16:59:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:43 smithi176 ceph-mon[131630]: pgmap v304: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:59:43.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:43 smithi176 ceph-mon[131630]: scrub summary: idle 2023-11-16T16:59:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:43 smithi077 ceph-mon[131204]: pgmap v304: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T16:59:43.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:43 smithi077 ceph-mon[131204]: scrub summary: idle 2023-11-16T16:59:45.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:45 smithi044 ceph-mon[128750]: pgmap v305: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2023-11-16T16:59:45.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:45 smithi176 ceph-mon[131630]: pgmap v305: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2023-11-16T16:59:45.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:45 smithi077 ceph-mon[131204]: pgmap v305: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2023-11-16T16:59:47.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:47 smithi044 ceph-mon[128750]: pgmap v306: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:59:47.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:47 smithi176 ceph-mon[131630]: pgmap v306: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:59:47.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:47 smithi077 ceph-mon[131204]: pgmap v306: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T16:59:49.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:49 smithi044 ceph-mon[128750]: pgmap v307: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3 op/s 2023-11-16T16:59:49.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:49 smithi176 ceph-mon[131630]: pgmap v307: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3 op/s 2023-11-16T16:59:49.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:49 smithi077 ceph-mon[131204]: pgmap v307: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3 op/s 2023-11-16T16:59:51.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:51 smithi044 ceph-mon[128750]: pgmap v308: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2 op/s 2023-11-16T16:59:51.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:51 smithi176 ceph-mon[131630]: pgmap v308: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2 op/s 2023-11-16T16:59:51.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:51 smithi077 ceph-mon[131204]: pgmap v308: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2 op/s 2023-11-16T16:59:53.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:53 smithi044 ceph-mon[128750]: pgmap v309: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2 op/s 2023-11-16T16:59:53.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:53 smithi176 ceph-mon[131630]: pgmap v309: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2 op/s 2023-11-16T16:59:53.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:53 smithi077 ceph-mon[131204]: pgmap v309: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2 op/s 2023-11-16T16:59:55.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:55 smithi044 ceph-mon[128750]: pgmap v310: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3 op/s 2023-11-16T16:59:55.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:55 smithi176 ceph-mon[131630]: pgmap v310: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3 op/s 2023-11-16T16:59:55.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:55 smithi077 ceph-mon[131204]: pgmap v310: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3 op/s 2023-11-16T16:59:57.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:57 smithi044 ceph-mon[128750]: pgmap v311: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2023-11-16T16:59:57.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:57 smithi176 ceph-mon[131630]: pgmap v311: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2023-11-16T16:59:57.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:57 smithi077 ceph-mon[131204]: pgmap v311: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2023-11-16T16:59:58.479 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2023-11-16T16:59:58.692 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.690+0000 7fd13b3ff700 1 -- 172.21.15.44:0/1429721037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134073780 msgr2=0x7fd13410c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:58.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.690+0000 7fd13b3ff700 1 --2- 172.21.15.44:0/1429721037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134073780 0x7fd13410c610 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd124009990 tx=0x7fd124009ca0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.690+0000 7fd13b3ff700 1 -- 172.21.15.44:0/1429721037 shutdown_connections 2023-11-16T16:59:58.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.690+0000 7fd13b3ff700 1 --2- 172.21.15.44:0/1429721037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd13410cb50 0x7fd13410ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.693 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.690+0000 7fd13b3ff700 1 --2- 172.21.15.44:0/1429721037 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd134072e70 0x7fd134073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.694 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.690+0000 7fd13b3ff700 1 --2- 172.21.15.44:0/1429721037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134073780 0x7fd13410c610 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.694 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.690+0000 7fd13b3ff700 1 -- 172.21.15.44:0/1429721037 >> 172.21.15.44:0/1429721037 conn(0x7fd1340781b0 msgr2=0x7fd1340785b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:58.694 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.690+0000 7fd13b3ff700 1 -- 172.21.15.44:0/1429721037 shutdown_connections 2023-11-16T16:59:58.694 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.690+0000 7fd13b3ff700 1 -- 172.21.15.44:0/1429721037 wait complete. 2023-11-16T16:59:58.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.691+0000 7fd13b3ff700 1 Processor -- start 2023-11-16T16:59:58.697 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.691+0000 7fd13b3ff700 1 -- start start 2023-11-16T16:59:58.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.692+0000 7fd13b3ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134072e70 0x7fd134198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:58.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.692+0000 7fd13b3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd134073780 0x7fd134198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:58.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.692+0000 7fd13b3ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd13410cb50 0x7fd13419ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:58.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.692+0000 7fd13b3ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13419d4f0 con 0x7fd134072e70 2023-11-16T16:59:58.698 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.692+0000 7fd13b3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13419d660 con 0x7fd134073780 2023-11-16T16:59:58.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.692+0000 7fd13b3ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13419d820 con 0x7fd13410cb50 2023-11-16T16:59:58.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.692+0000 7fd13919b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134072e70 0x7fd134198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:58.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.692+0000 7fd13919b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134072e70 0x7fd134198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60466/0 (socket says 172.21.15.44:60466) 2023-11-16T16:59:58.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.692+0000 7fd13919b700 1 -- 172.21.15.44:0/1489946610 learned_addr learned my addr 172.21.15.44:0/1489946610 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:58.699 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.692+0000 7fd13899a700 1 --2- 172.21.15.44:0/1489946610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd134073780 0x7fd134198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:58.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.692+0000 7fd13999c700 1 --2- 172.21.15.44:0/1489946610 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd13410cb50 0x7fd13419ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:58.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.692+0000 7fd13919b700 1 -- 172.21.15.44:0/1489946610 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd13410cb50 msgr2=0x7fd13419ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:58.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.692+0000 7fd13919b700 1 --2- 172.21.15.44:0/1489946610 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd13410cb50 0x7fd13419ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.692+0000 7fd13919b700 1 -- 172.21.15.44:0/1489946610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd134073780 msgr2=0x7fd134198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:58.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.692+0000 7fd13919b700 1 --2- 172.21.15.44:0/1489946610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd134073780 0x7fd134198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.700 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.692+0000 7fd13919b700 1 -- 172.21.15.44:0/1489946610 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1240095f0 con 0x7fd134072e70 2023-11-16T16:59:58.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.693+0000 7fd13919b700 1 --2- 172.21.15.44:0/1489946610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134072e70 0x7fd134198510 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd13000e9a0 tx=0x7fd13000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:58.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.693+0000 7fd12a7fc700 1 -- 172.21.15.44:0/1489946610 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130009790 con 0x7fd134072e70 2023-11-16T16:59:58.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.693+0000 7fd12a7fc700 1 -- 172.21.15.44:0/1489946610 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd13000cbd0 con 0x7fd134072e70 2023-11-16T16:59:58.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.693+0000 7fd13b3ff700 1 -- 172.21.15.44:0/1489946610 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd134112530 con 0x7fd134072e70 2023-11-16T16:59:58.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.693+0000 7fd12a7fc700 1 -- 172.21.15.44:0/1489946610 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130018680 con 0x7fd134072e70 2023-11-16T16:59:58.701 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.693+0000 7fd13b3ff700 1 -- 172.21.15.44:0/1489946610 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1341123a0 con 0x7fd134072e70 2023-11-16T16:59:58.702 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.694+0000 7fd13b3ff700 1 -- 172.21.15.44:0/1489946610 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1341126c0 con 0x7fd134072e70 2023-11-16T16:59:58.703 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.696+0000 7fd12a7fc700 1 -- 172.21.15.44:0/1489946610 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fd13000cd40 con 0x7fd134072e70 2023-11-16T16:59:58.703 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.697+0000 7fd12a7fc700 1 --2- 172.21.15.44:0/1489946610 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd1200a1340 0x7fd1200a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:58.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.698+0000 7fd12a7fc700 1 -- 172.21.15.44:0/1489946610 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fd130014070 con 0x7fd134072e70 2023-11-16T16:59:58.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.698+0000 7fd13899a700 1 --2- 172.21.15.44:0/1489946610 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd1200a1340 0x7fd1200a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:58.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.698+0000 7fd13899a700 1 --2- 172.21.15.44:0/1489946610 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd1200a1340 0x7fd1200a37f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd134199b20 tx=0x7fd124005fd0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:58.704 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.702+0000 7fd12a7fc700 1 -- 172.21.15.44:0/1489946610 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd130051730 con 0x7fd134072e70 2023-11-16T16:59:58.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.937+0000 7fd13b3ff700 1 -- 172.21.15.44:0/1489946610 --> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7fd13410dba0 con 0x7fd1200a1340 2023-11-16T16:59:58.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.946+0000 7fd12a7fc700 1 -- 172.21.15.44:0/1489946610 <== mgr.25447 v2:172.21.15.44:6800/3382499657 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+335 (secure 0 0 0) 0x7fd13410dba0 con 0x7fd1200a1340 2023-11-16T16:59:58.948 INFO:teuthology.orchestra.run.smithi044.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2023-11-16T00:00:00", "created": "2023-11-16T16:49:56", "first": "2023-11-16T16:50:00", "last": "2023-11-16T16:59:00", "last_pruned": "2023-11-16T16:59:00", "created_count": 10, "pruned_count": 4, "active": true} 2023-11-16T16:59:58.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.948+0000 7fd13b3ff700 1 -- 172.21.15.44:0/1489946610 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd1200a1340 msgr2=0x7fd1200a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:58.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.948+0000 7fd13b3ff700 1 --2- 172.21.15.44:0/1489946610 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd1200a1340 0x7fd1200a37f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd134199b20 tx=0x7fd124005fd0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.948+0000 7fd13b3ff700 1 -- 172.21.15.44:0/1489946610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134072e70 msgr2=0x7fd134198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:58.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.948+0000 7fd13b3ff700 1 --2- 172.21.15.44:0/1489946610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134072e70 0x7fd134198510 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd13000e9a0 tx=0x7fd13000ecb0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.949+0000 7fd13b3ff700 1 -- 172.21.15.44:0/1489946610 shutdown_connections 2023-11-16T16:59:58.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.949+0000 7fd13b3ff700 1 --2- 172.21.15.44:0/1489946610 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fd1200a1340 0x7fd1200a37f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.949+0000 7fd13b3ff700 1 --2- 172.21.15.44:0/1489946610 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd13410cb50 0x7fd13419ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.949+0000 7fd13b3ff700 1 --2- 172.21.15.44:0/1489946610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd134072e70 0x7fd134198510 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.949+0000 7fd13b3ff700 1 --2- 172.21.15.44:0/1489946610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd134073780 0x7fd134198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.949+0000 7fd13b3ff700 1 -- 172.21.15.44:0/1489946610 >> 172.21.15.44:0/1489946610 conn(0x7fd1340781b0 msgr2=0x7fd13410d490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:58.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.949+0000 7fd13b3ff700 1 -- 172.21.15.44:0/1489946610 shutdown_connections 2023-11-16T16:59:58.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:58.949+0000 7fd13b3ff700 1 -- 172.21.15.44:0/1489946610 wait complete. 2023-11-16T16:59:58.969 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2023-11-16T16:59:59.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.169+0000 7f8a53fff700 1 -- 172.21.15.44:0/1777594943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a54100c80 msgr2=0x7f8a54104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.169+0000 7f8a53fff700 1 --2- 172.21.15.44:0/1777594943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a54100c80 0x7f8a54104c40 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8a480099c0 tx=0x7f8a48009cd0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.171 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.169+0000 7f8a53fff700 1 -- 172.21.15.44:0/1777594943 shutdown_connections 2023-11-16T16:59:59.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.169+0000 7f8a53fff700 1 --2- 172.21.15.44:0/1777594943 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a54105320 0x7f8a5410cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.169+0000 7f8a53fff700 1 --2- 172.21.15.44:0/1777594943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a541002e0 0x7f8a541006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.169+0000 7f8a53fff700 1 --2- 172.21.15.44:0/1777594943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a54100c80 0x7f8a54104c40 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.169+0000 7f8a53fff700 1 -- 172.21.15.44:0/1777594943 >> 172.21.15.44:0/1777594943 conn(0x7f8a540fbb70 msgr2=0x7f8a540fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:59.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.170+0000 7f8a53fff700 1 -- 172.21.15.44:0/1777594943 shutdown_connections 2023-11-16T16:59:59.172 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.170+0000 7f8a53fff700 1 -- 172.21.15.44:0/1777594943 wait complete. 2023-11-16T16:59:59.174 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.170+0000 7f8a53fff700 1 Processor -- start 2023-11-16T16:59:59.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.171+0000 7f8a53fff700 1 -- start start 2023-11-16T16:59:59.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.171+0000 7f8a53fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a541002e0 0x7f8a541961f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:59.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.171+0000 7f8a53fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a54100c80 0x7f8a54196730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:59.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.171+0000 7f8a53fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a54105320 0x7f8a5419ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:59.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.171+0000 7f8a53fff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5419b180 con 0x7f8a54100c80 2023-11-16T16:59:59.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.171+0000 7f8a53fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5419b2f0 con 0x7f8a54105320 2023-11-16T16:59:59.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.171+0000 7f8a53fff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5419b4b0 con 0x7f8a541002e0 2023-11-16T16:59:59.175 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.171+0000 7f8a52ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a541002e0 0x7f8a541961f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:59.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.171+0000 7f8a527fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a54100c80 0x7f8a54196730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:59.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.171+0000 7f8a52ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a541002e0 0x7f8a541961f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:52650/0 (socket says 172.21.15.44:52650) 2023-11-16T16:59:59.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.171+0000 7f8a52ffd700 1 -- 172.21.15.44:0/2371132890 learned_addr learned my addr 172.21.15.44:0/2371132890 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T16:59:59.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.171+0000 7f8a537fe700 1 --2- 172.21.15.44:0/2371132890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a54105320 0x7f8a5419ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:59.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.172+0000 7f8a52ffd700 1 -- 172.21.15.44:0/2371132890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a54105320 msgr2=0x7f8a5419ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.172+0000 7f8a52ffd700 1 --2- 172.21.15.44:0/2371132890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a54105320 0x7f8a5419ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.172+0000 7f8a52ffd700 1 -- 172.21.15.44:0/2371132890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a54100c80 msgr2=0x7f8a54196730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.172+0000 7f8a52ffd700 1 --2- 172.21.15.44:0/2371132890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a54100c80 0x7f8a54196730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.176 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.172+0000 7f8a52ffd700 1 -- 172.21.15.44:0/2371132890 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a480095f0 con 0x7f8a541002e0 2023-11-16T16:59:59.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.172+0000 7f8a527fc700 1 --2- 172.21.15.44:0/2371132890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a54100c80 0x7f8a54196730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:59.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.172+0000 7f8a537fe700 1 --2- 172.21.15.44:0/2371132890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a54105320 0x7f8a5419ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:59:59.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.172+0000 7f8a52ffd700 1 --2- 172.21.15.44:0/2371132890 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a541002e0 0x7f8a541961f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400e9a0 tx=0x7f8a4400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:59.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.172+0000 7f8a3bfff700 1 -- 172.21.15.44:0/2371132890 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a44009790 con 0x7f8a541002e0 2023-11-16T16:59:59.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.172+0000 7f8a53fff700 1 -- 172.21.15.44:0/2371132890 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a541100a0 con 0x7f8a541002e0 2023-11-16T16:59:59.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.172+0000 7f8a3bfff700 1 -- 172.21.15.44:0/2371132890 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a44010430 con 0x7f8a541002e0 2023-11-16T16:59:59.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.173+0000 7f8a3bfff700 1 -- 172.21.15.44:0/2371132890 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a44018680 con 0x7f8a541002e0 2023-11-16T16:59:59.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.173+0000 7f8a53fff700 1 -- 172.21.15.44:0/2371132890 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5410ff10 con 0x7f8a541002e0 2023-11-16T16:59:59.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.174+0000 7f8a53fff700 1 -- 172.21.15.44:0/2371132890 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a54110230 con 0x7f8a541002e0 2023-11-16T16:59:59.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.176+0000 7f8a3bfff700 1 -- 172.21.15.44:0/2371132890 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f8a440105a0 con 0x7f8a541002e0 2023-11-16T16:59:59.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.177+0000 7f8a3bfff700 1 --2- 172.21.15.44:0/2371132890 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f8a3c0a12c0 0x7f8a3c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:59.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.177+0000 7f8a3bfff700 1 -- 172.21.15.44:0/2371132890 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f8a44014070 con 0x7f8a541002e0 2023-11-16T16:59:59.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.177+0000 7f8a527fc700 1 --2- 172.21.15.44:0/2371132890 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f8a3c0a12c0 0x7f8a3c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:59.180 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.178+0000 7f8a527fc700 1 --2- 172.21.15.44:0/2371132890 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f8a3c0a12c0 0x7f8a3c0a3770 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8a480058e0 tx=0x7f8a48005fb0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:59.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.179+0000 7f8a3bfff700 1 -- 172.21.15.44:0/2371132890 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8a440964e0 con 0x7f8a541002e0 2023-11-16T16:59:59.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.417+0000 7f8a53fff700 1 -- 172.21.15.44:0/2371132890 --> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f8a540fcd20 con 0x7f8a3c0a12c0 2023-11-16T16:59:59.427 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.426+0000 7f8a3bfff700 1 -- 172.21.15.44:0/2371132890 <== mgr.25447 v2:172.21.15.44:6800/3382499657 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f8a540fcd20 con 0x7f8a3c0a12c0 2023-11-16T16:59:59.428 INFO:teuthology.orchestra.run.smithi044.stdout:/ 1M 6M3h 2023-11-16T16:59:59.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.428+0000 7f8a53fff700 1 -- 172.21.15.44:0/2371132890 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f8a3c0a12c0 msgr2=0x7f8a3c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.428+0000 7f8a53fff700 1 --2- 172.21.15.44:0/2371132890 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f8a3c0a12c0 0x7f8a3c0a3770 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8a480058e0 tx=0x7f8a48005fb0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.428+0000 7f8a53fff700 1 -- 172.21.15.44:0/2371132890 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a541002e0 msgr2=0x7f8a541961f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.428+0000 7f8a53fff700 1 --2- 172.21.15.44:0/2371132890 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a541002e0 0x7f8a541961f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400e9a0 tx=0x7f8a4400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.428+0000 7f8a53fff700 1 -- 172.21.15.44:0/2371132890 shutdown_connections 2023-11-16T16:59:59.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.428+0000 7f8a53fff700 1 --2- 172.21.15.44:0/2371132890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a54105320 0x7f8a5419ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.428+0000 7f8a53fff700 1 --2- 172.21.15.44:0/2371132890 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f8a3c0a12c0 0x7f8a3c0a3770 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.428+0000 7f8a53fff700 1 --2- 172.21.15.44:0/2371132890 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a541002e0 0x7f8a541961f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.428+0000 7f8a53fff700 1 --2- 172.21.15.44:0/2371132890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a54100c80 0x7f8a54196730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.428+0000 7f8a53fff700 1 -- 172.21.15.44:0/2371132890 >> 172.21.15.44:0/2371132890 conn(0x7f8a540fbb70 msgr2=0x7f8a540fc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:59.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.428+0000 7f8a53fff700 1 -- 172.21.15.44:0/2371132890 shutdown_connections 2023-11-16T16:59:59.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T16:59:59.428+0000 7f8a53fff700 1 -- 172.21.15.44:0/2371132890 wait complete. 2023-11-16T16:59:59.448 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2023-11-16T16:59:59.459 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2023-11-16T16:59:59.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 16:59:59 smithi044 ceph-mon[128750]: pgmap v312: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2023-11-16T16:59:59.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 16:59:59 smithi176 ceph-mon[131630]: pgmap v312: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2023-11-16T16:59:59.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 16:59:59 smithi077 ceph-mon[131204]: pgmap v312: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2023-11-16T17:00:00.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:00 smithi176 ceph-mon[131630]: from='client.17145 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T17:00:00.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:00 smithi176 ceph-mon[131630]: from='client.27169 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T17:00:00.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:00 smithi176 ceph-mon[131630]: overall HEALTH_OK 2023-11-16T17:00:00.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:00 smithi077 ceph-mon[131204]: from='client.17145 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T17:00:00.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:00 smithi077 ceph-mon[131204]: from='client.27169 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T17:00:00.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:00 smithi077 ceph-mon[131204]: overall HEALTH_OK 2023-11-16T17:00:01.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:00 smithi044 ceph-mon[128750]: from='client.17145 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T17:00:01.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:00 smithi044 ceph-mon[128750]: from='client.27169 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T17:00:01.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:00 smithi044 ceph-mon[128750]: overall HEALTH_OK 2023-11-16T17:00:01.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:01 smithi176 ceph-mon[131630]: pgmap v313: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:00:01.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:01 smithi077 ceph-mon[131204]: pgmap v313: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:00:02.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:01 smithi044 ceph-mon[128750]: pgmap v313: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:00:03.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:03 smithi176 ceph-mon[131630]: pgmap v314: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:00:03.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:03 smithi077 ceph-mon[131204]: pgmap v314: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:00:04.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:03 smithi044 ceph-mon[128750]: pgmap v314: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:00:05.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:05 smithi176 ceph-mon[131630]: pgmap v315: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T17:00:05.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:05 smithi077 ceph-mon[131204]: pgmap v315: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T17:00:06.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:05 smithi044 ceph-mon[128750]: pgmap v315: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T17:00:07.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:07 smithi176 ceph-mon[131630]: pgmap v316: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 18 op/s 2023-11-16T17:00:07.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:07 smithi077 ceph-mon[131204]: pgmap v316: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 18 op/s 2023-11-16T17:00:08.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:07 smithi044 ceph-mon[128750]: pgmap v316: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 18 op/s 2023-11-16T17:00:08.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:08 smithi077 ceph-mon[131204]: osdmap e101: 12 total, 12 up, 12 in 2023-11-16T17:00:09.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:08 smithi044 ceph-mon[128750]: osdmap e101: 12 total, 12 up, 12 in 2023-11-16T17:00:09.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:08 smithi176 ceph-mon[131630]: osdmap e101: 12 total, 12 up, 12 in 2023-11-16T17:00:09.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:09 smithi077 ceph-mon[131204]: pgmap v318: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2023-11-16T17:00:10.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:09 smithi044 ceph-mon[128750]: pgmap v318: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2023-11-16T17:00:10.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:09 smithi176 ceph-mon[131630]: pgmap v318: 129 pgs: 129 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2023-11-16T17:00:11.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:11 smithi077 ceph-mon[131204]: pgmap v319: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2023-11-16T17:00:12.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:11 smithi044 ceph-mon[128750]: pgmap v319: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2023-11-16T17:00:12.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:11 smithi176 ceph-mon[131630]: pgmap v319: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2023-11-16T17:00:12.526 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-11-16T17:00:12.910 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.908+0000 7f41e11ff700 1 -- 172.21.15.44:0/2980261880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41dc101390 msgr2=0x7f41dc101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:12.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.908+0000 7f41e11ff700 1 --2- 172.21.15.44:0/2980261880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41dc101390 0x7f41dc101760 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f41d4009990 tx=0x7f41d4009ca0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.911 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.909+0000 7f41e11ff700 1 -- 172.21.15.44:0/2980261880 shutdown_connections 2023-11-16T17:00:12.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.909+0000 7f41e11ff700 1 --2- 172.21.15.44:0/2980261880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41dc10a8b0 0x7f41dc10ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.909+0000 7f41e11ff700 1 --2- 172.21.15.44:0/2980261880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41dc101390 0x7f41dc101760 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.909+0000 7f41e11ff700 1 --2- 172.21.15.44:0/2980261880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41dc101ca0 0x7f41dc10a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.909+0000 7f41e11ff700 1 -- 172.21.15.44:0/2980261880 >> 172.21.15.44:0/2980261880 conn(0x7f41dc0770b0 msgr2=0x7f41dc0774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:12.912 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.909+0000 7f41e11ff700 1 -- 172.21.15.44:0/2980261880 shutdown_connections 2023-11-16T17:00:12.913 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.909+0000 7f41e11ff700 1 -- 172.21.15.44:0/2980261880 wait complete. 2023-11-16T17:00:12.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.911+0000 7f41e11ff700 1 Processor -- start 2023-11-16T17:00:12.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.911+0000 7f41e11ff700 1 -- start start 2023-11-16T17:00:12.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.912+0000 7f41e11ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41dc101390 0x7f41dc196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:12.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.912+0000 7f41e11ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41dc101ca0 0x7f41dc196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:12.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.912+0000 7f41e11ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41dc10a8b0 0x7f41dc19add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:12.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.912+0000 7f41e11ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc19b410 con 0x7f41dc101390 2023-11-16T17:00:12.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.912+0000 7f41e11ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc19b580 con 0x7f41dc101ca0 2023-11-16T17:00:12.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.912+0000 7f41e11ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc19b740 con 0x7f41dc10a8b0 2023-11-16T17:00:12.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.912+0000 7f41db59e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41dc10a8b0 0x7f41dc19add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:12.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.912+0000 7f41db59e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41dc10a8b0 0x7f41dc19add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48078/0 (socket says 172.21.15.44:48078) 2023-11-16T17:00:12.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.912+0000 7f41db59e700 1 -- 172.21.15.44:0/1613440784 learned_addr learned my addr 172.21.15.44:0/1613440784 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:12.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.913+0000 7f41db59e700 1 -- 172.21.15.44:0/1613440784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41dc101ca0 msgr2=0x7f41dc196970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:12.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.913+0000 7f41db59e700 1 --2- 172.21.15.44:0/1613440784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41dc101ca0 0x7f41dc196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.913+0000 7f41db59e700 1 -- 172.21.15.44:0/1613440784 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41dc101390 msgr2=0x7f41dc196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:12.917 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.913+0000 7f41db59e700 1 --2- 172.21.15.44:0/1613440784 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41dc101390 0x7f41dc196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.913+0000 7f41db59e700 1 -- 172.21.15.44:0/1613440784 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d40095f0 con 0x7f41dc10a8b0 2023-11-16T17:00:12.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.914+0000 7f41db59e700 1 --2- 172.21.15.44:0/1613440784 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41dc10a8b0 0x7f41dc19add0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00c8d0 tx=0x7f41cc00cc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:12.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.915+0000 7f41d3fff700 1 -- 172.21.15.44:0/1613440784 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc004460 con 0x7f41dc10a8b0 2023-11-16T17:00:12.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.915+0000 7f41e11ff700 1 -- 172.21.15.44:0/1613440784 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41dc110400 con 0x7f41dc10a8b0 2023-11-16T17:00:12.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.915+0000 7f41d3fff700 1 -- 172.21.15.44:0/1613440784 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41cc007bb0 con 0x7f41dc10a8b0 2023-11-16T17:00:12.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.915+0000 7f41d3fff700 1 -- 172.21.15.44:0/1613440784 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc003d40 con 0x7f41dc10a8b0 2023-11-16T17:00:12.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.915+0000 7f41e11ff700 1 -- 172.21.15.44:0/1613440784 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41dc110270 con 0x7f41dc10a8b0 2023-11-16T17:00:12.924 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.923+0000 7f41d3fff700 1 -- 172.21.15.44:0/1613440784 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f41cc007d20 con 0x7f41dc10a8b0 2023-11-16T17:00:12.928 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.927+0000 7f41f4b33b80 1 Processor -- start 2023-11-16T17:00:12.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.929+0000 7f41f4b33b80 1 -- start start 2023-11-16T17:00:12.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.929+0000 7f41d3fff700 1 --2- 172.21.15.44:0/1613440784 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f41c80a13c0 0x7f41c80a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:12.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.930+0000 7f41d3fff700 1 -- 172.21.15.44:0/1613440784 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f41cc0cd0a0 con 0x7f41dc10a8b0 2023-11-16T17:00:12.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.930+0000 7f41f4b33b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b080b60 0x55a07b080f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:12.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.930+0000 7f41f4b33b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b081470 0x55a07b085370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:12.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.930+0000 7f41f4b33b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b0858b0 0x55a07b085d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:12.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.930+0000 7f41f4b33b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07af86060 con 0x55a07b081470 2023-11-16T17:00:12.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.930+0000 7f41f4b33b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07afd88a0 con 0x55a07b0858b0 2023-11-16T17:00:12.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.930+0000 7f41f4b33b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07afd1d30 con 0x55a07b080b60 2023-11-16T17:00:12.934 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.932+0000 7f41db59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b0858b0 0x55a07b085d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:12.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.932+0000 7f41db59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b0858b0 0x55a07b085d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56692/0 (socket says 172.21.15.44:56692) 2023-11-16T17:00:12.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.932+0000 7f41db59e700 1 -- 172.21.15.44:0/2815750296 learned_addr learned my addr 172.21.15.44:0/2815750296 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:12.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.932+0000 7f41db59e700 1 -- 172.21.15.44:0/2815750296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b080b60 msgr2=0x55a07b080f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T17:00:12.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.932+0000 7f41db59e700 1 --2- 172.21.15.44:0/2815750296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b080b60 0x55a07b080f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.932+0000 7f41db59e700 1 -- 172.21.15.44:0/2815750296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b081470 msgr2=0x55a07b085370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T17:00:12.935 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.932+0000 7f41db59e700 1 --2- 172.21.15.44:0/2815750296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b081470 0x55a07b085370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.932+0000 7f41db59e700 1 -- 172.21.15.44:0/2815750296 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a07ae53630 con 0x55a07b0858b0 2023-11-16T17:00:12.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.933+0000 7f41da59c700 1 --2- 172.21.15.44:0/1613440784 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f41c80a13c0 0x7f41c80a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:12.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.933+0000 7f41db59e700 1 --2- 172.21.15.44:0/2815750296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b0858b0 0x55a07b085d00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00cfd0 tx=0x7f41cc099be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22815376271828cd server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:12.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.934+0000 7f41d17fa700 1 -- 172.21.15.44:0/2815750296 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc09dd10 con 0x55a07b0858b0 2023-11-16T17:00:12.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.934+0000 7f41d17fa700 1 -- 172.21.15.44:0/2815750296 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41cc09c030 con 0x55a07b0858b0 2023-11-16T17:00:12.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.934+0000 7f41d17fa700 1 -- 172.21.15.44:0/2815750296 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc099f40 con 0x55a07b0858b0 2023-11-16T17:00:12.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.936+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2815750296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b0858b0 msgr2=0x55a07b085d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:12.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.936+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2815750296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b0858b0 0x55a07b085d00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00cfd0 tx=0x7f41cc099be0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.936+0000 7f41da59c700 1 --2- 172.21.15.44:0/1613440784 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f41c80a13c0 0x7f41c80a3870 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f41dc197a40 tx=0x7f41c40091f0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:12.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.936+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2815750296 shutdown_connections 2023-11-16T17:00:12.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.936+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2815750296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b0858b0 0x55a07b085d00 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.936+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2815750296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b080b60 0x55a07b080f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.936+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2815750296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b081470 0x55a07b085370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.936+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2815750296 >> 172.21.15.44:0/2815750296 conn(0x55a07b076360 msgr2=0x55a07b07bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:12.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.937+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2815750296 shutdown_connections 2023-11-16T17:00:12.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.937+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2815750296 wait complete. 2023-11-16T17:00:12.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.938+0000 7f41f4b33b80 1 -- start start 2023-11-16T17:00:12.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.940+0000 7f41f4b33b80 1 Processor -- start 2023-11-16T17:00:12.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.941+0000 7f41f4b33b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b080b60 0x55a07b08d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:12.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.941+0000 7f41f4b33b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b081470 0x55a07b08a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:12.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.941+0000 7f41f4b33b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b0858b0 0x55a07b08a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:12.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.941+0000 7f41f4b33b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07af738e0 con 0x55a07b081470 2023-11-16T17:00:12.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.941+0000 7f41f4b33b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07af73580 con 0x55a07b0858b0 2023-11-16T17:00:12.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.941+0000 7f41f4b33b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07ae55920 con 0x55a07b080b60 2023-11-16T17:00:12.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.941+0000 7f41f4b33b80 10 client.0 _ll_register_callbacks cb 0x55a07af91780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T17:00:12.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.943+0000 7f41db59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b0858b0 0x55a07b08a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:12.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.943+0000 7f41db59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b0858b0 0x55a07b08a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56704/0 (socket says 172.21.15.44:56704) 2023-11-16T17:00:12.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.943+0000 7f41db59e700 1 -- 172.21.15.44:0/2878922871 learned_addr learned my addr 172.21.15.44:0/2878922871 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:12.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.943+0000 7f41da59c700 1 --2- 172.21.15.44:0/2878922871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b081470 0x55a07b08a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:12.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.943+0000 7f41dad9d700 1 --2- 172.21.15.44:0/2878922871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b080b60 0x55a07b08d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:12.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.944+0000 7f41db59e700 1 -- 172.21.15.44:0/2878922871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b080b60 msgr2=0x55a07b08d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T17:00:12.946 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.944+0000 7f41db59e700 1 --2- 172.21.15.44:0/2878922871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b080b60 0x55a07b08d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.944+0000 7f41db59e700 1 -- 172.21.15.44:0/2878922871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b081470 msgr2=0x55a07b08a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T17:00:12.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.944+0000 7f41db59e700 1 --2- 172.21.15.44:0/2878922871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b081470 0x55a07b08a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.944+0000 7f41db59e700 1 -- 172.21.15.44:0/2878922871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41dc110270 con 0x55a07b0858b0 2023-11-16T17:00:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.945+0000 7f41c27fc700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T17:00:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.946+0000 7f41db59e700 1 --2- 172.21.15.44:0/2878922871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b0858b0 0x55a07b08a9a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0074a0 tx=0x7f41cc098290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9674ae3f32e742bb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.946+0000 7f41c27fc700 1 -- 172.21.15.44:0/2878922871 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0993d0 con 0x55a07b0858b0 2023-11-16T17:00:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.946+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2878922871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a07b073b20 con 0x55a07b0858b0 2023-11-16T17:00:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.946+0000 7f41c27fc700 1 -- 172.21.15.44:0/2878922871 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f41cc0b2470 con 0x55a07b0858b0 2023-11-16T17:00:12.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.947+0000 7f41c27fc700 1 -- 172.21.15.44:0/2878922871 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41cc0993d0 con 0x55a07b0858b0 2023-11-16T17:00:12.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.948+0000 7f41f4b33b80 10 client.27239 fetch_fsmap learned FSMap version 21 2023-11-16T17:00:12.951 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.948+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2878922871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55a07af98190 con 0x55a07b0858b0 2023-11-16T17:00:12.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.957+0000 7f41c27fc700 1 -- 172.21.15.44:0/2878922871 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a07b073b20 con 0x55a07b0858b0 2023-11-16T17:00:12.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.957+0000 7f41c27fc700 1 -- 172.21.15.44:0/2878922871 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f41cc07f050 con 0x55a07b0858b0 2023-11-16T17:00:12.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.957+0000 7f41f4b33b80 10 client.27239 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T17:00:12.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.957+0000 7f41f4b33b80 10 client.27239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T17:00:12.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.957+0000 7f41f4b33b80 20 client.27239 populate_metadata read hostname 'smithi044' 2023-11-16T17:00:12.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.957+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2878922871 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55a07b09a260 0x55a07b09c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:12.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.957+0000 7f41f4b33b80 4 client.27239 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T17:00:12.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.957+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2878922871 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a07af940a0 con 0x55a07b09a260 2023-11-16T17:00:12.961 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.959+0000 7f41dad9d700 1 --2- 172.21.15.44:0/2878922871 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55a07b09a260 0x55a07b09c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:12.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.960+0000 7f41dad9d700 1 --2- 172.21.15.44:0/2878922871 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55a07b09a260 0x55a07b09c640 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c013e356b3185004 server_cookie=fc2b7b08a48b0067 in_seq=0 out_seq=0 2023-11-16T17:00:12.962 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.960+0000 7f41c27fc700 10 client.27239 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T17:00:12.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.964+0000 7f41c27fc700 1 -- 172.21.15.44:0/2878922871 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a07af940a0 con 0x55a07b09a260 2023-11-16T17:00:12.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.964+0000 7f41c27fc700 10 client.27239 handle_command_reply: tid=0 2023-11-16T17:00:12.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.965+0000 7f41f4b33b80 1 client.27239 shutdown 2023-11-16T17:00:12.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.965+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2878922871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b0858b0 msgr2=0x55a07b08a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.965+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2878922871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b0858b0 0x55a07b08a9a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0074a0 tx=0x7f41cc098290 comp rx=0 tx=0).stop 2023-11-16T17:00:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.966+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2878922871 shutdown_connections 2023-11-16T17:00:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.966+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2878922871 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55a07b09a260 0x55a07b09c640 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.966+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2878922871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b0858b0 0x55a07b08a9a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.966+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2878922871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b080b60 0x55a07b08d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.966+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2878922871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b081470 0x55a07b08a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.966+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2878922871 >> 172.21.15.44:0/2878922871 conn(0x55a07b076360 msgr2=0x55a07b086d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.966+0000 7f41c27fc700 0 client.27239 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T17:00:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.967+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2878922871 shutdown_connections 2023-11-16T17:00:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.967+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2878922871 wait complete. 2023-11-16T17:00:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:12.967+0000 7f41f4b33b80 20 client.27239 trim_cache size 0 max 16384 2023-11-16T17:00:13.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.014+0000 7f41f4b33b80 1 Processor -- start 2023-11-16T17:00:13.016 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.015+0000 7f41f4b33b80 1 -- start start 2023-11-16T17:00:13.017 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.015+0000 7f41f4b33b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b080b60 0x55a07b07bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:13.017 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.015+0000 7f41f4b33b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b081470 0x55a07b091880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:13.017 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.015+0000 7f41f4b33b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b0858b0 0x55a07b096aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:13.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.015+0000 7f41f4b33b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07b07c7d0 con 0x55a07b081470 2023-11-16T17:00:13.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.015+0000 7f41f4b33b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07b084390 con 0x55a07b080b60 2023-11-16T17:00:13.018 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.015+0000 7f41f4b33b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07b073980 con 0x55a07b0858b0 2023-11-16T17:00:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.016+0000 7f41db59e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b0858b0 0x55a07b096aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.016+0000 7f41db59e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b0858b0 0x55a07b096aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48096/0 (socket says 172.21.15.44:48096) 2023-11-16T17:00:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.016+0000 7f41db59e700 1 -- 172.21.15.44:0/2539453642 learned_addr learned my addr 172.21.15.44:0/2539453642 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.017+0000 7f41dad9d700 1 --2- 172.21.15.44:0/2539453642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b080b60 0x55a07b07bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.017+0000 7f41db59e700 1 -- 172.21.15.44:0/2539453642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b080b60 msgr2=0x55a07b07bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T17:00:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.017+0000 7f41db59e700 1 --2- 172.21.15.44:0/2539453642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b080b60 0x55a07b07bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.017+0000 7f41db59e700 1 -- 172.21.15.44:0/2539453642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b081470 msgr2=0x55a07b091880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T17:00:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.017+0000 7f41db59e700 1 --2- 172.21.15.44:0/2539453642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b081470 0x55a07b091880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.017+0000 7f41db59e700 1 -- 172.21.15.44:0/2539453642 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a07b089dd0 con 0x55a07b0858b0 2023-11-16T17:00:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.017+0000 7f41dad9d700 1 --2- 172.21.15.44:0/2539453642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b080b60 0x55a07b07bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T17:00:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.017+0000 7f41db59e700 1 --2- 172.21.15.44:0/2539453642 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b0858b0 0x55a07b096aa0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0961e0 tx=0x7f41cc099be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3210c7178ebb49f2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:13.022 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.017+0000 7f41d1ffb700 1 -- 172.21.15.44:0/2539453642 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0a1ca0 con 0x55a07b0858b0 2023-11-16T17:00:13.022 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.017+0000 7f41d1ffb700 1 -- 172.21.15.44:0/2539453642 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41cc0ca120 con 0x55a07b0858b0 2023-11-16T17:00:13.022 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.018+0000 7f41d1ffb700 1 -- 172.21.15.44:0/2539453642 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc099f40 con 0x55a07b0858b0 2023-11-16T17:00:13.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.018+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2539453642 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b0858b0 msgr2=0x55a07b096aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:13.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.018+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2539453642 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b0858b0 0x55a07b096aa0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0961e0 tx=0x7f41cc099be0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.024 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.019+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2539453642 shutdown_connections 2023-11-16T17:00:13.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.019+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2539453642 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b0858b0 0x55a07b096aa0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.019+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2539453642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b080b60 0x55a07b07bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.019+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2539453642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b081470 0x55a07b091880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.019+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2539453642 >> 172.21.15.44:0/2539453642 conn(0x55a07b076360 msgr2=0x55a07b0a0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:13.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.019+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2539453642 shutdown_connections 2023-11-16T17:00:13.025 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.019+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2539453642 wait complete. 2023-11-16T17:00:13.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.020+0000 7f41f4b33b80 1 -- start start 2023-11-16T17:00:13.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.020+0000 7f41f4b33b80 1 Processor -- start 2023-11-16T17:00:13.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.021+0000 7f41f4b33b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b080b60 0x55a07b08f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:13.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.021+0000 7f41f4b33b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b081470 0x55a07b08fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:13.026 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.022+0000 7f41f4b33b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b0858b0 0x55a07b0ed0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:13.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.022+0000 7f41f4b33b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07b0906d0 con 0x55a07b080b60 2023-11-16T17:00:13.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.022+0000 7f41f4b33b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07b0ed600 con 0x55a07b081470 2023-11-16T17:00:13.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.022+0000 7f41f4b33b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07b0ed740 con 0x55a07b0858b0 2023-11-16T17:00:13.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.022+0000 7f41db59e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b0858b0 0x55a07b0ed0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:13.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.022+0000 7f41db59e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b0858b0 0x55a07b0ed0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48104/0 (socket says 172.21.15.44:48104) 2023-11-16T17:00:13.027 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.022+0000 7f41f4b33b80 10 client.0 _ll_register_callbacks cb 0x7f41cc099ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T17:00:13.028 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.022+0000 7f41db59e700 1 -- 172.21.15.44:0/2257224412 learned_addr learned my addr 172.21.15.44:0/2257224412 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:13.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.023+0000 7f41db59e700 1 -- 172.21.15.44:0/2257224412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b081470 msgr2=0x55a07b08fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T17:00:13.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.023+0000 7f41db59e700 1 --2- 172.21.15.44:0/2257224412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b081470 0x55a07b08fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.023+0000 7f41dad9d700 1 --2- 172.21.15.44:0/2257224412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b080b60 0x55a07b08f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:13.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.023+0000 7f41db59e700 1 -- 172.21.15.44:0/2257224412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b080b60 msgr2=0x55a07b08f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T17:00:13.030 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.023+0000 7f41db59e700 1 --2- 172.21.15.44:0/2257224412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b080b60 0x55a07b08f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.023+0000 7f41db59e700 1 -- 172.21.15.44:0/2257224412 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a07af98190 con 0x55a07b0858b0 2023-11-16T17:00:13.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.023+0000 7f41dad9d700 1 --2- 172.21.15.44:0/2257224412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b080b60 0x55a07b08f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T17:00:13.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.024+0000 7f41db59e700 1 --2- 172.21.15.44:0/2257224412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b0858b0 0x55a07b0ed0c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00cfd0 tx=0x7f41cc0a2230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a2fc208e21b0ae4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:13.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.024+0000 7f41c27fc700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T17:00:13.031 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.024+0000 7f41c27fc700 1 -- 172.21.15.44:0/2257224412 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0a1ca0 con 0x55a07b0858b0 2023-11-16T17:00:13.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.024+0000 7f41c27fc700 1 -- 172.21.15.44:0/2257224412 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f41cc0b2470 con 0x55a07b0858b0 2023-11-16T17:00:13.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.024+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2257224412 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f41cc09c030 con 0x55a07b0858b0 2023-11-16T17:00:13.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.024+0000 7f41c27fc700 1 -- 172.21.15.44:0/2257224412 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41cc09dd10 con 0x55a07b0858b0 2023-11-16T17:00:13.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.024+0000 7f41f4b33b80 10 client.27184 fetch_fsmap learned FSMap version 21 2023-11-16T17:00:13.032 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.024+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2257224412 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55a07adffde0 con 0x55a07b0858b0 2023-11-16T17:00:13.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.025+0000 7f41c27fc700 1 -- 172.21.15.44:0/2257224412 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41cc09c030 con 0x55a07b0858b0 2023-11-16T17:00:13.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.025+0000 7f41c27fc700 1 -- 172.21.15.44:0/2257224412 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f41cc07f070 con 0x55a07b0858b0 2023-11-16T17:00:13.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.025+0000 7f41f4b33b80 10 client.27184 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T17:00:13.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.025+0000 7f41f4b33b80 10 client.27184 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T17:00:13.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.025+0000 7f41f4b33b80 20 client.27184 populate_metadata read hostname 'smithi044' 2023-11-16T17:00:13.033 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.025+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2257224412 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55a07b0ee860 0x55a07b0f0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:13.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.025+0000 7f41f4b33b80 4 client.27184 mds_command: new command op to 24551 tid=0[{"prefix": "scrub status"}] 2023-11-16T17:00:13.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.025+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2257224412 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a07afa0660 con 0x55a07b0ee860 2023-11-16T17:00:13.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.025+0000 7f41dad9d700 1 --2- 172.21.15.44:0/2257224412 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55a07b0ee860 0x55a07b0f0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:13.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.026+0000 7f41dad9d700 1 --2- 172.21.15.44:0/2257224412 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55a07b0ee860 0x55a07b0f0c40 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a7888bb94d037f9 server_cookie=9d710441d7db6e1e in_seq=0 out_seq=0 2023-11-16T17:00:13.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.026+0000 7f41c27fc700 10 client.27184 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T17:00:13.034 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.026+0000 7f41c27fc700 1 -- 172.21.15.44:0/2257224412 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a07afa0660 con 0x55a07b0ee860 2023-11-16T17:00:13.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.026+0000 7f41c27fc700 10 client.27184 handle_command_reply: tid=0 2023-11-16T17:00:13.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.026+0000 7f41f4b33b80 1 client.27184 shutdown 2023-11-16T17:00:13.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.027+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2257224412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b0858b0 msgr2=0x55a07b0ed0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:13.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.027+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2257224412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b0858b0 0x55a07b0ed0c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00cfd0 tx=0x7f41cc0a2230 comp rx=0 tx=0).stop 2023-11-16T17:00:13.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.027+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2257224412 shutdown_connections 2023-11-16T17:00:13.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.027+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2257224412 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x55a07b0ee860 0x55a07b0f0c40 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.035 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.027+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2257224412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a07b0858b0 0x55a07b0ed0c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.027+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2257224412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a07b080b60 0x55a07b08f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.027+0000 7f41c27fc700 0 client.27184 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T17:00:13.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.027+0000 7f41f4b33b80 1 --2- 172.21.15.44:0/2257224412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a07b081470 0x55a07b08fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.027+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2257224412 >> 172.21.15.44:0/2257224412 conn(0x55a07b076360 msgr2=0x55a07b096aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:13.036 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.028+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2257224412 shutdown_connections 2023-11-16T17:00:13.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.028+0000 7f41f4b33b80 1 -- 172.21.15.44:0/2257224412 wait complete. 2023-11-16T17:00:13.037 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.029+0000 7f41f4b33b80 20 client.27184 trim_cache size 0 max 16384 2023-11-16T17:00:13.038 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T17:00:13.038 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2023-11-16T17:00:13.038 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2023-11-16T17:00:13.038 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T17:00:13.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.030+0000 7f41e11ff700 1 -- 172.21.15.44:0/1613440784 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f41c80a13c0 msgr2=0x7f41c80a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:13.039 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.030+0000 7f41e11ff700 1 --2- 172.21.15.44:0/1613440784 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f41c80a13c0 0x7f41c80a3870 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f41dc197a40 tx=0x7f41c40091f0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.031+0000 7f41e11ff700 1 -- 172.21.15.44:0/1613440784 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41dc10a8b0 msgr2=0x7f41dc19add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:13.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.031+0000 7f41e11ff700 1 --2- 172.21.15.44:0/1613440784 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41dc10a8b0 0x7f41dc19add0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00c8d0 tx=0x7f41cc00cc90 comp rx=0 tx=0).stop 2023-11-16T17:00:13.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.031+0000 7f41e11ff700 1 -- 172.21.15.44:0/1613440784 shutdown_connections 2023-11-16T17:00:13.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.031+0000 7f41e11ff700 1 --2- 172.21.15.44:0/1613440784 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f41c80a13c0 0x7f41c80a3870 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.040 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.031+0000 7f41e11ff700 1 --2- 172.21.15.44:0/1613440784 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41dc10a8b0 0x7f41dc19add0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.031+0000 7f41e11ff700 1 --2- 172.21.15.44:0/1613440784 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f41dc101390 0x7f41dc196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.031+0000 7f41e11ff700 1 --2- 172.21.15.44:0/1613440784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41dc101ca0 0x7f41dc196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.031+0000 7f41e11ff700 1 -- 172.21.15.44:0/1613440784 >> 172.21.15.44:0/1613440784 conn(0x7f41dc0770b0 msgr2=0x7f41dc0fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:13.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.031+0000 7f41e11ff700 1 -- 172.21.15.44:0/1613440784 shutdown_connections 2023-11-16T17:00:13.041 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.031+0000 7f41e11ff700 1 -- 172.21.15.44:0/1613440784 wait complete. 2023-11-16T17:00:13.066 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-16T17:00:13.067 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T17:00:13.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.444+0000 7f04d95ff700 1 -- 172.21.15.44:0/184368360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04d4100550 msgr2=0x7f04d4100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:13.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.444+0000 7f04d95ff700 1 --2- 172.21.15.44:0/184368360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04d4100550 0x7f04d4100920 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009920 tx=0x7f04cc009c30 comp rx=0 tx=0).stop 2023-11-16T17:00:13.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.444+0000 7f04d95ff700 1 -- 172.21.15.44:0/184368360 shutdown_connections 2023-11-16T17:00:13.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.444+0000 7f04d95ff700 1 --2- 172.21.15.44:0/184368360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04d4105540 0x7f04d410d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.444+0000 7f04d95ff700 1 --2- 172.21.15.44:0/184368360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04d4100550 0x7f04d4100920 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.444+0000 7f04d95ff700 1 --2- 172.21.15.44:0/184368360 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04d4100ef0 0x7f04d4104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.444+0000 7f04d95ff700 1 -- 172.21.15.44:0/184368360 >> 172.21.15.44:0/184368360 conn(0x7f04d4077020 msgr2=0x7f04d4077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:13.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.444+0000 7f04d95ff700 1 -- 172.21.15.44:0/184368360 shutdown_connections 2023-11-16T17:00:13.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.444+0000 7f04d95ff700 1 -- 172.21.15.44:0/184368360 wait complete. 2023-11-16T17:00:13.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.446+0000 7f04d95ff700 1 Processor -- start 2023-11-16T17:00:13.449 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.446+0000 7f04d95ff700 1 -- start start 2023-11-16T17:00:13.451 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.448+0000 7f04d95ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04d4100550 0x7f04d410bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:13.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.448+0000 7f04d95ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04d4100ef0 0x7f04d410c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:13.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.448+0000 7f04d95ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04d4105540 0x7f04d4199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:13.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.448+0000 7f04d95ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d419a4f0 con 0x7f04d4105540 2023-11-16T17:00:13.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.448+0000 7f04d95ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d419a660 con 0x7f04d4100550 2023-11-16T17:00:13.452 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.448+0000 7f04d95ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d419a820 con 0x7f04d4100ef0 2023-11-16T17:00:13.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.448+0000 7f04d2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04d4100550 0x7f04d410bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:13.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.448+0000 7f04d2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04d4100550 0x7f04d410bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56742/0 (socket says 172.21.15.44:56742) 2023-11-16T17:00:13.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.448+0000 7f04d2ffd700 1 -- 172.21.15.44:0/2208702292 learned_addr learned my addr 172.21.15.44:0/2208702292 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:13.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.448+0000 7f04d37fe700 1 --2- 172.21.15.44:0/2208702292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04d4105540 0x7f04d4199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:13.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.448+0000 7f04d2ffd700 1 -- 172.21.15.44:0/2208702292 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04d4100ef0 msgr2=0x7f04d410c520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T17:00:13.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.449+0000 7f04d2ffd700 1 --2- 172.21.15.44:0/2208702292 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04d4100ef0 0x7f04d410c520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.453 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.449+0000 7f04d2ffd700 1 -- 172.21.15.44:0/2208702292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04d4105540 msgr2=0x7f04d4199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:13.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.449+0000 7f04d2ffd700 1 --2- 172.21.15.44:0/2208702292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04d4105540 0x7f04d4199ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.449+0000 7f04d2ffd700 1 -- 172.21.15.44:0/2208702292 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04cc009520 con 0x7f04d4100550 2023-11-16T17:00:13.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.449+0000 7f04d2ffd700 1 --2- 172.21.15.44:0/2208702292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04d4100550 0x7f04d410bfe0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009840 tx=0x7f04cc00bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:13.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.449+0000 7f04cbfff700 1 -- 172.21.15.44:0/2208702292 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc01c070 con 0x7f04d4100550 2023-11-16T17:00:13.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.449+0000 7f04cbfff700 1 -- 172.21.15.44:0/2208702292 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f04cc005040 con 0x7f04d4100550 2023-11-16T17:00:13.454 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.449+0000 7f04cbfff700 1 -- 172.21.15.44:0/2208702292 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc003940 con 0x7f04d4100550 2023-11-16T17:00:13.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.449+0000 7f04d95ff700 1 -- 172.21.15.44:0/2208702292 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d4110350 con 0x7f04d4100550 2023-11-16T17:00:13.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.450+0000 7f04d95ff700 1 -- 172.21.15.44:0/2208702292 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d41101c0 con 0x7f04d4100550 2023-11-16T17:00:13.455 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.452+0000 7f04d95ff700 1 -- 172.21.15.44:0/2208702292 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d41104e0 con 0x7f04d4100550 2023-11-16T17:00:13.456 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.455+0000 7f04cbfff700 1 -- 172.21.15.44:0/2208702292 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f04cc020030 con 0x7f04d4100550 2023-11-16T17:00:13.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.457+0000 7f04cbfff700 1 --2- 172.21.15.44:0/2208702292 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f04c00a1340 0x7f04c00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:13.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.458+0000 7f04cbfff700 1 -- 172.21.15.44:0/2208702292 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f04cc0ce900 con 0x7f04d4100550 2023-11-16T17:00:13.460 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.458+0000 7f04d27fc700 1 --2- 172.21.15.44:0/2208702292 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f04c00a1340 0x7f04c00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:13.462 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.461+0000 7f04cbfff700 1 -- 172.21.15.44:0/2208702292 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f04cc097970 con 0x7f04d4100550 2023-11-16T17:00:13.463 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.462+0000 7f04d27fc700 1 --2- 172.21.15.44:0/2208702292 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f04c00a1340 0x7f04c00a37f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f04d4195b00 tx=0x7f04c400b040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:13.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.967+0000 7f04d95ff700 1 -- 172.21.15.44:0/2208702292 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04d4067a60 con 0x7f04d4100550 2023-11-16T17:00:13.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.968+0000 7f04cbfff700 1 -- 172.21.15.44:0/2208702292 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f04cc09af90 con 0x7f04d4100550 2023-11-16T17:00:13.972 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T17:00:13.972 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T17:00:13.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.973+0000 7f04d95ff700 1 -- 172.21.15.44:0/2208702292 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f04c00a1340 msgr2=0x7f04c00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:13.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.973+0000 7f04d95ff700 1 --2- 172.21.15.44:0/2208702292 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f04c00a1340 0x7f04c00a37f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f04d4195b00 tx=0x7f04c400b040 comp rx=0 tx=0).stop 2023-11-16T17:00:13.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.973+0000 7f04d95ff700 1 -- 172.21.15.44:0/2208702292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04d4100550 msgr2=0x7f04d410bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.973+0000 7f04d95ff700 1 --2- 172.21.15.44:0/2208702292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04d4100550 0x7f04d410bfe0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009840 tx=0x7f04cc00bf60 comp rx=0 tx=0).stop 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.973+0000 7f04d95ff700 1 -- 172.21.15.44:0/2208702292 shutdown_connections 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.973+0000 7f04d95ff700 1 --2- 172.21.15.44:0/2208702292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04d4105540 0x7f04d4199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.973+0000 7f04d95ff700 1 --2- 172.21.15.44:0/2208702292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04d4100550 0x7f04d410bfe0 secure :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009840 tx=0x7f04cc00bf60 comp rx=0 tx=0).stop 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.973+0000 7f04d95ff700 1 --2- 172.21.15.44:0/2208702292 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f04c00a1340 0x7f04c00a37f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.973+0000 7f04d95ff700 1 --2- 172.21.15.44:0/2208702292 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04d4100ef0 0x7f04d410c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.973+0000 7f04d95ff700 1 -- 172.21.15.44:0/2208702292 >> 172.21.15.44:0/2208702292 conn(0x7f04d4077020 msgr2=0x7f04d40fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.974+0000 7f04d95ff700 1 -- 172.21.15.44:0/2208702292 shutdown_connections 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:13.974+0000 7f04d95ff700 1 -- 172.21.15.44:0/2208702292 wait complete. 2023-11-16T17:00:13.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:13 smithi077 ceph-mon[131204]: pgmap v320: 129 pgs: 6 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 107 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2023-11-16T17:00:13.981 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T17:00:14.012 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-11-16T17:00:14.013 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:13 smithi044 ceph-mon[128750]: pgmap v320: 129 pgs: 6 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 107 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2023-11-16T17:00:14.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:13 smithi176 ceph-mon[131630]: pgmap v320: 129 pgs: 6 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 107 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2023-11-16T17:00:14.399 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.397+0000 7f2a9b1ff700 1 -- 172.21.15.44:0/31594909 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a94107af0 msgr2=0x7f2a94107ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:14.399 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.397+0000 7f2a9b1ff700 1 --2- 172.21.15.44:0/31594909 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a94107af0 0x7f2a94107ec0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00b0a0 tx=0x7f2a8c00b3b0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.398+0000 7f2a9b1ff700 1 -- 172.21.15.44:0/31594909 shutdown_connections 2023-11-16T17:00:14.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.398+0000 7f2a9b1ff700 1 --2- 172.21.15.44:0/31594909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a941133b0 0x7f2a94115790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.398+0000 7f2a9b1ff700 1 --2- 172.21.15.44:0/31594909 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a94107af0 0x7f2a94107ec0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.398+0000 7f2a9b1ff700 1 --2- 172.21.15.44:0/31594909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a94108400 0x7f2a94112e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.398+0000 7f2a9b1ff700 1 -- 172.21.15.44:0/31594909 >> 172.21.15.44:0/31594909 conn(0x7f2a9406c710 msgr2=0x7f2a940716d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:14.403 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.402+0000 7f2a9b1ff700 1 -- 172.21.15.44:0/31594909 shutdown_connections 2023-11-16T17:00:14.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.402+0000 7f2a9b1ff700 1 -- 172.21.15.44:0/31594909 wait complete. 2023-11-16T17:00:14.405 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.403+0000 7f2a9b1ff700 1 Processor -- start 2023-11-16T17:00:14.405 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.403+0000 7f2a9b1ff700 1 -- start start 2023-11-16T17:00:14.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.404+0000 7f2a9b1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a94107af0 0x7f2a94108cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.406 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.404+0000 7f2a9b1ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a94108400 0x7f2a94109210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.407 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.404+0000 7f2a9b1ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a941133b0 0x7f2a941ace90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.407 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.404+0000 7f2a9b1ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a94109a40 con 0x7f2a941133b0 2023-11-16T17:00:14.407 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.404+0000 7f2a9b1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a941ad3d0 con 0x7f2a94107af0 2023-11-16T17:00:14.407 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.404+0000 7f2a9b1ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a941ad510 con 0x7f2a94108400 2023-11-16T17:00:14.407 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.404+0000 7f2a98f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a94107af0 0x7f2a94108cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:14.407 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.404+0000 7f2a98f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a94107af0 0x7f2a94108cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56750/0 (socket says 172.21.15.44:56750) 2023-11-16T17:00:14.408 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.404+0000 7f2a98f9b700 1 -- 172.21.15.44:0/3375411722 learned_addr learned my addr 172.21.15.44:0/3375411722 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:14.408 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.405+0000 7f2a9979c700 1 --2- 172.21.15.44:0/3375411722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a941133b0 0x7f2a941ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:14.408 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.405+0000 7f2a93fff700 1 --2- 172.21.15.44:0/3375411722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a94108400 0x7f2a94109210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:14.409 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.405+0000 7f2a9979c700 1 -- 172.21.15.44:0/3375411722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a94108400 msgr2=0x7f2a94109210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:14.409 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.405+0000 7f2a9979c700 1 --2- 172.21.15.44:0/3375411722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a94108400 0x7f2a94109210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.405+0000 7f2a9979c700 1 -- 172.21.15.44:0/3375411722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a94107af0 msgr2=0x7f2a94108cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:14.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.405+0000 7f2a9979c700 1 --2- 172.21.15.44:0/3375411722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a94107af0 0x7f2a94108cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.405+0000 7f2a9979c700 1 -- 172.21.15.44:0/3375411722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c009d00 con 0x7f2a941133b0 2023-11-16T17:00:14.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.406+0000 7f2a9979c700 1 --2- 172.21.15.44:0/3375411722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a941133b0 0x7f2a941ace90 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2a9406a2e0 tx=0x7f2a8400da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:14.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.407+0000 7f2a91ffb700 1 -- 172.21.15.44:0/3375411722 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a84004220 con 0x7f2a941133b0 2023-11-16T17:00:14.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.407+0000 7f2a91ffb700 1 -- 172.21.15.44:0/3375411722 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a84004380 con 0x7f2a941133b0 2023-11-16T17:00:14.410 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.407+0000 7f2a91ffb700 1 -- 172.21.15.44:0/3375411722 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a84004220 con 0x7f2a941133b0 2023-11-16T17:00:14.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.408+0000 7f2a9b1ff700 1 -- 172.21.15.44:0/3375411722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a94118b20 con 0x7f2a941133b0 2023-11-16T17:00:14.411 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.408+0000 7f2a9b1ff700 1 -- 172.21.15.44:0/3375411722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a94118990 con 0x7f2a941133b0 2023-11-16T17:00:14.416 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.415+0000 7f2aaea90b80 1 Processor -- start 2023-11-16T17:00:14.417 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.415+0000 7f2aaea90b80 1 -- start start 2023-11-16T17:00:14.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.416+0000 7f2aaea90b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e1fb60 0x561e30e1ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.416+0000 7f2aaea90b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e20470 0x561e30e24370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.416+0000 7f2aaea90b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e248b0 0x561e30e24d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.416+0000 7f2aaea90b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e30d25060 con 0x561e30e20470 2023-11-16T17:00:14.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.416+0000 7f2aaea90b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e30d778a0 con 0x561e30e248b0 2023-11-16T17:00:14.418 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.416+0000 7f2aaea90b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e30d70d30 con 0x561e30e1fb60 2023-11-16T17:00:14.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.416+0000 7f2a93fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e1fb60 0x561e30e1ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:14.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.416+0000 7f2a93fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e1fb60 0x561e30e1ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48146/0 (socket says 172.21.15.44:48146) 2023-11-16T17:00:14.419 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.416+0000 7f2a93fff700 1 -- 172.21.15.44:0/1525003625 learned_addr learned my addr 172.21.15.44:0/1525003625 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:14.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.416+0000 7f2a93fff700 1 --2- 172.21.15.44:0/1525003625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e248b0 0x561e30e24d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:14.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.417+0000 7f2a93fff700 1 -- 172.21.15.44:0/1525003625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e248b0 msgr2=0x561e30e24d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T17:00:14.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.417+0000 7f2a93fff700 1 --2- 172.21.15.44:0/1525003625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e248b0 0x561e30e24d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.417+0000 7f2a93fff700 1 -- 172.21.15.44:0/1525003625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e20470 msgr2=0x561e30e24370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T17:00:14.420 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.417+0000 7f2a93fff700 1 --2- 172.21.15.44:0/1525003625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e20470 0x561e30e24370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.417+0000 7f2a93fff700 1 -- 172.21.15.44:0/1525003625 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e30bf2630 con 0x561e30e1fb60 2023-11-16T17:00:14.421 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.417+0000 7f2a93fff700 1 --2- 172.21.15.44:0/1525003625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e1fb60 0x561e30e1ff30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c014970 tx=0x7f2a7c00c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f67d556f62ff47c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.418+0000 7f2a8affd700 1 -- 172.21.15.44:0/1525003625 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c014310 con 0x561e30e1fb60 2023-11-16T17:00:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.418+0000 7f2a8affd700 1 -- 172.21.15.44:0/1525003625 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a7c01a030 con 0x561e30e1fb60 2023-11-16T17:00:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.418+0000 7f2a8affd700 1 -- 172.21.15.44:0/1525003625 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c0055c0 con 0x561e30e1fb60 2023-11-16T17:00:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.419+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1525003625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e1fb60 msgr2=0x561e30e1ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.419+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1525003625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e1fb60 0x561e30e1ff30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c014970 tx=0x7f2a7c00c470 comp rx=0 tx=0).stop 2023-11-16T17:00:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.420+0000 7f2a91ffb700 1 -- 172.21.15.44:0/3375411722 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f2a8400f690 con 0x7f2a941133b0 2023-11-16T17:00:14.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.422+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1525003625 shutdown_connections 2023-11-16T17:00:14.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.422+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1525003625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e248b0 0x561e30e24d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.424 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.422+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1525003625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e1fb60 0x561e30e1ff30 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.422+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1525003625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e20470 0x561e30e24370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.422+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1525003625 >> 172.21.15.44:0/1525003625 conn(0x561e30e15360 msgr2=0x561e30e1ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:14.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.422+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1525003625 shutdown_connections 2023-11-16T17:00:14.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.422+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1525003625 wait complete. 2023-11-16T17:00:14.425 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.423+0000 7f2aaea90b80 1 -- start start 2023-11-16T17:00:14.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.423+0000 7f2aaea90b80 1 Processor -- start 2023-11-16T17:00:14.426 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.425+0000 7f2a91ffb700 1 --2- 172.21.15.44:0/3375411722 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2a800a12e0 0x7f2a800a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.428 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.427+0000 7f2a93fff700 1 --2- 172.21.15.44:0/3375411722 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2a800a12e0 0x7f2a800a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:14.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.425+0000 7f2a91ffb700 1 -- 172.21.15.44:0/3375411722 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f2a840cc110 con 0x7f2a941133b0 2023-11-16T17:00:14.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.427+0000 7f2aaea90b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e1fb60 0x561e30e31c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.429 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.428+0000 7f2aaea90b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e20470 0x561e30e32180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.428+0000 7f2aaea90b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e248b0 0x561e30e326c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.428+0000 7f2a98f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e1fb60 0x561e30e31c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:14.430 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.428+0000 7f2a98f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e1fb60 0x561e30e31c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:56770/0 (socket says 172.21.15.44:56770) 2023-11-16T17:00:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.428+0000 7f2a98f9b700 1 -- 172.21.15.44:0/1419730017 learned_addr learned my addr 172.21.15.44:0/1419730017 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.428+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1419730017 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e30d128e0 con 0x561e30e248b0 2023-11-16T17:00:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.428+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1419730017 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e30d12580 con 0x561e30e1fb60 2023-11-16T17:00:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.428+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1419730017 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e30bf4920 con 0x561e30e20470 2023-11-16T17:00:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.428+0000 7f2a93fff700 1 --2- 172.21.15.44:0/3375411722 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2a800a12e0 0x7f2a800a3790 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c005140 tx=0x7f2a7c005f60 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.428+0000 7f2a93fff700 1 --2- 172.21.15.44:0/1419730017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e20470 0x561e30e32180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.429+0000 7f2aaea90b80 10 client.0 _ll_register_callbacks cb 0x561e30d30780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T17:00:14.432 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.429+0000 7f2a98f9b700 1 -- 172.21.15.44:0/1419730017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e20470 msgr2=0x561e30e32180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T17:00:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.429+0000 7f2a98f9b700 1 --2- 172.21.15.44:0/1419730017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e20470 0x561e30e32180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.429+0000 7f2a98f9b700 1 -- 172.21.15.44:0/1419730017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e248b0 msgr2=0x561e30e326c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T17:00:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.429+0000 7f2a98f9b700 1 --2- 172.21.15.44:0/1419730017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e248b0 0x561e30e326c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.429+0000 7f2a98f9b700 1 -- 172.21.15.44:0/1419730017 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7c014620 con 0x561e30e1fb60 2023-11-16T17:00:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.429+0000 7f2a93fff700 1 --2- 172.21.15.44:0/1419730017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e20470 0x561e30e32180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T17:00:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.430+0000 7f2a98f9b700 1 --2- 172.21.15.44:0/1419730017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e1fb60 0x561e30e31c40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c004c40 tx=0x7f2a8c004d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23e23ec22584122f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:14.433 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.430+0000 7f2a73fff700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-11-16T17:00:14.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.432+0000 7f2a73fff700 1 -- 172.21.15.44:0/1419730017 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c00e040 con 0x561e30e1fb60 2023-11-16T17:00:14.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.432+0000 7f2a73fff700 1 -- 172.21.15.44:0/1419730017 <== mon.2 v2:172.21.15.77:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f2a8c0076f0 con 0x561e30e1fb60 2023-11-16T17:00:14.434 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.433+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1419730017 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561e30e12b20 con 0x561e30e1fb60 2023-11-16T17:00:14.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.433+0000 7f2a73fff700 1 -- 172.21.15.44:0/1419730017 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a8c0078f0 con 0x561e30e1fb60 2023-11-16T17:00:14.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.433+0000 7f2aaea90b80 10 client.27257 fetch_fsmap learned FSMap version 21 2023-11-16T17:00:14.435 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.433+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1419730017 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x561e30d37190 con 0x561e30e1fb60 2023-11-16T17:00:14.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.434+0000 7f2a73fff700 1 -- 172.21.15.44:0/1419730017 <== mon.2 v2:172.21.15.77:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561e30e12b20 con 0x561e30e1fb60 2023-11-16T17:00:14.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.434+0000 7f2a73fff700 1 -- 172.21.15.44:0/1419730017 <== mon.2 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2a8c019050 con 0x561e30e1fb60 2023-11-16T17:00:14.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.434+0000 7f2aaea90b80 10 client.27257 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T17:00:14.436 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.434+0000 7f2aaea90b80 10 client.27257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T17:00:14.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.434+0000 7f2aaea90b80 20 client.27257 populate_metadata read hostname 'smithi044' 2023-11-16T17:00:14.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.434+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1419730017 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561e30e42250 0x561e30e44660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.434+0000 7f2aaea90b80 4 client.27257 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T17:00:14.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.434+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1419730017 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561e30d330a0 con 0x561e30e42250 2023-11-16T17:00:14.437 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.434+0000 7f2a93fff700 1 --2- 172.21.15.44:0/1419730017 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561e30e42250 0x561e30e44660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:14.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.436+0000 7f2a93fff700 1 --2- 172.21.15.44:0/1419730017 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561e30e42250 0x561e30e44660 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=742bd3a901fc6942 server_cookie=78508049f8ecb70c in_seq=0 out_seq=0 2023-11-16T17:00:14.438 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.436+0000 7f2a73fff700 10 client.27257 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T17:00:14.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.438+0000 7f2a73fff700 1 -- 172.21.15.44:0/1419730017 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x561e30d330a0 con 0x561e30e42250 2023-11-16T17:00:14.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.438+0000 7f2a73fff700 10 client.27257 handle_command_reply: tid=0 2023-11-16T17:00:14.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.439+0000 7f2aaea90b80 1 client.27257 shutdown 2023-11-16T17:00:14.441 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.439+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1419730017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e1fb60 msgr2=0x561e30e31c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:14.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.439+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1419730017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e1fb60 0x561e30e31c40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c004c40 tx=0x7f2a8c004d20 comp rx=0 tx=0).stop 2023-11-16T17:00:14.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.439+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1419730017 shutdown_connections 2023-11-16T17:00:14.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.439+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1419730017 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561e30e42250 0x561e30e44660 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.439+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1419730017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e248b0 0x561e30e326c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.439+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1419730017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e1fb60 0x561e30e31c40 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.439+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1419730017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e20470 0x561e30e32180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.442 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.439+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1419730017 >> 172.21.15.44:0/1419730017 conn(0x561e30e15360 msgr2=0x561e30e25d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:14.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.439+0000 7f2a73fff700 0 client.27257 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T17:00:14.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.440+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1419730017 shutdown_connections 2023-11-16T17:00:14.443 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.440+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1419730017 wait complete. 2023-11-16T17:00:14.444 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.442+0000 7f2aaea90b80 20 client.27257 trim_cache size 0 max 16384 2023-11-16T17:00:14.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.470+0000 7f2aaea90b80 1 Processor -- start 2023-11-16T17:00:14.472 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.470+0000 7f2aaea90b80 1 -- start start 2023-11-16T17:00:14.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.471+0000 7f2aaea90b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e1fb60 0x561e30e391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.471+0000 7f2aaea90b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e20470 0x561e30e22990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.473 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.471+0000 7f2aaea90b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e248b0 0x561e30e22ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.472+0000 7f2a98f9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e1fb60 0x561e30e391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:14.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.472+0000 7f2a93fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e20470 0x561e30e22990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:14.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.472+0000 7f2a9979c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e248b0 0x561e30e22ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:14.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.472+0000 7f2a98f9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e1fb60 0x561e30e391b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48160/0 (socket says 172.21.15.44:48160) 2023-11-16T17:00:14.474 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.472+0000 7f2a98f9b700 1 -- 172.21.15.44:0/1413965586 learned_addr learned my addr 172.21.15.44:0/1413965586 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:14.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.472+0000 7f2aaea90b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e30e331a0 con 0x561e30e20470 2023-11-16T17:00:14.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.472+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1413965586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e30e23520 con 0x561e30e248b0 2023-11-16T17:00:14.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.472+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1413965586 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e30e44970 con 0x561e30e1fb60 2023-11-16T17:00:14.475 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.472+0000 7f2a98f9b700 1 -- 172.21.15.44:0/1413965586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e248b0 msgr2=0x561e30e22ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T17:00:14.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.472+0000 7f2a98f9b700 1 --2- 172.21.15.44:0/1413965586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e248b0 0x561e30e22ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.472+0000 7f2a98f9b700 1 -- 172.21.15.44:0/1413965586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e20470 msgr2=0x561e30e22990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T17:00:14.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.472+0000 7f2a98f9b700 1 --2- 172.21.15.44:0/1413965586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e20470 0x561e30e22990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.472+0000 7f2a98f9b700 1 -- 172.21.15.44:0/1413965586 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e30e24f50 con 0x561e30e1fb60 2023-11-16T17:00:14.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.473+0000 7f2a98f9b700 1 --2- 172.21.15.44:0/1413965586 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e1fb60 0x561e30e391b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c002230 tx=0x7f2a8c00b830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d7a58340081711a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:14.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.474+0000 7f2a8b7fe700 1 -- 172.21.15.44:0/1413965586 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c00e070 con 0x561e30e1fb60 2023-11-16T17:00:14.476 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.474+0000 7f2a8b7fe700 1 -- 172.21.15.44:0/1413965586 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a8c027800 con 0x561e30e1fb60 2023-11-16T17:00:14.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.474+0000 7f2a8b7fe700 1 -- 172.21.15.44:0/1413965586 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c0279c0 con 0x561e30e1fb60 2023-11-16T17:00:14.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.475+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1413965586 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e1fb60 msgr2=0x561e30e391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:14.477 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.475+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1413965586 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e1fb60 0x561e30e391b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c002230 tx=0x7f2a8c00b830 comp rx=0 tx=0).stop 2023-11-16T17:00:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.476+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1413965586 shutdown_connections 2023-11-16T17:00:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.476+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1413965586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e248b0 0x561e30e22ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.476+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1413965586 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e1fb60 0x561e30e391b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.476+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1413965586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e20470 0x561e30e22990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.476+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1413965586 >> 172.21.15.44:0/1413965586 conn(0x561e30e15360 msgr2=0x561e30e44260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.476+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1413965586 shutdown_connections 2023-11-16T17:00:14.479 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.476+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1413965586 wait complete. 2023-11-16T17:00:14.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.477+0000 7f2aaea90b80 1 -- start start 2023-11-16T17:00:14.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.477+0000 7f2aaea90b80 1 Processor -- start 2023-11-16T17:00:14.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.478+0000 7f2aaea90b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e1fb60 0x561e30e391c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.478+0000 7f2aaea90b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e20470 0x561e30e371a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.480 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.478+0000 7f2aaea90b80 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e248b0 0x561e30e376e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.478+0000 7f2aaea90b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e30e37d00 con 0x561e30e1fb60 2023-11-16T17:00:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.478+0000 7f2aaea90b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e30e37e40 con 0x561e30e20470 2023-11-16T17:00:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.478+0000 7f2aaea90b80 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e30e38000 con 0x561e30e248b0 2023-11-16T17:00:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.478+0000 7f2aaea90b80 10 client.0 _ll_register_callbacks cb 0x7f2a8c0093e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T17:00:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.479+0000 7f2a9979c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e248b0 0x561e30e376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.479+0000 7f2a9979c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e248b0 0x561e30e376e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48172/0 (socket says 172.21.15.44:48172) 2023-11-16T17:00:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.479+0000 7f2a9979c700 1 -- 172.21.15.44:0/1440979399 learned_addr learned my addr 172.21.15.44:0/1440979399 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.479+0000 7f2a93fff700 1 --2- 172.21.15.44:0/1440979399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e20470 0x561e30e371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.480+0000 7f2a9979c700 1 -- 172.21.15.44:0/1440979399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e20470 msgr2=0x561e30e371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T17:00:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.480+0000 7f2a9979c700 1 --2- 172.21.15.44:0/1440979399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e20470 0x561e30e371a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.480+0000 7f2a9979c700 1 -- 172.21.15.44:0/1440979399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e1fb60 msgr2=0x561e30e391c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T17:00:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.480+0000 7f2a9979c700 1 --2- 172.21.15.44:0/1440979399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e1fb60 0x561e30e391c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.480+0000 7f2a9979c700 1 -- 172.21.15.44:0/1440979399 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e30d37190 con 0x561e30e248b0 2023-11-16T17:00:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.480+0000 7f2a9979c700 1 --2- 172.21.15.44:0/1440979399 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e248b0 0x561e30e376e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2a84098d40 tx=0x7f2a84004620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42353c376134d768 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.480+0000 7f2a73fff700 10 client.0 ms_handle_connect on v2:172.21.15.176:3300/0 2023-11-16T17:00:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.480+0000 7f2a73fff700 1 -- 172.21.15.44:0/1440979399 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a840b0070 con 0x561e30e248b0 2023-11-16T17:00:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.481+0000 7f2a73fff700 1 -- 172.21.15.44:0/1440979399 <== mon.1 v2:172.21.15.176:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f2a8409cde0 con 0x561e30e248b0 2023-11-16T17:00:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.481+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1440979399 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a7c01a030 con 0x561e30e248b0 2023-11-16T17:00:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.481+0000 7f2a73fff700 1 -- 172.21.15.44:0/1440979399 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a840a1200 con 0x561e30e248b0 2023-11-16T17:00:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.481+0000 7f2aaea90b80 10 client.27202 fetch_fsmap learned FSMap version 21 2023-11-16T17:00:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.481+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1440979399 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x561e30b9ede0 con 0x561e30e248b0 2023-11-16T17:00:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.481+0000 7f2a73fff700 1 -- 172.21.15.44:0/1440979399 <== mon.1 v2:172.21.15.176:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a840c7080 con 0x561e30e248b0 2023-11-16T17:00:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.482+0000 7f2a73fff700 1 -- 172.21.15.44:0/1440979399 <== mon.1 v2:172.21.15.176:3300/0 5 ==== fsmap(e 21) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2a840b3d20 con 0x561e30e248b0 2023-11-16T17:00:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.482+0000 7f2aaea90b80 10 client.27202 fetch_fsmap finished waiting for FSMap version 21 2023-11-16T17:00:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.482+0000 7f2aaea90b80 10 client.27202 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-11-16T17:00:14.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.482+0000 7f2aaea90b80 20 client.27202 populate_metadata read hostname 'smithi044' 2023-11-16T17:00:14.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.482+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1440979399 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561e30e96500 0x561e30e98910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:14.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.482+0000 7f2aaea90b80 4 client.27202 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-11-16T17:00:14.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.482+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1440979399 --> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561e30d3f660 con 0x561e30e96500 2023-11-16T17:00:14.486 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.485+0000 7f2a98f9b700 1 --2- 172.21.15.44:0/1440979399 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561e30e96500 0x561e30e98910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:14.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.486+0000 7f2a73fff700 10 client.27202 ms_handle_connect on v2:172.21.15.77:6838/1948116745 2023-11-16T17:00:14.487 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.486+0000 7f2a98f9b700 1 --2- 172.21.15.44:0/1440979399 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561e30e96500 0x561e30e98910 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=691dfd75bb601888 server_cookie=5408ac9f3ad8e167 in_seq=0 out_seq=0 2023-11-16T17:00:14.488 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.486+0000 7f2a73fff700 1 -- 172.21.15.44:0/1440979399 <== mds.0 v2:172.21.15.77:6838/1948116745 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561e30d3f660 con 0x561e30e96500 2023-11-16T17:00:14.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.486+0000 7f2a73fff700 10 client.27202 handle_command_reply: tid=0 2023-11-16T17:00:14.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.486+0000 7f2aaea90b80 1 client.27202 shutdown 2023-11-16T17:00:14.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.486+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1440979399 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e248b0 msgr2=0x561e30e376e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:14.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.486+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1440979399 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e248b0 0x561e30e376e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2a84098d40 tx=0x7f2a84004620 comp rx=0 tx=0).stop 2023-11-16T17:00:14.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.487+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1440979399 shutdown_connections 2023-11-16T17:00:14.489 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.487+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1440979399 >> [v2:172.21.15.77:6838/1948116745,v1:172.21.15.77:6839/1948116745] conn(0x561e30e96500 0x561e30e98910 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.487+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1440979399 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x561e30e248b0 0x561e30e376e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.487+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1440979399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e30e1fb60 0x561e30e391c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.487+0000 7f2aaea90b80 1 --2- 172.21.15.44:0/1440979399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e30e20470 0x561e30e371a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.487+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1440979399 >> 172.21.15.44:0/1440979399 conn(0x561e30e15360 msgr2=0x561e30e22990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.487+0000 7f2a73fff700 0 client.27202 ms_handle_reset on v2:172.21.15.77:6838/1948116745 2023-11-16T17:00:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.487+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1440979399 shutdown_connections 2023-11-16T17:00:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.487+0000 7f2aaea90b80 1 -- 172.21.15.44:0/1440979399 wait complete. 2023-11-16T17:00:14.492 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2023-11-16T17:00:14.492 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.489+0000 7f2aaea90b80 20 client.27202 trim_cache size 0 max 16384 2023-11-16T17:00:14.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.491+0000 7f2a9b1ff700 1 -- 172.21.15.44:0/3375411722 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2a800a12e0 msgr2=0x7f2a800a3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:14.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.491+0000 7f2a9b1ff700 1 --2- 172.21.15.44:0/3375411722 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2a800a12e0 0x7f2a800a3790 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c005140 tx=0x7f2a7c005f60 comp rx=0 tx=0).stop 2023-11-16T17:00:14.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.491+0000 7f2a9b1ff700 1 -- 172.21.15.44:0/3375411722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a941133b0 msgr2=0x7f2a941ace90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:14.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.491+0000 7f2a9b1ff700 1 --2- 172.21.15.44:0/3375411722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a941133b0 0x7f2a941ace90 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2a9406a2e0 tx=0x7f2a8400da00 comp rx=0 tx=0).stop 2023-11-16T17:00:14.493 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.491+0000 7f2a9b1ff700 1 -- 172.21.15.44:0/3375411722 shutdown_connections 2023-11-16T17:00:14.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.491+0000 7f2a9b1ff700 1 --2- 172.21.15.44:0/3375411722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a941133b0 0x7f2a941ace90 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.492+0000 7f2a9b1ff700 1 --2- 172.21.15.44:0/3375411722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a94107af0 0x7f2a94108cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.492+0000 7f2a9b1ff700 1 --2- 172.21.15.44:0/3375411722 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2a800a12e0 0x7f2a800a3790 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.494 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.492+0000 7f2a9b1ff700 1 --2- 172.21.15.44:0/3375411722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a94108400 0x7f2a94109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:14.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.492+0000 7f2a9b1ff700 1 -- 172.21.15.44:0/3375411722 >> 172.21.15.44:0/3375411722 conn(0x7f2a9406c710 msgr2=0x7f2a941154d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:14.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.492+0000 7f2a9b1ff700 1 -- 172.21.15.44:0/3375411722 shutdown_connections 2023-11-16T17:00:14.495 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:14.492+0000 7f2a9b1ff700 1 -- 172.21.15.44:0/3375411722 wait complete. 2023-11-16T17:00:14.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:14 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2208702292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T17:00:14.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:14 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2208702292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T17:00:15.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:14 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2208702292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T17:00:15.528 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2023-11-16T17:00:15.528 INFO:tasks.fwd_scrub:done joining 2023-11-16T17:00:15.529 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2023-11-16T17:00:15.540 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T17:00:15.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.913+0000 7ff80c99d700 1 -- 172.21.15.44:0/748681693 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff808068040 msgr2=0x7ff808068410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:15.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.913+0000 7ff80c99d700 1 --2- 172.21.15.44:0/748681693 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff808068040 0x7ff808068410 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800b0d0 tx=0x7ff7f800b3e0 comp rx=0 tx=0).stop 2023-11-16T17:00:15.915 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.914+0000 7ff80c99d700 1 -- 172.21.15.44:0/748681693 shutdown_connections 2023-11-16T17:00:15.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.914+0000 7ff80c99d700 1 --2- 172.21.15.44:0/748681693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff808105380 0x7ff80810cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:15.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.914+0000 7ff80c99d700 1 --2- 172.21.15.44:0/748681693 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff808068040 0x7ff808068410 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:15.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.914+0000 7ff80c99d700 1 --2- 172.21.15.44:0/748681693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8080689e0 0x7ff808104ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:15.916 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.914+0000 7ff80c99d700 1 -- 172.21.15.44:0/748681693 >> 172.21.15.44:0/748681693 conn(0x7ff808076fc0 msgr2=0x7ff8080773c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:15.918 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.916+0000 7ff80c99d700 1 -- 172.21.15.44:0/748681693 shutdown_connections 2023-11-16T17:00:15.919 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.916+0000 7ff80c99d700 1 -- 172.21.15.44:0/748681693 wait complete. 2023-11-16T17:00:15.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.918+0000 7ff80c99d700 1 Processor -- start 2023-11-16T17:00:15.920 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.918+0000 7ff80c99d700 1 -- start start 2023-11-16T17:00:15.921 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.919+0000 7ff80c99d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff808068040 0x7ff80810bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:15.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.920+0000 7ff80c99d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8080689e0 0x7ff80810c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:15.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.920+0000 7ff80c99d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff808105380 0x7ff8081b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:15.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.920+0000 7ff80c99d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80810ca40 con 0x7ff808105380 2023-11-16T17:00:15.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.920+0000 7ff8077fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff808068040 0x7ff80810bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:15.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.920+0000 7ff8077fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff808068040 0x7ff80810bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:55560/0 (socket says 172.21.15.44:55560) 2023-11-16T17:00:15.922 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.920+0000 7ff8077fe700 1 -- 172.21.15.44:0/81429932 learned_addr learned my addr 172.21.15.44:0/81429932 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:15.923 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.920+0000 7ff807fff700 1 --2- 172.21.15.44:0/81429932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff808105380 0x7ff8081b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:15.923 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.920+0000 7ff80c99d700 1 -- 172.21.15.44:0/81429932 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8081b1520 con 0x7ff808068040 2023-11-16T17:00:15.923 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.920+0000 7ff80c99d700 1 -- 172.21.15.44:0/81429932 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8081b1660 con 0x7ff8080689e0 2023-11-16T17:00:15.924 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.921+0000 7ff7feffd700 1 --2- 172.21.15.44:0/81429932 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8080689e0 0x7ff80810c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:15.924 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.921+0000 7ff807fff700 1 -- 172.21.15.44:0/81429932 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8080689e0 msgr2=0x7ff80810c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:15.925 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.921+0000 7ff807fff700 1 --2- 172.21.15.44:0/81429932 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8080689e0 0x7ff80810c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:15.925 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.921+0000 7ff807fff700 1 -- 172.21.15.44:0/81429932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff808068040 msgr2=0x7ff80810bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:15.925 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.921+0000 7ff807fff700 1 --2- 172.21.15.44:0/81429932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff808068040 0x7ff80810bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:15.925 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.921+0000 7ff807fff700 1 -- 172.21.15.44:0/81429932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f8009d00 con 0x7ff808105380 2023-11-16T17:00:15.925 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.922+0000 7ff807fff700 1 --2- 172.21.15.44:0/81429932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff808105380 0x7ff8081b0fe0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff80000b3f0 tx=0x7ff80000b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:15.925 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.922+0000 7ff8057fa700 1 -- 172.21.15.44:0/81429932 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800011630 con 0x7ff808105380 2023-11-16T17:00:15.925 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.922+0000 7ff8057fa700 1 -- 172.21.15.44:0/81429932 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff800011c50 con 0x7ff808105380 2023-11-16T17:00:15.925 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.923+0000 7ff8057fa700 1 -- 172.21.15.44:0/81429932 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800010ce0 con 0x7ff808105380 2023-11-16T17:00:15.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.923+0000 7ff80c99d700 1 -- 172.21.15.44:0/81429932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8081102c0 con 0x7ff808105380 2023-11-16T17:00:15.926 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.923+0000 7ff80c99d700 1 -- 172.21.15.44:0/81429932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff808110130 con 0x7ff808105380 2023-11-16T17:00:15.927 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.925+0000 7ff80c99d700 1 -- 172.21.15.44:0/81429932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff808110450 con 0x7ff808105380 2023-11-16T17:00:15.931 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.930+0000 7ff8057fa700 1 -- 172.21.15.44:0/81429932 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7ff800010380 con 0x7ff808105380 2023-11-16T17:00:15.936 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.935+0000 7ff8057fa700 1 --2- 172.21.15.44:0/81429932 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff7f40a1380 0x7ff7f40a3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:15.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.935+0000 7ff8057fa700 1 -- 172.21.15.44:0/81429932 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9599+0+0 (secure 0 0 0) 0x7ff8000ccde0 con 0x7ff808105380 2023-11-16T17:00:15.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.936+0000 7ff8077fe700 1 --2- 172.21.15.44:0/81429932 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff7f40a1380 0x7ff7f40a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:15.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.937+0000 7ff8077fe700 1 --2- 172.21.15.44:0/81429932 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff7f40a1380 0x7ff7f40a3830 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8000f80 tx=0x7ff7f800b040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:15.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:15.938+0000 7ff8057fa700 1 -- 172.21.15.44:0/81429932 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff800095ed0 con 0x7ff808105380 2023-11-16T17:00:15.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:15 smithi077 ceph-mon[131204]: pgmap v321: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2023-11-16T17:00:16.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:15 smithi044 ceph-mon[128750]: pgmap v321: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2023-11-16T17:00:16.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:15 smithi176 ceph-mon[131630]: pgmap v321: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2023-11-16T17:00:16.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.357+0000 7ff80c99d700 1 -- 172.21.15.44:0/81429932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff808067980 con 0x7ff808105380 2023-11-16T17:00:16.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.363+0000 7ff8057fa700 1 -- 172.21.15.44:0/81429932 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+25382 (secure 0 0 0) 0x7ff800015090 con 0x7ff808105380 2023-11-16T17:00:16.367 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T17:00:16.368 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":101,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T17:00:07.638718+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/4218362219":"2023-11-17T16:49:55.607497+0000","172.21.15.44:6801/304422349":"2023-11-17T16:49:55.607497+0000","172.21.15.44:0/1983838729":"2023-11-17T16:49:55.607497+0000","172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/2937650254":"2023-11-17T16:49:55.607497+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1915598320":"2023-11-17T16:49:55.607497+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/304422349":"2023-11-17T16:49:55.607497+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T17:00:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.369+0000 7ff80c99d700 1 -- 172.21.15.44:0/81429932 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff7f40a1380 msgr2=0x7ff7f40a3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.369+0000 7ff80c99d700 1 --2- 172.21.15.44:0/81429932 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff7f40a1380 0x7ff7f40a3830 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8000f80 tx=0x7ff7f800b040 comp rx=0 tx=0).stop 2023-11-16T17:00:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.369+0000 7ff80c99d700 1 -- 172.21.15.44:0/81429932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff808105380 msgr2=0x7ff8081b0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.369+0000 7ff80c99d700 1 --2- 172.21.15.44:0/81429932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff808105380 0x7ff8081b0fe0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff80000b3f0 tx=0x7ff80000b700 comp rx=0 tx=0).stop 2023-11-16T17:00:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.370+0000 7ff80c99d700 1 -- 172.21.15.44:0/81429932 shutdown_connections 2023-11-16T17:00:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.370+0000 7ff80c99d700 1 --2- 172.21.15.44:0/81429932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff808105380 0x7ff8081b0fe0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.370+0000 7ff80c99d700 1 --2- 172.21.15.44:0/81429932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff808068040 0x7ff80810bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.370+0000 7ff80c99d700 1 --2- 172.21.15.44:0/81429932 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7ff7f40a1380 0x7ff7f40a3830 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.370+0000 7ff80c99d700 1 --2- 172.21.15.44:0/81429932 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8080689e0 0x7ff80810c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.370+0000 7ff80c99d700 1 -- 172.21.15.44:0/81429932 >> 172.21.15.44:0/81429932 conn(0x7ff808076fc0 msgr2=0x7ff808103c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.370+0000 7ff80c99d700 1 -- 172.21.15.44:0/81429932 shutdown_connections 2023-11-16T17:00:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.370+0000 7ff80c99d700 1 -- 172.21.15.44:0/81429932 wait complete. 2023-11-16T17:00:16.407 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:03.059622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:16.211826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:17.601591+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:21.136084+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T17:00:16.407 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T17:00:16.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.791+0000 7f0c245ff700 1 -- 172.21.15.44:0/1905628500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c1c101390 msgr2=0x7f0c1c101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:16.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.791+0000 7f0c245ff700 1 --2- 172.21.15.44:0/1905628500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c1c101390 0x7f0c1c101760 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0c10009f90 tx=0x7f0c1000b330 comp rx=0 tx=0).stop 2023-11-16T17:00:16.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.792+0000 7f0c245ff700 1 -- 172.21.15.44:0/1905628500 shutdown_connections 2023-11-16T17:00:16.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.792+0000 7f0c245ff700 1 --2- 172.21.15.44:0/1905628500 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c1c10a8b0 0x7f0c1c10ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:16.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.792+0000 7f0c245ff700 1 --2- 172.21.15.44:0/1905628500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c1c101390 0x7f0c1c101760 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:16.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.792+0000 7f0c245ff700 1 --2- 172.21.15.44:0/1905628500 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c1c101ca0 0x7f0c1c10a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:16.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.792+0000 7f0c245ff700 1 -- 172.21.15.44:0/1905628500 >> 172.21.15.44:0/1905628500 conn(0x7f0c1c0770b0 msgr2=0x7f0c1c0774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:16.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.792+0000 7f0c245ff700 1 -- 172.21.15.44:0/1905628500 shutdown_connections 2023-11-16T17:00:16.794 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.792+0000 7f0c245ff700 1 -- 172.21.15.44:0/1905628500 wait complete. 2023-11-16T17:00:16.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.794+0000 7f0c245ff700 1 Processor -- start 2023-11-16T17:00:16.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.794+0000 7f0c245ff700 1 -- start start 2023-11-16T17:00:16.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.795+0000 7f0c245ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c1c101390 0x7f0c1c103b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:16.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.795+0000 7f0c245ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c1c101ca0 0x7f0c1c104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:16.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.795+0000 7f0c245ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c1c10a8b0 0x7f0c1c108490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:16.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.795+0000 7f0c245ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c1089d0 con 0x7f0c1c101ca0 2023-11-16T17:00:16.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.795+0000 7f0c245ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c108b40 con 0x7f0c1c10a8b0 2023-11-16T17:00:16.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.795+0000 7f0c245ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c108d00 con 0x7f0c1c101390 2023-11-16T17:00:16.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.796+0000 7f0c2239b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c1c101390 0x7f0c1c103b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:16.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.796+0000 7f0c2239b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c1c101390 0x7f0c1c103b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:49818/0 (socket says 172.21.15.44:49818) 2023-11-16T17:00:16.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.796+0000 7f0c2239b700 1 -- 172.21.15.44:0/2464493427 learned_addr learned my addr 172.21.15.44:0/2464493427 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:16.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.796+0000 7f0c21b9a700 1 --2- 172.21.15.44:0/2464493427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c1c101ca0 0x7f0c1c104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:16.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.796+0000 7f0c2239b700 1 -- 172.21.15.44:0/2464493427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c1c10a8b0 msgr2=0x7f0c1c108490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:16.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.796+0000 7f0c2239b700 1 --2- 172.21.15.44:0/2464493427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c1c10a8b0 0x7f0c1c108490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.796+0000 7f0c2239b700 1 -- 172.21.15.44:0/2464493427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c1c101ca0 msgr2=0x7f0c1c104080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.796+0000 7f0c2239b700 1 --2- 172.21.15.44:0/2464493427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c1c101ca0 0x7f0c1c104080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.796+0000 7f0c2239b700 1 -- 172.21.15.44:0/2464493427 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c140095f0 con 0x7f0c1c101390 2023-11-16T17:00:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.797+0000 7f0c2239b700 1 --2- 172.21.15.44:0/2464493427 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c1c101390 0x7f0c1c103b40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0c10004af0 tx=0x7f0c10004b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.797+0000 7f0c1b7fe700 1 -- 172.21.15.44:0/2464493427 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1000e040 con 0x7f0c1c101390 2023-11-16T17:00:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.797+0000 7f0c1b7fe700 1 -- 172.21.15.44:0/2464493427 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c1000bbf0 con 0x7f0c1c101390 2023-11-16T17:00:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.797+0000 7f0c1b7fe700 1 -- 172.21.15.44:0/2464493427 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c100052a0 con 0x7f0c1c101390 2023-11-16T17:00:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.797+0000 7f0c245ff700 1 -- 172.21.15.44:0/2464493427 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c10009c40 con 0x7f0c1c101390 2023-11-16T17:00:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.798+0000 7f0c245ff700 1 -- 172.21.15.44:0/2464493427 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c1c10e110 con 0x7f0c1c101390 2023-11-16T17:00:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.799+0000 7f0c245ff700 1 -- 172.21.15.44:0/2464493427 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c1c10df80 con 0x7f0c1c101390 2023-11-16T17:00:16.806 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.804+0000 7f0c1b7fe700 1 -- 172.21.15.44:0/2464493427 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f0c10005400 con 0x7f0c1c101390 2023-11-16T17:00:16.811 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.809+0000 7f0c1b7fe700 1 --2- 172.21.15.44:0/2464493427 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c0c0a13a0 0x7f0c0c0a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:16.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.809+0000 7f0c1b7fe700 1 -- 172.21.15.44:0/2464493427 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f0c100ced70 con 0x7f0c1c101390 2023-11-16T17:00:16.812 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.810+0000 7f0c1b7fe700 1 -- 172.21.15.44:0/2464493427 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0c100cf150 con 0x7f0c1c101390 2023-11-16T17:00:16.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.813+0000 7f0c21b9a700 1 --2- 172.21.15.44:0/2464493427 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c0c0a13a0 0x7f0c0c0a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:16.824 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:16.821+0000 7f0c21b9a700 1 --2- 172.21.15.44:0/2464493427 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c0c0a13a0 0x7f0c0c0a3850 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c105100 tx=0x7f0c0800a380 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:16.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:16 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/81429932' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T17:00:17.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:16 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/81429932' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T17:00:17.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:16 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/81429932' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T17:00:17.219 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.218+0000 7f0c245ff700 1 -- 172.21.15.44:0/2464493427 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0c1c10e2a0 con 0x7f0c1c101390 2023-11-16T17:00:17.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.218+0000 7f0c1b7fe700 1 -- 172.21.15.44:0/2464493427 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0c10097e10 con 0x7f0c1c101390 2023-11-16T17:00:17.222 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2023-11-16T17:00:17.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.224+0000 7f0c245ff700 1 -- 172.21.15.44:0/2464493427 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c0c0a13a0 msgr2=0x7f0c0c0a3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:17.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.224+0000 7f0c245ff700 1 --2- 172.21.15.44:0/2464493427 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c0c0a13a0 0x7f0c0c0a3850 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c105100 tx=0x7f0c0800a380 comp rx=0 tx=0).stop 2023-11-16T17:00:17.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.224+0000 7f0c245ff700 1 -- 172.21.15.44:0/2464493427 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c1c101390 msgr2=0x7f0c1c103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:17.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.224+0000 7f0c245ff700 1 --2- 172.21.15.44:0/2464493427 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c1c101390 0x7f0c1c103b40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0c10004af0 tx=0x7f0c10004b20 comp rx=0 tx=0).stop 2023-11-16T17:00:17.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.224+0000 7f0c245ff700 1 -- 172.21.15.44:0/2464493427 shutdown_connections 2023-11-16T17:00:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.224+0000 7f0c245ff700 1 --2- 172.21.15.44:0/2464493427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c1c10a8b0 0x7f0c1c108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.224+0000 7f0c245ff700 1 --2- 172.21.15.44:0/2464493427 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c0c0a13a0 0x7f0c0c0a3850 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.224+0000 7f0c245ff700 1 --2- 172.21.15.44:0/2464493427 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c1c101390 0x7f0c1c103b40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.224+0000 7f0c245ff700 1 --2- 172.21.15.44:0/2464493427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c1c101ca0 0x7f0c1c104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.224+0000 7f0c245ff700 1 -- 172.21.15.44:0/2464493427 >> 172.21.15.44:0/2464493427 conn(0x7f0c1c0770b0 msgr2=0x7f0c1c0fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.224+0000 7f0c245ff700 1 -- 172.21.15.44:0/2464493427 shutdown_connections 2023-11-16T17:00:17.227 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.224+0000 7f0c245ff700 1 -- 172.21.15.44:0/2464493427 wait complete. 2023-11-16T17:00:17.264 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T17:00:17.648 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.646+0000 7ff006bff700 1 -- 172.21.15.44:0/3071495734 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff000105630 msgr2=0x7ff00010d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:17.648 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.646+0000 7ff006bff700 1 --2- 172.21.15.44:0/3071495734 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff000105630 0x7ff00010d1c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feff800b0a0 tx=0x7feff800b3b0 comp rx=0 tx=0).stop 2023-11-16T17:00:17.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.648+0000 7ff006bff700 1 -- 172.21.15.44:0/3071495734 shutdown_connections 2023-11-16T17:00:17.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.648+0000 7ff006bff700 1 --2- 172.21.15.44:0/3071495734 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff000105630 0x7ff00010d1c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:17.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.648+0000 7ff006bff700 1 --2- 172.21.15.44:0/3071495734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff000100c50 0x7ff000101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:17.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.648+0000 7ff006bff700 1 --2- 172.21.15.44:0/3071495734 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0001015f0 0x7ff000104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:17.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.648+0000 7ff006bff700 1 -- 172.21.15.44:0/3071495734 >> 172.21.15.44:0/3071495734 conn(0x7ff0000775c0 msgr2=0x7ff0000779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:17.650 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.648+0000 7ff006bff700 1 -- 172.21.15.44:0/3071495734 shutdown_connections 2023-11-16T17:00:17.652 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.650+0000 7ff006bff700 1 -- 172.21.15.44:0/3071495734 wait complete. 2023-11-16T17:00:17.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.651+0000 7ff006bff700 1 Processor -- start 2023-11-16T17:00:17.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.652+0000 7ff006bff700 1 -- start start 2023-11-16T17:00:17.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.652+0000 7ff006bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff000100c50 0x7ff000107160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:17.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.652+0000 7ff006bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0001015f0 0x7ff0001076a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:17.654 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.652+0000 7ff006bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff000105630 0x7ff000107be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:17.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.652+0000 7ff006bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0001a8cf0 con 0x7ff0001015f0 2023-11-16T17:00:17.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.652+0000 7ff006bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0001a8e60 con 0x7ff000105630 2023-11-16T17:00:17.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.652+0000 7ff006bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0001a9020 con 0x7ff000100c50 2023-11-16T17:00:17.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.652+0000 7ff00499b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff000100c50 0x7ff000107160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:17.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.653+0000 7ff00499b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff000100c50 0x7ff000107160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:49830/0 (socket says 172.21.15.44:49830) 2023-11-16T17:00:17.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.653+0000 7ff00499b700 1 -- 172.21.15.44:0/426338703 learned_addr learned my addr 172.21.15.44:0/426338703 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:17.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.653+0000 7ff00519c700 1 --2- 172.21.15.44:0/426338703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff000105630 0x7ff000107be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:17.656 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.653+0000 7ff00499b700 1 -- 172.21.15.44:0/426338703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff000105630 msgr2=0x7ff000107be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:17.656 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.653+0000 7ff00499b700 1 --2- 172.21.15.44:0/426338703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff000105630 0x7ff000107be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:17.656 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.653+0000 7ff00499b700 1 -- 172.21.15.44:0/426338703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0001015f0 msgr2=0x7ff0001076a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:17.656 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.653+0000 7ff00499b700 1 --2- 172.21.15.44:0/426338703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0001015f0 0x7ff0001076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:17.656 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.653+0000 7ff00499b700 1 -- 172.21.15.44:0/426338703 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff8009d00 con 0x7ff000100c50 2023-11-16T17:00:17.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.655+0000 7ff00499b700 1 --2- 172.21.15.44:0/426338703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff000100c50 0x7ff000107160 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fefec00b500 tx=0x7fefec00b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:17.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.656+0000 7feffdffb700 1 -- 172.21.15.44:0/426338703 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec011830 con 0x7ff000100c50 2023-11-16T17:00:17.658 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.656+0000 7feffdffb700 1 -- 172.21.15.44:0/426338703 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fefec011e50 con 0x7ff000100c50 2023-11-16T17:00:17.658 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.656+0000 7feffdffb700 1 -- 172.21.15.44:0/426338703 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec00f2e0 con 0x7ff000100c50 2023-11-16T17:00:17.658 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.656+0000 7ff006bff700 1 -- 172.21.15.44:0/426338703 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff000110490 con 0x7ff000100c50 2023-11-16T17:00:17.658 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.656+0000 7ff006bff700 1 -- 172.21.15.44:0/426338703 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff000110300 con 0x7ff000100c50 2023-11-16T17:00:17.659 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.658+0000 7ff006bff700 1 -- 172.21.15.44:0/426338703 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff000110620 con 0x7ff000100c50 2023-11-16T17:00:17.664 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.662+0000 7feffdffb700 1 -- 172.21.15.44:0/426338703 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fefec010580 con 0x7ff000100c50 2023-11-16T17:00:17.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.665+0000 7feffdffb700 1 --2- 172.21.15.44:0/426338703 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7feff00a1340 0x7feff00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:17.667 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.665+0000 7feffffff700 1 --2- 172.21.15.44:0/426338703 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7feff00a1340 0x7feff00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:17.668 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.667+0000 7feffdffb700 1 -- 172.21.15.44:0/426338703 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9599+0+0 (secure 0 0 0) 0x7fefec010380 con 0x7ff000100c50 2023-11-16T17:00:17.669 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.667+0000 7feffffff700 1 --2- 172.21.15.44:0/426338703 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7feff00a1340 0x7feff00a37f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7feff4005950 tx=0x7feff40058e0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:17.671 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:17.669+0000 7feffdffb700 1 -- 172.21.15.44:0/426338703 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fefec095af0 con 0x7ff000100c50 2023-11-16T17:00:18.086 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.085+0000 7ff006bff700 1 -- 172.21.15.44:0/426338703 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff0001a9b90 con 0x7ff000100c50 2023-11-16T17:00:18.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.085+0000 7feffdffb700 1 -- 172.21.15.44:0/426338703 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) v1 ==== 104+0+11 (secure 0 0 0) 0x7fefec099110 con 0x7ff000100c50 2023-11-16T17:00:18.090 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T17:00:18.091 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.090+0000 7ff006bff700 1 -- 172.21.15.44:0/426338703 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7feff00a1340 msgr2=0x7feff00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:18.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.090+0000 7ff006bff700 1 --2- 172.21.15.44:0/426338703 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7feff00a1340 0x7feff00a37f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7feff4005950 tx=0x7feff40058e0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.090+0000 7ff006bff700 1 -- 172.21.15.44:0/426338703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff000100c50 msgr2=0x7ff000107160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:18.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.090+0000 7ff006bff700 1 --2- 172.21.15.44:0/426338703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff000100c50 0x7ff000107160 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fefec00b500 tx=0x7fefec00b8c0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.091+0000 7ff006bff700 1 -- 172.21.15.44:0/426338703 shutdown_connections 2023-11-16T17:00:18.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.091+0000 7ff006bff700 1 --2- 172.21.15.44:0/426338703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff000105630 0x7ff000107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.091+0000 7ff006bff700 1 --2- 172.21.15.44:0/426338703 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7feff00a1340 0x7feff00a37f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.091+0000 7ff006bff700 1 --2- 172.21.15.44:0/426338703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff000100c50 0x7ff000107160 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.091+0000 7ff006bff700 1 --2- 172.21.15.44:0/426338703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0001015f0 0x7ff0001076a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.091+0000 7ff006bff700 1 -- 172.21.15.44:0/426338703 >> 172.21.15.44:0/426338703 conn(0x7ff0000775c0 msgr2=0x7ff0000fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:18.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.091+0000 7ff006bff700 1 -- 172.21.15.44:0/426338703 shutdown_connections 2023-11-16T17:00:18.094 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.091+0000 7ff006bff700 1 -- 172.21.15.44:0/426338703 wait complete. 2023-11-16T17:00:18.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:17 smithi044 ceph-mon[128750]: pgmap v322: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2023-11-16T17:00:18.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:17 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2464493427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T17:00:18.126 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T17:00:18.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:17 smithi176 ceph-mon[131630]: pgmap v322: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2023-11-16T17:00:18.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:17 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2464493427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T17:00:18.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:17 smithi077 ceph-mon[131204]: pgmap v322: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2023-11-16T17:00:18.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:17 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2464493427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T17:00:18.501 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.500+0000 7fe1e9fff700 1 -- 172.21.15.44:0/2487767102 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1e40746f0 msgr2=0x7fe1e4072d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.500+0000 7fe1e9fff700 1 --2- 172.21.15.44:0/2487767102 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1e40746f0 0x7fe1e4072d20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009990 tx=0x7fe1cc009ca0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.500+0000 7fe1e9fff700 1 -- 172.21.15.44:0/2487767102 shutdown_connections 2023-11-16T17:00:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.500+0000 7fe1e9fff700 1 --2- 172.21.15.44:0/2487767102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1e410cec0 0x7fe1e410f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.500+0000 7fe1e9fff700 1 --2- 172.21.15.44:0/2487767102 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1e40746f0 0x7fe1e4072d20 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.500+0000 7fe1e9fff700 1 --2- 172.21.15.44:0/2487767102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1e4074ac0 0x7fe1e4073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.500+0000 7fe1e9fff700 1 -- 172.21.15.44:0/2487767102 >> 172.21.15.44:0/2487767102 conn(0x7fe1e4077560 msgr2=0x7fe1e4077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.500+0000 7fe1e9fff700 1 -- 172.21.15.44:0/2487767102 shutdown_connections 2023-11-16T17:00:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.500+0000 7fe1e9fff700 1 -- 172.21.15.44:0/2487767102 wait complete. 2023-11-16T17:00:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.502+0000 7fe1e9fff700 1 Processor -- start 2023-11-16T17:00:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.502+0000 7fe1e9fff700 1 -- start start 2023-11-16T17:00:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.502+0000 7fe1e9fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1e40746f0 0x7fe1e419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.502+0000 7fe1e9fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1e4074ac0 0x7fe1e419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.502+0000 7fe1e9fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1e410cec0 0x7fe1e41986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.503+0000 7fe1e9fff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e4198d80 con 0x7fe1e4074ac0 2023-11-16T17:00:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.503+0000 7fe1e9fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e4198ef0 con 0x7fe1e410cec0 2023-11-16T17:00:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.503+0000 7fe1e9fff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e41990b0 con 0x7fe1e40746f0 2023-11-16T17:00:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.503+0000 7fe1e2ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1e4074ac0 0x7fe1e419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.503+0000 7fe1e2ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1e4074ac0 0x7fe1e419eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49660/0 (socket says 172.21.15.44:49660) 2023-11-16T17:00:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.503+0000 7fe1e2ffd700 1 -- 172.21.15.44:0/208007146 learned_addr learned my addr 172.21.15.44:0/208007146 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:18.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.504+0000 7fe1e2ffd700 1 -- 172.21.15.44:0/208007146 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1e40746f0 msgr2=0x7fe1e419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:18.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.504+0000 7fe1e2ffd700 1 --2- 172.21.15.44:0/208007146 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1e40746f0 0x7fe1e419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.504+0000 7fe1e2ffd700 1 -- 172.21.15.44:0/208007146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1e410cec0 msgr2=0x7fe1e41986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:18.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.504+0000 7fe1e2ffd700 1 --2- 172.21.15.44:0/208007146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1e410cec0 0x7fe1e41986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.507 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.504+0000 7fe1e2ffd700 1 -- 172.21.15.44:0/208007146 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1cc0095f0 con 0x7fe1e4074ac0 2023-11-16T17:00:18.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.506+0000 7fe1e2ffd700 1 --2- 172.21.15.44:0/208007146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1e4074ac0 0x7fe1e419eb80 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe1d400b500 tx=0x7fe1d400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:18.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.506+0000 7fe1e0ff9700 1 -- 172.21.15.44:0/208007146 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4011830 con 0x7fe1e4074ac0 2023-11-16T17:00:18.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.506+0000 7fe1e0ff9700 1 -- 172.21.15.44:0/208007146 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe1d4011e50 con 0x7fe1e4074ac0 2023-11-16T17:00:18.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.506+0000 7fe1e9fff700 1 -- 172.21.15.44:0/208007146 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e4112720 con 0x7fe1e4074ac0 2023-11-16T17:00:18.508 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.507+0000 7fe1e9fff700 1 -- 172.21.15.44:0/208007146 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e4112590 con 0x7fe1e4074ac0 2023-11-16T17:00:18.509 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.507+0000 7fe1e0ff9700 1 -- 172.21.15.44:0/208007146 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d400f2e0 con 0x7fe1e4074ac0 2023-11-16T17:00:18.510 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.508+0000 7fe1e9fff700 1 -- 172.21.15.44:0/208007146 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c8005310 con 0x7fe1e4074ac0 2023-11-16T17:00:18.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.518+0000 7fe1e0ff9700 1 -- 172.21.15.44:0/208007146 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fe1d4011990 con 0x7fe1e4074ac0 2023-11-16T17:00:18.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.523+0000 7fe1e0ff9700 1 --2- 172.21.15.44:0/208007146 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe1c40a1380 0x7fe1c40a3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:18.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.524+0000 7fe1e0ff9700 1 -- 172.21.15.44:0/208007146 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9599+0+0 (secure 0 0 0) 0x7fe1d40ccc30 con 0x7fe1e4074ac0 2023-11-16T17:00:18.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.524+0000 7fe1e0ff9700 1 -- 172.21.15.44:0/208007146 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe1d40cd110 con 0x7fe1e4074ac0 2023-11-16T17:00:18.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.525+0000 7fe1e37fe700 1 --2- 172.21.15.44:0/208007146 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe1c40a1380 0x7fe1c40a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:18.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.526+0000 7fe1e37fe700 1 --2- 172.21.15.44:0/208007146 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe1c40a1380 0x7fe1c40a3830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc005cb0 tx=0x7fe1cc005c00 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:18.928 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.926+0000 7fe1e9fff700 1 -- 172.21.15.44:0/208007146 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe1c8005f60 con 0x7fe1e4074ac0 2023-11-16T17:00:18.929 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.927+0000 7fe1e0ff9700 1 -- 172.21.15.44:0/208007146 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe1d40150a0 con 0x7fe1e4074ac0 2023-11-16T17:00:18.929 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2023-11-16T17:00:18.932 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.930+0000 7fe1e9fff700 1 -- 172.21.15.44:0/208007146 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe1c40a1380 msgr2=0x7fe1c40a3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:18.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.931+0000 7fe1e9fff700 1 --2- 172.21.15.44:0/208007146 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe1c40a1380 0x7fe1c40a3830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc005cb0 tx=0x7fe1cc005c00 comp rx=0 tx=0).stop 2023-11-16T17:00:18.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.931+0000 7fe1e9fff700 1 -- 172.21.15.44:0/208007146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1e4074ac0 msgr2=0x7fe1e419eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:18.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.931+0000 7fe1e9fff700 1 --2- 172.21.15.44:0/208007146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1e4074ac0 0x7fe1e419eb80 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe1d400b500 tx=0x7fe1d400b8c0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.931+0000 7fe1e9fff700 1 -- 172.21.15.44:0/208007146 shutdown_connections 2023-11-16T17:00:18.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.931+0000 7fe1e9fff700 1 --2- 172.21.15.44:0/208007146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1e410cec0 0x7fe1e41986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.931+0000 7fe1e9fff700 1 --2- 172.21.15.44:0/208007146 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fe1c40a1380 0x7fe1c40a3830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.931+0000 7fe1e9fff700 1 --2- 172.21.15.44:0/208007146 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1e40746f0 0x7fe1e419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.933 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.931+0000 7fe1e9fff700 1 --2- 172.21.15.44:0/208007146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1e4074ac0 0x7fe1e419eb80 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:18.934 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.931+0000 7fe1e9fff700 1 -- 172.21.15.44:0/208007146 >> 172.21.15.44:0/208007146 conn(0x7fe1e4077560 msgr2=0x7fe1e4104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:18.934 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.931+0000 7fe1e9fff700 1 -- 172.21.15.44:0/208007146 shutdown_connections 2023-11-16T17:00:18.934 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:18.931+0000 7fe1e9fff700 1 -- 172.21.15.44:0/208007146 wait complete. 2023-11-16T17:00:18.967 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T17:00:19.096 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:18 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/426338703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T17:00:19.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:18 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/426338703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T17:00:19.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:18 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/426338703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T17:00:19.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.356+0000 7f7ada9ff700 1 -- 172.21.15.44:0/3928092196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ad41063a0 msgr2=0x7f7ad4106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:19.358 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.356+0000 7f7ada9ff700 1 --2- 172.21.15.44:0/3928092196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ad41063a0 0x7f7ad4106770 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400b0a0 tx=0x7f7ac400b3b0 comp rx=0 tx=0).stop 2023-11-16T17:00:19.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.356+0000 7f7ada9ff700 1 -- 172.21.15.44:0/3928092196 shutdown_connections 2023-11-16T17:00:19.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.356+0000 7f7ada9ff700 1 --2- 172.21.15.44:0/3928092196 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ad40feb20 0x7f7ad410d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:19.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.356+0000 7f7ada9ff700 1 --2- 172.21.15.44:0/3928092196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ad41063a0 0x7f7ad4106770 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:19.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.356+0000 7f7ada9ff700 1 --2- 172.21.15.44:0/3928092196 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad4106cb0 0x7f7ad40fe3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:19.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.356+0000 7f7ada9ff700 1 -- 172.21.15.44:0/3928092196 >> 172.21.15.44:0/3928092196 conn(0x7f7ad4075460 msgr2=0x7f7ad4075860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:19.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.357+0000 7f7ada9ff700 1 -- 172.21.15.44:0/3928092196 shutdown_connections 2023-11-16T17:00:19.359 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.357+0000 7f7ada9ff700 1 -- 172.21.15.44:0/3928092196 wait complete. 2023-11-16T17:00:19.362 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.358+0000 7f7ada9ff700 1 Processor -- start 2023-11-16T17:00:19.362 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.361+0000 7f7ada9ff700 1 -- start start 2023-11-16T17:00:19.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.361+0000 7f7ada9ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad40feb20 0x7f7ad41991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:19.363 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.362+0000 7f7ada9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ad41063a0 0x7f7ad41996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:19.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.362+0000 7f7ada9ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ad4106cb0 0x7f7ad410bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:19.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.362+0000 7f7ada9ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad410c570 con 0x7f7ad40feb20 2023-11-16T17:00:19.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.362+0000 7f7ada9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad410c6e0 con 0x7f7ad41063a0 2023-11-16T17:00:19.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.362+0000 7f7ada9ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad410c8a0 con 0x7f7ad4106cb0 2023-11-16T17:00:19.364 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.362+0000 7f7ad8f9c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ad4106cb0 0x7f7ad410bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:19.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.362+0000 7f7acb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ad41063a0 0x7f7ad41996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:19.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.362+0000 7f7ad3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad40feb20 0x7f7ad41991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:19.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.363+0000 7f7ad3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad40feb20 0x7f7ad41991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49674/0 (socket says 172.21.15.44:49674) 2023-11-16T17:00:19.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.363+0000 7f7ad3fff700 1 -- 172.21.15.44:0/2750261815 learned_addr learned my addr 172.21.15.44:0/2750261815 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:19.366 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.363+0000 7f7ad3fff700 1 -- 172.21.15.44:0/2750261815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ad4106cb0 msgr2=0x7f7ad410bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:19.366 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.363+0000 7f7ad3fff700 1 --2- 172.21.15.44:0/2750261815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ad4106cb0 0x7f7ad410bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:19.366 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.363+0000 7f7ad3fff700 1 -- 172.21.15.44:0/2750261815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ad41063a0 msgr2=0x7f7ad41996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:19.366 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.363+0000 7f7ad3fff700 1 --2- 172.21.15.44:0/2750261815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ad41063a0 0x7f7ad41996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:19.366 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.363+0000 7f7ad3fff700 1 -- 172.21.15.44:0/2750261815 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac4009d00 con 0x7f7ad40feb20 2023-11-16T17:00:19.366 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.364+0000 7f7ad3fff700 1 --2- 172.21.15.44:0/2750261815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad40feb20 0x7f7ad41991a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400b070 tx=0x7f7ac400bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:19.367 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.365+0000 7f7ad1ffb700 1 -- 172.21.15.44:0/2750261815 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac400e040 con 0x7f7ad40feb20 2023-11-16T17:00:19.367 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.365+0000 7f7ad1ffb700 1 -- 172.21.15.44:0/2750261815 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ac4007930 con 0x7f7ad40feb20 2023-11-16T17:00:19.367 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.365+0000 7f7ad1ffb700 1 -- 172.21.15.44:0/2750261815 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac401b6a0 con 0x7f7ad40feb20 2023-11-16T17:00:19.367 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.365+0000 7f7ada9ff700 1 -- 172.21.15.44:0/2750261815 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad41103c0 con 0x7f7ad40feb20 2023-11-16T17:00:19.368 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.365+0000 7f7ada9ff700 1 -- 172.21.15.44:0/2750261815 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad4110230 con 0x7f7ad40feb20 2023-11-16T17:00:19.368 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.367+0000 7f7ada9ff700 1 -- 172.21.15.44:0/2750261815 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad4110550 con 0x7f7ad40feb20 2023-11-16T17:00:19.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.371+0000 7f7ad1ffb700 1 -- 172.21.15.44:0/2750261815 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f7ac401b800 con 0x7f7ad40feb20 2023-11-16T17:00:19.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.374+0000 7f7ad1ffb700 1 --2- 172.21.15.44:0/2750261815 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7ac00a1490 0x7f7ac00a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:19.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.374+0000 7f7ad1ffb700 1 -- 172.21.15.44:0/2750261815 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f7ac40ce800 con 0x7f7ad40feb20 2023-11-16T17:00:19.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.375+0000 7f7acb7fe700 1 --2- 172.21.15.44:0/2750261815 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7ac00a1490 0x7f7ac00a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:19.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.376+0000 7f7acb7fe700 1 --2- 172.21.15.44:0/2750261815 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7ac00a1490 0x7f7ac00a3940 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7abc006fd0 tx=0x7f7abc008040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:19.385 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.383+0000 7f7ad1ffb700 1 -- 172.21.15.44:0/2750261815 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7ac4097810 con 0x7f7ad40feb20 2023-11-16T17:00:19.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.792+0000 7f7ada9ff700 1 -- 172.21.15.44:0/2750261815 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f7ad410d350 con 0x7f7ad40feb20 2023-11-16T17:00:19.795 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.793+0000 7f7ad1ffb700 1 -- 172.21.15.44:0/2750261815 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v101) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7ac409ae30 con 0x7f7ad40feb20 2023-11-16T17:00:19.795 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T17:00:19.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.797+0000 7f7ada9ff700 1 -- 172.21.15.44:0/2750261815 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7ac00a1490 msgr2=0x7f7ac00a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:19.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.797+0000 7f7ada9ff700 1 --2- 172.21.15.44:0/2750261815 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7ac00a1490 0x7f7ac00a3940 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7abc006fd0 tx=0x7f7abc008040 comp rx=0 tx=0).stop 2023-11-16T17:00:19.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.797+0000 7f7ada9ff700 1 -- 172.21.15.44:0/2750261815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad40feb20 msgr2=0x7f7ad41991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:19.799 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.797+0000 7f7ada9ff700 1 --2- 172.21.15.44:0/2750261815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad40feb20 0x7f7ad41991a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400b070 tx=0x7f7ac400bce0 comp rx=0 tx=0).stop 2023-11-16T17:00:19.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.797+0000 7f7ada9ff700 1 -- 172.21.15.44:0/2750261815 shutdown_connections 2023-11-16T17:00:19.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.797+0000 7f7ada9ff700 1 --2- 172.21.15.44:0/2750261815 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f7ac00a1490 0x7f7ac00a3940 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:19.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.797+0000 7f7ada9ff700 1 --2- 172.21.15.44:0/2750261815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ad4106cb0 0x7f7ad410bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:19.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.797+0000 7f7ada9ff700 1 --2- 172.21.15.44:0/2750261815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad40feb20 0x7f7ad41991a0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:19.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.797+0000 7f7ada9ff700 1 --2- 172.21.15.44:0/2750261815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ad41063a0 0x7f7ad41996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:19.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.797+0000 7f7ada9ff700 1 -- 172.21.15.44:0/2750261815 >> 172.21.15.44:0/2750261815 conn(0x7f7ad4075460 msgr2=0x7f7ad4102080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:19.800 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.798+0000 7f7ada9ff700 1 -- 172.21.15.44:0/2750261815 shutdown_connections 2023-11-16T17:00:19.801 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:19.798+0000 7f7ada9ff700 1 -- 172.21.15.44:0/2750261815 wait complete. 2023-11-16T17:00:19.835 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2023-11-16T17:00:20.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:19 smithi044 ceph-mon[128750]: pgmap v323: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 201 B/s wr, 2 op/s 2023-11-16T17:00:20.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:19 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/208007146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T17:00:20.148 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:19 smithi176 ceph-mon[131630]: pgmap v323: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 201 B/s wr, 2 op/s 2023-11-16T17:00:20.148 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:19 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/208007146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T17:00:20.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.214+0000 7f3f03dff700 1 -- 172.21.15.44:0/962222249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3efc100eb0 msgr2=0x7f3efc104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:20.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.214+0000 7f3f03dff700 1 --2- 172.21.15.44:0/962222249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3efc100eb0 0x7f3efc104e70 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3eec0099c0 tx=0x7f3eec009cd0 comp rx=0 tx=0).stop 2023-11-16T17:00:20.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.215+0000 7f3f03dff700 1 -- 172.21.15.44:0/962222249 shutdown_connections 2023-11-16T17:00:20.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.215+0000 7f3f03dff700 1 --2- 172.21.15.44:0/962222249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3efc105550 0x7f3efc10d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:20.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.215+0000 7f3f03dff700 1 --2- 172.21.15.44:0/962222249 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3efc100510 0x7f3efc1008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:20.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.215+0000 7f3f03dff700 1 --2- 172.21.15.44:0/962222249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3efc100eb0 0x7f3efc104e70 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:20.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.215+0000 7f3f03dff700 1 -- 172.21.15.44:0/962222249 >> 172.21.15.44:0/962222249 conn(0x7f3efc077050 msgr2=0x7f3efc077450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:20.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.217+0000 7f3f03dff700 1 -- 172.21.15.44:0/962222249 shutdown_connections 2023-11-16T17:00:20.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.217+0000 7f3f03dff700 1 -- 172.21.15.44:0/962222249 wait complete. 2023-11-16T17:00:20.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.218+0000 7f3f03dff700 1 Processor -- start 2023-11-16T17:00:20.221 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.219+0000 7f3f03dff700 1 -- start start 2023-11-16T17:00:20.221 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.219+0000 7f3f03dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3efc100510 0x7f3efc113400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:20.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.219+0000 7f3f03dff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3efc100eb0 0x7f3efc10f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:20.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.219+0000 7f3f03dff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3efc105550 0x7f3efc10fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:20.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.219+0000 7f3f03dff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3efc110410 con 0x7f3efc105550 2023-11-16T17:00:20.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.219+0000 7f3f03dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3efc110580 con 0x7f3efc100510 2023-11-16T17:00:20.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.219+0000 7f3f03dff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3efc1b1900 con 0x7f3efc100eb0 2023-11-16T17:00:20.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.220+0000 7f3f01b9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3efc100510 0x7f3efc113400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:20.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.220+0000 7f3f0139a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3efc100eb0 0x7f3efc10f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:20.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.220+0000 7f3f0139a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3efc100eb0 0x7f3efc10f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:49860/0 (socket says 172.21.15.44:49860) 2023-11-16T17:00:20.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.220+0000 7f3f01b9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3efc100510 0x7f3efc113400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:55648/0 (socket says 172.21.15.44:55648) 2023-11-16T17:00:20.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.220+0000 7f3f0139a700 1 -- 172.21.15.44:0/1933594234 learned_addr learned my addr 172.21.15.44:0/1933594234 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:00:20.223 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.220+0000 7f3f0239c700 1 --2- 172.21.15.44:0/1933594234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3efc105550 0x7f3efc10fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:20.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.220+0000 7f3f0139a700 1 -- 172.21.15.44:0/1933594234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3efc100510 msgr2=0x7f3efc113400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:20.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.220+0000 7f3f0139a700 1 --2- 172.21.15.44:0/1933594234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3efc100510 0x7f3efc113400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:20.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.220+0000 7f3f0139a700 1 -- 172.21.15.44:0/1933594234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3efc105550 msgr2=0x7f3efc10fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:20.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.220+0000 7f3f0139a700 1 --2- 172.21.15.44:0/1933594234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3efc105550 0x7f3efc10fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:20.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.220+0000 7f3f0139a700 1 -- 172.21.15.44:0/1933594234 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eec0095f0 con 0x7f3efc100eb0 2023-11-16T17:00:20.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.221+0000 7f3f0139a700 1 --2- 172.21.15.44:0/1933594234 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3efc100eb0 0x7f3efc10f780 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3eec0098e0 tx=0x7f3eec00bfb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:20.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.223+0000 7f3efaffd700 1 -- 172.21.15.44:0/1933594234 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eec01d070 con 0x7f3efc100eb0 2023-11-16T17:00:20.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.223+0000 7f3efaffd700 1 -- 172.21.15.44:0/1933594234 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3eec004150 con 0x7f3efc100eb0 2023-11-16T17:00:20.225 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.223+0000 7f3efaffd700 1 -- 172.21.15.44:0/1933594234 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eec00f460 con 0x7f3efc100eb0 2023-11-16T17:00:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.224+0000 7f3f03dff700 1 -- 172.21.15.44:0/1933594234 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3efc0739e0 con 0x7f3efc100eb0 2023-11-16T17:00:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.224+0000 7f3f03dff700 1 -- 172.21.15.44:0/1933594234 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3efc073850 con 0x7f3efc100eb0 2023-11-16T17:00:20.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:19 smithi077 ceph-mon[131204]: pgmap v323: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 201 B/s wr, 2 op/s 2023-11-16T17:00:20.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:19 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/208007146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T17:00:20.228 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.226+0000 7f3ef8ff9700 1 -- 172.21.15.44:0/1933594234 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee00052e0 con 0x7f3efc100eb0 2023-11-16T17:00:20.232 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.231+0000 7f3efaffd700 1 -- 172.21.15.44:0/1933594234 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f3eec022650 con 0x7f3efc100eb0 2023-11-16T17:00:20.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.233+0000 7f3efaffd700 1 --2- 172.21.15.44:0/1933594234 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3ee80a1340 0x7f3ee80a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:20.235 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.233+0000 7f3efaffd700 1 -- 172.21.15.44:0/1933594234 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9599+0+0 (secure 0 0 0) 0x7f3eec0ce1c0 con 0x7f3efc100eb0 2023-11-16T17:00:20.236 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.235+0000 7f3f01b9b700 1 --2- 172.21.15.44:0/1933594234 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3ee80a1340 0x7f3ee80a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:20.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.235+0000 7f3efaffd700 1 -- 172.21.15.44:0/1933594234 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3eec097230 con 0x7f3efc100eb0 2023-11-16T17:00:20.237 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.236+0000 7f3f01b9b700 1 --2- 172.21.15.44:0/1933594234 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3ee80a1340 0x7f3ee80a37f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4009f40 tx=0x7f3ee40094d0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:20.768 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.766+0000 7f3ef8ff9700 1 -- 172.21.15.44:0/1933594234 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f3ee0005150 con 0x7f3efc100eb0 2023-11-16T17:00:20.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.780+0000 7f3efaffd700 1 -- 172.21.15.44:0/1933594234 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) v1 ==== 81+0+733775 (secure 0 0 0) 0x7f3eec09a850 con 0x7f3efc100eb0 2023-11-16T17:00:20.793 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T17:00:20.794 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T17:00:20.794 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 25, 2023-11-16T17:00:20.794 INFO:teuthology.orchestra.run.smithi044.stdout: "active_gid": 25447, 2023-11-16T17:00:20.794 INFO:teuthology.orchestra.run.smithi044.stdout: "active_name": "x", 2023-11-16T17:00:20.794 INFO:teuthology.orchestra.run.smithi044.stdout: "active_addrs": { 2023-11-16T17:00:20.794 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2023-11-16T17:00:20.795 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:20.795 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2023-11-16T17:00:20.795 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:6800", 2023-11-16T17:00:20.795 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 3382499657 2023-11-16T17:00:20.795 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.795 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:20.795 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2023-11-16T17:00:20.796 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:6801", 2023-11-16T17:00:20.796 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 3382499657 2023-11-16T17:00:20.796 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:20.796 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2023-11-16T17:00:20.796 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.796 INFO:teuthology.orchestra.run.smithi044.stdout: "active_addr": "172.21.15.44:6801/3382499657", 2023-11-16T17:00:20.796 INFO:teuthology.orchestra.run.smithi044.stdout: "active_change": "2023-11-16T16:49:55.607750+0000", 2023-11-16T17:00:20.796 INFO:teuthology.orchestra.run.smithi044.stdout: "active_mgr_features": 4540138322906710015, 2023-11-16T17:00:20.796 INFO:teuthology.orchestra.run.smithi044.stdout: "available": true, 2023-11-16T17:00:20.797 INFO:teuthology.orchestra.run.smithi044.stdout: "standbys": [ 2023-11-16T17:00:20.797 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:20.797 INFO:teuthology.orchestra.run.smithi044.stdout: "gid": 25441, 2023-11-16T17:00:20.797 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "z", 2023-11-16T17:00:20.797 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr_features": 4540138322906710015, 2023-11-16T17:00:20.797 INFO:teuthology.orchestra.run.smithi044.stdout: "available_modules": [ 2023-11-16T17:00:20.797 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:20.797 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "alerts", 2023-11-16T17:00:20.798 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:20.798 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:20.798 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:20.798 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2023-11-16T17:00:20.798 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2023-11-16T17:00:20.798 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:20.798 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.798 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.799 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2023-11-16T17:00:20.799 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.799 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.799 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.799 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How frequently to reexamine health status", 2023-11-16T17:00:20.800 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.800 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.800 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.800 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.800 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:20.800 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:20.800 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.800 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.800 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.801 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.801 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.801 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.801 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:20.801 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:20.801 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:20.801 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:20.801 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:20.802 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:20.802 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:20.802 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:20.802 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.802 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.802 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.802 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.803 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.803 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:20.803 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:20.803 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.803 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.803 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.803 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:20.803 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.803 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.804 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.804 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.804 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.804 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.804 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.804 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.804 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:20.804 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:20.805 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.805 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.805 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.805 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:20.805 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.805 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.805 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:20.805 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:20.805 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:20.806 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:20.806 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:20.806 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:20.806 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:20.806 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:20.806 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.806 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.807 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.807 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.807 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.807 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:20.807 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:20.807 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.807 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.807 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.807 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:20.808 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.808 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.808 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.808 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.808 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.808 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.808 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.808 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.809 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_destination": { 2023-11-16T17:00:20.809 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_destination", 2023-11-16T17:00:20.809 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.809 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.809 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.809 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.809 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.809 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.810 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.810 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email address to send alerts to", 2023-11-16T17:00:20.810 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.810 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.810 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.810 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.810 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_from_name": { 2023-11-16T17:00:20.811 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_from_name", 2023-11-16T17:00:20.811 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.811 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.811 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.811 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "Ceph", 2023-11-16T17:00:20.811 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.811 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.811 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.812 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email From: name", 2023-11-16T17:00:20.812 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.812 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.812 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.812 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.812 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_host": { 2023-11-16T17:00:20.812 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_host", 2023-11-16T17:00:20.812 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.812 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.813 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.813 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.813 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.813 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.813 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.813 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP server", 2023-11-16T17:00:20.813 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.813 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.813 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.814 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.814 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_password": { 2023-11-16T17:00:20.814 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_password", 2023-11-16T17:00:20.814 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.814 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.814 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.814 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.815 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.815 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.815 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.815 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Password to authenticate with", 2023-11-16T17:00:20.815 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.815 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.815 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.815 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.816 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_port": { 2023-11-16T17:00:20.816 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_port", 2023-11-16T17:00:20.816 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:20.816 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.816 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.816 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "465", 2023-11-16T17:00:20.816 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.816 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.816 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.817 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP port", 2023-11-16T17:00:20.817 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.817 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.817 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.817 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.817 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_sender": { 2023-11-16T17:00:20.817 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_sender", 2023-11-16T17:00:20.817 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.818 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.818 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.818 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.818 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.818 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.818 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.818 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP envelope sender", 2023-11-16T17:00:20.818 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.818 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.819 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.819 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.819 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_ssl": { 2023-11-16T17:00:20.819 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_ssl", 2023-11-16T17:00:20.819 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.819 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.819 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.820 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.820 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.820 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.820 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.820 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use SSL to connect to SMTP server", 2023-11-16T17:00:20.820 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.820 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.820 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.821 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.821 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_user": { 2023-11-16T17:00:20.821 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_user", 2023-11-16T17:00:20.821 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.821 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.821 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.821 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.821 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.821 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.822 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.822 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "User to authenticate as", 2023-11-16T17:00:20.822 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.822 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.822 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.822 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:20.822 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:20.823 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.823 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:20.823 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "balancer", 2023-11-16T17:00:20.823 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:20.823 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:20.823 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:20.823 INFO:teuthology.orchestra.run.smithi044.stdout: "active": { 2023-11-16T17:00:20.823 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "active", 2023-11-16T17:00:20.824 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.824 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.824 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.824 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.824 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.824 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.824 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.824 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically balance PGs across cluster", 2023-11-16T17:00:20.825 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.825 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.825 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.825 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.825 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_time": { 2023-11-16T17:00:20.825 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_time", 2023-11-16T17:00:20.825 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.825 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.825 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.826 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0000", 2023-11-16T17:00:20.826 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.826 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.826 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.826 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "beginning time of day to automatically balance", 2023-11-16T17:00:20.826 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T17:00:20.826 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.827 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.827 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.827 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_weekday": { 2023-11-16T17:00:20.827 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_weekday", 2023-11-16T17:00:20.827 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2023-11-16T17:00:20.827 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.827 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.827 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2023-11-16T17:00:20.828 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2023-11-16T17:00:20.828 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2023-11-16T17:00:20.828 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.828 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-11-16T17:00:20.828 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T17:00:20.828 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.828 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.828 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.828 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_max_iterations": { 2023-11-16T17:00:20.829 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_max_iterations", 2023-11-16T17:00:20.829 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2023-11-16T17:00:20.829 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.829 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.829 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "25", 2023-11-16T17:00:20.829 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2023-11-16T17:00:20.829 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "250", 2023-11-16T17:00:20.830 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.830 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-11-16T17:00:20.830 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.830 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.830 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.830 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.830 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_metrics": { 2023-11-16T17:00:20.830 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_metrics", 2023-11-16T17:00:20.831 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.831 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.831 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.831 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "pgs,objects,bytes", 2023-11-16T17:00:20.831 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.831 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.831 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.831 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-11-16T17:00:20.832 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-11-16T17:00:20.832 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.832 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.832 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.832 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_step": { 2023-11-16T17:00:20.832 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_step", 2023-11-16T17:00:20.832 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:20.832 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.833 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.833 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.5", 2023-11-16T17:00:20.833 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0.001", 2023-11-16T17:00:20.833 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "0.999", 2023-11-16T17:00:20.833 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.833 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "aggressiveness of optimization", 2023-11-16T17:00:20.833 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-11-16T17:00:20.833 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.833 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.834 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.834 INFO:teuthology.orchestra.run.smithi044.stdout: "end_time": { 2023-11-16T17:00:20.834 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_time", 2023-11-16T17:00:20.834 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.834 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.834 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.834 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2359", 2023-11-16T17:00:20.835 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.835 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.835 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.835 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ending time of day to automatically balance", 2023-11-16T17:00:20.835 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T17:00:20.835 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.835 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.835 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.836 INFO:teuthology.orchestra.run.smithi044.stdout: "end_weekday": { 2023-11-16T17:00:20.836 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_weekday", 2023-11-16T17:00:20.836 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2023-11-16T17:00:20.836 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.836 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.836 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2023-11-16T17:00:20.836 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2023-11-16T17:00:20.836 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2023-11-16T17:00:20.836 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.837 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-11-16T17:00:20.837 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T17:00:20.837 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.837 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.837 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.837 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:20.837 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:20.837 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.838 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.838 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.838 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.838 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.838 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.838 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:20.838 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:20.839 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:20.839 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:20.839 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:20.839 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:20.839 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:20.839 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:20.839 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.839 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.840 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.840 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.840 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.840 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:20.840 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:20.840 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.840 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.840 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.840 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:20.841 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.841 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.841 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.841 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.841 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.841 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.841 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.841 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.842 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:20.842 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:20.842 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.842 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.842 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.842 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:20.842 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.842 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.843 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:20.843 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:20.843 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:20.843 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:20.843 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:20.843 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:20.843 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:20.843 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:20.844 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.844 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.844 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.844 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.844 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.844 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:20.844 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:20.844 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.844 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.845 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.845 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:20.845 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.845 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.845 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.845 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.845 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.845 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.846 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.846 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.846 INFO:teuthology.orchestra.run.smithi044.stdout: "min_score": { 2023-11-16T17:00:20.846 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_score", 2023-11-16T17:00:20.846 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:20.846 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.846 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.846 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2023-11-16T17:00:20.847 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.847 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.847 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.847 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-11-16T17:00:20.847 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.847 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.847 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.847 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.848 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2023-11-16T17:00:20.848 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2023-11-16T17:00:20.848 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.848 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.848 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.848 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "upmap", 2023-11-16T17:00:20.848 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.848 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.848 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:20.849 INFO:teuthology.orchestra.run.smithi044.stdout: "crush-compat", 2023-11-16T17:00:20.849 INFO:teuthology.orchestra.run.smithi044.stdout: "none", 2023-11-16T17:00:20.849 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap" 2023-11-16T17:00:20.849 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:20.849 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Balancer mode", 2023-11-16T17:00:20.849 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.849 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.849 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.850 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.850 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_ids": { 2023-11-16T17:00:20.850 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_ids", 2023-11-16T17:00:20.850 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.850 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.850 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.850 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.850 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.851 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.851 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.851 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-11-16T17:00:20.851 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.851 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.851 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.851 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.851 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2023-11-16T17:00:20.852 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2023-11-16T17:00:20.852 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:20.852 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.852 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.852 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2023-11-16T17:00:20.852 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.852 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.852 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.853 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-11-16T17:00:20.853 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.853 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.853 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.853 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.853 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_deviation": { 2023-11-16T17:00:20.853 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_deviation", 2023-11-16T17:00:20.853 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:20.853 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.854 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.854 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2023-11-16T17:00:20.854 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2023-11-16T17:00:20.854 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.854 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.854 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "deviation below which no optimization is attempted", 2023-11-16T17:00:20.854 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-11-16T17:00:20.855 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.855 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.855 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.855 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_optimizations": { 2023-11-16T17:00:20.855 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_optimizations", 2023-11-16T17:00:20.855 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2023-11-16T17:00:20.855 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.855 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.856 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:20.856 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.856 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.856 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.856 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-11-16T17:00:20.856 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.856 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.856 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.856 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:20.857 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:20.857 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.857 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:20.857 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cephadm", 2023-11-16T17:00:20.857 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:20.857 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:20.857 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:20.857 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_down_multiplier": { 2023-11-16T17:00:20.857 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_down_multiplier", 2023-11-16T17:00:20.858 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:20.858 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.858 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.858 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2023-11-16T17:00:20.858 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.858 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.858 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.859 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-11-16T17:00:20.859 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.859 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.859 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.859 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.859 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_refresh_rate": { 2023-11-16T17:00:20.859 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_refresh_rate", 2023-11-16T17:00:20.860 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:20.860 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.860 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.860 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "20", 2023-11-16T17:00:20.860 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.860 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.860 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.860 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-11-16T17:00:20.860 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.861 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.861 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.861 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.861 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_starting_port": { 2023-11-16T17:00:20.861 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_starting_port", 2023-11-16T17:00:20.861 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:20.861 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.861 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.861 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4721", 2023-11-16T17:00:20.861 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.862 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.862 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.862 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-11-16T17:00:20.862 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.862 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.862 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.862 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.863 INFO:teuthology.orchestra.run.smithi044.stdout: "alertmanager_web_password": { 2023-11-16T17:00:20.863 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "alertmanager_web_password", 2023-11-16T17:00:20.863 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.863 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.863 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.863 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:20.863 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.863 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.863 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.864 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Alertmanager web password", 2023-11-16T17:00:20.864 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.864 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.864 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.864 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.864 INFO:teuthology.orchestra.run.smithi044.stdout: "alertmanager_web_user": { 2023-11-16T17:00:20.864 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "alertmanager_web_user", 2023-11-16T17:00:20.864 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.865 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.865 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.865 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:20.865 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.865 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.865 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.865 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Alertmanager web user", 2023-11-16T17:00:20.865 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.865 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.866 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.866 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.866 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_ptrace": { 2023-11-16T17:00:20.866 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_ptrace", 2023-11-16T17:00:20.866 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.866 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.866 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.867 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:20.867 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.867 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.867 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.867 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-11-16T17:00:20.867 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-11-16T17:00:20.867 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.867 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.868 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.868 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_interval": { 2023-11-16T17:00:20.868 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_interval", 2023-11-16T17:00:20.868 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:20.868 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.868 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.868 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2023-11-16T17:00:20.868 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.868 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.869 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.869 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to autotune daemon memory", 2023-11-16T17:00:20.869 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.869 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.869 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.869 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.869 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_memory_target_ratio": { 2023-11-16T17:00:20.870 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_memory_target_ratio", 2023-11-16T17:00:20.870 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:20.870 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.870 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.870 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.7", 2023-11-16T17:00:20.870 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.870 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.870 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.871 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-11-16T17:00:20.871 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.871 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.871 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.871 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.871 INFO:teuthology.orchestra.run.smithi044.stdout: "cgroups_split": { 2023-11-16T17:00:20.871 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cgroups_split", 2023-11-16T17:00:20.871 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.872 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.872 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.872 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.872 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.872 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.872 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.872 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-11-16T17:00:20.872 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.873 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.873 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.873 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.873 INFO:teuthology.orchestra.run.smithi044.stdout: "config_checks_enabled": { 2023-11-16T17:00:20.873 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_checks_enabled", 2023-11-16T17:00:20.873 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.873 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.873 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.873 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:20.874 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.874 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.874 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.874 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-11-16T17:00:20.874 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.874 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.874 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.875 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.875 INFO:teuthology.orchestra.run.smithi044.stdout: "config_dashboard": { 2023-11-16T17:00:20.875 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_dashboard", 2023-11-16T17:00:20.875 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.875 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.875 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.875 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.875 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.876 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.876 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.876 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-11-16T17:00:20.876 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.876 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.876 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.876 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.876 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_alertmanager": { 2023-11-16T17:00:20.877 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_alertmanager", 2023-11-16T17:00:20.877 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.877 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.877 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.877 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-11-16T17:00:20.877 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.877 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.877 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.877 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:20.878 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.878 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.878 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.878 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.878 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_base": { 2023-11-16T17:00:20.878 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_base", 2023-11-16T17:00:20.878 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.879 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.879 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.879 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/ceph", 2023-11-16T17:00:20.879 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.879 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.879 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.879 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Container image name, without the tag", 2023-11-16T17:00:20.879 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.880 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.880 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.880 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.880 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_elasticsearch": { 2023-11-16T17:00:20.880 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_elasticsearch", 2023-11-16T17:00:20.880 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.880 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.880 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.881 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-11-16T17:00:20.881 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.881 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.881 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.881 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "elasticsearch container image", 2023-11-16T17:00:20.881 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.881 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.881 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.881 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.882 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_grafana": { 2023-11-16T17:00:20.882 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_grafana", 2023-11-16T17:00:20.882 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.882 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.882 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.882 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-11-16T17:00:20.882 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.883 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.883 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.883 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:20.883 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.883 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.883 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.883 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.883 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_haproxy": { 2023-11-16T17:00:20.884 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_haproxy", 2023-11-16T17:00:20.884 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.884 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.884 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.884 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-11-16T17:00:20.884 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.884 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.884 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.884 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "HAproxy container image", 2023-11-16T17:00:20.885 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.885 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.885 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.885 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.885 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_agent": { 2023-11-16T17:00:20.885 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_agent", 2023-11-16T17:00:20.885 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.885 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.886 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.886 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-11-16T17:00:20.886 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.886 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.886 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.886 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger agent container image", 2023-11-16T17:00:20.886 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.886 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.887 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.887 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.887 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_collector": { 2023-11-16T17:00:20.887 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_collector", 2023-11-16T17:00:20.887 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.887 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.887 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.887 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-11-16T17:00:20.888 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.888 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.888 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.888 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger collector container image", 2023-11-16T17:00:20.888 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.888 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.888 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.889 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.889 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_query": { 2023-11-16T17:00:20.889 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_query", 2023-11-16T17:00:20.889 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.889 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.889 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.889 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-11-16T17:00:20.889 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.890 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.890 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.890 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger query container image", 2023-11-16T17:00:20.890 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.890 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.890 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.890 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.891 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_keepalived": { 2023-11-16T17:00:20.891 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_keepalived", 2023-11-16T17:00:20.891 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.891 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.891 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.891 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-11-16T17:00:20.891 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.891 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.892 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.892 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Keepalived container image", 2023-11-16T17:00:20.892 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.892 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.892 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.892 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.892 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_loki": { 2023-11-16T17:00:20.892 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_loki", 2023-11-16T17:00:20.892 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.893 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.893 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.893 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-11-16T17:00:20.893 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.893 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.893 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.893 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Loki container image", 2023-11-16T17:00:20.893 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.894 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.894 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.894 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.894 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_node_exporter": { 2023-11-16T17:00:20.894 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_node_exporter", 2023-11-16T17:00:20.894 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.894 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.894 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.895 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-11-16T17:00:20.895 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.895 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.895 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.895 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:20.895 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.895 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.895 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.896 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.896 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_prometheus": { 2023-11-16T17:00:20.896 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_prometheus", 2023-11-16T17:00:20.896 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.896 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.896 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.896 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-11-16T17:00:20.896 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.897 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.897 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.897 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:20.897 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.897 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.897 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.897 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.897 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_promtail": { 2023-11-16T17:00:20.898 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_promtail", 2023-11-16T17:00:20.898 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.898 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.898 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.898 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-11-16T17:00:20.898 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.898 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.898 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.899 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Promtail container image", 2023-11-16T17:00:20.899 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.899 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.899 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.899 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.899 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_snmp_gateway": { 2023-11-16T17:00:20.900 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_snmp_gateway", 2023-11-16T17:00:20.900 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.900 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.900 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.900 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-11-16T17:00:20.900 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.900 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.900 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.901 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SNMP Gateway container image", 2023-11-16T17:00:20.901 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.901 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.901 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.901 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.901 INFO:teuthology.orchestra.run.smithi044.stdout: "container_init": { 2023-11-16T17:00:20.901 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_init", 2023-11-16T17:00:20.901 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.902 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.902 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.902 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.902 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.902 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.902 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.902 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Run podman/docker with `--init`", 2023-11-16T17:00:20.902 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.903 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.903 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.903 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.903 INFO:teuthology.orchestra.run.smithi044.stdout: "daemon_cache_timeout": { 2023-11-16T17:00:20.903 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "daemon_cache_timeout", 2023-11-16T17:00:20.903 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:20.903 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.903 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.904 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2023-11-16T17:00:20.904 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.904 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.904 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.904 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-11-16T17:00:20.904 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.904 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.904 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.905 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.905 INFO:teuthology.orchestra.run.smithi044.stdout: "default_cephadm_command_timeout": { 2023-11-16T17:00:20.905 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_cephadm_command_timeout", 2023-11-16T17:00:20.905 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:20.905 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.905 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.905 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "900", 2023-11-16T17:00:20.905 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.906 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.906 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.906 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-11-16T17:00:20.906 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.906 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.906 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.906 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.907 INFO:teuthology.orchestra.run.smithi044.stdout: "default_registry": { 2023-11-16T17:00:20.907 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_registry", 2023-11-16T17:00:20.907 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.907 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.907 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.907 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io", 2023-11-16T17:00:20.907 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.907 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.908 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.908 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-11-16T17:00:20.908 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.908 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.908 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.908 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.908 INFO:teuthology.orchestra.run.smithi044.stdout: "device_cache_timeout": { 2023-11-16T17:00:20.908 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_cache_timeout", 2023-11-16T17:00:20.909 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:20.909 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.909 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.909 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1800", 2023-11-16T17:00:20.909 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.909 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.909 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.909 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache device inventory", 2023-11-16T17:00:20.910 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.910 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.910 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.910 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.910 INFO:teuthology.orchestra.run.smithi044.stdout: "device_enhanced_scan": { 2023-11-16T17:00:20.910 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_enhanced_scan", 2023-11-16T17:00:20.910 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.911 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.911 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.911 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:20.911 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.911 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.911 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.911 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use libstoragemgmt during device scans", 2023-11-16T17:00:20.911 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.912 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.912 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.912 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.912 INFO:teuthology.orchestra.run.smithi044.stdout: "facts_cache_timeout": { 2023-11-16T17:00:20.912 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "facts_cache_timeout", 2023-11-16T17:00:20.912 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:20.912 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.912 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.913 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2023-11-16T17:00:20.913 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.913 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.913 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.913 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache host facts data", 2023-11-16T17:00:20.913 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.913 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.913 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.914 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.914 INFO:teuthology.orchestra.run.smithi044.stdout: "host_check_interval": { 2023-11-16T17:00:20.914 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "host_check_interval", 2023-11-16T17:00:20.914 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:20.914 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.914 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.914 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2023-11-16T17:00:20.914 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.915 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.915 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.915 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to perform a host check", 2023-11-16T17:00:20.915 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.915 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.915 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.915 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.915 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:20.916 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:20.916 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.916 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.916 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.916 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.916 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.916 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.916 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:20.917 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:20.917 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:20.917 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:20.917 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:20.917 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:20.917 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:20.917 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:20.917 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.917 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.918 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.918 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.918 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.918 INFO:teuthology.orchestra.run.smithi044.stdout: "log_refresh_metadata": { 2023-11-16T17:00:20.918 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_refresh_metadata", 2023-11-16T17:00:20.918 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.918 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.919 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.919 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:20.919 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.919 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.919 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.919 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-11-16T17:00:20.919 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.919 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.920 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.920 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.920 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:20.920 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:20.920 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.920 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.920 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.920 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.921 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.921 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.921 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.921 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-11-16T17:00:20.921 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.921 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.921 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.921 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.922 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:20.922 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:20.922 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.922 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.922 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.922 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:20.922 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.922 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.923 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:20.923 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:20.923 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:20.923 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:20.923 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:20.923 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:20.923 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:20.923 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:20.924 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.924 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.924 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.924 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.924 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.924 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:20.924 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:20.924 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.925 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.925 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.925 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:20.925 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.925 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.925 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.925 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.925 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.925 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.926 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.926 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.926 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf": { 2023-11-16T17:00:20.926 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-11-16T17:00:20.926 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.926 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.926 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.927 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:20.927 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.927 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.927 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.927 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-11-16T17:00:20.927 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.927 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.927 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.928 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.928 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-11-16T17:00:20.928 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-11-16T17:00:20.928 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.928 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.928 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.928 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "*", 2023-11-16T17:00:20.928 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.929 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.929 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.929 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-11-16T17:00:20.929 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.929 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.929 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.929 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.929 INFO:teuthology.orchestra.run.smithi044.stdout: "max_count_per_host": { 2023-11-16T17:00:20.930 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_count_per_host", 2023-11-16T17:00:20.930 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:20.930 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.930 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.930 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:20.930 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.930 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.930 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.931 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of daemons per service per host", 2023-11-16T17:00:20.931 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.931 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.931 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.931 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.931 INFO:teuthology.orchestra.run.smithi044.stdout: "max_osd_draining_count": { 2023-11-16T17:00:20.931 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_osd_draining_count", 2023-11-16T17:00:20.931 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:20.932 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.932 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.932 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:20.932 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.932 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.932 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.932 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-11-16T17:00:20.933 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.933 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.933 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.933 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.933 INFO:teuthology.orchestra.run.smithi044.stdout: "migration_current": { 2023-11-16T17:00:20.933 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "migration_current", 2023-11-16T17:00:20.933 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:20.933 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.933 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.934 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.934 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.934 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.934 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.934 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "internal - do not modify", 2023-11-16T17:00:20.934 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.934 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.935 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.935 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.935 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2023-11-16T17:00:20.935 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2023-11-16T17:00:20.935 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.935 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.935 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.935 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "root", 2023-11-16T17:00:20.936 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.936 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.936 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:20.936 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm-package", 2023-11-16T17:00:20.936 INFO:teuthology.orchestra.run.smithi044.stdout: "root" 2023-11-16T17:00:20.936 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:20.936 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "mode for remote execution of cephadm", 2023-11-16T17:00:20.936 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.937 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.937 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.937 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.937 INFO:teuthology.orchestra.run.smithi044.stdout: "prometheus_alerts_path": { 2023-11-16T17:00:20.937 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus_alerts_path", 2023-11-16T17:00:20.937 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.937 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.937 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.938 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-11-16T17:00:20.938 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.938 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.938 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.938 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-11-16T17:00:20.938 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.938 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.938 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.939 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.939 INFO:teuthology.orchestra.run.smithi044.stdout: "prometheus_web_password": { 2023-11-16T17:00:20.939 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus_web_password", 2023-11-16T17:00:20.939 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.939 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.939 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.939 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:20.939 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.940 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.940 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.940 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus web password", 2023-11-16T17:00:20.940 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.940 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.940 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.940 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.940 INFO:teuthology.orchestra.run.smithi044.stdout: "prometheus_web_user": { 2023-11-16T17:00:20.941 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus_web_user", 2023-11-16T17:00:20.941 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.941 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.941 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.941 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:20.941 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.941 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.941 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.942 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus web user", 2023-11-16T17:00:20.942 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.942 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.942 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.942 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.942 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_insecure": { 2023-11-16T17:00:20.942 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_insecure", 2023-11-16T17:00:20.942 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.943 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.943 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.943 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:20.943 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.943 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.943 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.943 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-11-16T17:00:20.943 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.944 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.944 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.944 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.944 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_password": { 2023-11-16T17:00:20.944 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_password", 2023-11-16T17:00:20.944 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.944 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.945 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.945 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.945 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.945 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.945 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.945 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-11-16T17:00:20.945 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.945 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.945 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.946 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.946 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_url": { 2023-11-16T17:00:20.946 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_url", 2023-11-16T17:00:20.946 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.946 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.946 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.946 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.947 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.947 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.947 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.947 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-11-16T17:00:20.947 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.947 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.947 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.947 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.948 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_username": { 2023-11-16T17:00:20.948 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_username", 2023-11-16T17:00:20.948 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.948 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.948 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.948 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.948 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.948 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.949 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.949 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-11-16T17:00:20.949 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.949 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.949 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.949 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.949 INFO:teuthology.orchestra.run.smithi044.stdout: "secure_monitoring_stack": { 2023-11-16T17:00:20.950 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "secure_monitoring_stack", 2023-11-16T17:00:20.950 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.950 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.950 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.950 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:20.950 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.950 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi044.stdout: "service_discovery_port": { 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "service_discovery_port", 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8765", 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "cephadm service discovery port", 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.953 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.953 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.953 INFO:teuthology.orchestra.run.smithi044.stdout: "ssh_config_file": { 2023-11-16T17:00:20.953 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssh_config_file", 2023-11-16T17:00:20.953 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.953 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.953 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.953 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi044.stdout: "use_agent": { 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_agent", 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi044.stdout: "use_repo_digest": { 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_repo_digest", 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_failed_host_check": { 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_failed_host_check", 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if the host check fails", 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_daemons": { 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_daemons", 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_hosts": { 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_hosts", 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.962 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.962 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.962 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.962 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.962 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-11-16T17:00:20.962 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.962 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.962 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crash", 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.966 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.966 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.966 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:20.966 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:20.966 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.966 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.966 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.966 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:20.968 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:20.968 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.968 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.968 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.968 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:20.968 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.968 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.968 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.970 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.970 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.970 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.970 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:20.970 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:20.970 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.970 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi044.stdout: "retain_interval": { 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retain_interval", 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "31536000", 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain crashes before pruning them", 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_recent_interval": { 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_recent_interval", 2023-11-16T17:00:20.974 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:20.974 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.974 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:20.974 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1209600", 2023-11-16T17:00:20.974 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.974 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.974 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.974 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "time interval in which to warn about recent crashes", 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dashboard", 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:20.976 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:20.976 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:20.976 INFO:teuthology.orchestra.run.smithi044.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-11-16T17:00:20.976 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-11-16T17:00:20.976 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:20.976 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.976 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.976 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_HOST": { 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_HOST", 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_ENABLED": { 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_ENABLED", 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi044.stdout: "ENABLE_BROWSABLE_API": { 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ENABLE_BROWSABLE_API", 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.986 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-11-16T17:00:20.986 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-11-16T17:00:20.986 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.986 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.986 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.986 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.986 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.986 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.990 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.990 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.990 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.990 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.990 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.990 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.990 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.990 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_NFS": { 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.994 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.994 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RBD": { 2023-11-16T17:00:20.994 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-11-16T17:00:20.994 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.994 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.994 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.995 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.995 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.995 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.995 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.995 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.995 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.995 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.995 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RGW": { 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi044.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.998 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.998 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:20.998 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:20.998 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:20.998 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:20.998 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:20.998 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:20.998 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_PASSWORD": { 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_PASSWORD", 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.000 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:21.000 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.000 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.000 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.000 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.000 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.000 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.000 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.002 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.002 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.002 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.002 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.002 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.002 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.002 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_URL": { 2023-11-16T17:00:21.002 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_URL", 2023-11-16T17:00:21.003 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.055 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.055 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.055 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.055 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.055 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.055 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_USERNAME": { 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_USERNAME", 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.059 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.059 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.059 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.059 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.059 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.059 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.059 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-11-16T17:00:21.059 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi044.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi044.stdout: "ISSUE_TRACKER_API_KEY": { 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_HOST": { 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_HOST", 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.076 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.076 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.076 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.076 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.076 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-11-16T17:00:21.076 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-11-16T17:00:21.076 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.076 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.076 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.076 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.077 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.077 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.077 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.077 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.077 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.077 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.077 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.077 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.077 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_ENABLED": { 2023-11-16T17:00:21.078 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_ENABLED", 2023-11-16T17:00:21.078 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.078 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.078 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.078 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.078 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.078 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.078 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.078 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.078 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.079 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.079 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.079 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.079 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-11-16T17:00:21.079 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-11-16T17:00:21.079 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.079 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.079 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.080 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-11-16T17:00:21.080 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.080 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.080 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.080 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.080 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.080 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.080 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.080 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.080 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-11-16T17:00:21.081 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-11-16T17:00:21.081 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.081 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.081 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.081 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:21.081 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.081 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.081 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.081 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.081 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.082 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.082 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.082 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.082 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-11-16T17:00:21.082 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-11-16T17:00:21.082 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.082 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.082 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.082 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8", 2023-11-16T17:00:21.083 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.083 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.083 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.083 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.083 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.083 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.083 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.083 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.083 INFO:teuthology.orchestra.run.smithi044.stdout: "REST_REQUESTS_TIMEOUT": { 2023-11-16T17:00:21.083 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-11-16T17:00:21.084 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.084 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.084 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.084 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "45", 2023-11-16T17:00:21.084 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.084 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.084 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.084 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.084 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.085 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.085 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.085 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.085 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ACCESS_KEY": { 2023-11-16T17:00:21.085 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ACCESS_KEY", 2023-11-16T17:00:21.085 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.085 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.085 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.085 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.086 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.086 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.086 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.086 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.086 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.086 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.086 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.086 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.086 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-11-16T17:00:21.087 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-11-16T17:00:21.087 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.087 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.087 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.087 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:21.087 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.087 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.087 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.087 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.088 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.088 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.088 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.088 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.088 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SECRET_KEY": { 2023-11-16T17:00:21.088 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SECRET_KEY", 2023-11-16T17:00:21.088 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.088 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.088 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.089 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.089 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.089 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.089 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.089 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.089 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.089 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.089 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.090 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.090 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SSL_VERIFY": { 2023-11-16T17:00:21.090 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SSL_VERIFY", 2023-11-16T17:00:21.090 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.090 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.090 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.090 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.090 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.090 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.091 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.091 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.091 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.091 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.091 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.091 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.091 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-11-16T17:00:21.091 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-11-16T17:00:21.091 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.092 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.092 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.092 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2023-11-16T17:00:21.092 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.092 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.092 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.092 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.092 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.092 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.093 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.093 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.093 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-11-16T17:00:21.093 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-11-16T17:00:21.093 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.093 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.093 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.093 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:21.093 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.094 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.094 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.094 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.094 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.094 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.094 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.094 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.094 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-11-16T17:00:21.094 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-11-16T17:00:21.095 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.095 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.095 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.095 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2023-11-16T17:00:21.095 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.095 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.095 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.095 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.095 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.096 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.096 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.096 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.096 INFO:teuthology.orchestra.run.smithi044.stdout: "cross_origin_url": { 2023-11-16T17:00:21.096 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cross_origin_url", 2023-11-16T17:00:21.096 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.096 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.096 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.096 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.097 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.097 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.097 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.097 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.097 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.097 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.097 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.097 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.097 INFO:teuthology.orchestra.run.smithi044.stdout: "crt_file": { 2023-11-16T17:00:21.098 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crt_file", 2023-11-16T17:00:21.098 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.098 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.098 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.098 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.098 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.098 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.098 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.098 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.099 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.099 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.099 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.099 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.099 INFO:teuthology.orchestra.run.smithi044.stdout: "debug": { 2023-11-16T17:00:21.099 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "debug", 2023-11-16T17:00:21.099 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.099 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.100 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.100 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.100 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.100 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.100 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.100 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable/disable debug options", 2023-11-16T17:00:21.100 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.100 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.101 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.101 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.101 INFO:teuthology.orchestra.run.smithi044.stdout: "jwt_token_ttl": { 2023-11-16T17:00:21.101 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "jwt_token_ttl", 2023-11-16T17:00:21.101 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.101 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.101 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.101 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "28800", 2023-11-16T17:00:21.101 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.102 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.102 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.102 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.102 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.102 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.102 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.102 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.102 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2023-11-16T17:00:21.103 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2023-11-16T17:00:21.103 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.103 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.103 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.103 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.103 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.103 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.103 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.103 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.104 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.104 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.104 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.104 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.104 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.104 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.104 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.104 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.105 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.105 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.105 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.105 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.105 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.105 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.105 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.105 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.105 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.106 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.106 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.106 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.106 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.106 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.106 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.106 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.106 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.107 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.107 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.107 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.107 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.107 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.107 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.107 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.107 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.107 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.108 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.108 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.108 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.108 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.108 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.108 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.108 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.108 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.108 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.109 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.109 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.109 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.109 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.109 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.109 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.109 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.109 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.109 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.110 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.110 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.110 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.110 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.110 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.110 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.110 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.110 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.111 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.111 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.111 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.111 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.111 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.111 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.111 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.111 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.112 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.112 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.112 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.112 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.112 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.112 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.112 INFO:teuthology.orchestra.run.smithi044.stdout: "motd": { 2023-11-16T17:00:21.112 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "motd", 2023-11-16T17:00:21.112 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.113 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.113 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.113 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.113 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.113 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.113 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.113 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "The message of the day", 2023-11-16T17:00:21.113 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.113 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.114 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.114 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.114 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect_resolve_ip_addr": { 2023-11-16T17:00:21.114 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "redirect_resolve_ip_addr", 2023-11-16T17:00:21.114 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.114 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.114 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.114 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.114 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.115 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.115 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.115 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.115 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.115 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.115 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.115 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.115 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2023-11-16T17:00:21.115 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2023-11-16T17:00:21.116 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.116 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.116 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.116 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2023-11-16T17:00:21.116 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.116 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.116 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.116 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.116 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.117 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.117 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.117 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.117 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2023-11-16T17:00:21.117 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2023-11-16T17:00:21.117 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.117 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.117 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.117 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8080", 2023-11-16T17:00:21.118 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.118 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.118 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.118 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.118 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.118 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.118 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.119 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.119 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2023-11-16T17:00:21.119 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2023-11-16T17:00:21.119 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.119 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.119 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.119 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.119 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.120 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.120 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.120 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.120 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.120 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.120 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.120 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.120 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl_server_port": { 2023-11-16T17:00:21.120 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl_server_port", 2023-11-16T17:00:21.121 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.121 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.121 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.121 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8443", 2023-11-16T17:00:21.121 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.121 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.121 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.121 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.122 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.122 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.122 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.122 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.122 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2023-11-16T17:00:21.122 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2023-11-16T17:00:21.122 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.122 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.123 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.123 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "redirect", 2023-11-16T17:00:21.123 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.123 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.123 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.123 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.123 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect" 2023-11-16T17:00:21.123 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.124 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.124 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.124 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.124 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.124 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.124 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2023-11-16T17:00:21.124 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2023-11-16T17:00:21.124 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.124 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.125 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.125 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2023-11-16T17:00:21.125 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2023-11-16T17:00:21.125 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2023-11-16T17:00:21.125 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.125 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.125 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.125 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.125 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.126 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.126 INFO:teuthology.orchestra.run.smithi044.stdout: "url_prefix": { 2023-11-16T17:00:21.126 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url_prefix", 2023-11-16T17:00:21.126 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.126 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.126 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.126 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.126 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.127 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.127 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.127 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.127 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.127 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.127 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.127 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.127 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.127 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.128 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.128 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "devicehealth", 2023-11-16T17:00:21.128 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.128 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.128 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.128 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_monitoring": { 2023-11-16T17:00:21.128 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_monitoring", 2023-11-16T17:00:21.128 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.129 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.129 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.129 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.129 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.129 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.129 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.129 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "monitor device health metrics", 2023-11-16T17:00:21.129 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.130 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.130 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.130 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.130 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.130 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.130 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.130 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.130 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.131 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.131 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.131 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.131 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.131 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.131 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.131 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.131 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.131 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.132 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.132 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.132 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.132 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.132 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.132 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.132 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.132 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.132 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.133 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.133 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.133 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.133 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.133 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.133 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.133 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.133 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.133 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.134 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.134 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.134 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.134 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.134 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.134 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.134 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.134 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.135 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.135 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.135 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.135 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.135 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.135 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.135 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.135 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.135 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.136 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.136 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.136 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.136 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.136 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.136 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.136 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.136 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.136 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.137 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.137 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.137 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.137 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.137 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.137 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.137 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.137 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.137 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.138 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.138 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.138 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.138 INFO:teuthology.orchestra.run.smithi044.stdout: "mark_out_threshold": { 2023-11-16T17:00:21.138 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mark_out_threshold", 2023-11-16T17:00:21.138 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.138 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.138 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.139 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2419200", 2023-11-16T17:00:21.139 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.139 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.139 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.139 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-11-16T17:00:21.139 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.139 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.139 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.139 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.140 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_name": { 2023-11-16T17:00:21.140 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_name", 2023-11-16T17:00:21.140 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.140 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.140 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.140 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "device_health_metrics", 2023-11-16T17:00:21.140 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.140 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.140 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.141 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name of pool in which to store device health metrics", 2023-11-16T17:00:21.141 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.141 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.141 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.141 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.141 INFO:teuthology.orchestra.run.smithi044.stdout: "retention_period": { 2023-11-16T17:00:21.141 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retention_period", 2023-11-16T17:00:21.141 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.142 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.142 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.142 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15552000", 2023-11-16T17:00:21.142 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.142 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.142 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.142 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain device health metrics", 2023-11-16T17:00:21.142 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.142 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.143 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.143 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.143 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_frequency": { 2023-11-16T17:00:21.143 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_frequency", 2023-11-16T17:00:21.143 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.143 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.143 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.143 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2023-11-16T17:00:21.143 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.144 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.144 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.144 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to scrape device health metrics", 2023-11-16T17:00:21.144 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.144 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.144 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.144 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.144 INFO:teuthology.orchestra.run.smithi044.stdout: "self_heal": { 2023-11-16T17:00:21.144 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "self_heal", 2023-11-16T17:00:21.145 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.145 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.145 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.145 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.145 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.145 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.145 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.145 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-11-16T17:00:21.145 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.146 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.146 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.146 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.146 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2023-11-16T17:00:21.146 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2023-11-16T17:00:21.146 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.146 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.146 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.147 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2023-11-16T17:00:21.147 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.147 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.147 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.147 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and check device health", 2023-11-16T17:00:21.147 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.147 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.147 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.147 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.148 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_threshold": { 2023-11-16T17:00:21.148 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_threshold", 2023-11-16T17:00:21.148 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.148 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.148 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.148 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7257600", 2023-11-16T17:00:21.148 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.148 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.149 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.149 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-11-16T17:00:21.149 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.149 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.149 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.149 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.149 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.149 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.150 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.150 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "diskprediction_local", 2023-11-16T17:00:21.150 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.150 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.150 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.150 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.150 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.151 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.152 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.153 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.153 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.153 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.153 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.153 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.153 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.153 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.153 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.153 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.154 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.154 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.154 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.154 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.154 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.154 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.154 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.154 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.155 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.155 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.155 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.155 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.155 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.155 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.155 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.155 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.155 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.156 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.156 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.156 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.156 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.156 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.156 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.156 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.156 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.156 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.157 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.157 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.157 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.157 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.157 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.157 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.157 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.157 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.157 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.158 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.158 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.158 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.158 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.158 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.158 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.158 INFO:teuthology.orchestra.run.smithi044.stdout: "predict_interval": { 2023-11-16T17:00:21.158 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predict_interval", 2023-11-16T17:00:21.158 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.159 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.159 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.159 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2023-11-16T17:00:21.159 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.159 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.159 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.159 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.159 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.160 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.160 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.160 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.160 INFO:teuthology.orchestra.run.smithi044.stdout: "predictor_model": { 2023-11-16T17:00:21.160 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predictor_model", 2023-11-16T17:00:21.160 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.160 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.160 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.160 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "prophetstor", 2023-11-16T17:00:21.161 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.161 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.161 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.161 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.161 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.161 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.161 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.161 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.161 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2023-11-16T17:00:21.162 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2023-11-16T17:00:21.162 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.162 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.162 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.162 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2023-11-16T17:00:21.162 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.162 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.162 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.162 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.163 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.163 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.163 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.163 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.163 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.163 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.163 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.163 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "influx", 2023-11-16T17:00:21.163 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": false, 2023-11-16T17:00:21.164 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "influxdb python module not found", 2023-11-16T17:00:21.164 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.164 INFO:teuthology.orchestra.run.smithi044.stdout: "batch_size": { 2023-11-16T17:00:21.164 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "batch_size", 2023-11-16T17:00:21.164 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.164 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.164 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.164 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5000", 2023-11-16T17:00:21.164 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.165 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.165 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.165 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-11-16T17:00:21.165 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.165 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.165 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.165 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.165 INFO:teuthology.orchestra.run.smithi044.stdout: "database": { 2023-11-16T17:00:21.166 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "database", 2023-11-16T17:00:21.166 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.166 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.166 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.166 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "ceph", 2023-11-16T17:00:21.166 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.166 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.166 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.166 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-11-16T17:00:21.167 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.167 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.167 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.167 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.167 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": { 2023-11-16T17:00:21.167 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "hostname", 2023-11-16T17:00:21.167 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.167 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.167 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.168 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.168 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.168 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.168 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.168 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server hostname", 2023-11-16T17:00:21.168 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.168 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.168 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.168 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.169 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2023-11-16T17:00:21.169 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2023-11-16T17:00:21.169 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.169 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.169 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.169 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "30", 2023-11-16T17:00:21.169 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "5", 2023-11-16T17:00:21.169 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.170 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.170 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-11-16T17:00:21.170 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.170 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.170 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.170 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.170 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.170 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.170 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.171 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.171 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.171 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.171 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.171 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.171 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.171 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.171 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.172 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.172 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.172 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.172 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.172 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.172 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.172 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.172 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.172 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.173 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.173 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.173 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.173 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.173 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.173 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.173 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.173 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.173 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.174 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.174 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.174 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.174 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.174 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.174 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.174 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.174 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.175 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.175 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.175 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.175 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.175 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.175 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.175 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.175 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.176 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.176 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.176 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.176 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.176 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.176 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.176 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.176 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.177 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.177 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.177 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.177 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.177 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.177 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.177 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.177 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.177 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.178 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.178 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.178 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.178 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.178 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.178 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.178 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.178 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.179 INFO:teuthology.orchestra.run.smithi044.stdout: "password": { 2023-11-16T17:00:21.179 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "password", 2023-11-16T17:00:21.179 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.179 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.179 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.179 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.179 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.179 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.180 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.180 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "password of InfluxDB server user", 2023-11-16T17:00:21.180 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.180 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.180 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.180 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.180 INFO:teuthology.orchestra.run.smithi044.stdout: "port": { 2023-11-16T17:00:21.180 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "port", 2023-11-16T17:00:21.181 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.181 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.181 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.181 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8086", 2023-11-16T17:00:21.181 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.181 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.181 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.181 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server port", 2023-11-16T17:00:21.181 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.182 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.182 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.182 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.182 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2023-11-16T17:00:21.182 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2023-11-16T17:00:21.182 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.182 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.182 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.182 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "false", 2023-11-16T17:00:21.183 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.183 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.183 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.183 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T17:00:21.183 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.183 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.183 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.183 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.184 INFO:teuthology.orchestra.run.smithi044.stdout: "threads": { 2023-11-16T17:00:21.184 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threads", 2023-11-16T17:00:21.184 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.184 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.184 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.184 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2023-11-16T17:00:21.184 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2023-11-16T17:00:21.184 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "32", 2023-11-16T17:00:21.184 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.185 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-11-16T17:00:21.185 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.185 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.185 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.185 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.185 INFO:teuthology.orchestra.run.smithi044.stdout: "username": { 2023-11-16T17:00:21.185 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "username", 2023-11-16T17:00:21.185 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.185 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.186 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.186 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.186 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.186 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.186 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.186 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "username of InfluxDB server user", 2023-11-16T17:00:21.186 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.186 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.186 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.187 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.187 INFO:teuthology.orchestra.run.smithi044.stdout: "verify_ssl": { 2023-11-16T17:00:21.187 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "verify_ssl", 2023-11-16T17:00:21.187 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.187 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.187 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.187 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "true", 2023-11-16T17:00:21.187 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.187 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.188 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.188 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T17:00:21.188 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.188 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.188 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.188 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.188 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.188 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.189 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.189 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "insights", 2023-11-16T17:00:21.189 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.189 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.189 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.189 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.189 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.189 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.190 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.190 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.190 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.190 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.190 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.190 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.190 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.190 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.191 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.191 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.191 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.191 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.191 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.191 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.191 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.191 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.191 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.192 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.192 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.192 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.192 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.192 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.192 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.192 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.192 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.192 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.193 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.193 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.193 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.193 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.193 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.193 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.193 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.193 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.193 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.194 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.194 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.194 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.194 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.194 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.194 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.194 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.194 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.195 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.195 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.195 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.195 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.195 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.195 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.195 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.195 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.195 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.196 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.196 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.196 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.196 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.196 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.196 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.196 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.196 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.196 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.197 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.197 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.197 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.197 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.197 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.197 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.197 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.197 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.197 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.198 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "iostat", 2023-11-16T17:00:21.198 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.198 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.198 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.198 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.198 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.198 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.198 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.198 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.199 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.199 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.199 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.199 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.199 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.199 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.199 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.199 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.200 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.200 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.200 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.200 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.200 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.200 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.200 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.200 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.201 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.201 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.201 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.201 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.201 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.201 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.201 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.201 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.201 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.202 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.202 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.202 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.202 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.202 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.202 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.202 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.202 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.202 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.203 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.203 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.203 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.203 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.203 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.203 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.203 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.203 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.204 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.204 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.204 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.204 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.204 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.204 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.204 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.204 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.204 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.205 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.205 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.205 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.205 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.205 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.205 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.205 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.205 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.205 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.206 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.206 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.206 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.206 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.206 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.206 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.206 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.206 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.206 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "k8sevents", 2023-11-16T17:00:21.207 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.207 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.207 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.207 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_event_retention_days": { 2023-11-16T17:00:21.207 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ceph_event_retention_days", 2023-11-16T17:00:21.207 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.207 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.207 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.207 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7", 2023-11-16T17:00:21.208 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.208 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.208 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.208 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Days to hold ceph event information within local cache", 2023-11-16T17:00:21.208 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.208 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.208 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.208 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.209 INFO:teuthology.orchestra.run.smithi044.stdout: "config_check_secs": { 2023-11-16T17:00:21.209 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_check_secs", 2023-11-16T17:00:21.209 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.209 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.209 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.209 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:21.209 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "10", 2023-11-16T17:00:21.209 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.210 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.210 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-11-16T17:00:21.210 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.210 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.210 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.210 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.210 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.210 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.210 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.211 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.211 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.211 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.211 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.211 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.211 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.211 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.211 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.212 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.212 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.212 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.212 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.212 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.212 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.212 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.212 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.212 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.213 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.213 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.213 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.213 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.213 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.213 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.213 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.213 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.213 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.214 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.214 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.214 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.214 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.214 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.214 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.214 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.214 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.214 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.215 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.215 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.215 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.215 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.215 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.215 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.215 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.215 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.215 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.216 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.216 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.216 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.216 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.216 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.216 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.216 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.216 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.216 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.217 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.217 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.217 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.217 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.217 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.217 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.217 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.217 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.217 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.217 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.218 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.218 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.218 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.218 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.218 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.218 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.218 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.218 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "localpool", 2023-11-16T17:00:21.218 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.219 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.219 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.219 INFO:teuthology.orchestra.run.smithi044.stdout: "failure_domain": { 2023-11-16T17:00:21.219 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "failure_domain", 2023-11-16T17:00:21.219 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.219 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.219 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.219 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "host", 2023-11-16T17:00:21.220 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.220 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.220 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.220 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "failure domain for any created local pool", 2023-11-16T17:00:21.220 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-11-16T17:00:21.220 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.220 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.220 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.220 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.220 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.221 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.221 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.221 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.221 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.221 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.221 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.221 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.221 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.222 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.222 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.222 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.222 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.222 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.222 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.222 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.222 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.222 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.223 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.223 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.223 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.223 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.223 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.223 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.223 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.223 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.223 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.224 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.224 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.224 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.224 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.224 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.224 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.224 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.224 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.224 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.225 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.225 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.225 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.225 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.225 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.225 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.225 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.225 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.225 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.226 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.226 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.226 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.226 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.226 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.226 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.226 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.226 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.226 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.227 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.227 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.227 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.227 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.227 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.227 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.227 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.227 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.227 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.228 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.228 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.228 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.228 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.228 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.228 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.228 INFO:teuthology.orchestra.run.smithi044.stdout: "min_size": { 2023-11-16T17:00:21.229 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_size", 2023-11-16T17:00:21.229 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.229 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.229 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.229 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.229 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.229 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.229 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.229 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default min_size for any created local pool", 2023-11-16T17:00:21.230 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-11-16T17:00:21.230 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.230 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.230 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.230 INFO:teuthology.orchestra.run.smithi044.stdout: "num_rep": { 2023-11-16T17:00:21.230 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "num_rep", 2023-11-16T17:00:21.230 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.230 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.231 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.231 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3", 2023-11-16T17:00:21.231 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.231 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.231 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.231 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default replica count for any created local pool", 2023-11-16T17:00:21.231 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.231 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.231 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.232 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.232 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_num": { 2023-11-16T17:00:21.232 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_num", 2023-11-16T17:00:21.232 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.232 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.232 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.232 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "128", 2023-11-16T17:00:21.232 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.232 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.233 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.233 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default pg_num for any created local pool", 2023-11-16T17:00:21.233 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.233 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.233 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.233 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.233 INFO:teuthology.orchestra.run.smithi044.stdout: "prefix": { 2023-11-16T17:00:21.233 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prefix", 2023-11-16T17:00:21.233 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.234 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.234 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.234 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.234 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.234 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.234 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.234 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name prefix for any created local pool", 2023-11-16T17:00:21.234 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.234 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.235 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.235 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.235 INFO:teuthology.orchestra.run.smithi044.stdout: "subtree": { 2023-11-16T17:00:21.235 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "subtree", 2023-11-16T17:00:21.235 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.235 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.235 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.235 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "rack", 2023-11-16T17:00:21.236 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.236 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.236 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.236 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "CRUSH level for which to create a local pool", 2023-11-16T17:00:21.236 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-11-16T17:00:21.236 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.236 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.236 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.236 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.237 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.237 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.237 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mds_autoscaler", 2023-11-16T17:00:21.237 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.237 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.237 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.237 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.237 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.237 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.238 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.238 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.238 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.238 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.238 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.238 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.238 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.238 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.239 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.239 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.239 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.239 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.239 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.239 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.239 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.239 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.239 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.240 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.240 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.240 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.240 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.240 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.240 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.240 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.240 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.241 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.241 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.241 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.241 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.241 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.241 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.241 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.241 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.241 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.242 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.242 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.242 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.242 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.242 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.242 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.242 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.242 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.242 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.243 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.243 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.243 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.243 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.243 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.243 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.243 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.243 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.243 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.244 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.244 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.244 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.244 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.244 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.244 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.244 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.244 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.244 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.245 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.245 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.245 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.245 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.245 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.245 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.245 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.245 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.245 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.245 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirroring", 2023-11-16T17:00:21.246 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.246 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.246 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.246 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.246 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.246 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.246 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.246 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.247 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.247 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.247 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.247 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.247 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.247 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.247 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.247 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.247 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.247 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.248 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.248 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.248 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.248 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.248 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.248 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.248 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.248 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.248 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.249 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.249 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.249 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.249 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.249 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.249 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.249 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.249 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.249 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.250 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.250 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.250 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.250 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.250 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.250 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.250 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.250 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.250 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.251 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.251 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.251 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.251 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.251 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.251 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.251 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.251 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.251 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.252 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.252 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.252 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.252 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.252 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.252 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.252 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.252 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.252 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.252 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.253 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.253 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.253 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.253 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.253 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.253 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.253 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.253 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.253 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.253 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.254 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.254 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.254 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "nfs", 2023-11-16T17:00:21.254 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.254 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.254 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.254 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.254 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.254 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.255 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.255 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.255 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.255 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.255 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.255 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.255 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.255 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.255 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.256 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.256 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.256 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.256 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.256 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.256 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.256 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.256 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.256 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.256 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.258 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.258 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.258 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.258 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.258 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.258 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.258 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.258 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.258 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.259 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.259 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.259 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.259 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.259 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.259 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.259 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.259 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.260 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.260 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.260 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.260 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.260 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.260 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.260 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.260 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.260 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.260 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.261 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.261 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.261 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.261 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.261 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.261 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.261 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.261 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.261 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.262 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.262 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.262 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.262 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.262 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.262 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.262 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2023-11-16T17:00:21.262 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.262 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.263 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.263 INFO:teuthology.orchestra.run.smithi044.stdout: "fail_fs": { 2023-11-16T17:00:21.263 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "fail_fs", 2023-11-16T17:00:21.263 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.263 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.263 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.263 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.263 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.263 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.263 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.264 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-11-16T17:00:21.264 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.264 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.264 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.264 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.264 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.264 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.264 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.264 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.265 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.265 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.265 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.265 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.265 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.265 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.265 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.265 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.265 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.265 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.266 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.266 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.266 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.266 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.266 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.266 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.266 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.266 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.267 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.267 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.267 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.267 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.267 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.267 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.267 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.267 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.268 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.268 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.268 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.268 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.268 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.268 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.268 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.268 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.268 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.269 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.269 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.269 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.269 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.269 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.269 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.269 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.269 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.270 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.270 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.270 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.270 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.270 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.270 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.270 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.270 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.271 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.271 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.271 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.271 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.271 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.271 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.271 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.271 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.271 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.272 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.272 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.272 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.272 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.272 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.272 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.272 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator": { 2023-11-16T17:00:21.272 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2023-11-16T17:00:21.272 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.272 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.273 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.273 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.273 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.273 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.273 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.273 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm", 2023-11-16T17:00:21.273 INFO:teuthology.orchestra.run.smithi044.stdout: "rook", 2023-11-16T17:00:21.273 INFO:teuthology.orchestra.run.smithi044.stdout: "test_orchestrator" 2023-11-16T17:00:21.273 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.274 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Orchestrator backend", 2023-11-16T17:00:21.274 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.274 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.274 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.274 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.274 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.274 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.274 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.274 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_perf_query", 2023-11-16T17:00:21.275 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.275 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.275 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.275 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.275 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.275 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.275 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.275 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.275 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.275 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.276 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.276 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.276 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.276 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.276 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.276 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.276 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.276 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.276 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.277 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.277 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.277 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.277 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.277 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.277 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.277 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.277 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.277 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.277 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.278 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.278 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.278 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.278 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.278 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.278 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.278 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.278 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.279 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.279 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.279 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.279 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.279 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.279 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.279 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.279 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.280 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.280 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.280 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.280 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.280 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.280 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.280 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.280 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.280 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.280 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.281 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.281 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.281 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.281 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.281 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.281 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.281 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.281 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.281 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.282 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.282 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.282 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.282 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.282 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.282 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.282 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.282 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.282 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.283 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.283 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.283 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.283 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_support", 2023-11-16T17:00:21.283 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.283 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.283 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.283 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.283 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.283 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.284 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.284 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.284 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.284 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.284 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.284 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.284 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.284 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.284 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.284 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.285 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.285 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.285 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.285 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.285 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.285 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.285 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.285 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.285 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.286 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.286 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.286 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.286 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.286 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.286 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.286 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.286 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.286 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.287 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.287 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.287 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.287 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.287 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.287 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.287 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.287 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.287 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.287 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.288 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.288 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.288 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.288 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.288 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.288 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.288 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.288 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.288 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.289 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.289 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.289 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.289 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.289 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.289 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.289 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.289 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.289 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.290 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.290 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.290 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.290 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.290 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.290 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.290 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.290 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.290 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.291 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.291 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.291 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.291 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.291 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.291 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_autoscaler", 2023-11-16T17:00:21.291 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.291 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.291 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.292 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.292 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.292 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.292 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.292 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.292 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.292 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.292 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.292 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.292 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.293 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.293 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.293 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.293 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.293 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.293 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.293 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.293 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.293 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.294 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.294 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.294 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.294 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.294 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.294 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.294 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.294 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.294 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.294 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.295 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.295 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.295 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.295 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.295 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.295 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.295 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.295 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.295 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.296 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.296 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.296 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.296 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.296 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.296 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.296 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.296 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.296 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.296 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.297 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.297 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.297 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.297 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.297 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.297 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.297 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.297 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.297 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.297 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.298 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.298 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.298 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.298 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.298 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.298 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.298 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.298 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.298 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.299 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.299 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.299 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.299 INFO:teuthology.orchestra.run.smithi044.stdout: "noautoscale": { 2023-11-16T17:00:21.299 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "noautoscale", 2023-11-16T17:00:21.299 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.299 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.299 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.299 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.300 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.300 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.300 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.300 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "global autoscale flag", 2023-11-16T17:00:21.300 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-11-16T17:00:21.300 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.300 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.300 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.300 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2023-11-16T17:00:21.301 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2023-11-16T17:00:21.301 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.301 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.301 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.301 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2023-11-16T17:00:21.301 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.301 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.301 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.301 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.301 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.302 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.302 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.302 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.302 INFO:teuthology.orchestra.run.smithi044.stdout: "threshold": { 2023-11-16T17:00:21.302 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threshold", 2023-11-16T17:00:21.302 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:21.302 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.302 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.302 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2023-11-16T17:00:21.303 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1.0", 2023-11-16T17:00:21.303 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.303 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.303 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "scaling threshold", 2023-11-16T17:00:21.303 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-11-16T17:00:21.303 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.303 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.303 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.303 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.304 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.304 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.304 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "progress", 2023-11-16T17:00:21.304 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.304 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.304 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.304 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_pg_recovery_event": { 2023-11-16T17:00:21.304 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_pg_recovery_event", 2023-11-16T17:00:21.304 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.304 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.305 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.305 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.305 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.305 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.305 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.305 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow the module to show pg recovery progress", 2023-11-16T17:00:21.305 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.305 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.305 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.305 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.306 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2023-11-16T17:00:21.306 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2023-11-16T17:00:21.306 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.306 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.306 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.306 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.306 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.306 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.306 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.307 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.307 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.307 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.307 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.307 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.307 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.307 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.307 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.307 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.308 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.308 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.308 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.308 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.308 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.308 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.308 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.308 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.308 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.308 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.309 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.309 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.309 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.309 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.309 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.309 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.309 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.309 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.309 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.310 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.310 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.310 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.310 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.310 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.310 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.310 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.310 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.310 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.311 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.311 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.311 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.311 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.311 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.311 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.311 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.311 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.311 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.311 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.312 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.312 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.312 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.312 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.312 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.312 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.312 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.312 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.312 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.313 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.313 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.313 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.313 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.313 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.313 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.313 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.313 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.313 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.313 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.314 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.314 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.314 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.314 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.314 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.314 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.314 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.314 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.314 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.314 INFO:teuthology.orchestra.run.smithi044.stdout: "max_completed_events": { 2023-11-16T17:00:21.315 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_completed_events", 2023-11-16T17:00:21.315 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.315 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.315 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.315 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "50", 2023-11-16T17:00:21.315 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.315 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.315 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.315 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "number of past completed events to remember", 2023-11-16T17:00:21.316 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.316 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.316 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.316 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.316 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2023-11-16T17:00:21.316 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2023-11-16T17:00:21.316 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.316 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.316 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.316 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2023-11-16T17:00:21.317 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.317 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.317 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.317 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long the module is going to sleep", 2023-11-16T17:00:21.317 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.317 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.317 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.317 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.317 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.317 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.318 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.318 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus", 2023-11-16T17:00:21.318 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.318 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.318 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.318 INFO:teuthology.orchestra.run.smithi044.stdout: "cache": { 2023-11-16T17:00:21.318 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cache", 2023-11-16T17:00:21.318 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.319 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.319 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.319 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.319 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.319 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.319 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.319 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.319 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.319 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.320 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.320 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.320 INFO:teuthology.orchestra.run.smithi044.stdout: "exclude_perf_counters": { 2023-11-16T17:00:21.320 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "exclude_perf_counters", 2023-11-16T17:00:21.320 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.320 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.320 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.320 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.320 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.320 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.321 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.321 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-11-16T17:00:21.321 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-11-16T17:00:21.321 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.321 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.321 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.321 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.321 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.321 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.322 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.322 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.322 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.322 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.322 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.322 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.322 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.322 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.322 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.323 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.323 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.323 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.323 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.323 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.323 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.323 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.323 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.323 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.324 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.324 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.324 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.324 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.324 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.324 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.324 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.324 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.324 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.324 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.325 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.325 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.325 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.325 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.325 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.325 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.325 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.325 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.325 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.326 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.326 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.326 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.326 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.326 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.326 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.326 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.326 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.326 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.327 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.327 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.327 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.327 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.327 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.327 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.327 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.327 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.327 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.328 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.328 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.328 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.328 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.328 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.328 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.328 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.328 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.328 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.328 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.329 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.329 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.329 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools": { 2023-11-16T17:00:21.329 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools", 2023-11-16T17:00:21.329 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.329 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.329 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.329 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.334 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.335 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.335 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.335 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.335 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.335 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.335 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.335 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.335 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools_refresh_interval": { 2023-11-16T17:00:21.336 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-11-16T17:00:21.336 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.336 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.336 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.336 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "300", 2023-11-16T17:00:21.336 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.336 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.336 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.336 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.337 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.337 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.337 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.337 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.337 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_interval": { 2023-11-16T17:00:21.337 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_interval", 2023-11-16T17:00:21.337 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:21.337 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.337 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.338 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15.0", 2023-11-16T17:00:21.338 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.338 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.338 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.338 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.338 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.338 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.338 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.339 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.339 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2023-11-16T17:00:21.339 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2023-11-16T17:00:21.339 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.339 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.339 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.339 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2023-11-16T17:00:21.339 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.339 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.340 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.340 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-11-16T17:00:21.340 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.340 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.340 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.340 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.340 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2023-11-16T17:00:21.340 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2023-11-16T17:00:21.341 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.341 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.341 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.341 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "9283", 2023-11-16T17:00:21.341 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.341 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.341 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.341 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-11-16T17:00:21.342 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.342 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.342 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.342 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.342 INFO:teuthology.orchestra.run.smithi044.stdout: "stale_cache_strategy": { 2023-11-16T17:00:21.342 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stale_cache_strategy", 2023-11-16T17:00:21.342 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.342 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.343 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.343 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "log", 2023-11-16T17:00:21.343 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.343 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.343 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.343 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.343 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.343 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.343 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.343 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.344 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2023-11-16T17:00:21.344 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2023-11-16T17:00:21.344 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.344 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.344 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.344 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "default", 2023-11-16T17:00:21.344 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.344 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.345 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.345 INFO:teuthology.orchestra.run.smithi044.stdout: "default", 2023-11-16T17:00:21.345 INFO:teuthology.orchestra.run.smithi044.stdout: "error" 2023-11-16T17:00:21.345 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.345 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.345 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.345 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.345 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.345 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.346 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2023-11-16T17:00:21.346 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2023-11-16T17:00:21.346 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.346 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.346 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.346 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2023-11-16T17:00:21.346 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2023-11-16T17:00:21.346 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2023-11-16T17:00:21.346 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.347 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.347 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.347 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.347 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.347 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.347 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.347 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.347 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.348 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_support", 2023-11-16T17:00:21.348 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.348 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.348 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.348 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.348 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.348 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.348 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.348 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.349 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.349 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.349 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.349 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.349 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.349 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.349 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.349 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.349 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.350 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.350 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.350 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.350 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.350 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.350 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.350 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.350 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.351 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.351 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.351 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.351 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.351 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.351 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.351 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.351 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.351 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.352 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.352 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.352 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.352 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.352 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.352 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.352 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.352 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.352 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.353 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.353 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.353 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.353 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.353 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.353 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.353 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.353 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.353 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.354 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.354 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.354 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.354 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.354 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.354 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.354 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.354 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.354 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.355 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.355 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.355 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.355 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.355 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.355 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.355 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.355 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.355 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.356 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.356 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.356 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.356 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_snap_create": { 2023-11-16T17:00:21.356 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_snap_create", 2023-11-16T17:00:21.356 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.356 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.356 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.356 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:21.357 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.357 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.357 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.357 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.357 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.357 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.357 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.357 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.357 INFO:teuthology.orchestra.run.smithi044.stdout: "mirror_snapshot_schedule": { 2023-11-16T17:00:21.358 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirror_snapshot_schedule", 2023-11-16T17:00:21.358 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.358 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.358 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.358 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.358 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.358 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.358 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.358 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.359 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.359 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.359 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.359 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.359 INFO:teuthology.orchestra.run.smithi044.stdout: "trash_purge_schedule": { 2023-11-16T17:00:21.359 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "trash_purge_schedule", 2023-11-16T17:00:21.359 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.359 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.359 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.360 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.360 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.360 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.360 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.360 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.360 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.360 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.360 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.361 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.361 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.361 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.361 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.361 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "restful", 2023-11-16T17:00:21.361 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.361 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.361 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.361 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_auth": { 2023-11-16T17:00:21.362 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_auth", 2023-11-16T17:00:21.362 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.362 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.362 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.362 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.362 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.362 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.362 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.362 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.363 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.363 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.363 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.363 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.363 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2023-11-16T17:00:21.363 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2023-11-16T17:00:21.363 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.364 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.364 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.364 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.364 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.364 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.364 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.364 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.364 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.364 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.365 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.365 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.365 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.365 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.365 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.365 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.365 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.365 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.365 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.366 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.366 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.366 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.366 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.366 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.366 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.366 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.366 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.367 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.367 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.367 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.367 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.367 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.367 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.367 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.367 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.367 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.368 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.368 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.368 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.368 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.368 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.368 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.368 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.368 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.368 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.369 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.369 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.369 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.369 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.369 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.369 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.369 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.370 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.370 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.370 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.370 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.370 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.370 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.370 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.371 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.371 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.371 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.371 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.371 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.371 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.371 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.371 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.372 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.372 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.372 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.372 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.372 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.372 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.372 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.372 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.372 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.373 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.373 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.373 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.373 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.373 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.373 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.373 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2023-11-16T17:00:21.373 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2023-11-16T17:00:21.373 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.373 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.374 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.374 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.374 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.374 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.374 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.374 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.374 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.374 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.375 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.375 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.375 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2023-11-16T17:00:21.375 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2023-11-16T17:00:21.375 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.375 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.375 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.375 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.375 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.376 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.376 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.376 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.376 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.376 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.376 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.376 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.376 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.376 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.377 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.377 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rgw", 2023-11-16T17:00:21.377 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.377 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.377 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.377 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.377 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.377 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.377 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.377 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.378 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.378 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.378 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.378 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.378 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.378 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.378 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.378 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.379 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.379 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.379 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.379 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.379 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.379 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.379 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.379 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.380 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.380 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.380 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.380 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.380 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.380 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.380 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.380 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.380 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.381 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.381 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.381 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.381 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.381 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.381 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.381 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.381 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.381 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.382 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.382 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.382 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.486 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.487 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.487 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.487 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.487 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.487 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.487 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.487 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.487 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.487 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.488 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.488 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.488 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.488 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.488 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.488 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.488 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.488 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.488 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.489 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.489 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.489 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.489 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.489 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.489 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.489 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.489 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.489 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.490 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.490 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.490 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.490 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rook", 2023-11-16T17:00:21.490 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.490 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.490 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.490 INFO:teuthology.orchestra.run.smithi044.stdout: "drive_group_interval": { 2023-11-16T17:00:21.491 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "drive_group_interval", 2023-11-16T17:00:21.491 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:21.491 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.491 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.491 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "300.0", 2023-11-16T17:00:21.491 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.491 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.491 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.492 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-11-16T17:00:21.492 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.492 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.492 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.492 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.492 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.492 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.492 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.492 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.493 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.493 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.493 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.493 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.493 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.493 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.493 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.493 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.493 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.494 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.494 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.494 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.494 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.494 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.494 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.494 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.494 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.494 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.495 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.495 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.495 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.495 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.495 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.495 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.495 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.495 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.495 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.496 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.496 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.496 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.496 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.496 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.496 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.496 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.496 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.497 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.497 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.497 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.497 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.497 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.497 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.497 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.497 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.497 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.498 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.498 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.498 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.498 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.498 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.498 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.498 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.498 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.498 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.499 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.499 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.499 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.499 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.499 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.499 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.499 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.499 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.500 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.500 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.500 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.500 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.500 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.500 INFO:teuthology.orchestra.run.smithi044.stdout: "storage_class": { 2023-11-16T17:00:21.500 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "storage_class", 2023-11-16T17:00:21.500 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.500 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.500 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.501 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "local", 2023-11-16T17:00:21.501 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.501 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.501 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.501 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-11-16T17:00:21.501 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.501 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.501 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.502 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.502 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.502 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.502 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.502 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "selftest", 2023-11-16T17:00:21.502 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.502 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.502 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.502 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.503 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.503 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.503 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.503 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.503 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.503 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.503 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.503 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.503 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.504 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.504 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.504 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.504 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.504 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.504 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.504 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.504 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.504 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.506 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.506 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.506 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.506 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.506 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.506 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.506 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.506 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.507 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.507 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.507 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.507 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.507 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.507 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.507 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.507 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.507 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.508 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.508 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.508 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.508 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.508 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.508 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.508 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.508 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.509 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.509 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.509 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.509 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.509 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.509 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.509 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.509 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.509 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.510 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.510 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.510 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.510 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.510 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.510 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.510 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.510 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.511 INFO:teuthology.orchestra.run.smithi044.stdout: "roption1": { 2023-11-16T17:00:21.511 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption1", 2023-11-16T17:00:21.511 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.511 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.511 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.511 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.511 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.511 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.511 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.512 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.512 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.512 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.512 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.512 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.512 INFO:teuthology.orchestra.run.smithi044.stdout: "roption2": { 2023-11-16T17:00:21.512 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption2", 2023-11-16T17:00:21.512 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.512 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.513 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.513 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "xyz", 2023-11-16T17:00:21.513 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.513 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.513 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.513 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.513 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.513 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.513 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.514 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.514 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption1": { 2023-11-16T17:00:21.514 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption1", 2023-11-16T17:00:21.514 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.514 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.514 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.514 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.514 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.515 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.515 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.515 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.515 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.515 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.515 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.515 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.515 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption2": { 2023-11-16T17:00:21.515 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption2", 2023-11-16T17:00:21.516 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.516 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.516 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.516 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.516 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.516 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.516 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.516 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.516 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.517 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.517 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.517 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.517 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption3": { 2023-11-16T17:00:21.517 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption3", 2023-11-16T17:00:21.517 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:21.517 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.517 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.517 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.518 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.518 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.518 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.518 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.518 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.518 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.518 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.518 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.518 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption4": { 2023-11-16T17:00:21.519 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption4", 2023-11-16T17:00:21.519 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.519 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.519 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.519 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.519 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.519 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.519 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.520 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.520 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.520 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.520 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.520 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.520 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption5": { 2023-11-16T17:00:21.520 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption5", 2023-11-16T17:00:21.520 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.520 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.521 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.521 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.521 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.521 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.521 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.521 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.521 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.521 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.522 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.522 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.522 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption6": { 2023-11-16T17:00:21.522 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption6", 2023-11-16T17:00:21.522 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.522 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.522 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.522 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.523 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.523 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.523 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.523 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.523 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.523 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.523 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.523 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.523 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption7": { 2023-11-16T17:00:21.524 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption7", 2023-11-16T17:00:21.524 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.524 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.524 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.524 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.524 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2023-11-16T17:00:21.524 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "42", 2023-11-16T17:00:21.524 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.525 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.525 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.525 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.525 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.525 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.525 INFO:teuthology.orchestra.run.smithi044.stdout: "testkey": { 2023-11-16T17:00:21.525 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testkey", 2023-11-16T17:00:21.525 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.525 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.526 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.526 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.526 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.526 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.526 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.526 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.526 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.526 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.526 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.527 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.527 INFO:teuthology.orchestra.run.smithi044.stdout: "testlkey": { 2023-11-16T17:00:21.527 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testlkey", 2023-11-16T17:00:21.527 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.527 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.527 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.527 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.527 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.528 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.528 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.528 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.528 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.528 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.528 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.528 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.528 INFO:teuthology.orchestra.run.smithi044.stdout: "testnewline": { 2023-11-16T17:00:21.528 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testnewline", 2023-11-16T17:00:21.529 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.529 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.529 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.529 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.529 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.529 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.529 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.529 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.529 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.530 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.530 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.530 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.530 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.530 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.530 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.530 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snap_schedule", 2023-11-16T17:00:21.530 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.530 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.531 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.531 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_m_granularity": { 2023-11-16T17:00:21.531 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_m_granularity", 2023-11-16T17:00:21.531 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.531 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.531 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.531 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.531 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.531 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.532 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.532 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow minute scheduled snapshots", 2023-11-16T17:00:21.532 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.532 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.532 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.532 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.532 INFO:teuthology.orchestra.run.smithi044.stdout: "dump_on_update": { 2023-11-16T17:00:21.533 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dump_on_update", 2023-11-16T17:00:21.533 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.533 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.533 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.533 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.533 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.533 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.533 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.533 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "dump database to debug log on update", 2023-11-16T17:00:21.534 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.534 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.534 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.534 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.534 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.534 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.534 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.535 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.535 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.535 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.535 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.535 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.535 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.535 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.535 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.535 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.536 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.536 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.536 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.536 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.536 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.536 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.536 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.536 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.536 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.537 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.537 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.537 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.537 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.537 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.537 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.537 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.537 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.537 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.538 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.538 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.538 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.538 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.538 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.538 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.538 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.538 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.539 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.539 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.539 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.539 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.539 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.539 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.539 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.539 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.539 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.540 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.540 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.540 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.540 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.540 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.540 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.540 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.540 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.540 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.541 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.541 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.541 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.541 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.541 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.541 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.541 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.541 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.542 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.542 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.542 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.542 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.542 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.542 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.542 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.542 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.542 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.543 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stats", 2023-11-16T17:00:21.543 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.543 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.543 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.543 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.543 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.543 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.543 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.544 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.544 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.544 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.544 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.544 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.544 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.544 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.544 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.544 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.545 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.545 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.545 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.545 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.545 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.545 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.545 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.545 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.545 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.546 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.546 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.546 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.546 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.546 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.546 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.546 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.546 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.546 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.547 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.547 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.547 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.547 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.547 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.547 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.547 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.547 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.548 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.548 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.548 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.548 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.548 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.548 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.548 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.548 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.548 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.549 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.549 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.549 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.549 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.549 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.549 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.549 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.549 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.549 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.550 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.550 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.550 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.550 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.550 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.550 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.550 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.550 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.550 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.551 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.551 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.551 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.551 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.551 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.551 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.551 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.551 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "status", 2023-11-16T17:00:21.551 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.552 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.552 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.552 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.552 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.552 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.552 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.552 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.552 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.553 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.553 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.553 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.553 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.553 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.553 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.553 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.553 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.553 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.554 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.554 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.554 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.554 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.554 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.554 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.554 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.554 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.555 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.555 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.555 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.555 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.555 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.555 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.555 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.555 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.555 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.556 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.556 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.556 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.556 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.556 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.556 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.556 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.556 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.556 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.557 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.557 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.557 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.557 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.557 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.557 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.557 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.557 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.557 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.558 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.558 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.558 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.558 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.558 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.558 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.558 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.558 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.559 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.559 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.559 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.559 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.559 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.559 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.559 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.559 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.559 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.560 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.560 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.560 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.560 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.560 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.560 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.560 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telegraf", 2023-11-16T17:00:21.560 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.561 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.561 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.561 INFO:teuthology.orchestra.run.smithi044.stdout: "address": { 2023-11-16T17:00:21.561 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "address", 2023-11-16T17:00:21.561 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.561 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.561 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.561 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-11-16T17:00:21.561 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.562 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.562 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.562 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.562 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.562 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.562 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.562 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.562 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2023-11-16T17:00:21.563 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2023-11-16T17:00:21.563 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.563 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.563 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.563 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15", 2023-11-16T17:00:21.563 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.563 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.563 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.563 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.564 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.564 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.564 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.564 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.564 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.564 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.564 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.564 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.565 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.565 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.565 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.565 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.565 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.565 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.565 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.565 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.565 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.566 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.566 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.566 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.566 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.566 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.566 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.566 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.566 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.567 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.567 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.567 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.567 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.567 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.567 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.567 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.567 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.567 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.568 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.568 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.568 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.568 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.568 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.568 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.568 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.568 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.569 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.569 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.569 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.569 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.569 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.569 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.569 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.569 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.569 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.570 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.570 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.570 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.570 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.570 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.570 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.570 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.570 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.571 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.571 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.571 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.571 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.571 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.571 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.571 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.571 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.572 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.572 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.572 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.572 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.572 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.572 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.572 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.572 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.572 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.573 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.573 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telemetry", 2023-11-16T17:00:21.573 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.573 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.573 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.573 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_basic": { 2023-11-16T17:00:21.573 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_basic", 2023-11-16T17:00:21.573 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.574 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.574 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.574 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.574 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.574 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.574 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.574 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share basic cluster information (size, version)", 2023-11-16T17:00:21.574 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.574 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.575 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.575 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.575 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_crash": { 2023-11-16T17:00:21.575 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_crash", 2023-11-16T17:00:21.575 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.575 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.575 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.575 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.575 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.576 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.576 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.576 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-11-16T17:00:21.576 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.576 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.576 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.576 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.576 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_device": { 2023-11-16T17:00:21.576 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_device", 2023-11-16T17:00:21.577 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.577 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.577 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.577 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.577 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.577 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.577 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.577 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-11-16T17:00:21.578 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.578 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.578 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.578 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.578 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_ident": { 2023-11-16T17:00:21.578 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_ident", 2023-11-16T17:00:21.578 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.578 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.578 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.579 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.579 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.579 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.579 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.579 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-11-16T17:00:21.579 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.579 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.579 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.579 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.580 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_perf": { 2023-11-16T17:00:21.580 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_perf", 2023-11-16T17:00:21.580 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.580 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.580 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.580 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.580 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.580 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.580 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.581 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share various performance metrics of a cluster", 2023-11-16T17:00:21.581 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.581 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.581 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.581 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.581 INFO:teuthology.orchestra.run.smithi044.stdout: "contact": { 2023-11-16T17:00:21.581 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "contact", 2023-11-16T17:00:21.581 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.581 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.582 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.582 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.582 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.582 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.582 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.582 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.582 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.582 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.583 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.583 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.583 INFO:teuthology.orchestra.run.smithi044.stdout: "description": { 2023-11-16T17:00:21.583 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "description", 2023-11-16T17:00:21.583 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.583 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.583 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.583 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.583 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.583 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.584 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.584 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.584 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.584 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.584 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.584 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.584 INFO:teuthology.orchestra.run.smithi044.stdout: "device_url": { 2023-11-16T17:00:21.584 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_url", 2023-11-16T17:00:21.585 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.585 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.585 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.585 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-11-16T17:00:21.585 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.585 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.585 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.585 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.585 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.586 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.586 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.586 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.586 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2023-11-16T17:00:21.586 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2023-11-16T17:00:21.586 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.586 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.586 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.587 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.587 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.587 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.587 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.587 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.587 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.587 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.587 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.587 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.588 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2023-11-16T17:00:21.588 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2023-11-16T17:00:21.588 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.588 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.588 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.588 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "24", 2023-11-16T17:00:21.588 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "8", 2023-11-16T17:00:21.588 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.588 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.589 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.589 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.589 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.589 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.589 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.589 INFO:teuthology.orchestra.run.smithi044.stdout: "last_opt_revision": { 2023-11-16T17:00:21.589 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "last_opt_revision", 2023-11-16T17:00:21.589 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.590 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.590 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.590 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1", 2023-11-16T17:00:21.590 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.590 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.590 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.590 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.590 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.590 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.591 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.591 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.591 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard": { 2023-11-16T17:00:21.591 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard", 2023-11-16T17:00:21.591 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.591 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.591 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.591 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.591 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.592 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.592 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.592 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.592 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.592 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.592 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.592 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.592 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard_description": { 2023-11-16T17:00:21.593 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard_description", 2023-11-16T17:00:21.593 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.593 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.593 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.593 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.593 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.593 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.593 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.593 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.594 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.594 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.594 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.594 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.594 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.594 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.594 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.594 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.594 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.595 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.595 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.595 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.595 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.595 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.595 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.595 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.595 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.596 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.596 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.596 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.596 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.596 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.596 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.596 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.596 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.596 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.597 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.597 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.597 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.597 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.597 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.597 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.597 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.597 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.597 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.598 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.598 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.598 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.598 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.598 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.598 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.598 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.598 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.599 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.599 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.599 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.599 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.599 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.599 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.599 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.599 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.599 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.600 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.600 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.600 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.600 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.600 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.600 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.600 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.600 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.600 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.601 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.601 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.601 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.601 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.601 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.601 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.601 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.601 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.602 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.602 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.602 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.602 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.602 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.602 INFO:teuthology.orchestra.run.smithi044.stdout: "organization": { 2023-11-16T17:00:21.602 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "organization", 2023-11-16T17:00:21.602 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.602 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.603 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.603 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.603 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.603 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.603 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.603 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.603 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.603 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.603 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.604 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.604 INFO:teuthology.orchestra.run.smithi044.stdout: "proxy": { 2023-11-16T17:00:21.604 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "proxy", 2023-11-16T17:00:21.604 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.604 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.604 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.604 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.604 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.605 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.605 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.605 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.605 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.605 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.605 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.605 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.605 INFO:teuthology.orchestra.run.smithi044.stdout: "url": { 2023-11-16T17:00:21.606 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url", 2023-11-16T17:00:21.606 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.606 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.606 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.606 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-11-16T17:00:21.606 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.606 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.606 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.607 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.607 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.607 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.607 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.607 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.607 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.607 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.607 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.607 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "test_orchestrator", 2023-11-16T17:00:21.608 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.608 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.608 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.608 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.608 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.608 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.608 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.608 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.609 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.609 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.609 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.609 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.609 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.609 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.609 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.609 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.609 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.610 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.610 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.610 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.610 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.610 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.610 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.610 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.610 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.610 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.611 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.611 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.611 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.611 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.611 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.611 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.611 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.611 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.612 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.612 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.612 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.612 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.612 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.612 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.612 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.612 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.612 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.613 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.613 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.613 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.613 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.613 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.613 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.613 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.613 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.613 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.614 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.614 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.614 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.614 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.614 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.614 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.614 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.614 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.615 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.615 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.615 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.615 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.615 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.615 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.615 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.615 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.615 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.616 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.616 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.616 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.616 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.616 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.616 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.616 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.616 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "volumes", 2023-11-16T17:00:21.617 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.617 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.617 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.617 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.617 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.617 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.617 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.617 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.617 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.618 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.618 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.618 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.618 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.618 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.618 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.618 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.618 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.619 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.619 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.619 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.619 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.619 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.619 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.619 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.619 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.619 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.620 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.620 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.620 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.620 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.620 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.620 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.620 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.620 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.620 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.621 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.621 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.621 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.621 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.621 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.621 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.621 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.621 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.621 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.622 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.622 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.622 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.622 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.622 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.622 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.622 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.622 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.623 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.623 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.623 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.623 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.623 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.623 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.623 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.623 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.623 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.624 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.624 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.624 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.624 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.624 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.624 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.624 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.624 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.624 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.625 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.625 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.625 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.625 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_clones": { 2023-11-16T17:00:21.625 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_clones", 2023-11-16T17:00:21.625 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.625 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.625 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.625 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4", 2023-11-16T17:00:21.626 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.626 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.626 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.626 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Number of asynchronous cloner threads", 2023-11-16T17:00:21.626 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.626 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.626 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.626 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.626 INFO:teuthology.orchestra.run.smithi044.stdout: "snapshot_clone_delay": { 2023-11-16T17:00:21.627 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snapshot_clone_delay", 2023-11-16T17:00:21.627 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.627 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.627 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.627 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2023-11-16T17:00:21.627 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.627 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.627 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.628 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-11-16T17:00:21.628 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.628 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.628 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.628 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.628 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.628 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.628 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.628 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix", 2023-11-16T17:00:21.629 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.629 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.629 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.629 INFO:teuthology.orchestra.run.smithi044.stdout: "discovery_interval": { 2023-11-16T17:00:21.629 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "discovery_interval", 2023-11-16T17:00:21.629 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2023-11-16T17:00:21.629 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.629 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.629 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "100", 2023-11-16T17:00:21.630 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.630 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.630 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.630 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.630 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.630 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.630 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.630 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.631 INFO:teuthology.orchestra.run.smithi044.stdout: "identifier": { 2023-11-16T17:00:21.631 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "identifier", 2023-11-16T17:00:21.631 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.631 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.631 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.631 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.631 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.631 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.631 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.632 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.632 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.632 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.632 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.632 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.632 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2023-11-16T17:00:21.632 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2023-11-16T17:00:21.632 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.632 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.633 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.633 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2023-11-16T17:00:21.633 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.633 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.633 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.633 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.633 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.633 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.633 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.634 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.634 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.634 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.634 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.634 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.634 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.634 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.634 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.635 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.635 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.635 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.635 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.635 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.635 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.635 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.635 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.635 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.636 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.636 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.636 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.636 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.636 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.636 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.636 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.636 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.636 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.636 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.637 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.637 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.637 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.637 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.637 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.637 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.637 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.637 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.637 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.638 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.638 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.638 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.638 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.638 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.638 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.638 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.638 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.639 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.639 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.639 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.639 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.639 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.639 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.639 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.639 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.639 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.640 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.640 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.640 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.640 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.640 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.640 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.640 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.640 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.640 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.641 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.641 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.641 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.641 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.641 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.641 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.641 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.641 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.641 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.642 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_host": { 2023-11-16T17:00:21.642 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_host", 2023-11-16T17:00:21.642 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.642 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.642 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.642 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.642 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.642 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.642 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.643 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.643 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.643 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.643 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.643 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.643 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_port": { 2023-11-16T17:00:21.643 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_port", 2023-11-16T17:00:21.643 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.643 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.644 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.644 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10051", 2023-11-16T17:00:21.644 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.644 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.644 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.644 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.644 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.644 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.644 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.645 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.645 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_sender": { 2023-11-16T17:00:21.645 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_sender", 2023-11-16T17:00:21.645 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.645 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.645 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.645 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-11-16T17:00:21.645 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.646 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.646 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.646 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.646 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.646 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.646 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.646 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.646 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.646 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.647 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2023-11-16T17:00:21.647 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.647 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.647 INFO:teuthology.orchestra.run.smithi044.stdout: "gid": 25445, 2023-11-16T17:00:21.647 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "y", 2023-11-16T17:00:21.647 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr_features": 4540138322906710015, 2023-11-16T17:00:21.647 INFO:teuthology.orchestra.run.smithi044.stdout: "available_modules": [ 2023-11-16T17:00:21.647 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.647 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "alerts", 2023-11-16T17:00:21.648 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.648 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.648 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.648 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2023-11-16T17:00:21.648 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2023-11-16T17:00:21.648 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.648 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.648 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.648 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2023-11-16T17:00:21.649 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.649 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.649 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.649 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How frequently to reexamine health status", 2023-11-16T17:00:21.649 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.649 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.649 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.649 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.649 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.650 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.650 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.650 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.650 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.650 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.650 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.650 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.650 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.651 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.651 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.651 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.651 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.651 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.651 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.651 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.651 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.651 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.652 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.652 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.652 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.652 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.652 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.652 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.652 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.652 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.652 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.653 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.653 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.653 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.653 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.653 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.653 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.653 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.653 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.653 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.654 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.654 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.654 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.654 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.654 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.654 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.654 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.655 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.655 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.655 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.655 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.655 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.655 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.655 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.655 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.655 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.655 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.656 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.656 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.656 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.656 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.656 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.656 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.656 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.656 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.656 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.657 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.657 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.657 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.657 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.657 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.657 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.657 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.657 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.658 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_destination": { 2023-11-16T17:00:21.658 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_destination", 2023-11-16T17:00:21.658 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.658 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.658 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.658 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.658 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.658 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.659 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.659 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email address to send alerts to", 2023-11-16T17:00:21.659 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.659 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.659 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.659 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.659 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_from_name": { 2023-11-16T17:00:21.659 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_from_name", 2023-11-16T17:00:21.659 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.659 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.660 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.660 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "Ceph", 2023-11-16T17:00:21.660 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.660 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.660 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.660 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email From: name", 2023-11-16T17:00:21.660 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.660 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.661 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.661 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.661 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_host": { 2023-11-16T17:00:21.661 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_host", 2023-11-16T17:00:21.661 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.661 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.661 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.661 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.661 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.662 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.662 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.662 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP server", 2023-11-16T17:00:21.662 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.662 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.662 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.662 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.662 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_password": { 2023-11-16T17:00:21.663 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_password", 2023-11-16T17:00:21.663 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.663 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.663 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.663 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.663 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.663 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.663 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.663 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Password to authenticate with", 2023-11-16T17:00:21.664 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.664 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.664 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.664 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.664 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_port": { 2023-11-16T17:00:21.664 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_port", 2023-11-16T17:00:21.664 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.664 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.664 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.665 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "465", 2023-11-16T17:00:21.665 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.665 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.665 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.665 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP port", 2023-11-16T17:00:21.665 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.665 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.665 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.665 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.666 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_sender": { 2023-11-16T17:00:21.666 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_sender", 2023-11-16T17:00:21.666 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.666 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.666 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.666 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.666 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.666 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.666 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.667 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP envelope sender", 2023-11-16T17:00:21.667 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.667 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.667 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.667 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.667 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_ssl": { 2023-11-16T17:00:21.667 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_ssl", 2023-11-16T17:00:21.667 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.667 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.668 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.668 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.668 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.668 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.668 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.668 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use SSL to connect to SMTP server", 2023-11-16T17:00:21.668 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.668 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.668 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.669 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.669 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_user": { 2023-11-16T17:00:21.669 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_user", 2023-11-16T17:00:21.669 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.669 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.669 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.669 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.669 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.669 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.670 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.670 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "User to authenticate as", 2023-11-16T17:00:21.670 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.670 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.670 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.670 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.670 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.670 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.671 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.671 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "balancer", 2023-11-16T17:00:21.671 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.671 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.671 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.671 INFO:teuthology.orchestra.run.smithi044.stdout: "active": { 2023-11-16T17:00:21.671 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "active", 2023-11-16T17:00:21.671 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.671 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.672 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.672 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.672 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.672 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.672 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.672 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically balance PGs across cluster", 2023-11-16T17:00:21.672 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.672 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.672 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.673 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.673 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_time": { 2023-11-16T17:00:21.673 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_time", 2023-11-16T17:00:21.673 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.673 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.673 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.673 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0000", 2023-11-16T17:00:21.673 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.673 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.674 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.674 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "beginning time of day to automatically balance", 2023-11-16T17:00:21.674 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T17:00:21.674 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.674 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.674 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.674 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_weekday": { 2023-11-16T17:00:21.674 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_weekday", 2023-11-16T17:00:21.675 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2023-11-16T17:00:21.675 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.675 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.675 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2023-11-16T17:00:21.675 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2023-11-16T17:00:21.675 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2023-11-16T17:00:21.675 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.675 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-11-16T17:00:21.675 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T17:00:21.676 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.676 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.676 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.676 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_max_iterations": { 2023-11-16T17:00:21.676 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_max_iterations", 2023-11-16T17:00:21.676 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2023-11-16T17:00:21.676 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.676 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.676 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "25", 2023-11-16T17:00:21.677 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2023-11-16T17:00:21.677 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "250", 2023-11-16T17:00:21.677 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.677 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-11-16T17:00:21.677 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.677 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.677 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.677 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.678 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_metrics": { 2023-11-16T17:00:21.678 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_metrics", 2023-11-16T17:00:21.678 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.678 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.678 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.678 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "pgs,objects,bytes", 2023-11-16T17:00:21.678 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.678 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.678 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.679 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-11-16T17:00:21.679 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-11-16T17:00:21.679 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.679 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.679 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.679 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_step": { 2023-11-16T17:00:21.679 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_step", 2023-11-16T17:00:21.679 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:21.679 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.680 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.680 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.5", 2023-11-16T17:00:21.680 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0.001", 2023-11-16T17:00:21.680 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "0.999", 2023-11-16T17:00:21.680 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.680 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "aggressiveness of optimization", 2023-11-16T17:00:21.680 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-11-16T17:00:21.680 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.680 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.681 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.681 INFO:teuthology.orchestra.run.smithi044.stdout: "end_time": { 2023-11-16T17:00:21.681 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_time", 2023-11-16T17:00:21.681 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.681 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.681 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.681 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2359", 2023-11-16T17:00:21.681 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.681 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.682 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.682 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ending time of day to automatically balance", 2023-11-16T17:00:21.682 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T17:00:21.682 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.682 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.682 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.682 INFO:teuthology.orchestra.run.smithi044.stdout: "end_weekday": { 2023-11-16T17:00:21.682 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_weekday", 2023-11-16T17:00:21.682 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2023-11-16T17:00:21.683 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.683 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.683 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2023-11-16T17:00:21.683 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2023-11-16T17:00:21.683 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2023-11-16T17:00:21.683 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.683 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-11-16T17:00:21.683 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T17:00:21.683 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.684 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.684 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.684 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.684 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.684 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.684 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.684 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.684 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.684 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.685 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.685 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.685 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.685 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.685 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.685 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.685 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.685 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.685 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.686 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.686 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.686 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.686 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.686 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.686 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.686 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.686 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.686 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.687 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.687 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.687 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.687 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.687 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.687 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.687 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.687 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.687 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.688 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.688 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.688 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.688 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.688 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.688 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.688 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.688 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.688 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.689 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.689 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.689 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.689 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.689 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.689 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.689 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.689 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.689 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.690 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.690 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.690 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.690 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.690 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.690 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.690 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.690 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.691 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.691 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.691 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.691 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.691 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.691 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.691 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.691 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.691 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.692 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.692 INFO:teuthology.orchestra.run.smithi044.stdout: "min_score": { 2023-11-16T17:00:21.692 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_score", 2023-11-16T17:00:21.692 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:21.692 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.692 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.692 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2023-11-16T17:00:21.692 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.692 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.693 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.693 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-11-16T17:00:21.693 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.693 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.693 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.693 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.693 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2023-11-16T17:00:21.693 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2023-11-16T17:00:21.693 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.694 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.694 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.694 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "upmap", 2023-11-16T17:00:21.694 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.694 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.694 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.694 INFO:teuthology.orchestra.run.smithi044.stdout: "crush-compat", 2023-11-16T17:00:21.694 INFO:teuthology.orchestra.run.smithi044.stdout: "none", 2023-11-16T17:00:21.695 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap" 2023-11-16T17:00:21.695 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.695 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Balancer mode", 2023-11-16T17:00:21.695 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.695 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.695 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.695 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.695 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_ids": { 2023-11-16T17:00:21.695 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_ids", 2023-11-16T17:00:21.696 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.696 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.696 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.696 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.696 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.696 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.696 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.696 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-11-16T17:00:21.696 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.697 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.697 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.697 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.697 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2023-11-16T17:00:21.697 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2023-11-16T17:00:21.697 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.697 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.697 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.697 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2023-11-16T17:00:21.698 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.698 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.698 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.698 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-11-16T17:00:21.698 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.698 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.698 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.698 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.698 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_deviation": { 2023-11-16T17:00:21.699 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_deviation", 2023-11-16T17:00:21.699 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.699 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.699 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.699 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2023-11-16T17:00:21.699 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2023-11-16T17:00:21.699 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.699 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.699 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "deviation below which no optimization is attempted", 2023-11-16T17:00:21.700 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-11-16T17:00:21.700 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.700 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.700 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.700 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_optimizations": { 2023-11-16T17:00:21.700 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_optimizations", 2023-11-16T17:00:21.700 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2023-11-16T17:00:21.700 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.700 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.701 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:21.701 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.701 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.701 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.701 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-11-16T17:00:21.701 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.701 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.701 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.701 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.702 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.702 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.702 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.702 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cephadm", 2023-11-16T17:00:21.702 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.702 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.702 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.702 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_down_multiplier": { 2023-11-16T17:00:21.703 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_down_multiplier", 2023-11-16T17:00:21.703 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:21.703 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.703 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.703 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2023-11-16T17:00:21.703 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.703 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.703 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.703 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-11-16T17:00:21.704 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.704 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.704 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.704 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.704 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_refresh_rate": { 2023-11-16T17:00:21.704 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_refresh_rate", 2023-11-16T17:00:21.704 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.704 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.704 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.705 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "20", 2023-11-16T17:00:21.705 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.705 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.705 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.705 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-11-16T17:00:21.705 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.705 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.705 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.705 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.706 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_starting_port": { 2023-11-16T17:00:21.706 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_starting_port", 2023-11-16T17:00:21.706 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.706 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.706 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.706 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4721", 2023-11-16T17:00:21.706 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.706 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.706 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.707 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-11-16T17:00:21.707 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.707 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.707 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.707 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.707 INFO:teuthology.orchestra.run.smithi044.stdout: "alertmanager_web_password": { 2023-11-16T17:00:21.707 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "alertmanager_web_password", 2023-11-16T17:00:21.707 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.707 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.708 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.708 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:21.708 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.708 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.708 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.708 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Alertmanager web password", 2023-11-16T17:00:21.708 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.708 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.708 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.709 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.709 INFO:teuthology.orchestra.run.smithi044.stdout: "alertmanager_web_user": { 2023-11-16T17:00:21.709 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "alertmanager_web_user", 2023-11-16T17:00:21.709 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.709 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.709 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.709 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:21.709 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.709 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.710 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.710 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Alertmanager web user", 2023-11-16T17:00:21.710 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.710 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.710 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.710 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.710 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_ptrace": { 2023-11-16T17:00:21.710 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_ptrace", 2023-11-16T17:00:21.711 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.711 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.711 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.711 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.711 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.711 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.711 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.711 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-11-16T17:00:21.711 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-11-16T17:00:21.712 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.712 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.712 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.712 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_interval": { 2023-11-16T17:00:21.712 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_interval", 2023-11-16T17:00:21.712 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.712 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.712 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.712 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2023-11-16T17:00:21.713 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.713 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.713 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.713 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to autotune daemon memory", 2023-11-16T17:00:21.713 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.713 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.713 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.713 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.713 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_memory_target_ratio": { 2023-11-16T17:00:21.714 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_memory_target_ratio", 2023-11-16T17:00:21.714 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:21.714 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.714 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.714 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.7", 2023-11-16T17:00:21.714 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.714 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.714 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.715 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-11-16T17:00:21.715 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.715 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.715 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.715 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.715 INFO:teuthology.orchestra.run.smithi044.stdout: "cgroups_split": { 2023-11-16T17:00:21.715 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cgroups_split", 2023-11-16T17:00:21.715 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.715 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.716 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.716 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.716 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.716 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.716 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.716 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-11-16T17:00:21.716 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.716 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.716 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.717 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.717 INFO:teuthology.orchestra.run.smithi044.stdout: "config_checks_enabled": { 2023-11-16T17:00:21.717 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_checks_enabled", 2023-11-16T17:00:21.717 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.717 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.717 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.717 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.717 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.717 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.718 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.718 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-11-16T17:00:21.718 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.718 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.718 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.718 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.718 INFO:teuthology.orchestra.run.smithi044.stdout: "config_dashboard": { 2023-11-16T17:00:21.718 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_dashboard", 2023-11-16T17:00:21.719 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.719 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.719 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.719 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.719 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.719 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.719 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.719 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-11-16T17:00:21.719 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.720 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.720 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.720 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.720 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_alertmanager": { 2023-11-16T17:00:21.720 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_alertmanager", 2023-11-16T17:00:21.720 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.720 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.720 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.720 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-11-16T17:00:21.721 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.721 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.721 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.721 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:21.721 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.721 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.721 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.721 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.721 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_base": { 2023-11-16T17:00:21.722 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_base", 2023-11-16T17:00:21.722 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.722 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.722 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.722 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/ceph", 2023-11-16T17:00:21.722 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.722 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.722 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.722 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Container image name, without the tag", 2023-11-16T17:00:21.723 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.723 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.723 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.723 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.723 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_elasticsearch": { 2023-11-16T17:00:21.723 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_elasticsearch", 2023-11-16T17:00:21.723 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.723 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.723 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.724 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-11-16T17:00:21.724 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.724 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.724 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.724 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "elasticsearch container image", 2023-11-16T17:00:21.724 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.724 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.724 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.724 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.725 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_grafana": { 2023-11-16T17:00:21.725 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_grafana", 2023-11-16T17:00:21.725 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.725 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.725 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.725 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-11-16T17:00:21.725 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.725 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.725 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.726 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:21.726 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.726 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.726 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.726 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.726 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_haproxy": { 2023-11-16T17:00:21.726 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_haproxy", 2023-11-16T17:00:21.726 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.726 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.727 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.727 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-11-16T17:00:21.727 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.727 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.727 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.727 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "HAproxy container image", 2023-11-16T17:00:21.727 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.727 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.728 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.728 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.728 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_agent": { 2023-11-16T17:00:21.728 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_agent", 2023-11-16T17:00:21.728 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.728 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.728 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.728 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-11-16T17:00:21.729 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.729 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.729 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.729 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger agent container image", 2023-11-16T17:00:21.729 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.729 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.729 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.729 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.730 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_collector": { 2023-11-16T17:00:21.730 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_collector", 2023-11-16T17:00:21.730 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.730 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.730 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.730 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-11-16T17:00:21.730 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.730 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.731 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.731 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger collector container image", 2023-11-16T17:00:21.731 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.731 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.731 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.731 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.731 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_query": { 2023-11-16T17:00:21.731 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_query", 2023-11-16T17:00:21.731 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.732 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.732 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.732 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-11-16T17:00:21.732 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.732 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.732 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.732 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger query container image", 2023-11-16T17:00:21.732 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.732 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.733 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.733 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.733 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_keepalived": { 2023-11-16T17:00:21.733 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_keepalived", 2023-11-16T17:00:21.733 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.733 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.733 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.733 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-11-16T17:00:21.733 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.734 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.734 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.734 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Keepalived container image", 2023-11-16T17:00:21.734 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.734 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.734 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.734 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.734 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_loki": { 2023-11-16T17:00:21.735 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_loki", 2023-11-16T17:00:21.735 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.735 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.735 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.735 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-11-16T17:00:21.735 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.735 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.735 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.736 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Loki container image", 2023-11-16T17:00:21.736 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.736 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.736 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.736 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.736 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_node_exporter": { 2023-11-16T17:00:21.736 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_node_exporter", 2023-11-16T17:00:21.736 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.737 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.737 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.737 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-11-16T17:00:21.737 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.737 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.737 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.737 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:21.737 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.737 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.738 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.738 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.738 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_prometheus": { 2023-11-16T17:00:21.738 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_prometheus", 2023-11-16T17:00:21.738 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.738 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.738 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.738 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-11-16T17:00:21.739 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.739 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.739 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.739 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:21.739 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.739 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.739 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.739 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.740 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_promtail": { 2023-11-16T17:00:21.740 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_promtail", 2023-11-16T17:00:21.740 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.740 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.740 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.740 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-11-16T17:00:21.740 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.740 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.740 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.741 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Promtail container image", 2023-11-16T17:00:21.741 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.741 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.741 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.741 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.741 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_snmp_gateway": { 2023-11-16T17:00:21.741 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_snmp_gateway", 2023-11-16T17:00:21.741 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.741 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.742 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.742 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-11-16T17:00:21.742 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.742 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.742 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.742 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SNMP Gateway container image", 2023-11-16T17:00:21.742 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.742 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.743 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.743 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.743 INFO:teuthology.orchestra.run.smithi044.stdout: "container_init": { 2023-11-16T17:00:21.743 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_init", 2023-11-16T17:00:21.743 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.743 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.743 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.743 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.744 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.744 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.744 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.744 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Run podman/docker with `--init`", 2023-11-16T17:00:21.744 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.744 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.744 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.744 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.744 INFO:teuthology.orchestra.run.smithi044.stdout: "daemon_cache_timeout": { 2023-11-16T17:00:21.745 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "daemon_cache_timeout", 2023-11-16T17:00:21.745 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.745 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.745 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.745 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2023-11-16T17:00:21.745 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.745 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.745 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.746 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-11-16T17:00:21.746 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.746 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.746 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.746 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.746 INFO:teuthology.orchestra.run.smithi044.stdout: "default_cephadm_command_timeout": { 2023-11-16T17:00:21.746 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_cephadm_command_timeout", 2023-11-16T17:00:21.746 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.747 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.747 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.747 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "900", 2023-11-16T17:00:21.747 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.747 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.747 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.747 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-11-16T17:00:21.747 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.748 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.748 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.748 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.748 INFO:teuthology.orchestra.run.smithi044.stdout: "default_registry": { 2023-11-16T17:00:21.748 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_registry", 2023-11-16T17:00:21.748 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.748 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.748 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.748 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io", 2023-11-16T17:00:21.749 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.749 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.749 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.749 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-11-16T17:00:21.749 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.749 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.749 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.749 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.750 INFO:teuthology.orchestra.run.smithi044.stdout: "device_cache_timeout": { 2023-11-16T17:00:21.750 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_cache_timeout", 2023-11-16T17:00:21.750 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.750 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.750 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.750 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1800", 2023-11-16T17:00:21.750 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.750 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.751 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.751 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache device inventory", 2023-11-16T17:00:21.751 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.751 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.751 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.751 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.751 INFO:teuthology.orchestra.run.smithi044.stdout: "device_enhanced_scan": { 2023-11-16T17:00:21.751 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_enhanced_scan", 2023-11-16T17:00:21.751 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.752 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.752 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.752 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.752 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.752 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.752 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.752 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use libstoragemgmt during device scans", 2023-11-16T17:00:21.752 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.753 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.753 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.753 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.753 INFO:teuthology.orchestra.run.smithi044.stdout: "facts_cache_timeout": { 2023-11-16T17:00:21.753 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "facts_cache_timeout", 2023-11-16T17:00:21.753 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.753 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.753 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.754 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2023-11-16T17:00:21.754 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.754 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.754 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.754 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache host facts data", 2023-11-16T17:00:21.754 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.754 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.754 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.755 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.755 INFO:teuthology.orchestra.run.smithi044.stdout: "host_check_interval": { 2023-11-16T17:00:21.755 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "host_check_interval", 2023-11-16T17:00:21.755 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.755 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.755 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.755 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2023-11-16T17:00:21.755 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.755 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.756 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.756 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to perform a host check", 2023-11-16T17:00:21.756 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.756 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.756 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.756 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.756 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.756 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.757 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.757 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.757 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.757 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.757 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.757 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.757 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.757 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.758 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.758 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.758 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.758 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.758 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.758 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.758 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.758 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.759 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.759 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.759 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.759 INFO:teuthology.orchestra.run.smithi044.stdout: "log_refresh_metadata": { 2023-11-16T17:00:21.759 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_refresh_metadata", 2023-11-16T17:00:21.759 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.759 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.759 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.759 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.760 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.760 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.760 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.760 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-11-16T17:00:21.760 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.760 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.760 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.760 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.761 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.761 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.761 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.761 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.761 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.761 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.761 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.761 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.762 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.762 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-11-16T17:00:21.762 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.762 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.762 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.762 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.762 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.762 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.763 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.763 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.763 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.763 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.763 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.763 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.763 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.763 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.764 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.764 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.764 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.764 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.764 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.764 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.764 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.764 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.764 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.765 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.765 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.765 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.765 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.765 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.765 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.765 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.765 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.766 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.766 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.766 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.766 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.766 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.766 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.766 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.766 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.767 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf": { 2023-11-16T17:00:21.767 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-11-16T17:00:21.767 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.767 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.767 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.767 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.767 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.767 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.767 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.768 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-11-16T17:00:21.768 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.768 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.768 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.768 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.768 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-11-16T17:00:21.768 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-11-16T17:00:21.768 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.768 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.769 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.769 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "*", 2023-11-16T17:00:21.769 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.769 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.769 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.769 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-11-16T17:00:21.769 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.769 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.769 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.770 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.770 INFO:teuthology.orchestra.run.smithi044.stdout: "max_count_per_host": { 2023-11-16T17:00:21.770 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_count_per_host", 2023-11-16T17:00:21.770 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.770 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.770 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.770 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:21.770 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.771 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.771 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.771 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of daemons per service per host", 2023-11-16T17:00:21.771 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.771 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.771 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.771 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.771 INFO:teuthology.orchestra.run.smithi044.stdout: "max_osd_draining_count": { 2023-11-16T17:00:21.772 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_osd_draining_count", 2023-11-16T17:00:21.772 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.772 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.772 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.772 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:21.772 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.772 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.772 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.773 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-11-16T17:00:21.773 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.773 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.773 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.773 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.773 INFO:teuthology.orchestra.run.smithi044.stdout: "migration_current": { 2023-11-16T17:00:21.773 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "migration_current", 2023-11-16T17:00:21.773 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.774 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.774 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.774 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.774 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.774 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.774 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.774 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "internal - do not modify", 2023-11-16T17:00:21.774 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.775 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.775 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.775 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.775 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2023-11-16T17:00:21.775 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2023-11-16T17:00:21.775 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.775 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.775 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.776 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "root", 2023-11-16T17:00:21.776 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.776 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.776 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.776 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm-package", 2023-11-16T17:00:21.776 INFO:teuthology.orchestra.run.smithi044.stdout: "root" 2023-11-16T17:00:21.776 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.776 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "mode for remote execution of cephadm", 2023-11-16T17:00:21.776 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.777 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.777 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.777 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.777 INFO:teuthology.orchestra.run.smithi044.stdout: "prometheus_alerts_path": { 2023-11-16T17:00:21.777 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus_alerts_path", 2023-11-16T17:00:21.777 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.777 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.777 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.777 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-11-16T17:00:21.778 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.778 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.778 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.778 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-11-16T17:00:21.778 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.778 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.778 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.778 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.779 INFO:teuthology.orchestra.run.smithi044.stdout: "prometheus_web_password": { 2023-11-16T17:00:21.779 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus_web_password", 2023-11-16T17:00:21.779 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.779 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.779 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.779 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:21.779 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.779 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.779 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.780 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus web password", 2023-11-16T17:00:21.780 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.780 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.780 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.780 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.780 INFO:teuthology.orchestra.run.smithi044.stdout: "prometheus_web_user": { 2023-11-16T17:00:21.780 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus_web_user", 2023-11-16T17:00:21.780 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.780 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.781 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.781 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:21.781 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.781 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.781 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.781 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus web user", 2023-11-16T17:00:21.781 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.781 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.781 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.782 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.782 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_insecure": { 2023-11-16T17:00:21.782 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_insecure", 2023-11-16T17:00:21.782 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.782 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.782 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.782 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.782 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.783 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.783 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.783 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-11-16T17:00:21.783 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.783 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.783 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.783 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.783 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_password": { 2023-11-16T17:00:21.783 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_password", 2023-11-16T17:00:21.784 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.784 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.784 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.784 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.784 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.784 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.784 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.784 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-11-16T17:00:21.785 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.785 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.785 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.785 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.785 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_url": { 2023-11-16T17:00:21.785 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_url", 2023-11-16T17:00:21.785 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.785 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.785 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.786 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.786 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.786 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.786 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.786 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-11-16T17:00:21.786 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.786 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.786 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.787 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.787 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_username": { 2023-11-16T17:00:21.787 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_username", 2023-11-16T17:00:21.787 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.787 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.787 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.787 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.787 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.788 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.788 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.788 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-11-16T17:00:21.788 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.788 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.788 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.788 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.788 INFO:teuthology.orchestra.run.smithi044.stdout: "secure_monitoring_stack": { 2023-11-16T17:00:21.789 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "secure_monitoring_stack", 2023-11-16T17:00:21.789 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.789 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.789 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.789 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.789 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.789 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.789 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.789 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-11-16T17:00:21.790 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.790 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.790 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.790 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.790 INFO:teuthology.orchestra.run.smithi044.stdout: "service_discovery_port": { 2023-11-16T17:00:21.790 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "service_discovery_port", 2023-11-16T17:00:21.790 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.791 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.791 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.791 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8765", 2023-11-16T17:00:21.791 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.791 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.791 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.791 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "cephadm service discovery port", 2023-11-16T17:00:21.791 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.791 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.792 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.792 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.792 INFO:teuthology.orchestra.run.smithi044.stdout: "ssh_config_file": { 2023-11-16T17:00:21.792 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssh_config_file", 2023-11-16T17:00:21.792 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.792 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.792 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.792 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.793 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.793 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.793 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.793 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-11-16T17:00:21.793 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.793 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.793 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.793 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.793 INFO:teuthology.orchestra.run.smithi044.stdout: "use_agent": { 2023-11-16T17:00:21.794 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_agent", 2023-11-16T17:00:21.794 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.794 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.794 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.794 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.794 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.794 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.794 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.794 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-11-16T17:00:21.795 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.795 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.795 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.795 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.795 INFO:teuthology.orchestra.run.smithi044.stdout: "use_repo_digest": { 2023-11-16T17:00:21.795 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_repo_digest", 2023-11-16T17:00:21.795 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.795 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.796 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.796 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.796 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.796 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.796 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.796 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-11-16T17:00:21.796 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.796 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.796 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.797 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.797 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_failed_host_check": { 2023-11-16T17:00:21.797 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_failed_host_check", 2023-11-16T17:00:21.797 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.797 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.797 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.797 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.797 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.798 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.798 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.798 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if the host check fails", 2023-11-16T17:00:21.798 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.798 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.798 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.798 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.798 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_daemons": { 2023-11-16T17:00:21.799 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_daemons", 2023-11-16T17:00:21.799 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.799 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.799 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.799 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.799 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.799 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.799 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.799 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-11-16T17:00:21.800 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.800 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.800 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.800 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.800 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_hosts": { 2023-11-16T17:00:21.800 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_hosts", 2023-11-16T17:00:21.800 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.800 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.801 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.801 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.801 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.801 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.801 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.801 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-11-16T17:00:21.801 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.801 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.801 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.802 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.802 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.802 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.802 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.802 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crash", 2023-11-16T17:00:21.802 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.802 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.802 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.803 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.803 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.803 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.803 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.803 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.803 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.803 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.803 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.804 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.804 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.804 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.804 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.804 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.804 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.804 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.804 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.804 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.805 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.805 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.805 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.805 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.805 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.805 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.805 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.805 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.806 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.806 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.806 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.806 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.806 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.806 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.806 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.807 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.807 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.807 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.807 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.807 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.807 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.807 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.807 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.807 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.808 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.808 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.808 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.808 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.808 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.808 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.808 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.808 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.809 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.809 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.809 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.809 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.809 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.809 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.809 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.809 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.809 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.810 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.810 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.810 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.810 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.810 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.810 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.810 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.810 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.811 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.811 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.811 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.811 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.811 INFO:teuthology.orchestra.run.smithi044.stdout: "retain_interval": { 2023-11-16T17:00:21.811 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retain_interval", 2023-11-16T17:00:21.811 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.811 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.812 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.812 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "31536000", 2023-11-16T17:00:21.812 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.812 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.812 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.812 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain crashes before pruning them", 2023-11-16T17:00:21.812 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.812 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.813 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.813 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.813 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_recent_interval": { 2023-11-16T17:00:21.813 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_recent_interval", 2023-11-16T17:00:21.813 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.813 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.813 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.813 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1209600", 2023-11-16T17:00:21.813 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.814 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.814 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.814 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "time interval in which to warn about recent crashes", 2023-11-16T17:00:21.814 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.814 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.814 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.814 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.815 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.815 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.815 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.815 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dashboard", 2023-11-16T17:00:21.815 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.815 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.815 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.815 INFO:teuthology.orchestra.run.smithi044.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-11-16T17:00:21.815 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-11-16T17:00:21.816 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.816 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.816 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.816 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:21.816 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.816 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.816 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.816 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.816 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.817 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.817 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.817 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.817 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_HOST": { 2023-11-16T17:00:21.817 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_HOST", 2023-11-16T17:00:21.817 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.817 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.817 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.817 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.818 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.818 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.818 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.818 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.818 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.818 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.818 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.818 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.819 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-11-16T17:00:21.819 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-11-16T17:00:21.819 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.819 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.819 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.819 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.819 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.819 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.819 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.819 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_ENABLED": { 2023-11-16T17:00:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_ENABLED", 2023-11-16T17:00:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.820 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.821 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.822 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.822 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-11-16T17:00:21.822 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-11-16T17:00:21.822 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.822 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.822 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.822 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.822 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.823 INFO:teuthology.orchestra.run.smithi044.stdout: "ENABLE_BROWSABLE_API": { 2023-11-16T17:00:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ENABLE_BROWSABLE_API", 2023-11-16T17:00:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.824 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-11-16T17:00:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-11-16T17:00:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.825 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.826 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-11-16T17:00:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-11-16T17:00:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.827 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.828 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.828 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.828 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.828 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.828 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.828 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.828 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-11-16T17:00:21.828 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-11-16T17:00:21.828 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.829 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.829 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.829 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.829 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.829 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.829 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.829 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.829 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-11-16T17:00:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-11-16T17:00:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.830 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.831 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.832 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.832 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_NFS": { 2023-11-16T17:00:21.832 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-11-16T17:00:21.832 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.832 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.832 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.832 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.832 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.832 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.833 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.833 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.833 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.833 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.833 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.833 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.833 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RBD": { 2023-11-16T17:00:21.833 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-11-16T17:00:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.834 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RGW": { 2023-11-16T17:00:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-11-16T17:00:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.835 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.836 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-11-16T17:00:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-11-16T17:00:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.837 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.838 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.838 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.838 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.838 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.838 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.838 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.838 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.838 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_PASSWORD": { 2023-11-16T17:00:21.839 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_PASSWORD", 2023-11-16T17:00:21.839 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.839 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.839 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.839 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:21.839 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.839 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.839 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.840 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.840 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.840 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.840 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.840 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.840 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-11-16T17:00:21.840 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-11-16T17:00:21.840 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.841 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.841 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.841 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.841 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.841 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.841 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.841 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.841 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.842 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.842 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.842 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.842 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_URL": { 2023-11-16T17:00:21.842 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_URL", 2023-11-16T17:00:21.842 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.842 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.842 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.842 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.843 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.843 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.843 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.843 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.843 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.843 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.843 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.843 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.844 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_USERNAME": { 2023-11-16T17:00:21.844 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_USERNAME", 2023-11-16T17:00:21.844 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.844 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.844 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.844 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:21.844 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.844 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.845 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.845 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.845 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.845 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.845 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.845 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.845 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-11-16T17:00:21.845 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-11-16T17:00:21.846 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.846 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.846 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.846 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.846 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.846 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.846 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.846 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.847 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.847 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.847 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.847 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.847 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-11-16T17:00:21.847 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-11-16T17:00:21.847 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.847 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.848 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.848 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.848 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.848 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.848 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.848 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.848 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.848 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.848 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.849 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.849 INFO:teuthology.orchestra.run.smithi044.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-11-16T17:00:21.849 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-11-16T17:00:21.849 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.849 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.849 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.849 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.849 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.850 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.850 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.850 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.850 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.850 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.850 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.850 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.850 INFO:teuthology.orchestra.run.smithi044.stdout: "ISSUE_TRACKER_API_KEY": { 2023-11-16T17:00:21.851 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-11-16T17:00:21.851 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.851 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.851 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.851 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.851 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.851 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.851 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.852 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.852 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.852 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.852 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.852 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.852 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_HOST": { 2023-11-16T17:00:21.852 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_HOST", 2023-11-16T17:00:21.852 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.852 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.853 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.853 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.853 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.853 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.853 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.853 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.853 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.853 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.854 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.854 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.854 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-11-16T17:00:21.854 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-11-16T17:00:21.854 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.854 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.854 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.854 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.855 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.855 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.855 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.855 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.855 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.855 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.855 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.855 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.856 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-11-16T17:00:21.856 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-11-16T17:00:21.856 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.856 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.856 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.856 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.856 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.856 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.856 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.857 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.857 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.857 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.857 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.857 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.857 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-11-16T17:00:21.857 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-11-16T17:00:21.857 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.858 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.858 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.858 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.858 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.858 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.858 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.858 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.858 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.859 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.859 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.859 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.859 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-11-16T17:00:21.859 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-11-16T17:00:21.859 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.859 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.859 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.859 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.860 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.860 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.860 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.860 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.860 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.860 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.860 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.860 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.861 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-11-16T17:00:21.861 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-11-16T17:00:21.861 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.861 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.861 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.861 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.861 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.861 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.862 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.862 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.862 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.862 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.862 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.862 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.862 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-11-16T17:00:21.862 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-11-16T17:00:21.863 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.863 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.863 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.863 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.863 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.863 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.863 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.863 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.864 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.864 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.864 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.864 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.864 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-11-16T17:00:21.864 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-11-16T17:00:21.864 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.864 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.865 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.865 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.865 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.865 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.865 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.865 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.865 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.865 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.866 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.866 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.866 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-11-16T17:00:21.866 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-11-16T17:00:21.866 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.866 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.866 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.866 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.866 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.867 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.867 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.867 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.867 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.867 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.867 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.867 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.867 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_ENABLED": { 2023-11-16T17:00:21.867 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_ENABLED", 2023-11-16T17:00:21.868 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.868 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.868 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.868 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.868 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.868 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.868 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.868 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.868 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.869 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.869 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.869 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.869 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-11-16T17:00:21.869 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-11-16T17:00:21.869 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.869 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.869 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.869 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-11-16T17:00:21.870 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.870 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.870 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.870 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.870 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.870 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.870 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.870 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.871 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-11-16T17:00:21.871 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-11-16T17:00:21.871 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.871 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.871 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.871 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:21.871 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.871 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.871 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.871 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.872 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.872 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.872 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.872 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.872 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-11-16T17:00:21.872 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-11-16T17:00:21.872 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.872 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.872 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.873 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8", 2023-11-16T17:00:21.873 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.873 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.873 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.873 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.873 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.873 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.873 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.874 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.874 INFO:teuthology.orchestra.run.smithi044.stdout: "REST_REQUESTS_TIMEOUT": { 2023-11-16T17:00:21.874 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-11-16T17:00:21.874 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.874 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.874 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.874 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "45", 2023-11-16T17:00:21.874 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.874 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.875 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.875 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.875 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.875 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.875 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.875 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.875 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ACCESS_KEY": { 2023-11-16T17:00:21.875 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ACCESS_KEY", 2023-11-16T17:00:21.875 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.876 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.876 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.876 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.876 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.876 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.876 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.876 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.876 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.876 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.877 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.877 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.877 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-11-16T17:00:21.877 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-11-16T17:00:21.877 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.877 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.877 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.877 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:21.878 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.878 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.878 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.878 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.878 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.878 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.878 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.878 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.878 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SECRET_KEY": { 2023-11-16T17:00:21.879 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SECRET_KEY", 2023-11-16T17:00:21.879 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.879 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.879 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.879 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.879 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.879 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.879 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.879 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.880 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.880 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.880 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.880 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.880 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SSL_VERIFY": { 2023-11-16T17:00:21.880 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SSL_VERIFY", 2023-11-16T17:00:21.880 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.880 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.880 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.881 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.881 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.881 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.881 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.881 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.881 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.881 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.881 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.881 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.881 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-11-16T17:00:21.882 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-11-16T17:00:21.882 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.882 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.882 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.882 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2023-11-16T17:00:21.882 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.882 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.882 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.882 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.883 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.883 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.883 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.883 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.883 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-11-16T17:00:21.883 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-11-16T17:00:21.883 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.883 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.883 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.883 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:21.884 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.884 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.884 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.884 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.884 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.884 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.884 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.884 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.884 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-11-16T17:00:21.885 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-11-16T17:00:21.885 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.885 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.885 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.885 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2023-11-16T17:00:21.885 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.885 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.885 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.885 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.885 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.886 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.886 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.886 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.886 INFO:teuthology.orchestra.run.smithi044.stdout: "cross_origin_url": { 2023-11-16T17:00:21.886 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cross_origin_url", 2023-11-16T17:00:21.886 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.886 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.886 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.886 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.887 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.887 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.887 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.887 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.887 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.887 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.887 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.887 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.887 INFO:teuthology.orchestra.run.smithi044.stdout: "crt_file": { 2023-11-16T17:00:21.887 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crt_file", 2023-11-16T17:00:21.888 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.888 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.888 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.888 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.888 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.888 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.888 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.888 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.888 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.889 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.889 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.889 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.889 INFO:teuthology.orchestra.run.smithi044.stdout: "debug": { 2023-11-16T17:00:21.889 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "debug", 2023-11-16T17:00:21.889 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.889 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.889 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.889 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.889 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.890 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.890 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.890 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable/disable debug options", 2023-11-16T17:00:21.890 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.890 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.890 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.890 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.890 INFO:teuthology.orchestra.run.smithi044.stdout: "jwt_token_ttl": { 2023-11-16T17:00:21.890 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "jwt_token_ttl", 2023-11-16T17:00:21.891 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.891 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.891 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.891 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "28800", 2023-11-16T17:00:21.891 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.891 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.891 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.891 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.891 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.891 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.892 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.892 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.892 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2023-11-16T17:00:21.892 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2023-11-16T17:00:21.892 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.892 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.892 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.892 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.892 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.893 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.893 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.893 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.893 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.893 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.893 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.893 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.893 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.893 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.893 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.894 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.894 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.894 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.894 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.894 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.894 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.894 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.894 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.895 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.895 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.895 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.895 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.895 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.895 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.895 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.895 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.895 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.895 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.896 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.896 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.896 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.896 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.896 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.896 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.896 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.896 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.896 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.896 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.897 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.897 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.897 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.897 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.897 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.897 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.897 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.897 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.897 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.898 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.898 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.898 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.898 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.898 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.898 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.898 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.898 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.898 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.898 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.899 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.899 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.899 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.899 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.899 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.899 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.899 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.899 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.899 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.899 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.900 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.900 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.900 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.900 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.900 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.900 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.900 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.900 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.900 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.901 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.901 INFO:teuthology.orchestra.run.smithi044.stdout: "motd": { 2023-11-16T17:00:21.901 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "motd", 2023-11-16T17:00:21.901 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.901 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.901 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.901 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.901 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.901 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.902 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.902 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "The message of the day", 2023-11-16T17:00:21.902 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.902 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.902 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.902 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.902 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect_resolve_ip_addr": { 2023-11-16T17:00:21.902 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "redirect_resolve_ip_addr", 2023-11-16T17:00:21.902 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.903 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.903 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.903 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.903 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.903 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.903 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.903 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.903 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.903 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.903 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.904 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.904 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2023-11-16T17:00:21.904 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2023-11-16T17:00:21.904 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.904 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.904 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.904 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2023-11-16T17:00:21.904 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.904 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.905 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.905 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.905 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.905 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.905 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.905 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.905 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2023-11-16T17:00:21.905 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2023-11-16T17:00:21.905 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.906 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.906 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.906 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8080", 2023-11-16T17:00:21.906 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.906 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.906 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.906 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.906 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.906 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.907 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.907 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.907 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2023-11-16T17:00:21.907 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2023-11-16T17:00:21.907 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.907 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.907 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.907 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.907 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.908 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.908 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.908 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.908 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.908 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.908 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.908 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.908 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl_server_port": { 2023-11-16T17:00:21.908 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl_server_port", 2023-11-16T17:00:21.909 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.909 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.909 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.909 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8443", 2023-11-16T17:00:21.909 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.909 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.909 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.909 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.909 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.910 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.910 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.910 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.910 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2023-11-16T17:00:21.910 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2023-11-16T17:00:21.910 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.910 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.910 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.910 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "redirect", 2023-11-16T17:00:21.911 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.911 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.911 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.911 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.911 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect" 2023-11-16T17:00:21.911 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.911 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.911 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.912 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.912 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.912 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.912 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2023-11-16T17:00:21.912 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2023-11-16T17:00:21.912 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:21.912 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.912 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.912 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2023-11-16T17:00:21.912 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2023-11-16T17:00:21.913 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2023-11-16T17:00:21.913 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.913 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.913 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.913 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.913 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.913 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.913 INFO:teuthology.orchestra.run.smithi044.stdout: "url_prefix": { 2023-11-16T17:00:21.913 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url_prefix", 2023-11-16T17:00:21.914 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.914 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.914 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:21.914 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.914 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.914 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.914 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.914 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.914 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.914 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.915 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.915 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.915 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:21.915 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.915 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:21.915 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "devicehealth", 2023-11-16T17:00:21.915 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:21.915 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:21.915 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:21.916 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_monitoring": { 2023-11-16T17:00:21.916 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_monitoring", 2023-11-16T17:00:21.916 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.916 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.916 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.916 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.916 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.916 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.916 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.916 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "monitor device health metrics", 2023-11-16T17:00:21.917 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.917 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.917 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.917 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.917 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:21.917 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:21.917 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.917 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.917 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.918 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:21.918 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.918 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.918 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.918 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.918 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.918 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.918 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.919 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.919 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.919 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.919 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.919 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.919 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.919 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.919 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.919 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:21.920 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:21.920 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.920 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.920 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.920 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.920 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.920 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.920 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.920 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.921 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.921 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.921 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.921 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.921 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:21.921 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:21.921 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.921 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.921 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.922 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:21.922 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.922 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.922 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:21.922 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:21.922 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:21.922 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:21.922 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:21.923 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:21.923 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:21.923 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:21.923 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.923 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.923 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.923 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.923 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.923 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:21.924 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:21.924 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.924 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.924 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.924 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:21.924 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.924 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.924 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.924 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:21.925 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.925 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.925 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.925 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.925 INFO:teuthology.orchestra.run.smithi044.stdout: "mark_out_threshold": { 2023-11-16T17:00:21.925 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mark_out_threshold", 2023-11-16T17:00:21.925 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.925 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.925 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.926 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2419200", 2023-11-16T17:00:21.926 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.926 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.926 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.926 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-11-16T17:00:21.926 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.926 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.926 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.927 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.927 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_name": { 2023-11-16T17:00:21.927 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_name", 2023-11-16T17:00:21.927 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:21.927 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.927 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.927 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "device_health_metrics", 2023-11-16T17:00:21.927 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.928 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.928 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.928 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name of pool in which to store device health metrics", 2023-11-16T17:00:21.928 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.928 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.928 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.928 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.928 INFO:teuthology.orchestra.run.smithi044.stdout: "retention_period": { 2023-11-16T17:00:21.928 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retention_period", 2023-11-16T17:00:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15552000", 2023-11-16T17:00:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain device health metrics", 2023-11-16T17:00:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.930 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.930 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.930 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.930 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_frequency": { 2023-11-16T17:00:21.930 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_frequency", 2023-11-16T17:00:21.930 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.930 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.930 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.930 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2023-11-16T17:00:21.931 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.931 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.931 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.931 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to scrape device health metrics", 2023-11-16T17:00:21.931 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.931 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.931 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.931 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.932 INFO:teuthology.orchestra.run.smithi044.stdout: "self_heal": { 2023-11-16T17:00:21.932 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "self_heal", 2023-11-16T17:00:21.932 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:21.932 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.932 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.932 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:21.932 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.932 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.932 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.933 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-11-16T17:00:21.933 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:21.933 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:21.933 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:21.933 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:21.933 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2023-11-16T17:00:21.933 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2023-11-16T17:00:21.933 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:21.933 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:21.934 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:21.934 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2023-11-16T17:00:21.934 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:21.934 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:21.934 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:21.934 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and check device health", 2023-11-16T17:00:27.524 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.525 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.525 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.525 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.525 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_threshold": { 2023-11-16T17:00:27.525 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_threshold", 2023-11-16T17:00:27.525 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:27.525 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.526 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.526 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7257600", 2023-11-16T17:00:27.526 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.526 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.526 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.526 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-11-16T17:00:27.527 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.527 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.527 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.527 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.527 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.527 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.527 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.528 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "diskprediction_local", 2023-11-16T17:00:27.528 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.528 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.528 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.528 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.528 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.528 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.529 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.529 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.529 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.529 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.529 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.529 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.529 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.529 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.530 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.530 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.530 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.530 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.530 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.530 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.530 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.531 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.531 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.531 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.531 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.531 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.531 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.531 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.531 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.532 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.532 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.532 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.532 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.532 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.532 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.532 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.533 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.533 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.533 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.533 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.533 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.533 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.533 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.533 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.534 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.534 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.534 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.534 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.534 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.534 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.534 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.535 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.535 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.535 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.535 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.535 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.535 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.535 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.536 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.536 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.536 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.536 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.536 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.536 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.536 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.537 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.537 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.537 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.537 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.537 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.537 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.537 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.538 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.538 INFO:teuthology.orchestra.run.smithi044.stdout: "predict_interval": { 2023-11-16T17:00:27.538 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predict_interval", 2023-11-16T17:00:27.538 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.538 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.538 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.538 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2023-11-16T17:00:27.539 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.539 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.539 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.539 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.539 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.539 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.539 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.539 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.540 INFO:teuthology.orchestra.run.smithi044.stdout: "predictor_model": { 2023-11-16T17:00:27.540 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predictor_model", 2023-11-16T17:00:27.540 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.540 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.540 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.540 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "prophetstor", 2023-11-16T17:00:27.540 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.540 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.541 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.541 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.541 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.541 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.541 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.541 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.541 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2023-11-16T17:00:27.541 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2023-11-16T17:00:27.542 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.542 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.542 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.542 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2023-11-16T17:00:27.542 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.542 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.542 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.543 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.543 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.543 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.543 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.543 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.543 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.543 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.543 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.544 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "influx", 2023-11-16T17:00:27.544 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": false, 2023-11-16T17:00:27.544 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "influxdb python module not found", 2023-11-16T17:00:27.544 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.544 INFO:teuthology.orchestra.run.smithi044.stdout: "batch_size": { 2023-11-16T17:00:27.544 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "batch_size", 2023-11-16T17:00:27.544 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.544 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.545 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.545 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5000", 2023-11-16T17:00:27.545 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.545 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.545 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.545 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-11-16T17:00:27.545 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.545 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.546 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.546 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.546 INFO:teuthology.orchestra.run.smithi044.stdout: "database": { 2023-11-16T17:00:27.546 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "database", 2023-11-16T17:00:27.546 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.546 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.546 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.547 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "ceph", 2023-11-16T17:00:27.547 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.547 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.547 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.547 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-11-16T17:00:27.547 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.547 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.547 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.548 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.548 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": { 2023-11-16T17:00:27.548 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "hostname", 2023-11-16T17:00:27.548 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.548 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.548 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.548 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.548 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.549 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.549 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.549 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server hostname", 2023-11-16T17:00:27.549 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.549 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.549 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.549 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.549 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2023-11-16T17:00:27.550 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2023-11-16T17:00:27.550 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:27.550 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.550 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.550 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "30", 2023-11-16T17:00:27.550 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "5", 2023-11-16T17:00:27.550 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.551 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.551 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-11-16T17:00:27.551 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.551 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.551 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.551 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.551 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.551 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.552 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.552 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.552 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.552 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.552 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.552 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.552 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.552 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.553 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.553 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.553 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.553 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.553 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.553 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.553 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.553 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.554 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.554 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.554 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.554 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.554 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.554 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.554 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.554 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.555 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.555 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.555 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.555 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.555 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.555 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.555 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.555 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.556 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.556 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.556 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.556 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.556 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.556 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.556 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.556 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.558 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.558 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.558 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.558 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.558 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.558 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.558 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.559 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.559 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.559 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.559 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.559 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.559 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.559 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.559 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.560 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.560 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.560 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.560 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.560 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.560 INFO:teuthology.orchestra.run.smithi044.stdout: "password": { 2023-11-16T17:00:27.560 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "password", 2023-11-16T17:00:27.560 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "password of InfluxDB server user", 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.562 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.562 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.562 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.562 INFO:teuthology.orchestra.run.smithi044.stdout: "port": { 2023-11-16T17:00:27.562 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "port", 2023-11-16T17:00:27.562 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.562 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.562 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8086", 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server port", 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.564 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.564 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2023-11-16T17:00:27.564 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2023-11-16T17:00:27.564 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.564 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.564 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.564 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "false", 2023-11-16T17:00:27.564 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.565 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.565 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.565 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T17:00:27.565 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.565 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.565 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.565 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.565 INFO:teuthology.orchestra.run.smithi044.stdout: "threads": { 2023-11-16T17:00:27.566 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threads", 2023-11-16T17:00:27.566 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.566 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.566 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.566 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2023-11-16T17:00:27.566 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2023-11-16T17:00:27.566 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "32", 2023-11-16T17:00:27.566 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.567 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-11-16T17:00:27.567 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.567 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.567 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.567 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.567 INFO:teuthology.orchestra.run.smithi044.stdout: "username": { 2023-11-16T17:00:27.567 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "username", 2023-11-16T17:00:27.567 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.568 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.568 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.568 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.568 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.568 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.568 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.568 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "username of InfluxDB server user", 2023-11-16T17:00:27.568 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.569 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.569 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.569 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.569 INFO:teuthology.orchestra.run.smithi044.stdout: "verify_ssl": { 2023-11-16T17:00:27.569 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "verify_ssl", 2023-11-16T17:00:27.569 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.569 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.569 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.570 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "true", 2023-11-16T17:00:27.570 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.570 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.570 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.570 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T17:00:27.570 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.570 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.570 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.571 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.571 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.571 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.571 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.571 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "insights", 2023-11-16T17:00:27.571 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.571 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.571 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.572 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.572 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.572 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.572 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.572 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.572 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.572 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.572 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.573 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.573 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.573 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.573 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.573 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.573 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.573 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.573 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.574 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.574 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.574 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.574 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.574 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.574 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.574 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.575 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.575 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.575 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.575 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.575 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.575 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.575 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.575 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.576 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.576 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.576 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.576 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.576 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.576 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.576 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.576 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.577 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.577 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.577 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.577 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.577 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.577 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.577 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.577 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.578 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.578 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.578 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.578 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.578 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.578 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.578 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.579 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.579 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.579 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.579 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.579 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.579 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.579 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.579 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.580 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.580 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.580 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.580 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.580 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.580 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.580 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.580 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.581 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.581 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.581 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.581 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "iostat", 2023-11-16T17:00:27.581 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.581 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.581 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.581 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.582 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.582 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.582 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.582 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.582 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.582 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.582 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.582 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.583 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.583 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.583 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.583 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.583 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.583 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.583 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.583 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.584 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.584 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.584 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.584 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.584 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.584 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.584 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.584 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.586 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.586 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.586 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.586 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.586 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.586 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.586 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.587 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.587 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.587 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.587 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.587 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.587 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.587 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.587 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.588 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.588 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.588 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.588 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.588 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.588 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.588 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.588 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.589 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.589 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.589 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.589 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.589 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.589 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.589 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.589 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.590 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.590 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.590 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.590 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.590 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.590 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.590 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.590 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.591 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.591 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "k8sevents", 2023-11-16T17:00:27.591 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.591 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.591 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.591 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_event_retention_days": { 2023-11-16T17:00:27.591 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ceph_event_retention_days", 2023-11-16T17:00:27.591 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.592 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.592 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.592 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7", 2023-11-16T17:00:27.592 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.592 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.592 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.592 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Days to hold ceph event information within local cache", 2023-11-16T17:00:27.592 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.593 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.593 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.593 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.593 INFO:teuthology.orchestra.run.smithi044.stdout: "config_check_secs": { 2023-11-16T17:00:27.593 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_check_secs", 2023-11-16T17:00:27.593 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.593 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.593 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "10", 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.596 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.596 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.596 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.596 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.596 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.596 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.596 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.596 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.597 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.597 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.597 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.597 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.597 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.597 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.597 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.597 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.598 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.598 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.598 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.598 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.598 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.598 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.598 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.598 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.599 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.599 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.599 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.599 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.599 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.599 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.599 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.599 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.600 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.600 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.600 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.600 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.600 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.600 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.600 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.600 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.602 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.602 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.602 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.602 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.602 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.602 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.602 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.602 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.603 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.603 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.603 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.603 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.603 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.603 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.603 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.603 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "localpool", 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi044.stdout: "failure_domain": { 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "failure_domain", 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.605 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.605 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "host", 2023-11-16T17:00:27.605 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.605 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.605 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.605 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "failure domain for any created local pool", 2023-11-16T17:00:27.605 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-11-16T17:00:27.605 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.606 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.606 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.606 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.606 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.606 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.606 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.606 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.606 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.607 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.607 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.607 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.607 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.607 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.607 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.607 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.607 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.609 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.609 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.609 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.609 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.609 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.609 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.609 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.609 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.610 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.610 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.610 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.610 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.610 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.610 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.610 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.611 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.611 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.611 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.611 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.611 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.611 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.611 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.611 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.611 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.612 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.612 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.612 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.612 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.612 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.612 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.612 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.613 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.613 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.613 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.613 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.613 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.613 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.613 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.613 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.614 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.614 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.614 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.614 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.614 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.614 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.614 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.615 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.615 INFO:teuthology.orchestra.run.smithi044.stdout: "min_size": { 2023-11-16T17:00:27.615 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_size", 2023-11-16T17:00:27.615 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.615 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.615 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.615 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.615 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.616 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.616 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.616 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default min_size for any created local pool", 2023-11-16T17:00:27.616 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-11-16T17:00:27.616 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.616 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.616 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.616 INFO:teuthology.orchestra.run.smithi044.stdout: "num_rep": { 2023-11-16T17:00:27.617 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "num_rep", 2023-11-16T17:00:27.617 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.617 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.617 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.617 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3", 2023-11-16T17:00:27.617 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.617 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.617 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.618 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default replica count for any created local pool", 2023-11-16T17:00:27.618 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.618 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.618 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.618 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.618 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_num": { 2023-11-16T17:00:27.618 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_num", 2023-11-16T17:00:27.618 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.619 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.619 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.619 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "128", 2023-11-16T17:00:27.619 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.619 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.619 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.619 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default pg_num for any created local pool", 2023-11-16T17:00:27.619 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.620 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.620 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.620 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.620 INFO:teuthology.orchestra.run.smithi044.stdout: "prefix": { 2023-11-16T17:00:27.620 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prefix", 2023-11-16T17:00:27.620 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.620 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.620 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.621 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.621 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.621 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.621 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.621 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name prefix for any created local pool", 2023-11-16T17:00:27.621 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.621 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.621 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.622 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.622 INFO:teuthology.orchestra.run.smithi044.stdout: "subtree": { 2023-11-16T17:00:27.622 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "subtree", 2023-11-16T17:00:27.622 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.622 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.622 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.622 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "rack", 2023-11-16T17:00:27.622 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.623 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.623 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.623 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "CRUSH level for which to create a local pool", 2023-11-16T17:00:27.623 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-11-16T17:00:27.623 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.623 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.623 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.623 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.624 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.624 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.624 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mds_autoscaler", 2023-11-16T17:00:27.624 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.624 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.624 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.624 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.624 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.625 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.625 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.625 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.625 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.625 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.625 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.625 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.625 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.626 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.626 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.626 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.626 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.626 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.626 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.626 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.626 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.627 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.627 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.627 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.627 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.627 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.627 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.627 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.627 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.628 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.628 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.628 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.628 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.628 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.628 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.628 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.628 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.629 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.629 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.629 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.629 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.629 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.629 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.629 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.629 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.629 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.630 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.630 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.630 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.630 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.630 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.630 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.630 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.631 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.631 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.631 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.631 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.631 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.631 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.631 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.631 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.633 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.633 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.633 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.633 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.633 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.634 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.634 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirroring", 2023-11-16T17:00:27.634 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.634 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.634 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.634 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.634 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.634 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.635 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.635 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.635 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.635 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.635 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.635 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.635 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.635 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.636 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.636 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.636 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.636 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.636 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.636 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.636 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.636 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.637 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.637 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.637 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.637 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.637 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.637 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.637 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.637 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.638 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.638 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.638 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.638 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.638 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.638 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.638 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.638 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.639 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.639 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.639 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.639 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.639 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.639 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.639 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.639 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.640 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.640 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.640 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.640 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.640 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.640 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.640 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.640 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.641 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.641 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.641 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.641 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.641 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.641 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.641 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.641 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.642 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.642 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.642 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.642 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.642 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.642 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.642 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.642 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.643 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.643 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.643 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.643 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.643 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.643 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.643 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "nfs", 2023-11-16T17:00:27.643 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.644 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.644 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.644 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.644 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.644 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.644 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.644 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.644 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.645 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.645 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.645 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.645 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.645 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.645 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.645 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.645 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.646 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.646 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.646 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.646 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.646 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.646 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.646 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.646 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.647 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.647 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.647 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.647 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.647 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.647 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.647 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.647 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.648 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.648 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.648 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.648 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.648 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.648 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.648 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.648 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.649 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.649 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.649 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.649 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.649 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.649 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.649 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.649 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.650 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.650 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.650 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.650 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.650 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.650 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.650 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.650 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.651 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.651 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.651 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.651 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.651 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.651 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.651 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.651 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.652 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.652 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.652 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.652 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.652 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.652 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.652 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.652 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.653 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.653 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.653 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.653 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2023-11-16T17:00:27.653 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.653 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.653 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.653 INFO:teuthology.orchestra.run.smithi044.stdout: "fail_fs": { 2023-11-16T17:00:27.654 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "fail_fs", 2023-11-16T17:00:27.654 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.654 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.654 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.654 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.654 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.654 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.654 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.655 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-11-16T17:00:27.655 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.655 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.655 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.655 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.655 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.655 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.655 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.656 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.656 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.656 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.656 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.656 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.656 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.656 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.656 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.657 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.657 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.657 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.657 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.657 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.657 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.657 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.657 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.658 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.658 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.658 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.658 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.658 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.658 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.658 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.658 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.659 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.659 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.659 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.659 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.659 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.659 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.659 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.659 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.660 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.660 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.660 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.660 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.660 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.660 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.660 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.660 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.661 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.661 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.661 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.661 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.661 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.661 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.661 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.661 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.662 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.662 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.662 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.662 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.662 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.662 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.662 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.662 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.663 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.663 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.663 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.663 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.663 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.663 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.663 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.663 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.664 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.664 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.664 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.664 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator": { 2023-11-16T17:00:27.664 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2023-11-16T17:00:27.664 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.664 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.664 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.665 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.665 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.665 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.665 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.665 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm", 2023-11-16T17:00:27.665 INFO:teuthology.orchestra.run.smithi044.stdout: "rook", 2023-11-16T17:00:27.665 INFO:teuthology.orchestra.run.smithi044.stdout: "test_orchestrator" 2023-11-16T17:00:27.665 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.666 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Orchestrator backend", 2023-11-16T17:00:27.666 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.666 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.666 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.666 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.666 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.666 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.666 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.667 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_perf_query", 2023-11-16T17:00:27.667 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.667 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.667 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.667 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.667 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.667 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.667 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.668 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.668 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.668 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.668 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.668 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.668 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.668 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.668 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.669 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.669 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.669 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.669 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.669 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.669 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.669 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.669 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.670 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.670 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.670 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.670 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.670 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.670 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.670 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.670 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.671 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.671 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.671 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.671 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.671 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.671 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.671 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.671 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.672 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.672 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.672 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.672 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.672 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.672 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.672 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.672 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.673 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.673 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.673 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.673 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.673 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.673 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.673 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.673 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.674 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.674 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.674 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.674 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.674 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.674 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.674 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.674 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.675 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.675 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.675 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.675 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.675 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.675 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.675 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.675 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.675 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.676 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.676 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.676 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.676 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.676 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_support", 2023-11-16T17:00:27.676 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.676 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.676 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.677 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.677 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.677 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.677 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.677 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.677 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.677 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.678 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.678 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.678 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.678 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.678 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.678 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.678 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.678 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.679 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.679 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.679 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.679 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.679 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.679 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.679 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.679 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.680 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.680 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.680 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.680 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.680 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.680 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.680 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.680 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.681 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.681 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.681 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.681 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.681 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.681 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.681 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.681 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.681 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.682 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.682 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.682 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.682 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.682 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.682 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.682 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.683 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.683 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.683 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.683 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.683 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.683 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.683 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.683 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.684 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.684 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.684 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.684 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.684 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.684 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.684 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.684 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.685 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.685 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.685 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.685 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.685 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.685 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.685 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.685 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.685 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.686 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.686 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_autoscaler", 2023-11-16T17:00:27.686 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.686 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.686 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.686 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.686 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.687 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.687 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.687 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.687 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.687 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.687 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.687 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.687 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.688 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.688 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.688 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.688 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.688 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.688 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.688 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.688 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.689 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.689 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.689 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.689 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.689 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.689 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.689 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.689 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.690 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.690 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.690 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.690 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.690 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.690 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.690 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.690 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.691 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.691 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.691 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.691 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.691 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.691 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.691 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.691 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.692 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.692 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.692 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.692 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.692 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.692 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.692 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.692 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.693 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.693 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.693 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.693 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.693 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.693 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.693 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.693 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.694 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.694 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.694 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.694 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.694 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.694 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.694 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.694 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.695 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.695 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.695 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.695 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.695 INFO:teuthology.orchestra.run.smithi044.stdout: "noautoscale": { 2023-11-16T17:00:27.695 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "noautoscale", 2023-11-16T17:00:27.695 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.695 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.696 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.696 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.696 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.696 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.696 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.696 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "global autoscale flag", 2023-11-16T17:00:27.696 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-11-16T17:00:27.696 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.697 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.697 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.697 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2023-11-16T17:00:27.697 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2023-11-16T17:00:27.697 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:27.697 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.697 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.697 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2023-11-16T17:00:27.697 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.698 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.698 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.698 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.698 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.698 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.698 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.698 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.698 INFO:teuthology.orchestra.run.smithi044.stdout: "threshold": { 2023-11-16T17:00:27.699 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threshold", 2023-11-16T17:00:27.699 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:27.699 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.699 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.699 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2023-11-16T17:00:27.699 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1.0", 2023-11-16T17:00:27.699 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.699 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.700 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "scaling threshold", 2023-11-16T17:00:27.700 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-11-16T17:00:27.700 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.700 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.700 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.700 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.700 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.700 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.701 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "progress", 2023-11-16T17:00:27.701 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.701 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.701 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.701 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_pg_recovery_event": { 2023-11-16T17:00:27.701 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_pg_recovery_event", 2023-11-16T17:00:27.701 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.701 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.702 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.702 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.702 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.702 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.702 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.702 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow the module to show pg recovery progress", 2023-11-16T17:00:27.702 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.702 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.703 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.703 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.703 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2023-11-16T17:00:27.703 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2023-11-16T17:00:27.703 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.703 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.703 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.703 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:27.704 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.704 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.704 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.704 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.704 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.704 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.704 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.704 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.705 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.705 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.705 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.705 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.705 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.705 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.705 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.705 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.706 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.706 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.706 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.706 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.706 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.706 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.706 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.707 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.707 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.707 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.707 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.707 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.707 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.707 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.707 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.708 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.708 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.708 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.708 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.708 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.708 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.708 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.708 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.709 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.709 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.709 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.709 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.709 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.709 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.709 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.709 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.710 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.710 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.710 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.710 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.710 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.710 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.710 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.710 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.711 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.711 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.711 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.711 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.711 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.711 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.711 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.711 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.712 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.712 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.712 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.712 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.712 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.712 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.712 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.712 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.713 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.713 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.713 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.713 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.713 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.713 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.713 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.713 INFO:teuthology.orchestra.run.smithi044.stdout: "max_completed_events": { 2023-11-16T17:00:27.714 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_completed_events", 2023-11-16T17:00:27.714 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.714 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.714 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.714 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "50", 2023-11-16T17:00:27.714 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.714 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.714 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.715 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "number of past completed events to remember", 2023-11-16T17:00:27.715 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.715 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.715 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.715 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.715 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2023-11-16T17:00:27.715 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2023-11-16T17:00:27.715 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:27.716 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.716 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.716 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2023-11-16T17:00:27.716 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.716 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.716 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.716 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long the module is going to sleep", 2023-11-16T17:00:27.716 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.717 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.717 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.717 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.717 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.717 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.717 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.717 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus", 2023-11-16T17:00:27.717 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.718 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.718 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.718 INFO:teuthology.orchestra.run.smithi044.stdout: "cache": { 2023-11-16T17:00:27.718 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cache", 2023-11-16T17:00:27.718 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.718 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.718 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.718 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:27.719 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.719 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.719 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.719 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.719 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.719 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.719 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.719 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.720 INFO:teuthology.orchestra.run.smithi044.stdout: "exclude_perf_counters": { 2023-11-16T17:00:27.720 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "exclude_perf_counters", 2023-11-16T17:00:27.720 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.720 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.720 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.720 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:27.720 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.720 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.721 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.721 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-11-16T17:00:27.721 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-11-16T17:00:27.721 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.721 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.721 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.721 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.721 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.722 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.722 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.722 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.722 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.722 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.722 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.722 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.722 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.723 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.723 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.723 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.723 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.723 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.723 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.723 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.723 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.724 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.724 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.724 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.724 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.724 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.724 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.724 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.724 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.725 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.725 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.725 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.725 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.725 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.725 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.725 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.725 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.726 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.726 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.726 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.726 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.726 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.726 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.726 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.726 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.727 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.727 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.727 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.727 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.727 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.727 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.727 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.727 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.728 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.728 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.728 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.728 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.728 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.728 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.728 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.728 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.729 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.729 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.729 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.729 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.729 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.729 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.729 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.729 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.730 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.730 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.730 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.730 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.730 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools": { 2023-11-16T17:00:27.730 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools", 2023-11-16T17:00:27.730 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.730 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.731 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.731 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.731 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.731 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.731 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.731 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.731 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.731 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.732 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.732 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.732 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools_refresh_interval": { 2023-11-16T17:00:27.732 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-11-16T17:00:27.732 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.732 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.732 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.732 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "300", 2023-11-16T17:00:27.733 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.733 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.733 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.733 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.733 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.733 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.733 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.733 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.734 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_interval": { 2023-11-16T17:00:27.734 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_interval", 2023-11-16T17:00:27.734 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:27.734 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.734 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.734 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15.0", 2023-11-16T17:00:27.734 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.734 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.735 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.735 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.735 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.735 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.735 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.735 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.735 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2023-11-16T17:00:27.735 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2023-11-16T17:00:27.736 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.736 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.736 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.736 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2023-11-16T17:00:27.736 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.736 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.736 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.736 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-11-16T17:00:27.737 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.737 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.737 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.737 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.737 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2023-11-16T17:00:27.737 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2023-11-16T17:00:27.737 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.737 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.737 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.738 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "9283", 2023-11-16T17:00:27.738 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.738 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.738 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.738 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-11-16T17:00:27.738 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.738 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.738 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.739 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.739 INFO:teuthology.orchestra.run.smithi044.stdout: "stale_cache_strategy": { 2023-11-16T17:00:27.739 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stale_cache_strategy", 2023-11-16T17:00:27.739 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.739 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.739 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.739 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "log", 2023-11-16T17:00:27.739 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.740 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.740 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.740 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.740 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.740 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.740 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.740 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.740 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2023-11-16T17:00:27.741 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2023-11-16T17:00:27.741 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.741 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.741 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.741 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "default", 2023-11-16T17:00:27.741 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.741 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.741 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.742 INFO:teuthology.orchestra.run.smithi044.stdout: "default", 2023-11-16T17:00:27.742 INFO:teuthology.orchestra.run.smithi044.stdout: "error" 2023-11-16T17:00:27.742 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.742 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.742 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.742 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.742 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.742 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.743 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2023-11-16T17:00:27.743 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2023-11-16T17:00:27.743 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.743 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.743 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.743 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2023-11-16T17:00:27.743 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2023-11-16T17:00:27.743 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2023-11-16T17:00:27.744 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.744 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.744 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.744 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.744 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.744 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.744 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.744 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.745 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.745 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_support", 2023-11-16T17:00:27.745 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.745 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.745 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.745 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.745 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.745 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.746 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.746 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.746 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.746 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.746 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.746 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.746 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.746 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.747 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.747 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.747 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.747 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.747 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.747 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.747 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.747 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.748 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.748 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.748 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.748 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.748 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.748 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.748 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.748 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.749 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.749 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.749 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.749 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.749 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.749 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.749 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.749 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.750 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.750 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.750 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.750 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.750 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.750 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.750 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.750 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.751 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.751 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.751 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.751 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.751 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.751 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.751 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.751 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.752 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.752 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.752 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.752 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.752 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.752 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.752 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.752 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.753 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.753 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.753 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.753 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.753 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.753 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.753 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.753 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.754 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.754 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.754 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.754 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_snap_create": { 2023-11-16T17:00:27.754 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_snap_create", 2023-11-16T17:00:27.754 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.754 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.755 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.755 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:27.755 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.755 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.755 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.756 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.756 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.756 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.756 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.756 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.756 INFO:teuthology.orchestra.run.smithi044.stdout: "mirror_snapshot_schedule": { 2023-11-16T17:00:27.756 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirror_snapshot_schedule", 2023-11-16T17:00:27.757 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.757 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.757 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.757 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.757 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.758 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.758 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.758 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.758 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.758 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.759 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.759 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.759 INFO:teuthology.orchestra.run.smithi044.stdout: "trash_purge_schedule": { 2023-11-16T17:00:27.759 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "trash_purge_schedule", 2023-11-16T17:00:27.759 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.760 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.760 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.760 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.760 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.760 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.761 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.761 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.761 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.761 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.761 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.762 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.762 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.762 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.762 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.762 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "restful", 2023-11-16T17:00:27.763 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.763 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.763 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.763 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_auth": { 2023-11-16T17:00:27.763 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_auth", 2023-11-16T17:00:27.764 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.764 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.764 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.764 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:27.764 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.765 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.765 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.765 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.765 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.765 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.765 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.765 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.765 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2023-11-16T17:00:27.766 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2023-11-16T17:00:27.766 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.766 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.766 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.766 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.766 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.766 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.766 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.767 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.767 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.767 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.767 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.767 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.767 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.767 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.767 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.768 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.768 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.768 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.768 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.768 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.768 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.768 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.768 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.769 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.769 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.769 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.769 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.769 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.769 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.769 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.769 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.770 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.770 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.770 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.770 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.770 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.770 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.770 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.770 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.771 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.771 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.771 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.771 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.771 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.771 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.771 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.771 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.772 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.772 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.772 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.772 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.772 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.772 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.772 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.772 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.773 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.773 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.773 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.773 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.773 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.773 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.773 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.773 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.774 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.774 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.774 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.774 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.774 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.774 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.774 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.774 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.775 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.775 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.775 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.775 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.775 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.775 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.775 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.775 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.776 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.776 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.776 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.776 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2023-11-16T17:00:27.776 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2023-11-16T17:00:27.776 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.776 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.776 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.777 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.777 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.777 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.777 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.777 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.777 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.777 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.777 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.777 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.778 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2023-11-16T17:00:27.778 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2023-11-16T17:00:27.778 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.778 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.778 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.778 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.778 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.779 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.779 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.779 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.779 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.779 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.779 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.779 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.779 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.780 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.780 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.780 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rgw", 2023-11-16T17:00:27.780 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.780 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.780 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.780 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.780 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.781 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.781 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.781 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.781 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.781 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.781 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.781 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.781 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.782 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.782 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.782 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.782 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.782 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.782 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.782 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.782 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.783 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.783 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.783 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.783 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.783 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.783 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.783 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.783 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.784 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.784 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.784 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.784 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.784 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.784 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.784 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.784 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.785 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.785 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.785 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.785 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.785 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.785 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.785 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.785 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.786 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.786 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.786 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.786 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.786 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.786 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.786 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.786 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.787 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.787 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.787 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.787 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.787 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.787 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.787 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.787 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.788 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.788 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.788 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.788 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.788 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.788 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.788 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.788 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.789 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.789 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.789 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.789 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.789 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.789 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.789 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.789 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rook", 2023-11-16T17:00:27.790 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.790 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.790 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.790 INFO:teuthology.orchestra.run.smithi044.stdout: "drive_group_interval": { 2023-11-16T17:00:27.790 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "drive_group_interval", 2023-11-16T17:00:27.790 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:27.790 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.791 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.791 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "300.0", 2023-11-16T17:00:27.791 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.791 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.791 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.791 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-11-16T17:00:27.791 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.791 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.791 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.792 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.792 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.792 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.792 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.792 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.792 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.792 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.793 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.793 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.793 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.793 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.793 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.793 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.793 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.793 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.793 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.794 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.794 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.794 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.794 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.794 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.794 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.794 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.794 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.795 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.795 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.795 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.795 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.795 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.795 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.795 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.795 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.796 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.796 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.796 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.796 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.796 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.796 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.796 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.796 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.797 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.797 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.797 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.797 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.797 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.797 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.797 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.797 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.798 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.798 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.798 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.798 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.798 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.798 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.798 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.798 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.799 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.799 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.799 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.799 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.799 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.799 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.799 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.799 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.800 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.800 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.800 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.800 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.800 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.800 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.800 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.800 INFO:teuthology.orchestra.run.smithi044.stdout: "storage_class": { 2023-11-16T17:00:27.801 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "storage_class", 2023-11-16T17:00:27.801 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.801 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.801 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.801 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "local", 2023-11-16T17:00:27.801 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.801 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.801 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.802 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-11-16T17:00:27.802 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.802 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.802 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.802 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.802 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.802 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.802 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.803 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "selftest", 2023-11-16T17:00:27.803 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.803 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.803 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.803 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.803 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.803 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.803 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.804 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.804 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.804 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.804 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.804 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.804 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.804 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.804 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.805 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.805 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.805 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.805 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.805 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.805 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.805 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.805 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.806 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.806 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.806 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.806 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.806 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.806 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.806 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.806 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.807 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.807 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.807 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.807 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.807 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.807 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.807 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.807 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.808 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.808 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.808 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.808 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.808 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.808 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.808 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.808 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.809 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.809 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.809 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.809 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.809 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.809 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.809 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.809 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.810 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.810 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.810 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.810 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.810 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.810 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.810 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.810 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.811 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.811 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.811 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.811 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.811 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.811 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.811 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.811 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.812 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.812 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.812 INFO:teuthology.orchestra.run.smithi044.stdout: "roption1": { 2023-11-16T17:00:27.812 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption1", 2023-11-16T17:00:27.812 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.812 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.812 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.812 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.813 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.813 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.813 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.813 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.813 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.813 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.813 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.813 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.813 INFO:teuthology.orchestra.run.smithi044.stdout: "roption2": { 2023-11-16T17:00:27.814 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption2", 2023-11-16T17:00:27.814 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.814 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.814 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.814 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "xyz", 2023-11-16T17:00:27.814 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.814 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.814 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.815 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.815 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.815 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.815 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.815 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.815 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption1": { 2023-11-16T17:00:27.815 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption1", 2023-11-16T17:00:27.816 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.816 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.816 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.816 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.816 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.816 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.816 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.816 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.817 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.817 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.817 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.817 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.817 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption2": { 2023-11-16T17:00:27.817 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption2", 2023-11-16T17:00:27.817 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.817 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.817 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.818 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.818 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.818 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.818 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.818 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.818 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.818 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.818 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.819 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.819 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption3": { 2023-11-16T17:00:27.819 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption3", 2023-11-16T17:00:27.819 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:27.819 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.819 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.819 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.819 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.820 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.820 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.820 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.820 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.820 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.820 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.820 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.820 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption4": { 2023-11-16T17:00:27.821 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption4", 2023-11-16T17:00:27.821 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.821 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.821 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.821 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.821 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.821 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.821 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.822 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.822 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.822 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.822 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.822 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.822 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption5": { 2023-11-16T17:00:27.822 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption5", 2023-11-16T17:00:27.822 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.823 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.823 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.823 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.823 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.823 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.823 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.823 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.823 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.824 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.824 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.824 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.824 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption6": { 2023-11-16T17:00:27.824 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption6", 2023-11-16T17:00:27.824 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.824 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.824 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.825 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:27.825 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.825 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.825 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.825 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.825 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.825 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.825 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.826 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.826 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption7": { 2023-11-16T17:00:27.826 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption7", 2023-11-16T17:00:27.826 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.826 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.826 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.826 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.826 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2023-11-16T17:00:27.827 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "42", 2023-11-16T17:00:27.827 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.827 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.827 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.827 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.827 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.827 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.827 INFO:teuthology.orchestra.run.smithi044.stdout: "testkey": { 2023-11-16T17:00:27.828 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testkey", 2023-11-16T17:00:27.828 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.828 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.828 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.828 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.828 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.828 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.828 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.829 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.829 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.829 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.829 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.829 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.829 INFO:teuthology.orchestra.run.smithi044.stdout: "testlkey": { 2023-11-16T17:00:27.829 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testlkey", 2023-11-16T17:00:27.829 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.830 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.830 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.830 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.830 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.830 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.830 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.830 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.830 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.831 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.831 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.831 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.831 INFO:teuthology.orchestra.run.smithi044.stdout: "testnewline": { 2023-11-16T17:00:27.831 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testnewline", 2023-11-16T17:00:27.831 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.831 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.831 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.832 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.832 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.832 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.832 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.832 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.832 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.832 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.832 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.833 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.833 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.833 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.833 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.833 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snap_schedule", 2023-11-16T17:00:27.833 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.833 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.833 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.834 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_m_granularity": { 2023-11-16T17:00:27.834 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_m_granularity", 2023-11-16T17:00:27.834 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.834 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.834 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.834 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.834 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.834 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.835 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.835 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow minute scheduled snapshots", 2023-11-16T17:00:27.835 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.835 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.835 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.835 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.835 INFO:teuthology.orchestra.run.smithi044.stdout: "dump_on_update": { 2023-11-16T17:00:27.835 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dump_on_update", 2023-11-16T17:00:27.836 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.836 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.836 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.836 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.836 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.836 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.836 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.836 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "dump database to debug log on update", 2023-11-16T17:00:27.837 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.837 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.837 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.837 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.837 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.837 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.837 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.837 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.838 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.838 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.838 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.838 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.838 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.838 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.838 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.838 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.839 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.839 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.839 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.839 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.839 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.839 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.839 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.839 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.840 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.840 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.840 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.840 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.840 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.840 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.840 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.840 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.841 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.841 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.841 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.841 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.841 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.841 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.841 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.841 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.842 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.842 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.842 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.842 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.842 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.842 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.842 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.842 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.843 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.843 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.843 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.843 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.843 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.843 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.843 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.843 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.844 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.844 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.844 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.844 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.844 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.844 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.844 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.844 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.845 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.845 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.845 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.845 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.845 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.845 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.845 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.845 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.846 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.846 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.846 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.846 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.846 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.846 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stats", 2023-11-16T17:00:27.846 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.846 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.847 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.847 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.847 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.847 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.847 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.847 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.847 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.847 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.848 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.848 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.848 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.848 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.848 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.848 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.848 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.848 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.849 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.849 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.849 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.849 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.849 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.849 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.849 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.849 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.850 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.850 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.850 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.850 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.850 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.850 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.850 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.850 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.851 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.851 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.851 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.851 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.851 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.851 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.851 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.851 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.852 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.852 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.852 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.852 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.852 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.852 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.852 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.852 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.853 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.853 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.853 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.853 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.853 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.853 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.853 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.853 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.854 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.854 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.854 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.854 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.854 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.854 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.854 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.854 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.855 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.855 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.855 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.855 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.855 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.855 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.855 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.855 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.856 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.856 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.856 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "status", 2023-11-16T17:00:27.856 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.856 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.856 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.856 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.856 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.857 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.857 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.857 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.857 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.857 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.857 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.857 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.857 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.858 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.858 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.858 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.858 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.858 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.858 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.858 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.858 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.859 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.859 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.859 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.859 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.859 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.859 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.859 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.859 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.860 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.860 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.860 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.860 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.860 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.860 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.860 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.860 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.861 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.861 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.861 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.861 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.861 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.861 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.861 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.861 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.861 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.862 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.862 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.862 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.862 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.862 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.862 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.862 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.863 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.863 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.863 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.863 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.863 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.863 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.863 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.863 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.863 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.864 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.864 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.864 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.864 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.864 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.864 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.864 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.864 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.865 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.865 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.865 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.865 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.865 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.865 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.865 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telegraf", 2023-11-16T17:00:27.865 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.866 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.866 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.866 INFO:teuthology.orchestra.run.smithi044.stdout: "address": { 2023-11-16T17:00:27.866 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "address", 2023-11-16T17:00:27.866 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.866 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.866 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.867 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-11-16T17:00:27.867 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.867 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.867 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.867 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.867 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.867 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.867 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.868 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.868 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2023-11-16T17:00:27.868 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2023-11-16T17:00:27.868 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:27.868 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.868 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.868 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15", 2023-11-16T17:00:27.868 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.868 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.869 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.869 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.869 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.869 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.869 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.869 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.869 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.869 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.870 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.870 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.870 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.870 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.870 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.870 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.870 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.870 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.871 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.871 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.871 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.871 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.871 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.871 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.871 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.871 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.872 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.872 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.872 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.872 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.872 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.872 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.872 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.872 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.873 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.873 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.873 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.873 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.873 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.873 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.873 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.873 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.874 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.874 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.874 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.874 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.874 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.874 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.874 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.874 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.875 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.875 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.875 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.875 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.875 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.875 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.875 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.875 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.876 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.876 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.876 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.876 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.876 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.876 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.876 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.876 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.877 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.877 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.877 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.877 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.877 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.877 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.877 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.877 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.877 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.878 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.878 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.878 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.878 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.878 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.878 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.878 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telemetry", 2023-11-16T17:00:27.878 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.879 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.879 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.879 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_basic": { 2023-11-16T17:00:27.879 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_basic", 2023-11-16T17:00:27.879 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.879 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.879 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.879 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:27.880 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.880 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.880 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.880 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share basic cluster information (size, version)", 2023-11-16T17:00:27.880 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.880 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.880 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.880 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.881 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_crash": { 2023-11-16T17:00:27.881 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_crash", 2023-11-16T17:00:27.881 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.881 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.881 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.881 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:27.881 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.882 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.882 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.882 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-11-16T17:00:27.882 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.882 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.882 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.882 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.882 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_device": { 2023-11-16T17:00:27.883 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_device", 2023-11-16T17:00:27.883 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.883 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.883 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.883 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:27.883 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.883 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.883 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.884 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-11-16T17:00:27.884 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.884 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.884 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.884 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.884 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_ident": { 2023-11-16T17:00:27.884 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_ident", 2023-11-16T17:00:27.884 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.885 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.885 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.885 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.885 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.885 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.885 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.885 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-11-16T17:00:27.885 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.886 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.886 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.886 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.886 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_perf": { 2023-11-16T17:00:27.886 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_perf", 2023-11-16T17:00:27.886 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.886 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.886 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.887 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.887 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.887 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.887 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.887 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share various performance metrics of a cluster", 2023-11-16T17:00:27.887 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.887 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.887 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.888 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.888 INFO:teuthology.orchestra.run.smithi044.stdout: "contact": { 2023-11-16T17:00:27.888 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "contact", 2023-11-16T17:00:27.888 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.888 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.888 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.888 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.888 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.889 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.889 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.889 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.889 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.889 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.889 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.889 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.889 INFO:teuthology.orchestra.run.smithi044.stdout: "description": { 2023-11-16T17:00:27.890 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "description", 2023-11-16T17:00:27.890 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.890 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.890 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.890 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.890 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.890 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.890 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.891 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.891 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.891 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.891 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.891 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.891 INFO:teuthology.orchestra.run.smithi044.stdout: "device_url": { 2023-11-16T17:00:27.891 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_url", 2023-11-16T17:00:27.891 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.892 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.892 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.892 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-11-16T17:00:27.892 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.892 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.892 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.892 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.892 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.893 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.893 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.893 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.893 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2023-11-16T17:00:27.893 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2023-11-16T17:00:27.893 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.893 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.893 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.894 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.894 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.894 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.894 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.894 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.894 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.894 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.894 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.895 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.895 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2023-11-16T17:00:27.895 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2023-11-16T17:00:27.895 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.895 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.895 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.895 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "24", 2023-11-16T17:00:27.895 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "8", 2023-11-16T17:00:27.896 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.896 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.896 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.896 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.896 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.896 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.896 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.896 INFO:teuthology.orchestra.run.smithi044.stdout: "last_opt_revision": { 2023-11-16T17:00:27.896 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "last_opt_revision", 2023-11-16T17:00:27.897 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.897 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.897 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.897 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1", 2023-11-16T17:00:27.897 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.897 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.897 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.897 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.898 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.898 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.898 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.898 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.898 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard": { 2023-11-16T17:00:27.898 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard", 2023-11-16T17:00:27.898 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.898 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.899 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.899 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.899 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.899 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.899 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.899 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.899 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.899 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.900 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.900 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.900 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard_description": { 2023-11-16T17:00:27.900 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard_description", 2023-11-16T17:00:27.900 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.900 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.900 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.900 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.901 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.901 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.901 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.901 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.901 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.901 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.901 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.901 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.902 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.902 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.902 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.902 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.902 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.902 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.902 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.902 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.903 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.903 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.903 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.903 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.903 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.903 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.903 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.903 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.903 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.904 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.904 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.904 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.904 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.904 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.904 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.904 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.904 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.905 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.905 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.905 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.905 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.905 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.905 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.905 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.905 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.905 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.906 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.906 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.906 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.906 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.906 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.906 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.906 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.906 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.907 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.907 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.907 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.907 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.907 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.907 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.907 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.907 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.908 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.908 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.908 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.908 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.908 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.908 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.908 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.908 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.909 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.909 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.909 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.909 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.909 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.909 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.909 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.909 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.910 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.910 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.910 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.910 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.910 INFO:teuthology.orchestra.run.smithi044.stdout: "organization": { 2023-11-16T17:00:27.910 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "organization", 2023-11-16T17:00:27.910 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.910 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.911 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.911 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.911 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.911 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.911 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.911 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.911 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.911 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.912 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.912 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.912 INFO:teuthology.orchestra.run.smithi044.stdout: "proxy": { 2023-11-16T17:00:27.912 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "proxy", 2023-11-16T17:00:27.912 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.912 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.912 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.912 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.913 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.913 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.913 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.913 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.913 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.913 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.913 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.913 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.914 INFO:teuthology.orchestra.run.smithi044.stdout: "url": { 2023-11-16T17:00:27.914 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url", 2023-11-16T17:00:27.914 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.914 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.914 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.914 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-11-16T17:00:27.914 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.915 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.915 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.915 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.915 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.915 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.915 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.915 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.915 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.916 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.916 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.916 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "test_orchestrator", 2023-11-16T17:00:27.916 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.916 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.916 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.916 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.916 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.916 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.917 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.917 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.917 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.917 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.917 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.917 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.917 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.917 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.918 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.918 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.918 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.918 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.918 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.918 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.918 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.918 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.919 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.919 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.919 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.919 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.919 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.919 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.919 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.919 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.920 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.920 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.920 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.920 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.920 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.920 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.920 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.920 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.921 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.921 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.921 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.921 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.921 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.921 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.921 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.921 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.922 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.922 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.922 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.922 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.922 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.922 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.922 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.922 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.923 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.923 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.923 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.923 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.923 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.923 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.923 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.923 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.924 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.924 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.924 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.924 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.924 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.924 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.924 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.924 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.925 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.925 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.925 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.925 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.925 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.925 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.925 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "volumes", 2023-11-16T17:00:27.925 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.926 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.926 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.926 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.926 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.926 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.926 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.926 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.926 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.927 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.927 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.927 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.927 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.927 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.927 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.927 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.927 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.928 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.928 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.928 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.928 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.928 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.928 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.928 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.928 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.929 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.929 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.929 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.929 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.929 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.929 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.929 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.929 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.930 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.930 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.930 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.930 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.930 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.930 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.930 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.930 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.931 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.931 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.931 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.931 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.931 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.931 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.931 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.931 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.932 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.932 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.932 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.932 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.932 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.932 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.932 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.932 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.932 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.933 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.933 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.933 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.933 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.933 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.933 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.933 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.933 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.934 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.934 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.934 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.934 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.934 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.934 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.934 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.934 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_clones": { 2023-11-16T17:00:27.935 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_clones", 2023-11-16T17:00:27.935 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.935 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.935 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.935 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4", 2023-11-16T17:00:27.935 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.935 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.935 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.936 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Number of asynchronous cloner threads", 2023-11-16T17:00:27.936 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.936 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.936 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.936 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.936 INFO:teuthology.orchestra.run.smithi044.stdout: "snapshot_clone_delay": { 2023-11-16T17:00:27.936 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snapshot_clone_delay", 2023-11-16T17:00:27.936 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.937 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.937 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.937 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2023-11-16T17:00:27.937 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.937 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.937 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.937 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-11-16T17:00:27.937 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.938 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.938 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.938 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.938 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.938 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.938 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.938 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix", 2023-11-16T17:00:27.938 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.939 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.939 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.939 INFO:teuthology.orchestra.run.smithi044.stdout: "discovery_interval": { 2023-11-16T17:00:27.939 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "discovery_interval", 2023-11-16T17:00:27.939 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2023-11-16T17:00:27.939 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.939 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.939 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "100", 2023-11-16T17:00:27.939 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.940 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.940 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.940 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.940 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.940 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.940 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.940 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.940 INFO:teuthology.orchestra.run.smithi044.stdout: "identifier": { 2023-11-16T17:00:27.940 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "identifier", 2023-11-16T17:00:27.941 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.941 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.941 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.941 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.941 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.941 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.941 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.941 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.942 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.942 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.942 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.942 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.942 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2023-11-16T17:00:27.942 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2023-11-16T17:00:27.942 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:27.942 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.943 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.943 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2023-11-16T17:00:27.943 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.943 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.943 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.943 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.943 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.943 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.944 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.944 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.944 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.944 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.944 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.944 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.944 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.944 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.945 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.945 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.945 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.945 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.945 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.945 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.945 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.945 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.946 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.946 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.946 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.946 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.946 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.946 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.946 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.946 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.947 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.947 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.947 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.947 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.947 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.947 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.947 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.947 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.948 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.948 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.948 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.948 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.948 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.948 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.948 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.948 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.949 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.949 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.949 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.949 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.949 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.949 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.949 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.950 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.950 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.950 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.950 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.950 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.950 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.950 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.950 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.951 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.951 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.951 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.951 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.951 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.951 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.951 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.951 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.952 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.952 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.952 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.952 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.952 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.952 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.952 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.952 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.952 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.953 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_host": { 2023-11-16T17:00:27.953 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_host", 2023-11-16T17:00:27.953 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.953 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.953 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.953 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.953 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.953 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.954 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.954 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.954 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.954 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.954 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.954 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.954 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_port": { 2023-11-16T17:00:27.954 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_port", 2023-11-16T17:00:27.955 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.955 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.955 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.955 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10051", 2023-11-16T17:00:27.955 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.955 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.955 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.955 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.956 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.956 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.956 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.956 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.956 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_sender": { 2023-11-16T17:00:27.956 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_sender", 2023-11-16T17:00:27.956 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.956 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.956 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:27.957 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-11-16T17:00:27.957 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.957 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.957 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.957 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.957 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.957 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.957 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.958 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.958 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.958 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.958 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2023-11-16T17:00:27.958 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.958 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.958 INFO:teuthology.orchestra.run.smithi044.stdout: "modules": [ 2023-11-16T17:00:27.958 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm", 2023-11-16T17:00:27.959 INFO:teuthology.orchestra.run.smithi044.stdout: "dashboard", 2023-11-16T17:00:27.959 INFO:teuthology.orchestra.run.smithi044.stdout: "iostat", 2023-11-16T17:00:27.959 INFO:teuthology.orchestra.run.smithi044.stdout: "nfs", 2023-11-16T17:00:27.959 INFO:teuthology.orchestra.run.smithi044.stdout: "restful", 2023-11-16T17:00:27.959 INFO:teuthology.orchestra.run.smithi044.stdout: "snap_schedule" 2023-11-16T17:00:27.959 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.959 INFO:teuthology.orchestra.run.smithi044.stdout: "available_modules": [ 2023-11-16T17:00:27.959 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.960 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "alerts", 2023-11-16T17:00:27.960 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.960 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.960 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.960 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2023-11-16T17:00:27.960 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2023-11-16T17:00:27.960 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:27.960 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.961 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.961 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2023-11-16T17:00:27.961 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.961 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.961 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.961 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How frequently to reexamine health status", 2023-11-16T17:00:27.961 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.961 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.962 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.962 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.962 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.962 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:27.962 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.962 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.962 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.963 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.963 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.963 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.963 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.963 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.963 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.963 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.963 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.964 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.964 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.964 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.964 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.964 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.964 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.964 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.964 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.965 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:27.965 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:27.965 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.965 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.965 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.965 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.965 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.965 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.966 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.966 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.966 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.966 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.966 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.966 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.966 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:27.966 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:27.967 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.967 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.967 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.967 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:27.967 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.967 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.967 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:27.967 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:27.968 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:27.968 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:27.968 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:27.968 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:27.968 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:27.968 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:27.968 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.968 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.969 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.969 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.969 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.969 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:27.969 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:27.969 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.969 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.969 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.970 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:27.970 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.970 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.970 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.970 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:27.970 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.970 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.970 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.970 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.971 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_destination": { 2023-11-16T17:00:27.971 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_destination", 2023-11-16T17:00:27.971 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.971 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.971 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.971 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.971 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.971 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.972 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.972 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email address to send alerts to", 2023-11-16T17:00:27.972 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.972 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.972 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.972 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.972 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_from_name": { 2023-11-16T17:00:27.972 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_from_name", 2023-11-16T17:00:27.972 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.973 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.973 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.973 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "Ceph", 2023-11-16T17:00:27.973 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.973 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.973 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.973 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email From: name", 2023-11-16T17:00:27.973 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.974 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.974 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.974 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.974 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_host": { 2023-11-16T17:00:27.974 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_host", 2023-11-16T17:00:27.974 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.974 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.974 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.975 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.975 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.975 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.975 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.975 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP server", 2023-11-16T17:00:27.975 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.975 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.975 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.976 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.976 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_password": { 2023-11-16T17:00:27.976 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_password", 2023-11-16T17:00:27.976 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.976 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.976 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.976 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.976 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.977 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.977 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.977 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Password to authenticate with", 2023-11-16T17:00:27.977 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.977 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.977 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.977 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.977 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_port": { 2023-11-16T17:00:27.978 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_port", 2023-11-16T17:00:27.978 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:27.978 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.978 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.978 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "465", 2023-11-16T17:00:27.978 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.978 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.978 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.979 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP port", 2023-11-16T17:00:27.979 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.979 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.979 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.979 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.979 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_sender": { 2023-11-16T17:00:27.979 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_sender", 2023-11-16T17:00:27.979 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.980 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.980 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.980 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.980 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.980 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.980 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.980 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP envelope sender", 2023-11-16T17:00:27.980 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.981 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.981 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.981 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.981 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_ssl": { 2023-11-16T17:00:27.981 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_ssl", 2023-11-16T17:00:27.981 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.981 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.981 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.982 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:27.982 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.982 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.982 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.982 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use SSL to connect to SMTP server", 2023-11-16T17:00:27.982 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.982 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.982 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.983 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.983 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_user": { 2023-11-16T17:00:27.983 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_user", 2023-11-16T17:00:27.983 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.983 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.983 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.983 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:27.983 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.984 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.984 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.984 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "User to authenticate as", 2023-11-16T17:00:27.984 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.984 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.984 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.984 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.984 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:27.985 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.985 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:27.985 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "balancer", 2023-11-16T17:00:27.985 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:27.985 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:27.985 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:27.985 INFO:teuthology.orchestra.run.smithi044.stdout: "active": { 2023-11-16T17:00:27.985 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "active", 2023-11-16T17:00:27.986 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:27.986 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.986 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.986 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:27.986 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.986 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.986 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.986 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically balance PGs across cluster", 2023-11-16T17:00:27.987 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.987 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.987 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.987 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.987 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_time": { 2023-11-16T17:00:27.987 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_time", 2023-11-16T17:00:27.987 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.987 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.988 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.988 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0000", 2023-11-16T17:00:27.988 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.988 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.988 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.988 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "beginning time of day to automatically balance", 2023-11-16T17:00:27.988 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T17:00:27.988 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.989 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.989 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.989 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_weekday": { 2023-11-16T17:00:27.989 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_weekday", 2023-11-16T17:00:27.989 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2023-11-16T17:00:27.989 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.989 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.989 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2023-11-16T17:00:27.990 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2023-11-16T17:00:27.990 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2023-11-16T17:00:27.990 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.990 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-11-16T17:00:27.990 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T17:00:27.990 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.990 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.990 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.991 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_max_iterations": { 2023-11-16T17:00:27.991 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_max_iterations", 2023-11-16T17:00:27.991 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2023-11-16T17:00:27.991 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.991 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.991 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "25", 2023-11-16T17:00:27.991 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2023-11-16T17:00:27.991 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "250", 2023-11-16T17:00:27.992 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.992 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-11-16T17:00:27.992 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:27.992 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.992 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.992 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.992 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_metrics": { 2023-11-16T17:00:27.992 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_metrics", 2023-11-16T17:00:27.993 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.993 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.993 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.993 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "pgs,objects,bytes", 2023-11-16T17:00:27.993 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.993 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.993 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.993 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-11-16T17:00:27.994 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-11-16T17:00:27.994 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.994 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.994 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.994 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_step": { 2023-11-16T17:00:27.994 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_step", 2023-11-16T17:00:27.994 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:27.994 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.995 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.995 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.5", 2023-11-16T17:00:27.995 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0.001", 2023-11-16T17:00:27.995 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "0.999", 2023-11-16T17:00:27.995 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.995 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "aggressiveness of optimization", 2023-11-16T17:00:27.995 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-11-16T17:00:27.995 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.996 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.996 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.996 INFO:teuthology.orchestra.run.smithi044.stdout: "end_time": { 2023-11-16T17:00:27.996 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_time", 2023-11-16T17:00:27.996 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:27.996 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.996 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.996 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2359", 2023-11-16T17:00:27.997 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:27.997 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:27.997 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.997 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ending time of day to automatically balance", 2023-11-16T17:00:27.997 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T17:00:27.997 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.997 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.997 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.998 INFO:teuthology.orchestra.run.smithi044.stdout: "end_weekday": { 2023-11-16T17:00:27.998 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_weekday", 2023-11-16T17:00:27.998 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2023-11-16T17:00:27.998 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:27.998 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:27.998 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2023-11-16T17:00:27.998 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2023-11-16T17:00:27.998 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2023-11-16T17:00:27.999 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:27.999 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-11-16T17:00:27.999 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T17:00:27.999 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:27.999 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:27.999 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:27.999 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:27.999 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.000 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.000 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.000 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.000 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.000 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.000 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.000 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.000 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.000 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.001 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.001 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.001 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.001 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.001 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.001 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.001 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.001 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.002 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.002 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.002 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.002 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.002 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.002 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.002 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.002 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.003 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.003 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.003 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.003 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.003 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.003 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.003 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.003 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.004 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.004 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.004 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.004 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.004 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.004 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.004 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.004 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.005 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.005 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.005 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.005 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.005 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.005 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.005 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.005 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.006 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.006 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.006 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.006 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.006 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.006 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.006 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.006 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.007 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.007 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.007 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.007 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.007 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.007 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.007 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.007 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.008 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.008 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.008 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.008 INFO:teuthology.orchestra.run.smithi044.stdout: "min_score": { 2023-11-16T17:00:28.008 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_score", 2023-11-16T17:00:28.008 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:28.008 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.008 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.009 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2023-11-16T17:00:28.009 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.009 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.009 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.009 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-11-16T17:00:28.009 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.009 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.009 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.009 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.010 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2023-11-16T17:00:28.010 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2023-11-16T17:00:28.010 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.010 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.010 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.010 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "upmap", 2023-11-16T17:00:28.010 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.011 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.011 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.011 INFO:teuthology.orchestra.run.smithi044.stdout: "crush-compat", 2023-11-16T17:00:28.011 INFO:teuthology.orchestra.run.smithi044.stdout: "none", 2023-11-16T17:00:28.011 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap" 2023-11-16T17:00:28.011 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.011 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Balancer mode", 2023-11-16T17:00:28.011 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.012 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.012 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.012 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.012 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_ids": { 2023-11-16T17:00:28.012 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_ids", 2023-11-16T17:00:28.012 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.012 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.012 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.013 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.013 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.013 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.013 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.013 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-11-16T17:00:28.013 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.013 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.013 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.014 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.014 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2023-11-16T17:00:28.014 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2023-11-16T17:00:28.014 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.014 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.014 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.014 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2023-11-16T17:00:28.014 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.015 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.015 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.015 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-11-16T17:00:28.015 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.015 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.015 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.015 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.015 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_deviation": { 2023-11-16T17:00:28.016 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_deviation", 2023-11-16T17:00:28.016 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.016 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.016 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.016 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2023-11-16T17:00:28.016 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2023-11-16T17:00:28.016 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.016 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.017 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "deviation below which no optimization is attempted", 2023-11-16T17:00:28.017 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-11-16T17:00:28.017 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.017 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.017 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.017 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_optimizations": { 2023-11-16T17:00:28.017 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_optimizations", 2023-11-16T17:00:28.017 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2023-11-16T17:00:28.018 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.018 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.018 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:28.018 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.018 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.018 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.018 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-11-16T17:00:28.018 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.019 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.019 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.019 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.019 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.019 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.019 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.019 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cephadm", 2023-11-16T17:00:28.019 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.020 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.020 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.020 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_down_multiplier": { 2023-11-16T17:00:28.020 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_down_multiplier", 2023-11-16T17:00:28.020 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:28.020 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.020 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.020 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2023-11-16T17:00:28.021 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.021 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.021 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.021 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-11-16T17:00:28.021 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.021 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.021 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.021 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.022 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_refresh_rate": { 2023-11-16T17:00:28.022 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_refresh_rate", 2023-11-16T17:00:28.022 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.022 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.022 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.022 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "20", 2023-11-16T17:00:28.022 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.023 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.023 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.023 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-11-16T17:00:28.023 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.023 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.023 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.023 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.023 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_starting_port": { 2023-11-16T17:00:28.023 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_starting_port", 2023-11-16T17:00:28.024 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.024 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.024 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.024 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4721", 2023-11-16T17:00:28.024 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.024 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.024 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.024 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-11-16T17:00:28.025 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.025 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.025 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.025 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.025 INFO:teuthology.orchestra.run.smithi044.stdout: "alertmanager_web_password": { 2023-11-16T17:00:28.025 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "alertmanager_web_password", 2023-11-16T17:00:28.025 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.025 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.026 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.026 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:28.026 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.026 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.026 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.026 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Alertmanager web password", 2023-11-16T17:00:28.026 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.027 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.027 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.027 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.027 INFO:teuthology.orchestra.run.smithi044.stdout: "alertmanager_web_user": { 2023-11-16T17:00:28.027 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "alertmanager_web_user", 2023-11-16T17:00:28.027 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.027 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.027 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.027 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:28.028 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.028 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.028 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.028 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Alertmanager web user", 2023-11-16T17:00:28.028 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.028 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.028 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.028 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.029 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_ptrace": { 2023-11-16T17:00:28.029 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_ptrace", 2023-11-16T17:00:28.029 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.029 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.029 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.029 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.029 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.030 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.030 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.030 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-11-16T17:00:28.030 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-11-16T17:00:28.030 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.030 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.030 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.030 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_interval": { 2023-11-16T17:00:28.031 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_interval", 2023-11-16T17:00:28.031 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.031 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.031 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.031 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2023-11-16T17:00:28.031 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.031 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.031 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.032 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to autotune daemon memory", 2023-11-16T17:00:28.032 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.032 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.032 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.032 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.032 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_memory_target_ratio": { 2023-11-16T17:00:28.032 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_memory_target_ratio", 2023-11-16T17:00:28.032 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:28.033 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.033 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.033 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.7", 2023-11-16T17:00:28.033 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.033 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.033 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.033 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-11-16T17:00:28.033 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.034 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.034 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.034 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.034 INFO:teuthology.orchestra.run.smithi044.stdout: "cgroups_split": { 2023-11-16T17:00:28.034 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cgroups_split", 2023-11-16T17:00:28.034 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.034 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.034 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.035 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.035 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.035 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.035 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.035 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-11-16T17:00:28.035 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.035 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.035 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.036 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.036 INFO:teuthology.orchestra.run.smithi044.stdout: "config_checks_enabled": { 2023-11-16T17:00:28.036 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_checks_enabled", 2023-11-16T17:00:28.036 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.036 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.036 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.036 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.036 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.037 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.037 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.037 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-11-16T17:00:28.037 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.037 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.037 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.037 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.037 INFO:teuthology.orchestra.run.smithi044.stdout: "config_dashboard": { 2023-11-16T17:00:28.038 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_dashboard", 2023-11-16T17:00:28.038 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.038 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.038 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.038 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.038 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.038 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.038 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.039 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-11-16T17:00:28.039 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.039 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.039 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.039 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.039 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_alertmanager": { 2023-11-16T17:00:28.039 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_alertmanager", 2023-11-16T17:00:28.040 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.040 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.040 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.040 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-11-16T17:00:28.040 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.040 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.040 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.040 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:28.041 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.041 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.041 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.041 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.041 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_base": { 2023-11-16T17:00:28.041 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_base", 2023-11-16T17:00:28.041 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.041 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.042 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.042 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/ceph", 2023-11-16T17:00:28.042 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.042 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.042 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.042 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Container image name, without the tag", 2023-11-16T17:00:28.042 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.042 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.043 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.043 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.043 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_elasticsearch": { 2023-11-16T17:00:28.043 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_elasticsearch", 2023-11-16T17:00:28.043 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.043 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.043 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.043 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-11-16T17:00:28.044 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.044 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.044 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.044 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "elasticsearch container image", 2023-11-16T17:00:28.044 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.044 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.044 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.044 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.045 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_grafana": { 2023-11-16T17:00:28.045 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_grafana", 2023-11-16T17:00:28.045 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.045 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.045 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.045 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-11-16T17:00:28.045 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.045 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.046 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.046 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:28.046 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.046 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.046 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.046 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.046 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_haproxy": { 2023-11-16T17:00:28.046 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_haproxy", 2023-11-16T17:00:28.047 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.047 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.047 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.047 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-11-16T17:00:28.047 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.047 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.047 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.047 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "HAproxy container image", 2023-11-16T17:00:28.048 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.048 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.048 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.048 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.048 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_agent": { 2023-11-16T17:00:28.048 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_agent", 2023-11-16T17:00:28.048 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.048 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.049 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.049 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-11-16T17:00:28.049 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.049 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.049 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.049 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger agent container image", 2023-11-16T17:00:28.049 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.049 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.050 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.050 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.050 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_collector": { 2023-11-16T17:00:28.050 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_collector", 2023-11-16T17:00:28.050 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.050 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.050 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.050 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-11-16T17:00:28.051 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.051 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.051 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.051 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger collector container image", 2023-11-16T17:00:28.051 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.051 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.051 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.051 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.052 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_query": { 2023-11-16T17:00:28.052 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_query", 2023-11-16T17:00:28.052 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.052 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.052 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.052 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-11-16T17:00:28.052 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.052 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.053 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.053 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger query container image", 2023-11-16T17:00:28.053 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.053 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.053 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.053 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.053 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_keepalived": { 2023-11-16T17:00:28.053 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_keepalived", 2023-11-16T17:00:28.054 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.054 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.054 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.054 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-11-16T17:00:28.054 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.054 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.054 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.054 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Keepalived container image", 2023-11-16T17:00:28.055 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.055 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.055 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.055 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.055 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_loki": { 2023-11-16T17:00:28.055 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_loki", 2023-11-16T17:00:28.055 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.055 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.056 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.056 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-11-16T17:00:28.056 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.056 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.056 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.056 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Loki container image", 2023-11-16T17:00:28.056 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.056 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.057 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.057 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.057 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_node_exporter": { 2023-11-16T17:00:28.057 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_node_exporter", 2023-11-16T17:00:28.057 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.057 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.057 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.057 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-11-16T17:00:28.058 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.058 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.058 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.058 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:28.058 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.058 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.058 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.059 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.059 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_prometheus": { 2023-11-16T17:00:28.059 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_prometheus", 2023-11-16T17:00:28.059 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.059 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.059 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.059 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-11-16T17:00:28.060 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.060 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.060 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.060 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:28.060 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.060 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.060 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.060 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.061 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_promtail": { 2023-11-16T17:00:28.061 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_promtail", 2023-11-16T17:00:28.061 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.061 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.061 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.061 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-11-16T17:00:28.061 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.061 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.062 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.062 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Promtail container image", 2023-11-16T17:00:28.062 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.062 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.062 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.062 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.062 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_snmp_gateway": { 2023-11-16T17:00:28.062 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_snmp_gateway", 2023-11-16T17:00:28.063 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.063 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.063 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.063 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-11-16T17:00:28.063 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.063 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.063 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.063 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SNMP Gateway container image", 2023-11-16T17:00:28.064 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.064 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.064 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.064 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.064 INFO:teuthology.orchestra.run.smithi044.stdout: "container_init": { 2023-11-16T17:00:28.064 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_init", 2023-11-16T17:00:28.064 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.064 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.065 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.065 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.065 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.065 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.065 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.065 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Run podman/docker with `--init`", 2023-11-16T17:00:28.065 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.065 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.066 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.066 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.066 INFO:teuthology.orchestra.run.smithi044.stdout: "daemon_cache_timeout": { 2023-11-16T17:00:28.066 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "daemon_cache_timeout", 2023-11-16T17:00:28.066 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.066 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.066 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.066 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2023-11-16T17:00:28.067 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.067 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.067 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.067 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-11-16T17:00:28.067 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.067 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.067 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.067 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.068 INFO:teuthology.orchestra.run.smithi044.stdout: "default_cephadm_command_timeout": { 2023-11-16T17:00:28.068 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_cephadm_command_timeout", 2023-11-16T17:00:28.068 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.068 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.068 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.068 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "900", 2023-11-16T17:00:28.068 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.068 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.069 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.069 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-11-16T17:00:28.069 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.069 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.069 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.069 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.069 INFO:teuthology.orchestra.run.smithi044.stdout: "default_registry": { 2023-11-16T17:00:28.069 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_registry", 2023-11-16T17:00:28.070 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.070 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.070 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.070 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io", 2023-11-16T17:00:28.070 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.070 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.070 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.070 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-11-16T17:00:28.071 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.071 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.071 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.071 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.071 INFO:teuthology.orchestra.run.smithi044.stdout: "device_cache_timeout": { 2023-11-16T17:00:28.071 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_cache_timeout", 2023-11-16T17:00:28.072 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.072 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.072 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.072 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1800", 2023-11-16T17:00:28.072 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.073 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.073 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.073 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache device inventory", 2023-11-16T17:00:28.073 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.073 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.073 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.073 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.073 INFO:teuthology.orchestra.run.smithi044.stdout: "device_enhanced_scan": { 2023-11-16T17:00:28.074 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_enhanced_scan", 2023-11-16T17:00:28.074 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.074 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.074 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.074 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.074 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.074 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.075 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.075 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use libstoragemgmt during device scans", 2023-11-16T17:00:28.075 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.075 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.075 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.075 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.075 INFO:teuthology.orchestra.run.smithi044.stdout: "facts_cache_timeout": { 2023-11-16T17:00:28.075 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "facts_cache_timeout", 2023-11-16T17:00:28.076 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.076 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.076 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.076 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2023-11-16T17:00:28.076 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.076 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.076 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.077 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache host facts data", 2023-11-16T17:00:28.077 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.077 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.077 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.077 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.077 INFO:teuthology.orchestra.run.smithi044.stdout: "host_check_interval": { 2023-11-16T17:00:28.077 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "host_check_interval", 2023-11-16T17:00:28.077 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.078 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.078 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.078 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2023-11-16T17:00:28.078 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.078 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.078 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.078 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to perform a host check", 2023-11-16T17:00:28.079 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.079 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.079 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.079 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.079 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.079 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.079 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.080 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.080 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.080 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.080 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.080 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.080 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.080 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.080 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.081 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.081 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.081 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.081 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.081 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.081 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.081 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.082 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.082 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.082 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.082 INFO:teuthology.orchestra.run.smithi044.stdout: "log_refresh_metadata": { 2023-11-16T17:00:28.082 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_refresh_metadata", 2023-11-16T17:00:28.082 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.082 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.082 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.083 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.083 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.083 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.083 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.083 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-11-16T17:00:28.083 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.083 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.084 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.084 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.084 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.084 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.084 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.084 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.084 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.085 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.085 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.085 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.085 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.085 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-11-16T17:00:28.085 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.085 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.085 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.086 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.086 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.086 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.086 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.086 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.086 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.086 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.087 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.087 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.087 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.087 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.087 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.087 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.087 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.087 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.088 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.088 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.088 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.088 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.088 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.088 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.088 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.089 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.089 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.089 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.089 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.089 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.089 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.089 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.089 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.090 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.090 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.090 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.090 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.090 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.090 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.090 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf": { 2023-11-16T17:00:28.091 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-11-16T17:00:28.091 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.091 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.091 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.091 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.091 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.091 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.092 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.092 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-11-16T17:00:28.092 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.092 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.092 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.092 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.092 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-11-16T17:00:28.092 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-11-16T17:00:28.093 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.093 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.093 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.093 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "*", 2023-11-16T17:00:28.093 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.093 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.093 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.093 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-11-16T17:00:28.094 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.094 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.094 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.094 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.094 INFO:teuthology.orchestra.run.smithi044.stdout: "max_count_per_host": { 2023-11-16T17:00:28.094 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_count_per_host", 2023-11-16T17:00:28.094 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.095 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.095 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.095 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:28.095 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.095 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.095 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.095 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of daemons per service per host", 2023-11-16T17:00:28.095 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.096 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.096 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.096 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.096 INFO:teuthology.orchestra.run.smithi044.stdout: "max_osd_draining_count": { 2023-11-16T17:00:28.096 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_osd_draining_count", 2023-11-16T17:00:28.096 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.096 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.097 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.097 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:28.097 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.097 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.097 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.097 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-11-16T17:00:28.097 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.097 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.098 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.098 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.098 INFO:teuthology.orchestra.run.smithi044.stdout: "migration_current": { 2023-11-16T17:00:28.098 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "migration_current", 2023-11-16T17:00:28.098 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.098 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.098 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.099 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.099 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.099 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.099 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.099 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "internal - do not modify", 2023-11-16T17:00:28.099 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.099 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.099 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.100 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.100 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2023-11-16T17:00:28.100 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2023-11-16T17:00:28.100 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.100 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.100 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.100 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "root", 2023-11-16T17:00:28.101 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.101 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.101 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.101 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm-package", 2023-11-16T17:00:28.101 INFO:teuthology.orchestra.run.smithi044.stdout: "root" 2023-11-16T17:00:28.101 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.101 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "mode for remote execution of cephadm", 2023-11-16T17:00:28.101 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.102 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.102 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.102 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.102 INFO:teuthology.orchestra.run.smithi044.stdout: "prometheus_alerts_path": { 2023-11-16T17:00:28.102 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus_alerts_path", 2023-11-16T17:00:28.102 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.102 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.103 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.103 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-11-16T17:00:28.103 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.103 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.103 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.103 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-11-16T17:00:28.103 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.103 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.104 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.104 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.104 INFO:teuthology.orchestra.run.smithi044.stdout: "prometheus_web_password": { 2023-11-16T17:00:28.104 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus_web_password", 2023-11-16T17:00:28.104 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.104 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.104 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.104 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:28.105 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.105 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.105 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.105 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus web password", 2023-11-16T17:00:28.105 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.105 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.105 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.106 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.106 INFO:teuthology.orchestra.run.smithi044.stdout: "prometheus_web_user": { 2023-11-16T17:00:28.106 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus_web_user", 2023-11-16T17:00:28.106 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.106 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.106 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.106 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:28.106 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.107 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.107 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.107 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus web user", 2023-11-16T17:00:28.107 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.107 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.107 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.107 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.108 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_insecure": { 2023-11-16T17:00:28.108 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_insecure", 2023-11-16T17:00:28.108 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.108 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.108 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.108 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.108 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.108 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.109 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.109 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-11-16T17:00:28.109 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.109 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.109 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.109 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.109 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_password": { 2023-11-16T17:00:28.110 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_password", 2023-11-16T17:00:28.110 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.110 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.110 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.110 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.110 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.110 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.111 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.111 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-11-16T17:00:28.111 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.111 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.111 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.111 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.111 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_url": { 2023-11-16T17:00:28.111 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_url", 2023-11-16T17:00:28.112 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.112 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.112 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.112 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.112 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.112 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.112 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.113 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-11-16T17:00:28.113 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.113 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.113 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.113 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.113 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_username": { 2023-11-16T17:00:28.113 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_username", 2023-11-16T17:00:28.113 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.114 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.114 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.114 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.114 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.114 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.114 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.114 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-11-16T17:00:28.115 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.115 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.115 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.115 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.115 INFO:teuthology.orchestra.run.smithi044.stdout: "secure_monitoring_stack": { 2023-11-16T17:00:28.115 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "secure_monitoring_stack", 2023-11-16T17:00:28.115 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.115 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.116 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.116 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.116 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.116 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.116 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.116 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-11-16T17:00:28.116 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.116 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.117 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.117 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.117 INFO:teuthology.orchestra.run.smithi044.stdout: "service_discovery_port": { 2023-11-16T17:00:28.117 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "service_discovery_port", 2023-11-16T17:00:28.117 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.117 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.117 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.118 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8765", 2023-11-16T17:00:28.118 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.118 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.118 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.118 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "cephadm service discovery port", 2023-11-16T17:00:28.118 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.118 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.119 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.119 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.119 INFO:teuthology.orchestra.run.smithi044.stdout: "ssh_config_file": { 2023-11-16T17:00:28.119 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssh_config_file", 2023-11-16T17:00:28.119 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.119 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.119 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.120 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.120 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.120 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.120 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.120 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-11-16T17:00:28.120 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.120 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.120 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.121 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.121 INFO:teuthology.orchestra.run.smithi044.stdout: "use_agent": { 2023-11-16T17:00:28.121 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_agent", 2023-11-16T17:00:28.121 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.121 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.121 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.121 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.121 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.122 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.122 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.122 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-11-16T17:00:28.122 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.122 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.122 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.122 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.123 INFO:teuthology.orchestra.run.smithi044.stdout: "use_repo_digest": { 2023-11-16T17:00:28.123 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_repo_digest", 2023-11-16T17:00:28.123 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.123 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.123 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.123 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.123 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.123 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.124 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.124 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-11-16T17:00:28.124 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.124 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.124 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.124 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.124 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_failed_host_check": { 2023-11-16T17:00:28.124 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_failed_host_check", 2023-11-16T17:00:28.125 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.125 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.125 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.125 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.125 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.125 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.125 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.126 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if the host check fails", 2023-11-16T17:00:28.126 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.126 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.126 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.126 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.126 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_daemons": { 2023-11-16T17:00:28.126 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_daemons", 2023-11-16T17:00:28.126 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.127 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.127 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.127 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.127 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.127 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.127 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.127 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-11-16T17:00:28.128 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.128 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.128 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.128 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.128 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_hosts": { 2023-11-16T17:00:28.128 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_hosts", 2023-11-16T17:00:28.128 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.128 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.129 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.129 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.129 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.129 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.129 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.129 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-11-16T17:00:28.129 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.130 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.130 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.130 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.130 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.130 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.130 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.130 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crash", 2023-11-16T17:00:28.131 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.131 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.131 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.131 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.131 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.131 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.131 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.131 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.132 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.132 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.132 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.132 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.132 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.132 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.132 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.133 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.133 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.133 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.133 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.133 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.133 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.133 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.133 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.134 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.134 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.134 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.134 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.134 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.134 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.134 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.134 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.135 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.135 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.135 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.135 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.135 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.135 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.135 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.136 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.136 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.136 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.136 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.136 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.136 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.136 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.136 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.137 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.137 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.137 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.137 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.137 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.137 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.137 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.137 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.138 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.138 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.138 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.138 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.138 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.138 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.138 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.139 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.139 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.139 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.139 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.139 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.139 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.139 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.140 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.140 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.140 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.140 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.140 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.140 INFO:teuthology.orchestra.run.smithi044.stdout: "retain_interval": { 2023-11-16T17:00:28.140 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retain_interval", 2023-11-16T17:00:28.140 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.141 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.141 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.141 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "31536000", 2023-11-16T17:00:28.141 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.141 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.141 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.141 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain crashes before pruning them", 2023-11-16T17:00:28.141 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.142 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.142 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.142 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.142 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_recent_interval": { 2023-11-16T17:00:28.142 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_recent_interval", 2023-11-16T17:00:28.142 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.142 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.143 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.143 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1209600", 2023-11-16T17:00:28.143 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.143 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.143 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.143 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "time interval in which to warn about recent crashes", 2023-11-16T17:00:28.143 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.143 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.144 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.144 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.144 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.144 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.144 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.144 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dashboard", 2023-11-16T17:00:28.144 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.145 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.145 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.145 INFO:teuthology.orchestra.run.smithi044.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-11-16T17:00:28.145 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-11-16T17:00:28.145 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.145 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.145 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.145 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:28.146 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.146 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.146 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.146 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.146 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.146 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.146 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.147 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.147 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_HOST": { 2023-11-16T17:00:28.147 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_HOST", 2023-11-16T17:00:28.147 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.147 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.147 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.147 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.147 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.148 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.148 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.148 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.148 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.148 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.148 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.148 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.149 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-11-16T17:00:28.149 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-11-16T17:00:28.149 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.149 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.149 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.149 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.149 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.150 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.150 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.150 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.150 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.150 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.150 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.150 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.150 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_ENABLED": { 2023-11-16T17:00:28.151 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_ENABLED", 2023-11-16T17:00:28.151 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.151 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.151 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.151 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.151 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.151 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.152 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.152 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.152 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.152 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.152 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.152 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.152 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-11-16T17:00:28.152 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-11-16T17:00:28.153 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.153 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.153 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.153 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.153 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.153 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.153 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.154 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.154 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.154 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.154 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.154 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.154 INFO:teuthology.orchestra.run.smithi044.stdout: "ENABLE_BROWSABLE_API": { 2023-11-16T17:00:28.154 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ENABLE_BROWSABLE_API", 2023-11-16T17:00:28.154 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.155 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.155 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.155 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.155 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.155 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.155 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.155 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.156 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.156 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.156 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.156 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.156 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-11-16T17:00:28.156 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-11-16T17:00:28.156 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.156 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.157 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.157 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.157 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.157 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.157 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.157 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.157 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.157 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.158 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.158 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.158 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-11-16T17:00:28.158 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-11-16T17:00:28.158 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.158 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.158 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.159 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.159 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.159 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.159 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.159 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.159 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.159 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.159 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.160 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.160 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-11-16T17:00:28.160 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-11-16T17:00:28.160 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.160 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.160 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.160 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.161 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.161 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.161 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.161 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.161 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.161 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.161 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.161 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.162 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-11-16T17:00:28.162 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-11-16T17:00:28.162 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.162 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.162 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.162 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.162 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.163 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.163 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.163 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.163 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.163 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.163 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.163 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.163 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_NFS": { 2023-11-16T17:00:28.164 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-11-16T17:00:28.164 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.164 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.164 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.164 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.164 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.164 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.164 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.165 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.165 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.165 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.165 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.165 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.165 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RBD": { 2023-11-16T17:00:28.165 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-11-16T17:00:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.166 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.167 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.167 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.167 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.167 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.167 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RGW": { 2023-11-16T17:00:28.167 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-11-16T17:00:28.167 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.167 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.168 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.168 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.168 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.168 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.168 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.168 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.168 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.169 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.169 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.169 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.169 INFO:teuthology.orchestra.run.smithi044.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-11-16T17:00:28.169 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-11-16T17:00:28.169 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.169 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.169 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.170 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.170 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.170 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.170 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.170 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.170 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.170 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.171 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.171 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.171 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_PASSWORD": { 2023-11-16T17:00:28.171 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_PASSWORD", 2023-11-16T17:00:28.171 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.171 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.171 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.171 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:28.172 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.172 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.172 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.172 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.172 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.172 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.172 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.173 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.173 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-11-16T17:00:28.173 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-11-16T17:00:28.173 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.173 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.173 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.173 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.173 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.174 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.174 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.174 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.174 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.174 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.174 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.174 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.175 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_URL": { 2023-11-16T17:00:28.175 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_URL", 2023-11-16T17:00:28.175 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.175 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.175 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.175 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.175 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.175 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.176 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.176 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.176 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.176 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.176 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.176 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.176 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_USERNAME": { 2023-11-16T17:00:28.177 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_USERNAME", 2023-11-16T17:00:28.177 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.177 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.177 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.177 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:28.177 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.177 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.177 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.178 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.178 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.178 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.178 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.178 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.178 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-11-16T17:00:28.178 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-11-16T17:00:28.179 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.179 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.179 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.179 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.179 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.179 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.179 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.180 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.180 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.180 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.180 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.180 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.180 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-11-16T17:00:28.180 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-11-16T17:00:28.180 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.181 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.181 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.181 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.181 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.181 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.181 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.181 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.181 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.182 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.182 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.182 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.182 INFO:teuthology.orchestra.run.smithi044.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-11-16T17:00:28.182 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-11-16T17:00:28.182 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.182 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.183 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.183 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.183 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.183 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.183 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.183 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.183 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.183 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.184 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.184 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.184 INFO:teuthology.orchestra.run.smithi044.stdout: "ISSUE_TRACKER_API_KEY": { 2023-11-16T17:00:28.184 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-11-16T17:00:28.184 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.184 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.184 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.184 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.185 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.185 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.185 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.185 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.185 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.185 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.185 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.186 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.186 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_HOST": { 2023-11-16T17:00:28.186 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_HOST", 2023-11-16T17:00:28.186 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.186 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.186 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.186 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.186 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.187 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.187 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.187 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.187 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.187 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.187 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.187 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.188 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-11-16T17:00:28.188 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-11-16T17:00:28.188 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.188 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.188 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.188 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.188 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.188 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.189 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.189 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.189 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.189 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.189 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.189 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.189 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-11-16T17:00:28.190 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-11-16T17:00:28.190 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.190 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.190 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.190 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.190 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.191 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.191 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.191 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.191 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.191 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.191 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.191 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.192 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-11-16T17:00:28.192 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-11-16T17:00:28.192 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.192 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.192 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.192 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.192 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.193 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.193 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.193 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.193 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.193 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.193 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.193 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.194 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-11-16T17:00:28.194 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-11-16T17:00:28.194 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.194 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.194 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.194 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.194 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.195 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.195 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.195 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.195 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.195 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.195 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.195 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.195 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-11-16T17:00:28.196 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-11-16T17:00:28.196 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.196 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.196 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.196 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.196 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.196 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.197 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.197 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.197 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.197 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.197 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.197 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.197 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-11-16T17:00:28.197 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-11-16T17:00:28.198 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.198 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.198 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.198 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.198 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.198 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.198 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.199 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.199 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.199 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.199 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.199 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.199 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-11-16T17:00:28.199 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-11-16T17:00:28.200 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.200 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.200 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.200 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.200 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.200 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.200 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.201 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.201 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.201 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.201 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.201 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.201 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-11-16T17:00:28.201 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-11-16T17:00:28.201 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.202 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.202 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.202 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.202 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.202 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.202 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.202 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.203 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.203 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.203 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.203 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.203 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_ENABLED": { 2023-11-16T17:00:28.203 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_ENABLED", 2023-11-16T17:00:28.203 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.203 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.204 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.204 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.204 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.204 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.204 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.204 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.204 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.205 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.205 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.205 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.205 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-11-16T17:00:28.205 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-11-16T17:00:28.205 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.205 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.205 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.206 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-11-16T17:00:28.206 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.206 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.206 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.206 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.206 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.206 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.207 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.207 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.207 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-11-16T17:00:28.207 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-11-16T17:00:28.207 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.207 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.207 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.207 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:28.208 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.208 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.208 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.208 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.208 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.208 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.208 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.209 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.209 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-11-16T17:00:28.209 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-11-16T17:00:28.209 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.209 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.209 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.209 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8", 2023-11-16T17:00:28.209 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.210 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.210 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.210 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.210 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.210 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.210 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.210 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.211 INFO:teuthology.orchestra.run.smithi044.stdout: "REST_REQUESTS_TIMEOUT": { 2023-11-16T17:00:28.211 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-11-16T17:00:28.211 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.211 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.211 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.211 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "45", 2023-11-16T17:00:28.211 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.211 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.212 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.212 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.212 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.212 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.212 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.212 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.212 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ACCESS_KEY": { 2023-11-16T17:00:28.213 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ACCESS_KEY", 2023-11-16T17:00:28.213 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.213 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.213 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.213 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.213 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.213 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.213 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.214 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.214 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.214 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.214 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.214 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.214 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-11-16T17:00:28.214 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-11-16T17:00:28.214 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.215 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.215 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.215 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2023-11-16T17:00:28.215 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.215 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.215 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.215 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.216 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.216 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.216 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.216 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.216 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SECRET_KEY": { 2023-11-16T17:00:28.216 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SECRET_KEY", 2023-11-16T17:00:28.216 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.217 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.217 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.217 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.217 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.217 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.217 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.217 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.218 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.218 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.218 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.218 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.218 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SSL_VERIFY": { 2023-11-16T17:00:28.218 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SSL_VERIFY", 2023-11-16T17:00:28.218 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.219 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.219 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.219 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.219 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.219 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.219 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.219 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.220 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.220 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.220 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.220 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.220 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-11-16T17:00:28.220 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-11-16T17:00:28.220 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.221 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.221 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.221 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2023-11-16T17:00:28.221 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.221 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.221 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.221 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.222 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.222 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.222 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.222 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.222 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-11-16T17:00:28.222 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-11-16T17:00:28.223 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.223 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.223 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.223 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:28.223 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.223 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.223 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.224 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.224 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.224 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.224 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.224 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.224 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-11-16T17:00:28.224 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-11-16T17:00:28.225 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.225 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.225 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.225 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2023-11-16T17:00:28.225 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.225 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.225 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.225 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.226 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.226 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.226 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.226 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.226 INFO:teuthology.orchestra.run.smithi044.stdout: "cross_origin_url": { 2023-11-16T17:00:28.226 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cross_origin_url", 2023-11-16T17:00:28.226 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.227 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.227 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.227 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.227 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.227 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.227 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.227 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.228 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.228 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.228 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.228 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.228 INFO:teuthology.orchestra.run.smithi044.stdout: "crt_file": { 2023-11-16T17:00:28.228 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crt_file", 2023-11-16T17:00:28.228 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.229 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.229 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.229 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.229 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.229 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.229 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.230 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.230 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.230 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.230 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.230 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.230 INFO:teuthology.orchestra.run.smithi044.stdout: "debug": { 2023-11-16T17:00:28.230 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "debug", 2023-11-16T17:00:28.231 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.231 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.231 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.231 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.231 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.231 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.231 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.231 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable/disable debug options", 2023-11-16T17:00:28.232 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.232 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.232 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.232 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.232 INFO:teuthology.orchestra.run.smithi044.stdout: "jwt_token_ttl": { 2023-11-16T17:00:28.232 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "jwt_token_ttl", 2023-11-16T17:00:28.232 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.233 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.233 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.233 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "28800", 2023-11-16T17:00:28.233 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.233 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.233 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.233 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.233 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.234 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.234 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.234 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.234 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2023-11-16T17:00:28.234 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2023-11-16T17:00:28.234 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.234 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.235 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.235 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.235 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.235 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.235 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.235 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.235 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.235 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.236 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.236 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.236 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.236 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.236 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.236 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.236 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.236 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.237 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.237 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.237 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.237 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.237 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.237 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.237 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.238 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.238 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.238 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.238 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.238 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.238 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.238 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.238 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.239 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.239 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.239 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.239 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.239 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.239 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.239 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.239 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.240 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.240 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.240 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.240 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.240 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.240 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.240 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.241 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.241 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.241 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.241 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.241 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.241 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.241 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.242 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.242 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.242 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.242 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.242 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.242 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.242 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.242 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.243 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.243 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.243 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.243 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.243 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.243 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.243 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.244 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.244 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.244 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.244 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.244 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.244 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.244 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.244 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.245 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.245 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.245 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.245 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.245 INFO:teuthology.orchestra.run.smithi044.stdout: "motd": { 2023-11-16T17:00:28.245 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "motd", 2023-11-16T17:00:28.245 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.245 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.246 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.246 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.246 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.246 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.246 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.246 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "The message of the day", 2023-11-16T17:00:28.246 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.247 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.247 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.247 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.247 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect_resolve_ip_addr": { 2023-11-16T17:00:28.247 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "redirect_resolve_ip_addr", 2023-11-16T17:00:28.247 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.247 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.248 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.248 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.248 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.248 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.248 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.248 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.248 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.249 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.249 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.249 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.249 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2023-11-16T17:00:28.249 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2023-11-16T17:00:28.249 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.250 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.250 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.250 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2023-11-16T17:00:28.250 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.250 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.250 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.250 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.251 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.251 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.251 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.251 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.251 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2023-11-16T17:00:28.251 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2023-11-16T17:00:28.251 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.252 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.252 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.252 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8080", 2023-11-16T17:00:28.252 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.252 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.252 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.252 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.252 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.253 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.253 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.253 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.253 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2023-11-16T17:00:28.253 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2023-11-16T17:00:28.253 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.253 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.253 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.254 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.254 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.254 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.254 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.254 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.254 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.254 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.255 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.255 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.255 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl_server_port": { 2023-11-16T17:00:28.255 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl_server_port", 2023-11-16T17:00:28.255 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.255 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.255 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.256 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8443", 2023-11-16T17:00:28.256 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.256 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.256 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.256 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.256 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.256 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.256 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.257 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.257 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2023-11-16T17:00:28.257 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2023-11-16T17:00:28.257 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.257 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.257 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.257 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "redirect", 2023-11-16T17:00:28.258 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.258 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.258 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.258 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.258 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect" 2023-11-16T17:00:28.258 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.258 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.259 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.259 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.259 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.259 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.259 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2023-11-16T17:00:28.259 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2023-11-16T17:00:28.259 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.260 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.260 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.260 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2023-11-16T17:00:28.260 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2023-11-16T17:00:28.260 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2023-11-16T17:00:28.260 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.260 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.261 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.261 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.261 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.261 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.261 INFO:teuthology.orchestra.run.smithi044.stdout: "url_prefix": { 2023-11-16T17:00:28.261 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url_prefix", 2023-11-16T17:00:28.261 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.262 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.262 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.262 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.262 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.262 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.262 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.262 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.263 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.263 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.263 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.263 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.263 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.263 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.263 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.263 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "devicehealth", 2023-11-16T17:00:28.264 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.264 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.264 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.264 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_monitoring": { 2023-11-16T17:00:28.264 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_monitoring", 2023-11-16T17:00:28.264 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.264 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "monitor device health metrics", 2023-11-16T17:00:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.266 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.266 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.266 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.266 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.266 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.266 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.266 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.267 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.267 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.267 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.267 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.267 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.267 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.267 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.268 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.268 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.268 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.268 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.268 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.268 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.268 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.268 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.269 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.269 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.269 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.269 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.269 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.269 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.269 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.270 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.270 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.270 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.270 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.270 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.270 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.270 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.271 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.271 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.271 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.271 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.271 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.271 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.271 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.271 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.272 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.272 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.272 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.272 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.272 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.272 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.272 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.272 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.273 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.273 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.273 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.273 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.273 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.273 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.273 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.274 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.274 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.274 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.274 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.274 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.274 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.274 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.274 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.275 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.275 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.275 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.275 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.275 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.275 INFO:teuthology.orchestra.run.smithi044.stdout: "mark_out_threshold": { 2023-11-16T17:00:28.275 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mark_out_threshold", 2023-11-16T17:00:28.276 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.276 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.276 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.276 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2419200", 2023-11-16T17:00:28.276 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.276 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.276 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.276 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-11-16T17:00:28.277 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.277 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.277 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.277 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.277 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_name": { 2023-11-16T17:00:28.277 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_name", 2023-11-16T17:00:28.277 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.277 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.278 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.278 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "device_health_metrics", 2023-11-16T17:00:28.278 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.278 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.278 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.278 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name of pool in which to store device health metrics", 2023-11-16T17:00:28.278 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.279 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.279 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.279 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.279 INFO:teuthology.orchestra.run.smithi044.stdout: "retention_period": { 2023-11-16T17:00:28.279 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retention_period", 2023-11-16T17:00:28.279 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.279 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.280 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.280 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15552000", 2023-11-16T17:00:28.280 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.280 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.280 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.280 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain device health metrics", 2023-11-16T17:00:28.280 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.281 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.281 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.281 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.281 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_frequency": { 2023-11-16T17:00:28.281 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_frequency", 2023-11-16T17:00:28.281 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.281 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.281 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.282 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2023-11-16T17:00:28.282 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.282 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.282 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.282 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to scrape device health metrics", 2023-11-16T17:00:28.282 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.282 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.283 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.283 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.283 INFO:teuthology.orchestra.run.smithi044.stdout: "self_heal": { 2023-11-16T17:00:28.283 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "self_heal", 2023-11-16T17:00:28.283 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.283 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.283 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.283 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.284 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.284 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.284 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.284 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-11-16T17:00:28.284 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.284 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.284 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.285 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.285 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2023-11-16T17:00:28.285 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2023-11-16T17:00:28.285 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.285 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.285 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.285 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2023-11-16T17:00:28.285 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.286 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.286 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.286 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and check device health", 2023-11-16T17:00:28.286 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.286 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.286 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.286 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.287 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_threshold": { 2023-11-16T17:00:28.287 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_threshold", 2023-11-16T17:00:28.287 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.287 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.287 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.287 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7257600", 2023-11-16T17:00:28.287 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.287 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.288 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.288 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-11-16T17:00:28.288 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.288 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.288 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.288 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.288 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.289 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.289 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.289 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "diskprediction_local", 2023-11-16T17:00:28.289 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.289 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.289 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.289 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.289 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.290 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.290 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.290 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.290 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.290 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.290 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.291 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.291 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.291 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.291 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.291 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.291 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.291 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.291 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.292 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.292 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.292 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.292 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.292 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.292 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.292 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.293 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.293 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.293 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.293 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.293 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.293 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.293 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.293 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.294 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.294 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.294 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.294 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.294 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.294 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.294 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.295 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.295 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.295 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.295 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.295 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.295 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.295 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.296 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.296 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.296 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.296 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.296 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.296 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.296 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.296 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.297 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.297 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.297 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.297 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.297 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.297 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.297 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.297 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.298 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.298 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.298 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.298 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.298 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.298 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.299 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.299 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.299 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.299 INFO:teuthology.orchestra.run.smithi044.stdout: "predict_interval": { 2023-11-16T17:00:28.299 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predict_interval", 2023-11-16T17:00:28.299 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.299 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.299 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.300 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2023-11-16T17:00:28.300 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.300 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.300 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.300 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.300 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.300 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.301 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.301 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.301 INFO:teuthology.orchestra.run.smithi044.stdout: "predictor_model": { 2023-11-16T17:00:28.301 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predictor_model", 2023-11-16T17:00:28.301 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.301 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.301 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.301 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "prophetstor", 2023-11-16T17:00:28.302 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.302 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.302 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.302 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.302 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.302 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.302 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.303 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.303 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2023-11-16T17:00:28.303 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2023-11-16T17:00:28.303 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.303 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.303 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.303 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2023-11-16T17:00:28.303 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.304 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.304 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.304 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.304 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.304 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.304 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.304 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.304 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.305 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.305 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.305 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "influx", 2023-11-16T17:00:28.305 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": false, 2023-11-16T17:00:28.305 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "influxdb python module not found", 2023-11-16T17:00:28.305 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.305 INFO:teuthology.orchestra.run.smithi044.stdout: "batch_size": { 2023-11-16T17:00:28.306 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "batch_size", 2023-11-16T17:00:28.306 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.306 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.306 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.306 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5000", 2023-11-16T17:00:28.306 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.306 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.306 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.307 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-11-16T17:00:28.307 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.307 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.307 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.307 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.307 INFO:teuthology.orchestra.run.smithi044.stdout: "database": { 2023-11-16T17:00:28.307 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "database", 2023-11-16T17:00:28.307 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.308 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.308 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.308 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "ceph", 2023-11-16T17:00:28.308 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.308 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.308 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.308 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-11-16T17:00:28.309 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.309 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.309 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.309 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.309 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": { 2023-11-16T17:00:28.309 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "hostname", 2023-11-16T17:00:28.309 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.309 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.310 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.310 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.310 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.310 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.310 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.310 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server hostname", 2023-11-16T17:00:28.311 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.311 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.311 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.311 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.311 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2023-11-16T17:00:28.311 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2023-11-16T17:00:28.311 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.311 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.312 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.312 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "30", 2023-11-16T17:00:28.312 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "5", 2023-11-16T17:00:28.312 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.312 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.312 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-11-16T17:00:28.312 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.312 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.313 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.313 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.313 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.313 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.313 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.313 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.313 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.314 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.314 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.314 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.314 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.314 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.314 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.314 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.314 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.315 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.315 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.315 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.315 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.315 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.315 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.315 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.316 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.316 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.316 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.316 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.316 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.316 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.316 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.316 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.317 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.317 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.317 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.317 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.317 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.317 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.317 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.318 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.318 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.318 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.318 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.318 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.318 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.318 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.318 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.319 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.319 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.319 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.319 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.319 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.319 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.319 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.320 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.320 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.320 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.320 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.320 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.320 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.320 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.320 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.321 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.321 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.321 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.321 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.321 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.321 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.321 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "password": { 2023-11-16T17:00:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "password", 2023-11-16T17:00:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.323 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.323 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.323 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.323 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.323 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.323 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.323 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "password of InfluxDB server user", 2023-11-16T17:00:28.323 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.324 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.324 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.324 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.324 INFO:teuthology.orchestra.run.smithi044.stdout: "port": { 2023-11-16T17:00:28.324 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "port", 2023-11-16T17:00:28.324 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.324 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.325 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.325 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8086", 2023-11-16T17:00:28.325 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.325 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.325 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.325 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server port", 2023-11-16T17:00:28.326 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.326 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.326 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.326 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.326 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2023-11-16T17:00:28.326 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2023-11-16T17:00:28.326 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.326 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.327 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.327 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "false", 2023-11-16T17:00:28.327 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.327 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.327 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.327 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T17:00:28.327 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.328 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.328 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.328 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.328 INFO:teuthology.orchestra.run.smithi044.stdout: "threads": { 2023-11-16T17:00:28.328 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threads", 2023-11-16T17:00:28.328 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.328 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.328 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.329 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2023-11-16T17:00:28.329 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2023-11-16T17:00:28.329 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "32", 2023-11-16T17:00:28.329 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.329 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-11-16T17:00:28.329 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.329 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.330 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.330 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.330 INFO:teuthology.orchestra.run.smithi044.stdout: "username": { 2023-11-16T17:00:28.330 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "username", 2023-11-16T17:00:28.330 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.330 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.330 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.331 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.331 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.331 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.331 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.331 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "username of InfluxDB server user", 2023-11-16T17:00:28.331 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.331 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.332 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.332 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.332 INFO:teuthology.orchestra.run.smithi044.stdout: "verify_ssl": { 2023-11-16T17:00:28.332 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "verify_ssl", 2023-11-16T17:00:28.332 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.332 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.332 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.332 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "true", 2023-11-16T17:00:28.333 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.333 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.333 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.333 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T17:00:28.333 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.333 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.333 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.333 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.334 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.334 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.334 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.334 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "insights", 2023-11-16T17:00:28.334 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.334 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.334 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.335 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.335 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.335 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.335 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.335 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.335 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.335 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.335 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.336 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.336 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.336 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.336 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.336 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.336 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.336 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.337 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.337 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.337 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.337 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.337 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.337 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.337 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.337 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.338 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.338 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.338 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.338 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.338 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.338 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.338 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.338 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.339 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.339 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.339 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.339 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.339 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.339 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.340 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.340 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.340 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.340 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.340 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.340 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.340 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.340 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.341 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.341 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.341 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.341 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.341 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.341 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.342 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.342 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.342 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.342 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.342 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.342 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.342 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.343 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.343 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.343 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.343 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.343 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.343 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.343 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.343 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.344 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.344 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.344 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.344 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.344 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.344 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.344 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.345 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "iostat", 2023-11-16T17:00:28.345 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.345 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.345 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.345 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.345 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.345 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.345 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.346 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.346 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.346 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.346 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.346 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.346 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.346 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.347 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.347 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.347 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.347 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.347 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.347 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.347 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.348 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.348 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.348 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.348 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.348 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.348 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.348 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.348 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.349 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.349 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.349 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.349 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.349 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.349 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.349 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.350 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.350 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.350 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.350 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.350 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.350 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.350 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.351 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.351 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.351 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.351 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.351 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.351 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.351 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.351 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.352 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.352 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.352 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.352 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.352 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.352 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.352 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.352 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.353 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.353 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.353 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.353 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.353 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.353 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.353 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.354 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.354 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.354 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.354 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.354 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.354 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.354 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.354 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.355 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.355 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.355 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "k8sevents", 2023-11-16T17:00:28.355 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.355 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.355 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.355 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_event_retention_days": { 2023-11-16T17:00:28.356 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ceph_event_retention_days", 2023-11-16T17:00:28.356 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.356 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.356 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.356 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7", 2023-11-16T17:00:28.356 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.356 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.356 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.357 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Days to hold ceph event information within local cache", 2023-11-16T17:00:28.357 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.357 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.357 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.357 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.357 INFO:teuthology.orchestra.run.smithi044.stdout: "config_check_secs": { 2023-11-16T17:00:28.357 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_check_secs", 2023-11-16T17:00:28.357 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.358 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.358 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.358 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:28.358 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "10", 2023-11-16T17:00:28.358 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.358 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.358 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-11-16T17:00:28.359 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.359 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.359 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.359 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.359 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.359 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.359 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.360 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.360 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.360 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.360 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.360 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.360 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.360 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.361 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.361 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.361 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.361 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.361 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.361 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.361 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.361 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.362 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.362 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.362 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.362 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.362 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.362 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.362 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.362 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.363 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.363 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.363 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.363 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.363 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.363 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.363 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.364 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.364 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.364 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.364 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.364 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.364 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.364 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.364 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.365 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.365 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.365 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.365 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.365 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.365 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.365 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.365 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.366 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.366 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.366 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.366 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.366 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.366 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.366 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.367 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.367 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.367 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.367 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.367 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.367 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.367 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.367 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.368 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.368 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.368 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.368 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.368 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.368 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.368 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.369 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.369 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.369 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "localpool", 2023-11-16T17:00:28.369 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.369 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.369 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.369 INFO:teuthology.orchestra.run.smithi044.stdout: "failure_domain": { 2023-11-16T17:00:28.370 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "failure_domain", 2023-11-16T17:00:28.370 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.370 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.370 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.370 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "host", 2023-11-16T17:00:28.370 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.370 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.370 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.371 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "failure domain for any created local pool", 2023-11-16T17:00:28.371 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-11-16T17:00:28.371 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.371 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.371 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.371 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.371 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.372 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.372 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.372 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.372 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.372 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.372 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.372 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.373 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.373 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.373 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.373 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.373 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.373 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.373 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.373 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.374 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.374 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.374 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.374 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.374 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.374 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.374 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.375 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.375 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.375 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.375 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.375 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.375 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.375 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.376 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.376 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.376 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.376 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.376 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.376 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.376 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.377 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.377 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.377 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.377 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.377 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.377 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.377 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.377 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.378 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.378 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.378 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.378 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.378 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.378 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.379 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.379 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.379 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.379 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.379 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.379 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.379 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.380 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.380 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.380 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.380 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.380 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.380 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.380 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.381 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.381 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.381 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.381 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.381 INFO:teuthology.orchestra.run.smithi044.stdout: "min_size": { 2023-11-16T17:00:28.381 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_size", 2023-11-16T17:00:28.381 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.381 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.382 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.382 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.382 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.382 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.382 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.382 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default min_size for any created local pool", 2023-11-16T17:00:28.382 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-11-16T17:00:28.382 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.383 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.383 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.383 INFO:teuthology.orchestra.run.smithi044.stdout: "num_rep": { 2023-11-16T17:00:28.383 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "num_rep", 2023-11-16T17:00:28.383 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.383 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.383 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.384 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3", 2023-11-16T17:00:28.384 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.384 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.384 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.384 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default replica count for any created local pool", 2023-11-16T17:00:28.384 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.384 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.384 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.385 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.385 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_num": { 2023-11-16T17:00:28.385 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_num", 2023-11-16T17:00:28.385 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.385 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.385 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.385 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "128", 2023-11-16T17:00:28.386 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.386 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.386 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.386 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default pg_num for any created local pool", 2023-11-16T17:00:28.386 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.386 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.386 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.386 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.387 INFO:teuthology.orchestra.run.smithi044.stdout: "prefix": { 2023-11-16T17:00:28.387 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prefix", 2023-11-16T17:00:28.387 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.387 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.387 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.387 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.387 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.388 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.388 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.388 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name prefix for any created local pool", 2023-11-16T17:00:28.388 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.388 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.388 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.388 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.388 INFO:teuthology.orchestra.run.smithi044.stdout: "subtree": { 2023-11-16T17:00:28.389 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "subtree", 2023-11-16T17:00:28.389 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.389 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.389 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.389 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "rack", 2023-11-16T17:00:28.389 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.389 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.390 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.390 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "CRUSH level for which to create a local pool", 2023-11-16T17:00:28.390 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-11-16T17:00:28.390 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.390 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.390 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.390 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.390 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.391 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.391 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mds_autoscaler", 2023-11-16T17:00:28.391 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.391 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.391 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.391 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.391 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.392 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.392 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.392 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.392 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.392 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.392 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.392 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.392 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.393 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.393 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.393 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.393 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.393 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.393 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.393 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.394 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.394 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.394 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.394 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.394 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.394 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.395 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.395 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.395 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.395 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.395 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.395 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.395 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.396 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.396 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.396 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.396 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.396 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.396 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.396 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.397 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.397 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.397 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.397 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.397 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.397 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.397 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.398 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.398 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.398 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.398 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.398 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.398 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.399 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.399 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.399 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.399 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.399 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.399 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.399 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.400 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.400 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.400 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.400 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.400 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.400 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.401 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.401 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.401 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.401 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.401 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.401 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.401 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.402 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.402 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.402 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.402 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirroring", 2023-11-16T17:00:28.402 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.402 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.402 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.403 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.403 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.403 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.403 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.403 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.404 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.404 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.404 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.404 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.404 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.404 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.404 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.405 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.405 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.405 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.405 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.405 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.405 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.405 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.406 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.406 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.406 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.406 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.406 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.406 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.406 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.407 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.407 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.407 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.407 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.407 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.407 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.407 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.407 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.408 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.408 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.408 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.408 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.408 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.408 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.408 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.408 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.409 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.409 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.409 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.409 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.409 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.409 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.409 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.410 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.410 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.410 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.410 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.410 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.410 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.411 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.411 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.411 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.411 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.411 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.411 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.411 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.412 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.412 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.412 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.412 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.412 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.412 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.412 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.412 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.413 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.413 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.413 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.413 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "nfs", 2023-11-16T17:00:28.413 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.413 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.413 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.414 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.414 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.414 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.414 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.414 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.414 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.414 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.415 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.415 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.415 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.415 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.415 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.415 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.415 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.415 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.416 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.416 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.416 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.416 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.416 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.416 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.416 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.417 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.417 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.417 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.417 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.417 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.417 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.417 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.418 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.418 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.418 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.418 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.418 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.418 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.418 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.419 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.419 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.419 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.419 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.419 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.419 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.419 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.420 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.420 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.420 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.420 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.420 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.420 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.420 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.421 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.421 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.421 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.421 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.421 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.421 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.421 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.422 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.422 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.422 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.422 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.422 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.422 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.422 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.423 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.423 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.423 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.423 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.423 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.423 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.423 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.423 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.424 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.424 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2023-11-16T17:00:28.424 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.424 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.424 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.424 INFO:teuthology.orchestra.run.smithi044.stdout: "fail_fs": { 2023-11-16T17:00:28.424 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "fail_fs", 2023-11-16T17:00:28.425 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.425 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.425 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.425 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.425 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.425 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.425 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.425 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-11-16T17:00:28.426 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.426 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.426 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.426 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.426 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.426 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.427 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.427 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.427 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.427 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.427 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.427 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.427 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.428 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.428 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.428 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.428 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.428 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.428 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.428 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.428 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.429 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.429 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.429 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.429 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.429 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.429 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.430 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.430 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.430 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.430 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.430 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.430 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.430 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.431 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.431 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.431 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.431 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.431 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.431 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.431 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.432 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.432 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.432 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.432 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.432 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.432 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.432 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.433 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.433 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.433 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.433 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.433 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.433 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.433 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.433 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.434 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.434 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.434 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.434 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.434 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.434 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.434 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.435 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.435 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.435 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.435 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.435 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.435 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.435 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.436 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.436 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.436 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.436 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.436 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator": { 2023-11-16T17:00:28.436 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2023-11-16T17:00:28.436 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.436 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.437 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.437 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.437 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.437 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.437 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.437 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm", 2023-11-16T17:00:28.437 INFO:teuthology.orchestra.run.smithi044.stdout: "rook", 2023-11-16T17:00:28.438 INFO:teuthology.orchestra.run.smithi044.stdout: "test_orchestrator" 2023-11-16T17:00:28.438 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.438 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Orchestrator backend", 2023-11-16T17:00:28.438 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.438 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.438 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.438 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.439 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.439 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.439 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.439 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_perf_query", 2023-11-16T17:00:28.439 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.439 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.439 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.440 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.440 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.440 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.440 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.440 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.440 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.440 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.440 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.441 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.441 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.441 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.441 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.441 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.441 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.441 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.441 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.442 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.442 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.442 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.442 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.442 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.442 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.442 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.443 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.443 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.443 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.443 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.443 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.443 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.443 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.443 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.444 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.444 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.444 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.444 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.444 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.444 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.444 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.444 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.445 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.445 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.445 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.445 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.445 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.445 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.445 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.446 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.446 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.446 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.446 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.446 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.446 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.446 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.446 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.447 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.447 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.447 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.447 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.447 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.447 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.447 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.448 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.448 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.448 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.448 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.448 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.448 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.448 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.448 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.449 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.449 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.449 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.449 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.449 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_support", 2023-11-16T17:00:28.449 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.449 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.450 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.450 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.450 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.450 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.450 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.450 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.450 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.451 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.451 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.451 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.451 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.451 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.451 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.451 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.451 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.452 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.452 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.452 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.452 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.452 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.452 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.452 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.452 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.453 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.453 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.453 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.453 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.453 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.453 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.453 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.454 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.454 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.454 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.454 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.454 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.454 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.454 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.454 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.455 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.455 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.455 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.455 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.455 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.455 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.455 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.455 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.456 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.456 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.456 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.456 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.456 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.456 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.456 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.457 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.457 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.457 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.457 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.457 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.457 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.457 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.457 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.458 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.458 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.458 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.458 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.458 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.458 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.458 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.459 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.459 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.459 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.459 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.459 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.459 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.459 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_autoscaler", 2023-11-16T17:00:28.459 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.460 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.460 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.460 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.460 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.460 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.460 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.460 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.461 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.461 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.461 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.461 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.461 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.461 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.461 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.461 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.462 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.462 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.462 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.462 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.462 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.462 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.462 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.463 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.463 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.463 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.463 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.463 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.463 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.463 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.463 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.464 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.464 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.464 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.464 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.464 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.464 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.464 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.464 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.465 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.465 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.465 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.465 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.465 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.465 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.465 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.465 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.466 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.466 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.466 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.466 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.466 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.466 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.466 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.467 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.467 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.467 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.467 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.467 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.467 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.467 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.467 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.468 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.468 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.468 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.468 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.468 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.468 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.468 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.469 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.469 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.469 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.469 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.469 INFO:teuthology.orchestra.run.smithi044.stdout: "noautoscale": { 2023-11-16T17:00:28.469 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "noautoscale", 2023-11-16T17:00:28.470 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.470 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.470 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.470 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.470 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.470 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.470 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.470 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "global autoscale flag", 2023-11-16T17:00:28.471 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-11-16T17:00:28.471 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.471 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.471 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.471 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2023-11-16T17:00:28.471 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2023-11-16T17:00:28.471 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.472 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.472 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.472 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2023-11-16T17:00:28.472 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.472 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.472 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.472 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.473 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.473 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.473 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.473 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.473 INFO:teuthology.orchestra.run.smithi044.stdout: "threshold": { 2023-11-16T17:00:28.473 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threshold", 2023-11-16T17:00:28.473 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:28.474 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.474 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.474 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2023-11-16T17:00:28.474 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1.0", 2023-11-16T17:00:28.474 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.474 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.474 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "scaling threshold", 2023-11-16T17:00:28.474 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-11-16T17:00:28.475 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.475 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.475 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.475 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.475 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.475 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.475 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "progress", 2023-11-16T17:00:28.476 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.476 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.476 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.476 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_pg_recovery_event": { 2023-11-16T17:00:28.476 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_pg_recovery_event", 2023-11-16T17:00:28.476 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.476 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.476 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.477 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.477 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.477 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.477 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.477 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow the module to show pg recovery progress", 2023-11-16T17:00:28.477 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.477 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.478 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.478 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.478 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2023-11-16T17:00:28.478 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2023-11-16T17:00:28.478 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.478 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.478 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.478 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.479 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.479 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.479 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.479 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.479 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.479 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.479 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.480 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.480 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.480 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.480 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.480 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.480 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.480 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.481 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.481 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.481 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.481 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.481 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.481 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.481 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.481 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.482 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.482 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.482 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.482 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.482 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.482 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.482 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.482 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.483 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.483 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.483 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.483 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.483 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.483 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.483 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.483 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.484 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.484 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.484 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.484 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.484 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.484 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.484 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.485 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.485 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.485 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.485 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.485 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.485 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.485 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.485 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.486 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.486 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.486 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.486 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.486 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.486 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.486 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.486 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.487 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.487 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.487 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.487 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.487 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.487 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.487 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.488 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.488 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.488 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.488 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.488 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.488 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.488 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.488 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.489 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.489 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.489 INFO:teuthology.orchestra.run.smithi044.stdout: "max_completed_events": { 2023-11-16T17:00:28.489 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_completed_events", 2023-11-16T17:00:28.489 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.489 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.489 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.490 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "50", 2023-11-16T17:00:28.490 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.490 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.490 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.490 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "number of past completed events to remember", 2023-11-16T17:00:28.490 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.490 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.490 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.491 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.491 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2023-11-16T17:00:28.491 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2023-11-16T17:00:28.491 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.491 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.491 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.491 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2023-11-16T17:00:28.492 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.492 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.492 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.492 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long the module is going to sleep", 2023-11-16T17:00:28.492 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.492 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.492 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.493 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.493 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.493 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.493 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.493 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus", 2023-11-16T17:00:28.493 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.493 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.494 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.494 INFO:teuthology.orchestra.run.smithi044.stdout: "cache": { 2023-11-16T17:00:28.494 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cache", 2023-11-16T17:00:28.494 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.494 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.494 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.494 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.495 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.495 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.495 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.495 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.495 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.495 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.495 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.496 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.496 INFO:teuthology.orchestra.run.smithi044.stdout: "exclude_perf_counters": { 2023-11-16T17:00:28.496 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "exclude_perf_counters", 2023-11-16T17:00:28.496 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.496 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.496 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.496 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.496 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.497 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.497 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.497 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-11-16T17:00:28.497 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-11-16T17:00:28.497 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.497 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.497 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.497 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.498 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.498 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.498 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.498 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.498 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.498 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.499 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.499 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.499 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.499 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.499 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.499 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.499 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.500 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.500 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.500 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.500 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.500 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.500 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.500 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.501 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.501 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.501 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.501 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.501 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.501 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.501 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.502 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.502 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.502 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.502 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.502 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.502 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.502 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.502 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.503 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.503 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.503 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.503 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.503 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.503 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.503 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.504 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.504 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.504 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.504 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.504 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.504 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.504 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.504 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.505 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.505 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.505 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.505 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.505 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.505 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.505 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.506 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.506 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.506 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.506 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.506 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.506 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.506 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.506 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.507 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.507 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.507 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.507 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.507 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools": { 2023-11-16T17:00:28.507 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools", 2023-11-16T17:00:28.507 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.508 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.508 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.508 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.508 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.508 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.508 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.508 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.508 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.509 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.509 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.509 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.509 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools_refresh_interval": { 2023-11-16T17:00:28.509 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-11-16T17:00:28.509 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.509 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.510 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.510 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "300", 2023-11-16T17:00:28.510 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.510 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.510 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.510 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.510 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.510 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.511 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.511 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.511 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_interval": { 2023-11-16T17:00:28.511 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_interval", 2023-11-16T17:00:28.511 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:28.511 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.511 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.512 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15.0", 2023-11-16T17:00:28.512 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.512 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.512 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.512 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.512 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.512 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.512 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.513 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.513 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2023-11-16T17:00:28.513 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2023-11-16T17:00:28.513 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.513 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.513 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.513 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2023-11-16T17:00:28.514 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.514 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.514 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.514 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-11-16T17:00:28.514 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.514 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.514 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.515 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.515 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2023-11-16T17:00:28.515 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2023-11-16T17:00:28.515 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.515 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.515 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.515 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "9283", 2023-11-16T17:00:28.516 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.516 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.516 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.516 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-11-16T17:00:28.516 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.516 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.516 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.516 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.517 INFO:teuthology.orchestra.run.smithi044.stdout: "stale_cache_strategy": { 2023-11-16T17:00:28.517 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stale_cache_strategy", 2023-11-16T17:00:28.517 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.517 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.517 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.517 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "log", 2023-11-16T17:00:28.517 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.517 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.518 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.518 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.518 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.518 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.518 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.518 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.519 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2023-11-16T17:00:28.519 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2023-11-16T17:00:28.519 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.519 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.519 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.519 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "default", 2023-11-16T17:00:28.519 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.520 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.520 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.520 INFO:teuthology.orchestra.run.smithi044.stdout: "default", 2023-11-16T17:00:28.520 INFO:teuthology.orchestra.run.smithi044.stdout: "error" 2023-11-16T17:00:28.520 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.520 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.520 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.520 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.521 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.521 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.521 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2023-11-16T17:00:28.521 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2023-11-16T17:00:28.521 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.521 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.521 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.522 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2023-11-16T17:00:28.522 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2023-11-16T17:00:28.522 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2023-11-16T17:00:28.522 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.522 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.522 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.522 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.522 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.523 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.523 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.523 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.523 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.523 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_support", 2023-11-16T17:00:28.523 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.523 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.523 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.524 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.524 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.524 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.524 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.524 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.524 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.524 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.525 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.525 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.525 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.525 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.525 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.525 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.525 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.525 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.526 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.526 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.526 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.526 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.526 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.526 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.527 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.527 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.527 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.527 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.527 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.527 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.527 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.527 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.528 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.528 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.528 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.528 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.528 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.528 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.528 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.529 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.529 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.529 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.529 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.529 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.529 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.529 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.530 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.530 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.530 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.530 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.530 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.530 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.530 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.531 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.531 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.531 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.531 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.531 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.531 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.531 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.532 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.532 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.532 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.532 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.532 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.532 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.532 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.533 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.533 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.533 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.533 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.533 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.533 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.533 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_snap_create": { 2023-11-16T17:00:28.533 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_snap_create", 2023-11-16T17:00:28.534 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.534 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.534 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.534 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2023-11-16T17:00:28.534 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.534 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.534 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.535 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.535 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.535 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.535 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.535 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.535 INFO:teuthology.orchestra.run.smithi044.stdout: "mirror_snapshot_schedule": { 2023-11-16T17:00:28.535 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirror_snapshot_schedule", 2023-11-16T17:00:28.535 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.536 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.537 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.537 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.537 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.537 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.537 INFO:teuthology.orchestra.run.smithi044.stdout: "trash_purge_schedule": { 2023-11-16T17:00:28.537 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "trash_purge_schedule", 2023-11-16T17:00:28.537 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.538 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.538 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.538 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.538 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.538 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.538 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.538 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.539 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.539 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.539 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.539 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.539 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.539 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.539 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.540 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "restful", 2023-11-16T17:00:28.540 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.540 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.540 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.540 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_auth": { 2023-11-16T17:00:28.540 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_auth", 2023-11-16T17:00:28.540 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.541 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.541 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.541 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.541 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.541 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.541 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.541 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.542 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.542 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.542 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.542 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.542 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2023-11-16T17:00:28.542 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2023-11-16T17:00:28.542 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.542 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.543 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.543 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.543 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.543 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.543 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.543 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.543 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.544 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.544 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.544 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.544 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.544 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.544 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.544 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.544 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.545 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.545 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.545 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.545 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.545 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.545 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.545 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.545 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.546 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.546 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.546 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.546 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.546 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.546 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.546 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.547 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.547 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.547 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.547 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.547 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.547 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.547 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.548 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.548 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.548 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.548 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.548 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.548 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.548 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.548 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.549 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.549 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.549 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.549 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.549 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.549 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.549 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.550 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.550 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.550 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.550 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.550 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.550 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.551 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.551 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.551 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.551 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.551 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.551 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.551 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.551 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.552 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.552 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.552 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.552 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.552 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.552 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.552 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.552 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.553 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.553 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.553 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.553 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.553 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.553 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.553 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2023-11-16T17:00:28.553 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2023-11-16T17:00:28.553 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.554 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.554 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.554 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.554 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.554 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.554 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.554 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.555 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.555 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.555 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.555 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.555 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2023-11-16T17:00:28.555 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2023-11-16T17:00:28.555 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.555 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.556 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.556 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.556 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.556 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.556 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.556 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.556 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.557 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.557 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.557 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.557 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.557 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.557 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.557 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rgw", 2023-11-16T17:00:28.558 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.558 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.558 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.558 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.558 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.558 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.558 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.559 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.559 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.559 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.559 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.559 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.559 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.559 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.560 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.560 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.560 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.560 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.560 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.560 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.561 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.561 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.561 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.561 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.561 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.561 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.561 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.561 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.562 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.562 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.562 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.562 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.562 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.562 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.562 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.563 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.563 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.563 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.563 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.563 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.563 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.563 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.563 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.564 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.564 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.564 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.564 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.564 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.564 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.564 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.565 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.565 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.565 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.565 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.565 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.565 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.565 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.565 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.566 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.566 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.566 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.566 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.566 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.566 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.566 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.567 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.567 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.567 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.567 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.567 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.567 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.567 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.567 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.568 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.568 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.568 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.568 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rook", 2023-11-16T17:00:28.568 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.568 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.568 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.569 INFO:teuthology.orchestra.run.smithi044.stdout: "drive_group_interval": { 2023-11-16T17:00:28.569 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "drive_group_interval", 2023-11-16T17:00:28.569 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:28.569 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.569 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.569 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "300.0", 2023-11-16T17:00:28.569 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.569 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.570 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.570 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-11-16T17:00:28.570 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.570 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.570 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.570 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.570 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.571 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.571 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.571 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.571 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.571 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.571 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.571 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.571 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.572 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.572 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.572 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.572 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.572 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.572 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.572 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.573 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.573 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.573 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.573 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.573 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.573 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.573 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.573 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.574 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.574 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.574 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.574 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.574 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.574 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.574 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.575 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.575 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.575 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.575 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.575 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.575 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.575 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.575 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.576 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.576 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.576 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.576 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.576 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.576 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.576 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.576 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.577 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.577 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.577 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.577 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.577 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.577 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.577 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.578 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.578 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.578 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.578 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.578 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.578 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.578 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.578 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.579 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.579 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.579 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.579 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.579 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.579 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.579 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.580 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.580 INFO:teuthology.orchestra.run.smithi044.stdout: "storage_class": { 2023-11-16T17:00:28.580 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "storage_class", 2023-11-16T17:00:28.580 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.580 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.580 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.580 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "local", 2023-11-16T17:00:28.580 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.581 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.581 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.581 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-11-16T17:00:28.581 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.581 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.581 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.581 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.582 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.582 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.582 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.582 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "selftest", 2023-11-16T17:00:28.582 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.582 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.582 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.582 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.583 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.583 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.583 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.583 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.583 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.583 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.583 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.584 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.584 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.584 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.584 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.584 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.584 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.584 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.584 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.585 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.585 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.585 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.585 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.585 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.585 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.585 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.585 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.586 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.586 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.586 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.586 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.586 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.586 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.586 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.587 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.587 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.587 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.587 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.587 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.587 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.587 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.587 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.588 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.588 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.588 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.588 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.588 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.588 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.588 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.589 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.589 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.589 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.589 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.589 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.589 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.589 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.590 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.590 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.590 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.590 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.590 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.590 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.590 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.590 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.591 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.591 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.591 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.591 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.591 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.591 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.591 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.592 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.592 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.592 INFO:teuthology.orchestra.run.smithi044.stdout: "roption1": { 2023-11-16T17:00:28.592 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption1", 2023-11-16T17:00:28.592 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.592 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.592 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.592 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.593 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.593 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.593 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.593 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.593 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.593 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.593 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.594 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.594 INFO:teuthology.orchestra.run.smithi044.stdout: "roption2": { 2023-11-16T17:00:28.594 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption2", 2023-11-16T17:00:28.594 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.594 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.594 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.594 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "xyz", 2023-11-16T17:00:28.594 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.595 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.595 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.595 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.595 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.595 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.595 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.595 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.595 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption1": { 2023-11-16T17:00:28.596 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption1", 2023-11-16T17:00:28.596 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.596 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.596 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.596 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.596 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.596 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.597 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.597 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.597 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.597 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.597 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.597 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.597 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption2": { 2023-11-16T17:00:28.597 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption2", 2023-11-16T17:00:28.598 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.598 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.598 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.598 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.598 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.598 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.598 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.599 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.599 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.599 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.599 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.599 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.599 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption3": { 2023-11-16T17:00:28.599 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption3", 2023-11-16T17:00:28.599 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2023-11-16T17:00:28.600 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.600 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.600 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.600 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.600 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.600 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.600 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.601 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.601 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.601 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.601 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.601 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption4": { 2023-11-16T17:00:28.601 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption4", 2023-11-16T17:00:28.601 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.601 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.602 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.602 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.602 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.602 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.602 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.602 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.602 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.603 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.603 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.603 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.603 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption5": { 2023-11-16T17:00:28.603 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption5", 2023-11-16T17:00:28.603 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.603 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.604 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.604 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.604 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.604 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.604 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.604 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.604 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.604 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.605 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.605 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.605 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption6": { 2023-11-16T17:00:28.605 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption6", 2023-11-16T17:00:28.605 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.605 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.605 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.606 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.606 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.606 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.606 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.606 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.606 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.606 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.607 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.607 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.607 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption7": { 2023-11-16T17:00:28.607 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption7", 2023-11-16T17:00:28.607 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.607 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.607 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.608 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.608 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2023-11-16T17:00:28.608 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "42", 2023-11-16T17:00:28.608 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.608 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.608 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.608 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.608 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.609 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.609 INFO:teuthology.orchestra.run.smithi044.stdout: "testkey": { 2023-11-16T17:00:28.609 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testkey", 2023-11-16T17:00:28.609 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.609 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.609 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.609 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.610 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.610 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.610 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.610 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.610 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.610 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.610 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.611 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.611 INFO:teuthology.orchestra.run.smithi044.stdout: "testlkey": { 2023-11-16T17:00:28.611 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testlkey", 2023-11-16T17:00:28.611 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.611 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.611 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.611 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.611 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.612 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.612 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.612 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.612 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.612 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.612 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.612 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.612 INFO:teuthology.orchestra.run.smithi044.stdout: "testnewline": { 2023-11-16T17:00:28.613 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testnewline", 2023-11-16T17:00:28.613 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.613 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.613 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.613 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.613 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.613 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.614 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.614 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.614 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.614 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.614 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.614 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.614 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.614 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.615 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.615 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snap_schedule", 2023-11-16T17:00:28.615 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.615 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.615 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.615 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_m_granularity": { 2023-11-16T17:00:28.615 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_m_granularity", 2023-11-16T17:00:28.616 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.616 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.616 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.616 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.616 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.616 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.616 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.616 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow minute scheduled snapshots", 2023-11-16T17:00:28.617 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.617 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.617 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.617 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.617 INFO:teuthology.orchestra.run.smithi044.stdout: "dump_on_update": { 2023-11-16T17:00:28.617 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dump_on_update", 2023-11-16T17:00:28.617 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.617 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.618 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.618 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.618 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.618 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.618 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.618 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "dump database to debug log on update", 2023-11-16T17:00:28.618 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.619 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.619 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.619 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.619 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.619 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.619 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.619 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.620 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.620 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.620 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.620 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.620 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.620 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.620 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.621 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.621 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.621 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.621 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.621 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.621 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.621 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.621 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.622 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.622 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.622 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.622 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.622 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.622 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.622 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.623 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.623 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.623 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.623 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.623 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.623 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.623 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.623 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.624 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.624 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.624 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.624 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.624 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.624 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.624 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.625 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.625 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.625 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.625 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.625 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.625 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.625 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.625 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.626 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.626 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.626 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.626 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.626 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.626 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.626 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.626 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.627 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.627 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.627 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.627 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.627 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.627 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.627 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.628 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.628 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.628 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.628 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.628 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.628 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.628 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.628 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.629 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.629 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stats", 2023-11-16T17:00:28.629 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.629 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.629 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.629 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.629 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.630 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.630 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.630 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.630 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.630 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.630 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.630 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.631 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.631 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.631 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.631 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.631 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.631 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.631 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.632 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.632 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.632 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.632 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.632 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.632 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.632 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.632 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.633 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.633 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.633 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.633 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.633 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.633 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.633 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.633 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.634 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.634 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.634 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.634 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.634 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.634 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.634 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.635 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.635 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.635 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.635 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.635 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.635 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.635 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.635 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.636 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.636 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.636 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.636 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.636 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.636 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.636 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.637 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.637 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.637 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.637 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.637 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.637 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.637 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.638 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.638 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.638 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.638 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.638 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.638 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.638 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.639 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.639 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.639 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.639 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.639 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.639 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "status", 2023-11-16T17:00:28.639 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.640 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.640 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.640 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.640 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.640 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.640 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.641 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.641 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.641 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.641 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.641 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.641 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.641 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.642 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.642 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.642 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.642 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.642 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.642 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.642 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.643 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.643 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.643 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.643 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.643 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.643 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.644 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.644 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.644 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.644 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.644 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.644 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.645 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.645 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.645 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.645 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.645 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.645 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.645 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.646 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.646 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.646 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.646 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.646 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.646 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.647 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.647 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.647 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.647 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.647 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.647 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.648 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.648 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.648 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.648 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.648 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.648 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.648 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.649 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.649 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.649 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.649 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.649 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.649 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.649 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.650 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.650 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.650 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.650 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.650 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.650 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.651 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.651 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.651 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.651 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.651 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telegraf", 2023-11-16T17:00:28.651 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.651 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.652 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.652 INFO:teuthology.orchestra.run.smithi044.stdout: "address": { 2023-11-16T17:00:28.652 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "address", 2023-11-16T17:00:28.652 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.652 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.652 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.652 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-11-16T17:00:28.652 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.653 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.653 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.653 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.653 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.653 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.653 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.653 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.654 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2023-11-16T17:00:28.654 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2023-11-16T17:00:28.654 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.654 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.654 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.654 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15", 2023-11-16T17:00:28.654 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.655 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.655 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.655 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.655 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.655 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.655 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.655 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.656 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.656 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.656 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.656 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.656 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.656 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.656 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.657 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.657 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.657 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.657 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.657 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.657 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.657 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.658 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.658 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.658 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.658 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.658 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.658 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.658 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.659 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.659 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.659 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.659 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.659 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.659 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.659 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.660 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.660 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.660 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.660 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.660 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.660 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.660 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.661 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.661 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.661 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.661 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.661 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.661 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.661 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.661 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.662 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.662 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.662 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.662 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.662 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.662 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.662 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.663 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.663 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.663 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.663 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.663 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.663 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.663 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.664 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.664 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.664 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.664 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.664 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.664 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.664 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.664 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.665 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.665 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.665 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.665 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.665 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.665 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.665 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.665 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.666 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telemetry", 2023-11-16T17:00:28.666 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.666 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.666 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.666 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_basic": { 2023-11-16T17:00:28.666 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_basic", 2023-11-16T17:00:28.666 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.667 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.667 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.667 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.667 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.667 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.667 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.667 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share basic cluster information (size, version)", 2023-11-16T17:00:28.668 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.668 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.668 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.668 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.668 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_crash": { 2023-11-16T17:00:28.668 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_crash", 2023-11-16T17:00:28.668 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.668 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.669 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.669 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.669 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.669 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.669 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.669 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-11-16T17:00:28.670 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.670 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.670 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.670 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.670 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_device": { 2023-11-16T17:00:28.670 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_device", 2023-11-16T17:00:28.670 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.671 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.671 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.671 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2023-11-16T17:00:28.671 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.671 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.671 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.671 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-11-16T17:00:28.672 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.672 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.672 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.672 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.672 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_ident": { 2023-11-16T17:00:28.672 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_ident", 2023-11-16T17:00:28.672 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.673 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.673 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.673 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.673 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.673 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.673 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.673 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-11-16T17:00:28.674 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.674 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.674 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.674 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.674 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_perf": { 2023-11-16T17:00:28.674 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_perf", 2023-11-16T17:00:28.674 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.675 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.675 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.675 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.675 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.675 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.675 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.675 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share various performance metrics of a cluster", 2023-11-16T17:00:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: "contact": { 2023-11-16T17:00:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "contact", 2023-11-16T17:00:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.677 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.677 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.677 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.677 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.677 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.677 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.677 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.678 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.678 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.678 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.678 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.678 INFO:teuthology.orchestra.run.smithi044.stdout: "description": { 2023-11-16T17:00:28.678 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "description", 2023-11-16T17:00:28.679 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.679 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.679 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.679 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.679 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.679 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.679 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.680 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.680 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.680 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.680 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.680 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.680 INFO:teuthology.orchestra.run.smithi044.stdout: "device_url": { 2023-11-16T17:00:28.680 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_url", 2023-11-16T17:00:28.681 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.681 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.681 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.681 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-11-16T17:00:28.681 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.681 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.681 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.682 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.682 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.682 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.682 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.682 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.682 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2023-11-16T17:00:28.682 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2023-11-16T17:00:28.683 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.683 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.683 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.683 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.683 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.683 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.683 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.684 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.684 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.684 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.684 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.684 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.684 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2023-11-16T17:00:28.684 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2023-11-16T17:00:28.685 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.685 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.685 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.685 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "24", 2023-11-16T17:00:28.685 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "8", 2023-11-16T17:00:28.685 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.686 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.686 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.686 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.686 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.686 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.686 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.686 INFO:teuthology.orchestra.run.smithi044.stdout: "last_opt_revision": { 2023-11-16T17:00:28.687 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "last_opt_revision", 2023-11-16T17:00:28.687 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.687 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.687 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.687 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1", 2023-11-16T17:00:28.687 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.688 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.688 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.688 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.688 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.688 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.688 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.688 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.689 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard": { 2023-11-16T17:00:28.689 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard", 2023-11-16T17:00:28.689 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.689 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.689 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.689 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.690 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.690 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.690 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.690 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.690 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.690 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.690 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.691 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.691 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard_description": { 2023-11-16T17:00:28.691 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard_description", 2023-11-16T17:00:28.691 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.691 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.691 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.691 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.692 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.692 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.692 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.692 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.692 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.692 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.692 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.693 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.693 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.693 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.693 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.693 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.693 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.693 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.694 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.694 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.694 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.694 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.694 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.694 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.694 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.695 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.695 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.695 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.695 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.695 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.695 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.695 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.696 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.696 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.696 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.696 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.696 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.696 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.697 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.697 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.697 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.697 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.697 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.697 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.697 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.698 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.698 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.698 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.698 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.698 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.698 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.699 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.699 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.699 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.699 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.699 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.699 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.699 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.700 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.700 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.700 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.700 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.700 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.700 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.700 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.701 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.701 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.701 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.701 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.701 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.701 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.701 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.702 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.702 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.702 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.702 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.702 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.702 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.703 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.703 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.703 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.703 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.703 INFO:teuthology.orchestra.run.smithi044.stdout: "organization": { 2023-11-16T17:00:28.703 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "organization", 2023-11-16T17:00:28.703 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.704 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.704 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.704 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.704 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.704 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.704 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.704 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.705 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.705 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.705 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.705 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.705 INFO:teuthology.orchestra.run.smithi044.stdout: "proxy": { 2023-11-16T17:00:28.705 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "proxy", 2023-11-16T17:00:28.705 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.706 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.706 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.706 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.706 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.706 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.706 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.706 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.707 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.707 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.707 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.707 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.707 INFO:teuthology.orchestra.run.smithi044.stdout: "url": { 2023-11-16T17:00:28.707 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url", 2023-11-16T17:00:28.707 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.708 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.708 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.708 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-11-16T17:00:28.708 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.708 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.708 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.708 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.709 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.709 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.709 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.709 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.709 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.709 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.710 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.710 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "test_orchestrator", 2023-11-16T17:00:28.710 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.710 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.710 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.710 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.710 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.711 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.711 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.711 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.711 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.711 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.711 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.711 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.712 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.712 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.712 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.712 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.712 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.712 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.712 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.712 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.713 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.713 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.713 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.713 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.713 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.713 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.713 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.714 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.714 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.714 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.714 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.714 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.714 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.714 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.715 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.715 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.715 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.715 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.715 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.715 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.715 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.716 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.716 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.716 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.716 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.716 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.716 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.717 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.717 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.717 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.717 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.717 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.717 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.717 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.717 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.718 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.718 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.718 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.718 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.718 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.718 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.719 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.719 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.719 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.719 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.719 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.719 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.719 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.720 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.720 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.720 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.720 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.720 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.720 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.720 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.721 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.721 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "volumes", 2023-11-16T17:00:28.721 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.721 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.721 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.721 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.721 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.722 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.722 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.722 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.722 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.722 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.722 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.723 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.723 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.723 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.723 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.723 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.723 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.723 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.724 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.724 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.724 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.724 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.724 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.724 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.724 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.724 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.725 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.725 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.725 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.725 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.725 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.725 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.725 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.726 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.726 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.726 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.726 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.726 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.726 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.726 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.727 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.727 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.727 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.727 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.727 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.727 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.727 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.728 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.728 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.728 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.728 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.728 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.728 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.728 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.728 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.729 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.729 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.729 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.729 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.729 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.729 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.730 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.730 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.730 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.730 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.730 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.730 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.730 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.731 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.731 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.731 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.731 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.731 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.731 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_clones": { 2023-11-16T17:00:28.732 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_clones", 2023-11-16T17:00:28.732 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.732 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.732 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.732 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4", 2023-11-16T17:00:28.732 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.732 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.733 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.733 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Number of asynchronous cloner threads", 2023-11-16T17:00:28.733 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.733 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.733 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.733 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.733 INFO:teuthology.orchestra.run.smithi044.stdout: "snapshot_clone_delay": { 2023-11-16T17:00:28.734 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snapshot_clone_delay", 2023-11-16T17:00:28.734 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.734 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.734 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.734 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2023-11-16T17:00:28.734 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.734 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.734 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.735 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-11-16T17:00:28.735 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.735 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.735 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.735 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.735 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.735 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.736 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.736 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix", 2023-11-16T17:00:28.736 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2023-11-16T17:00:28.736 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2023-11-16T17:00:28.736 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2023-11-16T17:00:28.736 INFO:teuthology.orchestra.run.smithi044.stdout: "discovery_interval": { 2023-11-16T17:00:28.736 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "discovery_interval", 2023-11-16T17:00:28.737 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2023-11-16T17:00:28.737 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.737 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.737 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "100", 2023-11-16T17:00:28.737 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.737 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.737 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.737 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.738 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.738 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.738 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.738 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.738 INFO:teuthology.orchestra.run.smithi044.stdout: "identifier": { 2023-11-16T17:00:28.738 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "identifier", 2023-11-16T17:00:28.739 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.739 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.739 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.739 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.739 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.739 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.739 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.740 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.740 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.740 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.740 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.740 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.740 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2023-11-16T17:00:28.740 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2023-11-16T17:00:28.741 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2023-11-16T17:00:28.741 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.741 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.741 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2023-11-16T17:00:28.741 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.741 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.741 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.742 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.742 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.742 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.742 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.742 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.742 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2023-11-16T17:00:28.742 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2023-11-16T17:00:28.742 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.743 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.743 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.743 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.743 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.743 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.743 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.743 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.744 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.744 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.744 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.744 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.744 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.744 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.744 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.744 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.745 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.745 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.745 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.745 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2023-11-16T17:00:28.745 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2023-11-16T17:00:28.745 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.745 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.745 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.746 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.746 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.746 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.746 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.746 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.746 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.747 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.747 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.747 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.747 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2023-11-16T17:00:28.747 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:28.747 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.747 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.748 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.748 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2023-11-16T17:00:28.748 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.748 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.748 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2023-11-16T17:00:28.748 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2023-11-16T17:00:28.748 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2023-11-16T17:00:28.748 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2023-11-16T17:00:28.749 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2023-11-16T17:00:28.749 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2023-11-16T17:00:28.749 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2023-11-16T17:00:28.749 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.749 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.749 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.750 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.750 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.750 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.750 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2023-11-16T17:00:28.750 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2023-11-16T17:00:28.750 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2023-11-16T17:00:28.750 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.751 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2023-11-16T17:00:28.751 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2023-11-16T17:00:28.751 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.751 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.751 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.751 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.751 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.752 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.752 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.752 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.752 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_host": { 2023-11-16T17:00:28.752 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_host", 2023-11-16T17:00:28.752 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.752 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.752 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.753 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2023-11-16T17:00:28.753 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.753 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.753 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.753 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.753 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.753 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.754 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.754 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.754 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_port": { 2023-11-16T17:00:28.754 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_port", 2023-11-16T17:00:28.754 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2023-11-16T17:00:28.754 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.755 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.755 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10051", 2023-11-16T17:00:28.755 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.755 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.755 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.755 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.756 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.756 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.756 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.756 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.756 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_sender": { 2023-11-16T17:00:28.756 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_sender", 2023-11-16T17:00:28.756 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2023-11-16T17:00:28.757 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2023-11-16T17:00:28.757 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2023-11-16T17:00:28.757 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-11-16T17:00:28.757 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2023-11-16T17:00:28.757 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2023-11-16T17:00:28.757 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2023-11-16T17:00:28.758 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2023-11-16T17:00:28.758 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2023-11-16T17:00:28.758 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2023-11-16T17:00:28.758 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2023-11-16T17:00:28.758 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.758 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.758 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.759 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.759 INFO:teuthology.orchestra.run.smithi044.stdout: "services": { 2023-11-16T17:00:28.759 INFO:teuthology.orchestra.run.smithi044.stdout: "dashboard": "https://172.21.15.44:8443/" 2023-11-16T17:00:28.759 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.759 INFO:teuthology.orchestra.run.smithi044.stdout: "always_on_modules": { 2023-11-16T17:00:28.759 INFO:teuthology.orchestra.run.smithi044.stdout: "octopus": [ 2023-11-16T17:00:28.760 INFO:teuthology.orchestra.run.smithi044.stdout: "balancer", 2023-11-16T17:00:28.760 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2023-11-16T17:00:28.760 INFO:teuthology.orchestra.run.smithi044.stdout: "devicehealth", 2023-11-16T17:00:28.760 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator", 2023-11-16T17:00:28.760 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_autoscaler", 2023-11-16T17:00:28.760 INFO:teuthology.orchestra.run.smithi044.stdout: "progress", 2023-11-16T17:00:28.760 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_support", 2023-11-16T17:00:28.760 INFO:teuthology.orchestra.run.smithi044.stdout: "status", 2023-11-16T17:00:28.761 INFO:teuthology.orchestra.run.smithi044.stdout: "telemetry", 2023-11-16T17:00:28.761 INFO:teuthology.orchestra.run.smithi044.stdout: "volumes" 2023-11-16T17:00:28.761 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.761 INFO:teuthology.orchestra.run.smithi044.stdout: "pacific": [ 2023-11-16T17:00:28.761 INFO:teuthology.orchestra.run.smithi044.stdout: "balancer", 2023-11-16T17:00:28.761 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2023-11-16T17:00:28.761 INFO:teuthology.orchestra.run.smithi044.stdout: "devicehealth", 2023-11-16T17:00:28.762 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator", 2023-11-16T17:00:28.762 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_autoscaler", 2023-11-16T17:00:28.762 INFO:teuthology.orchestra.run.smithi044.stdout: "progress", 2023-11-16T17:00:28.762 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_support", 2023-11-16T17:00:28.762 INFO:teuthology.orchestra.run.smithi044.stdout: "status", 2023-11-16T17:00:28.762 INFO:teuthology.orchestra.run.smithi044.stdout: "telemetry", 2023-11-16T17:00:28.763 INFO:teuthology.orchestra.run.smithi044.stdout: "volumes" 2023-11-16T17:00:28.763 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.763 INFO:teuthology.orchestra.run.smithi044.stdout: "quincy": [ 2023-11-16T17:00:28.763 INFO:teuthology.orchestra.run.smithi044.stdout: "balancer", 2023-11-16T17:00:28.763 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2023-11-16T17:00:28.763 INFO:teuthology.orchestra.run.smithi044.stdout: "devicehealth", 2023-11-16T17:00:28.763 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator", 2023-11-16T17:00:28.763 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_autoscaler", 2023-11-16T17:00:28.764 INFO:teuthology.orchestra.run.smithi044.stdout: "progress", 2023-11-16T17:00:28.764 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_support", 2023-11-16T17:00:28.764 INFO:teuthology.orchestra.run.smithi044.stdout: "status", 2023-11-16T17:00:28.764 INFO:teuthology.orchestra.run.smithi044.stdout: "telemetry", 2023-11-16T17:00:28.764 INFO:teuthology.orchestra.run.smithi044.stdout: "volumes" 2023-11-16T17:00:28.764 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2023-11-16T17:00:28.764 INFO:teuthology.orchestra.run.smithi044.stdout: "reef": [ 2023-11-16T17:00:28.765 INFO:teuthology.orchestra.run.smithi044.stdout: "balancer", 2023-11-16T17:00:28.765 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2023-11-16T17:00:28.765 INFO:teuthology.orchestra.run.smithi044.stdout: "devicehealth", 2023-11-16T17:00:28.765 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator", 2023-11-16T17:00:28.765 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_autoscaler", 2023-11-16T17:00:28.765 INFO:teuthology.orchestra.run.smithi044.stdout: "progress", 2023-11-16T17:00:28.765 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_support", 2023-11-16T17:00:28.766 INFO:teuthology.orchestra.run.smithi044.stdout: "status", 2023-11-16T17:00:28.766 INFO:teuthology.orchestra.run.smithi044.stdout: "telemetry", 2023-11-16T17:00:28.766 INFO:teuthology.orchestra.run.smithi044.stdout: "volumes" 2023-11-16T17:00:28.766 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2023-11-16T17:00:28.766 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.766 INFO:teuthology.orchestra.run.smithi044.stdout: "last_failure_osd_epoch": 92, 2023-11-16T17:00:28.766 INFO:teuthology.orchestra.run.smithi044.stdout: "active_clients": [ 2023-11-16T17:00:28.767 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.767 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "devicehealth", 2023-11-16T17:00:28.767 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2023-11-16T17:00:28.767 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.767 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2023-11-16T17:00:28.767 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2023-11-16T17:00:28.767 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 348100948 2023-11-16T17:00:28.768 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.768 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2023-11-16T17:00:28.768 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.768 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.768 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "libcephsqlite", 2023-11-16T17:00:28.768 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2023-11-16T17:00:28.768 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.769 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2023-11-16T17:00:28.769 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2023-11-16T17:00:28.769 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2873353693 2023-11-16T17:00:28.769 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.769 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2023-11-16T17:00:28.769 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.769 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.770 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_support", 2023-11-16T17:00:28.770 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2023-11-16T17:00:28.770 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.770 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2023-11-16T17:00:28.770 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2023-11-16T17:00:28.770 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2670487113 2023-11-16T17:00:28.770 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.770 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2023-11-16T17:00:28.771 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.771 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.771 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snap_schedule", 2023-11-16T17:00:28.771 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2023-11-16T17:00:28.771 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.771 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2023-11-16T17:00:28.772 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2023-11-16T17:00:28.772 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2316259355 2023-11-16T17:00:28.772 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.772 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2023-11-16T17:00:28.772 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.772 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.772 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snap_schedule", 2023-11-16T17:00:28.773 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2023-11-16T17:00:28.773 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.773 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2023-11-16T17:00:28.773 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2023-11-16T17:00:28.773 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 1281978030 2023-11-16T17:00:28.773 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.773 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2023-11-16T17:00:28.774 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:28.774 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.774 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "volumes", 2023-11-16T17:00:28.774 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2023-11-16T17:00:28.774 INFO:teuthology.orchestra.run.smithi044.stdout: { 2023-11-16T17:00:28.774 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2023-11-16T17:00:28.774 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2023-11-16T17:00:28.774 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2133227912 2023-11-16T17:00:28.775 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.775 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2023-11-16T17:00:28.775 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:28.775 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2023-11-16T17:00:28.775 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T17:00:28.778 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:20 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/2750261815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T17:00:28.778 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:21 smithi077 ceph-mon[131204]: pgmap v324: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2023-11-16T17:00:28.778 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:21 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1933594234' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-11-16T17:00:28.778 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:23 smithi077 ceph-mon[131204]: pgmap v325: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2023-11-16T17:00:28.778 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:25 smithi077 ceph-mon[131204]: pgmap v326: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2023-11-16T17:00:28.778 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:27 smithi077 ceph-mon[131204]: pgmap v327: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2023-11-16T17:00:28.779 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:20 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/2750261815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T17:00:28.779 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:21 smithi176 ceph-mon[131630]: pgmap v324: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2023-11-16T17:00:28.779 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:21 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1933594234' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-11-16T17:00:28.779 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:23 smithi176 ceph-mon[131630]: pgmap v325: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2023-11-16T17:00:28.779 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:25 smithi176 ceph-mon[131630]: pgmap v326: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2023-11-16T17:00:28.779 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:27 smithi176 ceph-mon[131630]: pgmap v327: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2023-11-16T17:00:28.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.793+0000 7f3ef8ff9700 1 -- 172.21.15.44:0/1933594234 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3ee80a1340 msgr2=0x7f3ee80a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:28.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.793+0000 7f3ef8ff9700 1 --2- 172.21.15.44:0/1933594234 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3ee80a1340 0x7f3ee80a37f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4009f40 tx=0x7f3ee40094d0 comp rx=0 tx=0).stop 2023-11-16T17:00:28.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.793+0000 7f3ef8ff9700 1 -- 172.21.15.44:0/1933594234 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3efc100eb0 msgr2=0x7f3efc10f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:28.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.793+0000 7f3ef8ff9700 1 --2- 172.21.15.44:0/1933594234 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3efc100eb0 0x7f3efc10f780 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3eec0098e0 tx=0x7f3eec00bfb0 comp rx=0 tx=0).stop 2023-11-16T17:00:28.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.794+0000 7f3ef8ff9700 1 -- 172.21.15.44:0/1933594234 shutdown_connections 2023-11-16T17:00:28.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.794+0000 7f3ef8ff9700 1 --2- 172.21.15.44:0/1933594234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3efc105550 0x7f3efc10fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:28.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.794+0000 7f3ef8ff9700 1 --2- 172.21.15.44:0/1933594234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3efc100510 0x7f3efc113400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:28.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.794+0000 7f3ef8ff9700 1 --2- 172.21.15.44:0/1933594234 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3ee80a1340 0x7f3ee80a37f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:28.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.794+0000 7f3ef8ff9700 1 --2- 172.21.15.44:0/1933594234 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3efc100eb0 0x7f3efc10f780 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:28.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.794+0000 7f3ef8ff9700 1 -- 172.21.15.44:0/1933594234 >> 172.21.15.44:0/1933594234 conn(0x7f3efc077050 msgr2=0x7f3efc103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:28.781 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.794+0000 7f3ef8ff9700 1 -- 172.21.15.44:0/1933594234 shutdown_connections 2023-11-16T17:00:28.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:00:20.794+0000 7f3ef8ff9700 1 -- 172.21.15.44:0/1933594234 wait complete. 2023-11-16T17:00:28.788 DEBUG:tasks.check_counter:Getting stats from a 2023-11-16T17:00:28.789 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.a perf dump 2023-11-16T17:00:28.790 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:20 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/2750261815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T17:00:28.790 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:21 smithi044 ceph-mon[128750]: pgmap v324: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2023-11-16T17:00:28.790 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:21 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1933594234' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-11-16T17:00:28.790 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:23 smithi044 ceph-mon[128750]: pgmap v325: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2023-11-16T17:00:28.790 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:25 smithi044 ceph-mon[128750]: pgmap v326: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2023-11-16T17:00:28.791 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:27 smithi044 ceph-mon[128750]: pgmap v327: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2023-11-16T17:00:29.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:29 smithi077 ceph-mon[131204]: pgmap v328: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2023-11-16T17:00:29.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:29 smithi044 ceph-mon[128750]: pgmap v328: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2023-11-16T17:00:29.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:29 smithi176 ceph-mon[131630]: pgmap v328: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2023-11-16T17:00:30.267 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T17:00:31.726 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:31 smithi077 ceph-mon[131204]: pgmap v329: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:00:31.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:31 smithi044 ceph-mon[128750]: pgmap v329: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:00:31.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:31 smithi176 ceph-mon[131630]: pgmap v329: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:00:33.730 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:33 smithi077 ceph-mon[131204]: pgmap v330: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:00:33.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:33 smithi044 ceph-mon[128750]: pgmap v330: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:00:33.887 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T17:00:33.887 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:00:33.887 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 230, 2023-11-16T17:00:33.888 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 218, 2023-11-16T17:00:33.888 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 1425760, 2023-11-16T17:00:33.888 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 123328, 2023-11-16T17:00:33.888 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 4, 2023-11-16T17:00:33.888 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 0, 2023-11-16T17:00:33.889 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.056226615, 2023-11-16T17:00:33.889 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.021274736, 2023-11-16T17:00:33.889 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.033902260, 2023-11-16T17:00:33.889 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.005236555, 2023-11-16T17:00:33.889 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:33.890 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 218, 2023-11-16T17:00:33.890 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.023585433, 2023-11-16T17:00:33.890 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000108190 2023-11-16T17:00:33.890 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:33.890 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:33.890 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:33.891 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:33.891 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:33.891 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:33.891 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 1425760, 2023-11-16T17:00:33.891 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 123328 2023-11-16T17:00:33.892 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:33.892 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:00:33.892 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 2, 2023-11-16T17:00:33.892 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 165, 2023-11-16T17:00:33.892 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 320, 2023-11-16T17:00:33.893 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 180640, 2023-11-16T17:00:33.893 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 5, 2023-11-16T17:00:33.893 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 3, 2023-11-16T17:00:33.893 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.020913402, 2023-11-16T17:00:33.893 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.014531302, 2023-11-16T17:00:33.893 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.002832447, 2023-11-16T17:00:33.894 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:33.894 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:33.894 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 162, 2023-11-16T17:00:33.894 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.023680187, 2023-11-16T17:00:33.894 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000146173 2023-11-16T17:00:33.895 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:33.895 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:33.895 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:33.895 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:33.895 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:33.895 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:33.896 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T17:00:33.896 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 180640 2023-11-16T17:00:33.896 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:33.896 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:00:33.896 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 0, 2023-11-16T17:00:33.897 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 2, 2023-11-16T17:00:33.897 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 0, 2023-11-16T17:00:33.897 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 0, 2023-11-16T17:00:33.897 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 2, 2023-11-16T17:00:33.897 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 1, 2023-11-16T17:00:33.897 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.000849758, 2023-11-16T17:00:33.898 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.000000000, 2023-11-16T17:00:33.898 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.000739236, 2023-11-16T17:00:33.898 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:33.898 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:33.898 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:33.899 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:33.899 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:33.899 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:33.899 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:33.899 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:33.899 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:33.900 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:33.900 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:33.900 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T17:00:33.900 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T17:00:33.900 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:33.901 INFO:teuthology.orchestra.run.smithi044.stdout: "cct": { 2023-11-16T17:00:33.901 INFO:teuthology.orchestra.run.smithi044.stdout: "total_workers": 0, 2023-11-16T17:00:33.901 INFO:teuthology.orchestra.run.smithi044.stdout: "unhealthy_workers": 0 2023-11-16T17:00:33.901 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:33.901 INFO:teuthology.orchestra.run.smithi044.stdout: "mempool": { 2023-11-16T17:00:33.901 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:00:33.902 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_items": 0, 2023-11-16T17:00:33.902 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:00:33.902 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:00:33.902 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:00:33.902 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:00:33.902 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:00:33.903 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:00:33.903 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:00:33.903 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:00:33.903 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:00:33.903 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:00:33.903 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:00:33.904 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:00:33.904 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:00:33.904 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_items": 0, 2023-11-16T17:00:33.904 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:00:33.904 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_items": 0, 2023-11-16T17:00:33.905 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:00:33.905 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:00:33.905 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:00:33.905 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:00:33.905 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:00:33.905 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:00:33.906 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:00:33.906 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_items": 0, 2023-11-16T17:00:33.906 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:00:33.906 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:00:33.906 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:00:33.907 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_items": 0, 2023-11-16T17:00:33.907 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_bytes": 0, 2023-11-16T17:00:33.907 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_items": 0, 2023-11-16T17:00:33.907 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:00:33.907 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:00:33.907 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:00:33.908 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:00:33.908 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_bytes": 148977, 2023-11-16T17:00:33.908 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_items": 30, 2023-11-16T17:00:33.908 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:00:33.908 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_items": 0, 2023-11-16T17:00:33.908 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_bytes": 0, 2023-11-16T17:00:33.909 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_items": 0, 2023-11-16T17:00:33.909 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:00:33.909 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_items": 0, 2023-11-16T17:00:33.909 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:00:33.909 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_items": 0, 2023-11-16T17:00:33.910 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_bytes": 0, 2023-11-16T17:00:33.910 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_items": 0, 2023-11-16T17:00:33.910 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:00:33.910 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:00:33.910 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_bytes": 0, 2023-11-16T17:00:33.910 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_items": 0, 2023-11-16T17:00:33.911 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_bytes": 928, 2023-11-16T17:00:33.911 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_items": 2, 2023-11-16T17:00:33.911 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_bytes": 0, 2023-11-16T17:00:33.911 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_items": 0, 2023-11-16T17:00:33.911 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_bytes": 0, 2023-11-16T17:00:33.912 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_items": 0 2023-11-16T17:00:33.912 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:33.912 INFO:teuthology.orchestra.run.smithi044.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:00:33.912 INFO:teuthology.orchestra.run.smithi044.stdout: "val": 0, 2023-11-16T17:00:33.912 INFO:teuthology.orchestra.run.smithi044.stdout: "max": 104857600, 2023-11-16T17:00:33.912 INFO:teuthology.orchestra.run.smithi044.stdout: "get_started": 0, 2023-11-16T17:00:33.913 INFO:teuthology.orchestra.run.smithi044.stdout: "get": 228, 2023-11-16T17:00:33.913 INFO:teuthology.orchestra.run.smithi044.stdout: "get_sum": 1390918, 2023-11-16T17:00:33.913 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:33.913 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_success": 228, 2023-11-16T17:00:33.913 INFO:teuthology.orchestra.run.smithi044.stdout: "take": 0, 2023-11-16T17:00:33.913 INFO:teuthology.orchestra.run.smithi044.stdout: "take_sum": 0, 2023-11-16T17:00:33.914 INFO:teuthology.orchestra.run.smithi044.stdout: "put": 228, 2023-11-16T17:00:33.914 INFO:teuthology.orchestra.run.smithi044.stdout: "put_sum": 1390918, 2023-11-16T17:00:33.914 INFO:teuthology.orchestra.run.smithi044.stdout: "wait": { 2023-11-16T17:00:33.914 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:33.914 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:33.914 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:33.915 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:33.915 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:33.915 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T17:00:33.915 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:33 smithi176 ceph-mon[131630]: pgmap v330: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:00:34.454 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2023-11-16T17:00:34.454 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2023-11-16T17:00:34.455 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2023-11-16T17:00:34.455 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2023-11-16T17:00:34.455 DEBUG:tasks.check_counter:Getting stats from d 2023-11-16T17:00:34.455 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.d perf dump 2023-11-16T17:00:34.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:34 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T17:00:34.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:34 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T17:00:34.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:34 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T17:00:35.853 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T17:00:35.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:35 smithi044 ceph-mon[128750]: pgmap v331: 129 pgs: 23 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 37 active+clean; 6.1 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:00:35.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:35 smithi176 ceph-mon[131630]: pgmap v331: 129 pgs: 23 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 37 active+clean; 6.1 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:00:35.983 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:35 smithi077 ceph-mon[131204]: pgmap v331: 129 pgs: 23 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 37 active+clean; 6.1 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:00:37.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:37 smithi044 ceph-mon[128750]: pgmap v332: 129 pgs: 23 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 37 active+clean; 6.1 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:00:37.900 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:37 smithi176 ceph-mon[131630]: pgmap v332: 129 pgs: 23 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 37 active+clean; 6.1 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:00:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:37 smithi077 ceph-mon[131204]: pgmap v332: 129 pgs: 23 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 37 active+clean; 6.1 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:00:38.473 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T17:00:38.473 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:00:38.473 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 232, 2023-11-16T17:00:38.473 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 222, 2023-11-16T17:00:38.474 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 1424448, 2023-11-16T17:00:38.474 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 126432, 2023-11-16T17:00:38.474 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 4, 2023-11-16T17:00:38.474 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 0, 2023-11-16T17:00:38.474 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.057216407, 2023-11-16T17:00:38.475 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.019901847, 2023-11-16T17:00:38.475 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.038732355, 2023-11-16T17:00:38.475 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.005737265, 2023-11-16T17:00:38.475 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:38.475 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 221, 2023-11-16T17:00:38.476 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.021701201, 2023-11-16T17:00:38.476 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000098195 2023-11-16T17:00:38.476 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:38.476 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:38.476 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:38.476 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:38.477 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:38.477 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:38.477 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 1424448, 2023-11-16T17:00:38.477 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 126432 2023-11-16T17:00:38.477 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:38.478 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:00:38.478 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 2, 2023-11-16T17:00:38.478 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 169, 2023-11-16T17:00:38.478 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 320, 2023-11-16T17:00:38.478 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 181344, 2023-11-16T17:00:38.479 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 5, 2023-11-16T17:00:38.479 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 3, 2023-11-16T17:00:38.479 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.024652900, 2023-11-16T17:00:38.479 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.018096254, 2023-11-16T17:00:38.479 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.003128504, 2023-11-16T17:00:38.479 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:38.480 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:38.480 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 166, 2023-11-16T17:00:38.480 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.033972351, 2023-11-16T17:00:38.480 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000204652 2023-11-16T17:00:38.480 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:38.481 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:38.481 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:38.481 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:38.481 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:38.481 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:38.481 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T17:00:38.482 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 181344 2023-11-16T17:00:38.482 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:38.482 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:00:38.482 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 4, 2023-11-16T17:00:38.482 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 4, 2023-11-16T17:00:38.483 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 2944, 2023-11-16T17:00:38.483 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 448, 2023-11-16T17:00:38.483 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 2, 2023-11-16T17:00:38.483 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 1, 2023-11-16T17:00:38.483 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.001529621, 2023-11-16T17:00:38.483 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.000073842, 2023-11-16T17:00:38.484 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.001391397, 2023-11-16T17:00:38.484 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:38.484 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:38.484 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 3, 2023-11-16T17:00:38.485 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.001501017, 2023-11-16T17:00:38.485 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000500339 2023-11-16T17:00:38.485 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:38.485 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:38.485 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:38.485 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:38.486 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:38.486 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:38.486 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 2944, 2023-11-16T17:00:38.486 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 448 2023-11-16T17:00:38.486 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:38.487 INFO:teuthology.orchestra.run.smithi044.stdout: "cct": { 2023-11-16T17:00:38.487 INFO:teuthology.orchestra.run.smithi044.stdout: "total_workers": 0, 2023-11-16T17:00:38.487 INFO:teuthology.orchestra.run.smithi044.stdout: "unhealthy_workers": 0 2023-11-16T17:00:38.487 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:38.487 INFO:teuthology.orchestra.run.smithi044.stdout: "mempool": { 2023-11-16T17:00:38.487 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:00:38.488 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_items": 0, 2023-11-16T17:00:38.488 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:00:38.488 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:00:38.488 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:00:38.488 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:00:38.488 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:00:38.489 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:00:38.489 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:00:38.489 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:00:38.489 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:00:38.489 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:00:38.490 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:00:38.490 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:00:38.490 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:00:38.490 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_items": 0, 2023-11-16T17:00:38.490 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:00:38.490 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_items": 0, 2023-11-16T17:00:38.491 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:00:38.491 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:00:38.491 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:00:38.491 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:00:38.491 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:00:38.492 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:00:38.492 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:00:38.492 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_items": 0, 2023-11-16T17:00:38.492 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:00:38.492 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:00:38.492 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:00:38.493 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_items": 0, 2023-11-16T17:00:38.493 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_bytes": 0, 2023-11-16T17:00:38.493 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_items": 0, 2023-11-16T17:00:38.493 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:00:38.493 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:00:38.493 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:00:38.494 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:00:38.494 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_bytes": 148714, 2023-11-16T17:00:38.494 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_items": 30, 2023-11-16T17:00:38.494 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:00:38.494 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_items": 0, 2023-11-16T17:00:38.494 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_bytes": 0, 2023-11-16T17:00:38.495 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_items": 0, 2023-11-16T17:00:38.495 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:00:38.495 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_items": 0, 2023-11-16T17:00:38.495 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:00:38.495 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_items": 0, 2023-11-16T17:00:38.496 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_bytes": 0, 2023-11-16T17:00:38.496 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_items": 0, 2023-11-16T17:00:38.496 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:00:38.496 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:00:38.496 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_bytes": 0, 2023-11-16T17:00:38.496 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_items": 0, 2023-11-16T17:00:38.496 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_bytes": 928, 2023-11-16T17:00:38.497 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_items": 2, 2023-11-16T17:00:38.497 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_bytes": 0, 2023-11-16T17:00:38.497 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_items": 0, 2023-11-16T17:00:38.497 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_bytes": 0, 2023-11-16T17:00:38.497 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_items": 0 2023-11-16T17:00:38.497 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:38.498 INFO:teuthology.orchestra.run.smithi044.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:00:38.498 INFO:teuthology.orchestra.run.smithi044.stdout: "val": 0, 2023-11-16T17:00:38.498 INFO:teuthology.orchestra.run.smithi044.stdout: "max": 104857600, 2023-11-16T17:00:38.498 INFO:teuthology.orchestra.run.smithi044.stdout: "get_started": 0, 2023-11-16T17:00:38.498 INFO:teuthology.orchestra.run.smithi044.stdout: "get": 234, 2023-11-16T17:00:38.499 INFO:teuthology.orchestra.run.smithi044.stdout: "get_sum": 1391698, 2023-11-16T17:00:38.499 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:38.499 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_success": 234, 2023-11-16T17:00:38.499 INFO:teuthology.orchestra.run.smithi044.stdout: "take": 0, 2023-11-16T17:00:38.499 INFO:teuthology.orchestra.run.smithi044.stdout: "take_sum": 0, 2023-11-16T17:00:38.499 INFO:teuthology.orchestra.run.smithi044.stdout: "put": 234, 2023-11-16T17:00:38.500 INFO:teuthology.orchestra.run.smithi044.stdout: "put_sum": 1391698, 2023-11-16T17:00:38.500 INFO:teuthology.orchestra.run.smithi044.stdout: "wait": { 2023-11-16T17:00:38.500 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:38.500 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:38.500 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:38.500 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:38.501 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:38.501 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T17:00:38.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:38 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:38.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:38 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:38.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:38 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:38.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:38 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:38.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:38 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:38.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:38 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:38.978 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2023-11-16T17:00:38.978 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2023-11-16T17:00:38.978 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2023-11-16T17:00:38.978 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2023-11-16T17:00:38.979 DEBUG:tasks.check_counter:Getting stats from g 2023-11-16T17:00:38.979 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.g perf dump 2023-11-16T17:00:39.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:39 smithi044 ceph-mon[128750]: pgmap v333: 129 pgs: 22 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 38 active+clean; 6.1 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:00:39.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:39 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:39.872 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:39 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.872 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:39 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:39.872 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:39 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.872 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:39 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:39 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:39 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi176 to 196.2M 2023-11-16T17:00:39.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:39 smithi044 ceph-mon[128750]: Unable to set osd_memory_target on smithi176 to 205772492: error parsing value: Value '205772492' is below minimum 939524096 2023-11-16T17:00:39.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:39 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:39 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.873 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:39 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:39 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.896 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:39 smithi176 ceph-mon[131630]: pgmap v333: 129 pgs: 22 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 38 active+clean; 6.1 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:00:39.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:39 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:39.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:39 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:39 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:39.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:39 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:39 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:39 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:39 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi176 to 196.2M 2023-11-16T17:00:39.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:39 smithi176 ceph-mon[131630]: Unable to set osd_memory_target on smithi176 to 205772492: error parsing value: Value '205772492' is below minimum 939524096 2023-11-16T17:00:39.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:39 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:39 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:39 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:39 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:39 smithi077 ceph-mon[131204]: pgmap v333: 129 pgs: 22 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 38 active+clean; 6.1 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:00:39.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:39 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:39 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:39 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:39 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:39 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:39 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:39 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi176 to 196.2M 2023-11-16T17:00:39.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:39 smithi077 ceph-mon[131204]: Unable to set osd_memory_target on smithi176 to 205772492: error parsing value: Value '205772492' is below minimum 939524096 2023-11-16T17:00:39.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:39 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:39 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:39 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:39.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:39 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:40.328 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T17:00:40.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:40 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi077 to 196.2M 2023-11-16T17:00:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:40 smithi077 ceph-mon[131204]: Unable to set osd_memory_target on smithi077 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T17:00:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:40 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:40.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:40 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:41.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:40 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi077 to 196.2M 2023-11-16T17:00:41.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:40 smithi044 ceph-mon[128750]: Unable to set osd_memory_target on smithi077 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T17:00:41.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:40 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:41.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:40 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:41.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:40 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi077 to 196.2M 2023-11-16T17:00:41.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:40 smithi176 ceph-mon[131630]: Unable to set osd_memory_target on smithi077 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T17:00:41.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:40 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:41.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:40 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:41.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:41 smithi077 ceph-mon[131204]: pgmap v334: 129 pgs: 22 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 38 active+clean; 6.1 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:00:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:41 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:41 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:41 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:41 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:41 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T17:00:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:41 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T17:00:41.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:41 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:42.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:41 smithi044 ceph-mon[128750]: pgmap v334: 129 pgs: 22 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 38 active+clean; 6.1 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:00:42.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:41 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:42.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:41 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:42.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:41 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:42.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:41 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:42.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:41 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T17:00:42.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:41 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T17:00:42.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:41 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:42.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:41 smithi176 ceph-mon[131630]: pgmap v334: 129 pgs: 22 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 38 active+clean; 6.1 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:00:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:41 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:41 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:41 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:41 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:41 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T17:00:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:41 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T17:00:42.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:41 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:00:42.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:42 smithi077 ceph-mon[131204]: Adjusting osd_memory_target on smithi044 to 223.0M 2023-11-16T17:00:42.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:42 smithi077 ceph-mon[131204]: Unable to set osd_memory_target on smithi044 to 233877504: error parsing value: Value '233877504' is below minimum 939524096 2023-11-16T17:00:42.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:42 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T17:00:42.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:42 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T17:00:42.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:42 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T17:00:42.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:42 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T17:00:42.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:42 smithi044 ceph-mon[128750]: Adjusting osd_memory_target on smithi044 to 223.0M 2023-11-16T17:00:42.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:42 smithi044 ceph-mon[128750]: Unable to set osd_memory_target on smithi044 to 233877504: error parsing value: Value '233877504' is below minimum 939524096 2023-11-16T17:00:42.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:42 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T17:00:43.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:42 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T17:00:43.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:42 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T17:00:43.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:42 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T17:00:43.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:42 smithi176 ceph-mon[131630]: Adjusting osd_memory_target on smithi044 to 223.0M 2023-11-16T17:00:43.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:42 smithi176 ceph-mon[131630]: Unable to set osd_memory_target on smithi044 to 233877504: error parsing value: Value '233877504' is below minimum 939524096 2023-11-16T17:00:43.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:42 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T17:00:43.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:42 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T17:00:43.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:42 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T17:00:43.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:42 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T17:00:43.361 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T17:00:43.362 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:00:43.362 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 230, 2023-11-16T17:00:43.362 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 220, 2023-11-16T17:00:43.362 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 1423904, 2023-11-16T17:00:43.363 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 125248, 2023-11-16T17:00:43.363 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 4, 2023-11-16T17:00:43.363 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 0, 2023-11-16T17:00:43.363 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.057311257, 2023-11-16T17:00:43.363 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.022559940, 2023-11-16T17:00:43.363 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.034314504, 2023-11-16T17:00:43.363 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.005315362, 2023-11-16T17:00:43.364 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:43.364 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 219, 2023-11-16T17:00:43.364 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.021617446, 2023-11-16T17:00:43.364 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000098709 2023-11-16T17:00:43.364 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:43.364 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:43.365 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:43.365 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:43.365 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:43.365 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:43.365 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 1423904, 2023-11-16T17:00:43.365 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 125248 2023-11-16T17:00:43.365 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:43.366 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:00:43.366 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 2, 2023-11-16T17:00:43.366 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 167, 2023-11-16T17:00:43.366 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 320, 2023-11-16T17:00:43.366 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 180992, 2023-11-16T17:00:43.366 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 5, 2023-11-16T17:00:43.367 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 3, 2023-11-16T17:00:43.367 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.023156452, 2023-11-16T17:00:43.367 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.014905482, 2023-11-16T17:00:43.367 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.002384974, 2023-11-16T17:00:43.367 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:43.367 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:43.367 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 164, 2023-11-16T17:00:43.368 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.027122318, 2023-11-16T17:00:43.368 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000165379 2023-11-16T17:00:43.368 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:43.368 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:43.368 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:43.368 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:43.368 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:43.369 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:43.369 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T17:00:43.369 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 180992 2023-11-16T17:00:43.369 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:43.369 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:00:43.369 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 4, 2023-11-16T17:00:43.370 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 4, 2023-11-16T17:00:43.370 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 2944, 2023-11-16T17:00:43.370 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 448, 2023-11-16T17:00:43.370 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 2, 2023-11-16T17:00:43.370 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 1, 2023-11-16T17:00:43.370 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.000916338, 2023-11-16T17:00:43.371 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.000045890, 2023-11-16T17:00:43.371 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.000820480, 2023-11-16T17:00:43.371 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:43.371 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:43.371 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 3, 2023-11-16T17:00:43.371 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.001188924, 2023-11-16T17:00:43.371 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000396308 2023-11-16T17:00:43.372 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:43.372 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:43.372 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:43.372 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:43.372 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:43.372 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:43.372 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 2944, 2023-11-16T17:00:43.373 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 448 2023-11-16T17:00:43.373 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:43.373 INFO:teuthology.orchestra.run.smithi044.stdout: "cct": { 2023-11-16T17:00:43.373 INFO:teuthology.orchestra.run.smithi044.stdout: "total_workers": 0, 2023-11-16T17:00:43.373 INFO:teuthology.orchestra.run.smithi044.stdout: "unhealthy_workers": 0 2023-11-16T17:00:43.373 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:43.373 INFO:teuthology.orchestra.run.smithi044.stdout: "mempool": { 2023-11-16T17:00:43.374 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:00:43.374 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_items": 0, 2023-11-16T17:00:43.374 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:00:43.374 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:00:43.374 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:00:43.374 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:00:43.375 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:00:43.375 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:00:43.375 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:00:43.375 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:00:43.375 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:00:43.375 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:00:43.375 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:00:43.376 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:00:43.376 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:00:43.376 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_items": 0, 2023-11-16T17:00:43.376 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:00:43.376 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_items": 0, 2023-11-16T17:00:43.376 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:00:43.376 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:00:43.377 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:00:43.377 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:00:43.377 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:00:43.377 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:00:43.377 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:00:43.377 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_items": 0, 2023-11-16T17:00:43.377 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:00:43.377 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:00:43.378 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:00:43.378 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_items": 0, 2023-11-16T17:00:43.378 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_bytes": 0, 2023-11-16T17:00:43.378 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_items": 0, 2023-11-16T17:00:43.378 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:00:43.378 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:00:43.378 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:00:43.379 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:00:43.379 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_bytes": 148977, 2023-11-16T17:00:43.379 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_items": 30, 2023-11-16T17:00:43.379 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:00:43.379 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_items": 0, 2023-11-16T17:00:43.379 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_bytes": 0, 2023-11-16T17:00:43.379 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_items": 0, 2023-11-16T17:00:43.380 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:00:43.380 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_items": 0, 2023-11-16T17:00:43.380 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:00:43.380 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_items": 0, 2023-11-16T17:00:43.380 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_bytes": 0, 2023-11-16T17:00:43.380 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_items": 0, 2023-11-16T17:00:43.380 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:00:43.381 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:00:43.381 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_bytes": 0, 2023-11-16T17:00:43.381 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_items": 0, 2023-11-16T17:00:43.381 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_bytes": 928, 2023-11-16T17:00:43.381 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_items": 2, 2023-11-16T17:00:43.381 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_bytes": 0, 2023-11-16T17:00:43.381 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_items": 0, 2023-11-16T17:00:43.382 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_bytes": 0, 2023-11-16T17:00:43.382 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_items": 0 2023-11-16T17:00:43.382 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:43.382 INFO:teuthology.orchestra.run.smithi044.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:00:43.382 INFO:teuthology.orchestra.run.smithi044.stdout: "val": 0, 2023-11-16T17:00:43.382 INFO:teuthology.orchestra.run.smithi044.stdout: "max": 104857600, 2023-11-16T17:00:43.382 INFO:teuthology.orchestra.run.smithi044.stdout: "get_started": 0, 2023-11-16T17:00:43.383 INFO:teuthology.orchestra.run.smithi044.stdout: "get": 232, 2023-11-16T17:00:43.383 INFO:teuthology.orchestra.run.smithi044.stdout: "get_sum": 1391438, 2023-11-16T17:00:43.383 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:43.383 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_success": 232, 2023-11-16T17:00:43.383 INFO:teuthology.orchestra.run.smithi044.stdout: "take": 0, 2023-11-16T17:00:43.383 INFO:teuthology.orchestra.run.smithi044.stdout: "take_sum": 0, 2023-11-16T17:00:43.383 INFO:teuthology.orchestra.run.smithi044.stdout: "put": 232, 2023-11-16T17:00:43.384 INFO:teuthology.orchestra.run.smithi044.stdout: "put_sum": 1391438, 2023-11-16T17:00:43.384 INFO:teuthology.orchestra.run.smithi044.stdout: "wait": { 2023-11-16T17:00:43.384 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:43.384 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:43.384 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:43.384 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:43.384 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:43.384 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T17:00:43.907 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2023-11-16T17:00:43.908 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2023-11-16T17:00:43.908 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2023-11-16T17:00:43.908 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2023-11-16T17:00:43.908 DEBUG:tasks.check_counter:Getting stats from j 2023-11-16T17:00:43.909 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.j perf dump 2023-11-16T17:00:43.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:43 smithi077 ceph-mon[131204]: pgmap v335: 129 pgs: 22 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 38 active+clean; 6.1 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2023-11-16T17:00:44.040 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:43 smithi044 ceph-mon[128750]: pgmap v335: 129 pgs: 22 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 38 active+clean; 6.1 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2023-11-16T17:00:44.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:43 smithi176 ceph-mon[131630]: pgmap v335: 129 pgs: 22 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 38 active+clean; 6.1 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2023-11-16T17:00:45.216 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T17:00:45.872 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:45 smithi044 ceph-mon[128750]: pgmap v336: 129 pgs: 22 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 38 active+clean; 6.1 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:00:45.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:45 smithi077 ceph-mon[131204]: pgmap v336: 129 pgs: 22 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 38 active+clean; 6.1 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:00:46.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:45 smithi176 ceph-mon[131630]: pgmap v336: 129 pgs: 22 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 38 active+clean; 6.1 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:00:48.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:47 smithi044 ceph-mon[128750]: pgmap v337: 129 pgs: 22 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 38 active+clean; 6.1 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2023-11-16T17:00:48.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:47 smithi176 ceph-mon[131630]: pgmap v337: 129 pgs: 22 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 38 active+clean; 6.1 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2023-11-16T17:00:48.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:47 smithi077 ceph-mon[131204]: pgmap v337: 129 pgs: 22 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 38 active+clean; 6.1 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2023-11-16T17:00:50.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:49 smithi044 ceph-mon[128750]: pgmap v338: 129 pgs: 19 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 53 active+clean; 6.1 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:00:50.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:49 smithi176 ceph-mon[131630]: pgmap v338: 129 pgs: 19 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 53 active+clean; 6.1 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:00:50.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:49 smithi077 ceph-mon[131204]: pgmap v338: 129 pgs: 19 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 53 active+clean; 6.1 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:00:52.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:51 smithi044 ceph-mon[128750]: pgmap v339: 129 pgs: 19 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 53 active+clean; 6.1 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2023-11-16T17:00:52.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:51 smithi176 ceph-mon[131630]: pgmap v339: 129 pgs: 19 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 53 active+clean; 6.1 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2023-11-16T17:00:52.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:51 smithi077 ceph-mon[131204]: pgmap v339: 129 pgs: 19 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 53 active+clean; 6.1 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2023-11-16T17:00:54.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:53 smithi044 ceph-mon[128750]: pgmap v340: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 6.1 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:00:54.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:53 smithi176 ceph-mon[131630]: pgmap v340: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 6.1 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:00:54.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:53 smithi077 ceph-mon[131204]: pgmap v340: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 6.1 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:00:54.678 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2023-11-16T17:00:54.678 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:00:54.679 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 237, 2023-11-16T17:00:54.679 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 227, 2023-11-16T17:00:54.679 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 1425808, 2023-11-16T17:00:54.679 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 129392, 2023-11-16T17:00:54.679 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 4, 2023-11-16T17:00:54.680 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 0, 2023-11-16T17:00:54.680 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.056566328, 2023-11-16T17:00:54.680 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.020025712, 2023-11-16T17:00:54.680 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.037867643, 2023-11-16T17:00:54.680 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.005356342, 2023-11-16T17:00:54.681 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:54.681 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 226, 2023-11-16T17:00:54.681 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.019415746, 2023-11-16T17:00:54.681 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000085910 2023-11-16T17:00:54.682 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:54.682 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:54.682 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:54.682 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:54.682 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:54.683 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:54.683 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 1425808, 2023-11-16T17:00:54.683 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 129392 2023-11-16T17:00:54.683 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:54.683 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:00:54.684 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 6, 2023-11-16T17:00:54.684 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 175, 2023-11-16T17:00:54.684 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 3264, 2023-11-16T17:00:54.684 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 182496, 2023-11-16T17:00:54.684 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 5, 2023-11-16T17:00:54.685 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 3, 2023-11-16T17:00:54.685 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.022063857, 2023-11-16T17:00:54.685 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.015308393, 2023-11-16T17:00:54.685 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.003256083, 2023-11-16T17:00:54.685 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:54.686 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:54.686 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 173, 2023-11-16T17:00:54.686 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.022996870, 2023-11-16T17:00:54.686 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000132929 2023-11-16T17:00:54.686 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:54.687 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:54.687 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:54.687 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:54.687 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:54.687 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:54.688 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 3264, 2023-11-16T17:00:54.688 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 182496 2023-11-16T17:00:54.688 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:54.688 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:00:54.688 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 0, 2023-11-16T17:00:54.689 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 2, 2023-11-16T17:00:54.689 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 0, 2023-11-16T17:00:54.689 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 0, 2023-11-16T17:00:54.689 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 2, 2023-11-16T17:00:54.689 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 1, 2023-11-16T17:00:54.690 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.000697792, 2023-11-16T17:00:54.690 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.000000000, 2023-11-16T17:00:54.690 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.000576608, 2023-11-16T17:00:54.690 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:54.690 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:54.691 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:54.691 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:54.691 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:54.691 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:54.691 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:54.692 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:54.692 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:54.692 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:54.692 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:54.692 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T17:00:54.693 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T17:00:54.693 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:54.693 INFO:teuthology.orchestra.run.smithi044.stdout: "cct": { 2023-11-16T17:00:54.693 INFO:teuthology.orchestra.run.smithi044.stdout: "total_workers": 0, 2023-11-16T17:00:54.693 INFO:teuthology.orchestra.run.smithi044.stdout: "unhealthy_workers": 0 2023-11-16T17:00:54.693 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:54.694 INFO:teuthology.orchestra.run.smithi044.stdout: "mempool": { 2023-11-16T17:00:54.694 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:00:54.694 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_items": 0, 2023-11-16T17:00:54.694 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:00:54.695 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:00:54.695 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:00:54.695 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:00:54.695 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:00:54.695 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:00:54.695 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:00:54.696 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:00:54.696 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:00:54.696 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:00:54.696 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:00:54.696 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:00:54.697 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:00:54.697 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_items": 0, 2023-11-16T17:00:54.697 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:00:54.697 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_items": 0, 2023-11-16T17:00:54.697 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:00:54.697 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:00:54.698 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:00:54.698 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:00:54.698 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:00:54.698 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:00:54.699 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:00:54.699 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_items": 0, 2023-11-16T17:00:54.699 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:00:54.699 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:00:54.699 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:00:54.700 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_items": 0, 2023-11-16T17:00:54.700 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_bytes": 0, 2023-11-16T17:00:54.700 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_items": 0, 2023-11-16T17:00:54.700 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:00:54.700 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:00:54.700 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:00:54.701 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:00:54.701 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_bytes": 150776, 2023-11-16T17:00:54.701 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_items": 32, 2023-11-16T17:00:54.701 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:00:54.701 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_items": 0, 2023-11-16T17:00:54.701 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_bytes": 0, 2023-11-16T17:00:54.702 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_items": 0, 2023-11-16T17:00:54.702 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:00:54.702 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_items": 0, 2023-11-16T17:00:54.702 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:00:54.702 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_items": 0, 2023-11-16T17:00:54.703 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_bytes": 0, 2023-11-16T17:00:54.703 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_items": 0, 2023-11-16T17:00:54.703 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:00:54.703 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:00:54.703 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_bytes": 0, 2023-11-16T17:00:54.704 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_items": 0, 2023-11-16T17:00:54.704 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_bytes": 928, 2023-11-16T17:00:54.704 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_items": 2, 2023-11-16T17:00:54.704 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_bytes": 0, 2023-11-16T17:00:54.704 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_items": 0, 2023-11-16T17:00:54.705 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_bytes": 0, 2023-11-16T17:00:54.705 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_items": 0 2023-11-16T17:00:54.705 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2023-11-16T17:00:54.705 INFO:teuthology.orchestra.run.smithi044.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:00:54.705 INFO:teuthology.orchestra.run.smithi044.stdout: "val": 0, 2023-11-16T17:00:54.706 INFO:teuthology.orchestra.run.smithi044.stdout: "max": 104857600, 2023-11-16T17:00:54.706 INFO:teuthology.orchestra.run.smithi044.stdout: "get_started": 0, 2023-11-16T17:00:54.706 INFO:teuthology.orchestra.run.smithi044.stdout: "get": 239, 2023-11-16T17:00:54.706 INFO:teuthology.orchestra.run.smithi044.stdout: "get_sum": 1392348, 2023-11-16T17:00:54.707 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:54.707 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_success": 239, 2023-11-16T17:00:54.707 INFO:teuthology.orchestra.run.smithi044.stdout: "take": 0, 2023-11-16T17:00:54.707 INFO:teuthology.orchestra.run.smithi044.stdout: "take_sum": 0, 2023-11-16T17:00:54.707 INFO:teuthology.orchestra.run.smithi044.stdout: "put": 239, 2023-11-16T17:00:54.707 INFO:teuthology.orchestra.run.smithi044.stdout: "put_sum": 1392348, 2023-11-16T17:00:54.708 INFO:teuthology.orchestra.run.smithi044.stdout: "wait": { 2023-11-16T17:00:54.708 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2023-11-16T17:00:54.708 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2023-11-16T17:00:54.708 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2023-11-16T17:00:54.708 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:54.709 INFO:teuthology.orchestra.run.smithi044.stdout: } 2023-11-16T17:00:54.709 INFO:teuthology.orchestra.run.smithi044.stdout:} 2023-11-16T17:00:55.195 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2023-11-16T17:00:55.195 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2023-11-16T17:00:55.196 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2023-11-16T17:00:55.196 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2023-11-16T17:00:55.196 DEBUG:tasks.check_counter:Getting stats from b 2023-11-16T17:00:55.197 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.b perf dump 2023-11-16T17:00:56.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:55 smithi044 ceph-mon[128750]: pgmap v341: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 6.1 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:00:56.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:55 smithi176 ceph-mon[131630]: pgmap v341: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 6.1 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:00:56.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:55 smithi077 ceph-mon[131204]: pgmap v341: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 6.1 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:00:57.336 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T17:00:58.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:57 smithi044 ceph-mon[128750]: pgmap v342: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 6.1 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:00:58.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:57 smithi176 ceph-mon[131630]: pgmap v342: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 6.1 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:00:58.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:57 smithi077 ceph-mon[131204]: pgmap v342: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 6.1 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:01:00.100 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:00:59 smithi077 ceph-mon[131204]: pgmap v343: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 6.1 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:01:00.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:00:59 smithi044 ceph-mon[128750]: pgmap v343: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 6.1 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:01:00.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:00:59 smithi176 ceph-mon[131630]: pgmap v343: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 6.1 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:01:01.915 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:01 smithi077 ceph-mon[131204]: pgmap v344: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 6.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:01:02.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:01 smithi044 ceph-mon[128750]: pgmap v344: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 6.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:01:02.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:01 smithi176 ceph-mon[131630]: pgmap v344: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 6.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:01:03.870 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2023-11-16T17:01:03.871 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:01:03.871 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 131170, 2023-11-16T17:01:03.871 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 130545, 2023-11-16T17:01:03.871 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 53329892, 2023-11-16T17:01:03.871 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 462197039, 2023-11-16T17:01:03.871 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 12, 2023-11-16T17:01:03.872 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 5, 2023-11-16T17:01:03.872 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 8.049735486, 2023-11-16T17:01:03.872 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 2.469695675, 2023-11-16T17:01:03.872 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 1.393459727, 2023-11-16T17:01:03.872 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 3.846364960, 2023-11-16T17:01:03.872 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:03.872 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 130543, 2023-11-16T17:01:03.872 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 215.369469716, 2023-11-16T17:01:03.873 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.001649797 2023-11-16T17:01:03.873 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.873 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:03.873 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 975, 2023-11-16T17:01:03.873 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000144027, 2023-11-16T17:01:03.873 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000147 2023-11-16T17:01:03.873 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.873 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 350912, 2023-11-16T17:01:03.874 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 110720 2023-11-16T17:01:03.874 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.874 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:01:03.874 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 102159, 2023-11-16T17:01:03.874 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 102022, 2023-11-16T17:01:03.874 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 42306875, 2023-11-16T17:01:03.874 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 393241393, 2023-11-16T17:01:03.874 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 200, 2023-11-16T17:01:03.875 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 8, 2023-11-16T17:01:03.875 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 6.559921940, 2023-11-16T17:01:03.875 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 2.058344600, 2023-11-16T17:01:03.875 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 1.134614311, 2023-11-16T17:01:03.875 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 3.096471512, 2023-11-16T17:01:03.875 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:03.875 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 102019, 2023-11-16T17:01:03.875 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 223.089661415, 2023-11-16T17:01:03.876 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.002186746 2023-11-16T17:01:03.876 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.876 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:03.876 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 399, 2023-11-16T17:01:03.876 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000046940, 2023-11-16T17:01:03.876 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000117 2023-11-16T17:01:03.876 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.876 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 86368, 2023-11-16T17:01:03.877 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 142896 2023-11-16T17:01:03.877 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.877 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:01:03.877 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 285407, 2023-11-16T17:01:03.877 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 489664, 2023-11-16T17:01:03.877 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 108886770, 2023-11-16T17:01:03.877 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 354748867, 2023-11-16T17:01:03.877 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 13, 2023-11-16T17:01:03.878 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 6, 2023-11-16T17:01:03.878 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 15.046580952, 2023-11-16T17:01:03.878 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 5.985978691, 2023-11-16T17:01:03.878 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 27.733159615, 2023-11-16T17:01:03.878 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 2.800189927, 2023-11-16T17:01:03.878 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:03.878 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 489664, 2023-11-16T17:01:03.878 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 1416.754978806, 2023-11-16T17:01:03.879 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.002893320 2023-11-16T17:01:03.879 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.879 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:03.879 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 447470, 2023-11-16T17:01:03.879 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.074067291, 2023-11-16T17:01:03.879 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000165 2023-11-16T17:01:03.879 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.879 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 1534016, 2023-11-16T17:01:03.880 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 511248 2023-11-16T17:01:03.880 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.880 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2023-11-16T17:01:03.880 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 1, 2023-11-16T17:01:03.880 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2023-11-16T17:01:03.880 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.880 INFO:teuthology.orchestra.run.smithi077.stdout: "finisher-MDSRank": { 2023-11-16T17:01:03.880 INFO:teuthology.orchestra.run.smithi077.stdout: "queue_len": 0, 2023-11-16T17:01:03.881 INFO:teuthology.orchestra.run.smithi077.stdout: "complete_latency": { 2023-11-16T17:01:03.881 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 56395, 2023-11-16T17:01:03.881 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 253.269389189, 2023-11-16T17:01:03.881 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.004490990 2023-11-16T17:01:03.881 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-11-16T17:01:03.881 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.881 INFO:teuthology.orchestra.run.smithi077.stdout: "finisher-PurgeQueue": { 2023-11-16T17:01:03.881 INFO:teuthology.orchestra.run.smithi077.stdout: "queue_len": 0, 2023-11-16T17:01:03.882 INFO:teuthology.orchestra.run.smithi077.stdout: "complete_latency": { 2023-11-16T17:01:03.882 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 69656, 2023-11-16T17:01:03.882 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 53.657864226, 2023-11-16T17:01:03.882 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000770326 2023-11-16T17:01:03.882 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-11-16T17:01:03.882 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.882 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2023-11-16T17:01:03.882 INFO:teuthology.orchestra.run.smithi077.stdout: "request": 105752, 2023-11-16T17:01:03.883 INFO:teuthology.orchestra.run.smithi077.stdout: "reply": 105752, 2023-11-16T17:01:03.883 INFO:teuthology.orchestra.run.smithi077.stdout: "reply_latency": { 2023-11-16T17:01:03.883 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 105752, 2023-11-16T17:01:03.883 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 3884.479434374, 2023-11-16T17:01:03.883 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.036731971 2023-11-16T17:01:03.883 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.883 INFO:teuthology.orchestra.run.smithi077.stdout: "slow_reply": 0, 2023-11-16T17:01:03.883 INFO:teuthology.orchestra.run.smithi077.stdout: "forward": 0, 2023-11-16T17:01:03.884 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_fetch_complete": 0, 2023-11-16T17:01:03.884 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_fetch_keys": 0, 2023-11-16T17:01:03.884 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_commit": 27575, 2023-11-16T17:01:03.884 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_split": 379, 2023-11-16T17:01:03.884 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_merge": 10, 2023-11-16T17:01:03.884 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes": 70941, 2023-11-16T17:01:03.884 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_top": 49637, 2023-11-16T17:01:03.884 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_bottom": 21304, 2023-11-16T17:01:03.885 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_pin_tail": 0, 2023-11-16T17:01:03.885 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_pinned": 29, 2023-11-16T17:01:03.885 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_expired": 0, 2023-11-16T17:01:03.885 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_with_caps": 6, 2023-11-16T17:01:03.885 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": 7, 2023-11-16T17:01:03.885 INFO:teuthology.orchestra.run.smithi077.stdout: "subtrees": 2, 2023-11-16T17:01:03.885 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse": 106433, 2023-11-16T17:01:03.885 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_hit": 105770, 2023-11-16T17:01:03.886 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_forward": 0, 2023-11-16T17:01:03.886 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_discover": 0, 2023-11-16T17:01:03.886 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_dir_fetch": 0, 2023-11-16T17:01:03.886 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_remote_ino": 0, 2023-11-16T17:01:03.886 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_lock": 0, 2023-11-16T17:01:03.886 INFO:teuthology.orchestra.run.smithi077.stdout: "load_cent": 0, 2023-11-16T17:01:03.886 INFO:teuthology.orchestra.run.smithi077.stdout: "q": 0, 2023-11-16T17:01:03.887 INFO:teuthology.orchestra.run.smithi077.stdout: "exported": 0, 2023-11-16T17:01:03.887 INFO:teuthology.orchestra.run.smithi077.stdout: "exported_inodes": 0, 2023-11-16T17:01:03.887 INFO:teuthology.orchestra.run.smithi077.stdout: "imported": 0, 2023-11-16T17:01:03.887 INFO:teuthology.orchestra.run.smithi077.stdout: "imported_inodes": 0, 2023-11-16T17:01:03.887 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_dir_fetch": 0, 2023-11-16T17:01:03.887 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_backtrace_fetch": 96243, 2023-11-16T17:01:03.887 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_peer_discover": 0, 2023-11-16T17:01:03.887 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rfiles": 4, 2023-11-16T17:01:03.888 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rbytes": 246, 2023-11-16T17:01:03.888 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rsnaps": 6, 2023-11-16T17:01:03.888 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_backtrace_fetch": 96243, 2023-11-16T17:01:03.888 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_set_tag": 0, 2023-11-16T17:01:03.888 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_backtrace_repaired": 0, 2023-11-16T17:01:03.888 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_inotable_repaired": 0, 2023-11-16T17:01:03.888 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dir_inodes": 179, 2023-11-16T17:01:03.888 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dir_base_inodes": 17, 2023-11-16T17:01:03.889 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dirfrag_rstats": 162, 2023-11-16T17:01:03.889 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_file_inodes": 96064, 2023-11-16T17:01:03.889 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_inode_file_caps": 0, 2023-11-16T17:01:03.889 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_revoke": 3275, 2023-11-16T17:01:03.889 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_grant": 105215, 2023-11-16T17:01:03.889 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_trunc": 1, 2023-11-16T17:01:03.889 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-16T17:01:03.889 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-16T17:01:03.890 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_caps": 77305, 2023-11-16T17:01:03.890 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_caps_dirty": 73956, 2023-11-16T17:01:03.890 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_cap_release": 422, 2023-11-16T17:01:03.890 INFO:teuthology.orchestra.run.smithi077.stdout: "process_request_cap_release": 55298 2023-11-16T17:01:03.890 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.890 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_cache": { 2023-11-16T17:01:03.890 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_update": 0, 2023-11-16T17:01:03.890 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_update_receipt": 0, 2023-11-16T17:01:03.891 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_try_discover": 0, 2023-11-16T17:01:03.891 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_send_discover": 0, 2023-11-16T17:01:03.891 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_handle_discover": 0, 2023-11-16T17:01:03.891 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays": 10, 2023-11-16T17:01:03.891 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays_delayed": 0, 2023-11-16T17:01:03.891 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays_enqueuing": 0, 2023-11-16T17:01:03.891 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_created": 50124, 2023-11-16T17:01:03.891 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_enqueued": 50114, 2023-11-16T17:01:03.892 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_reintegrated": 0, 2023-11-16T17:01:03.892 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_migrated": 0, 2023-11-16T17:01:03.892 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_processing": 0, 2023-11-16T17:01:03.892 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_enqueued": 0, 2023-11-16T17:01:03.892 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_prioritized": 0, 2023-11-16T17:01:03.892 INFO:teuthology.orchestra.run.smithi077.stdout: "recovery_started": 0, 2023-11-16T17:01:03.892 INFO:teuthology.orchestra.run.smithi077.stdout: "recovery_completed": 0, 2023-11-16T17:01:03.892 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_enqueue_scrub": 17, 2023-11-16T17:01:03.893 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_exportdir": 0, 2023-11-16T17:01:03.893 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_flush": 0, 2023-11-16T17:01:03.893 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_fragmentdir": 444, 2023-11-16T17:01:03.893 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_fragstats": 0, 2023-11-16T17:01:03.893 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_inodestats": 0 2023-11-16T17:01:03.893 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.893 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_log": { 2023-11-16T17:01:03.893 INFO:teuthology.orchestra.run.smithi077.stdout: "evadd": 226588, 2023-11-16T17:01:03.894 INFO:teuthology.orchestra.run.smithi077.stdout: "evex": 226536, 2023-11-16T17:01:03.894 INFO:teuthology.orchestra.run.smithi077.stdout: "evtrm": 226536, 2023-11-16T17:01:03.894 INFO:teuthology.orchestra.run.smithi077.stdout: "ev": 52, 2023-11-16T17:01:03.894 INFO:teuthology.orchestra.run.smithi077.stdout: "evexg": 0, 2023-11-16T17:01:03.894 INFO:teuthology.orchestra.run.smithi077.stdout: "evexd": 36, 2023-11-16T17:01:03.894 INFO:teuthology.orchestra.run.smithi077.stdout: "segadd": 344, 2023-11-16T17:01:03.894 INFO:teuthology.orchestra.run.smithi077.stdout: "segex": 343, 2023-11-16T17:01:03.894 INFO:teuthology.orchestra.run.smithi077.stdout: "segtrm": 343, 2023-11-16T17:01:03.895 INFO:teuthology.orchestra.run.smithi077.stdout: "seg": 2, 2023-11-16T17:01:03.895 INFO:teuthology.orchestra.run.smithi077.stdout: "segexg": 0, 2023-11-16T17:01:03.895 INFO:teuthology.orchestra.run.smithi077.stdout: "segexd": 1, 2023-11-16T17:01:03.895 INFO:teuthology.orchestra.run.smithi077.stdout: "expos": 774436220, 2023-11-16T17:01:03.895 INFO:teuthology.orchestra.run.smithi077.stdout: "wrpos": 774581079, 2023-11-16T17:01:03.895 INFO:teuthology.orchestra.run.smithi077.stdout: "rdpos": 0, 2023-11-16T17:01:03.895 INFO:teuthology.orchestra.run.smithi077.stdout: "jlat": { 2023-11-16T17:01:03.895 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 2097, 2023-11-16T17:01:03.896 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 829.386821281, 2023-11-16T17:01:03.896 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.395511121 2023-11-16T17:01:03.896 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.896 INFO:teuthology.orchestra.run.smithi077.stdout: "replayed": 0 2023-11-16T17:01:03.896 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.896 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_mem": { 2023-11-16T17:01:03.896 INFO:teuthology.orchestra.run.smithi077.stdout: "ino": 66711, 2023-11-16T17:01:03.896 INFO:teuthology.orchestra.run.smithi077.stdout: "ino+": 116842, 2023-11-16T17:01:03.897 INFO:teuthology.orchestra.run.smithi077.stdout: "ino-": 50131, 2023-11-16T17:01:03.897 INFO:teuthology.orchestra.run.smithi077.stdout: "dir": 2614, 2023-11-16T17:01:03.897 INFO:teuthology.orchestra.run.smithi077.stdout: "dir+": 3179, 2023-11-16T17:01:03.897 INFO:teuthology.orchestra.run.smithi077.stdout: "dir-": 565, 2023-11-16T17:01:03.897 INFO:teuthology.orchestra.run.smithi077.stdout: "dn": 70941, 2023-11-16T17:01:03.897 INFO:teuthology.orchestra.run.smithi077.stdout: "dn+": 166853, 2023-11-16T17:01:03.897 INFO:teuthology.orchestra.run.smithi077.stdout: "dn-": 95912, 2023-11-16T17:01:03.897 INFO:teuthology.orchestra.run.smithi077.stdout: "cap": 7, 2023-11-16T17:01:03.897 INFO:teuthology.orchestra.run.smithi077.stdout: "cap+": 57761, 2023-11-16T17:01:03.898 INFO:teuthology.orchestra.run.smithi077.stdout: "cap-": 57754, 2023-11-16T17:01:03.898 INFO:teuthology.orchestra.run.smithi077.stdout: "rss": 421476, 2023-11-16T17:01:03.898 INFO:teuthology.orchestra.run.smithi077.stdout: "heap": 207156 2023-11-16T17:01:03.898 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.898 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_server": { 2023-11-16T17:01:03.898 INFO:teuthology.orchestra.run.smithi077.stdout: "dispatch_client_request": 109297, 2023-11-16T17:01:03.898 INFO:teuthology.orchestra.run.smithi077.stdout: "dispatch_server_request": 0, 2023-11-16T17:01:03.899 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_request": 105752, 2023-11-16T17:01:03.899 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_session": 36, 2023-11-16T17:01:03.899 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_peer_request": 0, 2023-11-16T17:01:03.899 INFO:teuthology.orchestra.run.smithi077.stdout: "req_create_latency": { 2023-11-16T17:01:03.899 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 50012, 2023-11-16T17:01:03.899 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 3262.569775981, 2023-11-16T17:01:03.899 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.065235738 2023-11-16T17:01:03.899 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.899 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getattr_latency": { 2023-11-16T17:01:03.900 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 158, 2023-11-16T17:01:03.900 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.167764396, 2023-11-16T17:01:03.900 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.001061799 2023-11-16T17:01:03.900 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.900 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getfilelock_latency": { 2023-11-16T17:01:03.901 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.901 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.901 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.901 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.901 INFO:teuthology.orchestra.run.smithi077.stdout: "req_link_latency": { 2023-11-16T17:01:03.901 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.901 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.901 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.902 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.902 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookup_latency": { 2023-11-16T17:01:03.902 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 125, 2023-11-16T17:01:03.902 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.042727062, 2023-11-16T17:01:03.902 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000341816 2023-11-16T17:01:03.902 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.902 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookuphash_latency": { 2023-11-16T17:01:03.902 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.902 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.903 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.903 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.903 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupino_latency": { 2023-11-16T17:01:03.903 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 2, 2023-11-16T17:01:03.903 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000339879, 2023-11-16T17:01:03.903 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000169939 2023-11-16T17:01:03.903 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.903 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupname_latency": { 2023-11-16T17:01:03.903 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.904 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.904 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.904 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.904 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupparent_latency": { 2023-11-16T17:01:03.904 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.904 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.904 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.904 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.904 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupsnap_latency": { 2023-11-16T17:01:03.905 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.905 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.905 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.905 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.905 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lssnap_latency": { 2023-11-16T17:01:03.905 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 12, 2023-11-16T17:01:03.905 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.028216289, 2023-11-16T17:01:03.905 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.002351357 2023-11-16T17:01:03.906 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.906 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mkdir_latency": { 2023-11-16T17:01:03.906 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 125, 2023-11-16T17:01:03.906 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.126548530, 2023-11-16T17:01:03.906 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.001012388 2023-11-16T17:01:03.906 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.906 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mknod_latency": { 2023-11-16T17:01:03.906 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.906 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.907 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.907 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.907 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mksnap_latency": { 2023-11-16T17:01:03.907 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 12, 2023-11-16T17:01:03.907 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 10.019695624, 2023-11-16T17:01:03.907 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.834974635 2023-11-16T17:01:03.907 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.907 INFO:teuthology.orchestra.run.smithi077.stdout: "req_open_latency": { 2023-11-16T17:01:03.907 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.908 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.908 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.908 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.908 INFO:teuthology.orchestra.run.smithi077.stdout: "req_readdir_latency": { 2023-11-16T17:01:03.908 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 5129, 2023-11-16T17:01:03.908 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 18.031849989, 2023-11-16T17:01:03.908 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.003515665 2023-11-16T17:01:03.908 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.909 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rename_latency": { 2023-11-16T17:01:03.909 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 2, 2023-11-16T17:01:03.909 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.002437868, 2023-11-16T17:01:03.909 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.001218934 2023-11-16T17:01:03.909 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.909 INFO:teuthology.orchestra.run.smithi077.stdout: "req_renamesnap_latency": { 2023-11-16T17:01:03.909 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.909 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.910 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.910 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.910 INFO:teuthology.orchestra.run.smithi077.stdout: "req_snapdiff_latency": { 2023-11-16T17:01:03.910 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.910 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.910 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.910 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.910 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmdir_latency": { 2023-11-16T17:01:03.911 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 116, 2023-11-16T17:01:03.911 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.351213604, 2023-11-16T17:01:03.911 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.003027703 2023-11-16T17:01:03.911 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.911 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmsnap_latency": { 2023-11-16T17:01:03.911 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 6, 2023-11-16T17:01:03.911 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 8.113599615, 2023-11-16T17:01:03.911 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 1.352266602 2023-11-16T17:01:03.911 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.912 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmxattr_latency": { 2023-11-16T17:01:03.912 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 3, 2023-11-16T17:01:03.912 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.002539630, 2023-11-16T17:01:03.912 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000846543 2023-11-16T17:01:03.912 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.912 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setattr_latency": { 2023-11-16T17:01:03.912 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 4, 2023-11-16T17:01:03.912 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.010661787, 2023-11-16T17:01:03.912 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.002665446 2023-11-16T17:01:03.913 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.913 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setdirlayout_latency": { 2023-11-16T17:01:03.913 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.913 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.913 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.913 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.913 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setfilelock_latency": { 2023-11-16T17:01:03.913 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 22, 2023-11-16T17:01:03.913 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 10.025971184, 2023-11-16T17:01:03.914 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.455725962 2023-11-16T17:01:03.914 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.914 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setlayout_latency": { 2023-11-16T17:01:03.914 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.914 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.914 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.914 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.914 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setxattr_latency": { 2023-11-16T17:01:03.915 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 16, 2023-11-16T17:01:03.915 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.029395257, 2023-11-16T17:01:03.915 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.001837203 2023-11-16T17:01:03.915 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.915 INFO:teuthology.orchestra.run.smithi077.stdout: "req_symlink_latency": { 2023-11-16T17:01:03.915 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.915 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.915 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.915 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.916 INFO:teuthology.orchestra.run.smithi077.stdout: "req_unlink_latency": { 2023-11-16T17:01:03.916 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 50008, 2023-11-16T17:01:03.916 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 574.956697679, 2023-11-16T17:01:03.916 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.011497294 2023-11-16T17:01:03.916 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.916 INFO:teuthology.orchestra.run.smithi077.stdout: "cap_revoke_eviction": 0, 2023-11-16T17:01:03.916 INFO:teuthology.orchestra.run.smithi077.stdout: "cap_acquisition_throttle": 0, 2023-11-16T17:01:03.916 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getvxattr_latency": { 2023-11-16T17:01:03.916 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.917 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.917 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.917 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-11-16T17:01:03.917 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.917 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_sessions": { 2023-11-16T17:01:03.917 INFO:teuthology.orchestra.run.smithi077.stdout: "session_count": 2, 2023-11-16T17:01:03.917 INFO:teuthology.orchestra.run.smithi077.stdout: "session_add": 6, 2023-11-16T17:01:03.917 INFO:teuthology.orchestra.run.smithi077.stdout: "session_remove": 4, 2023-11-16T17:01:03.917 INFO:teuthology.orchestra.run.smithi077.stdout: "sessions_open": 2, 2023-11-16T17:01:03.917 INFO:teuthology.orchestra.run.smithi077.stdout: "sessions_stale": 0, 2023-11-16T17:01:03.918 INFO:teuthology.orchestra.run.smithi077.stdout: "total_load": 2954, 2023-11-16T17:01:03.918 INFO:teuthology.orchestra.run.smithi077.stdout: "average_load": 1477, 2023-11-16T17:01:03.918 INFO:teuthology.orchestra.run.smithi077.stdout: "avg_session_uptime": 674 2023-11-16T17:01:03.918 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.918 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2023-11-16T17:01:03.918 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:01:03.918 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2023-11-16T17:01:03.918 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:01:03.918 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:01:03.919 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:01:03.919 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:01:03.919 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:01:03.919 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:01:03.919 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:01:03.919 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:01:03.919 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:01:03.919 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:01:03.920 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:01:03.920 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:01:03.920 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:01:03.920 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2023-11-16T17:01:03.920 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:01:03.920 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2023-11-16T17:01:03.920 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:01:03.920 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:01:03.921 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:01:03.921 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:01:03.921 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:01:03.921 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:01:03.921 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:01:03.921 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2023-11-16T17:01:03.921 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:01:03.921 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:01:03.921 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:01:03.921 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2023-11-16T17:01:03.922 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2023-11-16T17:01:03.922 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2023-11-16T17:01:03.922 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:01:03.922 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:01:03.922 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:01:03.922 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:01:03.922 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 2071158, 2023-11-16T17:01:03.922 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 49496, 2023-11-16T17:01:03.922 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:01:03.922 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2023-11-16T17:01:03.923 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2023-11-16T17:01:03.923 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2023-11-16T17:01:03.923 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:01:03.923 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2023-11-16T17:01:03.923 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:01:03.923 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2023-11-16T17:01:03.923 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 10040, 2023-11-16T17:01:03.923 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 194, 2023-11-16T17:01:03.923 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:01:03.924 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:01:03.924 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2023-11-16T17:01:03.924 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2023-11-16T17:01:03.924 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 201565872, 2023-11-16T17:01:03.924 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 1678786, 2023-11-16T17:01:03.924 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2023-11-16T17:01:03.924 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2023-11-16T17:01:03.924 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2023-11-16T17:01:03.924 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2023-11-16T17:01:03.924 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.925 INFO:teuthology.orchestra.run.smithi077.stdout: "objecter": { 2023-11-16T17:01:03.925 INFO:teuthology.orchestra.run.smithi077.stdout: "op_active": 0, 2023-11-16T17:01:03.925 INFO:teuthology.orchestra.run.smithi077.stdout: "op_laggy": 0, 2023-11-16T17:01:03.925 INFO:teuthology.orchestra.run.smithi077.stdout: "op_send": 333222, 2023-11-16T17:01:03.925 INFO:teuthology.orchestra.run.smithi077.stdout: "op_send_bytes": 965242166, 2023-11-16T17:01:03.925 INFO:teuthology.orchestra.run.smithi077.stdout: "op_resend": 37, 2023-11-16T17:01:03.925 INFO:teuthology.orchestra.run.smithi077.stdout: "op_reply": 333222, 2023-11-16T17:01:03.925 INFO:teuthology.orchestra.run.smithi077.stdout: "op_latency": { 2023-11-16T17:01:03.925 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 333222, 2023-11-16T17:01:03.925 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 167116.385857946, 2023-11-16T17:01:03.926 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.501516664 2023-11-16T17:01:03.926 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.926 INFO:teuthology.orchestra.run.smithi077.stdout: "op_inflight": 0, 2023-11-16T17:01:03.926 INFO:teuthology.orchestra.run.smithi077.stdout: "oplen_avg": { 2023-11-16T17:01:03.926 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 333222, 2023-11-16T17:01:03.926 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 536194 2023-11-16T17:01:03.926 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.926 INFO:teuthology.orchestra.run.smithi077.stdout: "op": 333222, 2023-11-16T17:01:03.927 INFO:teuthology.orchestra.run.smithi077.stdout: "op_r": 96647, 2023-11-16T17:01:03.927 INFO:teuthology.orchestra.run.smithi077.stdout: "op_w": 236575, 2023-11-16T17:01:03.927 INFO:teuthology.orchestra.run.smithi077.stdout: "op_rmw": 0, 2023-11-16T17:01:03.927 INFO:teuthology.orchestra.run.smithi077.stdout: "op_pg": 0, 2023-11-16T17:01:03.927 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_stat": 24497, 2023-11-16T17:01:03.927 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_create": 104681, 2023-11-16T17:01:03.927 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_read": 387, 2023-11-16T17:01:03.927 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_write": 2467, 2023-11-16T17:01:03.927 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_writefull": 238, 2023-11-16T17:01:03.927 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_writesame": 0, 2023-11-16T17:01:03.928 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_append": 0, 2023-11-16T17:01:03.928 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_zero": 0, 2023-11-16T17:01:03.928 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_truncate": 19, 2023-11-16T17:01:03.928 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_delete": 101442, 2023-11-16T17:01:03.928 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_mapext": 0, 2023-11-16T17:01:03.928 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_sparse_read": 0, 2023-11-16T17:01:03.928 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_clonerange": 0, 2023-11-16T17:01:03.928 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_getxattr": 96260, 2023-11-16T17:01:03.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_setxattr": 159366, 2023-11-16T17:01:03.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_cmpxattr": 0, 2023-11-16T17:01:03.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_rmxattr": 0, 2023-11-16T17:01:03.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_resetxattrs": 0, 2023-11-16T17:01:03.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_call": 0, 2023-11-16T17:01:03.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_watch": 0, 2023-11-16T17:01:03.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_notify": 0, 2023-11-16T17:01:03.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_src_cmpxattr": 0, 2023-11-16T17:01:03.930 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_pgls": 0, 2023-11-16T17:01:03.930 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_pgls_filter": 0, 2023-11-16T17:01:03.930 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_other": 1, 2023-11-16T17:01:03.930 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_active": 0, 2023-11-16T17:01:03.930 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_send": 0, 2023-11-16T17:01:03.930 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_resend": 0, 2023-11-16T17:01:03.930 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_ping": 0, 2023-11-16T17:01:03.930 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_active": 0, 2023-11-16T17:01:03.930 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_send": 0, 2023-11-16T17:01:03.931 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_resend": 0, 2023-11-16T17:01:03.931 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_active": 0, 2023-11-16T17:01:03.931 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_send": 0, 2023-11-16T17:01:03.931 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_resend": 0, 2023-11-16T17:01:03.931 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_active": 0, 2023-11-16T17:01:03.931 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_send": 0, 2023-11-16T17:01:03.931 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_resend": 0, 2023-11-16T17:01:03.931 INFO:teuthology.orchestra.run.smithi077.stdout: "command_active": 0, 2023-11-16T17:01:03.931 INFO:teuthology.orchestra.run.smithi077.stdout: "command_send": 0, 2023-11-16T17:01:03.932 INFO:teuthology.orchestra.run.smithi077.stdout: "command_resend": 0, 2023-11-16T17:01:03.932 INFO:teuthology.orchestra.run.smithi077.stdout: "map_epoch": 101, 2023-11-16T17:01:03.932 INFO:teuthology.orchestra.run.smithi077.stdout: "map_full": 0, 2023-11-16T17:01:03.932 INFO:teuthology.orchestra.run.smithi077.stdout: "map_inc": 19, 2023-11-16T17:01:03.932 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_sessions": 12, 2023-11-16T17:01:03.932 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_session_open": 12, 2023-11-16T17:01:03.932 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_session_close": 0, 2023-11-16T17:01:03.932 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_laggy": 0, 2023-11-16T17:01:03.933 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_wr": 43957, 2023-11-16T17:01:03.933 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_rd": 0, 2023-11-16T17:01:03.933 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_del": 2879 2023-11-16T17:01:03.933 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.933 INFO:teuthology.orchestra.run.smithi077.stdout: "oft": { 2023-11-16T17:01:03.933 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_objs": 1, 2023-11-16T17:01:03.933 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_kv_pairs": 0, 2023-11-16T17:01:03.933 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_updates": 0, 2023-11-16T17:01:03.933 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_removes": 0 2023-11-16T17:01:03.934 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.934 INFO:teuthology.orchestra.run.smithi077.stdout: "osd-slow-ops": { 2023-11-16T17:01:03.934 INFO:teuthology.orchestra.run.smithi077.stdout: "slow_ops_count": 106230 2023-11-16T17:01:03.934 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.934 INFO:teuthology.orchestra.run.smithi077.stdout: "purge_queue": { 2023-11-16T17:01:03.934 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_ops": 0, 2023-11-16T17:01:03.934 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_ops_high_water": 48, 2023-11-16T17:01:03.934 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing": 0, 2023-11-16T17:01:03.935 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_high_water": 24, 2023-11-16T17:01:03.935 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executed": 50114, 2023-11-16T17:01:03.935 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_item_in_journal": 0 2023-11-16T17:01:03.935 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.935 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:01:03.935 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2023-11-16T17:01:03.935 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2023-11-16T17:01:03.935 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2023-11-16T17:01:03.936 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 518732, 2023-11-16T17:01:03.936 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 157766774, 2023-11-16T17:01:03.936 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2023-11-16T17:01:03.936 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 518732, 2023-11-16T17:01:03.936 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2023-11-16T17:01:03.936 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2023-11-16T17:01:03.936 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 518732, 2023-11-16T17:01:03.936 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 157766774, 2023-11-16T17:01:03.936 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2023-11-16T17:01:03.937 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.937 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.937 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.937 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-11-16T17:01:03.937 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.937 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-objecter_bytes": { 2023-11-16T17:01:03.937 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2023-11-16T17:01:03.937 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2023-11-16T17:01:03.937 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2023-11-16T17:01:03.938 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2023-11-16T17:01:03.938 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2023-11-16T17:01:03.938 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2023-11-16T17:01:03.938 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2023-11-16T17:01:03.938 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 333222, 2023-11-16T17:01:03.938 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 971743680, 2023-11-16T17:01:03.938 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 333222, 2023-11-16T17:01:03.939 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 971743680, 2023-11-16T17:01:03.939 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2023-11-16T17:01:03.939 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.939 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.939 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.939 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-11-16T17:01:03.939 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.939 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-objecter_ops": { 2023-11-16T17:01:03.939 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2023-11-16T17:01:03.940 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 1024, 2023-11-16T17:01:03.940 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2023-11-16T17:01:03.940 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2023-11-16T17:01:03.940 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2023-11-16T17:01:03.940 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2023-11-16T17:01:03.940 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2023-11-16T17:01:03.940 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 333222, 2023-11-16T17:01:03.940 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 333222, 2023-11-16T17:01:03.941 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 333222, 2023-11-16T17:01:03.941 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 333222, 2023-11-16T17:01:03.941 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2023-11-16T17:01:03.941 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.941 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.941 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.941 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-11-16T17:01:03.941 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.941 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-write_buf_throttle": { 2023-11-16T17:01:03.942 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2023-11-16T17:01:03.942 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 3758096384, 2023-11-16T17:01:03.942 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2023-11-16T17:01:03.942 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 50114, 2023-11-16T17:01:03.942 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 6501514, 2023-11-16T17:01:03.942 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2023-11-16T17:01:03.942 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 50114, 2023-11-16T17:01:03.942 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2023-11-16T17:01:03.942 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2023-11-16T17:01:03.943 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 370, 2023-11-16T17:01:03.943 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 6501514, 2023-11-16T17:01:03.943 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2023-11-16T17:01:03.943 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.943 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.943 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.943 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-11-16T17:01:03.943 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:03.943 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-write_buf_throttle-0x555d2d2380a0": { 2023-11-16T17:01:03.944 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 60, 2023-11-16T17:01:03.944 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 3758096384, 2023-11-16T17:01:03.944 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2023-11-16T17:01:03.944 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 226588, 2023-11-16T17:01:03.944 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 770386775, 2023-11-16T17:01:03.944 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2023-11-16T17:01:03.944 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 226588, 2023-11-16T17:01:03.944 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2023-11-16T17:01:03.944 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2023-11-16T17:01:03.945 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 2097, 2023-11-16T17:01:03.945 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 770386715, 2023-11-16T17:01:03.945 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2023-11-16T17:01:03.945 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:03.945 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:03.945 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:03.945 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-11-16T17:01:03.945 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-11-16T17:01:03.945 INFO:teuthology.orchestra.run.smithi077.stdout:} 2023-11-16T17:01:03.946 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:03 smithi077 ceph-mon[131204]: pgmap v345: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 6.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:01:04.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:03 smithi044 ceph-mon[128750]: pgmap v345: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 6.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:01:04.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:03 smithi176 ceph-mon[131630]: pgmap v345: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 6.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2023-11-16T17:01:04.959 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2023-11-16T17:01:04.959 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=12 2023-11-16T17:01:04.960 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=6 2023-11-16T17:01:04.960 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=379 2023-11-16T17:01:04.960 DEBUG:tasks.check_counter:Getting stats from e 2023-11-16T17:01:04.960 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.e perf dump 2023-11-16T17:01:05.763 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T17:01:06.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:05 smithi044 ceph-mon[128750]: pgmap v346: 129 pgs: 14 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 74 active+clean; 6.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 23 KiB/s wr, 5 op/s 2023-11-16T17:01:06.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:05 smithi176 ceph-mon[131630]: pgmap v346: 129 pgs: 14 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 74 active+clean; 6.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 23 KiB/s wr, 5 op/s 2023-11-16T17:01:06.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:05 smithi077 ceph-mon[131204]: pgmap v346: 129 pgs: 14 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 74 active+clean; 6.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 23 KiB/s wr, 5 op/s 2023-11-16T17:01:06.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:06 smithi077 ceph-mon[131204]: pgmap v347: 129 pgs: 14 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 74 active+clean; 6.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 45 KiB/s wr, 16 op/s 2023-11-16T17:01:07.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:06 smithi044 ceph-mon[128750]: pgmap v347: 129 pgs: 14 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 74 active+clean; 6.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 45 KiB/s wr, 16 op/s 2023-11-16T17:01:07.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:06 smithi176 ceph-mon[131630]: pgmap v347: 129 pgs: 14 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 74 active+clean; 6.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 45 KiB/s wr, 16 op/s 2023-11-16T17:01:08.096 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2023-11-16T17:01:08.096 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:01:08.096 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 4, 2023-11-16T17:01:08.096 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 4, 2023-11-16T17:01:08.096 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 2944, 2023-11-16T17:01:08.097 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 448, 2023-11-16T17:01:08.097 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 4, 2023-11-16T17:01:08.097 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 0, 2023-11-16T17:01:08.097 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.001565554, 2023-11-16T17:01:08.097 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.000036096, 2023-11-16T17:01:08.097 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.000659163, 2023-11-16T17:01:08.097 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:01:08.097 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:08.098 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 3, 2023-11-16T17:01:08.098 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000722088, 2023-11-16T17:01:08.098 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000240696 2023-11-16T17:01:08.098 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:08.098 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:08.098 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:08.098 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:08.098 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:08.099 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:08.099 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 2944, 2023-11-16T17:01:08.099 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 448 2023-11-16T17:01:08.099 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:08.099 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:01:08.099 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 239, 2023-11-16T17:01:08.099 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 229, 2023-11-16T17:01:08.099 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 1426352, 2023-11-16T17:01:08.100 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 130560, 2023-11-16T17:01:08.100 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 2, 2023-11-16T17:01:08.100 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 1, 2023-11-16T17:01:08.100 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.042988115, 2023-11-16T17:01:08.100 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.013154287, 2023-11-16T17:01:08.100 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.030283258, 2023-11-16T17:01:08.100 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.006496176, 2023-11-16T17:01:08.100 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:08.101 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 228, 2023-11-16T17:01:08.101 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.018222496, 2023-11-16T17:01:08.101 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000079923 2023-11-16T17:01:08.101 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:08.101 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:08.101 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:08.101 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:08.101 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:08.101 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:08.102 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 1426352, 2023-11-16T17:01:08.102 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 130560 2023-11-16T17:01:08.102 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:08.102 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:01:08.102 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 2, 2023-11-16T17:01:08.102 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 175, 2023-11-16T17:01:08.102 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 320, 2023-11-16T17:01:08.102 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 182400, 2023-11-16T17:01:08.103 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 5, 2023-11-16T17:01:08.103 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 3, 2023-11-16T17:01:08.103 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.010207232, 2023-11-16T17:01:08.103 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.006403636, 2023-11-16T17:01:08.103 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.001588378, 2023-11-16T17:01:08.103 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:01:08.103 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:08.103 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 172, 2023-11-16T17:01:08.103 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.033005435, 2023-11-16T17:01:08.104 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000191892 2023-11-16T17:01:08.104 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:08.104 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:08.104 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:08.104 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:08.104 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:08.104 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:08.104 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T17:01:08.104 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 182400 2023-11-16T17:01:08.105 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:08.105 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2023-11-16T17:01:08.105 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 0, 2023-11-16T17:01:08.105 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2023-11-16T17:01:08.105 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:08.105 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2023-11-16T17:01:08.105 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:01:08.105 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2023-11-16T17:01:08.106 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:01:08.106 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:01:08.106 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:01:08.106 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:01:08.106 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:01:08.106 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:01:08.106 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:01:08.106 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:01:08.107 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:01:08.107 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:01:08.107 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:01:08.107 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:01:08.107 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:01:08.107 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2023-11-16T17:01:08.107 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:01:08.107 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2023-11-16T17:01:08.107 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:01:08.108 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:01:08.108 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:01:08.108 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:01:08.108 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:01:08.108 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:01:08.108 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:01:08.108 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2023-11-16T17:01:08.108 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:01:08.108 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:01:08.109 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:01:08.109 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2023-11-16T17:01:08.109 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2023-11-16T17:01:08.109 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2023-11-16T17:01:08.109 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:01:08.109 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:01:08.109 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:01:08.109 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:01:08.110 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 148731, 2023-11-16T17:01:08.110 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 31, 2023-11-16T17:01:08.110 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:01:08.110 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2023-11-16T17:01:08.110 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2023-11-16T17:01:08.110 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2023-11-16T17:01:08.110 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:01:08.110 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2023-11-16T17:01:08.111 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:01:08.111 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2023-11-16T17:01:08.111 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 0, 2023-11-16T17:01:08.111 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 0, 2023-11-16T17:01:08.111 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:01:08.111 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:01:08.111 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2023-11-16T17:01:08.112 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2023-11-16T17:01:08.112 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 928, 2023-11-16T17:01:08.112 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 2, 2023-11-16T17:01:08.112 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2023-11-16T17:01:08.112 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2023-11-16T17:01:08.112 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2023-11-16T17:01:08.112 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2023-11-16T17:01:08.112 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:08.113 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:01:08.113 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2023-11-16T17:01:08.113 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2023-11-16T17:01:08.113 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2023-11-16T17:01:08.113 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 241, 2023-11-16T17:01:08.113 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 1392608, 2023-11-16T17:01:08.113 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2023-11-16T17:01:08.113 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 241, 2023-11-16T17:01:08.113 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2023-11-16T17:01:08.114 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2023-11-16T17:01:08.114 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 241, 2023-11-16T17:01:08.114 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 1392608, 2023-11-16T17:01:08.114 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2023-11-16T17:01:08.114 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:08.114 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:08.114 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:08.114 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-11-16T17:01:08.115 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-11-16T17:01:08.115 INFO:teuthology.orchestra.run.smithi077.stdout:} 2023-11-16T17:01:08.590 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2023-11-16T17:01:08.590 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2023-11-16T17:01:08.590 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2023-11-16T17:01:08.590 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.e 2023-11-16T17:01:08.591 DEBUG:tasks.check_counter:Getting stats from h 2023-11-16T17:01:08.591 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.h perf dump 2023-11-16T17:01:08.825 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:08 smithi077 ceph-mon[131204]: osdmap e102: 12 total, 12 up, 12 in 2023-11-16T17:01:08.825 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:08 smithi077 ceph-mon[131204]: pgmap v349: 129 pgs: 11 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 85 active+clean; 6.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 54 KiB/s wr, 19 op/s 2023-11-16T17:01:09.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:08 smithi044 ceph-mon[128750]: osdmap e102: 12 total, 12 up, 12 in 2023-11-16T17:01:09.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:08 smithi044 ceph-mon[128750]: pgmap v349: 129 pgs: 11 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 85 active+clean; 6.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 54 KiB/s wr, 19 op/s 2023-11-16T17:01:09.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:08 smithi176 ceph-mon[131630]: osdmap e102: 12 total, 12 up, 12 in 2023-11-16T17:01:09.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:08 smithi176 ceph-mon[131630]: pgmap v349: 129 pgs: 11 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 85 active+clean; 6.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 54 KiB/s wr, 19 op/s 2023-11-16T17:01:09.590 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T17:01:10.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:10 smithi077 ceph-mon[131204]: pgmap v350: 129 pgs: 11 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 85 active+clean; 6.1 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 54 KiB/s wr, 19 op/s 2023-11-16T17:01:11.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:10 smithi044 ceph-mon[128750]: pgmap v350: 129 pgs: 11 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 85 active+clean; 6.1 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 54 KiB/s wr, 19 op/s 2023-11-16T17:01:11.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:10 smithi176 ceph-mon[131630]: pgmap v350: 129 pgs: 11 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 85 active+clean; 6.1 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 54 KiB/s wr, 19 op/s 2023-11-16T17:01:11.940 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2023-11-16T17:01:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:01:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 4, 2023-11-16T17:01:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 4, 2023-11-16T17:01:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 2944, 2023-11-16T17:01:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 448, 2023-11-16T17:01:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 4, 2023-11-16T17:01:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 0, 2023-11-16T17:01:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.001446818, 2023-11-16T17:01:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.000041849, 2023-11-16T17:01:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.000535532, 2023-11-16T17:01:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:01:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 3, 2023-11-16T17:01:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000649879, 2023-11-16T17:01:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000216626 2023-11-16T17:01:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 2944, 2023-11-16T17:01:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 448 2023-11-16T17:01:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:01:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 238, 2023-11-16T17:01:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 228, 2023-11-16T17:01:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 1426080, 2023-11-16T17:01:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 129968, 2023-11-16T17:01:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 2, 2023-11-16T17:01:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 1, 2023-11-16T17:01:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.043336200, 2023-11-16T17:01:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.013157765, 2023-11-16T17:01:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.031012745, 2023-11-16T17:01:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.006498349, 2023-11-16T17:01:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 227, 2023-11-16T17:01:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.018292460, 2023-11-16T17:01:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000080583 2023-11-16T17:01:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 1426080, 2023-11-16T17:01:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 129968 2023-11-16T17:01:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:01:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 2, 2023-11-16T17:01:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 174, 2023-11-16T17:01:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 320, 2023-11-16T17:01:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 182224, 2023-11-16T17:01:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 5, 2023-11-16T17:01:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 3, 2023-11-16T17:01:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.010038845, 2023-11-16T17:01:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.006554136, 2023-11-16T17:01:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.001477872, 2023-11-16T17:01:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:01:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 171, 2023-11-16T17:01:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.033462526, 2023-11-16T17:01:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000195687 2023-11-16T17:01:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T17:01:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 182224 2023-11-16T17:01:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2023-11-16T17:01:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 0, 2023-11-16T17:01:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2023-11-16T17:01:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2023-11-16T17:01:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:01:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2023-11-16T17:01:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:01:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:01:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:01:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:01:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:01:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:01:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:01:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:01:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:01:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:01:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:01:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:01:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:01:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2023-11-16T17:01:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:01:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2023-11-16T17:01:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:01:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:01:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:01:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:01:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:01:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:01:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:01:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2023-11-16T17:01:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:01:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:01:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:01:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2023-11-16T17:01:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2023-11-16T17:01:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2023-11-16T17:01:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:01:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:01:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:01:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:01:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 148994, 2023-11-16T17:01:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 31, 2023-11-16T17:01:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:01:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2023-11-16T17:01:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2023-11-16T17:01:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2023-11-16T17:01:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:01:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2023-11-16T17:01:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:01:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2023-11-16T17:01:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 0, 2023-11-16T17:01:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 0, 2023-11-16T17:01:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:01:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:01:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2023-11-16T17:01:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2023-11-16T17:01:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 928, 2023-11-16T17:01:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 2, 2023-11-16T17:01:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2023-11-16T17:01:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2023-11-16T17:01:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2023-11-16T17:01:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2023-11-16T17:01:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:01:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2023-11-16T17:01:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2023-11-16T17:01:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2023-11-16T17:01:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 240, 2023-11-16T17:01:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 1392478, 2023-11-16T17:01:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2023-11-16T17:01:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 240, 2023-11-16T17:01:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2023-11-16T17:01:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2023-11-16T17:01:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 240, 2023-11-16T17:01:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 1392478, 2023-11-16T17:01:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2023-11-16T17:01:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-11-16T17:01:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-11-16T17:01:11.966 INFO:teuthology.orchestra.run.smithi077.stdout:} 2023-11-16T17:01:13.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:12 smithi077 ceph-mon[131204]: pgmap v351: 129 pgs: 14 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 70 active+clean; 6.1 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 54 KiB/s wr, 19 op/s 2023-11-16T17:01:13.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:12 smithi044 ceph-mon[128750]: pgmap v351: 129 pgs: 14 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 70 active+clean; 6.1 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 54 KiB/s wr, 19 op/s 2023-11-16T17:01:13.395 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:12 smithi176 ceph-mon[131630]: pgmap v351: 129 pgs: 14 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 70 active+clean; 6.1 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 54 KiB/s wr, 19 op/s 2023-11-16T17:01:13.825 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.h 2023-11-16T17:01:13.826 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.h 2023-11-16T17:01:13.826 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.h 2023-11-16T17:01:13.826 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.h 2023-11-16T17:01:13.826 DEBUG:tasks.check_counter:Getting stats from k 2023-11-16T17:01:13.826 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.k perf dump 2023-11-16T17:01:14.783 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T17:01:15.728 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:15 smithi077 ceph-mon[131204]: pgmap v352: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 KiB/s wr, 16 op/s 2023-11-16T17:01:15.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:15 smithi044 ceph-mon[128750]: pgmap v352: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 KiB/s wr, 16 op/s 2023-11-16T17:01:15.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:15 smithi176 ceph-mon[131630]: pgmap v352: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 KiB/s wr, 16 op/s 2023-11-16T17:01:17.380 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2023-11-16T17:01:17.380 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:01:17.380 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 0, 2023-11-16T17:01:17.381 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 2, 2023-11-16T17:01:17.381 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 0, 2023-11-16T17:01:17.381 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 0, 2023-11-16T17:01:17.381 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 4, 2023-11-16T17:01:17.382 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 0, 2023-11-16T17:01:17.382 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.001111914, 2023-11-16T17:01:17.382 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.000000000, 2023-11-16T17:01:17.382 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.000204674, 2023-11-16T17:01:17.382 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:01:17.383 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:17.383 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:17.383 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:17.383 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:17.383 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:17.384 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:17.384 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:17.384 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:17.384 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:17.384 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:17.385 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T17:01:17.385 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T17:01:17.385 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:17.385 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:01:17.385 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 6, 2023-11-16T17:01:17.385 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 180, 2023-11-16T17:01:17.386 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 3264, 2023-11-16T17:01:17.386 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 183376, 2023-11-16T17:01:17.386 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 5, 2023-11-16T17:01:17.386 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 3, 2023-11-16T17:01:17.386 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.010415410, 2023-11-16T17:01:17.387 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.006453173, 2023-11-16T17:01:17.387 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.001878341, 2023-11-16T17:01:17.387 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:01:17.387 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:17.387 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 178, 2023-11-16T17:01:17.388 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.029797795, 2023-11-16T17:01:17.388 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000167403 2023-11-16T17:01:17.388 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:17.388 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:17.388 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:17.389 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:17.389 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:17.389 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:17.389 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 3264, 2023-11-16T17:01:17.389 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 183376 2023-11-16T17:01:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:01:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 243, 2023-11-16T17:01:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 233, 2023-11-16T17:01:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 1427440, 2023-11-16T17:01:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 132928, 2023-11-16T17:01:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 2, 2023-11-16T17:01:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 1, 2023-11-16T17:01:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.043705516, 2023-11-16T17:01:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.013061530, 2023-11-16T17:01:17.392 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.031699179, 2023-11-16T17:01:17.392 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.006468215, 2023-11-16T17:01:17.392 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:17.392 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 232, 2023-11-16T17:01:17.392 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.016382922, 2023-11-16T17:01:17.393 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000070616 2023-11-16T17:01:17.393 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:17.393 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:17.393 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:17.393 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:17.393 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:17.394 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:17.394 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 1427440, 2023-11-16T17:01:17.394 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 132928 2023-11-16T17:01:17.394 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:17.394 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2023-11-16T17:01:17.395 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 0, 2023-11-16T17:01:17.395 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2023-11-16T17:01:17.395 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:17.395 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2023-11-16T17:01:17.395 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:01:17.396 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2023-11-16T17:01:17.396 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:01:17.396 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:01:17.396 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:01:17.396 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:01:17.396 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:01:17.397 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:01:17.397 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:01:17.397 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:01:17.397 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:01:17.397 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:01:17.398 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:01:17.398 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:01:17.398 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:01:17.398 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2023-11-16T17:01:17.398 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:01:17.399 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2023-11-16T17:01:17.399 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:01:17.399 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:01:17.399 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:01:17.399 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:01:17.400 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:01:17.400 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:01:17.400 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:01:17.400 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2023-11-16T17:01:17.400 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:01:17.400 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:01:17.401 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:01:17.401 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2023-11-16T17:01:17.401 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2023-11-16T17:01:17.401 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2023-11-16T17:01:17.401 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:01:17.402 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:01:17.402 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:01:17.402 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:01:17.402 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 148794, 2023-11-16T17:01:17.402 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 32, 2023-11-16T17:01:17.402 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:01:17.403 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2023-11-16T17:01:17.403 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2023-11-16T17:01:17.403 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2023-11-16T17:01:17.403 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:01:17.403 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2023-11-16T17:01:17.403 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:01:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2023-11-16T17:01:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 0, 2023-11-16T17:01:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 0, 2023-11-16T17:01:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:01:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:01:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2023-11-16T17:01:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2023-11-16T17:01:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 928, 2023-11-16T17:01:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 2, 2023-11-16T17:01:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2023-11-16T17:01:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2023-11-16T17:01:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2023-11-16T17:01:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2023-11-16T17:01:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-11-16T17:01:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:01:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2023-11-16T17:01:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2023-11-16T17:01:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2023-11-16T17:01:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 245, 2023-11-16T17:01:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 1393128, 2023-11-16T17:01:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2023-11-16T17:01:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 245, 2023-11-16T17:01:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2023-11-16T17:01:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2023-11-16T17:01:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 245, 2023-11-16T17:01:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 1393128, 2023-11-16T17:01:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2023-11-16T17:01:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2023-11-16T17:01:17.409 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2023-11-16T17:01:17.409 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2023-11-16T17:01:17.409 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-11-16T17:01:17.409 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-11-16T17:01:17.409 INFO:teuthology.orchestra.run.smithi077.stdout:} 2023-11-16T17:01:17.727 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:17 smithi077 ceph-mon[131204]: pgmap v353: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2023-11-16T17:01:17.846 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2023-11-16T17:01:17.846 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2023-11-16T17:01:17.846 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2023-11-16T17:01:17.846 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2023-11-16T17:01:17.847 DEBUG:tasks.check_counter:Getting stats from c 2023-11-16T17:01:17.847 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.c perf dump 2023-11-16T17:01:17.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:17 smithi044 ceph-mon[128750]: pgmap v353: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2023-11-16T17:01:17.900 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:17 smithi176 ceph-mon[131630]: pgmap v353: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2023-11-16T17:01:18.755 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T17:01:19.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:19 smithi044 ceph-mon[128750]: pgmap v354: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 203 B/s wr, 2 op/s 2023-11-16T17:01:19.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:19 smithi176 ceph-mon[131630]: pgmap v354: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 203 B/s wr, 2 op/s 2023-11-16T17:01:19.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:19 smithi077 ceph-mon[131204]: pgmap v354: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 203 B/s wr, 2 op/s 2023-11-16T17:01:21.636 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:21 smithi176 ceph-mon[131630]: pgmap v355: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2023-11-16T17:01:21.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:21 smithi044 ceph-mon[128750]: pgmap v355: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2023-11-16T17:01:21.952 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2023-11-16T17:01:21.952 INFO:teuthology.orchestra.run.smithi176.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:01:21.952 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_messages": 0, 2023-11-16T17:01:21.952 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages": 2, 2023-11-16T17:01:21.952 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_bytes": 0, 2023-11-16T17:01:21.952 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_bytes": 0, 2023-11-16T17:01:21.952 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_created_connections": 4, 2023-11-16T17:01:21.952 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_active_connections": 0, 2023-11-16T17:01:21.953 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_total_time": 0.001328438, 2023-11-16T17:01:21.953 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_send_time": 0.000000000, 2023-11-16T17:01:21.953 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_recv_time": 0.000343544, 2023-11-16T17:01:21.953 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:01:21.953 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:21.953 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:21.953 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:21.954 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:21.954 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:21.954 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:21.954 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:21.954 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:21.954 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:21.954 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:21.954 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T17:01:21.954 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T17:01:21.954 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:21.955 INFO:teuthology.orchestra.run.smithi176.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:01:21.955 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_messages": 245, 2023-11-16T17:01:21.955 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages": 233, 2023-11-16T17:01:21.955 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_bytes": 1429840, 2023-11-16T17:01:21.955 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_bytes": 132192, 2023-11-16T17:01:21.955 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_created_connections": 2, 2023-11-16T17:01:21.955 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_active_connections": 1, 2023-11-16T17:01:21.955 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_total_time": 0.044860631, 2023-11-16T17:01:21.955 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_send_time": 0.013830345, 2023-11-16T17:01:21.956 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_recv_time": 0.031126327, 2023-11-16T17:01:21.956 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_fast_dispatch_time": 0.006698607, 2023-11-16T17:01:21.956 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:21.956 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 233, 2023-11-16T17:01:21.956 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.018623942, 2023-11-16T17:01:21.956 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000079931 2023-11-16T17:01:21.956 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:21.956 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:21.956 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:21.956 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:21.957 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:21.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:21.957 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_encrypted_bytes": 1429840, 2023-11-16T17:01:21.957 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_encrypted_bytes": 132192 2023-11-16T17:01:21.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:21.957 INFO:teuthology.orchestra.run.smithi176.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:01:21.957 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_messages": 2, 2023-11-16T17:01:21.957 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages": 177, 2023-11-16T17:01:21.957 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_bytes": 320, 2023-11-16T17:01:21.958 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_bytes": 182752, 2023-11-16T17:01:21.958 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_created_connections": 5, 2023-11-16T17:01:21.958 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_active_connections": 3, 2023-11-16T17:01:21.958 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_total_time": 0.010346916, 2023-11-16T17:01:21.958 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_send_time": 0.006606480, 2023-11-16T17:01:21.958 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_recv_time": 0.001494905, 2023-11-16T17:01:21.958 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:01:21.958 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:21.959 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 174, 2023-11-16T17:01:21.959 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.028148340, 2023-11-16T17:01:21.959 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000161772 2023-11-16T17:01:21.959 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:21.959 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:21.959 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:21.959 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:21.959 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:21.959 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:21.960 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T17:01:21.960 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_encrypted_bytes": 182752 2023-11-16T17:01:21.960 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:21.960 INFO:teuthology.orchestra.run.smithi176.stdout: "cct": { 2023-11-16T17:01:21.960 INFO:teuthology.orchestra.run.smithi176.stdout: "total_workers": 0, 2023-11-16T17:01:21.960 INFO:teuthology.orchestra.run.smithi176.stdout: "unhealthy_workers": 0 2023-11-16T17:01:21.960 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:21.960 INFO:teuthology.orchestra.run.smithi176.stdout: "mempool": { 2023-11-16T17:01:21.960 INFO:teuthology.orchestra.run.smithi176.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:01:21.961 INFO:teuthology.orchestra.run.smithi176.stdout: "bloom_filter_items": 0, 2023-11-16T17:01:21.961 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:01:21.961 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:01:21.961 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:01:21.961 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:01:21.961 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:01:21.961 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:01:21.961 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:01:21.961 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:01:21.962 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:01:21.962 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:01:21.962 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:01:21.962 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:01:21.962 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:01:21.962 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_extent_items": 0, 2023-11-16T17:01:21.962 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:01:21.962 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_blob_items": 0, 2023-11-16T17:01:21.962 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:01:21.963 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:01:21.963 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:01:21.963 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:01:21.963 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:01:21.963 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:01:21.963 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:01:21.963 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_txc_items": 0, 2023-11-16T17:01:21.963 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:01:21.963 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:01:21.964 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:01:21.964 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_writing_items": 0, 2023-11-16T17:01:21.964 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_bytes": 0, 2023-11-16T17:01:21.964 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_items": 0, 2023-11-16T17:01:21.964 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:01:21.964 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:01:21.964 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:01:21.964 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:01:21.964 INFO:teuthology.orchestra.run.smithi176.stdout: "buffer_anon_bytes": 149017, 2023-11-16T17:01:21.965 INFO:teuthology.orchestra.run.smithi176.stdout: "buffer_anon_items": 31, 2023-11-16T17:01:21.965 INFO:teuthology.orchestra.run.smithi176.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:01:21.965 INFO:teuthology.orchestra.run.smithi176.stdout: "buffer_meta_items": 0, 2023-11-16T17:01:21.965 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_bytes": 0, 2023-11-16T17:01:21.965 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_items": 0, 2023-11-16T17:01:21.965 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:01:21.965 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_mapbl_items": 0, 2023-11-16T17:01:21.965 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:01:21.965 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_pglog_items": 0, 2023-11-16T17:01:21.966 INFO:teuthology.orchestra.run.smithi176.stdout: "osdmap_bytes": 0, 2023-11-16T17:01:21.966 INFO:teuthology.orchestra.run.smithi176.stdout: "osdmap_items": 0, 2023-11-16T17:01:21.966 INFO:teuthology.orchestra.run.smithi176.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:01:21.966 INFO:teuthology.orchestra.run.smithi176.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:01:21.966 INFO:teuthology.orchestra.run.smithi176.stdout: "pgmap_bytes": 0, 2023-11-16T17:01:21.966 INFO:teuthology.orchestra.run.smithi176.stdout: "pgmap_items": 0, 2023-11-16T17:01:21.966 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_co_bytes": 928, 2023-11-16T17:01:21.966 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_co_items": 2, 2023-11-16T17:01:21.966 INFO:teuthology.orchestra.run.smithi176.stdout: "unittest_1_bytes": 0, 2023-11-16T17:01:21.967 INFO:teuthology.orchestra.run.smithi176.stdout: "unittest_1_items": 0, 2023-11-16T17:01:21.967 INFO:teuthology.orchestra.run.smithi176.stdout: "unittest_2_bytes": 0, 2023-11-16T17:01:21.967 INFO:teuthology.orchestra.run.smithi176.stdout: "unittest_2_items": 0 2023-11-16T17:01:21.967 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:21.967 INFO:teuthology.orchestra.run.smithi176.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:01:21.967 INFO:teuthology.orchestra.run.smithi176.stdout: "val": 0, 2023-11-16T17:01:21.967 INFO:teuthology.orchestra.run.smithi176.stdout: "max": 104857600, 2023-11-16T17:01:21.967 INFO:teuthology.orchestra.run.smithi176.stdout: "get_started": 0, 2023-11-16T17:01:21.967 INFO:teuthology.orchestra.run.smithi176.stdout: "get": 243, 2023-11-16T17:01:21.967 INFO:teuthology.orchestra.run.smithi176.stdout: "get_sum": 1392868, 2023-11-16T17:01:21.968 INFO:teuthology.orchestra.run.smithi176.stdout: "get_or_fail_fail": 0, 2023-11-16T17:01:21.968 INFO:teuthology.orchestra.run.smithi176.stdout: "get_or_fail_success": 243, 2023-11-16T17:01:21.968 INFO:teuthology.orchestra.run.smithi176.stdout: "take": 0, 2023-11-16T17:01:21.968 INFO:teuthology.orchestra.run.smithi176.stdout: "take_sum": 0, 2023-11-16T17:01:21.968 INFO:teuthology.orchestra.run.smithi176.stdout: "put": 243, 2023-11-16T17:01:21.968 INFO:teuthology.orchestra.run.smithi176.stdout: "put_sum": 1392868, 2023-11-16T17:01:21.968 INFO:teuthology.orchestra.run.smithi176.stdout: "wait": { 2023-11-16T17:01:21.968 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:21.968 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:21.969 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:21.969 INFO:teuthology.orchestra.run.smithi176.stdout: } 2023-11-16T17:01:21.969 INFO:teuthology.orchestra.run.smithi176.stdout: } 2023-11-16T17:01:21.969 INFO:teuthology.orchestra.run.smithi176.stdout:} 2023-11-16T17:01:21.981 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:21 smithi077 ceph-mon[131204]: pgmap v355: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2023-11-16T17:01:22.472 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.c 2023-11-16T17:01:22.472 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.c 2023-11-16T17:01:22.472 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.c 2023-11-16T17:01:22.472 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.c 2023-11-16T17:01:22.473 DEBUG:tasks.check_counter:Getting stats from f 2023-11-16T17:01:22.473 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.f perf dump 2023-11-16T17:01:23.290 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T17:01:23.807 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:23 smithi176 ceph-mon[131630]: pgmap v356: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2023-11-16T17:01:23.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:23 smithi044 ceph-mon[128750]: pgmap v356: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2023-11-16T17:01:23.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:23 smithi077 ceph-mon[131204]: pgmap v356: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2023-11-16T17:01:24.850 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2023-11-16T17:01:24.850 INFO:teuthology.orchestra.run.smithi176.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:01:24.850 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_messages": 0, 2023-11-16T17:01:24.850 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages": 2, 2023-11-16T17:01:24.851 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_bytes": 0, 2023-11-16T17:01:24.851 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_bytes": 0, 2023-11-16T17:01:24.851 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_created_connections": 4, 2023-11-16T17:01:24.851 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_active_connections": 0, 2023-11-16T17:01:24.851 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_total_time": 0.001216634, 2023-11-16T17:01:24.852 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_send_time": 0.000000000, 2023-11-16T17:01:24.852 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_recv_time": 0.000291708, 2023-11-16T17:01:24.852 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:01:24.852 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:24.852 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:24.852 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:24.853 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:24.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:24.853 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:24.853 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:24.853 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:24.853 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:24.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:24.854 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T17:01:24.854 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T17:01:24.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:24.854 INFO:teuthology.orchestra.run.smithi176.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:01:24.855 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_messages": 2, 2023-11-16T17:01:24.855 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages": 177, 2023-11-16T17:01:24.855 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_bytes": 320, 2023-11-16T17:01:24.855 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_bytes": 182752, 2023-11-16T17:01:24.855 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_created_connections": 5, 2023-11-16T17:01:24.856 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_active_connections": 3, 2023-11-16T17:01:24.856 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_total_time": 0.009782644, 2023-11-16T17:01:24.856 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_send_time": 0.006479643, 2023-11-16T17:01:24.856 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_recv_time": 0.001196783, 2023-11-16T17:01:24.856 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:01:24.857 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:24.857 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 174, 2023-11-16T17:01:24.857 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.026628429, 2023-11-16T17:01:24.857 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000153036 2023-11-16T17:01:24.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:24.857 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:24.858 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:24.858 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:24.858 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:24.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:24.858 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T17:01:24.859 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_encrypted_bytes": 182752 2023-11-16T17:01:24.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:24.859 INFO:teuthology.orchestra.run.smithi176.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:01:24.859 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_messages": 246, 2023-11-16T17:01:24.859 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages": 234, 2023-11-16T17:01:24.859 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_bytes": 1430112, 2023-11-16T17:01:24.860 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_bytes": 132784, 2023-11-16T17:01:24.860 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_created_connections": 2, 2023-11-16T17:01:24.860 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_active_connections": 1, 2023-11-16T17:01:24.860 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_total_time": 0.044842869, 2023-11-16T17:01:24.860 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_send_time": 0.013554369, 2023-11-16T17:01:24.860 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_recv_time": 0.030840596, 2023-11-16T17:01:24.861 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_fast_dispatch_time": 0.007013452, 2023-11-16T17:01:24.861 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:24.861 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 234, 2023-11-16T17:01:24.861 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.013314322, 2023-11-16T17:01:24.861 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000056898 2023-11-16T17:01:24.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:24.862 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:24.862 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:24.862 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:24.862 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:24.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:24.863 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_encrypted_bytes": 1430112, 2023-11-16T17:01:24.863 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_encrypted_bytes": 132784 2023-11-16T17:01:24.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:24.863 INFO:teuthology.orchestra.run.smithi176.stdout: "cct": { 2023-11-16T17:01:24.863 INFO:teuthology.orchestra.run.smithi176.stdout: "total_workers": 0, 2023-11-16T17:01:24.863 INFO:teuthology.orchestra.run.smithi176.stdout: "unhealthy_workers": 0 2023-11-16T17:01:24.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:24.864 INFO:teuthology.orchestra.run.smithi176.stdout: "mempool": { 2023-11-16T17:01:24.864 INFO:teuthology.orchestra.run.smithi176.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:01:24.864 INFO:teuthology.orchestra.run.smithi176.stdout: "bloom_filter_items": 0, 2023-11-16T17:01:24.864 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:01:24.864 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:01:24.865 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:01:24.865 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:01:24.865 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:01:24.865 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:01:24.865 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:01:24.865 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:01:24.866 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:01:24.866 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:01:24.866 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:01:24.866 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:01:24.866 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:01:24.867 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_extent_items": 0, 2023-11-16T17:01:24.867 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:01:24.867 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_blob_items": 0, 2023-11-16T17:01:24.867 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:01:24.867 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:01:24.867 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:01:24.868 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:01:24.868 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:01:24.868 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:01:24.868 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:01:24.868 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_txc_items": 0, 2023-11-16T17:01:24.868 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:01:24.868 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:01:24.869 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:01:24.869 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_writing_items": 0, 2023-11-16T17:01:24.869 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_bytes": 0, 2023-11-16T17:01:24.869 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_items": 0, 2023-11-16T17:01:24.869 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:01:24.870 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:01:24.870 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:01:24.870 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:01:24.870 INFO:teuthology.orchestra.run.smithi176.stdout: "buffer_anon_bytes": 149034, 2023-11-16T17:01:24.870 INFO:teuthology.orchestra.run.smithi176.stdout: "buffer_anon_items": 32, 2023-11-16T17:01:24.870 INFO:teuthology.orchestra.run.smithi176.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:01:24.871 INFO:teuthology.orchestra.run.smithi176.stdout: "buffer_meta_items": 0, 2023-11-16T17:01:24.871 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_bytes": 0, 2023-11-16T17:01:24.871 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_items": 0, 2023-11-16T17:01:24.871 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:01:24.871 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_mapbl_items": 0, 2023-11-16T17:01:24.871 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:01:24.872 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_pglog_items": 0, 2023-11-16T17:01:24.872 INFO:teuthology.orchestra.run.smithi176.stdout: "osdmap_bytes": 0, 2023-11-16T17:01:24.872 INFO:teuthology.orchestra.run.smithi176.stdout: "osdmap_items": 0, 2023-11-16T17:01:24.872 INFO:teuthology.orchestra.run.smithi176.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:01:24.872 INFO:teuthology.orchestra.run.smithi176.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:01:24.872 INFO:teuthology.orchestra.run.smithi176.stdout: "pgmap_bytes": 0, 2023-11-16T17:01:24.873 INFO:teuthology.orchestra.run.smithi176.stdout: "pgmap_items": 0, 2023-11-16T17:01:24.873 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_co_bytes": 928, 2023-11-16T17:01:24.873 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_co_items": 2, 2023-11-16T17:01:24.873 INFO:teuthology.orchestra.run.smithi176.stdout: "unittest_1_bytes": 0, 2023-11-16T17:01:24.873 INFO:teuthology.orchestra.run.smithi176.stdout: "unittest_1_items": 0, 2023-11-16T17:01:24.873 INFO:teuthology.orchestra.run.smithi176.stdout: "unittest_2_bytes": 0, 2023-11-16T17:01:24.874 INFO:teuthology.orchestra.run.smithi176.stdout: "unittest_2_items": 0 2023-11-16T17:01:24.874 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:24.874 INFO:teuthology.orchestra.run.smithi176.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:01:24.874 INFO:teuthology.orchestra.run.smithi176.stdout: "val": 0, 2023-11-16T17:01:24.874 INFO:teuthology.orchestra.run.smithi176.stdout: "max": 104857600, 2023-11-16T17:01:24.874 INFO:teuthology.orchestra.run.smithi176.stdout: "get_started": 0, 2023-11-16T17:01:24.875 INFO:teuthology.orchestra.run.smithi176.stdout: "get": 244, 2023-11-16T17:01:24.875 INFO:teuthology.orchestra.run.smithi176.stdout: "get_sum": 1392998, 2023-11-16T17:01:24.875 INFO:teuthology.orchestra.run.smithi176.stdout: "get_or_fail_fail": 0, 2023-11-16T17:01:24.875 INFO:teuthology.orchestra.run.smithi176.stdout: "get_or_fail_success": 244, 2023-11-16T17:01:24.875 INFO:teuthology.orchestra.run.smithi176.stdout: "take": 0, 2023-11-16T17:01:24.875 INFO:teuthology.orchestra.run.smithi176.stdout: "take_sum": 0, 2023-11-16T17:01:24.875 INFO:teuthology.orchestra.run.smithi176.stdout: "put": 244, 2023-11-16T17:01:24.876 INFO:teuthology.orchestra.run.smithi176.stdout: "put_sum": 1392998, 2023-11-16T17:01:24.876 INFO:teuthology.orchestra.run.smithi176.stdout: "wait": { 2023-11-16T17:01:24.876 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:24.876 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:24.876 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:24.876 INFO:teuthology.orchestra.run.smithi176.stdout: } 2023-11-16T17:01:24.877 INFO:teuthology.orchestra.run.smithi176.stdout: } 2023-11-16T17:01:24.877 INFO:teuthology.orchestra.run.smithi176.stdout:} 2023-11-16T17:01:25.260 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.f 2023-11-16T17:01:25.261 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.f 2023-11-16T17:01:25.261 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.f 2023-11-16T17:01:25.261 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.f 2023-11-16T17:01:25.261 DEBUG:tasks.check_counter:Getting stats from i 2023-11-16T17:01:25.261 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.i perf dump 2023-11-16T17:01:25.552 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:25 smithi176 ceph-mon[131630]: pgmap v357: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2023-11-16T17:01:25.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:25 smithi044 ceph-mon[128750]: pgmap v357: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2023-11-16T17:01:25.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:25 smithi077 ceph-mon[131204]: pgmap v357: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2023-11-16T17:01:26.176 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T17:01:27.752 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2023-11-16T17:01:27.752 INFO:teuthology.orchestra.run.smithi176.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:01:27.752 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_messages": 1413, 2023-11-16T17:01:27.752 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages": 1402, 2023-11-16T17:01:27.753 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_bytes": 243716696, 2023-11-16T17:01:27.753 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_bytes": 493578, 2023-11-16T17:01:27.753 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_created_connections": 7, 2023-11-16T17:01:27.753 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_active_connections": 3, 2023-11-16T17:01:27.753 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_total_time": 0.248731450, 2023-11-16T17:01:27.754 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_send_time": 0.046705088, 2023-11-16T17:01:27.754 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_recv_time": 0.159084054, 2023-11-16T17:01:27.754 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_fast_dispatch_time": 0.040482358, 2023-11-16T17:01:27.754 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:27.754 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 1401, 2023-11-16T17:01:27.755 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.096895580, 2023-11-16T17:01:27.755 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000069161 2023-11-16T17:01:27.755 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.755 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:27.755 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.756 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.756 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.756 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_encrypted_bytes": 1457600, 2023-11-16T17:01:27.756 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_encrypted_bytes": 135616 2023-11-16T17:01:27.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.757 INFO:teuthology.orchestra.run.smithi176.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:01:27.757 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_messages": 1896, 2023-11-16T17:01:27.757 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages": 2069, 2023-11-16T17:01:27.757 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_bytes": 344552752, 2023-11-16T17:01:27.758 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_bytes": 1040075, 2023-11-16T17:01:27.758 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_created_connections": 9, 2023-11-16T17:01:27.758 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_active_connections": 7, 2023-11-16T17:01:27.758 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_total_time": 0.351919981, 2023-11-16T17:01:27.759 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_send_time": 0.055300982, 2023-11-16T17:01:27.759 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_recv_time": 0.214359625, 2023-11-16T17:01:27.759 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_fast_dispatch_time": 0.063973740, 2023-11-16T17:01:27.759 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:27.759 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 2067, 2023-11-16T17:01:27.760 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.155914156, 2023-11-16T17:01:27.760 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000075430 2023-11-16T17:01:27.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.760 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:27.760 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.761 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.761 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.761 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_encrypted_bytes": 3264, 2023-11-16T17:01:27.761 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_encrypted_bytes": 452224 2023-11-16T17:01:27.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.762 INFO:teuthology.orchestra.run.smithi176.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:01:27.762 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_messages": 695, 2023-11-16T17:01:27.762 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages": 697, 2023-11-16T17:01:27.762 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_bytes": 184760357, 2023-11-16T17:01:27.763 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_bytes": 216160, 2023-11-16T17:01:27.763 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_created_connections": 6, 2023-11-16T17:01:27.763 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_active_connections": 5, 2023-11-16T17:01:27.763 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_total_time": 0.147351270, 2023-11-16T17:01:27.763 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_send_time": 0.018010572, 2023-11-16T17:01:27.764 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_recv_time": 0.102329623, 2023-11-16T17:01:27.764 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_fast_dispatch_time": 0.021921781, 2023-11-16T17:01:27.764 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:27.764 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 695, 2023-11-16T17:01:27.764 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.058927185, 2023-11-16T17:01:27.765 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000084787 2023-11-16T17:01:27.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.765 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:27.765 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.765 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.766 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.766 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T17:01:27.766 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T17:01:27.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.767 INFO:teuthology.orchestra.run.smithi176.stdout: "cct": { 2023-11-16T17:01:27.767 INFO:teuthology.orchestra.run.smithi176.stdout: "total_workers": 1, 2023-11-16T17:01:27.767 INFO:teuthology.orchestra.run.smithi176.stdout: "unhealthy_workers": 0 2023-11-16T17:01:27.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.767 INFO:teuthology.orchestra.run.smithi176.stdout: "finisher-MDSRank": { 2023-11-16T17:01:27.768 INFO:teuthology.orchestra.run.smithi176.stdout: "queue_len": 0, 2023-11-16T17:01:27.768 INFO:teuthology.orchestra.run.smithi176.stdout: "complete_latency": { 2023-11-16T17:01:27.768 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 5496, 2023-11-16T17:01:27.768 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 1.506441925, 2023-11-16T17:01:27.768 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000274097 2023-11-16T17:01:27.769 INFO:teuthology.orchestra.run.smithi176.stdout: } 2023-11-16T17:01:27.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.769 INFO:teuthology.orchestra.run.smithi176.stdout: "finisher-PurgeQueue": { 2023-11-16T17:01:27.769 INFO:teuthology.orchestra.run.smithi176.stdout: "queue_len": 0, 2023-11-16T17:01:27.769 INFO:teuthology.orchestra.run.smithi176.stdout: "complete_latency": { 2023-11-16T17:01:27.770 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.770 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.770 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.770 INFO:teuthology.orchestra.run.smithi176.stdout: } 2023-11-16T17:01:27.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.771 INFO:teuthology.orchestra.run.smithi176.stdout: "mds": { 2023-11-16T17:01:27.771 INFO:teuthology.orchestra.run.smithi176.stdout: "request": 0, 2023-11-16T17:01:27.771 INFO:teuthology.orchestra.run.smithi176.stdout: "reply": 0, 2023-11-16T17:01:27.771 INFO:teuthology.orchestra.run.smithi176.stdout: "reply_latency": { 2023-11-16T17:01:27.771 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.772 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.772 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.772 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.772 INFO:teuthology.orchestra.run.smithi176.stdout: "slow_reply": 0, 2023-11-16T17:01:27.772 INFO:teuthology.orchestra.run.smithi176.stdout: "forward": 0, 2023-11-16T17:01:27.772 INFO:teuthology.orchestra.run.smithi176.stdout: "dir_fetch_complete": 0, 2023-11-16T17:01:27.773 INFO:teuthology.orchestra.run.smithi176.stdout: "dir_fetch_keys": 0, 2023-11-16T17:01:27.773 INFO:teuthology.orchestra.run.smithi176.stdout: "dir_commit": 0, 2023-11-16T17:01:27.773 INFO:teuthology.orchestra.run.smithi176.stdout: "dir_split": 0, 2023-11-16T17:01:27.773 INFO:teuthology.orchestra.run.smithi176.stdout: "dir_merge": 0, 2023-11-16T17:01:27.773 INFO:teuthology.orchestra.run.smithi176.stdout: "inodes": 163484, 2023-11-16T17:01:27.774 INFO:teuthology.orchestra.run.smithi176.stdout: "inodes_top": 114421, 2023-11-16T17:01:27.774 INFO:teuthology.orchestra.run.smithi176.stdout: "inodes_bottom": 49055, 2023-11-16T17:01:27.774 INFO:teuthology.orchestra.run.smithi176.stdout: "inodes_pin_tail": 8, 2023-11-16T17:01:27.774 INFO:teuthology.orchestra.run.smithi176.stdout: "inodes_pinned": 25, 2023-11-16T17:01:27.774 INFO:teuthology.orchestra.run.smithi176.stdout: "inodes_expired": 3359, 2023-11-16T17:01:27.775 INFO:teuthology.orchestra.run.smithi176.stdout: "inodes_with_caps": 0, 2023-11-16T17:01:27.775 INFO:teuthology.orchestra.run.smithi176.stdout: "caps": 0, 2023-11-16T17:01:27.775 INFO:teuthology.orchestra.run.smithi176.stdout: "subtrees": 2, 2023-11-16T17:01:27.775 INFO:teuthology.orchestra.run.smithi176.stdout: "traverse": 0, 2023-11-16T17:01:27.775 INFO:teuthology.orchestra.run.smithi176.stdout: "traverse_hit": 0, 2023-11-16T17:01:27.776 INFO:teuthology.orchestra.run.smithi176.stdout: "traverse_forward": 0, 2023-11-16T17:01:27.776 INFO:teuthology.orchestra.run.smithi176.stdout: "traverse_discover": 0, 2023-11-16T17:01:27.776 INFO:teuthology.orchestra.run.smithi176.stdout: "traverse_dir_fetch": 0, 2023-11-16T17:01:27.776 INFO:teuthology.orchestra.run.smithi176.stdout: "traverse_remote_ino": 0, 2023-11-16T17:01:27.776 INFO:teuthology.orchestra.run.smithi176.stdout: "traverse_lock": 0, 2023-11-16T17:01:27.777 INFO:teuthology.orchestra.run.smithi176.stdout: "load_cent": 0, 2023-11-16T17:01:27.777 INFO:teuthology.orchestra.run.smithi176.stdout: "q": 0, 2023-11-16T17:01:27.777 INFO:teuthology.orchestra.run.smithi176.stdout: "exported": 0, 2023-11-16T17:01:27.777 INFO:teuthology.orchestra.run.smithi176.stdout: "exported_inodes": 0, 2023-11-16T17:01:27.777 INFO:teuthology.orchestra.run.smithi176.stdout: "imported": 0, 2023-11-16T17:01:27.777 INFO:teuthology.orchestra.run.smithi176.stdout: "imported_inodes": 0, 2023-11-16T17:01:27.778 INFO:teuthology.orchestra.run.smithi176.stdout: "openino_dir_fetch": 0, 2023-11-16T17:01:27.778 INFO:teuthology.orchestra.run.smithi176.stdout: "openino_backtrace_fetch": 0, 2023-11-16T17:01:27.778 INFO:teuthology.orchestra.run.smithi176.stdout: "openino_peer_discover": 0, 2023-11-16T17:01:27.778 INFO:teuthology.orchestra.run.smithi176.stdout: "root_rfiles": 4, 2023-11-16T17:01:27.779 INFO:teuthology.orchestra.run.smithi176.stdout: "root_rbytes": 246, 2023-11-16T17:01:27.779 INFO:teuthology.orchestra.run.smithi176.stdout: "root_rsnaps": 6, 2023-11-16T17:01:27.779 INFO:teuthology.orchestra.run.smithi176.stdout: "scrub_backtrace_fetch": 0, 2023-11-16T17:01:27.779 INFO:teuthology.orchestra.run.smithi176.stdout: "scrub_set_tag": 0, 2023-11-16T17:01:27.780 INFO:teuthology.orchestra.run.smithi176.stdout: "scrub_backtrace_repaired": 0, 2023-11-16T17:01:27.780 INFO:teuthology.orchestra.run.smithi176.stdout: "scrub_inotable_repaired": 0, 2023-11-16T17:01:27.780 INFO:teuthology.orchestra.run.smithi176.stdout: "scrub_dir_inodes": 0, 2023-11-16T17:01:27.780 INFO:teuthology.orchestra.run.smithi176.stdout: "scrub_dir_base_inodes": 0, 2023-11-16T17:01:27.780 INFO:teuthology.orchestra.run.smithi176.stdout: "scrub_dirfrag_rstats": 0, 2023-11-16T17:01:27.781 INFO:teuthology.orchestra.run.smithi176.stdout: "scrub_file_inodes": 0, 2023-11-16T17:01:27.781 INFO:teuthology.orchestra.run.smithi176.stdout: "handle_inode_file_caps": 0, 2023-11-16T17:01:27.781 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_cap_op_revoke": 0, 2023-11-16T17:01:27.781 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_cap_op_grant": 0, 2023-11-16T17:01:27.782 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_cap_op_trunc": 0, 2023-11-16T17:01:27.782 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-16T17:01:27.782 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-16T17:01:27.782 INFO:teuthology.orchestra.run.smithi176.stdout: "handle_client_caps": 0, 2023-11-16T17:01:27.783 INFO:teuthology.orchestra.run.smithi176.stdout: "handle_client_caps_dirty": 0, 2023-11-16T17:01:27.783 INFO:teuthology.orchestra.run.smithi176.stdout: "handle_client_cap_release": 0, 2023-11-16T17:01:27.783 INFO:teuthology.orchestra.run.smithi176.stdout: "process_request_cap_release": 0 2023-11-16T17:01:27.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.783 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_cache": { 2023-11-16T17:01:27.784 INFO:teuthology.orchestra.run.smithi176.stdout: "dir_update": 0, 2023-11-16T17:01:27.784 INFO:teuthology.orchestra.run.smithi176.stdout: "dir_update_receipt": 0, 2023-11-16T17:01:27.784 INFO:teuthology.orchestra.run.smithi176.stdout: "dir_try_discover": 0, 2023-11-16T17:01:27.784 INFO:teuthology.orchestra.run.smithi176.stdout: "dir_send_discover": 0, 2023-11-16T17:01:27.785 INFO:teuthology.orchestra.run.smithi176.stdout: "dir_handle_discover": 0, 2023-11-16T17:01:27.785 INFO:teuthology.orchestra.run.smithi176.stdout: "num_strays": 0, 2023-11-16T17:01:27.785 INFO:teuthology.orchestra.run.smithi176.stdout: "num_strays_delayed": 0, 2023-11-16T17:01:27.785 INFO:teuthology.orchestra.run.smithi176.stdout: "num_strays_enqueuing": 0, 2023-11-16T17:01:27.785 INFO:teuthology.orchestra.run.smithi176.stdout: "strays_created": 0, 2023-11-16T17:01:27.786 INFO:teuthology.orchestra.run.smithi176.stdout: "strays_enqueued": 0, 2023-11-16T17:01:27.786 INFO:teuthology.orchestra.run.smithi176.stdout: "strays_reintegrated": 0, 2023-11-16T17:01:27.786 INFO:teuthology.orchestra.run.smithi176.stdout: "strays_migrated": 0, 2023-11-16T17:01:27.786 INFO:teuthology.orchestra.run.smithi176.stdout: "num_recovering_processing": 0, 2023-11-16T17:01:27.787 INFO:teuthology.orchestra.run.smithi176.stdout: "num_recovering_enqueued": 0, 2023-11-16T17:01:27.787 INFO:teuthology.orchestra.run.smithi176.stdout: "num_recovering_prioritized": 0, 2023-11-16T17:01:27.787 INFO:teuthology.orchestra.run.smithi176.stdout: "recovery_started": 0, 2023-11-16T17:01:27.787 INFO:teuthology.orchestra.run.smithi176.stdout: "recovery_completed": 0, 2023-11-16T17:01:27.787 INFO:teuthology.orchestra.run.smithi176.stdout: "ireq_enqueue_scrub": 0, 2023-11-16T17:01:27.788 INFO:teuthology.orchestra.run.smithi176.stdout: "ireq_exportdir": 0, 2023-11-16T17:01:27.788 INFO:teuthology.orchestra.run.smithi176.stdout: "ireq_flush": 0, 2023-11-16T17:01:27.788 INFO:teuthology.orchestra.run.smithi176.stdout: "ireq_fragmentdir": 0, 2023-11-16T17:01:27.788 INFO:teuthology.orchestra.run.smithi176.stdout: "ireq_fragstats": 0, 2023-11-16T17:01:27.788 INFO:teuthology.orchestra.run.smithi176.stdout: "ireq_inodestats": 0 2023-11-16T17:01:27.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.789 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_log": { 2023-11-16T17:01:27.789 INFO:teuthology.orchestra.run.smithi176.stdout: "evadd": 0, 2023-11-16T17:01:27.789 INFO:teuthology.orchestra.run.smithi176.stdout: "evex": 0, 2023-11-16T17:01:27.789 INFO:teuthology.orchestra.run.smithi176.stdout: "evtrm": 0, 2023-11-16T17:01:27.790 INFO:teuthology.orchestra.run.smithi176.stdout: "ev": 0, 2023-11-16T17:01:27.790 INFO:teuthology.orchestra.run.smithi176.stdout: "evexg": 0, 2023-11-16T17:01:27.790 INFO:teuthology.orchestra.run.smithi176.stdout: "evexd": 0, 2023-11-16T17:01:27.790 INFO:teuthology.orchestra.run.smithi176.stdout: "segadd": 0, 2023-11-16T17:01:27.791 INFO:teuthology.orchestra.run.smithi176.stdout: "segex": 0, 2023-11-16T17:01:27.791 INFO:teuthology.orchestra.run.smithi176.stdout: "segtrm": 0, 2023-11-16T17:01:27.791 INFO:teuthology.orchestra.run.smithi176.stdout: "seg": 2, 2023-11-16T17:01:27.791 INFO:teuthology.orchestra.run.smithi176.stdout: "segexg": 0, 2023-11-16T17:01:27.792 INFO:teuthology.orchestra.run.smithi176.stdout: "segexd": 0, 2023-11-16T17:01:27.792 INFO:teuthology.orchestra.run.smithi176.stdout: "expos": 774436220, 2023-11-16T17:01:27.792 INFO:teuthology.orchestra.run.smithi176.stdout: "wrpos": 0, 2023-11-16T17:01:27.792 INFO:teuthology.orchestra.run.smithi176.stdout: "rdpos": 774581079, 2023-11-16T17:01:27.792 INFO:teuthology.orchestra.run.smithi176.stdout: "jlat": { 2023-11-16T17:01:27.793 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.793 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.793 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.794 INFO:teuthology.orchestra.run.smithi176.stdout: "replayed": 226589 2023-11-16T17:01:27.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.794 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_mem": { 2023-11-16T17:01:27.794 INFO:teuthology.orchestra.run.smithi176.stdout: "ino": 66711, 2023-11-16T17:01:27.794 INFO:teuthology.orchestra.run.smithi176.stdout: "ino+": 116825, 2023-11-16T17:01:27.795 INFO:teuthology.orchestra.run.smithi176.stdout: "ino-": 50114, 2023-11-16T17:01:27.795 INFO:teuthology.orchestra.run.smithi176.stdout: "dir": 2614, 2023-11-16T17:01:27.795 INFO:teuthology.orchestra.run.smithi176.stdout: "dir+": 3179, 2023-11-16T17:01:27.795 INFO:teuthology.orchestra.run.smithi176.stdout: "dir-": 565, 2023-11-16T17:01:27.796 INFO:teuthology.orchestra.run.smithi176.stdout: "dn": 163484, 2023-11-16T17:01:27.796 INFO:teuthology.orchestra.run.smithi176.stdout: "dn+": 166851, 2023-11-16T17:01:27.796 INFO:teuthology.orchestra.run.smithi176.stdout: "dn-": 3367, 2023-11-16T17:01:27.796 INFO:teuthology.orchestra.run.smithi176.stdout: "cap": 0, 2023-11-16T17:01:27.796 INFO:teuthology.orchestra.run.smithi176.stdout: "cap+": 0, 2023-11-16T17:01:27.797 INFO:teuthology.orchestra.run.smithi176.stdout: "cap-": 0, 2023-11-16T17:01:27.797 INFO:teuthology.orchestra.run.smithi176.stdout: "rss": 388800, 2023-11-16T17:01:27.797 INFO:teuthology.orchestra.run.smithi176.stdout: "heap": 207156 2023-11-16T17:01:27.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.798 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_server": { 2023-11-16T17:01:27.798 INFO:teuthology.orchestra.run.smithi176.stdout: "dispatch_client_request": 0, 2023-11-16T17:01:27.798 INFO:teuthology.orchestra.run.smithi176.stdout: "dispatch_server_request": 0, 2023-11-16T17:01:27.798 INFO:teuthology.orchestra.run.smithi176.stdout: "handle_client_request": 0, 2023-11-16T17:01:27.799 INFO:teuthology.orchestra.run.smithi176.stdout: "handle_client_session": 0, 2023-11-16T17:01:27.799 INFO:teuthology.orchestra.run.smithi176.stdout: "handle_peer_request": 0, 2023-11-16T17:01:27.799 INFO:teuthology.orchestra.run.smithi176.stdout: "req_create_latency": { 2023-11-16T17:01:27.799 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.799 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.800 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.800 INFO:teuthology.orchestra.run.smithi176.stdout: "req_getattr_latency": { 2023-11-16T17:01:27.800 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.801 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.801 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.801 INFO:teuthology.orchestra.run.smithi176.stdout: "req_getfilelock_latency": { 2023-11-16T17:01:27.802 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.802 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.802 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.803 INFO:teuthology.orchestra.run.smithi176.stdout: "req_link_latency": { 2023-11-16T17:01:27.803 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.803 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.803 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.804 INFO:teuthology.orchestra.run.smithi176.stdout: "req_lookup_latency": { 2023-11-16T17:01:27.804 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.804 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.804 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.805 INFO:teuthology.orchestra.run.smithi176.stdout: "req_lookuphash_latency": { 2023-11-16T17:01:27.805 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.805 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.805 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.806 INFO:teuthology.orchestra.run.smithi176.stdout: "req_lookupino_latency": { 2023-11-16T17:01:27.806 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.806 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.807 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.807 INFO:teuthology.orchestra.run.smithi176.stdout: "req_lookupname_latency": { 2023-11-16T17:01:27.807 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.808 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.808 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.808 INFO:teuthology.orchestra.run.smithi176.stdout: "req_lookupparent_latency": { 2023-11-16T17:01:27.808 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.809 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.809 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.809 INFO:teuthology.orchestra.run.smithi176.stdout: "req_lookupsnap_latency": { 2023-11-16T17:01:27.810 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.810 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.810 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.811 INFO:teuthology.orchestra.run.smithi176.stdout: "req_lssnap_latency": { 2023-11-16T17:01:27.811 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.811 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.811 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.812 INFO:teuthology.orchestra.run.smithi176.stdout: "req_mkdir_latency": { 2023-11-16T17:01:27.812 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.812 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.813 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.813 INFO:teuthology.orchestra.run.smithi176.stdout: "req_mknod_latency": { 2023-11-16T17:01:27.813 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.813 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.814 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.814 INFO:teuthology.orchestra.run.smithi176.stdout: "req_mksnap_latency": { 2023-11-16T17:01:27.814 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.815 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.815 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.815 INFO:teuthology.orchestra.run.smithi176.stdout: "req_open_latency": { 2023-11-16T17:01:27.816 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.816 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.816 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.816 INFO:teuthology.orchestra.run.smithi176.stdout: "req_readdir_latency": { 2023-11-16T17:01:27.817 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.817 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.817 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.818 INFO:teuthology.orchestra.run.smithi176.stdout: "req_rename_latency": { 2023-11-16T17:01:27.818 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.818 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.818 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.819 INFO:teuthology.orchestra.run.smithi176.stdout: "req_renamesnap_latency": { 2023-11-16T17:01:27.819 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.819 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.819 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.820 INFO:teuthology.orchestra.run.smithi176.stdout: "req_snapdiff_latency": { 2023-11-16T17:01:27.820 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.820 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.821 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.821 INFO:teuthology.orchestra.run.smithi176.stdout: "req_rmdir_latency": { 2023-11-16T17:01:27.821 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.821 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.821 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.821 INFO:teuthology.orchestra.run.smithi176.stdout: "req_rmsnap_latency": { 2023-11-16T17:01:27.821 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: "req_rmxattr_latency": { 2023-11-16T17:01:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: "req_setattr_latency": { 2023-11-16T17:01:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: "req_setdirlayout_latency": { 2023-11-16T17:01:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.824 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.824 INFO:teuthology.orchestra.run.smithi176.stdout: "req_setfilelock_latency": { 2023-11-16T17:01:27.824 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.824 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.824 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.824 INFO:teuthology.orchestra.run.smithi176.stdout: "req_setlayout_latency": { 2023-11-16T17:01:27.825 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.825 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.825 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.825 INFO:teuthology.orchestra.run.smithi176.stdout: "req_setxattr_latency": { 2023-11-16T17:01:27.825 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.825 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.825 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.826 INFO:teuthology.orchestra.run.smithi176.stdout: "req_symlink_latency": { 2023-11-16T17:01:27.826 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.826 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.826 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.826 INFO:teuthology.orchestra.run.smithi176.stdout: "req_unlink_latency": { 2023-11-16T17:01:27.826 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.826 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.827 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.827 INFO:teuthology.orchestra.run.smithi176.stdout: "cap_revoke_eviction": 0, 2023-11-16T17:01:27.827 INFO:teuthology.orchestra.run.smithi176.stdout: "cap_acquisition_throttle": 0, 2023-11-16T17:01:27.827 INFO:teuthology.orchestra.run.smithi176.stdout: "req_getvxattr_latency": { 2023-11-16T17:01:27.827 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.827 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.827 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.828 INFO:teuthology.orchestra.run.smithi176.stdout: } 2023-11-16T17:01:27.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.828 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_sessions": { 2023-11-16T17:01:27.828 INFO:teuthology.orchestra.run.smithi176.stdout: "session_count": 2, 2023-11-16T17:01:27.828 INFO:teuthology.orchestra.run.smithi176.stdout: "session_add": 6, 2023-11-16T17:01:27.828 INFO:teuthology.orchestra.run.smithi176.stdout: "session_remove": 4, 2023-11-16T17:01:27.828 INFO:teuthology.orchestra.run.smithi176.stdout: "sessions_open": 4, 2023-11-16T17:01:27.828 INFO:teuthology.orchestra.run.smithi176.stdout: "sessions_stale": 0, 2023-11-16T17:01:27.829 INFO:teuthology.orchestra.run.smithi176.stdout: "total_load": 0, 2023-11-16T17:01:27.829 INFO:teuthology.orchestra.run.smithi176.stdout: "average_load": 0, 2023-11-16T17:01:27.829 INFO:teuthology.orchestra.run.smithi176.stdout: "avg_session_uptime": 4927 2023-11-16T17:01:27.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.829 INFO:teuthology.orchestra.run.smithi176.stdout: "mempool": { 2023-11-16T17:01:27.829 INFO:teuthology.orchestra.run.smithi176.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:01:27.829 INFO:teuthology.orchestra.run.smithi176.stdout: "bloom_filter_items": 0, 2023-11-16T17:01:27.829 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:01:27.830 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:01:27.830 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:01:27.830 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:01:27.830 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:01:27.830 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:01:27.830 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:01:27.830 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:01:27.830 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:01:27.831 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:01:27.831 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:01:27.831 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:01:27.831 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:01:27.831 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_extent_items": 0, 2023-11-16T17:01:27.831 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:01:27.831 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_blob_items": 0, 2023-11-16T17:01:27.831 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:01:27.832 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:01:27.832 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:01:27.832 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:01:27.832 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:01:27.832 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:01:27.832 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:01:27.832 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_txc_items": 0, 2023-11-16T17:01:27.832 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:01:27.832 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:01:27.833 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:01:27.833 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_writing_items": 0, 2023-11-16T17:01:27.833 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_bytes": 0, 2023-11-16T17:01:27.833 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_items": 0, 2023-11-16T17:01:27.833 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:01:27.833 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:01:27.833 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:01:27.833 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:01:27.834 INFO:teuthology.orchestra.run.smithi176.stdout: "buffer_anon_bytes": 2508441, 2023-11-16T17:01:27.834 INFO:teuthology.orchestra.run.smithi176.stdout: "buffer_anon_items": 66814, 2023-11-16T17:01:27.834 INFO:teuthology.orchestra.run.smithi176.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:01:27.834 INFO:teuthology.orchestra.run.smithi176.stdout: "buffer_meta_items": 0, 2023-11-16T17:01:27.834 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_bytes": 0, 2023-11-16T17:01:27.834 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_items": 0, 2023-11-16T17:01:27.834 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:01:27.834 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_mapbl_items": 0, 2023-11-16T17:01:27.835 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:01:27.835 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_pglog_items": 0, 2023-11-16T17:01:27.835 INFO:teuthology.orchestra.run.smithi176.stdout: "osdmap_bytes": 10072, 2023-11-16T17:01:27.835 INFO:teuthology.orchestra.run.smithi176.stdout: "osdmap_items": 196, 2023-11-16T17:01:27.835 INFO:teuthology.orchestra.run.smithi176.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:01:27.835 INFO:teuthology.orchestra.run.smithi176.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:01:27.835 INFO:teuthology.orchestra.run.smithi176.stdout: "pgmap_bytes": 0, 2023-11-16T17:01:27.835 INFO:teuthology.orchestra.run.smithi176.stdout: "pgmap_items": 0, 2023-11-16T17:01:27.836 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_co_bytes": 268881542, 2023-11-16T17:01:27.836 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_co_items": 2202346, 2023-11-16T17:01:27.836 INFO:teuthology.orchestra.run.smithi176.stdout: "unittest_1_bytes": 0, 2023-11-16T17:01:27.836 INFO:teuthology.orchestra.run.smithi176.stdout: "unittest_1_items": 0, 2023-11-16T17:01:27.836 INFO:teuthology.orchestra.run.smithi176.stdout: "unittest_2_bytes": 0, 2023-11-16T17:01:27.836 INFO:teuthology.orchestra.run.smithi176.stdout: "unittest_2_items": 0 2023-11-16T17:01:27.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.836 INFO:teuthology.orchestra.run.smithi176.stdout: "objecter": { 2023-11-16T17:01:27.836 INFO:teuthology.orchestra.run.smithi176.stdout: "op_active": 0, 2023-11-16T17:01:27.837 INFO:teuthology.orchestra.run.smithi176.stdout: "op_laggy": 0, 2023-11-16T17:01:27.837 INFO:teuthology.orchestra.run.smithi176.stdout: "op_send": 3736, 2023-11-16T17:01:27.837 INFO:teuthology.orchestra.run.smithi176.stdout: "op_send_bytes": 26, 2023-11-16T17:01:27.837 INFO:teuthology.orchestra.run.smithi176.stdout: "op_resend": 4, 2023-11-16T17:01:27.837 INFO:teuthology.orchestra.run.smithi176.stdout: "op_reply": 3736, 2023-11-16T17:01:27.837 INFO:teuthology.orchestra.run.smithi176.stdout: "op_latency": { 2023-11-16T17:01:27.837 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 3736, 2023-11-16T17:01:27.837 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 18.970832586, 2023-11-16T17:01:27.838 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.005077845 2023-11-16T17:01:27.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.838 INFO:teuthology.orchestra.run.smithi176.stdout: "op_inflight": 0, 2023-11-16T17:01:27.838 INFO:teuthology.orchestra.run.smithi176.stdout: "oplen_avg": { 2023-11-16T17:01:27.838 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 3736, 2023-11-16T17:01:27.838 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 3737 2023-11-16T17:01:27.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.838 INFO:teuthology.orchestra.run.smithi176.stdout: "op": 3736, 2023-11-16T17:01:27.839 INFO:teuthology.orchestra.run.smithi176.stdout: "op_r": 3736, 2023-11-16T17:01:27.839 INFO:teuthology.orchestra.run.smithi176.stdout: "op_w": 0, 2023-11-16T17:01:27.839 INFO:teuthology.orchestra.run.smithi176.stdout: "op_rmw": 0, 2023-11-16T17:01:27.839 INFO:teuthology.orchestra.run.smithi176.stdout: "op_pg": 0, 2023-11-16T17:01:27.839 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_stat": 2468, 2023-11-16T17:01:27.839 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_create": 0, 2023-11-16T17:01:27.839 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_read": 1265, 2023-11-16T17:01:27.839 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_write": 0, 2023-11-16T17:01:27.840 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_writefull": 0, 2023-11-16T17:01:27.840 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_writesame": 0, 2023-11-16T17:01:27.840 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_append": 0, 2023-11-16T17:01:27.840 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_zero": 0, 2023-11-16T17:01:27.840 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_truncate": 0, 2023-11-16T17:01:27.840 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_delete": 0, 2023-11-16T17:01:27.840 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_mapext": 0, 2023-11-16T17:01:27.840 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_sparse_read": 0, 2023-11-16T17:01:27.841 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_clonerange": 0, 2023-11-16T17:01:27.841 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_getxattr": 2, 2023-11-16T17:01:27.841 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_setxattr": 0, 2023-11-16T17:01:27.841 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_cmpxattr": 0, 2023-11-16T17:01:27.841 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_rmxattr": 0, 2023-11-16T17:01:27.841 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_resetxattrs": 0, 2023-11-16T17:01:27.841 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_call": 0, 2023-11-16T17:01:27.841 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_watch": 0, 2023-11-16T17:01:27.842 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_notify": 0, 2023-11-16T17:01:27.842 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_src_cmpxattr": 0, 2023-11-16T17:01:27.842 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_pgls": 0, 2023-11-16T17:01:27.842 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_pgls_filter": 0, 2023-11-16T17:01:27.842 INFO:teuthology.orchestra.run.smithi176.stdout: "osdop_other": 0, 2023-11-16T17:01:27.842 INFO:teuthology.orchestra.run.smithi176.stdout: "linger_active": 0, 2023-11-16T17:01:27.842 INFO:teuthology.orchestra.run.smithi176.stdout: "linger_send": 0, 2023-11-16T17:01:27.842 INFO:teuthology.orchestra.run.smithi176.stdout: "linger_resend": 0, 2023-11-16T17:01:27.843 INFO:teuthology.orchestra.run.smithi176.stdout: "linger_ping": 0, 2023-11-16T17:01:27.843 INFO:teuthology.orchestra.run.smithi176.stdout: "poolop_active": 0, 2023-11-16T17:01:27.843 INFO:teuthology.orchestra.run.smithi176.stdout: "poolop_send": 0, 2023-11-16T17:01:27.843 INFO:teuthology.orchestra.run.smithi176.stdout: "poolop_resend": 0, 2023-11-16T17:01:27.843 INFO:teuthology.orchestra.run.smithi176.stdout: "poolstat_active": 0, 2023-11-16T17:01:27.843 INFO:teuthology.orchestra.run.smithi176.stdout: "poolstat_send": 0, 2023-11-16T17:01:27.843 INFO:teuthology.orchestra.run.smithi176.stdout: "poolstat_resend": 0, 2023-11-16T17:01:27.843 INFO:teuthology.orchestra.run.smithi176.stdout: "statfs_active": 0, 2023-11-16T17:01:27.844 INFO:teuthology.orchestra.run.smithi176.stdout: "statfs_send": 0, 2023-11-16T17:01:27.844 INFO:teuthology.orchestra.run.smithi176.stdout: "statfs_resend": 0, 2023-11-16T17:01:27.844 INFO:teuthology.orchestra.run.smithi176.stdout: "command_active": 0, 2023-11-16T17:01:27.844 INFO:teuthology.orchestra.run.smithi176.stdout: "command_send": 0, 2023-11-16T17:01:27.844 INFO:teuthology.orchestra.run.smithi176.stdout: "command_resend": 0, 2023-11-16T17:01:27.844 INFO:teuthology.orchestra.run.smithi176.stdout: "map_epoch": 102, 2023-11-16T17:01:27.844 INFO:teuthology.orchestra.run.smithi176.stdout: "map_full": 0, 2023-11-16T17:01:27.844 INFO:teuthology.orchestra.run.smithi176.stdout: "map_inc": 13, 2023-11-16T17:01:27.844 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_sessions": 11, 2023-11-16T17:01:27.845 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_session_open": 11, 2023-11-16T17:01:27.845 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_session_close": 0, 2023-11-16T17:01:27.845 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_laggy": 0, 2023-11-16T17:01:27.845 INFO:teuthology.orchestra.run.smithi176.stdout: "omap_wr": 0, 2023-11-16T17:01:27.845 INFO:teuthology.orchestra.run.smithi176.stdout: "omap_rd": 2, 2023-11-16T17:01:27.845 INFO:teuthology.orchestra.run.smithi176.stdout: "omap_del": 0 2023-11-16T17:01:27.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.845 INFO:teuthology.orchestra.run.smithi176.stdout: "oft": { 2023-11-16T17:01:27.846 INFO:teuthology.orchestra.run.smithi176.stdout: "omap_total_objs": 0, 2023-11-16T17:01:27.846 INFO:teuthology.orchestra.run.smithi176.stdout: "omap_total_kv_pairs": 0, 2023-11-16T17:01:27.846 INFO:teuthology.orchestra.run.smithi176.stdout: "omap_total_updates": 0, 2023-11-16T17:01:27.846 INFO:teuthology.orchestra.run.smithi176.stdout: "omap_total_removes": 0 2023-11-16T17:01:27.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.846 INFO:teuthology.orchestra.run.smithi176.stdout: "osd-slow-ops": { 2023-11-16T17:01:27.846 INFO:teuthology.orchestra.run.smithi176.stdout: "slow_ops_count": 0 2023-11-16T17:01:27.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.847 INFO:teuthology.orchestra.run.smithi176.stdout: "purge_queue": { 2023-11-16T17:01:27.847 INFO:teuthology.orchestra.run.smithi176.stdout: "pq_executing_ops": 0, 2023-11-16T17:01:27.847 INFO:teuthology.orchestra.run.smithi176.stdout: "pq_executing_ops_high_water": 0, 2023-11-16T17:01:27.847 INFO:teuthology.orchestra.run.smithi176.stdout: "pq_executing": 0, 2023-11-16T17:01:27.847 INFO:teuthology.orchestra.run.smithi176.stdout: "pq_executing_high_water": 0, 2023-11-16T17:01:27.847 INFO:teuthology.orchestra.run.smithi176.stdout: "pq_executed": 0, 2023-11-16T17:01:27.847 INFO:teuthology.orchestra.run.smithi176.stdout: "pq_item_in_journal": 0 2023-11-16T17:01:27.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.848 INFO:teuthology.orchestra.run.smithi176.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:01:27.848 INFO:teuthology.orchestra.run.smithi176.stdout: "val": 0, 2023-11-16T17:01:27.848 INFO:teuthology.orchestra.run.smithi176.stdout: "max": 104857600, 2023-11-16T17:01:27.848 INFO:teuthology.orchestra.run.smithi176.stdout: "get_started": 0, 2023-11-16T17:01:27.848 INFO:teuthology.orchestra.run.smithi176.stdout: "get": 4000, 2023-11-16T17:01:27.848 INFO:teuthology.orchestra.run.smithi176.stdout: "get_sum": 772653426, 2023-11-16T17:01:27.848 INFO:teuthology.orchestra.run.smithi176.stdout: "get_or_fail_fail": 0, 2023-11-16T17:01:27.848 INFO:teuthology.orchestra.run.smithi176.stdout: "get_or_fail_success": 4000, 2023-11-16T17:01:27.848 INFO:teuthology.orchestra.run.smithi176.stdout: "take": 0, 2023-11-16T17:01:27.849 INFO:teuthology.orchestra.run.smithi176.stdout: "take_sum": 0, 2023-11-16T17:01:27.849 INFO:teuthology.orchestra.run.smithi176.stdout: "put": 4000, 2023-11-16T17:01:27.849 INFO:teuthology.orchestra.run.smithi176.stdout: "put_sum": 772653426, 2023-11-16T17:01:27.849 INFO:teuthology.orchestra.run.smithi176.stdout: "wait": { 2023-11-16T17:01:27.849 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.849 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.849 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.849 INFO:teuthology.orchestra.run.smithi176.stdout: } 2023-11-16T17:01:27.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.850 INFO:teuthology.orchestra.run.smithi176.stdout: "throttle-objecter_bytes": { 2023-11-16T17:01:27.850 INFO:teuthology.orchestra.run.smithi176.stdout: "val": 0, 2023-11-16T17:01:27.850 INFO:teuthology.orchestra.run.smithi176.stdout: "max": 104857600, 2023-11-16T17:01:27.850 INFO:teuthology.orchestra.run.smithi176.stdout: "get_started": 0, 2023-11-16T17:01:27.850 INFO:teuthology.orchestra.run.smithi176.stdout: "get": 0, 2023-11-16T17:01:27.850 INFO:teuthology.orchestra.run.smithi176.stdout: "get_sum": 0, 2023-11-16T17:01:27.850 INFO:teuthology.orchestra.run.smithi176.stdout: "get_or_fail_fail": 0, 2023-11-16T17:01:27.851 INFO:teuthology.orchestra.run.smithi176.stdout: "get_or_fail_success": 0, 2023-11-16T17:01:27.851 INFO:teuthology.orchestra.run.smithi176.stdout: "take": 3736, 2023-11-16T17:01:27.851 INFO:teuthology.orchestra.run.smithi176.stdout: "take_sum": 770549433, 2023-11-16T17:01:27.851 INFO:teuthology.orchestra.run.smithi176.stdout: "put": 3736, 2023-11-16T17:01:27.851 INFO:teuthology.orchestra.run.smithi176.stdout: "put_sum": 770549433, 2023-11-16T17:01:27.851 INFO:teuthology.orchestra.run.smithi176.stdout: "wait": { 2023-11-16T17:01:27.851 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.851 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.852 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.852 INFO:teuthology.orchestra.run.smithi176.stdout: } 2023-11-16T17:01:27.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.852 INFO:teuthology.orchestra.run.smithi176.stdout: "throttle-objecter_ops": { 2023-11-16T17:01:27.852 INFO:teuthology.orchestra.run.smithi176.stdout: "val": 0, 2023-11-16T17:01:27.852 INFO:teuthology.orchestra.run.smithi176.stdout: "max": 1024, 2023-11-16T17:01:27.852 INFO:teuthology.orchestra.run.smithi176.stdout: "get_started": 0, 2023-11-16T17:01:27.852 INFO:teuthology.orchestra.run.smithi176.stdout: "get": 0, 2023-11-16T17:01:27.853 INFO:teuthology.orchestra.run.smithi176.stdout: "get_sum": 0, 2023-11-16T17:01:27.853 INFO:teuthology.orchestra.run.smithi176.stdout: "get_or_fail_fail": 0, 2023-11-16T17:01:27.853 INFO:teuthology.orchestra.run.smithi176.stdout: "get_or_fail_success": 0, 2023-11-16T17:01:27.853 INFO:teuthology.orchestra.run.smithi176.stdout: "take": 3736, 2023-11-16T17:01:27.853 INFO:teuthology.orchestra.run.smithi176.stdout: "take_sum": 3736, 2023-11-16T17:01:27.853 INFO:teuthology.orchestra.run.smithi176.stdout: "put": 3736, 2023-11-16T17:01:27.853 INFO:teuthology.orchestra.run.smithi176.stdout: "put_sum": 3736, 2023-11-16T17:01:27.853 INFO:teuthology.orchestra.run.smithi176.stdout: "wait": { 2023-11-16T17:01:27.853 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.854 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.854 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.854 INFO:teuthology.orchestra.run.smithi176.stdout: } 2023-11-16T17:01:27.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.854 INFO:teuthology.orchestra.run.smithi176.stdout: "throttle-write_buf_throttle": { 2023-11-16T17:01:27.854 INFO:teuthology.orchestra.run.smithi176.stdout: "val": 0, 2023-11-16T17:01:27.854 INFO:teuthology.orchestra.run.smithi176.stdout: "max": 3758096384, 2023-11-16T17:01:27.854 INFO:teuthology.orchestra.run.smithi176.stdout: "get_started": 0, 2023-11-16T17:01:27.855 INFO:teuthology.orchestra.run.smithi176.stdout: "get": 0, 2023-11-16T17:01:27.855 INFO:teuthology.orchestra.run.smithi176.stdout: "get_sum": 0, 2023-11-16T17:01:27.855 INFO:teuthology.orchestra.run.smithi176.stdout: "get_or_fail_fail": 0, 2023-11-16T17:01:27.855 INFO:teuthology.orchestra.run.smithi176.stdout: "get_or_fail_success": 0, 2023-11-16T17:01:27.855 INFO:teuthology.orchestra.run.smithi176.stdout: "take": 0, 2023-11-16T17:01:27.855 INFO:teuthology.orchestra.run.smithi176.stdout: "take_sum": 0, 2023-11-16T17:01:27.855 INFO:teuthology.orchestra.run.smithi176.stdout: "put": 0, 2023-11-16T17:01:27.855 INFO:teuthology.orchestra.run.smithi176.stdout: "put_sum": 0, 2023-11-16T17:01:27.856 INFO:teuthology.orchestra.run.smithi176.stdout: "wait": { 2023-11-16T17:01:27.856 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.856 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.856 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.856 INFO:teuthology.orchestra.run.smithi176.stdout: } 2023-11-16T17:01:27.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:27.856 INFO:teuthology.orchestra.run.smithi176.stdout: "throttle-write_buf_throttle-0x55a7d43b4460": { 2023-11-16T17:01:27.856 INFO:teuthology.orchestra.run.smithi176.stdout: "val": 0, 2023-11-16T17:01:27.856 INFO:teuthology.orchestra.run.smithi176.stdout: "max": 3758096384, 2023-11-16T17:01:27.857 INFO:teuthology.orchestra.run.smithi176.stdout: "get_started": 0, 2023-11-16T17:01:27.857 INFO:teuthology.orchestra.run.smithi176.stdout: "get": 0, 2023-11-16T17:01:27.857 INFO:teuthology.orchestra.run.smithi176.stdout: "get_sum": 0, 2023-11-16T17:01:27.857 INFO:teuthology.orchestra.run.smithi176.stdout: "get_or_fail_fail": 0, 2023-11-16T17:01:27.857 INFO:teuthology.orchestra.run.smithi176.stdout: "get_or_fail_success": 0, 2023-11-16T17:01:27.857 INFO:teuthology.orchestra.run.smithi176.stdout: "take": 0, 2023-11-16T17:01:27.857 INFO:teuthology.orchestra.run.smithi176.stdout: "take_sum": 0, 2023-11-16T17:01:27.857 INFO:teuthology.orchestra.run.smithi176.stdout: "put": 0, 2023-11-16T17:01:27.858 INFO:teuthology.orchestra.run.smithi176.stdout: "put_sum": 0, 2023-11-16T17:01:27.858 INFO:teuthology.orchestra.run.smithi176.stdout: "wait": { 2023-11-16T17:01:27.858 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:27.858 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:27.858 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:27.858 INFO:teuthology.orchestra.run.smithi176.stdout: } 2023-11-16T17:01:27.858 INFO:teuthology.orchestra.run.smithi176.stdout: } 2023-11-16T17:01:27.858 INFO:teuthology.orchestra.run.smithi176.stdout:} 2023-11-16T17:01:27.859 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:27 smithi176 ceph-mon[131630]: pgmap v358: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2023-11-16T17:01:27.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:27 smithi044 ceph-mon[128750]: pgmap v358: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2023-11-16T17:01:27.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:27 smithi077 ceph-mon[131204]: pgmap v358: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2023-11-16T17:01:28.143 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2023-11-16T17:01:28.143 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2023-11-16T17:01:28.144 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2023-11-16T17:01:28.144 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2023-11-16T17:01:28.144 DEBUG:tasks.check_counter:Getting stats from l 2023-11-16T17:01:28.144 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.l perf dump 2023-11-16T17:01:29.004 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T17:01:29.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:29 smithi044 ceph-mon[128750]: pgmap v359: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2023-11-16T17:01:29.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:29 smithi176 ceph-mon[131630]: pgmap v359: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2023-11-16T17:01:29.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:29 smithi077 ceph-mon[131204]: pgmap v359: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2023-11-16T17:01:30.600 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2023-11-16T17:01:30.601 INFO:teuthology.orchestra.run.smithi176.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:01:30.601 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_messages": 0, 2023-11-16T17:01:30.601 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages": 2, 2023-11-16T17:01:30.601 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_bytes": 0, 2023-11-16T17:01:30.601 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_bytes": 0, 2023-11-16T17:01:30.601 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_created_connections": 4, 2023-11-16T17:01:30.601 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_active_connections": 0, 2023-11-16T17:01:30.602 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_total_time": 0.001069443, 2023-11-16T17:01:30.602 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_send_time": 0.000000000, 2023-11-16T17:01:30.602 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_recv_time": 0.000157560, 2023-11-16T17:01:30.602 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:01:30.602 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:30.602 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:30.602 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:30.603 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:30.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:30.603 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:30.603 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:30.603 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:30.603 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:30.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:30.603 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T17:01:30.604 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T17:01:30.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:30.604 INFO:teuthology.orchestra.run.smithi176.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:01:30.604 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_messages": 251, 2023-11-16T17:01:30.604 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages": 239, 2023-11-16T17:01:30.604 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_bytes": 1431472, 2023-11-16T17:01:30.604 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_bytes": 135744, 2023-11-16T17:01:30.604 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_created_connections": 2, 2023-11-16T17:01:30.605 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_active_connections": 1, 2023-11-16T17:01:30.605 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_total_time": 0.047276952, 2023-11-16T17:01:30.605 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_send_time": 0.015088009, 2023-11-16T17:01:30.605 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_recv_time": 0.032631955, 2023-11-16T17:01:30.605 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_fast_dispatch_time": 0.006789392, 2023-11-16T17:01:30.605 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:30.605 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 239, 2023-11-16T17:01:30.605 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.021879501, 2023-11-16T17:01:30.606 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000091546 2023-11-16T17:01:30.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:30.606 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:30.606 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:30.606 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:30.606 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:30.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:30.607 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_encrypted_bytes": 1431472, 2023-11-16T17:01:30.607 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_encrypted_bytes": 135744 2023-11-16T17:01:30.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:30.607 INFO:teuthology.orchestra.run.smithi176.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:01:30.607 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_messages": 2, 2023-11-16T17:01:30.607 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages": 181, 2023-11-16T17:01:30.607 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_bytes": 320, 2023-11-16T17:01:30.608 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_bytes": 183456, 2023-11-16T17:01:30.608 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_created_connections": 5, 2023-11-16T17:01:30.608 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_active_connections": 3, 2023-11-16T17:01:30.608 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_total_time": 0.010751982, 2023-11-16T17:01:30.608 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_send_time": 0.007171634, 2023-11-16T17:01:30.608 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_recv_time": 0.001259552, 2023-11-16T17:01:30.608 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:01:30.608 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:01:30.609 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 178, 2023-11-16T17:01:30.609 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.022471346, 2023-11-16T17:01:30.609 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000126243 2023-11-16T17:01:30.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:30.609 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:01:30.609 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:30.609 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:30.610 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:30.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:30.610 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T17:01:30.610 INFO:teuthology.orchestra.run.smithi176.stdout: "msgr_send_encrypted_bytes": 183456 2023-11-16T17:01:30.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:30.610 INFO:teuthology.orchestra.run.smithi176.stdout: "cct": { 2023-11-16T17:01:30.610 INFO:teuthology.orchestra.run.smithi176.stdout: "total_workers": 0, 2023-11-16T17:01:30.611 INFO:teuthology.orchestra.run.smithi176.stdout: "unhealthy_workers": 0 2023-11-16T17:01:30.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:30.611 INFO:teuthology.orchestra.run.smithi176.stdout: "mempool": { 2023-11-16T17:01:30.611 INFO:teuthology.orchestra.run.smithi176.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:01:30.611 INFO:teuthology.orchestra.run.smithi176.stdout: "bloom_filter_items": 0, 2023-11-16T17:01:30.611 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:01:30.611 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:01:30.611 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:01:30.612 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:01:30.612 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:01:30.612 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:01:30.612 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:01:30.612 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:01:30.612 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:01:30.612 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:01:30.612 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:01:30.613 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:01:30.613 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:01:30.613 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_extent_items": 0, 2023-11-16T17:01:30.613 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:01:30.613 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_blob_items": 0, 2023-11-16T17:01:30.613 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:01:30.613 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:01:30.613 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:01:30.614 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:01:30.614 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:01:30.614 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:01:30.614 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:01:30.614 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_txc_items": 0, 2023-11-16T17:01:30.614 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:01:30.614 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:01:30.614 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:01:30.615 INFO:teuthology.orchestra.run.smithi176.stdout: "bluestore_writing_items": 0, 2023-11-16T17:01:30.615 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_bytes": 0, 2023-11-16T17:01:30.615 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_items": 0, 2023-11-16T17:01:30.615 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:01:30.615 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:01:30.615 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:01:30.615 INFO:teuthology.orchestra.run.smithi176.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:01:30.616 INFO:teuthology.orchestra.run.smithi176.stdout: "buffer_anon_bytes": 148732, 2023-11-16T17:01:30.616 INFO:teuthology.orchestra.run.smithi176.stdout: "buffer_anon_items": 31, 2023-11-16T17:01:30.616 INFO:teuthology.orchestra.run.smithi176.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:01:30.616 INFO:teuthology.orchestra.run.smithi176.stdout: "buffer_meta_items": 0, 2023-11-16T17:01:30.616 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_bytes": 0, 2023-11-16T17:01:30.616 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_items": 0, 2023-11-16T17:01:30.616 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:01:30.616 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_mapbl_items": 0, 2023-11-16T17:01:30.617 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:01:30.617 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_pglog_items": 0, 2023-11-16T17:01:30.617 INFO:teuthology.orchestra.run.smithi176.stdout: "osdmap_bytes": 0, 2023-11-16T17:01:30.617 INFO:teuthology.orchestra.run.smithi176.stdout: "osdmap_items": 0, 2023-11-16T17:01:30.617 INFO:teuthology.orchestra.run.smithi176.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:01:30.617 INFO:teuthology.orchestra.run.smithi176.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:01:30.617 INFO:teuthology.orchestra.run.smithi176.stdout: "pgmap_bytes": 0, 2023-11-16T17:01:30.617 INFO:teuthology.orchestra.run.smithi176.stdout: "pgmap_items": 0, 2023-11-16T17:01:30.617 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_co_bytes": 928, 2023-11-16T17:01:30.618 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_co_items": 2, 2023-11-16T17:01:30.618 INFO:teuthology.orchestra.run.smithi176.stdout: "unittest_1_bytes": 0, 2023-11-16T17:01:30.618 INFO:teuthology.orchestra.run.smithi176.stdout: "unittest_1_items": 0, 2023-11-16T17:01:30.618 INFO:teuthology.orchestra.run.smithi176.stdout: "unittest_2_bytes": 0, 2023-11-16T17:01:30.618 INFO:teuthology.orchestra.run.smithi176.stdout: "unittest_2_items": 0 2023-11-16T17:01:30.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2023-11-16T17:01:30.618 INFO:teuthology.orchestra.run.smithi176.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:01:30.618 INFO:teuthology.orchestra.run.smithi176.stdout: "val": 0, 2023-11-16T17:01:30.619 INFO:teuthology.orchestra.run.smithi176.stdout: "max": 104857600, 2023-11-16T17:01:30.619 INFO:teuthology.orchestra.run.smithi176.stdout: "get_started": 0, 2023-11-16T17:01:30.619 INFO:teuthology.orchestra.run.smithi176.stdout: "get": 249, 2023-11-16T17:01:30.619 INFO:teuthology.orchestra.run.smithi176.stdout: "get_sum": 1393648, 2023-11-16T17:01:30.619 INFO:teuthology.orchestra.run.smithi176.stdout: "get_or_fail_fail": 0, 2023-11-16T17:01:30.619 INFO:teuthology.orchestra.run.smithi176.stdout: "get_or_fail_success": 249, 2023-11-16T17:01:30.619 INFO:teuthology.orchestra.run.smithi176.stdout: "take": 0, 2023-11-16T17:01:30.620 INFO:teuthology.orchestra.run.smithi176.stdout: "take_sum": 0, 2023-11-16T17:01:30.620 INFO:teuthology.orchestra.run.smithi176.stdout: "put": 249, 2023-11-16T17:01:30.620 INFO:teuthology.orchestra.run.smithi176.stdout: "put_sum": 1393648, 2023-11-16T17:01:30.620 INFO:teuthology.orchestra.run.smithi176.stdout: "wait": { 2023-11-16T17:01:30.620 INFO:teuthology.orchestra.run.smithi176.stdout: "avgcount": 0, 2023-11-16T17:01:30.620 INFO:teuthology.orchestra.run.smithi176.stdout: "sum": 0.000000000, 2023-11-16T17:01:30.620 INFO:teuthology.orchestra.run.smithi176.stdout: "avgtime": 0.000000000 2023-11-16T17:01:30.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2023-11-16T17:01:30.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2023-11-16T17:01:30.621 INFO:teuthology.orchestra.run.smithi176.stdout:} 2023-11-16T17:01:31.052 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2023-11-16T17:01:31.053 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2023-11-16T17:01:31.053 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2023-11-16T17:01:31.053 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2023-11-16T17:01:31.053 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-11-16T17:01:31.063 INFO:tasks.kclient:Unmounting kernel clients... 2023-11-16T17:01:31.063 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T17:01:31.063 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T17:01:31.168 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-11-16T17:01:31.168 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T17:01:31.254 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T17:01:31.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.638+0000 7f68431ff700 1 -- 172.21.15.44:0/3368399924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f683c1045d0 msgr2=0x7f683c104a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:31.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.638+0000 7f68431ff700 1 --2- 172.21.15.44:0/3368399924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f683c1045d0 0x7f683c104a50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6834009990 tx=0x7f6834009ca0 comp rx=0 tx=0).stop 2023-11-16T17:01:31.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.638+0000 7f68431ff700 1 -- 172.21.15.44:0/3368399924 shutdown_connections 2023-11-16T17:01:31.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.638+0000 7f68431ff700 1 --2- 172.21.15.44:0/3368399924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f683c1045d0 0x7f683c104a50 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:31.640 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.638+0000 7f68431ff700 1 --2- 172.21.15.44:0/3368399924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f683c0ff2b0 0x7f683c0ff680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:31.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.638+0000 7f68431ff700 1 --2- 172.21.15.44:0/3368399924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f683c0ffbc0 0x7f683c104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:31.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.638+0000 7f68431ff700 1 -- 172.21.15.44:0/3368399924 >> 172.21.15.44:0/3368399924 conn(0x7f683c076310 msgr2=0x7f683c076710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:31.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.638+0000 7f68431ff700 1 -- 172.21.15.44:0/3368399924 shutdown_connections 2023-11-16T17:01:31.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.638+0000 7f68431ff700 1 -- 172.21.15.44:0/3368399924 wait complete. 2023-11-16T17:01:31.641 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.640+0000 7f68431ff700 1 Processor -- start 2023-11-16T17:01:31.642 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.640+0000 7f68431ff700 1 -- start start 2023-11-16T17:01:31.643 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.641+0000 7f68431ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f683c0ff2b0 0x7f683c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:31.643 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.641+0000 7f68431ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f683c0ffbc0 0x7f683c19cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:31.643 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.641+0000 7f68431ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f683c1045d0 0x7f683c1a1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:31.643 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.641+0000 7f68431ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c1a1920 con 0x7f683c0ff2b0 2023-11-16T17:01:31.643 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.641+0000 7f68431ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c1a1a90 con 0x7f683c0ffbc0 2023-11-16T17:01:31.643 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.641+0000 7f68431ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c1a1c50 con 0x7f683c1045d0 2023-11-16T17:01:31.644 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.641+0000 7f684179c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f683c1045d0 0x7f683c1a1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:31.644 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.641+0000 7f684179c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f683c1045d0 0x7f683c1a1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:33476/0 (socket says 172.21.15.44:33476) 2023-11-16T17:01:31.644 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.641+0000 7f684179c700 1 -- 172.21.15.44:0/3412200062 learned_addr learned my addr 172.21.15.44:0/3412200062 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:01:31.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.642+0000 7f6840f9b700 1 --2- 172.21.15.44:0/3412200062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f683c0ff2b0 0x7f683c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:31.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.642+0000 7f684179c700 1 -- 172.21.15.44:0/3412200062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f683c0ffbc0 msgr2=0x7f683c19cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:31.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.642+0000 7f684179c700 1 --2- 172.21.15.44:0/3412200062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f683c0ffbc0 0x7f683c19cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:31.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.642+0000 7f684179c700 1 -- 172.21.15.44:0/3412200062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f683c0ff2b0 msgr2=0x7f683c19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:31.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.642+0000 7f684179c700 1 --2- 172.21.15.44:0/3412200062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f683c0ff2b0 0x7f683c19c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:31.645 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.642+0000 7f684179c700 1 -- 172.21.15.44:0/3412200062 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68340095f0 con 0x7f683c1045d0 2023-11-16T17:01:31.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.644+0000 7f684179c700 1 --2- 172.21.15.44:0/3412200062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f683c1045d0 0x7f683c1a1350 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6834006010 tx=0x7f6834003c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:31.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.644+0000 7f683a7fc700 1 -- 172.21.15.44:0/3412200062 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683401d070 con 0x7f683c1045d0 2023-11-16T17:01:31.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.644+0000 7f683a7fc700 1 -- 172.21.15.44:0/3412200062 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6834003e90 con 0x7f683c1045d0 2023-11-16T17:01:31.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.644+0000 7f683a7fc700 1 -- 172.21.15.44:0/3412200062 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68340173f0 con 0x7f683c1045d0 2023-11-16T17:01:31.646 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.645+0000 7f68431ff700 1 -- 172.21.15.44:0/3412200062 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f683c1169b0 con 0x7f683c1045d0 2023-11-16T17:01:31.647 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.645+0000 7f68431ff700 1 -- 172.21.15.44:0/3412200062 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f683c116820 con 0x7f683c1045d0 2023-11-16T17:01:31.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.650+0000 7f683a7fc700 1 -- 172.21.15.44:0/3412200062 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f68340052d0 con 0x7f683c1045d0 2023-11-16T17:01:31.651 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.649+0000 7f68431ff700 1 -- 172.21.15.44:0/3412200062 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f683c116b40 con 0x7f683c1045d0 2023-11-16T17:01:31.653 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.652+0000 7f683a7fc700 1 --2- 172.21.15.44:0/3412200062 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f68280a13c0 0x7f68280a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:31.656 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.655+0000 7f683a7fc700 1 -- 172.21.15.44:0/3412200062 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9631+0+0 (secure 0 0 0) 0x7f68340ce470 con 0x7f683c1045d0 2023-11-16T17:01:31.657 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.656+0000 7f6840f9b700 1 --2- 172.21.15.44:0/3412200062 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f68280a13c0 0x7f68280a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:31.658 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.656+0000 7f6840f9b700 1 --2- 172.21.15.44:0/3412200062 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f68280a13c0 0x7f68280a3870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f682c00e270 tx=0x7f682c009040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:31.666 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:31.665+0000 7f683a7fc700 1 -- 172.21.15.44:0/3412200062 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6834097460 con 0x7f683c1045d0 2023-11-16T17:01:31.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:31 smithi044 ceph-mon[128750]: pgmap v360: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:01:31.894 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:31 smithi176 ceph-mon[131630]: pgmap v360: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:01:31.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:31 smithi077 ceph-mon[131204]: pgmap v360: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:01:32.084 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.083+0000 7f68431ff700 1 -- 172.21.15.44:0/3412200062 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f683c06a930 con 0x7f683c1045d0 2023-11-16T17:01:32.087 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.085+0000 7f683a7fc700 1 -- 172.21.15.44:0/3412200062 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+25430 (secure 0 0 0) 0x7f683409aa80 con 0x7f683c1045d0 2023-11-16T17:01:32.094 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T17:01:32.094 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":102,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T17:01:07.811232+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/4218362219":"2023-11-17T16:49:55.607497+0000","172.21.15.44:6801/304422349":"2023-11-17T16:49:55.607497+0000","172.21.15.44:0/1983838729":"2023-11-17T16:49:55.607497+0000","172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/2937650254":"2023-11-17T16:49:55.607497+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1915598320":"2023-11-17T16:49:55.607497+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/304422349":"2023-11-17T16:49:55.607497+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":2},{"begin":17,"length":1},{"begin":19,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":2},{"begin":17,"length":1},{"begin":19,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]},{"pool":4,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T17:01:32.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.093+0000 7f68431ff700 1 -- 172.21.15.44:0/3412200062 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f68280a13c0 msgr2=0x7f68280a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:32.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.093+0000 7f68431ff700 1 --2- 172.21.15.44:0/3412200062 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f68280a13c0 0x7f68280a3870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f682c00e270 tx=0x7f682c009040 comp rx=0 tx=0).stop 2023-11-16T17:01:32.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.093+0000 7f68431ff700 1 -- 172.21.15.44:0/3412200062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f683c1045d0 msgr2=0x7f683c1a1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:32.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.093+0000 7f68431ff700 1 --2- 172.21.15.44:0/3412200062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f683c1045d0 0x7f683c1a1350 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6834006010 tx=0x7f6834003c40 comp rx=0 tx=0).stop 2023-11-16T17:01:32.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.093+0000 7f68431ff700 1 -- 172.21.15.44:0/3412200062 shutdown_connections 2023-11-16T17:01:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.093+0000 7f68431ff700 1 --2- 172.21.15.44:0/3412200062 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f68280a13c0 0x7f68280a3870 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.093+0000 7f68431ff700 1 --2- 172.21.15.44:0/3412200062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f683c1045d0 0x7f683c1a1350 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.093+0000 7f68431ff700 1 --2- 172.21.15.44:0/3412200062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f683c0ff2b0 0x7f683c19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.093+0000 7f68431ff700 1 --2- 172.21.15.44:0/3412200062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f683c0ffbc0 0x7f683c19cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.093+0000 7f68431ff700 1 -- 172.21.15.44:0/3412200062 >> 172.21.15.44:0/3412200062 conn(0x7f683c076310 msgr2=0x7f683c102060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.094+0000 7f68431ff700 1 -- 172.21.15.44:0/3412200062 shutdown_connections 2023-11-16T17:01:32.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.094+0000 7f68431ff700 1 -- 172.21.15.44:0/3412200062 wait complete. 2023-11-16T17:01:32.128 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:03.059622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:16.211826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:17.601591+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:21.136084+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T17:01:32.129 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T17:01:32.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.517+0000 7f482e7ff700 1 -- 172.21.15.44:0/316223905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4828105620 msgr2=0x7f482810d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:32.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.517+0000 7f482e7ff700 1 --2- 172.21.15.44:0/316223905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4828105620 0x7f482810d1a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f482000b0d0 tx=0x7f482000b3e0 comp rx=0 tx=0).stop 2023-11-16T17:01:32.519 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.517+0000 7f482e7ff700 1 -- 172.21.15.44:0/316223905 shutdown_connections 2023-11-16T17:01:32.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.517+0000 7f482e7ff700 1 --2- 172.21.15.44:0/316223905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4828105620 0x7f482810d1a0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:32.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.517+0000 7f482e7ff700 1 --2- 172.21.15.44:0/316223905 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4828100cc0 0x7f4828101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:32.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.517+0000 7f482e7ff700 1 --2- 172.21.15.44:0/316223905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4828101660 0x7f4828104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:32.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.517+0000 7f482e7ff700 1 -- 172.21.15.44:0/316223905 >> 172.21.15.44:0/316223905 conn(0x7f4828077560 msgr2=0x7f4828077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:32.520 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.518+0000 7f482e7ff700 1 -- 172.21.15.44:0/316223905 shutdown_connections 2023-11-16T17:01:32.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.519+0000 7f482e7ff700 1 -- 172.21.15.44:0/316223905 wait complete. 2023-11-16T17:01:32.521 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.520+0000 7f482e7ff700 1 Processor -- start 2023-11-16T17:01:32.523 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.522+0000 7f482e7ff700 1 -- start start 2023-11-16T17:01:32.524 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.522+0000 7f482e7ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4828100cc0 0x7f482819e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:32.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.522+0000 7f482e7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4828101660 0x7f482819eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:32.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.522+0000 7f482e7ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4828105620 0x7f4828198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:32.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.522+0000 7f482e7ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4828198cd0 con 0x7f4828105620 2023-11-16T17:01:32.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.522+0000 7f482e7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4828198e40 con 0x7f4828101660 2023-11-16T17:01:32.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.522+0000 7f482e7ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4828199000 con 0x7f4828100cc0 2023-11-16T17:01:32.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.523+0000 7f482cd9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4828105620 0x7f4828198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:32.525 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.523+0000 7f482cd9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4828105620 0x7f4828198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54516/0 (socket says 172.21.15.44:54516) 2023-11-16T17:01:32.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.523+0000 7f482cd9c700 1 -- 172.21.15.44:0/192553374 learned_addr learned my addr 172.21.15.44:0/192553374 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:01:32.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.523+0000 7f482cd9c700 1 -- 172.21.15.44:0/192553374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4828100cc0 msgr2=0x7f482819e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:32.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.523+0000 7f482cd9c700 1 --2- 172.21.15.44:0/192553374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4828100cc0 0x7f482819e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:32.526 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.523+0000 7f482cd9c700 1 -- 172.21.15.44:0/192553374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4828101660 msgr2=0x7f482819eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:32.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.523+0000 7f482cd9c700 1 --2- 172.21.15.44:0/192553374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4828101660 0x7f482819eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:32.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.524+0000 7f482cd9c700 1 -- 172.21.15.44:0/192553374 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4820009d00 con 0x7f4828105620 2023-11-16T17:01:32.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.524+0000 7f482cd9c700 1 --2- 172.21.15.44:0/192553374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4828105620 0x7f4828198660 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f4820009fd0 tx=0x7f4820007910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:32.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.525+0000 7f4825ffb700 1 -- 172.21.15.44:0/192553374 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482000e040 con 0x7f4828105620 2023-11-16T17:01:32.527 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.525+0000 7f4825ffb700 1 -- 172.21.15.44:0/192553374 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4820003e10 con 0x7f4828105620 2023-11-16T17:01:32.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.525+0000 7f482e7ff700 1 -- 172.21.15.44:0/192553374 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4828110490 con 0x7f4828105620 2023-11-16T17:01:32.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.525+0000 7f4825ffb700 1 -- 172.21.15.44:0/192553374 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482001a990 con 0x7f4828105620 2023-11-16T17:01:32.528 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.525+0000 7f482e7ff700 1 -- 172.21.15.44:0/192553374 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4828110300 con 0x7f4828105620 2023-11-16T17:01:32.537 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.527+0000 7f482e7ff700 1 -- 172.21.15.44:0/192553374 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4828110620 con 0x7f4828105620 2023-11-16T17:01:32.538 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.530+0000 7f4825ffb700 1 -- 172.21.15.44:0/192553374 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f4820005630 con 0x7f4828105620 2023-11-16T17:01:32.538 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.532+0000 7f4825ffb700 1 --2- 172.21.15.44:0/192553374 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f48140a1340 0x7f48140a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:32.538 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.533+0000 7f4825ffb700 1 -- 172.21.15.44:0/192553374 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9631+0+0 (secure 0 0 0) 0x7f48200ce490 con 0x7f4828105620 2023-11-16T17:01:32.538 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.533+0000 7f4827fff700 1 --2- 172.21.15.44:0/192553374 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f48140a1340 0x7f48140a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:32.538 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.537+0000 7f4827fff700 1 --2- 172.21.15.44:0/192553374 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f48140a1340 0x7f48140a37f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f48180097b0 tx=0x7f4818006d20 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:32.542 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.541+0000 7f4825ffb700 1 -- 172.21.15.44:0/192553374 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f48200974e0 con 0x7f4828105620 2023-11-16T17:01:32.895 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:32 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3412200062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T17:01:32.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.950+0000 7f482e7ff700 1 -- 172.21.15.44:0/192553374 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f48281a5820 con 0x7f4828105620 2023-11-16T17:01:32.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.951+0000 7f4825ffb700 1 -- 172.21.15.44:0/192553374 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f482009ab00 con 0x7f4828105620 2023-11-16T17:01:32.953 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2023-11-16T17:01:32.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.956+0000 7f482e7ff700 1 -- 172.21.15.44:0/192553374 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f48140a1340 msgr2=0x7f48140a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:32.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.956+0000 7f482e7ff700 1 --2- 172.21.15.44:0/192553374 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f48140a1340 0x7f48140a37f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f48180097b0 tx=0x7f4818006d20 comp rx=0 tx=0).stop 2023-11-16T17:01:32.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.956+0000 7f482e7ff700 1 -- 172.21.15.44:0/192553374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4828105620 msgr2=0x7f4828198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:32.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.956+0000 7f482e7ff700 1 --2- 172.21.15.44:0/192553374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4828105620 0x7f4828198660 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f4820009fd0 tx=0x7f4820007910 comp rx=0 tx=0).stop 2023-11-16T17:01:32.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.956+0000 7f482e7ff700 1 -- 172.21.15.44:0/192553374 shutdown_connections 2023-11-16T17:01:32.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.956+0000 7f482e7ff700 1 --2- 172.21.15.44:0/192553374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4828105620 0x7f4828198660 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:32.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.956+0000 7f482e7ff700 1 --2- 172.21.15.44:0/192553374 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f48140a1340 0x7f48140a37f0 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f48180097b0 tx=0x7f4818006d20 comp rx=0 tx=0).stop 2023-11-16T17:01:32.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.956+0000 7f482e7ff700 1 --2- 172.21.15.44:0/192553374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4828100cc0 0x7f482819e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:32.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.956+0000 7f482e7ff700 1 --2- 172.21.15.44:0/192553374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4828101660 0x7f482819eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:32.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.956+0000 7f482e7ff700 1 -- 172.21.15.44:0/192553374 >> 172.21.15.44:0/192553374 conn(0x7f4828077560 msgr2=0x7f48280fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:32.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.956+0000 7f482e7ff700 1 -- 172.21.15.44:0/192553374 shutdown_connections 2023-11-16T17:01:32.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:32.956+0000 7f482e7ff700 1 -- 172.21.15.44:0/192553374 wait complete. 2023-11-16T17:01:32.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:32 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3412200062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T17:01:32.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:32 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3412200062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T17:01:32.991 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T17:01:33.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.368+0000 7f26cf9ff700 1 -- 172.21.15.44:0/1370128956 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c8100c50 msgr2=0x7f26c8101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:33.370 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.368+0000 7f26cf9ff700 1 --2- 172.21.15.44:0/1370128956 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c8100c50 0x7f26c8101020 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f26c000b0a0 tx=0x7f26c000b3b0 comp rx=0 tx=0).stop 2023-11-16T17:01:33.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.368+0000 7f26cf9ff700 1 -- 172.21.15.44:0/1370128956 shutdown_connections 2023-11-16T17:01:33.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.368+0000 7f26cf9ff700 1 --2- 172.21.15.44:0/1370128956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c8105630 0x7f26c810d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:33.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.368+0000 7f26cf9ff700 1 --2- 172.21.15.44:0/1370128956 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c8100c50 0x7f26c8101020 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:33.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.368+0000 7f26cf9ff700 1 --2- 172.21.15.44:0/1370128956 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26c81015f0 0x7f26c8104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:33.371 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.368+0000 7f26cf9ff700 1 -- 172.21.15.44:0/1370128956 >> 172.21.15.44:0/1370128956 conn(0x7f26c80775c0 msgr2=0x7f26c80779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:33.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.368+0000 7f26cf9ff700 1 -- 172.21.15.44:0/1370128956 shutdown_connections 2023-11-16T17:01:33.372 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.369+0000 7f26cf9ff700 1 -- 172.21.15.44:0/1370128956 wait complete. 2023-11-16T17:01:33.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.370+0000 7f26cf9ff700 1 Processor -- start 2023-11-16T17:01:33.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.370+0000 7f26cf9ff700 1 -- start start 2023-11-16T17:01:33.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.371+0000 7f26cf9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c8100c50 0x7f26c8107160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:33.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.371+0000 7f26cf9ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26c81015f0 0x7f26c81076a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:33.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.371+0000 7f26cf9ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c8105630 0x7f26c8107be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:33.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.371+0000 7f26cf9ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c81a8f00 con 0x7f26c81015f0 2023-11-16T17:01:33.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.371+0000 7f26cf9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c81a9070 con 0x7f26c8100c50 2023-11-16T17:01:33.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.371+0000 7f26cf9ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c81a9230 con 0x7f26c8105630 2023-11-16T17:01:33.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.371+0000 7f26cd79b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c8100c50 0x7f26c8107160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:33.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.371+0000 7f26cd79b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c8100c50 0x7f26c8107160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:48954/0 (socket says 172.21.15.44:48954) 2023-11-16T17:01:33.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.371+0000 7f26cd79b700 1 -- 172.21.15.44:0/3583398577 learned_addr learned my addr 172.21.15.44:0/3583398577 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:01:33.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.371+0000 7f26ccf9a700 1 --2- 172.21.15.44:0/3583398577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26c81015f0 0x7f26c81076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:33.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.371+0000 7f26cdf9c700 1 --2- 172.21.15.44:0/3583398577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c8105630 0x7f26c8107be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:33.376 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.372+0000 7f26cd79b700 1 -- 172.21.15.44:0/3583398577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c8105630 msgr2=0x7f26c8107be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:33.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.372+0000 7f26cd79b700 1 --2- 172.21.15.44:0/3583398577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c8105630 0x7f26c8107be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:33.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.372+0000 7f26cd79b700 1 -- 172.21.15.44:0/3583398577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26c81015f0 msgr2=0x7f26c81076a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:33.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.372+0000 7f26cd79b700 1 --2- 172.21.15.44:0/3583398577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26c81015f0 0x7f26c81076a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:33.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.372+0000 7f26cd79b700 1 -- 172.21.15.44:0/3583398577 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26bc009520 con 0x7f26c8100c50 2023-11-16T17:01:33.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.372+0000 7f26ccf9a700 1 --2- 172.21.15.44:0/3583398577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26c81015f0 0x7f26c81076a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T17:01:33.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.372+0000 7f26cdf9c700 1 --2- 172.21.15.44:0/3583398577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c8105630 0x7f26c8107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T17:01:33.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.373+0000 7f26cd79b700 1 --2- 172.21.15.44:0/3583398577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c8100c50 0x7f26c8107160 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f26c00077f0 tx=0x7f26c00078d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:33.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.373+0000 7f26ba7fc700 1 -- 172.21.15.44:0/3583398577 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c000e040 con 0x7f26c8100c50 2023-11-16T17:01:33.379 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.373+0000 7f26ba7fc700 1 -- 172.21.15.44:0/3583398577 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f26c0004e60 con 0x7f26c8100c50 2023-11-16T17:01:33.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.373+0000 7f26cf9ff700 1 -- 172.21.15.44:0/3583398577 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c0009d00 con 0x7f26c8100c50 2023-11-16T17:01:33.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.373+0000 7f26ba7fc700 1 -- 172.21.15.44:0/3583398577 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c001b6c0 con 0x7f26c8100c50 2023-11-16T17:01:33.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.374+0000 7f26cf9ff700 1 -- 172.21.15.44:0/3583398577 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c8110440 con 0x7f26c8100c50 2023-11-16T17:01:33.380 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.375+0000 7f26b3fff700 1 -- 172.21.15.44:0/3583398577 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c81102b0 con 0x7f26c8100c50 2023-11-16T17:01:33.381 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.378+0000 7f26ba7fc700 1 -- 172.21.15.44:0/3583398577 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f26c001d470 con 0x7f26c8100c50 2023-11-16T17:01:33.382 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.380+0000 7f26ba7fc700 1 --2- 172.21.15.44:0/3583398577 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f26b40a1400 0x7f26b40a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:33.382 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.380+0000 7f26ba7fc700 1 -- 172.21.15.44:0/3583398577 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9631+0+0 (secure 0 0 0) 0x7f26c00ce1f0 con 0x7f26c8100c50 2023-11-16T17:01:33.382 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.381+0000 7f26ccf9a700 1 --2- 172.21.15.44:0/3583398577 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f26b40a1400 0x7f26b40a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:33.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.385+0000 7f26ccf9a700 1 --2- 172.21.15.44:0/3583398577 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f26b40a1400 0x7f26b40a38b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f26c8108950 tx=0x7f26bc009260 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:33.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.391+0000 7f26ba7fc700 1 -- 172.21.15.44:0/3583398577 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f26c00971e0 con 0x7f26c8100c50 2023-11-16T17:01:33.802 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.800+0000 7f26b3fff700 1 -- 172.21.15.44:0/3583398577 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f26c81105d0 con 0x7f26c8100c50 2023-11-16T17:01:33.808 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.807+0000 7f26ba7fc700 1 -- 172.21.15.44:0/3583398577 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) v1 ==== 104+0+11 (secure 0 0 0) 0x7f26c009a800 con 0x7f26c8100c50 2023-11-16T17:01:33.809 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T17:01:33.815 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.814+0000 7f26b3fff700 1 -- 172.21.15.44:0/3583398577 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f26b40a1400 msgr2=0x7f26b40a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:33.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.814+0000 7f26b3fff700 1 --2- 172.21.15.44:0/3583398577 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f26b40a1400 0x7f26b40a38b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f26c8108950 tx=0x7f26bc009260 comp rx=0 tx=0).stop 2023-11-16T17:01:33.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.814+0000 7f26b3fff700 1 -- 172.21.15.44:0/3583398577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c8100c50 msgr2=0x7f26c8107160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:33.816 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.814+0000 7f26b3fff700 1 --2- 172.21.15.44:0/3583398577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c8100c50 0x7f26c8107160 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f26c00077f0 tx=0x7f26c00078d0 comp rx=0 tx=0).stop 2023-11-16T17:01:33.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.814+0000 7f26b3fff700 1 -- 172.21.15.44:0/3583398577 shutdown_connections 2023-11-16T17:01:33.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.814+0000 7f26b3fff700 1 --2- 172.21.15.44:0/3583398577 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f26b40a1400 0x7f26b40a38b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:33.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.814+0000 7f26b3fff700 1 --2- 172.21.15.44:0/3583398577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26c8105630 0x7f26c8107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:33.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.814+0000 7f26b3fff700 1 --2- 172.21.15.44:0/3583398577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c8100c50 0x7f26c8107160 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:33.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.814+0000 7f26b3fff700 1 --2- 172.21.15.44:0/3583398577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26c81015f0 0x7f26c81076a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:33.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.814+0000 7f26b3fff700 1 -- 172.21.15.44:0/3583398577 >> 172.21.15.44:0/3583398577 conn(0x7f26c80775c0 msgr2=0x7f26c80fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:33.818 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.815+0000 7f26b3fff700 1 -- 172.21.15.44:0/3583398577 shutdown_connections 2023-11-16T17:01:33.819 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:33.815+0000 7f26b3fff700 1 -- 172.21.15.44:0/3583398577 wait complete. 2023-11-16T17:01:33.852 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T17:01:33.897 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:33 smithi176 ceph-mon[131630]: pgmap v361: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:01:33.898 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:33 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/192553374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T17:01:33.970 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:33 smithi044 ceph-mon[128750]: pgmap v361: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:01:33.970 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:33 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/192553374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T17:01:33.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:33 smithi077 ceph-mon[131204]: pgmap v361: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:01:33.979 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:33 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/192553374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T17:01:34.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.236+0000 7f34ee3ff700 1 -- 172.21.15.44:0/2430256602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e81063f0 msgr2=0x7f34e81067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:34.238 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.236+0000 7f34ee3ff700 1 --2- 172.21.15.44:0/2430256602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e81063f0 0x7f34e81067c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f34d800b0d0 tx=0x7f34d800b3e0 comp rx=0 tx=0).stop 2023-11-16T17:01:34.239 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.238+0000 7f34ee3ff700 1 -- 172.21.15.44:0/2430256602 shutdown_connections 2023-11-16T17:01:34.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.238+0000 7f34ee3ff700 1 --2- 172.21.15.44:0/2430256602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34e8101400 0x7f34e810d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:34.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.238+0000 7f34ee3ff700 1 --2- 172.21.15.44:0/2430256602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e81063f0 0x7f34e81067c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:34.240 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.238+0000 7f34ee3ff700 1 --2- 172.21.15.44:0/2430256602 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f34e8100a50 0x7f34e8100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:34.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.238+0000 7f34ee3ff700 1 -- 172.21.15.44:0/2430256602 >> 172.21.15.44:0/2430256602 conn(0x7f34e8077590 msgr2=0x7f34e8077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:34.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.238+0000 7f34ee3ff700 1 -- 172.21.15.44:0/2430256602 shutdown_connections 2023-11-16T17:01:34.241 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.240+0000 7f34ee3ff700 1 -- 172.21.15.44:0/2430256602 wait complete. 2023-11-16T17:01:34.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.242+0000 7f34ee3ff700 1 Processor -- start 2023-11-16T17:01:34.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.242+0000 7f34ee3ff700 1 -- start start 2023-11-16T17:01:34.244 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.242+0000 7f34ee3ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f34e8100a50 0x7f34e810cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:34.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.242+0000 7f34ee3ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34e8101400 0x7f34e8109db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:34.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.242+0000 7f34ee3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e81063f0 0x7f34e810a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:34.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.242+0000 7f34ee3ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e810a990 con 0x7f34e8101400 2023-11-16T17:01:34.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.242+0000 7f34ee3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e810ab00 con 0x7f34e81063f0 2023-11-16T17:01:34.245 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.242+0000 7f34ee3ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e81a91c0 con 0x7f34e8100a50 2023-11-16T17:01:34.246 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.243+0000 7f34e7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f34e8100a50 0x7f34e810cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:34.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.243+0000 7f34e7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f34e8100a50 0x7f34e810cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:33544/0 (socket says 172.21.15.44:33544) 2023-11-16T17:01:34.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.243+0000 7f34e7fff700 1 -- 172.21.15.44:0/970499265 learned_addr learned my addr 172.21.15.44:0/970499265 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:01:34.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.243+0000 7f34ec99c700 1 --2- 172.21.15.44:0/970499265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e81063f0 0x7f34e810a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:34.247 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.244+0000 7f34ec99c700 1 -- 172.21.15.44:0/970499265 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f34e8100a50 msgr2=0x7f34e810cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:34.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.244+0000 7f34ec99c700 1 --2- 172.21.15.44:0/970499265 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f34e8100a50 0x7f34e810cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:34.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.244+0000 7f34ec99c700 1 -- 172.21.15.44:0/970499265 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34e8101400 msgr2=0x7f34e8109db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:34.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.244+0000 7f34ec99c700 1 --2- 172.21.15.44:0/970499265 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34e8101400 0x7f34e8109db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:34.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.244+0000 7f34ec99c700 1 -- 172.21.15.44:0/970499265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d8009d00 con 0x7f34e81063f0 2023-11-16T17:01:34.248 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.244+0000 7f34ec99c700 1 --2- 172.21.15.44:0/970499265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e81063f0 0x7f34e810a2f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f34e000d720 tx=0x7f34e000da30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:34.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.245+0000 7f34e5ffb700 1 -- 172.21.15.44:0/970499265 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0004220 con 0x7f34e81063f0 2023-11-16T17:01:34.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.245+0000 7f34e5ffb700 1 -- 172.21.15.44:0/970499265 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f34e0004cf0 con 0x7f34e81063f0 2023-11-16T17:01:34.249 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.245+0000 7f34ee3ff700 1 -- 172.21.15.44:0/970499265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e80748c0 con 0x7f34e81063f0 2023-11-16T17:01:34.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.245+0000 7f34e5ffb700 1 -- 172.21.15.44:0/970499265 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e000f690 con 0x7f34e81063f0 2023-11-16T17:01:34.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.245+0000 7f34ee3ff700 1 -- 172.21.15.44:0/970499265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e8074730 con 0x7f34e81063f0 2023-11-16T17:01:34.250 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.248+0000 7f34ee3ff700 1 -- 172.21.15.44:0/970499265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e8074a50 con 0x7f34e81063f0 2023-11-16T17:01:34.252 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.250+0000 7f34e5ffb700 1 -- 172.21.15.44:0/970499265 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f34e0004410 con 0x7f34e81063f0 2023-11-16T17:01:34.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.252+0000 7f34e5ffb700 1 --2- 172.21.15.44:0/970499265 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f34d40a1340 0x7f34d40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:34.254 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.252+0000 7f34e5ffb700 1 -- 172.21.15.44:0/970499265 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9631+0+0 (secure 0 0 0) 0x7f34e00ccd10 con 0x7f34e81063f0 2023-11-16T17:01:34.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.253+0000 7f34e7fff700 1 --2- 172.21.15.44:0/970499265 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f34d40a1340 0x7f34d40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:34.255 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.254+0000 7f34e7fff700 1 --2- 172.21.15.44:0/970499265 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f34d40a1340 0x7f34d40a37f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009fd0 tx=0x7f34d800b040 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:34.264 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.263+0000 7f34e5ffb700 1 -- 172.21.15.44:0/970499265 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f34e0095cd0 con 0x7f34e81063f0 2023-11-16T17:01:34.655 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.653+0000 7f34ee3ff700 1 -- 172.21.15.44:0/970499265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f34e8067b00 con 0x7f34e81063f0 2023-11-16T17:01:34.672 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.670+0000 7f34e5ffb700 1 -- 172.21.15.44:0/970499265 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) v1 ==== 100+0+11 (secure 0 0 0) 0x7f34e00160a0 con 0x7f34e81063f0 2023-11-16T17:01:34.672 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2023-11-16T17:01:34.679 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.677+0000 7f34ee3ff700 1 -- 172.21.15.44:0/970499265 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f34d40a1340 msgr2=0x7f34d40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.677+0000 7f34ee3ff700 1 --2- 172.21.15.44:0/970499265 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f34d40a1340 0x7f34d40a37f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009fd0 tx=0x7f34d800b040 comp rx=0 tx=0).stop 2023-11-16T17:01:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.677+0000 7f34ee3ff700 1 -- 172.21.15.44:0/970499265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e81063f0 msgr2=0x7f34e810a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.677+0000 7f34ee3ff700 1 --2- 172.21.15.44:0/970499265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e81063f0 0x7f34e810a2f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f34e000d720 tx=0x7f34e000da30 comp rx=0 tx=0).stop 2023-11-16T17:01:34.680 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.678+0000 7f34ee3ff700 1 -- 172.21.15.44:0/970499265 shutdown_connections 2023-11-16T17:01:34.681 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.678+0000 7f34ee3ff700 1 --2- 172.21.15.44:0/970499265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e81063f0 0x7f34e810a2f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:34.681 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.678+0000 7f34ee3ff700 1 --2- 172.21.15.44:0/970499265 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f34d40a1340 0x7f34d40a37f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:34.681 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.678+0000 7f34ee3ff700 1 --2- 172.21.15.44:0/970499265 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f34e8100a50 0x7f34e810cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:34.681 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.678+0000 7f34ee3ff700 1 --2- 172.21.15.44:0/970499265 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34e8101400 0x7f34e8109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:34.681 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.678+0000 7f34ee3ff700 1 -- 172.21.15.44:0/970499265 >> 172.21.15.44:0/970499265 conn(0x7f34e8077590 msgr2=0x7f34e80ff720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:34.682 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.678+0000 7f34ee3ff700 1 -- 172.21.15.44:0/970499265 shutdown_connections 2023-11-16T17:01:34.682 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:34.678+0000 7f34ee3ff700 1 -- 172.21.15.44:0/970499265 wait complete. 2023-11-16T17:01:34.715 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T17:01:34.978 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:34 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3583398577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T17:01:35.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.090+0000 7fa9345ff700 1 -- 172.21.15.44:0/2560009830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa92c108400 msgr2=0x7fa92c112e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:35.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.090+0000 7fa9345ff700 1 --2- 172.21.15.44:0/2560009830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa92c108400 0x7fa92c112e70 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa91c009990 tx=0x7fa91c009ca0 comp rx=0 tx=0).stop 2023-11-16T17:01:35.092 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.090+0000 7fa9345ff700 1 -- 172.21.15.44:0/2560009830 shutdown_connections 2023-11-16T17:01:35.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.090+0000 7fa9345ff700 1 --2- 172.21.15.44:0/2560009830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa92c1133b0 0x7fa92c115790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:35.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.090+0000 7fa9345ff700 1 --2- 172.21.15.44:0/2560009830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa92c107af0 0x7fa92c107ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:35.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.090+0000 7fa9345ff700 1 --2- 172.21.15.44:0/2560009830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa92c108400 0x7fa92c112e70 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:35.093 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.090+0000 7fa9345ff700 1 -- 172.21.15.44:0/2560009830 >> 172.21.15.44:0/2560009830 conn(0x7fa92c06c7b0 msgr2=0x7fa92c071770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:35.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.094+0000 7fa9345ff700 1 -- 172.21.15.44:0/2560009830 shutdown_connections 2023-11-16T17:01:35.096 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.094+0000 7fa9345ff700 1 -- 172.21.15.44:0/2560009830 wait complete. 2023-11-16T17:01:35.097 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.096+0000 7fa9345ff700 1 Processor -- start 2023-11-16T17:01:35.098 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.096+0000 7fa9345ff700 1 -- start start 2023-11-16T17:01:35.101 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.099+0000 7fa9345ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa92c107af0 0x7fa92c108cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:35.101 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.099+0000 7fa9345ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa92c108400 0x7fa92c109210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:35.101 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.099+0000 7fa9345ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa92c1133b0 0x7fa92c1acf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:35.101 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.099+0000 7fa9345ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c109a40 con 0x7fa92c1133b0 2023-11-16T17:01:35.102 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.099+0000 7fa9345ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c1ad470 con 0x7fa92c107af0 2023-11-16T17:01:35.102 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.099+0000 7fa9345ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c1ad5b0 con 0x7fa92c108400 2023-11-16T17:01:35.102 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.099+0000 7fa931b9a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa92c108400 0x7fa92c109210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:35.102 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.100+0000 7fa931b9a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa92c108400 0x7fa92c109210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48846/0 (socket says 172.21.15.44:48846) 2023-11-16T17:01:35.102 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.100+0000 7fa931b9a700 1 -- 172.21.15.44:0/1602082971 learned_addr learned my addr 172.21.15.44:0/1602082971 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:01:35.102 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.100+0000 7fa932b9c700 1 --2- 172.21.15.44:0/1602082971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa92c1133b0 0x7fa92c1acf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:35.102 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.100+0000 7fa931b9a700 1 -- 172.21.15.44:0/1602082971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa92c107af0 msgr2=0x7fa92c108cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:35.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.100+0000 7fa931b9a700 1 --2- 172.21.15.44:0/1602082971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa92c107af0 0x7fa92c108cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:35.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.100+0000 7fa931b9a700 1 -- 172.21.15.44:0/1602082971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa92c1133b0 msgr2=0x7fa92c1acf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:35.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.100+0000 7fa931b9a700 1 --2- 172.21.15.44:0/1602082971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa92c1133b0 0x7fa92c1acf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:35.103 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.100+0000 7fa931b9a700 1 -- 172.21.15.44:0/1602082971 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91c0095f0 con 0x7fa92c108400 2023-11-16T17:01:35.107 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.105+0000 7fa931b9a700 1 --2- 172.21.15.44:0/1602082971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa92c108400 0x7fa92c109210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa91c00c010 tx=0x7fa91c00f720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:35.107 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.106+0000 7fa9237fe700 1 -- 172.21.15.44:0/1602082971 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c01d070 con 0x7fa92c108400 2023-11-16T17:01:35.107 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.106+0000 7fa9345ff700 1 -- 172.21.15.44:0/1602082971 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa92c118c90 con 0x7fa92c108400 2023-11-16T17:01:35.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.106+0000 7fa9237fe700 1 -- 172.21.15.44:0/1602082971 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa91c00fc40 con 0x7fa92c108400 2023-11-16T17:01:35.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.106+0000 7fa9237fe700 1 -- 172.21.15.44:0/1602082971 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c017650 con 0x7fa92c108400 2023-11-16T17:01:35.108 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.106+0000 7fa9345ff700 1 -- 172.21.15.44:0/1602082971 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa92c118b00 con 0x7fa92c108400 2023-11-16T17:01:35.110 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.108+0000 7fa9217fa700 1 -- 172.21.15.44:0/1602082971 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa92c118e20 con 0x7fa92c108400 2023-11-16T17:01:35.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:34 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3583398577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T17:01:35.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.118+0000 7fa9237fe700 1 -- 172.21.15.44:0/1602082971 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fa91c0178b0 con 0x7fa92c108400 2023-11-16T17:01:35.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.120+0000 7fa9237fe700 1 --2- 172.21.15.44:0/1602082971 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa9180a1380 0x7fa9180a3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:35.123 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.120+0000 7fa9237fe700 1 -- 172.21.15.44:0/1602082971 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9631+0+0 (secure 0 0 0) 0x7fa91c0cf090 con 0x7fa92c108400 2023-11-16T17:01:35.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.122+0000 7fa9237fe700 1 -- 172.21.15.44:0/1602082971 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa91c0980e0 con 0x7fa92c108400 2023-11-16T17:01:35.124 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.122+0000 7fa93239b700 1 --2- 172.21.15.44:0/1602082971 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa9180a1380 0x7fa9180a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:35.125 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.124+0000 7fa93239b700 1 --2- 172.21.15.44:0/1602082971 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa9180a1380 0x7fa9180a3830 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa9280097b0 tx=0x7fa928006d20 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:35.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:34 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3583398577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T17:01:35.540 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.538+0000 7fa9217fa700 1 -- 172.21.15.44:0/1602082971 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa92c067b00 con 0x7fa92c108400 2023-11-16T17:01:35.541 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.539+0000 7fa9237fe700 1 -- 172.21.15.44:0/1602082971 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v102) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa91c09b700 con 0x7fa92c108400 2023-11-16T17:01:35.542 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2023-11-16T17:01:35.544 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.543+0000 7fa9217fa700 1 -- 172.21.15.44:0/1602082971 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa9180a1380 msgr2=0x7fa9180a3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:35.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.543+0000 7fa9217fa700 1 --2- 172.21.15.44:0/1602082971 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa9180a1380 0x7fa9180a3830 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa9280097b0 tx=0x7fa928006d20 comp rx=0 tx=0).stop 2023-11-16T17:01:35.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.543+0000 7fa9217fa700 1 -- 172.21.15.44:0/1602082971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa92c108400 msgr2=0x7fa92c109210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:35.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.543+0000 7fa9217fa700 1 --2- 172.21.15.44:0/1602082971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa92c108400 0x7fa92c109210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa91c00c010 tx=0x7fa91c00f720 comp rx=0 tx=0).stop 2023-11-16T17:01:35.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.543+0000 7fa9217fa700 1 -- 172.21.15.44:0/1602082971 shutdown_connections 2023-11-16T17:01:35.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.543+0000 7fa9217fa700 1 --2- 172.21.15.44:0/1602082971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa92c1133b0 0x7fa92c1acf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:35.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.543+0000 7fa9217fa700 1 --2- 172.21.15.44:0/1602082971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa92c107af0 0x7fa92c108cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:35.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.543+0000 7fa9217fa700 1 --2- 172.21.15.44:0/1602082971 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa9180a1380 0x7fa9180a3830 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:35.545 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.543+0000 7fa9217fa700 1 --2- 172.21.15.44:0/1602082971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa92c108400 0x7fa92c109210 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:35.546 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.543+0000 7fa9217fa700 1 -- 172.21.15.44:0/1602082971 >> 172.21.15.44:0/1602082971 conn(0x7fa92c06c7b0 msgr2=0x7fa92c10ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:35.546 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.545+0000 7fa9217fa700 1 -- 172.21.15.44:0/1602082971 shutdown_connections 2023-11-16T17:01:35.546 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.545+0000 7fa9217fa700 1 -- 172.21.15.44:0/1602082971 wait complete. 2023-11-16T17:01:35.582 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-16T17:01:35.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:35 smithi044 ceph-mon[128750]: pgmap v362: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:01:35.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:35 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/970499265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T17:01:35.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:35 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1602082971' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T17:01:35.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.945+0000 7fdf8c7ff700 1 -- 172.21.15.44:0/1245389550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf84101430 msgr2=0x7fdf84101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:35.947 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.945+0000 7fdf8c7ff700 1 --2- 172.21.15.44:0/1245389550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf84101430 0x7fdf84101800 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c00b0a0 tx=0x7fdf7c00b3b0 comp rx=0 tx=0).stop 2023-11-16T17:01:35.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.946+0000 7fdf8c7ff700 1 -- 172.21.15.44:0/1245389550 shutdown_connections 2023-11-16T17:01:35.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.946+0000 7fdf8c7ff700 1 --2- 172.21.15.44:0/1245389550 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf8410a770 0x7fdf8410cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:35.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.946+0000 7fdf8c7ff700 1 --2- 172.21.15.44:0/1245389550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf84101430 0x7fdf84101800 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:35.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.946+0000 7fdf8c7ff700 1 --2- 172.21.15.44:0/1245389550 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf84101d40 0x7fdf8410a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:35.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.946+0000 7fdf8c7ff700 1 -- 172.21.15.44:0/1245389550 >> 172.21.15.44:0/1245389550 conn(0x7fdf84076370 msgr2=0x7fdf84076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:35.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.947+0000 7fdf8c7ff700 1 -- 172.21.15.44:0/1245389550 shutdown_connections 2023-11-16T17:01:35.949 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.947+0000 7fdf8c7ff700 1 -- 172.21.15.44:0/1245389550 wait complete. 2023-11-16T17:01:35.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.949+0000 7fdf8c7ff700 1 Processor -- start 2023-11-16T17:01:35.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.950+0000 7fdf8c7ff700 1 -- start start 2023-11-16T17:01:35.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.951+0000 7fdf8c7ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf84101430 0x7fdf8419e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:35.953 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.951+0000 7fdf8c7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf84101d40 0x7fdf8419eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:35.953 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.951+0000 7fdf8c7ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf8410a770 0x7fdf841985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:35.953 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.951+0000 7fdf8c7ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf84198d80 con 0x7fdf8410a770 2023-11-16T17:01:35.953 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.951+0000 7fdf8c7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf84198ef0 con 0x7fdf84101d40 2023-11-16T17:01:35.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.951+0000 7fdf8c7ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf841990b0 con 0x7fdf84101430 2023-11-16T17:01:35.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.952+0000 7fdf8ad9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf8410a770 0x7fdf841985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:35.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.952+0000 7fdf8ad9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf8410a770 0x7fdf841985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34158/0 (socket says 172.21.15.44:34158) 2023-11-16T17:01:35.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.952+0000 7fdf8ad9c700 1 -- 172.21.15.44:0/1352966725 learned_addr learned my addr 172.21.15.44:0/1352966725 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:01:35.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.953+0000 7fdf8ad9c700 1 -- 172.21.15.44:0/1352966725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf84101430 msgr2=0x7fdf8419e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:35.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.953+0000 7fdf8ad9c700 1 --2- 172.21.15.44:0/1352966725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf84101430 0x7fdf8419e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:35.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.953+0000 7fdf8ad9c700 1 -- 172.21.15.44:0/1352966725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf84101d40 msgr2=0x7fdf8419eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:35.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.953+0000 7fdf8ad9c700 1 --2- 172.21.15.44:0/1352966725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf84101d40 0x7fdf8419eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:35.955 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.953+0000 7fdf8ad9c700 1 -- 172.21.15.44:0/1352966725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c009d00 con 0x7fdf8410a770 2023-11-16T17:01:35.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.956+0000 7fdf8ad9c700 1 --2- 172.21.15.44:0/1352966725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf8410a770 0x7fdf841985b0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400d980 tx=0x7fdf7400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:35.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.956+0000 7fdf7b7fe700 1 -- 172.21.15.44:0/1352966725 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74009940 con 0x7fdf8410a770 2023-11-16T17:01:35.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.956+0000 7fdf7b7fe700 1 -- 172.21.15.44:0/1352966725 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf74004830 con 0x7fdf8410a770 2023-11-16T17:01:35.958 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.956+0000 7fdf7b7fe700 1 -- 172.21.15.44:0/1352966725 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf740054e0 con 0x7fdf8410a770 2023-11-16T17:01:35.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.957+0000 7fdf8c7ff700 1 -- 172.21.15.44:0/1352966725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf84110280 con 0x7fdf8410a770 2023-11-16T17:01:35.959 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.957+0000 7fdf8c7ff700 1 -- 172.21.15.44:0/1352966725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf841100f0 con 0x7fdf8410a770 2023-11-16T17:01:35.960 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.958+0000 7fdf8c7ff700 1 -- 172.21.15.44:0/1352966725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf84110410 con 0x7fdf8410a770 2023-11-16T17:01:35.965 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.963+0000 7fdf7b7fe700 1 -- 172.21.15.44:0/1352966725 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fdf74004ca0 con 0x7fdf8410a770 2023-11-16T17:01:35.968 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.967+0000 7fdf7b7fe700 1 --2- 172.21.15.44:0/1352966725 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fdf700a1410 0x7fdf700a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:35.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.967+0000 7fdf7b7fe700 1 -- 172.21.15.44:0/1352966725 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9631+0+0 (secure 0 0 0) 0x7fdf740cd360 con 0x7fdf8410a770 2023-11-16T17:01:35.969 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.968+0000 7fdf8a59b700 1 --2- 172.21.15.44:0/1352966725 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fdf700a1410 0x7fdf700a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:35.971 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.969+0000 7fdf8a59b700 1 --2- 172.21.15.44:0/1352966725 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fdf700a1410 0x7fdf700a38c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c00b070 tx=0x7fdf7c00b7b0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:35 smithi077 ceph-mon[131204]: pgmap v362: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:01:35.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:35 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/970499265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T17:01:35.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:35 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1602082971' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T17:01:35.977 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:35.975+0000 7fdf7b7fe700 1 -- 172.21.15.44:0/1352966725 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdf74096420 con 0x7fdf8410a770 2023-11-16T17:01:36.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:35 smithi176 ceph-mon[131630]: pgmap v362: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2023-11-16T17:01:36.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:35 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/970499265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T17:01:36.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:35 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1602082971' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T17:01:36.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:36.392+0000 7fdf8c7ff700 1 -- 172.21.15.44:0/1352966725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdf84107d80 con 0x7fdf8410a770 2023-11-16T17:01:36.396 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:36.394+0000 7fdf7b7fe700 1 -- 172.21.15.44:0/1352966725 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v102) v1 ==== 81+0+943 (secure 0 0 0) 0x7fdf74017030 con 0x7fdf8410a770 2023-11-16T17:01:36.396 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/4218362219 2023-11-17T16:49:55.607497+0000 2023-11-16T17:01:36.397 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/304422349 2023-11-17T16:49:55.607497+0000 2023-11-16T17:01:36.397 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1983838729 2023-11-17T16:49:55.607497+0000 2023-11-16T17:01:36.397 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1018560948 2023-11-17T16:40:42.449496+0000 2023-11-16T17:01:36.397 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1599940692 2023-11-17T16:40:42.449496+0000 2023-11-16T17:01:36.397 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2937650254 2023-11-17T16:49:55.607497+0000 2023-11-16T17:01:36.397 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3409868354 2023-11-17T16:40:42.449496+0000 2023-11-16T17:01:36.397 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1915598320 2023-11-17T16:49:55.607497+0000 2023-11-16T17:01:36.397 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/4234382393 2023-11-17T16:40:42.449496+0000 2023-11-16T17:01:36.398 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/1852059921 2023-11-17T16:40:15.683996+0000 2023-11-16T17:01:36.398 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3135397911 2023-11-17T16:40:15.683996+0000 2023-11-16T17:01:36.398 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/1852059921 2023-11-17T16:40:15.683996+0000 2023-11-16T17:01:36.398 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3599504154 2023-11-17T16:40:15.683996+0000 2023-11-16T17:01:36.398 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/304422349 2023-11-17T16:49:55.607497+0000 2023-11-16T17:01:36.398 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/264068429 2023-11-17T16:40:15.683996+0000 2023-11-16T17:01:36.398 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/4234382393 2023-11-17T16:40:42.449496+0000 2023-11-16T17:01:36.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:36.399+0000 7fdf8c7ff700 1 -- 172.21.15.44:0/1352966725 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fdf700a1410 msgr2=0x7fdf700a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:36.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:36.399+0000 7fdf8c7ff700 1 --2- 172.21.15.44:0/1352966725 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fdf700a1410 0x7fdf700a38c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c00b070 tx=0x7fdf7c00b7b0 comp rx=0 tx=0).stop 2023-11-16T17:01:36.400 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:36.399+0000 7fdf8c7ff700 1 -- 172.21.15.44:0/1352966725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf8410a770 msgr2=0x7fdf841985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:36.399+0000 7fdf8c7ff700 1 --2- 172.21.15.44:0/1352966725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf8410a770 0x7fdf841985b0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400d980 tx=0x7fdf7400dd40 comp rx=0 tx=0).stop 2023-11-16T17:01:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:36.399+0000 7fdf8c7ff700 1 -- 172.21.15.44:0/1352966725 shutdown_connections 2023-11-16T17:01:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:36.399+0000 7fdf8c7ff700 1 --2- 172.21.15.44:0/1352966725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf8410a770 0x7fdf841985b0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:36.399+0000 7fdf8c7ff700 1 --2- 172.21.15.44:0/1352966725 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fdf700a1410 0x7fdf700a38c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:36.399+0000 7fdf8c7ff700 1 --2- 172.21.15.44:0/1352966725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf84101430 0x7fdf8419e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:36.399+0000 7fdf8c7ff700 1 --2- 172.21.15.44:0/1352966725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf84101d40 0x7fdf8419eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:36.399+0000 7fdf8c7ff700 1 -- 172.21.15.44:0/1352966725 >> 172.21.15.44:0/1352966725 conn(0x7fdf84076370 msgr2=0x7fdf840ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:36.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:36.401+0000 7fdf8c7ff700 1 -- 172.21.15.44:0/1352966725 shutdown_connections 2023-11-16T17:01:36.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:36.401+0000 7fdf8c7ff700 1 -- 172.21.15.44:0/1352966725 wait complete. 2023-11-16T17:01:36.405 INFO:teuthology.orchestra.run.smithi044.stderr:listed 16 entries 2023-11-16T17:01:36.435 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-16T17:01:36.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T17:01:36.435 DEBUG:teuthology.orchestra.run.smithi044:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-16T17:01:36.952 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:36 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1352966725' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T17:01:36.981 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2023-11-16T17:01:36.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T17:01:36.982 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-16T17:01:36.982 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:36 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1352966725' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T17:01:37.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T17:01:37.090 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2023-11-16T17:01:37.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:36 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1352966725' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T17:01:37.217 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-16T17:01:37.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T17:01:37.218 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-16T17:01:37.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T17:01:37.351 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link delete ceph-brx 2023-11-16T17:01:37.547 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-16T17:01:37.556 INFO:tasks.cephadm:Teardown begin 2023-11-16T17:01:37.557 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-16T17:01:37.557 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T17:01:37.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:37 smithi044 ceph-mon[128750]: pgmap v363: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:01:37.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.935+0000 7f91fadff700 1 -- 172.21.15.44:0/1667217625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91f4108580 msgr2=0x7f91f4108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:37.937 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.935+0000 7f91fadff700 1 --2- 172.21.15.44:0/1667217625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91f4108580 0x7f91f4108950 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f91ec00b0a0 tx=0x7f91ec00b3b0 comp rx=0 tx=0).stop 2023-11-16T17:01:37.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.936+0000 7f91fadff700 1 -- 172.21.15.44:0/1667217625 shutdown_connections 2023-11-16T17:01:37.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.936+0000 7f91fadff700 1 --2- 172.21.15.44:0/1667217625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91f4102f00 0x7f91f410f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:37.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.936+0000 7f91fadff700 1 --2- 172.21.15.44:0/1667217625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91f4108580 0x7f91f4108950 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:37.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.936+0000 7f91fadff700 1 --2- 172.21.15.44:0/1667217625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91f4102550 0x7f91f41029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:37.938 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.936+0000 7f91fadff700 1 -- 172.21.15.44:0/1667217625 >> 172.21.15.44:0/1667217625 conn(0x7f91f40fdff0 msgr2=0x7f91f4100400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:37.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.936+0000 7f91fadff700 1 -- 172.21.15.44:0/1667217625 shutdown_connections 2023-11-16T17:01:37.939 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.936+0000 7f91fadff700 1 -- 172.21.15.44:0/1667217625 wait complete. 2023-11-16T17:01:37.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.938+0000 7f91fadff700 1 Processor -- start 2023-11-16T17:01:37.940 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.939+0000 7f91fadff700 1 -- start start 2023-11-16T17:01:37.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.939+0000 7f91fadff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91f4102550 0x7f91f419e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:37.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.939+0000 7f91fadff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91f4102f00 0x7f91f419eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:37.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.939+0000 7f91fadff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91f4108580 0x7f91f4198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:37.941 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.940+0000 7f91f3fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91f4102f00 0x7f91f419eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:37.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.940+0000 7f91f3fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91f4102f00 0x7f91f419eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48858/0 (socket says 172.21.15.44:48858) 2023-11-16T17:01:37.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.940+0000 7f91f3fff700 1 -- 172.21.15.44:0/897918385 learned_addr learned my addr 172.21.15.44:0/897918385 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:01:37.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.940+0000 7f91fadff700 1 -- 172.21.15.44:0/897918385 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f4198c80 con 0x7f91f4102550 2023-11-16T17:01:37.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.940+0000 7f91fadff700 1 -- 172.21.15.44:0/897918385 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f4198df0 con 0x7f91f4108580 2023-11-16T17:01:37.942 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.940+0000 7f91fadff700 1 -- 172.21.15.44:0/897918385 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f4198fb0 con 0x7f91f4102f00 2023-11-16T17:01:37.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.941+0000 7f91f3fff700 1 -- 172.21.15.44:0/897918385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91f4108580 msgr2=0x7f91f4198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:37.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.941+0000 7f91f3fff700 1 --2- 172.21.15.44:0/897918385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91f4108580 0x7f91f4198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:37.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.941+0000 7f91f3fff700 1 -- 172.21.15.44:0/897918385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91f4102550 msgr2=0x7f91f419e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:37.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.941+0000 7f91f3fff700 1 --2- 172.21.15.44:0/897918385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91f4102550 0x7f91f419e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:37.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.941+0000 7f91f3fff700 1 -- 172.21.15.44:0/897918385 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec009d00 con 0x7f91f4102f00 2023-11-16T17:01:37.943 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.941+0000 7f91f3fff700 1 --2- 172.21.15.44:0/897918385 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91f4102f00 0x7f91f419eae0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f91e800b530 tx=0x7f91e800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:37.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.942+0000 7f91f1ffb700 1 -- 172.21.15.44:0/897918385 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e8011830 con 0x7f91f4102f00 2023-11-16T17:01:37.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.942+0000 7f91fadff700 1 -- 172.21.15.44:0/897918385 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f41126a0 con 0x7f91f4102f00 2023-11-16T17:01:37.944 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.942+0000 7f91f1ffb700 1 -- 172.21.15.44:0/897918385 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91e8011e50 con 0x7f91f4102f00 2023-11-16T17:01:37.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.942+0000 7f91f1ffb700 1 -- 172.21.15.44:0/897918385 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e800f2e0 con 0x7f91f4102f00 2023-11-16T17:01:37.945 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.942+0000 7f91fadff700 1 -- 172.21.15.44:0/897918385 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f4112510 con 0x7f91f4102f00 2023-11-16T17:01:37.948 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.947+0000 7f91fadff700 1 -- 172.21.15.44:0/897918385 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f4112830 con 0x7f91f4102f00 2023-11-16T17:01:37.950 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.948+0000 7f91f1ffb700 1 -- 172.21.15.44:0/897918385 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f91e8010580 con 0x7f91f4102f00 2023-11-16T17:01:37.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.951+0000 7f91f1ffb700 1 --2- 172.21.15.44:0/897918385 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f91dc0a1340 0x7f91dc0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:37.952 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.951+0000 7f91f1ffb700 1 -- 172.21.15.44:0/897918385 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9631+0+0 (secure 0 0 0) 0x7f91e80ccb00 con 0x7f91f4102f00 2023-11-16T17:01:37.953 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.951+0000 7f91f8b9b700 1 --2- 172.21.15.44:0/897918385 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f91dc0a1340 0x7f91dc0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:37.954 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.953+0000 7f91f8b9b700 1 --2- 172.21.15.44:0/897918385 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f91dc0a1340 0x7f91dc0a37f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f91ec00b070 tx=0x7f91ec00b7b0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:37.957 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:37.956+0000 7f91f1ffb700 1 -- 172.21.15.44:0/897918385 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f91e8095bd0 con 0x7f91f4102f00 2023-11-16T17:01:37.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:37 smithi077 ceph-mon[131204]: pgmap v363: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:01:38.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:37 smithi176 ceph-mon[131630]: pgmap v363: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2023-11-16T17:01:38.439 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.438+0000 7f91fadff700 1 -- 172.21.15.44:0/897918385 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91f4199bc0 con 0x7f91f4102f00 2023-11-16T17:01:38.440 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.439+0000 7f91f1ffb700 1 -- 172.21.15.44:0/897918385 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f91e80991f0 con 0x7f91f4102f00 2023-11-16T17:01:38.441 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T17:01:38.443 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T17:01:38.446 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.444+0000 7f91fadff700 1 -- 172.21.15.44:0/897918385 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f91dc0a1340 msgr2=0x7f91dc0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:38.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.444+0000 7f91fadff700 1 --2- 172.21.15.44:0/897918385 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f91dc0a1340 0x7f91dc0a37f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f91ec00b070 tx=0x7f91ec00b7b0 comp rx=0 tx=0).stop 2023-11-16T17:01:38.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.444+0000 7f91fadff700 1 -- 172.21.15.44:0/897918385 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91f4102f00 msgr2=0x7f91f419eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:38.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.444+0000 7f91fadff700 1 --2- 172.21.15.44:0/897918385 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91f4102f00 0x7f91f419eae0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f91e800b530 tx=0x7f91e800b8f0 comp rx=0 tx=0).stop 2023-11-16T17:01:38.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.444+0000 7f91fadff700 1 -- 172.21.15.44:0/897918385 shutdown_connections 2023-11-16T17:01:38.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.444+0000 7f91fadff700 1 --2- 172.21.15.44:0/897918385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91f4108580 0x7f91f4198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:38.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.444+0000 7f91fadff700 1 --2- 172.21.15.44:0/897918385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91f4102550 0x7f91f419e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:38.447 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.444+0000 7f91fadff700 1 --2- 172.21.15.44:0/897918385 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f91dc0a1340 0x7f91dc0a37f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:38.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.444+0000 7f91fadff700 1 --2- 172.21.15.44:0/897918385 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91f4102f00 0x7f91f419eae0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:38.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.444+0000 7f91fadff700 1 -- 172.21.15.44:0/897918385 >> 172.21.15.44:0/897918385 conn(0x7f91f40fdff0 msgr2=0x7f91f4106070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:38.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.445+0000 7f91fadff700 1 -- 172.21.15.44:0/897918385 shutdown_connections 2023-11-16T17:01:38.448 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.445+0000 7f91fadff700 1 -- 172.21.15.44:0/897918385 wait complete. 2023-11-16T17:01:38.448 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T17:01:38.481 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T17:01:38.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.850+0000 7f2dfb7ff700 1 -- 172.21.15.44:0/144569780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2df41133b0 msgr2=0x7f2df4115790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:38.852 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.850+0000 7f2dfb7ff700 1 --2- 172.21.15.44:0/144569780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2df41133b0 0x7f2df4115790 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2dec00b0a0 tx=0x7f2dec00b3b0 comp rx=0 tx=0).stop 2023-11-16T17:01:38.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.850+0000 7f2dfb7ff700 1 -- 172.21.15.44:0/144569780 shutdown_connections 2023-11-16T17:01:38.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.850+0000 7f2dfb7ff700 1 --2- 172.21.15.44:0/144569780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2df41133b0 0x7f2df4115790 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:38.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.850+0000 7f2dfb7ff700 1 --2- 172.21.15.44:0/144569780 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2df4107af0 0x7f2df4107ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:38.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.850+0000 7f2dfb7ff700 1 --2- 172.21.15.44:0/144569780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2df4108400 0x7f2df4112e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:38.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.850+0000 7f2dfb7ff700 1 -- 172.21.15.44:0/144569780 >> 172.21.15.44:0/144569780 conn(0x7f2df406c550 msgr2=0x7f2df40716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:38.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.850+0000 7f2dfb7ff700 1 -- 172.21.15.44:0/144569780 shutdown_connections 2023-11-16T17:01:38.853 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.850+0000 7f2dfb7ff700 1 -- 172.21.15.44:0/144569780 wait complete. 2023-11-16T17:01:38.854 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.852+0000 7f2dfb7ff700 1 Processor -- start 2023-11-16T17:01:38.854 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.853+0000 7f2dfb7ff700 1 -- start start 2023-11-16T17:01:38.855 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.853+0000 7f2dfb7ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2df4107af0 0x7f2df41a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:38.855 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.853+0000 7f2dfb7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2df4108400 0x7f2df41a1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:38.855 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.854+0000 7f2dfb7ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2df41133b0 0x7f2df41a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:38.855 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.854+0000 7f2dfb7ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df41a5f10 con 0x7f2df41133b0 2023-11-16T17:01:38.855 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.854+0000 7f2dfb7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df41a6080 con 0x7f2df4108400 2023-11-16T17:01:38.856 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.854+0000 7f2dfb7ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df41a6240 con 0x7f2df4107af0 2023-11-16T17:01:38.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.854+0000 7f2df9d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2df41133b0 0x7f2df41a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:38.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.854+0000 7f2df9d9c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2df41133b0 0x7f2df41a58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34188/0 (socket says 172.21.15.44:34188) 2023-11-16T17:01:38.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.854+0000 7f2df9d9c700 1 -- 172.21.15.44:0/1374494519 learned_addr learned my addr 172.21.15.44:0/1374494519 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:01:38.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.854+0000 7f2df959b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2df4107af0 0x7f2df41a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:38.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.854+0000 7f2df8d9a700 1 --2- 172.21.15.44:0/1374494519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2df4108400 0x7f2df41a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:38.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.855+0000 7f2df9d9c700 1 -- 172.21.15.44:0/1374494519 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2df4107af0 msgr2=0x7f2df41a0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:38.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.855+0000 7f2df9d9c700 1 --2- 172.21.15.44:0/1374494519 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2df4107af0 0x7f2df41a0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:38.857 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.855+0000 7f2df9d9c700 1 -- 172.21.15.44:0/1374494519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2df4108400 msgr2=0x7f2df41a1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:38.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.855+0000 7f2df9d9c700 1 --2- 172.21.15.44:0/1374494519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2df4108400 0x7f2df41a1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:38.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.855+0000 7f2df9d9c700 1 -- 172.21.15.44:0/1374494519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec009d00 con 0x7f2df41133b0 2023-11-16T17:01:38.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.855+0000 7f2df8d9a700 1 --2- 172.21.15.44:0/1374494519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2df4108400 0x7f2df41a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T17:01:38.858 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.855+0000 7f2df9d9c700 1 --2- 172.21.15.44:0/1374494519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2df41133b0 0x7f2df41a58d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009fd0 tx=0x7f2dec00ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:38.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.856+0000 7f2dea7fc700 1 -- 172.21.15.44:0/1374494519 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec00e040 con 0x7f2df41133b0 2023-11-16T17:01:38.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.856+0000 7f2dea7fc700 1 -- 172.21.15.44:0/1374494519 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2dec004de0 con 0x7f2df41133b0 2023-11-16T17:01:38.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.856+0000 7f2dea7fc700 1 -- 172.21.15.44:0/1374494519 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec01b6b0 con 0x7f2df41133b0 2023-11-16T17:01:38.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.856+0000 7f2dfb7ff700 1 -- 172.21.15.44:0/1374494519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df4118b90 con 0x7f2df41133b0 2023-11-16T17:01:38.859 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.856+0000 7f2dfb7ff700 1 -- 172.21.15.44:0/1374494519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df4118a00 con 0x7f2df41133b0 2023-11-16T17:01:38.860 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.858+0000 7f2dfb7ff700 1 -- 172.21.15.44:0/1374494519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df4118d20 con 0x7f2df41133b0 2023-11-16T17:01:38.866 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.864+0000 7f2dea7fc700 1 -- 172.21.15.44:0/1374494519 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f2dec003a00 con 0x7f2df41133b0 2023-11-16T17:01:38.868 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.867+0000 7f2dea7fc700 1 --2- 172.21.15.44:0/1374494519 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2de00a1490 0x7f2de00a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:38.869 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.867+0000 7f2dea7fc700 1 -- 172.21.15.44:0/1374494519 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9631+0+0 (secure 0 0 0) 0x7f2dec0cf0d0 con 0x7f2df41133b0 2023-11-16T17:01:38.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:38 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/897918385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T17:01:38.870 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.868+0000 7f2df959b700 1 --2- 172.21.15.44:0/1374494519 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2de00a1490 0x7f2de00a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:38.871 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.870+0000 7f2df959b700 1 --2- 172.21.15.44:0/1374494519 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2de00a1490 0x7f2de00a3940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009a10 tx=0x7f2df0009190 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:38.874 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:38.873+0000 7f2dea7fc700 1 -- 172.21.15.44:0/1374494519 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2dec098140 con 0x7f2df41133b0 2023-11-16T17:01:38.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:38 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/897918385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T17:01:39.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:38 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/897918385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T17:01:39.365 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.364+0000 7f2dfb7ff700 1 -- 172.21.15.44:0/1374494519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2df4067a60 con 0x7f2df41133b0 2023-11-16T17:01:39.367 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.366+0000 7f2dea7fc700 1 -- 172.21.15.44:0/1374494519 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f2dec09b760 con 0x7f2df41133b0 2023-11-16T17:01:39.369 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T17:01:39.369 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14460,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/3970258683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":3970258683},{"type":"v1","addr":"172.21.15.44:6835","nonce":3970258683}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/159805710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":159805710},{"type":"v1","addr":"172.21.15.44:6837","nonce":159805710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14523,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1792053180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1792053180},{"type":"v1","addr":"172.21.15.44:6839","nonce":1792053180}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/2080079420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":2080079420},{"type":"v1","addr":"172.21.15.44:6841","nonce":2080079420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2897858995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2897858995},{"type":"v1","addr":"172.21.15.77:6833","nonce":2897858995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6833/1226611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1226611},{"type":"v1","addr":"172.21.15.176:6833","nonce":1226611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1355935860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1355935860},{"type":"v1","addr":"172.21.15.77:6835","nonce":1355935860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1665062606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1665062606},{"type":"v1","addr":"172.21.15.176:6835","nonce":1665062606}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/440017436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":440017436},{"type":"v1","addr":"172.21.15.77:6837","nonce":440017436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3753678422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3753678422},{"type":"v1","addr":"172.21.15.176:6837","nonce":3753678422}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:18.150861+0000","modified":"2023-11-16T16:47:50.656307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.176:6839/2828920421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6838","nonce":2828920421},{"type":"v1","addr":"172.21.15.176:6839","nonce":2828920421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6839/1948116745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1948116745},{"type":"v1","addr":"172.21.15.77:6839","nonce":1948116745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T17:01:39.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.371+0000 7f2dfb7ff700 1 -- 172.21.15.44:0/1374494519 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2de00a1490 msgr2=0x7f2de00a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:39.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.371+0000 7f2dfb7ff700 1 --2- 172.21.15.44:0/1374494519 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2de00a1490 0x7f2de00a3940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009a10 tx=0x7f2df0009190 comp rx=0 tx=0).stop 2023-11-16T17:01:39.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.371+0000 7f2dfb7ff700 1 -- 172.21.15.44:0/1374494519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2df41133b0 msgr2=0x7f2df41a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:39.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.371+0000 7f2dfb7ff700 1 --2- 172.21.15.44:0/1374494519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2df41133b0 0x7f2df41a58d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009fd0 tx=0x7f2dec00ce90 comp rx=0 tx=0).stop 2023-11-16T17:01:39.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.371+0000 7f2dfb7ff700 1 -- 172.21.15.44:0/1374494519 shutdown_connections 2023-11-16T17:01:39.373 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.371+0000 7f2dfb7ff700 1 --2- 172.21.15.44:0/1374494519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2df41133b0 0x7f2df41a58d0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:39.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.371+0000 7f2dfb7ff700 1 --2- 172.21.15.44:0/1374494519 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f2de00a1490 0x7f2de00a3940 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:39.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.371+0000 7f2dfb7ff700 1 --2- 172.21.15.44:0/1374494519 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2df4107af0 0x7f2df41a0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:39.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.371+0000 7f2dfb7ff700 1 --2- 172.21.15.44:0/1374494519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2df4108400 0x7f2df41a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:39.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.371+0000 7f2dfb7ff700 1 -- 172.21.15.44:0/1374494519 >> 172.21.15.44:0/1374494519 conn(0x7f2df406c550 msgr2=0x7f2df40716c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:39.374 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.372+0000 7f2dfb7ff700 1 -- 172.21.15.44:0/1374494519 shutdown_connections 2023-11-16T17:01:39.375 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.372+0000 7f2dfb7ff700 1 -- 172.21.15.44:0/1374494519 wait complete. 2023-11-16T17:01:39.376 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2023-11-16T17:01:39.407 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T17:01:39.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.776+0000 7f4421bff700 1 -- 172.21.15.44:0/1121282954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f441c100510 msgr2=0x7f441c1008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:39.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.776+0000 7f4421bff700 1 --2- 172.21.15.44:0/1121282954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f441c100510 0x7f441c1008e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f440c0099c0 tx=0x7f440c009cd0 comp rx=0 tx=0).stop 2023-11-16T17:01:39.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.776+0000 7f4421bff700 1 -- 172.21.15.44:0/1121282954 shutdown_connections 2023-11-16T17:01:39.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.776+0000 7f4421bff700 1 --2- 172.21.15.44:0/1121282954 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f441c105550 0x7f441c10d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:39.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.776+0000 7f4421bff700 1 --2- 172.21.15.44:0/1121282954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f441c100510 0x7f441c1008e0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:39.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.776+0000 7f4421bff700 1 --2- 172.21.15.44:0/1121282954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c100eb0 0x7f441c104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:39.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.776+0000 7f4421bff700 1 -- 172.21.15.44:0/1121282954 >> 172.21.15.44:0/1121282954 conn(0x7f441c077050 msgr2=0x7f441c077450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:39.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.776+0000 7f4421bff700 1 -- 172.21.15.44:0/1121282954 shutdown_connections 2023-11-16T17:01:39.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.777+0000 7f4421bff700 1 -- 172.21.15.44:0/1121282954 wait complete. 2023-11-16T17:01:39.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.778+0000 7f4421bff700 1 Processor -- start 2023-11-16T17:01:39.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.779+0000 7f4421bff700 1 -- start start 2023-11-16T17:01:39.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.779+0000 7f4421bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c100510 0x7f441c06fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:39.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.779+0000 7f4421bff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f441c100eb0 0x7f441c070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:39.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.779+0000 7f4421bff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f441c105550 0x7f441c06ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:39.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.779+0000 7f4421bff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c06b640 con 0x7f441c105550 2023-11-16T17:01:39.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.779+0000 7f4421bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c06b7b0 con 0x7f441c100510 2023-11-16T17:01:39.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.779+0000 7f4421bff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c06b970 con 0x7f441c100eb0 2023-11-16T17:01:39.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.780+0000 7f441b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c100510 0x7f441c06fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:39.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.780+0000 7f441b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c100510 0x7f441c06fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:52674/0 (socket says 172.21.15.44:52674) 2023-11-16T17:01:39.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.780+0000 7f441b7fe700 1 -- 172.21.15.44:0/1491216826 learned_addr learned my addr 172.21.15.44:0/1491216826 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:01:39.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.780+0000 7f441b7fe700 1 -- 172.21.15.44:0/1491216826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f441c100eb0 msgr2=0x7f441c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:39.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.780+0000 7f441b7fe700 1 --2- 172.21.15.44:0/1491216826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f441c100eb0 0x7f441c070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:39.784 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.780+0000 7f441b7fe700 1 -- 172.21.15.44:0/1491216826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f441c105550 msgr2=0x7f441c06ae70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T17:01:39.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.782+0000 7f441b7fe700 1 --2- 172.21.15.44:0/1491216826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f441c105550 0x7f441c06ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:39.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.782+0000 7f441b7fe700 1 -- 172.21.15.44:0/1491216826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440c0095f0 con 0x7f441c100510 2023-11-16T17:01:39.785 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.783+0000 7f441b7fe700 1 --2- 172.21.15.44:0/1491216826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c100510 0x7f441c06fdb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f440c000c00 tx=0x7f440c0057d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:39.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.787+0000 7f4418ff9700 1 -- 172.21.15.44:0/1491216826 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c01d070 con 0x7f441c100510 2023-11-16T17:01:39.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.787+0000 7f4421bff700 1 -- 172.21.15.44:0/1491216826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441c110430 con 0x7f441c100510 2023-11-16T17:01:39.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.787+0000 7f4418ff9700 1 -- 172.21.15.44:0/1491216826 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f440c0040f0 con 0x7f441c100510 2023-11-16T17:01:39.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.787+0000 7f4418ff9700 1 -- 172.21.15.44:0/1491216826 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c021410 con 0x7f441c100510 2023-11-16T17:01:39.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.787+0000 7f4421bff700 1 -- 172.21.15.44:0/1491216826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441c1102a0 con 0x7f441c100510 2023-11-16T17:01:39.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.790+0000 7f4421bff700 1 -- 172.21.15.44:0/1491216826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f441c1105c0 con 0x7f441c100510 2023-11-16T17:01:39.793 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.792+0000 7f4418ff9700 1 -- 172.21.15.44:0/1491216826 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f440c02b440 con 0x7f441c100510 2023-11-16T17:01:39.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.794+0000 7f4418ff9700 1 --2- 172.21.15.44:0/1491216826 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f44040a1400 0x7f44040a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:39.796 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.794+0000 7f4418ff9700 1 -- 172.21.15.44:0/1491216826 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9631+0+0 (secure 0 0 0) 0x7f440c0ce150 con 0x7f441c100510 2023-11-16T17:01:39.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.795+0000 7f441affd700 1 --2- 172.21.15.44:0/1491216826 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f44040a1400 0x7f44040a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:39.797 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.796+0000 7f441affd700 1 --2- 172.21.15.44:0/1491216826 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f44040a1400 0x7f44040a38b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f441c06c640 tx=0x7f4410005c80 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:39.806 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:39.804+0000 7f4418ff9700 1 -- 172.21.15.44:0/1491216826 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f440c097140 con 0x7f441c100510 2023-11-16T17:01:39.976 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:39 smithi077 ceph-mon[131204]: pgmap v364: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 170 B/s wr, 2 op/s 2023-11-16T17:01:39.977 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:39 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1374494519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T17:01:40.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:39 smithi044 ceph-mon[128750]: pgmap v364: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 170 B/s wr, 2 op/s 2023-11-16T17:01:40.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:39 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1374494519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T17:01:40.148 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:39 smithi176 ceph-mon[131630]: pgmap v364: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 170 B/s wr, 2 op/s 2023-11-16T17:01:40.148 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:39 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1374494519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T17:01:40.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.210+0000 7f4421bff700 1 -- 172.21.15.44:0/1491216826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f441c06c420 con 0x7f441c100510 2023-11-16T17:01:40.216 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-11-16T17:01:40.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.214+0000 7f4418ff9700 1 -- 172.21.15.44:0/1491216826 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+25430 (secure 0 0 0) 0x7f440c09a760 con 0x7f441c100510 2023-11-16T17:01:40.217 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":102,"fsid":"7f2b12a8-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:41.796646+0000","modified":"2023-11-16T17:01:07.811232+0000","last_up_change":"2023-11-16T16:46:28.585321+0000","last_in_change":"2023-11-16T16:46:11.932762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:03.059622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:16.211826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:17.601591+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:21.136084+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a062dd5b-f36d-4686-86b7-b93b81bf317d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6803","nonce":2561264157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6805","nonce":2561264157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6809","nonce":2561264157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":2561264157},{"type":"v1","addr":"172.21.15.44:6807","nonce":2561264157}]},"public_addr":"172.21.15.44:6803/2561264157","cluster_addr":"172.21.15.44:6805/2561264157","heartbeat_back_addr":"172.21.15.44:6809/2561264157","heartbeat_front_addr":"172.21.15.44:6807/2561264157","state":["exists","up"]},{"osd":1,"uuid":"48ba3e8d-abd4-4253-a4aa-bc3d123a8bec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6801","nonce":1505198975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6803","nonce":1505198975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6807","nonce":1505198975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1505198975},{"type":"v1","addr":"172.21.15.77:6805","nonce":1505198975}]},"public_addr":"172.21.15.77:6801/1505198975","cluster_addr":"172.21.15.77:6803/1505198975","heartbeat_back_addr":"172.21.15.77:6807/1505198975","heartbeat_front_addr":"172.21.15.77:6805/1505198975","state":["exists","up"]},{"osd":2,"uuid":"98830193-aa89-42b5-a96b-ea72147286c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6801","nonce":1877349342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6803","nonce":1877349342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6807","nonce":1877349342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1877349342},{"type":"v1","addr":"172.21.15.176:6805","nonce":1877349342}]},"public_addr":"172.21.15.176:6801/1877349342","cluster_addr":"172.21.15.176:6803/1877349342","heartbeat_back_addr":"172.21.15.176:6807/1877349342","heartbeat_front_addr":"172.21.15.176:6805/1877349342","state":["exists","up"]},{"osd":3,"uuid":"ba9c2299-3850-4b08-b361-6ed673629019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6811","nonce":1041267584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6813","nonce":1041267584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6817","nonce":1041267584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1041267584},{"type":"v1","addr":"172.21.15.44:6815","nonce":1041267584}]},"public_addr":"172.21.15.44:6811/1041267584","cluster_addr":"172.21.15.44:6813/1041267584","heartbeat_back_addr":"172.21.15.44:6817/1041267584","heartbeat_front_addr":"172.21.15.44:6815/1041267584","state":["exists","up"]},{"osd":4,"uuid":"6a7ee641-fb5c-4a14-83b5-07c5b5842cfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6809","nonce":2688407497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6811","nonce":2688407497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6815","nonce":2688407497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2688407497},{"type":"v1","addr":"172.21.15.77:6813","nonce":2688407497}]},"public_addr":"172.21.15.77:6809/2688407497","cluster_addr":"172.21.15.77:6811/2688407497","heartbeat_back_addr":"172.21.15.77:6815/2688407497","heartbeat_front_addr":"172.21.15.77:6813/2688407497","state":["exists","up"]},{"osd":5,"uuid":"d16999a0-6bd4-4447-b88e-3271c8ab904a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6809","nonce":609622683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6811","nonce":609622683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6815","nonce":609622683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":609622683},{"type":"v1","addr":"172.21.15.176:6813","nonce":609622683}]},"public_addr":"172.21.15.176:6809/609622683","cluster_addr":"172.21.15.176:6811/609622683","heartbeat_back_addr":"172.21.15.176:6815/609622683","heartbeat_front_addr":"172.21.15.176:6813/609622683","state":["exists","up"]},{"osd":6,"uuid":"a31ca92c-d241-4731-8e79-a2aa0fda2c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6819","nonce":2634951849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6821","nonce":2634951849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6825","nonce":2634951849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2634951849},{"type":"v1","addr":"172.21.15.44:6823","nonce":2634951849}]},"public_addr":"172.21.15.44:6819/2634951849","cluster_addr":"172.21.15.44:6821/2634951849","heartbeat_back_addr":"172.21.15.44:6825/2634951849","heartbeat_front_addr":"172.21.15.44:6823/2634951849","state":["exists","up"]},{"osd":7,"uuid":"bb57b463-2ccc-4a4e-96d9-b4ecf5d2c969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6817","nonce":1651452313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6819","nonce":1651452313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6823","nonce":1651452313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1651452313},{"type":"v1","addr":"172.21.15.77:6821","nonce":1651452313}]},"public_addr":"172.21.15.77:6817/1651452313","cluster_addr":"172.21.15.77:6819/1651452313","heartbeat_back_addr":"172.21.15.77:6823/1651452313","heartbeat_front_addr":"172.21.15.77:6821/1651452313","state":["exists","up"]},{"osd":8,"uuid":"ba086b54-d6b6-4563-ade8-59c99f2ca473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6817","nonce":832350338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6819","nonce":832350338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6823","nonce":832350338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":832350338},{"type":"v1","addr":"172.21.15.176:6821","nonce":832350338}]},"public_addr":"172.21.15.176:6817/832350338","cluster_addr":"172.21.15.176:6819/832350338","heartbeat_back_addr":"172.21.15.176:6823/832350338","heartbeat_front_addr":"172.21.15.176:6821/832350338","state":["exists","up"]},{"osd":9,"uuid":"88e6c45c-9bac-49a2-b71d-2a6932bae792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6827","nonce":1897410398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6829","nonce":1897410398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6833","nonce":1897410398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1897410398},{"type":"v1","addr":"172.21.15.44:6831","nonce":1897410398}]},"public_addr":"172.21.15.44:6827/1897410398","cluster_addr":"172.21.15.44:6829/1897410398","heartbeat_back_addr":"172.21.15.44:6833/1897410398","heartbeat_front_addr":"172.21.15.44:6831/1897410398","state":["exists","up"]},{"osd":10,"uuid":"a6f87a5d-f4a5-45a6-aa9d-1cd84e948d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6825","nonce":3918521093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6827","nonce":3918521093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6831","nonce":3918521093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3918521093},{"type":"v1","addr":"172.21.15.77:6829","nonce":3918521093}]},"public_addr":"172.21.15.77:6825/3918521093","cluster_addr":"172.21.15.77:6827/3918521093","heartbeat_back_addr":"172.21.15.77:6831/3918521093","heartbeat_front_addr":"172.21.15.77:6829/3918521093","state":["exists","up"]},{"osd":11,"uuid":"ca5bd483-91ad-4385-922d-129f21cf955a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6825","nonce":1098488101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6827","nonce":1098488101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6831","nonce":1098488101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1098488101},{"type":"v1","addr":"172.21.15.176:6829","nonce":1098488101}]},"public_addr":"172.21.15.176:6825/1098488101","cluster_addr":"172.21.15.176:6827/1098488101","heartbeat_back_addr":"172.21.15.176:6831/1098488101","heartbeat_front_addr":"172.21.15.176:6829/1098488101","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:17.443697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:39.147351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:59.854065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:26.345424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:48.154691+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:09.120539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:35.690802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:57.788048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:18.503848+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:45.137646+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:07.071894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:27.505225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/4218362219":"2023-11-17T16:49:55.607497+0000","172.21.15.44:6801/304422349":"2023-11-17T16:49:55.607497+0000","172.21.15.44:0/1983838729":"2023-11-17T16:49:55.607497+0000","172.21.15.44:0/1018560948":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1599940692":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/2937650254":"2023-11-17T16:49:55.607497+0000","172.21.15.44:0/3409868354":"2023-11-17T16:40:42.449496+0000","172.21.15.44:0/1915598320":"2023-11-17T16:49:55.607497+0000","172.21.15.44:6801/4234382393":"2023-11-17T16:40:42.449496+0000","172.21.15.44:6801/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3135397911":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/1852059921":"2023-11-17T16:40:15.683996+0000","172.21.15.44:0/3599504154":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/304422349":"2023-11-17T16:49:55.607497+0000","172.21.15.44:0/264068429":"2023-11-17T16:40:15.683996+0000","172.21.15.44:6800/4234382393":"2023-11-17T16:40:42.449496+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":2},{"begin":17,"length":1},{"begin":19,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":2},{"begin":17,"length":1},{"begin":19,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]},{"pool":4,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T17:01:40.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.218+0000 7f4421bff700 1 -- 172.21.15.44:0/1491216826 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f44040a1400 msgr2=0x7f44040a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:40.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.218+0000 7f4421bff700 1 --2- 172.21.15.44:0/1491216826 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f44040a1400 0x7f44040a38b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f441c06c640 tx=0x7f4410005c80 comp rx=0 tx=0).stop 2023-11-16T17:01:40.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.218+0000 7f4421bff700 1 -- 172.21.15.44:0/1491216826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c100510 msgr2=0x7f441c06fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:40.220 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.218+0000 7f4421bff700 1 --2- 172.21.15.44:0/1491216826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c100510 0x7f441c06fdb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f440c000c00 tx=0x7f440c0057d0 comp rx=0 tx=0).stop 2023-11-16T17:01:40.221 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.219+0000 7f4421bff700 1 -- 172.21.15.44:0/1491216826 shutdown_connections 2023-11-16T17:01:40.221 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.219+0000 7f4421bff700 1 --2- 172.21.15.44:0/1491216826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f441c105550 0x7f441c06ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:40.221 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.219+0000 7f4421bff700 1 --2- 172.21.15.44:0/1491216826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c100510 0x7f441c06fdb0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:40.221 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.219+0000 7f4421bff700 1 --2- 172.21.15.44:0/1491216826 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f44040a1400 0x7f44040a38b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:40.221 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.219+0000 7f4421bff700 1 --2- 172.21.15.44:0/1491216826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f441c100eb0 0x7f441c070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:40.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.219+0000 7f4421bff700 1 -- 172.21.15.44:0/1491216826 >> 172.21.15.44:0/1491216826 conn(0x7f441c077050 msgr2=0x7f441c103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:40.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.219+0000 7f4421bff700 1 -- 172.21.15.44:0/1491216826 shutdown_connections 2023-11-16T17:01:40.222 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.219+0000 7f4421bff700 1 -- 172.21.15.44:0/1491216826 wait complete. 2023-11-16T17:01:40.256 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-16T17:01:40.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.610+0000 7f3a6f9ff700 1 -- 172.21.15.44:0/3892034594 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a6810cb80 msgr2=0x7f3a6810ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:40.612 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.610+0000 7f3a6f9ff700 1 --2- 172.21.15.44:0/3892034594 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a6810cb80 0x7f3a6810ef60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009f90 tx=0x7f3a6000b330 comp rx=0 tx=0).stop 2023-11-16T17:01:40.614 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.613+0000 7f3a6f9ff700 1 -- 172.21.15.44:0/3892034594 shutdown_connections 2023-11-16T17:01:40.614 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.613+0000 7f3a6f9ff700 1 --2- 172.21.15.44:0/3892034594 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a6810cb80 0x7f3a6810ef60 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:40.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.613+0000 7f3a6f9ff700 1 --2- 172.21.15.44:0/3892034594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a680fefd0 0x7f3a680ff3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:40.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.613+0000 7f3a6f9ff700 1 --2- 172.21.15.44:0/3892034594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a680ff8e0 0x7f3a6810c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:40.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.613+0000 7f3a6f9ff700 1 -- 172.21.15.44:0/3892034594 >> 172.21.15.44:0/3892034594 conn(0x7f3a68076370 msgr2=0x7f3a68076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:40.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.613+0000 7f3a6f9ff700 1 -- 172.21.15.44:0/3892034594 shutdown_connections 2023-11-16T17:01:40.615 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.613+0000 7f3a6f9ff700 1 -- 172.21.15.44:0/3892034594 wait complete. 2023-11-16T17:01:40.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.614+0000 7f3a6f9ff700 1 Processor -- start 2023-11-16T17:01:40.616 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.615+0000 7f3a6f9ff700 1 -- start start 2023-11-16T17:01:40.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.615+0000 7f3a6f9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a680fefd0 0x7f3a68194c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:40.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.615+0000 7f3a6f9ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a680ff8e0 0x7f3a68195160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:40.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.615+0000 7f3a6f9ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a6810cb80 0x7f3a68198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:40.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.615+0000 7f3a6f9ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a681992f0 con 0x7f3a680ff8e0 2023-11-16T17:01:40.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.615+0000 7f3a6f9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a68199460 con 0x7f3a680fefd0 2023-11-16T17:01:40.617 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.615+0000 7f3a6f9ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a68199620 con 0x7f3a6810cb80 2023-11-16T17:01:40.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.616+0000 7f3a6cf9a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a680ff8e0 0x7f3a68195160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:40.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.616+0000 7f3a6df9c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a6810cb80 0x7f3a68198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:40.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.616+0000 7f3a6df9c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a6810cb80 0x7f3a68198ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48900/0 (socket says 172.21.15.44:48900) 2023-11-16T17:01:40.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.616+0000 7f3a6df9c700 1 -- 172.21.15.44:0/1497942937 learned_addr learned my addr 172.21.15.44:0/1497942937 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:01:40.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.616+0000 7f3a6df9c700 1 -- 172.21.15.44:0/1497942937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a680fefd0 msgr2=0x7f3a68194c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:40.619 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.616+0000 7f3a6df9c700 1 --2- 172.21.15.44:0/1497942937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a680fefd0 0x7f3a68194c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:40.620 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.616+0000 7f3a6df9c700 1 -- 172.21.15.44:0/1497942937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a680ff8e0 msgr2=0x7f3a68195160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:40.620 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.616+0000 7f3a6df9c700 1 --2- 172.21.15.44:0/1497942937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a680ff8e0 0x7f3a68195160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:40.620 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.616+0000 7f3a6df9c700 1 -- 172.21.15.44:0/1497942937 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a60009c40 con 0x7f3a6810cb80 2023-11-16T17:01:40.620 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.617+0000 7f3a6df9c700 1 --2- 172.21.15.44:0/1497942937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a6810cb80 0x7f3a68198ce0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3a6000b6c0 tx=0x7f3a60004ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:40.620 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.617+0000 7f3a5e7fc700 1 -- 172.21.15.44:0/1497942937 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6000e040 con 0x7f3a6810cb80 2023-11-16T17:01:40.620 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.617+0000 7f3a6f9ff700 1 -- 172.21.15.44:0/1497942937 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a681039f0 con 0x7f3a6810cb80 2023-11-16T17:01:40.620 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.617+0000 7f3a5e7fc700 1 -- 172.21.15.44:0/1497942937 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a6000bbf0 con 0x7f3a6810cb80 2023-11-16T17:01:40.620 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.618+0000 7f3a6f9ff700 1 -- 172.21.15.44:0/1497942937 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a68103860 con 0x7f3a6810cb80 2023-11-16T17:01:40.621 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.618+0000 7f3a5e7fc700 1 -- 172.21.15.44:0/1497942937 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6000e040 con 0x7f3a6810cb80 2023-11-16T17:01:40.621 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.619+0000 7f3a6f9ff700 1 -- 172.21.15.44:0/1497942937 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a68103b80 con 0x7f3a6810cb80 2023-11-16T17:01:40.626 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.623+0000 7f3a5e7fc700 1 -- 172.21.15.44:0/1497942937 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f3a6001d470 con 0x7f3a6810cb80 2023-11-16T17:01:40.627 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.626+0000 7f3a5e7fc700 1 --2- 172.21.15.44:0/1497942937 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3a540a13a0 0x7f3a540a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:40.628 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.626+0000 7f3a5e7fc700 1 -- 172.21.15.44:0/1497942937 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9631+0+0 (secure 0 0 0) 0x7f3a600ce460 con 0x7f3a6810cb80 2023-11-16T17:01:40.628 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.626+0000 7f3a6d79b700 1 --2- 172.21.15.44:0/1497942937 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3a540a13a0 0x7f3a540a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:40.629 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.628+0000 7f3a6d79b700 1 --2- 172.21.15.44:0/1497942937 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3a540a13a0 0x7f3a540a3850 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3a640097b0 tx=0x7f3a64006d20 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:40.633 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:40.631+0000 7f3a5e7fc700 1 -- 172.21.15.44:0/1497942937 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3a60097450 con 0x7f3a6810cb80 2023-11-16T17:01:41.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:40 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1491216826' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T17:01:41.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:40 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1491216826' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T17:01:41.147 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:41.146+0000 7f3a6f9ff700 1 -- 172.21.15.44:0/1497942937 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3a680679f0 con 0x7f3a6810cb80 2023-11-16T17:01:41.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:40 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1491216826' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T17:01:41.751 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:41.748+0000 7f3a5e7fc700 1 -- 172.21.15.44:0/1497942937 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v22) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3a6009aa70 con 0x7f3a6810cb80 2023-11-16T17:01:41.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:41.755+0000 7f3a6f9ff700 1 -- 172.21.15.44:0/1497942937 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3a540a13a0 msgr2=0x7f3a540a3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:41.756 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:41.755+0000 7f3a6f9ff700 1 --2- 172.21.15.44:0/1497942937 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3a540a13a0 0x7f3a540a3850 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3a640097b0 tx=0x7f3a64006d20 comp rx=0 tx=0).stop 2023-11-16T17:01:41.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:41.755+0000 7f3a6f9ff700 1 -- 172.21.15.44:0/1497942937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a6810cb80 msgr2=0x7f3a68198ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:41.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:41.755+0000 7f3a6f9ff700 1 --2- 172.21.15.44:0/1497942937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a6810cb80 0x7f3a68198ce0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3a6000b6c0 tx=0x7f3a60004ae0 comp rx=0 tx=0).stop 2023-11-16T17:01:41.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:41.755+0000 7f3a6f9ff700 1 -- 172.21.15.44:0/1497942937 shutdown_connections 2023-11-16T17:01:41.757 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:41.755+0000 7f3a6f9ff700 1 --2- 172.21.15.44:0/1497942937 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f3a540a13a0 0x7f3a540a3850 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:41.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:41.755+0000 7f3a6f9ff700 1 --2- 172.21.15.44:0/1497942937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a6810cb80 0x7f3a68198ce0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:41.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:41.755+0000 7f3a6f9ff700 1 --2- 172.21.15.44:0/1497942937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a680fefd0 0x7f3a68194c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:41.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:41.755+0000 7f3a6f9ff700 1 --2- 172.21.15.44:0/1497942937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a680ff8e0 0x7f3a68195160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:41.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:41.755+0000 7f3a6f9ff700 1 -- 172.21.15.44:0/1497942937 >> 172.21.15.44:0/1497942937 conn(0x7f3a68076370 msgr2=0x7f3a680fc9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:41.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:41.756+0000 7f3a6f9ff700 1 -- 172.21.15.44:0/1497942937 shutdown_connections 2023-11-16T17:01:41.758 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:41.756+0000 7f3a6f9ff700 1 -- 172.21.15.44:0/1497942937 wait complete. 2023-11-16T17:01:41.759 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-16T17:01:41.793 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-16T17:01:42.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:41 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mon-a[128726]: 2023-11-16T17:01:41.735+0000 7fd6d5704700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-16T17:01:42.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:41 smithi044 ceph-mon[128750]: pgmap v365: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 170 B/s wr, 1 op/s 2023-11-16T17:01:42.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:41 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1497942937' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-11-16T17:01:42.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:41 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-11-16T17:01:42.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:41 smithi044 ceph-mon[128750]: osdmap e103: 12 total, 12 up, 12 in 2023-11-16T17:01:42.146 INFO:journalctl@ceph.mds.i.smithi176.stdout:Nov 16 17:01:41 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-i[157991]: ignoring --setuser ceph since I am not root 2023-11-16T17:01:42.146 INFO:journalctl@ceph.mds.i.smithi176.stdout:Nov 16 17:01:41 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-i[157991]: ignoring --setgroup ceph since I am not root 2023-11-16T17:01:42.146 INFO:journalctl@ceph.mds.i.smithi176.stdout:Nov 16 17:01:41 smithi176 ceph-mds[158015]: -- 172.21.15.176:0/493575122 <== mon.1 v2:172.21.15.176:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5580b1d27520 con 0x5580b1cfc800 2023-11-16T17:01:42.146 INFO:journalctl@ceph.mds.i.smithi176.stdout:Nov 16 17:01:41 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-i[157991]: starting mds.i at 2023-11-16T17:01:42.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:41 smithi176 ceph-mon[131630]: pgmap v365: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 170 B/s wr, 1 op/s 2023-11-16T17:01:42.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:41 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1497942937' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-11-16T17:01:42.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:41 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-11-16T17:01:42.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:41 smithi176 ceph-mon[131630]: osdmap e103: 12 total, 12 up, 12 in 2023-11-16T17:01:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.176+0000 7fa9a35ff700 1 -- 172.21.15.44:0/1828465670 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa99c107af0 msgr2=0x7fa99c10eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:42.178 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.176+0000 7fa9a35ff700 1 --2- 172.21.15.44:0/1828465670 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa99c107af0 0x7fa99c10eb50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa994009f90 tx=0x7fa99400b330 comp rx=0 tx=0).stop 2023-11-16T17:01:42.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.176+0000 7fa9a35ff700 1 -- 172.21.15.44:0/1828465670 shutdown_connections 2023-11-16T17:01:42.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.177+0000 7fa9a35ff700 1 --2- 172.21.15.44:0/1828465670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa99c119a40 0x7fa99c11be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:42.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.177+0000 7fa9a35ff700 1 --2- 172.21.15.44:0/1828465670 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa99c107af0 0x7fa99c10eb50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:42.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.177+0000 7fa9a35ff700 1 --2- 172.21.15.44:0/1828465670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa99c107ec0 0x7fa99c108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:42.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.177+0000 7fa9a35ff700 1 -- 172.21.15.44:0/1828465670 >> 172.21.15.44:0/1828465670 conn(0x7fa99c06c7b0 msgr2=0x7fa99c071770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:42.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.177+0000 7fa9a35ff700 1 -- 172.21.15.44:0/1828465670 shutdown_connections 2023-11-16T17:01:42.179 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.177+0000 7fa9a35ff700 1 -- 172.21.15.44:0/1828465670 wait complete. 2023-11-16T17:01:42.181 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.180+0000 7fa9a35ff700 1 Processor -- start 2023-11-16T17:01:42.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.181+0000 7fa9a35ff700 1 -- start start 2023-11-16T17:01:42.183 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.181+0000 7fa9a35ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa99c107af0 0x7fa99c10fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:42.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.181+0000 7fa9a35ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa99c107ec0 0x7fa99c110220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:42.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.182+0000 7fa9a35ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa99c119a40 0x7fa99c1b9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:42.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.182+0000 7fa9a139b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa99c107af0 0x7fa99c10fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:42.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.182+0000 7fa9a139b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa99c107af0 0x7fa99c10fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:52718/0 (socket says 172.21.15.44:52718) 2023-11-16T17:01:42.184 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.182+0000 7fa9a35ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c1b9d80 con 0x7fa99c107ec0 2023-11-16T17:01:42.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.182+0000 7fa9a139b700 1 -- 172.21.15.44:0/315366737 learned_addr learned my addr 172.21.15.44:0/315366737 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:01:42.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.182+0000 7fa9a35ff700 1 -- 172.21.15.44:0/315366737 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c1b9ef0 con 0x7fa99c107af0 2023-11-16T17:01:42.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.182+0000 7fa9a35ff700 1 -- 172.21.15.44:0/315366737 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c1ba0b0 con 0x7fa99c119a40 2023-11-16T17:01:42.185 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.183+0000 7fa9a1b9c700 1 --2- 172.21.15.44:0/315366737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa99c119a40 0x7fa99c1b9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:42.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.183+0000 7fa9a0b9a700 1 --2- 172.21.15.44:0/315366737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa99c107ec0 0x7fa99c110220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:42.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.183+0000 7fa9a139b700 1 -- 172.21.15.44:0/315366737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa99c119a40 msgr2=0x7fa99c1b9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:42.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.183+0000 7fa9a139b700 1 --2- 172.21.15.44:0/315366737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa99c119a40 0x7fa99c1b9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:42.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.183+0000 7fa9a139b700 1 -- 172.21.15.44:0/315366737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa99c107ec0 msgr2=0x7fa99c110220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:42.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.183+0000 7fa9a139b700 1 --2- 172.21.15.44:0/315366737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa99c107ec0 0x7fa99c110220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:42.186 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.183+0000 7fa9a139b700 1 -- 172.21.15.44:0/315366737 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa994009c40 con 0x7fa99c107af0 2023-11-16T17:01:42.187 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.184+0000 7fa9a139b700 1 --2- 172.21.15.44:0/315366737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa99c107af0 0x7fa99c10fce0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa99400bc20 tx=0x7fa99400bc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:42.189 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.185+0000 7fa9927fc700 1 -- 172.21.15.44:0/315366737 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99400e040 con 0x7fa99c107af0 2023-11-16T17:01:42.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.185+0000 7fa9927fc700 1 -- 172.21.15.44:0/315366737 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa994007910 con 0x7fa99c107af0 2023-11-16T17:01:42.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.185+0000 7fa9927fc700 1 -- 172.21.15.44:0/315366737 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99401b750 con 0x7fa99c107af0 2023-11-16T17:01:42.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.185+0000 7fa9a35ff700 1 -- 172.21.15.44:0/315366737 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99c11f220 con 0x7fa99c107af0 2023-11-16T17:01:42.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.185+0000 7fa9a35ff700 1 -- 172.21.15.44:0/315366737 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99c11f090 con 0x7fa99c107af0 2023-11-16T17:01:42.190 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.187+0000 7fa9a35ff700 1 -- 172.21.15.44:0/315366737 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa980005310 con 0x7fa99c107af0 2023-11-16T17:01:42.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.203+0000 7fa9927fc700 1 -- 172.21.15.44:0/315366737 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fa99401b8b0 con 0x7fa99c107af0 2023-11-16T17:01:42.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.205+0000 7fa9927fc700 1 --2- 172.21.15.44:0/315366737 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa9880a1400 0x7fa9880a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:42.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.205+0000 7fa9927fc700 1 -- 172.21.15.44:0/315366737 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9629+0+0 (secure 0 0 0) 0x7fa9940ce9a0 con 0x7fa99c107af0 2023-11-16T17:01:42.208 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.206+0000 7fa9927fc700 1 -- 172.21.15.44:0/315366737 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa9940fa7d0 con 0x7fa99c107af0 2023-11-16T17:01:42.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.207+0000 7fa9a0b9a700 1 --2- 172.21.15.44:0/315366737 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa9880a1400 0x7fa9880a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:42.209 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.208+0000 7fa9a0b9a700 1 --2- 172.21.15.44:0/315366737 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa9880a1400 0x7fa9880a38b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa99c118950 tx=0x7fa998009310 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:42.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:41 smithi077 ceph-mon[131204]: pgmap v365: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 170 B/s wr, 1 op/s 2023-11-16T17:01:42.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:41 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1497942937' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-11-16T17:01:42.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:41 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-11-16T17:01:42.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:41 smithi077 ceph-mon[131204]: osdmap e103: 12 total, 12 up, 12 in 2023-11-16T17:01:42.228 INFO:journalctl@ceph.mds.b.smithi077.stdout:Nov 16 17:01:41 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-b[156523]: ignoring --setuser ceph since I am not root 2023-11-16T17:01:42.228 INFO:journalctl@ceph.mds.b.smithi077.stdout:Nov 16 17:01:41 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-b[156523]: ignoring --setgroup ceph since I am not root 2023-11-16T17:01:42.228 INFO:journalctl@ceph.mds.b.smithi077.stdout:Nov 16 17:01:41 smithi077 ceph-mds[156547]: -- 172.21.15.77:0/850493623 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5565013eb520 con 0x5565013c0c00 2023-11-16T17:01:42.228 INFO:journalctl@ceph.mds.b.smithi077.stdout:Nov 16 17:01:41 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-b[156523]: starting mds.b at 2023-11-16T17:01:42.746 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.744+0000 7fa9a35ff700 1 -- 172.21.15.44:0/315366737 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa980005f60 con 0x7fa99c107af0 2023-11-16T17:01:43.373 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:42 smithi077 ceph-mon[131204]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-11-16T17:01:43.374 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:42 smithi077 ceph-mon[131204]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-16T17:01:43.374 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:42 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-11-16T17:01:43.374 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:42 smithi077 ceph-mon[131204]: fsmap cephfs:0/1 10 up:standby, 1 failed 2023-11-16T17:01:43.374 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:42 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T17:01:43.374 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:42 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/315366737' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T17:01:43.374 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:42 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T17:01:43.374 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:42 smithi176 ceph-mon[131630]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-11-16T17:01:43.375 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:42 smithi176 ceph-mon[131630]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-16T17:01:43.375 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:42 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-11-16T17:01:43.375 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:42 smithi176 ceph-mon[131630]: fsmap cephfs:0/1 10 up:standby, 1 failed 2023-11-16T17:01:43.375 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:42 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T17:01:43.375 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:42 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/315366737' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T17:01:43.375 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:42 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T17:01:43.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.789+0000 7fa9927fc700 1 -- 172.21.15.44:0/315366737 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa994097990 con 0x7fa99c107af0 2023-11-16T17:01:43.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.796+0000 7fa9a35ff700 1 -- 172.21.15.44:0/315366737 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa9880a1400 msgr2=0x7fa9880a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:43.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.796+0000 7fa9a35ff700 1 --2- 172.21.15.44:0/315366737 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa9880a1400 0x7fa9880a38b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa99c118950 tx=0x7fa998009310 comp rx=0 tx=0).stop 2023-11-16T17:01:43.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.796+0000 7fa9a35ff700 1 -- 172.21.15.44:0/315366737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa99c107af0 msgr2=0x7fa99c10fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:43.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.796+0000 7fa9a35ff700 1 --2- 172.21.15.44:0/315366737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa99c107af0 0x7fa99c10fce0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa99400bc20 tx=0x7fa99400bc50 comp rx=0 tx=0).stop 2023-11-16T17:01:43.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.796+0000 7fa9a35ff700 1 -- 172.21.15.44:0/315366737 shutdown_connections 2023-11-16T17:01:43.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.796+0000 7fa9a35ff700 1 --2- 172.21.15.44:0/315366737 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fa9880a1400 0x7fa9880a38b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:43.377 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.796+0000 7fa9a35ff700 1 --2- 172.21.15.44:0/315366737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa99c119a40 0x7fa99c1b9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:43.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.796+0000 7fa9a35ff700 1 --2- 172.21.15.44:0/315366737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa99c107af0 0x7fa99c10fce0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:43.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.796+0000 7fa9a35ff700 1 --2- 172.21.15.44:0/315366737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa99c107ec0 0x7fa99c110220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:43.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.796+0000 7fa9a35ff700 1 -- 172.21.15.44:0/315366737 >> 172.21.15.44:0/315366737 conn(0x7fa99c06c7b0 msgr2=0x7fa99c071750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:43.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.800+0000 7fa9a35ff700 1 -- 172.21.15.44:0/315366737 shutdown_connections 2023-11-16T17:01:43.378 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:42.802+0000 7fa9a35ff700 1 -- 172.21.15.44:0/315366737 wait complete. 2023-11-16T17:01:43.379 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-16T17:01:43.379 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:42 smithi044 ceph-mon[128750]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-11-16T17:01:43.379 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:42 smithi044 ceph-mon[128750]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-16T17:01:43.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:42 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-11-16T17:01:43.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:42 smithi044 ceph-mon[128750]: fsmap cephfs:0/1 10 up:standby, 1 failed 2023-11-16T17:01:43.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:42 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T17:01:43.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:42 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/315366737' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T17:01:43.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:42 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T17:01:43.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.771+0000 7f0c5359e700 1 -- 172.21.15.44:0/3532147321 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c4c095a20 msgr2=0x7f0c4c098f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:43.773 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.771+0000 7f0c5359e700 1 --2- 172.21.15.44:0/3532147321 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c4c095a20 0x7f0c4c098f60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0c540640e0 tx=0x7f0c54066fd0 comp rx=0 tx=0).stop 2023-11-16T17:01:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.772+0000 7f0c5359e700 1 -- 172.21.15.44:0/3532147321 shutdown_connections 2023-11-16T17:01:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.772+0000 7f0c5359e700 1 --2- 172.21.15.44:0/3532147321 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c4c099690 0x7f0c4c0a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.772+0000 7f0c5359e700 1 --2- 172.21.15.44:0/3532147321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c4c095080 0x7f0c4c095450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.772+0000 7f0c5359e700 1 --2- 172.21.15.44:0/3532147321 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c4c095a20 0x7f0c4c098f60 secure :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0c540640e0 tx=0x7f0c54066fd0 comp rx=0 tx=0).stop 2023-11-16T17:01:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.772+0000 7f0c5359e700 1 -- 172.21.15.44:0/3532147321 >> 172.21.15.44:0/3532147321 conn(0x7f0c4c00a400 msgr2=0x7f0c4c00a800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:43.774 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.772+0000 7f0c5359e700 1 -- 172.21.15.44:0/3532147321 shutdown_connections 2023-11-16T17:01:43.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.773+0000 7f0c5359e700 1 -- 172.21.15.44:0/3532147321 wait complete. 2023-11-16T17:01:43.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.774+0000 7f0c5359e700 1 Processor -- start 2023-11-16T17:01:43.776 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.775+0000 7f0c5359e700 1 -- start start 2023-11-16T17:01:43.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.775+0000 7f0c5359e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c4c095080 0x7f0c4c129570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:43.777 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.775+0000 7f0c5359e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c4c099690 0x7f0c4c129ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:43.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.775+0000 7f0c5359e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c4c12df10 0x7f0c4c12e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:43.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.776+0000 7f0c51d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c4c099690 0x7f0c4c129ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:43.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.776+0000 7f0c51d9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c4c099690 0x7f0c4c129ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:48942/0 (socket says 172.21.15.44:48942) 2023-11-16T17:01:43.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.776+0000 7f0c5259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c4c095080 0x7f0c4c129570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:43.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.776+0000 7f0c5259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c4c095080 0x7f0c4c129570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.44:52738/0 (socket says 172.21.15.44:52738) 2023-11-16T17:01:43.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.776+0000 7f0c51d9b700 1 -- 172.21.15.44:0/981968232 learned_addr learned my addr 172.21.15.44:0/981968232 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:01:43.778 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.776+0000 7f0c5359e700 1 -- 172.21.15.44:0/981968232 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c12e980 con 0x7f0c4c12df10 2023-11-16T17:01:43.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.776+0000 7f0c5359e700 1 -- 172.21.15.44:0/981968232 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c12eaf0 con 0x7f0c4c095080 2023-11-16T17:01:43.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.776+0000 7f0c5359e700 1 -- 172.21.15.44:0/981968232 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c12ecb0 con 0x7f0c4c099690 2023-11-16T17:01:43.779 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.776+0000 7f0c52d9d700 1 --2- 172.21.15.44:0/981968232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c4c12df10 0x7f0c4c12e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.777+0000 7f0c51d9b700 1 -- 172.21.15.44:0/981968232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c4c095080 msgr2=0x7f0c4c129570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.777+0000 7f0c51d9b700 1 --2- 172.21.15.44:0/981968232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c4c095080 0x7f0c4c129570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.777+0000 7f0c51d9b700 1 -- 172.21.15.44:0/981968232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c4c12df10 msgr2=0x7f0c4c12e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.777+0000 7f0c51d9b700 1 --2- 172.21.15.44:0/981968232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c4c12df10 0x7f0c4c12e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.777+0000 7f0c51d9b700 1 -- 172.21.15.44:0/981968232 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5404f760 con 0x7f0c4c099690 2023-11-16T17:01:43.780 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.777+0000 7f0c51d9b700 1 --2- 172.21.15.44:0/981968232 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c4c099690 0x7f0c4c129ab0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0c54050070 tx=0x7f0c5407e810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:43.782 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.781+0000 7f0c437fe700 1 -- 172.21.15.44:0/981968232 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5407a040 con 0x7f0c4c099690 2023-11-16T17:01:43.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.781+0000 7f0c437fe700 1 -- 172.21.15.44:0/981968232 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c5407f8e0 con 0x7f0c4c099690 2023-11-16T17:01:43.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.781+0000 7f0c437fe700 1 -- 172.21.15.44:0/981968232 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c54088630 con 0x7f0c4c099690 2023-11-16T17:01:43.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.781+0000 7f0c5359e700 1 -- 172.21.15.44:0/981968232 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c4c0a3240 con 0x7f0c4c099690 2023-11-16T17:01:43.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.781+0000 7f0c5359e700 1 -- 172.21.15.44:0/981968232 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4c0a30b0 con 0x7f0c4c099690 2023-11-16T17:01:43.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.783+0000 7f0c417fa700 1 -- 172.21.15.44:0/981968232 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c380052e0 con 0x7f0c4c099690 2023-11-16T17:01:43.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.786+0000 7f0c437fe700 1 -- 172.21.15.44:0/981968232 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f0c5407f420 con 0x7f0c4c099690 2023-11-16T17:01:43.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.788+0000 7f0c437fe700 1 --2- 172.21.15.44:0/981968232 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c3c0a1330 0x7f0c3c0a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:43.791 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.789+0000 7f0c437fe700 1 -- 172.21.15.44:0/981968232 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9629+0+0 (secure 0 0 0) 0x7f0c54138360 con 0x7f0c4c099690 2023-11-16T17:01:43.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.789+0000 7f0c5259c700 1 --2- 172.21.15.44:0/981968232 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c3c0a1330 0x7f0c3c0a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:43.792 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.790+0000 7f0c5259c700 1 --2- 172.21.15.44:0/981968232 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c3c0a1330 0x7f0c3c0a37e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0c44005fd0 tx=0x7f0c4400a560 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:43.798 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:43.797+0000 7f0c437fe700 1 -- 172.21.15.44:0/981968232 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0c540c2730 con 0x7f0c4c099690 2023-11-16T17:01:44.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:43 smithi044 ceph-mon[128750]: pgmap v367: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 204 B/s wr, 2 op/s 2023-11-16T17:01:44.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:43 smithi044 ceph-mon[128750]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-11-16T17:01:44.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:43 smithi044 ceph-mon[128750]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-11-16T17:01:44.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:43 smithi044 ceph-mon[128750]: Cluster is now healthy 2023-11-16T17:01:44.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:43 smithi044 ceph-mon[128750]: mds.? [v2:172.21.15.176:6838/2809415908,v1:172.21.15.176:6839/2809415908] up:boot 2023-11-16T17:01:44.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:43 smithi044 ceph-mon[128750]: mds.? [v2:172.21.15.77:6838/1069670609,v1:172.21.15.77:6839/1069670609] up:boot 2023-11-16T17:01:44.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:43 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-11-16T17:01:44.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:43 smithi044 ceph-mon[128750]: fsmap 12 up:standby 2023-11-16T17:01:44.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:43 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T17:01:44.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:43 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T17:01:44.121 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:43 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:01:44.122 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:43 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:01:44.122 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:43 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:01:44.122 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:43 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:01:44.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:43 smithi176 ceph-mon[131630]: pgmap v367: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 204 B/s wr, 2 op/s 2023-11-16T17:01:44.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:43 smithi176 ceph-mon[131630]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-11-16T17:01:44.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:43 smithi176 ceph-mon[131630]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-11-16T17:01:44.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:43 smithi176 ceph-mon[131630]: Cluster is now healthy 2023-11-16T17:01:44.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:43 smithi176 ceph-mon[131630]: mds.? [v2:172.21.15.176:6838/2809415908,v1:172.21.15.176:6839/2809415908] up:boot 2023-11-16T17:01:44.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:43 smithi176 ceph-mon[131630]: mds.? [v2:172.21.15.77:6838/1069670609,v1:172.21.15.77:6839/1069670609] up:boot 2023-11-16T17:01:44.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:43 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-11-16T17:01:44.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:43 smithi176 ceph-mon[131630]: fsmap 12 up:standby 2023-11-16T17:01:44.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:43 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T17:01:44.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:43 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T17:01:44.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:43 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:01:44.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:43 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:01:44.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:43 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:01:44.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:43 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:01:44.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:43 smithi077 ceph-mon[131204]: pgmap v367: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 204 B/s wr, 2 op/s 2023-11-16T17:01:44.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:43 smithi077 ceph-mon[131204]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-11-16T17:01:44.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:43 smithi077 ceph-mon[131204]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-11-16T17:01:44.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:43 smithi077 ceph-mon[131204]: Cluster is now healthy 2023-11-16T17:01:44.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:43 smithi077 ceph-mon[131204]: mds.? [v2:172.21.15.176:6838/2809415908,v1:172.21.15.176:6839/2809415908] up:boot 2023-11-16T17:01:44.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:43 smithi077 ceph-mon[131204]: mds.? [v2:172.21.15.77:6838/1069670609,v1:172.21.15.77:6839/1069670609] up:boot 2023-11-16T17:01:44.230 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:43 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-11-16T17:01:44.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:43 smithi077 ceph-mon[131204]: fsmap 12 up:standby 2023-11-16T17:01:44.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:43 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T17:01:44.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:43 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T17:01:44.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:43 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:01:44.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:43 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:01:44.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:43 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:01:44.231 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:43 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:01:44.239 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:44.237+0000 7f0c417fa700 1 -- 172.21.15.44:0/981968232 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0c38005f30 con 0x7f0c4c099690 2023-11-16T17:01:44.783 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:44.783+0000 7f0c437fe700 1 -- 172.21.15.44:0/981968232 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v104) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0c54104e40 con 0x7f0c4c099690 2023-11-16T17:01:44.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:44.786+0000 7f0c417fa700 1 -- 172.21.15.44:0/981968232 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c3c0a1330 msgr2=0x7f0c3c0a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:44.787 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:44.786+0000 7f0c417fa700 1 --2- 172.21.15.44:0/981968232 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c3c0a1330 0x7f0c3c0a37e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0c44005fd0 tx=0x7f0c4400a560 comp rx=0 tx=0).stop 2023-11-16T17:01:44.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:44.786+0000 7f0c417fa700 1 -- 172.21.15.44:0/981968232 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c4c099690 msgr2=0x7f0c4c129ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:44.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:44.786+0000 7f0c417fa700 1 --2- 172.21.15.44:0/981968232 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c4c099690 0x7f0c4c129ab0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0c54050070 tx=0x7f0c5407e810 comp rx=0 tx=0).stop 2023-11-16T17:01:44.788 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:44.788+0000 7f0c417fa700 1 -- 172.21.15.44:0/981968232 shutdown_connections 2023-11-16T17:01:44.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:44.788+0000 7f0c417fa700 1 --2- 172.21.15.44:0/981968232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c4c12df10 0x7f0c4c12e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:44.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:44.788+0000 7f0c417fa700 1 --2- 172.21.15.44:0/981968232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c4c095080 0x7f0c4c129570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:44.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:44.788+0000 7f0c417fa700 1 --2- 172.21.15.44:0/981968232 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f0c3c0a1330 0x7f0c3c0a37e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:44.789 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:44.788+0000 7f0c417fa700 1 --2- 172.21.15.44:0/981968232 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c4c099690 0x7f0c4c129ab0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:44.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:44.788+0000 7f0c417fa700 1 -- 172.21.15.44:0/981968232 >> 172.21.15.44:0/981968232 conn(0x7f0c4c00a400 msgr2=0x7f0c4c093020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:44.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:44.788+0000 7f0c417fa700 1 -- 172.21.15.44:0/981968232 shutdown_connections 2023-11-16T17:01:44.790 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:44.788+0000 7f0c417fa700 1 -- 172.21.15.44:0/981968232 wait complete. 2023-11-16T17:01:44.796 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' removed 2023-11-16T17:01:44.828 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-16T17:01:45.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:44 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/981968232' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:45.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:44 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:45.144 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:44 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/981968232' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:45.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:44 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.211+0000 7fca4ebff700 1 -- 172.21.15.44:0/1734251538 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca48105620 msgr2=0x7fca4810d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.211+0000 7fca4ebff700 1 --2- 172.21.15.44:0/1734251538 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca48105620 0x7fca4810d1a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0098f0 tx=0x7fca3c009c00 comp rx=0 tx=0).stop 2023-11-16T17:01:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.211+0000 7fca4ebff700 1 -- 172.21.15.44:0/1734251538 shutdown_connections 2023-11-16T17:01:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.211+0000 7fca4ebff700 1 --2- 172.21.15.44:0/1734251538 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca48105620 0x7fca4810d1a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.211+0000 7fca4ebff700 1 --2- 172.21.15.44:0/1734251538 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca48100cc0 0x7fca48101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.211+0000 7fca4ebff700 1 --2- 172.21.15.44:0/1734251538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca48101660 0x7fca48104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.211+0000 7fca4ebff700 1 -- 172.21.15.44:0/1734251538 >> 172.21.15.44:0/1734251538 conn(0x7fca48077560 msgr2=0x7fca48077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.211+0000 7fca4ebff700 1 -- 172.21.15.44:0/1734251538 shutdown_connections 2023-11-16T17:01:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.211+0000 7fca4ebff700 1 -- 172.21.15.44:0/1734251538 wait complete. 2023-11-16T17:01:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.213+0000 7fca4ebff700 1 Processor -- start 2023-11-16T17:01:45.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.213+0000 7fca4ebff700 1 -- start start 2023-11-16T17:01:45.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.214+0000 7fca4ebff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca48100cc0 0x7fca48198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:45.214 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.214+0000 7fca4ebff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca48101660 0x7fca48198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.214+0000 7fca4ebff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca48105620 0x7fca4819c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.214+0000 7fca4ebff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4819c610 con 0x7fca48105620 2023-11-16T17:01:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.214+0000 7fca4ebff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4819c780 con 0x7fca48101660 2023-11-16T17:01:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.214+0000 7fca4ebff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4819c940 con 0x7fca48100cc0 2023-11-16T17:01:45.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.214+0000 7fca4c99b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca48100cc0 0x7fca48198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:45.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.214+0000 7fca4c99b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca48100cc0 0x7fca48198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:50804/0 (socket says 172.21.15.44:50804) 2023-11-16T17:01:45.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.214+0000 7fca4c99b700 1 -- 172.21.15.44:0/3892380976 learned_addr learned my addr 172.21.15.44:0/3892380976 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:01:45.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.214+0000 7fca47fff700 1 --2- 172.21.15.44:0/3892380976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca48101660 0x7fca48198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:45.216 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.215+0000 7fca4d19c700 1 --2- 172.21.15.44:0/3892380976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca48105620 0x7fca4819c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.215+0000 7fca4c99b700 1 -- 172.21.15.44:0/3892380976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca48101660 msgr2=0x7fca48198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.215+0000 7fca4c99b700 1 --2- 172.21.15.44:0/3892380976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca48101660 0x7fca48198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.215+0000 7fca4c99b700 1 -- 172.21.15.44:0/3892380976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca48105620 msgr2=0x7fca4819c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.215+0000 7fca4c99b700 1 --2- 172.21.15.44:0/3892380976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca48105620 0x7fca4819c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.215+0000 7fca4c99b700 1 -- 172.21.15.44:0/3892380976 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c009520 con 0x7fca48100cc0 2023-11-16T17:01:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.215+0000 7fca4c99b700 1 --2- 172.21.15.44:0/3892380976 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca48100cc0 0x7fca48198720 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fca4000ba50 tx=0x7fca4000bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.216+0000 7fca45ffb700 1 -- 172.21.15.44:0/3892380976 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4000c9b0 con 0x7fca48100cc0 2023-11-16T17:01:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.216+0000 7fca45ffb700 1 -- 172.21.15.44:0/3892380976 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca40014440 con 0x7fca48100cc0 2023-11-16T17:01:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.216+0000 7fca45ffb700 1 -- 172.21.15.44:0/3892380976 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca400075d0 con 0x7fca48100cc0 2023-11-16T17:01:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.216+0000 7fca4ebff700 1 -- 172.21.15.44:0/3892380976 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca48110450 con 0x7fca48100cc0 2023-11-16T17:01:45.219 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.216+0000 7fca4ebff700 1 -- 172.21.15.44:0/3892380976 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca481102c0 con 0x7fca48100cc0 2023-11-16T17:01:45.219 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.218+0000 7fca4ebff700 1 -- 172.21.15.44:0/3892380976 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca481105e0 con 0x7fca48100cc0 2023-11-16T17:01:45.224 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.224+0000 7fca45ffb700 1 -- 172.21.15.44:0/3892380976 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fca400145b0 con 0x7fca48100cc0 2023-11-16T17:01:45.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:44 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/981968232' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:45.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:44 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:45.231 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.230+0000 7fca45ffb700 1 --2- 172.21.15.44:0/3892380976 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fca300a1410 0x7fca300a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:45.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.230+0000 7fca45ffb700 1 -- 172.21.15.44:0/3892380976 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9017+0+0 (secure 0 0 0) 0x7fca40006e40 con 0x7fca48100cc0 2023-11-16T17:01:45.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.232+0000 7fca47fff700 1 --2- 172.21.15.44:0/3892380976 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fca300a1410 0x7fca300a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:45.233 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.232+0000 7fca45ffb700 1 -- 172.21.15.44:0/3892380976 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fca40095870 con 0x7fca48100cc0 2023-11-16T17:01:45.234 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.233+0000 7fca47fff700 1 --2- 172.21.15.44:0/3892380976 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fca300a1410 0x7fca300a38c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fca48199d30 tx=0x7fca3800b410 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:45.637 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.636+0000 7fca4ebff700 1 -- 172.21.15.44:0/3892380976 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fca48067b00 con 0x7fca48100cc0 2023-11-16T17:01:45.970 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.969+0000 7fca45ffb700 1 -- 172.21.15.44:0/3892380976 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v105) v1 ==== 169+0+0 (secure 0 0 0) 0x7fca40098e90 con 0x7fca48100cc0 2023-11-16T17:01:45.974 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.974+0000 7fca4ebff700 1 -- 172.21.15.44:0/3892380976 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fca300a1410 msgr2=0x7fca300a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:45.974 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.974+0000 7fca4ebff700 1 --2- 172.21.15.44:0/3892380976 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fca300a1410 0x7fca300a38c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fca48199d30 tx=0x7fca3800b410 comp rx=0 tx=0).stop 2023-11-16T17:01:45.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.974+0000 7fca4ebff700 1 -- 172.21.15.44:0/3892380976 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca48100cc0 msgr2=0x7fca48198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:45.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.974+0000 7fca4ebff700 1 --2- 172.21.15.44:0/3892380976 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca48100cc0 0x7fca48198720 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fca4000ba50 tx=0x7fca4000bd60 comp rx=0 tx=0).stop 2023-11-16T17:01:45.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.974+0000 7fca4ebff700 1 -- 172.21.15.44:0/3892380976 shutdown_connections 2023-11-16T17:01:45.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.974+0000 7fca4ebff700 1 --2- 172.21.15.44:0/3892380976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca48105620 0x7fca4819c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:45.975 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.974+0000 7fca4ebff700 1 --2- 172.21.15.44:0/3892380976 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7fca300a1410 0x7fca300a38c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:45.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.974+0000 7fca4ebff700 1 --2- 172.21.15.44:0/3892380976 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca48100cc0 0x7fca48198720 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:45.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.974+0000 7fca4ebff700 1 --2- 172.21.15.44:0/3892380976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca48101660 0x7fca48198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:45.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.974+0000 7fca4ebff700 1 -- 172.21.15.44:0/3892380976 >> 172.21.15.44:0/3892380976 conn(0x7fca48077560 msgr2=0x7fca480fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:45.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.974+0000 7fca4ebff700 1 -- 172.21.15.44:0/3892380976 shutdown_connections 2023-11-16T17:01:45.976 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:45.975+0000 7fca4ebff700 1 -- 172.21.15.44:0/3892380976 wait complete. 2023-11-16T17:01:45.978 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' removed 2023-11-16T17:01:46.013 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-11-16T17:01:46.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:45 smithi044 ceph-mon[128750]: pgmap v368: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 307 B/s wr, 2 op/s 2023-11-16T17:01:46.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:45 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T17:01:46.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:45 smithi044 ceph-mon[128750]: osdmap e104: 12 total, 12 up, 12 in 2023-11-16T17:01:46.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:45 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T17:01:46.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:45 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T17:01:46.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:45 smithi044 ceph-mon[128750]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:01:46.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:45 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/3892380976' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:46.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:45 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:46.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:45 smithi176 ceph-mon[131630]: pgmap v368: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 307 B/s wr, 2 op/s 2023-11-16T17:01:46.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:45 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T17:01:46.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:45 smithi176 ceph-mon[131630]: osdmap e104: 12 total, 12 up, 12 in 2023-11-16T17:01:46.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:45 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T17:01:46.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:45 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T17:01:46.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:45 smithi176 ceph-mon[131630]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:01:46.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:45 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/3892380976' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:46.147 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:45 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:46.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:45 smithi077 ceph-mon[131204]: pgmap v368: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 6.1 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 307 B/s wr, 2 op/s 2023-11-16T17:01:46.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:45 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T17:01:46.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:45 smithi077 ceph-mon[131204]: osdmap e104: 12 total, 12 up, 12 in 2023-11-16T17:01:46.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:45 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T17:01:46.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:45 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T17:01:46.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:45 smithi077 ceph-mon[131204]: from='mgr.25447 172.21.15.44:0/173605006' entity='mgr.x' 2023-11-16T17:01:46.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:45 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/3892380976' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:46.229 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:45 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:46.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.385+0000 7f708c7ff700 1 -- 172.21.15.44:0/3324647449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70841063f0 msgr2=0x7f70841067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:46.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.385+0000 7f708c7ff700 1 --2- 172.21.15.44:0/3324647449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70841063f0 0x7f70841067c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f70800099c0 tx=0x7f7080009cd0 comp rx=0 tx=0).stop 2023-11-16T17:01:46.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.385+0000 7f708c7ff700 1 -- 172.21.15.44:0/3324647449 shutdown_connections 2023-11-16T17:01:46.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.385+0000 7f708c7ff700 1 --2- 172.21.15.44:0/3324647449 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7084101400 0x7f708410d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:46.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.385+0000 7f708c7ff700 1 --2- 172.21.15.44:0/3324647449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70841063f0 0x7f70841067c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:46.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.385+0000 7f708c7ff700 1 --2- 172.21.15.44:0/3324647449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7084100a50 0x7f7084100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:46.386 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.385+0000 7f708c7ff700 1 -- 172.21.15.44:0/3324647449 >> 172.21.15.44:0/3324647449 conn(0x7f7084077590 msgr2=0x7f7084077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:46.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.385+0000 7f708c7ff700 1 -- 172.21.15.44:0/3324647449 shutdown_connections 2023-11-16T17:01:46.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.386+0000 7f708c7ff700 1 -- 172.21.15.44:0/3324647449 wait complete. 2023-11-16T17:01:46.387 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.387+0000 7f708c7ff700 1 Processor -- start 2023-11-16T17:01:46.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.387+0000 7f708c7ff700 1 -- start start 2023-11-16T17:01:46.388 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.388+0000 7f708c7ff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7084100a50 0x7f708419e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:46.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.388+0000 7f708c7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7084101400 0x7f708419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:46.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.388+0000 7f708c7ff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70841063f0 0x7f70841984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:46.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.388+0000 7f708c7ff700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7084198b30 con 0x7f70841063f0 2023-11-16T17:01:46.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.388+0000 7f708c7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7084198ca0 con 0x7f7084101400 2023-11-16T17:01:46.389 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.388+0000 7f708c7ff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7084198e60 con 0x7f7084100a50 2023-11-16T17:01:46.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.388+0000 7f708a59b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7084100a50 0x7f708419e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:46.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.389+0000 7f708a59b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7084100a50 0x7f708419e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.44:50826/0 (socket says 172.21.15.44:50826) 2023-11-16T17:01:46.390 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.389+0000 7f708a59b700 1 -- 172.21.15.44:0/1759382752 learned_addr learned my addr 172.21.15.44:0/1759382752 (peer_addr_for_me v2:172.21.15.44:0/0) 2023-11-16T17:01:46.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.389+0000 7f708a59b700 1 -- 172.21.15.44:0/1759382752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7084101400 msgr2=0x7f708419e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:46.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.389+0000 7f708a59b700 1 --2- 172.21.15.44:0/1759382752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7084101400 0x7f708419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:46.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.389+0000 7f708a59b700 1 -- 172.21.15.44:0/1759382752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70841063f0 msgr2=0x7f70841984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:46.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.389+0000 7f708a59b700 1 --2- 172.21.15.44:0/1759382752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70841063f0 0x7f70841984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:46.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.389+0000 7f708a59b700 1 -- 172.21.15.44:0/1759382752 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70800095f0 con 0x7f7084100a50 2023-11-16T17:01:46.391 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.389+0000 7f708a59b700 1 --2- 172.21.15.44:0/1759382752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7084100a50 0x7f708419e400 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7080004730 tx=0x7f7080005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:46.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.390+0000 7f707b7fe700 1 -- 172.21.15.44:0/1759382752 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708001d070 con 0x7f7084100a50 2023-11-16T17:01:46.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.390+0000 7f707b7fe700 1 -- 172.21.15.44:0/1759382752 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7080004cd0 con 0x7f7084100a50 2023-11-16T17:01:46.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.390+0000 7f708c7ff700 1 -- 172.21.15.44:0/1759382752 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7084110450 con 0x7f7084100a50 2023-11-16T17:01:46.392 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.390+0000 7f707b7fe700 1 -- 172.21.15.44:0/1759382752 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7080005290 con 0x7f7084100a50 2023-11-16T17:01:46.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.390+0000 7f708c7ff700 1 -- 172.21.15.44:0/1759382752 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70841102c0 con 0x7f7084100a50 2023-11-16T17:01:46.393 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.392+0000 7f708c7ff700 1 -- 172.21.15.44:0/1759382752 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70841105e0 con 0x7f7084100a50 2023-11-16T17:01:46.399 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.398+0000 7f707b7fe700 1 -- 172.21.15.44:0/1759382752 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 25) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f7080003c50 con 0x7f7084100a50 2023-11-16T17:01:46.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.401+0000 7f707b7fe700 1 --2- 172.21.15.44:0/1759382752 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f70700a1490 0x7f70700a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:01:46.401 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.401+0000 7f707b7fe700 1 -- 172.21.15.44:0/1759382752 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 8458+0+0 (secure 0 0 0) 0x7f70800cda00 con 0x7f7084100a50 2023-11-16T17:01:46.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.401+0000 7f707b7fe700 1 -- 172.21.15.44:0/1759382752 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7080096e80 con 0x7f7084100a50 2023-11-16T17:01:46.402 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.402+0000 7f7089d9a700 1 --2- 172.21.15.44:0/1759382752 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f70700a1490 0x7f70700a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:01:46.404 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.403+0000 7f7089d9a700 1 --2- 172.21.15.44:0/1759382752 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f70700a1490 0x7f70700a3940 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7074005950 tx=0x7f70740058e0 comp rx=0 tx=0).ready entity=mgr.25447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:01:46.817 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:46.816+0000 7f708c7ff700 1 -- 172.21.15.44:0/1759382752 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7084067b00 con 0x7f7084100a50 2023-11-16T17:01:47.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:46 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T17:01:47.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:46 smithi044 ceph-mon[128750]: Service mds.all has no related daemon already placed. 2023-11-16T17:01:47.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:46 smithi044 ceph-mon[128750]: Service mgr has no related daemon already placed. 2023-11-16T17:01:47.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:46 smithi044 ceph-mon[128750]: Service mon has no related daemon already placed. 2023-11-16T17:01:47.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:46 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T17:01:47.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:46 smithi044 ceph-mon[128750]: osdmap e105: 12 total, 12 up, 12 in 2023-11-16T17:01:47.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:46 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T17:01:47.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:46 smithi176 ceph-mon[131630]: Service mds.all has no related daemon already placed. 2023-11-16T17:01:47.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:46 smithi176 ceph-mon[131630]: Service mgr has no related daemon already placed. 2023-11-16T17:01:47.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:46 smithi176 ceph-mon[131630]: Service mon has no related daemon already placed. 2023-11-16T17:01:47.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:46 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T17:01:47.146 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:46 smithi176 ceph-mon[131630]: osdmap e105: 12 total, 12 up, 12 in 2023-11-16T17:01:47.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:46 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T17:01:47.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:46 smithi077 ceph-mon[131204]: Service mds.all has no related daemon already placed. 2023-11-16T17:01:47.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:46 smithi077 ceph-mon[131204]: Service mgr has no related daemon already placed. 2023-11-16T17:01:47.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:46 smithi077 ceph-mon[131204]: Service mon has no related daemon already placed. 2023-11-16T17:01:47.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:46 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T17:01:47.228 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:46 smithi077 ceph-mon[131204]: osdmap e105: 12 total, 12 up, 12 in 2023-11-16T17:01:47.841 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:47.840+0000 7f707b7fe700 1 -- 172.21.15.44:0/1759382752 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v106) v1 ==== 178+0+0 (secure 0 0 0) 0x7f708009a4a0 con 0x7f7084100a50 2023-11-16T17:01:47.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:47.845+0000 7f708c7ff700 1 -- 172.21.15.44:0/1759382752 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f70700a1490 msgr2=0x7f70700a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:47.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:47.845+0000 7f708c7ff700 1 --2- 172.21.15.44:0/1759382752 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f70700a1490 0x7f70700a3940 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7074005950 tx=0x7f70740058e0 comp rx=0 tx=0).stop 2023-11-16T17:01:47.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:47.845+0000 7f708c7ff700 1 -- 172.21.15.44:0/1759382752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7084100a50 msgr2=0x7f708419e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:01:47.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:47.845+0000 7f708c7ff700 1 --2- 172.21.15.44:0/1759382752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7084100a50 0x7f708419e400 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7080004730 tx=0x7f7080005e80 comp rx=0 tx=0).stop 2023-11-16T17:01:47.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:47.845+0000 7f708c7ff700 1 -- 172.21.15.44:0/1759382752 shutdown_connections 2023-11-16T17:01:47.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:47.845+0000 7f708c7ff700 1 --2- 172.21.15.44:0/1759382752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70841063f0 0x7f70841984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:47.846 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:47.845+0000 7f708c7ff700 1 --2- 172.21.15.44:0/1759382752 >> [v2:172.21.15.44:6800/3382499657,v1:172.21.15.44:6801/3382499657] conn(0x7f70700a1490 0x7f70700a3940 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:47.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:47.845+0000 7f708c7ff700 1 --2- 172.21.15.44:0/1759382752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7084100a50 0x7f708419e400 secure :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7080004730 tx=0x7f7080005e80 comp rx=0 tx=0).stop 2023-11-16T17:01:47.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:47.845+0000 7f708c7ff700 1 --2- 172.21.15.44:0/1759382752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7084101400 0x7f708419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:01:47.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:47.845+0000 7f708c7ff700 1 -- 172.21.15.44:0/1759382752 >> 172.21.15.44:0/1759382752 conn(0x7f7084077590 msgr2=0x7f708410b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:01:47.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:47.845+0000 7f708c7ff700 1 -- 172.21.15.44:0/1759382752 shutdown_connections 2023-11-16T17:01:47.847 INFO:teuthology.orchestra.run.smithi044.stderr:2023-11-16T17:01:47.846+0000 7f708c7ff700 1 -- 172.21.15.44:0/1759382752 wait complete. 2023-11-16T17:01:47.851 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data_ec' removed 2023-11-16T17:01:47.883 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T17:01:47.993 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T17:01:48.040 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T17:01:48.068 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:47 smithi176 ceph-mon[131630]: pgmap v370: 97 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 1 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T17:01:48.068 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:47 smithi176 ceph-mon[131630]: from='client.? 172.21.15.44:0/1759382752' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:48.068 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:47 smithi176 ceph-mon[131630]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:48.073 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-16T17:01:48.073 DEBUG:teuthology.orchestra.run.smithi044:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-16T17:01:48.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:47 smithi044 ceph-mon[128750]: pgmap v370: 97 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 1 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T17:01:48.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:47 smithi044 ceph-mon[128750]: from='client.? 172.21.15.44:0/1759382752' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:48.118 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:47 smithi044 ceph-mon[128750]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:48.148 DEBUG:teuthology.orchestra.run.smithi077:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-16T17:01:48.164 DEBUG:teuthology.orchestra.run.smithi176:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-16T17:01:48.181 INFO:tasks.cephadm:Stopping all daemons... 2023-11-16T17:01:48.181 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-11-16T17:01:48.182 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mon.a 2023-11-16T17:01:48.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:47 smithi077 ceph-mon[131204]: pgmap v370: 97 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 1 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T17:01:48.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:47 smithi077 ceph-mon[131204]: from='client.? 172.21.15.44:0/1759382752' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:48.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:47 smithi077 ceph-mon[131204]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:01:48.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:48 smithi044 systemd[1]: Stopping Ceph mon.a for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:01:49.090 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:48 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mon-a[128726]: 2023-11-16T17:01:48.766+0000 7fd6d870a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:01:49.090 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:48 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mon-a[128726]: 2023-11-16T17:01:48.766+0000 7fd6d870a700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-11-16T17:01:49.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Nov 16 17:01:49 smithi044 bash[294036]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mon-a 2023-11-16T17:01:49.655 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mon.a.service' 2023-11-16T17:01:49.756 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:01:49.757 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-11-16T17:01:49.757 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-11-16T17:01:49.757 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mon.b 2023-11-16T17:01:50.226 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:49 smithi077 systemd[1]: Stopping Ceph mon.b for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:01:50.619 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mon-b[131180]: 2023-11-16T17:01:50.255+0000 7f9293fda700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:01:50.619 INFO:journalctl@ceph.mon.b.smithi077.stdout:Nov 16 17:01:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mon-b[131180]: 2023-11-16T17:01:50.255+0000 7f9293fda700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-11-16T17:01:50.819 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mon.b.service' 2023-11-16T17:01:50.863 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:01:50.863 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-11-16T17:01:50.863 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-11-16T17:01:50.863 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mon.c 2023-11-16T17:01:51.145 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:50 smithi176 systemd[1]: Stopping Ceph mon.c for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:01:51.515 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:51 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mon-c[131600]: 2023-11-16T17:01:51.222+0000 7f954d3ad700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:01:51.515 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:51 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mon-c[131600]: 2023-11-16T17:01:51.222+0000 7f954d3ad700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-11-16T17:01:51.766 INFO:journalctl@ceph.mon.c.smithi176.stdout:Nov 16 17:01:51 smithi176 bash[164385]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mon-c 2023-11-16T17:01:51.773 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mon.c.service' 2023-11-16T17:01:51.824 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:01:51.825 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-11-16T17:01:51.825 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-11-16T17:01:51.825 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mgr.x 2023-11-16T17:01:52.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 17:01:51 smithi044 systemd[1]: Stopping Ceph mgr.x for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:01:52.656 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 17:01:52 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T17:01:52.374+0000 7fc1c0d5a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:01:52.656 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 17:01:52 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x[129000]: 2023-11-16T17:01:52.374+0000 7fc1c0d5a700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-11-16T17:01:52.924 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Nov 16 17:01:52 smithi044 bash[294182]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-x 2023-11-16T17:01:53.256 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mgr.x.service' 2023-11-16T17:01:53.351 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:01:53.351 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-11-16T17:01:53.351 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-11-16T17:01:53.351 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mgr.y 2023-11-16T17:01:53.726 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 17:01:53 smithi077 systemd[1]: Stopping Ceph mgr.y for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:01:54.239 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 17:01:54 smithi077 bash[162857]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-y 2023-11-16T17:01:54.239 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 17:01:54 smithi077 systemd[1]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-11-16T17:01:54.243 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mgr.y.service' 2023-11-16T17:01:54.267 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 17:01:54 smithi077 systemd[1]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mgr.y.service: Failed with result 'exit-code'. 2023-11-16T17:01:54.267 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Nov 16 17:01:54 smithi077 systemd[1]: Stopped Ceph mgr.y for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T17:01:54.279 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:01:54.279 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-11-16T17:01:54.279 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-11-16T17:01:54.280 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mgr.z 2023-11-16T17:01:54.645 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 17:01:54 smithi176 systemd[1]: Stopping Ceph mgr.z for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:01:54.974 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 17:01:54 smithi176 bash[164524]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mgr-z 2023-11-16T17:01:54.974 INFO:journalctl@ceph.mgr.z.smithi176.stdout:Nov 16 17:01:54 smithi176 systemd[1]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-11-16T17:01:55.170 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mgr.z.service' 2023-11-16T17:01:55.209 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:01:55.209 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-11-16T17:01:55.210 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-11-16T17:01:55.210 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.0 2023-11-16T17:01:55.618 INFO:journalctl@ceph.osd.0.smithi044.stdout:Nov 16 17:01:55 smithi044 systemd[1]: Stopping Ceph osd.0 for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:01:56.118 INFO:journalctl@ceph.osd.0.smithi044.stdout:Nov 16 17:01:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-0[140968]: 2023-11-16T17:01:55.722+0000 7f9b5aae3700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:01:56.119 INFO:journalctl@ceph.osd.0.smithi044.stdout:Nov 16 17:01:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-0[140968]: 2023-11-16T17:01:55.722+0000 7f9b5aae3700 -1 osd.0 106 *** Got signal Terminated *** 2023-11-16T17:01:56.119 INFO:journalctl@ceph.osd.0.smithi044.stdout:Nov 16 17:01:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-0[140968]: 2023-11-16T17:01:55.722+0000 7f9b5aae3700 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:02:02.118 INFO:journalctl@ceph.osd.0.smithi044.stdout:Nov 16 17:02:01 smithi044 bash[294327]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-0 2023-11-16T17:02:05.101 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.0.service' 2023-11-16T17:02:05.198 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:05.198 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-11-16T17:02:05.199 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-11-16T17:02:05.199 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.1 2023-11-16T17:02:05.476 INFO:journalctl@ceph.osd.1.smithi077.stdout:Nov 16 17:02:05 smithi077 systemd[1]: Stopping Ceph osd.1 for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:05.976 INFO:journalctl@ceph.osd.1.smithi077.stdout:Nov 16 17:02:05 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-1[136286]: 2023-11-16T17:02:05.549+0000 7f1401b55700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:05.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Nov 16 17:02:05 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-1[136286]: 2023-11-16T17:02:05.549+0000 7f1401b55700 -1 osd.1 106 *** Got signal Terminated *** 2023-11-16T17:02:05.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Nov 16 17:02:05 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-1[136286]: 2023-11-16T17:02:05.549+0000 7f1401b55700 -1 osd.1 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:02:11.476 INFO:journalctl@ceph.osd.1.smithi077.stdout:Nov 16 17:02:11 smithi077 bash[163000]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-1 2023-11-16T17:02:12.543 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.1.service' 2023-11-16T17:02:12.584 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:12.584 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-11-16T17:02:12.585 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-11-16T17:02:12.585 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.2 2023-11-16T17:02:12.895 INFO:journalctl@ceph.osd.2.smithi176.stdout:Nov 16 17:02:12 smithi176 systemd[1]: Stopping Ceph osd.2 for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:13.394 INFO:journalctl@ceph.osd.2.smithi176.stdout:Nov 16 17:02:12 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-2[137428]: 2023-11-16T17:02:12.927+0000 7f829bee9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:13.395 INFO:journalctl@ceph.osd.2.smithi176.stdout:Nov 16 17:02:12 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-2[137428]: 2023-11-16T17:02:12.927+0000 7f829bee9700 -1 osd.2 106 *** Got signal Terminated *** 2023-11-16T17:02:13.395 INFO:journalctl@ceph.osd.2.smithi176.stdout:Nov 16 17:02:12 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-2[137428]: 2023-11-16T17:02:12.927+0000 7f829bee9700 -1 osd.2 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:02:19.145 INFO:journalctl@ceph.osd.2.smithi176.stdout:Nov 16 17:02:18 smithi176 bash[164672]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-2 2023-11-16T17:02:20.121 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.2.service' 2023-11-16T17:02:20.163 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:20.163 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-11-16T17:02:20.163 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-11-16T17:02:20.164 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.3 2023-11-16T17:02:20.618 INFO:journalctl@ceph.osd.3.smithi044.stdout:Nov 16 17:02:20 smithi044 systemd[1]: Stopping Ceph osd.3 for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:21.118 INFO:journalctl@ceph.osd.3.smithi044.stdout:Nov 16 17:02:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-3[146392]: 2023-11-16T17:02:20.727+0000 7fae6439d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:21.118 INFO:journalctl@ceph.osd.3.smithi044.stdout:Nov 16 17:02:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-3[146392]: 2023-11-16T17:02:20.727+0000 7fae6439d700 -1 osd.3 106 *** Got signal Terminated *** 2023-11-16T17:02:21.119 INFO:journalctl@ceph.osd.3.smithi044.stdout:Nov 16 17:02:20 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-3[146392]: 2023-11-16T17:02:20.727+0000 7fae6439d700 -1 osd.3 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:02:22.145 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:21 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:21.837+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:22.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:21 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:21.910+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:22.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:21 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:21.954+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:23.145 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:22 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:22.874+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:23.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:22 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:22.951+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:23.368 INFO:journalctl@ceph.osd.3.smithi044.stdout:Nov 16 17:02:22 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-3[146392]: 2023-11-16T17:02:22.997+0000 7fae60192700 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.438109+0000 front 2023-11-16T17:01:59.438227+0000 (oldest deadline 2023-11-16T17:02:22.938161+0000) 2023-11-16T17:02:23.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:22 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:22.945+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:23.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:23 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:23.581+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:24.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:23 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:23.959+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:24.368 INFO:journalctl@ceph.osd.3.smithi044.stdout:Nov 16 17:02:23 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-3[146392]: 2023-11-16T17:02:23.970+0000 7fae60192700 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.438109+0000 front 2023-11-16T17:01:59.438227+0000 (oldest deadline 2023-11-16T17:02:22.938161+0000) 2023-11-16T17:02:24.369 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:23 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:23.949+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:24.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:23 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:23.910+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:24.925 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:24 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:24.627+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:25.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:24 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:24.912+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:25.368 INFO:journalctl@ceph.osd.3.smithi044.stdout:Nov 16 17:02:25 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-3[146392]: 2023-11-16T17:02:25.014+0000 7fae60192700 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.438109+0000 front 2023-11-16T17:01:59.438227+0000 (oldest deadline 2023-11-16T17:02:22.938161+0000) 2023-11-16T17:02:25.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:24 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:24.923+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:25.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:24 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:24.893+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:25.947 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:25 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:25.651+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:26.145 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:25 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:25.850+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:26.145 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:26 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:26.025+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:26.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:25 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:25.919+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:26.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:25 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:25.945+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:26.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:26 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:26.615+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:26.868 INFO:journalctl@ceph.osd.3.smithi044.stdout:Nov 16 17:02:26 smithi044 bash[294607]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-3 2023-11-16T17:02:27.144 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:26 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:26.858+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:27.145 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:27 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:27.055+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:27.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:26 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:26.950+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:27.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:26 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:26.930+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:27.889 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:27 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:27.631+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:28.145 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:27 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:27.848+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:28.145 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:28 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:28.015+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:28.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:27 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:27.929+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:28.257 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:27 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:27.887+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:28.738 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.3.service' 2023-11-16T17:02:28.831 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:28.832 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-11-16T17:02:28.832 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-11-16T17:02:28.832 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.4 2023-11-16T17:02:29.118 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:28 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:28.863+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:29.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:28 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:28.652+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:29.145 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:28 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:28.869+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:29.145 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:29 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:29.030+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:29.200 INFO:journalctl@ceph.osd.4.smithi077.stdout:Nov 16 17:02:28 smithi077 systemd[1]: Stopping Ceph osd.4 for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:29.201 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:28 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:28.900+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:29.476 INFO:journalctl@ceph.osd.4.smithi077.stdout:Nov 16 17:02:29 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-4[141799]: 2023-11-16T17:02:29.199+0000 7f6cef1de700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:29.476 INFO:journalctl@ceph.osd.4.smithi077.stdout:Nov 16 17:02:29 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-4[141799]: 2023-11-16T17:02:29.199+0000 7f6cef1de700 -1 osd.4 106 *** Got signal Terminated *** 2023-11-16T17:02:29.477 INFO:journalctl@ceph.osd.4.smithi077.stdout:Nov 16 17:02:29 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-4[141799]: 2023-11-16T17:02:29.199+0000 7f6cef1de700 -1 osd.4 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:02:30.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:29 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:29.661+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:30.119 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:29 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:29.880+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:30.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:29 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:29.899+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:30.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:29 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:29.914+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:30.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:30 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:30.031+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:30.926 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:30 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:30.665+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:31.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:30 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:30.915+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:31.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:30 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:30.925+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:31.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:30 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:30.925+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:31.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:31 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:31.003+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:31.971 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:31 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:31.627+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:32.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:31 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:31.873+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:32.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:31 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:31.873+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:32.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:31 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:31.969+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:32.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:31 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:31.969+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:32.394 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:31 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:31.910+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:32.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:31 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:31.910+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.968889+0000 front 2023-11-16T17:02:09.968897+0000 (oldest deadline 2023-11-16T17:02:31.069036+0000) 2023-11-16T17:02:32.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:32 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:32.020+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:32.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:32 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:32.020+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:32.939 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:32 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:32.621+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:33.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:32 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:32.843+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:33.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:32 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:32.843+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:33.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:32 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:32.938+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:33.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:32 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:32.938+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:33.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:32 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:32.911+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:33.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:32 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:32.911+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.968889+0000 front 2023-11-16T17:02:09.968897+0000 (oldest deadline 2023-11-16T17:02:31.069036+0000) 2023-11-16T17:02:33.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:33 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:33.060+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:33.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:33 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:33.060+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:33.726 INFO:journalctl@ceph.osd.4.smithi077.stdout:Nov 16 17:02:33 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-4[141799]: 2023-11-16T17:02:33.296+0000 7f6ce97d0700 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.993363+0000 front 2023-11-16T17:02:08.993339+0000 (oldest deadline 2023-11-16T17:02:32.394293+0000) 2023-11-16T17:02:33.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:33 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:33.600+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:33.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:33 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:33.600+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:34.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:33 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:33.823+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:34.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:33 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:33.823+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:34.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:33 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:33.930+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:34.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:33 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:33.930+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:34.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:33 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:33.930+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:34.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:33 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:33.930+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.968889+0000 front 2023-11-16T17:02:09.968897+0000 (oldest deadline 2023-11-16T17:02:31.069036+0000) 2023-11-16T17:02:34.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:34 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:34.027+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:34.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:34 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:34.027+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:34.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:34 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:34.561+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:34.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:34 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:34.561+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:35.226 INFO:journalctl@ceph.osd.4.smithi077.stdout:Nov 16 17:02:35 smithi077 bash[163268]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-4 2023-11-16T17:02:35.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:34 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:34.871+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:35.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:34 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:34.871+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:35.369 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:34 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:34.960+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:35.369 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:34 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:34.960+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:35.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:34 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:34.927+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:35.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:34 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:34.927+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.968889+0000 front 2023-11-16T17:02:09.968897+0000 (oldest deadline 2023-11-16T17:02:31.069036+0000) 2023-11-16T17:02:35.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:35 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:35.009+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:35.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:35 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:35.071+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:35.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:35 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:35.071+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:35.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:35 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:35.396+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:35.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:35 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:35.593+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:35.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:35 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:35.593+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:36.151 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:35 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:35.919+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:36.152 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:35 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:35.919+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:36.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:36 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:36.003+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:36.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:36 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:36.003+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:36.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:35 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:35.983+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:36.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:35 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:35.955+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:36.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:35 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:35.955+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.968889+0000 front 2023-11-16T17:02:09.968897+0000 (oldest deadline 2023-11-16T17:02:31.069036+0000) 2023-11-16T17:02:36.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:36 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:36.052+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:36.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:36 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:36.052+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:36.468 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:36 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:36.373+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:36.476 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.4.service' 2023-11-16T17:02:36.516 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:36.516 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-11-16T17:02:36.516 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-11-16T17:02:36.516 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.5 2023-11-16T17:02:36.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:36 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:36.601+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:36.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:36 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:36.601+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:36.883 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:36 smithi176 systemd[1]: Stopping Ceph osd.5 for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:37.145 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:37 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:37.015+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:37.145 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:37 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:37.063+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:37.146 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:37 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:37.063+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:37.146 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:36 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:36.882+0000 7f20f7b3d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:37.146 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:36 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:36.882+0000 7f20f7b3d700 -1 osd.5 106 *** Got signal Terminated *** 2023-11-16T17:02:37.146 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:36 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:36.882+0000 7f20f7b3d700 -1 osd.5 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:02:37.146 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:37 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:37.002+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:37.146 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:37 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:37.002+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.968889+0000 front 2023-11-16T17:02:09.968897+0000 (oldest deadline 2023-11-16T17:02:31.069036+0000) 2023-11-16T17:02:37.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:36 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:36.920+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:37.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:36 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:36.920+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:37.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:37 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:37.024+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:37.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:37 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:37.024+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:37.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:37 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:37.342+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:37.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:37 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:37.594+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:37.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:37 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:37.594+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:38.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:37 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:37.970+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:38.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:37 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:37.970+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:38.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:38 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:38.007+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:38.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:38 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:38.007+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:38.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:38 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:38.003+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:38.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:38 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:38.046+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:38.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:38 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:38.046+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.968889+0000 front 2023-11-16T17:02:09.968897+0000 (oldest deadline 2023-11-16T17:02:31.069036+0000) 2023-11-16T17:02:38.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:38 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:38.042+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:38.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:38 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:38.042+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:38.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:38 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:38.391+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:38.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:38 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:38.601+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:38.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:38 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:38.601+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:39.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:39 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:39.018+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:39.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:39 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:39.018+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:39.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:39 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:39.032+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:39.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:39 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:39.022+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:39.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:39 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:39.022+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.968889+0000 front 2023-11-16T17:02:09.968897+0000 (oldest deadline 2023-11-16T17:02:31.069036+0000) 2023-11-16T17:02:39.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:39 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:39.007+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:39.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:39 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:39.007+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:39.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:39 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:39.007+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:39.399 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:38 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:38.975+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:39.400 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:38 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:38.975+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:39.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:39 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:39.398+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:39.978 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:39 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:39.628+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:39.978 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:39 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:39.628+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:40.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:39 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:39.977+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:40.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:39 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:39.977+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:40.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:39 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:39.977+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.164634+0000 front 2023-11-16T17:02:16.164603+0000 (oldest deadline 2023-11-16T17:02:39.664845+0000) 2023-11-16T17:02:40.370 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:40 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:40.024+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:40.370 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:40 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:40.024+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:40.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:40 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:40.006+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:40.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:40 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:40.006+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:40.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:39 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:39.981+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:40.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:39 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:39.981+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.968889+0000 front 2023-11-16T17:02:09.968897+0000 (oldest deadline 2023-11-16T17:02:31.069036+0000) 2023-11-16T17:02:40.396 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:39 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:39.981+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.371242+0000 front 2023-11-16T17:02:17.371224+0000 (oldest deadline 2023-11-16T17:02:39.071406+0000) 2023-11-16T17:02:40.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:40 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:40.030+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:40.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:40 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:40.030+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:40.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:40 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:40.030+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:40.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:40 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:40.368+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:41.006 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:40 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:40.657+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:41.007 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:40 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:40.657+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:41.321 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:40 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:40.975+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:41.321 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:40 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:40.975+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:41.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:41 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:41.005+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:41.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:41 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:41.005+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:41.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:41 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:41.005+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.164634+0000 front 2023-11-16T17:02:16.164603+0000 (oldest deadline 2023-11-16T17:02:39.664845+0000) 2023-11-16T17:02:41.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:41 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:41.022+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:41.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:41 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:41.022+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:41.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:41 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:41.018+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:57.067105+0000 front 2023-11-16T17:01:57.067143+0000 (oldest deadline 2023-11-16T17:02:21.767274+0000) 2023-11-16T17:02:41.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:41 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:41.018+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.968889+0000 front 2023-11-16T17:02:09.968897+0000 (oldest deadline 2023-11-16T17:02:31.069036+0000) 2023-11-16T17:02:41.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:41 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5[142915]: 2023-11-16T17:02:41.018+0000 7f20f012b700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.371242+0000 front 2023-11-16T17:02:17.371224+0000 (oldest deadline 2023-11-16T17:02:39.071406+0000) 2023-11-16T17:02:41.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:41 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:41.056+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:41.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:41 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:41.056+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:41.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:41 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:41.056+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:41.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:41 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:41.320+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:41.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:41 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:41.614+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:41.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:41 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:41.614+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:42.350 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:42 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:42.020+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:42.350 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:42 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:42.020+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:42.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:42 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:42.031+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:42.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:42 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:42.031+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:42.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:42 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:42.031+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.164634+0000 front 2023-11-16T17:02:16.164603+0000 (oldest deadline 2023-11-16T17:02:39.664845+0000) 2023-11-16T17:02:42.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:42 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:42.049+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:42.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:42 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:42.049+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:42.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:42 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:42.059+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:42.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:42 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:42.059+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:42.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:42 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:42.059+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:42.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:42 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:42.348+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:42.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:42 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:42.348+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:42.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:42 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:42.574+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:42.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:42 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:42.574+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:43.357 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:42 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:42.995+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:43.358 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:42 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:42.995+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:43.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:43 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:43.021+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:43.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:43 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:43.021+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:43.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:43 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:43.021+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.164634+0000 front 2023-11-16T17:02:16.164603+0000 (oldest deadline 2023-11-16T17:02:39.664845+0000) 2023-11-16T17:02:43.395 INFO:journalctl@ceph.osd.5.smithi176.stdout:Nov 16 17:02:43 smithi176 bash[164947]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-5 2023-11-16T17:02:43.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:43 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:43.051+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:43.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:43 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:43.051+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:43.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:43 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:43.024+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:43.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:43 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:43.024+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:43.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:43 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:43.024+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:43.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:43 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:43.356+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:43.727 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:43 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:43.356+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:43.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:43 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:43.598+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:43.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:43 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:43.598+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:44.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:43 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:43.955+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:44.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:43 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:43.955+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:44.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:43 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:43.955+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:02:44.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:44 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:44.012+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:44.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:44 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:44.012+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:44.369 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:44 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:44.012+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.164634+0000 front 2023-11-16T17:02:16.164603+0000 (oldest deadline 2023-11-16T17:02:39.664845+0000) 2023-11-16T17:02:44.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:44 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:44.040+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:44.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:44 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:44.040+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:44.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:43 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:43.997+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:44.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:43 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:43.997+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:44.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:43 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:43.997+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:44.418 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.5.service' 2023-11-16T17:02:44.458 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:44.459 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-11-16T17:02:44.459 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-11-16T17:02:44.459 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.6 2023-11-16T17:02:44.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:44 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:44.402+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:44.727 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:44 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:44.402+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:44.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:44 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:44.587+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:44.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:44 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:44.587+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:44.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:44 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:44.587+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:02:44.868 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:44 smithi044 systemd[1]: Stopping Ceph osd.6 for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:45.368 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:44 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:44.998+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:45.368 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:44 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:44.998+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:45.368 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:44 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:44.998+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:02:45.369 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:44 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:44.978+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:45.369 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:44 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:44.978+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:45.369 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:44 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:44.978+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.164634+0000 front 2023-11-16T17:02:16.164603+0000 (oldest deadline 2023-11-16T17:02:39.664845+0000) 2023-11-16T17:02:45.369 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:44 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:44.982+0000 7fbd9b083700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:45.369 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:44 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:44.982+0000 7fbd9b083700 -1 osd.6 106 *** Got signal Terminated *** 2023-11-16T17:02:45.369 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:44 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:44.982+0000 7fbd9b083700 -1 osd.6 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:02:45.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:45 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:45.026+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:45.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:45 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:45.026+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:45.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:44 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:44.952+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:45.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:44 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:44.952+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:45.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:44 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:44.952+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:45.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:45 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:45.366+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:45.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:45 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:45.366+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:45.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:45 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:45.611+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:45.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:45 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:45.611+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:45.868 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:45 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:45.611+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:02:46.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:45 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:45.977+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:46.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:45 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:45.977+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:46.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:45 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:45.977+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.164634+0000 front 2023-11-16T17:02:16.164603+0000 (oldest deadline 2023-11-16T17:02:39.664845+0000) 2023-11-16T17:02:46.394 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:45 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:45.977+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:46.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:45 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:45.977+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:46.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:45 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:45.955+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:46.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:45 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:45.955+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:46.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:45 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:45.955+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:46.397 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:46 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:46.019+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:46.398 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:46 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:46.019+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:46.398 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:46 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:46.019+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:02:46.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:46 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:46.395+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:46.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:46 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:46.395+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:46.996 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:46 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:46.657+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:46.997 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:46 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:46.657+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:46.997 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:46 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:46.657+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:02:47.365 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:46 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:46.988+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:47.365 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:46 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:46.988+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:47.365 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:46 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:46.988+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:02:47.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:46 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:46.995+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:47.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:46 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:46.995+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:47.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:46 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:46.995+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.164634+0000 front 2023-11-16T17:02:16.164603+0000 (oldest deadline 2023-11-16T17:02:39.664845+0000) 2023-11-16T17:02:47.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:46 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:46.996+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:47.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:46 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:46.996+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:47.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:46 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:46.995+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:47.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:46 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:46.995+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:47.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:46 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:46.995+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:47.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:47 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:47.362+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:47.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:47 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:47.362+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:48.028 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:47 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:47.655+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:48.029 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:47 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:47.655+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:48.029 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:47 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:47.655+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:02:48.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:47 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:47.966+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:48.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:47 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:47.966+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:48.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:47 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:47.966+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:02:48.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:48 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:48.027+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:48.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:48 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:48.027+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:48.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:48 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:48.027+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.164634+0000 front 2023-11-16T17:02:16.164603+0000 (oldest deadline 2023-11-16T17:02:39.664845+0000) 2023-11-16T17:02:48.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:47.950+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:48.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:47.950+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:48.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:47.965+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:48.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:47.965+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:48.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:47 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:47.965+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:48.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:48 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:48.413+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:48.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:48 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:48.413+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:49.046 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:48 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:48.664+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:49.046 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:48 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:48.664+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:49.046 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:48 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:48.664+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:02:49.047 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:48 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:48.664+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:02:49.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:49 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:49.045+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:55.760451+0000 front 2023-11-16T17:01:55.760376+0000 (oldest deadline 2023-11-16T17:02:21.660445+0000) 2023-11-16T17:02:49.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:49 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:49.045+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:06.862662+0000 front 2023-11-16T17:02:06.862348+0000 (oldest deadline 2023-11-16T17:02:31.562459+0000) 2023-11-16T17:02:49.369 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:49 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6[151956]: 2023-11-16T17:02:49.045+0000 7fbd97679700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.164634+0000 front 2023-11-16T17:02:16.164603+0000 (oldest deadline 2023-11-16T17:02:39.664845+0000) 2023-11-16T17:02:49.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:48 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:48.990+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:49.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:48 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:48.990+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:49.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:48 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:48.990+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:02:49.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:48 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:48.996+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:49.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:48 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:48.996+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:49.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:48 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:48.996+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:49.411 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:49.008+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:49.412 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:49.008+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:49.412 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:49.008+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:02:49.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:49.410+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:49.727 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:49.410+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:49.727 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:49.410+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:25.667400+0000 front 2023-11-16T17:02:25.667414+0000 (oldest deadline 2023-11-16T17:02:48.567268+0000) 2023-11-16T17:02:50.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:49 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:49.704+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:50.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:49 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:49.704+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:50.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:49 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:49.704+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:02:50.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:49 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:49.704+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:02:50.380 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:49.995+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:50.380 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:49.995+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:50.380 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:49 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:49.995+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:02:50.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:49 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:49.997+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:50.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:49 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:49.997+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:50.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:49 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:49.997+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:02:50.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:50 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:50.024+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:50.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:50 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:50.024+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:50.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:50 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:50.024+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:50.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:50.377+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:50.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:50.377+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:50.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:50.377+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:25.667400+0000 front 2023-11-16T17:02:25.667414+0000 (oldest deadline 2023-11-16T17:02:48.567268+0000) 2023-11-16T17:02:51.014 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:50 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:50.707+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:51.015 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:50 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:50.707+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:51.015 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:50 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:50.707+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:02:51.015 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:50 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:50.707+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:02:51.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:50.952+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:51.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:50.952+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:51.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:50.952+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:02:51.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:50.952+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:02:51.368 INFO:journalctl@ceph.osd.6.smithi044.stdout:Nov 16 17:02:51 smithi044 bash[294875]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-6 2023-11-16T17:02:51.394 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:50 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:50.997+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:51.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:50 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:50.997+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:51.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:50 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:50.997+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:02:51.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:51 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:51.025+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:51.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:51 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:51.025+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:51.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:51 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:51.025+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:51.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:51 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:51.421+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:51.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:51 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:51.421+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:51.727 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:51 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:51.421+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:25.667400+0000 front 2023-11-16T17:02:25.667414+0000 (oldest deadline 2023-11-16T17:02:48.567268+0000) 2023-11-16T17:02:52.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:51.705+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:52.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:51.705+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:52.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:51.705+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:02:52.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:51 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:51.705+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:02:52.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:51 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:51.922+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:52.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:51 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:51.922+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:52.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:51 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:51.922+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:02:52.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:51 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:51.922+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:02:52.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:52 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:52.018+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:52.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:52 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:52.018+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:52.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:52 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:52.018+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:02:52.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:52 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:52.004+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:52.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:52 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:52.005+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:52.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:52 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:52.005+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:52.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:52 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:52.459+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:52.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:52 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:52.459+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:52.727 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:52 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:52.459+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:25.667400+0000 front 2023-11-16T17:02:25.667414+0000 (oldest deadline 2023-11-16T17:02:48.567268+0000) 2023-11-16T17:02:53.108 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:52 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:52.722+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:53.109 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:52 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:52.722+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:53.109 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:52 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:52.722+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:02:53.109 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:52 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:52.722+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:02:53.133 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.6.service' 2023-11-16T17:02:53.224 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:53.225 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-11-16T17:02:53.225 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-11-16T17:02:53.225 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.7 2023-11-16T17:02:53.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:52 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:52.921+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:53.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:52 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:52.921+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:53.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:52 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:52.921+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:02:53.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:52 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:52.921+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:02:53.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:53 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:53.005+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:53.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:53 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:53.005+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:53.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:53 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:53.005+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:02:53.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:53 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:53.024+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:53.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:53 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:53.024+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:53.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:53 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:53.024+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:53.579 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:53 smithi077 systemd[1]: Stopping Ceph osd.7 for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:53.579 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:53 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:53.486+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:53.580 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:53 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:53.486+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:53.580 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:53 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:53.486+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:25.667400+0000 front 2023-11-16T17:02:25.667414+0000 (oldest deadline 2023-11-16T17:02:48.567268+0000) 2023-11-16T17:02:53.580 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:53 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:53.578+0000 7f14fd958700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:53.580 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:53 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:53.578+0000 7f14fd958700 -1 osd.7 106 *** Got signal Terminated *** 2023-11-16T17:02:53.878 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:53 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:53.578+0000 7f14fd958700 -1 osd.7 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:02:54.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:53 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:53.707+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:54.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:53 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:53.707+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:54.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:53 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:53.707+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:02:54.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:53 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:53.707+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:02:54.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:53 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:53.876+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:54.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:53 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:53.876+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:54.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:53 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:53.876+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:02:54.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:53 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:53.876+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:02:54.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:54 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:54.045+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:54.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:54 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:54.045+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:54.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:54 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:54.045+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:54.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:54 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:54.028+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:54.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:54 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:54.028+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:54.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:54 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:54.028+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:02:54.887 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:54 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:54.478+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:54.887 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:54 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:54.478+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:54.887 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:54 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:54.478+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:25.667400+0000 front 2023-11-16T17:02:25.667414+0000 (oldest deadline 2023-11-16T17:02:48.567268+0000) 2023-11-16T17:02:55.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:54 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:54.658+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:55.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:54 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:54.658+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:55.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:54 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:54.658+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:02:55.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:54 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:54.658+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:02:55.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:54 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:54.886+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:55.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:54 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:54.886+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:55.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:54 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:54.886+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:02:55.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:54 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:54.886+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:02:55.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:55 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:55.055+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:55.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:55 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:55.055+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:55.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:55 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:55.055+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:02:55.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:55 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:55.073+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:55.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:55 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:55.073+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:55.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:55 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:55.073+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:55.846 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:55 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:55.504+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:55.846 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:55 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:55.504+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:55.846 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:55 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:55.504+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:25.667400+0000 front 2023-11-16T17:02:25.667414+0000 (oldest deadline 2023-11-16T17:02:48.567268+0000) 2023-11-16T17:02:56.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:55.686+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:56.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:55.686+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:56.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:55.686+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:02:56.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:55 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:55.686+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:02:56.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:55 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:55.844+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:56.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:55 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:55.844+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:56.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:55 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:55.844+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:02:56.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:55 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:55.844+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:02:56.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:55 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:55.844+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:02:56.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:56 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:56.033+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:56.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:56 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:56.033+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:56.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:56 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:56.033+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:02:56.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:56 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:56.033+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.234279+0000 front 2023-11-16T17:02:33.234249+0000 (oldest deadline 2023-11-16T17:02:55.534386+0000) 2023-11-16T17:02:56.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:56 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:56.090+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:56.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:56 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:56.090+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:56.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:56 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:56.090+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:56.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:56 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:56.090+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:02:56.863 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:56 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:56.528+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:56.864 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:56 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:56.529+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:56.864 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:56 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:56.529+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:25.667400+0000 front 2023-11-16T17:02:25.667414+0000 (oldest deadline 2023-11-16T17:02:48.567268+0000) 2023-11-16T17:02:56.864 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:56 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:56.529+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.668353+0000 front 2023-11-16T17:02:32.668385+0000 (oldest deadline 2023-11-16T17:02:56.168405+0000) 2023-11-16T17:02:57.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:56 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:56.703+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:57.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:56 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:56.703+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:57.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:56 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:56.703+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:02:57.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:56 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:56.703+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:02:57.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:56 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:56.703+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:02:57.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:56 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:56.862+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:57.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:56 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:56.862+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:57.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:56 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:56.862+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:02:57.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:56 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:56.862+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:02:57.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:56 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:56.862+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:02:57.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:57 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:57.035+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:57.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:57 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:57.035+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:57.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:57 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:57.035+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:02:57.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:57 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:57.035+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.234279+0000 front 2023-11-16T17:02:33.234249+0000 (oldest deadline 2023-11-16T17:02:55.534386+0000) 2023-11-16T17:02:57.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:57 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:57.126+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:57.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:57 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:57.126+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:57.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:57 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:57.126+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:57.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:57 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:57.126+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:02:57.887 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:57 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:57.481+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:57.888 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:57 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:57.481+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:57.888 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:57 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:57.481+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:25.667400+0000 front 2023-11-16T17:02:25.667414+0000 (oldest deadline 2023-11-16T17:02:48.567268+0000) 2023-11-16T17:02:57.888 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:57 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:57.481+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.668353+0000 front 2023-11-16T17:02:32.668385+0000 (oldest deadline 2023-11-16T17:02:56.168405+0000) 2023-11-16T17:02:58.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:57 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:57.708+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:58.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:57 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:57.708+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:58.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:57 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:57.708+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:02:58.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:57 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:57.708+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:02:58.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:57 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:57.708+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:02:58.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:57 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:57.886+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:58.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:57 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:57.886+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:58.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:57 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:57.886+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:02:58.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:57 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:57.886+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:02:58.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:57 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:57.886+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:02:58.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:57 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:57.992+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:58.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:57 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:57.992+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:58.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:57 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:57.992+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:02:58.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:57 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:57.992+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.234279+0000 front 2023-11-16T17:02:33.234249+0000 (oldest deadline 2023-11-16T17:02:55.534386+0000) 2023-11-16T17:02:58.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:58 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:58.108+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:58.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:58 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:58.108+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:58.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:58 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:58.108+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:58.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:58 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:58.108+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:02:58.727 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:58 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:58.466+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.265060+0000 front 2023-11-16T17:02:09.265041+0000 (oldest deadline 2023-11-16T17:02:34.565219+0000) 2023-11-16T17:02:58.727 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:58 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:58.466+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:14.565826+0000 front 2023-11-16T17:02:14.565808+0000 (oldest deadline 2023-11-16T17:02:42.166389+0000) 2023-11-16T17:02:58.727 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:58 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:58.466+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:25.667400+0000 front 2023-11-16T17:02:25.667414+0000 (oldest deadline 2023-11-16T17:02:48.567268+0000) 2023-11-16T17:02:58.727 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:58 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7[147206]: 2023-11-16T17:02:58.466+0000 7f14f5f46700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.668353+0000 front 2023-11-16T17:02:32.668385+0000 (oldest deadline 2023-11-16T17:02:56.168405+0000) 2023-11-16T17:02:59.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:58 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:58.721+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:02:59.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:58 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:58.721+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:02:59.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:58 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:58.721+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:02:59.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:58 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:58.721+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:02:59.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:58 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:58.721+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:02:59.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:58 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:58.864+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:02:59.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:58 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:58.864+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:02:59.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:58 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:58.864+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:02:59.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:58 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:58.864+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:02:59.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:58 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:58.864+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:02:59.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:59 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:59.097+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:02:59.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:59 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:59.097+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:02:59.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:59 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:59.097+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:02:59.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:02:59 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:02:59.097+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:02:59.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:58 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:58.980+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:02:59.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:58 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:58.980+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:02:59.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:58 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:58.980+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:02:59.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:58 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:58.980+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.234279+0000 front 2023-11-16T17:02:33.234249+0000 (oldest deadline 2023-11-16T17:02:55.534386+0000) 2023-11-16T17:02:59.726 INFO:journalctl@ceph.osd.7.smithi077.stdout:Nov 16 17:02:59 smithi077 bash[163543]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-7 2023-11-16T17:03:00.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:59 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:59.756+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:03:00.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:59 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:59.756+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:03:00.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:59 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:59.756+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:03:00.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:59 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:59.756+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:03:00.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:02:59 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:02:59.756+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:03:00.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:59 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:59.863+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:00.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:59 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:59.863+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:00.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:59 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:59.863+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:00.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:59 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:59.863+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:00.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:02:59 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:02:59.863+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:00.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:59 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:59.941+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:03:00.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:59 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:59.941+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:03:00.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:59 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:59.941+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:03:00.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:02:59 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:02:59.941+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.234279+0000 front 2023-11-16T17:02:33.234249+0000 (oldest deadline 2023-11-16T17:02:55.534386+0000) 2023-11-16T17:03:00.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:00 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:00.132+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:00.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:00 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:00.132+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:00.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:00 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:00.132+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:00.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:00 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:00.132+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:00.622 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.7.service' 2023-11-16T17:03:00.662 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:00.663 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-11-16T17:03:00.663 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-11-16T17:03:00.663 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.8 2023-11-16T17:03:00.976 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:00 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:00.861+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:00.976 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:00 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:00.861+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:00.977 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:00 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:00.861+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:00.977 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:00 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:00.861+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:00.977 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:00 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:00.861+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:01.021 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:00 smithi176 systemd[1]: Stopping Ceph osd.8 for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:01.021 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:00 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:00.897+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:03:01.021 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:00 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:00.897+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:03:01.022 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:00 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:00.897+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:03:01.022 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:00 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:00.897+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.234279+0000 front 2023-11-16T17:02:33.234249+0000 (oldest deadline 2023-11-16T17:02:55.534386+0000) 2023-11-16T17:03:01.022 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:01 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:01.020+0000 7f8e0f401700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:01.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:00 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:00.717+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:03:01.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:00 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:00.717+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:03:01.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:00 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:00.717+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:03:01.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:00 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:00.717+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:03:01.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:00 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:00.717+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:03:01.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:01 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:01.020+0000 7f8e0f401700 -1 osd.8 106 *** Got signal Terminated *** 2023-11-16T17:03:01.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:01 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:01.020+0000 7f8e0f401700 -1 osd.8 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:03:01.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:01 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:01.113+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:01.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:01 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:01.113+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:01.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:01 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:01.113+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:01.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:01 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:01.113+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:02.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:01 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:01.689+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:03:02.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:01 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:01.689+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:03:02.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:01 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:01.689+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:03:02.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:01 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:01.689+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:03:02.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:01 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:01.689+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:03:02.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:01 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:01.818+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:02.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:01 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:01.818+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:02.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:01 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:01.818+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:02.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:01 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:01.818+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:02.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:01 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:01.818+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:02.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:02 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:02.115+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:02.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:02 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:02.115+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:02.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:02 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:02.115+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:02.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:02 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:02.115+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:02.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:01 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:01.945+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:03:02.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:01 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:01.945+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:03:02.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:01 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:01.945+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:03:02.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:01 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:01.945+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.234279+0000 front 2023-11-16T17:02:33.234249+0000 (oldest deadline 2023-11-16T17:02:55.534386+0000) 2023-11-16T17:03:03.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:02 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:02.716+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:03:03.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:02 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:02.716+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:03:03.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:02 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:02.716+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:03:03.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:02 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:02.716+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:03:03.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:02 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:02.716+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:03:03.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:02 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:02.786+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:03.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:02 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:02.786+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:03.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:02 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:02.786+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:03.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:02 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:02.786+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:03.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:02 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:02.786+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:03.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:03 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:03.079+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:03.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:03 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:03.079+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:03.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:03 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:03.079+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:03.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:03 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:03.079+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:03.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:02 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:02.952+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:03:03.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:02 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:02.952+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:03:03.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:02 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:02.952+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:03:03.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:02 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:02.952+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.234279+0000 front 2023-11-16T17:02:33.234249+0000 (oldest deadline 2023-11-16T17:02:55.534386+0000) 2023-11-16T17:03:04.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:03 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:03.719+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:03:04.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:03 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:03.719+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:03:04.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:03 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:03.719+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:03:04.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:03 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:03.719+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:03:04.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:03 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:03.719+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:03:04.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:03 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:03.719+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:40.194609+0000 front 2023-11-16T17:02:40.194578+0000 (oldest deadline 2023-11-16T17:03:03.694366+0000) 2023-11-16T17:03:04.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:03 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:03.833+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:04.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:03 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:03.833+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:04.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:03 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:03.833+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:04.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:03 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:03.833+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:04.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:03 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:03.833+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:04.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:03 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:03.943+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:03:04.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:03 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:03.943+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:03:04.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:03 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:03.943+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:03:04.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:03 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:03.943+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.234279+0000 front 2023-11-16T17:02:33.234249+0000 (oldest deadline 2023-11-16T17:02:55.534386+0000) 2023-11-16T17:03:04.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:04 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:04.108+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:04.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:04 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:04.108+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:04.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:04 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:04.108+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:04.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:04 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:04.108+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:04.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:04 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:04.108+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:05.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:04 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:04.727+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:03:05.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:04 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:04.727+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:03:05.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:04 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:04.727+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:03:05.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:04 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:04.727+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:03:05.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:04 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:04.727+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:03:05.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:04 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:04.727+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:40.194609+0000 front 2023-11-16T17:02:40.194578+0000 (oldest deadline 2023-11-16T17:03:03.694366+0000) 2023-11-16T17:03:05.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:04 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:04.835+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:05.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:04 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:04.835+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:05.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:04 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:04.835+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:05.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:04 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:04.835+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:05.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:04 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:04.835+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:05.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:04 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:04.835+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:05.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:05 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:05.113+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:05.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:05 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:05.113+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:05.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:05 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:05.113+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:05.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:05 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:05.113+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:05.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:05 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:05.113+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:05.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:04 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:04.900+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:03:05.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:04 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:04.900+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:03:05.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:04 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:04.900+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:03:05.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:04 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:04.900+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.234279+0000 front 2023-11-16T17:02:33.234249+0000 (oldest deadline 2023-11-16T17:02:55.534386+0000) 2023-11-16T17:03:05.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:04 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:04.900+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:38.935586+0000 front 2023-11-16T17:02:38.935629+0000 (oldest deadline 2023-11-16T17:03:04.235791+0000) 2023-11-16T17:03:06.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:05 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:05.743+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:03:06.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:05 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:05.743+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:03:06.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:05 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:05.743+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:03:06.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:05 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:05.743+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:03:06.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:05 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:05.743+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:03:06.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:05 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:05.743+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:40.194609+0000 front 2023-11-16T17:02:40.194578+0000 (oldest deadline 2023-11-16T17:03:03.694366+0000) 2023-11-16T17:03:06.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:05 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:05.854+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:06.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:05 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:05.854+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:06.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:05 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:05.854+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:06.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:05 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:05.854+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:06.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:05 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:05.854+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:06.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:05 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:05.854+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:06.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:05 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:05.928+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:09.529283+0000 front 2023-11-16T17:02:09.529516+0000 (oldest deadline 2023-11-16T17:02:34.829465+0000) 2023-11-16T17:03:06.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:05 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:05.928+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.030919+0000 front 2023-11-16T17:02:17.030949+0000 (oldest deadline 2023-11-16T17:02:39.331047+0000) 2023-11-16T17:03:06.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:05 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:05.928+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.532662+0000 front 2023-11-16T17:02:24.532625+0000 (oldest deadline 2023-11-16T17:02:48.632756+0000) 2023-11-16T17:03:06.395 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:05 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:05.928+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.234279+0000 front 2023-11-16T17:02:33.234249+0000 (oldest deadline 2023-11-16T17:02:55.534386+0000) 2023-11-16T17:03:06.396 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:05 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8[148453]: 2023-11-16T17:03:05.928+0000 7f8e0a9f5700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:38.935586+0000 front 2023-11-16T17:02:38.935629+0000 (oldest deadline 2023-11-16T17:03:04.235791+0000) 2023-11-16T17:03:06.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:06 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:06.136+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:06.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:06 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:06.136+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:06.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:06 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:06.136+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:06.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:06 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:06.136+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:06.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:06 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:06.136+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:06.895 INFO:journalctl@ceph.osd.8.smithi176.stdout:Nov 16 17:03:06 smithi176 bash[165218]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-8 2023-11-16T17:03:07.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:06 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:06.740+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:03:07.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:06 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:06.740+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:03:07.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:06 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:06.740+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:03:07.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:06 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:06.740+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:03:07.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:06 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:06.740+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:03:07.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:06 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:06.740+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:40.194609+0000 front 2023-11-16T17:02:40.194578+0000 (oldest deadline 2023-11-16T17:03:03.694366+0000) 2023-11-16T17:03:07.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:06 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:06.828+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:07.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:06 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:06.828+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:07.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:06 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:06.828+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:07.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:06 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:06.828+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:07.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:06 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:06.828+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:07.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:06 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:06.828+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:07.386 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:07 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:07.111+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:07.386 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:07 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:07.111+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:07.386 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:07 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:07.111+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:07.386 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:07 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:07.111+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:07.387 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:07 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:07.111+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:08.061 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.8.service' 2023-11-16T17:03:08.104 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:08.104 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-11-16T17:03:08.105 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-11-16T17:03:08.105 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.9 2023-11-16T17:03:08.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:07 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:07.725+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:03:08.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:07 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:07.725+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:03:08.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:07 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:07.725+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:03:08.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:07 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:07.725+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:03:08.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:07 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:07.725+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:03:08.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:07 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:07.725+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:40.194609+0000 front 2023-11-16T17:02:40.194578+0000 (oldest deadline 2023-11-16T17:03:03.694366+0000) 2023-11-16T17:03:08.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:07 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:07.839+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:08.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:07 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:07.839+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:08.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:07 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:07.839+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:08.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:07 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:07.839+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:08.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:07 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:07.839+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:08.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:07 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:07.839+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:08.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:08 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:08.122+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:08.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:08 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:08.122+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:08.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:08 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:08.122+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:08.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:08 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:08.122+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:08.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:08 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:08.122+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:08.618 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:08 smithi044 systemd[1]: Stopping Ceph osd.9 for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:09.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:08 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:08.630+0000 7fce876fd700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:09.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:08 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:08.630+0000 7fce876fd700 -1 osd.9 106 *** Got signal Terminated *** 2023-11-16T17:03:09.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:08 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:08.630+0000 7fce876fd700 -1 osd.9 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:03:09.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:08 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:08.710+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:03:09.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:08 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:08.710+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:03:09.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:08 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:08.710+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:03:09.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:08 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:08.710+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:03:09.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:08 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:08.710+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:03:09.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:08 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:08.710+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:40.194609+0000 front 2023-11-16T17:02:40.194578+0000 (oldest deadline 2023-11-16T17:03:03.694366+0000) 2023-11-16T17:03:09.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:08 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:08.889+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:09.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:08 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:08.889+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:09.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:08 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:08.889+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:09.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:08 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:08.889+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:09.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:08 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:08.889+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:09.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:08 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:08.889+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:09.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:09 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:09.083+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:09.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:09 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:09.083+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:09.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:09 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:09.084+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:09.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:09 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:09.084+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:09.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:09 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:09.084+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:10.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:09 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:09.701+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:03:10.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:09 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:09.701+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:03:10.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:09 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:09.701+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:03:10.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:09 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:09.701+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:03:10.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:09 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:09.701+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:03:10.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:09 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:09.701+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:40.194609+0000 front 2023-11-16T17:02:40.194578+0000 (oldest deadline 2023-11-16T17:03:03.694366+0000) 2023-11-16T17:03:10.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:09 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:09.855+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:10.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:09 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:09.855+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:10.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:09 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:09.855+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:10.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:09 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:09.855+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:10.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:09 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:09.855+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:10.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:09 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:09.855+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:10.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:10 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:10.129+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:10.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:10 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:10.129+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:10.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:10 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:10.129+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:10.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:10 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:10.129+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:10.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:10 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:10.129+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:11.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:10 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:10.675+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:03:11.260 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:10 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:10.675+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:03:11.261 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:10 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:10.675+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:03:11.261 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:10 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:10.675+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:03:11.261 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:10 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:10.675+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:03:11.261 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:10 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:10.675+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:40.194609+0000 front 2023-11-16T17:02:40.194578+0000 (oldest deadline 2023-11-16T17:03:03.694366+0000) 2023-11-16T17:03:11.262 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:10 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:10.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:11.262 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:10 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:10.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:11.262 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:10 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:10.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:11.262 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:10 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:10.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:11.262 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:10 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:10.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:11.263 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:10 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:10.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:11.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:11 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:11.130+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:11.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:11 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:11.130+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:11.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:11 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:11.130+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:11.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:11 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:11.130+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:11.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:11 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:11.130+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:12.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:11 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:11.637+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:03:12.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:11 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:11.637+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:03:12.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:11 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:11.637+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:03:12.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:11 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:11.637+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:03:12.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:11 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:11.637+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:03:12.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:11 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:11.637+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:40.194609+0000 front 2023-11-16T17:02:40.194578+0000 (oldest deadline 2023-11-16T17:03:03.694366+0000) 2023-11-16T17:03:12.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:11 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:11.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:12.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:11 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:11.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:12.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:11 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:11.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:12.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:11 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:11.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:12.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:11 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:11.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:12.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:11 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:11.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:12.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:12 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:12.123+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:12.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:12 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:12.123+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:12.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:12 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:12.123+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:12.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:12 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:12.123+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:12.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:12 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:12.123+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:13.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:12 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:12.619+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:03:13.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:12 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:12.619+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:03:13.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:12 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:12.619+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:03:13.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:12 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:12.619+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:03:13.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:12 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:12.619+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:03:13.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:12 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:12.619+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:40.194609+0000 front 2023-11-16T17:02:40.194578+0000 (oldest deadline 2023-11-16T17:03:03.694366+0000) 2023-11-16T17:03:13.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:12 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:12.865+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:13.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:12 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:12.865+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:13.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:12 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:12.865+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:13.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:12 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:12.865+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:13.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:12 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:12.865+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:13.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:12 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:12.865+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:13.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:13 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:13.112+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:13.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:13 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:13.112+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:13.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:13 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:13.112+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:13.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:13 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:13.112+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:13.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:13 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:13.112+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:13.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:13 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:13.112+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:14.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:13 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:13.629+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:59.887435+0000 front 2023-11-16T17:01:59.887457+0000 (oldest deadline 2023-11-16T17:02:23.387593+0000) 2023-11-16T17:03:14.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:13 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:13.629+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.088861+0000 front 2023-11-16T17:02:08.088939+0000 (oldest deadline 2023-11-16T17:02:33.388874+0000) 2023-11-16T17:03:14.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:13 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:13.629+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:16.290417+0000 front 2023-11-16T17:02:16.290252+0000 (oldest deadline 2023-11-16T17:02:43.890631+0000) 2023-11-16T17:03:14.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:13 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:13.629+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:23.891239+0000 front 2023-11-16T17:02:23.891263+0000 (oldest deadline 2023-11-16T17:02:48.591420+0000) 2023-11-16T17:03:14.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:13 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:13.629+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:33.193196+0000 front 2023-11-16T17:02:33.193086+0000 (oldest deadline 2023-11-16T17:02:56.693294+0000) 2023-11-16T17:03:14.119 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:13 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9[157477]: 2023-11-16T17:03:13.629+0000 7fce7fceb700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:40.194609+0000 front 2023-11-16T17:02:40.194578+0000 (oldest deadline 2023-11-16T17:03:03.694366+0000) 2023-11-16T17:03:14.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:13 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:13.895+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:14.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:13 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:13.895+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:14.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:13 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:13.895+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:14.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:13 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:13.895+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:14.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:13 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:13.895+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:14.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:13 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:13.895+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:14.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:14 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:14.122+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:14.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:14 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:14.122+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:14.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:14 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:14.122+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:14.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:14 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:14.122+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:14.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:14 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:14.122+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:14.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:14 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:14.122+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:15.118 INFO:journalctl@ceph.osd.9.smithi044.stdout:Nov 16 17:03:14 smithi044 bash[295157]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-9 2023-11-16T17:03:15.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:14 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:14.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:15.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:14 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:14.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:15.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:14 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:14.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:15.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:14 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:14.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:15.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:14 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:14.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:15.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:14 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:14.858+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:15.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:15 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:15.089+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:15.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:15 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:15.089+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:15.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:15 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:15.089+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:15.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:15 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:15.089+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:15.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:15 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:15.089+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:15.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:15 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:15.089+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:16.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:15 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:15.822+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:16.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:15 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:15.822+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:16.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:15 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:15.822+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:16.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:15 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:15.822+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:16.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:15 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:15.822+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:16.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:15 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:15.822+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:16.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:16 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:16.102+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:16.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:16 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:16.102+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:16.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:16 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:16.102+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:16.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:16 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:16.102+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:16.397 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:16 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:16.102+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:16.397 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:16 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:16.102+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:17.055 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.9.service' 2023-11-16T17:03:17.189 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:17.189 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-11-16T17:03:17.190 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-11-16T17:03:17.190 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.10 2023-11-16T17:03:17.219 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:16 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:16.807+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:17.219 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:16 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:16.807+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:17.220 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:16 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:16.807+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:17.220 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:16 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:16.807+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:17.220 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:16 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:16.807+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:17.220 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:16 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:16.807+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:17.220 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:16 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:16.807+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:49.593861+0000 front 2023-11-16T17:02:49.593904+0000 (oldest deadline 2023-11-16T17:03:16.594040+0000) 2023-11-16T17:03:17.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:17 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:17.076+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:17.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:17 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:17.076+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:17.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:17 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:17.076+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:17.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:17 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:17.076+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:17.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:17 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:17.076+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:17.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:17 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:17.076+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:17.476 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:17 smithi077 systemd[1]: Stopping Ceph osd.10 for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:17.829 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:17 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:17.541+0000 7f6fb3330700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:17.829 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:17 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:17.541+0000 7f6fb3330700 -1 osd.10 106 *** Got signal Terminated *** 2023-11-16T17:03:17.830 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:17 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:17.541+0000 7f6fb3330700 -1 osd.10 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:03:17.830 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:17 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:17.829+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:18.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:17 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:17.829+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:18.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:17 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:17.829+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:18.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:17 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:17.829+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:18.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:17 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:17.829+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:18.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:17 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:17.829+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:18.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:17 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:17.829+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:49.593861+0000 front 2023-11-16T17:02:49.593904+0000 (oldest deadline 2023-11-16T17:03:16.594040+0000) 2023-11-16T17:03:18.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:18 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:18.073+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:18.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:18 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:18.073+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:18.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:18 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:18.073+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:18.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:18 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:18.073+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:18.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:18 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:18.073+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:18.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:18 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:18.073+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:19.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:18 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:18.855+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:19.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:18 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:18.855+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:19.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:18 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:18.855+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:19.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:18 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:18.855+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:19.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:18 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:18.855+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:19.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:18 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:18.855+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:19.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:18 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:18.855+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:49.593861+0000 front 2023-11-16T17:02:49.593904+0000 (oldest deadline 2023-11-16T17:03:16.594040+0000) 2023-11-16T17:03:19.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:19 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:19.120+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:19.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:19 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:19.120+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:19.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:19 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:19.120+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:19.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:19 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:19.120+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:19.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:19 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:19.120+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:19.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:19 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:19.120+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:20.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:19 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:19.880+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:20.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:19 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:19.880+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:20.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:19 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:19.880+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:20.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:19 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:19.880+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:20.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:19 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:19.880+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:20.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:19 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:19.880+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:20.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:19 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:19.880+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:49.593861+0000 front 2023-11-16T17:02:49.593904+0000 (oldest deadline 2023-11-16T17:03:16.594040+0000) 2023-11-16T17:03:20.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:20 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:20.108+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:20.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:20 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:20.108+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:20.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:20 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:20.108+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:20.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:20 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:20.108+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:20.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:20 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:20.108+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:20.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:20 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:20.108+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:21.226 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:20 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:20.836+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:21.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:20 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:20.836+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:21.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:20 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:20.836+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:21.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:20 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:20.836+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:21.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:20 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:20.836+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:21.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:20 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:20.836+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:21.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:20 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:20.836+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:49.593861+0000 front 2023-11-16T17:02:49.593904+0000 (oldest deadline 2023-11-16T17:03:16.594040+0000) 2023-11-16T17:03:21.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:21 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:21.082+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:21.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:21 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:21.082+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:21.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:21 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:21.082+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:21.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:21 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:21.082+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:21.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:21 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:21.082+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:21.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:21 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:21.082+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:22.227 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:21 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:21.838+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:01:56.486160+0000 front 2023-11-16T17:01:56.486019+0000 (oldest deadline 2023-11-16T17:02:21.786060+0000) 2023-11-16T17:03:22.229 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:21 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:21.838+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:07.086798+0000 front 2023-11-16T17:02:07.086813+0000 (oldest deadline 2023-11-16T17:02:31.786970+0000) 2023-11-16T17:03:22.229 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:21 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:21.838+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:17.588765+0000 front 2023-11-16T17:02:17.588717+0000 (oldest deadline 2023-11-16T17:02:43.488541+0000) 2023-11-16T17:03:22.229 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:21 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:21.838+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2023-11-16T17:02:24.589593+0000 front 2023-11-16T17:02:24.589690+0000 (oldest deadline 2023-11-16T17:02:50.489589+0000) 2023-11-16T17:03:22.229 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:21 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:21.838+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:30.489970+0000 front 2023-11-16T17:02:30.489930+0000 (oldest deadline 2023-11-16T17:02:55.790048+0000) 2023-11-16T17:03:22.230 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:21 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:21.838+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:36.791523+0000 front 2023-11-16T17:02:36.791455+0000 (oldest deadline 2023-11-16T17:03:04.392051+0000) 2023-11-16T17:03:22.230 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:21 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10[152642]: 2023-11-16T17:03:21.838+0000 7f6fab91e700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:49.593861+0000 front 2023-11-16T17:02:49.593904+0000 (oldest deadline 2023-11-16T17:03:16.594040+0000) 2023-11-16T17:03:22.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:22 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:22.132+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:22.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:22 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:22.132+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:22.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:22 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:22.132+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:22.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:22 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:22.132+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:22.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:22 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:22.132+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:22.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:22 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:22.132+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:23.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:23 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:23.097+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:23.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:23 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:23.097+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:23.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:23 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:23.097+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:23.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:23 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:23.097+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:23.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:23 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:23.097+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:23.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:23 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:23.097+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:23.726 INFO:journalctl@ceph.osd.10.smithi077.stdout:Nov 16 17:03:23 smithi077 bash[163804]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-10 2023-11-16T17:03:24.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:24 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:24.073+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:24.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:24 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:24.073+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:24.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:24 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:24.073+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:24.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:24 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:24.073+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:24.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:24 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:24.073+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:24.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:24 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:24.073+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:24.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:24 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:24.073+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2023-11-16T17:02:57.623883+0000 front 2023-11-16T17:02:57.623842+0000 (oldest deadline 2023-11-16T17:03:23.523624+0000) 2023-11-16T17:03:24.687 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.10.service' 2023-11-16T17:03:24.767 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:24.767 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-11-16T17:03:24.768 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-11-16T17:03:24.768 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.11 2023-11-16T17:03:25.104 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:24 smithi176 systemd[1]: Stopping Ceph osd.11 for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:25.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:25 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:25.103+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:25.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:25 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:25.103+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:25.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:25 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:25.103+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:25.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:25 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:25.103+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:25.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:25 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:25.103+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:25.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:25 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:25.103+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:25.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:25 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:25.103+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2023-11-16T17:02:57.623883+0000 front 2023-11-16T17:02:57.623842+0000 (oldest deadline 2023-11-16T17:03:23.523624+0000) 2023-11-16T17:03:25.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:25 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:25.116+0000 7f188d635700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:25.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:25 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:25.116+0000 7f188d635700 -1 osd.11 106 *** Got signal Terminated *** 2023-11-16T17:03:25.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:25 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:25.116+0000 7f188d635700 -1 osd.11 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:03:26.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:26 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:26.130+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:26.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:26 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:26.130+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:26.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:26 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:26.130+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:26.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:26 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:26.130+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:26.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:26 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:26.130+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:26.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:26 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:26.130+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:26.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:26 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:26.130+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2023-11-16T17:02:57.623883+0000 front 2023-11-16T17:02:57.623842+0000 (oldest deadline 2023-11-16T17:03:23.523624+0000) 2023-11-16T17:03:27.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:27 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:27.134+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:27.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:27 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:27.134+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:27.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:27 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:27.134+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:27.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:27 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:27.134+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:27.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:27 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:27.134+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:27.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:27 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:27.134+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:27.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:27 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:27.134+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2023-11-16T17:02:57.623883+0000 front 2023-11-16T17:02:57.623842+0000 (oldest deadline 2023-11-16T17:03:23.523624+0000) 2023-11-16T17:03:28.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:28 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:28.118+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:28.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:28 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:28.118+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:28.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:28 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:28.118+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:28.395 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:28 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:28.118+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:28.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:28 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:28.118+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:28.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:28 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:28.118+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:28.396 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:28 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:28.118+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2023-11-16T17:02:57.623883+0000 front 2023-11-16T17:02:57.623842+0000 (oldest deadline 2023-11-16T17:03:23.523624+0000) 2023-11-16T17:03:29.645 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:29 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:29.160+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2023-11-16T17:02:00.714002+0000 front 2023-11-16T17:02:00.714105+0000 (oldest deadline 2023-11-16T17:02:26.014163+0000) 2023-11-16T17:03:29.645 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:29 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:29.160+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2023-11-16T17:02:08.315089+0000 front 2023-11-16T17:02:08.315055+0000 (oldest deadline 2023-11-16T17:02:31.215069+0000) 2023-11-16T17:03:29.645 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:29 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:29.160+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6804 osd.2 since back 2023-11-16T17:02:15.915928+0000 front 2023-11-16T17:02:15.915814+0000 (oldest deadline 2023-11-16T17:02:38.815954+0000) 2023-11-16T17:03:29.645 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:29 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:29.160+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2023-11-16T17:02:32.218590+0000 front 2023-11-16T17:02:32.218653+0000 (oldest deadline 2023-11-16T17:02:55.718542+0000) 2023-11-16T17:03:29.646 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:29 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:29.160+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6812 osd.5 since back 2023-11-16T17:02:39.020398+0000 front 2023-11-16T17:02:39.020413+0000 (oldest deadline 2023-11-16T17:03:03.120545+0000) 2023-11-16T17:03:29.646 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:29 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:29.160+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2023-11-16T17:02:48.322384+0000 front 2023-11-16T17:02:48.322320+0000 (oldest deadline 2023-11-16T17:03:13.022133+0000) 2023-11-16T17:03:29.646 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:29 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:29.160+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2023-11-16T17:02:57.623883+0000 front 2023-11-16T17:02:57.623842+0000 (oldest deadline 2023-11-16T17:03:23.523624+0000) 2023-11-16T17:03:29.646 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:29 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11[153976]: 2023-11-16T17:03:29.160+0000 7f1887c27700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.176:6820 osd.8 since back 2023-11-16T17:03:05.224852+0000 front 2023-11-16T17:03:05.224835+0000 (oldest deadline 2023-11-16T17:03:28.124988+0000) 2023-11-16T17:03:31.145 INFO:journalctl@ceph.osd.11.smithi176.stdout:Nov 16 17:03:30 smithi176 bash[165484]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-osd-11 2023-11-16T17:03:32.169 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@osd.11.service' 2023-11-16T17:03:32.210 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:32.211 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-11-16T17:03:32.211 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-11-16T17:03:32.211 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.a 2023-11-16T17:03:32.618 INFO:journalctl@ceph.mds.a.smithi044.stdout:Nov 16 17:03:32 smithi044 systemd[1]: Stopping Ceph mds.a for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:33.118 INFO:journalctl@ceph.mds.a.smithi044.stdout:Nov 16 17:03:32 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-a[162673]: 2023-11-16T17:03:32.726+0000 7f16b18da700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:33.119 INFO:journalctl@ceph.mds.a.smithi044.stdout:Nov 16 17:03:32 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-a[162673]: 2023-11-16T17:03:32.726+0000 7f16b18da700 -1 mds.a *** got signal Terminated *** 2023-11-16T17:03:34.618 INFO:journalctl@ceph.mds.a.smithi044.stdout:Nov 16 17:03:34 smithi044 bash[295425]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-a 2023-11-16T17:03:34.872 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.a.service' 2023-11-16T17:03:34.961 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:34.961 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-11-16T17:03:34.961 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-11-16T17:03:34.962 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.d 2023-11-16T17:03:35.118 INFO:journalctl@ceph.mds.d.smithi044.stdout:Nov 16 17:03:35 smithi044 systemd[1]: Stopping Ceph mds.d for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:35.868 INFO:journalctl@ceph.mds.d.smithi044.stdout:Nov 16 17:03:35 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-d[161662]: 2023-11-16T17:03:35.483+0000 7fc93ca17700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:35.868 INFO:journalctl@ceph.mds.d.smithi044.stdout:Nov 16 17:03:35 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-d[161662]: 2023-11-16T17:03:35.483+0000 7fc93ca17700 -1 mds.d *** got signal Terminated *** 2023-11-16T17:03:37.368 INFO:journalctl@ceph.mds.d.smithi044.stdout:Nov 16 17:03:37 smithi044 bash[295571]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-d 2023-11-16T17:03:37.577 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.d.service' 2023-11-16T17:03:37.662 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:37.662 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-11-16T17:03:37.662 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-11-16T17:03:37.663 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.g 2023-11-16T17:03:37.868 INFO:journalctl@ceph.mds.g.smithi044.stdout:Nov 16 17:03:37 smithi044 systemd[1]: Stopping Ceph mds.g for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:38.618 INFO:journalctl@ceph.mds.g.smithi044.stdout:Nov 16 17:03:38 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-g[162285]: 2023-11-16T17:03:38.158+0000 7f6db7fd1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:38.618 INFO:journalctl@ceph.mds.g.smithi044.stdout:Nov 16 17:03:38 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-g[162285]: 2023-11-16T17:03:38.158+0000 7f6db7fd1700 -1 mds.g *** got signal Terminated *** 2023-11-16T17:03:42.618 INFO:journalctl@ceph.mds.g.smithi044.stdout:Nov 16 17:03:42 smithi044 bash[295717]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-g 2023-11-16T17:03:42.841 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.g.service' 2023-11-16T17:03:42.924 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:42.925 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-11-16T17:03:42.925 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-11-16T17:03:42.925 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.j 2023-11-16T17:03:43.118 INFO:journalctl@ceph.mds.j.smithi044.stdout:Nov 16 17:03:43 smithi044 systemd[1]: Stopping Ceph mds.j for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:43.868 INFO:journalctl@ceph.mds.j.smithi044.stdout:Nov 16 17:03:43 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-j[161268]: 2023-11-16T17:03:43.424+0000 7f696fb00700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:43.868 INFO:journalctl@ceph.mds.j.smithi044.stdout:Nov 16 17:03:43 smithi044 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-j[161268]: 2023-11-16T17:03:43.424+0000 7f696fb00700 -1 mds.j *** got signal Terminated *** 2023-11-16T17:03:46.206 INFO:journalctl@ceph.mds.j.smithi044.stdout:Nov 16 17:03:45 smithi044 bash[295863]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-j 2023-11-16T17:03:46.394 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.j.service' 2023-11-16T17:03:46.457 INFO:journalctl@ceph.mds.j.smithi044.stdout:Nov 16 17:03:46 smithi044 systemd[1]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.j.service: Succeeded. 2023-11-16T17:03:46.457 INFO:journalctl@ceph.mds.j.smithi044.stdout:Nov 16 17:03:46 smithi044 systemd[1]: Stopped Ceph mds.j for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T17:03:46.478 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:46.479 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-11-16T17:03:46.479 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-11-16T17:03:46.479 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.b 2023-11-16T17:03:46.831 INFO:journalctl@ceph.mds.b.smithi077.stdout:Nov 16 17:03:46 smithi077 systemd[1]: Stopping Ceph mds.b for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:47.226 INFO:journalctl@ceph.mds.b.smithi077.stdout:Nov 16 17:03:46 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-b[156523]: 2023-11-16T17:03:46.829+0000 7f1d5a7c3700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:47.227 INFO:journalctl@ceph.mds.b.smithi077.stdout:Nov 16 17:03:46 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-b[156523]: 2023-11-16T17:03:46.829+0000 7f1d5a7c3700 -1 mds.b *** got signal Terminated *** 2023-11-16T17:03:50.206 INFO:journalctl@ceph.mds.b.smithi077.stdout:Nov 16 17:03:50 smithi077 bash[164073]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-b 2023-11-16T17:03:50.395 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.b.service' 2023-11-16T17:03:50.434 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:50.434 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-11-16T17:03:50.434 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-11-16T17:03:50.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.e 2023-11-16T17:03:50.726 INFO:journalctl@ceph.mds.e.smithi077.stdout:Nov 16 17:03:50 smithi077 systemd[1]: Stopping Ceph mds.e for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:51.226 INFO:journalctl@ceph.mds.e.smithi077.stdout:Nov 16 17:03:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-e[155846]: 2023-11-16T17:03:50.820+0000 7fbbbb8db700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:51.227 INFO:journalctl@ceph.mds.e.smithi077.stdout:Nov 16 17:03:50 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-e[155846]: 2023-11-16T17:03:50.820+0000 7fbbbb8db700 -1 mds.e *** got signal Terminated *** 2023-11-16T17:03:55.408 INFO:journalctl@ceph.mds.e.smithi077.stdout:Nov 16 17:03:55 smithi077 bash[164212]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-e 2023-11-16T17:03:55.414 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.e.service' 2023-11-16T17:03:55.448 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:55.449 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-11-16T17:03:55.449 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-11-16T17:03:55.449 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.h 2023-11-16T17:03:55.726 INFO:journalctl@ceph.mds.h.smithi077.stdout:Nov 16 17:03:55 smithi077 systemd[1]: Stopping Ceph mds.h for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:56.226 INFO:journalctl@ceph.mds.h.smithi077.stdout:Nov 16 17:03:55 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-h[156184]: 2023-11-16T17:03:55.820+0000 7effda059700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:56.227 INFO:journalctl@ceph.mds.h.smithi077.stdout:Nov 16 17:03:55 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-h[156184]: 2023-11-16T17:03:55.820+0000 7effda059700 -1 mds.h *** got signal Terminated *** 2023-11-16T17:03:57.976 INFO:journalctl@ceph.mds.h.smithi077.stdout:Nov 16 17:03:57 smithi077 bash[164357]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-h 2023-11-16T17:03:58.173 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.h.service' 2023-11-16T17:03:58.207 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:58.208 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-11-16T17:03:58.208 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-11-16T17:03:58.208 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.k 2023-11-16T17:03:58.476 INFO:journalctl@ceph.mds.k.smithi077.stdout:Nov 16 17:03:58 smithi077 systemd[1]: Stopping Ceph mds.k for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:58.976 INFO:journalctl@ceph.mds.k.smithi077.stdout:Nov 16 17:03:58 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-k[155507]: 2023-11-16T17:03:58.585+0000 7f1261504700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:58.976 INFO:journalctl@ceph.mds.k.smithi077.stdout:Nov 16 17:03:58 smithi077 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-k[155507]: 2023-11-16T17:03:58.585+0000 7f1261504700 -1 mds.k *** got signal Terminated *** 2023-11-16T17:04:03.054 INFO:journalctl@ceph.mds.k.smithi077.stdout:Nov 16 17:04:02 smithi077 bash[164495]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-k 2023-11-16T17:04:03.189 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.k.service' 2023-11-16T17:04:03.223 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:04:03.224 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-11-16T17:04:03.224 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-11-16T17:04:03.224 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.c 2023-11-16T17:04:03.578 INFO:journalctl@ceph.mds.c.smithi176.stdout:Nov 16 17:04:03 smithi176 systemd[1]: Stopping Ceph mds.c for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:04:03.894 INFO:journalctl@ceph.mds.c.smithi176.stdout:Nov 16 17:04:03 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-c[157652]: 2023-11-16T17:04:03.575+0000 7f12be5b9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:04:03.895 INFO:journalctl@ceph.mds.c.smithi176.stdout:Nov 16 17:04:03 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-c[157652]: 2023-11-16T17:04:03.575+0000 7f12be5b9700 -1 mds.c *** got signal Terminated *** 2023-11-16T17:04:07.836 INFO:journalctl@ceph.mds.c.smithi176.stdout:Nov 16 17:04:07 smithi176 bash[165752]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-c 2023-11-16T17:04:07.843 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.c.service' 2023-11-16T17:04:07.881 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:04:07.881 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-11-16T17:04:07.882 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-11-16T17:04:07.882 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.f 2023-11-16T17:04:08.145 INFO:journalctl@ceph.mds.f.smithi176.stdout:Nov 16 17:04:07 smithi176 systemd[1]: Stopping Ceph mds.f for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:04:08.644 INFO:journalctl@ceph.mds.f.smithi176.stdout:Nov 16 17:04:08 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-f[157313]: 2023-11-16T17:04:08.249+0000 7f799f581700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:04:08.645 INFO:journalctl@ceph.mds.f.smithi176.stdout:Nov 16 17:04:08 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-f[157313]: 2023-11-16T17:04:08.249+0000 7f799f581700 -1 mds.f *** got signal Terminated *** 2023-11-16T17:04:09.644 INFO:journalctl@ceph.mds.f.smithi176.stdout:Nov 16 17:04:09 smithi176 bash[165890]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-f 2023-11-16T17:04:09.760 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.f.service' 2023-11-16T17:04:09.793 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:04:09.793 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-11-16T17:04:09.793 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-11-16T17:04:09.793 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.i 2023-11-16T17:04:10.145 INFO:journalctl@ceph.mds.i.smithi176.stdout:Nov 16 17:04:09 smithi176 systemd[1]: Stopping Ceph mds.i for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:04:10.644 INFO:journalctl@ceph.mds.i.smithi176.stdout:Nov 16 17:04:10 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-i[157991]: 2023-11-16T17:04:10.166+0000 7fa4024b4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:04:10.645 INFO:journalctl@ceph.mds.i.smithi176.stdout:Nov 16 17:04:10 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-i[157991]: 2023-11-16T17:04:10.166+0000 7fa4024b4700 -1 mds.i *** got signal Terminated *** 2023-11-16T17:04:14.335 INFO:journalctl@ceph.mds.i.smithi176.stdout:Nov 16 17:04:14 smithi176 bash[166035]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-i 2023-11-16T17:04:14.340 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.i.service' 2023-11-16T17:04:14.375 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:04:14.376 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-11-16T17:04:14.376 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-11-16T17:04:14.377 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.l 2023-11-16T17:04:14.645 INFO:journalctl@ceph.mds.l.smithi176.stdout:Nov 16 17:04:14 smithi176 systemd[1]: Stopping Ceph mds.l for 7f2b12a8-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:04:15.145 INFO:journalctl@ceph.mds.l.smithi176.stdout:Nov 16 17:04:14 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-l[156882]: 2023-11-16T17:04:14.739+0000 7f6532303700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:04:15.145 INFO:journalctl@ceph.mds.l.smithi176.stdout:Nov 16 17:04:14 smithi176 ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-l[156882]: 2023-11-16T17:04:14.739+0000 7f6532303700 -1 mds.l *** got signal Terminated *** 2023-11-16T17:04:16.813 INFO:journalctl@ceph.mds.l.smithi176.stdout:Nov 16 17:04:16 smithi176 bash[166181]: ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7-mds-l 2023-11-16T17:04:17.008 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7f2b12a8-849e-11ee-8dbe-212e2dc638e7@mds.l.service' 2023-11-16T17:04:17.043 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:04:17.043 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-11-16T17:04:17.044 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 --force --keep-logs 2023-11-16T17:04:21.513 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 --force --keep-logs 2023-11-16T17:04:23.491 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 --force --keep-logs 2023-11-16T17:04:25.449 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T17:04:25.531 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T17:04:25.566 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T17:04:25.597 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-16T17:04:25.600 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920/remote/smithi044/crash 2023-11-16T17:04:25.601 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-16T17:04:25.703 INFO:teuthology.orchestra.run.smithi044.stderr:tar: /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-16T17:04:25.703 INFO:teuthology.orchestra.run.smithi044.stderr:tar: Error is not recoverable: exiting now 2023-11-16T17:04:25.706 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920/remote/smithi077/crash 2023-11-16T17:04:25.708 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-16T17:04:25.765 INFO:teuthology.orchestra.run.smithi077.stderr:tar: /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-16T17:04:25.765 INFO:teuthology.orchestra.run.smithi077.stderr:tar: Error is not recoverable: exiting now 2023-11-16T17:04:25.767 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920/remote/smithi176/crash 2023-11-16T17:04:25.769 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-16T17:04:25.815 INFO:teuthology.orchestra.run.smithi176.stderr:tar: /var/lib/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-16T17:04:25.815 INFO:teuthology.orchestra.run.smithi176.stderr:tar: Error is not recoverable: exiting now 2023-11-16T17:04:25.816 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-16T17:04:25.816 DEBUG:teuthology.orchestra.run.smithi044:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-11-16T17:04:25.898 INFO:teuthology.orchestra.run.smithi044.stderr:grep: /var/log/ceph/7f2b12a8-849e-11ee-8dbe-212e2dc638e7/ceph.log: No such file or directory 2023-11-16T17:04:25.903 INFO:tasks.cephadm:Compressing logs... 2023-11-16T17:04:25.903 DEBUG:teuthology.orchestra.run.smithi044:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T17:04:25.908 DEBUG:teuthology.orchestra.run.smithi077:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T17:04:25.911 DEBUG:teuthology.orchestra.run.smithi176:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T17:04:25.937 INFO:teuthology.orchestra.run.smithi176.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-16T17:04:25.962 INFO:teuthology.orchestra.run.smithi077.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-16T17:04:25.991 INFO:teuthology.orchestra.run.smithi044.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-16T17:09:50.760 INFO:tasks.cephadm:Archiving logs... 2023-11-16T17:09:50.760 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/log/ceph to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920/remote/smithi044/log 2023-11-16T17:09:50.762 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-16T17:10:09.675 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920/remote/smithi077/log 2023-11-16T17:10:09.677 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-16T17:10:47.719 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920/remote/smithi176/log 2023-11-16T17:10:47.720 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-16T17:11:08.963 INFO:tasks.cephadm:Removing cluster... 2023-11-16T17:11:08.963 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 --force 2023-11-16T17:11:10.620 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 --force 2023-11-16T17:11:11.783 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7f2b12a8-849e-11ee-8dbe-212e2dc638e7 --force 2023-11-16T17:11:12.822 INFO:tasks.cephadm:Removing cephadm ... 2023-11-16T17:11:12.822 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-16T17:11:12.872 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-16T17:11:12.911 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-16T17:11:12.928 INFO:tasks.cephadm:Teardown complete 2023-11-16T17:11:12.929 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-16T17:11:12.942 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-16T17:11:12.942 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-16T17:11:12.947 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-16T17:11:12.954 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-16T17:11:13.093 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-16T17:11:13.103 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-16T17:11:13.103 DEBUG:teuthology.orchestra.run.smithi044:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T17:11:13.108 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T17:11:13.111 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T17:11:13.126 INFO:teuthology.orchestra.run.smithi176.stderr:bash: ntpq: command not found 2023-11-16T17:11:13.127 INFO:teuthology.orchestra.run.smithi077.stderr:bash: ntpq: command not found 2023-11-16T17:11:13.131 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T17:11:13.131 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2023-11-16T17:11:13.131 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +1335us[+1373us] +/- 76ms 2023-11-16T17:11:13.131 INFO:teuthology.orchestra.run.smithi176.stdout:^* hv02.front.sepia.ceph.com 3 6 377 53 -457us[ -419us] +/- 77ms 2023-11-16T17:11:13.131 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 -733us[ -695us] +/- 74ms 2023-11-16T17:11:13.132 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T17:11:13.147 INFO:teuthology.orchestra.run.smithi044.stderr:bash: ntpq: command not found 2023-11-16T17:11:13.159 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T17:11:13.159 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2023-11-16T17:11:13.159 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 53 +1297us[+1297us] +/- 76ms 2023-11-16T17:11:13.159 INFO:teuthology.orchestra.run.smithi077.stdout:^* hv02.front.sepia.ceph.com 3 6 377 56 -507us[ -507us] +/- 77ms 2023-11-16T17:11:13.159 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 -752us[ -752us] +/- 74ms 2023-11-16T17:11:13.159 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T17:11:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T17:11:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2023-11-16T17:11:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 55 +1302us[+1302us] +/- 76ms 2023-11-16T17:11:13.191 INFO:teuthology.orchestra.run.smithi044.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -496us[ -480us] +/- 77ms 2023-11-16T17:11:13.191 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 -726us[ -726us] +/- 74ms 2023-11-16T17:11:13.191 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T17:11:13.192 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-16T17:11:13.202 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-16T17:11:13.202 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-16T17:11:13.212 DEBUG:teuthology.orchestra.run.smithi044:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-16T17:11:13.517 DEBUG:teuthology.orchestra.run.smithi077:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-16T17:11:13.637 DEBUG:teuthology.orchestra.run.smithi176:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-16T17:11:13.739 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T17:11:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T17:11:13.817 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T17:11:13.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T17:11:13.851 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T17:11:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T17:11:13.885 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-16T17:11:13.896 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-16T17:11:13.911 INFO:teuthology.task.internal:Duration was 3040.663027 seconds 2023-11-16T17:11:13.911 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-16T17:11:13.921 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-16T17:11:13.921 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-16T17:11:13.926 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-16T17:11:13.928 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-16T17:11:13.975 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T17:11:14.006 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T17:11:14.095 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T17:11:14.408 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-16T17:11:14.409 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi044.front.sepia.ceph.com 2023-11-16T17:11:14.409 DEBUG:teuthology.orchestra.run.smithi044:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-16T17:11:14.512 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2023-11-16T17:11:14.512 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-16T17:11:14.551 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2023-11-16T17:11:14.551 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-16T17:11:14.574 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-16T17:11:14.574 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T17:11:14.578 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T17:11:14.594 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T17:11:14.713 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-16T17:11:14.713 DEBUG:teuthology.orchestra.run.smithi044:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-16T17:11:14.718 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-16T17:11:14.721 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-16T17:11:14.896 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-16T17:11:14.908 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-16T17:11:14.908 DEBUG:teuthology.orchestra.run.smithi044:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-16T17:11:14.990 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-16T17:11:15.019 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-16T17:11:15.046 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-16T17:11:15.058 DEBUG:teuthology.orchestra.run.smithi044:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-16T17:11:15.063 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-16T17:11:15.066 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-16T17:11:15.091 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2023-11-16T17:11:15.112 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2023-11-16T17:11:15.148 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = core 2023-11-16T17:11:15.186 DEBUG:teuthology.orchestra.run.smithi044:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-16T17:11:15.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T17:11:15.228 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-16T17:11:15.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T17:11:15.244 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-16T17:11:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T17:11:15.260 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-16T17:11:15.272 INFO:teuthology.task.internal:Transferring archived files... 2023-11-16T17:11:15.273 DEBUG:teuthology.misc:Transferring archived files from smithi044:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920/remote/smithi044 2023-11-16T17:11:15.273 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-16T17:11:15.360 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920/remote/smithi077 2023-11-16T17:11:15.360 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-16T17:11:15.400 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920/remote/smithi176 2023-11-16T17:11:15.400 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-16T17:11:15.442 INFO:teuthology.task.internal:Removing archive directory... 2023-11-16T17:11:15.442 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-16T17:11:15.447 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-16T17:11:15.449 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-16T17:11:15.490 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-16T17:11:15.503 INFO:teuthology.task.internal:Not uploading archives. 2023-11-16T17:11:15.503 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-16T17:11:15.515 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-16T17:11:15.515 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-16T17:11:15.532 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-16T17:11:15.534 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-16T17:11:15.548 INFO:teuthology.orchestra.run.smithi077.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 16 17:11 /home/ubuntu/cephtest 2023-11-16T17:11:15.550 INFO:teuthology.orchestra.run.smithi176.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 16 17:11 /home/ubuntu/cephtest 2023-11-16T17:11:15.571 INFO:teuthology.orchestra.run.smithi044.stdout: 524293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 16 17:11 /home/ubuntu/cephtest 2023-11-16T17:11:15.575 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-16T17:11:15.587 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-16T17:11:15.612 INFO:teuthology.nuke:Checking targets against current locks 2023-11-16T17:11:15.640 DEBUG:teuthology.nuke:shortname: smithi044 2023-11-16T17:11:15.641 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T17:11:15.665 DEBUG:teuthology.nuke:shortname: smithi077 2023-11-16T17:11:15.665 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T17:11:15.672 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 16:10:26.568556', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T17:11:15.691 DEBUG:teuthology.nuke:shortname: smithi176 2023-11-16T17:11:15.691 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T17:11:15.698 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 16:10:26.570346', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T17:11:15.710 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460920', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 16:10:26.571771', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T17:11:15.730 INFO:teuthology.orchestra.console.smithi044:Power off 2023-11-16T17:11:15.730 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-16T17:11:15.745 INFO:teuthology.orchestra.console.smithi077:Power off 2023-11-16T17:11:15.745 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-16T17:11:15.754 DEBUG:teuthology.orchestra.console.smithi044:power off output: Chassis Power Control: Down/Off 2023-11-16T17:11:15.754 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:11:15.765 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2023-11-16T17:11:15.766 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:11:15.781 INFO:teuthology.orchestra.console.smithi176:Power off 2023-11-16T17:11:15.782 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-16T17:11:15.788 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2023-11-16T17:11:15.789 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2023-11-16T17:11:15.802 DEBUG:teuthology.orchestra.console.smithi176:power off output: Chassis Power Control: Down/Off 2023-11-16T17:11:15.802 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:11:15.822 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2023-11-16T17:11:19.790 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:11:19.798 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:11:19.823 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:11:19.900 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2023-11-16T17:11:19.906 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2023-11-16T17:11:19.933 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2023-11-16T17:11:23.905 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:11:23.913 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:11:23.934 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:11:24.015 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is off 2023-11-16T17:11:24.023 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2023-11-16T17:11:24.043 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2023-11-16T17:11:24.116 INFO:teuthology.orchestra.console.smithi044:Power off completed 2023-11-16T17:11:28.028 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:11:28.044 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:11:28.138 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2023-11-16T17:11:28.152 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2023-11-16T17:11:28.239 INFO:teuthology.orchestra.console.smithi077:Power off completed 2023-11-16T17:11:28.253 INFO:teuthology.orchestra.console.smithi176:Power off completed 2023-11-16T17:11:28.359 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 5b73517e0e2fa132ea79290e5fc06ea5349b2288 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} duration: 3040.6630272865295 failure_reason: 'Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi044 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh''' flavor: default owner: scheduled_choffman@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7a0a097208514f73992dbf62c029bd9c status: fail success: false 2023-11-16T17:11:28.360 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-16T17:11:28.452 INFO:teuthology.run:FAIL